PowerScale OneFS

Upgrade Planning and Process Guide

9.0.0.0 - 9.8.0.0

April 2024
Notes, cautions, and warnings

NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid
the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

© 2013 - 2024 Dell Inc. or its subsidiaries. All rights reserved. Dell Technologies, Dell, and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners.
 Contents

Chapter 1: Introduction to this guide............................................................................................. 6

About this guide................................................................................................................................................................... 6
Where to get help................................................................................................................................................................6
Additional options for getting help............................................................................................................................ 6

Chapter 2: Planning an upgrade.....................................................................................................7

Upgrade planning overview ..............................................................................................................................................7
Planning an upgrade - summary................................................................................................................................. 7
Prerequisites for completing an upgrade....................................................................................................................... 7
Review required documentation...................................................................................................................................... 8
Types of OneFS upgrades................................................................................................................................................. 8
Parallel upgrades............................................................................................................................................................ 8
Rolling upgrades............................................................................................................................................................. 8
Simultaneous upgrades................................................................................................................................................ 9
Verify the upgrade path..................................................................................................................................................... 9
Upgrade Paths for supported versions of OneFS................................................................................................ 10
Check supportability and compatibility requirements............................................................................................... 10
Assess upgrade impact..................................................................................................................................................... 10
Consider upgrade limitations...................................................................................................................................... 11
Review feature changes and known issues............................................................................................................ 11
Plan an upgrade schedule................................................................................................................................................. 11

Chapter 3: Completing pre-upgrade tasks....................................................................................12

Pre-upgrade process - Overview................................................................................................................................... 12
Pre-upgrade - Summary............................................................................................................................................. 12
Collect cluster information...............................................................................................................................................13
Review and save information about cluster status .............................................................................................13
Gathering cluster logs................................................................................................................................................. 13
Check cluster readiness....................................................................................................................................................14
Check hardware health - web administration interface......................................................................................14
Check hardware health - CLI.................................................................................................................................... 14
Check the available free space................................................................................................................................. 14
Resolve outstanding events and errors.................................................................................................................. 16
Verify configurations and settings................................................................................................................................. 16
Upgrading in a Cloudpools environment................................................................................................................. 16
Upgrading in a FIPS or STIG environment............................................................................................................. 16
Preserve the Kerberos keytab file............................................................................................................................16
Install a supported version of DataIQ or InsightIQ............................................................................................... 17
Download the OneFS installation bundle file............................................................................................................... 17
Upgrade compatibility check utility................................................................................................................................18
Run the upgrade compatibility check utility...........................................................................................................18
Upgrade compatibility check summary................................................................................................................... 18
Reconfigure unsupported SMB settings................................................................................................................ 19
On-Cluster Analysis tool................................................................................................................................................... 19

Contents 3
 HealthCheck tool...............................................................................................................................................................20
Backup data........................................................................................................................................................................20
SyncIQ backup............................................................................................................................................................. 20
NDMP backup.............................................................................................................................................................. 20
Back up custom settings........................................................................................................................................... 20
Complete or stop jobs in progress................................................................................................................................. 21
Complete system jobs................................................................................................................................................. 21
Update Node Firmware Package...................................................................................................................................22
Update Drive Support Package..................................................................................................................................... 22
Configure IPMI ports........................................................................................................................................................ 22
Disable SupportAssist or Secure Remote Services (SRS)...................................................................................... 22

Chapter 4: Performing the OneFS upgrade.................................................................................. 23

Upgrade process overview............................................................................................................................................. 23
Upgrade - Summary....................................................................................................................................................23
Completing a parallel upgrade of OneFS..................................................................................................................... 23
Performing a parallel upgrade using the web administration interface.......................................................... 24
Performing a parallel upgrade using the CLI......................................................................................................... 24
Client connections during parallel upgrades......................................................................................................... 25
Completing a rolling upgrade of OneFS....................................................................................................................... 26
Perform a rolling upgrade using the web administration interface................................................................. 26
Perform a rolling upgrade using the CLI................................................................................................................ 27
Configuration changes during rolling upgrades....................................................................................................28
Client connections during rolling upgrades........................................................................................................... 28
Completing a simultaneous upgrade of OneFS.......................................................................................................... 28
Perform a simultaneous upgrade using the web administration interface.................................................... 28
Perform a simultaneous upgrade using the CLI................................................................................................... 29
Adding a node to the cluster while an upgrade is in progress................................................................................29
Committing an upgrade of OneFS................................................................................................................................ 29
Commit an upgrade using the web interface....................................................................................................... 30
Commit an upgrade using the CLI........................................................................................................................... 30
Rolling back an upgrade of OneFS................................................................................................................................30
Roll back an upgrade using the web interface..................................................................................................... 30
Roll back an upgrade using the CLI.........................................................................................................................30
Verify the OneFS installation using the CLI.................................................................................................................31

Chapter 5: Completing post-upgrade tasks................................................................................. 32

Post-upgrade process - Overview................................................................................................................................ 32
Post-upgrade - Summary.......................................................................................................................................... 32
Allow the Upgrade job to run..........................................................................................................................................32
Verify operational status using the CLI........................................................................................................................32
Re-establish user privileges............................................................................................................................................ 34
Check client connections, and test the workflow.....................................................................................................34
Verify Kerberos migration................................................................................................................................................34
Verify Kerberos migration using the web administration interface................................................................. 34
Verify Kerberos migration using the CLI................................................................................................................35
Reapply custom settings................................................................................................................................................. 35
Test custom scripts.......................................................................................................................................................... 35
Install the latest roll-up patch........................................................................................................................................ 35

4 Contents
Chapter 6: Troubleshooting your upgrade....................................................................................36
Troubleshooting an upgrade........................................................................................................................................... 36

Contents 5
 1
Introduction to this guide
Topics:
• About this guide
• Where to get help

About this guide

This guide provides important information and steps that you must review and follow when upgrading your OneFS 9.0.x through
9.8.x clusters.

NOTE: For upgrades from OneFS 9.0.0.0 and earlier, contact your Dell Technologies account team.

Where to get help

The Dell Technologies Support site contains important information about products and services including drivers, installation
packages, product documentation, knowledge base articles, and advisories.
A valid support contract and account might be required to access all the available information about a specific Dell Technologies
product or service.

Additional options for getting help

This section contains resources for getting answers to questions about PowerScale products.

Dell Technologies Support ● Contact Technical Support

Telephone support ● United States: 1-800-SVC-4EMC (1-800-782-4362)
● Canada: 1-800-543-4782
● Worldwide: 1-312-725-5401
● Local phone numbers for a specific country or region are available at Contact
Technical Support .
PowerScale OneFS Documentation ● PowerScale OneFS Info Hubs
Info Hubs

6 Introduction to this guide

 2
Planning an upgrade
Topics:
• Upgrade planning overview
• Prerequisites for completing an upgrade
• Review required documentation
• Types of OneFS upgrades
• Verify the upgrade path
• Check supportability and compatibility requirements
• Assess upgrade impact
• Plan an upgrade schedule

Upgrade planning overview

This chapter provides information to help you create an upgrade plan. Your upgrade plan helps you estimate the time that it
takes to complete tasks and determine who must complete them.
Create an upgrade plan by evaluating how the upgrade affects users and workflow, assessing the impact of the new version of
OneFS on your cluster, and analyzing upgrade risks.
NOTE: To provide you with the best user experience, Dell Technologies recommends using Dell Remote Proactive Services
for upgrade assistance. The services team is available to perform OneFS pre-upgrade health checks and can complete the
OneFS upgrade remotely for most customers. Contact your Account Team for more information.

Planning an upgrade - summary

The following is a summary of steps to perform during the upgrade planning phase:
1. Review required documentation.
2. Determine the type of upgrade.
3. Verify the upgrade path.
4. Check supportability and compatibility.
5. Assess upgrade impact.
6. Schedule the upgrade.
7. (Optional) Upgrade a test cluster.

Prerequisites for completing an upgrade

Permissions and interfaces
To complete the tasks described in this guide, you must be able to log in to the cluster as the root user through the following
interfaces:
● OneFS command-line interface
● OneFS web administration interface
NOTE: If you are not logged into the OneFS cluster with root privileges, you might not be able to run all of the commands in
this guide.

Planning an upgrade 7
Review required documentation
Reviewing the documentation in this list helps you to understand the upgrade process and the impact the upgrade could have on
your workflow. Some links require a support login.

Required documentation
See the PowerScale OneFS InfoHub for the version of OneFS that corresponds with your deployment.
● PowerScale OneFS Supportability and Compatibility Guide
Confirm that your PowerScale software and PowerScale hardware is compatible with the version of OneFS to which you are
upgrading. Check on releases, release dates, and version status.
● PowerScale OneFS Release Notes
Read the OneFS release notes for each version between your current version and your target version for information about
new features, changes, resolved issues, and known issues.
● PowerScale OneFS Current Patches
Review patches that have been released for the version of OneFS to which you are upgrading.

Other documentation
● OneFS Technical and Security Advisories
Determine whether any PowerScale Technical Advisories or Security Advisories have been issued for the version of OneFS
to which you are upgrading.
● For more information about tips and tricks from the services team, see KB article 200890

Types of OneFS upgrades

Depending on the version of OneFS installed on your cluster, there are different types of upgrades available: parallel upgrades,
rolling upgrades, or simultaneous upgrades.
NOTE: See the Verify the upgrade path topic in this guide for information about which types of upgrades are supported
between OneFS versions.

Parallel upgrades
Dell Technologies recommends using the parallel upgrade option when upgrading any size cluster running OneFS 8.2.2 and later.
Parallel upgrades require a smaller maintenance window than rolling upgrades, and do not require the interruption of service like
simultaneous upgrades.
A parallel upgrade installs the new operating system on a subset of nodes and restarts that subset of nodes at the same time.
Each subset of nodes attempts to make a reservation for their turn to upgrade until all nodes are upgraded. Node subsets and
reservations are based on diskpool and node availability.
During a parallel upgrade, node subsets that are not being upgraded remain online and can continue serving clients. However,
clients that are connected to a restarting node are disconnected and reconnected. How the client connection behaves when
a node is restarted depends on several factors including client type, client configuration (mount type, timeout settings), IP
allocation method, and how the client connected to the cluster. In OneFS 9.2.0.0 and later, client connection behavior is
managed by the disruption manager settings.

Rolling upgrades
A rolling upgrade installs the new operating system and restarts each node individually in the OneFS cluster so that only one
node is offline at a time. A rolling upgrade takes longer to complete than a simultaneous upgrade. During a rolling upgrade, nodes

8 Planning an upgrade
that are not currently being upgraded remain online and can continue serving clients. However, clients that are connected to
a restarting node are disconnected and reconnected. How the client connection behaves when a node is restarted depends
on several factors including client type, client configuration (mount type, timeout settings), IP allocation method, and how the
client connected to the cluster.

Simultaneous upgrades
A simultaneous upgrade installs the new operating system and restarts all nodes in the OneFS cluster at the same time.
Simultaneous upgrades are faster than rolling upgrades but require a temporary interruption of service during the upgrade
process. All client connections to the cluster must be terminated before completing the upgrade and data is inaccessible until
the installation of the new OneFS operating system is complete and the cluster is back online.

Verify the upgrade path

Verify that the current version of OneFS on your cluster can be upgraded to the target version.
All upgrades follow one of these scenarios:
● Upgrade to the patched target OneFS version
● Upgrade to the target OneFS version, and then add patches
● Upgrade to an intermediate OneFS version, upgrade to the target OneFS version, and then add patches
To view which version of OneFS is running on all nodes in your cluster, run the following command:

isi_for_array uname -r

Find your version of OneFS in the supported OneFS Upgrade Paths matrix and confirm which versions of OneFS are supported
upgrade paths.

Planning an upgrade 9
Upgrade Paths for supported versions of OneFS
The following table can be used to determine the supported upgrade path from the current version of OneFS on your cluster to
a target version of OneFS. The upgrade paths table displays the supported upgrade paths from major OneFS version to major
OneFS version, but does not include patch information.
Key:
● "O" : Rolling and Simultaneous upgrades available
● "=" : Parallel, Rolling, and Simultaneous upgrades available
● "x" : Unsupported upgrade path
● "–": For upgrades from OneFS 9.0.0 and earlier, contact your Dell Technologies account team.

Table 1. OneFS Upgrade Paths

Upgrade Upgrade to target version
from
current 9.0.0 9.1.0 9.2.0 9.2.1 9.3.0 9.4.0 9.5.0 9.6.0 9.7.0 9.8.0
version
9.0.0 – = = = = = = x x x
9.1.0 – = = = = = x x x
9.2.0 – = = = = x = =
9.2.1 – = = = x = =
9.3.0 – = = x = =
9.4.0 – = x = =
9.5.0 – x = =
9.6.0 – = =
9.7.0 – =
9.8.0 –

NOTE: These supported upgrade paths do not guarantee bug fix and feature parity. For more information about using
patches to achieve bug fix and feature parity, contact your account team.

NOTE: If you are upgrading from an earlier version of OneFS than is listed in the upgrade matrix above, contact your
account team for assistance.

Check supportability and compatibility requirements

Review the PowerScale Supportability and Compatibility Guide on the Dell Support site to confirm that your current
hardware components, software components, and protocol versions are compatible with the version of OneFS to which you are
upgrading.
For information about OneFS compatibility with Hadoop, see the Hadoop - Info Hub page.

Assess upgrade impact

Consider all the areas of your environment that might be affected by upgrading to a new version and plan a strategy for
performing the tasks that must be completed and addressing issues that might occur. Assessing the impact of the upgrade on
your environment, clients, and performance helps ensure that the upgrade does not disrupt your cluster, workflow, or users.
NOTE: If you have a performance-sensitive workload, PowerScale recommends that you consult your Sales Engineer for
assistance during the pre-upgrade planning process.

10 Planning an upgrade
Consider upgrade limitations
If the upgrade cannot be completed for any reason—for example, if there is insufficient space on the cluster or if the upgrade
process detects a stalled drive—the system will revert to the current version and the upgrade will be cancelled. Preparing your
cluster as recommended in this guide will help you to avoid situations that might result in a cancelled upgrade.
NOTE: In OneFS 8.2.0 and newer, you have the ability to pause and resume the upgrade process in order to resolve
blocking issues.

Review feature changes and known issues

Familiarize yourself with new, modified, and deprecated features as well as known issues in the target version of OneFS to
which you are upgrading.
Functionality changes and new features in the target version might impact the performance, configuration, or functionality of
your cluster after upgrading. If the target version contains an issue that affects your environment, you should consider waiting
to upgrade until after the issue is resolved in a later version of OneFS.
See the release notes for the target version of OneFS for a summary of new features, feature changes, and known issues. It
is recommended to also review the release notes for each OneFS release between your current OneFS version and the target
version.
NOTE: The new features and known issues for all supported OneFS versions have been compiled into this guide in the
appendix.

Plan an upgrade schedule

An upgrade schedule can help ensure that the upgrade goes smoothly. The schedule should consider all the factors that go into
an upgrade and estimate how long each stage of the upgrade process might take.
The upgrade process begins with ensuring that your OneFS cluster is ready to be upgraded. It is recommended that you set
aside two weeks to check the cluster health and resolve software and hardware issues before performing the upgrade itself.

Upgrade The maintenance window should encompass the pre-upgrade, upgrade, and post-upgrade phases.
maintenance Estimate the time that it takes to run the upgrade considering cluster size and upgrade type (parallel,
window rolling, or simultaneous). Schedule time to inform users when the upgrade will take place and that client
connections might be slow, file access might be affected, and clients might be disconnected. A best
practice is to upgrade the cluster during an off-hours maintenance window.
Schedule time for node and drive health checks and replacement of bad hardware. Include time to update
configurations and settings that are not supported in the new version.
Estimate the time that it takes to back up your data, considering cluster size, number of files, types of
files, and file size. Also include time to collect information about the cluster such as status, logs, and
settings.
If performing a parallel or rolling upgrade, consider whether you will configure client connection drain
times, which will extend the required maintenance window, but lower the impact on client connections.
Build in time to let the upgrade jobs run to completion and to reestablish permissions and connections.
Schedule time or extend the maintenance window to accommodate post-upgrade tasks such as
reconfiguring custom settings, updating scripts to reflect command and functionality changes in the
upgrade version, and potential troubleshooting.

(Optional) If available, upgrading a test cluster with the same current version of OneFS before you upgrade your
Upgrade a test production cluster can expose issues that could slow down or prevent the upgrade of your production
cluster system.
After you upgrade a test cluster, verify that the cluster is operational and validate key workflows on the
test cluster by simulating how administrators, users, and applications interact with the system.

Planning an upgrade 11
 3
Completing pre-upgrade tasks
Topics:
• Pre-upgrade process - Overview
• Collect cluster information
• Check cluster readiness
• Verify configurations and settings
• Download the OneFS installation bundle file
• Upgrade compatibility check utility
• On-Cluster Analysis tool
• HealthCheck tool
• Backup data
• Complete or stop jobs in progress
• Update Node Firmware Package
• Update Drive Support Package
• Configure IPMI ports
• Disable SupportAssist or Secure Remote Services (SRS)

Pre-upgrade process - Overview

Performing the pre-upgrade tasks described in this guide helps to ensure that the OneFS cluster hardware, software modules,
configuration, features, and file system do not have preexisting issues that could adversely affect the upgrade process.
Performing the pre-upgrade tasks also ensures that important cluster data is collected and any custom configuration settings
are recorded.

Pre-upgrade - Summary
The following is a summary of steps to perform during the pre-upgrade phase:
1. Collect cluster information.
a. Collect cluster status.
b. Gather cluster logs.
c. Check cluster hardware health.
d. Check cluster available space.
2. Resolve events and errors.
3. Verify cluster configuration.
a. Preserve the Kerberos keytab file.
b. Install DataIQ or InsightIQ.
4. Download the OneFS installation file.
5. Run the upgrade compatibility check utility.
a. Reconfigure unsupported SMB settings.
6. Run the On-Cluster Analysis tool.
7. Run the HealthCheck tool.
8. Back-up cluster data.
9. Complete system jobs.
10. Update Node Firmware Package.
11. Update Drive Support Package.
12. Configure IPMI ports.
13. Disable SupportAssist.

12 Completing pre-upgrade tasks

Collect cluster information
Before you upgrade, collect and record key information about your OneFS cluster and how it is configured.

Review and save information about cluster status

Before you upgrade, run the isi status command to get the status of your cluster, events, and jobs then save the
information to a file.
Run the isi status > /ifs/data/isi_status_output command to save the output of the isi status command
to a file named isi_status_output in the /ifs/data directory.
NOTE: Run the isi status command to view the status of the cluster, events, and jobs without saving the output to a
file.

Gathering cluster logs

You can gather cluster logs and send the logs to Dell Technologies Support for analysis. Cluster logs can be sent automatically or
manually through the cluster command-line and web administration interfaces.
NOTE: Your cluster must be connected to the internet to be able to send log files directly. In newer versions of OneFS,
you must also have remote support and SRS enabled. If your cluster does not have an internet connection or if your upload
has failed, you can copy the log file from the cluster and upload the log file with an FTP client to an FTP server. For more
information about how to gather and send cluster log files, including the command parameters for configuring how logs are
uploaded to Dell Technologies Support, refer to the administration guide for your version of OneFS.

Gather cluster logs - Web UI

You must have root access to run log gathering commands.
1. To gather the log files, go to Cluster Management > Diagnostics.
2. Click Start Gather to open the Start Gather window.
3. Configure the log gather and click Start Gather.
For information about configuring the log gather, see the administration guide for your version of OneFS.
After the log gathering process is complete, a link to the file that the process generates appears under Filename in the
Archived Info Manager section.

Gather cluster logs - CLI

You must have root access to run log gathering commands.
To gather the log files in OneFS 9.0.0.0 and earlier, run the following command:

isi_gather_info

To gather the log files in OneFS 9.1.0.0 and later, run the following command:

isi diagnostics gather start

The files generated during the log gathering process are stored on the cluster in the /ifs/data/Isilon_Support/pkg
directory.

Completing pre-upgrade tasks 13

Check cluster readiness
Completing pre-upgrade tasks such as checking the health of the hardware components on the cluster, ensuring that storage
space requirements are met, and managing outstanding cluster events and errors are recommended for a successful upgrade.

Check hardware health - web administration interface

Use the OneFS web administration interface to evaluate the health of the clusters hardware components. Log in as root user
and perform the following steps on your cluster:
1. Click Cluster Management > Hardware Configuration.
2. In the Nodes tab, check that each node has a healthy status.
3. In the Drives tab, check that each drive has a healthy status.
4. If any correctable or fatal errors are reported, do not continue the upgrade. If the errors cannot be resolved, contact Dell
Technologies Support
NOTE: Dell Networking switches running Dell Network Operating System (DNOS) version 10.5.0.6C1 or earlier, that are
used for back-end traffic and configured as a single switch (also known as a flat switch, or a Top of Rack (TOR) switch)
might be affected by a certificate expiration issue. For more information, including a list of affected switches and a
workaround, see DTA 190714.

Check hardware health - CLI

Run the following commands from the OneFS command-line interface as root user to evaluate the health of the cluster's
hardware components and the status of job engine jobs.
1. To return information about cluster health and check for jobs or devices that report a status of ATTENTION, SMARTFAIL,
or DOWN.

isi status -v

2. To check for drives that do not report a status of HEALTHY, L3, or JOURNAL.

isi_for_array -s 'isi devices drive list | egrep -v "HEALTHY|L3|JOURNAL"'

3. To check the mirror status of the boot drives on each node.

isi_for_array -s 'gmirror status'

NOTE: If a drive is degraded, do not continue with the upgrade until the issue is resolved.

4. If the cluster has an InfiniBand network, confirm whether a node has been assigned the OpenSM (subnet manager) main
role.

isi_for_array -s 'ps -auwwwx | grep opensm' | grep master

Confirm that the output displays only one node in the cluster as the main (opensm). The output should be similar to the
following:

node-2: root 1610 0.0 2.3 436292 384672 ?? Ss 20May15 97:31.63

opensm: 0x00151b00007a671b main (opensm)

Check the available free space

Ensure that the minimum available-space requirements for the cluster, nodes, node pools, and critical directories are met before
you upgrade. Do not continue with the upgrade if the minimum available-space requirements are not met.

14 Completing pre-upgrade tasks

Available space requirements for clusters, nodes, node pools, and
directories
The cluster, nodes, node pools, and several critical directories in the file system must meet available space requirements before
upgrading OneFS. If the available space for any of these items falls below the minimum requirement, make more space available
before upgrading. Otherwise, the upgrade process fails and might not return an error indicating available space as the cause.

Table 2. Item Requirements

Item Minimum Requirement Task
Cluster The cluster cannot be more than Make more available space.
90 percent capacity.
Node Each node cannot be more than Make more available space.
92 percent capacity.
Node pool Each node pool cannot be more Make more available space.
than 90 percent capacity.
root partition (/) The root partition cannot be more If this directory is at or near the minimum available-
than 97 percent capacity. space requirement, see the following resources for
steps to address the issue:
● Event ID 100010003, The /(root) partition is near
capacity.

/ifs The /ifs directory cannot be If this directory is at or near the minimum available-
more than 90 percent capacity. space requirement, see the following resources for
steps to address the issue:
● Event ID 100010004, The cluster's /ifs partition
is near capacity.

/var The /var partition cannot be If this directory is at or near the minimum available-
more than 90 percent capacity. space requirement, see the following resources for
steps to address the issue:
● Event ID 100010001, The /var partition is near
capacity.

/var/crash The /var/crash directory If this directory is at or near the minimum available-
cannot be more than 90 percent space requirement, see the following resources for
capacity. steps to address the issue:
● Event ID 100010002, The /var/crash partition is
near capacity.

Confirm used and available space

● To confirm how much space is being used on the cluster and on each node, run the following command:

isi stat

● To confirm how much space is being used in each node pool, run the following command:

isi stat -p

● To confirm how much space is being used by critical directories on the cluster, run the following command:

isi_for_array -s 'df -h'

The isi_for_array output is similar to the following for each node in the cluster:

Completing pre-upgrade tasks 15

 virtual-demo-3: Filesystem Size Used Avail Capacity Mounted on
--------------------------------------------------------------------
virtual-demo-3: /dev/imdd0a 495M 418M 38M 92% /
virtual-demo-3: devfs 1.0K 1.0K 0B 100% /dev
virtual-demo-3: /dev/imdd1a 495M 6.7M 449M 1% /var
virtual-demo-3: /dev/imdd2a 496M 5.9M 451M 1% /var/crash
virtual-demo-3: OneFS 40G 283M 20G 1% /ifs

virtual-demo-2: Filesystem Size Used Avail Capacity Mounted on

--------------------------------------------------------------------
virtual-demo-2: /dev/imdd0a 495M 417M 38M 92% /
virtual-demo-2: devfs 1.0K 1.0K 0B 100% /dev
virtual-demo-2: /dev/imdd1a 495M 6.1M 449M 1% /var
virtual-demo-2: /dev/imdd2a 496M 5.9M 451M 1% /var/crash
virtual-demo-2: OneFS 40G 283M 20G 1% /ifs

Resolve outstanding events and errors

Before you begin your upgrade, resolve any outstanding critical events, errors, and failures, as these issues can disrupt the
OneFS upgrade process.
1. Run the following command to view all events on the cluster:

isi event groups list --sort=severity

● If the command returns any critical errors, check the log files in the following directories for more information:
○ /var/log
○ /var/log/messages
○ /var/crash
● NOTE: If any log files contain messages about a dynamic sector recovery (DSR) failure or a Data Integrity (IDI)
failure, contact PowerScale before you upgrade.
2. Cancel non-critical events before upgrading to prevent a recurrence of notifications that you know to be harmless.
If you have critical events that you are unable to resolve, contact support before upgrading.

Verify configurations and settings

Confirm that your OneFS configurations for CloudPools, NIC aggregation, Kerberos, InsightIQ, and SMB are supported by the
target version of OneFS to which your are upgrading.

Upgrading in a Cloudpools environment

If you're upgrading a OneFS 8.x cluster to OneFS 8.2.x. or later with a Cloudpools environment, there are additional items to
consider. For more information on pre-upgrade and upgrade steps, see KB article 000020736

Upgrading in a FIPS or STIG environment

If you are upgrading to OneFS 9.5 or later, and you have enabled Federal Information Processing Standard (FIPS) or Security
Technical Implementation Guides (STIG) on your cluster, you must disable FIPS or STIG before you upgrade. For more
information about FIPS or STIG, see the OneFS Security Configuration Guide.

Preserve the Kerberos keytab file

If Kerberos authentication is configured on the cluster, upgrading OneFS might delete the local /etc/<name>.keytab file
that is on each node. Choose one of the following methods to preserve the keytab file during the upgrade:

16 Completing pre-upgrade tasks

● Copy the keytab file to the /ifs directory where it can be accessed by all nodes on the cluster instead of saved locally on
each node.
● Add the following text to the /etc/mcp/override/user_preserve_files.xml file of each node on the cluster
where <name> is the name of the keytab file:

<?xml version="1.0" encoding="UTF-8"?>

<user_preserve>
<files>
<file name="etc/allow_unsupported_boot" recursive="no">
</file>
...
...
<file name="etc/<name>.keytab" recursive="no" />

</files>
</user_preserve>

Install a supported version of DataIQ or InsightIQ

Before you upgrade OneFS, confirm the version of DataIQ or InsightIQ you are running is compatible with the target version of
OneFS to which you are upgrading. If the versions are not compatible, upgrade DataIQ or InsightIQ before upgrading OneFS.
See the Supportability and Compatibility Guide on the Dell Support site for OneFS compatibility information.
See the DataIQ - info hub for documents and content related to DataIQ, including release notes, installation guides, user guides,
and troubleshooting guides.
See the InsightIQ - Info hub for documents and content related to InsightIQ.

Download the OneFS installation bundle file

1. From the OneFS Downloads page, download the installation bundle file for the target version of OneFS to which you are
upgrading.
NOTE: It is recommended that you use a OneFS installation bundle file that includes the latest patches. If the
installation bundle is used, you can skip the patch installation task in the post-upgrade steps. Contact your support
representative for access to an installation file that does not include patches.

NOTE: If you use the web administration interface to upgrade from OneFS 9.3.0.x or earlier, to OneFS 9.4.0.0 or later,
you must first change the file extension from .isi to .tar.gz. (example: change OneFS_v9.4.0.0_Install.isi to
OneFS_v9.4.0.0.Install.tar.gz)

2. Open a secure shell (SSH) connection to any node in the cluster and log in using the root account.
3. Move the installation file that you downloaded into the /ifs/data directory on the cluster you want to upgrade.
4. Validate the integrity of the installation file.
For OneFS 9.4.0.0 and later, use the following instructions:
a. To import the package into the OneFS Catalog, run the isi upgrade catalog import /ifs/data/
<filename>.isi command.
b. To verify the package, run the isi upgrade catalog verify --file /ifs/data/<filename>.isi
command.
For OneFS 9.3.0.0 and earlier, use the following instructions:
a. Optional: On the OneFS downloads page, click Checksum Values and record the SHA-256 checksum value displayed.
b. Optional: In OneFS CLI, run the following command where <installation-file-name> is the name of the downloaded
installation file:
● sha256 /ifs/data/<installation-file-name>
c. Optional: Compare the SHA-256 checksum value that you recorded from the downloads page on the Online Support site
to the checksum value returned from the SHA-256 command. If the values do not match, re-download the installation
file.

Completing pre-upgrade tasks 17

5. (OneFS 9.3.0.0 and earlier) To unzip the downloaded tgz file, run the following command, where <installation-file-name> is
the name of the downloaded tgz file:

gzip -d /ifs/data/<installation-file-name>

NOTE: Clusters running OneFS 9.4.0.0 and later are not required to unzip .tgz files.

6. (OneFS 9.3.0.0 and earlier) To unbundle the .tar file, run the following command where <installation-file-name> is the name
of the OneFS install tar file:

tar -vxf /ifs/data/<installation-file-name> /ifs/data/

NOTE: Clusters running OneFS 9.4.0.0 and later are not required to unbundle .tar files.

7. (OneFS 9.3.0.0 and earlier) Validate the manifest file (example: OneFS_v9.3.0.0_Install.tar.gz). See the Checking Manifests
section of the PowerScale OneFS Security Configuration Guide for more information.
NOTE: Clusters running OneFS 9.4.0.0 and later use the OneFS Catalog to verify packages.

Upgrade compatibility check utility

The upgrade compatibility check utility examines cluster settings, activities, and statuses to verify whether your cluster is
compatible with the target version of OneFS to which you are upgrading.
The upgrade compatibility check utility is included in each OneFS installation package. You can run the utility alone or as part of
the upgrade process.

Run the upgrade compatibility check utility

To confirm that your OneFS cluster is compatible with the target version that you are upgrading to, run the upgrade
compatibility check utility.
NOTE: The upgrade compatibility check utility is included in each OneFS installation package, which must be downloaded
and accessible in an /ifs directory.

1. Open a secure shell (SSH) connection to any node in the cluster and log in to the cluster with the root account.
2. Start the upgrade compatibility check utility by running the following command, where <install-image-path> is the file path
of the upgrade installation file.
isi upgrade cluster assess <install-image-path>
NOTE: The upgrade compatibility check utility might take several minutes to run. If the utility returns errors, resolve the
errors before continuing with the upgrade. Warnings are informational and do not prevent an upgrade.
3. View the results of the upgrade compatibility check by running the following command:
isi_upgrade_logs -a

Upgrade compatibility check summary

The upgrade compatibility check utility examines the cluster and returns warnings or errors if something is not compatible with
the version of OneFS to which you are upgrading. The upgrade compatibility check utility examines the following areas:

Table 3. Check Specifications

Check name Description Recommendation
Disk load Checks the cluster usage level and Disconnect all client connections and
returns a warning if the disk load is stop all system jobs before upgrading.
greater than 50%.

18 Completing pre-upgrade tasks

Table 3. Check Specifications (continued)
Check name Description Recommendation
Free Space Checks cluster free space and returns Free up space in the necessary partition
a warning if capacity thresholds are before upgrading.
exceeded for the following partitions:
● /
● /var
● /ifs
● Node pools
Drive stalls Checks the health of the drives in the Smartfail and resolve the stalled drive
cluster. issue before upgrading.
Smartfail operation status Determines if a Smartfail operation is Wait for the Smartfail operation to
running on the cluster. complete before upgrading.
IntegrityScan job status Determines if an IntegrityScan job is Wait for the job to complete before
running on the cluster. upgrading.
Unresolved critical alerts Checks for any critical alerts on the Resolve all critical alerts before
cluster. upgrading.
Unsupported SMB configuration Checks for unsupported SMB settings Reconfigure any unsupported SMB
on the cluster. settings before upgrading.
SMB access zone association Checks if any SMB shares are associated Resolve issues where the share paths
with multiple access zones where the overlap before upgrading. The upgrade
share paths overlap. will succeed, but you cannot create new
SMB shares until the overlap issue is
resolved.
Kerberos keytab Checks if the Kerberos keytab file will be You must configure the Kerberos
preserved after the upgrade. settings to preserve the keytab file
before upgrading.
HDFS Kerberos keytab Checks if the HDFS Kerberos keytab file You must configure the Kerberos
will be preserved after the upgrade. settings to preserve the keytab file
before upgrading.

Reconfigure unsupported SMB settings

If the SMB settings on the cluster are not supported by the target version of OneFS to which you are upgrading, the upgrade
might fail. The upgrade compatibility check utility confirms whether your current SMB settings are supported.
If the upgrade compatibility check utility detects unsupported SMB settings, remove or modify the unsupported SMB settings
through the command-line interface or web administration interface before you upgrade.
NOTE: SMB1 is disabled during the upgrade process which might cause some clients to have authentication issues.

On-Cluster Analysis tool

The PowerScale On-Cluster Analysis (OCA) tool analyzes the health status of a running cluster and helps you prepare for
an upgrade. The OCA tool analyzes your cluster and displays any issues that it encounters, a summary of the issue, and any
associated KB articles to help you resolve the issue. The OCA tool also has a feature that provides you with an estimated
upgrade plan based on the analysis.
For more information about the OCA tool, please see the KB article: How to run the On-Cluster Analysis tool

Completing pre-upgrade tasks 19

HealthCheck tool
The HealthCheck tool enables you to evaluate the status of specific software and hardware components of your cluster and
your cluster environment. For more information about the HealthCheck tool, see this site: HealthCheck Info Hub

Backup data
It is recommended to backup your cluster data immediately before you upgrade. Schedule sufficient time for the back up to
complete before the upgrade window.

SyncIQ backup
SyncIQ is one option you can use to backup your OneFS cluster. SyncIQ creates and references snapshots to replicate a
consistent point-in-time image of a root directory.
For more information about backing up your OneFS cluster, see the OneFS CLI Administration Guide or the OneFS Web
Administration Guide for your version of OneFS.
NOTE: If you are upgrading your cluster from OneFS 8.1.0.x or earlier to OneFS 8.1.1.x, 8.1.2.x, or 8.1.3.x, and your cluster
is in Compliance mode, you must ensure that all SyncIQ partners are on the same code and patch level OneFS before
restarting SyncIQ backups, or the backups fail. This issue is resolved in OneFS 8.2.0 and later.

NOTE: If you are upgrading your cluster to OneFS 8.2.0 through 9.2.1.0, are using SyncIQ, and have non-networked
(NANON) nodes on your source cluster, SyncIQ jobs might fail after the upgrade. For more information including a
workaround, see the KB article SyncIQ jobs failing with NANON after upgrade

NDMP backup
Other OneFS cluster backup options include using the Network Data Management Protocol (NDMP).
From a backup server, you can perform both NDMP three-way backup and NDMP two-way backup processes between a cluster
and backup devices such as tape devices, media servers, and virtual tape libraries (VTLs).
See the OneFS Web Administration Guide or the OneFS CLI Administration Guide for information about backing up data using
NDMP.

Complete NDMP backups

Before you upgrade, you must wait for Network Data Management Protocol (NDMP) backups to finish so you have saved
copies of your data.
NOTE: If you cannot wait for NDMP backups to finish, stop the active NDMP backups 30– 60 minutes before the upgrade,
as the NDMP backup process requires this additional time to come to a stop.
For instructions on how to stop backup jobs, see the documentation for your backup application.
NOTE: If you are upgrading from OneFS version 8.0.x or 8.1.x to OneFS 8.2.x, you must disable the NDMP service before
performing the upgrade.

Back up custom settings

Most settings are preserved during a OneFS upgrade. However, documenting and backing up custom settings enables you to
reapply any settings that are not preserved during the upgrade process.
Document and back up the following custom settings:

20 Completing pre-upgrade tasks

Table 4. Custom Setting Specifications
Setting Description Recommendation
SMB audit If you have an audit log directory After upgrading, you must reconfigure SMB audit logging. See File
logging in /ifs, you might have custom System Auditing with the Common Event Enabler (CEE) for more
SMB logging settings configured. information about SMB audit logging.

Passwords for After you upgrade, you might Make a list of the local accounts and their passwords before you
local user have to reset the passwords upgrade.
accounts of the local user accounts that
you configured on the cluster.
Other users should be prepared to
reset the passwords of their local
accounts after the upgrade.
sysctl If you changed the default PowerScale does not recommend modifying sysctl parameters
parameters value that is assigned to one unless you are instructed to do so by Dell Technologies Support. If
or more sysctl parameters you must modify a sysctl parameter, configure the parameter in
by editing the /etc/mcp/ the /etc/mcp/override/sysctl.conf file to ensure that the
override/sysctl.conf file, change is preserved when you upgrade a node or a cluster.
the etc/mcp/templates
Before you upgrade, document your custom sysctl parameters and
directory, or the /etc/local/
back up the /etc/mcp/override/sysctl.conf, the etc/mcp/
sysctl.conf file, you might
templates directory, and the /etc/local/sysctl.conf files.
need to reset the parameter after
you upgrade. For more information about making sysctl changes persist through
upgrades, see KB article 000102543 and KB article 000083411.
If you modified a sysctl
parameter by editing another
file—for example, the /etc/
sysctl.conf file—the change
will not be preserved during the
upgrade.

Cron jobs Cron jobs settings that were Document and back up custom cron job settings or configure them
not configured in the /etc/mcp/ in the /etc/mcp/override/crontab.smbtime file before you
override/crontab.smbtime upgrade.
file are not preserved during an
upgrade. After you upgrade, you might have to modify a cron job to
accommodate changes to OneFS commands.

Complete or stop jobs in progress

You should ensure that there are no jobs running on your OneFS cluster before beginning the upgrade. Wait for jobs to fully
complete or stop jobs before upgrading.

Complete system jobs

Ensure that no system jobs are running during the upgrade by allowing system jobs to finish before the upgrade starts, or by
cancelling them. OneFS performs system jobs through a service that runs in the background, and if any system jobs are running
during the upgrade, the upgrade process might fail.
1. To check for running system jobs, run the following command, and make a note of the job ID for any jobs that you want to
cancel:

isi job status

2. To cancel a job, run the following command where <job_id> is the ID of the job you want to cancel:

isi job jobs cancel <job_id>

Completing pre-upgrade tasks 21

 NOTE: Do not cancel the Upgrade, FlexProtect, FlexProtectLin, or IntegrityScan jobs. If any of these four system jobs
are running, you cannot continue with the upgrade. If an Upgrade, FlexProtect, FlexProtectLin, or IntegrityScan system
job takes longer than expected to complete, contact Dell Technologies Support.

NOTE: Sync polices and jobs must be canceled or paused in order for the upgrade to complete successfully.

Update Node Firmware Package

Before upgrading your OneFS cluster, ensure that your nodes are running the most recent version of firmware found in the
Node Firmware Package.
For additional information on updating node firmware, see the OneFS Node Firmware Package site.
NOTE: In OneFS 9.2.0.0 and later, you can choose to update the Node Firmware Package during a parallel upgrade. If you
choose to perform the Node Firmware update during an upgrade, delay this step until the Performing a Parallel Upgrade
chapter.

Update Drive Support Package

Before upgrading your OneFS cluster, ensure that your drives are running the most recent version of firmware found in the
Drive Support Package.
For more information, see the OneFS Drive Support Package site.

Configure IPMI ports

If you have enabled Intelligent Platform Management Interface (IPMI) ports, it is recommended that you change the IPMI
port IP configuration to static in the BIOS settings for each affected node before the upgrade. If you change the IPMI port
configuration during the upgrade process, and your workflow requires the IPMI ports be enabled, you must manually re-enable
the ports after the upgrade process is complete.

NOTE: Use of IPMI ports is supported in OneFS version 8.2.2 and later.

Disable SupportAssist or Secure Remote Services

(SRS)
SupportAssist and SRS allow remote support access to your cluster.
It is recommended that you disable SupportAssist or SRS before you upgrade your OneFS cluster.

22 Completing pre-upgrade tasks

 4
Performing the OneFS upgrade
Topics:
• Upgrade process overview
• Completing a parallel upgrade of OneFS
• Completing a rolling upgrade of OneFS
• Completing a simultaneous upgrade of OneFS
• Adding a node to the cluster while an upgrade is in progress
• Committing an upgrade of OneFS
• Rolling back an upgrade of OneFS
• Verify the OneFS installation using the CLI

Upgrade process overview

The OneFS upgrade process consists of downloading the OneFS installation image, starting the upgrade, and verifying that the
upgrade is complete.
The following upgrade types are available:
● Parallel upgrade (for OneFS 8.2.2 and later)
● Rolling upgrade
● Simultaneous upgrade
A OneFS installation image is required to upgrade your cluster. See the Download the OneFS image section for more
information.

NOTE: You can upgrade OneFS using the command-line interface or the web administration interface.

Upgrade - Summary
The following is a summary of steps to perform during the upgrade phase:
1. Perform the upgrade.
2. Commit the upgrade.
3. Verify the upgrade.

Completing a parallel upgrade of OneFS

If you perform a parallel upgrade, subsets of nodes within the OneFS cluster are restarted in an undetermined order. During
a parallel upgrade, client connections to the restarting subset of nodes are disconnected, but other subsets of nodes remain
available for client connection.
NOTE: The parallel upgrade feature is available in OneFS version 8.2.2 and later. Upgrades where the source cluster is on
OneFS 8.2.1 or earlier must use the rolling or simultaneous option.

Performing the OneFS upgrade 23

Performing a parallel upgrade using the web administration
interface
You can use the web administration interface to upgrade the version of OneFS that is running on your cluster.
Perform the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the
new version of OneFS.
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Enter the file path location of the upgrade image to install. The file path must be accessible in an /ifs directory.
5. (Optional) In OneFS 9.2.0.0 and later, to update node firmware during the upgrade, enter the file path location of the node
firmware package.
6. In the Upgrade Type drop-down list, select Parallel Upgrade.
7. (Optional) To skip optional pre-upgrade checks related to compatibility, select Skip optional pre-upgrade checks.
8. (Optional) In OneFS 9.2.0.0 and later, to manage disruption to SMB client connections, follow these steps:
a. Check Enable disruption manager
b. (Optional) Check Wait Forever to wait for all SMB clients to disconnect from a node before upgrading.
c. Enter the Drain time out to specify the amount of time OneFS waits for clients to manually disconnect before the node
reboots and upgrades.
d. Enter the CELOG time out to specify the amount of time OneFS waits for clients to disconnect before alerting the
OneFS administrator. This allows the administrator to work with the connected clients before rebooting the node.
9. Click Start Upgrade to begin the upgrade.
The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.

NOTE: In 9.2.0.0 and later, you can use the upgrade status screen to monitor the progress of your upgrade, view
connected clients, and delay draining clients from specific nodes.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will
return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.

Performing a parallel upgrade using the CLI

You can use the command-line interface to upgrade the version of OneFS that is running on your cluster.
Perform the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the
new version of OneFS.
1. Open a secure shell (SSH) connection on any node in the cluster and log in with the root account.
2. In OneFS 9.2.0.0 and later, you can manage the amount of time OneFS waits to disconnect client connections using the isi
upgrade cluster drain command and sub-commands.
3. To perform a parallel upgrade, run the following command, where <install-image-path> is the file path of the upgrade install
image. The file path must be accessible in an /ifs directory.

isi upgrade cluster start --parallel <install-image-path>

In OneFS 9.2.0.0 and later, you can include the firmware upgrade with your OneFS upgrade. To perform a parallel upgrade
that includes a firmware upgrade, run the following command where <install-image-path> is the file path of the upgrade
install image and <firmware-path> is the file path of the firmware package.

isi upgrade cluster start --parallel <install-image-path> --fw-pkg <firmware-path>

NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web administration
interface.

24 Performing the OneFS upgrade

4. You can specify the following parallel upgrade options:

Options Description
Nodes to Upgrade specific nodes with the --nodes <integer_range_list> option.
select for
Specify the nodes in their upgrade order as a comma-separated list (for example, --nodes 7,3,2,5) or as
upgrade
a dash-separated range (for example, --nodes 1-7) of logical node numbers (LNNs).
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is
sent to confirm that the upgrade is making progress. Do not leave the cluster in a partially upgraded
state for a prolonged period. Some new features in the upgrade might not be available until all the nodes
in the cluster have been upgraded and the upgrade is committed. Refer to the release notes for the
OneFS version that you are upgrading to for information about features that require all the nodes to be
upgraded.

OneFS 9.2.0.0 and later ignores the node restart order. Instead, OneFS restarts any node that is ready, as long as only one
node is restarting at a time.
The following example for OneFS 9.0.0.0 through OneFS 9.1.0.x starts a parallel upgrade on nodes 7,3,2,5, in that order:

isi upgrade cluster start --parallel <install-image-path> --nodes 7,3,2,5

5. To add nodes to a current upgrade, run the following command, where <nodes> is the list of LNNs:

isi upgrade add-nodes <nodes>

6. To add any remaining nodes to the upgrade, run the following command:

isi upgrade add-remaining-nodes

After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will
return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.

Client connections during parallel upgrades

Parallel upgrades allow users to access data before, during, and after the upgrade. However, as groups of nodes are upgraded
and restarted, users may experience brief pauses in the time it takes to complete a read or write operation.
If the cluster is configured for dynamic IP allocation, client connections to restarted nodes are automatically reconnected. If
a client is reconnected to a node that has not yet been upgraded and restarted, the client might be required to reestablish a
connection to the cluster more than once.
The following table describes the expected client behavior when a node is restarted on a cluster that is configured for dynamic
IP allocation:

Table 5. Client Behaviors

Client Expected client behavior
SMB2 Client transitions from the restarting node to a new node, but the client is disrupted.
NOTE: In OneFS 9.3.0.0 and later, client transitions from the restarting node to a new node
without disruption.

SMB3 Client transitions from the restarting node to a new node without disruption.
NFSv2 and NFSv3 Client transitions from the restarting node to a new node without disruption.
NFSv4 Client transitions from the restarting node to a new node without disruption. Clients use NFSv4
failover support.

NOTE: For more information about NFS settings, see article 457328, Best practices for NFS client settings.

Performing the OneFS upgrade 25

Completing a rolling upgrade of OneFS
If you perform a rolling upgrade, the OneFS cluster nodes are restarted one at a time. During a rolling upgrade, client
connections to the restarting node are disconnected, but other nodes remain available for client connection.
NOTE: When you perform a rolling upgrade between OneFS releases, you can upgrade all the nodes in the cluster, or you
can select specific nodes to upgrade. If you upgrade only some of the nodes, the remaining nodes in the cluster are not
upgraded but can be upgraded later in the upgrade process. The upgrade is not complete until all nodes are upgraded, and
only one upgrade can be in progress at a time. You cannot upgrade some nodes to one version of OneFS and then upgrade
another group of nodes to a different version of OneFS.

Perform a rolling upgrade using the web administration interface

You can use the web administration interface to upgrade the version of OneFS that is running on the cluster.
You must download the OneFS installation image OneFS installation image. The installation image is used to run compatibility
checks. You must download the image and run the checks before upgrading. You must resolve any compatibility issues found by
the checks before upgrading.
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs
directory. Then click Select.
5. In the Upgrade Type list, select Rolling Upgrade.
6. Under Nodes to Upgrade, specify an upgrade option:
● To upgrade all the nodes in the cluster, click All nodes.
● To upgrade specific nodes, click Specify nodes and reboot order.
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is sent to confirm
that the upgrade is making progress. Do not leave the cluster in a partially upgraded state for a prolonged period.
Some new features in the upgrade might not be available until all the nodes in the cluster have been upgraded
and the upgrade is committed. Refer to the release notes for the OneFS version that you are upgrading to for
information about features that require all the nodes to be upgraded.
7. If you are upgrading specific nodes, select the node or nodes in the Nodes Available for Upgrade list, and then click Add.
The selected node or nodes are added to the Nodes Selected for Upgrade section. To arrange the nodes in the preferred
restart order, click the arrows next to the nodes in the Nodes Selected for Upgrade section.
NOTE: In OneFS 9.2.0.0 and later, the restart order is ignored. Instead, OneFS restarts any node that is ready, as long
as only one node is restarting at a time. This process allows for quicker rolling upgrades.
8. Optional: To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
NOTE: It is recommended that you run the optional pre-upgrade checks. Before starting an upgrade, OneFS checks that
your cluster is healthy enough to complete the upgrade process. Some of the pre-upgrade checks are mandatory, and
will be performed even if you choose to skip the optional checks. All pre-upgrade checks contribute to a safer upgrade.

9. Click Start Upgrade.

The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.

NOTE: In 9.2.0.0 and later, you can use the upgrade status screen to monitor the progress of your upgrade and view
connected clients.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will
return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.

26 Performing the OneFS upgrade

Perform a rolling upgrade using the CLI
You can use the command-line interface or the web administration interface to upgrade the version of OneFS that is running on
the cluster.
Review the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the
new version of OneFS.
Command-line Interface instructions:
1. Open a secure shell (SSH) connection on any node in the cluster and log in with the root account.
2. To perform a rolling upgrade, run the following command, where <install-image-path> is the file path of the upgrade install
image. The file path must be accessible in an /ifs directory.
For OneFS 8.2.2 and later:

isi upgrade cluster start --rolling <install-image-path>

For OneFS 8.2.1 and earlier:

isi upgrade cluster start <install-image-path>

NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web administration
interface.
3. Optional: You can specify the following rolling upgrade options:

Options Description
Nodes to Upgrade specific nodes with the --nodes <integer_range_list> option.
select for
Specify the nodes in their upgrade order as a comma-separated list (for example, --nodes 7,3,2,5) or as
upgrade
a dash-separated range (for example, --nodes 1-7) of logical node numbers (LNNs).
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is
sent to confirm that the upgrade is making progress. Do not leave the cluster in a partially upgraded
state for a prolonged period. Some new features in the upgrade might not be available until all the nodes
in the cluster have been upgraded and the upgrade is committed. Refer to the release notes for the
OneFS version that you are upgrading to for information about features that require all the nodes to be
upgraded.

OneFS 9.2.0.0 and later ignores the node restart order. Instead, OneFS restarts any node that is ready, as long as only one
node is restarting at a time.
The following example for OneFS 8.2.2 through OneFS 9.1.0.x starts a rolling upgrade on nodes 7,3,2,5, in that order:

isi upgrade cluster start --rolling <install-image-path> --nodes 7,3,2,5

The following example for OneFS 8.2.1 and earlier starts a rolling upgrade on nodes 7,3,2,5 in that order:

isi upgrade cluster start <install-image-path> --nodes 7,3,2,5

4. Optional: To add nodes to a current upgrade, run the following command, where <nodes> is the list of LNNs:

isi upgrade add-nodes <nodes>

5. Optional: To add any remaining nodes to the upgrade, run the following command:

isi upgrade add-remaining-nodes

After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will

Performing the OneFS upgrade 27

return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.

Configuration changes during rolling upgrades

You can continue to manage data and modify some cluster configurations during a rolling upgrade. For example, you can modify
SMB shares and NFS exports. It is recommended that you make configuration changes from the node with the highest devid.
The node with the highest devid is the last node to be upgraded and helps you avoid being disconnected during the upgrade
process.
NOTE: Attempts to enable or configure settings on an upgraded node fail until the upgrade is complete. New OneFS
features and software modules are not functional until all nodes in the cluster are upgraded.

Client connections during rolling upgrades

Rolling upgrades allow users to access data before, during, and after the upgrade. However, as nodes are upgraded and
restarted, users may experience brief pauses in the time it takes to complete a read or write operation.
If the cluster is configured for dynamic IP allocation, client connections to restarted nodes are automatically reconnected. If
a client is reconnected to a node that has not yet been upgraded and restarted, the client might be required to reestablish a
connection to the cluster more than once.
The following table describes the expected client behavior when a node is restarted on a cluster that is configured for dynamic
IP allocation:

Table 6. Client Behaviors

Client Expected client behavior
SMB2 Client transitions from the restarting node to a new node, but the client is disrupted.
NOTE: In OneFS 9.3.0.0 and later, the client transitions from the restarting node to a new node
without disruption.

SMB3 Client transitions from the restarting node to a new node without disruption.
NFSv2 and NFSv3 Client transitions from the restarting node to a new node without disruption.
NFSv4 Client transitions from the restarting node to a new node without disruption. Clients use NFSv4
failover support.

NOTE: If a client is reconnected to a node that has not yet been upgraded, the client might be required to reestablish a
connection to the cluster more than once.

NOTE: For more information about NFS settings, see article 457328, Best practices for NFS client settings.

Completing a simultaneous upgrade of OneFS

If you run a simultaneous upgrade, the OneFS cluster is upgraded simultaneously and then all of the nodes in the cluster are
restarted simultaneously. During a simultaneous upgrade, the entire cluster is down and unavailable for client connections.

Perform a simultaneous upgrade using the web administration

interface
You can use the web administration interface to upgrade the OneFS version that is running on a cluster.
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs
directory. Then click Select.

28 Performing the OneFS upgrade

5. In the Upgrade Type list, select Simultaneous Upgrade.
6. (Optional) To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
NOTE: It is recommended that you run the optional pre-upgrade checks. Before starting an upgrade, OneFS checks that
your cluster is healthy enough to complete the upgrade process. Some of the pre-upgrade checks are mandatory, and
will be performed even if you choose to skip the optional checks. All pre-upgrade checks contribute to a safer upgrade.

7. Click Start Upgrade.

The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will
return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.

Perform a simultaneous upgrade using the CLI

You can use the command-line interface to upgrade the version of OneFS that is running on the cluster.
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To perform a simultaneous upgrade, run the following command, where <install-image-path> is the file path of the upgrade
install image. The file path must be accessible in an /ifs directory.

isi upgrade cluster start --simultaneous <install-image-path>

NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web administration
interface.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the
cluster is accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will
return to normal. At this stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of
OneFS. Some new features in the upgrade might not be available until the upgrade is committed.

Adding a node to the cluster while an upgrade is in

progress
In some circumstances, you might need to add a node to the cluster while an upgrade is in progress and before the upgrade has
been committed.
To add a node during an upgrade, at least one node in the cluster must have been successfully upgraded to the target version
of OneFS. When a new node is added to the cluster during an upgrade, the system images the new node with the previously
committed version of OneFS. Then as the cluster upgrade continues, the new node is upgraded to the target version of OneFS.
If the cluster upgrade is stopped and rolled back, all the nodes in the cluster are returned to the previously committed version of
OneFS.
NOTE: Before adding new hardware to the cluster, first confirm that the hardware is compatible with the current version
of OneFS that is installed and the target version of OneFS that you are upgrading to. See the OneFS Supportability and
Compatibility Guide for more information.

Committing an upgrade of OneFS

You must commit an upgrade to complete the upgrade process. Once you commit the upgrade, you cannot roll back to the
previous version of OneFS.
New features in the target version of OneFS are not available until the upgrade has been committed.

Performing the OneFS upgrade 29

Commit an upgrade using the web interface
You can use the web interface to commit an upgrade of OneFS. Once you commit the upgrade, you cannot roll back to the
previous version of OneFS.
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click the Commit Upgrade to OneFS <version> button.
The cluster might display several confirmation messages. Confirm each message to continue the commit process.

Commit an upgrade using the CLI

You can use the command-line interface to commit an upgrade of OneFS. Once you commit the upgrade, you cannot roll back to
the previous version of OneFS.
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To commit the upgrade, run the following command:

isi upgrade cluster commit

Rolling back an upgrade of OneFS

The upgrade rollback feature allows you to stop the upgrade that is in progress and restore all the upgraded nodes to the
previous committed version of OneFS.
NOTE: You can roll back an upgrade only if the upgrade process is still active and the upgrade has not been committed. A
committed upgrade cannot be rolled back.

Roll back an upgrade using the web interface

You can use the web administration interface to stop an upgrade and return to the previous committed version of OneFS.
NOTE: The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster
services and data availability.
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click the Roll Back to OneFS <version> button. In OneFS 9.2.0.0 and later, click the Halt and roll back current upgrade
button.
The cluster might display several confirmation messages. Confirm each message to continue the rollback process.
The cluster displays the rollback progress.

Roll back an upgrade using the CLI

Rolling back stops an upgrade and returns the cluster to the previous committed version of OneFS.
NOTE: The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster
services and data availability.
Command-Line Interface instructions:
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To roll back the upgrade, run the following command:

isi upgrade cluster rollback

30 Performing the OneFS upgrade

Verify the OneFS installation using the CLI
After you install OneFS, verify that the installation was successful.
1. Confirm that the health of all the nodes in the cluster is OK by running the following command:

isi stat

2. Remove the installation files from the /ifs/data directory by running the following command where
<installation_file_name> is the name of the installation file:

rm /ifs/data/<installation_file_name>

3. Collect information about the cluster in OneFS 9.0.0.0 and earlier, by running the following command:

isi_gather_info

Collect information about the cluster in OneFS 9.1.0.0 and later, by running the following command:

isi diagnostics gather start

Performing the OneFS upgrade 31

 5
Completing post-upgrade tasks
Topics:
• Post-upgrade process - Overview
• Allow the Upgrade job to run
• Verify operational status using the CLI
• Re-establish user privileges
• Check client connections, and test the workflow
• Verify Kerberos migration
• Reapply custom settings
• Test custom scripts
• Install the latest roll-up patch

Post-upgrade process - Overview

After an upgrade, you should perform restoration and change management tasks to ensure that your cluster performs and
behaves as expected. It is recommended that you build time into your upgrade plan to re-establish custom settings and
privileges, and re-enable connections and features. You should also make time to modify settings for new and changed features.

Post-upgrade - Summary
The following is a summary of steps to perform during the post-upgrade phase:
1. Allow upgrade jobs to run.
2. Verify operational status.
3. Re-establish user privileges.
4. Restore client connections and workflows.
5. Verify Kerberos migration.
6. Restore custom settings.
7. Test custom scripts.
8. (Optional) Install latest patch.

Allow the Upgrade job to run

After an upgrade, an important job titled Upgrade that upgrades on-disk data structures, might continue to run for a while. The
Upgrade job must be allowed to run to completion.
Although the cluster remains accessible while the Upgrade job is running, the job might temporarily decrease the cluster's
performance.

NOTE: For more information about the Upgrade job, see KB Article 194551.

Verify operational status using the CLI

You can run a series of CLI commands as root to help verify that the PowerScale cluster is working correctly after an upgrade.
If you find an unresolvable issue, contact Dell Technologies Support.

32 Completing post-upgrade tasks

1. Check the new version number of the cluster:

isi_for_array -s uname -a

2. View the status of the cluster and ensure all the nodes are operational:

isi status

3. Check the devices in the nodes to validate the status of the drives:

isi_for_array -s "isi devices list"

4. Check the status of jobs and resume any paused jobs:

isi job status

isi job jobs resume <jobID>

5. Review the list of events, and address any critical events:

isi event groups list --sort=severity

6. To verify network connectivity and SmartConnect functionality, ping all internal and external interfaces on the cluster.
7. Verify the network interfaces:

isi network interfaces list

8. Verify the subnets:

isi network subnets list --verbose

9. Verify the pools:

isi network pools list --verbose

10. To check for issues, review the log files on the cluster:

isi_upgrade_logs

cat /var/log/messages

11. Check the input and output:

isi statistics system

12. Check the global SMB settings:

isi smb settings global view

13. Check the status of the node firmware to ensure it is consistent across nodes:
For OneFS 9.0.0 and later:

isi_upgrade_logs --get-fw-report

Completing post-upgrade tasks 33

 For OneFS 8.2.2 and earlier:

isi upgrade firmware devices

14. Ensure that all the licenses carried over and remain up to date:

isi license licenses list

15. Check the status of the authentication providers to ensure that they remain active:

isi auth status --verbose

16. Review the list of SyncIQ jobs:

isi sync jobs list

17. Check the SyncIQ job reports:

isi sync reports list

18. Review the list of the scheduled snapshots:

isi snapshot schedules list

19. If you use SupportAssist or SRS, confirm that the service is reenabled.

Re-establish user privileges

After you upgrade, re-establish user privileges and roles. You can log in to the cluster through SSH as root or though an
administrator account if that role has been assigned to any users.
NOTE: If the system administered the user roles through RBAC before the upgrade, any custom roles that existed are still
in place. However, if the privileges assigned to built-in roles have changed in the new version, the users who are assigned to
those built-in roles will have those new privileges.
See the OneFS CLI Administration Guide or the OneFS Web Administration Guide for more information.

Check client connections, and test the workflow

After the OneFS upgrade, check client connections and confirm that all users, clients, and applications can access the cluster.
Test your workflows to ensure that they function correctly.

NOTE: If you are using NDMP backups on your cluster, re-enable the NDMP service and test that it's working correctly.

Verify Kerberos migration

If you are using Kerberos authentication, you must verify that the Kerberos providers and settings have been migrated
successfully.

Verify Kerberos migration using the web administration interface

Verify that Kerberos authentication providers, and settings are correct:
1. Click Access > Authentication Providers > Kerberos Provider.
2. In the Kerberos Realms, Kerberos Domains, and Kerberos Providers tables, verify that the Kerberos providers are
correct.

34 Completing post-upgrade tasks

3. In the Kerberos Settings area, verify that the Kerberos authentication settings are correct.

Verify Kerberos migration using the CLI

Verify that Kerberos authentication providers, and settings are correctly represented in the output of each of these commands:
● isi auth krb5 realm list
● isi auth krb5 domain list
● isi auth krb5 spn list
● isi auth settings krb5 view

Reapply custom settings

Some custom settings might not have been preserved during the upgrade. Reapply the custom settings that you backed up and
recorded when you performed the pre-upgrade tasks.
The custom settings include:
● SMB audit logging
● Passwords for local user accounts
● Changes to system controls
● Aspera
● Cron jobs
● Certificates
● Static routes

Test custom scripts

Test any custom scripts you have implemented to confirm they continue to work. If command syntax changes were made in the
target version of OneFS, some custom scripts might not work as intended.
NOTE: It is recommended to ensure that your custom scripts work on a test cluster before implementing them on a
production system.

Install the latest roll-up patch

Install the latest patch for the target version of OneFS to which you upgraded.
Confirm if your version of OneFS requires a patch. Most OneFS images already include the latest patch. For more information,
check the OneFS image download, or contact your account team.
1. See PowerScale OneFS Current Patches to view a list of patches that are available for your version of OneFS.
2. Download and install the latest roll-up patch using the instructions found in the README.txt file included with the patch
download.

Completing post-upgrade tasks 35

 6
Troubleshooting your upgrade
Topics:
• Troubleshooting an upgrade

Troubleshooting an upgrade
If you experience problems with your upgrade, try the following:
● Check the upgrade logs and review for errors.
● Search for OneFS upgrade information within the knowledge base on the PowerScale support site.
● Contact your Dell Technologies Support representative.

36 Troubleshooting your upgrade