|Article Number: 524285||Article Version: 5||Article Type: Break Fix|
Isilon,Isilon OneFS,Isilon OneFS 8.0,Isilon OneFS 8.1
When running a OneFS upgrade the upgrade might fail during the isi_version_checker phase stating that it is not able to decode a file which is not encoded with UTF-8.
You can verify the same by performing an upgrade assessment which will give out same results when viewing the upgrade summary report.
isi upgrade nodes list displays the following error :
Error Details: Failed Step: HookPreUpgradeOptional
The failure is likely to occur when running the uid_gid_check script which checks for any files under ‘/’ with a large UID/GID as this could impact the upgrade. If one of the files contains non-UTF-8 characters in its name. The script looks for any characters with non UTF encoding under / on all nodes.
OneFS 184.108.40.206, CLI parser can handle characters that were not valid UTF-8 avoiding such issues.
New pre-upgrade check introduced in 220.127.116.11
[A] If there is an OneFS upgrade which has failed follow the below steps:
(If there is an upgrade assessment which is giving an error jump to [B])
- Check the upgrade state first:
# isi upgrade view
Cluster Upgrade State: error >> error state in upgrade
(see output of `isi upgrade nodes list` to know which node failed)
Current Upgrade Activity: Upgrade
Upgrade Install Image: /ifs/data/Isilon_Support/OneFS_v18.104.22.168_Install.tar.gz
Upgrade Start Time: 2018-07-26T10:59:55
Current OS Version: 22.214.171.124_build(131)style(5)
Upgrade OS Version: 126.96.36.199_build(240)style(5)
Total Cluster Nodes: 5
Nodes On Older OS: 5
Nodes Upgraded: 0
Nodes Transitioning/Down: 0
- Check on which node it fails
# isi upgrade nodes list
Node LNN: 2
Node Upgrade State: error
Error Details: Failed Step: HookPreUpgradeOptional
Error Log: /ifs/.ifsvar/upgrade/Agent-32-000e1ea67780a95c3c57e400d1411b40b520.log
Last Upgrade Action: –
Last Action Result: –
Node Upgrade Progress: None
Node OS Version: 188.8.131.52_build(131)style(5)
- You will notice that the upgrade is in an error state, run “# isi_upgrade_logs” to generate the log files to check the error.
- View the log files generated from the nodes in the previous step, and you will see the below message at the start of the file :
isi_version_checker encountered an unexpected error.
‘utf8’ codec can’t decode bytes in position 0-2: invalid data
This means there is a file under “/” [except /ifs]on the node that throws the upgrade error which is not encoded with UTF-8 or there are files under ‘/’ which have a large UID and GID.
Check directories, say “/tmp” “/var/tmp“. A simple “ls -al” under these directories should help check files with large UID and GID’s.
# ls -al /tmp
-rw——- 1 0 0 – 0 Apr 7 10:55 apache2.py.lock
-rw-r–r– 1 0 0 – 0 Apr 7 09:32 cpool_stat_cpool_query_stat_map_
-rw-r–r– 1 0 0 – 0 Apr 7 09:32 cpool_stat_cpool_stat_map_
-rw——- 1 0 0 – 0 Apr 7 10:55 generic_copy.py.lock
-rw——- 1 0 0 – 0 Apr 7 11:14 hostupdate.py
-rw-r–r– 1 0 0 – 0 Apr 7 10:55 isi_flexnet_d_sem_agg_update.key
-rw-r–r– 1 0 0 – 0 Apr 7 10:53 isi_pkg_outcmdck1
-rw-r–r– 1 0 0 – 0 Apr 7 10:49 isi_pkg_outcmderr
-rw——- 1 1000002 1000000 – 5045 Jul 23 08:09 krb5cc_1000002 >> Files with large UID and GID
-rw——- 1 1000376 1000000 – 4677 Jul 13 07:44 krb5cc_1000376
-rw——- 1 0 0 – 0 Apr 7 09:32 metadata-generator
-rw——- 1 0 0 – 0 Apr 7 11:10 nsswitch.py.lock
-rw——- 1 0 0 – 0 Jul 21 07:24 ntpd.py.lock
-rw——- 1 0 0 – 524 Apr 7 10:54 portmap.
To check if there are files with non printable characters (files not encoded with UTF-8) use the below command, the command will take time to time to return an output :
# isi_for_array -s “find / -type f -not -path ‘/ifs/*’| grep -E ‘.*[^[:print:]].*'”
Once you identify the files move it under “/ifs/data/Isilon_Support” with user permission and retry the upgrade.
# isi upgrade retry-last-action –nodes=all
[B]If there is an assessment which is giving an error, follow the below steps:
- Generate the assessment report by running ” # isi_upgrade_logs -a “ :
# isi_upgrade_logs -a
For Assessment with Upgrade ID [2018-07-26T09:52:19.000Z]
Assessment errors found attempting to find and extract logs from archive
Assessment Archive File found here [/ifs/.ifsvar/upgrade/archive/UpgradeArchive-2018-07-26T09:57:30.000Z.tgz]
Extracting [UpgradeArchive-2018-07-26T09:57:30.000Z/Agent-32-000e1ea67780a95c3c57e400d1411b40b520.log] to [/ifs/upgrade_error_logs/Agent-32-000e1ea67780a95c3c57e400d1411b40b520.log]
Extracting [UpgradeArchive-2018-07-26T09:57:30.000Z/32-000e1ea67780a95c3c57e400d1411b40b520/logs/hooklogs/pre-upgrade-optional/isi_upgrade_checker.log] to [/ifs/upgrade_error_logs/32-000e1ea67780a95c3c57e400d1411b40b520/logs/hooklogs/pre-upgrade-optional/isi_upgrade_checker.log]
Extracting [UpgradeArchive-2018-07-26T09:57:30.000Z/Assessment-summary.report] to [/ifs/upgrade_error_logs/Assessment-summary.report]
Suggest running ‘grep -ilr error /ifs/upgrade_error_logs/*’ to find the log files with errors
- Follow the suggestion in the assessment report, run ‘grep -ilr error /ifs/upgrade_error_logs/*‘ to find the log files with errors, and you can view the assessment summary report [cat /ifs/upgrade_error_logs/Assessment-summary.report] to verify which node the upgrade assessment is pointing the error.
- While viewing the log files if you see the error “‘utf8′ codec can’t decode bytes“, follow the steps mentioned in [A] on the node on which the assessment fails.
UTF–8 (U from Universal Character Set + Transformation Format—8-bit) is a character encoding capable of encoding all possible characters (called code points) in Unicode.