*************************************************************************** ** ** ** Software Information for vSphere 7.0, vSphere 8.0 ** ** ** ** VMware Multi-Pathing plug-in for ETERNUS AX/HX ** ** 1.0.0 ** ** ** ** Dec 2024 FUJITSU LIMITED ** *************************************************************************** Preface This guide explains installation/uninstallation procedures and notes of VMware Multi-Pathing plug-in for ETERNUS AX/HX. Abbreviations In this document, the disk storage system supported by this product is referred to as "ETERNUS storage system". In this document, Broadcom's Knowledge Base is described as "KB". More information on each KB can be found on the Broadcom website. Reference - Fujitsu Disk storage system ETERNUS AX/HX series https://www.fujitsu.com/global/products/computing/storage/all-flash-hybrid.html Trademarks Microsoft, Windows, Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and other countries. VMware is a registered trademark of VMware,Inc, in the United States and other countries. All other company/product names mentioned herein maybe trademarks or registered trademarks of their respective holders and are used for identification purposes only. High Safety Required This Product is designed, developed and manufactured as contemplated for general use, including without limitation, general office use, personal use, household use, and ordinary industrial use, but is not designed, developed and manufactured for use in situations with accompanying fatal risks or dangers that, unless extremely high safety is secured, could lead directly to death, personal injury, severe physical damage or other loss (hereinafter "High Safety Required Use"), including without limitation, nuclear reaction control in nuclear facility, aircraft flight control, air traffic control, mass transport control, medical life support system, and missile launch control in weapon systems. Do not use this Product for High Safety Required Use without securing the sufficient safety level required. If you wish to use this Product for High Safety Required Use, please consult with our sales representative before such use. 1. Introduction 1-1. VMware Multi-Pathing plug-in for ETERNUS AX/HX VMware Multi-Pathing plug-in for ETERNUS AX/HX is offered as a sub plug-in of Native Multipathing Plugin (NMP) of the VMware standard used when the multipath is connected with the ETERNUS storage system. VMware Multi-Pathing plug-in for ETERNUS AX/HX is Storage Array Type Plug-in(SATP) that controls error handling corresponding to the ETERNUS storage system. 1-2. Audience This document is intended for VMware administrators who are using ETERNUS storage system as an infrastructure of virtualization deployments. It is assumed that the reader has a general knowledge of VMware and ETERNUS storage systems. 1-3. Release Information 1-3-1. Modification History +----------------+--------------------------------------------------+ | version | Modification effect | +----------------+--------------------------------------------------+ | 1.0.0 | New Release | +----------------+--------------------------------------------------+ 1-3-2. Updates +--------------+---------+------------------------------------------+ | date | version | changes | +--------------+---------+------------------------------------------+ | July 31, 2024| 1.0.0 | first version | +--------------+---------+------------------------------------------+ | Dec 26, 2024 | 1.0.0 | - This document was corrected. | | | | - Added 7. Notes Regarding to PDL | | | | (Permanent Device Loss) | +--------------+---------+------------------------------------------+ 2. Requirements This section describes support environment and necessary resources. Please be sure to check this section before installing VMware Multi-Pathing plug-in for ETERNUS AX/HX. 2-1. Supported Environment Please visit the following product's web site for the latest information (supported models and operating systems, important notes, etc.). https://www.fujitsu.com/global/support/products/computing/storage/download/vmware-mp-plug-in-axhx.html 2-2. Required Disk Capacity for Installation The following table shows the static disk space requirements for each directory used to install VMware Multi-Pathing plug-in for ETERNUS AX/HX. +----------+----------------+----------------+----------------------+ |No |Directory |Disk Capacity |Note | +----------+----------------+----------------+----------------------+ |1 |/usr |0.1MB | | +----------+----------------+----------------+----------------------+ |2 |/etc |0.1MB | | +----------+----------------+----------------+----------------------+ 3. Product configuration The product configuration of VMware Multi-Pathing plug-in for ETERNUS AX/HX is as follows. VMwareVAMPAHX_1.0.0 +-- FJT_satp_eternus_AHX_100.zip File used for installation +-- readme.txt Software Information(JP) +-- readme_en.txt Software Information(EN) 4. Installation and Uninstallation procedure 4-1. Installation This section describes three methods to install VMware Multi-Pathing plug-in for ETERNUS AX/HX. You may select method based on your environment. - Using ESXCLI - Using esxcli command - Using vSphere Lifecycle Manager All commands must be run as root user in this section. Stop GuestOS or move GuestOS to other hosts before it installs. Specify the IP address of ESXi host at of the following procedures. The procedure to install the upgrade of VMware Multi-Pathing plug-in for ETERNUS AX/HX is the same as a new installation. 4-1-1. Installation VMware Multi-Pathing plug-in for ETERNUS AX/HX 4-1-1-1. Installation using ESXCLI This section describes how to install using ESXCLI. 1) Copy the installation file to a local directory on your ESXi host. FJT_satp_eternus_AHX_100.zip is preserved in "/tmp" in the example of executing the following commands. 2) Open a command prompt and change to the directory in which ESXCLI is installed only Windows. For default, installed pass is as follows. >cd C:\Program Files (x86)\VMware\esxcli 3) Run the following command to place the ESXi host into maintenance mode. >esxcli --server system maintenanceMode set --enable=1 4) Run the following command to install VMware Multi-Pathing plug-in for ETERNUS AX/HX into the ESXi host. >esxcli --server software vib install -d /tmp/FJT_satp_eternus_AHX_100.zip 5) Run the following command to reboot the ESXi host. >esxcli --server system shutdown reboot -d 10 -r satpinstall 6) Run the following command to confirm VMware Multi-Pathing plug-in for ETERNUS AX/HX installation. >esxcli --server software vib list example: Name Version Vendor Acceptance Level Install Date -------------------- --------- ------- ---------------- ------------ fjt-satp-eternus-ahx 1.0.0-0 FJT VMwareAccepted 2024-05-31 7) Run the following command to confirm it is a setting that uses FJT_SATP_ETERNUS_AHX. >esxcli --server storage nmp satp rule list --satp FJT_SATP_ETERNUS_AHX example: Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description -------------------- ------ ------- ------------- ------ --------- ------- ---------- ------------- ----------- ----------- ----------- FJT_SATP_ETERNUS_AHX NETAPP LUN C-Mode user VMW_PSP_RR 8) Run the following command to confirm it is a setting that uses SATP. Please confirm FJT_SATP_ETERNUS_AHX is displayed in "Storage Array Type:". >esxcli --server storage nmp device list example: naa.600a098038314c70452b567743536570 Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314c70452b567743536570) Storage Array Type: FJT_SATP_ETERNUS_AHX Storage Array Type Device Config: {FJT_SATP_ETERNUS_AHX device config options dtime=20 mtime=3 limit=6} Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0} Path Selection Policy Device Custom Config: Working Paths: vmhba6:C0:T0:L7 Is USB: false 9) Run the following command to exit maintenance mode. >esxcli --server system maintenanceMode set --enable=0 4-1-1-2. Installation using esxcli command This section describes how to install using esxcli command. Perform the following steps on the ESXi Tech Support Mode. 1) Copy the installation file to a local directory on your ESXi host. FJT_satp_eternus_AHX_100.zip is preserved in "/tmp" in the example of executing the following commands. 2) Place the ESXi host into maintenance mode. Use vSphere Client or Host Client. 3) Run the following command from the local directory where you copied the FJT_satp_eternus_AHX_100.zip file to install VMware Multi-Pathing plug-in for ETERNUS AX/HX into the ESXi host. # esxcli software vib install -d /tmp/FJT_satp_eternus_AHX_100.zip 4) Reboot the ESXi host. Use vSphere Client or Host Client. 5) Run the following command to VMware Multi-Pathing plug-in for ETERNUS AX/HX installation is confirmed. # esxcli software vib list example: Name Version Vendor Acceptance Level Install Date -------------------- --------- ------ ---------------- ------------ fjt-satp-eternus-ahx 1.0.0-0 FJT VMwareAccepted 2024-05-31 6) Run the following command to confirm it is a setting that uses FJT_SATP_ETERNUS_AHX. # esxcli storage nmp satp rule list --satp FJT_SATP_ETERNUS_AHX example: Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description -------------------- ------ ------- ------------ ------ --------- ------- ---------- ------------- ----------- ----------- ----------- FJT_SATP_ETERNUS_AHX NETAPP LUN C-Mode user VMW_PSP_RR 7) Run the following command to confirm it is a setting that uses SATP. Please confirm FJT_SATP_ETERNUS_AHX is displayed in "Storage Array Type:". # esxcli storage nmp device list example: naa.600a098038314c70452b567743536570 Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314c70452b567743536570) Storage Array Type: FJT_SATP_ETERNUS_AHX Storage Array Type Device Config: {FJT_SATP_ETERNUS_AHX device config options dtime=20 mtime=3 limit=6} Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0} Path Selection Policy Device Custom Config: Working Paths: vmhba6:C0:T0:L7 Is USB: false 8) ESXi host exit maintenance mode. Use vSphere Client or Host Client. 4-1-1-3. Installation using vSphere Lifecycle Manager Refer to the "Managing Host and Cluster Lifecycle". Reboot the ESXi host after installation. 4-2. Uninstallation This section describes two methods to uninstall VMware Multi-Pathing plug-in for ETERNUS AX/HX. You can use one of two methods for your environment. - Using ESXCLI - Using esxcli command Stop GuestOS or move GuestOS to other hosts before it uninstalls. Specify the IP address of ESXi host at of the following procedures. 4-2-1. Uninstallation VMware Multi-Pathing plug-in for ETERNUS AX/HX 4-2-1-1. Uninstallation using ESXCLI This section describes how to uninstall VMware Multi-Pathing plug-in for ETERNUS AX/HX using ESXCLI. Perform the following steps on server in which ESXCLI is installed. 1) Open a command prompt and change to the directory in which ESXCLI is installed only Windows. For default, installed pass is as follows. >cd C:\Program Files (x86)\VMware\esxcli 2) Run the following command to place the ESXi host into maintenance mode. >esxcli --server system maintenanceMode set --enable=1 3) Run the following command to uninstall. >esxcli --server software vib remove -n fjt-satp-eternus-ahx 4) Run the following command to confirm the package is not displayed. >esxcli --server software vib list 5) Run the following command to release the setting that uses FJT_SATP_ETERNUS_AHX. >esxcli --server storage nmp satp rule remove --satp="FJT_SATP_ETERNUS_AHX" --vendor="NETAPP" --model="LUN C-Mode" --psp="VMW_PSP_RR" After that, the following error occur depending on the case. Error deleting SATP rule: No matching user rule found for SATP FJT_SATP_ETERNUS_AHX matching vendor NETAPP model LUN C-Mode claim Options PSP VMW_PSP_RR and PSP Options If the error occur, please execute the following (7) command, and there is no display FJT_SATP_ETERNUS_AHX information, then that is no problem. >esxcli --server storage nmp satp rule list 6) Run the following command to reboot the ESXi host. >esxcli --server system shutdown reboot -d 10 -r satpinstall 7) Run the following command to confirm the setting has been deleted. >esxcli --server storage nmp satp rule list --satp FJT_SATP_ETERNUS_AHX 8) Run the following command to confirm it is a return to standard VMW_SATP_ALUA. Please confirm VMW_SATP_ALUA is displayed in "Storage Array Type:". >esxcli --server storage nmp device list example: naa.600a098038314c70452b567743536570 Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314c70452b567743536570) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1000,TPG_state=ANO}{TPG_id=1001,TPG_state=AO}} Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0} Path Selection Policy Device Custom Config: Working Paths: vmhba6:C0:T0:L7 Is USB: false 9) Run the following command to exit maintenance mode. >esxcli --server system maintenanceMode set --enable=0 4-2-1-2. Uninstallation using esxcli command This section describes how to uninstall VMware Multi-Pathing plug-in for ETERNUS AX/HX using esxcli command. Perform the following steps on the ESXi host Tech Support Mode. 1) Place the ESXi host into maintenance mode. Use vSphere Client or Host Client. 2) Run the following command to uninstall. # esxcli software vib remove -n fjt-satp-eternus-ahx 3) Run the following command to confirm the package is not displayed. # esxcli software vib list 4) Run the following command to release the setting that uses FJT_SATP_ETERNUS_AHX. # esxcli storage nmp satp rule remove --satp="FJT_SATP_ETERNUS_AHX" --vendor="NETAPP" --model="LUN C-Mode" --psp="VMW_PSP_RR" After that, the following error occur depending on the case. Error deleting SATP rule: No matching user rule found for SATP FJT_SATP_ETERNUS_AHX matching vendor NETAPP model LUN C-Mode claim Options PSP VMW_PSP_RR and PSP Options If the error occur, please execute the following (6) command, and there is no display FJT_SATP_ETERNUS_AHX information, then that is no problem. # esxcli storage nmp satp rule list 5) Reboot the ESXi host. Use vSphere Client or Host Client. 6) Run the following command to confirm the setting has been deleted. # esxcli storage nmp satp rule list --satp FJT_SATP_ETERNUS_AHX 7) Run the following command to confirm it is a return to standard VMW_SATP_ALUA. Please confirm VMW_SATP_ALUA is displayed in "Storage Array Type:". # esxcli storage nmp device list example: naa.600a098038314c70452b567743536570 Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314c70452b567743536570) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1000,TPG_state=ANO}{TPG_id=1001,TPG_state=AO}} Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0} Path Selection Policy Device Custom Config: Working Paths: vmhba6:C0:T0:L7 Is USB: false 8) ESXi host exit maintenance mode. Use vSphere Client or Host Client. 5. About Path Recovery If the diagnostic continues to respond normally for more than 20 minutes, return the path state from dead to online. Also, if the online to dead transition is repeated 6 times within 3 hours from the first dead transition, it is judged to be an unstable path and the status changes to "fataldead". "fataldead" is not automatically recovered by diagnostics. 5-1 How to recover from fataldead After correcting the error, restore the path as follows. 1) Check for fataldead in the following ways:. # esxcli storage nmp path list or # esxcli storage nmp path list -d naa.XXXXXXXXXXXXXXX example: # esxcli storage nmp path list -d naa.600a098038314c70452b567743536631 fc.2000f4e9d45a1abf:2100f4e9d45a1abf-fc.2003d039eaaaf83d:2005d039eaaaf83d-naa.600a098038314c70452b567743536631 Runtime Name: vmhba6:C0:T0:L17 Device: naa.600a098038314c70452b567743536631 Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314c70452b567743536631) Group State: dead Array Priority: 0 Storage Array Type Path Config: {FJT_SATP_ETERNUS_AHX path vmhba6:C0:T0:L17 config info: vendor NETAPP model LUN C-Mode FATALDEAD} Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration. A path in the fataldead state will show FATALDEAD in "Storage Array Type Path Config". 2) Release fataldead with the following command: # esxcli storage nmp satp generic pathconfig set -p "Runtime Name" --config="fataldead=off" -p must be the "Runtime Name" of 1). example: # esxcli storage nmp satp generic pathconfig set -p vmhba3:C0:T2:L0 --config="fataldead=off" 3) Verify that the fataldead state of the path is cleared with the following command: # esxcli storage nmp path list -p vmhbaX:X:X:X example: # esxcli storage nmp path list -p vmhba3:C0:T0:L17 fc.2000f4e9d45a1abf:2100f4e9d45a1abf-fc.2003d039eaaaf83d:2005d039eaaaf83d-naa.600a098038314c70452b567743536631 Runtime Name: vmhba6:C0:T0:L17 Device: naa.600a098038314c70452b567743536631 Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314c70452b567743536631) Group State: dead Array Priority: 0 Storage Array Type Path Config: {FJT_SATP_ETERNUS_AHX path vmhba6:C0:T0:L17 config info: vendor NETAPP model LUN C-Mode} Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration. Notes: * The Group State immediately following the command is dead. Transition to active at next diagnostic. * Only the Runtime Name specified by -p will be fataldead. Do 1) to 3) for each. 6. Message Output to /var/log/vmkernel.log. Message: FJT_SATP_ETERNUS_AHX: vmhbaX:CX:TX:LX XX->YY Meaning: Received XX host error and converted to YY. Action: If a path transition occurs, investigate the cause of the XX error. Message: FJT_SATP_ETERNUS_AHX: vmhbaX:CX:TX:LX is fataldead Meaning: Transition to fataldead successful. Action: Investigate the cause of the error. After removing the cause, release fataldead as described in "5-1 How to recover from fataldead". 7. Notes - About updating the ONTAP of ETERNUS storage system Please execute the following steps. 1) Run the 5 -1 2) Command on all LUNs and paths. or the following command: # for i in `esxcfg-mpath -L | grep naa.XXXXXXXXXXX | awk '{print $1}'`; do esxcli storage nmp satp generic pathconfig set -p $i --config="fataldead=off" ; done naa.XXXXXXXXXX should specify the common part of the farm replacement unit. 2) Then, Upgrade the firmware. In case of changing to dead state during upgrading ONTAP, 1)'s step is prevented from changing to fataldead state when reaching to recurrence count. The current recurrence count will be cleared by 1)'s command. - Timeout on Guest OS If activating virtual disk of RDM on GuestOS of Linux, set the timeout period to at least 40 seconds. Ex. Red Hat Enterprise Linux7) Create "/etc/udev/rules.d/99-rdm-scsi-udev.rules" file. Add the following three lines. ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NETAPP ", ATTRS{model}=="LUN C-Mode ", RUN+="/bin/sh -c 'echo 40 >/sys$DEVPATH/timeout'" Ex. Red Hat Enterprise Linux8, and Red Hat Enterprise Linux9) Create "/etc/udev/rules.d/99-rdm-scsi-udev.rules" file. Add the following three lines. ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NETAPP ", ATTRS{model}=="LUN C-Mode ", ENV{DEVTYPE}=="disk", RUN+="/bin/sh -c 'echo 40 >/sys$DEVPATH/device/timeout'" If you are not using an RDM to show the LUN to the guest, refer to the KB below to set the timeout value. https://knowledge.broadcom.com/external/article?legacyId=1009465 If you do not change the timeout value, the guest may detect the timeout before the host detects the timeout, causing an I/O error or LUN unavailability. - Regarding to upgrading VMware ESXi If the using version of VMware Multi-Pathing plug-in for ETERNUS AX/HX doesn't adapt VMware ESXi's version, please do the following. 1. First, Un-install VMware Multi-Pathing plug-in for ETERNUS AX/HX. 2. Upgrade VMware ESXi. 3. After that, install the compatible version's VMware Multi-Pathing plug-in for ETERNUS AX/HX, and setup it. (Please refer the install/uninstall sequences.) - Regarding to PDL(Permanent Device Loss) VMware Multi-Pathing plug-in for ETERNUS AX/HX doesn't become PDL. If a hardware trouble occurred, VMware Multi-Pathing plug-in for ETERNUS AX/HX detect APD. Then, even you reduce LUN, the path state keep on active. If so, you may do Rescan Storage, and LUN will be decreased on device list.