Powervault ME4024 firmwares update issues

Hi,

Has any of you have had issues upgrading the controllers firmwares on their ME4 units ?

We have 4 Powervault ME4024 head unit that are all running GT280R004-01. I am trying to update to GT280R008-04 and I am having issues. I've tried 2 units so far and both have given me troubles.

I have dual controllers on all my units for redundancy. There, I enable PFU in the firmware update dialog so that both controllers will update one by one.

Out of the 2 units I've tried, I was able to update one successfully, but it wasn't as easy as the installation instructions told me.

Before doing any firmware update, I always do the followings :

• Check that the system health is OK using a check firmware-upgrade-health on both controllers. Every time, it says PASS.
• I also do a show controllers on both controllers to make sure that both controllers can see each other.
• show system on both controllers to make sure that the health is OK. It is always OK.
• show unwritable-cache on both controllers to ensure that the cache is empty. It is always empty.

So once all this is done, I expect my system to be Healthy and ready for upgrade. Using the GUI, I invoke the Upgrade firmware wizard, load the file and select FRU to update both controllers.

It takes a few minutes and then will complain saying : Firmware update progress: System health is insufficient to support firmware update. These are the events :

2021-04-21 09:18:31 [139] #A3653: ME4024 Array SN#00C0FF501D68 Controller A INFORMATIONAL The Management Controller booted up. MC firmware version: GXM280R004-01
2021-04-21 09:18:31 [363] #A3652: ME4024 Array SN#00C0FF501D68 Controller A INFORMATIONAL Firmware versions match those in the firmware bundle. (controller: A)
2021-04-21 09:17:05 [623] #A3651: ME4024 Array SN#00C0FF501D68 Controller A INFORMATIONAL Management Controller configuration parameters were set.
2021-04-21 09:06:09 [237] #A3650: ME4024 Array SN#00C0FF501D68 Controller A ERROR Firmware update progress: System health is insufficient to support firmware update.
2021-04-21 09:04:57 [237] #A3649: ME4024 Array SN#00C0FF501D68 Controller A INFORMATIONAL Firmware update progress: System health is sufficient to support firmware update.

You can clearly see that the controllers were healthy prior the upgrade and something went wrong while doing it.

Once this happens, the management interface of both controllers reboot without killing the controllers and the iSCSI connections. So this is good...I guess.

Now the controllers are in a weird state. If I issue a show controllers command on both, one of them is not seeing the other controllers settings. On my first ME4, what I eventually did is rebooting manually each controller using restart sc a and few minutes after it has come back, issue a restart sc b. After all this, my controllers were back online and I was able to do a firmware upgrade successfully on that ME4. Before doing restart sc and tried doing restart mc, but only restarting sc was able to fix this.

My second ME4 is at the point where it has failed and both controllers don't see each other configs. iSCSI still works and this is a production unit that I don't want to mess too much with since it has already failed once. The first ME4 was more a test unit and could be mess with during the day.

So am I doing something wrong ? Why is it always failing the first time and I have to manually reboot each controllers before attempting an upgrade ?

I know you guys can't troubleshoot my ME4 remotely and that's not really what I am looking for. I'm more looking to see if the procedure that I am doing is right and if others have had issues in the past with their ME4 firmwares upgrades.

Thanks,

Guillaume.