FX2S FC640 loses storage controllers

Question

I have a FX2S with a FD332, 2x FC640, all firmwares current. The FD332 is dual controller set to split mode, each FC640 gets 8 drives. FC640#1 I have left running, but FC640#2 I have rebooted several times or sometimes left it off just doing some testing. At one point I noticed #2 had no storage, it had booted to the OS from the BOSS card SSDs, and could not access the array on the FD332. I checked the iDrac and it now does not even list the BOSS card as available storage, it thinks there are no controllers at all. I just turned both nodes off (chassis left on), and turned them on a couple hours later, both have no controllers, but they boot from the BOSS, but have no access to the FD332. If I boot in to the bios and go to Device settings, just the list of network cards is there, when it is working properly the list includes the BOSS RAID config and FD332 config, but even in this state of missing the BOSS from setup it still boots from the BOSS. Often even turning the chassis off doesnt work, *usually* if I pull the power cords everything comes right back on with all controllers visible.

On the faulty one :

On the working one (which has also failed occasionally):

Chassis config:

DELL-Erman O · Answer

Hi,

It looks like your system is losing the host storage information. When this happens, you might see error codes RAC0501 and RAC0503. Here’s what you can do:

You can open an SSH session and run racadm racreset.https://dell.to/4gtxotT
Please power down the system and drain flea power.
Normally all firmware is updated to the latest version. However you said that they are already up to date.
You can perform a repurpose and select the iDRAC and LCC options. https://dell.to/49Uzd0G

If you’ve tried all these steps and the problem still persists, it might be time to consider a hardware issue.

mcrholdings · Answer

@DELL-Erman O​ There are no errors logged in the CMC logs or iDrac logs, I ran the Repurpose and chose everything on the top section, accept formatting drives on the bottom, I gave it 20 minutes and eventually the FC640 powered off, when I powered it on again the iDrac had defaulted to 192.168.0.120, and the default password.... but after opening the iDrac Configuration > Bios Settings, several of my original settings were still there. I feel like the latest firmware has some bugs.  Your racadmin command I have done vie the iDrac webpage under Troubleshooting > Reboot iDrac. Upon bootup it had no storage... but the BIOS settings in the iDrac showed the BOSS card array as the boot device, but iDrac shows no controllers no drives. I booted it once and turned off, booted OS again, OS detects all drives on all controllers (iDrac still shows no controllers). This is after pulling the power cords on the chassis, and even using the CMC to Reseat each sled. I feel like the Lifecycle controller is not properly collecting system inventory, and it did not properly Repurpose the system, and this behavior is on both nodes, so hardware malfunctions are possible, but less so on multiple nodes, I have another FX2+nodes coming and I suspect they will behave the same on the latest firmware... if there is a firmware bug what are the steps to report that to the firmware dev team?

PowerEdge Hardware General

FX2S FC640 loses storage controllers

Was this post helpful?