Unsolved
1 Rookie
•
2 Posts
0
688
June 20th, 2023 04:00
iDRAC 7 complete failure after FW update. How to update CPLD or "management engine"?
Hi,
we have issues with the iDRAC on our R420 (out of support of course).
After upgrading the iDRAC 7 firmware(*) to 2.65.65.65 I receive below error messages after reboot in POST and the iDRAC card seems to be completely dead:
"initializing iDRAC...." --> "no IDRAC found", later then:
"Warning: iDRAC initialization error. Continuing to boot... unexpected power loss" etc.
The OS (Debian) boots fine of course.
Consequently, in BIOS there is no way to access iDRAC settings or (re-)activate the LCC. The only available information: in BIOS, system information it tells "iDRAC settings version: 1.65.65". This version also appears when I unplug the iDRAC card and reboot into BIOS. So, this can't be the card's actual firmware version, more like the driver version inside the BIOS, when it tries to talk to iDRAC...
The iDRAC card (DP/N: 02827M) has an SD-card slot and sits in the "iDRAC enterprise" slot of the PCIe riser card, and that one of course attached to the MB. The blue "i" LED lights up during POST, but apart there is no sign of life. The network port link LED stays off.
There is an INTEL network card, also plugged into the PCIe riser card, which works just fine. So, the riser card itself is fine.
After having read similar threads in this forum, this is what I tried, in order to recover:
- complete power drain
- reset NVRAM via jumper
- reset iDRAC via "i"-button
- prepared DELLs platform specific, bootable ISO, but the Linux fails to recognize the iDRAC and hangs before I could ever upgrady anything
- I managed to upgraded the BIOS via Bootmanager-->System utilities-->BIOS flash to the latest 2.9.0
- FAT-formatted an SD-card and tried different versions of the "firmimg.d7". No yellow blinking LED during all these attempts. It seems it doesn't start, as documented the iDRAC autoloader / flash procedure.
- got myself a new iDRAC card (2d hand of course)
- exactly the same symptoms with the new card
- even more: if I completely remove the iDRAC card, the system behaves the same: very long period of "iDRAC initialization..." always ends with errors
- I also went through above steps with the new card to no avail
- I also tried to downgrade the BIOS (back from 2.0.9 to 1.6.7), assuming the iDRAC firmware never updated and there might be a mismatch between (latest) BIOS and current --in fact unknown-- iDRAC version. Frustrating result: "your hardware configuration is not compatible with this firmware". What??
I can't believe that the iDRAC firmware upgrade burned the mainboard. I mean, this is a separate piece of hardware... The system seems to ignore: my original iDRAC, a different iDRAC card or just no card at all. So, the error sits on the mainboard. As a result of a FW update?
That brings me to the "CPLD" and "Management Engine"- what about this?
Is this part of the BIOS, is it related to the communication between BIOS and iDRAC and if yes, how would I update that piece of firmware?
Any help or suggestions would be very much appreciated!
best
Peter
(*) BTW: the upgrade finished with no visible error message. I know, I should have better upgrade BIOS and iDRAC stepwise, instead of using the latest available... afterwards one is always wiser.
DELL-Charles R
Moderator
•
4.5K Posts
0
June 20th, 2023 10:00
Hello PeterHH609,
You have done quite a bit of troubleshooting and all are things I would recommend. You even tried the firmimg.d7 file.
The iDRAC in integrated to the system board. The part you are changing is an Optional daughter card for a Dedicated iDRAC port. Removing or changing this does not remove the iDRAC.
There is no update for the CPLD so you are at latest version.
See if you can enable the LCC this way:
-boot to F2 -> System Setup -> iDRAC Settings -> Lifecycle Controller to check status
-Choose: Lifecycle Controller set to Enabled
-Collect System Inventory on Restart set to Enabled.
-Cancel Lifecycle Controller Actions set to Yes.
Since the iDRAC was at 1.66.65 you would want to step that up.
Using the list below working bottom to top:
>iDRAC with Lifecycle Controller v. 2.65.65.65
>2.63.60.62, A00
27 May 2019
Urgent
>2.50.50.50, A00
26 Sep 2017
Recommended
>2.40.40.40, A00
12 Oct 2016
Recommended
>2.21.21.21, A00
09 Nov 2015
Urgent
>2.10.10.10, A00
05 Apr 2015
Recommended
1.66.65, A00
01 Dec 2014
Recommended
Have you tried the steps in this article:
PowerEdge: iDRAC Recovery Procedure with firmimg.d7
https://dell.to/3p66y5X
If none of those steps help then you would be replacing the system board.
There should be a sticker near CPU2 socket and DIMM sockets with a DP/N number
PeterHH609
1 Rookie
•
2 Posts
0
June 22nd, 2023 01:00
Hello Charles,
thanks a lot for your support.
> See if you can enable the LCC this way:
> -boot to F2 -> System Setup -> iDRAC Settings -> Lifecycle Controller to check status
I don't get that far. As soon as I click "iDRAC settings" there is an error message. No further options available and therefor no way to activate LCC.
> Have you tried the steps in this article:
> PowerEdge: iDRAC Recovery Procedure with firmimg.d7
> https://dell.to/3p66y5X
yes, as written previously I tried to flash with the SD card and firmimg.d7. But I would say there is no communication at all between daughter card and the mainboard. There is no yellow blinking when I insert the card.
Note: I'm using a 256GB SD card with a 512MB primary FAT partition. Might this cause issues?
Second thought: if the mainboard no longer communicates with the daughter card and so the dedicated iDRAC network port is dead: should I try the TFTP flash procedure using the onboard/embedded NIC? Is this supposed to work?
> Since the iDRAC was at 1.66.65 you would want to step that up.
> Using the list below working bottom to top:...
Can you confirm -to start stepwise with iDRAC beginning with the earliest version- even if my BIOS is on the latest version? It seems I can't downgrade anymore, as explained above
Thanks and best regards
Peter
DELL-Joey C
Moderator
•
3.9K Posts
0
June 22nd, 2023 02:00
Hi @PeterHH609,
Perhaps you can try a lower capacity SD card, probably 32GB or 64GB. I vaguely remember other users did try and it's because of the capacity.
I say give it a try, if it's a success. Share the experience.
What version of iDRAC firmimg.d7 did you use? Try different versions. You did mention BIOS is showing iDRAC version, 1.65.65. Try the image file version 1.66.65.
Try reset the iDRAC through iSM, but I'm unsure if Debian would be the same as Linux procedure/commands, but worth a try: https://dell.to/42VmIgi. The iSM download is here: https://dell.to/3p9YpgR