PowerEdge R620 iDrac Failure

I should add that this is the error message which I receive in the setup bios

SWC0700: iDRAC is not ready. The configuration values cannot be accessed.

Using the SLI boot iso, I get the following error...

[sliuser@localhost sbin]$ sudo ./racadm racresetcfg
ERROR: Unable to perform requested operation

The result code of racadm is 23

[sliuser@localhost sbin]$ sudo ./racadm racreset ; echo $?
ERROR: Unable to perform requested operation
23

Bueller? Bueller? Any suggested feedback?

I read there’s an iso media I can boot from that will update the server. I’m not sure where to find it for the r620. The few links that I see reference it go to a Dell.TO domain, which doesn’t work.

I bought a motherboard but in worried this will happen again. I’d prefer to save the motherboard as a backup - but it seems so fragile now. What precautions can we take moving forward?

Is there a way to access IDRAC via the serial port? As documented here, I can’t seem to reproduce the result: https://buildingtents.com/2014/04/24/idrac6-recovery-through-tftp-and-serial/

I set redirection to com1 because the server has only 1 com port. I rebooted and I can see the console output on my serial terminal. However, I am not seeing any IDRAC configuration menus or anything else. 

Can someone from Dell help with the dot TO links in the forums? I find all forum posts have suggestions for my issue but they link to dot TO Dell webserver that doesn’t work. 

first, the certificate is expired, and second there’s an error on the site.

we really need to get this server back online. Also, we need to understand what happened to prevent it from happening again. Did pressing the three front lcd buttons actually brick an entire server? 

Uhm, I wrote a lengthy response on my laptop and hit save. It’s about how I used the uart to debug busybox and see the EMMC has degraded. Hitting save now gives me this…

For whatever reason I keep getting access denied on my laptop for the forum here. So I have to use my phone to post details. 

It appears Dell choose emmc with a low write count. Essentially Dell has put a hardware expiry on all IDRAC servers. Because IDRAC writes logs and abuses the emmc, the expiry of a server can pretty much be predicted. 

Hello overall when you can't access iDRAC, we eventually see mainboard replacement in the end. You can either opt for it or choosing to not use iDRAC since your system is out of warranty. You can also get a second opinion by raising an official ticket through https://www.dell.com/support/incidents-online/en-us/contactus/dynamic
Respectfully,

Or, I fix it by building my own IDRAC and lifecycle image, configure uboot to boot from the sd card, and not have to pay Dell for a new server. Dell should be ashamed of yourselves for adding a life expectancy to your servers via emmc write limits. I’ll publish a guide so people can continue using their hardware without needing to purchase additional Dell replacements. 

Hopefully, this will be useful to the tens of thousands of people buying new motherboards or throwing out their PowerEdge servers after experiencing iDRAC EMMC failures.  This allows running iDRAC from the SD card rather than the internal EMMC. This is how Dell should have shipped the servers: by running the iDRAC off the SD card.

This document outlines how to boot the iDRAC system from an SD card on a server where the onboard eMMC is corrupt. The process involves configuring U-Boot environment variables, flashing the SD card with a firmware image using TFTP, and modifying the squashfs filesystem to reference the SD card (mmcblk1) instead of the corrupted eMMC (mmcblk0). This ensures the iDRAC system can run entirely from the SD card and avoids any failed interactions with the broken eMMC.

UBoot is the boot loader that loads the iDRAC Linux from partition 2 of the EMMC or SD.

*Note: always need to reboot and drain the flea power before booting off new sd card

            (until you hear a click from the rear of the server)

*Note: SD card that is 32gb seems to work best AND make sure you check uboot log that the capacity is not a negative number. A bad sd card or incorrect size gives a negative reading and it'll never work. Can't have any bad sectors so use a new and high quality card. i'm using samsung evo's.

*Note: If you don't edit the squashfs to swap the mmcblk0 references from the scripts, it'll still boot off the SD but the scripts will mount the mmcblk0 partitions. That may not work for you if the EMMC is really screwed

*Note: Some conversations on the internet show to use the uboot "recover" command. The recover command always seems to want to write to the internal EMMC even if the SD card is set as target, as it doesn't work as expected - so use my instructions below.

# ------------------------------------------------------------------------------------------------------------
# Boot into U-Boot console fro the UART at 115,200 N-8-1
#
# Environment variables. Set these for your network
# Connect ethernet to NIC1 (don't need to use the iDRAC port)
# ------------------------------------------------------------------------------------------------------------

setenv ipaddr '10.0.0.21'
setenv netmask '255.255.255.0'
setenv gatewayip '10.0.0.1'
setenv serverip '10.0.0.10'
setenv bootargs 'root=/dev/mmcblk1p2 rootwait rw rootfstype=squashfs mem=239616k console=ttyS2,115200 <NULL> mac1=F8:BC:12:50:D3:F6 mac2=F8:BC:12:50:D3:F7  mode=vhard reset_cause=board nmi_buf=0x83000000 quiet'
setenv bootcmd 'fwu dev 1; fwu mmc sd; sleep 5; fwu boot'
setenv verify 'no'
saveenv

# ------------------------------------------------------------------------------------------------------------
# Flash the SD card with the firmimg.d7 from the TFTP server
# Get the firmimg.d7 file from Dell, I'm using 1.65.65.0
# ------------------------------------------------------------------------------------------------------------

tftpboot firmimg.d7
fwu mmc sd
fwu check 0x81000000
fwu update 0x81000000 

# ------------------------------------------------------------------------------------------------------------
# Boot the SD card to make sure proper copy, you should see endless messages about being unable to write to mmcblk0
# This is becuase while we are booting from the SD card, the configuration uses mmcblk0 partitions.
# ------------------------------------------------------------------------------------------------------------

boot

# ------------------------------------------------------------------------------------------------------------
# After you have successfully booted iDRAC, you know it works.
# Put the SD Card on a Linux machine and extract, edit, rebuild...
#
# extract the second partition (live filesystem)
# ------------------------------------------------------------------------------------------------------------

mkdir ~/idrac
cd ~/idrac
unsquashfs /dev/sda/sda2

# ------------------------------------------------------------------------------------------------------------
# find references to the mmcblk0. So far they are only found in /etc/sys_apps_scripts of the extracted partition
# edit any files that have references to mmcblk0 and change to mmcblk0
# ------------------------------------------------------------------------------------------------------------

cd ~/idrac/squashfs-root/etc
grep -rv 'mmcblk0'

# ------------------------------------------------------------------------------------------------------------
# Write the squashfs file system back to the SD card. There are two partitions that contain system data.
# ------------------------------------------------------------------------------------------------------------

cd ~/idrac
mksquashfs squashfs-root/ /dev/sda2 -noappend
mksquashfs squashfs-root/ /dev/sda6 -noappend

9. Completely power off the server, disconnect power etc... put in the SD card and there you go.
   It'll take a while to boot because the startup scripts will create the empty configs and such
   on the SD card.

Hey @FixMyServer,

Massive thanks for sharing your experience, your post is honestly a lifesaver. I've been digging around for months and this is the first proper bit of info I’ve found on getting around an iDRAC failure like this.

I was wondering if you could share exactly how you managed to interrupt U-Boot? Like what you pressed, when you pressed it, or anything specific you had to do? I'm working on a T420 but the setup looks really similar so I’d love to know how you did it.

Really appreciate any tips you can share, you’ve already helped loads just by posting what you did.

Hi @luigi034, I plan to make a short video on the complete process. We have several more of these servers and expect to see this happen to them all soon. I wish Dell had put the SD card on the MB instead of soldered EMMC. That would have solved so many issues, including de-bricking and easier updating. Imagine simply inserting an SD card with the latest update to lessen downtime. 

Anyway, let me get a short video that will explain the process. I can squeeze it in next week when I return home.

@FixMyServer​ Hi there, did you progress or went ahead with this task or project or the video you was mentioning?, thank you

Enrico

Well I keep researching about this problem and various methods to try to sort out, I copy/paste here below a Reddit thread in which the latest post also about a bricked iDRAC for T620 and the guy looks like arguing was successful

https://www.reddit.com/r/homelab/comments/zhegir/dell_12g_bricked_idrac_emmc_replacement/