Help diagnosing random reboots on new (to me) hardware?

Sure! Here are some quick steps to diagnose random reboots on your PowerEdge server:

Look for errors around reboot times, ensure everything is up to date.
Run Diagnostics: Use Dell SupportAssist or built-in diagnostics.
Check Hardware Connections: Reseat components like memory and CPUs.
Monitor Temperatures: Ensure components aren't overheating.
Power Supply: Test with a known good power supply.
Minimum to POST: Reduce to essential hardware and add back components one by one.

Thanks

These are very general and I've done most of them already:

Look for errors around reboot times, ensure everything is up to date.

-- Logs provided, I'm not seeing any definitive errors.  There's some correlation about a disk reset, but that also happens when there isn't a reboot and the message says it's a normal operation, so it feels like a distraction.

Run Diagnostics: Use Dell SupportAssist or built-in diagnostics.

- I will run this, though I'm not sure how it works on a proxmox host

Check Hardware Connections: Reseat components like memory and CPUs.
- Completed, no change

Monitor Temperatures: Ensure components aren't overheating.

- I monitor the systems with zabbix, they are well within ranges.  They're in a dedicated server space with an air conditioner.

Power Supply: Test with a known good power supply.
- Completed, it's dual power supplies, both tested well and swapped out.  No change

Minimum to POST: Reduce to essential hardware and add back components one by one.
- Completed as possible, the only hardware is cpu, ram, and a fiber card for 10G fiber.  The OS can't boot without the fiber card.

Are there any other hardware clues here I can get?  I'm going to pursue supportassist, but I would think the idrac logs would be enough for a clue.  Should I pursue replacing the backplane, in case that's connecting to the disk issue, even though I suspect it's a red herring?

Hi,

We're not good with Proxmox, troubleshooting might be limited. But most of Proxmox post I've replied, the RAID controller is being flashed as IT mode? - I don't know much about that. But was any of the R710 hardware being installed in the R6525 or just the data being imported?

For PDR87 error, the physical device is being reset. Perhaps you are right, it could be backplane. Have you tried swapping to other slots or try another spare drive? 

Have you also confirm that the server is running on latest firmware?

No, it has a PERC H345 Front which can be configured for passthrough to support the ZFS, so we aren't doing anything weird here. 

I've ordered a new backplane for $150, and I'll try that.  I did try swapping the drives around, but haven't noticed a trend.

The idrac auto-updates the firmware the first Saturday of the month. 

I am not sure how to run support assist as it shows as incompatible with debian based distros, and I'm not sure what other diagnostics to run.

Hi,

you've already done a thorough hardware sanity check, which narrows down the likelihood of simple issues like reseated components, power, or basic temperature concerns. The correlation between disk resets, CPU resets, and reboots suggests a possible hardware fault or firmware/hardware interaction, but pinpointing it is tricky.

Next steps for diagnosis:
1. Check for Hardware Faults via iDRAC
Firmware updates: Keep iDRAC firmware current—known issues sometimes get resolved this way.
iDRAC logs: Double-check for any critical hardware warnings or errors not evident in your current logs, including power supplies, CPUs, memory, and storage components.
Run thorough diagnostics:
Use Dell's Lifecycle Controller Diagnostics—accessible via iDRAC or during POST (F10) — for thorough hardware testing.
Even if these tools are designed primarily for Windows, they often boot from diagnostics USBs or via iDRAC remote virtual media. You can create a Dell diagnostic ISO and run full hardware tests outside the OS.
2. Monitor for Hardware Errors & Correlations
Enable dell_syslog_server integration or check iDRAC telemetrics to log hardware events over time.
Correlate these logs with your reboot times, especially before and after specific events like disk resets or backplane logs.
3. Disable or Bypass Certain Hardware Components
Remove the fiber NIC temporarily if possible or disable it, to rule out NIC-related issues in the reboot cycle.
If the server has IPMI or additional hardware management, monitor those metrics for anomalies.
4. Memory and CPU Testing
Run memtest86+ (bootable from USB) outside of Proxmox to test RAM thoroughly.
Consider swapping to known-good, compatible RAM if feasible.
CPU stress tests (e.g., stress-ng, or running a CPU burn-in tool) can help identify hardware faults, but these require a live OS.
5. Firmware & BIOS
Even though firmware updates are scheduled, double-check and manually verify the latest firmware for BIOS, iDRAC, PERC controller, and other components. Sometimes, new firmware releases fix rare hardware instability issues.
6. Check Power and Environment
Use a power quality monitor (e.g., a kill-a-watt or more advanced device) to verify stable power delivery—none of the logs suggest power issues, but it's worth confirming if you haven't yet.
7. Further Hardware Replacement
Backplane: Considering you've ordered a new backplane, that’s a good move. Since disk resets can sometimes indicate backplane issues, replacing it might help.
Power Supplies: Swapping for known-good units, if not already done.
Motherboard/CPU: As extreme measures, testing with a different CPU/motherboard or running the current setup in a different environment may clarify if hardware failure is inevitable.
Additional notes:
If you can’t run Dell diagnostics directly (since they prefer their utility), creating a bootable diagnostics USB from Dell's official support page is often the easiest way.
Keep an eye on IDRAC logs for specific errors just before or after reboots.
Consider disabling all non-essential hardware one by one (e.g., NICs, PCIe cards) to see if stability improves.

Rough update, I replaced the backplane and got another reboot.  My monitoring system caught the node reboot at 5:35, and these were the logs from the idrac around that, the reboot happens at the red arrow:

So, I get that disk backplane reset a lot, but it is pretty adamant that it's a normal part of operations, and I'm not clear which disk that even is as I've swapped them all out.  Between swapping disks and swapping the backplane, I'm not sure what else to check in that hardware chain, though it's a strong correlation that it happens right before a reboot.

I'm pretty frustrated; I don't have the resources to get new Power Supplies or MB/CPU's right now.  The UPS shows clean power and all memtests were fine.  Firmware updates are confirmed up to date.

Hi, 

You mentioned swapping the drive around and did not notice any trend. Does it mean that it only occurs on slot 0 and 1? Curious, are the drives Dell drives? Is this server in production? Are you able to test this without booting to OS, say leave it in BIOS mode for a period to check if it's a hardware issue that cause the reboot.