T5610 Hard Reseting

I started having the same issue in Jan 2022.  T5610 with 128GB RAM, 2x E5-2667 v2 CPUs, RTX 3060 with an 825W PSU.  The RTX was installed in August and has been running fine.  I just ordered a 1300W PSU because my Kill-a-Watt meter was seeing occasional (every few minutes) short bursts (~1s) of 600W and 1200W power draws occasionally.  I ran diagnostics and Passmark benchmarks to test and stress the system.  All the tests pass.  My drop outs are typically around a initial boot, sometimes not even making it through post.  Once it "warms up" it seems to be fine, but the problem is getting worse.  I'm thinking the either the PSU is failing, or is over sensitive, or the video card is pulsing too high of a current draw and potentially failing.  My next steps are to swap back to an older video card, swap PSUs with my 7810 which is also 825W (similar config and running fine with a GTX 680), and try the 1300W PSU when it arrives.  I'll post any updates as I continue to troubleshoot the issue.

I solved the problem. Bottom line... I think it was heat.

Unfortunately I broke the cardinal rule of making too many adjustments in the fix cycle so I can't say for sure what THE fix was.

I found a program for monitoring heat and ran it while I was working in Daz. The temperatures did climb into the 70s celsius and I recall from the Daz forums a user with the same rig and a lesser card reporting the same shut down issue around mid 70s. He had asked if there was a way to disable the safety shut off... bad idea anyway... but there is no way to do it... it's there for a reason.  Anyway, I also recalled another post - I think here in a Dell forum where someone was talking about cooling in general and the stock fans on this rig.

So what did I do?

(1) I went into BIOS and changed the fan setting from auto to high. The reason for this was I suspected that my shut down was due to heat on CPU #2.  It was already getting "cooled" with warmer air than CPU #1 as the air flow comes from the front and runs over/through CPU#1 sink first. Next, my Titan X generates heat which goes... yep... UP and adds to the heat on CPU#2.  I theorized that CPU was getting just a little too hot before the fan could ever get the signal to "ramp up" or it might not even be doing that.

(2) I was running a second display off the Titan. Why? Because I could and I might need it? Anyway, I disconnected the second display. 

With the increased airflow and no second display... no problem.  That said, Arhannan, IF you are only running one display, go into BIOS and kick your fans into high.  Yes, it will sound like a server but let me know if you get shut downs still.

Heat or thermal issue would caused throttling first before reboot occur.  Also, you would have noticed it when all the fans revved up and down.  Thermal paste on CPUs could use a replacement at this age.  A clean install of OS on a spare drive for testing could rule out drivers issue.  

Heat issues were more common on the SFF models but not on these towers.  

Not the thermal paste this machine was recently rebuilt and that was part of the rebuild. Additionally, this is a clean version of OS on a new WD drive. Also… the fans never increased at any point leading up to the shut down Which is part of what lead me to believe that it was overheating. That and the fact that with the fans on high, I reduced my temps during graphics work from mid to high 70s and system shut downs to 30s and no shut downs.

Pretty sure it was heat. Also, prior to my fan setting change I could never do work in Iray mode without a crash. Now I can. 

The decision now is do I mark it as resolved or add another fan on the rear to serve as exhaust fan so I can lower the speed of the main fans. They honestly are not that bothersome at speed.

Thanks for the info.  If you were shutting down for temperatures, there should be entries in the BIOS log records.  My drop outs would occur while the system was basically idling and nothing in the BIOS log.  Thermally it wasn't warm at all.  Here is what I have done so far.

I swapped a GTX 950 (90W) with the RTX 3060 (170W) between the T5500 (2x X5687, 144GB RAM, 875W PSU) and the T5610. The T5610 ran fine with no issues with the GTX 950. I noted some current spikes on the Kill-a-watt, 1 overload chirp like with the RTX 3060 card in the T5610, but it never dropped out. The RTX 3060 card ran fine in the T5500. I never saw any high current spikes or chirps on the Kill-a-watt with the RTX 3060 in the T5500.

Next I'll try the Kill-a-watt on the 7810 and look for the spikes as another data point. The high current spikes could be a sign the power supply is failing internally (bad caps). This might make sense if you are having the same issue using a 1300W PS and no entries in the BIOS log.  Are there are any entries in your BIOS log associated with your shutdowns?

Since I turned the fans up… no issue. In fact, prior to that I could not work in Iray mode without my program crashing due to GPU. 

Update:  In running the T5610 with the GTX 950, the machine did end up dropping out a couple times.  The 1300W PSU arrived and I swapped it into the T5610 with the RTX 3060.  There were no high current spikes detected on the Kill-a-watt meter.  The maximum current was under 400W during all the diagnostic and Passmark testing.  In my case, the 825W PSU was failing (Dell # G57YP) and dropping out due to what I believe was self induced overload detection.

So my T5610 is misbehaving again.  The hard shutdown with no log is still happening.  It is only with my Daz Studio program. At this point I'm narrowing down to my Titan X card itself or possibly the HDD?  The HDD was a brand new 1TB WD blue drive.  What are the possibilities that my HDD is the cause?  I read a post in a different forum which suggested that this drive may just not be fast enough.

So here is the thing... there is literally no warning at all.  It is like someone just pushed the power switch.  In fact... right before it happens, I do hear an audible 'click' and the box shuts off.  I have it set to turn on after power failure so it does reboot itself.  If that were turned off I suspect it would shut off and stay off.  I have the fan on high and it's been pretty cold where I have it.

Dell T5610

Xeon E5-2670 x2
Clock speed 2600
8 Cores Each... total of 16
64GB memory 4x 16 GB ECC RDIMM DDR3 at 800MHz
Nvidia GForce Titan X 12gb Card (Maxwell) in slot 4

So... I found that my PCI MMIO was set in BIOS to "small"... switching that to "large"
Also... Memory Map IO above 4GB was turn off... I am turning that on.

The clicking sound could be from the psu. My personal idea is that it could be a sudden power draw higher than the safe allowed by specs, triggering a cutoff.

Again, this is just me musing... Do you have another gpu to temporarily swap in ?

I don't. I just got this PSU and is the most powerful one you can get for this box... 1300w. That 1300w is a heck of a lot of power.

And i agree it is, but to rule out if by chance the gpu is misbehaving , trying to see if it's stable with a different one is not a bad plan.
How otherwise can you make sure that the gpu isn't drawing too much due to starting to randomly having issues at full load ? All in all everything is subject to wear and the Titan is not exactly new ( 2016 released )

(and are you 100% sure the psu hasn't some defect ?)

mmmmm well, could be a sw glitch ( wrong driver version + wrong sw subversion + anything else in the mix ).

You could try the same scene with a different release of the drivers, to see what happens, maybe

You’re never sure with used stuff.

I did run a full diagnostic from BIOS and there were no issues. The shut down is ONLY when running Daz studio and so far it is when navigating within a scene that is fairly detailed.

Drivers have always been an issue. Nvidia is horrible when it comes to good, stable tested drivers