Equallogic VSS Requestor Memory Leak? (HIT 4.7.1 Windows)

Question

Hello,

I have two machines here (one W2K8 R2, one W2K12R2) that show a steady climb of memory usage (commit size) for the Equallogic VSS Requestor (EqlReqService.exe) service - 5.711.888K Commit Size after 47days uptime.

This is with HIT 4.7.1.7382.

I don't see this increase on the machines still running 4.6.0.6882 (16.292K Commit Size after 29days uptime).

Do others see the same behavior? Known issue?

Thanks,

Florian

sbuerger · Accepted Answer

I have this issue since 09/2014. I reported it and have an open case with the number 901094060.

They acknowledged the problem and said they would implement a fix, maybe in the next version.

The workaround for us is to stop the service through task scheduler every day at midnight. By default the startup for this service is set to manual and it would be started when needed.

sl1200mk2 · Answer

I have the same issue on 2012 R2 hosts. We have 3 clusters with HIT installed and all 36 hosts experience the same issue. It's not uncommon at all for me to see eqlreqservice.exe using 20GB or more of memory. I've been searching to see if others have had this issue as well, but until seeing your post thought I was alone. I haven't yet opened a support ticket, but plan to soon if no suggestions arise.

I'm using the same HIT 4.7.1 version which I see is still the latest release available.

Wayne

RichardSkinner · Answer

I have the same issue on 2012 R2 hosts.  We have two Hyper-V clusters with 4 and 5 nodes each all running HIT 4.7.1.  We're using Veeam to backup the VMs using hardware VSS.  I too have seen eqlreqservice.exe using 20GB+ of RAM.

fzb · Answer

Using Symantec BackupExec here, also configured to use VSS HW snapshots

sl1200mk2 · Answer

That's a good point about mentioning backup software.  I'm using Veeam as well (latest 8.x) version which is performing snapshots using EQL VSS. Wayne

fzb · Answer

I am not sure if this is the issue I see. Both servers in question (in my scenario) are SQL Servers using not CSVs but exclusive attached LUNs. And additionally the 2012 R2 server has the mentioned December 2014 update rollup (KB 3013769) installed that's supposed to have the fix for the leak but still shows the issue.

The article about the leak also mentions the pool memory is leaked, not sure if that would show up in the commit memory of the process at all

sl1200mk2 · Answer

After searching some on the Veeam forums this looks be an MS issue and not EQL. I won't have ability to test for a couple weeks, but it looks like there's a hotfix available which was also part of the December 2014 rollup.

But is this the same R. Skinner posting above?

Reference: forums.veeam.com/.../hardware-vss-t17425.html

User 'rskinner': "I'm having a similar issue with the EqualLogic hardware VSS and Server 2012 R2. Each time a VSS snapshot is taken memory is leaked. I've opened a call with MS who've told me to use the software VSS. This is completely unworkable as many snapshots have timed out and the backups are taking hours longer than with the hardware VSS. I opened a call with Dell who say it's a known issue since Server 2012 and there's no resolution."

Later updated, "It's been several months, but I have an update. Microsoft have released a patch for the issue I was seeing for Windows Server 2012. I've been testing a pre-release version of the patch for Windows Server 2012 R2, which is due to be released as part of the December 2014 rollup on patch Tuesday. The good news is that the patch has resolved the memory leak."

---------------------------------------------------------------------

MS KB: "Memory leak occurs when you create or delete CSV snapshots by using a VSS hardware provider in Windows"

support.microsoft.com/.../en-us

RichardSkinner · Answer

Yes I am the rskinner on the Veeam forum. However, as fzb says the December update was for a memory leak in pool memory and I've applied the fix to all my hosts. The memory leak I'm seeing with eqlreqservice.exe is a separate issue.

As fzb is seeing the leak on SQL servers I have a hunch that this is a memory leak related to EqualLogic hardware VSS snapshots rather than a specific technology, e.g. Hyper-V or SQL Server or backup software.

fzb · Answer

thanks for the feedback. should next release have no fix, will open a case too and reference to your case number.

mviel · Answer

I can also confirm that the Problem is still out there , the Equallogic Support 1st and 2nd confirmed the iussue.

But there is no planned Date which is communicated, when the Problem is going to be fixed . The Hit Kit Development Team hasnt have a history of communicating too much !

They just dont have those information .

Memory Leak for the Service : Equallogic VSS Requestor is so imperativ that is very hard to understand that the QA hasnt found it

Problem : the Service Equallogic VSS Requestor Service which starts with 10MB Memory USAGE is aquiring up to 35000MB ( thats when we restarted the Service after 90 Days ) . Its a Hyper-V in a 12 Node Enviroment . ( It took 10% of the RAM away until noticed )

Maybe someone reads that with the same "Problem", we have now a scheduled Task in place which restarts the service on a daily Count , that "fixes" that Problem .

( The Funny part is , we dont use that Funktion for what the Service is used never ... it is used for the Auto Snapshot Manager ASM )

on Top of that Error we now experience a Bluecscreen of Death ( BSOD) on one of our Hyper-V 2012 R2 Cluster , the cause was the eqlASMAgent.exe Service , only 30 VMs booted .... great News for the Production Enviroment this week.

Enviroment : 12 Node Cluster W2012R2 + Equallogic Group with 5 Members ( PS6100) ( Intel Nics, R720, ISCSI )

Reference : 925869406 | [PS6100] BSOD caused by EqlASMAgent.ex 0x139

EqualLogic

Equallogic VSS Requestor Memory Leak? (HIT 4.7.1 Windows)

Was this post helpful?