Equallogic Auto-Snapshot manager conflicting with Veeam

VMware Esxi is a physical server that hosts all of our Windows VM's. So, I am talking about ASM on a Windows OS VM. I appreciate all your information - very detailed and explanatory.

So, I extrapolate from your reply, since I created the schedule with Group Manager then ASM is not required. I only have one VM with the Veeam conflict because the VM's E: drive 5 TB resides on the Equallogic SAN. I'm going to disable/stop the ASM service and verify it does not effect the replication and also run a Veeam backup. 

Thanks, 

Milty 

Hello Milty, 

 AutoSnapshot Manager (ASM) is for Windows hosts, but you said ESXi?    I am pretty sure you mean VMware Storage Manager? (VSM)?  Neither ASM nor VSM are required for EQL replication.  They just add host integration and more consistent snapshots/replication to the process.  VSM also has additional features like creating or expanding Datastores. 

Re: Replication  It depends on where you have it scheduled. If the schedule was created by the Group Manager then ASM/VSM are not required. 

 With VSM, while you can select a VM, regardless all VMs on that shared Datastore will be snapshotted or replicated. There's no way to just snapshot or replicate a single VM on a shared Datastore, unless that's the only VM on that Datastore. 

 VSM will work with VMware to first create a VMware snapshot of each VM, then create the EQL snapshot/replication.  Then turn around and ask VMware to delete that VMware snapshot.   This normally does not interfere with any other backup solution for VMware. 

 I'm going to guess and suggest maybe the problem is with Change Block Tracking (CBT)? 

 So if you want to take VSM out of the process, pause or remove the Replication schedule in VSM, and create it on the EQL array itself.  This will prevent any host awareness of the process, since it operates only at the array level. 

 Regards, 

Don

Hello, 

 You are very welcome. 

  Just to complete it, when no HW provider is available, VSS uses the unclaimed space on the volume to hold the VSS snapshot.  Which VEEAM uses to transfer the files.  Insuring there are no open or missed files. 

 Regards, 

Don 

Hi Milty, 

 That's for the additional information. 

 Now I know what the issue you are having is.  When you installed HIT/ME.  In addition to ASM GUI, you also installed the Volume Shadow Service (VSS) Hardware Provider for EQL storage. 

 When VEEAM does snapshots or replicas it calls on VSS to quiet the filesystem first.  Especially important with Exchange and SQL. 

 When VSS runs it looks at the drive E: and see's that it is an EQUALLOGIC drive.  Then finding an EQL H/W provider installed it connects to the EQL storage via the vss-control volume and creates a 'Hardware' snapshot on the array.  (or replication same process).  However, VEEAM doesn't support EQL HW snapshots/replicas. 

 If you disabled ASM, the HW provider would still be there.   You just need to disable it with the EQLVSS command. 

 In fact you can disable it before VEEAM runs, then re-enable it after.  So in fact you can have both ASM/ME snapshot/replicas and VEEAM if you wanted to.  I believe like most backup related products VEEAM will allow you to run commands PRE and POST operations.  You could put these commands there. 

Regards, 

Don 

Thanks for the details. I'll report back following tonight's backup. 

.....Milty 

Don, 

Appreciate the thoroughness. 

..... Milty

Hello, 

 You are very welcome.  I am glad that I could help you!

Don 

Can the disable/renable process happen with say the exchange VM still running on that particular node? (as long as it's not doing a veeam backup, I would assume?)

Don,

Thank you for your response. I think that answers the question - my only other concern is that it might disconnect volumes? Or am I mistaken?

Eric

Hello, 

 Yes you can disable the HW provider at anytime. It does not impact any ongoing IO operations at all. 

It only comes in to use when called by a VSS writer.  Like VEEAM, or the ASM GUI or scheduler. 

 Hence you can do this just BEFORE VEEAM runs then re-enable afterwards. 

 It's done on a per server/node basis.  All EQL volumes will either be enabled or disabled. 

 Does that answer your question? 

 Regards,

Don

Hello, 

re: Disconnect.  Not at all.  This is operating mostly at the host OS level.  With the HW provider enabled commands are sent to the SAN that create a snapshot of that selected volume(s).   The base volume is untouched.  The problem is VEEAM isn't supporting this  HW snapshot. It isn't programmed to connect to the EQL san and mount the snapshot. 

 When the Volume Shadow Service (VSS) H/W provider is disabled or not available for that storage vendos, VSS creates a hidden snapshot on the volume using unallocted space.  So VEEAM can directly access that software snapshot w/ol issue. 

 So VSS won't have any impact on iSCSI connections to the base volume at all. 

 Regards,

Don 

Don - thanks for your help. I did try your unreg/reg process on the node and attempted to run the backup inbetween toggling but no luck. I worked with veeam today for over an hour on trying to get things moving. We weren't able to get our exchange server to successfully use the equallogic snapshot process. It would fail over to the microsoft csv shadow copy provider and seemed to work but I went ahead and killed that process since I was a tad worried about shadow copy space on our c: drive. I've tried using the host to do the processing - and letting our repo server do the processing but all fail at the hardware snap. We got it to quit hard failing at guest processing by restarting the shadow copy services on the exchange server but it still fails when doing the hardware snap shot and fails over to microsofts eventually. I've been working on this pretty heavily for a few weeks now and seem to be finding no way out. It's got to be something simple - no idea why it just quit working a few weeks ago. We've ruled out a ton of stuff like windows updates, the hardware itself (snapshots work for all of our dozens of VM's on our hyper-v cluster) - just exchange is the one being fussy. The latest things I've tried - I looked at our equallogic for our exchange data volume and saw an old snap sitting out there from a couple of months ago ( the other 3 for that volume were from each night within the last 3 days - they're supposed to age out with the hit kit installed) - so I deleted that thinking it had something to do with it but no luck. The latest message it is giving us is this:

9/25/2019 3:16:26 PM :: Failed to create snapshot (Dell EqualLogic VSS HW Provider) (mode: Veeam application-aware processing) Details: Failed to add volume [\\?\Volume{ca664c21-7988-4908-827b-7397fe3cb708}\] to the VSS snapshot set
The shadow copy provider had an unexpected error while trying to process the specified operation.
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.

There are no open snapshots on the exchange VM or the host where exchange resides so I'm not sure what volume its trying to add. It could be because of the retryable error I'm just now noticing on the exchange vm:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {ae01a084-c813-4fcd-a598-98a36b2d18eb}
State: [1] Stable
Last error: Retryable error

I really don't want to have to restart exchange again. I read somewhere that just letting the vm do its backup can resolve this retryable error? I might try that but I feel the other issues will still be there.

Eric

Hello, 

 If you are using VEEAM then you can't use the EQL VSS Hardware Provider.  You have to use the local space option. 

re: C: drive.  The hidden VSS snapshot should reside on the volume itself, not your local drive.  Of are you using mount points off the C:\ drive?    I always had problems doing that. 

I haven't done it in a long time but you used to be able to specify a dedicated drive to store VSS snapshots. But that was the W2K3/2K8 days.   

Regards, 

Don 

Hello, 

 I don't believe HIT/ME truncates Exchange Logs.  For SQL it will checkpoint the logs.  But for Exchange it's about flushing anything unwritten to disk.  To makes the DB and logs are in sync. 

 The HW providers only job is to take the command to create snapshot and connect to the EQL array. Select the volume(s) in question and create HW snapshots of that volume or volumes. 

 Anything with exchange or SQL is done before that creation process.  Since it must complete to make sure the data on the snapshots are up to date. 

 HIT/ME only works on EQL Volumes that are directly conected to the EQL array.  VHDs themselves aren't supported.  You can snapshot the volume storing the VHDs but that's done at the hypervisor level. Not the Exchange server. 

Regards,

Don 

Whats interesting is that the hardware provider actually truncated logs for a few months. I'm not sure why it quit exactly. We may be going down a bit of a rabbit hole. Maybe exchange is the exception? I know the hardware works for everything else we use right now.

I will look into the drive storage option for the snapshots - I think I found something in there the other day about that. Right now it's just making shadow copies on each of our mounted vhds in the exchange VM.