VMware Image backups failing with 10056 as seen in the GUI. Too many extra snapshot files.

Hello everyone, this is an issue that is seen often, and should be seen by the community. 

These are seen in the GUI as Failed with error code 10056.  This is identifying orphaned snapshots, and aborting the backup process.

This occurs in the backup log (vmimage[w/l].log) file.

avvcbimage Warning <16002>: Too many extra snapshot files (3) were found on the VMs datastore. This can cause a problem for the backup or restore. 
avvcbimage FATAL <16018>: The datastore information from VMX '[MLUN] Datastore/TestMachine.vmx' will not permit a restore or backup.
avvcbimage Info <0000>: Starting graceful (staged) termination, Too many pre-existing snapshots on the datastore will not permit a restore or backup. (wrap-up stage) 
avvcbimage Error <9759>: createSnapshot: snapshot creation failed

The log shows that Avamar did not detect a Snapshot for this VM initially:

avvcbimage Info <14664>: no snapshots are present on this VM.

However, there are Snapshot files on the Datastore:

avvcbimage Info <14677>: All VM file Info:
File:'TEST-000003-ctk.vmdk                   ' last modified on '07-Dec-2015 15:58'  size=3932672
File:'TEST-000003-delta.vmdk                 ' last modified on '07-Dec-2015 15:57'  size=134344704
File:'TEST-000003.vmdk                       ' last modified on '07-Dec-2015 15:55'  size=408

In Avamar 19.7 and above, this issue has been resolved and this error will no longer occur for this issue.

*Note 10056 can still occur if a user has too deep of a snapshot hierarchy.  (Create Virtual Machine Snapshot task fails with a VMware error of: "Snapshot hierarchy is too deep.").  This will happen when customer reaches 32 snapshots

To resolve this issue

In Avamar 19.4 and below, this issue is fixed with the goav tool.

To obtain goav

1. Download from the ftp site, to /home/admin on the Avamar Utility Node.

Download Location
ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/goav
OR (if running from Avamar)
admin@ave:~/>: wget ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/goav

2. To make it executable in order to be able to run it, once downloaded on Avamar do this:

admin@ave:~/>: chmod +x goav

3. Run goav vm snapshot clean.  This removes orphaned snapshots from VMs and any disks that are stuck to a proxy. 
This can be run two ways:
    - This can be run against a single virtual machine with the --name switch.
       - Example: goav vm snapshot clean --name winpe-testing-2
    - This can run against ALL VMs, as seen below.

admin@ave:~/>: ./goav vm snapshot clean
===========================================================
GoAv    :        1.49
Avamar  :        19.7
Date    :        03 Mar 2023 10:29 EST
===========================================================
COMMAND :  ./goav vm snapshot clean
NOTE: This is not an official tool
===========================================================
Running "./goav proxy clean" to clean proxies : ConsolidateTask=True
================== Checking Active Vmware Image backups =================
No active Vmware backups found
================== Checking PROXIES for unwanted DISK =================
------------ 192proxy.lab ------------
Proxy 192proxy.lab is clean
------------ 197proxy.lab (19.7.100-82) ------------
Proxy 197proxy.lab (19.7.100-82) is clean
================== Checking Vms for leftover snapshots and disk consolidation =================
....

The full KB for this issue can be found here: https://www.dell.com/support/kbdoc/000068694