1 Rookie
•
2 Posts
0
133
May 15th, 2024 08:52
VM Backup Slowness
Dear All,
We have Networker backup infra running 19.8.0.2-build 86 version for VMWare VM backups. The VM full backups are noticed as very slow compared to previous rates. recently we have upgraded the vproxies to the latest for FLR issue and some DAS release related.
The vproxy are configured as NBD mode for the backup, as there are some dependency issue with nutanix to configure backups as Hot-add.
we have performed ddpconncheck and found the bandwidth and transfer are OK from DD and Vproxy log and check says the vproxy throughput shows more than 100MB/s. But in NMC the vmbackup transfer rate only shows below 10MB.
Is there something we miss to check or add in the configuration level or connectivity level ?.
How to get maximum transfer rate for VM backups in NBD mode.
We have Networker backup infra running 19.8.0.2-build 86 version for VMWare VM backups. The VM full backups are noticed as very slow compared to previous rates. recently we have upgraded the vproxies to the latest for FLR issue and some DAS release related.
The vproxy are configured as NBD mode for the backup, as there are some dependency issue with nutanix to configure backups as Hot-add.
we have performed ddpconncheck and found the bandwidth and transfer are OK from DD and Vproxy log and check says the vproxy throughput shows more than 100MB/s. But in NMC the vmbackup transfer rate only shows below 10MB.
Is there something we miss to check or add in the configuration level or connectivity level ?.
How to get maximum transfer rate for VM backups in NBD mode.
No Events found!
V_Thayyil
1 Rookie
•
2 Posts
0
September 30th, 2024 09:06
@bbeckers1 The issue with Nutanix Hot-add backup is known and there is VMware verified KB release for that which as following. The solution suggested in the VMware KB 2010953 - using NFSv4 is not applicable in our case because Stargate only supports NFSv3.
For AOS 5.x and ESXi 6.x, change the backup policy transport mode to use the NBD or NBDSSL instead of HotAdd.
For ARCSERV, change the transport mode in the proxy server is overridden by the backup plan settings. In the backup, ensure that the NBD transport mode is moved to the top of the list.
VMware docs on the issue:
Even we are running the backups in NBD mode we didn't got desired backup speed and the full backups of VMs took more than 5 days to complete. After a full verification and analysis on the backup infra structure Networker Backend team found that the issue is with the read throughput from the VMware storage is making the issue .
We have confirmed the end to end infra is getting 10G bandwidth but the read throughput from the storage is very less. As the backend storage is with Nutanix we engaged them and they came up with another known issue with VMware version and Nutanix.
https://knowledge.broadcom.com/external/article?legacyId=59548 [knowledge.broadcom.com]
#: hostssh 'esxcli system settings advanced list | grep -A10 /SunRPC/SetNoDelayedAck '
============= 192.168.xx.xx ============
Path: /SunRPC/SetNoDelayedAck
Type: integer
Int Value: 0
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Set socket option to disable TCP delayed acknowledgements (Set this option under guidance of VMware Support. Requir
nxesxi03:~] ps -Tcjstv | egrep "WID|vmk1"
WID CID WorldName GID Type State Wait CPU Time Command
2098367 0 vmk1-rx-0 2097153 System WAIT KTCPLO 0-87 54239.470919
2098368 0 vmk1-rx-1 2097153 System WAIT KTCPLO 0-87 0.79852
2098369 0 vmk1-rx-2 2097153 System WAIT KTCPLO 0-87 0.68835
2098370 0 vmk1-rx-3 2097153 System WAIT KTCPLO 0-87 0.81727
2098371 0 vmk1-tx 2097153 System WAIT KTCPLO 0-87 92141.23041
8975308 8975308 egrep 8975308 User WAIT UPIPER 0-87 0.1695 egrep WID|vmk1
The TCP delay was higher which makes the read throughput affected.
And this was impacted us after a recent ESXi Upgrade as this parameters were Active as default. And we have implemented this fix . After implementation the backup rate in NBD mode is better and acceptable range.
bbeckers1
2 Intern
•
191 Posts
0
May 17th, 2024 13:02
you performed the ddpconnchk from the vproxy end I assume? or from where was the test performed? Dunno what transfer speeds would be that are deemed OK, based on the check?
the vm's are not using IDE virtual disks as those also degrade backup performance compared to SCSI virtual disks?
As we almost exclusively use Hotadd, we don't have too much experience with NBD. However the times we did, we also made sure that all the backups use the correct vproxy as we deal with multiple geographical locations. Don't wanna have one location use the vproxy of another in case Hotadd doesn't work and then it would use the vproxy that has NBD enabled. This as we have all of our policies configured without specifying which vproxy to use, which makes for easy deployments and configurations. However it can have undesired effects in certain situations, hence we restricted the vproxy that was supposed to only use NBD for specific datastores on that esxi cluster only, so that vm's on other clusters would not be able to use it as they would be located on other datastores. So if multiple locations are involved, make sure the correct vproxy is used, as any backup might use any vproxy with NBD, unlike Hotadd which restricts it to vproxies on esxi hosts within the same cluster only.
What is the issue wrg to Nutanix using Hotadd?
(edited)