This post is more than 5 years old
38 Posts
0
2626
July 7th, 2011 15:00
NDMP Multiplexing and Concurrency
I have an SR open at the moment where I've enquired about what the possibilities are for improving the throughput of NDMP backups on DART 5.6. So far EMC have confirmed that 4 concurrent NDMP streams are supported where this means one stream per filesystem. Multiplexing the streams to a single tape device should also be possible.
We are using EMC Networker 7.4.x at the moment for the NDMP backup agent with a Fibre attached library. One tape drive is currently visible to the NS20.
Does anyone else out there have any experience of this or any ideas which elements of the DART and Networker config are involved in getting this concurrency to work. I believe the options are under the PAX and NDMP facilities of the DART server params and various of the parallelism options in Networker. We have tried a number of these but can't seem to get more than one filesystem to stream at a time. This is an issue as some of the filesystems have many small files and seem to be restricted by filesystem metadata access rather than raw throughput.
The intention in the longer term is to make another tape drive visible to the Celerra and use both multiple drives and NDMP stream multiplexing (2x2).
Anyone?
gbarretoxx1
366 Posts
0
July 8th, 2011 12:00
Hi,
there are some flowchatrs on the NDMP manual that explains how to tune NDMP backups and restores.
Please, see emc218290.
Gustavo Barreto.
dynamox
9 Legend
•
20.4K Posts
1
July 8th, 2011 10:00
you want to stream more than one file system to single tape drive ?
Rainer_EMC
4 Operator
•
8.6K Posts
0
July 8th, 2011 11:00
I dont think multiplexing will help his problem
There is still the limit of 4 concurrent streams (unless you have a system with more than 4GB data mover memory)
Besides the Celerra itself doesn’t do multiplexing when writing to a local (FC) tape drive.
If you use heterogenous 3-way NDMP that sends the data stream over IP to your backup server then the backup software there can put multiple streams on a single tape drive
I don’t think this is used that much these days – its typically done when a single backup stream isn’t fast enough to keep the tape streaming
Rainer
carwynedwards
38 Posts
0
July 8th, 2011 12:00
We don't "want" to multiplex the filesystems to a single tape drive, it's just it would be interesting to see if we can get more total throughput by doing it. We have student home directory filesystems that have quite high filecounts that are in no way saturating the tape drive. (It's an NS20 backed by a CX3-10). Other filesystems with fewer larger files are happily saturating the tape drive.
The SR response has noted that there are options under the NDMP facility in DART 6 relating to this but we're on 5.6
I assume that the multiple NDMP stream capability is more aimed at multiple tape drives?
Rainer_EMC
4 Operator
•
8.6K Posts
0
July 11th, 2011 07:00
I think we are talking different things here - multiplexing typically means interleaving several backups onto one backup device
multiple streams - running multiple NDMP backup jobs - each to its own tape drive
multi-threaded - the Celerra internally uses several multiple threads reading/writing to the file system for one NDMP backup to improve performance.
As to how to get your backup to run faster:
- look at the tuning options in the NDMP manual
(see the Tune backup and restore parameters - it is present also in the 5.6 NDMP manual)
- run multiple NDMP backups in parallel (not really feasible for one file system)
- check the disk stats and layout if that is the bottleneck - if yes reorganize
- get faster hardware
- if the problem is due to lots of very small files investigate block based NDMP VBB
- do less backups and rely more on replication or snapshots
Rainer