1 Rookie
•
82 Posts
0
6440
November 3rd, 2020 04:00
nsr clone fails mmd reservation failed with nonretryable error
Hi again:
i am trying to clone from tape to disk (trought the cli ) but i have an mmd error. Something to consider.
- The saveset do exist on tape.
- The saveset is from an old client exchangedag, this client doesnt exist anymore y networker, it was deleted.
- A month ago we upgraded NW 9.1 to 19.X.
- Today i see the same error on disk to tape offsite backup(scheduled trought NW console).
The output from tape to disk is:
180169:nsrclone: Step (1 of 16): NSRCLONE_INIT_PHASE_ONE: nsrclone has been started on host 'nwserver', with command 'nsrclone -s nwserver -b Default Clone -S 3277621574 '.
10/27/20 08:50:29.783451 ============== nsrcloneconfig parameters ===============================
10/27/20 08:50:29.783451 max_threads_per_client(0) is default value.
10/27/20 08:50:29.783451 max_savesets_per_thread(0) is default value.
10/27/20 08:50:29.783451 max_client_threads(0) is default value.
10/27/20 08:50:29.783451 mmd_reservation_timeout(0) is default value.
10/27/20 08:50:29.783451 disable_vsr(0) is default value.
10/27/20 08:50:29.783451 concurrent_clone_enabled(1) is default value.
10/27/20 08:50:29.783451 ams_enabled(0) is default value.
10/27/20 08:50:29.783451 ams_slice_size_factor(31) is default value.
10/27/20 08:50:29.783451 ams_preferred_slice_count(0) is default value.
10/27/20 08:50:29.783451 ams_min_concurrent_slice_count(1) is default value.
10/27/20 08:50:29.783451 ams_max_concurrent_slice_count(60) is default value.
10/27/20 08:50:29.783451 ams_force_multithreaded_slicecopies(0) is default value.
10/27/20 08:50:29.783451 co_use_ip_if(0) is default value.
10/27/20 08:50:29.783451 ========================================================================
180170:nsrclone: Step (2 of 16): NSRCLONE_INIT_PHASE_TWO: Obtaining nsrclone options.
180171:nsrclone: Step (3 of 16): NSRCLONE_INIT_PHASE_THREE: Performing Recover Pipe to Save enabled cloning.
180173:nsrclone: Step (4 of 16): NSRCLONE_INIT_PHASE_FOUR: Validating the command line arguments.
180174:nsrclone: Step (5 of 16): NSRCLONE_PROCESSING_PHASE_ONE: Submitting all save sets for Recover Pipe to Save processing.
180184:nsrclone: Step (6 of 16): NSRCLONE_PROCESSING_PHASE_TWO: List of save sets is requested for cloning.
3277621574;
180206:nsrclone: Step (7 of 16): NSRCLONE_PROCESSING_PHASE_THREE: Sending the media reservation request for the current save set list.
160812:nsrclone: Retry reservation after it was rejected. Retry counter [4]
116284:nsrclone: Sending cancel broker request 18338
180876:nsrclone: Request 18338 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
160812:nsrclone: Retry reservation after it was rejected. Retry counter [3]
116284:nsrclone: Sending cancel broker request 18339
180876:nsrclone: Request 18339 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
160812:nsrclone: Retry reservation after it was rejected. Retry counter [2]
116284:nsrclone: Sending cancel broker request 18340
180876:nsrclone: Request 18340 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
160812:nsrclone: Retry reservation after it was rejected. Retry counter [1]
116284:nsrclone: Sending cancel broker request 18341
180876:nsrclone: Request 18341 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
160812:nsrclone: Retry reservation after it was rejected. Retry counter [0]
116284:nsrclone: Sending cancel broker request 18342
180876:nsrclone: Request 18342 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18343
180876:nsrclone: Request 18343 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18344
180876:nsrclone: Request 18344 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18345
180876:nsrclone: Request 18345 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18346
180876:nsrclone: Request 18346 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18347
180876:nsrclone: Request 18347 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18348
180876:nsrclone: Request 18348 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18349
180876:nsrclone: Request 18349 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18350
180876:nsrclone: Request 18350 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18351
180876:nsrclone: Request 18351 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18352
180876:nsrclone: Request 18352 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18353
180876:nsrclone: Request 18353 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18354
180876:nsrclone: Request 18354 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18355
180876:nsrclone: Request 18355 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18356
180876:nsrclone: Request 18356 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18357
180876:nsrclone: Request 18357 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18358
180876:nsrclone: Request 18358 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18359
180876:nsrclone: Request 18359 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18360
180876:nsrclone: Request 18360 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18361
180876:nsrclone: Request 18361 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18362
180876:nsrclone: Request 18362 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18363
180876:nsrclone: Request 18363 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18364
180876:nsrclone: Request 18364 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18365
180876:nsrclone: Request 18365 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18366
180876:nsrclone: Request 18366 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18367
180876:nsrclone: Request 18367 was cancelled.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18368
180876:nsrclone: Request 18368 was cancelled.
114982:nsrclone: NSRCLONE being aborted since broker request for mmd reservation failed with nonretryable error.
180208:nsrclone: Step (8 of 16): NSRCLONE_PROCESSING_PHASE_FOUR: Found the recover storage node 'nwserver' for source volume '240651030'.
124479:nsrclone: Failed to reserve one of the mmd. Sending cancel broker request 18337
180882:nsrclone: Step (9 of 16): NSRCLONE_PROCESSING_PHASE_FIVE: Failed to receive the media multiplexor daemon reservations for both the source volume '240651030' and destination volume '0'. Removing the reservation ID 18337.
No valid input savesets - check CLONE SKIPPED SAVESETS!
RPS CLONE RESULTS
[ORIGINAL REQUESTED SAVESETS]
3277621574;
[CLONE SKIPPED SAVESETS]
3277621574/1583206588;
142778:nsrclone: NSRCLONE failed for one or more savesets.
Thanks in advance.
No Events found!
crazyrov
4 Operator
•
1.3K Posts
1
November 3rd, 2020 20:00
@Dsantibanez, If the name in the error is correct for the host that you are trying to clone the data of then you can add this hostname to the aliases of the respective client instance.
crazyrov
4 Operator
•
1.3K Posts
0
November 6th, 2020 05:00
@Dsantibanez, Since cloning process is multi streamed it might be taking up the available sessions for the device. Can you try increasing the target and max session or create a new device to the same destination pool ?
crazyrov
4 Operator
•
1.3K Posts
0
November 3rd, 2020 05:00
Hi@Dsantibanez , Check if this saveset is already present in the destination disk that you are trying to lone to? Also make sure you have a drive free for reading the saveset from the Tape.
Dsantibanez
1 Rookie
•
82 Posts
0
November 3rd, 2020 06:00
Hi @crazyrov , when i tried to look pool Default clone , i figureout that this pool wasnt propertly mapped to a disk volume and device, so a created a new device and the mmd error dissapear.Thanks, without your suggestion , I had never seen if the savedset was in the default clone pool and I might have never discovered it.
Now the log shows me this, because the client isnt there , How do i resolve this ? Thanks in advance.
bingo.1
2.4K Posts
1
November 4th, 2020 04:00
Another method is to create a fake entry in your hosts file and recreate the missing client. Make sure that it will not be backed up.
Dsantibanez
1 Rookie
•
82 Posts
0
November 5th, 2020 03:00
Right know the library is busy , so i going to try the alias tip then @bingo.1 suggestion.
Dsantibanez
1 Rookie
•
82 Posts
1
November 6th, 2020 05:00
Thanks a lot, put de alias on the client works for me.
Question: before the update from NW 9.1 to 19.2
i can send several nsrclone commands to cmd with out issues, it 's was and standalone process and worked like a charm, but with 19.2 after a fiew commands nsrclone fail with this error. One solution is to restart the NW server.
116283:nsrclone: retrying mmd reservation request!!!
116284:nsrclone: Sending cancel broker request 18357
PD: Do I open a new forum thread?
Thanks.
Dsantibanez
1 Rookie
•
82 Posts
0
November 6th, 2020 08:00
Thanks @crazyrov i going to try to edit that config on tape devices (the issue is only in the clone to tape process)
Thanks in advance.
Dsantibanez
1 Rookie
•
82 Posts
0
November 9th, 2020 07:00
Today i checked the weekly clone workflows and every one was succesfully.
Thanks @crazyrov
clebeuva
1 Rookie
•
58 Posts
0
March 5th, 2021 09:00
Hi @Dsantibanez
What did you do with the tape devices config to fix this? I think I am seeing the same issue when one of my tape drives in the library is busy (I have 2 drives).
CLEB
Dsantibanez
1 Rookie
•
82 Posts
0
March 5th, 2021 10:00
Hi, after nearly 5 months, the issue persist, DELL EMC told us tha we have to run an update.We are waiting a backup windows to do it.
But i have to restart every time that i clone to a different pool.
clebeuva
1 Rookie
•
58 Posts
0
March 5th, 2021 16:00
Ok thanks for the info. I am using v 19.0 and I haven't had the issue with changing clone pool or are you talking about multiple tape clone pools? I only use one.
I found this article which isn't very informative
https://www.dell.com/support/kbdoc/en-uk/000050305/ccr-cloning-to-dr-data-domain-fails-with-mmd-reservation-error-messages-nsrclone-being-aborted-since-broker-request-for-mmd-reservation-failed-with-nonretryable-error
barry_beckers
393 Posts
0
March 9th, 2021 03:00
wrg to cloning to tape you might also want to look into RPS (recover pipe to save) as according to your logs it is enabled.
This RPS feature might still not be working as good and effective as it was intended to be, when it was introduced. There have been multiple KB articles stating it to be set to disabled ("Disable RPS Clone" attribute of NW server properties). Depending on certain settings when upgrading to nw19 it will be either left enabled, but if it was set to disabled, it will be left disabled.
Cloning of VMware image level backups will use RPS method automatically anyways, regardless if you have en/disabled it on NW server level.
"
"
from the nw19.4 manual:
"Cloning of saveset is not supported from a non-Data Domain device to a DD Cloud Tier device. Cloning fails with the following error:
Failed to get mmd reservation with err: Clone saveset(s) operation from a nonData Domain device to a DD Cloud Tier device is not supported.
● When you perform a fresh installation of NetWorker Server 19.4, RPS cloning is disabled by default.
● If you have NetWorker 19.4 server installed, Dell EMC recommends that you maintain parity with the NetWorker 19.4 storage node in both the RPS Enabled and Disabled mode. However, if you have compatibility and migration challenges and want to maintain earlier versions of the storage node, that is, the N-2 version, then Dell EMC recommends that you use the RPS Disabled mode for cloning. In cases where VMware vProxy save sets are used for cloning, RPS Enabled mode is supported by default. Therefore, Dell EMC recommends that you maintain NetWorker server and storage node compatibility and configure clone actions with the appropriate storage node based on the workloads."
So even regardless of what the recommended RPS state would be, depending on what you use in a NW environment might lead to less than optimal settings, if for example you'd have DD to DD and DD to CB in the same environment. So then you'd have to set RPS to disabled on NW server level.
Dsantibanez
1 Rookie
•
82 Posts
0
May 13th, 2021 07:00
UPDATE:
I again, yesterday we upgraded both networkers sites from NW 19.2.1.2 to 19.3.0.4
early cloning jobs says that the issue is gone (we hope), tapes are mounting faster and the GUI-console is faster too.
bingo.1
2.4K Posts
1
May 13th, 2021 08:00
May I suggest that you also read this thread: https://www.dell.com/community/NetWorker/retrying-mmd-reservation-request/td-p/7859085