Unsolved
1 Rookie
•
27 Posts
0
1682
September 27th, 2021 06:00
Backup Override did not run properly
Hi,
I'm running Networker 19.5.0.1 on 2 different environments. We are backing up dailies to data domains and monthlies to LTO7 tapes.
Both environment are scheduled create Full backups on Last Friday of every month. These full backups are then cloned to tapes for offsite backups. Every other day of the month are incremental backups; only the last friday of the month is full.
I am configuring/scheduling this through each job and using overrides. In the backup job, I set the period as "Weekly by day" and setup incremental for every day. Then, in the overrides pane I setup "full last friday every month". We've been running like this since the past 2 years with no issues. This is the same for all backup jobs and for both environments.
However, last friday (24th September 2021) was the Last Friday of the month and the override did not work. Incremental backups were taken instead of Full and this caused issue for the cloning process. This happened for both environment.
I am trying to figure out why the override didn't work for Last Friday but worked for the past 2 years. The logs indicate that an incremental backup was taken (level=incr); no sign of a reason why the override didn't occur.
The backups jobs configuration did not change since the last Full; only the clone job was disabled since I'm now cloning using Powershell instead.
I know this may be a complicated issue to troubleshoot, but I'm mostly looking for pointers on where to look.
Thank you,
Guillaume.
bingo.1
2.4K Posts
0
September 27th, 2021 07:00
If it worked for 2 years then I suspect the new NW version to be buggy as it did not exist at that point in time.
Did you already contact EMC support?
guillaume.leonard
1 Rookie
•
27 Posts
0
September 27th, 2021 07:00
Thanks bingo.1.
I did not contact EMC Support yet. I tried the forums first.
I will get in touch with them. Thanks again !
barry_beckers
393 Posts
0
October 4th, 2021 03:00
I expect this to become addressed for the next nw19.5.0.2 or later release as nw19.3 and 19.4 appear to have been receiving a fix in the latest CHF releases for exactly this issue when dealing with months that have 30 days.
Too bad that for ticket 332314 it is also not explained clearly what it is actually about? Does it mean that nw19.4 already contains a fix for that?
NW19.3.0.6 (released 8 sept 2021):
332314 NetWorker Escalation 40575: override rules in backup action is not honored. Change all uses of unsafe lg_localtime() to safe lg_localtime_r() (edit)
331233 NetWorker Escalation 40597:Schedules with override of 'full on last saturday of month' on 30-day months do not work, skipping backups
NW19.4.0.4 (released 1 sept 2021):
331233 NetWorker Escalation 40597: Schedules with override of 'full on last saturday of month' on 30 day months do not work, skipping backups
guillaume.leonard
1 Rookie
•
27 Posts
0
October 4th, 2021 05:00
Thank you barry. I did not have time to contact DELL last week, but seeing this fix on the other branches of Networker gives me hope that it will be fixed in 19.5.
I have been encountering quite a lot of bugs lately with Networker. Pretty much every version I upgrade to fix a bug, introduces another one that impacts us
barry_beckers
393 Posts
0
October 4th, 2021 05:00
That is indeed as always cumbersome, not really helped by the fact that at times the higher release frequency, seems to prevent thorough testing at Dell end also.
Otherwise I can't explain why nw19.5.0.0 had a very quick follow up within 3 weeks with a fix that addressed nsrck to fail to run due to too many open files error? That is basic NW functionality, part of the server protection policy, to run that daily. Basic functionality testing should have shown that.
nw19.5.1.0 fix list:
Escalation 40667: nsrck:Cannot lock file index for client (Too many open files)
Also the shift from a stable, thoroughly tested and used DPS software version (the whole data protection suite) listing the whole stack and advised versions, was let go for a shift towards advising the latest and greatest.
However why is then nw19.5 not yet target code? nw19.4 is target code. As we tend to have even one version lower than the target code as internal advised version, that would make it nw19.3. But due to some DTA's and DSA's, it would have to become nw19.4.0.4 as not all fixes are back-ported into nw19.3 (yet... if even at all).
The same for datadomain, the target code is even lower, still ddos7.2, while 7.6 is released already, which might make for a great version disparity between a DD on-prem and if you'd have one in the cloud, which offers the latest an greatest by default (in AWS at the moment ddos7.6.0.6, nothing lower than ddos7.1). Even more so if still running ddos6.2 on physical dd's, especially for the ones that do not even allow any upgrade beyond ddos6.x, like dd2500's or dd990's. This already means if you also use DDMC to manage multiple DD"s, you'd have to have multiple DDMC versions to be able to manage ddos6.2 and 7.x as well.
But keeps us busy and of the street...
bingo.1
2.4K Posts
0
October 4th, 2021 06:00
Hi Barry,
you did mention NW 19.5.1.0 - is this correct? If so, from where do you get it?
The latest version I can download is 19.5.0.1.
Thanks and regards
barry_beckers
393 Posts
0
October 4th, 2021 07:00
that is indeed a typo. should have stated nw19.5.0.1 fixlist.
barry_beckers
393 Posts
0
October 5th, 2021 04:00
So nw19.5.0.2, just released on Oct. 4th should contain the fix:
331233 NetWorker Escalation 40597: Schedules with override of 'full on last saturday of month' on 30 day months do not work, skipping backups
https://dl.dell.com/downloads/DL106442_NetWorker-19.5.0.2-for-Linux-for-x64.gz
https://dl.dell.com/downloads/DL106448_NetWorker-19.5.0.2-for-Windows-x64.zip
barry_beckers
393 Posts
0
October 5th, 2021 04:00
nw19.5.0.2 has just been released. It indeed mentions:
331233 NetWorker Escalation 40597: Schedules with override of 'full on last saturday of month' on 30 day months do not work, skipping backups
contains all these fixes, some of which rather sound severe making nw19.5.0.2 the go-to release for nw19.5 (and others become important when you use the NW REST API):
Ticket Product Description
330156 NMM Escalation 40191: Fix to include Microsoft OLE DB Driver for SQL Server for SP Farm Backup
331538 NetWorker Escalation 40679: nsrim removes a saveset that has "dependent" savesets
332314 NetWorker Change all uses of unsafe lg_localtime() to safe lg_localtime_r()
331930 NetWorker Escalation 40336: nsrd crashes when an old SN talks to it.
331697 NetWorker Escalation 40606: implement longer timeout for starting dispatcher service
331374 NetWorker VMware backup action should attempt retry if exception is caught
331233 NetWorker Escalation 40597: Schedules with override of 'full on last saturday of month' on 30 day months do not work, skipping backups
331184 NetWorker DD to Tape Cloning (RPS Enabled) of 100 filesystem clients fails for one client with "Invalid mmd reservation request"
331175 NetWorker Escalation 40458: Huge difference in nsrcapinfo outputs for Oracle when queried from a 18.2 and 19.4 host
331080 NetWorker Auth services not coming up in cluster on failover
331079 NetWorker Escalation 40357: nsrtrap syntax error after upgrading to 19.4
330943 NetWorker Escalation 40381: PSS backup is marked as "complete/successful", despite unrecoverable read file error
330783 NetWorker Escalation 40405: Restapi return empty list when use query string for client index with linux path "/"
330748 NetWorker Escalation 40352: REGRESSION(nsrclone/nsrstage fails): The DDRL has expired but the save set cannot be staged or nsrclone -m
330313 NetWorker Escalation 39142: hanging nsrmmdbd on networker 19.3.0.1 backup server due to an exclusive-lock(SQLITE_LOCKED) not getting removed by a thread and other threads not able to get their operations done.
327740 NetWorker Escalation 39289: Networker REST API: Client index custom header X-NW-CONTINUATION-TOKEN pagination intermitently returns duplicate values
331609 NetWorker Escalation 40369: daemon.raw realtime rendered log incomplete
332258 NetWorker libddobbost integration 19.5 CHF 7.4.0.8_b688525 and libddboost integration to 7.7.0.0_b1002040
guillaume.leonard
1 Rookie
•
27 Posts
0
October 6th, 2021 10:00
Thanks guys ! I'll update within the new weeks