networker skipping oracle backup files, but still saying backup completed with no issues

Sorry for my lack of experience with oracle and other databases - I can just help you with some general thoughts.

I think you are mixing some facts here:

  -  VSS writers are part of the OS (Windows), not part of NetWorker.

  -  That's why NW works with its own 'exceptions' aka directives, either globally defined (as a resource) or locally (via a .nsr file).

  -  On Windows, there is also a registry entry called HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToBackup which also defines exceptions.

If NW does not detect the filenames during the backup startup phase (during the so-called 'probe' process), it will not add them to its internal worklist. But if all listed could be backed up successfully, then the backup will end successfully.

So if the 'exception' will be set by Windows or a VSS writer, then NW will not overrule these definitions.

However, if NW would have set the exception, then you should see an appropriate directive (file or resource) which you may change of course.

But by default, there is no general rule that NW will skip *.bkp files.

I still do not completely understand your configuration. I think, the VSS oracle writer will automatically be installed with the product itself.

As you say, you are not working with a module but exporting the databases to these 'backups' will be treated as ordinary files. Consequently their backup configuration can be manipulated by NW directives. And I am really surprised that NW mentions an '... internal skip directive ...' as it will by default not skip *.bkp files just for their names.

Maybe the person who configured the system added such directive. If so, you could simply modify it. If not, you might add a NW 'always' directive to explicitly backup these files.

But be careful to apply it correctly - if you have a 'skip' on a directory an 'always' on a subdirectory will not help as NW will already skip this directory branch.

Sorry I was not clear . The issue is indeed with oracle and its VVS writer that has a very odd VSS "directive" (for want of a less confusing term). The servers were created by the same person so it could be there is a config issue, however i suspect it is with oracle assuming any backup of "fast recovery area" would be part of a "live backup" and the *.bkp files should not be part of that.

The test was to run the backup with oracle VSS running and then with it turned off. Sure enough  when off, no file was skipped

It was only when we did a "-vvvvv" that we could see networker obeying the oracle vss "directive" to skip certain files.

walk(\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\XX21\1C18...4C\BACKUPSET\2022_05_04\O1_MF_NNNDF_TAG20220504T183928_K75JH0LN_.BKP, O1_MF_NNNDF_TAG20220504T183928_K75JH0LN_.BKP)
32246:save: matched internal `skip' on `O1_MF_NNNDF_TAG20220504T183928_K75JH0LN_.BKP' for `\\?

However I would say that there should be an option to report "skipped files" in backup notifications. 

Oracle VSS writer is installed by default, however it is independent of the product. It only comes into its own when a backup program is wanting to do a live backup of the actual database files while it is in use. If you don't backup oracle db like this then you might as well turn it off. 

I think oracle VSS is hardcoded to exclude certain filetypes in specified folders, as it presumes you only backup those folders during a "live" db backup. 

I should add that despite me searching I could not find any oracle VSS "directives". Unlike windows (and a few other apps), there is no registry setting or "*.conf" file.

I came across a couple of posts with an associated issue, but not quite matching. Here a user of another backup product says they are getting a successful backup despite control and redo not being backed up.

"Oracle CTL and REDO files are not able to be backed up in spite of exit status 0" (I have removed the link to a rival product of respect)

To summarise networker is not at fault (but a "some files were skipped" notification would be good), all it is doing is simply following a bizarre "directive" from oracle vss writer to skip certain files. 

not exaclty the same, but we ran into a situation with an application that NW does not have a module for. Also on Windows. Unlike with Unix/Linux, whenever files are accessed/locked in a specific way, silently any filesystem backup would skip the files. Only when trying to restore data it was noticed certain files where not there. Of course this should have been tested and evaluated at a much earlier state and not when actually needing the files in question.

But still the NW documentation is not clear in which cases actually files would be skipped, nor does it even log this anywhere clearly.

So while unix/linux mentions data possibly being pointless (changed during save,if you forgot to skip DB files from the filesystem backup), on Windows files simply certain - let's call them "open" - files will not be backed up AND not being reported at all, not even when enabling verbose (maybe if we'd have used -vvvvv it might have, based on above, but I believe very verbose settings were tried, also by specifying only the specific file as saveset, it simply returned to save 0 bytes).

So the reporting of NW has much to be desired for wrg to skipping files in these kind of cases.

We didn't even get a way to check this, for example if files have this or that locked status, then NW will not be able to backup these files, so that one could perform a pre-active/preventive check for these kind of files by the OS team, so that they might report this to application teams, that certain files might not end up in the backup.

Now it might only turn up when specifically checking for them in a backup or when restoring. Don't get me wrong, anyone responsible for specific data, should always check and test their backups regularly (*), but the we know that that is not always the case...

(*) which is not the backup team responsibility, but the rather the responsible team for the respective data, as only they can actually assess the data and see if it is valid and working.

So for the SR that we had last year, we asked for a RFE, but up until now, no response yet when and if that will be introduced. Now one can still have files being skipped silently, with no way to check/report this from NW end...

Something akin to it, is also the skipping of MSSQL DB files, when not having even the NW NMM module installed (as NW NMM has build-in feature for this, being applied to filesystem backups if I recall correctly?). So when the DB is running - unlike on unix/linux - the DB files will not be backed-up, to the amazement of some DB admins, who expect it to be in the backup (without having tested nor validated this, simply by "assuming" it would).

We from backup end, would always regard the filesystem backup having open DB files in it as pointless, as there is no guarantee whatsoever for any consistency. We wouldn't even want to call it a crash-consistent backup, like you might have with a NW vproxy based vmware image level backup of a VM. And even then it would require the DB to be able to handle a crash-consistent backup even.

But a normal filesystem backup saving database files of a running DB, is worthless data, and hence should be skipped always, but it should show that somewhere, it does do this, when using NW directives for example.

I don't consider silent skipping of any files as good practice for an enterprise level data protection solution... If you skip it for any reason, please mention doing so, as then you give the customers a way to try to mitigate against it. And if you know you product through and through, you also should know when this behavior is taking place, hence inform the customers about how you product works and deals with certain situations. Up until nw19.6 the NW manuals still don't, it is a simple rehash of pretty much the same statements going back to at least nw8 or so.

agree regarding "I don't consider silent skipping of any files as good practice for an enterprise level data protection solution". I guess it's to reduce the mess of consistently seeing files that will always be skipped (swap/hyber/temp etc). 

We don't backup live databases but instead backup export/rman files (we also take the db down every two months and backup "all"). It is either rman or export that creates bkp files in "fast recovery area" (a oracle term). Those files do not change (until we do another rman/export) . So we thought "we are backing up local backup so we have a copy on tape", only to discover key files were being skipped.

again not networkers fault (but "notification").

I can only guess that oracle VSS writer assumes that everything is a live backup and that there is no need to backup the bkp files in the recover folder as otherwise you are backing up two backup (live backup and a previous rman/export). However it does not make complete sense. 

could you also report the lines just above the "matched internal skip"?

As that is where I'd expect it would state where the skip comes from?

For example :
https://www.dell.com/support/kbdoc/en-us/000105046/how-to-test-if-the-skip-directive-is-working?lang=en
shows :

Scenario:

So the skip comes from a directive defined on client end inside file /.nsr (so within the root directory). However in your case it also reports about an internal skip? So do for example there also exist any skip directives on client end? Are there any local directives being created on client end?

As stated in the nw19.5 admin manual:

Depending on user privileges and the operating system version, the networkr.cfg file is created in one of the following locations:
● If you are logged in with local Windows Administrator or Backup Operator privileges, networkr.cfg is created in the root of the system volume (usually C:\).
● If you are not logged in with local Windows Administrator or Backup Operator privileges, networkr.cfg is created in %SystemDrive%\Documents and Settings\User_name\Application Data\EMC NetWorker.
NOTE: The Application Data directories are hidden by default. To view these directories by using Windows Explorer, select Tools > Folder Options. On the View tab of the View Options dialog box, select the Show hidden files and folders option.

What is it with dell. Every time I post, or click on a link someone sends i get asked to pick traffic lights.

174422:save: Step (6 of 7) for PID-22396: Reading the save sets and writing to the target device.

chdir(\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\1C18FD4DE2594513AE4261B3526DA74C\BACKUPSET\2022_05_04)

Name=`\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\1C18FD4DE2594513AE4261B3526DA74C\BACKUPSET\2022_05_04\', name=`\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\1C18FD4DE2594513AE4261B3526DA74C\BACKUPSET\2022_05_04\', fname=`.\'87713:save: Found prototype specification for '/':  mntasm : C:

87713:save: Found prototype specification for '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\':  skip : hiberfil.sys

87713:save: Found prototype specification for '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\':  skip : CONTROL02.CTL

87713:save: Found prototype specification for '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\1C18FD4DE2594513AE4261B3526DA74C\BACKUPSET\2022_05_04\':+ skip : O1_MF_NNNDF_TAG20220504T183928_K75JH0LN_.BKP

walk(\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\1C18FD4DE2594513AE4261B3526DA74C\BACKUPSET\2022_05_04\, .\)

87333:save: '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy76\app\oracle\fast_recovery_area\CDB21\1C18FD4DE2594513AE4261B3526DA74C\BACKUPSET\2022_05_04\' change time Wed May 04 19:11:44 2022

For readability, it would be best to reply to the actual post instead of creating a new reply, but the Dell community interface is far from ideal (and the never ending captchas annoying as heck). Also not very helpful, that it doesn't even state who and when the last reply was made. Comparing that with other current community websites, one would have thought Dell might have been able to get a better solution in place that this. (end rant)

I indeed don't see a line akin to:

66135:save: NSR directive file (C:\NETWORKR.CFG) parsed

(example output from https://www.dell.com/support/kbdoc/en-us/000163738/networker-for-windows-18-2-0-1-archiving-with-save-e-forever-or-save-y-forever-does-not-set-the-retention-to?lang=en).

...after the log line with "Found prototype specification for '/': mntasm : C:" in it.

So regular client side directives do not seem to be used. So if NW uses some other method to skip anything, it would be helpful about how that works exactly, so to be able to look for it (ideally without having to increase the verbosity as that is almost unworkable on a larger system as it would create way too large log files).