NDMP request header with error: 4; ... no password for user root

Looking at the provided log, u are using the root acc

it seems the root password is expired.

i have seen that a couple of times, logging in with root directly prompts for a password change, nasadmin and su works fine.

Dont use root acc for NDMP, use a NDMP acc, if u already have one, check that acc so it dont have a expired pass

Hi,

we have a ndmp user for backup configured and backup admin tells he use this ndmp user.

Backup has worked until 13. of Jan. 2015, and now we try to take new backups and get these error.

dack

Why you are used root account for that?

Need to create an ndmp user first, and use this one for all the backups.

Is a networker server or another vendor? Can you confirm the configuration for the user on the backup software?

NDMP can be tricky sometimes and standard NDMP ERR log dosn´t always show what u need to see.

Enable NDMP debug log on the CS

Enable

.server_config server_2 "logsys set severity NDMP=LOG_DBG2"

.server_config server_2 "logsys set severity PAX=LOG_DBG2"

then start a backup and dig in to the ndmp logfile on the CS

to change back to standard logging just replace LOG_DBG2 wirh LOG_ERR

Hi,

we use CommVault and have created a new ndmp account.

Now we get an new error:

2015-01-26 15:52:12: NDMP: 6: Active NDMP backup/restore streams: 3, system configured concurrent streams: 4, maximum concurrent sessions supported: 4.

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) < Backup type: dump >

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: DEBUG Value: n

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: DIRECT Value: y

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: EMC_OFFLINE_DATA Value: n

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: FILESYSTEM Value: /root_vdm_5/fs-hqcifs-d

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: HIST Value: y

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: LEVEL Value: 1

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: SNAPSURE Value: y

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: TYPE Value: dump

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: UPDATE Value: y

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: VBB Value: n

2015-01-26 15:57:12: NDMP: 6: Thread ndmp510 SnapUtils::snapCreate() timeout srcFsId: 6451 snapsureMountPath: /root_vdm_5/automaticNDMPCkpts/automaticTempNDMPCkpt369-6451-1422283932

2015-01-26 15:57:12: NDMP: 3: < LOG type: 2, msg_id: 0, entry: SnapSure file system creation fails, hasAssociatedMsg: 0, associatedMsgSeq: 0 >

2015-01-26 15:57:12: NDMP: 4: Session 510 (thread ndmp510) NdmpdData::startBackup, SnapSure creation for backup failed

2015-01-26 15:57:14: NDMP: 6: Cleanup: Active NDMP backup/restore sessions back to: 2, system configured concurrent streams: 4, maximum concurrent sessions allowed: 4.

BUT

I can manually create a checkpoint...

[dradmin@CS03 ~]$ fs_ckpt fs-hqcifs-d -name ckpt_hqcifs-d-test -Create

operation in progress (not interruptible)...id        = 6451

name      = fs-hqcifs-d

acl       = 0

in_use    = True

type      = uxfs

worm      = off

volume    = v10228

pool      = cmata_r6

member_of = root_avm_fs_group_20

rw_servers= server_2

ro_servers=

rw_vdms   = vdm4

ro_vdms   =

auto_ext  = no,virtual_provision=no

deduplication   = Off

ckpts     = ckpt_hqcifs-d-test

stor_devs = CKM00093400053-0190,CKM00093400053-0193,CKM00093400053-0191,CKM00093400053-0194,CKM00093400053-0192,CKM00093400053-0195

disks     = d24,d23,d26,d25,d28,d27

disk=d24   stor_dev=CKM00093400053-0190 addr=c16t2l0        server=server_2

disk=d24   stor_dev=CKM00093400053-0190 addr=c0t2l0         server=server_2

disk=d23   stor_dev=CKM00093400053-0193 addr=c0t2l1         server=server_2

disk=d23   stor_dev=CKM00093400053-0193 addr=c16t2l1        server=server_2

disk=d26   stor_dev=CKM00093400053-0191 addr=c16t2l2        server=server_2

disk=d26   stor_dev=CKM00093400053-0191 addr=c0t2l2         server=server_2

disk=d25   stor_dev=CKM00093400053-0194 addr=c0t2l3         server=server_2

disk=d25   stor_dev=CKM00093400053-0194 addr=c16t2l3        server=server_2

disk=d28   stor_dev=CKM00093400053-0192 addr=c16t2l4        server=server_2

disk=d28   stor_dev=CKM00093400053-0192 addr=c0t2l4         server=server_2

disk=d27   stor_dev=CKM00093400053-0195 addr=c0t2l5         server=server_2

disk=d27   stor_dev=CKM00093400053-0195 addr=c16t2l5        server=server_2

id        = 14786

name      = ckpt_hqcifs-d-test

acl       = 0

in_use    = True

type      = ckpt

worm      = off

volume    = vp18351

pool      = cmata_r6

member_of =

rw_servers=

ro_servers= server_2

rw_vdms   =

ro_vdms   = vdm4

checkpt_of= fs-hqcifs-d Mon Jan 26 16:07:22 CET 2015

deduplication   = Off

used      = 1%

full(mark)= 90%

stor_devs = CKM00093400053-0192,CKM00093400053-0195

disks     = d28,d27

disk=d28   stor_dev=CKM00093400053-0192 addr=c16t2l4        server=server_2

disk=d28   stor_dev=CKM00093400053-0192 addr=c0t2l4         server=server_2

disk=d27   stor_dev=CKM00093400053-0195 addr=c0t2l5         server=server_2

disk=d27   stor_dev=CKM00093400053-0195 addr=c16t2l5        server=server_2

Hi,

thank all for help unfortunately without a result.

but

because of using DR-Data Mover (of mirrored Arrays) with loss of function we decided to abort ndmp backups.

Best regards

dack

If you ever decide to resume NDMP backups, your problem could be very easy to fix:

2015-01-26 15:52:12: NDMP: 4: Session 510 (thread ndmp510) Name: VBB Value: n

2015-01-26 15:57:12: NDMP: 6: Thread ndmp510 SnapUtils::snapCreate() timeout srcFsId: 6451 snapsureMountPath: /root_vdm_5/automaticNDMPCkpts/automaticTempNDMPCkpt369-6451-1422283932

This is a very common occurrence with NDMP backups.  Basically, the backup application waits a fixed amount of time for the NAS to create a snapshot.  If the NAS takes longer than this window, the backup application receives a timeout.  You can increase this delay on the NAS very easily.

In your log above, notice that the NDMP backup began at 15:52:12, then you get a snapCreate() timeout at 15:57:12.  That tells me your snap Timeout value is set to 5 minutes.  This is the default on the EMC Celerra.  You can check this from the CLI:

[nasadmin@vnx-test~]$ server_param server_2 -facility NDMP -info snapTimeout

server_2 :

name                    = snapTimeout

facility_name          = NDMP

default_value          = 5

current_value          = 5

configured_value        =

user_action            = none

change_effective        = immediate

range                  = (2,60)

description            = NDMP backup SnapSure file system creation timeout value (in minutes)

You can increase this value without a reboot by using a command on the CLI:

[nasadmin@vnx-test ~]$ server_param server_2 -facility NDMP -modify snapTimeout -value 15

Done.

[nasadmin@vnx-test~]$ server_param server_2 -facility NDMP -info snapTimeout

server_2 :

name                    = snapTimeout

facility_name          = NDMP

default_value          = 5

current_value          = 15

configured_value        =

user_action            = none

change_effective        = immediate

range                  = (2,60)

description            = NDMP backup SnapSure file system creation timeout value (in minutes)

My snapTimeout value is now set to 15 minutes. 

Please give this a try before you give up on NDMP backups.  Just because you have a copy of your data replicated to the DR site does not mean someone may need a much older file!

Let us know if this helps!

Karl

Karl,

should it take that long to create a snapshot ? I remember on my aging NS80 i could create a snapshot of 16TB file system under a minute.

Normally not – but there are exceptions

For example if it’s the first ckpt than the savvol has to be created first

fair enough, but 5 minutes for an empty savvol ? ...must be a very busy array.

I'm with Rainer on this one - there's always a gotcha.  Our old NS80 never took more than 5 minutes to create a snap from the CLI.  But whenever Netbackup would call a snap as part of the backup?  Some of them took as long as 6 minutes each time!  Symantec gave us a patch for NBU 7.0.2 that didn't help, so we increased the timeout and never looked back.

Is this a backup of filesystem a DR Box which is the destination of a IP-Replication?

As long as the IP-Replication is "updating" the destination FS, the creation of a checkpoint on this destination FS is waiting for the end of the replication update. If this is taking longer than the configured snapTimeout, the checkpoint creation will fail.

Hi Guys,

thank you all again.

We've stopped investigating this problem. Support has found a ecm-issue which describe this to assume because on DR-Box of a sync mirrored celerra are restricted rights for dradmin.

Instead of NDMP Backup we backup over UNC-Path now.

Anyway, it's a final backup of these dates; after that we shutdown the box.

Best Regards

dack