dpnctl: ERROR: gsan is not up or is exhibiting errors - not performing MCS restore

Hi all,

 

After crash of VM in Azure AVE 7.5.1 did not come back:

admin@avamarveazure:~/>: dpnctl status
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: gsan status: not running
dpnctl: INFO: MCS status: down.
dpnctl: INFO: emt status: down.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: unknown.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: down.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

admin@avamarveazure:~/>: dpnctl start
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
ssh: connect to host 10.71.232.54 port 22: Network is unreachable
admin@avamarveazure:~/>: nslookup 10.71.232.54
Server: 10.2.132.213
Address: 10.2.132.213#53

54.232.71.10.in-addr.arpa name = avamarveazure.Go.JohnsonControls.com.

admin@avamarveazure:~/>: dpnctl start
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)

- - - - - - - - - - - - - - - - - - - -
Action: starting all
Have you contacted Avamar Technical Support to ensure that this
is the right thing to do?

Answering y(es) proceeds with starting all;
n(o) or q(uit) exits

y(es), n(o), q(uit/exit): y
dpnctl: INFO: Checking that gsan was shut down cleanly...

- - - - - - - - - - - - - - - - - - - -
Here are the most recent validated and non-validated checkpoints:
Wed Sep 16 16:01:53 2020 UTC Validated(type=rolling)
Wed Sep 16 16:05:40 2020 UTC Not Validated

A rollback is recommended: the gsan was not shut down cleanly.

The choices are as follows:
1 roll back to the most recent checkpoint, whether or not validated
2 roll back to the most recent validated checkpoint
3 select a specific checkpoint to which to roll back
4 do not restart
q quit/exit

(Entering an empty (blank) line twice quits/exits.)
> 2

- - - - - - - - - - - - - - - - - - - -
You have selected this checkpoint:

name: cp.20200916160153
date: Wed Sep 16 16:01:53 2020 UTC
validated: yes
age: 1 day, 16 hours

Roll back to this checkpoint?

Answering y(es) accepts this checkpoint and initiates rollback
n(o) rejects this checkpoint and returns to the main menu
q(uit) exits

y(es), n(o), q(uit/exit): y
dpnctl: INFO: Initiating rollback to "cp.20200916160153" with gsan restart (this may take some time)...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-gsan-restart-output-7121
dpnctl: ERROR: error return from "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20200916160153"" - exit status 1
dpnctl: ERROR: 10 errors seen in output of "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20200916160153""
dpnctl: ERROR: error return from "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20200916160153"" - exit status 1
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-subsystem-control-action-output-7121
dpnctl: ERROR: gsan is not up or is exhibiting errors - not performing MCS restore
dpnctl: ERROR: MCS restore did not succeed, so not restarting MCS
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-subsystem-control-action-output-7121
dpnctl: INFO: EM Tomcat started.
dpnctl: ERROR: cannot resume backup scheduler: gsan is down and MCS is down
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
dpnctl: INFO: AvInstaller is already running.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

After changing user under root I ran again dpnctl start and it went successful. Not sure exactly what was the problem in this case but hopefully it will be useful

root@avamarveazure:~/#: dpnctl start
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)

- - - - - - - - - - - - - - - - - - - -
Action: starting all
Have you contacted Avamar Technical Support to ensure that this
is the right thing to do?

Answering y(es) proceeds with starting all;
n(o) or q(uit) exits

y(es), n(o), q(uit/exit): y
dpnctl: INFO: Checking that gsan was shut down cleanly...

- - - - - - - - - - - - - - - - - - - -
Here are the most recent validated and non-validated checkpoints:
Wed Sep 16 16:01:53 2020 UTC Validated(type=rolling)
Wed Sep 16 16:05:40 2020 UTC Not Validated

A rollback is recommended: the gsan was not shut down cleanly.

The choices are as follows:
1 roll back to the most recent checkpoint, whether or not validated
2 roll back to the most recent validated checkpoint
3 select a specific checkpoint to which to roll back
4 do not restart
q quit/exit

(Entering an empty (blank) line twice quits/exits.)
> 2

- - - - - - - - - - - - - - - - - - - -
You have selected this checkpoint:

name: cp.20200916160153
date: Wed Sep 16 16:01:53 2020 UTC
validated: yes
age: 1 day, 16 hours

Roll back to this checkpoint?

Answering y(es) accepts this checkpoint and initiates rollback
n(o) rejects this checkpoint and returns to the main menu
q(uit) exits

y(es), n(o), q(uit/exit): y
dpnctl: INFO: Initiating rollback to "cp.20200916160153" with gsan restart (this may take some time)...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-gsan-restart-output-10123

 

dpnctl: WARNING: 5 warnings seen in output of "/usr/bin/yes no | /usr/local/avamar/bin/rollback.dpn --cptag="cp.20200916160153""
dpnctl: WARNING: 4 warnings seen in gsan error logs:
- - - - - - - - - - - - - - - - - - - - BEGIN
(0.0) 2020/09/18-08:56:44.81226 {P0.0} [gsan] WARN: <1359> stripedns::validateandupdatediskstats update previous devno=0x861 to new devno=0x841 [dev=841 total=262009MB avail=256870MB reserved=4621MB state=online]
(0.0) 2020/09/18-08:56:44.81227 {P0.0} [gsan] WARN: <1359> stripedns::validateandupdatediskstats update previous devno=0x841 to new devno=0x861 [dev=861 total=262009MB avail=256981MB reserved=4611MB state=online]
(0.0) 2020/09/18-08:57:47.84184 {0.0} [perfbeat.3:286] WARN: <0966> perfbeat::restore devname mismatch oldname=/dev/sdg1 dev=[/dev/sde1 fd=1698 flags=16384 maxoff=0 appending=0 stat=[dev=f ino=1222 mode=60660 nlink=1 uid=0 gid=6 rdev=2113 size=0 blksize=4096 blocks=0 atime=1600418260 mtime=1600418258 ctime=1600418258]]
(0.0) 2020/09/18-08:57:47.85251 {0.0} [perfbeat.5:290] WARN: <0966> perfbeat::restore devname mismatch oldname=/dev/sde1 dev=[/dev/sdg1 fd=1699 flags=16384 maxoff=0 appending=0 stat=[dev=f ino=2725 mode=60660 nlink=1 uid=0 gid=6 rdev=2145 size=0 blksize=4096 blocks=0 atime=1600418260 mtime=1600418258 ctime=1600418258]]
- - - - - - - - - - - - - - - - - - - - END

dpnctl: INFO: rolling back to checkpoint "cp.20200916160153" and restarting the gsan succeeded.
dpnctl: INFO: gsan started.
dpnctl: INFO: Restoring MCS data...
dpnctl: INFO: MCS data restored.
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-10123
dpnctl: WARNING: 5 warnings seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start"
dpnctl: INFO: MCS started.
dpnctl: INFO: EM Tomcat is already running, not attempting to restart it
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
dpnctl: INFO: AvInstaller is already running.
root@avamarveazure:~/#: dpnctl status
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
root@avamarveazure:~/#: