Unsolved
This post is more than 5 years old
6 Posts
0
3096
December 6th, 2013 07:00
NAS (Celerra) 6.0.40 gui not coming up
I am trying to decommission the storage and wipe the drives. Yesterday I was able to access the Unisphere most of the day. After a few hours the connection hung and since them I have not been able to get back into it.
Yesterday as part of the wipe I removed all shares, luns, disk groups (EXECEPT I kept the raid group for the "NAS/OS"). I then regrouped all the disks into 2 arrays so I could wipe them (again leaving the NAS/OS raid group alone). However when I went back into the Unisphere it had hung and i have been unable to get it to come up so I can create a share that to my raid groups.
I have powered off the data movers and the nas and I am still unable to get it to work.
Any suggestion on what has gone wrong?
p.s. i can still access the navisphere.
Thanks
bg
----------------------
[nasadmin@cotys ~]$ su -
Password:
[root@cotys ~]#
[root@cotys ~]# /nasmcd/sbin/getreason; /nas/tools/nas_summary; /nas/bin/nas_checkup;
10 - slot_0 primary control station
0 - slot_2 reset
0 - slot_3 reset
nas_summary v2.1
cat: /nas/server/servers: No such file or directory
cat: /nas/server/servers: No such file or directory
cat: /nas/server/servers: No such file or directory
cat: /nas/server/servers: No such file or directory
cat: /nas/server/vdm/vdms: No such file or directory
cat: /nas/server/vdm/vdms: No such file or directory
cat: /nas/server/vdm/vdms: No such file or directory
cat: /nas/server/vdm/vdms: No such file or directory
cat: /nas/volume/symms: No such file or directory
cat: /nas/volume/filesys: No such file or directory
grep: /nas/server/slot_*/netd: No such file or directory
grep: /nas/server/slot_*/export: No such file or directory
grep: /nas/site/cshosts: No such file or directory
/nas/tools/nas_summary: line 349: [: -gt: unary operator expected
Collecting Replication info...Please Wait
#####################################################
System type: NS20FC (Gateway)
Version: 6.0.40-8
#####################################################
Number of data movers: 0 VDMs: 0
#####################################################
Number of arrays: 0
NAS: NAS NAS
#####################################################
Filesystems: UXFS:1 Checkpoint:1 Total:0
Number of disk vols (LUNs): 1
#####################################################
Replication mode: RepV2 Replication license not enabled
# of Interconnects: 1 Remote Celerras: 1
#####################################################
Features in use: CIFS: False iSCSI: *True* NFS: False
Rep: False SRDF: False DHSM: *True*
#####################################################
DBCHK State: Errors found!
Check Version: 6.0.40.8
Check Command: /nas/bin/nas_checkup
Check Log : /nas/log/checkup-run.131206-141509.log
-------------------------------------Checks-------------------------------------
Control Station: Checking if NBS clients are started....................... Pass
Control Station: Checking if NBS configuration exists...................... Pass
Control Station: Checking if NBS devices are accessible.................... Fail
Control Station: Checking if NBS service is started........................ Fail
Control Station: Checking if NAS partitions are mounted.................... Pass
Blades : Checking status........................................... Warn
--------------------------------------------------------------------------------
One or more errors have occurred. Follow the instructions provided to correct
the problem and try again.
------------------------------------Warnings------------------------------------
Blades : Check status
Warning HC_DM_18800115755: The following Blades are rebooting:
* server_2
* server_3
Because of this, several other Blade checks may return a
question mark "?".
Action : Please wait a few minutes to allow the Blade to finish
rebooting. Then use the /nasmcd/sbin/getreason command to obtain the
reason code and ensure that it completes the boot cycle (reason code
should be 4 or 5). Once the boot cycle is complete, re-run nas_checkup
so that all Blade checks can be completed. If the Blade does not
complete the boot cycle, use server_log to check for errors on the
Blade. Escalate this issue through your support organization if
necessary. Provide this output and any output or errors received from
running these commands in the escalation.
--------------------------------------------------------------------------------
-------------------------------------Errors-------------------------------------
Control Station: Check if NBS devices are accessible
Error HC_CS_14505082914: Failed NBS (nd-clnt processes) devices
access check
NOTE: Several checks depend on NBS device access to run. These checks
were not run.
Action :
1. This may occur if NBS is not configured correctly or if the NBS
service is not started. Look in the "Checks" section to see if the
following checks passed:
* Control Station: Check if NBS configuration exists
* Control Station: Check if NBS service is started
* Control Station: Check if NBS clients are started
If either of those checks did not pass, follow the instructions for
that check to correct the problem, then rerun nas_checkup to verify
that the NBS devices can now be accessed.
2. This may also occur if Blades are powered down or pulled out. If
you are on the primary Control Station, look in the "Checks"
section to see if the following check passed:
* Blades: Check status
If this check failed, follow its instructions to correct the
problem, then rerun nas_checkup to verify that the NBS devices can
now be accessed.
3. If the problem persists, escalate this issue through your support
organization. Provide this output and any errors or output that
occurred running the commands in this procedure in the escalation.
Control Station: Check if NBS service is started
Error HC_CS_14505082915: NBS (nd-clnt processes) service is not (or
not fully) started
Action :
Use the command "/sbin/service nbs start" to restart the NBS service
or reboot the Control Station.
--------------------------------------------------------------------------------
[root@cotys ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 2.0G 1.4G 526M 73% /
none 1012M 0 1012M 0% /dev/shm
/dev/hda1 259M 12M 237M 5% /boot
/dev/mapper/emc_vg_pri_ide-emc_lv_home
591M 17M 544M 3% /home
/dev/mapper/emc_vg_pri_ide-emc_lv_celerra_backup
827M 216M 570M 28% /celerra/backup
/dev/mapper/emc_vg_pri_ide-emc_lv_celerra_backendmonitor
7.8M 1.2M 6.3M 16% /celerra/backendmonitor
/dev/mapper/emc_vg_pri_ide-emc_lv_celerra_audit
117M 5.6M 105M 6% /celerra/audit
/dev/hda5 2.0G 796M 1.1G 42% /nas
ohboy1
6 Posts
0
December 6th, 2013 08:00
I attached an image from the storage system.
Rainer_EMC
4 Operator
•
8.6K Posts
0
December 6th, 2013 09:00
You won’t get NAS services starting or UniSphere working if the data movers aren’t running
ohboy1
6 Posts
0
December 6th, 2013 09:00
Hi Rainer:
How does one get the data movers running?
[nasadmin@cotys ~]$ /nas/bin/server_cpu server_2 -reboot now
NAS DB is needed to execute the current command but is unavailable at this time
bg
ohboy1
6 Posts
0
December 6th, 2013 10:00
I have waited. The last time has been 1 hour. Previously occasions it has been a few hours.
I have verified they have power by looking at them.
yes it was working yesterday.
bg
Rainer_EMC
4 Operator
•
8.6K Posts
0
December 6th, 2013 10:00
Power on and wait
I assume the system was working before
ohboy1
6 Posts
0
December 6th, 2013 11:00
I just found these..
would this be useful?? in particular i see a backup from "Oct 2 17:02 nasdb_backup.b.tar.gz" which is 2 months before I mucked around.
[root@cotys celerra]# cd /celerra/backup
[root@cotys backup]# ls -lrt
-rw-r--r-- 1 nasadmin nasadmin 16194129 Oct 2 17:02 nasdb_backup.b.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571082 Dec 5 06:01 _nasbkup.03.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571079 Dec 5 07:01 _nasbkup.04.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571087 Dec 5 08:01 _nasbkup.05.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571087 Dec 5 09:01 _nasbkup.06.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571086 Dec 5 10:01 _nasbkup.07.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571084 Dec 5 11:01 _nasbkup.08.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571086 Dec 5 12:01 _nasbkup.09.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571085 Dec 5 13:01 _nasbkup.10.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571094 Dec 5 14:01 _nasbkup.11.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571081 Dec 5 15:01 _nasbkup.12.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 70261 Dec 5 16:01 _dbms_backup.01.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571597 Dec 5 16:01 _nasbkup.01.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 70264 Dec 5 17:01 _dbms_backup.OK.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 70264 Dec 5 17:01 _dbms_backup.02.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 12475 Dec 5 17:01 stat_groups_backup.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 16235524 Dec 5 17:01 nasdb_backup.1.tar.gz
-rw-r--r-- 1 nasadmin nasadmin 14571580 Dec 5 17:01 _nasbkup.02.tar.gz
Rainer_EMC
4 Operator
•
8.6K Posts
0
December 6th, 2013 12:00
No
I would suggest to engage support
ohboy1
6 Posts
0
December 9th, 2013 12:00
I have spent some more time trying to figure this out.
I notice that only one of my SP make a login to my host initiator. However what I notice is that
it flips from NO to YES and then back to NO again and then YES and then..
I am not seeing any hardware or software events in the events for the SP.
Well I do see it saying it zeroing out the LUN (a CLI shows that pcrc rebuild/bound is at 100%)
and i see that the channel loop goes up and then goes down.
Are there any other logs or ways I can get more visibility into this.
Does the NAS db have to be operational for a permanent login to occur?
thanks
bg