VNX: HC_BE_18799984808: The system does not have a Global User of type 'System'.

some additional test...

I'm using temporary user tuser/password to create sysadmin2 with role administrator and type system

First lets be sure that sysadmin2 does not exist neither on global / local:

[root@emc2-cs0 domain]# /nas/sbin/naviseccli -User tuser -Password password -scope 0 -h SPA Security -rmuser -user sysadmin2 -scope global
Delete user operation failed.  User does not exist
[root@emc2-cs0 domain]# /nas/sbin/naviseccli -User tuser -Password password -scope 0 -h SPA Security -rmuser -user sysadmin2 -scope global -o
Delete user operation failed.  User does not exist
[root@emc2-cs0 domain]# /nas/sbin/naviseccli -User tuser -Password password -scope 0 -h SPA Security -rmuser -user sysadmin2 -scope local
Delete user operation failed.  User does not exist
[root@emc2-cs0 domain]# /nas/sbin/naviseccli -User tuser -Password password -scope 0 -h SPA Security -rmuser -user sysadmin2 -scope local -o
Delete user operation failed.  User does not exist
[root@emc2-cs0 domain]# 

Now lets add sysadmin2 as administator using following flags:

[...]
usage:  security

  -adduser -user USER -password PASS -scope global|local 
      -role administrator|storageadmin|operator|securityadministrator|dataprotection|localdataprotection|datarecovery|sanadmin|networkadmin|nasadmin|vmadmin 
      [-type user|system] <-o>
  -rmuser -user USER -scope global|local <-o>
[...]

[root@emc2-cs0 domain]# /nas/sbin/naviseccli -User tuser -Password password -scope 0 -h SPA Security -adduser -user sysadmin2 -password sysadmin2 -scope 0 -role administrator -type system 
WARNING: You are about to add user:  sysadmin2  

Proceed?(y/n) y
Add user operation failed. System user already exists.
[root@emc2-cs0 domain]# 

It says user already exists (!). It allow to add it without the "-type system" flag:

[root@emc2-cs0 domain]# /nas/sbin/naviseccli -User tuser -Password password -scope 0 -h SPA Security -adduser -user sysadmin2 -password sysadmin2 -scope 0 -role administrator 
WARNING: You are about to add user:  sysadmin2  

Proceed?(y/n) y

[root@emc2-cs0 domain]# 
[root@emc2-cs0 domain]# 

Hello abuzzi,

Are you logging in using the control station or direct to SP?

@DELL-Sam L​ 

When accessing via Unisphere GUI I get the warning:

An error occurred during the login process. This may be because:
1. Certificates are not accepted.
2. Both storage processors or the control station are not accessible.
3. You have logged in to a File or Block system using a local user account not defined on both the File and Block systems.

and some UI sections appear to be grayed out.

If I connect from CS and run the nas_check I get this post topic error:
"VNX: HC_BE_18799984808: The system does not have a Global User of type 'System'."

The console access does not seems to work anymore on this system (they are still fine when connecting to the other node).

Is there any other method to connect directly to SP ?

Can you reseat the SP and see if it is acessible then?

Hello,

I see that even if I can't get console access right now... I can browse SP via

https://SPA_IP/setup
https://SPB_IP/setup

Here the displayed options:

-spa (master)-
Create New Account
Change Service Password
Manage SSL/TLS Certificate
Restart Management Server
Set Update Parameters
Recover Domain
Turn Automanage On/Off
Set Administration Access Restrictions
Set Remotely/Anywhere Access Restrictions
Refresh Management Server State

-spb-
Change Service Password
Manage SSL/TLS Certificate
Restart Management Server
Set Update Parameters
Recover Domain
Turn Automanage On/Off
Set Administration Access Restrictions
Set Remotely/Anywhere Access Restrictions
Refresh Management Server State

What should I then do ? Maybe trying to rebuild the domain ?

That and restarting the management server are the best options. 

I should probably have restarted the management server first, but I did not.....

After selecting the "Recover Domain" the:
https://SPA_IP/setup https://SPB_IP/setup

says now:

"Domain Security Uninitialized 



Please initialize the security using Unisphere or Naviseccli"

which I normally fix connecting via serial cable but on this system I get no answer.

(same cable, laptop and configuration works on the other vnx system allowing to reach console).

Does the NMI button on SP would reset the console port process ?
Since I can't soft failover/reset SP...  do I have to hard shut one at the time hoping no service is impacted ?

This VNX is used for NFS mountpoint so File only.

Thx!
A.

The NMI button doesn't reset the console port. So yeah you will have to hard shut them down.

Today I unplugged spb power cable hoping to hard reset the standby but it remained up.

Should I remove the spb power supplies on the front (or some different card) to cause some panic hoping to recover spb then do the same to switch spa to spb ?

Otherwise I have to wait for next maintenance window where I can turn down all VMs using the NFS…

Thx!

A.

today I made some experiments on the other VNX unit.
I spin up a VM on a local NFS and verified its network/filesystem access keep working while:
- I first extracted then reseated SPB and get console access.
- I then extracted then reseated SPA and get console access.
On this unit I selected the "recover domain" option and notices it correctly recreate it along with the default sysadmin credentials.

With such positive feedback I turn to the production VNX unix.
- I first extracted then reseated SPB and get console access.
- I then extracted then reseated SPA and get console access.

When attempted to perform the "recover domain" I've got:

"Result of recovering the directory: InitializeSecurity() call failed The internal credential to the Control Station is invalid. Log in to the Control Station as root and run /nas/http/bin/set_passphrase IP_SP"

followed by

Domain Security Uninitialized
Please initialize the security using Unisphere or Naviseccli

I then tried "Create New Account" (sysadmin/sysadmin) but

Result of account creation: The internal credential to the Control Station is invalid. Log in to the Control Station as root and run /nas/http/bin/set passphrase IP_SP
Result of domain creation: Operation successfully completed.

then I "Create New Account" (sysadmin2/sysadmin2) 

Result of account creation: Operation successfully completed.

Using sysadmin2/sysadmin2 I added sysadmin/sysadmin

I then run nas_checkup and got the initial complain of sysadmin not being of type system

Storage System : Check for domain and federations health on VNX
Warning HC_BE_18799984808: The system does not have a Global User of
         type 'System'.
Action : IMPORTANT NOTE: If the system will be added to an existing
         VNX domain immediately after an upgrade, the following steps are NOT
         necessary.
         
         WARNING: If this check continues to fail after the VNX OE for Block
         upgrade is complete, then follow the instructions below.
         
         To create the Global User of type 'System', use the following
         instructions:
         
         From Unisphere:
         
           1. Log in using the 'sysadmin' account or an account with an
              'Administrator' role
           2. Click on 'Domains', then under the 'Users' section in the right
              pane click on 'Manage Global Users'
           3. Add an account with username 'temp_user' with role
              'Administrator', without enabling the 'System Account' checkbox,
              and click OK then Yes
           4. Log off
           5. Log in with the 'temp_user' account
           6. Click on 'Domains', then under the 'Users' section in the right
              pane click on 'Manage Global Users'
           7. Select the account used to log on in step 1, and click
              Delete then Yes
           8. Add an account with the same username as the deleted account.
              Make sure the role is 'Administrator' and check the 'System
              Account' checkbox, then click OK then Yes
           9. Delete the 'temp_user' account and click Yes. You will be
              logged off
          10. Log back in using the newly added account from step 8 to verify
              that it is working
         
         OR from the CLI:
         
           1. SSH to the control station using the 'nasadmin' account
           2. Perform the following commands:
              NOTE: If you do not use 'sysadmin' as your domain account, then
                    replace 'sysadmin' in the commands below with the username
                    of your domain account.
           3. /nas/sbin/naviseccli -User sysadmin -Password password -scope 0
              -h SPA Security -adduser -user temp_user -password password -
              scope 0 -role administrator
           4. /nas/sbin/naviseccli -User temp_user -Password password -scope 0
              -h SPA Security -rmuser -user sysadmin -scope 0
           5. /nas/sbin/naviseccli -User temp_user -Password password -scope 0
              -h SPA Security -adduser -user sysadmin -password password -scope
              0 -role administrator -type system
           6. /nas/sbin/naviseccli -User sysadmin -Password password -scope 0
              -h SPA Security -rmuser -user temp_user -scope 0
           7.  Log in to Unisphere using the newly added account from step 2c
               to verify that it is working
         
         If there are any failures, retry the steps. Refer to EMC Knowledgebase
         article emc270977 for more information. Otherwise, contact your
         service provider.

btw... after disconnecting from SPA's console port it does not allow to reconnect anymore...
So I guess I'll have to repeat the extract/reseat procedure...

Hello abuzzi,

You are correct that you will need to pull the sp so that you can gain access to it again to perform the steps that are needed to create the global user “system”.

Hello abuzzi,

Are you preforming the steps logged in as root or are you logged in as a different user?

I tried the sequence both as root:

[root@emc2-cs0 ~]# id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
[root@emc2-cs0 ~]# 
[root@emc2-cs0 ~]# /nas/sbin/naviseccli -User sysadmin -Password sysadmin -scope 0 -h SPA Security -adduser -user temp_user -password password -scope 0 -role administrator
WARNING: You are about to add user:  temp_user  

Proceed?(y/n) y

[root@emc2-cs0 ~]# 
[root@emc2-cs0 ~]# /nas/sbin/naviseccli -User temp_user -Password password -scope 0 -h SPA Security -rmuser -user sysadmin -scope 0
WARNING: You are about to remove user:  sysadmin (global)  

Proceed?(y/n) y

[root@emc2-cs0 ~]# 
[root@emc2-cs0 ~]# /nas/sbin/naviseccli -User temp_user -Password password -scope 0 -h SPA Security -adduser -user sysadmin -password sysadmin -scope 0 -role administrator -type system
WARNING: You are about to add user:  sysadmin  

Proceed?(y/n) y

[root@emc2-cs0 ~]# 
[root@emc2-cs0 ~]# /nas/sbin/naviseccli -User sysadmin -Password sysadmin -scope 0 -h SPA Security -rmuser -user temp_user -scope 0
WARNING: You are about to remove user:  temp_user (global)  

Proceed?(y/n) y

[root@emc2-cs0 ~]# 

as well as from nasadmin

[nasadmin@emc2-cs0 bin]$ id
uid=201(nasadmin) gid=201(nasadmin) groups=201(nasadmin),504(fullnas)
[nasadmin@emc2-cs0 bin]$
[nasadmin@emc2-cs0 bin]$ 
[nasadmin@emc2-cs0 bin]$ /nas/sbin/naviseccli -User sysadmin -Password sysadmin -scope 0 -h SPA Security -adduser -user temp_user -password password -scope 0 -role administrator
WARNING: You are about to add user:  temp_user  

Proceed?(y/n) y

[nasadmin@emc2-cs0 bin]$ /nas/sbin/naviseccli -User temp_user -Password password -scope 0 -h SPA Security -rmuser -user sysadmin -scope 0
WARNING: You are about to remove user:  sysadmin (global)
Deleting of this account will cause all CLI commands issued from File to Block to fail.   

Proceed?(y/n) y

[nasadmin@emc2-cs0 bin]$ /nas/sbin/naviseccli -User temp_user -Password password -scope 0 -h SPA Security -adduser -user sysadmin -password sysadmin -scope 0 -role administrator -type system
WARNING: You are about to add user:  sysadmin  

Proceed?(y/n) y

[nasadmin@emc2-cs0 bin]$ /nas/sbin/naviseccli -User sysadmin -Password sysadmin -scope 0 -h SPA Security -rmuser -user temp_user -scope 0
WARNING: You are about to remove user:  temp_user (global)  

Proceed?(y/n) y

[nasadmin@emc2-cs0 bin]$ 

in both cases no errors but the subsequent nas_checkup still complain for:

Storage System : Check for domain and federations health on VNX
Warning HC_BE_18799984808: The system does not have a Global User of
         type 'System'.

Accessing the GUI display:

An error occurred during the login process. This may be because:
1. Certificates are not accepted.
2. Both storage processors or the control station are not accessible.
3. You have logged in to a File or Block system using a local user account not defined on both the File and Block systems.

and some options are grayed out...

I just realized that on the working VNX (emc1), I can connect to the GUI Unisphere with the sysadmin user using cs, spa, and spb addresses without any issues.

However, on the affected VNX (emc2), I can use the sysadmin user to connect to spa and spb addresses, but I receive the following error:

"""
An error occurred during the login process. This may be because:
1. Certificates are not accepted.
2. Both storage processors or the control station are not accessible.
3. You have logged in to a File or Block system using a local user account not defined on both the File and Block systems.
"""

When I try connecting using the cs address, I get an "Authentication Failed" message.

On emc1, I see different sysadmin local accounts, with one pointing to a bash shell:

[root@emc1-cs0 log]# cat /etc/passwd
[...]
nasadmin:x:201:201::/home/nasadmin:/bin/bash
sysadmin:x:508:201::/home/sysadmin:/sbin/nologin
sysadmin1:x:509:201::/home/sysadmin1:/sbin/nologin
sysadmin2:x:510:201::/home/sysadmin2:/sbin/nologin
sysadmin3:x:511:201::/home/sysadmin3:/sbin/nologin
sysadmin4:x:512:201::/home/sysadmin4:/sbin/nologin
sysadmin21:x:513:201::/home/sysadmin21:/sbin/nologin
sysadmin22:x:514:201::/home/sysadmin22:/sbin/nologin
sysadmin23:x:515:201::/home/sysadmin23:/sbin/nologin
sysadmin5:x:516:201::/home/sysadmin5:/sbin/nologin
sysadmin6:x:517:201::/home/sysadmin6:/sbin/nologin
sysadmin7:x:518:201::/home/sysadmin7:/sbin/nologin
sysadmin8:x:519:201::/home/sysadmin8:/bin/bash
[root@emc1-cs0 log]# 
[root@emc1-cs0 ~]# su - sysadmin8

        *** slot_0 primary control station ***

[sysadmin8@emc1-cs0 ~]$ 

But on emc2, I only see one account set to nologin:

[root@emc2-cs0 log]# cat /etc/passwd
[...]
nasadmin:x:201:201::/home/nasadmin:/bin/bash
sysadmin3:x:503:201::/home/sysadmin3:/sbin/nologin
[root@emc2-cs0 log]# 
[root@emc2-cs0 log]# su - sysadmin3
This account is currently not available.
[root@emc2-cs0 log]# 

Even if I change emc2's sysadmin3 shell to /bin/bash, it doesn't help as the /var/log/messages still show an unknown UID:

Nov  6 10:23:49 emc2-cs0 Unisphere: Successfully got failed_auth_record lock 
Nov  6 10:23:49 emc2-cs0 Unisphere: Authentication failed for user sysadmin (uid=Unknown) from 10.58.56.159 
Nov  6 10:24:18 emc2-cs0 Unisphere: Successfully got failed_auth_record lock 
Nov  6 10:24:18 emc2-cs0 Unisphere: Authentication failed for user sysadmin (uid=Unknown) from 10.58.56.159 
Nov  6 10:24:18 emc2-cs0 Unisphere: Successfully got failed_auth_record lock 
Nov  6 10:24:18 emc2-cs0 Unisphere: Authentication failed for user sysadmin (uid=Unknown) from 10.58.56.159 
Nov  6 10:24:34 emc2-cs0 Unisphere: Successfully got failed_auth_record lock 
Nov  6 10:24:34 emc2-cs0 Unisphere: Authentication failed for user sysadmin (uid=Unknown) from 10.58.56.159 
Nov  6 10:24:35 emc2-cs0 Unisphere: Successfully got failed_auth_record lock 
Nov  6 10:24:35 emc2-cs0 Unisphere: Authentication failed for user sysadmin (uid=Unknown) from 10.58.56.159 
Nov  6 10:24:37 emc2-cs0 Unisphere: Successfully got failed_auth_record lock 
Nov  6 10:24:37 emc2-cs0 Unisphere: Too many authentication failures from 10.58.56.159 
Nov  6 10:24:37 emc2-cs0 apache: Authentication failed for user sysadmin (uid=Unknown) from 10.58.56.159 

My guess is that emc2's CS is unable to properly set the current sysadmin's UID, resulting in the CS not responding to Unisphere UI requests, although it is fully in charge of initiating CLI commands towards SPs.

What are your recommendations here?

Thx!

A.