VNX5600 UNIFIED CS0 problem

We have a VNX5600 unified in our environment. Regret the CS0 has failed and stuck on boot after power cycling. below is the result to getreason and nas_checkup.

nas_getreason

0 - slot_0 reset

10 - slot_1 primary control station

5 - slot_2 contacted

5 - slot_3 contacted

Check Version: 8.1.3.79

Check Command: /nas/bin/nas_checkup

nas_version:8.1  

nas_release:3-79   slot:slot_1   platform:NS   subplatform:NS-ARGONAUT

-------------------------------------Checks-------------------------------------

2018-02-21T20:21:33+03:30 cs::check_statistics_groups_database: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_file_system_usage_is_under_limit_: (Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass Pass

Pass Pass Pass Pass Pass Pass)

2018-02-21T20:21:33+03:30 cs::check_if_NAS_Storage_API_is_installed_correctly_: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_NAS_Storage_APIs_match_: (?)

2018-02-21T20:21:33+03:30 cs::check_if_NBS_clients_are_started: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_NBS_configuration_exists: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_NBS_devices_are_accessible: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_NBS_service_is_started: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_PXE_service_is_stopped: (Pass)

2018-02-21T20:21:33+03:30 cs::check_if_standby_is_up: (Warn)

2018-02-21T20:21:34+03:30 cs::check_integrity_of_NASDB: (Pass)

2018-02-21T20:21:34+03:30 cs::check_if_primary_is_active: (Fail)

2018-02-21T20:21:34+03:30 cs::check_all_callhome_files_delivered_: (Pass)

2018-02-21T20:21:34+03:30 cs::check_resolv_conf: (Pass)

2018-02-21T20:21:34+03:30 cs::check_if_NAS_partitions_are_mounted: (Pass)

2018-02-21T20:21:34+03:30 cs::check_ipmi_connection: (Pass)

2018-02-21T20:21:34+03:30 cs::check_nas_site_eventlog_configuration_: (Pass)

2018-02-21T20:21:34+03:30 cs::check_nas_sys_mcd_configuration: (Pass)

2018-02-21T20:21:34+03:30 cs::check_nas_sys_eventlog_configuration_: (Pass)

2018-02-21T20:21:34+03:30 cs::check_logical_volume_status: (Pass)

2018-02-21T20:21:34+03:30 cs::check_valid_nasdb_backup_files: (Pass)

2018-02-21T20:21:35+03:30 cs::check_root_disk_reserved_region_: (Pass)

2018-02-21T20:21:35+03:30 cs::check_if_RDF_configuration_is_valid: (N/A)

2018-02-21T20:21:35+03:30 cs::check_if_fstab_contains_duplicate_entries: (Pass)

2018-02-21T20:21:35+03:30 cs::check_if_sufficient_swap_memory_available: (Pass)

2018-02-21T20:21:35+03:30 cs::check_for_IP_and_subnet_configuration: (Pass)

2018-02-21T20:21:35+03:30 cs::check_auto_transfer_status: (Warn)

2018-02-21T20:22:15+03:30 cs::check_for_invalid_entries_in_etc_hosts: (Pass)

2018-02-21T20:22:15+03:30 cs::check_for_correct_filesystem_mount_options: (Pass)

2018-02-21T20:22:15+03:30 cs::check_the_hard_drive_in_the_control_station: (Pass)

2018-02-21T20:22:16+03:30 cs::check_ssh_connection: (Fail)

2018-02-21T20:22:27+03:30 cs::check_if_Symapi_data_is_present: (Pass)

2018-02-21T20:22:45+03:30 cs::check_if_Symapi_is_synced_with_Storage_System: (Pass)

2018-02-21T20:22:45+03:30 dm::check_boot_files: (Pass Pass)

2018-02-21T20:22:45+03:30 dm::check_if_primary_is_active: (Pass Pass)

2018-02-21T20:22:45+03:30 dm::check_if_root_filesystem_is_too_large: (Pass N/A)

2018-02-21T20:22:45+03:30 dm::check_if_root_filesystem_has_enough_free_space: (Pass N/A)

2018-02-21T20:22:45+03:30 dm::check_Server_IfConfig: (Pass Pass Pass Pass Pass Pass Pass Pass N/A)

2018-02-21T20:22:45+03:30 dm::check_network_connectivity: (Pass Pass Pass Pass)

2018-02-21T20:22:54+03:30 dm::check_status: (Pass Pass)

2018-02-21T20:22:54+03:30 dm::check_VNX_FILE_OE_release_compatibility: (Pass Pass)

2018-02-21T20:22:54+03:30 dm::check_VNX_FILE_OE_version_compatibility: (Pass Pass)

2018-02-21T20:22:54+03:30 dm::check_server_name: (Pass Pass)

2018-02-21T20:22:54+03:30 dm::check_unique_id: (Pass Pass)

2018-02-21T20:22:54+03:30 dm::check_CIFS_file_server_configuration: (Pass N/A)

2018-02-21T20:22:54+03:30 dm::check_domain_controller_connectivity_and_configuration: (Pass N/A)

2018-02-21T20:22:54+03:30 dm::check_DNS_connectivity_and_configuration: (Pass N/A)

2018-02-21T20:22:54+03:30 dm::check_connectivity_to_WINS_servers: (Pass N/A)

2018-02-21T20:22:54+03:30 dm::check_I18N_mode_and_unicode_translation_tables: (Pass N/A)

2018-02-21T20:22:54+03:30 dm::check_connectivity_to_NTP_servers: (Warn N/A)

2018-02-21T20:22:55+03:30 dm::check_connectivity_to_NIS_servers: (Pass N/A)

2018-02-21T20:22:55+03:30 dm::check_virus_checker_server_configuration: (Pass N/A)

2018-02-21T20:22:55+03:30 dm::check_if_workpart_is_OK: (Pass)

2018-02-21T20:22:56+03:30 dm::check_if_free_full_dump_is_available: (Pass)

2018-02-21T20:22:56+03:30 dm::check_if_each_primary_Blade_has_standby: (Pass)

2018-02-21T20:22:56+03:30 dm::check_if_Blade_parameters_use_EMC_default_values: (Pass)

2018-02-21T20:22:56+03:30 dm::check_VDM_root_filesystem_space_usage: (N/A)

2018-02-21T20:22:56+03:30 dm::check_if_file_system_usage_is_under_limit: (Pass N/A)

2018-02-21T20:22:56+03:30 dm::check_slic_signature: (Pass)

2018-02-21T20:22:56+03:30 be::check_disk_emulation_type: (Pass)

2018-02-21T20:22:56+03:30 be::check_disk_high_availability_access: (Pass Pass Pass Pass)

2018-02-21T20:22:56+03:30 be::check_disks_read_cache_enabled: (Pass)

2018-02-21T20:22:58+03:30 be::check_disks_and_storage_processors_write_cache_enabled: (Pass)

2018-02-21T20:22:58+03:30 be::check_if_VNX_BLOCK_OE_is_committed: (Pass)

2018-02-21T20:23:03+03:30 be::check_if_VNX_BLOCK_OE_is_supported_: (Pass)

2018-02-21T20:23:09+03:30 be::check_array_model: (Pass)

2018-02-21T20:23:14+03:30 be::check_if_microcode_is_supported: (N/A)

2018-02-21T20:23:14+03:30 be::check_no_disks_or_storage_processors_are_failed_over: (Pass)

2018-02-21T20:23:14+03:30 be::check_that_no_disks_or_storage_processors_are_faulted: (Pass)

2018-02-21T20:23:14+03:30 be::check_backend_faults: (Pass)

2018-02-21T20:23:16+03:30 be::check_that_no_disks_are_rebuilding: (Pass)

2018-02-21T20:23:17+03:30 be::check_minimum_control_lun_size_: (Pass)

2018-02-21T20:23:18+03:30 be::check_maximum_control_lun_size_: (N/A)

2018-02-21T20:23:18+03:30 be::check_maximum_lun_address_limit: (Pass)

2018-02-21T20:23:36+03:30 be::check_system_lun_configuration: (Pass)

2018-02-21T20:23:36+03:30 be::check_if_storage_processors_are_read_cache_enabled: (Pass)

2018-02-21T20:23:37+03:30 be::check_if_auto_assign_are_disabled_for_all_luns: (N/A)

2018-02-21T20:23:37+03:30 be::check_if_auto_trespass_are_disabled_for_all_luns: (N/A)

2018-02-21T20:23:37+03:30 be::check_storage_processor_connectivity: (Pass Pass)

2018-02-21T20:23:42+03:30 be::check_control_lun_ownership: (N/A)

2018-02-21T20:23:42+03:30 be::check_if_Fibre_Channel_zone_checker_is_set_up: (N/A)

2018-02-21T20:23:42+03:30 be::check_if_Fibre_Channel_zoning_is_OK: (N/A)

2018-02-21T20:23:42+03:30 be::check_if_proxy_arp_is_setup: (Pass)

2018-02-21T20:23:46+03:30 be::check_if_Product_Serial_Number_is_Correct: (Pass)

2018-02-21T20:23:49+03:30 be::check_SPA_SPB_communication: (Pass)

2018-02-21T20:23:57+03:30 be::check_if_secure_communications_is_enabled: (Pass)

2018-02-21T20:23:58+03:30 be::check_if_backend_has_mixed_disk_types: (Pass)

2018-02-21T20:23:58+03:30 be::check_for_file_and_block_enabler: (Pass)

2018-02-21T20:23:59+03:30 be::check_if_nas_storage_command_generates_discrepancies: (Warn Warn)

2018-02-21T20:23:59+03:30 be::check_if_Repset_and_CG_configuration_are_consistent: (Pass)

2018-02-21T20:23:59+03:30 be::check_block_operating_environment: (Pass)

2018-02-21T20:24:00+03:30 be::check_thin_pool_usage: (N/A)

2018-02-21T20:24:01+03:30 be::check_for_domain_and_federations_health_on_VNX_: (Pass)

--------------------------------------------------------------------------------

One or more errors have occurred. Follow the instructions provided to correct

the problem and try again.

------------------------------------Warnings------------------------------------

Control Station: Check if NAS Storage APIs match

Warning HC_CS_18800050320: The standby Control Station has been

powered off.

Action : Power on the standby control station and then run nas_checkup

to rerun this check.

Control Station: Check if standby is up

Warning HC_CS_18800050192: The control stations are failed over.

Action :

 

The standby is the acting primary. You can run in this state, but if

the acting primary has problems, there is no standby to take over.

1. Use various logs on the system to determine the problem that caused

     the Control Stations to fail over. Make sure that you know what the

     cause of the failure was and that it has been fixed before

     attempting to failback the Control Stations.

2. On the Control Station in slot 1, run command

     "/nasmcd/sbin/cs_standby -failback". The Control Station in

     slot 1 will reboot, and the Control Station in slot 0 will

     become the primary.

3. Wait 5 minutes for services to come up fully and verify that the

     problem has been fixed by logging into both Control stations as

     nasadmin. When you log into the Control Station in slot 0, it

     should log you in normally. When you log into the Control Station

     in slot 1, it should tell you that slot 1 is not the primary

     Control Station.

Control Station: Check auto transfer status

Warning HC_CS_18800050420: Cannot establish connection to FTP server

sftphost sftp.emc.com

Action : Command "/nas/tools/automaticcollection -verifyftp" failed to

connect to FTP connection. Check the configuration in

/nas/site/automaticcollection.cfg file. For more information, search

the Knowledgebase on Powerlink as follows:

1. Log in to http://support.emc.com and go to Support > Knowledgebase

     Search> Support Solutions Search.

2. Use ID emc221733 to search.

Blades : Check connectivity to NTP servers

Warning HC_DM_18800115743:

* server_2: Only one NTP server is configured. It is recommended to

    define at least two different NTP servers for a high availability.

    If the clock of the Data Mover is not correct, potential errors

    during Kerberos authentication may happen (timeskew).

Action : Use the server_date command to define another NTP server on

the Data Mover. Read the man pages for details and examples.

Storage System : Check if nas storage command generates discrepancies

Warning HC_BE_18799984777: Storage health check command

"/nas/bin/nas_storage -check -all" returned the following warning

messages: * Warning 17717198868: The FileSharing-PL1 on CKM00144801488

storage pool contains disk volumes that use different data service

configurations. Creating and extending file systems using this storage

pool may cause performance inconsistency.

* Warning 17718509596: The component volumes contain disk volumes with

    different data service configurations. As a result of that, the file

    system rightel-fs created from them may have inconsistent

    performance.

Action :

 

Use "/nas/bin/nas_message -info " to review warning

details and recommended actions for the aforementioned warnings. If

the problem remains unresolved, escalate the issue through your

support organization and provide this output in the escalation.

--------------------------------------------------------------------------------

-------------------------------------Errors-------------------------------------

Control Station: Check if primary is active

Error HC_CS_14505082898: Either the Primary or the Secondary Control

Station is in an inappropriate state. The reason code for the Primary

is 0, and the reason code for the Secondary is 10.

Action : Run "/nasmcd/sbin/getreason" command to investigate the

problem. Provided that NAS services are started, the command will

return "10" for Primary, and "11" for Secondary. If NAS services are

not started, the command will return a reason code of "6". Escalate

this issue through your support organization if needed. Provide this

output and any errors received from the command in the escalation.

Control Station: Check ssh connection

Error HC_CS_14505083143: The Internal network emcnasotherCS_i0

is not up.

Action :

 

1. Check that the "Check if standby is up" check passed. If it did not

     pass, then follow the instructions for that check and try to bring

     up the standby Control Station.

2. Ping the standby internal IP addresses from this Control Station by

     running "ping emcnasotherCS_i0" and "ping emcnasotherCS_i1". If you

     cannot ping the standby Control Station:

     * Check for physical problems with the network cabling between the

       two Control Stations.

     * Check network cards for lights that indicate a failed card

     * Replace hardware as necessary.

3. Check that /etc/hosts is set up with the correct internal IP

     address. Run the command "ifconfig" on the standby Control Station

     to determine its internal IP address. Verify that two of the IP

     addresses listed match the address for emcnasotherCS_i0 and

     emcnasotherCS_i1 in /etc/hosts on this Control Station.

4. If the problem persists, escalate this issue through your support

     organization. Provide this output and the output from any commands

     that you ran for this procedure in the escalation.

--------------------------------------------------------------------------------