MD3000i - one controller keeps rebooting

hi Sam,

thank you for a quick answer.

Do we have to turn storage off for this? Maybe it'd be possible to just put second controller offline to remove it and remove batery for few minutes? I'm asking becouse we can't turn the whole storage off during week, so if it's really necessery, we have to wait till saturday to check it

Hello mkay1,

Based on the Serial capture that you posted in the log it looks like the secondary controller is not able to match the firmware & setting of the primary controller.  What you can try to do is to power down the MD3000i & removes the secondary controller (one the serial cable is connected to pull the capture) & remove the raid battery from the controller & leave it out for about 5 minutes.   Once that is done then power back on the MD3000i with just the primary controller in & wait for the MD to a ready state.  After the MD is up then insert the raid battery back into the controller and put the controller back into the MD3000i.  Connect the serial cable again & & capture the boot to see if the controller will sync with the primary controller.  

Please let us know if you have any other questions.

Hello mkay1,

You can do this with the MD on & putting the controller in offline in MDSM.  

Please let us know if you have any other questions.

Hi,

I have the same problem. I try with online storage to remove the battery from second controller, but after reactivation I got the same probem.

My log after make a reset:

Reset, Power-Up Diagnostics - Loop 1 of 1

3600 Processor DRAM

    01 Data lines                                                  Passed

    02 Address lines                                               Passed

3300 NVSRAM

    01 Data lines                                                  Passed

5900 Ethernet 91c111 #1

    01 Register read                                               Passed

    02 Register test                                               Passed

3A00 NAND Flash

    06 Bad Blocks Test                                             Passed

2310 Application Accelerator Unit

    01 AAU Register Test                                           Passed

6D00 LSI SAS 1068 IOC--Base Board

    01 IOC Register Read Test                                      Passed

    02 IOC Register Address Lines Test                             Passed

    03 IOC Register Data Lines Test                                Passed

6F01 QLOGIC EP4032 CHIP 0

    01 Register Read Test                                          Passed

    02 Register Address Lines Test                                 Passed

    03 Register Data Lines Test                                    Passed

3900 Real-Time Clock

    01 RT Clock Tick                                               Passed

Diagnostic Manager exited normally.

Current date: 05/27/15  time: 01:11:52

Send for Service Interface or baud rate change

05/26/15-18:41:21 (GMT) (tRAID): NOTE:  Set Powerup State

05/26/15-18:41:21 (GMT) (tRAID): NOTE:  SOD Sequence is Normal, 0

05/26/15-18:41:21 (GMT) (tRAID): NOTE:  SOD: removed SAS host from index 0

05/26/15-18:41:21 (GMT) (tRAID): NOTE:  In iscsiIOQLIscsiInitDq.  iscsiIoFstrBase = 0x0

05/26/15-18:41:22 (GMT) (tRAID): NOTE:  Turning on tray summary fault LED

05/26/15-18:41:23 (GMT) (tNetCfgInit): NOTE:  Network Ready

esmc0: Link change detected, LinkDown may take a long time to detect

05/26/15-18:41:23 (GMT) (tRAID): NOTE:  SYMBOL: SYMbolAPI registered.

05/26/15-18:41:23 (GMT) (tRAID): NOTE:  lost persistent dq data because buffer was modified or size changed.

0x36d600 (tNetTask): esmc0: LinkUp event

05/26/15-18:41:27 (GMT) (tRAID): NOTE:  Initiating Drive channel: ioc:0 bringup

05/26/15-18:41:29 (GMT) (tRAID): NOTE:  IOC Firmware Version: 00-24-63-00

05/26/15-18:41:46 (GMT) (tSasEvtWkr): NOTE:  sasIocPhyUp: chan:1 phy:0 prevNumActivePhys:2 numActivePhys:2

05/26/15-18:41:46 (GMT) (tSasEvtWkr): NOTE:  sasIocPhyUp: chan:1 phy:1 prevNumActivePhys:2 numActivePhys:2

05/26/15-18:41:46 (GMT) (tSasEvtWkr): NOTE:  sasIocPhyUp: chan:0 phy:2 prevNumActivePhys:2 numActivePhys:2

05/26/15-18:41:46 (GMT) (tSasEvtWkr): NOTE:  sasIocPhyUp: chan:0 phy:3 prevNumActivePhys:2 numActivePhys:2

05/26/15-18:41:46 (GMT) (tSasCfg021): NOTE:  Alt Controller path up - chan:1 phy:18 itn:1

05/26/15-18:41:46 (GMT) (tSasCfg022): NOTE:  Alt Controller path up - chan:0 phy:16 itn:2

05/26/15-18:41:47 (GMT) (tRAID): NOTE:  IonMgr: Drive Interface Enabled

05/26/15-18:41:48 (GMT) (tRAID): NOTE:  SOD: Instantiation Phase Complete

05/26/15-18:41:48 (GMT) (tRAID): NOTE:  Inter-Controller Communication Channels Opened

05/26/15-18:41:48 (GMT) (IOSched): NOTE:  New Initiator:  1 - channel: 0,devHandle: x28, SAS Address: 5a4badb453af1c00

05/26/15-18:41:48 (GMT) (tSasDiscCom): NOTE:  SAS Discovery complete task spawned

05/26/15-18:41:48 (GMT) (tRAID): NOTE:  LockMgr Role is Slave

05/26/15-18:41:48 (GMT) (IOSched): NOTE:  discoveredEncl: enclosure:1, enclProp: x2c5a0a0, trayId: 1, slotCount: 15

05/26/15-18:41:48 (GMT) (IOSched): NOTE:  New Initiator:  2 - channel: 1,devHandle: x18, SAS Address: 5a4badb453af1c01

05/26/15-18:41:48 (GMT) (sasCheckExpanderSet): NOTE:  Expander Firmware Version: 0116-e05c

05/26/15-18:41:48 (GMT) (sasCheckExpanderSet): NOTE:  Expander SAS address: Hi = x5842b2b4 Low = x09247d10

05/26/15-18:41:48 (GMT) (IOSched): NOTE:  discoveredEncl: enclosure:1, enclProp: x3124260, trayId: 1, slotCount: 15

05/26/15-18:41:50 (GMT) (tRAID): NOTE:  spmEarlyData: Using data from alternate

05/26/15-18:41:54 (GMT) (tSasDiscCom): WARN:  SAS: Initial Discovery Complete Time: 30 seconds

05/26/15-18:41:54 (GMT) (tRAID): NOTE:  WWN baseName 0004842b-2b09298e (valid==>SigMatch)

05/26/15-18:41:54 (GMT) (tRAID): NOTE:  IonMgr: Host Interface Enabled

05/26/15-18:41:54 (GMT) (tRAID): NOTE:  SOD: Pre-Initialization Phase Complete

05/26/15-18:41:54 (GMT) (tRAID): WARN:  BID: initialize(): Power latched!

05/26/15-18:42:09 (GMT) (tRAID): NOTE:  ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 1

05/26/15-18:42:09 (GMT) (tRAID): NOTE:  SOD: Code Synchronization Initialization Phase Complete

05/26/15-18:42:10 (GMT) (NvpsPersistentSyncM): NOTE:  NVSRAM Persistent Storage updated successfully

05/26/15-18:42:10 (GMT) (tRAID): NOTE:  USM Mgr initialization complete with 0 records.

05/26/15-18:42:10 (GMT) (tRAID): NOTE:  EDR - recieved 1 small records

05/26/15-18:42:10 (GMT) (tRAID): NOTE:  EDR - recieved 0 large records

05/26/15-18:42:11 (GMT) (tRAID): NOTE:  Acquire              0.044 secs

05/26/15-18:42:13 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058c3a0 4c0c8 bytes , result 0

05/26/15-18:42:40 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion timeout

05/26/15-18:42:40 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion timeout, cmd = 0x69

05/26/15-18:42:41 (GMT) (tRAID): NOTE:  Qlogic coredump file written to '79HGS4J:/tmp/QLogic_Coredump_port_0_79HGS4J',rc 204E50, expected 204E50

05/26/15-18:42:41 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose returned -1

05/26/15-18:42:41 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:42:41 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:42:41 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:42:41 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

05/26/15-18:42:42 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058c3a0 4c0c8 bytes , result 0

05/26/15-18:43:09 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion timeout

05/26/15-18:43:09 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion timeout, cmd = 0x69

05/26/15-18:43:10 (GMT) (tRAID): NOTE:  Qlogic coredump file written to '79HGS4J:/tmp/QLogic_Coredump_port_0_79HGS4J',rc 204E50, expected 204E50

05/26/15-18:43:10 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose returned -1

05/26/15-18:43:10 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:43:10 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:43:10 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:43:10 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

05/26/15-18:43:11 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058c3a0 4c0c8 bytes , result 0

05/26/15-18:43:38 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion timeout

05/26/15-18:43:38 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion timeout, cmd = 0x69

05/26/15-18:43:39 (GMT) (tRAID): NOTE:  Qlogic coredump file written to '79HGS4J:/tmp/QLogic_Coredump_port_0_79HGS4J',rc 204E50, expected 204E50

05/26/15-18:43:39 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose returned -1

05/26/15-18:43:39 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:43:39 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:43:39 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:43:39 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

05/26/15-18:43:41 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058c3a0 4c0c8 bytes , result 0

05/26/15-18:44:08 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion timeout

05/26/15-18:44:08 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion timeout, cmd = 0x69

05/26/15-18:44:08 (GMT) (tRAID): NOTE:  Qlogic coredump file written to '79HGS4J:/tmp/QLogic_Coredump_port_0_79HGS4J',rc 204E50, expected 204E50

05/26/15-18:44:08 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose returned -1

05/26/15-18:44:08 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:44:08 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:44:08 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:44:08 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

05/26/15-18:44:10 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058c3a0 4c0c8 bytes , result 0

05/26/15-18:44:37 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion timeout

05/26/15-18:44:37 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion timeout, cmd = 0x69

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  Qlogic coredump file written to '79HGS4J:/tmp/QLogic_Coredump_port_0_79HGS4J',rc 204E50, expected 204E50

05/26/15-18:44:38 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose returned -1

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:44:38 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

05/26/15-18:44:39 (GMT) (tRAID): WARN:  QLStartAdapter: ControllerErrorCount exceeds threshold.

05/26/15-18:44:39 (GMT) (tRAID): ERROR: QLInitializeDevice: QLStartAdapter failed

05/26/15-18:44:39 (GMT) (tRAID): ERROR: QLAddDevice: controller/device/chip initialization failed.

05/26/15-18:44:39 (GMT) (tRAID): ERROR: qlgEnableHostInterface: QLInitializeDevice failed.

05/26/15-18:44:39 (GMT) (tRAID): NOTE:  ********************************************************************************

05/26/15-18:44:39 (GMT) (tRAID): NOTE:    QLogic Target Appl

Hello akoscomp,

Thanks for posting your serial capture about your controller as it helps to determine your error.  Now based on the serial capture when you see the following:

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  Qlogic coredump file written to '79HGS4J:/tmp/QLogic_Coredump_port_0_79HGS4J',rc 204E50, expected 204E50

05/26/15-18:44:38 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose returned -1

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:44:38 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

05/26/15-18:44:39 (GMT) (tRAID): WARN:  QLStartAdapter: ControllerErrorCount exceeds threshold.

05/26/15-18:44:39 (GMT) (tRAID): ERROR: QLInitializeDevice: QLStartAdapter failed

That means that your controller has failed & is in need of being replaced.  There is a chance that the controller can be brought back online but in most cases that I have seen it will fail pretty quickly again & won’t be able to be brought back online.

Please let us know if you have any other questions.

Hi Sam, L

I have  the same  problem

what do you mean  by  " There is a chance that the controller can be brought back online "  

How can this be done ??

I have seen  a post  you can see below   but I am not sure what is meant  by  number 4 Flash controller firmware with the same FW (if step 3 was successful).  ?

Also I have  two attached MD1000's   so do I have pull all the disks  ? and  I also have  dual controllers

Raymond

Posted by

DELL-Kenny K  

on  13 Jan 2014 1:14 PM  

Most of the time we have seen this issue occurs due to power outages. You can try the following troubleshooting steps that may bring some controllers out of bad state:

1. Turn off array and pull all the HDD’s from array.

2. Pull one controller out and leave one controller in any slot.

3. Wait until controller LEDs are off, Power ON the array with single controller (that had the issue). You may be able to break that controller to get to shell prompt.

4. Flash controller firmware with the same FW (if step 3 was successful).

5. If “sodMain complete”, install other controller HOT till it sync up and does the “sodMain complete”.  Reboot both controllers.

6. If both controller has completed SODMAIN, turn off the array and install all the HDD’s. Turn on the array.

Note: We recommend, when the array is optimal, update the controllers to the latest MD3000i code level and capture the Support Bundle

Hi SAM L

From

///

05/26/15-18:44:39 (GMT) (tRAID): ERROR: QLInitializeDevice: QLStartAdapter failed

That means that your controller has failed & is in need of being replaced.  There is a chance that the controller can be brought back online but in most cases that I have seen it will fail pretty quickly again & won’t be able to be brought back online.

Please let us know if you have any other questions.

///

I have the same  error   but  how can you bring back the controller back on Line

I have seen   post  below  but it a bit confuing in what is meant by controller firmware with the same FW

Also I have two MD1000 attached as well   so do you disconnect all the drives in the MD 1000s as well

???

any help would be welcome  (Please I have two controller )  

Posted by

DELL-Kenny K  

on  13 Jan 2014 1:14 PM  

Most of the time we have seen this issue occurs due to power outages. You can try the following troubleshooting steps that may bring some controllers out of bad state:

1. Turn off array and pull all the HDD’s from array.

2. Pull one controller out and leave one controller in any slot.

3. Wait until controller LEDs are off, Power ON the array with single controller (that had the issue). You may be able to break that controller to get to shell prompt.

4. Flash controller firmware with the same FW (if step 3 was successful).

5. If “sodMain complete”, install other controller HOT till it sync up and does the “sodMain complete”.  Reboot both controllers.

6. If both controller has completed SODMAIN, turn off the array and install all the HDD’s. Turn on the array.

Note: We recommend, when the array is optimal, update the controllers to the latest MD3000i code level and capture the Support Bundle

Hi All

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC

05/26/15-18:44:38 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed.  Stat f000

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State 4543

05/26/15-18:44:38 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed

These errors mean that the chip image is corrupted and as a result the firmware cannot load.

The controller needs to be replaced.