Skip to main content
Start a Conversation

Unsolved

This post is more than 5 years old

phil.nexus

9 Posts

80271

July 17th, 2014 02:00

Both Controllers in reboot cycle after Thermal sensor issue

Hi,

we have a MD3000i which the controller in bay 0 stopped working with a thermal sensor error.

We read that it could be maybe brought back by upgrading the firmware on the controller in bay 1, but that failed to complete, and now both controllers just loop.

the upgrade was tried when the controller was in bay 1.

the output is from the controller in bay 1 which was ok prior to the failed firmware upgrade.i have taken the battery out for 5 mins, and also tried the controller in bay 0, but no luck

Current date: 07/16/14 time: 05:42:09

Send for Service Interface or baud rate change
07/16/14-14:19:47 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0
07/16/14-14:19:47 (GMT) (tRAID): NOTE: SOD: removed SAS host from index 0
07/16/14-14:19:47 (GMT) (tRAID): NOTE: In iscsiIOQLIscsiInitDq. iscsiIoFstrBase = 0x0
07/16/14-14:19:47 (GMT) (tRAID): NOTE: Turning on tray summary fault LED
07/16/14-14:19:48 (GMT) (tNetCfgInit): NOTE: Acquiring network parameters for interface esmc0 using DHCP
07/16/14-14:19:49 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
07/16/14-14:19:52 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup
07/16/14-14:19:55 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00
07/16/14-14:20:03 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:0 prevNumActivePhys:2 numActivePhys:2
07/16/14-14:20:03 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:1 prevNumActivePhys:2 numActivePhys:2
07/16/14-14:20:12 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled
07/16/14-14:20:13 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete
07/16/14-14:20:13 (GMT) (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels
07/16/14-14:20:13 (GMT) (tRAID): NOTE: LockMgr Role is Master
07/16/14-14:20:13 (GMT) (tRAID): WARN: FBM:validateSubModel: Exception - Alt controller not ready
07/16/14-14:20:13 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned
07/16/14-14:20:13 (GMT) (tRAID): NOTE: spmEarlyData: Using cached data
07/16/14-14:20:13 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c
07/16/14-14:20:13 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x50026b94 Low = x585a9d10
07/16/14-14:20:20 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 30 seconds
07/16/14-14:20:20 (GMT) (tRAID): NOTE: WWN baseName 00040026-b9585a2f (valid==>SigMatch)
07/16/14-14:20:20 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled
07/16/14-14:20:20 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete
07/16/14-14:20:29 (GMT) (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0
07/16/14-14:20:29 (GMT) (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0
07/16/14-14:20:29 (GMT) (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
07/16/14-14:20:29 (GMT) (tRAID): NOTE: SOD: Code Synchronization Initialization Phase Complete
07/16/14-14:20:29 (GMT) (sntpEvent): NOTE: sntpEventHandler: VNI_GET_SYNC_TIME failed
07/16/14-14:20:29 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleException Error in iop::requestAltIopDelay
07/16/14-14:20:29 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleException Error)
07/16/14-14:20:29 (GMT) (NvpsPersistentSyncM): NOTE: NVSRAM Persistent Storage updated successfully
07/16/14-14:20:29 (GMT) (tRAID): NOTE: USM Mgr initialization complete with 0 records.
07/16/14-14:20:30 (GMT) (tRAID): WARN: Received IconSendInfeasibleException Error adding small edr records from alt controller
07/16/14-14:20:30 (GMT) (tRAID): WARN: spm: unable to exchange features, assuming none
07/16/14-14:20:30 (GMT) (tRAID): NOTE: SPM acquireObjects exception: IconSendInfeasibleException Error
07/16/14-14:20:30 (GMT) (tRAID): NOTE: DBRead 0.275 secs
07/16/14-14:20:30 (GMT) (tRAID): NOTE: sas: Peering Disabled (Alt Unavailable)
07/16/14-14:20:32 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0326e1a0 4c0c8 bytes , result 0
07/16/14-14:20:59 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-14:20:59 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-14:20:59 (GMT) (tRAID): WARN: File open failed for filename 1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J.

07/16/14-14:20:59 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-14:20:59 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-14:20:59 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543
07/16/14-14:20:59 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed
07/16/14-14:21:01 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0326e1a0 4c0c8 bytes , result 0
07/16/14-14:21:28 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-14:21:28 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-14:21:29 (GMT) (tRAID): WARN: File open failed for filename 1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J.

07/16/14-14:21:29 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-14:21:29 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-14:21:29 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543
07/16/14-14:21:29 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed
07/16/14-14:21:30 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0326e1a0 4c0c8 bytes , result 0
07/16/14-14:21:57 (GMT) (tNetCfgInit): WARN: DHCP failed to obtain a lease for interface esmc0
07/16/14-14:21:57 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-14:21:57 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-14:21:57 (GMT) (tNetCfgInit): NOTE: Network Ready
07/16/14-14:21:58 (GMT) (tRAID): NOTE: Qlogic coredump file written to '1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J',rc 204E50, expected 204E50
07/16/14-14:21:58 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1

07/16/14-14:21:58 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-14:21:58 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-14:21:58 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543
07/16/14-14:21:58 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed
07/16/14-14:21:59 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0326e1a0 4c0c8 bytes , result 0
07/16/14-14:22:26 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-14:22:26 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-14:22:27 (GMT) (tRAID): NOTE: Qlogic coredump file written to '1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J',rc 204E50, expected 204E50
07/16/14-14:22:27 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1

07/16/14-14:22:27 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-14:22:27 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-14:22:27 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543
07/16/14-14:22:27 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed
07/16/14-14:22:28 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0326e1a0 4c0c8 bytes , result 0
07/16/14-14:22:56 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-14:22:56 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-14:22:56 (GMT) (tRAID): NOTE: Qlogic coredump file written to '1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J',rc 204E50, expected 204E50
07/16/14-14:22:56 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1

07/16/14-14:22:56 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-14:22:56 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-14:22:56 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543
07/16/14-14:22:56 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed
07/16/14-14:22:57 (GMT) (tRAID): WARN: QLStartAdapter: ControllerErrorCountÿê
ëÇþError
07/16/14-14:23:11 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00

WARNING: Restart by watchdog time out

this is the output from the controller that failed with the thermal sensor error

Current date: 07/16/14 time: 00:51:09

Send for Service Interface or baud rate change
07/16/14-09:28:48 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0
07/16/14-09:28:48 (GMT) (tRAID): NOTE: SOD: removed SAS host from index 0
07/16/14-09:28:48 (GMT) (tRAID): NOTE: In iscsiIOQLIscsiInitDq. iscsiIoFstrBase = 0x0
07/16/14-09:28:48 (GMT) (tRAID): NOTE: Turning on tray summary fault LED
07/16/14-09:28:49 (GMT) (tNetCfgInit): NOTE: Acquiring network parameters for interface esmc0 using DHCP
07/16/14-09:28:50 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
07/16/14-09:28:53 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup
esmc0: Link change detected, LinkDown may take a long time to detect
0x36d600 (tNetTask): esmc0: LinkUp event
07/16/14-09:28:56 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00
07/16/14-09:29:03 (GMT) (tNetCfgInit): NOTE: DHCP obtained a lease for interface esmc0
07/16/14-09:29:03 (GMT) (tNetCfgInit): NOTE: DHCP server name:
07/16/14-09:29:03 (GMT) (tNetCfgInit): NOTE: DHCP server: 192.168.96.20
07/16/14-09:29:03 (GMT) (tNetCfgInit): NOTE: Assigned IP address: 192.168.96.167
07/16/14-09:29:03 (GMT) (tNetCfgInit): NOTE: Assigned subnet mask: 255.255.255.0
07/16/14-09:29:03 (GMT) (tNetCfgInit): WARN: **WARNING** The DHCP Server did not assign a permanent IP for esmc0.
07/16/14-09:29:03 (GMT) (tNetCfgInit): WARN: Network access to this controller may eventually fail.
07/16/14-09:29:03 (GMT) (tNetCfgInit): NOTE: Network Ready
07/16/14-09:29:05 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:0 prevNumActivePhys:2 numActivePhys:2
07/16/14-09:29:06 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:1 prevNumActivePhys:2 numActivePhys:2
07/16/14-09:29:06 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:2 prevNumActivePhys:2 numActivePhys:2
07/16/14-09:29:06 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:3 prevNumActivePhys:2 numActivePhys:2
07/16/14-09:29:06 (GMT) (tSasCfg011): NOTE: Alt Controller path up - chan:1 phy:18 itn:1
07/16/14-09:29:06 (GMT) (tSasCfg021): NOTE: Alt Controller path up - chan:0 phy:16 itn:2
07/16/14-09:29:15 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled
07/16/14-09:29:16 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete
07/16/14-09:29:16 (GMT) (tRAID): NOTE: Inter-Controller Communication Channels Opened
07/16/14-09:29:16 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned
07/16/14-09:29:16 (GMT) (IOSched): NOTE: New Initiator: 1 - channel: 0,devHandle: x21, SAS Address: 50026b94585a2f00
07/16/14-09:29:16 (GMT) (IOSched): NOTE: New Initiator: 2 - channel: 1,devHandle: x15, SAS Address: 50026b94585a2f01
07/16/14-09:29:16 (GMT) (tRAID): NOTE: LockMgr Role is Slave
07/16/14-09:29:16 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c
07/16/14-09:29:16 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x50026b94 Low = x585a9d10
07/16/14-09:29:16 (GMT) (tRAID): NOTE: spmEarlyData: Using cached data
07/16/14-09:29:20 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 30 seconds
07/16/14-09:29:20 (GMT) (tRAID): NOTE: WWN baseName 00040026-b9585a2f (valid==>SoftRst)
07/16/14-09:29:20 (GMT) (tRAID): NOTE: ionEnableHostInterfaces is waiting for a channel to become ready
07/16/14-09:29:21 (GMT) (tRAID): NOTE: ionEnableHostInterfaces waited 1800ms for a channel to become ready
07/16/14-09:29:21 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled
07/16/14-09:29:21 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete
07/16/14-09:29:33 (GMT) (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 1
07/16/14-09:29:33 (GMT) (iacTask8): NOTE: ACS: Acs Needed on Alt: No, StateCode: 2, ReasonCode = 6
07/16/14-09:29:33 (GMT) (tRAID): NOTE: SOD: Code Synchronization Initialization Phase Complete
07/16/14-09:29:34 (GMT) (NvpsPersistentSyncM): NOTE: NVSRAM Persistent Storage updated successfully
07/16/14-09:29:35 (GMT) (tRAID): NOTE: USM Mgr initialization complete with 0 records.
07/16/14-09:29:35 (GMT) (tRAID): NOTE: EDR - recieved 1 small records
07/16/14-09:29:35 (GMT) (tRAID): NOTE: EDR - recieved 0 large records
07/16/14-09:29:37 (GMT) (tRAID): NOTE: Acquire 0.044 secs
07/16/14-09:29:39 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 032ca440 4c0c8 bytes , result 0
07/16/14-09:29:42 (GMT) (tRAID): NOTE: ********************************************************************************
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLogic Target Application, Version 2.01.08 6-13-2005 (W2K)
07/16/14-09:29:42 (GMT) (tRAID): NOTE: iSCSI Target Application
07/16/14-09:29:42 (GMT) (tRAID): NOTE: ********************************************************************************
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: iSNS Server 10.10.10.22:3205
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: ISNSServerIPv6Addr 00:00:00:00:00:00:00:00 :3205
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: iSCSI Name iqn.1984-05.com.dell:powervault.md3000i.60026b9000585a2f000000004b571182
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: port = 0, IPv4 Enable = 1, IPv6 Enable = 1
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: IP Address 10.10.10.3:3260
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: IPv6InterfaceID fe80:0:0:0:00:00:00:00 :3260
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: Firmware waiting for DHCP lease. State 18
07/16/14-09:29:42 (GMT) (tRAID): NOTE: QLInitializeFW: Time 000/010 FwState 18
07/16/14-09:29:43 (GMT) (tRAID): NOTE: QLInitializeFW: Time 001/010 FwState 18
07/16/14-09:29:44 (GMT) (tRAID): NOTE: QLInitializeFW: Time 002/010 FwState 18
07/16/14-09:29:45 (GMT) (IOSched): NOTE: QLIsrDecodeMailbox: Port 0 Link up.
07/16/14-09:29:45 (GMT) (IOSched): NOTE: QLIsrDecodeMailbox: Async Event Code 8002 received
07/16/14-09:29:45 (GMT) (IOSched): ERROR: QLDoInterruptServiceRoutine: PortFatal interrupt. PortFatalErrorStatus 00002000 CSR 0000c508 AS 2 AF 800031
07/16/14-09:29:45 (GMT) (IOSched): ERROR: QLDoInterruptServiceRoutine: Local RAM Parity Fatal Error occured
07/16/14-09:29:45 (GMT) (IOSched): NOTE: QLProcessSystemError: Restart RISC
07/16/14-09:30:10 (GMT) (IOSched): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-09:30:10 (GMT) (IOSched): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-09:30:11 (GMT) (IOSched): NOTE: Qlogic coredump file written to '1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J',rc 204E50, expected 204E50
07/16/14-09:30:11 (GMT) (IOSched): WARN: Qlogic coredump file write failed.fclose returned -1

07/16/14-09:30:11 (GMT) (IOSched): NOTE: QLProcessSystemError: Restart RISC
07/16/14-09:30:11 (GMT) (IOSched): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-09:30:11 (GMT) (IOSched): WARN: QLProcessAen: QLGetFwState failed
07/16/14-09:30:36 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout
07/16/14-09:30:36 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x69
07/16/14-09:30:36 (GMT) (tRAID): NOTE: Qlogic coredump file written to '1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J',rc 204E50, expected 204E50
07/16/14-09:30:36 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1

07/16/14-09:30:36 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-09:30:36 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
07/16/14-09:30:36 (GMT) (tRAID): WARN: QLInitializeFW: QLGetFwState failed.
07/16/14-09:30:36 (GMT) (tRAID): NOTE: QLInitializeAdapter: QLInitializeFW failed
07/16/14-09:30:36 (GMT) (tRAID): ERROR: QLEnable: Enable lun error
07/16/14-09:30:38 (GMT) (tHckReset): WARN: Alt Ctl Reboot:
Reboot CompID: 0x408
Reboot reason: 0x6
Reboot reason extra: 0x1
07/16/14-09:30:38 (GMT) (tHckReset): NOTE: holding alt ctl in reset
07/16/14-09:30:39 (GMT) (tHckReset): NOTE: releasing alt ctl from reset
07/16/14-09:30:39 (GMT) (tHckReset): NOTE: LockMgr Role is Master
07/16/14-09:30:39 (GMT) (tHckReset): NOTE: sas: Peering Disabled (Health Check) Event: -4
07/16/14-09:30:39 (GMT) (IOSched): NOTE: Disconnect Initiator 1 - channel: 0, devHandle: x21, I/O: 0, SAS Address: 50026b94585a2f00
07/16/14-09:30:39 (GMT) (IOSched): WARN: sasRemoveInitiator - No I/O's - send EventAck: event: 00000018, eventContext: 00020021
07/16/14-09:30:39 (GMT) (sasTgtSendEventAck): WARN: sasTgtSendEventAckTask - SendEventAck for host: 1
07/16/14-09:30:39 (GMT) (tSasEvtWkr): NOTE: sasIocPhyDown: chan:0 phy:2 prevNumActivePhys:2 numActivePhys:1
07/16/14-09:30:39 (GMT) (tSasEvtWkr): NOTE: sasIocPhyDown: IOCPort: chan:0, OPTIMAL to DEGRADED
07/16/14-09:30:39 (GMT) (IOSched): NOTE: Disconnect Initiator 2 - channel: 1, devHandle: x15, I/O: 0, SAS Address: 50026b94585a2f01
07/16/14-09:30:39 (GMT) (IOSched): WARN: sasRemoveInitiator - No I/O's - send EventAck: event: 00000018, eventContext: 00020015
07/16/14-09:30:39 (GMT) (sasTgtSendEventAck): WARN: sasTgtSendEventAckTask - SendEventAck for host: 2
07/16/14-09:30:40 (GMT) (tSasEvtWkr): NOTE: sasIocPhyDown: chan:0 phy:3 prevNumActivePhys:1 numActivePhys:0
07/16/14-09:30:40 (GMT) (tSasEvtWkr): NOTE: sasIocPhyDown: IOCPort: chan:0, DEGRADED to FAILED
07/16/14-09:30:40 (GMT) (tSasEvtWkr): NOTE: sasRemoveExpanders: removing expanders on chan:0 level:0 and below
07/16/14-09:30:40 (GMT) (tSasEvtWkr): NOTE: sasLinkDown: expDevHandle:x9 - WidePort: #1 - Alt Controller, chan:1 encl:0, subExp:255, OPTIMAL to DEGRADED
07/16/14-09:30:40 (GMT) (tSasEvtWkr): NOTE: sasLinkDown: expDevHandle:x9 - WidePort: #1 - Alt Controller, chan:1 encl:0, subExp:255, DEGRADED to FAILED
07/16/14-09:30:53 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:2 prevNumActivePhys:0 numActivePhys:1
07/16/14-09:30:53 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: IOCPort: chan:0, FAILED to DEGRADED
07/16/14-09:30:53 (GMT) (tSasEvtWkr): WARN: sasIocPhyUp: Initializing Channel 0: Attached SAS Address: 50026b94585a2f10
07/16/14-09:30:55 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:3 prevNumActivePhys:1 numActivePhys:2
07/16/14-09:30:55 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: IOCPort: chan:0, DEGRADED to OPTIMAL
07/16/14-09:31:02 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0026, completion timeout
07/16/14-09:31:02 (GMT) (tRAID): WARN: QLMailboxCommand: command completion timeout, cmd = 0x26
07/16/14-09:31:02 (GMT) (tRAID): NOTE: Qlogic coredump file written to '1768M4J:/tmp/QLogic_Coredump_port_0_1768M4J',rc 204E50, expected 204E50
07/16/14-09:31:02 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1

07/16/14-09:31:02 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC
07/16/14-09:31:02 (GMT) (tRAID): NOTE: QLInitializeAdapter: MBOX_CMD_GET_FLASH f000. Unable to check MAC
07/16/14-09:31:02 (GMT) (tRAID): ERROR: QLEnable: Enable lun error

Exception: Reset
cpsr: 60000013 (Unknown Program Counter)


Registers:
07/16/14-09:31:02 (GMT) (t4): WARN: QLUtmEventNotify: pDevExt 3279338 port 1 Event code 8002 pUtmTaGetTeb is null.
r0 = 0 r1 = 35c31a8 r2 = 35c31a8 r3 = 0
r4 = 1bf6512 r5 = 1 r6 = 389d188 r7 = 0
r8 = 400 r9 = 400 r10 = 32b5f44 r11/fp = 1c2c2d0
r12/ip = 1 r13/sp = 1c2c294 r14/lr = 58b83c pc = 0
cpsr = 60000013

Stack Trace:


======== STACK SHOW ========
Showing for task id = 0x1c2c7b0 (tRAID), Running
FP=0x1c2c2d0, SP=0x1c2c294, PC=0x0
Current executing task id = 0x1c2c7b0 (tRAID); not interrupted

Frame Ptr Ret Addr Return Name + Offset Called Name + Offset
========== ========== ================================ ========================
0x1c2c780 0x0019f960 vxTaskEntry + 0x14 [fuzzy]
0x1c2c778 0x0019f960 vxTaskEntry + 0x14 sodMain
0x1c2c704 0x0061f2e8 sodMain + 0x1c8 _Z17sodInitializationv
0x1c2c6f4 0x0061e318 _Z17sodInitializationv + 0x18 _Z32sodInitializeApplicationServicesv
0x1c2c6e4 0x0061e0b8 _Z32sodInitializeApplicationServicesv + 0xb8 _Z13sodLogStartupPFvvE
0x1c2c588 0x0061dbf0 _Z13sodLogStartupPFvvE + 0xb0 _ZN3ion10initializeEv
0x1c2c524 0x00ae2f3c _ZN3ion10initializeEv + 0x7c _ZN3ion10IonManager10initializeEv
0x1c2c4a4 0x00ab1a78 _ZN3ion10IonManager10initializeEv + 0x438 _ZN5b_isn19IscsiNetworkManager10initializeEv
0x1c2c324 0x0052013c _ZN5b_isn19IscsiNetworkManager10initializeEv + 0x4fc QLTA_Main
0x1c2c2d8 0x00502078 QLTA_Main + 0x238 QLBM_RegisterImmDataBufs
0x1c2c2c4 0x0058bad0 QLBM_RegisterImmDataBufs + 0x30 QLBM_Register4032ImmDataBufs

Note: At least one "[fuzzy]" is indicated. A fuzzy frame entry is not a true
stack frame; rather, an address within VxWorks code space was found in the
stack, but it may not be a legitimate entry in the call list (or it may be).
Error in task 0x1c2c7b0: Bad stack poinÂÿçÄ
LÎëError
07/16/14-09:31:10 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00

any help would great

thanks

phil

DELL-Sam L

Moderator

 • 

7.6K Posts

July 17th, 2014 12:00

Hello phil,

First off thanks for posting the serial capture from the controller as it really helps to figure out your problem.  As I can see from the serial capture the raid controller in slot 1 is going to need to be replaced as it has failed and is not coming backup.

Please let us know if you have any other questions.

phil.nexus

9 Posts

July 18th, 2014 02:00

Hi sam

The controller in slot 1 was fine before the failed firmware update.

is there a way to redo the firmware, or do a roll back?

the controller in slot 0 was the 1 that initially failed.

you only mentioned the controller in slot 1, what about slot 0?

cheers

phil

DELL-Sam L

Moderator

 • 

7.6K Posts

July 18th, 2014 08:00

Hello phil,

The controller that you posted the serial capture from has failed & needs to be replaced.  I thought you had said that you put the failed controller in slot 1 so that is why I said that controller needed to be replaced, but if that controller is in slot 0 then that is the one that will need to be replaced.  

No there is not a way to roll back the firmware on the controllers.

Please let us know if you have any other questions.

phil.nexus

9 Posts

July 22nd, 2014 07:00

Hi,

I have managed to get what was the initial failed controller in Slot 0 to fire up after reseating the memory and connect to the storage manager.

But i cannot connect to the ISCSi ports or ping them, and when i make a change to the ports i get the below

Application version: 03.35.G6.50

Storage array management version: 03.35.G6.50

Storage array name: MD3000

Firmware version: 07.35.38.60

Management class: devmgr.v1035api01.Manager

**************************************************

ERROR DATA

Command sent to RAID controller module in slot: 0

Host name: 192.168.96.203

IP address: 192.168.96.203

Return code: Error 2 - The operation cannot complete because either (1) the current state of a component does not allow the operation to be completed, (2) the operation has been disabled in NVSRAM (example, you are modifying media scan parameters when that option (offset 0x31, bit 5) is disabled), or (3) there is a problem with the storage array.

Please check your storage array and its various components for possible problems and then retry the operation.

Operation when error occurred: PROC_setIscsiInterfaceProperties

Timestamp: 22-Jul-2014 13:02:25

STACK DATA

devmgr.v1035api01.sam.jal.ManagementOperationFailedException:

at devmgr.v1035api01.sam.jal.SYMbolClient.dispatchOperation(Unknown Source)

at devmgr.v1035api01.sam.jal.StorageArrayFacade.issueCommand(Unknown Source)

at devmgr.v1035api01.sam.jal.StorageArrayFacade.sendCommandCommon(Unknown Source)

at devmgr.v1035api01.sam.jal.StorageArrayFacade.sendCommand(Unknown Source)

at devmgr.v1035api01.sam.jal.StorageArrayFacade.setIscsiInterfaceProperties(Unknown Source)

at devmgr.v1035api01.sam.configuration.interfaces.ChangeISCSIConfigDialog.changePortsConfiguration(Unknown Source)

at devmgr.v1035api01.sam.configuration.interfaces.ChangeNetworkConfigDialog$2.performOp(Unknown Source)

at devmgr.v1035api01.shared.AbstractTaskAdapter.run(Unknown Source)

THREAD DATA

Thread[Reference Handler,10,system]

Thread[Finalizer,8,system]

Thread[Signal Dispatcher,9,system]

Thread[Java2D Disposer,10,system]

Thread[TimerQueue,5,system]

Thread[AWT-Windows,6,main]

Thread[AWT-Shutdown,5,main]

Thread[DestroyJavaVM,5,main]

Thread[GarbageCollectorThread,6,main]

Thread[ChangeDetector,6,main]

Thread[LogMsgThread,6,main]

Thread[Np_Link_Monitor0,6,main]

Thread[RecoveryProfile-10,6,main]

Thread[RecoveryProfile-171,6,main]

Thread[AEN-172,6,main]

Thread[Timer-1,6,main]

Thread[AWT-EventQueue-0,6,main]

Thread[Image Fetcher 3,8,main]

Thread[Image Animator 0,3,main]

Thread[Image Fetcher 0,8,main]

Thread[Image Fetcher 1,8,main]

Thread[Image Fetcher 2,8,main]

===============================

is there an update for the NVSRAM i can run or maybe retry the firmware on this controller?

Controller in Slot 1 is still rebooting though and showing Local RAM Parity Fatal Error occured.

Thanks

phil.nexus

9 Posts

July 22nd, 2014 09:00

Thanks Sam,

At this point im willing to try anything, and i have nothing to lose as i migrated all the data to an alternative SAN soon as the 1st controller went down.

Ill give it a go shortly.

If it fails, i will have to redistribute the disks to another SAN, but thats for an other thread.

cheers

DELL-Sam L

Moderator

 • 

7.6K Posts

July 22nd, 2014 09:00

Hello phil,

You can attempt to flash the firmware on the controller and see if it resolves your issue where you can get the controller in slot 0 to come back up. I have seen it work a few times but in most cases I haven’t seen it work. Here is a link to the firmware just in case you need it. http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=R315164&fileId=2731121564&osCode=WNET&productCode=powervault-md3000i&languageCode=EN&categoryId=SI

Please let us know if you have any other questions.

phil.nexus

9 Posts

July 23rd, 2014 06:00

well that also failed, and once again made the controller uncontrollable.

i guess i just have to accept its now dead.

I am going to redistribute the disks if possible to a second SAN.

Thanks for the help.

Was this post helpful?

View All

No Events found!

Top