This post is more than 5 years old
2 Posts
0
4031
August 10th, 2012 10:00
What's best practice for power outage and a Clariion AX-4?
Our site has Clariion Ax-4 with one expansion rack and dual SPS which is used as the datastore for all our VM's. The SPS is connected to the power bar in the server rack which is connected to an APC Symmetra LX UPS. The UPS is configured with 'PowerChute Network Shutdown' to send a shutdown command to all servers once it has been running on battery power for 2 minutes. After 4 minutes of battery run time the UPS will send a shutdown command to the VM hosts. The servers and hosts are configured to remain powered off even once AC power is restored.
What I want to do is have the AX-4 beging to shutdown after 6 minutes of battery run time. How can I automate this process? I understand that there is no PCNS client available for the Clariion. Are there other option like creating a script that would run from the command line and shut the AX-4 down? What do other shops do with their SANs in the event of power loss?
Storagesavvy
474 Posts
1
August 10th, 2012 16:00
Rereading your question and discussing with a colleague, I want to make sure I understand how your power is connected, and provide some more insight.
What is seems like from your question is that both the hosts AND the AX's SPS are connected to the APC Symmetra UPS and in the event of a power failure, the Symmetra maintains power for the entire rack of servers as well as the AX4 array for some period of time until the batteries are drained. Correct?
It also sounds like you've configured APC PowerChute management software in the Guest VM's as well as on the VMware hosts so that all of the guests and hosts monitor the status of primary power via the APC Symmetra's management interface.
You have then configured PowerChute in the Guest VM's to start a graceful shutdown 2 minutes after the loss of primary power. Then, you've assumed that the VM's will power down within 2 minutes because the Host servers will begin powering off after another 2 minutes. Is this correct?
What you'd like to do, understandably, is to make sure that the AX storage stays powered on long enough for the guests and hosts to successfully shut down, then to power itself off. You also state that the hosts will not automatically power on when primary power is restored. (By the way, this is a good thing).
As I mentioned in my first reply, you cannot set a specific delay for the AX to wait and then shutdown. But the built-in power failure handling of the AX will probably get you what you want.
First, a little background on how the AX handles a power outage in general. The SPS is essentially a UPS battery (two in your case) that powers the processors and first shelf of disks. When power is lost to BOTH SPS's, the SPS powers the CPUs, Cache, and Vault disks long enough to copy any unwritten data in cache to disk, then gracefully shutdown. My coolleague points out that assuming the AX's SPS battery is fully charge, the cache vaulting process and power down sequence starts approximately 120 seconds after power is lost to the SPS.
In your case, since the AX's SPS is connected to a second UPS (the APC Symmetra), the AX will not see the power outage when it first occurs, and will happily serve up data throughout the host shutdown process and after. Once the hosts have powered down, the AX will continue to run normally until the Symmetra either powers off automatically, or runs out of battery power and dies. The AX's SPS will then see the power is lost, count to 120 seconds, and start shutting down.
If you really want the array to power off before the APC Symmetra battery is fully depleted, you would need to tell the Symmetra itself to power off after some delay time.
In any case, the natural occurance of events based on the configuration you have now will work just fine and prevent data loss. There are two key things to keep in mind to prevent data loss, corruption, and other problems with SAN storage during power outages..
1.) The hosts MUST power off (either immediately with the power outage, or gracefully through scripts) before the array powers off.
2.) The array MUST be online and healthy BEFORE the hosts are powered back on.
I hope that helps.
Richard
Storagesavvy
474 Posts
0
August 10th, 2012 15:00
CLARiiON arrays will automatically vault cache to disk and power down when there is a power outage. To my knowledge there is not a way to modify the delay time. The assumption is that the array could have more than one disk shelf and only the first shelf is protected by the included SPS. So a power outage will immediately take down any disks not in the first enclosure, effectively off-lining the data until power is restored. There is a serial cable connected between the SPS and the AX4 processor which communicates power status to the array.
Richard J Anderson
AnkitMehta
1.4K Posts
0
August 10th, 2012 17:00
Thank you Richard for the detailed response. Didn't know the discussion would be this fruitful. It was nice talking to you!
However, as discussed I believe you forgot to mention one more thing about making APC Symmetra LX UPS shutdown (if possible), once the all hosts are shutdown so in that way we can perfectly time the graceful shutdown of AX.
Little Correction here: Once the CLARiiON loses primary power. It starts the Vaulting Process. SPS powers the SP's and data is dumped in Vault Drives. I mean it would be Powerdown starts appx. after 120s after power is lost to the primary PDU.
Refer https://community.emc.com/docs/DOC-16041 for more information on Vaulting Operation.
AnkitMehta
1.4K Posts
0
August 13th, 2012 07:00
The Storage System will powerback on automatically. However, manual intervention is required to start the hosts.
Since, me and Richard Anderson discussed this I would like to inform you here Fully Charged SPS holds 120s for AX4-5. However, this time depends on Storage system to storage system, newer arrays Batteries, are designed to hold for a longer period of time. (I am still in process of getting the exact time. I will post it once I have it)
SaskIT
2 Posts
0
August 13th, 2012 07:00
Amazing reply. Thanks a ton. This is what I was hoping would happen. Its good to get that confirmation.
Just to finish off the discussion, your assumptions in your post are correct; the UPS will shut itself down when there is 10% battery left, and the VM hosts will begin their shutdown sequence 2 minutes after the VM quests have started their shutdown sequence. After that all VM hosts will remain powered off until they are manually powered back on.
Which leads me to one last question, will the AX-4 power itself back on once AC power is restored or is manual intervention required? The UPS will power itself back on once and begin to recharge it's batteries and provide AC power to the server rack once it starts getting AC power.