Isilon performance issues caused by service tasks

Hi Community,

Is there any experience of how maintenance and service tasks affect the performance of a Isilon cluster for a certain time?

I have now seen yet again serious performance issues for the whole cluster caused by doing service tasks at a Gen6 Node pool.

Example 1:

12 Node Cluster: 8x X410, 4x A2000

The A2000 Nodes are used for archive data with SmartPools with no high performance clients connected. All relevant productive clients are connected to the X410 Node pool. Last day a fan failed at one A2000 node and the node shutdown itself automatically. Today a technician came and replaced the defective fan. During or shortly after the service task, the smb write performance at the X410 pool collapsed to such an extent that the write speed was no longer sufficient for some video recording applications and the applications ran in timeouts. I checked (recently run) jobs on Isilon but there were no jobs running during that time that could have affect the overall cluster performance.

Example 2:

17 Node Cluster: 6x X200, 7x X410, 4x H500

In order to test the process and the effects of a hard disk failure, the customer removed and added again one of the H500 drive sleds according to the drive replacement instructions. First time with all HDDs working fine and second time with one hard disk drive smartfailed.

In both cases, the performance of clients connected to the other node pools was negatively affected and in the first case the write/read performance even went down to 0 for several seconds and applications run into issues.

SRs have already been created, but I would be interested to know if anyone else has made such observations, especially in relation to Gen6 nodes.

Phil