DELL MD3220i consistancy mismatch messages

Hi @itims,

This might be caused by a corrupted parity in the logical volume(VD) which it needs a few sets of command that need to be executed, to try to detect the parity and repair it. This involves the support to verify your storage logs to identify the effected logical volume and perform command syntax to try repair the logical volume. If the repair is unable to resolve the issue, disks that are involved in the logical volume must be replaced. 

Hi, I don't know root cause of the issue. However please check to related thread and Sam's answers in here https://dell.to/3H59gio

It seems like the same issue.

Thank you for replying back Joey-C.

What may be the root cause of this corrupted partiy? Also, where can we find these commands needed to be executed for this issue to be resolved? Else, please let me know how to get those storage logs from where you can identify the affected volume. Are you talking about the support bundle here?

Thank you, Erman.

That thread sounds a bit useful.

Can you please send a private link so I can also share you my support bundle and if possible can you please check that for me too?

Also, the thread does not give any specific resolution steps. Can you please provide any link which has the steps and commands to check this issue as suggested by Joey-C?

Your help would be highly appreciated.

Hello itims

You can upload the support bundle here under the MD3220i service tag then Private Message me the service tag for me to retrieve the report:

https://upload.dell.com/

You may like to review this Knowledge base article:

MD Series: Isolation of physical disk causing consistency mismatch

https://www.dell.com/support/kbdoc/en-us/000083860

To see the article, you may need to log in to your account or create an account to log in if you don't have one.

Hello itims,

Please note I removed the service tag from the post as that is private information.

I will collect the log for review.

Service tag : {moderator removed private information}

Can't see the support document. It gives me the following error even when I am logged on to the portal :

This article is permission based. Find another article.

Hello itims,

I get that too if I use Chrome. You may try Firefox but I'll include the text for you.

After reviewing the log I see the system is on old controller firmware and drive firmware, new fw has a fix for data loss.

This exposed to all the problems that could cause data inconsistencies. Its hard to say whether its because of the unexpected controller restarts the week before these started or if its because of an issue with vmware vmotion, or the ODX issues that could cause it, etc.

Findings:

Drives are ST1200MM0007 on firmware IS05

Controllers are on 08.20.05.60

KB 000083860, included below, explains these errors.

Isolation of physical disk causing consistency mismatch are seen because the system is basically saying that there is a mismatch between data and parity. The system is self-correcting (reconstructing the drives) each time we hit the mismatch.

First one started. Date/Time: 12/17/20 7:39:01 AM and has continued to get them on various drives since.

About a week prior to these starting, there does appear to have been events for controllers being reset.

Drive firmware version IS06 mentions these fixes, which include data loss and drive errors.

Fix for corner case data loss issue

Corrects several issues where a drive hang or timeout conditions can result

Drive error recovery FW improvements and enhancements

Maintenance and health update

Controller firmware 08.20.05.60 is many versions behind the latest. Current firmware 08.20.24.60

It is missing many updates, that have various fixes for things that could have caused the data inconsistency. Just to name a few:

08.20.12.60 resolved issues that started in 08.20.05.60 which can cause corruption.

08.20.16.60 also resolved issues that can cause data inconsistency from an unexpected controller reboot.

08.20.20.60 has a fix for Failed VMware Vmotion read operation caused data inconsistency

Assuming the server OS versions support the latest firmware, I would strongly recommend upgrading to latest controller firmware.

POA:

Find out server OS version to confirm compatibility with latest controller firmware. You can check the Dell EMC PowerVault MD32XX/36XX Series Support Matrix on the Support Documents site.

Update controller firmware to latest, assuming server OS supports it.

Controller update would also require first having the latest version of MD Storage Manager.

After controllers, update drive firmware.

Once updates for drives and controllers have been applied I recommend running through 000083860, included below, to check and repair consistency errors for the two volumes. Then run a second parity scan on volumes to confirm no further errors are encountered.

Support site:

Drivers and Firmware

https://www.dell.com/support/home/en-us/product-support/product/powervault-md3220i/drivers

Support Documents

https://www.dell.com/support/home/en-us/product-support/product/powervault-md3220i/docs

Article Number: 000083860

MD Series: Isolation of physical disk causing consistency mismatch

Article Content

The following critical error is reported during a background media scan operation:

Event type: 2046 Isolation of physical disk causing consistency mismatch.

Cause:

A drive reports the above error.

If the following are true, a parity scan and repair on the affected volume are performed:

• The error is only logged against one drive
• There are no other drive check conditions
  
  
  
  To determine which volume is affected, review the following raw data for the MEL event:
  
  Raw data:
  4d 45 4c 48 03 00 00 00 3d a4 00 00 00 00 00 00
  46 20 41 01 3b 86 8f 57 17 05 01 00 00 80 00 00
  06 00 00 00 04 00 00 00 22 00 00 00 22 00 00 00
  01 00 00 00 02 00 00 00 18 00 00 00 01 00 00 00
  0a 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00
  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  01 00 00 00 00 00 00 03 44 00 00 00 20 00 20 06
  53 34 32 31 36 57 30 33 00 00 00 00 00 00 00 00
  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  08 00 20 86 49 28 80 00 00 00 00 00 10 00 43 87
  50 00 c5 00 96 c8 1c 5f 00 00 00 00 00 00 00 00
  
  Byte 33 shows the SSID of the affected volume as 06. Matching this to the evfShowVol output, you can determine the name of the volume that needs to be scanned.
   

Byte 33 showing the SSID of the affected volume is in Hexadecimal and must be converted to Decimal before matching it to the value of evfShowVol.

Executing evfShowVol(6,99,0,0,0,0,0,0,0,0) on controller A:
Volume 0x6(RAIDVolume
     Volume Type:       8+2 RAID 6
     User Label:        SLVSAN01_VMWARE_DS07

Resolution

Perform the following steps to resolve the issue:

In MDSM, right-click on the discovered array and click on "Execute Script"

1. Perform a parity scan on the affected volumes as follows: (Verify below command before executing it in the script windows).

check virtualDisk [VD1] consistency consistencyErrorFile="C:\Temp\ConsistencyError.txt" mediaErrorFile="C:\Temp\mediaError.txt" priority=high verbose=true;

2. After this scan is completed, use the Error file to perform a parity repair on the affected volumes as follows. (Verify below command before executing it in the script windows).

repair virtualDisk [VD1] consistency consistencyErrorFile="C:\Temp\ConsistencyError.txt" verbose=true;

If no Parity Error file is generated, then no errors are identified. During this scan, if new errors are encountered, additional events might be logged against this drive.

3. To verify that all errors have been resolved, perform a second parity scan on this volume. No further errors are encountered. Further, wait for the next background media scan to execute; however, immediate confirmation is required that there are no more errors.

If there are any additional errors, then replace the drive.