[csi-vxflexos] Create Volume with Snapshot datasource results in Pods with "mount failed" error

I attempted to create a sample application, take a snapshot and restore it using the steps described in this gist-

https://gist.github.com/bathina2/1c453f9722181cce31c3e07931f85317

The final Pod that uses the PVC with a snapshot as its datasource is stuck with the error-

Events:

  Type     Reason                  Age                  From                              Message

  ----     ------                  ----                 ----                              -------

  Warning  FailedScheduling        4m8s (x2 over 4m8s)  default-scheduler                 pod has unbound immediate PersistentVolumeClaims (repeated 8 times)

  Normal   Scheduled               4m7s                 default-scheduler                 Successfully assigned default/clone-of-k10-primer-app to sl73ddcls8pw01.xyz.com

  Normal   SuccessfulAttachVolume  4m6s                 attachdetach-controller           AttachVolume.Attach succeeded for volume "ddcls8-bs-b870de85f7"

  Warning  FailedMount             2m4s                 kubelet, sl73ddcls8pw01.xyz.com  Unable to mount volumes for pod "app-restored": timeout expired waiting for volumes to attach or mount for pod "default"/"app-restored". list of unmounted volumes=[persistent-storage]. list of unattached volumes=[]

  Warning  FailedMount             111s (x9 over 4m2s)  kubelet, asdfafds.xyz.com  MountVolume.SetUp failed for volume "ddcls8-bs-b870de85f7" : rpc error: code = Internal desc = error performing private mount: mount failed: exit status 32

mounting arguments: -t xfs -o defaults /dev/scinib /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430bba0000038c

output: mount: wrong fs type, bad option, bad superblock on /dev/scinib,

       missing codepage or helper program, or other error

 

       In some cases useful info is found in syslog - try

       dmesg | tail or so.

Looking at node logs, I see similar messages-

time="2020-09-01T19:51:01Z" level=debug msg="/csi.v1.Node/NodePublishVolume: REQ 0011: VolumeId=ee430b5000000384, TargetPath=/var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount, VolumeCapability=mount: access_mode: , Readonly=false, VolumeContext=map[CreationTime:2020-09-01 19:50:52 +0000 UTC FsType:xfs Name:ddcls8-bs-70d165a5ec StoragePoolID:6aa9fbd800000001 StoragePoolName:pd8_08 storage.kubernetes.io/csiProvisionerIdentity:1598905765098-8081-csi-vxflexos.dellemc.com], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"

time="2020-09-01T19:51:01Z" level=info msg="VolumeContext:"

time="2020-09-01T19:51:01Z" level=info msg="    [StoragePoolID]=6aa9fbd800000001"

time="2020-09-01T19:51:01Z" level=info msg="    [StoragePoolName]=pd8_08"

time="2020-09-01T19:51:01Z" level=info msg="    [storage.kubernetes.io/csiProvisionerIdentity]=1598905765098-8081-csi-vxflexos.dellemc.com"

time="2020-09-01T19:51:01Z" level=info msg="    [CreationTime]=2020-09-01 19:50:52 +0000 UTC"

time="2020-09-01T19:51:01Z" level=info msg="    [FsType]=xfs"

time="2020-09-01T19:51:01Z" level=info msg="    [Name]=ddcls8-bs-70d165a5ec"

time="2020-09-01T19:51:01Z" level=info msg="NodePublishVolume id: ee430b5000000384"

time="2020-09-01T19:51:01Z" level=debug msg="attempting mount to private area" CSIRequestID=11 device=/dev/scinib id=ee430b5000000384 privateMount=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384 target="/var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount" volumePath=/dev/scinib

time="2020-09-01T19:51:01Z" level=debug msg="created directory" path=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384

time="2020-09-01T19:51:01Z" level=info msg="attempting to mount disk" fsType=xfs options="[defaults]" reqID=11 source=/dev/scinib target=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384

time="2020-09-01T19:51:01Z" level=info msg="mount command" args="-t xfs -o defaults /dev/scinib /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384" cmd=mount

time="2020-09-01T19:51:01Z" level=error msg="mount Failed" args="-t xfs -o defaults /dev/scinib /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384" cmd=mount error="exit status 32" output="mount: wrong fs type, bad option, bad superblock on /dev/scinib,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n"

time="2020-09-01T19:51:01Z" level=info msg="Mount attempt failed" mountErr="mount failed: exit status 32\nmounting arguments: -t xfs -o defaults /dev/scinib /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384\noutput: mount: wrong fs type, bad option, bad superblock on /dev/scinib,\n       missing codepage or helper program, or other error\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n"

time="2020-09-01T19:51:01Z" level=info msg="checking if disk is formatted using lsblk" args="[-n -o FSTYPE /dev/scinib]" disk=/dev/scinib

time="2020-09-01T19:51:01Z" level=debug msg="lsblk output" output="\n"

time="2020-09-01T19:51:01Z" level=info msg="getDiskFormat returned after initial mount failed" existingFormat= reqID=11 source=/dev/scinib

time="2020-09-01T19:51:01Z" level=info msg="disk is unformatted" existingFormat= reqID=11 source=/dev/scinib

time="2020-09-01T19:51:01Z" level=info msg="disk appears unformatted, attempting format" existingFormat= fsType=xfs reqID=11 source=/dev/scinib

time="2020-09-01T19:51:02Z" level=info msg="disk successfully formatted" existingFormat= fsType=xfs reqID=11 source=/dev/scinib

time="2020-09-01T19:51:02Z" level=info msg="mount command" args="-t xfs -o defaults /dev/scinib /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384" cmd=mount

time="2020-09-01T19:51:02Z" level=info msg="mount command" args="-o bind /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384 /var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount" cmd=mount

time="2020-09-01T19:51:02Z" level=info msg="mount command" args="-o remount /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384 /var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount" cmd=mount

time="2020-09-01T19:51:02Z" level=debug msg="/csi.v1.Node/NodePublishVolume: REP 0011: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"

time="2020-09-01T19:51:09Z" level=debug msg="/csi.v1.Node/NodeUnpublishVolume: REQ 0012: VolumeId=ee430b5000000384, TargetPath=/var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"

time="2020-09-01T19:51:09Z" level=info msg="NodeUnublishVolume id: ee430b5000000384"

time="2020-09-01T19:51:09Z" level=debug msg="Unmounting /var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount" CSIRequestID=12 device=/dev/scinib privTgt=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384 target="/var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount"

time="2020-09-01T19:51:09Z" level=info msg="unmount command" cmd=umount path="/var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount"

time="2020-09-01T19:51:09Z" level=debug msg="Unmounting /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384" CSIRequestID=12 device=/dev/scinib privTgt=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384 target="/var/lib/kubelet/pods/70dd2ae1-ec8c-11ea-8de0-0242ac110005/volumes/kubernetes.io~csi/ddcls8-bs-70d165a5ec/mount"

time="2020-09-01T19:51:09Z" level=info msg="unmount command" cmd=umount path=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384

time="2020-09-01T19:51:09Z" level=debug msg="removing directory" directory=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/ee430b5000000384

How do I resolve this issue? The driver I am using is csi-vxflexos.dellemc.com