MountVolume.MountDevice failed for volume

Hi Experts, 

I'm deploying a pod on Unity storage using iSCSI protocol. I have the pvc created, and volume looks good on Unity side. 

[root@master helm]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-iscsi-vol-claim Bound jij8-csivol-369833ea70 10Gi RWO unity-iscsi 24m  

But my pod is not able to start correct, below are from kubectl describe pod:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled default-scheduler Successfully assigned default/task-pv-pod to worker
Normal SuccessfulAttachVolume 7m23s attachdetach-controller AttachVolume.Attach succeeded for volume "jij8-csivol-369833ea70"
Warning FailedMount 6m59s kubelet, worker MountVolume.MountDevice failed for volume "jij8-csivol-369833ea70" : rpc error: code = Internal desc = runid=87 Unable to find device after multiple discovery attempts: [registered device not found]

My pod yaml file is just a simple one, to mount the Unity volume to the pod, see below yaml file:

apiVersion: v1
kind: Pod
metadata:
name: task-pv-pod
spec:
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: test-iscsi-vol-claim
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: task-pv-storage

 

From the my work node, journal log is displaying below message :

 

Aug 04 03:40:04 worker kubelet[5354]: E0804 03:40:04.086921 5354 nestedpendingoperations.go:270] Operation for "\\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\"" failed. No retries permitted until 2020-08-04 03:40:04.586904855 -0400 EDT m=+812.451239824 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \\"jij8-csivol-efcff9ba60\\" (UniqueName: \\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\") pod \\"task-pv-pod\\" (UID: \\"a78741d4-f040-4fbd-a96b-06cdb601c197\\") "\cf1\highlight2

Aug 04 03:40:04 worker kubelet[5354]: E0804 03:40:04.590183 5354 nestedpendingoperations.go:270] Operation for "\\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\"" failed. No retries permitted until 2020-08-04 03:40:05.590141672 -0400 EDT m=+813.454476645 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \\"jij8-csivol-efcff9ba60\\" (UniqueName: \\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\") pod \\"task-pv-pod\\" (UID: \\"a78741d4-f040-4fbd-a96b-06cdb601c197\\") "\cf1\highlight2

Aug 04 03:40:05 worker kubelet[5354]: E0804 03:40:05.597634 5354 nestedpendingoperations.go:270] Operation for "\\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\"" failed. No retries permitted until 2020-08-04 03:40:07.597609413 -0400 EDT m=+815.461944410 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \\"jij8-csivol-efcff9ba60\\" (UniqueName: \\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\") pod \\"task-pv-pod\\" (UID: \\"a78741d4-f040-4fbd-a96b-06cdb601c197\\") "\cf1\highlight2

Aug 04 03:40:07 worker kubelet[5354]: E0804 03:40:07.611613 5354 nestedpendingoperations.go:270] Operation for "\\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\"" failed. No retries permitted until 2020-08-04 03:40:11.611593657 -0400 EDT m=+819.475928639 (durationBeforeRetry 4s). Error: "Volume not attached according to node status for volume \\"jij8-csivol-efcff9ba60\\" (UniqueName: \\"kubernetes.io/csi/csi-unity.dellemc.com^jij8-csivol-efcff9ba60-iSCSI-apm00194717505-sv_128\\") pod \\"task-pv-pod\\" (UID: \\"a78741d4-f040-4fbd-a96b-06cdb601c197\\") "\cf1\highlight2

 

Could you please help understand why the attach volume failed?