csi-powerscale fsGroup behavior changed from 1.3.0 to 1.4.0+

Hi @drb45 

We don't remember any such change being made in the driver which would lead to this behavior however we will try to reproduce this in our lab and we will revert back to you.

Regards

Thar_J

Hi@drb45 

Following are the fsGroup behaviour observed in csi-powerscale 1.3. 1.4 and 1.5 :

--> Scenario 1: I had set these in security-context-pod.yaml, 

securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000

if runAsUser, runAsGroup and fsGroup is set like above given, data are written to the pod and check the id in the mount path then it displayed 

id
uid=1000 gid=3000 groups=2000

>> It returned the same gid as runAsGroup set. 

--> Scenario 2: Only if the runAsGroup is omitted and if pod is created as below 

securityContext: runAsUser: 1000 fsGroup: 2000

$ id

uid =1000 gid=0(root) groups=2000

>>  If the runAsGroup was omitted the gid would remain as 0(root) and the process will be able to interact with files that are owned by root(0) group and that have the required group permissions for root(0) group.

--> Concerning fsGroupChangePolicy:

fsGroupChangePolicy - fsGroupChangePolicy defines behavior for changing ownership and permission of the volume before being exposed inside a Pod. This field only applies to volume types that support fsGroup controlled ownership and permissions. This field has two possible values:

• OnRootMismatch: Only change permissions and ownership if permission and ownership of root directory does not match with expected permissions of the volume. This could help shorten the time it takes to change ownership and permission of a volume.

Kindly refer the kubernetes document for further clarity: 

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

Regards

Thar_J

Thanks @Thar_J 

I ran a few more tests. This is on K8s 1.19.10 with a Helm install of the CSI (using the csi-installer.sh script). And to keep things clean, none of these tests involve fsGroupChangePolicy - the ConfigurableFSGroupPolicy feature gate is NOT enabled on this cluster.

I have a pod with the following defined:

  securityContext:
    runAsUser: 1000
    runAsGroup: 100
    fsGroup: 100

The image has a local user 1000 "myuser", and the pod is mounting an Isilon PVC at /home/myuser.

With v1.3.0:

myuser@test-pod:~$ id
uid=1000(myuser) gid=100(users) groups=100(users)
myuser@test-pod:~$ touch test.file-1.3
myuser@test-pod:~$ ls -l
-rw-r--r--. 1 myuser users 0 Jun 23 16:23 test.file-1.3

All good - group owner of the file is users (100) as expected.

I then deleted that pod (but not the PVC), upgraded to v1.4.0, and redeployed the pod.

myuser@test-pod:~$ id
uid=1000(myuser) gid=100(users) groups=100(users)
myuser@test-pod:~$ touch test.file-1.4
myuser@test-pod:~$ ls -l
-rw-rw-r--. 1 myuser users 0 Jun 23 16:23 test.file-1.3
-rw-r--r--. 1 myuser users 0 Jun 23 16:34 test.file-1.4

Still good - new file also has the right group owner.

Next I created a new PVC, and a second pod which mounts it.

myuser@test-pod2:~$ id
uid=1000(myuser) gid=100(users) groups=100(users)
myuser@test-pod2:~$ touch test-file-1.4
myuser@test-pod2:~$ ls -l
-rw-r--r--. 1 myuser root 0 Jun 23 16:39 test-file-1.4

Uh oh - file group owner is root.

Here's what the directories look like on the Isilon cluster...k8s-6b54099747 is the one created with v1.3.0, k8s-789bcdc544 with 1.4.0. Note the difference in group ownership. Under v1.3.0, when the volume got mounted the first time, the directory group owner was set to the fsGroup value. Not so with v1.4.0 and 1.5.0.

ISI-12# ls -l
total 128
drwxrwsrwx    4 csi-test-cluster  100    141 Jun 23 12:34 k8s-6b54099747
drwxrwxrwx    2 csi-test-cluster  wheel    0 Jun 23 12:36 k8s-789bcdc544

Another observation - if I delete and re-launch the pod with the volume that was created by v1.3.0, the recursive fsGroup behavior is happening as expected... g+w gets set on existing files and if I manually make a file in there owned by some other group, it gets updated to gid 100 on a re-mount. This is NOT happening with the volume created with v1.4.0. So it seems the difference is not so much the version of the CSI running at mount-time, but rather the version that created the volume in the first place.

v1.5.0 behavior is identical to v1.4.0.

Another observation that you can see in the Isilon cluster's directory listing above... besides setting the group owner on the volume's directory (k8s-6b54099747) on first mount, v1.3.0 also set the setgid bit (g+s). v1.4.0 and 1.5.0 don't do that either.

Examining the spec of the volumes created by v1.3.0 compared to v1.4.0+, only one thing stands out...v1.3.0 was explicitly setting spec.csi.fsType: ext4, whereas the new versions omit that. Not sure if that matters since the docs say ext4 is the default for fsType.

Hi @drb45 

Kindly share the YAML files that you used to create the PVC and POD 

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: testvolclaim1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: isilon

The pod is using a jupyter notebook image which has a user "jovyan" with uid 1000:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 100
    fsGroup: 100
  containers:
  - name: task-pv-container
    image: jupyter/base-notebook
    volumeMounts:
    - mountPath: "/home/jovyan"
      name: task-pv-storage
  volumes:
  - name: task-pv-storage
    persistentVolumeClaim:
      claimName: testvolclaim1

H @drb45 

We will get back to post analyzing the behavior using your YAML files shared in our lab.

Regards

Thar_J

Hi @drb45 

I apologize for the delay in response. 

Kindly provide us the user through which you installed the driver. Provide me the secret.json output

also, let me know the privileges applied for this user. 

Regards

Thar_J

@Thar_J, do you mean the Isilon user that is being used for the API permissions? I can't provide the secret.json contents because I would have to redact it past the point of usability.

The API user is a Local account in the System Access Zone, assigned to a role with permissions as shown below. The top-level directory for volumes is owned by this same user.

A few other notes in case they're helpful: the cluster is on OneFS 8.1.2.0, and the volumes are being created in a non-system Access Zone (isiAccessZone is not "System").

       Name: CSIDriver
Description: Role for Kubernetes CSI driver accounts
    Members: csi-test-cluster
 Privileges
             ID: ISI_PRIV_LOGIN_PAPI
      Read Only: True

             ID: ISI_PRIV_NFS
      Read Only: False

             ID: ISI_PRIV_QUOTA
      Read Only: False

             ID: ISI_PRIV_SNAPSHOT
      Read Only: False

             ID: ISI_PRIV_IFS_RESTORE
      Read Only: True

             ID: ISI_PRIV_NS_TRAVERSE
      Read Only: True

             ID: ISI_PRIV_NS_IFS_ACCESS
      Read Only: True

Hi @drb45 

Kindly share me the logs concerning - v1.3.0 was explicitly setting spec.csi.fsType: ext4, whereas the new versions omit that.

Also I would like to understand the user through which you install the csi-powerscale driver, is it admin user or a non root user. 

Kindly share the privileges assigned to that particular user through which you installed the driver.

Regards

Thar_J

Hi @drb45 

Kindly share the storage class yaml  also. 

Would like to understand how the rootClientEnable field is set. 

Regards

Thar_J

@Thar_J,

Re: the fsType thing, this is a snippent from the yaml of a volume created with 1.3. The fsType setting is not present on a volume created with 1.4+:

spec:
  csi:
    driver: csi-isilon.dellemc.com
    fsType: ext4

I used an admin user to install the CSI on the K8s cluster:

[~]$ kubectl auth can-i '*' '*'
yes

And here's the Storage Class with rootClientEnabled true:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    helm.sh/resource-policy: keep
    meta.helm.sh/release-name: isilon
    meta.helm.sh/release-namespace: isilon
    storageclass.beta.kubernetes.io/is-default-class: "false"
  creationTimestamp: "2021-06-23T16:16:54Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: isilon
  resourceVersion: "247158"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/isilon
  uid: be613f2e-eafe-11eb-9a03-0242ac130003
parameters:
  AccessZone: REDACTED
  AzServiceIP: REDACTED
  IsiPath: /ifs/REDACTED
  RootClientEnabled: "true"
provisioner: csi-isilon.dellemc.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Hi @drb45 

I would like to have a call concerning this issue, Kindly let me know the time to connect. We work on IST time zone.

Regards

Thar_J

I will PM you.

Hi @drb45 

Kindly let us know the time to connect with you to have a look at your environment.

Regards

Thar_J