Skip to main content
Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

ZhangFan4

13 Posts

4378

June 4th, 2019 23:00

CSI 1.0 deployment failed on k8s 1.13.3

Customer Env: 

K8S 1.13.3

K8S host CentOS Linux 7.4

VxFlex 3.0.769

***********************************************

3SDS/MDM：
Data1:192.168.1.2-4
Data2:192.168.2.2-4
App：10.8.192.89-91

Gateway 10.8.192.89
Primary MDM 10.8.192.90

System ID MDPM IPs
7a5ada7d5a833e0f 10.8.192.89 10.8.192.90

**************************************************
K8S：

kube-master

10.8.192.27 kube-m1 
10.8.192.28 kube-m2 
10.8.192.40 kube-m3 

kube-worker 10.8.192.39/41/44

**************************************************

As follow the below video and product guide:

https://github.com/coulof/vxflexos-csi-examples/tree/master/demos#enable-csi-in-k8s-cluster 

  1. We modified:/var/lib/kubelet/config.yaml   Please see attachment
  2. There is no below file in customer env:/etc/systemd/system/multi-user.target.wants/docker.service
  3. We modified: /etc/kubernetes/manifests/kube-controller-manager.yaml Please see attachment
  4. We modiefied: /etc/kubernetes/manifests/kube-scheduler.yaml Please see attachment
  5. We modified: /etc/kubernetes/manifests/kube-apiserver.yaml Please see attachment
  6. There is no in customer env /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
    or  /etc/ systemd/system/kubelet.service.d/10-kubeadm.conf
    We modified  /etc/systemd/system/multi-user.target.wants/kubelet.service Please see attachment

 

  1. Attach myvalues,yaml as well

****************************************************************************************

[root@kube-m1 helm]# sh install.vxflexos
Kubernetes version v1.13.3
Kubernetes master nodes: 10.8.192.27 10.8.192.28 10.8.192.40
Kubernetes minion nodes: 10.8.192.41 10.8.192.39 10.8.192.44
Verifying the SDC installation.
Retrieved 1 mdm(s)
MDM-ID 7a5ada7d5a833e0f SDC ID e71b7b7500000004 INSTALLATION ID 0f7a6e0e2eb88757 IPs [0]-10.8.192.89 [1]-10.8.192.90
Retrieved 1 mdm(s)
MDM-ID 7a5ada7d5a833e0f SDC ID e71ba28500000007 INSTALLATION ID 0f7a6e0e2eb88757 IPs [0]-10.8.192.89 [1]-10.8.192.90
Retrieved 1 mdm(s)
MDM-ID 7a5ada7d5a833e0f SDC ID e71ba28600000008 INSTALLATION ID 0f7a6e0e2eb88757 IPs [0]-10.8.192.89 [1]-10.8.192.90
Verifying the feature gates.
node 10.8.192.27 proc /usr/bin/kubelet gate VolumeSnapshotDataSource=true failed
node 10.8.192.27 proc /usr/bin/kubelet gate KubeletPluginsWatcher=true failed
node 10.8.192.27 proc /usr/bin/kubelet gate CSINodeInfo=true failed
node 10.8.192.27 proc /usr/bin/kubelet gate CSIDriverRegistry=true failed
node 10.8.192.27 proc /usr/bin/kubelet gate BlockVolume=true failed
node 10.8.192.27 proc /usr/bin/kubelet gate CSIBlockVolume=true failed
node 10.8.192.28 proc /usr/bin/kubelet gate VolumeSnapshotDataSource=true failed
node 10.8.192.28 proc /usr/bin/kubelet gate KubeletPluginsWatcher=true failed
node 10.8.192.28 proc /usr/bin/kubelet gate CSINodeInfo=true failed
node 10.8.192.28 proc /usr/bin/kubelet gate CSIDriverRegistry=true failed
node 10.8.192.28 proc /usr/bin/kubelet gate BlockVolume=true failed
node 10.8.192.28 proc /usr/bin/kubelet gate CSIBlockVolume=true failed
node 10.8.192.40 proc /usr/bin/kubelet gate VolumeSnapshotDataSource=true failed
node 10.8.192.40 proc /usr/bin/kubelet gate KubeletPluginsWatcher=true failed
node 10.8.192.40 proc /usr/bin/kubelet gate CSINodeInfo=true failed
node 10.8.192.40 proc /usr/bin/kubelet gate CSIDriverRegistry=true failed
node 10.8.192.40 proc /usr/bin/kubelet gate BlockVolume=true failed
node 10.8.192.40 proc /usr/bin/kubelet gate CSIBlockVolume=true failed
node 10.8.192.41 proc /usr/bin/kubelet gate VolumeSnapshotDataSource=true failed
node 10.8.192.41 proc /usr/bin/kubelet gate KubeletPluginsWatcher=true failed
node 10.8.192.41 proc /usr/bin/kubelet gate CSINodeInfo=true failed
node 10.8.192.41 proc /usr/bin/kubelet gate CSIDriverRegistry=true failed
node 10.8.192.41 proc /usr/bin/kubelet gate BlockVolume=true failed
node 10.8.192.41 proc /usr/bin/kubelet gate CSIBlockVolume=true failed
node 10.8.192.39 proc /usr/bin/kubelet gate VolumeSnapshotDataSource=true failed
node 10.8.192.39 proc /usr/bin/kubelet gate KubeletPluginsWatcher=true failed
node 10.8.192.39 proc /usr/bin/kubelet gate CSINodeInfo=true failed
node 10.8.192.39 proc /usr/bin/kubelet gate CSIDriverRegistry=true failed
node 10.8.192.39 proc /usr/bin/kubelet gate BlockVolume=true failed
node 10.8.192.39 proc /usr/bin/kubelet gate CSIBlockVolume=true failed
node 10.8.192.44 proc /usr/bin/kubelet gate VolumeSnapshotDataSource=true failed
node 10.8.192.44 proc /usr/bin/kubelet gate KubeletPluginsWatcher=true failed
node 10.8.192.44 proc /usr/bin/kubelet gate CSINodeInfo=true failed
node 10.8.192.44 proc /usr/bin/kubelet gate CSIDriverRegistry=true failed
node 10.8.192.44 proc /usr/bin/kubelet gate BlockVolume=true failed
node 10.8.192.44 proc /usr/bin/kubelet gate CSIBlockVolume=true failed
fail: 1
*******************************************************************************
Warning: Kubernetes --feature-gates not correctly configured... it may not work
*******************************************************************************
NAME: vxflexos
LAST DEPLOYED: Wed Jun 5 13:48:46 2019
NAMESPACE: vxflexos
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
vxflexos-node-55crv 0/2 ContainerCreating 0 0s
vxflexos-node-5dfg6 0/2 ContainerCreating 0 0s
vxflexos-node-qpfzj 0/2 ContainerCreating 0 0s
vxflexos-controller-0 0/4 ContainerCreating 0 0s

==> v1/StorageClass
NAME PROVISIONER AGE
vxflexos-xfs csi-vxflexos 1s
vxflexos (default) csi-vxflexos 1s

==> v1/ServiceAccount
NAME SECRETS AGE
vxflexos-controller 1 1s
vxflexos-node 1 1s

==> v1/ClusterRole
NAME AGE
vxflexos-controller 1s
vxflexos-node 1s

==> v1/ClusterRoleBinding
NAME AGE
vxflexos-controller 0s
vxflexos-node 0s

==> v1beta2/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
vxflexos-node 3 3 0 3 0 0s

==> v1beta1/StatefulSet
NAME DESIRED CURRENT AGE
vxflexos-controller 1 1 0s

==> v1alpha1/CSIDriver
NAME AGE
vxflexos 0s


NAME READY STATUS RESTARTS AGE
vxflexos-controller-0 0/4 ErrImagePull 0 10s
vxflexos-node-55crv 0/2 ErrImagePull 0 10s
vxflexos-node-5dfg6 0/2 ImagePullBackOff 0 10s
vxflexos-node-qpfzj 0/2 ErrImagePull 0 10s
NAME READY STATUS RESTARTS AGE
vxflexos-controller-0 0/4 ErrImagePull 0 15s
vxflexos-node-55crv 0/2 ErrImagePull 0 15s
vxflexos-node-5dfg6 0/2 ImagePullBackOff 0 15s
vxflexos-node-qpfzj 0/2 ErrImagePull 0 15s
NAME READY STATUS RESTARTS AGE
vxflexos-controller-0 0/4 ImagePullBackOff 0 20s
vxflexos-node-55crv 0/2 ImagePullBackOff 0 20s
vxflexos-node-5dfg6 0/2 ImagePullBackOff 0 20s
vxflexos-node-qpfzj 0/2 ImagePullBackOff 0 20s
CSIDrivers:
NAME AGE
vxflexos 20s
CSINodeInfos:
NAME AGE
kube-n1 1d
kube-n2 1d
kube-n3 1d
StorageClasses:
NAME PROVISIONER AGE
vxflexos (default) csi-vxflexos 22s
vxflexos-xfs csi-vxflexos 22s
No resources found.
installing volumesnapshotclass
volumesnapshotclass.snapshot.storage.k8s.io/vxflexos-snapclass created
VolumeSnapshotClasses:
NAME AGE
vxflexos-snapclass 1s

 

gashof

46 Posts

July 2nd, 2019 09:00

I just wanted to follow up on ZhangFan4's question.  He reports that he was able to work around his issues.

The user reports that he does have CentOS 7.4.  

His paths for the feature gates and Docker service were different than that in the docs.  He was able to find the files in different locations and modify them appropriately.  

He did modify the verify.kubernetes script MINION_PROCS  to point to his location. 

He also installed the VxFlex OS Storage Data Client (SDC) on the Master.

I also had issues with setting up the feature gates.  I found that copy and paste from the Prod Guide did not work as expected.  To help I ran the yaml through a yaml checker to assist with spacing, alignment, and dashes.

I also ran Flo_csl's recommended command to check the feature gates.  Mine now show enabled: 

systemctl status kubelet.service -l

â kubelet.service - kubelet: The Kubernetes Node Agent

Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
ââ10-kubeadm.conf
Active: active (running) since Thu 2019-06-27 16:48:23 EDT; 4 days ago
Docs: https://kubernetes.io/docs/
Main PID: 7838 (kubelet)
Tasks: 30
Memory: 143.9M
CGroup: /system.slice/kubelet.service
ââ7838 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --allow-privileged=true --feature-gates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIBlockVolume=true--config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

ZhangFan4

13 Posts

June 4th, 2019 23:00

Anyone know how to attach?

Flo_csI

2 Intern

 • 

166 Posts

June 5th, 2019 05:00

Hi Zhang,

From what I see there are several issues here :

1. Oracle Linux is not officially supported for VxFlex OS + CSI. That probably explains why the path for systemd is different than the one documented

2. You have `ErrImagePull` errors, this means your k8s nodes can't access to the docker hub to pull images. You can get more details with : kubectl describe pod vxflexos-controller-0 -n vxflexos

3. For the missing flags, if you do a systemctl status kubelet.service -l can you see them configured ?

 

Thx.

 

aron_dell

1 Rookie

 • 

3 Posts

June 5th, 2019 05:00

Good evening. Are you trying to validate an environment with Kubernetes running with a HostOS of Oracle Linux?

ZhangFan4

13 Posts

June 10th, 2019 01:00

Hi, Aron. The current PoC K8S host is CentOS 7.4

Was this post helpful?

View All

No Events found!

Top