K8s - CNI not ready

2020.12.01

a. Problem: Worker node - 재기동 후 상태가 not ready임 (cni)

- Environments
  Kubernetes 1.16.15

 

  [iap@iap01 ~]$ k get nodes iap11

  NAME    STATUS     ROLES    AGE   VERSION

  iap11   NotReady   <none>   60d   v1.16.15

  [iap@iap01 ~]$ k describe nodes iap11 | grep Conditions -A6

  Conditions:

    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message

    ----             ------  -----------------                 ------------------                ------                       -------

    MemoryPressure   False   Tue, 24 Nov 2020 17:32:08 +0900   Tue, 24 Nov 2020 16:57:59 +0900   KubeletHasSufficientMemory   kubelet has sufficient memory available

    DiskPressure     False   Tue, 24 Nov 2020 17:32:08 +0900   Tue, 24 Nov 2020 16:57:59 +0900   KubeletHasNoDiskPressure     kubelet has no disk pressure

    PIDPressure      False   Tue, 24 Nov 2020 17:32:08 +0900   Tue, 24 Nov 2020 16:57:59 +0900   KubeletHasSufficientPID      kubelet has sufficient PID available

    Ready            False   Tue, 24 Nov 2020 17:32:08 +0900   Tue, 24 Nov 2020 16:57:59 +0900   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

  [iap@iap01 ~]$

 

 

b.  Cause analysis 

  [root@iap11 ~]# journalctl -f -u kubelet | egrep "kubelet.go|cni.go"

  11월 24 17:35:35 iap11 kubelet[33909]: W1124 17:35:35.987764   33909 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d

  11월 24 17:35:39 iap11 kubelet[33909]: E1124 17:35:39.334611   33909 kubelet.go:2188] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

  [root@iap11 net.d]# ls -l /etc/cni/net.d/10-flannel.conflist

  -rw-r--r--. 1 root root 267 11월 24 17:53 /etc/cni/net.d/10-flannel.conflist

  [root@iap11 net.d]# ip addr sh | grep cni0

  [root@iap11 net.d]#

 

 

c. Solution:

    cniVersion 값이 없는 worker node도 존재하며 정상 되작 됨. 다만 아래와 같이 수정 후 ready 상태로 변경 됨.

    kubelet을 재기동하지 않아도 cni interface가 생성됨.

 

  [root@iap11 net.d]# vi /etc/cni/net.d/10-flannel.conflist

  {

    "name": "cbr0",

    "cniVersion": "0.2.0",    # appended

  …

  [root@iap11 net.d]# systemctl restart kubelet

  [root@iap11 net.d]# ip addr sh | grep "cni0:"

  8: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000

  [root@iap11 net.d]#

 

  [iap@iap01 ~]$ k get nodes iap11

  NAME    STATUS   ROLES    AGE   VERSION

  iap11   Ready    <none>   60d   v1.16.15

  [iap@iap01 ~]$