Rook Ceph - DiskPressure

2020.11.30

 

a. Problem: DiskPressure

- Environments

  Kubernetes 1.16.15, Rook Ceph 1.3.8, CentOS 7.8

 

  [iap@iap01 ~]$ k get pod -n rook-ceph -o wide| egrep -v "Run|Com"

  NAME                    READY  STATUS   RESTARTS  AGE  IP      NODE   ...

  csi-cephfsplugin-tf82b  0/3    Evicted  0         13m  <none>  iap04  <none>  <none>

  csi-rbdplugin-jzkxk     0/3    Evicted  0         1s   <none>  iap04  <none>  <none>

  [iap@iap01 ~]$ k describe pod csi-cephfsplugin-tf82b -n rook-ceph | grep Events -A10

  Events:

    Type     Reason     Age        From               Message

    ----     ------     ----       ----               -------

    Normal   Scheduled  <unknown>  default-scheduler  Successfully assigned rook-ceph/csi-cephfsplugin-tf82b to iap04

    Warning  Evicted    13m        kubelet, iap04     The node had condition: [DiskPressure].

  [iap@iap01 ~]$

 

 

b. Casue analysis: Rook ceph에서 PG rebalancing 작업을 하면서 추가로 디스크 공간이 필요함에 따라 발생

  [root@iap04 ~]# journalctl -u kubelet | grep -i DiskPressure | grep csi-rbdplugin-jzkxk

 11월 19 10:10:35 iap04 kubelet[8698]: W1119 10:10:35.770633    8698 eviction_manager.go:164] Failed to admit pod csi-rbdplugin-jzkxk_rook-ceph(7de79c15-425c-4930-b983-8d6ba3f3e7b9) - node has conditions: [DiskPressure]

 [root@iap04 ~]# df -h | egrep "File|centos-root"

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root   50G   43G  7.3G  86% /

  [root@iap04 ~]# df -Hi | egrep "File|centos-root"

 Filesystem              Inodes IUsed IFree IUse% Mounted on

 /dev/mapper/centos-root    48M  885k   47M    2% /

 [root@iap04 ~]#

 

 

c. Solution:

    lvextend 명령어로 lv(/dev/centos/root) 추가 할당 (Case study: LVM(Logical Volume Manager 문서 참조)

 

  [root@iap04 ~]# df -h | egrep "File|centos-root"

  Filesystem               Size  Used Avail Use% Mounted on

  /dev/mapper/centos-root   91G   43G   48G  47% /

  [root@iap04 ~]#

 

  [iap@iap01 ~]$ k describe nodes iap04 | grep Conditions -A6

  Conditions:

    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message

    ----             ------  -----------------                 ------------------                ------                       -------

    MemoryPressure   False   Thu, 19 Nov 2020 13:32:24 +0900   Wed, 09 Sep 2020 14:41:56 +0900   KubeletHasSufficientMemory   kubelet has sufficient memory available

    DiskPressure     False   Thu, 19 Nov 2020 13:32:24 +0900   Thu, 19 Nov 2020 13:05:31 +0900   KubeletHasNoDiskPressure     kubelet has no disk pressure

    PIDPressure      False   Thu, 19 Nov 2020 13:32:24 +0900   Wed, 09 Sep 2020 14:41:56 +0900   KubeletHasSufficientPID      kubelet has sufficient PID available

    Ready            True    Thu, 19 Nov 2020 13:32:24 +0900   Thu, 19 Nov 2020 13:31:24 +0900   KubeletReady                 kubelet is posting ready status

  [iap@iap01 ~]$