Rook Ceph - failed to get status

2020.12.01

a. Problem: “op-cluster: failed to get ceph status. failed to get status”

- Environments
  Kubernetes 1.16.15, Rook Ceph 1.3.8

 

  [iap@iap01 ~]$ k logs rook-ceph-operator-674d4db4cf-zpp8g -n rook-ceph | egrep " E "

  …

  2020-11-30 07:16:22.362561 E | op-cluster: failed to create cluster in namespace "rook-ceph". failed to start the mons: failed to start mon pods: failed to check mon quorum q: failed to wait for mon quorum: exceeded max retry count waiting for monitors to reach quorum

  …

  2020-11-30 07:16:24.094371 E | op-cluster: failed to get ceph status. failed to get status. . timed out: exit status 1

  2020-11-30 07:16:24.112068 E | op-cluster: failed to get ceph status. failed to get status. . timed out: exit status 1

  2020-11-30 07:16:24.133962 E | op-cluster: failed to get ceph status. failed to get status. . timed out: exit status 1

  …

  [iap@iap01 ~]$ k get pod -n rook-ceph -o wide | grep mon

  rook-ceph-mon-o-85c9bb5c-8nfnb    1/1  Running  0  6d     10.244.6.164  iap04  <none>  <none>

  rook-ceph-mon-p-5b7f4fcc65-6xrct  1/1  Running  0  6d2h   10.244.8.212  iap06  <none>  <none>

  rook-ceph-mon-q-777c755fcc-pgn4m  1/1  Running  0  5d22h  10.244.9.186  iap11  <none>  <none>

  [iap@iap01 ~]$ k get po -n rook-ceph | grep mgr

  rook-ceph-mgr-a-6db8458579-jzc5x    0/1     Init:1/3    28     23h

  [iap@iap01 ~]$

 

 

b. Solution : roo-ceph-mon-[a-z] pod 재기동

  [iap@iap01 ~]$ k rollout restart deployment rook-ceph-mon-o rook-ceph-mon-p rook-ceph-mon-q -n rook-ceph

  deployment.apps/rook-ceph-mon-o restarted

  deployment.apps/rook-ceph-mon-p restarted

  deployment.apps/rook-ceph-mon-q restarted

  [iap@iap01 ~]$ k get pod -n rook-ceph -o wide | grep mon

  rook-ceph-mon-o-8794d79c6-znmzx  1/1  Running  0  13s  10.244.6.209  iap04  <none>  <none>

  rook-ceph-mon-p-7888f6c74-5l6f4  1/1  Running  0  5s   10.244.8.12   iap06  <none>  <none>

  rook-ceph-mon-q-84f448568d-mgd4f 1/1  Running  0  8s   10.244.9.149  iap11  <none>  <none>

  [iap@iap01 ~]$

  [iap@iap01 ~]$ ceph-status.sh

    cluster:

      id:     1ef6e249-005e-477e-999b-b874f9fa0854

      health: HEALTH_WARN

              Degraded data redundancy: 33191/912962 objects degraded (3.636%), 20 pgs degraded, 20 pgs undersized

              69 pgs not deep-scrubbed in time

              1 pgs not scrubbed in time