Rook Ceph - OSD autoout

2021.05.14

a. Problem : OSD autoout

- Environments

  Kubernetes 1.16.15, Rook Ceph 1.3.8

-  특정 OSD(Object storage devices)가 autoout 상태이며, 관련 rook-ceph-osd-[number] POD가 기동되지 않은 상태

 

[root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph osd status

+----+-------+-------+-------+--------+---------+--------+---------+----------------+

| id |  host |  used | avail | wr ops | wr data | rd ops | rd data |     state      |

+----+-------+-------+-------+--------+---------+--------+---------+----------------+

| 0  | iap12 |  479G | 14.0T |    0   |     0   |    0   |     0   |   exists,up    |

| 1  | iap13 |    0  |    0  |  126   |  5932k  |   27   |  2676k  | autoout,exists |

| 12 | iap10 |  448G | 14.1T |    0   |  9011   |    0   |     0   |   exists,up    |

…

[root@rook-ceph-tools-79d7c49c8d-kp6xh /]#

 

 

b. Cause Analysis

[iap@iap01 ~]$ k get deployments.apps -n rook-ceph | egrep -w 'NAME|rook-ceph-osd-1'

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE

rook-ceph-osd-1                  0/1     0            0           244d

[iap@iap01 ~]$

[iap@iap01 ~]$ k get pod -n rook-ceph | egrep 'rook-ceph-osd-1-'

[iap@iap01 ~]$ k get replicasets.apps -n rook-ceph | egrep 'rook-ceph-osd-1-'

[iap@iap01 ~]$

[iap@iap01 ~]$ k rollout restart deployment rook-ceph-osd-1 -n rook-ceph

deployment.apps/rook-ceph-osd-1 restarted

[iap@iap01 ~]$ k rollout status  deployment rook-ceph-osd-1 -n rook-ceph

Waiting for deployment "rook-ceph-osd-1" rollout to finish: 0 out of 1 new replicas have been updated...

 

 

c. Solution:

    There is no command for bringing the osd up because the osd daemon is responsible for reporting that it is up.

 

[iap@iap01 ~]$ k get deployments.apps rook-ceph-osd-1 -n rook-ceph -o yaml > rook-ceph-osd-1.yaml

[iap@iap01 ~]$ k delete -f rook-ceph-osd-1.yaml

…

[iap@iap01 ~]$ k apply -f rook-ceph-osd-1.yaml

…

[iap@iap01 ~]$

[iap@iap01 ~]$ k get replicasets.apps -n rook-ceph | egrep ‘NAME|rook-ceph-osd-1-'

NAME                                        DESIRED   CURRENT   READY   AGE

rook-ceph-osd-1-756969f685                  1         1         1       13s

[iap@iap01 ~]$ ceph-toolbox.sh ceph osd status

+----+-------+-------+-------+--------+---------+--------+---------+----------------+

| id |  host |  used | avail | wr ops | wr data | rd ops | rd data |     state      |

+----+-------+-------+-------+--------+---------+--------+---------+----------------+

| 0  | iap12 | 2276G | 12.3T |    0   |     0   |    6   |   106   |   exists,up    |

| 1  | iap13 |  642G | 13.9T |    0   |     0   |    0   |     0   |   exists,up    |

…

[iap@iap01 ~]$