Rook Ceph - scrub error

2021.04.14

 

a. Problem: scrub error

• Environments: Kubernetes 1.16.15, Rook Ceph 1.3.8
• 특정 PG(placement groups)에서 data damage 발생
• A Placement Group (PG) is a logical collection of objects that are replicated on OSDs to provide reliability in a storage system.

 

  [iap@iap01 ~]$ ceph-toolbox.sh

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph status

    cluster:

      id:     1ef6e249-005e-477e-999b-b874f9fa0854

      health: HEALTH_ERR

              1 scrub errors

              Possible data damage: 1 pg inconsistent

 

    services:

      mon: 3 daemons, quorum v,w,z (age 9h)

      mgr: a(active, since 27h)

      mds: myfs:1 {0=myfs-b=up:active} 1 up:standby-replay

      osd: 10 osds: 9 up (since 22h), 9 in (since 27h)

  …

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]#

 

 

b. Solution

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph health detail

  HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent

  OSD_SCRUB_ERRORS 1 scrub errors

  PG_DAMAGED Possible data damage: 1 pg inconsistent

      pg 6.1d is active+clean+inconsistent, acting [19,13]

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph pg repair 6.1d

  instructing pg 6.1d on osd.19 to repair

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph health detail

  HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent

  OSD_SCRUB_ERRORS 1 scrub errors

  PG_DAMAGED Possible data damage: 1 pg inconsistent

      pg 6.1d is active+clean+scrubbing+deep+inconsistent+repair, acting [19,13]

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]#

 

  … <<1~2 시간 경과>>…

 

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph health detail

  HEALTH_OK

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]# ceph status

    cluster:

      id:     1ef6e249-005e-477e-999b-b874f9fa0854

      health: HEALTH_OK

 

    services:

      mon: 3 daemons, quorum v,w,z (age 11h)

      mgr: a(active, since 29h)

      mds: myfs:1 {0=myfs-b=up:active} 1 up:standby-replay

      osd: 10 osds: 9 up (since 27m), 9 in (since 28h)

 

    task status:

      scrub status:

          mds.myfs-a: idle

          mds.myfs-b: idle

 

    data:

      pools:   5 pools, 192 pgs

      objects: 531.08k objects, 2.0 TiB

      usage:   4.0 TiB used, 127 TiB / 131 TiB avail

      pgs:     192 active+clean

 

    io:

      client:   5.4 KiB/s rd, 110 KiB/s wr, 1 op/s rd, 7 op/s wr

 

  [root@rook-ceph-tools-79d7c49c8d-kp6xh /]#