Rook ceph vs NFS

2020.11.20

1. 개요

    K8s storage로 Rook ceph와 NFS 중 어떤 것을 선택할지 검토하기 위하여 작성

 

 

2. 검토 의견

- 개발 환경 (in GiGA Tech Hub)는  Rook ceph와 NFS 스토리지 모두를 제공하며 기본으로 Rook ceph를 사용

- 운영환경과 시스템별로 별도 구축되는 개발 환경은 NFS 스토리지만 제공 

   운영 이관 고려 :  Rook Ceph 엔지니어 부재 및 적용 사례가 없어 이관 협의 시 이슈화 될 수 있기 때문에

 

 

3. Rook ceph vs NFS 비교

a. 기능 / 디스크 사용율 관점

	기능	디스크 사용율 (Block storage)	SPOF	구성 관리 / 운영
Rook Ceph	- Block storage, Shared Filesystem,   Object storage(Amazon S3 I/F) 기능 - Dynamic provisioning 지원 - PVC resize 지원 (K8s 1.16 이상)	Erasure coding 구성시 75% Usable    (Data:6, Coding: 2  설정시 OSD 8개 이상 필요) Replica로 3 구성시 33%, 2 구성시 50% Usable	M+N 제공	매우 복잡
NFS	- Block storage 기능   (ReadWriteOnce, ReadWriteMany) - Dynamic provisioning 지원 - PVC Size 및 resize 미지원	NAS에서 RAID 5 구성 시 75% (일반적), RAID 10 구성시 50% Usable	None	간단

- Rook ceph와 NFS 스토리지에 PV/PVC(Size:5 GB) 생성 결과

 $ k get pvc | egrep "NAME|perf-test"

 NAME               STATUS VOLUME                                   CAPACITY ACCESS MODES STORAGECLASS           AGE

 perf-test-ceph-pvc Bound  pvc-25ea09e2-cb14-47cb-a165-260b2dece744 5Gi      RWO          rook-ceph-block-sc-iap 2d

 perf-test-nfs-pvc  Bound  pvc-b886c47b-7278-4ebc-b7c4-44baf72da24c 5Gi      RWO          nfs-sc-iap             9d

 $

  POD 내에 Mount된 내용 조회

 sh-4.2# df -h | egrep 'File|nfs|ceph'

 Filesystem                                                         Size  Used Avail Use% Mounted on

 14.52.244.215:/nfs_01/default-perf-test-nfs-pvc-pvc-b886c47b-...   21T   18G   21T   1%  /nfs

 /dev/rbd10                                                         4.8G   20M  4.8G   1% /ceph

 sh-4.2#

 

- Erasure Coding                                                      

  Erasure coding uses storage capacity more efficiently than replication. 

  The n-replication approach maintains n copies of an object (3x by default in Ceph), whereas erasure coding maintains only k + m chunks. 

  For example, 3 data and 2 coding chunks use 1.5x the storage space of the original object.

 

b. 성능 관점

테스트 환경에서 각각 항목별 3회 수행한 결과이며, 테스트 환경에 따라 성능 결과가 상이할 수 있으므로 단순 참조만 할 것

테스트 시점에 Rook ceph는 PostgreSQL, Elasticsearch, Kafka에서 사용 중 이었음

	dd Command	fio Command
		Random R/W	Random Read	Random Write	Sequential Write
K8s (Rook Ceph)	12.3 s, 87.3 MB/s	R: 3.3 MB/s, iops=834  W: 1.1 MB/s, iops=279	2.4 MB/s, iops=604	4.7 MB/s, iops=1181	17.9 MB/s, iops=4478
K8s (NFS)	15.3 s, 69.9 MB/s	R: 70.2 MB/s, iops=17555  W: 23.3 MB/s, iops=5837	78.2 MB/s, iops=19557	56.6 MB/s, iops=14169	91.3 MB/s, iops=22848
Local (NFS)	14.5 s, 74.0 MB/s
Local (HDD)	0.7 s, 1.4 GB/s

- 소프트웨어 환경

   CentOS Linux release 7.8

   Kubernetes 1.16, Rook v1.3.6, ceph image version: "14.2.9-0 nautilus”

   fio-2.0.9

 

- 스토리지 환경

           NAS : QNAP TS-873U-RP-8G (RAID 5 구성)

           Rook Ceph : 

               [iap@iap01 bin]$ kubectl -n rook-ceph exec -it \ 

               $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') \

       ceph osd status

               +----+-------+-------+-------+--------+---------+--------+---------+-----------+

               | id |  host |  used | avail | wr ops | wr data | rd ops | rd data |   state   |

               +----+-------+-------+-------+--------+---------+--------+---------+-----------+

               | 0  | iap09 | 1115M |  278G |    0   |     0   |    0   |     0   | exists,up |

               | 1  | iap07 | 1116M |  278G |    0   |     0   |    0   |     0   | exists,up |

               | 2  | iap08 | 1115M |  278G |    0   |     0   |    0   |     0   | exists,up |

               | 3  | iap07 | 1437M |  557G |    0   |     0   |    0   |     0   | exists,up |

               | 4  | iap09 | 1115M |  557G |    0   |     0   |    0   |     0   | exists,up |

               | 5  | iap08 | 1115M |  557G |    0   |     0   |    0   |     0   | exists,up |

               | 6  | iap04 | 55.3G |  410G |  103   |  1134k  |    1   |  16.0k  | exists,up |

               | 7  | iap05 | 1115M |  464G |    0   |     0   |    0   |     0   | exists,up |

               | 8  | iap06 | 12.1G |  453G |    0   |     0   |    0   |     0   | exists,up |

               | 9  | iap05 | 1115M |  464G |    0   |     0   |    0   |     0   | exists,up |

               | 10 | iap04 | 1115M |  464G |    0   |     0   |    0   |     0   | exists,up |

               | 11 | iap06 | 1115M |  464G |    0   |     0   |    0   |     0   | exists,up |

               | 12 | iap10 |  539G | 14.0T |  646   |  2603k  |    7   |   828k  | exists,up |

               | 13 | iap10 |  324G | 14.2T |  417   |  2546k  |   13   |   852k  | exists,up |

               | 14 | iap10 |  487G | 14.0T |  224   |  2592k  |   14   |  1638k  | exists,up |

               | 15 | iap10 |  378G | 14.1T |   17   |   822k  |    0   |  1638   | exists,up |

               | 16 | iap11 |  431G | 14.1T |  421   |  3060k  |   32   |  3834k  | exists,up |

               | 17 | iap11 |  270G | 14.2T |    8   |  3337k  |    0   |     0   | exists,up |

               | 18 | iap11 |  432G | 14.1T |  640   |  6540k  |   15   |  1657k  | exists,up |

               | 19 | iap11 |  540G | 14.0T | 1122   |  9405k  |   24   |  2502k  | exists,up |

               +----+-------+-------+-------+--------+---------+--------+---------+-----------+

 

 

4. 테스트 

- https://support.binarylane.com.au/support/solutions/articles/1000055889-how-to-benchmark-disk-i-o

a. 성능 테스트 환경 구축

  $ mkdir -p ~/k8s/storage-perf-test && cd ~/k8s/storage-perf-test

  $ k get storageclasses.storage.k8s.io | egrep "NAME|nfs|ceph"

  NAME                               PROVISIONER                  AGE

  nfs-sc-iap                         nfs-provisioner              14d

  rook-ceph-block-sc-iap (default)   rook-ceph.rbd.csi.ceph.com   106d   # Replica 2 적용

  $ vi perf-test-ceph-pvc.yaml

  apiVersion: v1

  kind: PersistentVolumeClaim

  metadata:

    name: perf-test-ceph-pvc

  spec:

    storageClassName: rook-ceph-block-sc-iap

    accessModes:

      - ReadWriteOnce

    resources:

      requests:

        storage: 5Gi

  $ vi perf-test-nfs-pvc.yaml

  apiVersion: v1

  kind: PersistentVolumeClaim

  metadata:

    name: perf-test-nfs-pvc

  spec:

    storageClassName: nfs-sc-iap

    accessModes:

      - ReadWriteOnce

    resources:

      requests:

        storage: 5Gi

  $ kubectl apply -f perf-test-ceph-pvc.yaml

  $ kubectl apply -f perf-test-nfs-pvc.yaml

  $ vi perf-test.yaml

  apiVersion: v1

  kind: Pod

  metadata:

    name: perf-test

  spec:

    containers:

    - name: perf-test

      image: centos:7

      command:

        - sleep

        - "3600"

      volumeMounts:

      - name: host-volume-nfs

        mountPath: /nfs

      - name: host-volume-ceph

        mountPath: /ceph

    volumes:

    - name: host-volume-nfs

      persistentVolumeClaim:

        claimName: perf-test-nfs-pvc

    - name: host-volume-ceph

      persistentVolumeClaim:

        claimName: perf-test-ceph-pvc

  $ kubectl get pod perf-test -o wide

  NAME        READY   STATUS    RESTARTS   AGE     IP              NODE    NOMINATED NODE   READINESS GATES

  perf-test   1/1     Running   0          9m15s   10.244.10.137   iap10   <none>           <none>

  $ kubectl exec perf-test -it -- sh

  sh-4.2# cat /etc/centos-release

  CentOS Linux release 7.8.2003 (Core)

  sh-4.4# df -h | egrep "File|nfs|ceph"

  Filesystem                                                                                Size  Used Avail Use% Mounted on

  14.52.244.215:/nfs_01/default-perf-test-nfs-pvc-pvc-b886c47b-7278-4ebc-b7c4-44baf72da24c   21T   18G   21T   1% /nfs

  /dev/rbd10                                                                                4.8G   20M  4.8G   1% /ceph

  sh-4.4# yum install -y make gcc libaio-devel wget

  sh-4.2# cd /nfs &&  wget https://github.com/Crowd9/Benchmark/raw/master/fio-2.0.9.tar.gz ; tar xf fio*

  sh-4.2# cd fio-2.0.9 && make

  sh-4.2#

 

b. 테스트 명령어

- Random read/write

      sh-4.2# cd /ceph

      sh-4.2# /nfs/fio-2.0.9/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \

              --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --rwmixread=75

      fio-2.0.9

      Starting 1 process

      Jobs: 1 (f=1): [m] [100.0% done] [1662K/579K /s] [415 /144  iops] [eta 00m:00s]

      test: (groupid=0, jobs=1): err= 0: pid=445: Thu Nov 12 07:31:18 2020

        read : io=785452KB, bw=3339.8KB/s, iops=834 , runt=235182msec

        write: io=263124KB, bw=1118.9KB/s, iops=279 , runt=235182msec

        cpu          : usr=0.85%, sys=4.24%, ctx=427536, majf=0, minf=431

        IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%

           submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

           complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%

           issued    : total=r=196363/w=65781/d=0, short=r=0/w=0/d=0

      Run status group 0 (all jobs):

         READ: io=785452KB, aggrb=3339KB/s, minb=3339KB/s, maxb=3339KB/s, mint=235182msec, maxt=235182msec

        WRITE: io=263124KB, aggrb=1118KB/s, minb=1118KB/s, maxb=1118KB/s, mint=235182msec, maxt=235182msec

      Disk stats (read/write):

        rbd10: ios=196300/65850, merge=0/46, ticks=10077045/4837541, in_queue=14932174, util=100.00%

      sh-4.2#

- Random read

      # /nfs/fio-2.0.9/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \

        --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randread

- Random write

      # /nfs/fio-2.0.9/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \

        --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randwrite

- Sequantial write

      # /nfs/fio-2.0.9/fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \

        --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=write

- dd command

      # time dd if=/dev/zero of=test bs=64k count=16k