NFS subdir external provisioner

1. 개요

- 쿠버네티스에는 내장 NFS 프로비저너가 없다. NFS를 위한 스토리지클래스를 생성하려면 외부 프로비저너를 사용해야 한다. 

   ✓ NFS Ganesha server and external provisioner

   ✓ NFS subdir external provisioner 

- NFS subdir external provisioner

   ✓ PVC(Persistent Volume Claim)에 대한 쿠버네티스 PV(Persistent Volume)를동적으로 프로비저닝하기 위하여 사전에 구성된 NFS server를 사용하는 automatic provisioner이다.

   ✓ PV는 "${namespace}-${pvcName}-${pvName}"로 프로비저닝된다.

   ✓ NFS-Client Provisioner를 마이그레이션 하여 NFS subdir external provisioner를 개발하였다. NFS-Client Provisioner에 대한 상세 정보는 아래와 같다.

       최종 버전: quay.io/external_storage/nfs-client-provisioner v3.1.0-k8s1.11

       참조 문서: https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client

       설치 문서: https://1week.tistory.com/12?category=1229730 

 

 

2. 구성 환경

- NFS subdir external provisioner 4.0.14

- Kubernetes 1.20.11

 

 

3. NFS subdir external provisioner 구성

- Helm으로 구성하는 방법과 직접 구성하는 방법을 제공한다. 본 문서에서는 직접 구성하였다.

   https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner#with-helm

 

a. Get connection information for your NFS server 

- NFS server IP: 14.52.244.xxx

- NFS export path: /nfs_03

 

b. Get the NFS Subdir External Provisioner files

$ git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git
Cloning into 'nfs-subdir-external-provisioner'...
remote: Enumerating objects: 1518, done.
remote: Counting objects: 100% (322/322), done.
remote: Compressing objects: 100% (173/173), done.
remote: Total 1518 (delta 170), reused 260 (delta 146), pack-reused 1196
Receiving objects: 100% (1518/1518), 493.83 KiB | 0 bytes/s, done.
Resolving deltas: 100% (814/814), done.
$ cd nfs-subdir-external-provisioner
$

 

c. Setup authorization

- NFS subdir external provisioner가 설치될 쿠버네티스 네임스테이스를 지정한다. 본 예제에서는 kube-system을 지정하였다.

- 권한 관련 리소스를 배포한다.

$ NAMESPACE=kube-system
$ sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml ./deploy/deployment.yaml
$ kubectl create -f deploy/rbac.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
$

 

d. Configure the NFS subdir external provisioner

- NFS server ip와 NFS export path 정보를 입력한다.

   spec.template.spec.containers[0].env.name[PROVISIONER_NAME].value

   spec.template.spec.containers[0].env.name[PROVISIONER_PATH].value  

   spec.template.spec.containers[0].volumes[0].server

   spec.template.spec.containers[0].volumes[0].path  

- NFS subdir external provisioner 디플로이먼트를 배포한다.

$ vi deploy/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 14.52.xxx.xxx
            - name: NFS_PATH
              value: /nfs_03
      volumes:
        - name: nfs-client-root
          nfs:
            server: 14.52.xxx.xxx
            path: /nfs_03
$

$ kubectl create -f deploy/deployment.yaml
deployment.apps/nfs-client-provisioner created
$ 
$ k get pod -n kube-system | egrep 'NAME|nfs'
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-5bc64f4cbb-xv7v6   1/1     Running   0          4m20s
$

 

e. Deploying your storage class

- Storage class를 생성한다.

   Storage class는 관리자가 제공하는 스토리지의 "classes"를 설명할 수 있는 방법을 제공하며, PV를 동적으로 프로비저닝 할 때 사용되는 정보가 포함되어 있다.

- Default Storage class로 지정한다.

   PVC에서 "storageClassName"을 지정되어 있지 않는 경우 사용할 Storage class이다. 

$ kubectl create -f deploy/class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
$ kubectl get storageclass
NAME                  PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  12s
$
$ kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/managed-nfs-storage patched
$ kubectl get storageclass
NAME                            PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage (default)   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  2m13s
$

 

 

f. Finally, test your environment!

- 테스트 PVC와 Pod를 배포해서 정상적으로 동작하는지 확인한다.

- NFS Server를 사전에 연결해 놓았기 때문에 OS상에서 직접 접근이 가능하다.

$ kubectl create -f deploy/test-claim.yaml -f deploy/test-pod.yaml
persistentvolumeclaim/test-claim created
$
$ k get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Bound    pvc-23634b67-1265-4125-8516-2d3d234c1b96   1Mi        RWX            managed-nfs-storage   16s
$ k get pod
NAME       READY   STATUS      RESTARTS   AGE
test-pod   0/1     Completed   0          23s
$ k get pv pvc-23634b67-1265-4125-8516-2d3d234c1b96
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS          REASON   AGE
pvc-23634b67-1265-4125-8516-2d3d234c1b96   1Mi        RWX            Delete           Bound    default/test-claim   managed-nfs-storage            38s
$ ls /nfs_03/default-test-claim-pvc-23634b67-1265-4125-8516-2d3d234c1b96/
SUCCESS
$ kubectl delete -f deploy/test-pod.yaml -f deploy/test-claim.yaml
pod "test-pod" deleted
persistentvolumeclaim "test-claim" deleted
$

 

 

4. NFS provisioner limitations/pitfalls

- The provisioned storage is not guaranteed. You may allocate more than the NFS share's total size. The share may also not have enough storage space left to actually accommodate the request.
- The provisioned storage limit is not enforced. The application can expand to use all the available storage regardless of the provisioned size.
- Storage resize/expansion operations are not presently supported in any form. You will end up in an error state: Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.