NFS-Client Provisioner

2020.11.09

 

22.01.11: NFS 프로비저너를 구성시최신 버전(NFS subdir external provisioner)으로 구성할 것을 권고한다.

 

1. 개요

    - nfs-client is an automatic provisioner that use your existing and already configured NFS server

      to support dynamic provisioning of Kubernetes Persistent Volumes via Persistent Volume Claims.

      Persistent volumes are provisioned as ${namespace}-${pvcName}-${pvName}.

    - 제약사항

      nfs-client는 PVC 생성시 설정한 size 값이 적용되지 않음

   - References

      https://blog.exxactcorp.com/deploying-dynamic-nfs-provisioning-in-kubernetes/

      https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

 

 

2. Configure Dynamic NFS Provisioner

- Environment

  Kubernetes 1.16.15

  quay.io/external_storage/nfs-client-provisioner v3.1.0-k8s1.11

 

a. Deploying Service Account and Role Bindings

$ vi nfs-rbac.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-provisioner-sa
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1 # auth API
metadata:
  name: nfs-provisioner-clusterRole
rules:
  - apiGroups: [""]                      # rules on persistentvolumes
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-rolebinding
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner-sa
    namespace: kube-system
roleRef:                               # binding cluster role to service account
  kind: ClusterRole
  name: nfs-provisioner-clusterRole    # name defined in clusterRole
  apiGroup: rbac.authorization.k8s.io
---  
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-otherRoles
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch”]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-otherRoles
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner-sa           # same as top of the file
                                       # replace with namespace where provisioner is deployed
    namespace: kube-system
roleRef:
  kind: Role
  name: nfs-provisioner-otherRoles
  apiGroup: rbac.authorization.k8s.io
$ kubectl apply -f nfs-rbac.yaml
...
$

 

b. Deploying Storage Class

- Storage class를 생성하고 default로 지정한다.

$ vi nfs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-sc-iap             # IMPORTANT pvc needs to mention this name
provisioner: nfs-provisioner   # name can be anything
parameters:
  archiveOnDelete: "false"
$ kubectl apply -f nfs-storageclass.yaml
...
$ k get storageclasses.storage.k8s.io nfs-sc-iap
NAME                   PROVISIONER        AGE
nfs-sc-iap             nfs-provisioner    3d5h
$
$ kubectl patch storageclass nfs-sc-iap -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
...
$ k get storageclasses.storage.k8s.io nfs-sc-iap
NAME                   PROVISIONER       AGE
nfs-sc-iap (default)   nfs-provisioner   3d5h
$

 

c. Deploying NFS Provisioner

$ df -h | egrep 'File|nfs'
Filesystem               Size  Used Avail Use% Mounted on
14.52.244.215:/nfs_01     21T   17G   21T   1% /nfs_01
14.52.244.215:/nfs_02     12T  1.4T   11T  11% /nfs_02
14.52.244.215:/nfs_03    3.0T  812M  3.0T   1% /nfs_03
$
$ vi nfs-provisioner.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccountName: nfs-provisioner-sa # name of service account
      containers:
        - name: nfs-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-provisioner-volume
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME         # do not change
              value: nfs-provisioner         # SAME AS PROVISIONER NAME VALUE IN STORAGECLASS
            - name: NFS_SERVER               # do not change
              value: 14.52.244.215           # Ip of the NFS SERVER
            - name: NFS_PATH                 # do not change
              value: /nfs_01                 # path to nfs directory setup
      volumes:
       - name: nfs-provisioner-volume        # same as volumemouts name
         nfs:
           server: 14.52.244.215
           path: /nfs_01
$ kubectl apply -f nfs-provisioner.yaml
...
$

 

 

3.  Creating Persistent Volume and Persistent Volume Claims

    - Persistent Volume Claims are objects that request storage resources from your cluster. They’re similar to a voucher that your deployment can redeem for storage access.

    - Persistent Volume is resource that can be used by a pod to store data that will persist beyond the lifetime of the pod. It is a storage volume that in this case is a nfs volume.

$ cat nfs-test-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-test-pvc
spec:
  storageClassName: nfs-sc-iap
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Mi
$ k apply -f nfs-test-pvc.yaml
persistentvolumeclaim/nfs-test-pvc created
$
$ k get pvc | egrep 'NAME|nfs'
NAME          STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
nfs-test-pvc  Bound   pvc-c94789b6-aabd-4e73-8f24-df33b784c407  500Mi     RWX           nfs-sc-iap    31s
$ k get pv | egrep 'NAME|nfs-test-pvc'
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS    REASON   AGE
pvc-b0bdaeac-3e8e-434e-a683-daf6a7879085   500Mi      RWX            Delete           Bound    default/nfs-test-pvc    nfs-sc-iap               39s
$ ls -l /nfs_01
drwxrwxrwx. 2 root root 4096 10월 14 15:24 @Recently-Snapshot
drwxrwxrwx. 2 root root 4096 10월 14 15:24 @Recycle
drwxrwxrwx. 2 root root 4096 11월  9 13:57 default-nfs-test-pvc-pvc-c94789b6-aabd-4e73-8f24-df33b784c407
$

 

 

4. Trouble shooting

    - TS #1

      Problem: Docker image 설치 시 접근에러 발생

    $ kubectl describe pod nfs-provisioner-665cbd454b-br28k -n kube-system

      Cause:     

    # docker load -i /nfs_03/docker-images/quay.io_external_storage-nfs-client-provisioner_latest

    Loaded image: quay.io/external_storage/nfs-client-provisioner:latest

    # docker images | grep nfs

    quay.io/external_storage/nfs-client-provisioner     latest    16d2f904b0d8    2 years ago    45.5MB

    # docker pull quay.io/external_storage/nfs-client-provisioner:latest

    latest: Pulling from external_storage/nfs-client-provisioner

    a073c86ecf9e: Downloading

    d9d714ee28a7: Downloading

    36dfde95678a: Downloading

    dial tcp 99.84.238.50:443: i/o timeout

    # host 99.84.238.50

    50.238.84.99.in-addr.arpa domain name pointer server-99-84-238-50.sfo5.r.cloudfront.net.

    #

     Solution:

        방화벽 설정 : Outbound로 Any IP 요청 (docker image 설치 시마다 접속 IP(ex. 50.238.84.99)가 변경 되기 때문에)

                            Amazon CloudFront는 인터넷 상의 빠른 컨텐츠 전달을 위한 CDN