NFS-Client Provisioner - PV 마운트 위치

1. 개요

- Docker container가 PV(Pysical Volume)를 어떻게 연결해서 사용하는지 이해한다.

- 본 환경은 Dynamic provisioning이 구성되어 있어 PVC(Pysical Volume Claim리소스를 생성하면 자동으로 PV가 생성된다.

  Storage : NFS

  Provisioner: quay.io/external_storage/nfs-client-provisioner v3.1.0-k8s1.11

  구성 관련 문서: NFS-Client Provisioner

- Docker container에 마운트된 PV는 '/var/lib/kubelet/pods' 디렉토리 이하에 위치한다.

 

 

2. Docker container 정보 확인

- NFS에 생성된 PV을 사용하는 예제 Pod에 대한 정보를 조회한다.

   예제 Pod는 study-0이고, iap12 노드에서 실행 중이며 하나의 container(study)를 가지고 있다.

   study cotainer ID는 131af95d96cb0a0182393f3ab22800a0ac7eb0f1860feb5d285bc1ad6a75e283이다.

   study container가 사용하는 PV는 pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f이다.

$ k describe pod study-0 -n yoosung-jeon
Name:         study-0
Namespace:    yoosung-jeon
Node:         iap12/14.52.244.216
…
Containers:
  study:
    Container ID:   docker://131af95d96cb0a0182393f3ab22800a0ac7eb0f1860feb5d285bc1ad6a75e283
…
    Mounts:
      /dev/shm from dshm (rw)
      /home/jovyan from workspace-study (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-editor-token-j6l58 (ro)
…
Volumes:
  workspace-study:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  workspace-study
    ReadOnly:   false

…
$ k get pvc -n yoosung-jeon workspace-study
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
workspace-study   Bound    pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f   10Gi       RWO            nfs-sc-iap     130d
k get pv pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS   REASON   AGE
pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f   10Gi       RWO            Delete           Bound    yoosung-jeon/workspace-study   nfs-sc-iap              130d
$

 

- iap12 서버에서 Container ID로 docker container 프로세스를 찾는다. 프로세스명은 [docker-]containerd-shim이다.

   해당 프로세스가 사용하는 주요 디렉토리 및 파일은 아래와 같다. 

   ✓ Root directory of persistent Docker state (default "/var/lib/docker")

   ✓ Root directory for execution state files (default "/var/run/docker")

   ✓ Path to use for daemon PID file (default "/var/run/docker.pid")

# ps -ef | grep 131af95d96cb0a0182393f3ab22800a0ac7eb0f1860feb5d285bc1ad6a75e283 | grep -v grep
root   34706   3181  0 15:13 ?   00:00:00 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/131af95d96cb0a0182393f3ab22800a0ac7eb0f1860feb5d285bc1ad6a75e283 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup
#

 

 

3. PV 마운트 위치 찾기

- Container가 실행하고 있는 노드에서 아래와 같이 PV 마운트 디렉토리 위치를 찾는다. 

   마운트 디렉토리: /var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f

 

a. PV 명 (pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f)

# df -h | grep 'pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f' | awk '{print $6}'
/var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f
#
# tree /var/lib/kubelet/pods/*/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f | head -n 1
/var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f
#

 

b. Pod 명 (study-0)

# grep study-0  /var/lib/kubelet/pods/*/etc-hosts
/var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/etc-hosts:10.244.2.176       study-0
# ls /var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26
containers  etc-hosts  plugins  volumes
#

 

- Pod의 마운트 항목들

# df -h | egrep 'Mounted|69ca289f-aa1e-4c90-9643-377731e5ec26'
Filesystem                                                                                    Size  Used Avail Use% Mounted on
tmpfs                                                                                          63G     0   63G   0% /var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~empty-dir/dshm
tmpfs                                                                                          63G   12K   63G   1% /var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~secret/default-editor-token-j6l58
14.52.244.215:/nfs_01/yoosung-jeon-workspace-study-pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f    21T  5.9T   16T  29% /var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f
#

 

- 마운트 된 디렉토리 내용 확인

# tree /var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f | head
/var/lib/kubelet/pods/69ca289f-aa1e-4c90-9643-377731e5ec26/volumes/kubernetes.io~nfs/pvc-4da89616-ba82-4273-8e1f-a5d3dde76b4f
└── kf-distributed-training-exam
    ├── 1.local-dev-fairing
    │   ├── Dockerfile
    │   ├── fairing.py
    │   ├── mnist-mwms.py
    │   ├── run.sh
    │   └── yaml
    │       ├── mnist-pvc.yaml
    │       └── serviceac

$ k exec study-0 -n yoosung-jeon -it -- ls /home/jovyan
kf-distributed-training-exam
$