2021.4.29
1. Harbor (Private Docker Registry, https://goharbor.io/)
- What is Harbor?
Harbor is an open source container image registry that secures images with role-based access control, scans images for vulnerabilities, and signs images as trusted. A CNCF Incubating project, Harbor delivers compliance, performance, and interoperability to help you consistently and securely manage images across cloud native compute platforms like Kubernetes and Docker.
- Main Features
✓ Cloud native registry
With support for both container images and Helm charts, Harbor serves as registry for cloud native environments like container runtimes and orchestration platforms.
✓ Role based access control (Multi-Tenant)
Users access different repositories through 'projects' and a user can have different permission for images or Helm charts under a project.
✓ Vulnerability Scanning
Harbor scans images regularly for vulnerabilities and has policy checks to prevent vulnerable images from being deployed.
2. Environments
- Harbor helm chart 9.4.4 / Harbor 2.1.3
- Kubernetes 1.16.15
3. Usage
- https://goharbor.io/docs/2.1.0/working-with-projects/
a. create a project and a user
- https://repo.acp.kt.co.kr or https://14.52.244.136:32443
Username: admin
- create a project and a user
Administration > Users > New User > User Name: emo-dev, Email: emo-dev@kt.co.kr, Password: Emo*****
Projects > New Project > Project Name: emo-dev
Projects > emo-dev > Members > User > Name: emo-dev, role: Maintainer
Projects > emo-dev > Configuration > Vulnerability scanning > Automatically scan images on push: Check
b. Pushing Images using docker command
- Syntax
docker tag SOURCE_IMAGE[:TAG] repo.acp.kt.co.kr/emo-dev/REPOSITORY[:TAG]
docker push repo.acp.kt.co.kr/emo-dev/REPOSITORY[:TAG]
- push
$ docker login repo.acp.kt.co.kr
Username: emo-dev
Password:
$ docker pull centos:7
7: Pulling from library/centos
2d473b07cdd5: Pull complete
Digest: sha256:0f4ec88e21daf75124b8a9e5ca03c37a5e937e0e108a255d890492430789b60e
Status: Downloaded newer image for centos:7
$ docker tag centos:7 repo.acp.kt.co.kr/emo-dev/centos:7
$ docker push repo.acp.kt.co.kr/emo-dev/centos:7
The push refers to repository [repo.acp.kt.co.kr/emo-dev/centos]
174f56854903: Pushed
7: digest: sha256:e4ca2ed0202e76be184e75fb26d14bf974193579039d5573fb2348664deef76e size: 529
$ docker pull repo.acp.kt.co.kr/emo-dev/centos:7
...
$ docker logout repo.acp.kt.co.kr
$ docker images | egrep "REPOSITORY|centos"
REPOSITORY TAG IMAGE ID CREATED SIZE
repo.acp.kt.co.kr/emo-dev/centos 7 8652b9f0cb4c 2 months ago 204MB
centos 7 8652b9f0cb4c 2 months ago 204MB
$- Push 결과 조회
- Push된 이미지에 대하여 Vulnerability (취약점) 검사를 자동으로 실행하고, 특정 severity 이상인 경우 Pull 되지 않도록 설정 할 수 있음
- Harbor에서 Self-signed certificate 사용하는 경우 & On Remote (MacOS)
‣ Certificate 저장
Projects > emo-dev > Repositories > Registry Certificate 선택
~/.docker/certs.d/repo.acp.kt.co.kr 디렉토리에 저장
yoosungjeon@ysjeon-Dev ~ % ls .docker/certs.d/repo.acp.kt.co.kr
ca.crt
yoosungjeon@ysjeon-Dev ~ %
‣ Docker 재기동
‣ Docker login
yoosungjeon@ysjeon-Dev ~ % docker login repo.acp.kt.co.kr
Username: emo-dev
Password:
Login Succeeded
yoosungjeon@ysjeon-Dev ~ %
c. Pull an Image from a Private Registry in K8s
- https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
- Create a Secret by providing credentials on the command line
$ k create ns emo-dev
$ k create secret docker-registry reg-cred -n emo-dev \
--docker-server=repo.acp.kt.co.kr --docker-username=emo-dev --docker-password=Emo*****
secret/regcred created
$ kubectl get secret reg-cred -n emo-dev -o jsonpath='{.data.\.dockerconfigjson}' | base64 --decode
{"auths":{"repo.acp.kt.co.kr":{"username":"emo-dev","password":"Emo*****","auth":"ZW1vLWRldjpFbW8xMjM0NQ=="}}}
$ echo ZW1vLWRldjpFbW8xMjM0NQ== | base64 --decode
emo-dev:Emo*****
$ # 특정 namespace의 default serviceaccounts에 Docker login을 위한 자격증명을 아래와 같이 등록하면
# POD의 imagePullSecrets 항목을 설정하지 않아도 됨
$ k get serviceaccounts -n emo-dev
NAME SECRETS AGE
default 1 102s
$ kubectl patch serviceaccount default -n emo-dev -p "{\"imagePullSecrets\": [{\"name\": \"reg-cred\"}]}"
serviceaccount/default patched
$
- Pull an image
$ vi private-reg.yaml
apiVersion: v1
kind: Pod
metadata:
name: private-reg
namespace: emo-dev
spec:
containers:
- name: private-reg-container
image: repo.acp.kt.co.kr/emo-dev/centos:7
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
$ k apply -f private-reg.yaml
$ k get pod private-reg -n emo-dev
NAME READY STATUS RESTARTS AGE
private-reg 1/1 Running 0 36s
$
4. Harbor Install
- https://hub.kubeapps.com/charts/bitnami/harbor
4-1. Self-signed certificate
a. Install Harbor helm chart
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install harbor-registry bitnami/harbor --create-namespace --namespace harbor-registry \
--set service.type=NodePort --set service.nodePorts.https=32443 \
--set service.tls.commonName=repo.acp.kt.co.kr \
--set externalURL=https://repo.acp.kt.co.kr --set harborAdminPassword=admin****
NAME: harbor-registry
LAST DEPLOYED: Fri Jan 29 16:08:10 2021
NAMESPACE: harbor-registry
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
1. Get the Harbor URL:
export NODE_PORT=$(kubectl get --namespace harbor-registry -o jsonpath="{.spec.ports[0].nodePort}" services harbor-registry)
export NODE_IP=$(kubectl get nodes --namespace harbor-registry -o jsonpath="{.items[0].status.addresses[0].address}")
echo "Harbor URL: http://$NODE_IP:$NODE_PORT/"
2. Login with the following credentials to see your Harbor application
echo Username: "admin"
echo Password: $(kubectl get secret --namespace harbor-registry harbor-registry-core-envvars -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 --decode)
$
$ helm list -A | egrep "NAME|harbor"
NAME NAMESPACE REVISION UPDATED STATUS CHART PP VERSION
harbor-registry harbor-registry 1 2021-01-29 16:08:10.45626081 +0900 KST deployed harbor-9.4.4 2.1.3
$ k get pod -n harbor-registry
NAME READY STATUS RESTARTS AGE
harbor-registry-chartmuseum-5c77c4b856-qqxdp 1/1 Running 0 70s
harbor-registry-clair-7d574f64bd-dpfrc 2/2 Running 1 70s
harbor-registry-core-77c8984fbc-kc2rb 1/1 Running 0 70s
harbor-registry-jobservice-7dfb4659f6-2xm92 1/1 Running 0 70s
harbor-registry-nginx-6c44bf766d-2p2br 1/1 Running 0 70s
harbor-registry-notary-server-64976dff8d-5rzvq 1/1 Running 0 70s
harbor-registry-notary-signer-79746db695-22fbx 1/1 Running 0 70s
harbor-registry-portal-6bc8846c99-zwh66 1/1 Running 0 70s
harbor-registry-postgresql-0 1/1 Running 0 70s
harbor-registry-redis-master-0 1/1 Running 0 70s
harbor-registry-registry-56b775d89d-d22mj 2/2 Running 0 70s
harbor-registry-trivy-0 1/1 Running 0 70s
$ k get svc -n harbor-registry | egrep -w "NAME|harbor-registry"
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
harbor-registry NodePort 10.102.89.219 <none> 80:31027/TCP,443:32443/TCP,4443:31800/TCP 2m5s
...
$ k get pvc -n harbor-registry
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-harbor-registry-postgresql-0 Bound pvc-e848dffe-df17-4631-8367-cd0c6ae66036 8Gi RWO rook-ceph-block-sc-iap 16m
data-harbor-registry-trivy-0 Bound pvc-94581ed5-1692-44cb-8fb7-df9aa38ccd93 5Gi RWO rook-ceph-block-sc-iap 16m
harbor-registry-chartmuseum Bound pvc-046622b7-9db7-4b3d-be1f-5bfa9fab9b73 5Gi RWO rook-ceph-block-sc-iap 16m
harbor-registry-jobservice Bound pvc-ffabc960-a9ba-4e5b-8654-5146ad1ab0a2 1Gi RWO rook-ceph-block-sc-iap 16m
harbor-registry-registry Bound pvc-87e4b185-c54a-4cb4-97ce-ab3a056779a7 5Gi RWO rook-ceph-block-sc-iap 16m
redis-data-harbor-registry-redis-master-0 Bound pvc-5297606c-e652-4861-8a2c-6caebabe252d 8Gi RWO rook-ceph-block-sc-iap 16m
$
b. apply Self-signed certificate
- Harbor가 설치되면서 자동 생성된 CA(Certificate authority) 파일을 NFS로 저장
$ mkdir -p /nfs_03/docker/certs.d/repo.acp.kt.co.kr && cd /nfs_03/docker/certs.d/repo.acp.kt.co.kr
$ k get secrets harbor-registry-nginx -n harbor-registry -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt
- K8s Master node와 Worker node에서 CA 파일을 Docker 설정디렉토리에 적용, docker 재기동 불 필요
# mkdir -p /etc/docker/certs.d && cd /etc/docker/certs.d && ln -s /nfs_03/docker/certs.d/repo.acp.kt.co.kr
#
# docker login https://repo.acp.kt.co.kr
Username: admin
Password:
# docker logout https://repo.acp.kt.co.kr
Removing login credentials for repo.acp.kt.co.kr
#
4-2. SSL Certificate
a. Install Harbor helm chart
Harbor에서 사용할 Certificate가 있는 경우는 아래와 같이 적용한다.
$ k create ns harbor-registry
$ openssl rsa -in gsd_kt_co_kr_key.pem -out gsd_kt_co_kr_key-no_pass_phrase.pem
Enter pass phrase for gsd_kt_co_kr_key.pem:
$ kubectl create secret generic gsd-certificate -n harbor-registry \
--from-file=tls.crt=gsd_kt_co_kr_cert.pem \
--from-file=tls.key=gsd_kt_co_kr_key-no_pass_phrase.pem \
--from-file=ca.crt=Chain_RootCA_Bundle.crt
$ helm install harbor-registry bitnami/harbor --create-namespace --namespace harbor-registry \
--set service.type=NodePort --set service.nodePorts.https=32443 --set harborAdminPassword=admin**** \
--set service.tls.existingSecret=gsd-certificate \
--set externalURL=https://gsd.kt.co.kr:32443
$b. 적용된 Certificate
'Kubernetes > CI-CD' 카테고리의 다른 글
| CI/CD 적용 가이드 #2 (CI 편) (0) | 2021.09.26 |
|---|---|
| CI/CD 적용 가이드 #1 (개요) (0) | 2021.09.26 |
| Jenkins (0) | 2021.09.18 |
| Giblab (0) | 2021.09.17 |
| Argo CD (0) | 2021.09.16 |
댓글