2021.04.06
1. CD (Continuous Deployment) & Gitops
- CD
✓ 개발자의 변경 사항을 레포지토리에서 개발/프로덕션 환경으로 자동으로 릴리스
✓ 지속적 배포가 제대로 이루어지려면 테스트 자동화가 제대로 설계되어 있어야 함
- Gitops
✓ 클라우드 네이티브 애플리케이션을 대상으로 한 지속적 배포(Continuous Deployment)에 초점
✓ 애플리케이션의 배포와 운영에 관련된 모든 요소를 코드화하여 깃(Git)에서 관리(Ops)하는 것이 핵심
2. Kubernetes Configuration Management
- popular config tools
✓ Kustomize applications
▷ a template-free way to customize application configuration that simplifies the use of off-the-shelf applications.
▷ built into kubectl as apply -k.
▷ customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.
▷ References
https://kustomize.io/
https://github.com/kubernetes-sigs/kustomize
https://github.com/kubernetes-sigs/kustomize/tree/master/examples
✓ Helm charts
▷ it is a self-described package manager for Kubernetes, and doesn’t claim to be a configuration management tool.
✓ Ksonnet applications
▷ A CLI-supported framework for extensible Kubernetes configurations.
$ ks generate deployed-service guestbook-ui \
--image gcr.io/heptio-images/ks-guestbook-demo:0.1 --type ClusterIP
▷ Built on the JSON templating language Jsonnet, ksonnet provides an organizational structure and specialized features for managing configurations across different clusters and environments.
▷ References
https://ksonnet.io/
https://ksonnet.io/docs/tutorial/guestbook/
✓ jsonnect files
▷ A data templating language for app and tool developers. A simple extension of JSON, pronounced "jay sonnet"
$ cat guestbook-ui.jsonnet
local params = import 'params.libsonnet';
[
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"name": params.name
},
"spec": {
"ports": [
{
"port": params.servicePort,
"targetPort": params.containerPort
}
],
"selector": {
"app": params.name
},
"type": params.type
}
},
...
$▷ Reference
https://jsonnet.org/
- Argo CD는 Raw YAML file, Kustomize, Helm, Ksonnet, Jsonnet를 지원
- 참고: https://blog.argoproj.io/the-state-of-kubernetes-configuration-management-d8b06c1205
3. Deploy using deployment and service yaml files
$ kubectl create secret docker-registry chelsea-nexus -n yoosung-jeon \
--docker-server=repo.chelsea.kt.co.kr --docker-username=agp --docker-password=*****
$ vi deployment.yaml
kind: Deployment
metadata:
name: agp-app1
namespace: yoosung-jeon
spec:
replicas: 1
selector:
matchLabels:
name: agp-app1
template:
metadata:
labels:
name: agp-app1
spec:
containers:
- name: agp-app1
image: repo.chelsea.kt.co.kr/agp/agp-app1:latest
ports:
- containerPort: 5000
imagePullSecrets:
- name: chelsea-nexus
$ vi service.yaml
kind: Service
apiVersion: v1
metadata:
name: agp-app1
namespace: yoosung-jeon
spec:
selector:
name: agp-app1
type: NodePort
ports:
- port: 80
protocol: TCP
targetPort: 5000
$ kubectl apply -f deployment.yaml
$ kubectl apply -f service.yaml
$
4. Deploy using Kustomize
$ mkdir -p agp-app1/base && agp-app1/base
$ vi deployment.yaml
…
$ vi service.yaml
…
$ touch kustomization.yaml
$ kustomize edit add resource deployment.yaml
$ kustomize edit add resource service.yaml
$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
$ cd ..
$ mkdir -p overlays/development && mkdir -p overlays/production
$ cd overlays/development
$ touch kustomization.yaml
$ kustomize edit add base ../../base
$ kustomize edit set nameprefix dev-
$ cd - && cd overlays/production
$ touch kustomization.yaml
$ kustomize edit add base ../../base
$ kustomize edit set name-prefix prd-
$ kustomize edit add patch --path deployment-patch.yaml
$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namePrefix: prd-
patches:
- path: deployment-patch.yaml
$ cd ../..
$ tree .
.
├── base
│ ├── deployment.yaml
│ ├── kustomization.yaml
│ └── service.yaml
└── overlays
├── development
│ └── kustomization.yaml
└── production
├── deployment-patch.yaml
└── kustomization.yaml
4 directories, 6 files
$ diff -u <(kustomize build overlays/development) <(kustomize build overlays/production)
--- /dev/fd/63 2021-03-29 15:36:37.134988502 +0900
+++ /dev/fd/62 2021-03-29 15:36:37.135988524 +0900
@@ -1,7 +1,7 @@
apiVersion: v1
kind: Service
metadata:
- name: dev-agp-app1
+ name: prd-agp-app1
namespace: yoosung-jeon
spec:
ports:
@@ -15,10 +15,10 @@
apiVersion: apps/v1
kind: Deployment
metadata:
- name: dev-agp-app1
+ name: prd-agp-app1
namespace: yoosung-jeon
spec:
- replicas: 1
+ replicas: 3
selector:
matchLabels:
name: agp-app1
$ kustomize build overlays/development/ | k apply -f -
service/dev-agp-app1 created
deployment.apps/dev-agp-app1 created
$ kustomize build overlays/production/ | k apply -f -
…
$ k get svc -n yoosung-jeon
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dev-agp-app1 NodePort 10.100.174.77 <none> 80:30078/TCP 70s
prd-agp-app1 NodePort 10.102.179.175 <none> 80:30031/TCP 30s
$ k get pod -n yoosung-jeon -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dev-agp-app1-68b5b9cdf4-2v25x 1/1 Running 0 71s 10.244.14.49 iap12 <none> <none>
prd-agp-app1-68b5b9cdf4-k5vc2 1/1 Running 0 31s 10.244.15.74 iap13 <none> <none>
prd-agp-app1-68b5b9cdf4-p2lfb 1/1 Running 0 31s 10.244.9.91 iap11 <none> <none>
prd-agp-app1-68b5b9cdf4-sr8jc 1/1 Running 0 31s 10.244.10.120 iap10 <none> <none>
$
5. Continous Deployment using Argo CD
a. create project in Gitlab
Menu: + > New project
Project Name: agp-gitopts
Project URL: agp
Visibility Level: Private
Initialize repository with a README: uncheck
b. push Gitops files
$ pwd
/home/iap/k8s/kustomize
$ ls
agp-app1
$ git init
Initialized empty Git repository in /nfs_03/k8s/kustomize/.git/
$ git remote add origin https://scm.chelsea.kt.co.kr/agp/agp-gitops.git
$ git config --global user.email "yoosung.jeon@kt.com"
$ git config --global user.name "유성 전"
$ git add .
$ git commit -m "Initial commit"
[master (root-commit) e5803a9] Initial commit
6 files changed, 62 insertions(+)
create mode 100644 agp-app1/base/deployment.yaml
create mode 100644 agp-app1/base/kustomization.yaml
create mode 100644 agp-app1/base/service.yaml
create mode 100644 agp-app1/overlays/development/kustomization.yaml
create mode 100644 agp-app1/overlays/production/deployment-patch.yaml
create mode 100644 agp-app1/overlays/production/kustomization.yaml
$ git push -u origin master
Username for 'https://scm.chelsea.kt.co.kr': agp
Password for 'https://agp@scm.chelsea.kt.co.kr':
Counting objects: 13, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (13/13), 1.30 KiB | 0 bytes/s, done.
Total 13 (delta 2), reused 0 (delta 0)
...
To https://scm.chelsea.kt.co.kr/agp/agp-gitops.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.
$
c. login n Argo CD : https://14.52.244.139/
d. create repo
Repository 권한: admin 계정에서 생성/삭제 권한, 일반 계정은 조회 권한 부여
Menu: Settings > Repositories > Connect repo using HTTPS
생성할 Repo 값 입력 후 "CONNECT" 클릭
Type: git
Repository URL: https://scm.chelsea.kt.co.kr/agp/agp-gitops.git
Username: agp
e. create new app
- Menu: Applications > NEW APP
- agp-app1 생성
GENERAL
Application Name: agp-app1
Project: agp
SYNC POLICY: Automatic
PRUNE RESOURCES: unchekced
SELF HEAL: checked
SOURCE
Repository URL: https://scm.chelsea.kt.co.kr/agp/agp-gitops.git
Path: agp-app1/overlays/development
DESTINATION
Cluster URL: https://kubernetes.default.svc
- prd-agp-app2 생성
GENERAL
Application Name: prd-agp-app1
Project: default
SYNC POLICY: Manual
SOURCE
Repository URL: https://scm.chelsea.kt.co.kr/agp/agp-gitops.git
Path: agp-app1/overlays/production
DESTINATION
Cluster URL: https://kubernetes.default.svc
- SYNC POLICY Options (Automatic)
✓ Pruning
automated sync will delete resources when Argo CD detects the resource is no longer defined in Git.
✓ Self-Healing
automatic sync when the live cluster's state deviates from the state defined in Git
- Kubernetes에 배포된 내용 확인
$ k get all -n yoosung-jeon
NAME READY STATUS RESTARTS AGE
pod/dev-agp-app1-844bb64b7c-lns8g 1/1 Running 0 61m
pod/dev-agp-app1-844bb64b7c-r8z4n 1/1 Running 0 61m
pod/prd-agp-app1-674df6fb97-2pljd 1/1 Running 0 88s
pod/prd-agp-app1-674df6fb97-6wnfl 1/1 Running 0 88s
pod/prd-agp-app1-674df6fb97-q64dh 1/1 Running 0 88s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dev-agp-app1 NodePort 10.106.242.162 <none> 80:32757/TCP 3h2m
service/prd-agp-app1 NodePort 10.104.32.54 <none> 80:32504/TCP 90s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dev-agp-app1 2/2 2 2 3h2m
deployment.apps/prd-agp-app1 3/3 3 3 90s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dev-agp-app1-844bb64b7c 2 2 2 61m
replicaset.apps/prd-agp-app1-674df6fb97 3 3 3 88s
$
f. Docker images rebuild 시 배포
- Argo CD follows the GitOps model of deployment, where desired configuration changes are first pushed to Git, and the cluster state then syncs to the desired state in git.
- 즉 Argo CD는 Kubernetes manifest 변경을 체크하지만, manifest에서 참조하고 있는 Docker image 변경을 체크 하지 않음
- Jenkins pipeline에서 Docker image 빌드 후 Kubernetes manifest (Image tag 정보)를 변경하고 push 하면 Argo CD에서 인지 할 수 있음
- Jenkinsfile pipeline 일부
def scmURL = "scm.chelsea.kt.co.kr" // GitLab
def scmCredential = "chelsea-gitlab-agp"
def scmAccount = "agp"
def scmProject = "agp-gitops"
def registryURLPull = "repo.chelsea.kt.co.kr" // Docker registry의 Pull와 Push 포트가 다른 경우
stage('Deploy to Dev') {
container('git') {
withCredentials([usernamePassword(credentialsId: "$scmCredential",
usernameVariable: 'SCM_USER',
passwordVariable: 'SCM_PASSWORD')]) {
sh """
git clone https://${SCM_USER}:${SCM_PASSWORD}@${scmURL}/${scmAccount}/${scmProject}.git
"""
}
}
container('kustomize') {
sh """
cd $scmProject/$dockerImage/overlays/development
kustomize edit set image $registryURLPull/$registryAccount/$dockerImage:${env.BUILD_ID}
"""
}
container('git') {
withCredentials([usernamePassword(credentialsId: "$scmCredential",
usernameVariable: 'SCM_USER',
passwordVariable: 'SCM_PASSWORD')]) {
sh """
cd $scmProject
git config --global user.email "jenkins@kt.co.kr"
git config --global user.name "Jenkins pipeline"
git add .
git commit -m "changed Image tag by Jenkins pipeline"
git push
"""
}
}
..
g. Argo CD 살펴보기
- App details
- history And Rollback
- App diff (prd-agp-app1: Manual sync)
'Kubernetes > CI-CD' 카테고리의 다른 글
| CI/CD 적용 가이드 #2 (CI 편) (0) | 2021.09.26 |
|---|---|
| CI/CD 적용 가이드 #1 (개요) (0) | 2021.09.26 |
| Jenkins (0) | 2021.09.18 |
| Harbor (0) | 2021.09.18 |
| Giblab (0) | 2021.09.17 |
댓글