2021.03.23, 2020.07.13
Cert-manager with LetsEncrypt (DNS challenge): https://1week.tistory.com/2
Cert-manager with LetsEncrypt (HTTP challenge)
1. 개요
- Cert-manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self signed.
Cert-manager can be used to obtain certificates from a CA using the ACME protocol.
- The ACME protocol supports various challenge mechanisms which are used to prove ownership of a domain
so that a valid certificate can be issued for that domain.
- ACME Architecture
https://vocon-it.com/2019/01/08/kubernetes-automatic-tls-certificates-with-lets-encrypt/
✓ ACME(Automated Certificate Management Environment) challenge mechanisms
https://medium.com/@gregoire.waymel/istio-cert-manager-lets-encrypt-demystified-c1cbed011d67
▷ HTTP (https://cert-manager.io/docs/configuration/acme/http01/)
▷ DNS (https://cert-manager.io/docs/configuration/acme/dns01/)
✓ ACME Debugging 사이트
https://letsdebug.net/, https://tools.letsdebug.net/cert-search, http://dnsviz.net
2. 고려사항
2.1 Requirements
- 방화벽 룰 설정
✓ Inbound (Src: Any, Dest: 14.52.244.138:80)
For the “http-01” ACME challenge, you need to allow inbound port 80 traffic.
We don’t publish the IP ranges from which we perform validation, and they will change without notice.
https://community.letsencrypt.org/t/lets-encrypt-server-addresses-for-certificate-renewal/83466/2
$ k get svc -n gitlab | grep gitlab-nginx-ingress-controller
gitlab-nginx-ingress-controller LoadBalancer 10.109.176.173 14.52.244.138 80:31681/TCP,443:32274/TCP,22:32336/TCP
$KT GTH 사외망에 있는 서버는 Any port로 오픈 불가 - http 대신 DNS로 인증 진행 해야 함
✓ Outbound(Src: K8s worker nodes, Dest: acme-v02.api.letsencrypt.org:443)
Outbound(Src: K8s worker nodes, Dest: acme-staging-v02.api.letsencrypt.org:443)
2.2 TLS Certificate 발급을 위한 순서
- orders.certmanager.k8s.io ⇢ challenges.certmanager.k8s.io ⇢
certificaterequests.certmanager.k8s.io ⇢ certificates.certmanager.k8s.io
- 관련 Objects 조회
k get orders.certmanager.k8s.io -n gitlab
k get challenges.certmanager.k8s.io -n gitlab
k get certificaterequests.certmanager.k8s.io -n gitlab
k get certificates.certmanager.k8s.io -n gitlab
2.3. 강제 Certificate 발급 요청
- 명령어
k delete orders.certmanager.k8s.io -n gitlab --all
- 빈번하게 수행될 경우 Let’s Encrypt에서 일정 시간 대기 발생 됨
Let’s Encrypt 서버를 Production에서 Staging environment 변경
$ k edit configmap gitlab-certmanager-issuer-certmanager -n gitlab
…
# server: "https://acme-v02.api.letsencrypt.org/directory"
server: "https://acme-staging-v02.api.letsencrypt.org/directory"
…
$
2.4 challenge 테스트
- Token 조회
$ k describe pod -n gitlab -l certmanager.k8s.io/acme-http01-solver=true | grep "Args:" -A4
Args:
--listen-port=8089
--domain=gitlab.14.52.244.138.sslip.io
--token=-04WA6YyKJb2bud0Vs3ZokLcjAwMu5l8NwcZhuNDPg0
--key=-04WA6YyKJb2bud0Vs3ZokLcjAwMu5l8NwcZhuNDPg0.FkjJQsPabF0wZf33O6NFCZ6wX_wHs4WHhYZu7EaPiPE
$
- challenge 요청
$ curl http://gitlab.14.52.244.138.sslip.io/.well-known/acme-challenge/Gp-1AtU5tllDnerGyfEb0SPTQgItf0wezTdjwkCk4d4 -vk
* About to connect() to gitlab.14.52.244.138.sslip.io port 80 (#0)
* Trying 14.52.244.138...
* Connected to gitlab.14.52.244.138.sslip.io (14.52.244.138) port 80 (#0)
> GET /.well-known/acme-challenge/Gp-1AtU5tllDnerGyfEb0SPTQgItf0wezTdjwkCk4d4 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: gitlab.14.52.244.138.sslip.io
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 23 Mar 2021 10:23:33 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 87
< Connection: keep-alive
< Cache-Control: no-cache, no-store, must-revalidate
< Referrer-Policy: strict-origin-when-cross-origin
<
* Connection #0 to host gitlab.14.52.244.138.sslip.io left intact
Gp-1AtU5tllDnerGyfEb0SPTQgItf0wezTdjwkCk4d4.FkjJQsPabF0wZf33O6NFCZ6wX_wHs4WHhYZu7EaPiPE
$
- challenge용 ingress/POD (임시 생성 후 삭제 됨)
$ k describe ingresses. cm-acme-http-solver-5j4cm -n gitlab
Name: cm-acme-http-solver-5j4cm
Namespace: gitlab
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
registry.14.52.244.138.sslip.io
/.well-known/acme-challenge/af6RJnDqaC_P47wU9k0Yb0LS1LP62EqARup6Ue4pu2U cm-acme-http-solver-j2cbp:8089 (10.244.15.159:8089)
...
$ kubectl get pod -n gitlab | grep -i acme
gitlab cm-acme-http-solver-8wxmk 1/1 Running 0 9s
gitlab cm-acme-http-solver-f9zxh 1/1 Running 0 10s
gitlab cm-acme-http-solver-fsp9h 1/1 Running 0 9s
$
2.5 로그 조회
$ k logs -l app=nginx-ingress -n gitlab -f | grep GET
$ k logs cm-acme-http-solver-2f4tm -n gitlab -f
3. Kubeflow
3.1. 환경
- Kubernetes v1.15.12
- Case1: Kubeflow 1.0.2 with Istio 1.3, cert-manager v0.11, Dex <- Certificate 적용 불가 - istio SDS 비 활성화
Case2: Kubeflow 1.0.2 with cert-manager v0.11, Dex + istio 1.5.8 <- Certificate 적용
3.2. ACME with HTTP 구성하기
- cert-manager에서 istio https 서비스용 인증서를 Let’s Encrypt에서 무료로 발급 받아 적용하고자 할 경우 구성
- Let’s Encrypt에서 인증서를 발급받기 위해서 도메인이 필요하며, 없는 경우 IP기반의 도메인 활용 (xxx.xxx.xxx.xxx.sslip.io)
- Kubeflow 1.0.2 (istio 1.3)에서는 istio SDS(Secret Discovery Service) 비활성화 되어 발급된 인증서를 적용할 수 없음
- cert-manager resource name
API group changing to be cert-manager.io instead of certmanager.k8s.io
ex) clusterissuers.certmanager.k8s.io -> clusterissuers.cert-manager.io
- cert-manager로 생성한 Certificate를 적용하기 위해서는 istio SDS(Secure Discovery Service) 기능이 활성화 되야 함
a. ClusterIssuer 생성
- ClusterIssuers are a resource type similar to Issuers. They are specified in exactly the same way, but they do not belong to a single namespace and can be referenced by Certificate resources from multiple different namespaces.
- They are particularly useful when you want to provide the ability to obtain certificates from a central authority (e.g. Letsencrypt, or your internal CA) and you run single-tenant clusters.
$ vi clusterissuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: kubeflow-letsencrypt-issuer
spec:
# https://cert-manager.io/docs/configuration/acme/
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring certificates, and issues related to your account.
email: ysjeon71@gmail.com
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: kubeflow-letsencrypt-issuer-secret
server: https://acme-v02.api.letsencrypt.org/directory # Let’s Encrypt’s production environment
# server: https://acme-staging-v02.api.letsencrypt.org/directory # Let’s Encrypt’s staging environment
# Add a single challenge solver
solvers:
- http01:
ingress:
class: istio
$ k apply -f clusterissuer.yaml
b. Certificate 생성
- Certificate의 namespace는 istio-ingressgateway와 동일한 namespace로 지정해야 함
$ vi certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: kubeflow-cert
namespace: istio-system
spec:
secretName: kubeflow-cert
commonName: 34.75.224.5.sslip.io
dnsNames:
- 34.75.224.5.sslip.io
# - 34-75-224-5.sslip.io
issuerRef:
name: kubeflow-letsencrypt-issuer
kind: ClusterIssuer
$ k apply -f certificate.yaml
…
$ k get pod -n istio-system | grep cm-acme
$ k get svc -n istio-system | grep cm-acme
$ k get ingresses.networking.k8s.io -n istio-system
$
c. Virtualservice 생성
- HTTP ACME Challenge를 위하여 ingress 구성과 Let’s Encrypt이 Domain 검증 요청에 대한 응답 처리를 하는 POD이 자동 생성되며, 검증이 완료되면 삭제 됨
Kubeflow 1.0.2 환경에서는 ingress 대신 Istio에서 사용되기 때문에 수동으로 VirtualService를 생성 해 주어야 함
$ k describe ingresses.networking.k8s.io cm-acme-http-solver-s2hcd -n istio-system | grep Rules -A10
Rules:
Host Path Backends
---- ---- --------
34.75.224.5.sslip.io
/.well-known/acme-challenge/SiUOnSN4E4cR3uvjYz52v3YfCRDGJ9NKuS67rRXlEAg cm-acme-http-solver-vqxbl:8089 (10.43.0.21:8089)
…
$ vi virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: acme-challenge
namespace: kubeflow
spec:
gateways:
- kubeflow-gateway
hosts:
- '*'
http:
- match:
- uri:
exact: /.well-known/acme-challenge/SiUOnSN4E4cR3uvjYz52v3YfCRDGJ9NKuS67rRXlEAg
route:
- destination:
host: cm-acme-http-solver-vqxbl.istio-system.svc.cluster.local # Service name
$ k apply -f virtualservice.yaml
- Istio-ingress의 Service type이 NodePort 인 경우 (MeltalLB 설정 이전)
$ sudo kubectl port-forward pod/cm-acme-http-solver-m2sdv 80:8089 -n istio-system --address 0.0.0.0 &- Istio-ingress의 Service type이 LoadBalancer 인 경우 (아래 Troubleshooting 참조)
$ kubectl delete envoyfilters.networking.istio.io authn-filter -n istio-system
sleep ??? : HTTP Challenge 수행 및 Certificate 발급될 때 까지
$ kubectl apply -f authn-filter.yaml
$ curl http://34.75.224.5.sslip.io/.well-known/acme-challenge/e1R7Vom4Dx4DkzO1uUlKjdN6d5NZ6a_37jPVfwpuO_Q -vk
* About to connect() to 34.75.224.5.sslip.io port 80 (#0)
* Trying 34.75.224.5...
* Connected to 34.75.224.5.sslip.io (34.75.224.5) port 80 (#0)
> GET /.well-known/acme-challenge/e1R7Vom4Dx4DkzO1uUlKjdN6d5NZ6a_37jPVfwpuO_Q HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 34.75.224.5.sslip.io
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: no-cache, no-store, must-revalidate
< date: Tue, 07 Jul 2020 05:34:14 GMT
< content-length: 87
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 0
< server: istio-envoy
<
* Connection #0 to host 34.75.224.5.sslip.io left intact
e1R7Vom4Dx4DkzO1uUlKjdN6d5NZ6a_37jPVfwpuO_Q.tWPCvTv4o6m7WcBuVTGiyLDxT9zjo2fOiXOJg8JBwGM
d. 인증서 발급 확인
$ k get secrets kubeflow-cert -n istio-system
NAME TYPE DATA AGE
kubeflow-cert kubernetes.io/tls 3 5d11h
[ysjeon71_kubeflow3@master solver-http]$ k describe secrets kubeflow-cert -n istio-system
Name: kubeflow-cert
Namespace: istio-system
…
Type: kubernetes.io/tls
Data
====
tls.crt: 3566 bytes
tls.key: 1679 bytes
ca.crt: 0 bytes
$ k describe certificates.cert-manager.io kubeflow-cert -n istio-system | grep Spec -A50
Spec:
Common Name: 34.75.224.5.sslip.io
Dns Names:
34.75.224.5.sslip.io
Issuer Ref:
Kind: ClusterIssuer
Name: kubeflow-letsencrypt-issuer
Secret Name: kubeflow-cert
Status:
Conditions:
Last Transition Time: 2020-07-07T13:34:53Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2020-10-05T12:34:52Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issued 4m24s cert-manager Certificate issued successfully
$
$ k describe certificaterequests.cert-manager.io kubeflow-cert-3909740735 -n istio-system | grep Status -A50
…
$ k describe challenges.acme.cert-manager.io -n istio-system | grep Status -A50
…
$ k describe orders.acme.cert-manager.io kubeflow-cert-3909740735-2606464957 -n istio-system | grepStatus -A50
…
$
e. 인증서 적용 및 확인
$ k edit gateways.networking.istio.io kubeflow-gateway -n kubeflow
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: kubeflow-gateway
namespace: kubeflow
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: false
- hosts:
- 34.75.224.5.sslip.io # This should match a DNS name in the Certificate
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: kubeflow-cert # This should match the Certifcate secretName
$Chrome에서 https://34.75.224.5.sslip.io 접속
3.3 ACME with HTTP - Trouble shooting (Redirect)
- Problem: istio로 요청시 무조건 “/dex/“ URI로 리다이텍트 되 http challenge를 수행할 수 없음
Environments: kubeflow 1.0.2 (istio 1.3)에서 발생, kubeflow 1.0.2 + istio 1.5.8에서는 미 발생
$ curl http://35.196.36.171.sslip.io:31380/.well-known/acme-challenge/6sUQoKwWi4GYW80CTMkURV1yNeV4MY2MMBwXiTF7qfM -vk
* About to connect() to 35.196.36.171.sslip.io port 31380 (#0)
* Trying 35.196.36.171...
* Connected to 35.196.36.171.sslip.io (35.196.36.171) port 31380 (#0)
> GET /.well-known/acme-challenge/6sUQoKwWi4GYW80CTMkURV1yNeV4MY2MMBwXiTF7qfM HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 35.196.36.171.sslip.io:31380
> Accept: */*
>
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTU5NDA5NjY2NHxFd3dBRURka1RFZEhkbFIzVDFoUFIwSllhR2s9fLhnBPTSfNf1Gw0CMcgaRXSG3Ida0QBVY-yHTG5P1jI0
< date: Tue, 07 Jul 2020 04:37:44 GMT
< content-length: 269
< x-envoy-upstream-service-time: 41
< server: istio-envoy
<
<a href="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTU5NDA5NjY2NHxFd3dBRURka1RFZEhkbFIzVDFoUFIwSllhR2s9fLhnBPTSfNf1Gw0CMcgaRXSG3Ida0QBVY-yHTG5P1jI0">Found</a>.
* Connection #0 to host 35.196.36.171.sslip.io left intact
$- Cause: Envoy filter(authn-filter) 설정 때문에 “/dex/auth”로 redirect 됨
$ k logs istio-ingressgateway-cc4bc5bbb-bstbm -n istio-system -f
[2020-07-07T04:39:23.817Z] "GET /.well-known/acme-challenge/6sUQoKwWi4GYW80CTMkURV1yNeV4MY2MMBwXiTF7qfM HTTP/1.1" 302 UAEX "-" "-" 0 269 13 13 "10.34.0.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" "9f29570e-b5a7-4cff-bbd4-29fce12dda29" "35.196.36.171.sslip.io:31380" "-" - - 10.32.0.36:80 10.34.0.0:50245 - -
[2020-07-07T04:39:24.164Z] "GET /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTU5NDA5Njc2M3xFd3dBRUVkelNuSlBVVWMzUVVKS00ycE5WWG89fIsqpvUT9NcmRcyDP5Boyg2rClX6qOHJl-HWU44Q4HMv HTTP/1.1" 200 - "-" "-" 0 1443 9 6 "10.34.0.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" "cf5149cd-fbb9-4fad-aabf-31167850d5a0" "35.196.36.171.sslip.io:31380" "10.38.0.19:5556" outbound|5556||dex.auth.svc.cluster.local - 10.32.0.36:80 10.34.0.0:50245 - -
$ k get envoyfilters.networking.istio.io -n istio-system
NAME AGE
authn-filter 3m39s
$https://www.kubeflow.org/docs/other-guides/istio-in-kubeflow/
- Solution: 인증서를 발급받는 과정에서 임시로 authn-filter를 삭제하고, 발급이 완료되면 재 생성 (임시)
$ k get envoyfilters.networking.istio.io authn-filter -n istio-system -o yaml > authn-filter.yaml
$ k delete -f authn-filter.yaml
envoyfilter.networking.istio.io "authn-filter” deleted
$
…
$ k apply -f authn-filter.yaml
envoyfilter.networking.istio.io/authn-filter created
$
'Kubernetes > Management' 카테고리의 다른 글
| Istio - Virtual service config (0) | 2021.09.23 |
|---|---|
| Istio 1.5 구성 (0) | 2021.09.23 |
| ClusterIP, NodePort, Ingress 개념 (0) | 2021.09.23 |
| K8s 잡학다식 (0) | 2021.09.23 |
| Cert-manager with LetsEncrypt (DNS challenge) (1) | 2021.09.23 |
댓글