istio - Access logs 설정

1. 개요

- istio ingress gateway의 access log를 stdout으로 출력하도록 설정

 

 

2. Environments

- istio 1.3

- Kubernetes 1.16.15

 

 

3. 설정

- istio configmap의 accessLogFile 항목을 "/dev/stdout"으로 지정하고, istio ingressgateway를 재 기동 한다.

$ k edit configmap istio -n istio-system
apiVersion: v1
data:
  mesh: |-
..
    # Set accessLogFile to empty string to disable access log.
    accessLogFile: "/dev/stdout"

    # Set accessLogEncoding to JSON or TEXT to configure sidecar access log
    accessLogEncoding: 'TEXT'
...
$ k rollout restart deployment istio-ingressgateway -n istio-system
$

 

- Access log 출력하기

$ k logs istio-ingressgateway-79f4b68899-d2w75 -n istio-system -f
2021-10-08T05:55:07.543948Z	info	FLAG: --applicationPorts="[]"
...
2021-10-08T05:55:16.663128Z	info	Envoy proxy is ready

[2021-10-08T05:55:50.464Z] "GET /?sleep=5000&prime=10000&bloat=5 HTTP/1.1" 200 - "-" "-" 0 101 5013 5012 "10.244.2.1" "-" "f6fe8a44-cfee-41cc-9d16-e41caf0d2d9f" "autoscale-go.yoosung-jeon.kf-serv.acp.kt.co.kr" "10.244.2.64:8012" outbound|80||autoscale-go-hnfpq.yoosung-jeon.svc.cluster.local - 10.244.2.203:80 10.244.2.1:17126 - -
[2021-10-08T05:55:50.462Z] "GET /?sleep=5000&prime=10000&bloat=5 HTTP/1.1" 200 - "-" "-" 0 101 5016 5014 "10.244.2.1" "-" "27112e0a-3582-4047-8cc7-3c526644262c" "autoscale-go.yoosung-jeon.kf-serv.acp.kt.co.kr" "10.244.2.64:8012" outbound|80||autoscale-go-hnfpq.yoosung-jeon.svc.cluster.local - 10.244.2.203:80 10.244.2.1:30895 - -
[2021-10-08T05:57:30.210Z] "GET / HTTP/1.1" 302 UAEX "-" "-" 0 269 3 2 "10.244.2.1" "-" "468d0965-11d1-41c0-ba30-dc159156f91a" "autoscale-go-new.yoosung-jeon.kf-serv.acp.kt.co.kr" "-" - - 10.244.2.203:80 10.244.2.1:23741 - -
[2021-10-08T06:56:48.825Z] "GET / HTTP/1.1" 302 UAEX "-" "-" 0 269 7 6 "10.244.2.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "f6781d5e-3148-4f0b-9e6f-a3335f944c44" "kf.acp.kt.co.kr" "-" - - 10.244.2.206:80 10.244.2.1:7917 - -
[2021-10-08T06:56:48.845Z] "GET /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTYzMzY3NjIwOHxFd3dBRUZoelVubDFNblpWV0hWRk16WnFhMU09fO6u_GDZyCn_EZJWRLXHZ0kqAqjJhy0sa5GrYrJWMla9 HTTP/1.1" 302 - "-" "-" 0 68 8 7 "10.244.2.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "4837f9f7-28ba-40ef-8724-448e93045253" "kf.acp.kt.co.kr" "10.244.4.137:5556" outbound|5556||dex.auth.svc.cluster.local - 10.244.2.206:80 10.244.2.1:7917 - -
[2021-10-08T06:56:48.861Z] "GET /dex/auth/local?req=go3vantq3fjtsrecbbzqcyut6 HTTP/1.1" 200 - "-" "-" 0 1497 13 11 "10.244.2.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "e036df62-a1dd-4f0a-8e88-f1ce096d1e5e" "kf.acp.kt.co.kr" "10.244.4.137:5556" outbound|5556||dex.auth.svc.cluster.local - 10.244.2.206:80 10.244.2.1:7917 - -

 

- Access log format 

Log operator                                                    access log
--------------------------------------------------------------  ---------------------------------------------------------------
[%START_TIME%]                                                  [2021-10-08T05:55:50.462Z] 
"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"  "GET /?sleep=5000&prime=10000&bloat=5 HTTP/1.1" 
%RESPONSE_CODE%                                                 200
%RESPONSE_FLAGS%                                                - 
?                                                               "-" 
?                                                               "-" 
%BYTES_RECEIVED%                                                0 
%BYTES_SENT%                                                    101 
%DURATION%                                                      5016 
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%                           5014 
%REQ(X-FORWARDED-FOR)%"                                         "10.244.2.1" 
"%REQ(USER-AGENT)%"                                             "-" 
"%REQ(X-REQUEST-ID)%"                                           "27112e0a-3582-4047-8cc7-3c526644262c" 
"%REQ(:AUTHORITY)%"                                             "autoscale-go.yoosung-jeon.kf-serv.acp.kt.co.kr" 
"%UPSTREAM_HOST%"                                               "10.244.2.64:8012" 
"%UPSTREAM_CLUSTER%"                                            outbound|80||autoscale-go-hnfpq.yoosung-jeon.svc.cluster.local 
?                                                               - 
%DOWNSTREAM_LOCAL_ADDRESS%                                      10.244.2.203:80 
%DOWNSTREAM_REMOTE_ADDRESS%                                     10.244.2.1:30895 
?                                                               - 
?                                                               -

   ✓ 공식 문서에서 누락된 access log 항목이 존재했으며, accessLogEncoding을 JSON으로 설정해서 일부 항목들을 추가로 파악하였다.

       Default Format String: https://www.envoyproxy.io/docs/envoy/v1.11.1/configuration/access_log

                                             https://istio.io/latest/docs/tasks/observability/logs/access-log/

   ✓ 위 Log operator 중에서 "?"로 표시된 항목들은 아래 값 중 하나이며, 정확하게 판단할 수 없어 "?"로 표시하였다.

       ISTIO_POLICY_STATUS 
       REQUESTED_SERVER_NAME 
       ROUTE_NAME
       UPSTREAM_LOCAL_ADDRESS
       UPSTREAM_TRANSPORT_FAILURE_REASON 

   ✓ UPSTREAM, DOWNSTREAM ?

       Downstream: A downstream host connects to Envoy, sends requests, and receives responses.
       Upstream: An upstream host receives connections and requests from Envoy and returns responses.

       위 로그에서 Upstream_host는 knative-serving의 activator-55f9fdc55d-k64tg pod에 해당한다.

https://rinormaloku.com/wp-content/uploads/2019/02/Envoy-Edge-Proxy-900x481.png

   ✓ UPSTREAM_CLUSTER

       A cluster is a group of logically similar upstream hosts that Envoy connects to. Envoy discovers the members of a cluster via service discovery. It optionally determines the health of cluster members via active health checking. The cluster member that Envoy routes a request to is determined by the load balancing policy.