Knative - SKS Mode (Proxy, Serve) 이해

1. 개요

- Knative Service request flow 

https://help-static-aliyun-doc.aliyuncs.com/assets/img/en-US/0032790261/p172886.png

   

   ✓ istio(Route)에서 Knative serving service를 요청하는 경우 

       a. 'scale to zero' 상태로 Activator가 호출된다. 이 모드가 Proxy이다. Activator는 Revision을 활성화(Pod 기동), 요청을 전달하고 SKS 모드를 Proxy에서 Serve로 변경한다. 

       b. Revision이 활성화 된 경우로 Pod가 호출된다. 이 모드가 Serve이다.

   ✓ Activator 역할과 Request flow에 대한 상세한 설명은 Knative 이해 (3. Knative 이해)를 참조하기 바란다. 

 

- SKS(Serverless Knative Service)는 knative serving 리소스 중 일부이며, Knative service의 모드(Proxy, Serve) 정보를 제공한다.

   SKS는 Knative serving service를 생성할 때 자동 생성된다.

$ k api-resources | egrep -w 'NAME|sks'
NAME                 SHORTNAMES   APIVERSION                                  NAMESPACED   KIND
serverlessservices   sks          networking.internal.knative.dev/v1alpha1    true         ServerlessService
$

$ k get sks -n yoosung-jeon -l serving.knative.dev/service=autoscale-go
NAME                 MODE    ACTIVATORS   SERVICENAME          PRIVATESERVICENAME           READY     REASON
autoscale-go-n5c78   Proxy   3            autoscale-go-n5c78   autoscale-go-n5c78-private   True
autoscale-go-n9hm2   Proxy                autoscale-go-n9hm2   autoscale-go-n9hm2-private   Unknown   NoHealthyBackends
$

 

 

2. 테스트 환경

- knative v0.14.3, istio 1.3, Kubernetes 1.16.15

- Kubeflow 1.2에 포함된 knative, istios를 사용함

 

 

3. Why Proxy mode?

- 이슈

   'scale to zero'가 아니면서 Revision(Pod)이 활성화된 상태인데, SKS 모드가 Serve가 아닌 Proxy이며 요청이 항상 Activator로 전달된다.

 

- Knative service 상태

   autoscaling.knative.dev/minScale은 1로 설정되어 있고, 1개의 Pod가 기동 중이다. (0인 경우가 scale to zero)

$ k get ksvc -n yoosung-jeon autoscale-go -o yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: autoscale-go
  namespace: yoosung-jeon
  ...
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/metric: concurrency
        autoscaling.knative.dev/minScale: "1"
        autoscaling.knative.dev/target: "10"
...
$ k get ksvc -n yoosung-jeon autoscale-go
NAME                               URL                                                                         LATESTCREATED                            LATESTREADY                              READY   REASON
autoscale-go                       http://autoscale-go.yoosung-jeon.kf-serv.acp.kt.co.kr                       autoscale-go-g7g2b                       autoscale-go-g7g2b                       True
$ k get pod -n yoosung-jeon -l app=autoscale-go-g7g2b
NAME                                             READY   STATUS    RESTARTS   AGE
autoscale-go-g7g2b-deployment-65975796fd-tdhct   2/2     Running   0          4m6s
$

 

- SKS Mode 및 Service endpoint 조회

   autoscale-go-g7g2b revision의 SKS Mode는 Proxy이다.

   autoscale-go-g7g2b 서비스에 연결된 endpoint는 Activator(activator-55f9fdc55d-k64tg)이다.

$ k get sks -n yoosung-jeon -l serving.knative.dev/service=autoscale-go
NAME                 MODE    ACTIVATORS   SERVICENAME          PRIVATESERVICENAME           READY     REASON
autoscale-go-g7g2b   Proxy   3            autoscale-go-g7g2b   autoscale-go-g7g2b-private   True
autoscale-go-n5c78   Proxy   2            autoscale-go-n5c78   autoscale-go-n5c78-private   Unknown   NoHealthyBackends
$ k get svc -n yoosung-jeon | egrep 'NAME|autoscale-go-g7g2b'
NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP                                            PORT(S)                             AGE
autoscale-go-g7g2b                               ClusterIP      10.101.66.37     <none>                                                 80/TCP                              9m20s
autoscale-go-g7g2b-private                       ClusterIP      10.103.13.182    <none>                                                 80/TCP,9090/TCP,9091/TCP,8022/TCP   9m20s
$ k describe svc autoscale-go-g7g2b -n yoosung-jeon | grep Endpoints
Endpoints:         10.244.2.64:8012
$ k get pod -A -o wide | egrep 'NAME|10.244.2.64'
NAMESPACE                NAME                                                              READY   STATUS                  RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
knative-serving          activator-55f9fdc55d-k64tg                                        1/1     Running                 2          18d     10.244.2.64     iap12    <none>           <none>
$

 

 

4. 원인 파악

- autoscaler 로그 파악

   'SKS should be in proxy mode' 로그와 함께 ebc 값이 -190이다. 

 

$ k logs autoscaler-7d57f954bc-wlzwn -n knative-serving -f | grep autoscale-go | egrep 'SKS|PodCount'
...
{"level":"debug","ts":"2021-10-13T04:11:52.940Z","logger":"autoscaler","caller":"scaling/autoscaler.go:260","msg":"PodCount=1 Total1PodCapacity=10 ObsStableValue=0 ObsPanicValue=0 TargetBC=200 ExcessBC=-190 NumActivators=3","commit":"bcda051","knative.dev/key":"yoosung-jeon/autoscale-go-g7g2b","metric":"concurrency"}
{"level":"info","ts":"2021-10-13T04:11:52.967Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:126","msg":"SKS should be in proxy mode: want = 1, ebc = -190, PA Inactive? = false","commit":"bcda051","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/traceid":"a6e9b1e2-8bc0-4ada-9ff4-9863abb063c0","knative.dev/key":"yoosung-jeon/autoscale-go-g7g2b"}

 

- 원인

   ✓ SKS is in proxy mode due to your revision not having enough pods to have positive effective bust capacity (ebc = -190)

   ✓ EBC = TotCapacity - Cur#ReqInFlight - TargetBurstCapacity 

       Compute the excess burst capacity based on stable value for now, since we don't want to be making knee-jerk decisions about Activator in the request path.

       Negative EBC means that the deployment does not have enough capacity to serve the desired burst off hand. 

 

 

5. 조치 사항

- Knative service의 Pod 수를 충분히 증가시킨다.

   autoscaling.knative.dev/minScale의 값을 1에서 20으로 변경 후 저장한다.

   Knative service가 수정되었기 때문에 신규로 revision(autoscale-go-6q85r)이 생성되었으며, SKS mode는 Serve로 설정되었다.

$ k edit ksvc -n yoosung-jeon autoscale-go
...
      annotations:
        autoscaling.knative.dev/metric: concurrency
        autoscaling.knative.dev/minScale: "20"
        autoscaling.knative.dev/target: "10"
...
$ k get ksvc -n yoosung-jeon autoscale-go
NAME           URL                                                     LATESTCREATED        LATESTREADY          READY   REASON
autoscale-go   http://autoscale-go.yoosung-jeon.kf-serv.acp.kt.co.kr   autoscale-go-6q85r   autoscale-go-6q85r   True
$ k get sks -n yoosung-jeon -l serving.knative.dev/service=autoscale-go
NAME                 MODE    ACTIVATORS   SERVICENAME          PRIVATESERVICENAME           READY     REASON
autoscale-go-6q85r   Serve   4            autoscale-go-6q85r   autoscale-go-6q85r-private   True
autoscale-go-g7g2b   Proxy   2            autoscale-go-g7g2b   autoscale-go-g7g2b-private   Unknown   NoHealthyBackends
autoscale-go-n5c78   Proxy   2            autoscale-go-n5c78   autoscale-go-n5c78-private   Unknown   NoHealthyBackends
$

 

- autoscaler 로그

  ebc(ExcessBC) 값은 0으로 변경되었다.

$ k logs autoscaler-7d57f954bc-wlzwn -n knative-serving -f | grep autoscale-go | egrep 'SKS|PodCount'
...
{"level":"debug","ts":"2021-10-13T04:46:16.041Z","logger":"autoscaler","caller":"scaling/autoscaler.go:260","msg":"PodCount=20 Total1PodCapacity=10 ObsStableValue=0 ObsPanicValue=0 TargetBC=200 ExcessBC=0 NumActivators=4","commit":"bcda051","knative.dev/key":"yoosung-jeon/autoscale-go-6q85r","metric":"concurrency"}
{"level":"info","ts":"2021-10-13T04:46:16.041Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"autoscaling/reconciler.go:69","msg":"SKS autoscale-go-6q85r changed; reconciling, want mode: Serve","commit":"bcda051","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/traceid":"42f742da-fb5f-4fda-81f7-22af7b6c4352","knative.dev/key":"yoosung-jeon/autoscale-go-6q85r"}

 

- autoscale-go-6q85r 서비스는 Activator가 아닌 Revision(Pod)에 연결된 것을 확인 할 수 있다.

$ k describe svc autoscale-go-6q85r -n yoosung-jeon | grep Endpoints
Endpoints:         10.244.2.11:8012,10.244.2.12:8012,10.244.2.13:8012 + 17 more...
$ k get pod -A -o wide | egrep -w 'NAME|10.244.2.11'
NAMESPACE      NAME                                             READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
yoosung-jeon   autoscale-go-6q85r-deployment-7c48c8db98-9f65f   2/2     Running   0          15m   10.244.2.11   iap12    <none>           <none>
$

 

- 요청량에 따라 ebs 값은 변동되며, 음수인 경우 Proxy 모드로 변경된다.

$ k get sks autoscale-go-6q85r -n yoosung-jeon -f
NAME                  MODE    ACTIVATORS   SERVICENAME          PRIVATESERVICENAME           READY     REASON
autoscale-go-6q85r    Serve   4            autoscale-go-6q85r   autoscale-go-6q85r-private   True
autoscale-go-6q85r    Proxy   4            autoscale-go-6q85r   autoscale-go-6q85r-private   True
autoscale-go-6q85r    Proxy   4            autoscale-go-6q85r   autoscale-go-6q85r-private   True
autoscale-go-6q85r    Serve   4            autoscale-go-6q85r   autoscale-go-6q85r-private   True
autoscale-go-6q85r    Serve   4            autoscale-go-6q85r   autoscale-go-6q85r-private   True
...