Istio - Virtual service config

2020.04.27

 

1. 참고 사이트

- Istio Traffic management (https://bcho.tistory.com/1367)

- https://istio.io/docs/ops/best-practices/traffic-management/

- https://istio.io/docs/ops/best-practices/traffic-management/#split-virtual-services

 

 

2. 등록 고려 대상

- 대상

   ✓ kubernetes-dashboard

   ✓ weave scope

   ✓ istio tool : kiali, jaeger, prometheus, grafana

 

- 관련 서비스

$ k get svc grafana jaeger-query kiali prometheus -n istio-system
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
grafana        ClusterIP   10.100.208.29    <none>        3000/TCP    17h
jaeger-query   ClusterIP   10.100.113.191   <none>        16686/TCP   17h
kiali          ClusterIP   10.107.43.9      <none>        20001/TCP   17h
prometheus     ClusterIP   10.97.165.192    <none>        9090/TCP    17h
$ k get svc kubernetes-dashboard -n kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.101.50.211   <none>        443:32333/TCP   39h
$ k get svc -n weave
NAME              TYPE        CLUSTER-IP      EXTERNaAL-IP   PORT(S)   AGE
weave-scope-app   ClusterIP   10.107.238.54   <none>        80/TCP    5d22h
$

 

 

3. Istio 설정

- 설정 참고 사항

   The hosts field

         The virtual service hostname can be an IP address, a DNS name, or, depending on the platform, a short name (such as a Kubernetes service short name) that resolves, implicitly or explicitly, 

         to a fully qualified domain name (FQDN).

         You can also use wildcard (”*”) prefixes, letting you create a single set of routing rules for all matching services.

 

   - Virtual service Config 작성 및 적용

$ vi vs-admin-tool.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: admin-tool
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        prefix: /kiali
    route:
    - destination:
        host: kiali.istio-system.svc.cluster.local
  - match:
    - uri:
        prefix: /jaeger
    route:
    - destination:
        host: jaeger-query.istio-system.svc.cluster.local
$ kubectl apply -f vs-admin-tool.yaml

   

- Virtual service 호출 주소 조회

   Service type이 NodePort이므로 NodePort용 http와 https 포트를 조회 함

   On-prem 환경에서 Service type을 LoadBalancer로 설정할 경우 추가적으로 MetalLB 설정이 필요 함

$ kubectl -n istio-system get service istio-ingressgateway
NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                         AGE
istio-ingressgateway   NodePort   10.106.99.75   <none>        15020:30546/TCP,80:32549/TCP,443:30773/TCP,..   19h
$ export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}') && echo $INGRESS_PORT
32549
$ export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}') && echo $SECURE_INGRESS_PORT
30773
$

ysjeon71_kubeflow3@cs-6000-devshell-vm-5ca63e10-d56a-47dd-88e2-f04ee058197c:~$ gcloud compute instances list
NAME      ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
master    us-east1-b  n1-standard-2               10.142.0.2   35.229.68.219  RUNNING
worker-1  us-east1-c  n1-standard-1               10.142.0.3   34.74.59.60    RUNNING
worker-2  us-east1-d  n1-standard-1               10.142.0.4   34.73.30.174   RUNNING
ysjeon71_kubeflow3@cs-6000-devshell-vm-5ca63e10-d56a-47dd-88e2-f04ee058197c:~$

   - 요청 URL

     http://35.229.68.219:32549/kiali

     http://35.229.68.219:32549/jaeger

     http://35.229.68.219:32549/kubernetes-dashboard

 

 

4. Trouble-shooting

a. 404 page not found

- Problem

   ingressgateway를 통해서 prometheus, grafana, weave-scope를 Virtual service에 등록 후 요청시 “404 page not found”에러가 발생됨

$ cat vs-admin-tool.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: admin-tool
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        prefix: /prometheus
    route:
    - destination:
        host: prometheus.istio-system.svc.cluster.local
  - match:
    - uri:
        prefix: /istio-grafana
    route:
    - destination:
        host: grafana.istio-system.svc.cluster.local
  - match:
    - uri:
        prefix: /weave-scope
    rewrite:
      uri: /
    route:
    - destination:
        host: weave-scope-app.weave.svc.cluster.local
$

- Cause

   요청한 POD내의 어플리케이션의 Base url이 “/“이기 때문에 발생

- Solution

   Virtual services에서 각각 DNS name으로 분리하고 매칭할 URI를 “/“로 선언하거나  어플리케이션에서 base_url 설정을 지원할 경우 변경

 

 

 b. 404 (Not found)

- Problem

   Virtual Service에서 DNS Name, IP address 등으로 라우팅 설정하였으나 404 (Not found) 에러가 발생

$ cat virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
…
spec:
  hosts:
  - "admin.iap.kt.com"
…
$

 

- Casue

   ingress gateway가 NodePort 방식이면 HTTP(80) 또는 HTTPS(443)이 아닌 임의의 포트(INGRESS_PORT)로 요청을 해야 하며,

   클라이언트 Request Header의 Host는 “admin.iap.kt.com:INGRESS_PORT”로 설정되어 Virtual service의 설정된 hosts 값과 일치되지 않아 발생 됨

- Solution

   You just need to make sure your application is available on the default HTTP (80) and/or HTTPS (443) ports. The easiest way for this is to use a load balancer or proxy in front of your cluster

   MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.

   https://medium.com/01001101/how-to-bind-hostnames-in-gateway-virtualservice-istio-resources-with-nodeport-exposed-istio-ingress-59b5650967c7