MetalLB

2021.03.16

 

1. MetalLB 0.9.5 ?
- MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.
- https://metallb.universe.tf/

 

 

2. MetalLB 배포 & Kubeflow (istio) 설정 변경
- https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#expose-with-a-loadbalancer
a. Install MetalLB

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

 

b. Config MetalLB

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 14.52.244.137-14.52.244.138   # IP Pool range
EOF
$

        Layer 2 mode does not require the IPs to be bound to the network interfaces of your worker nodes.
        It works by responding to ARP requests on your local network directly, to give the machine’s MAC address to clients.


c. To expose Kubeflow with a LoadBalancer Service
- Service type을 'LoadBalancer'로 지정하면 IP 주소가 할당되어 직접 접근이 가능한다.

   istio-ingressgateway service의 External-IP 항목에 할당된 IP 주소(14.52.244.137)가 출력된다. 

$ kubectl get svc -n istio-system istio-ingressgateway
NAME                  TYPE      CLUSTER-IP      EXTERNAL-IP  PORT(S)                         AGE
istio-ingressgateway  NodePort  10.100.224.181  <none>       15020:30707/TCP,80:31380/TCP,…  4d16h
$ kubectl patch service -n istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'
$ kubectl get svc -n istio-system istio-ingressgateway
NAME                  TYPE          CLUSTER-IP      EXTERNAL-IP    PORT(S)                         AGE
istio-ingressgateway  LoadBalancer  10.100.224.181  14.52.244.137  15020:30707/TCP,80:31380/TCP,…  4d20h
$ kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0]}’
14.52.244.137
$

 

- arp 명령어로 할당된 IP 주소(14.52.244.137)의 MAC 주소('34:48:ed:f7:18:78') 조회할 수 있다.

   MAC 주소('34:48:ed:f7:18:78')는 iap11서버의 em1 인터페이스의 MAC주소이다.

   iap11서버에는  istio-ingressgateway service에 연결된 Pod(istio-ingressgateway-767b85d746-qcs2b)가 실행 중이다.

[iap@iap01 ~]$ k get pod -n metallb-system -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
controller-6b8d7594db-hpswj   1/1     Running   0          15d   10.244.14.137   iap12   <none>           <none>
speaker-jvjzq                 1/1     Running   0          15d   14.52.244.213   iap10   <none>           <none>
speaker-mqxmp                 1/1     Running   0          15d   14.52.244.214   iap11   <none>           <none>
speaker-hqpfs                 1/1     Running   0          15d   14.52.244.216   iap12   <none>           <none>
speaker-krpjf                 1/1     Running   0          15d   14.52.244.217   iap13   <none>           <none>
[iap@iap01 ~]$ k logs speaker-mqxmp -n metallb-system | grep "senderMAC" | tail -n 1
{"caller":"arp.go:102","interface":"em1","ip":"14.52.244.137","msg":"got ARP request for service IP,
 sending response","responseMAC":"34:48:ed:f7:18:78","senderIP":"14.52.244.3","senderMAC":"00:e0:4c:99:28:36",
 "ts":"2021-03-30T23:04:09.265724976Z"}
[iap@iap01 ~]$
[iap@iap01 ~]$ k logs speaker-krpjf -n metallb-system | grep "senderMAC" | tail -n 1
{"caller":"arp.go:102","interface":"em1","ip":"14.52.244.139","msg":"got ARP request for service IP,
 sending response","responseMAC":"34:48:ed:f9:07:9c","senderIP":"14.52.244.216","senderMAC":"34:48:ed:f9:09:3c",
 "ts":"2021-03-31T02:02:04.71020998Z"}
[iap@iap01 ~]$
[iap@iap01 ~]$ arp -a | egrep "14.52.244.13[7-9]"
iap07 (14.52.244.139) at 34:48:ed:f9:07:9c [ether] on enp2s0
? (14.52.244.138) at 34:48:ed:f9:09:3c [ether] on enp2s0
? (14.52.244.137) at 34:48:ed:f7:18:78 [ether] on enp2s0
[iap@iap01 ~]$

[root@iap11 ~]# ip addr sh | grep "em1:" -A4
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 34:48:ed:f7:18:78 brd ff:ff:ff:ff:ff:ff
    inet 14.52.244.214/24 brd 14.52.244.255 scope global noprefixroute em1
       valid_lft forever preferred_lft forever
    inet6 fe80::3648:edff:fef7:1878/64 scope link
[root@iap11 ~]#

[iap@iap01 ~]$ k get pod istio-ingressgateway-767b85d746-qcs2b -n istio-system -o wide
NAME                                    READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
istio-ingressgateway-767b85d746-qcs2b   1/1     Running   1          86d   10.244.9.56   iap11   <none>           <none>
[iap@iap01 ~]$

 

 

3. MetalLB Test (Ensure that MetalLB works as expected)

$ kubectl create service loadbalancer nginx --tcp=80:80
$ kubectl describe service nginx | grep Events -A10
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
  Normal  IPAllocated  2m9s  metallb-controller  Assigned IP "14.52.244.137"
$ k get svc nginx
NAME    TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)        AGE
nginx   LoadBalancer   10.97.76.95   14.52.244.137   80:31327/TCP   7s
$ kubectl run nginx --image nginx --restart=Never -l app=nginx
$ kubectl describe service nginx | grep Events -A10
Events:
  Type    Reason        Age    From                Message
  ----    ------        ----   ----                -------
  Normal  IPAllocated   4m21s  metallb-controller  Assigned IP "14.52.244.137"
  Normal  nodeAssigned  23s    metallb-speaker     announcing from node "iap07"
$
$ arping -I enp2s0 14.52.244.137
ARPING 14.52.244.137 from 14.52.244.134 enp2s0
Unicast reply from 14.52.244.137 [E4:11:5B:12:6C:94]  2.292ms
Unicast reply from 14.52.244.137 [E4:11:5B:12:6C:94]  0.992ms
Unicast reply from 14.52.244.137 [E4:11:5B:12:6C:94]  0.905ms
…
$ arp -a | grep -i E4:11:5B:12:6C:94
iap07 (14.52.244.139) at e4:11:5b:12:6c:94 [ether] on enp2s0
$ curl http://14.52.244.137/
<!DOCTYPE html>
…
$ kubectl delete service nginx
$ kubectl delete pod nginx