2021.03.31, 2020.06.10
1. 개요
Kubernetes master를 HA 구성시 L4 스위치가 없을 경우 Software load balancing로 설정하는 방법
2. 환경
- keepalived 2.0.20, HAProxy 2.0.14
The keepalived service provides a virtual IP managed by a configurable health check.
The haproxy service can be configured for simple stream-based load balancing thus allowing TLS termination
- 구성 내역
Server IP : iap1(14.52.244.134:6443), iap2(14.52.244.135:6443), iap3(14.52.244.207:6443),
VIP (14.52.244.136:7443)
3. 구성
https://keepalived.readthedocs.io/
3.1 keepalived
- Install
# yum install -y keepalived
or
[root@iap01 ~]# yum -y install gcc openssl-devel libnl3-devel
...
[root@iap01 ~]# wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
[root@iap01 ~]# tar xzf keepalived-2.0.20.tar.gz
[root@iap01 ~]# cd keepalived-2.0.20
[root@iap01 keepalived-2.0.20]# ./configure
...
[root@iap01 keepalived-2.0.20]# make && make install
...
[root@iap01 keepalived-2.0.20]# /usr/local/sbin/keepalived -v
Keepalived v2.0.20 (01/22,2020)
...
[root@iap01 keepalived-2.0.20]#
- Configure (Node #1) - https://www.keepalived.org/manpage.html
[root@iap01 ~]# systemctl stop firewalld
[root@iap01 ~]# mkdir -p /etc/keepalived
[root@iap01 ~]# vi /etc/keepalived/keepalived.conf
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 3 # check every 3 seconds
fall 5 # require 5 failures for KO
rise 2 # require 2 successes for OK
weight -2
}
vrrp_instance VI_1 {
state MASTER
interface enp2s0
virtual_router_id 51
priority 101
authentication {
auth_type PASS
auth_pass 42
}
virtual_ipaddress {
14.52.244.136
}
track_script {
check_apiserver
}
}
[root@iap01 ~]#
## The above keepalived configuration uses a health check script /etc/keepalived/check_apiserver.sh responsible
## for making sure that on the node holding the virtual IP the API Server is available.
[root@iap01 ~]# vi /etc/keepalived/check_apiserver.sh
#!/bin/sh
errorExit() {
echo "*** $*" 1>&2
exit 1
}
curl --silent --max-time 2 --insecure https://localhost:7443/ -o /dev/null || errorExit "Error GET https://localhost:7443/"
if ip addr | grep -q {{ virtual_ip }}; then
curl --silent --max-time 2 --insecure https://{{ virtual_ip }}:7443/ -o /dev/null || errorExit "Error GET https://{{ virtual_ip }}:7443/"
fi
[root@iap01 ~]# chmod 755 /etc/keepalived/check_apiserver.sh
[root@iap01 ~]# systemctl enable keepalived --now
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@iap01 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-06-08 23:09:01 EDT; 9s ago
Process: 22157 ExecStart=/usr/local/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 22158 (keepalived)
CGroup: /system.slice/keepalived.service
├─22158 /usr/local/sbin/keepalived -D
└─22159 /usr/local/sbin/keepalived -D
Jun 08 23:09:10 iap01 Keepalived_vrrp[22159]: Sending gratuitous ARP on enp2s0 for 14.52.244.136
Jun 08 23:09:10 iap01 Keepalived_vrrp[22159]: (VI_1) Sending/queueing gratuitous ARPs on enp2s0 for 14.52.244.136
Jun 08 23:09:10 iap01 Keepalived_vrrp[22159]: Sending gratuitous ARP on enp2s0 for 14.52.244.136
[root@iap01 ~]# ip addr | grep -A 8 enp2s0
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:25:90:a0:1f:46 brd ff:ff:ff:ff:ff:ff
inet 14.52.244.134/8 brd 14.255.255.255 scope global noprefixroute enp2s0
valid_lft forever preferred_lft forever
inet 14.52.244.136/32 scope global enp2s0
valid_lft forever preferred_lft forever
inet6 fe80::dd86:5cfd:646f:f172/64 scope link noprefixroute
valid_lft forever preferred_lft forever
…
[root@iap01 ~]#
- Configure (Node #2)
Node #1과 keepalived.conf 설정 파일만 차이가 있고, 그외는 동일
[root@iap02 keepalived-2.0.20]# vi /etc/keepalived/keepalived.conf
…
vrrp_instance VI_1 {
state BACKUP
interface eno2
virtual_router_id 51
priority 100
authentication {
auth_type PASS
auth_pass 42
}
virtual_ipaddress {
14.52.244.136
}
track_script {
check_apiserver
}
}
[root@iap02 keepalived-2.0.20]# systemctl enable keepalived --now
…
CGroup: /system.slice/keepalived.service
├─7972 /usr/local/sbin/keepalived -D
└─7973 /usr/local/sbin/keepalived -D
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: Assigned address 14.52.244.135 for interface eno2
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: Assigned address fe80::3a18:18ff:7d54:da24 for interface eno2
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: Registering gratuitous ARP shared channel
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: (VI_1) removing VIPs.
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: (VI_1) Entering BACKUP STATE (init)
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: VRRP sockpool: [ifindex(3), family(IPv4), proto(112), unicast(0), fd(11,12)]
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: Script `check_apiserver` now returning 1
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: VRRP_Script(check_apiserver) failed (exited with status 1)
Jun 09 13:32:56 iap02 Keepalived_vrrp[7973]: (VI_1) Changing effective priority from 100 to 98
[root@iap02 keepalived-2.0.20]#
- Test
[root@iap03 ~]# ping 14.52.244.136
...
64 bytes from 14.52.244.136: icmp_seq=52 ttl=64 time=0.253 ms
From 14.52.244.134: icmp_seq=53 Redirect Host(New nexthop: 14.52.244.136)
From 14.52.244.134: icmp_seq=53 Redirect Host(New nexthop: 14.52.244.136)
64 bytes from 14.52.244.136: icmp_seq=54 ttl=64 time=0.259 ms
...
root@iap01 log]# reboot
[root@iap02 keepalived-2.0.20]# tail -f /var/log/messages | grep -i keepalived_vrrp
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: (VI_1) Backup received priority 0 advertisement
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: (VI_1) Receive advertisement timeout
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: (VI_1) Entering MASTER STATE
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: (VI_1) setting VIPs.
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: (VI_1) Sending/queueing gratuitous ARPs on eno2 for 14.52.244.136
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:40:56 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:41:01 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:41:01 iap02 Keepalived_vrrp[7973]: (VI_1) Sending/queueing gratuitous ARPs on eno2 for 14.52.244.136
Jun 9 13:41:01 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:41:01 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:41:01 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:41:01 iap02 Keepalived_vrrp[7973]: Sending gratuitous ARP on eno2 for 14.52.244.136
Jun 9 13:41:44 iap02 Keepalived_vrrp[7973]: (VI_1) Master received advert from 14.52.244.134 with higher priority 99, ours 98
Jun 9 13:41:44 iap02 Keepalived_vrrp[7973]: (VI_1) Entering BACKUP STATE
Jun 9 13:41:44 iap02 Keepalived_vrrp[7973]: (VI_1) removing VIPs.
3.2 HAproxy
- Install
# yum install -y systemd-devel
or
[root@iap01 ~]# wget http://www.haproxy.org/download/2.0/src/haproxy-2.0.14.tar.gz
[root@iap01 ~]# tar zxf haproxy-2.0.14.tar.gz
[root@iap01 ~]# cd haproxy-2.0.14
[root@iap01 haproxy-2.0.14]# make -j $(nproc) TARGET=linux-glibc USE_OPENSSL=1 USE_ZLIB=1 USE_PCRE=1 USE_SYSTEMD=1
…
[root@iap01 haproxy-2.0.14]# make install
…
[root@iap01 haproxy-2.0.14]# /usr/local/sbin/haproxy -v
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
- Configure
[root@iap01 haproxy-2.0.14]# mkdir /etc/haproxy
[root@iap01 haproxy-2.0.14]# vi /etc/haproxy/haproxy.cfg
# /etc/haproxy/haproxy.cfg
# Global settings
global
log /dev/log local0
log /dev/log local1 notice
daemon
# common defaults that all the 'listen' and 'backend' sections will use if not designated in their block
defaults
mode http
log global
option httplog
option dontlognull # disable logging of null connection
# option http-server-close
option forwardfor except 127.0.0.0/8 # insertion of the X-Forwarded-For header to requests sent to servers
option redispatch
retries 1
# the maximum time to wait for a connection to a backend server
timeout connect 5s
# mode tcp: 클라이언트가 연결을 맺고 유지할 수 있는 시간
# mode http: timeout client와 timeout http-keep-alive 중 작은 값 적용
# Kubernetes에서 kubectl 명령어가 client 역활을 하기 때문에 timeout server와 같게 설정
timeout client 60s
# backend로 요청을 보내고 응답을 기다리는 시간
# in TCP mode: server side inactivity
# in HTTP mode: time for the server to process the response (504 returned)
timeout server 60s
# the maximum time to wait in the queue for a connection slot to be free
# When a server's maxconn is reached, connections are left pending in a queue
timeout queue 5s
timeout http-request 10s # 클라이언트가 연결한 후 Request 요청을 보내기까지 대기할 수 있는 시간
timeout http-keep-alive 10s # http의 keep alive 유지를 위한 시간
# apiserver frontend which proxys to the masters
frontend apiserver
bind *:7443
mode tcp
option tcplog
default_backend apiserver
# round robin balancing for apiserver
backend apiserver
option httpchk GET /healthz
http-check expect status 200
mode tcp
option ssl-hello-chk
balance roundrobin
server iap01 14.52.244.134:6443 check
server iap02 14.52.244.135:6443 check
server iap03 14.52.244.207:6443 check
# [...]
[root@iap01 haproxy]#
- haproxy timeout (https://blog.naver.com/bumsukoh/222179667781)
[root@iap01 haproxy]# vi /lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target
[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid"
ExecStartPre=/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
ExecReload=/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify
[Install]
WantedBy=multi-user.target
[root@iap01 haproxy]# systemctl enable haproxy --now
4. Troubleshooting
a. Problem: "No route to host" or Dual master, 192.168.100.10 is VIP
[root@gmd01 ~]# nc 192.168.100.10 7443
Ncat: No route to host.
[root@gmd01 ~]#
b. Cause
[root@gmd01 ~]# nc 192.168.100.10 22 -v
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.100.10:22.
SSH-2.0-OpenSSH_7.4
^C
[root@gmd01 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 125.133.49.190 0.0.0.0 UG 101 0 0 eno1
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1
10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 flannel.1
125.133.49.128 0.0.0.0 255.255.255.192 U 101 0 0 eno1
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.100.0 0.0.0.0 255.255.255.0 U 100 0 0 enp7s0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[root@gmd02 ~]#
[root@gmd01 ~]# ip addr sh | egrep "enp7s0:" -A5
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 70:5a:0f:2f:3c:57 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.11/24 brd 192.168.100.255 scope global noprefixroute enp7s0
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp7s0
valid_lft forever preferred_lft forever
[root@gmd01 ~]#
c. Workaround:
[root@gmd01 ~]# systemctl stop firewalld
'Kubernetes > Install' 카테고리의 다른 글
| Helm (0) | 2021.09.21 |
|---|---|
| MetalLB (0) | 2021.09.15 |
| K8s 구성 - MiniKube on MacOS (0) | 2021.09.14 |
| K8s 구성 - KinD on MacOS (0) | 2021.09.14 |
| K8s 구성 - Single on GCE (0) | 2021.09.14 |
댓글