Redis cluster

2021.05.20, 2021.01.20

 

1. Redis cluster

- Redis is an advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs.

- Redis cluster chart bootstraps a Redis deployment on a Kubernetes cluster using the Helm package manager.

- Reference

   https://redis.io/  

   https://bitnami.com/stack/redis-cluster/helm 

   https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster

 

 

2.  Environments

- Environments (2021.05.20)

   ✓ helm chart redis-cluster-6.0.3 / redis 6.2.3

   ✓ Helm 3.3.1 / Kubernetes 1.16.15 

- Environments (2021.01.20)

   ✓ helm chart redis-cluster-4.2.5 / redis 6.0.10

   ✓ Helm 3.3.1 / Kubernetes 1.16.15 

 

 

3.  Configure Redis cluster

a. Install helm repo

- The Redis Helm Chart will deploy a master-slave cluster using Redis Sentinel

- The Redis Cluster Helm Chart will deploy a Redis Cluster with sharding.

   ✓ Data sharding (partitioning) is the technique to split all data across multiple Redis instances so that every instance will only contain a subset of the keys.

  ✓ Such a process allows mitigating data grown by adding more and more instances and dividing the data to smaller parts (shards or partitions)

$ helm search repo bitnami | egrep 'NAME|redis'
NAME                      CHART VERSION   APP VERSION     DESCRIPTION
bitnami/redis             14.1.1          6.2.3           Open source, advanced key-value store. It is of...
bitnami/redis-cluster     6.0.3           6.2.3           Open source, advanced key-value store. It is of...
$

 

b. Create a Redis cluster

- https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster#parameters

$ helm inspect values bitnami/redis-cluster --version 4.2.5 > redis-cluster.values
$ vi redis-cluster.values
…

cluster:
  # The Helm Chart will deploy by default 3 redis masters and 3 replicas.
  nodes: 6             # default: 6
  replicas: 1          # default: 1 (Master:3, Slave:3)
…

redis:
…
  resources: {}        # default
  # resources:
  #   requests:
  #    memory: 256Mi   # default: Commented
  #     cpu: 100m      # default: Commented
…

service:
…
  type: NodePort       # default: ClusterIP
…

persistence:
…
  storageClass:
  size: 8Gi            # default: 8Gi
…

metrics:
  enabled: true        # default: false
…
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
…

sysctlImage:
  enabled: true        # default: false
  # command: []        # default
  # listen backlog, 즉 listen()으로 바인딩 된 서버 소켓에서 accept()를 기다리는 소켓 개수에 관련된 커널 파라미터는 'net.core.somaxconn'임
  command:
    - /bin/sh
    - -c
    - |-
      install_packages procps
      sysctl -w net.core.somaxconn=10000
      echo never > /host-sys/kernel/mm/transparent_hugepage/enabled
  mountHostSys: true   # default: false
…
$ helm install aicc bitnami/redis-cluster --create-namespace --namespace redis-cluster \
                   --values redis-cluster.values --version 6.0.3
NAME: aicc
LAST DEPLOYED: Thu May 20 21:01:35 2021
NAMESPACE: redis-cluster
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "redis-cluster" aicc-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)

You have deployed a Redis(TM) Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis(TM) cluster:

1. Run a Redis(TM) pod that you can use as a client:
kubectl run --namespace redis-cluster aicc-redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:6.2.3-debian-10-r0 -- bash

2. Connect using the Redis(TM) CLI:
redis-cli -c -h aicc-redis-cluster -a $REDIS_PASSWORD

$
$ helm list -A | egrep 'NAME|redis'
NAME  NAMESPACE      REVISION  UPDATED                                  STATUS    CHART                APP VERSION
aicc  redis-cluster            2021-05-20 21:01:35.500552385 +0900 KST  deployed  redis-cluster-6.0.3  6.2.3
$ k get all -n redis-cluster -o wide
NAME                      READY  STATUS   RESTARTS  AGE    IP            NODE   NOMINATED NODE  READINESS GATES
pod/aicc-redis-cluster-0  2/2    Running  0         6m49s  10.244.14.40  iap12  <none>          <none>
pod/aicc-redis-cluster-1  2/2    Running  0         6m49s  10.244.2.165  iap13  <none>          <none>
pod/aicc-redis-cluster-2  2/2    Running  0         6m49s  10.244.3.189  iap11  <none>          <none>
pod/aicc-redis-cluster-3  2/2    Running  0         6m48s  10.244.4.55   iap10  <none>          <none>
pod/aicc-redis-cluster-4  2/2    Running  0         6m48s  10.244.14.41  iap12  <none>          <none>
pod/aicc-redis-cluster-5  2/2    Running  0         6m48s  10.244.2.164  iap13  <none>          <none>

NAME                                 TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)             AGE    SELECTOR
service/aicc-redis-cluster           NodePort   10.104.19.172  <none>       6379:32243/TCP      6m49s  app.kubernetes.io/instance=aicc,app.kubernetes.io/name=redis-cluster
service/aicc-redis-cluster-headless  ClusterIP  None           <none>       6379/TCP,16379/TCP  6m49s  app.kubernetes.io/instance=aicc,app.kubernetes.io/name=redis-cluster
service/aicc-redis-cluster-metrics   ClusterIP  10.97.207.120  <none>       9121/TCP            6m49s  app.kubernetes.io/instance=aicc,app.kubernetes.io/name=redis-cluster

NAME                                 READY  AGE    CONTAINERS                  IMAGES
statefulset.apps/aicc-redis-cluster  6/6    6m49s  aicc-redis-cluster,metrics  docker.io/bitnami/redis-cluster:6.2.3-debian-10-r0,docker.io/bitnami/redis-exporter:1.23.0-debian-10-r8

NAME                                         COMPLETIONS  DURATION  AGE   CONTAINERS  IMAGES                                               SELECTOR
job.batch/aicc-redis-cluster-cluster-create  1/1          71s       121d  trigger     docker.io/bitnami/redis-cluster:6.0.10-debian-10-r1  controller-uid=30546a8f-c46a-404d-b8ae-759c5fea8770
$ k get pvc -n redis-cluster
NAME                             STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
redis-data-aicc-redis-cluster-0  Bound   pvc-9571935e-2d2f-4b70-abeb-832986d19f99  8Gi       RWO           nfs-sc-iap    7m24s
redis-data-aicc-redis-cluster-1  Bound   pvc-9496d5b9-9eac-4ec8-97d4-22a974b0a994  8Gi       RWO           nfs-sc-iap    7m24s
redis-data-aicc-redis-cluster-2  Bound   pvc-17e16ab3-23af-4ba2-9217-981493c019f2  8Gi       RWO           nfs-sc-iap    7m24s
redis-data-aicc-redis-cluster-3  Bound   pvc-dad0da38-d7f9-417f-b76f-f0d2bbcce490  8Gi       RWO           nfs-sc-iap    7m24s
redis-data-aicc-redis-cluster-4  Bound   pvc-ea23741b-1c9f-4aa6-85e6-7733ce9240a4  8Gi       RWO           nfs-sc-iap    7m23s
redis-data-aicc-redis-cluster-5  Bound   pvc-b9b322f1-e335-4379-9568-f5c5185bee10  8Gi       RWO           nfs-sc-iap    7m23s
$

 

c. Connect using the Redis CLI

$ export REDIS_PASSWORD=$(kubectl get secret --namespace redis-cluster aicc-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
$ kubectl run --namespace redis-cluster aicc-redis-cluster-client --rm --tty -i --restart='Never' \
                   --env REDIS_PASSWORD=$REDIS_PASSWORD --image docker.io/bitnami/redis-cluster:6.2.3-debian-10-r0 -- bash
I have no name!@aicc-redis-cluster-client:/$ redis-cli -c -h aicc-redis-cluster -a $REDIS_PASSWORD cluster nodes
da98d8a7a2b50bb3df8444507248b3361cb0ebd2 10.244.9.104:6379@16379  master       -                                        0 1611042479000 1 connected 0-5460
61b3a53b5505ae6a158d0497200a9450de12dc72 10.244.15.53:6379@16379  master       -                                        0 1611042478000 2 connected 5461-10922
c7b722525e33daa4b85287e77642784e92a2b2eb 10.244.14.106:6379@16379 master       -                                        0 1611042478585 3 connected 10923-16383
2ef12cb4ee50c09e1684efcc038d407c8f5cf7c9 10.244.10.120:6379@16379 slave        da98d8a7a2b50bb3df8444507248b3361cb0ebd2 0 1611042480592 1 connected
1f2c3c3d53e751361782a495003ae7115748e5d7 10.244.9.103:6379@16379  slave        61b3a53b5505ae6a158d0497200a9450de12dc72 0 1611042479588 2 connected
e4731bf19c34907ed914a4c28616ab6b68f523c6 10.244.15.52:6379@16379  myself,slave c7b722525e33daa4b85287e77642784e92a2b2eb 0 1611042479000 3 connected
I have no name!@aicc-redis-cluster-client:/$
I have no name!@aicc-redis-cluster-client:/$ redis-cli -c -h aicc-redis-cluster -a $REDIS_PASSWORD
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
aicc-redis-cluster:6379> SET mykey1 "Hello"
-> Redirected to slot [14687] located at 10.244.14.106:6379
OK
10.244.14.106:6379> SET mykey2 GoodJob
-> Redirected to slot [1860] located at 10.244.9.104:6379
OK
10.244.9.104:6379> GET mykey1
-> Redirected to slot [14687] located at 10.244.14.106:6379
"Hello"
10.244.14.106:6379> hset key1 field1 10 field2 20
-> Redirected to slot [9189] located at 10.244.2.165:6379
(integer) 1
10.244.14.106:6379> exit
I have no name!@aicc-redis-cluster-client:/$

- Cluster nodes command

   ✓ format

       <id> <ip:port@cport> <flags> <master> <ping-sent> <pong-recv> <config-epoch> <link-state> <slot> <slot> ... <slot>

   ✓ Description

       ▷ id: The node ID, a 40 characters random string generated when a node is created and never changed again (unless CLUSTER RESET HARD is used).

       ▷ ip:port@cport: The node address where clients should contact the node to run queries.

       ▷ flags: A list of comma separated flags: myself, master, slave, fail?, fail, handshake, noaddr, noflags.

       ▷ master: If the node is a replica, and the master is known, the master node ID, otherwise the "-" character.

       ▷ ping-sent: Milliseconds unix time at which the currently active ping was sent, or zero if there are no pending pings.

       ▷ pong-recv: Milliseconds unix time the last pong was received.

       ▷ config-epoch: The configuration epoch (or version) of the current node (or of the current master if the node is a replica).

           Each time there is a failover, a new, unique, monotonically increasing configuration epoch is created.

           If multiple nodes claim to serve the same hash slots, the one with higher configuration epoch wins.

       ▷ link-state: The state of the link used for the node-to-node cluster bus. We use this link to communicate with the node. Can be connected or disconnected.

       ▷ slot: A hash slot number or range. Starting from argument number 9, but there may be up to 16384 entries in total (limit never reached).

       This is the list of hash slots served by this node. and means that the node is responsible for all the hash slots from start to end including the start and end values.

 

- Cluster topology

   By default only one service is exposed (when not using the external access mode). You will connect your client to the exposed service, regardless you need to read or write.

   When a write operation arrives to a replica it will redirect the client to the proper master node. For example, using redis-cli you will need to provide the -c flag for redis-cli to follow the redirection automatically.

   Using the external access mode, you can connect to any of the pods and the slaves will redirect the client in the same way as explained before, but the all the IPs will be public.

 

 

4.  Monitoring Redis cluster

- 공통으로 사용할 kube-prometheus-stack을 구성하고 다수의 오픈소스 SW를 수용(Metric 수집, Grafana dashboard 제공)하는 방식으로 구성

- Scraping Annotations

$ k describe pod aicc-redis-cluster-0 -n redis-cluster | grep "prometheus.io"
 prometheus.io/port: 9121
 prometheus.io/scrape: true
$

- Redis dashboard (Unofficial)

- Grafana