Strimzi #2 Configuration

2020.08.20

 

1. Strimzi ?     

- Strimzi simplifies the process of running Apache Kafka in a Kubernetes cluster.

- https://strimzi.io/docs/operators/latest/quickstart.html

- Environments (2020.08.20)

   Strimzi 0.19.0

   Kafka 2.5.0

   Kubernetes 1.15.12

 

 

2. Strimzi Installation 

a. Downloading Strimzi

$ wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.23.0/strimzi-0.23.0.tar.gz
$ tar xzf strimzi-0.23.0.tar.gz
$ cd strimzi-0.23.0

b. Installing Strimzi

# Create a new kafka namespace for the Strimzi Kafka Cluster Operator.
$ kubectl create ns kafka-operator

# Create a new my-kafka-project namespace where you will deploy your Kafka cluster.
$ kubectl create ns kafka-cluster

$ sed -i 's/namespace: .*/namespace: kafka-operator/' install/cluster-operator/*RoleBinding*.yaml    # linux
$ sed -i '' 's/namespace: .*/namespace: kafka-operator/' install/cluster-operator/*RoleBinding*.yaml # MacOS
$ vi install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml
env:
   - name: STRIMZI_NAMESPACE
     value: kafka-cluster  # appended
     # below lines was Commented
     # valueFrom:
     #   fieldRef:
     #     fieldPath: metadata.namespace
$ kubectl apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n kafka-cluster
$ kubectl apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n kafka-cluster
$ kubectl apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n kafka-cluster
$ kubectl apply -f install/cluster-operator/ -n kafka-operator

$ k get pod -n kafka-operator
NAME                                        READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-64ffbb889f-7vx9m   1/1     Running   4          8m26s
$ k logs strimzi-cluster-operator-64ffbb889f-7vx9m -n kafka-operator
…
2020-09-01 01:26:53 INFO  ClusterOperator:81 - Creating ClusterOperator for namespace kafka-cluster
2020-09-01 01:26:53 INFO  ClusterOperator:98 - Starting ClusterOperator for namespace kafka-cluster
2020-09-01 01:26:54 INFO  ClusterOperator:112 - Opened watch for Kafka operator
2020-09-01 01:26:54 INFO  ClusterOperator:112 - Opened watch for KafkaMirrorMaker operator
2020-09-01 01:26:54 INFO  ClusterOperator:112 - Opened watch for KafkaConnect operator
2020-09-01 01:26:54 INFO  ClusterOperator:112 - Opened watch for KafkaBridge operator
2020-09-01 01:26:54 INFO  ClusterOperator:112 - Opened watch for KafkaMirrorMaker2 operator
2020-09-01 01:26:54 INFO  ClusterOperator:123 - Setting up periodic reconciliation for namespace kafka-cluster
2020-09-01 01:26:55 INFO  ClusterOperator:185 - ClusterOperator is now ready (health server listening on 8080)
2020-09-01 01:26:55 INFO  Main:160 - Cluster Operator verticle started in namespace kafka-cluster
$

 

 

3. Creating a cluster

- Create a new Kafka cluster with ZooKeepers and one Kafka brokers.

   ✓ Use persistent-claim storage

   ✓ Expose the Kafka cluster outside of the Kubernetes cluster using an external listener configured to use a nodeport.

   ✓ JBOD (Just a Bunch Of Disks)

       JBOD storage allows you to use multiple disks in each Kafka broker for storing commit logs. Strimzi already added support for JBOD storage in Kafka brokers in version 0.11.

      https://strimzi.io/blog/2019/07/08/persistent-storage-improvements/

$ vi emo-dev-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: emo-dev-cluster
spec:
  kafka:
    replicas: 3
    listeners:
      plain: {}
      tls: {}

    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi               # default 100Gi
        deleteClaim: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi                 # default 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
$ k apply -f kafka-my-cluster.yaml -n kafka-cluster

$ k get statefulsets.apps -n kafka-cluster
NAME                        READY   AGE
emo-dev-cluster-kafka       3/3     58s
emo-dev-cluster-zookeeper   3/3     2m46s
$ k get deployments.apps -n kafka-cluster
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
emo-dev-cluster-entity-operator   1/1     1            1           42s
emo-dev-cluster-kafka-exporter    1/1     1            1           22s
$ k get persistentvolumeclaims -n kafka-cluster
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS             AGE
data-0-emo-dev-cluster-kafka-0     Bound    pvc-9e97d9aa-36d4-49c3-8c48-3cdc50c88931   10Gi       RWO            rook-ceph-block-sc-iap   2m2s
data-0-emo-dev-cluster-kafka-1     Bound    pvc-a7174971-d8a9-4056-8835-7104b38bdac2   10Gi       RWO            rook-ceph-block-sc-iap   2m2s
data-0-emo-dev-cluster-kafka-2     Bound    pvc-0dc90630-ab7b-493c-b8c6-8a5493d95875   10Gi       RWO            rook-ceph-block-sc-iap   2m2s
data-emo-dev-cluster-zookeeper-0   Bound    pvc-4f8dd4c8-c1a3-4ae9-903a-49ad1063edc6   10Gi       RWO            rook-ceph-block-sc-iap   3m50s
data-emo-dev-cluster-zookeeper-1   Bound    pvc-1f5e5757-82f5-4ed1-bfe1-342c7625ff5e   10Gi       RWO            rook-ceph-block-sc-iap   3m50s
data-emo-dev-cluster-zookeeper-2   Bound    pvc-ba356076-045f-4632-9317-b61e705357bd   10Gi       RWO            rook-ceph-block-sc-iap   3m50s
$ kubectl get service emo-dev-cluster-kafka-external-bootstrap -n kafka-cluster -o=jsonpath='{.spec.ports[0].nodePort}{"\n"}'
30170
$

 

 

4. Sending and receiving messages from a topic

a. Create a topic

$ vi kafkaTopic-my-topic.yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: “emo-dev-cluster"
spec:
  partitions: 5
  replicas: 3
$ k apply -f kafkaTopic-my-topic.yaml -n kafka-cluster
$ k get kafkatopics.kafka.strimzi.io -n kafka-cluster
NAME       PARTITIONS   REPLICATION FACTOR
my-topic   5            3
$

 

b. Sending messages

$ wget http://apache.mirror.cdnetworks.com/kafka/2.6.1/kafka_2.13-2.6.1.tgz
$ tar xzf kafka_2.13-2.6.1.tgz
$ cd kafka_2.13-2.6.1
$ bin/kafka-console-producer.sh --bootstrap-server 14.52.244.208:30170,14.52.244.210:30170,14.52.244.211:30170 --topic my-topic
>hello kafka

 

c. Receiving messages

$ bin/kafka-console-consumer.sh --bootstrap-server 14.52.244.208:30170,14.52.244.210:30170,14.52.244.211:30170 --group my-topic --topic my-topic --from-beginning
hello kafka

 

 

5. ZooKeeper connection

- https://strimzi.io/docs/operators/master/using.html#assembly-zookeeper-connection-deployment-configuration-kafka

   ZooKeeper services are secured with encryption and authentication and are not intended to be used by external applications that are not part of Strimzi.

   However, if you want to use Kafka CLI tools that require a connection to ZooKeeper, such as the kafka-topics tool, you can use a terminal inside a Kafka container and connect to the local end of the TLS tunnel to ZooKeeper by using localhost:2181 as the ZooKeeper address.

$ kubectl get pod -n kafka-cluster | grep zookeeper
emo-dev-cluster-zookeeper-0                        1/1     Running   1          21d
emo-dev-cluster-zookeeper-1                        1/1     Running   2          21d
emo-dev-cluster-zookeeper-2                        1/1     Running   1          45h
$ kubectl exec emo-dev-cluster-zookeeper-0 -n kafka-cluster -it -- bin/kafka-topics.sh --list --zookeeper localhost:12181
__consumer_offsets
emo-raw
my-topic
# kubectl exec emo-dev-cluster-zookeeper-0 -n kafka-cluster -it -- sh
sh-4.2$ bin/zookeeper-shell.sh localhost:12181
Connecting to localhost:12181
Welcome to ZooKeeper!
JLine support is disabled

ls /brokers/topics
[__consumer_offsets, emo-raw, my-topic]
ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, isr_change_notification,   latest_producer_id_block, log_dir_event_notification, strimzi, zookeeper]
sh-4.2$