Strimzi #1 Overview (0.19.0)

2020.08.20

 

1. Key features

- Strimzi simplifies the process of running Apache Kafka in a Kubernetes cluster.

- The Operators provided with Strimzi are purpose-built with specialist operational knowledge to effectively manage Kafka.

- https://strimzi.io/docs/operators/latest/overview.html#key-features-kafka_str

 

 

2. Kafka

- Apache Kafka is an open-source distributed publish-subscribe messaging system for fault-tolerant real-time data feeds.

- A Kafka cluster comprises multiple brokers Topics are used to receive and store data in a Kafka cluster. Topics are split by partitions, where the data is written. Partitions are replicated across topics for fault tolerance.

- Kafka brokers and topics

- Producers and consumers

- Producing and consuming data

- Apache Kafka combines three key capabilities so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:

   To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.

   To store streams of events durably and reliably for as long as you want.

   To process streams of events as they occur or retrospectively.

- Apache ZooKeeper 

   a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

 

 

3. Strimzi deployment of Kafka

- Apache Kafka components are provided for deployment to Kubernetes with the Strimzi distribution.

 

3.1. Kafka component architecture  

  * Kafka cluster of broker nodes

  * ZooKeeper cluster of replicated ZooKeeper instances

  * Kafka Connect cluster for external data connections

  * Kafka MirrorMaker cluster to mirror the Kafka cluster in a secondary cluster

  * Kafka Exporter to extract additional Kafka metrics data for monitoring

  * Kafka Bridge to make HTTP-based requests to the Kafka cluster

 

3.2. Kafka Bridge interface

- Strimzi Kafka Bridge provides a RESTful interface that allows HTTP-based clients to interact with a Kafka cluster.

- Clients can produce and consume messages without the requirement to use the native Kafka protocol.

- The available protocols are:

    AMQP 1.0 (Advanced Message Queuing Protocol)

    HTTP 1.1 (Hypertext Transfer Protocol)

 

 

4. Strimzi Operators

- Operators

  Operators are a method of packaging, deploying, and managing a Kubernetes application.

- Strimzi provides Operators for managing a Kafka cluster running within a Kubernetes cluster.

   ✓ Cluster Operator

       Deploys and manages Apache Kafka clusters, Kafka Connect, Kafka MirrorMaker, Kafka Bridge, Kafka Exporter, and the Entity Operator

   ✓ Entity Operator

       Comprises the Topic Operator and User Operator

   ✓ Topic Operator

       Manages Kafka topics

   ✓ User Operator

       Manages Kafka users

   ✓ Operators within the Strimzi architecture

 

 

5. Monitoring

- Strimzi can employ the following tools for metrics and monitoring:

  ✓ Prometheus pulls metrics from Kafka, ZooKeeper and Kafka Connect clusters. The Prometheus Alertmanager plugin handles alerts and routes them to a notification service.

  ✓ Kafka Exporter adds additional Prometheus metrics

      Kafka Exporter is an open source project to enhance monitoring of Apache Kafka brokers and clients.

  ✓ Grafana provides dashboard visualizations of Prometheus metrics

  ✓ Jaeger provides distributed tracing support to track transactions between applications

 

- Prometheus (https://prometheus.io/)

   ✓ An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

   ✓ Prometheus's main features are:

       * a multi-dimensional data model with time series data identified by metric name and key/value pairs

       * PromQL, a flexible query language to leverage this dimensionality

       * no reliance on distributed storage; single server nodes are autonomous

       * time series collection happens via a pull model over HTTP

       * pushing time series is supported via an intermediary gateway

       * targets are discovered via service discovery or static configuration

       * multiple modes of graphing and dashboarding support

    ✓ Prometheus Architecture

- Kafka Exporter connects to Kafka as a client and collects different information about topics, partitions and consumer groups. It then exposes this information as a Prometheus metric endpoint.

- Kafka Exporter is provided with Strimzi for deployment with a Kafka cluster to extract additional metrics data from Kafka brokers related to offsets, consumer groups, consumer lag, and topics.

- Consumer lag indicates the difference in the rate of production and consumption of messages.

   it is critical to monitor consumer lag to check that it does not become too big. 

 

 

6. Red Hat AMQ Steams

- Red Hat AMQ Streams is an enterprise-grade Apache Kafka (event streaming) solution, which enables systems to exchange data at high throughput and low latency

- These challenges are resolved using the Operator pattern from the Strimzi project.