Strimzi #4 Performance test

2020.09.04

 

1. 개요

  -  별도의 튜닝 없이  초 간단하게 Kubernetes 환경에서 동작하는 Kafka 성능 측정

 

 

2. Performance test 

- Test # 1  

   ✓ Produce message: max 132.3K/sec

  ✓ Consumer message: max 229.1K/sec

% kafka-producer-perf-test.sh --topic my-topic --num-records 10000000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=14.52.244.135:30170
  792577 records sent, 158483.7 records/sec (15.11 MB/sec), 1020.3 ms avg latency,  2584.0 ms max latency.
  743201 records sent, 148610.5 records/sec (14.17 MB/sec), 2051.0 ms avg latency,  3296.0 ms max latency.
  866938 records sent, 173352.9 records/sec (16.53 MB/sec), 1804.7 ms avg latency,  2908.0 ms max latency.
  811016 records sent, 162170.8 records/sec (15.47 MB/sec), 1713.7 ms avg latency,  3218.0 ms max latency.
  801158 records sent, 160167.5 records/sec (15.27 MB/sec), 1879.5 ms avg latency,  3959.0 ms max latency.
  131301 records sent,  17743.4 records/sec  (1.69 MB/sec), 3183.7 ms avg latency,  6412.0 ms max latency.
  611444 records sent, 122288.8 records/sec (11.66 MB/sec), 5546.0 ms avg latency, 10597.0 ms max latency.
  951911 records sent, 190382.2 records/sec (18.16 MB/sec), 1663.2 ms avg latency,  2652.0 ms max latency.
  582386 records sent, 116430.6 records/sec (11.10 MB/sec), 2501.5 ms avg latency,  4594.0 ms max latency.
  738236 records sent, 147617.7 records/sec (14.08 MB/sec), 2043.6 ms avg latency,  3525.0 ms max latency.
  408232 records sent,  81646.4 records/sec  (7.79 MB/sec), 2503.0 ms avg latency,  4098.0 ms max latency.
  280267 records sent,  56031.0 records/sec  (5.34 MB/sec), 4885.7 ms avg latency,  7911.0 ms max latency.
  971119 records sent, 194185.0 records/sec (18.52 MB/sec), 2229.0 ms avg latency,  8191.0 ms max latency.
 1001436 records sent, 200207.1 records/sec (19.09 MB/sec), 1526.9 ms avg latency,  2822.0 ms max latency.
10000000 records sent, 133965.6 records/sec (12.78 MB/sec), 2185.16 ms avg latency, 10597.00 ms max latency, 1996 ms 50th, 5824 ms 95th, 10112 ms 99th, 10575 ms 99.9th.
%
% kafka-consumer-perf-test.sh --topic my-topic --group my-topic --threads 10 --messages 10000000 --show-detailed-stats --bootstrap-server 14.52.244.135:30170
time                   , threadId, data.consumed.in.MB,  MB.sec, data.consumed.in.nMsg,    nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-09-03 09:30:14:391,        0,             17.1395,  3.3926,                179721,  35574.2280,     1599093012764, -1599093007712,      0.0000,         0.0000
2020-09-03 09:30:19:429,        0,            152.5199, 26.8718,               1599287, 281771.7348,                 0,           5038,     26.8718,    281771.7348
2020-09-03 09:30:24:476,        0,            290.3814, 27.3155,               3044870, 286424.2124,                 0,           5047,     27.3155,    286424.2124
2020-09-03 09:30:29:484,        0,            382.5201, 18.3983,               4011014, 192920.1278,                 0,           5008,     18.3983,    192920.1278
2020-09-03 09:30:34:532,        0,            478.2718, 18.9682,               5015043, 198896.3946,                 0,           5048,     18.9682,    198896.3946
2020-09-03 09:30:39:564,        0,            583.0569, 20.8238,               6113795, 218352.9412,                 0,           5032,     20.8238,    218352.9412
2020-09-03 09:30:44:578,        0,            695.0686, 22.3398,               7288323, 234249.7008,                 0,           5014,     22.3398,    234249.7008
2020-09-03 09:30:49:587,        0,            800.7571, 21.0997,               8396547, 221246.5562,                 0,           5009,     21.0997,    221246.5562
WARNING: Exiting before consuming the expected number of messages: timeout (10000 ms) exceeded. You can use the --timeout option to increase the timeout.
%

- Test # 2

   차이점 : Producer/Comsumer를 Kubernetes 내 POD에서 수행

 

 

 

3. Test Environments

a. Kafka Server

- Hardware (Kubernetes worker node 6ea)

   Unknown (Intel Xeon E31260L 4Core 1ea, Mem 32GB) * 3ea

   HP ProLiant Dl380 G7 (Intel Xeon X5660 6Core, Mem 32GB, SAS 10000 RPM 2ea) * 3ea - Ceph 운영

- Software

   Kubernetes 1.15.12

   Kafka 2.5.0 / Strimzi 0.19

   Ceph 14.2 / Rook v1.3.6

 

b. Kafka Client

- Software

   MacOS 10.15

   Kafka 2.6.0  (Scala 2.13)

 

c. Kafka Config

$ k describe kafka -n kafka-cluster
Name:         emo-dev-cluster
Namespace:    kafka-cluster
…
Spec:
  Kafka:
    Config:
      log.message.format.version:                2.5
      offsets.topic.replication.factor:          3
      transaction.state.log.min.isr:             2
      transaction.state.log.replication.factor:  3
    Replicas:                 3
    Storage:
      Type:  jbod
      Volumes:
        Delete Claim:  false
        Id:            0
        Size:          10Gi
        Type:          persistent-claim
…
$ kubectl get pvc -n kafka-cluster | egrep ’NAME|kafka'
NAME                            STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS            AGE
data-0-emo-dev-cluster-kafka-0  Bound   pvc-9e97d9aa-36d4-49c3-8c48-3cdc50c88931  10Gi      RWO           rook-ceph-block-sc-iap  2d
data-0-emo-dev-cluster-kafka-1  Bound   pvc-a7174971-d8a9-4056-8835-7104b38bdac2  10Gi      RWO           rook-ceph-block-sc-iap  2d
data-0-emo-dev-cluster-kafka-2  Bound   pvc-0dc90630-ab7b-493c-b8c6-8a5493d95875  10Gi      RWO           rook-ceph-block-sc-iap  2d
$ kubectl get pod -n kafka-cluster -o wide | egrep 'NAME|kafka-[0-2]'
NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE    NOMINATED NODE   READINESS GATES
emo-dev-cluster-kafka-0           2/2     Running   3          43h   10.244.8.11    iap06   <none>           <none>
emo-dev-cluster-kafka-1           2/2     Running   4          43h   10.244.8.15    iap06   <none>           <none>
emo-dev-cluster-kafka-2           2/2     Running   4          43h   10.244.8.17    iap06   <none>           <none>
$

 

d. Kafka topic config

$ k describe kafkatopics.kafka.strimzi.io my-topic -n kafka-cluster | grep Spec -A3
Spec:
  Partitions:  5
  Replicas:    3
$

 

e. Storage Config

[root@rook-ceph-tools-8548f548ff-tq6hm /]# ceph osd status
+----+-------+-------+-------+--------+---------+--------+---------+----------------+
| id |  host |  used | avail | wr ops | wr data | rd ops | rd data |     state      |
+----+-------+-------+-------+--------+---------+--------+---------+----------------+
| 0  | iap09 | 8648M |  270G |    0   |  8192   |    1   |     0   |   exists,up    |
| 1  | iap07 |    0  |    0  |    0   |     0   |    0   |     0   | autoout,exists |
| 2  | iap08 | 6972M |  272G |    0   |  2457   |    0   |     0   |   exists,up    |
| 3  | iap07 |    0  |    0  |    0   |     0   |    1   |     0   | autoout,exists |
| 4  | iap09 | 10.5G |  548G |    0   |   819   |    1   |     0   |   exists,up    |
| 5  | iap08 | 12.1G |  546G |    1   |  17.6k  |    0   |     0   |   exists,up    |
+----+-------+-------+-------+--------+---------+--------+---------+----------------+
[root@rook-ceph-tools-8548f548ff-tq6hm /]# [iap@iap01 ~]$

$ k describe CephBlockPool replica-block-iap -n rook-ceph | grep "Replicated" -A3
  Replicated:
    Require Safe Replica Size:  true
    Size:                       3
    Target Size Ratio:          0
$

## GPU Node
# smartctl -a /dev/sdb -d cciss,0 | egrep 'Rotation|Transport protocol'
Rotation Rate:        10020 rpm
Transport protocol:   SAS (SPL-3)
# smartctl -a /dev/sdc -d cciss,0 | egrep 'Rotation|Transport protocol'
Rotation Rate:        10020 rpm
Transport protocol:   SAS (SPL-3)
#