Kafka Performance Benchmarking

Apache Kafka comes with the kafka-*-perf-test tool, which can be used to benchmark the performance of the Kafka cluster. This can be used as the first level of benchmarking tool before you run your production load tests. 

Kafka Producer Test

You can use kafka-producer-perf-test to measure the Kafka write performance.  The below command demonstrates running the Kafka producer test for 100K records.

bin/kafka-producer-perf-test.sh \ 
--num-records 100000 --topic test2 \
--record-size 10 --throughput -1 \
--producer.config producer.properties

Once you run the above command you will see the result like below

100000 records sent, 158730.158730 records/sec (1.51 MB/sec), 104.48 ms avg latency, 202.00 ms max latency, 117 ms 50th, 154 ms 95th, 158 ms 99th, 159 ms 99.9th.

The producer.properties should contain details like below

linger.ms=100
acks=1
batch.size=1000
buffer.memory=4294967296
request.timeout.ms=300000
bootstrap.servers=localhost:9092

The "--help" option provides more details on controlling different parameters for the test.

Kafka Consumer Test

Similar to the producer test you can run a consumer test as well to measure the read performance from Kafka. The below command demonstrates consumer test on broker localhost and topic test2. 

bin/kafka-consumer-perf-test.sh \
--broker-list localhost:9092 \
--consumer.config consumer.properties \
--topic test2 --print-metrics --threads 1 \
--num-fetch-threads 1 --messages 10000 \
--show-detailed-stats

Once you run the consumer perf test with print-metrics it will provide consuming stats as well like below.

time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-27 16:33:37:363, 0, 0.0041, 0.0006, 1, 0.1597, 1603816414867, -1603816408604, 0.0000, 0.0000
2020-10-27 16:33:42:384, 0, 842.1655, 167.7278, 211001, 42023.5013, 0, 5021, 167.7278, 42023.5013

You can control different parameters of the consumer test by providing the consumer.properties file as specified below

security.protocol=PLAINTEXT
max.partition.fetch.bytes=30485760
fetch.max.bytes=51971520
fetch.min.bytes=5242880
max.poll.records=500
fetch.max.wait.ms=50
enable.auto.commit=false
auto.offset.reset=earliest