Troubleshooting Common Kafka Performance Bottlenecks: A Practical Handbook

Kafka performance work gets messy when every slow thing is called a Kafka problem. Sometimes the broker is saturated. Sometimes producers are sending tiny uncompressed records. Sometimes consumers are waiting on a database and Kafka is only the messenger. A useful troubleshooting pass starts by locating where time is being spent: producer send, broker append and replication, consumer fetch, or application processing after the fetch.

This handbook is written for that kind of investigation. It keeps the focus on observable symptoms, likely causes, and changes that are worth testing one at a time.

Understanding Kafka Performance Metrics

Before diving into troubleshooting, it's essential to understand the key metrics that indicate performance health. Monitoring these metrics regularly will help you spot anomalies early:

• Broker Metrics:
  • BytesInPerSec and BytesOutPerSec: Measures the incoming and outgoing data rate. High values can indicate high load, while low values might suggest a bottleneck elsewhere.
  • RequestQueueTimeMs: Average time a request waits in the request queue. High values point to broker overload.
  • NetworkProcessorAvgIdlePercent: Percentage of time network threads are idle. Low percentage indicates high network I/O load.
  • LogFlushRateAndTimeMs: Measures disk flush operations. High latency here directly impacts producer and follower replication.
  • UnderReplicatedPartitions: Number of partitions with fewer replicas than desired. This can indicate replication lag and potential data loss.
• Producer Metrics:
  • RecordBatchSize: Average size of record batches. Large batches can improve throughput but increase latency.
  • RecordSendRate: Number of records sent per second.
  • CompressionRate: Effectiveness of compression. Higher rates mean less data transferred.
• Consumer Metrics:
  • FetchRate: Number of fetch requests per second.
  • BytesConsumedPerSec: Data consumed per second.
  • OffsetLagMax: The maximum offset lag for a consumer group. This is a critical indicator of consumer performance.
• Controller Metadata Metrics: On ZooKeeper-based clusters, watch ZooKeeper request latency and connection health. On KRaft-based clusters, watch controller quorum health and metadata request latency. The exact metric names vary by Kafka version and monitoring stack.

Common Bottleneck Scenarios and Solutions

1. Throughput Limitations

Limited throughput can manifest as slow data ingestion or consumption, impacting the overall speed of your event streams.

1.1. Insufficient Network Bandwidth

• Symptoms: High BytesInPerSec or BytesOutPerSec approaching network interface limits, slow producer/consumer throughput.
• Diagnosis: Monitor network utilization on brokers, producers, and consumers. Compare with available bandwidth.
• Solutions:
  • Scale Network: Upgrade network interfaces or NICs on broker machines.
  • Distribute Load: Add more brokers to distribute network traffic. Ensure topics are partitioned appropriately across brokers.
  • Optimize Serialization: Use efficient serialization formats (e.g., Avro, Protobuf) over less efficient ones (e.g., JSON).
  • Compression: Enable producer-side compression (Gzip, Snappy, LZ4, Zstd) to reduce the amount of data sent over the network. For example, configure your producer:

  # producer.properties
  compression.type=snappy
  

1.2. Disk I/O Bottlenecks

• Symptoms: High LogFlushRateAndTimeMs metrics, slow disk read/write operations, producers and followers falling behind.
• Diagnosis: Monitor disk I/O utilization (IOPS, throughput) on broker machines. Kafka heavily relies on sequential disk writes.
• Solutions:
  • Faster Disks: Use faster SSDs or NVMe drives for Kafka logs. Ensure adequate IOPS and throughput for your workload.
  • RAID Configuration: Use RAID configurations that favor write performance (e.g., RAID 0, RAID 10), but be mindful of redundancy trade-offs.
  • Separate Disks: Distribute Kafka logs across multiple physical disks to parallelize I/O operations.
  • Tune log.flush.interval.messages and log.flush.interval.ms: These settings control how often logs are flushed to disk. While larger values can improve throughput by reducing flush frequency, they increase the risk of data loss if a broker fails before flushing.
  • Be careful with durability tradeoffs: Broker flush settings and producer acks affect how much failure risk you accept. Lowering durability expectations can reduce latency in some workloads, but it should be a business decision with a documented failure model, not a casual tuning trick.

1.3. Insufficient Broker Resources (CPU/Memory)

• Symptoms: High CPU utilization on brokers, high RequestQueueTimeMs, low NetworkProcessorAvgIdlePercent.
• Diagnosis: Monitor CPU and memory usage on broker machines.
• Solutions:
  • Scale Up: Increase CPU cores or RAM on existing broker instances.
  • Scale Out: Add more brokers to the cluster. Ensure topics are well-partitioned to distribute load.
  • Tune JVM Heap: Adjust the JVM heap size for Kafka brokers. Too small a heap can lead to frequent garbage collection pauses, while too large a heap can also cause issues. A common starting point is 6GB or 8GB for many workloads.
  • Offload Operations: Avoid running other resource-intensive applications on Kafka broker machines.

2. High Latency

High latency means a noticeable delay between when an event is produced and when it's consumed.

2.1. Producer Latency

• Symptoms: Producers report high request.timeout.ms or delivery.timeout.ms values being hit.
• Diagnosis: Analyze producer configurations and network conditions.
• Solutions:
  • acks Setting: acks=all waits for the required in-sync replicas and is usually the right choice when durability matters. Pair it with a sensible min.insync.replicas, commonly greater than 1 on replicated production topics. acks=1 can reduce waiting, but it accepts more loss risk during broker failures.
  • linger.ms: Setting linger.ms to a small value (e.g., 0-10ms) sends messages immediately, reducing latency but potentially increasing request overhead. Increasing it batches more messages, improving throughput but increasing latency.
  • batch.size: Larger batch sizes improve throughput but can increase latency. Tune this based on your latency requirements.
  • Network: Ensure low latency network paths between producers and brokers.
  • Broker Load: If brokers are overloaded, producer requests will queue up.

2.2. Consumer Latency (Offset Lag)

• Symptoms: Consumers report significant OffsetLagMax for their consumer groups.
• Diagnosis: Monitor consumer group lag using tools like kafka-consumer-groups.sh or monitoring dashboards.
• Solutions:
  • Scale Consumers: Increase the number of consumer instances within a consumer group, up to the number of partitions for the topic. Each consumer instance can only process messages from one or more partitions, and partitions cannot be shared by multiple consumers within the same group.
  • Increase Partitions: If a topic has too few partitions to keep up with the producer's write rate, increase the number of partitions. Note: This is a permanent change and requires careful consideration as it affects existing consumers and producers.

  # Example to increase partitions for a topic
  kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my-topic --partitions 12
  

  • Optimize Consumer Logic: Ensure the processing logic within your consumers is efficient. Avoid blocking operations or long-running tasks. Process messages in batches if possible.
  • Fetch Configuration: Tune fetch.min.bytes and fetch.max.wait.ms on the consumer. Larger fetch.min.bytes can improve throughput but increase latency, while fetch.max.wait.ms controls how long the consumer waits for data before returning even if the minimum bytes aren't met.
  • Broker Performance: If brokers are struggling (disk, network, CPU), it will directly impact fetch requests and consumer lag.

3. ZooKeeper Bottlenecks

While Kafka is moving towards KRaft (Kafka Raft) for controller quorum, many deployments still rely on ZooKeeper. ZooKeeper issues can cripple Kafka operations.

• Symptoms: Slow broker startup, issues with topic/partition reassignments, zk_avg_latency is high, brokers reporting connection errors to ZooKeeper.
• Diagnosis: Monitor ZooKeeper performance metrics. Check ZooKeeper logs for errors.
• Solutions:
  • Dedicated ZooKeeper Cluster: Run ZooKeeper on dedicated machines, separate from Kafka brokers.
  • Sufficient Resources: Ensure ZooKeeper nodes have adequate CPU, memory, and fast I/O (especially SSDs).
  • ZooKeeper Tuning: Tune ZooKeeper's tickTime, syncLimit, and initLimit settings based on your network and cluster size.
  • Reduce ZooKeeper Traffic: Minimize operations that frequently update ZooKeeper, such as frequent topic creation/deletion or aggressive controller failover.
  • Migrate to KRaft: Consider migrating to KRaft mode to eliminate the ZooKeeper dependency.

Best Practices for Performance Optimization

• Monitor Continuously: Implement robust monitoring and alerting for all key Kafka and ZooKeeper metrics.
• Tune Configurations: Understand the impact of each configuration parameter and tune them based on your specific workload and hardware. Start with sensible defaults and iterate.
• Partitioning Strategy: Choose an appropriate number of partitions per topic. Too few can limit parallelism, while too many can increase overhead.
• Hardware Selection: Invest in appropriate hardware, especially fast disks and sufficient network bandwidth, for your Kafka brokers.
• Producer and Consumer Tuning: Optimize batch.size, linger.ms, acks for producers, and fetch.min.bytes, fetch.max.wait.ms, max.poll.records for consumers.
• Keep Kafka Updated: Newer versions often bring performance improvements and bug fixes.
• Load Testing: Regularly perform load tests to simulate production traffic and identify potential bottlenecks before they impact live systems.

How to Run a Performance Investigation

Change one layer at a time. If producers are slow, first check producer metrics such as request latency, batch size, compression ratio, retries, and buffer exhaustion. If brokers are slow, check request queue time, network thread idle percent, disk await, page cache pressure, under-replicated partitions, and controller stability. If consumers are slow, check lag by partition, processing time per batch, downstream dependency latency, and rebalance frequency.

A real example: an orders topic shows rising lag after a marketing campaign. Broker CPU is fine, disk writes are fine, and producer retries are normal. kafka-consumer-groups.sh --describe shows one partition with most of the lag. That points away from broker capacity and toward partition skew. If records are keyed by customer ID and one large customer is generating most events, adding consumers will not help that partition because a partition is still assigned to only one consumer in the group. You may need to change the keying strategy for future data, split the workload by topic, or make that consumer path faster.

Another example: all partitions lag together, and consumer logs show calls to a payment API taking several seconds. Kafka fetch tuning will not fix that. You either need bounded concurrency inside the consumer, a queue between Kafka and the slow dependency, bulk writes, or a product decision about backpressure and retries.

Good Kafka tuning is mostly disciplined measurement. Keep a baseline, make one change, load test with realistic record sizes and keys, then compare p95 and p99 latency as well as throughput. Average latency can look fine while a small number of partitions are already falling behind.

What I Check Before Changing Config

Before tuning Kafka, I like to prove the bottleneck is actually in Kafka. Pick one slow path and trace it end to end. For a produced event, how long does the producer spend waiting for send completion? How long before the record appears in the topic? How long before the consumer fetches it? How long does the consumer spend after fetch? Those four numbers prevent a lot of random configuration changes.

If producer send time is high, inspect batching, compression, retries, acks, delivery.timeout.ms, and broker request latency. If broker append is slow, inspect disk, network, ISR churn, controller events, and request queues. If consumer fetch is fast but processing is slow, stop tuning broker threads and look at application code. If everything is fast until a downstream database write, Kafka is not the bottleneck.

Here is a realistic pattern. A team sees high end-to-end latency and increases broker memory. Nothing changes. Then they check consumer timing and find each message performs three serial HTTP calls. Kafka was delivering batches quickly; the consumer was spending most of its time waiting outside the cluster. The useful fix was bounded concurrency, timeouts, and a dead-letter path for repeated downstream failures.

Another common pattern is tiny producer batches. A service sends one small JSON record at a time with no linger and no compression. Broker CPU rises, network overhead rises, and throughput is poor even though no single machine looks completely saturated. A small linger.ms, a larger batch.size, and a faster serialization format may improve throughput more than adding brokers. The right values depend on latency tolerance, so test them with real record sizes instead of copying defaults from another system.

The safest performance changes are reversible and measurable. Client settings are usually easier to roll back than partition-count changes. Compression changes are usually easier to test than hardware changes. Partition increases can be useful, but they affect ordering and future key distribution, so they deserve more care than a normal client-side tuning change.