Demystifying Kafka's Exactly-Once Semantics: A Comprehensive Guide

Kafka exactly-once semantics can protect a stream-processing pipeline from duplicate output records when producers retry, brokers fail over, or an application restarts. The guarantee is powerful, but it is narrower than the phrase sounds: Kafka can make Kafka writes and consumed offsets transactional. It cannot automatically make your external database, payment gateway, or HTTP API exactly-once.

Use exactly-once semantics when duplicate output would be expensive or hard to clean up, such as inventory adjustments, account balance events, or derived state topics consumed by other services.

Delivery Guarantees in Plain English

Kafka applications usually talk about three delivery models.

• At-most-once: Your app may lose records, but it should not process the same record twice. This can happen when offsets are committed before processing finishes.
• At-least-once: Your app should not lose records, but it may process a record more than once after a retry or restart.
• Exactly-once: A Kafka read-process-write loop commits its output records and its consumed offsets as one transaction.

The last point is the key. Exactly-once semantics are strongest when the application reads from Kafka, writes results back to Kafka, and commits offsets within the same transaction.

Idempotent Producers

An idempotent producer prevents duplicate writes caused by producer retries. Kafka assigns the producer an ID and tracks sequence numbers for each producer and partition. If the broker already accepted a batch and then receives the retry, it can reject the duplicate instead of appending it again.

For current Kafka clients, idempotence is enabled by default when you do not configure conflicting producer settings. You can still set it explicitly:

enable.idempotence=true
acks=all

acks=all means the leader waits for all in-sync replicas before acknowledging the write. Idempotence also depends on compatible retry and in-flight request settings, so avoid overriding producer reliability settings unless you know the effect in your client version.

Idempotence protects producer retries, but it does not make a full processing workflow atomic. If your app consumes from one topic and produces to another, you need transactions to tie the output and offset commit together.

Kafka Transactions

Transactions let one producer group multiple writes into an atomic unit. The producer needs a stable transactional.id.

transactional.id=inventory-adjuster-0
enable.idempotence=true
acks=all

A typical transaction flow is:

1. Initialize transactions when the application starts.
2. Begin a transaction.
3. Consume records from the input topic.
4. Produce output records.
5. Send the consumed offsets to the transaction.
6. Commit the transaction, or abort it on failure.

If the process crashes before commit, Kafka does not expose the uncommitted output to read_committed consumers. On restart, the application can read the same input records again and produce one committed result.

Consumer Settings That Matter

Consumers that read transactional output should use:

isolation.level=read_committed
enable.auto.commit=false

read_committed hides records from aborted transactions. enable.auto.commit=false prevents the consumer from committing offsets outside the transaction.

The property name matters. Kafka's consumer setting is enable.auto.commit, not auto.commit.enable.

For a manual consumer-producer app, the offset commit must be part of the producer transaction. In the Java client, that means using the transactional producer APIs, including sending offsets to the transaction before committing it.

A Concrete Scenario

Imagine an orders topic and an inventory-events output topic. Your service reads an order, checks the SKU, and writes an inventory deduction event.

Without transactions, a crash after writing the output but before committing the input offset can create a duplicate deduction after restart. With transactions, the output event and the input offset commit succeed or fail together. A restart may re-read the order, but only one committed inventory event becomes visible to downstream read_committed consumers.

Limits to Keep in Mind

Kafka exactly-once semantics do not cover side effects outside Kafka unless you design for them. If the same service also writes to PostgreSQL or calls a billing API, that external side effect needs its own idempotency key, unique constraint, transaction strategy, or outbox pattern.

Transactions also add coordination overhead. For simple log ingestion where duplicates are acceptable, idempotent producers plus at-least-once consumers may be enough.

Practical Checklist

Use a stable transactional.id per application instance or task. Do not let two live producers use the same transactional ID at the same time.

Set consumers of transactional output to read_committed. Disable automatic offset commits in transactional processing loops.

Keep transactions short. Large transactions can increase latency and make recovery slower.

Treat external systems separately. Kafka can protect Kafka state, but your database writes still need an idempotent design.

The useful takeaway: exactly-once semantics are not a magic switch. They are a set of producer, consumer, and transaction choices that work best for Kafka-to-Kafka stream processing.