Exactly-Once Semantics — What It Actually Means in Practice

“Exactly-once” is one of the most misused terms in distributed systems. I’ve seen it promised in design documents, ignored in implementations, and confused with things it doesn’t actually cover. Before you configure enable.idempotence=true and tell your team you have exactly-once delivery, it’s worth being precise about what that means — and what it doesn’t.

The Three Guarantees

Kafka gives you a choice of three delivery semantics between producer and broker:

At-most-once — the producer fires and forgets. Messages may be lost if the broker doesn’t acknowledge. No retries. Fast, but unsuitable for anything where data loss is unacceptable.

At-least-once — the producer retries on failure. Messages will eventually be delivered, but a retry after a network timeout where the original write actually succeeded means duplicates are possible. This is the default and the most common real-world behaviour.

Exactly-once — a message is written to the broker exactly once, and consumed exactly once by a Kafka Streams topology. Not “at most once” or “at least once” — once, period.

The important caveat is that scope. Kafka’s exactly-once guarantee covers the Kafka-to-Kafka path: producer writing to a topic, and Kafka Streams consuming and producing within the same cluster. It does not automatically extend to your database writes, your external API calls, or any side effect outside Kafka. That’s a different problem — which is why the Outbox Pattern from the previous post still matters even with exactly-once enabled.

Idempotent Producers

The foundation of exactly-once is the idempotent producer. Enable it with a single config:

@Configuration
public class KafkaProducerConfig {

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        // these are set automatically when idempotence is enabled, but being explicit is clearer:
        config.put(ProducerConfig.ACKS_CONFIG, "all");
        config.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
        config.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
        return new DefaultKafkaProducerFactory<>(config);
    }
}

When idempotence is enabled, the broker assigns each producer a Producer ID (PID) and tracks a sequence number for every message per partition. If the producer retries a message that was already written (because a network timeout hid the broker’s acknowledgement), the broker recognises the duplicate sequence number and discards it rather than writing again. The producer gets a success acknowledgement and moves on.

This eliminates duplicates within a single producer session. It doesn’t survive a producer restart — a new process gets a new PID, and the broker can’t correlate it with the previous one.

Kafka Transactions

For writes that span multiple partitions or topics, and for the produce-consume-produce pattern in stream processing, Kafka provides transactions. A transactional producer writes atomically across any number of partitions:

@Configuration
public class KafkaTransactionalConfig {

    @Bean
    public ProducerFactory<String, String> transactionalProducerFactory() {
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        config.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "claims-producer-1");
        return new DefaultKafkaProducerFactory<>(config);
    }

    @Bean
    public KafkaTemplate<String, String> transactionalKafkaTemplate() {
        KafkaTemplate<String, String> template =
            new KafkaTemplate<>(transactionalProducerFactory());
        template.setTransactionIdPrefix("claims-tx-");
        return template;
    }
}

The transactional.id must be stable across restarts — it’s the identifier the broker uses to fence zombie producers (a previous instance that didn’t shut down cleanly). Using a stable, unique ID per logical producer is essential; generating it randomly at startup defeats the purpose.

Using executeInTransaction in Spring

Spring Kafka’s KafkaTemplate.executeInTransaction() wraps the Kafka transaction API:

@Service
@RequiredArgsConstructor
public class ClaimEventPublisher {

    private final KafkaTemplate<String, String> kafkaTemplate;
    private final ObjectMapper objectMapper;

    public void publishDecisionAndAudit(ClaimDecision decision,
                                         AuditEvent audit) throws JsonProcessingException {

        kafkaTemplate.executeInTransaction(ops -> {
            ops.send("claim.decisions",
                decision.claimId(),
                objectMapper.writeValueAsString(decision));

            ops.send("claim.audit",
                decision.claimId(),
                objectMapper.writeValueAsString(audit));

            return null;
        });
    }
}

Both messages land in both topics atomically. Either both are written and visible to consumers, or neither is — there’s no state where a consumer sees the decision event but not the audit event.

Consumers: Isolation Level

Transactional writes don’t help if your consumer reads uncommitted messages. Set the isolation level on consumers that must only see committed data:

@Bean
public ConsumerFactory<String, String> consumerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    config.put(ConsumerConfig.GROUP_ID_CONFIG, "claims-processor");
    config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    config.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
    return new DefaultKafkaConsumerFactory<>(config);
}

The default is read_uncommitted. With read_committed, the consumer skips messages from aborted transactions and waits for in-progress transactions to complete before advancing past them. This is required for exactly-once to be meaningful end-to-end.

The Exactly-Once Scope Problem

Here’s where most explanations stop, and where most production bugs start. Consider this consumer:

@KafkaListener(topics = "claim.decisions")
@Transactional("kafkaTransactionManager")
public void consume(ClaimDecision decision) {
    claimRepository.save(buildProjection(decision));   // MongoDB write
    kafkaTemplate.send("claim.projections.updated", decision.claimId(), "updated");
}

This looks like exactly-once. It isn’t. The MongoDB write and the Kafka produce happen in two separate transaction scopes. If the MongoDB write succeeds but the Kafka produce fails (or vice versa), you have an inconsistency. Kafka’s transaction covers the Kafka side; MongoDB has no knowledge of it.

For true end-to-end exactly-once behaviour into a database, you need either:

1. The Outbox Pattern — write to the database first, publish from the outbox separately
2. Idempotent consumers — make the database write safe to repeat, so at-least-once delivery from Kafka is sufficient

Kafka Streams is the one case where Kafka’s exactly-once guarantee is genuinely end-to-end: consume from a source topic, apply a transformation, produce to a sink topic — all within Kafka’s transaction coordinator. If you’re not using Kafka Streams, exactly-once covers the broker side only.

Configuring Exactly-Once in Spring Boot Kafka Streams

For Kafka Streams applications, the configuration is a single property:

spring:
  kafka:
    streams:
      properties:
        processing.guarantee: exactly_once_v2

exactly_once_v2 (introduced in Kafka 2.5) is the preferred setting over the older exactly_once — it uses one transaction per partition rather than one per thread, which reduces transaction coordinator load significantly.

The Performance Cost

Exactly-once is not free. The transactional overhead adds latency because:

• Each transaction requires a beginTransaction, sendOffsetsToTransaction, and commitTransaction round-trip to the broker
• The transaction coordinator on the broker must maintain transaction state
• read_committed consumers may lag behind read_uncommitted ones waiting for open transactions to close

In practice, for low-to-medium throughput services — a claims processor, an audit pipeline, a financial data enrichment service — the overhead is negligible. For very high-throughput systems (tens of thousands of messages per second), it’s measurable and worth benchmarking.

My rule of thumb: use at-least-once with idempotent consumers by default. Add Kafka transactions when you genuinely need atomic multi-topic writes or are using Kafka Streams. Don’t enable transactions just because you can.

Summary

Getting exactly-once right is less about Kafka configuration and more about understanding which part of your pipeline each guarantee actually covers. The configuration is the easy part.

If you’re designing event-driven systems where correctness under failure matters, get in touch.