Kafka topics and partitions

Kafka is a durable, partitioned log. Most production surprises come from misunderstanding what that log guarantees — and what it does not.

Topics are contracts

A topic is not “a queue for my service.” For domain events it is a published history other teams will depend on.

Decide deliberately:

• who owns the topic (usually the producing domain)
• retention (time, size, or compact for keyed state)
• who may produce (ideally one writer)
• compatibility rules for payloads

Dumping unrelated event types onto one kitchen-sink topic makes authorization, retention, and consumer filtering harder than they need to be.

Partitions are the unit of parallelism

A topic is split into partitions. Ordering is per partition, not per topic.

• Same key → same partition → ordered relative to that key
• Different keys → may interleave across partitions
• More partitions → more consumer parallelism — until you hit broker and client limits

Choose a partition key that matches the ordering you care about (payment id, account id). Random keys maximize throughput and destroy per-entity order.

Consumer groups

Each consumer group tracks its own offsets. Multiple groups can read the same topic independently — that is the fan-out model.

Within one group, Kafka assigns partitions so that each partition is processed by one member. Scale consumers up to the partition count; beyond that, extras sit idle.

What Kafka does not promise by itself

• Exactly-once business effects end to end without careful producer/consumer design
• Global order across a whole topic
• That every consumer processed a message just because it was acknowledged by the broker
• That compacted topics still hold every historical event

Design around at-least-once delivery as the default mental model, then add idempotency and transactional patterns where money or ledgers demand stronger guarantees.