Producing reliably

Publishing is easy until the process dies between the database write and the Kafka send. Seniors design for that gap.

Publish after the fact exists

The domain write is the source of truth. The event is a notification that the write happened. If you publish first and fail to commit, consumers act on a lie.

Prefer an outbox

A durable pattern:

1. In the same database transaction, write business state and an outbox row.
2. A publisher (poller or CDC) reads the outbox and sends to Kafka.
3. Mark the outbox row published only after a successful send (with retries).

This keeps “state changed” and “event available” aligned without distributed transactions between your DB and the broker.

Producer settings that matter

• Idempotent producer — reduces duplicates on producer retries for the same producer session
• Acks — all (or equivalent) when loss is unacceptable
• Keys — set them when ordering per entity matters
• Timeouts and retries — explicit, bounded, observed

Idempotent producers help transport-level duplicates. They do not replace consumer idempotency or an outbox.

What to put on the wire

• Stable event id (or outbox id) for dedupe
• Occurred-at time from the domain, not only broker timestamp
• Business identifiers consumers will key on
• Schema version or subject name if you use a registry

Anti-patterns

• Fire-and-forget after HTTP 200 with no durability story
• Dual-writing to DB and Kafka in two separate commits “and hoping”
• Giant payloads that force every consumer to parse fields they ignore
• Reusing the same topic for commands, events, and DLQ leftovers

Reliable production is boring infrastructure: outbox, clear keys, strong acks, and metrics on publish lag.