Consuming correctly

Consumers are where business side effects happen. Treat “message received” and “work finished” as different moments.

At-least-once is the default

Expect duplicates. Network retries, rebalances, and crash recovery will redeliver. Design handlers to be safe under replay.

Common approaches:

• store processed event ids (or outbox ids) and skip duplicates
• make writes upserts keyed by business identity
• use DB constraints so a second apply is a no-op

“Exactly once” marketing usually means effectively once after you combine broker features with idempotent application logic.

Commit after side effects

Commit offsets only when the work is durable — or accept that a crash will reprocess. Committing early risks lost effects. Committing late risks duplicates. Prefer duplicates you can ignore over silent loss when money is involved.

Ordering within a key

If you keyed by entity id, one partition gives you order for that entity inside one consumer group. Do not parallelize the same key across threads in a way that reorders applies.

When order across entities does not matter, parallelize freely. When a saga depends on sequence for one payment, keep that payment’s events on one key and process them serially.

Poison messages

A bad payload should not block a partition forever. Define:

• retry with backoff for transient errors
• a dead-letter path after N failures
• alerts on DLQ depth and oldest age
• a runbook to fix and replay

Keep consumers boring

Thin handlers: deserialize, validate, apply idempotent state change, emit metrics. Heavy orchestration belongs in explicit workflows — not in an ever-growing listener class that calls six services synchronously per message.