Learning · Software Engineering Manifesto
Publish reliably, consume idempotently
Dual-writes lie — durable outbox (or equivalent) for facts; consumers must tolerate duplicates and replay.
Principle
In distributed systems, “I wrote the DB and I sent the message” as two separate commits is a lie under failure. Delivery is at-least-once unless you engineer stronger guarantees end to end.
Therefore we practice
- Persist the business change and the intent to publish in one atomic step when both must happen (transactional outbox or equivalent)
- Give events stable identities so consumers can dedupe
- Design consumers to be safe under retry and replay (idempotent applies, upserts, natural keys)
- Commit consumer side effects before or with offset/ack progress — never ack then hope
- Prefer facts that already happened over fake “events” that are really RPCs
Smells
- Fire-and-forget publish after HTTP 200 with no durability story
- Consumers that “usually” work until a rebalance doubles charges
- No replay path when a bug corrupts a projection
- Poison messages blocking a partition with no DLQ/runbook