Resilience and failure

In a microservice system, failure is normal. Latency spikes, partial deploys, dependency brownouts, and message lag will happen. Senior design assumes that and contains the blast radius.

Timeout everything remote

Every outbound call needs a timeout. Infinite waits turn one stuck thread pool into a cascading outage.

Choose timeouts from SLOs and dependency behavior, not guesswork. A 30s default on an internal call that should return in 50ms is how queues melt.

Retries with care

Retry transient failures (timeouts, 429, 503) with:

• bounded attempts
• exponential backoff and jitter
• idempotent operations only (or an idempotency key)

Do not blindly retry non-idempotent POSTs. Do not retry faster than the dependency can recover — that is a self-inflicted DDoS.

Isolate capacity

Bulkheads keep one dependency from exhausting shared resources: separate thread pools / connection pools / queue consumers per critical path.

If “PSP adapter” hangs, checkout may degrade; search and account reads should not die with it.

Circuit breakers and degradation

When error rates or latency cross a threshold, stop calling for a cool-down and fail fast (or serve a fallback). Pair this with a degraded mode product can accept: queue the request, show stale data, skip an optional enrichment.

Failing loudly and quickly is better than failing slowly and silently.

Backpressure

Protect yourself from overload: limit concurrency, reject or shed load when saturated, and make producers slow down (HTTP 429, consumer lag alerts, rate limits). Infinite buffering hides problems until memory or disk is gone.

Design questions

• What happens if this dependency is down for 5 minutes? 1 hour?
• Which features are optional vs hard requirements?
• Who gets paged, and what is the runbook?

Resilience is not a library checkbox. It is the set of explicit answers to those questions, implemented in code and operations.