Timeouts, Retries, Backoff & Circuit Breakers

Build a Go HTTP client that calls a flaky upstream without amplifying failure — timeout, bounded retries, exponential backoff with jitter, and a circuit breaker.

The patterns that keep one service's bad day from becoming everyone's: timeouts, retries, backoff and circuit breakers. Here we build a small Go client for calling an unreliable upstream and add each pattern a few lines at a time, watching how each one prevents a specific failure — a hung request, a retry storm, a thundering herd, a cascading outage. The goal is a client that fails gracefully instead of making things worse.

What you'll build

• Bound every outbound call with a timeout
• Retry a fixed number of times, only for retryable errors
• Space retries with exponential backoff and jitter
• Stop hammering a dead upstream with a circuit breaker

Contents

1. An empty Go module
2. A client for the upstream
3. Never wait forever
4. Try again, but a bounded number of times
5. Wait between attempts
6. Back off exponentially
7. Add jitter
8. Use the backoff
9. Don't retry what won't change
10. A circuit breaker
11. Ask before you call
12. Trip on sustained failure
13. Reset on success
14. Put it all together