Table of contents
A retry loop looks like the simplest code in the world: try, and if it failed, wait and try again. The difficult questions, though, live in the gap between one attempt and the next. Was that outcome worth retrying at all? What must be released before another attempt is safe? Can the request even be sent a second time? How long is the wait, and who has the final say on it? The httpretrier
package in nurago
wraps any client with a Do(req) (*http.Response, error) method in retry orchestration, so this post walks through that gap, decision by decision, in the order the code makes them.
import "github.com/tecnickcom/nurago/pkg/httpretrier"
r, err := httpretrier.New(client,
httpretrier.WithRetryIfFn(httpretrier.RetryIfForReadRequests),
httpretrier.WithAttempts(5),
)
resp, err := r.Do(req)
A note on sharing before the walk: an HTTPRetrier holds only immutable configuration after construction, so one instance can serve concurrent Do calls, each of which keeps its mutable state in a per-call value. Each concurrent call needs its own *http.Request, though, since a retry mutates the request’s body.
Decision one: was that outcome worth retrying?
The retry decision is a single pluggable function receiving both the response and the error, so policy can react to transport failures and HTTP status codes alike. The default is the narrowest defensible policy: retry only when err != nil, that is, only when no HTTP response arrived at all.
Two predefined policies widen it. RetryIfForWriteRequests, for state-changing requests, adds only 429, 502, and 503: statuses that generally indicate throttling or a gateway that could not reach the application. RetryIfForReadRequests adds a much longer list (404, 408, 409, 423, 425, 429, 500, 502, 503, 504, 507), and the surprising entries are deliberate. Retrying a 404 or 409 looks wrong until you consider read-after-write eventual consistency: a resource created moments ago may simply not be visible yet, and for an idempotent read the cost of asking again is small. The package documents this reasoning where the policy is defined, and if it does not fit your semantics, a custom RetryIfFn replaces it wholesale. RetryIfFnByHTTPMethod picks the read policy for GET and the write policy for everything else.
One thing no policy can settle is worth stating: a transport error can arrive after the server has processed the request, with only the response lost in transit, so even a conservative retry of a write can execute it twice. Retrying writes is reasonable only when the writes themselves are idempotent, or fenced at the application level; the policy function decides when to retry, not whether that is safe for your data.
Decision two: release what you hold
Before the next attempt may run, the current one must be cleaned up. The response body of the failed attempt is closed, and this is not mere hygiene: an unclosed body keeps its connection out of the transport’s pool, so a retry loop that forgets it degrades the very connection reuse it depends on. A close failure stops the loop and is returned as the error, rather than being ignored.
The ordering has a subtle constraint: the retry-decision function runs before the body is closed, since a policy may want to inspect the body. That is why the RetryIfFn contract says it must not panic; a panic there would leak the open response. The OnRetryFn observability callback, by contrast, runs after the close, and its documentation says so.
There is also a small piece of contract enforcement. Do upholds the standard library’s response-XOR-error convention even when the wrapped client does not: if a non-conforming client returns both a response and an error, the response is closed and dropped so the caller of Do sees one or the other, not both. A test pins it.
Decision three: can the request be sent again?
An http.Request body is a stream, and the first attempt consumed it. Replaying the request relies on Request.GetBody, the standard library’s own mechanism for recreating a body (it is set automatically by http.NewRequest for *bytes.Buffer, *bytes.Reader, and *strings.Reader bodies). When a retry is needed and the body cannot be recreated, the loop stops with ErrBodyNotReplayable instead of silently sending a truncated or empty request. Bodyless requests retry without restriction.
The reopening is lazy, immediately before the retry attempt runs, rather than eagerly when the retry is scheduled. The difference shows up on cancellation: a scheduled retry that is pre-empted by the context does not reopen the body, so nothing is left dangling, which the tests exercise directly.
Decision four: how long is the wait?
The delay arithmetic is delegated to nurago’s backoff
package, whose schedule handles exponential growth, jitter strategies, and the overflow clamping that post covers. The defaults here: 4 total attempts, an initial 1-second delay, a factor of 2, a 100-millisecond additive jitter ceiling, and a 30-second cap on the computed delay.
Then the server gets a say. With WithRespectRetryAfter, a Retry-After header (delta-seconds or an HTTP date; absent, malformed, and non-positive values are ignored) can lengthen the wait: it applies only when it exceeds the computed backoff delay, so it can stretch the schedule but not shorten it below what backoff chose. Jitter is still added on top of the server’s value, and additively regardless of the configured jitter strategy, because a full-jitter draw in [0, delay) could land below the server’s requested minimum, while an additive draw only lands at or above it. Since fleets of clients often receive the same Retry-After value, that jitter is what keeps them from re-synchronising on it.
Trust has a ceiling: the honoured value is capped, by default at 24 hours (WithMaxRetryAfter), so a hostile or misconfigured server does not get to park a caller for an arbitrary time. A Retry-After wait can legitimately exceed WithMaxDelay, which bounds only the exponential schedule; the request context’s deadline remains the outer bound on any wait.
Decision five: who gets told?
WithOnRetry registers a callback invoked before each scheduled retry with the 1-based number of the attempt that just failed, the computed delay, and the response (body already closed) or error that triggered it. It is an observability hook for logs and metrics, with one precise caveat in its contract: it counts scheduled retries, and a scheduled retry can still be pre-empted by cancellation before it runs, so the count can exceed the attempts that actually execute by one. A test pins that the callback is not invoked when cancellation wins the race outright.
Decision six: when does it end?
Three terminators, in decreasing order of good news. The attempt succeeds, or the policy declines to retry: the current response and error are returned as-is. The attempts cap is reached: same. Or the context ends: a context that is already done fails fast before the first attempt, and cancellation during a wait fires through the select that guards the retry timer, so a cancelled caller does not sit out the remainder of a long backoff delay.
One race is documented rather than denied: if the context is cancelled at the same moment a retry timer fires, one further attempt may run with the cancelled request before Do returns the context error. The attempt itself then fails promptly under the dead context; the loop does not pretend the race cannot happen.
The pattern worth taking away is that “retry” names the easy part. What separates a robust retry loop from a harmful one is everything in the gap: a policy matched to the request’s semantics, resources released between attempts, a body that can actually be replayed, waits that respect both the mathematics and the server, and a caller whose cancellation is honoured. Each of those is a small decision, and each has a wrong default that works fine until production finds it.