Talking to Other Services Without Losing the Thread

Every outbound call is a place the service can hang, leak a goroutine, lose the trace, or make somebody else’s outage worse, and http.Get handles none of it. What a call needs is a timeout, structured logging of what went out and came back, a metric, the trace ID forwarded, and secrets kept out of the log line. Every client needs the same list, so it is built once.

One base, many clients

httpClientOpts := []httpclient.Option{
	httpclient.WithLogger(l),
	httpclient.WithRoundTripper(m.InstrumentRoundTripper),
	httpclient.WithTraceIDHeaderName(traceid.DefaultHeader),
	httpclient.WithComponent(appInfo.ProgramName),
	httpclient.WithRedactFn(logRedactor.BytesToString),
}

Built once in bind and passed to each client constructor, which appends only its own; inside newIpifyClient the parameter is baseHTTPClientOpts:

ipifyHTTPClient := httpclient.New(append(
	baseHTTPClientOpts,
	httpclient.WithTimeout(ipifyTimeout),
)...)

WithRoundTripper records duration and status for outbound calls the way page 09’s InstrumentHandler does for inbound ones, WithComponent tags the log lines so one query separates this client’s calls from the others, and WithTraceIDHeaderName is the propagation half of page 06.

The rest of the package is mostly transport configuration, and one of its defaults is worth knowing: the base transport is a clone of http.DefaultTransport with MaxIdleConnsPerHost raised from the standard library’s 2 to 100, because a service concentrating traffic on a few hosts otherwise spends its time reopening connections it just closed.

Option order matters

Options are applied in the order given, and four of them touch the same field:

WithTransport      // installs the base *http.Transport
WithDialContext    // mutates that transport
WithTLSClientConfig// mutates that transport
WithRoundTripper   // wraps it in something that is no longer an *http.Transport

WithDialContext and WithTLSClientConfig type-assert the current transport to *http.Transport and do nothing when the assertion fails. Once WithRoundTripper has wrapped it, that assertion fails, and both options become silent no-ops: no error, no log line, a client that dials normally and verifies against the system roots while the code says otherwise.

So: WithTransport first, then WithDialContext and WithTLSClientConfig, then WithRoundTripper. The base slice above puts WithRoundTripper second and every per-client option is appended after it, which closes the door on the other three for anything built from that base. Nothing breaks today, because the only option appended is WithTimeout and that does not touch the transport. The first client needing a dialer or a TLS config has to prepend it or build a separate slice.

Why the append is safe

The source comment on newIpifyClient is precise about it:

// The base slice is left untouched: it has no spare capacity, so the append
// always copies rather than mutating the caller's slice, which keeps it safe
// to reuse for the next client.

append writes into the existing backing array when capacity allows and allocates a copy when it does not, and a slice built with a literal has length equal to capacity, so the first append always copies. Two clients appending different timeouts to the same base get independent slices.

That stops being true the moment somebody writes make([]httpclient.Option, 0, 10). Then the first append writes into the shared array, the second overwrites the first, and both clients end up with whichever timeout was appended last: working code with wrong values.

opts := slices.Clone(baseHTTPClientOpts)
opts = append(opts, httpclient.WithTimeout(ipifyTimeout))

slices.Clone does not depend on how the base was built. Write that in code you did not construct yourself.

Timeouts

The client defaults to one minute; http.DefaultClient has no timeout at all. A hung upstream then holds a handler goroutine, a connection, and whatever the handler had already acquired, until the process restarts.

Per-client timeouts, because the calls are different. The ipify diagnostic gets one second, since nothing depends on it. A payment authorisation might get ten, a search backend two hundred milliseconds against a latency budget.

Inheriting the request budget

Page 09 established the rule: the client timeout bounds this call and the request deadline bounds everything, so pass the request context and both apply:

req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
	return fmt.Errorf("building request: %w", err)
}

resp, err := s.client.Do(req)

http.NewRequestWithContext, never http.NewRequest. Without the context the call is bounded only by the client timeout, so a client that disconnected two seconds ago is still being worked for.

A client timeout longer than the request budget is a number that describes nothing that will happen: with a 60-second budget and three sequential 30-second calls, the third can never use its allowance.

Retries

retrier, err := httpretrier.New(client,
	httpretrier.WithRetryIfFn(httpretrier.RetryIfForReadRequests),
	httpretrier.WithAttempts(4),
	httpretrier.WithDelay(200*time.Millisecond),
	httpretrier.WithDelayFactor(2),
	httpretrier.WithJitter(100*time.Millisecond),
	httpretrier.WithMaxDelay(5*time.Second),
	httpretrier.WithRespectRetryAfter(true),
)

Wrapping rather than replacing, so the client keeps its logging, instrumentation and trace propagation.

Read and write are different

httpretrier.RetryIfForReadRequests   // idempotent: GET, HEAD
httpretrier.RetryIfForWriteRequests  // POST, PUT, PATCH
httpretrier.RetryIfFnByHTTPMethod    // picks one from the method

A GET that times out can be retried, because the worst case is a wasted request. A POST cannot safely be: the response was lost, and the side effect may not have been, so retrying a create that actually succeeded creates a second row.

The default policy is conservative: retry only when err != nil, a transport failure with no response.

Retrying writes is possible when the upstream supports idempotency keys, the mechanism page 15 describes on the inbound side: send the same key on every attempt and the upstream deduplicates. Without that guarantee, a write retry is a bet.

Backoff

s := backoff.New(backoff.Config{
	Base:     100 * time.Millisecond,
	Factor:   2,
	Jitter:   50 * time.Millisecond,
	MaxDelay: 30 * time.Second,
})

100ms, 200ms, 400ms, 800ms, capped at 30 seconds, each with a random addition.

Jitter is the easiest of those to leave out and the most expensive. Without it, clients that failed at the same moment retry at the same moment, and an upstream that was recovering gets a synchronised wave that knocks it over again. That thundering herd is why an outage sometimes takes several attempts to end.

The additive default adds a fixed random ceiling and desynchronises less as the delay grows, so a long retry sequence wants JitterFull or JitterEqual, which scale with the delay.

The backoff article covers the overflow arithmetic. Exponential growth in time.Duration reaches the int64 limit quickly, and an unclamped calculation produces a negative delay: a time.Sleep that returns immediately and a retry loop with no delay at all.

Retry-After

httpretrier.WithRespectRetryAfter(true)

A 429 or 503 may carry Retry-After, in seconds or as an HTTP date: the upstream telling you when it expects to serve you. Using your own backoff instead means retrying before it is ready.

The wait becomes the larger of the computed backoff and the server’s value, bounded by MaxRetryAfter, which defaults to 24 hours. That value comes from another system, so set the cap against your own request budget: a retrier waiting ten minutes inside a request with 60 seconds left is waiting for a deadline that fires first.

This is page 08’s rate limiting from the other side.

Replaying the body

var ErrBodyNotReplayable = errors.New("cannot retry: the request body has already been consumed and Request.GetBody is not set")

A request body is a stream, consumed by the first attempt. The retrier recreates it through Request.GetBody, which http.NewRequest sets for *bytes.Buffer, *bytes.Reader and *strings.Reader. A body from an arbitrary io.Reader gets none, and the retry fails with the error above rather than sending an empty body.

To retry a streamed body, buffer it first and pay the memory.

Retries can make outages worse

An upstream is at capacity and answering slowly. Every client retries three times, so it receives four times its normal rate at the moment it is least able to serve it, and a brownout becomes an outage. Retries help with transient, uncorrelated failures and hurt with overload, correlated by definition.

The circuit breaker

A breaker distinguishes the two. Count recent failures, and when they cross a threshold stop calling and fail immediately for a cooldown, then let one request through: if it works, close the breaker; if not, extend the cooldown.

Most of the value is in the open state, converting a slow failure into a fast one: a caller failing in a microsecond instead of a two-second timeout stops holding a handler goroutine, and the upstream gets room to recover.

A breaker can also be wrong. A brief blip opens it and the service refuses calls that would have worked, so a partial outage becomes a total one for the cooldown. Threshold and cooldown are the tuning, and both depend on traffic shape.

Placement: outside the retrier, so a burst of retries counts toward opening it rather than being hidden inside one logical call.

If a mesh is already doing this

A service mesh sidecar typically does retries, timeouts, circuit breaking and outlier detection for mesh-internal traffic, and an in-process retrier duplicates it. The mesh opens its breaker after five failures, your retrier reads each rejection as worth retrying and sends four attempts per call, and the mesh counts those toward its thresholds. The effective retry count is the product rather than the sum.

The mesh owns mesh-internal traffic, and the process owns calls that leave the mesh. A third-party API on the public internet has no sidecar in front of it and needs an in-process retrier. An internal service reached through the mesh does not.

Say which is which in the code, because it is not visible from the call site:

// Outside the mesh: no sidecar retries this, so the retrier is ours.
paymentClient := httpretrier.New(...)

// Inside the mesh: the sidecar owns retry and breaking for this path.
inventoryClient := httpclient.New(...)

Retries and the budget

A retry sequence has to fit the remaining request budget, or it produces delay for a response nobody will read.

deadline, ok := ctx.Deadline()
if ok && time.Until(deadline) < minimumUsefulTime {
	return nil, fmt.Errorf("insufficient time budget: %w", ErrUnavailable)
}

The context deadline preempts every wait inside the retrier, so checking before you start turns a wasted attempt into an immediate, accurate failure.

The ipify client

The one worked example:

ipifyClient, err := ipify.New(
	ipify.WithHTTPClient(ipifyHTTPClient),
	ipify.WithTimeout(ipifyTimeout),
	ipify.WithURL(cfg.Clients.Ipify.Address),
)

It backs the monitoring server’s /ip route, and its comment carries the design decision:

// ipify is used only as a diagnostic (the monitoring /ip route); it is
// intentionally not part of the health checks.

A health check including a third-party address lookup means that when api.ipify.org has a bad afternoon, your service reports itself unhealthy, the orchestrator stops sending it traffic, and a service that was working perfectly is taken out of rotation by an unrelated outage.

Health-check what you need, not what you use, per page 13.

DNS caching

cache := dnscache.New(...)

client := httpclient.New(
	httpclient.WithDialContext(cache.DialContext), // before the round tripper
	httpclient.WithRoundTripper(m.InstrumentRoundTripper),
	httpclient.WithLogger(l),
)

The order is the trap from the ordering section above: the dialer has to be installed while the transport is still an *http.Transport, so appended to a base slice that already contains WithRoundTripper it silently does nothing and the cache is never consulted.

Go’s resolver does not cache, so a service making a thousand calls a second to one host performs a thousand lookups, and a slow resolver adds that latency to every call.

dnscache caches resolved hosts for a cache-wide TTL, collapses concurrent lookups for the same host into one, and dials the resolved addresses in preference order, interleaving address families so a dead IPv6 path is not exhausted before IPv4 is tried.

The package documentation states the trade: the cache uses one TTL and ignores the authoritative record TTLs, so a host whose address changes is stale for up to your TTL. In an orchestrated environment, where addresses are reassigned often, a TTL of a few seconds still removes most lookups.

Reach for it when the resolver is measurably a cost, rather than by default.

Reverse proxying

When the service fronts something else rather than calling it:

proxy, err := httpreverseproxy.New(targetURL, opts...)

httpreverseproxy wraps net/http/httputil.ReverseProxy with the same logging, redaction and trace propagation as the client. The article on it covers what a proxy has to refuse to do: header sanitisation, hop-by-hop headers, and the requests it must not forward.

What every outbound call needs

A timeout set explicitly, the request context passed, structured logging with redaction, a metric on duration and status, the trace ID forwarded, a retry policy matching the method’s idempotency with jitter, and a decision about circuit breaking, including that the mesh may own it.

Next: The Database Connection Is a Long-Lived Thing.