Table of contents
The service in this guide serves HTTP requests and nothing else. Growth past that
follows a few recognisable shapes: scheduled work, queue work, a second data
store, and secrets that cannot sit in a file, with one structural question
underneath all four: how bind stays readable while they arrive.
Work that has to happen on a schedule
The idempotency keys from page 15 accumulate, items nobody has touched in a year can be archived, and a cache warms better if something warms it. The shape is a goroutine that wakes on an interval, registered the way page 05 registers everything:
p, err := periodic.New(
30*time.Minute, // interval
5*time.Minute, // random jitter added to each pause
2*time.Minute, // deadline for each invocation
expireIdempotencyKeys,
)
if err != nil {
return fmt.Errorf("building the expiry job: %w", err)
}
p.Start(ctx)
The jitter spreads the fleet. Six replicas started by the same rolling deploy
run in lockstep, so an unjittered 30-minute job means six simultaneous bursts every
half hour. One caveat: the first call fires immediately and is not jittered by
default, so a fleet that starts together still fires together.
WithInitialJitter spreads that one too.
The per-call deadline stops a slow run overlapping the next. Without it, a job that occasionally takes 40 minutes on a 30-minute interval eventually has two copies running, which for a cleanup job means two deleters racing.
The context is the service context from bind, so cancelling it stops the
loop after the current invocation returns.
What changes with three replicas
Six replicas means six copies of every scheduled job. For an idempotent cleanup
that is six DELETE statements where one would do; for anything that sends an
email or charges a card, it happens six times.
Making it idempotent and letting it run everywhere is the simplest answer, and it
works when the operation is a DELETE ... WHERE expired or an upsert.
Running it as its own workload is the next one: a CronJob, or a deployment with
one replica, running the same binary with a different subcommand. Scheduling
becomes the platform’s problem, at the cost of a second deployable artefact.
The third is to elect a leader. Every replica tries to take a named lock and only the winner proceeds:
locker := mysqllock.New(db)
release, err := locker.Acquire(ctx, "expire-idempotency-keys", 10*time.Second)
if errors.Is(err, mysqllock.ErrTimeout) {
// Another replica is running it. Nothing to do.
return nil
}
if err != nil {
return fmt.Errorf("acquiring the expiry lock: %w", err)
}
defer func() {
rerr := release()
if rerr != nil {
l.ErrorContext(ctx, "failed releasing the expiry lock", slog.Any("error", rerr))
}
}()
// The critical section.
MySQL’s GET_LOCK and RELEASE_LOCK, a distributed lock without a new dependency
when you already have MySQL. One property makes it both usable and dangerous:
a named lock lives only as long as its owning session. Dropped by an idle
timeout, a proxy, a network failure or a server restart, the lock is released
while your code still believes it holds it. The package runs a keep-alive query
and treats a failure as a presumed loss, which converts a silent double-run into a
detectable error without eliminating the window.
So the rule for any lock across replicas: the critical section must survive being run twice. A lock reduces the probability of a double run; only idempotency removes the consequence. The article on this lock goes through every way the session can betray you, and applies to Redis and etcd locks too.
Work that arrives from a queue
Stock changes published to a downstream system, events consumed from another service, work too slow to do inside a request. A queue consumer has a different lifecycle from a handler and the same one as everything else in page 05: long-lived, registered with the wait group, watching the shutdown signals.
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-sc:
return
case <-ctx.Done():
return
default:
}
msgs, err := consumer.Receive(ctx)
if err != nil {
// A receive that failed because shutdown started is not an error.
if ctx.Err() != nil {
return
}
l.ErrorContext(ctx, "receiving messages", slog.Any("error", err))
time.Sleep(backoffSchedule.Next())
continue
}
for _, m := range msgs {
process(ctx, m)
}
}
}()
Acknowledgement is explicit. An HTTP response is the acknowledgement; a queue message is only removed when you delete it. Delete after the work succeeded, or a crash mid-processing loses the message.
At-least-once is the normal guarantee. A consumer that crashed between processing and acknowledging, or whose visibility timeout expired while it worked, sees the message twice, so every handler must be idempotent. That is page 15’s problem with a different trigger and the same solution.
There is no client waiting, so the deadline has to be constructed: give each message its own context with a timeout rather than the service context, which never expires.
Backpressure is yours. An HTTP server refuses connections when saturated; a consumer pulls more work than it can do, so the concurrency is a number you choose.
On shutdown, returning immediately abandons messages being worked on, which under at-least-once delivery means redelivery. Draining first is kinder to downstream systems and costs shutdown time. Either is defensible; acknowledging a message and then abandoning it is not.
kafka and sqs are the two implementations here, with sqs also exposing the
HealthCheck(ctx) error method page 13’s
aggregator takes directly.
A second data store
A cache in front of the database. Object storage for attachments. A key-value store for session state.
The technical part is short: another connection built in bind, another entry in
the wait group, and another health check if the service cannot work without it,
for which the redis, valkey and s3 packages all expose
HealthCheck(ctx) error.
The hard part is who owns consistency between them. By default nobody does. No transaction spans MySQL and Redis, so a write to one can succeed while a write to the other fails and the two disagree until somebody notices. Three arrangements work, in order of cost.
One store is authoritative and the other is derived. The database holds the
truth and the cache holds a copy with a TTL. A failed cache write is a log line
rather than an error, because the cache is allowed to be wrong for a bounded
time. Page 16’s sfcache is this
arrangement in-process.
Write to one, derive the other asynchronously. In the same transaction as the write, insert an outbox row describing what to publish, and let a separate process read the outbox and publish. The two are eventually consistent and the window is observable, because the outbox has a measurable depth. This is the pattern that does not lose events.
Genuinely need both atomically. Then they are one store, and the second one is a mistake to unwind.
The arrangement to avoid is writing to both in sequence and hoping:
// Wrong: two writes, no relationship between them.
err := repo.Create(ctx, it)
if err != nil {
return err
}
// If this fails, the item exists and the event does not.
// If this succeeds and the transaction later rolls back, the reverse.
return publisher.Publish(ctx, itemCreated{ID: it.ID})
It works in testing, because the second call rarely fails there.
Secrets that must not sit in a configuration file
Page 04 put secrets in environment variables. That is the right first answer, with two limits: the value is set at start and cannot be rotated without a restart, and it is visible to anything that can read the process environment. For a secret that rotates, fetch it at use time with a cache in front:
cache, err := awssecretcache.New(ctx, 128, 5*time.Minute)
if err != nil {
return fmt.Errorf("building the secret cache: %w", err)
}
value, err := cache.GetSecretString(ctx, "prod/inventorysvc/upstream-api-key")
Single-flight, bounded in size, cached for a TTL. The secret store stays off the
request path. The TTL is also a rotation window: afterwards, callers get the
old value for up to that long and every one of them fails against the new
credential, so a rotation webhook should call Remove.
Two things it does not solve. The credential for reaching the secret store has to
come from somewhere, which in a cloud environment is an instance role rather than
another secret. And a secret in memory is readable from a heap dump, one more
reason page 17 keeps /pprof off the
public listener.
Keeping bind readable
Every section above adds lines to one function: the most valuable file in the service, and the easiest to ruin.
The reference implementation uses named helpers, visible already at the size it ships at:
func bind(cfg *appConfig, appInfo *jsendx.AppInfo, mtr instr.Metrics, wg *sync.WaitGroup, sc chan struct{}) bootstrap.BindFunc {
return func(ctx context.Context, l *slog.Logger, m metrics.Client) error {
jsx := jsendx.NewJSXResp(httputil.NewHTTPResp(l))
logRedactor := newLogRedactor()
httpClientOpts := []httpclient.Option{ /* ... */ }
ipifyClient, err := newIpifyClient(cfg, httpClientOpts)
if err != nil {
return err
}
serviceBinderPrivate, serviceBinderPublic, statusHandler, err :=
bindServiceHandlers(ctx, cfg, appInfo, jsx, l, mtr, wg, sc)
if err != nil {
return err
}
// ... three servers
}
}
with newIpifyClient, bindServiceHandlers, newDatabases, newDatabase,
newLogRedactor and startServiceServer below it.
bind reads as an outline. Its body is a sequence of named steps in dependency
order, and anything that takes more than a few lines becomes a helper with a name
that says what it produces.
Those helpers return values rather than mutating shared state. newIpifyClient
returns a client. newDatabases returns the connections and the health checks.
The dependency graph is visible in the return values, so reading the outline
tells you what depends on what.
When the file gets long, split it by subsystem and not by type: bind_http.go,
bind_data.go, bind_workers.go. Splitting into bind_clients.go and
bind_servers.go puts one subsystem’s parts in two files.
At some size a dependency-injection framework starts to look appealing. Resist it. It buys less typing and costs the wiring being readable Go: the graph resolves at runtime, and errors surface at startup as reflection failures rather than at compile time as type errors.
What to delete
The scaffold ships with things a real service does not need.
The ipify client, unless the /ip diagnostic earns its place: it is an
outbound call to a third party. Remove newIpifyClient, the WithIPHandlerFunc
option, the clients.ipify config section and its defaults, and the smocker
container from the integration stack.
The example collector in internal/metrics, with the
IncExampleCounter("START") call in bind. Keep the shape and give it a metric
you will look at.
The database, if the service has none. Set db.enabled to false and
page 04’s conditional validation stops
asking for DSNs. Delete internal/db, newDatabases, the resources/db tree
and the MySQL and Flyway containers when you are sure.
The /uid handlers in httphandlerpub and httphandlerpriv, which are
placeholders. Part V
replaced the public one; the private one is still there.
The custom redactor, if the default suits you. newLogRedactor exists to
show how to depart from redact.Default(), and deleting it plus the
WithRedactFn options that reference it restores the default.
Page 06 was clear that redaction is never lost
by omission, so this is a safe deletion.
The unused default routes. WithEnableAllDefaultRoutes() on the monitoring
server is convenient and it enables /ip, which you may have just deleted. List
what you want.
A component whose only consumer is the example that demonstrates it is scaffolding. Where that is unclear, the health check question from page 13 is a reasonable proxy: something the service genuinely needs is something whose absence should make it unready.
Where to go from here
The guide covered one service, end to end, from a bound socket to a deployment manifest. What it deliberately left out is on page 01, and the largest omission is authentication, whose starting points are on page 17.
For a specific package’s engineering in more depth than a guide page allows, the articles go further: the HTTP server’s edges, redaction on a performance budget, what a retry loop must decide, backoff without the overflow footgun, single-flight caching, filtering untrusted client queries, and a distributed lock in MySQL.
The reference documentation is on pkg.go.dev, and nurago.org has the per-package pages, the short-form documentation, and a comparison against other libraries.
What is left after twenty pages is smaller than any of it. Most of the interesting engineering in a service sits outside the handler: in what happens before the first request, what the request carries with it, and what happens after the last one.