From singleflight to a Production Cache: a Gap Analysis

What golang.org/x/sync/singleflight gives you, what a production cache still needs, and how the nurago sfcache package closes each gap: TTL caching, context provenance, failure-anchored stale windows, and bounded O(log n) eviction.

- Nicola Asuni

Table of contents

Some lookups are too expensive to run on every request: a Domain Name System (DNS) resolution, a secret from a vault, slow remote metadata. Caching them is the natural move, and it brings a well-known hazard: when a hot value expires, a crowd of goroutines can rush the upstream at once, the cache stampede. In Go, the usual remedy is golang.org/x/sync/singleflight, which lets one goroutine do the work while the other forty-nine wait for its result.

singleflight is a good primitive, and it is only a primitive. This post is a gap analysis: what the primitive provides, where its remit ends, and how the sfcache package in nurago closes each gap.

The baseline, and where its remit ends

singleflight.Group.Do(key, fn) runs fn once per key at a time; every caller that arrives while the call is in flight receives the same value, the same error, and a flag saying the result was shared. That is deduplication in space (across concurrent callers) but not in time: the moment the call completes, the group forgets it, and the next caller runs fn again.

The remit ends there, deliberately:

  • No memory. Results are shared among concurrent callers, never cached.
  • No expiry. With nothing retained, there is nothing like a time to live (TTL) to enforce.
  • No bound. No capacity concept, because no capacity is used.
  • No context. Do does not take one. fn typically closes over the first caller’s context, and whatever error it produces, including that caller’s cancellation, is handed to every waiter.
  • No failure policy. An error is a result like any other; what to do about a flaky upstream is your problem.

Each of these is the right scope for a primitive and a gap for a production cache. sfcache closes them one at a time.

Gap one: remembering results, on a TTL

import "github.com/tecnickcom/nurago/pkg/sfcache"

cache := sfcache.New(
    func(ctx context.Context, key string) (Secret, error) {
        return fetchSecretFromVault(ctx, key) // the expensive lookup
    },
    sfcache.Config{
        Size: 1000,            // maximum number of values held
        TTL:  5 * time.Minute, // time to live of a successful value
    },
)

val, err := cache.Lookup(ctx, "db/password")

The cache is generic, so Lookup returns typed values with no assertions, and a warm hit takes only a read lock. Successful values are cached for the TTL. Errors are shared with the callers coalesced onto the same lookup but never cached, so the very next call retries: there is no negative caching to let a transient failure poison a key for five minutes. What a failure leaves behind is an already-expired residue entry, reclaimed when its room is next needed, and not at the price of a live value (gap four explains why).

Two quiet decisions carry this section. Expiry is measured against the monotonic clock, so a stepped system clock or a Network Time Protocol (NTP) adjustment does not make entries expire hours early or live forever (one caveat: on most platforms the monotonic clock does not advance during system suspend, so TTLs stretch by the suspended time). And entries are immutable once stored: an update replaces the whole entry rather than mutating it, so an entry read under the lock stays valid after the lock is released, which is exactly what keeps the fast path a read lock and nothing more. Values are shared by reference; treat them as read-only.

Gap two: the shared failure, and whose context cancelled

The scenario: goroutine A starts the lookup and becomes the producer; B and C coalesce as waiters. A’s client disconnects, A’s context is cancelled, and the lookup returns a context error. With plain singleflight, that cancellation is the shared result: B and C, whose contexts are live and who cancelled nothing, receive A’s context.Canceled.

sfcache treats an error’s provenance as part of the result. When a lookup fails, it asks whether the error is the producing context’s own (an errors.Is test against that context’s error, so an upstream error that merely wraps context.Canceled counts once the producer’s context has in fact ended). If it is, the cache publishes nothing at all: no entry, no flight. The waiters wake to a terminal state and re-run the lookup under their own live contexts. The price of not sharing a cancellation is one extra lookup. A genuine upstream error, by contrast, is shared with every waiter, because it is a fact about the key rather than an artefact of one caller giving up.

A caller receives ErrLookupAborted only when its own context ends: while parked on someone else’s flight, or arriving after its context already died. The sentinel wraps the context error, so errors.Is with context.Canceled or context.DeadlineExceeded keeps working. One asymmetry is worth knowing: a fresh cached value is served even to a dead context, since no work is needed to produce it, but a stale value is not, because serving stale first requires attempting a refresh.

Gap three: serving through the outage

For keys where a slightly stale answer beats an error, the cache offers stale-if-error, in two windows that differ in what anchors them:

cache := sfcache.New(lookupFn, sfcache.Config{
    Size:              1000,
    TTL:               5 * time.Minute,
    MaxStale:          30 * time.Second, // anchored to the value's expiration (RFC 5861)
    MaxStaleOnFailure: 2 * time.Minute,  // anchored to the first failed refresh
})

MaxStale is the classic stale-if-error of RFC 5861: when a refresh of an expired key fails, the last known good value is returned, with a nil error, until the value’s original expiration plus MaxStale. Because the window hangs off the expiration, a key idle for longer than TTL + MaxStale gets no protection at all; the window closed before anyone needed it.

MaxStaleOnFailure fixes that for rarely read keys by anchoring the window to the first failed refresh instead, however long the key sat idle. It is anchored exactly once: later failures keep serving the value until that deadline but do not push it back, so a permanently failing upstream does not make a value immortal. When both windows are set, the later deadline wins.

A revived entry is stored still expired, so every subsequent call attempts a fresh lookup and the first success replaces it: recovery is immediate, not deferred to the end of a window. An entry whose last outcome was an error is not served stale. And the stale window takes precedence over the gap-two retry: during an outage, the waiters behind a cancelled producer get the stale value rather than another round trip to a dead upstream.

Gap four: bounded memory that evicts the right thing

Config.Size bounds the values held, not the keys in play. Lookups in flight live in their own map and hold no cache entry, so a burst of requests for cold keys does not evict the warm working set, and nothing in flight is evicted. (Len can temporarily exceed Size by the lookups in flight, plus failure residue, plus at most one stale revive; the excess is reclaimed as flights complete and the next value is stored.)

Eviction is a heap discipline, not a scan. Every stored entry sits in exactly one of three deadline-ordered queues: live values ordered by expiration, revived stale values ordered by their anchored deadlines, and error residue. The victim of an eviction is the head of one of those queues, so a store at capacity costs O(log Size) under the write lock, and a store that is not allowed a victim discovers that in constant time. Nothing on the lookup path walks the cache; the only linear pass is the explicit PurgeExpired, which is rarely worth calling and forfeits stale protection for everything it purges.

Which head a store may take depends on what the store adds, in three levels. A failed lookup stores no value, so it may reclaim only entries nobody can be served (residue, and values past any stale window); when none exists, it leaves the cache over capacity rather than take something better. A stale revive may additionally take a value that is itself only being served stale, preferring the one no caller has asked for during the outage. Only a successful lookup may displace a valid entry, and it takes the one closest to expiring. The invariant is short: a lookup that is merely attempted, and may yet fail, does not cost the cache a live value.

Gap five: policy you can read

All of the above is declared at the construction site. Config holds the settings that do not depend on the cache’s key and value types; functional options carry the ones that do, so their type parameters stay inferable:

cache := sfcache.New(lookupFn, sfcache.Config{Size: 256},
    sfcache.WithTTLFunc(func(_ string, r Record) time.Duration {
        return r.TTL // the data carries its own freshness
    }),
)

WithTTLFunc suits data with intrinsic freshness, such as DNS records with authoritative TTLs or credentials with a known expiry. The zero Config is valid and instructive: it yields a single-entry cache that caches no value and only coalesces concurrent lookups, which is to say roughly singleflight itself, plus the context rules of gap two.

Local by design

sfcache is a local, in-process cache: the right tool for deduplicating and caching expensive lookups inside one service instance, and within nurago it is the foundation under dnscache and awssecretcache . It is deliberately not distributed: each process keeps its own copy, with no network hop and no external dependency, and cross-instance consistency is out of scope.

The point of the gap analysis is that none of these gaps is exotic. Teams that wrap singleflight tend to meet the same five, usually one incident at a time: results forgotten, cancellations shared, outages unbuffered, memory unbounded, policy implicit. Closing them well is the difference between a primitive and a cache you can leave alone.