Table of contents
It is surprisingly easy for a service to leak secrets into its own logs. Someone dumps an inbound HTTP request to debug a flaky integration, and the Authorization header goes straight to disk. A JSON body with a password field lands in an error log. A connection string with the database password ends up in a stack trace. None of it is malicious; it is a side effect of logging the things you need to see when something breaks.
The redact
package in nurago
sits at that boundary, the moment just before text is written to a log, and removes the sensitive parts. What shapes its design is where it runs: it is the fallback redact function for the logged request and response dumps of nurago’s httpclient, httpserver, and httpreverseproxy packages, and the intended usage is on every log line. Every byte of every line passes through it, which sets a hard performance budget: on ordinary text it has to cost about as much as a copy, or someone will turn it off. This post is about how that budget is met, and about the correctness properties the speed is not allowed to compromise.
A naive redactor is a pile of regular expressions: one for headers, one for JSON keys, one for card numbers, and so on, each one a full scan of the string, with backtracking that can degrade badly on adversarial input, exactly the input a log redactor is most likely to see. redact spends its budget differently.
One table load per byte
The engine makes a single pass. For each byte it consults a 256-entry byte-class table (bulkTrigger in the source) that answers one question: could a redaction rule possibly start here? Almost every byte of a log line classifies as “no”, and for those the scan costs exactly one table load before moving on. When the run of safe bytes ends, the whole run is appended to the output in one bulk copy.
The remaining byte classes are filtered before any rule runs. Hard stops (", =, <, newline, digits) always hand over to the rules. Candidate bytes get a short inline prefilter first: a : only stops the scan when // follows (a URL that may carry userinfo credentials), an e only as eyJ at a word boundary (the start of every JSON Web Token), a - only as the -----B of a PEM boundary, and the nine first letters of vendor token prefixes only when the three bytes starting there spell a known prefix (ghp, sk-, AKI, …). Ordinary prose containing “skip” or “energy” does not leave the bulk-copy loop.
Digits get their own fast path ahead of the rule dispatch, because identifier-heavy log lines (trace IDs, UUIDs, ports, durations) are dominated by digit runs: a run glued to word characters is copied verbatim as part of an identifier, and only a free-standing run is checked as a candidate card number.
Only the survivors of all that fan out to per-rule dispatch, where each trigger byte maps to one rule class: sensitive HTTP headers and JSON keys, key=value pairs in URL-encoded form data, XML elements, URL userinfo passwords, JWT and JSON Web Encryption (JWE) compact tokens, vendor credential literals, Privacy-Enhanced Mail (PEM) private-key blocks, and card numbers. Each class can be disabled per instance with WithoutRules.
The rules’ lookahead is bounded too. A rule that inspects a candidate and rejects it does not consume input, so an unbounded forward search could be re-entered at every trigger byte, and a hostile line packed with -----BEGIN markers or unterminated XML comments would make redaction quadratic: a redaction denial of service (DoS) in the logging path. The engine bounds those searches (the inline PEM end-marker search to a 16 KB window, the XML comment and CDATA terminator search to 8 KB) so total work stays linear, and the fallbacks consume what they scan, in the safe direction: an unterminated private-key body is redacted through to the end of its value rather than left visible.
Keyword matching without allocating
Deciding whether a key name like X-Api-Key or dbPassword2 is sensitive happens thousands of times per second, so it cannot allocate. The matcher walks the key’s tokens in place, splitting on camelCase, snake_case, kebab-case, and acronym-run boundaries as it goes, lowercases each token into a fixed stack buffer, and looks it up via Go’s allocation-free map[string(bytes)] optimisation, so the keyword set stays an ordinary map without costing a string conversion per token. Only keys containing non-ASCII bytes take a slower normalising path, and its results are memoised in a bounded per-instance cache.
The output side is allocation-free as well when you want it to be: AppendTo writes into a caller-owned buffer (the repo’s benchmark measures zero allocations per call), and Pooled and BytesToString draw their scratch buffer from a sync.Pool.
re := redact.Default()
var dst []byte
for _, payload := range payloads {
dst = re.AppendTo(dst, payload)
logger.Info("request", "payload", string(dst))
}
The correctness layer
Speed is only worth having if the output can be trusted, in both directions: no leaked secrets, and no logs shredded into a wall of markers. Three properties carry that.
Boundary-exact key matching
Keyword matching is token-exact, never a substring search: apiKey, api_key, API-KEY, and APIKey all tokenise to api + key and match, but monkey does not match key and wildcard does not match card. Around the exact match sit a few deliberately bounded generalisations: a trailing digit run is stripped (password2, cvv2), a trailing plural s is retried against a short list of unambiguous roots only (tokens redacts; keys, a JSON Web Key Set array, stays visible), all-lowercase glued compounds match when they end in one of those roots (newpassword, awssecretkey), and a few two-word pairs match together where neither word is sensitive alone (firstName, nationalId, connectionString). House-style names outside these rules are added per instance with WithExtraTokens, and over-eager tokens are removed with WithoutTokens (keep amount and balance readable in fintech logs).
Convergent output
In real systems the same string often passes through more than one logging layer, so redacting already-redacted text must be safe. The property the package is built to hold: re-redacting output does not reveal more, output is byte-stable in a single pass on well-formed input, and on pathological, structurally ambiguous input it reaches a fixed point after at most one extra pass. The rules are written to preserve this: the marker is inert text, redaction does not consume structural bytes that would change how a second pass parses the surroundings, and the property is pinned by tests and a dedicated fuzzer.
Cards: over-redact by default, verify on request
Card detection defaults to deliberate over-redaction: any free-standing run of 13 to 19 digits (contiguous, or grouped by single spaces or dashes) matching a known network prefix and length is masked, even though that also catches unrelated identifiers that share the shape. Letting a real Primary Account Number (PAN) through is the worse failure. Grouped detection excludes a few legacy ranges whose prefixes collide with phone-number formats (“1 800 555 0199 1234”).
Callers that prefer fewer false positives can enable a Luhn-checksum gate, which then requires both the prefix match and a valid checksum, and as a side effect unlocks detection of short 12-15 digit Maestro numbers, too collision-prone to match on prefix alone. The gate is deliberately per instance, fixed at construction rather than exposed as a process-global toggle, so one component flipping it does not silently change what every other component logs:
re := redact.New(redact.WithLuhnCheck(true))
safe := re.String(rawPayload)
One entry point, one named bypass
All redaction runs through a Redactor, immutable after construction and safe for concurrent use. redact.Default() returns the shared zero-configuration instance (marker ***, all rules on, Luhn gate off); redact.New builds an independent one:
import "github.com/tecnickcom/nurago/pkg/redact"
safe := redact.Default().String(rawLogLine)
re := redact.New(
redact.WithMarker("#REDACTED#"), // custom placeholder
redact.WithExtraTokens("floof"), // company-specific key names
redact.WithoutTokens("amount", "balance"), // keep fintech fields readable
redact.WithoutRules(redact.RuleCards), // or disable a whole rule class
)
safe = re.String(rawPayload)
Disabling redaction outright is possible, but only explicitly: redact.InsecureNoRedaction is a ready-made pass-through for the redact-function options of the HTTP packages, named after the crypto/tls.InsecureSkipVerify convention to be conspicuous in review. An unset option falls back to Default() and a nil function is ignored, so redaction is not lost by omission, only by writing that name into a diff.
Where it fits, and where it stops
redact is not a Data Loss Prevention (DLP) product: it does not classify data across an organisation or police every channel it can leave through. It solves the smaller problem of a fast, predictable sanitisation step at the boundary where a service turns internal state into text that persists. It pairs naturally with structured logging (route your handler or HTTP dump output through it) and with the jwt
package, whose default responder logs issued tokens at debug level unless handed a redacting logger.
The scope limitation is structural: redaction is pattern-based, so it only catches shapes it can anchor on. A bespoke credential format it was never told about passes through (that is what WithExtraTokens is for), and so does anything without a structure the rules can anchor on: Go’s %+v rendering of a struct or map has none of the quoted-key, key=value, or header shapes, and a multipart form body separates each field name from its value across lines, so both must be redacted field by field or marshalled to JSON first. Treat it as a safety net over disciplined logging, not a licence to log recklessly. Used that way, it goes a long way towards turning “we leaked a token in the logs again” from a recurring incident into a rarity.