Filtering Untrusted Client Queries in Go: Threat Model First

What a hostile filter query can send, and how the nurago filter package neutralises it: bounded input, RE2 linear-time matching, exact int64/uint64 comparison past 2^53, strict schema validation, and boxing-free in-place evaluation.

- Nicola Asuni

Table of contents

A filter= query parameter is a common way to let clients narrow down a list endpoint: the client asks for “users named Doe, at most 42 years old, in England or France”, and the server has to turn that into a predicate over an in-memory slice. It looks like a small convenience, but accepting that parameter means exposing a little expression language to the open internet, and a sensible way to design one is to start from what a hostile client can send. The filter package in nurago is built that way, so this post walks the threat model first and the machinery second.

What a hostile client can send

Treat the filter parameter as an attacker-controlled program. Its instruction set is small, but each instruction class has a failure mode:

  1. Bulk. A multi-megabyte payload that costs memory and CPU merely to decode before you can reject it.
  2. Breadth and depth. Hundreds of rules, or field selectors dozens of segments deep, multiplying evaluation cost per element. Against a recursive element type there are infinitely many distinct valid selectors, so an unbounded per-selector cache is also a memory leak on demand.
  3. Pathological patterns. A regular expression crafted for catastrophic backtracking, the classic regular-expression denial of service (ReDoS).
  4. Corrupting numbers. An integer just past 2^53. Decoded through float64, as encoding/json does by default, it silently rounds to its neighbour, so a filter for one 64-bit ID can match a different record.
  5. Malformed shape. Misspelled fields, omitted keys, arrays where scalars belong. These should be a clean HTTP 400, not a silent empty result indistinguishable from “no data”.

The grammar under attack is deliberately small: rules arrive as JSON [][]Rule, outer list AND, inner list OR, so [A, [B, C], D] means A AND (B OR C) AND D. Ten operators (regular expression, equality, case-fold equality, prefix, suffix, contains, and the four orderings), each negatable with a leading !, plus dot-path selectors like address.country for nested structs. Wiring it into a handler:

import "github.com/tecnickcom/nurago/pkg/filter"

// WithFieldNameTag maps JSON-style selectors ("address.country") to struct tags;
// without it, selectors resolve against Go field names, case-sensitively.
p, err := filter.New(filter.WithFieldNameTag("json"))

rules, err := p.ParseURLQuery(r.URL.Query()) // reads ?filter=<json>
if err != nil {
    // errors.Is(err, filter.ErrInvalidFilter) -> respond 400, log the detail
}

// users already scoped to what this request is allowed to see
n, total, err := p.Apply(rules, &users)

Apply removes non-matching elements in place and reports the kept length and the total match count. Now, class by class, how the input is neutralised.

Bulk, breadth, and depth: bounded before they cost anything

The limits are on by default; a bare New() is the hardened configuration:

  • The raw payload is capped at 64 KiB (WithMaxFilterBytes) and rejected before JSON decoding, so an oversized body does not buy a large allocation.
  • At most 8 rules (WithMaxRules). The same limit bounds two counts: the number of AND groups and the total rules summed across all OR groups, so a swarm of single-rule groups does not slip past a per-group view of it. Checked at parse time and again at apply time.
  • Any string rule value, including a regular-expression pattern and any named string type, is capped at 4 KiB (WithMaxValueLength), checked before the pattern reaches regexp.Compile.
  • Field selectors are capped at 32 dot-separated segments (WithMaxFieldDepth), rejected before they are resolved or cached. The resolved-path cache itself is additionally capped internally, so a stream of distinct selectors against a recursive type recomputes past the ceiling instead of growing memory without limit.

For pathological patterns, the answer is structural rather than a limit: Go’s regexp is the RE2 engine, which matches in time linear in the input with no backtracking, so the catastrophic-backtracking class of attack does not apply. What remains is input size, and input size is exactly what the caps above govern.

Numbers: exact past 2^53

float64 has 53 bits of mantissa; every integer above 2^53 is rounded. 64-bit identifiers, nanosecond timestamps, and balances in minor units all live up there, and once widened, two different values can compare equal. The package closes this at both ends. On the way in, JSON numbers are decoded via json.Number and converted to the narrowest exact type: int64 when it fits, uint64 for values above math.MaxInt64, float64 otherwise, and a number no Go type can represent (say 1e400) is rejected outright. On the comparison side, values are normalised into a type that widens an integer only when it has to:

// numeric is a normalized numeric value preserving the exactness of integers.
// Integers are kept as int64/uint64 (not widened to float64) so that values
// beyond 2^53 still compare correctly for equality and ordering.
type numeric struct {
	kind numericKind
	i    int64
	u    uint64
	f    float64
}

When neither operand is a float, comparison stays in integer space, with a dedicated signed/unsigned path so a large uint64 and a negative int64 order correctly rather than through a lossy common type. Only a genuine float32/float64 operand drops the comparison to floating point, where a NaN is reported as “no ordering applies” instead of the usual nonsense. Exactness holds for integer literals: a client who writes 9.007199254740993e15 has sent a float and gets float semantics. One deliberate overload rounds it out: the ordering operators applied to a string, array, slice, or map compare its length, which is usually what a client means by tags > 3.

Shape: strict schema validation

Everything the JSON parser accepts is held to the package’s published schema, and everything else is an explicit error, not a silent no-op. A filter must contain at least one AND group, and no group may be empty (an empty disjunction is always false and would silently drop every element, so it is treated as malformed rather than as a “match nothing” request). Every rule object must carry all three of field, type, and value; an omitted key is rejected, and is distinguished from an explicit null, which is a legitimate nil reference. Array and object values are rejected, keeping the untrusted surface to scalars, and trailing data after the top-level JSON value is rejected too.

Misspelled fields get the same treatment. A selector that cannot be resolved against a concrete element type (no such field, descent into a non-struct, an unexported target, or over-depth) is rejected with ErrInvalidFilter before any element is touched, deterministically even on an empty slice, so a typo answers with a 400 instead of a plausible-looking empty list. Only conditions knowable per element stay per element: a nil pointer along the path, or an element of a []any slice whose concrete type lacks the field, makes that element a non-match rather than failing the request.

Every error attributable to the client’s filter is wrapped in ErrInvalidFilter, so a handler can test errors.Is, return a flat 400, and log the detail server-side rather than echoing client input or internal type names back to the caller.

The machinery: evaluate a reflect.Value without boxing the element

Filtering is reflection-driven, since the element type is unknown at compile time. The tempting implementation pulls each element out with Value.Interface() and type-switches on the resulting any, which heap-allocates a copy of the element every time. Here the evaluator interface takes the reflect.Value itself:

type evaluator interface {
	// Evaluate returns true when the value satisfies the evaluator condition.
	//
	// The value is passed as a [reflect.Value] so that filtering large slices does not box
	// every element and field into an any: elements are never boxed, and string and numeric
	// operands are read directly from the reflect.Value without allocating. A field is boxed
	// only by the deep-equal fallback of the equality evaluators, which is reached when the
	// rule's reference value is neither numeric, nor a string, nor nil (a boolean, or an
	// uncomparable map or slice value). An invalid Value (the zero [reflect.Value])
	// represents a nil or absent operand.
	Evaluate(v reflect.Value) bool
}

The matching pass iterates with slice.Index(i) and hands that reflect.Value straight through; only the one field a rule reads is materialised, via v.Int(), v.String(), and friends. Filter a thousand fat structs on one small field and you read that field a thousand times without copying the structs, the caveat in the comment aside. Pointer and interface leaves are unwrapped to their concrete value first, with the indirection bounded, so a cyclic reference in the data resolves to a nil operand instead of hanging Apply.

Match fully, then commit

In-place compaction is where a filter can hurt its caller: fail halfway through a naive read-and-overwrite loop and the slice is torn. The pass is therefore split. Matching runs to completion first, collecting matched indices and mutating nothing; only then does compaction begin:

// Commit phase: only reached after a fully successful pass, so a mid-iteration
// error above can never truncate or clobber the caller's slice.

An error leaves the input untouched. And because evaluators, regular expressions, and field paths are all compiled and resolved up front, before the slice is examined, a misconfigured rule fails early, and both a Processor and a compiled rule set are safe to share across goroutines (the slice being filtered is not).

The boundary worth stating out loud

filter filters; it does not authorise. The grammar lets a client select and compare any exported field of the elements, including nested ones, so match counts and result windows can probe values. The contract is explicit: apply it only to data the requesting user is already entitled to see, pre-filtered to their permitted records, with unreadable fields projected away, before calling Apply. Get that boundary right and the package gives you expressive, validated, allocation-frugal filtering; get it wrong and no comparison logic will save you, because the vulnerability is in what you handed it.