Tests That Would Have Caught It

Four layers, each catching a class of bug the others cannot, ordered by how fast they run and how much they need.

Unit tests

make test

expands to:

go test -shuffle=on -tags=unit,benchmark -covermode=atomic \
  -bench=. -benchtime=1x -race -failfast \
  -coverprofile=target/report/coverage.out -v ./...

-race finds concurrent access to shared state at runtime, the failure page 02 warned about, which otherwise appears in production under load and nowhere else. It costs several times the runtime, worth paying where handlers run concurrently by definition.

-shuffle=on randomises test order, so a test that only passes after another one becomes a failure you can see. A failure may then not reproduce without the seed, and the output prints it.

-failfast stops at the first failure: good locally, arguable in CI.

-benchtime=1x runs each benchmark once, for compilation and execution coverage rather than measurement. Real measurement is make bench, without -race and without coverage, because both distort the numbers.

Doubles

The service tests run against a hand-written fake. That is what the Store interface on page 14 is declared for:

// internal/item/item_test.go

// fakeStore is a hand-written Store used by the service tests. The service
// declares the Store interface, so its rules are testable without a database.
type fakeStore struct {
	item    *Item
	items   []Item
	created *Item
	limit   uint
	err     error
}

func (f *fakeStore) List(_ context.Context, limit uint) ([]Item, error) {
	f.limit = limit

	return f.items, f.err
}

Canned answers in fields, and the fields also record what the service passed down. That is how the test that a zero limit becomes DefaultPageSize asserts anything at all: it reads f.limit afterwards. A fake whose fields are functions instead lets each test set behaviour next to its assertion, and is worth the extra weight once one interface needs different behaviour per call. Generated mocks, through make generate, are for large interfaces and call-order assertions.

The repository is the layer with no interface below it, so its tests drive go-sqlmock and assert on the SQL itself:

// internal/item/repository_test.go

mock.ExpectBegin()
mock.ExpectExec(regexp.QuoteMeta(sqlInsertItem)).
	WillReturnError(&mysql.MySQLError{Number: mysqlErrDupEntry, Message: "duplicate entry"})
mock.ExpectRollback()

err := repo.Create(t.Context(), newItem())

require.ErrorIs(t, err, ErrConflict)

The expectation is the assertion. This one covers the branch a real database will not produce on demand: error 1062 arrives, the transaction rolls back, and the caller receives ErrConflict rather than a driver error. Page 15 built that mapping, and its pair asserts NotErrorIs(err, ErrConflict) for any other failure, so a broader errors.As that swallows unrelated errors fails here.

regexp.QuoteMeta(sqlInsertItem) matches the statement the repository actually holds rather than a hand-copied pattern, so the expectation cannot drift from the constant. newTestRepository points the read and write handles at the same mock and asserts ExpectationsWereMet in a t.Cleanup, turning a statement the code stopped issuing into a failure.

The seams that exist for testing

Page 03 pointed at exitFn and bootstrapFn, page 04 at validatorNewFn, page 12 at WithSQLOpenFunc. Each is a function-shaped hole where something untestable would otherwise be hard-coded, and each is a small readability cost paid for a whole class of test.

testutil covers the rest: forcing I/O failures on demand, capturing process output, and normalising time-variant values so an assertion on a response containing a timestamp is possible at all.

Coverage

make coverage

renders the profile as HTML. Coverage tells you what was executed rather than what was asserted, so a test that calls a function and checks nothing counts the same as one that checks everything. Read it as a map of what is untested: uncovered lines cluster in the error paths, the ones that run during an incident.

Handlers, with and without a server

The handler tests in the example call the handler directly, with a recorder and a request carrying the path parameters the router would have set:

// internal/httphandlerpub/httphandlerpub_test.go

// newItemRequest builds a request carrying the httprouter path parameters the
// item handlers read.
func newItemRequest(t *testing.T, method, target, body string, params httprouter.Params) *http.Request {
	t.Helper()

	ctx := context.WithValue(t.Context(), httprouter.ParamsKey, params)

	req, err := http.NewRequestWithContext(ctx, method, target, strings.NewReader(body))
	require.NoError(t, err)

	return req
}

httputil.PathParam reads the identifier out of the request context, so a test that wants /items/:id to resolve puts the parameters there itself. That is the cost of calling the handler directly, and what it buys is a table of thirteen bodies against thirteen status codes with no server anywhere:

{name: "accepts a whole quantity written with a fraction", body: `{"name":"test","quantity":3.0}`, wantCode: http.StatusCreated},
{name: "rejects a fractional quantity",                    body: `{"name":"test","quantity":3.5}`, wantCode: http.StatusUnprocessableEntity},
{name: "rejects a quantity above the column bound",        body: `{"name":"test","quantity":4294967296}`, wantCode: http.StatusUnprocessableEntity},
{name: "fails with an unknown field",                      body: `{"name":"test","unknown":1}`, wantCode: http.StatusBadRequest},

Every branch of page 15’s itemQuantity, one line each. This is the layer where that kind of coverage is cheap, and the arrangement worth copying: the cases are data, the assertion is one line, and adding a body somebody sent in production is adding a row.

The layer above it

What the recorder cannot test is the router, the middleware chain and the real encoder. httpserver.New binds the listener before returning (page 07) and :0 gets an ephemeral port from the operating system (page 02), so the real server runs inside go test:

// startTestServer binds the real server on an ephemeral port and returns its
// base URL. It exercises routing, the middleware chain and response encoding
// without a container.
func startTestServer(t *testing.T, svc ItemService) string {
	t.Helper()

	srv, err := httpserver.New(t.Context(), New(svc, slog.Default()),
		httpserver.WithServerAddr("127.0.0.1:0"),
		httpserver.WithRequestTimeout(5*time.Second),
		httpserver.WithoutRouteLogger(),
	)
	if err != nil {
		t.Fatalf("building the server: %v", err)
	}

	srv.StartServer()

	t.Cleanup(func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()

		_ = srv.Shutdown(ctx)
	})

	return "http://" + srv.Addr().String()
}

127.0.0.1:0 rather than :0, so nothing outside the machine can reach a test server, and Addr() reports what the operating system assigned, so parallel tests never collide. The shutdown context comes from context.Background() for page 05’s reason: a context derived from the test’s would already be cancelled. WithoutRouteLogger() keeps the output readable.

Real TCP, real HTTP parsing, the real router, the real middleware chain and the real response encoder, in milliseconds, under t.Parallel(), in the same go test invocation as everything else. Four things become testable that the recorder cannot reach.

Routing, including the not-found and method-not-allowed handlers from page 07. A route registered with a lowercase method fails at httpserver.New, and a path that matches nothing produces the 404 a client would actually receive.

The registration switch from page 14. Passing a nil service and asserting that /items is a 404 while /uid is a 200 is the one test that proves the feature flag removes the routes rather than breaking them.

Path parameters resolved by the router rather than injected by the test, the half newItemRequest has to fake.

The whole middleware chain, so an ordering mistake that breaks the response shape fails here.

It still uses a fake service, so it exercises no SQL, and it runs in one process, so it tests no configuration loading, no container image, no migration and no network policy. The next layer covers those.

Container integration

docker compose -f docker-compose-int.yml up

Five containers.

The service, built from the shipping image and configured through environment variables from target/<app>.cfg.env, page 04’s override path exercised for real. MySQL, seeded with create.sql. Flyway, applying the migrations from resources/db/mysql/, the only layer where a migration runs. smocker, a programmable mock standing in for api.ipify.org, so the suite has no third-party dependency and can be made to return errors on demand. And the test container.

Ordering them is two mechanisms, and the file uses both:

    # The service waits for the migrations to complete, not just to start:
    # it serves endpoints that read and write the migrated tables.
    depends_on:
      inventorysvc_smocker_ipify:
        condition: service_started
      inventorysvc_mysql:
        condition: service_healthy
      inventorysvc_flyway_mysql:
        condition: service_completed_successfully
    entrypoint: [
        "/usr/bin/dockerize",
        "-wait", "tcp://inventorysvc_mysql:3306",
        "-wait", "tcp://inventorysvc_smocker_ipify:8081",
        "/usr/bin/inventorysvc"
    ]

A bare depends_on waits for a container to start rather than for the process inside it to be ready, so each entry carries a condition. service_healthy reads the MySQL container’s own mysqladmin ping healthcheck, and service_completed_successfully holds the service back until Flyway has exited zero, because the item endpoints read a table that does not exist until it has.

dockerize then waits on the ports from inside the container. The overlap is deliberate: conditions are Compose’s view of another container’s state, and the -wait is this process’s own precondition, which still holds when the service runs somewhere Compose is not. Without either, page 12’s startup ping fails against a MySQL that is still initialising and the service exits 2: correct behaviour producing a flaky test. A service that started anyway and retried forever would paper over it here and fail differently in production.

What only this layer catches

The migration applies to a real database, cleanly, on top of what came before.

The SQL is real SQL. sqlmock matches a pattern against a string and proves nothing about whether the database accepts it, and MySQL and PostgreSQL agree on page 16’s row comparison and disagree on plenty else.

Configuration loading runs end to end, the container image is the one that ships, and the wiring in bind.go executes for real, the only place outside production where the whole graph is built.

Contract tests

make apitest

which fans out to venom suites, one per listener per environment:

resources/test/venom/{dev,int,prod}/{public,private,monitoring}/

The matrix is the point: the same suite runs against the monitoring, private and public listeners in each environment. That is how you find an endpoint answering on a port it should not be, and it makes page 17’s exposure decision tested rather than asserted. The cases are YAML, readable by someone who does not write Go:

# resources/test/venom/int/public/api.yaml

- name: create_item
  steps:
    - type: http
      ignore_verify_ssl optional: true
      method: POST
      url: '{{.inventorysvc.url}}/items'
      headers:
        Content-Type: application/json
      body: '{"name":"venom-widget","quantity":5}'
      vars:
        id:
          from: result.bodyjson.id
      assertions:
        - result.statuscode ShouldEqual 201
        - result.headers.Location ShouldStartWith /items/
        - result.bodyjson.name ShouldEqual "venom-widget"

- name: create_duplicate_item
  steps:
    - type: http
      ignore_verify_ssl optional: true
      method: POST
      url: '{{.inventorysvc.url}}/items'
      headers:
        Content-Type: application/json
      body: '{"name":"venom-widget","quantity":9}'
      assertions:
        - result.statuscode ShouldEqual 409

vars captures the identifier out of the first response and the later cases spend it as {{.create_item.id}}, so the suite creates, reads, lists, deletes and then asserts that reading and deleting again are both 404. That is a sequence, and it is the shape the unit tests cannot have: the 409 is produced by the unique index on a real MySQL, not by a mock returning ErrConflict.

The list assertions read on the seeded rows:

        - result.bodyjson ShouldHaveLength 4
        - result.body ShouldContainSubstring '"name":"seed-bolt"'

Four, because Flyway loaded three fixture rows from resources/db/mysql/int/V1001__example_table.sql and this suite created one. That coupling is why the integration entrypoint runs the suites in a fixed order:

# Run tests. The API tests come first: they assert on the seeded rows, and the
# schemathesis run that follows creates and deletes rows of its own.
DEPLOY_ENV=int make apitest openapitest

A count assertion against a database another suite is writing to is a flaky test waiting for a scheduler change. Stating the dependency in the script that owns it is cheaper than making every assertion count-independent.

result.bodyjson.name and not result.bodyjson.data.name. The item handlers answer with httputil.HTTPResp.SendJSON, which writes the value it is given, while the JSend envelope from page 10 reaches the monitoring listener’s routes, because bind.go wires the jsendx handlers there. Two shapes on two listeners is a decision worth making deliberately rather than discovering from an assertion path that does not resolve.

The prod suite earns its place: a read-only subset run against production after a deploy is a smoke test under real conditions, and it turns “the deployment succeeded” into “the service works”.

make ping is the smallest version of it:

if [ "200_$(VERSION)_$(RELEASE)_" != "$(shell curl --silent --insecure \
  '$(MONITORING_URL)/ping' | jq -r '.code,.version,.release' | tr '\n' '_')" ]; then exit 1; fi

It asserts that the code, the version and the release match what was just built, which catches the deploy that reported success and left the old version running. make rping retries for a minute while the rollout finishes.

Spec fuzzing

make openapitest

runs schemathesis against each document:

schemathesis run \
  --request-timeout=2000 \
  --max-examples=100 \
  --max-time=10 \
  --url='${API_TEST_URL}' \
  ${OPENAPI_FILE}

It reads the OpenAPI specification, generates requests from the schemas, sends them, and checks the responses against what the specification promised. Four kinds of finding come out of it.

Undocumented status codes. The specification lists 201, 409 and 422, the service returns 400 for a malformed body, and the specification is wrong. The usual first finding, and a real one: a generated client does not know 400 is possible.

Responses that do not match their schema, such as a required field that is absent, or a string where the schema says integer.

Inputs that produce a 500. Boundary values, empty strings, very long strings, Unicode, deeply nested structures. The specification is what tells the fuzzer which inputs are supposed to be valid.

Contradictions, such as maxLength: 128 on the item name against a service that refuses at 100. The name bound lives in the schema, in the column and in MaxNameLength: the duplication from page 14, and the reason this layer exists.

Configuring the checks

Which checks run is in schemathesis.toml rather than on the command line, and the file says why:

# The check selection lives here rather than on the schemathesis command line:
# a --checks flag on the command line replaces the settings below, including
# the per-operation overrides, so the two cannot be combined.

[[operations]]
include-name = "GET /items"
# A query parameter is a string on the wire, so schemathesis cannot build a
# type-violating value for the integer "limit" that a server could tell apart
# from a valid one: negating the type of 2 yields "2", which is the same
# request. The check reports that accepted request as a failure, so it is off
# for this operation. Every other check still applies to it, and this one still
# applies everywhere else.
checks.negative_data_rejection.enabled = false

That is the shape a suppression should have. One check, one operation, a stated reason, and everything else left on, against the alternative of dropping negative_data_rejection globally and losing it on the request bodies where it finds real bugs. A fuzzer whose findings are switched off wholesale stops being evidence of anything.

--max-time=10 bounds the run for a different reason: the stateful phase explores API links for as long as it keeps finding new behaviour, which on a specification with several linked operations takes an unpredictable amount of time. An unbounded stage in CI is a stage somebody eventually deletes.

The cost of this layer is that fuzzing needs a service it can send anything to, including requests that create and delete rows, so it belongs in an environment you can reset and it runs after the suites that count rows.

make qa

qa: linter govulncheck confcheck test coverage

linter runs golangci-lint with the project’s .golangci.yml, a coding standard that enforces itself.

govulncheck checks the dependency graph against Go’s vulnerability database, reporting only vulnerabilities in code paths the program can reach, so the findings are actionable rather than a list of CVEs in code you do not call.

confcheck validates config.json against config.schema.json for both the shipped configuration and the test one. That makes page 04’s rule enforceable: a configuration section added without its schema entry fails the build.

test and coverage are the unit layer above. modcheck is not in qa and belongs in CI anyway: it fails if go mod tidy would change anything, which stops a go.mod drifting from the imports.

What to test, and what not to

Test the failure paths. They are the ones that run during an incident and the ones nobody exercises by hand. Every error return in a handler, every sentinel in the mapping on page 10, the timeout, the rollback.

Test the boundaries. A quantity of zero and one of 4,294,967,296. A limit of one, a limit of 101, an absent limit and ?limit=. An empty result set. A delete of a row that is already gone.

Test what a comment explains. The comment on createItemRequest says a client may write a whole number as 3.0, and the case named “accepts a whole quantity written with a fraction” is what stops somebody simplifying the field back to a uint32. A comment without a test is a note; a comment with one is a constraint.

Do not test the standard library. A test that json.Marshal produces JSON is noise.

Do not assert on log output unless the log line is the feature. Log messages change and the tests break for no reason, which trains people to change tests without reading them.

Do not test private functions only through their exported callers, or only directly. The behaviour worth locking down is what another package can see, and a private function complicated enough to need its own test usually wants to be a type.

Next: Building, Packaging, and the Probes That Watch It.