Table of contents
Everything from page 03 onward has been the process’s own behaviour. This is the machinery that produces the binary, wraps it, and puts it somewhere it will be watched. A shutdown timeout the orchestrator ignores does nothing.
The build
build:
CGO_ENABLED=0 $(GOBUILDENV) \
$(GO) build \
-tags ${STATIC_TAG} \
-ldflags '-w -s -X main.programVersion=${VERSION} -X main.programRelease=${RELEASE} -extldflags "-fno-PIC ${STATIC_FLAG}"' \
-o "./target/${BINPATH}$(PROJECT)" "$(CMDDIR)"
CGO_ENABLED=0 produces a binary with no dynamic libraries, so it runs in a
FROM scratch container with no libc and does not care which glibc the host has.
The cost is that os/user and the default DNS resolver fall back to pure-Go
implementations, which matters only if you rely on nsswitch.conf.
-w -s strips the DWARF debug information and the symbol table, taking a
meaningful fraction off the binary size and making a core dump much less useful.
pprof still works without them.
-X main.programVersion=... is the interesting one.
Version injection
Page 03 declared them:
var (
programVersion = "0.0.0"
programRelease = "0"
)
-X importpath.name=value overwrites a string variable at link time, from the
VERSION and RELEASE files in the repository root. They are compile-time
constants rather than configuration, because a version that can be set in a
configuration file is a version that can lie.
From there it propagates: page 06 puts it on every
log line, so an error rate can be grouped by release,
page 10 puts it in every response envelope, and
page 13’s /ping returns it. make ping
asserts that:
ping:
if [ "200_$(VERSION)_$(RELEASE)_" != "$(shell curl --silent --insecure \
'$(MONITORING_URL)/ping' | jq -r '.code,.version,.release' | tr '\n' '_')" ]; then exit 1; fi
A deploy that reported success and left the previous version running fails there.
The version is the semantic version of the code and the release is the packaging build number, so rebuilding the same code on a new base image is a new release at the same version. The pair identifies an artefact rather than a commit.
Containers
make docker
builds a runtime image from the static binary and a Flyway image carrying the migrations.
Two images, because page 12 runs migrations
outside the service: the migration job runs to completion and the service starts
afterwards. make dcenvup brings up a local environment that exercises
page 04’s environment-variable path before
it reaches a cluster.
The image
A FROM scratch image with a static binary is a few tens of megabytes with
nothing else in it: no shell, no package manager, no utilities, and no curl for
an attacker to pull the next stage with. It is also inconvenient, since debugging
goes through the monitoring listener’s /pprof and the logs, one more reason those exist. distroless is the middle ground, with a certificate bundle
and a timezone database and still no shell.
Two things a scratch image needs and people forget: a CA bundle, or every outbound
call from page 11 fails certificate verification,
and the timezone database, or time.LoadLocation fails outside UTC. Go embeds the
latter with a time/tzdata import.
Packages
make deb and make rpm produce native packages, which the teams that cannot run
a container still need: on-premises installs, appliances, regulated environments.
The layout is conventional, and that is the point of using the native format:
/usr/bin/inventorysvc the binary
/etc/inventorysvc/config.json the configuration
/usr/lib/systemd/system/inventorysvc.service the unit
/etc/init.d/inventorysvc the init script
/usr/share/man/man1/inventorysvc.1 the man page
/usr/share/doc/inventorysvc/ the documentation
/etc/inventorysvc/config.json is the third entry in
page 04’s search path: the package puts a
default there, and an operator overriding it in $HOME or with --configDir wins
without editing a package-managed file.
The systemd unit
[Unit]
Description=inventorysvc service
After=network.target
Wants=network.target
[Service]
Type=simple
ExecStart=/usr/bin/inventorysvc
Restart=on-failure
RestartSec=5
StandardOutput=append:/var/log/inventorysvc.log
StandardError=append:/var/log/inventorysvc.log
# Security hardening options
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Restart=on-failure with RestartSec=5 is the host-level equivalent of an
orchestrator’s restart policy. It makes
page 12’s fail-fast startup ping right here too:
the service exits, systemd waits, and tries again.
Type=simple says the process does not fork, correct for a Go binary, and leaves systemd with no readiness tracking: that needs Type=notify. A host
deployment learns the service is ready by polling /status.
The hardening options each close something. ProtectSystem=full mounts /usr and
/boot read-only, ProtectHome=true makes /home inaccessible,
NoNewPrivileges=true breaks a common escalation step through setuid binaries,
and PrivateTmp=true gives the process its own /tmp. Change one thing from this
template: add User= and Group= with a service account, because without them
the unit runs as root and none of the above compensates.
Sending log lines to a file rather than the journal is the choice here that argues
with page 06, since a file needs rotation.
Dropping StandardOutput and StandardError sends both to the journal, which
handles rotation and is queryable with journalctl -u inventorysvc.
Orchestrated deployment
Something has to run the image, restart it when it dies, decide which instances receive traffic, and replace them one at a time during a rollout. Kubernetes and Nomad both do that, and the service is written against neither. Each needs the same list: the three ports from page 17, the environment and secrets from page 04, which endpoint answers “is it alive” and which “can it serve” from page 13, and the drain window from page 05.
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventorysvc
spec:
replicas: 3
template:
spec:
containers:
- name: inventorysvc
image: example/inventorysvc:1.4.2-873
ports:
- { name: monitoring, containerPort: 8071 }
- { name: private, containerPort: 8072 }
- { name: public, containerPort: 8073 }
env:
- name: INVENTORYSVC_LOG_LEVEL
value: "INFO"
- name: INVENTORYSVC_DB_MAIN_DSN
valueFrom:
secretKeyRef: { name: inventorysvc-db, key: main-dsn }
- name: INVENTORYSVC_DB_READ_DSN
valueFrom:
secretKeyRef: { name: inventorysvc-db, key: read-dsn }
startupProbe:
httpGet: { path: /status, port: monitoring }
failureThreshold: 30
periodSeconds: 2
livenessProbe:
httpGet: { path: /ping, port: monitoring }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /status, port: monitoring }
periodSeconds: 5
failureThreshold: 2
terminationGracePeriodSeconds: 90
Every value there connects to something earlier in the guide.
The three ports
Declared separately, per page 17. The
Service in front exposes only public, and a NetworkPolicy restricts
private to the pods that should reach it and monitoring to the scraper and the
operator namespace. That policy is load-bearing: without it the three ports are
three ports on the same cluster network, and page 17’s argument is about naming
rather than enforcement.
The environment
Page 04’s fifth layer. The image ships a
config.json with defaults and the deployment overrides what differs. Every key
has a registered default because a key without one is not overridable here,
silently.
Secrets come from a Secret rather than the config file, and RuleUserinfo
from page 06 is what stops a DSN in a connection
error putting the password in the logs.
The probes
startupProbe on /status, with failureThreshold: 30 and
periodSeconds: 2, allowing 60 seconds to start, with liveness and readiness
suspended while it runs. A slow start is then a slow start rather than the crash
loop you get when liveness begins immediately.
livenessProbe on /ping, never on /status. Liveness answers “kill and
replace this process”. Pointed at /status, a 30-second database blip fails the
probe on every replica at once, the orchestrator kills all of them, they restart,
fail page 12’s startup ping, and crash-loop, so
the database gets a thundering herd of restarting pods and a degradation outlasts
its cause. /ping consults nothing, so nothing external can trigger a restart.
readinessProbe on /status, never on /ping. Readiness answers “send this
pod traffic”. Pointed at /ping, a pod whose database is unreachable reports itself ready, receives traffic, and fails every request. During a rolling deploy that is a window of hard failures on every new pod.
failureThreshold: 2 on readiness with periodSeconds: 5 takes a pod out after
about ten seconds; liveness at 3 and 10 restarts it after thirty. Liveness should
be the slower of the two, because its action is destructive and readiness’s is
reversible.
terminationGracePeriodSeconds
This number has to agree with page 05. The pod
is removed from the Service endpoints, the orchestrator sends SIGTERM, waits
terminationGracePeriodSeconds, then sends SIGKILL. The service’s own shutdown
is bounded by shutdown_timeout, 60 seconds in the reference configuration, so
the grace period has to exceed it with margin: 90 against 60 leaves room, and 30
against 60 kills the process halfway through a drain the platform asked for.
Endpoint removal is asynchronous, so SIGTERM often arrives before every proxy
has stopped routing to the pod and requests land after shutdown started. A
preStop hook delays the signal:
lifecycle:
preStop:
sleep:
seconds: 5
The sleep handler is timed by the kubelet rather than run inside the container,
so it needs nothing in the image and a scratch build stays usable. It is beta
and on by default from Kubernetes 1.30 and stable from 1.34; the older
exec: command: ["/bin/sleep", "5"] form is what requires a sleep binary, and
therefore a distroless or similar base image instead of scratch.
The delay is counted inside terminationGracePeriodSeconds, not added to it, so
the budget is the hook plus shutdown_timeout: 5 and 60 against 90 still leaves
margin.
The same deployment in Nomad
group "inventorysvc" {
count = 3
network {
port "monitoring" { to = 8071 }
port "private" { to = 8072 }
port "public" { to = 8073 }
}
service {
name = "inventorysvc"
port = "public"
shutdown_delay = "5s"
check {
name = "alive"
type = "http"
port = "monitoring"
path = "/ping"
interval = "10s"
timeout = "2s"
check_restart {
limit = 3
grace = "60s"
}
}
check {
name = "ready"
type = "http"
port = "monitoring"
path = "/status"
interval = "5s"
timeout = "2s"
}
}
task "inventorysvc" {
driver = "docker"
config {
image = "example/inventorysvc:1.4.2-873"
ports = ["monitoring", "private", "public"]
}
kill_timeout = "90s"
env {
INVENTORYSVC_LOG_LEVEL = "INFO"
}
template {
destination = "secrets/env"
env = true
data = <<-EOT
{{ with secret "kv/data/inventorysvc" -}}
INVENTORYSVC_DB_MAIN_DSN={{ .Data.data.main_dsn }}
INVENTORYSVC_DB_READ_DSN={{ .Data.data.read_dsn }}
{{- end }}
EOT
}
}
}
Same image, same environment variables, same two endpoints, and three differences to know before translating one to the other.
Nomad has no separate liveness and readiness probe. It has health checks, which
decide whether the instance stays in the service catalogue, and check_restart,
which kills the task after a check has been failing long enough. The split from
page 13 survives as where check_restart is
attached: on the /ping check, never on the /status one, or a database outage
restarts every allocation at once. grace is the startup allowance that
startupProbe provides on the other side.
kill_timeout is terminationGracePeriodSeconds, and it has the same job of
exceeding shutdown_timeout. It also has a trap the Kubernetes field does not:
the client agent caps it at max_kill_timeout, which defaults to 30 seconds, so a
task asking for 90 gets 30 and a truncated drain until someone changes the agent
configuration. The failure is silent from inside the job specification.
shutdown_delay is the preStop sleep. Nomad deregisters the service, waits,
and only then signals the task, handling the endpoint-removal race directly
rather than working around it. Unlike the Kubernetes hook, the wait sits
outside kill_timeout rather than being taken out of it.
The rest maps across. A template block with env = true fills
page 04’s fifth layer from Vault instead of
a Secret, and restricting the monitoring and private ports is Consul intentions
or the host firewall rather than a NetworkPolicy.
What the platform now owns
Page 08’s edge concerns have names in this
manifest. The Ingress, or whatever load balancer routes on the Nomad job’s
registered service, terminates TLS, applies a body cap, and often applies rate
limiting and compression, so the in-process rate limiter drops to a floor well
above the real policy and CORS and compression are each handled at one end only. A
mesh sidecar likewise owns retries and circuit breaking for mesh-internal traffic,
leaving page 11’s retrier the calls that leave it.
The proxy depth is countable here, because the topology is in this file, so keep the number in the same manifest and the two move together.
The deploy checklist
Five things, each a specific outage if it is wrong.
terminationGracePeriodSeconds, or kill_timeout under an agent whose
max_kill_timeout allows it, is greater than shutdown_timeout, itself greater than the per-server httpserver shutdown timeout.
Liveness is on /ping and readiness is on /status, and nothing that restarts
the process is watching a dependency.
A NetworkPolicy, or the Consul and firewall equivalent, restricts 8071 and 8072,
and only 8073 is behind an ingress.
Migrations run as a job before the rollout, and the migration is backward compatible with the code currently running.
make rping passes after the rollout, so the version answering is the version
that was built.
Next: Growing the Service Without Growing the Wiring, the last page.