Packing an Entire Country Record into a Single uint64

Country metadata is one of those unglamorous needs that turns up in many backends: translate US to USA to 840, find which region a country belongs to, validate a country top-level domain (TLD). The usual solution is a pile of maps, one per lookup direction, each holding strings. It works, and it is unremarkable.

The countrycode package in nurago takes a more interesting route. Internally, an entire International Organization for Standardization (ISO) 3166 country record is encoded into a single 64-bit integer. This post dissects that integer field by field.

The API is ordinary; the internals are not

From the outside, the package looks like any other lookup library:

import "github.com/tecnickcom/nurago/pkg/countrycode"

data, err := countrycode.New(nil) // embedded ISO 3166 defaults
if err != nil {
    log.Fatal(err)
}

c, err := data.CountryByAlpha2Code("US")
// c.Alpha3Code == "USA", c.NumericCode == "840",
// c.NameEnglish == "United States of America (the)", c.TLD == "us"

italy, _ := data.CountryByAlpha3Code("ITA")
europe, _ := data.CountriesByRegionName("Europe")

Failures come back as two exported sentinels, ErrInvalidCode for malformed input and ErrNotFound for a miss, both matchable with errors.Is. You can also pass your own []*CountryData to New for private overrides or curated subsets. The returned Data is read-only after construction and safe for concurrent use. What is unusual is what sits behind it.

Anatomy of a country key

The internal representation is what the package calls a country key: a uint64 that packs every identifying field of a record into a fixed bit range. This is an application of the Reversible Numeric Composite Key (RNCK) technique, cited in the source itself (N. Asuni, Reversible Numeric Composite Key, arXiv:2306.04353, 2023). The layout is declared as constants, with bitLenChar = 5:

// Binary length of each CountryKey section.
const (
	bitLenTLD       int = 2 * bitLenChar // 2 characters, 5 bit per character.
	bitLenIntRegion int = 5              // max 2^5 = 32 distinct values.
	bitLenSubRegion int = 5              // max 2^5 = 32 distinct values.
	bitLenRegion    int = 5              // max 2^5 = 32 distinct values.
	bitLenNumeric   int = 10             // max 3 numerical digits: log2(999) ~> 10.
	bitLenAlpha3    int = 3 * bitLenChar // 3 characters, 5 bit per character.
	bitLenAlpha2    int = 2 * bitLenChar // 2 characters, 5 bit per character.
	bitLenStatus    int = 3              // max 2^3 = 8 distinct values.
)

Stacked from the least significant bit, that gives the TLD at position 0, intermediate region at 10, sub-region at 15, region at 20, numeric code at 25, alpha-3 at 35, alpha-2 at 50, and the 3-bit assignment status at 60. That is 63 bits, leaving the top bit unused. The status field enumerates the seven ISO 3166 states, from “Unassigned” through “Officially assigned” to “Formerly assigned”. The three region fields do not store United Nations M49 codes directly; they store an index into a small catalogue that maps the index back to its code and name.

The best way to see it is to dissect a real key. The embedded dataset stores the United States as 0x1ACEB30E903102B3, which unpacks like this:

Encoding is a sequence of shifts and ORs (encodeCountryKey), decoding is masks and shifts (decodeCountryKey), and each mirrors the other. The same uint64 round-trips to the record it came from, which is the property the whole package is built on.

Five bits per letter

The part that makes the letter codes fit is the observation that an ISO country code is not arbitrary text. Alpha-2 and alpha-3 codes are drawn from a 26-letter alphabet, so each letter fits in 5 bits (2^5 = 32) as its 1-based offset from A, with 0 left over to mean “no value here”:

func charOffset(b byte, offset uint16) (uint16, error) {
	c := (uint16(b) - offset)
	if c < 1 || c > 26 { // A-Z or a-z
		return 0, errInvalidCharacter
	}

	return c, nil
}

Two letters make a 10-bit alpha-2, three letters a 15-bit alpha-3, and the two-character TLD is another 10 bits with a lowercase offset. Decoding is the mirror image, three masks and three shifts:

func decodeAlpha3(code uint16) string {
	return string([]byte{
		byte(((code & bitMaskChar2) >> bitPosChar2) + chrOffsetUpper),
		byte(((code & bitMaskChar1) >> bitPosChar1) + chrOffsetUpper),
		byte(((code & bitMaskChar0) >> bitPosChar0) + chrOffsetUpper),
	})
}

The package fuzz-tests this round trip: any string accepted by the encoder must decode back to itself, for alpha-2, alpha-3, and TLD alike.

Why bit-packing beats maps and string tables

For static reference data, the packed form has three concrete advantages.

One source of truth. All the reverse indexes (alpha-3 to alpha-2, numeric to alpha-2, groupings by region, status, and TLD) are generated at construction time by decoding the packed keys. The “alpha-2 to alpha-3” table and its inverse are derived from the same integer rather than maintained by hand, so they do not drift apart.

Memory. The identifying core of every country is 8 bytes in a map[uint16]uint64. The English and French names live in a separate map keyed by the same 10-bit alpha-2 ID and are only referenced when a record is materialised.

Cheap decoding. Turning a key back into text costs almost nothing. Alpha-2 and TLD strings are sliced out of two precomputed 2 KiB tables covering the whole 10-bit code space, and the zero-padded numeric code is sliced from a precomputed "000001...999" string instead of going through fmt.Sprintf. The one deliberate exception is alpha-3: a dense table over its 15-bit space would cost 96 KiB, so those three bytes are computed per call. The result, measured by the package benchmarks, is exactly two heap allocations per single-country lookup (the returned CountryData and that 3-byte alpha-3 string), at roughly 130 nanoseconds per CountryByAlpha2Code call on a recent laptop CPU.

Partial records

ISO 3166 is messier than its reputation. The embedded dataset resolves every two-letter combination from AA to ZZ, all 676 of them, but many are reserved or unassigned codes with almost no metadata: EU is “Exceptionally reserved” and carries nothing beyond its status and alpha-2 code, and plain unused combinations come back as “Unassigned”. The decoder handles this by populating each optional field only when it is present in the key, so a record keeps whatever data it actually has instead of being silently reduced to the fields common to every entry.

Absence itself is encoded as zero. Index 0 of each region catalogue is an empty sentinel meaning “no region”, and it is kept strictly internal: the EnumRegion family of methods skips it, and resolving an empty region code or name returns ErrNotFound rather than leaking the sentinel as if it were a valid value.

Custom datasets, same binary form

Passing your own records to New does not switch the package into a slower generic mode. Loading works in two passes: the first collects the region, sub-region, and intermediate-region catalogues from your records (sorted by code, so catalogue indexes are stable across builds), and the second encodes each record into a country key through the same path that produced the embedded defaults. Only Status and Alpha2Code are required; every other field is best-effort, and an absent or malformed optional field encodes to zero rather than failing the record. If two records share an alpha-2 code, the last one wins. After that, custom data and embedded data are indistinguishable: the same uint64 keys, the same derived indexes, the same lookup cost.

When it matters

For a single occasional lookup, you would be unlikely to notice the difference between this and a naive map. The technique earns its place when country resolution sits on a hot path: request validation and enrichment on every inbound call, or geographic joins across a large dataset. That is where a compact, reversible, integer-keyed representation stops being a curiosity and starts being the reason the lookup does not show up in your profile. And if the bit-packing idea itself is what interests you, the RNCK paper generalises the approach well beyond country codes.

• Source: https://github.com/tecnickcom/nurago/tree/main/pkg/countrycode
• Docs: https://pkg.go.dev/github.com/tecnickcom/nurago/pkg/countrycode
• RNCK paper: https://arxiv.org/abs/2306.04353
• Project: https://github.com/tecnickcom/nurago