Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260713-office1-netbox-deployed.md

changelog 2026-07-13 -- office1-netbox DEPLOYED; NetBox 4.6.4 live on the Office1 headend

What

office1-netbox -- the MAAS-composed LXD VM inside voffice1 -- is Deployed (Ubuntu 24.04.4, laid down by MAAS) and runs NetBox 4.6.4 (netbox-docker, Django 6.0.6). Its API authenticates.

This completes the D-114 chain in its final form: MAAS composed the machine, PXE-booted it, commissioned it, deployed an OS onto it, and we installed a real service on it -- all three levels of nesting deep, inside a containment VM, on a host that is itself a KVM guest.

FOUR findings, all hit for real. Two are now guarded in code.

1. MAAS-discovered subnets have NO gateway -- a deployed machine gets NO default route (FIXED)

MAAS auto-discovers the compose subnet from lxdbr0's interface, and that discovery carries no gateway_ip. So the first deploy produced a machine that looked healthy -- it had an address, and DNS even resolved (MAAS's bind9 is link-local) -- but had no default route and no egress. The failure only surfaces at the first apt install, which hangs.

Fixed in scripts/site-headend-install.sh: it now sets gateway_ip=<bridge-ip> on the compose subnet (lxdbr0 IS the gateway -- it does the NAT). tests/site-headend-install/ asserts it (19/19). office1-netbox was released and redeployed to prove the fix rather than hand-patching a route.

2. NetBox 4.6 v2 API tokens: the wire format is nbt_<key>.<plaintext> -- NOT the token field

This cost the most time and is the least guessable thing here.

NetBox 4.6 hashes API tokens. The Token model has key (a 12-char PREFIX), plaintext (40), hmac_digest (64) and a version. Authentication (netbox/api/authentication.py) infers the version from a prefix:

version = 2 if auth_value.startswith(TOKEN_PREFIX) else 1   # TOKEN_PREFIX = 'nbt_'
...
key, plaintext = auth_value.removeprefix(TOKEN_PREFIX).split('.', 1)

So a v2 token must be presented as nbt_<key>.<plaintext> (57 chars). POST /api/users/tokens/provision/ returns key and token as separate fields, and the token field ALONE IS NOT USABLE -- present it by itself and NetBox parses it as a legacy v1 token, fails the lookup, and returns 403 {"detail":"Invalid v1 token"}. That error names v1 while you are holding a v2 token, which is thoroughly misleading.

Consequences for anything scripted against this NetBox (incl. netbox/dc-dc-prefixes-import.py and any pynetbox client): assemble the token, do not use the API's token field raw.

Related: SUPERUSER_API_TOKEN in netbox-docker's env is NOT honored on this version (the entrypoint logs "No API token was created ... SUPERUSER_API_TOKEN and SUPERUSER_API_KEY are not set" even when it IS set). Use the provisioning endpoint instead. And do NOT try to construct a Token row by hand -- setting plaintext trips the DB's enforce_version_dependent_fields check constraint. Call the interface; do not re-implement it. (This repo already learned that lesson minting OPNsense API keys -- and I re-learned it here the hard way.)

3. A WRONG HYPOTHESIS, recorded because the mistake is reusable

When the token kept 403-ing I concluded that rotating SECRET_KEY after NetBox had initialised had broken the token HMAC pepper, and I wiped the database (docker compose down -v) to re-init with the key in place. That hypothesis was WRONG. The 403 was entirely the v2 wire format above. The wipe was harmless -- the DB was empty -- but the reasoning was not, and a wipe on a populated NetBox would have been destructive. The lesson: Invalid v1 token is a FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a destructive reset.

4. netbox-docker ships a PUBLIC default SECRET_KEY

env/netbox.env in the public netbox-docker repo carries a real, published SECRET_KEY. Left alone it is session-forgery material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the shipped default, so at least no default admin/admin exists.)

As deployed (measured)

  • office1-netbox: Ubuntu 24.04.4, 2 vCPU / 4096 MiB / 40 GB, 10.10.1.201 (MAAS-assigned, outside the dynamic range), gateway 10.10.1.1, egress OK. systemd-detect-virt -> kvm (L3).
  • NetBox 4.6.4 on Django 6.0.6, containers netbox + netbox-worker + postgres + redis + redis-cache, published on :8000. GET /api/status/ -> HTTP 200.
  • Secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under /root/netbox-secrets/, never printed into the session.

Reachability caveat (not yet solved)

NetBox is on 10.10.1.201, which sits behind voffice1's NAT on lxdbr0. It is reachable from voffice1 but NOT from office1-local or the operator's workstation. To make it reachable, the OPNsense edge needs a static route for 10.10.1.0/24 via 10.10.0.20. That is a live edge change (REST API, D-113(a2)) and is NOT done -- it should be gated with the operator.

Still to do

  • The D-115 carve is NOT yet imported into this NetBox (it is the next step).
  • 10.10.1.0/24 + the D-115 Office/Edge roles need registering here, then feeding back upstream.

Verification

tests/site-headend-install/run-tests.sh 19/19 PASS. repo-lint 0 fail.