diff --git a/docs/changelog-20260713-office1-netbox-deployed.md b/docs/changelog-20260713-office1-netbox-deployed.md new file mode 100644 index 0000000..ef3b55a --- /dev/null +++ b/docs/changelog-20260713-office1-netbox-deployed.md @@ -0,0 +1,93 @@ +# changelog 2026-07-13 -- office1-netbox DEPLOYED; NetBox 4.6.4 live on the Office1 headend + +## What + +`office1-netbox` -- the MAAS-composed LXD VM inside `voffice1` -- is **Deployed** (Ubuntu 24.04.4, +laid down by MAAS) and runs **NetBox 4.6.4** (netbox-docker, Django 6.0.6). Its API authenticates. + +This completes the D-114 chain in its final form: MAAS composed the machine, PXE-booted it, +commissioned it, deployed an OS onto it, and we installed a real service on it -- all three levels +of nesting deep, inside a containment VM, on a host that is itself a KVM guest. + +## FOUR findings, all hit for real. Two are now guarded in code. + +### 1. MAAS-discovered subnets have NO gateway -- a deployed machine gets NO default route (FIXED) + +MAAS auto-discovers the compose subnet from `lxdbr0`'s interface, and that discovery carries **no +`gateway_ip`**. So the first deploy produced a machine that looked healthy -- it had an address, +and DNS even resolved (MAAS's bind9 is link-local) -- but had **no default route and no egress**. +The failure only surfaces at the first `apt install`, which hangs. + +Fixed in `scripts/site-headend-install.sh`: it now sets `gateway_ip=` on the compose +subnet (lxdbr0 IS the gateway -- it does the NAT). `tests/site-headend-install/` asserts it (19/19). +`office1-netbox` was released and redeployed to prove the fix rather than hand-patching a route. + +### 2. NetBox 4.6 v2 API tokens: the wire format is `nbt_.` -- NOT the `token` field + +This cost the most time and is the least guessable thing here. + +NetBox 4.6 **hashes** API tokens. The `Token` model has `key` (a 12-char PREFIX), `plaintext` (40), +`hmac_digest` (64) and a `version`. Authentication (`netbox/api/authentication.py`) infers the +version **from a prefix**: + +```python +version = 2 if auth_value.startswith(TOKEN_PREFIX) else 1 # TOKEN_PREFIX = 'nbt_' +... +key, plaintext = auth_value.removeprefix(TOKEN_PREFIX).split('.', 1) +``` + +So a v2 token must be presented as **`nbt_<key>.<plaintext>`** (57 chars). `POST +/api/users/tokens/provision/` returns `key` and `token` as **separate fields**, and **the `token` +field ALONE IS NOT USABLE** -- present it by itself and NetBox parses it as a legacy v1 token, +fails the lookup, and returns `403 {"detail":"Invalid v1 token"}`. That error names v1 while you +are holding a v2 token, which is thoroughly misleading. + +Consequences for anything scripted against this NetBox (incl. `netbox/dc-dc-prefixes-import.py` +and any `pynetbox` client): **assemble the token, do not use the API's `token` field raw.** + +Related: `SUPERUSER_API_TOKEN` in netbox-docker's env is **NOT honored** on this version (the +entrypoint logs "No API token was created ... SUPERUSER_API_TOKEN and SUPERUSER_API_KEY are not +set" even when it IS set). Use the provisioning endpoint instead. And do NOT try to construct a +Token row by hand -- setting `plaintext` trips the DB's `enforce_version_dependent_fields` check +constraint. Call the interface; do not re-implement it. (This repo already learned that lesson +minting OPNsense API keys -- and I re-learned it here the hard way.) + +### 3. A WRONG HYPOTHESIS, recorded because the mistake is reusable + +When the token kept 403-ing I concluded that rotating `SECRET_KEY` **after** NetBox had initialised +had broken the token HMAC pepper, and I wiped the database (`docker compose down -v`) to re-init +with the key in place. **That hypothesis was WRONG.** The 403 was entirely the v2 wire format above. +The wipe was harmless -- the DB was empty -- but the reasoning was not, and a wipe on a populated +NetBox would have been destructive. The lesson: `Invalid v1 token` is a FORMAT error, not a crypto +or key-rotation error. Read the auth code before reaching for a destructive reset. + +### 4. netbox-docker ships a PUBLIC default `SECRET_KEY` + +`env/netbox.env` in the public netbox-docker repo carries a real, published `SECRET_KEY`. Left +alone it is session-forgery material on any reachable instance. **Rotated** to a generated value. +(`SKIP_SUPERUSER=true` is the shipped default, so at least no default admin/admin exists.) + +## As deployed (measured) + +- `office1-netbox`: Ubuntu 24.04.4, 2 vCPU / 4096 MiB / 40 GB, **10.10.1.201** (MAAS-assigned, + outside the dynamic range), gateway `10.10.1.1`, egress OK. `systemd-detect-virt` -> `kvm` (L3). +- NetBox **4.6.4** on Django 6.0.6, containers `netbox` + `netbox-worker` + `postgres` + `redis` + + `redis-cache`, published on `:8000`. `GET /api/status/` -> **HTTP 200**. +- Secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored `0600` under + `/root/netbox-secrets/`, never printed into the session. + +## Reachability caveat (not yet solved) + +NetBox is on `10.10.1.201`, which sits **behind `voffice1`'s NAT** on `lxdbr0`. It is reachable +from `voffice1` but NOT from `office1-local` or the operator's workstation. To make it reachable, +the OPNsense edge needs a static route for `10.10.1.0/24` via `10.10.0.20`. That is a live edge +change (REST API, D-113(a2)) and is NOT done -- it should be gated with the operator. + +## Still to do + +- The D-115 carve is NOT yet imported into this NetBox (it is the next step). +- `10.10.1.0/24` + the D-115 Office/Edge roles need registering here, then feeding back upstream. + +## Verification + +`tests/site-headend-install/run-tests.sh` 19/19 PASS. `repo-lint` 0 fail. diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh index 44a13db..ef94b02 100644 --- a/scripts/site-headend-install.sh +++ b/scripts/site-headend-install.sh @@ -239,6 +239,14 @@ " "$COMPOSE_CIDR")" RACK="$(maas admin rack-controllers read 2>/dev/null | python3 -c "import sys,json;print(json.load(sys.stdin)[0]['system_id'])")" + # GATEWAY: MAAS auto-discovers the compose subnet from lxdbr0's interface, and that discovery + # carries NO gateway. Without this, a DEPLOYED machine gets a netplan with NO DEFAULT ROUTE -- + # it comes up looking fine (DNS even resolves, because MAAS's bind9 is link-local) but has NO + # EGRESS, so the first `apt install` on it hangs. lxdbr0 IS the gateway (it does the NAT). + # Measured 2026-07-13: office1-netbox deployed with gateway_ip=None and had to be redeployed. + maas admin subnet update "$SUB_ID" gateway_ip="$BRIDGE_IP" >/dev/null 2>&1 \ + && echo " -> gateway_ip=$BRIDGE_IP set on $COMPOSE_CIDR" + maas admin ipranges create type=dynamic subnet="$SUB_ID" start_ip="$RANGE_LO" end_ip="$RANGE_HI" \ comment="MAAS DHCP/PXE for LXD-composed VMs (D-114)" >/dev/null 2>&1 maas admin vlan update "$FAB_ID" 0 dhcp_on=True primary_rack="$RACK" >/dev/null 2>&1 \ diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh index 8880a78..4aa5c18 100644 --- a/tests/site-headend-install/run-tests.sh +++ b/tests/site-headend-install/run-tests.sh @@ -63,6 +63,10 @@ grep -q '5.21/stable' "$S" && ok || bad "trap 3 LOST: LXD must be pinned to the 5.21 LTS track (MAAS breaks on LXD >= 6.7)" grep -q 'is ON for a subnet that is NOT the compose network' "$S" \ && ok || bad "trap 4 LOST: the guard asserting MAAS DHCP is on NO other subnet is gone" +# trap 5: MAAS discovers the compose subnet from lxdbr0 with NO gateway. A deployed machine then +# gets a netplan with no default route -- it LOOKS healthy (DNS resolves; bind9 is link-local) but +# has no egress, and the first apt install on it hangs. Cost a redeploy on 2026-07-13. +grep -q 'subnet update .*gateway_ip' "$S" && ok || bad "trap 5 LOST: gateway_ip is never set on the compose subnet -- deployed machines will have NO default route" echo total=$((pass+fail))