diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md index 4755be7..44419f9 100644 --- a/.claude/skills/openstack-cloud-ops/references/platform-traps.md +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -395,3 +395,33 @@ | a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c | | MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a | | a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c | + +--- + +## NetBox (upstream, `netbox.baldurkeep.com`) + +### `HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl` + +**Symptom.** `urllib`/`requests`/`pynetbox` get `403 Forbidden` from the upstream NetBox on every +endpoint, including ones a valid token plainly reads. `curl` with the identical token and URL +returns `200`. It reads exactly like a bad or revoked token; it is not. + +**Cause.** Something in front of upstream NetBox (WAF / reverse proxy) filters on **User-Agent**. +The default Python UA (`Python-urllib/3.12`) is blocked; `curl/8.5.0` is allowed. Measured +2026-07-13: same token, same URL, UA the only variable -- `None -> 403`, `curl/8.5.0 -> 200`. + +**Fix.** Send an accepted `User-Agent` header. This applies to **`pynetbox`** too (it sends +`python-requests/...`), so the repo's NetBox tooling must set it: + +```python +nb = pynetbox.api(url, token=tok) +nb.http_session.headers["User-Agent"] = "curl/8.5.0" +``` + +**Do not** diagnose this as a token/permission problem and go re-mint credentials -- that is the +trap. Confirm with `curl` first; if curl works and Python does not, it is the UA. + +**Note:** this is an UPSTREAM-only trap so far. The Office1 sandbox NetBox (4.6.4, no proxy in +front) accepts the default Python UA. Upstream is 4.5.8 and also uses **v1** (40-char, bare) tokens, +while the sandbox is 4.6 and uses **v2** (`nbt_.`). Any tool pointed at BOTH must +handle both auth shapes and the UA -- do not assume they are the same NetBox. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 06efd89..04a878d 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2782,3 +2782,70 @@ `docs/dc-dc-deployment-workflow.md`, from the Office1 standup runbook's sequence and its open questions, and from `docs/vr1-office1-as-built.md`. Office1's composed service machines are therefore exactly TWO: `office1-netbox` and `office1-tailscale`, both LIVE. + +--- + +## D-117: VR1 site naming -- the apex says DC0/DC1, the repo says DC1/DC2 (G5) + +**Status:** **PROPOSED 2026-07-13.** Blocks the NetBox import. Options below; operator rules. + +**This is the G5 gap** (`docs/dc-dc-netbox-buildout-scope.md`), promoted to a decision because it +blocks the Office1 NetBox import and because the tool that trips over it WRITES BY DEFAULT. + +**The collision, measured live against `netbox.baldurkeep.com` (4.5.8) on 2026-07-13:** + +| v6 site `/48` | Prod NetBox site (the apex) | `dc-dc-prefixes-import.py` | +|---|---|---| +| `2602:f3e2:f02::/48` | **VR1 DC0** (`vr1-dc0`, id 43) | `--dc dc1` (its docstring: "`f02::/48` for dc1") | +| `2602:f3e2:f03::/48` | **VR1 DC1** (`vr1-dc1`, id 44) | `--dc dc2` | + +The **addressing agrees**; only the **labels are off by one**. The repo's "DC1" and the apex's "DC0" +are the SAME datacenter. Same off-by-one on the office: the repo's "Office1" carve +(`2602:f3e2:f01:100::/56`, D-115) exactly mirrors VR0's **Off0** (`e01:100::/56`). + +**Why this is dangerous, not merely cosmetic.** `dc-dc-prefixes-import.py` maps `dc1 -> slug +vr1-dc1` and **has no `--commit`** -- `--verify-only` is its read-only mode, so it WRITES BY +DEFAULT. Run it as documented against a NetBox seeded from prod and it binds DC1's `f02::/48` +prefixes to the site `vr1-dc1`, which upstream holds `f03::/48` -- i.e. **the other DC**, silently. +(`roles-aggregates-import.py` is the opposite: dry by default. Do not assume the tools are uniform.) + +**How it happened, recorded because the mistake is reusable.** The script's own comment: *"Slugs are +new -- no prior convention existed in this repo (grepped `opentofu/*.tf` and the buildout design; +none found)."* It grepped the **repo**. The convention lived in the **apex**. NetBox is declared the +IPAM apex (D-010/D-103) precisely so that naming is NOT a repo-local invention -- but the check for +prior art was run against the repo. Same class of error as concluding "no VR0 precedent for +MAAS-composed LXD VMs" after grepping only `bundle.yaml` (2026-07-13, same session). + +**What the site byte does and does NOT mean.** Within a region `/40`: `00`=Infrastructure, +`01`=Offices, `02`/`03`=the DCs. That is **sequential allocation, not a DC index** -- there is no +`02 -> DC0` mnemonic to preserve. An earlier draft of this entry claimed there was; that was wrong +and is corrected here, because it materially weakens the case against Option A. + +**Blast radius, measured.** `vr1-dc0` and `vr1-dc1` are **skeletal**: 1 prefix each (their own `/48` +container), and **0** devices / racks / VLANs / clusters. (Contrast `vr0-dc0`: 18 prefixes.) Renaming +them upstream touches 2 site records + their 2 prefix descriptions. Renaming the REPO instead touches +the live surfaces (`lib-net.sh`, the `--dc` flag, `opentofu/`, runbooks, SKILL.md, and the text of +D-101/D-106/D-111/D-115) and contradicts **already-ratified D-106** (Designate `dc1`/`dc2` zones). + +### Options + +**A. Rename the two skeletal apex sites to `vr1-dc1`/`vr1-dc2`; add `vr1-off1`. (RECOMMENDED)** +The scope doc's standing recommendation. Cheapest by a wide margin, and it preserves the `dc1`/`dc2` +language already ADOPTED in D-106 and wired through every live repo surface. Cost: VR1 reads DC1/DC2 +while VR0 reads DC0/DC1 -- a cosmetic cross-region inconsistency, and VR1's office becomes `Off1` +against VR0's `Off0`. +**TRAP -- rename ORDER matters:** `vr1-dc0 -> vr1-dc1` while `vr1-dc1` still exists is a **slug +collision**. Rename **`vr1-dc1` -> `vr1-dc2` FIRST**, then `vr1-dc0` -> `vr1-dc1`. Update the two +`/48` descriptions ("Datacenter 0 (DC0)") in the same change or they contradict their own site. + +**B. Rename the repo to the apex's 0-indexed `DC0`/`DC1` + `Off0`.** +Uniform 0-indexing across VR0 and VR1, and the apex is never bent to fit the repo. Cost: reopens +ratified D-106, and rewrites every live repo surface. Historical changelogs are NOT rewritten. + +**C. Re-point the tool only (`SITES = {dc1: vr1-dc0, dc2: vr1-dc1}`).** +Two lines, nothing else moves. **Argued against:** it makes `--dc dc1` write to a site *displayed* as +"VR1 DC0" forever -- a permanent hidden indirection, which is how this class of bug got here. + +**Whichever is chosen, one fix is unconditional:** give `dc-dc-prefixes-import.py` a `--commit` flag +so it is DRY BY DEFAULT, matching `roles-aggregates-import.py`. A write-by-default IPAM tool is a +standing hazard regardless of how the naming lands.