diff --git a/docs/session-ledger.md b/docs/session-ledger.md index f70d48b..f61e58f 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -274,3 +274,42 @@ - Consolidate the 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. - Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials. - v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. + +## NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13) + +**Done.** The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop +(`netbox/prod-draft-dump.py` READ-ONLY -> `netbox/draft/vr1-draft.json` -> `netbox/sandbox-seed.py`), +seeded 129 objects from the upstream draft, then applied the **D-115 office carve** +(`netbox/d115-office-carve.py`): site `vr1-off1`, roles `office` + **`edge`** (new), and 8 prefixes. +Idempotent, verified. See `docs/vr1-office1-as-built.md` section 6. + +**D-117 ADOPTED (operator ruling, Option B, DC-only):** VR1's DCs are **dc0/dc1**, matching the apex. +The **office KEEPS its number** (Off1). Fixed the importer's off-by-one site binding and made it +**dry by default**. + +### BLOCKED: the VR1 DC six-plane import needs `ORG_ULA_48` RATIFIED + +`netbox/dc-dc-prefixes-import.py --dc dc0` cannot run. It requires `ORG_ULA_48`, and that literal is +**still unassigned** -- it is gap **G3** in `docs/dc-dc-netbox-buildout-scope.md`, which *recommends* +`fd50:840e:74e2::/48` (CSPRNG-generated, Global ID `0x50840e74e2`) but **no ADOPTED decision assigns +it** (`grep ORG_ULA docs/design-decisions.md` -> nothing). D-101 explicitly leaves it NetBox-assigned. + +Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. +This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready. + +**To unblock:** ratify an org ULA `/48` under its own D-number. Then: +``` +ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # VR1 DC0 +ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f03::/48 \ + DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # VR1 DC1 +``` +(`DC_GUA_PREFIX` values are MEASURED off the apex; `DC1_V4_SUPERNET` is assigned by D-115. +`DC2_V4_SUPERNET` is RETIRED -- the tool rejects it by name.) + +### Also open (recorded, not blocking) +- **`$DC` shell-selector namespace collision** (D-117 amendment): `lib-net.sh`'s `dc0` means **VR0's** + DC0, while the importer's `dc0` now means **VR1's**. NOT interchangeable. Proposed fix: region- + qualify the shell selector to the apex slugs (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`). +- Feed the D-115 carve upstream to `netbox.baldurkeep.com` (operator-gated; the tool refuses a + non-sandbox target without `--yes-write-upstream`). +- Office1 LAN IPv6 (`2602:f3e2:f01:100::/64` + RA via the OPNsense REST API) -- still pinned. diff --git a/docs/vr1-office1-as-built.md b/docs/vr1-office1-as-built.md index e708b34..8793a7d 100644 --- a/docs/vr1-office1-as-built.md +++ b/docs/vr1-office1-as-built.md @@ -8,7 +8,9 @@ **Authority note.** NetBox is the IPAM apex -- this table is an OPERATIONAL reference (how to reach things, where credentials live), NOT the address authority. Where the two ever disagree, NetBox wins -and the divergence gets fixed here. The D-115 carve is not yet imported into NetBox (open item). +and the divergence gets fixed here. **The D-115 carve IS now imported** into the Office1 sandbox +NetBox (2026-07-13) -- site `vr1-off1`, roles `office` + `edge`, and every prefix below. It has NOT +yet been fed back upstream to `netbox.baldurkeep.com`; that is an operator-gated step. Last measured: 2026-07-13. @@ -62,7 +64,7 @@ | **OPNsense GUI** | `https://10.10.0.1` | `ssh -L 8443:10.10.0.1:443 jessea123@10.17.11.248` -> `https://localhost:8443` | | **OPNsense shell** | `ssh -i root@10.10.0.1` | via vcloud. **root's shell is `tcsh`** -- feed commands to `sh -s`. | | **MAAS UI** | `http://10.10.0.20:5240/MAAS` | `ssh -L 5240:10.10.0.20:5240 jessea123@10.17.11.248` -> `http://localhost:5240/MAAS` | -| **NetBox UI/API** | `http://10.10.1.201:8000` (needs the route below) | `ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.201:8000 jessea123@10.10.0.20` -> `http://localhost:8000` | +| **NetBox UI/API** | `http://10.10.1.201:8000` (needs the route below) | on tailnet: `http://10.10.1.201:8000` direct. Else `ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.201:8000 jessea123@10.10.0.20` -> `http://localhost:8000` | | **office1-netbox / office1-tailscale** (shell) | `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201` | chain both jumps | **Routing to the compose net.** The OPNsense edge has a static route `10.10.1.0/24 -> 10.10.0.20` @@ -81,9 +83,11 @@ - **`--accept-dns=false` is DELIBERATE** (a deviation from the operator's usual command): this is a MAAS-managed machine and MAAS's bind9 serves its internal names. Letting the tailnet take over DNS would break that resolution. Flip it if MagicDNS is wanted here. -- **OPERATOR ACTION STILL REQUIRED:** the `10.10.0.0/22` route is ADVERTISED but must be **ENABLED - on the control server** (Headscale: `headscale routes enable`) before other tailnet nodes route to - Office1. Until then the node is reachable but the subnet is not. +- **ROUTE IS ENABLED** (operator, 2026-07-13). The `10.10.0.0/22` subnet route is live on the + self-hosted control server and **verified from the workstation** -- the OPNsense GUI answers + directly at `https://10.10.0.1` with **no SSH tunnel**. This SUPERSEDES the tunnel commands in the + table above: from a tailnet-joined workstation, every Office1 address in the `/22` is reachable + directly. The tunnels remain valid as a fallback when off-tailnet. - The auth key was consumed by path from a `0600` file and **shredded from the node** after use -- tailscaled holds its own node key now. It never entered an operator session or the repo. @@ -127,3 +131,43 @@ (seen twice: docker registry, tailscale.com). Retry before diagnosing. 6. **A `tofu apply` touching a `libvirt_domain`'s devices BOUNCES THE GUEST**, even when the plan says "updated in-place". Schedule it as an outage. + +--- + +## 6. IPAM -- what NetBox now says about this site (D-115 carve, imported 2026-07-13) + +The Office1 **sandbox** NetBox (`10.10.1.201:8000`) was seeded from the upstream draft (129 objects: +1 RIR, 23 roles, 5 regions, 10 sites, 90 prefixes) and then had the D-115 carve applied on top. +Site **`vr1-off1` ("VR1 Off1")**, region VR1. + +| Prefix | Role | Status | What it is | +|---|---|---|---| +| `10.10.0.0/16` | office | container | The Office role's v4 block -- `/22` per office (D-115) | +| `10.10.0.0/22` | office | container | **VR1 Off1's office `/22`** (`.2`/`.3` spare) | +| `10.10.0.0/24` | office | active | `office1-local` LAN -- Kea DHCP on the edge | +| `10.10.1.0/24` | office | active | `lxdbr0` LXD compose net -- MAAS DHCP | +| `172.30.0.0/16` | **edge** | container | The NEW Edge role -- simulated ISP/WAN (mirrors v6 `2602:f3e2:fe::/48`) | +| `172.30.1.0/24` | edge | active | `office1-wan` -- the simulated ISP uplink | +| `2602:f3e2:f01:100::/56` | office | container | VR1 Off1's v6 `/56` (mirrors VR0 Off0's `e01:100::/56`) | +| `2602:f3e2:f01:100::/64` | office | active | Office subnet -- **NOT YET DEPLOYED** (v6 does not egress the lab) | + +**Every one of these addresses was already running.** D-115 blesses what is deployed rather than +renumbering it -- the carve cost ZERO renumber. What it fixed is that `10.10.0.0/24`, `10.10.1.0/24` +and `172.30.1.0/24` were **SQUAT**: on the wire with no allocation behind them. + +**Naming (D-117):** VR1's datacenters are **DC0/DC1** (matching the apex, which already held both +sites) but **the office keeps its number** -- it is **Off1**. VR1 therefore reads DC0/DC1/**Off1** +while VR0 reads DC0/DC1/Off0. Accepted cosmetic asymmetry; the site is NEW upstream so nothing was +renamed and **no deployed object changed**. + +**Not yet upstream.** The sandbox is a simulation. Feeding this carve back to +`netbox.baldurkeep.com` is a separate, operator-gated write (`netbox/d115-office-carve.py` refuses a +non-sandbox target unless given `--yes-write-upstream`). + +**Reproduce the sandbox from scratch:** +``` +python3 netbox/prod-draft-dump.py --out netbox/draft/vr1-draft.json # on vcloud, READ-ONLY +# then on office1-netbox, with the sandbox token: +python3 sandbox-seed.py --draft draft/vr1-draft.json --commit +python3 d115-office-carve.py --commit +``` diff --git a/netbox/d115-office-carve.py b/netbox/d115-office-carve.py new file mode 100755 index 0000000..72d4cc7 --- /dev/null +++ b/netbox/d115-office-carve.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +""" +Apply the D-115 Office carve (and the roles it needs) to a NetBox. + +D-115 option (a), ADOPTED 2026-07-13. This BLESSES what is already running -- +every address here was MEASURED off the live Office1 build, not invented, so the +carve costs ZERO renumbering. What it fixes is that those addresses were SQUAT: +they existed on the wire with no allocation behind them. + +What D-115 found, and what this creates: + + 1. There is NO Office role in IPv4. Office1's LAN (10.10.0.0/24) and its LXD + compose net (10.10.1.0/24) were squatting. + -> new role `office` at 10.10.0.0/16, carved /22 per office. + VR1 Off1 = 10.10.0.0/22 -> LAN .0/24 + compose .1/24 (+ .2/.3 spare). + + 2. "Edge Networks" has a v6 /48 (2602:f3e2:fe::/48) but NO v4 counterpart, so + the simulated ISP uplink 172.30.1.0/24 was squatting inside the OOB /12. + -> new role `edge` at 172.30.0.0/16, its OWN role (NOT oob), carved out of + the existing 172.16.0.0/12 container. Office1's WAN becomes legitimate. + + 3. IPv6 has NO design freedom here -- the region pattern determines it: + VR1's office /48 (2602:f3e2:f01::/48) already exists upstream, and VR0's + Off0 shows the shape (e01:100::/56 container -> e01:100::/64 subnet). + -> VR1 Off1 = 2602:f3e2:f01:100::/56 container + :100::/64 office subnet. + + 4. That /56 needs a SITE. -> `vr1-off1` ("VR1 Off1"), region VR1. + +NAMING (D-117): VR1's DCs are dc0/dc1 (matching the apex) but the OFFICE KEEPS +ITS NUMBER -- it is Off1, site `vr1-off1`. That site does not exist upstream, so +it is CREATED, not renamed, and nothing deployed changes. VR1 therefore reads +DC0/DC1/Off1 while VR0 reads DC0/DC1/Off0: an accepted cosmetic asymmetry. + +NOT DONE HERE: VR1 DC1's v4 supernet (10.12.64.0/19, also D-115) belongs to +netbox/dc-dc-prefixes-import.py --dc dc1, which owns the six-plane carve. Doing +it in two places is how they drift. + +DRY BY DEFAULT. Nothing is written without --commit. + +WRITING UPSTREAM IS GATED IN CODE. Per the standing architecture the sim never +writes upstream casually: feeding refinements back to the production apex is an +operator decision. Pointing this at anything that is not an obvious sandbox +requires --yes-write-upstream ON TOP of --commit. + +Usage (sandbox, on office1-netbox): + sudo NETBOX_URL=http://localhost:8000 \ + NETBOX_TOKEN="$(cat /root/netbox-secrets/api.token)" \ + python3 d115-office-carve.py # preview + ... --commit # apply +""" + +import argparse +import ipaddress +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request + +UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md + +SITE = {"slug": "vr1-off1", "name": "VR1 Off1", "region": "vr1"} + +ROLES = [ + {"slug": "office", "name": "Office"}, # exists upstream; created if absent + {"slug": "edge", "name": "Edge"}, # NEW -- v6 had Edge Networks, v4 had nothing +] + +# Every CIDR below was MEASURED on the live Office1 build (docs/vr1-office1-as-built.md). +# status: container = an allocation block; active = a real subnet on the wire. +PREFIXES = [ + # --- IPv4: the Office role (D-115 gap 1) --- + ("10.10.0.0/16", "office", "container", None, + "Office (v4) -- /22 per office (D-115)"), + ("10.10.0.0/22", "office", "container", "vr1-off1", + "VR1 Off1 -- office /22 (D-115)"), + ("10.10.0.0/24", "office", "active", "vr1-off1", + "VR1 Off1 office1-local LAN -- Kea DHCP on the OPNsense edge (D-115)"), + ("10.10.1.0/24", "office", "active", "vr1-off1", + "VR1 Off1 LXD compose net (lxdbr0) -- MAAS DHCP (D-114/D-115)"), + + # --- IPv4: the Edge role (D-115 gap 4) --- + ("172.30.0.0/16", "edge", "container", None, + "Edge (v4) -- simulated ISP/WAN segments; mirrors v6 2602:f3e2:fe::/48 (D-115)"), + ("172.30.1.0/24", "edge", "active", "vr1-off1", + "VR1 Off1 office1-wan -- simulated ISP uplink (D-115)"), + + # --- IPv6: forced by the region pattern (D-115 gap 3) --- + ("2602:f3e2:f01:100::/56", "office", "container", "vr1-off1", + "VR1 Off1 (D-115) -- mirrors VR0 Off0 e01:100::/56"), + ("2602:f3e2:f01:100::/64", "office", "active", "vr1-off1", + "VR1 Off1 office subnet -- NOT YET DEPLOYED (v6 does not egress the lab)"), +] + +# A sandbox is local, or the known Office1 sandbox address. Anything else is +# treated as production and requires the explicit upstream flag. +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.201"} + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +class NB: + def __init__(self, base, token): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, + headers={"Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + "User-Agent": UA}) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " + "nbt_., not the API's bare `token` field.") + if exc.code == 403: + die(f"403 on {path}. If curl works with this token, it is the upstream " + f"User-Agent filter, NOT the token (references/platform-traps.md).") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def one(self, path, **flt): + res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") + return res["results"][0] if res["results"] else None + + def create(self, path, payload): + return self._req("POST", f"{path}/", payload) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--commit", action="store_true", + help="WRITE. Default is a DRY RUN that writes nothing.") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox. " + "Feeding the production apex is an operator decision.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set.") + + host = (urllib.parse.urlparse(url).hostname or url).lower() + is_sandbox = host in SANDBOX_HOSTS + + print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") + if args.commit and not is_sandbox and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{host}': it is not a known sandbox, so this is " + f"treated as the PRODUCTION apex. Feeding validated refinements upstream is " + f"an operator decision, not a side effect of running an import. Re-run with " + f"--yes-write-upstream if that is genuinely what you intend.") + + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" + if not args.commit else "\n*** COMMITTING. ***") + + nb = NB(url, token) + created = existing = 0 + + print("\nIPAM roles:") + for r in ROLES: + if nb.one("ipam/roles", slug=r["slug"]): + print(f" EXISTS role {r['slug']}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE role {r['slug']} ({r['name']})") + created += 1 + continue + o = nb.create("ipam/roles", {"name": r["name"], "slug": r["slug"]}) + print(f" CREATED role {r['slug']} (id={o['id']})") + created += 1 + + print("\nSite:") + site = nb.one("dcim/sites", slug=SITE["slug"]) + site_id = site["id"] if site else None + if site: + print(f" EXISTS site {SITE['slug']} (id={site_id})") + existing += 1 + else: + region = nb.one("dcim/regions", slug=SITE["region"]) + if region is None: + die(f"region '{SITE['region']}' is absent -- seed the draft first " + f"(netbox/sandbox-seed.py).") + if not args.commit: + print(f" [dry-run] would CREATE site {SITE['slug']} ({SITE['name']}) in region {SITE['region']}") + created += 1 + else: + o = nb.create("dcim/sites", {"name": SITE["name"], "slug": SITE["slug"], + "region": region["id"], "status": "active"}) + site_id = o["id"] + print(f" CREATED site {SITE['slug']} (id={site_id})") + created += 1 + + print("\nPrefixes:") + for cidr, role_slug, status, scope_slug, desc in PREFIXES: + ipaddress.ip_network(cidr) # fail loud on a typo, before any write + if nb.one("ipam/prefixes", prefix=cidr): + print(f" EXISTS {cidr}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE {cidr:<24} role={role_slug:<7} {status:<9} " + f"scope={scope_slug or '-'}") + created += 1 + continue + role = nb.one("ipam/roles", slug=role_slug) + if role is None: + die(f"role '{role_slug}' absent -- cannot place {cidr}") + payload = {"prefix": cidr, "role": role["id"], "status": status, "description": desc} + if scope_slug: + if site_id is None: + die(f"site '{scope_slug}' absent -- cannot scope {cidr}") + payload["scope_type"] = "dcim.site" + payload["scope_id"] = site_id + o = nb.create("ipam/prefixes", payload) + print(f" CREATED {cidr} (id={o['id']}) role={role_slug}") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") + if not args.commit: + print("DRY RUN -- nothing was written. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/d115-office-carve/run-tests.sh b/tests/d115-office-carve/run-tests.sh new file mode 100755 index 0000000..d3a5242 --- /dev/null +++ b/tests/d115-office-carve/run-tests.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# tests/d115-office-carve/run-tests.sh +# +# Harness for netbox/d115-office-carve.py. OFFLINE -- touches no NetBox. +# +# The carve's VALUES are the point. Every CIDR in that script was MEASURED off +# the live Office1 build; D-115 blesses exactly what is running, so a "tidy-up" +# that changes one of these silently turns a zero-renumber carve into a +# renumber. This harness pins them. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/d115-office-carve.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" + +# no env -> must fail loud, not guess a target +NETBOX_URL= NETBOX_TOKEN= python3 "$S" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" + +# DRY BY DEFAULT (the D-117 lesson). +grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" +grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" + +# WRITING UPSTREAM IS GATED IN CODE, not by discipline. Feeding the production +# apex is an operator decision; it must never be a side effect of an import. +grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" --commit 2>&1)" +printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ + || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" + +# THE D-115 VALUES. Each was measured on the live build; changing one costs a renumber. +for cidr in \ + "10.10.0.0/16" "10.10.0.0/22" "10.10.0.0/24" "10.10.1.0/24" \ + "172.30.0.0/16" "172.30.1.0/24" \ + "2602:f3e2:f01:100::/56" "2602:f3e2:f01:100::/64"; do + grep -qF "\"$cidr\"" "$S" && ok || bad "D-115 prefix $cidr is GONE from the carve" +done + +# The two roles D-115 exists to create. `edge` is the NEW one -- v6 had "Edge +# Networks", v4 had nothing, so the WAN was squatting inside the OOB /12. +grep -q '"slug": "office"' "$S" && ok || bad "the office role is gone" +grep -q '"slug": "edge"' "$S" && ok || bad "the edge role is gone -- the WAN goes back to squatting in OOB" + +# The office KEEPS its number (D-117 operator ruling): site is vr1-off1, NOT off0. +grep -q '"slug": "vr1-off1"' "$S" && ok || bad "site slug is not vr1-off1 (D-117: the office keeps its number)" +grep -q 'vr1-off0' "$S" && bad "vr1-off0 appears -- the office was NOT renamed (D-117)" || ok + +# The DC v4 supernet must NOT be duplicated here -- it belongs to the six-plane +# importer. Two owners for one prefix is how they drift apart. +# (match the PREFIXES table, not the docstring that explains why it is absent) +grep -qE '^\s*\("10\.12\.64\.0/19"' "$S" \ + && bad "DC1's v4 supernet is duplicated here -- it belongs to dc-dc-prefixes-import.py" || ok + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "d115-office-carve: $pass/$total PASS"; exit 0; fi +echo "d115-office-carve: $fail/$total FAIL"; exit 1