diff --git a/docs/changelog-20260715-d120-office1-service-reip.md b/docs/changelog-20260715-d120-office1-service-reip.md new file mode 100644 index 0000000..34273c3 --- /dev/null +++ b/docs/changelog-20260715-d120-office1-service-reip.md @@ -0,0 +1,66 @@ +# Changelog 2026-07-15 -- D-120 ADOPTED: Office1 service re-IP into the static band + +## What changed + +Adopted **D-120** (static-services addressing convention for MAAS-managed compose /24s) as +**option (a), executed**, and re-IP'd Office1's two running services out of the MAAS node band into +the `.2-.49` static band: + + office1-netbox 10.10.1.201 -> 10.10.1.10 + office1-tailscale 10.10.1.202 -> 10.10.1.11 + +No wipe. Both services verified on the wire: +- NetBox: HTTP 200 at `http://10.10.1.10:8000/login/`, reachable vcloud-side; old `.201` dead. +- Tailscale: `BackendState=Running`, `PrimaryRoutes=["10.10.0.0/22"]` unchanged, tailnet + `100.64.0.53` unchanged; old `.202` dead. + +## How (method, for the DC replication that reuses this) + +- **MAAS could NOT be updated** -- it refuses interface edits on a **Deployed** machine ("Cannot + unlink subnet interface because the machine is not New, Ready, Allocated, or Broken"). The only + lift is Release->redeploy = WIPE (forbidden; destroys the seeded sandbox). So MAAS-model drift is + ACCEPTED: MAAS still shows `.201`/`.202` `mode=auto`. It reconciles for free at the next + teardown/redeploy. No collision: MAAS auto-assigns nodes from `.201+` upward (observed -- the two + services themselves landed at `.201`/`.202`), so the low `.2-.49` band is off its allocation path. + (Being outside the dynamic range is NOT the guarantee; the whole non-dynamic space is the static pool.) +- The live address was set on each **guest** via BOTH `/etc/netplan/50-cloud-init.yaml` and + `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (the latter so it survives a reboot / + cloud-init re-render), then `netplan apply`. Backups saved on each guest: `.bak-reip201` / + `.bak-reip202`. +- Driven over **`lxc exec` from voffice1**, NOT SSH-to-the-guest-IP -- the LXD socket is immune to + the `netplan apply` that swaps the address, so the control channel never dropped. `lxc exec`/ + `console` was also confirmed as the recovery backstop BEFORE the first apply. +- NetBox `ALLOWED_HOSTS` is `*` (config default, not overridden in `env/netbox.env`), so it served + at `.10` with no restart. Tailscale re-asserted automatically (no `tailscale up` needed). + +## Repo references swept (`.201->.10`, `.202->.11`) + +- **Code:** `SANDBOX_HOSTS` allowlist in `netbox/dc-dc-prefixes-import.py`, + `netbox/roles-aggregates-import.py`, `netbox/d115-office-carve.py`; + `tests/dc-dc-prefixes-import/test_logic.py` (both sandbox-URL hosts). The on-box working copy + `/home/ubuntu/nbimport/netbox/d115-office-carve.py` was updated too (else a `--commit` against the + now-`.10` sandbox would be REFUSED as non-sandbox). +- **Docs:** `docs/vr1-office1-as-built.md` (+ a MAAS-model-drift caveat), `docs/session-ledger.md`, + `docs/dc-dc-netbox-buildout-scope.md`, `docs/dc-dc-deployment-workflow.md`, + `runbooks/dc-dc-phase1-office1-standup.md`, `docs/design-decisions.md` (D-120 flip + Execution), + `runbooks/dc-dc-office1-service-reip.md` (PENDING markers resolved to the measured method). +- Dated changelogs left as historical snapshots (they record the deploy-time `.201`/`.202`). + +## Gauntlet + +`bash scripts/run-tests-all.sh` -> ALL GREEN (58 harnesses). `bash scripts/repo-lint.sh` -> 0 fail, +1 documented legacy WARN (design-decisions non-ASCII carve-out; my additions are ASCII-clean). + +## Still owed (NOT done here) + +- **D-120 Step 6 / C2:** register the `10.10.1.0/24` child ranges (static `.2-.49` / dynamic + `.100-.200` / node `.201-.254`) and the two service IPs IN the sandbox NetBox, then feed upstream. + Rides with C2. + +## Revert + +- **Repo:** `git revert ` restores the `.201`/`.202` references. +- **Live services:** on each guest via `lxc exec`, restore `/etc/netplan/50-cloud-init.yaml` and + `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` from the `.bak-reip201` / `.bak-reip202` backups, + then `netplan apply`. MAAS needs no rollback (its model was never changed). The on-box importer + copy reverts with `sed -i s#10.10.1.10#10.10.1.201#g /home/ubuntu/nbimport/netbox/d115-office-carve.py`. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index bcec3ae..91ef8bf 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -113,8 +113,8 @@ EXISTS and is RUNNING** (`module.voffice1` is in `opentofu/terraform.tfstate`). - **DONE (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5 ON `voffice1`; that LXD **registered back into MAAS as an LXD VM host**; and both service - machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.201, NetBox 4.6.4) and - `office1-tailscale` (10.10.1.202, subnet router for `10.10.0.0/22`, route now ENABLED on the + machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.10, NetBox 4.6.4) and + `office1-tailscale` (10.10.1.11, subnet router for `10.10.0.0/22`, route now ENABLED on the self-hosted control server). GitBucket is out of scope (D-116), so Office1's composed machines are exactly TWO. Codified in `scripts/site-headend-install.sh` (harness 18/18) -- reusable for the DCs. diff --git a/docs/dc-dc-netbox-buildout-scope.md b/docs/dc-dc-netbox-buildout-scope.md index d992dd1..c815cb2 100644 --- a/docs/dc-dc-netbox-buildout-scope.md +++ b/docs/dc-dc-netbox-buildout-scope.md @@ -195,7 +195,7 @@ NetBox `netbox.baldurkeep.com` is the IPAM apex and source of truth (D-010, D-103). The section-4 buildout mutates it (roles, aggregates, org ULA /48, two DCs of six-plane prefixes). We do NOT -rehearse those mutations on the apex. A SECOND NetBox -- `office1-netbox`, `10.10.1.201:8000`, +rehearse those mutations on the apex. A SECOND NetBox -- `office1-netbox`, `10.10.1.10:8000`, NetBox 4.6.4 on the Office1 VM (`docs/changelog-20260713-office1-netbox-deployed.md`) -- is the SANDBOX. The upstream draft is copied in, the planned changes are applied and proven there, and only then is the validated delta fed back upstream as a separate, operator-gated write. @@ -247,7 +247,7 @@ |---|---|---| | `prod-draft-dump.py` | read-only; no `--commit` | N/A -- performs no writes | | `sandbox-seed.py` | DRY; `--commit` writes | STRUCTURAL: holds no upstream URL/token and dies if pointed at the draft's own source host -- it CANNOT reach upstream | -| `d115-office-carve.py` | DRY; `--commit` writes | `SANDBOX_HOSTS` allowlist `{localhost,127.0.0.1,10.10.1.201}`; a non-sandbox `--commit` REFUSES without `--yes-write-upstream` | +| `d115-office-carve.py` | DRY; `--commit` writes | `SANDBOX_HOSTS` allowlist `{localhost,127.0.0.1,10.10.1.10}`; a non-sandbox `--commit` REFUSES without `--yes-write-upstream` | | `roles-aggregates-import.py` | PREVIEW; `--commit` writes | **`SANDBOX_HOSTS` + `--yes-write-upstream` (added 2026-07-14)** -- plus a whole-plan PREFLIGHT so it cannot half-write | | `dc-dc-prefixes-import.py` | DRY; `--commit` writes | **`SANDBOX_HOSTS` + `--yes-write-upstream` (added 2026-07-14)** -- plus a whole-plan role PREFLIGHT so it cannot half-write | diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 67783a5..1f19fb5 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -3153,12 +3153,15 @@ in a repo that operates more than one cloud is a latent off-by-one waiting for a reader who does not know which cloud the sentence is about. -## D-120: static-services addressing convention for the MAAS-managed compose /24s (PROPOSED) +## D-120: static-services addressing convention for the MAAS-managed compose /24s (ADOPTED) -**Status:** **PROPOSED 2026-07-15.** Operator endorsed the direction (static services belong in a low -band, not mixed with deployed nodes); this entry ratifies the exact band boundaries and the Office1 -service addresses. Refines D-115 (which ratified that `10.10.1.0/24` exists as the Office1 compose -net but left its INTERNAL layout as just "MAAS DHCP/PXE"). Does NOT reopen D-115's /22 carve. +**Status:** **ADOPTED 2026-07-15 -- option (a), EXECUTED.** Operator endorsed the direction (static +services belong in a low band, not mixed with deployed nodes) and ruled "it doesn't matter where the +IPs land in the sub-50 scope; go ahead with changing the IPs." The band layout below and the Office1 +service addresses (`.10`/`.11`) are ratified, and the Office1 re-IP was executed the same day (see +"Execution" and `runbooks/dc-dc-office1-service-reip.md`). Refines D-115 (which ratified that +`10.10.1.0/24` exists as the Office1 compose net but left its INTERNAL layout as just "MAAS +DHCP/PXE"). Does NOT reopen D-115's /22 carve. ### The problem @@ -3199,10 +3202,12 @@ ### Consequence / cost (why this is not a free change) -Re-IPing Office1's two RUNNING services cascades to 20 repo references, INCLUDING the hardcoded -`SANDBOX_HOSTS = {localhost, 127.0.0.1, 10.10.1.201}` allowlist in `roles-aggregates-import.py` and -`dc-dc-prefixes-import.py` (added 2026-07-14) plus its test. See the re-IP plan -(`runbooks/dc-dc-office1-service-reip.md`). It is a gated mutation on live services, not done here. +Re-IPing Office1's two RUNNING services cascaded to the repo references, INCLUDING the hardcoded +`SANDBOX_HOSTS` allowlist (`{localhost, 127.0.0.1, 10.10.1.201}` -> `.10`) in +`roles-aggregates-import.py`, `dc-dc-prefixes-import.py`, and `d115-office-carve.py` (added +2026-07-14) plus its test. The sweep + gauntlet ran with the re-IP (see "Execution" above and +`runbooks/dc-dc-office1-service-reip.md`). Dated changelogs keep `.201`/`.202` as historical +snapshots. ### Options @@ -3216,3 +3221,36 @@ exists to remove. **Recommendation: (a).** Get the template right once. + +### Execution (2026-07-15) -- option (a), done + +Re-IP executed on the two running services, no wipe, verified on the wire: + + office1-netbox 10.10.1.201 -> 10.10.1.10 NetBox HTTP 200 at .10, reachable vcloud-side + office1-tailscale 10.10.1.202 -> 10.10.1.11 tailscale Running, still advertises 10.10.0.0/22 + +Two findings that shaped the method (and that the DC replication inherits): + +- **MAAS will NOT edit an interface on a Deployed machine** ("Cannot unlink subnet interface because + the machine is not New, Ready, Allocated, or Broken"). The only lift is Release->redeploy, which + WIPES the OS -- forbidden here (it would destroy the seeded sandbox). So the durable MAAS-model + update is unavailable without a wipe. Resolution: set the live address on the GUEST only, via BOTH + `/etc/netplan/50-cloud-init.yaml` and `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (so it + survives reboot), and ACCEPT MAAS-model drift (MAAS still shows `.201`/`.202` `mode=auto`). The + drift reconciles for free at the next teardown/redeploy from a corrected model. `.10`/`.11` are + off MAAS's auto-assign path -- MAAS allocates nodes from `.201+` upward (observed: the two services + themselves landed at `.201`/`.202`), so the low `.2-.49` band is never auto-handed out. NB: the + whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee; + the allocation DIRECTION is. A DC building fresh sidesteps this by MAAS-static-assigning `.2-.49` + before deploy (Step 1 below), which pins the addresses explicitly rather than relying on direction. +- **NetBox `ALLOWED_HOSTS` = `*`** (config default; not overridden in `env/netbox.env`), so NetBox + served at `.10` with no `DisallowedHost` change needed. +- **Method note for the DCs:** drive the guest change over `lxc exec` from the containment VM, NOT + SSH-to-the-guest-IP -- the LXD socket is immune to the `netplan apply` that swaps the address, so + the control channel never drops. `lxc console`/`exec` is also the recovery backstop if the new + address does not come up. For a DC building fresh, prefer static-assigning `.2-.49` in MAAS BEFORE + deploy (a Ready machine accepts the link edit) so there is no drift to accept -- the Deployed-block + above only bit Office1 because its services were already deployed in the node band. + +The NetBox-side carve record (Step 6 -- register the `.2-.49`/dynamic/node child ranges + the two +service IPs in the sandbox, then feed upstream) is still OWED and rides with C2; it is NOT done here. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 38790ef..3c05954 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -88,14 +88,16 @@ ## OPEN WORK -- VR1 DC-DC (the current mission) -**D-120 PROPOSED (2026-07-15) -- static-services addressing for the compose /24s.** Office1's -`office1-netbox` (10.10.1.201) and `office1-tailscale` (10.10.1.202) sit in MAAS's deployed-node -band, not distinguishable from ephemeral nodes. D-120 proposes a functional band layout -(`.2-.49` static services / `.100-.200` dynamic / `.201+` nodes) for every MAAS compose /24, and -re-IPing Office1's two services to `.10`/`.11`. **AWAITS operator ratification of the addresses.** -Re-IP plan written: `runbooks/dc-dc-office1-service-reip.md` (gated; NOT executed). Cascades to 20 -repo refs incl. the hardcoded `SANDBOX_HOSTS` `.201` in two importers. DCs inherit the layout from -day one. This is a NEW open item, separate from Stage 2 close-out (C2 is still the only C-item left). +**D-120 ADOPTED + EXECUTED (2026-07-15) -- static-services addressing for the compose /24s.** +Office1's two services were re-IP'd OUT of the MAAS node band into the `.2-.49` static band: +`office1-netbox .201 -> 10.10.1.10`, `office1-tailscale .202 -> 10.10.1.11`. On the wire, no wipe +(NetBox HTTP 200 at .10; tailscale still advertises `10.10.0.0/22`). Band layout ratified: `.2-.49` +static / `.100-.200` dynamic / `.201+` nodes, for every MAAS compose /24; DCs inherit it from day +one. Method finding: MAAS won't edit a Deployed machine's interface (lift = redeploy = WIPE), so the +address is set on the guest (netplan + curtin, over `lxc exec`) and MAAS-model drift is accepted +(reconciles at next teardown/redeploy). Sweep + gauntlet done; `runbooks/dc-dc-office1-service-reip.md` +now records the measured method. **Still owed:** D-120 Step 6 -- register the /24 child ranges + the +two service IPs in the sandbox NetBox and feed upstream; rides with C2 (the only C-item left). The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling @@ -113,8 +115,8 @@ **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. - **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.201) runs **NetBox - 4.6.4**; `office1-tailscale` (10.10.1.202, tailnet 100.64.0.53) is a subnet router advertising + **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.10) runs **NetBox + 4.6.4**; `office1-tailscale` (10.10.1.11, tailnet 100.64.0.53) is a subnet router advertising `10.10.0.0/22` on the operator's **SELF-HOSTED** control plane. The edge routes `10.10.1.0/24 -> 10.10.0.20`. Full connection reference: **`docs/vr1-office1-as-built.md`** -- read that first for any "how do I reach X" question. diff --git a/docs/vr1-office1-as-built.md b/docs/vr1-office1-as-built.md index fd42910..c210ffc 100644 --- a/docs/vr1-office1-as-built.md +++ b/docs/vr1-office1-as-built.md @@ -41,13 +41,27 @@ | **vcloud** | `10.17.11.248` (lab), `10.10.0.10` on `virbr2`, `172.30.1.1` on `virbr11` | libvirt/KVM hypervisor; OpenTofu runs from here | L1 (itself a KVM guest) | | **office1-opnsense** | LAN `10.10.0.1` / WAN `172.30.1.2` | OPNsense 26.1 edge router: routing, NAT, **Kea DHCP**, firewall | L2 | | **voffice1** | `10.10.0.20` (Kea reservation) | **MAAS 3.7.2** region+rack + PostgreSQL 16.14; **LXD 5.21.5** (registered to MAAS as VM host `office1-lxd`) | L2 | -| **office1-netbox** | `10.10.1.201` | **NetBox 4.6.4** (netbox-docker; Django 6.0.6) on `:8000` | L3 (LXD VM) | -| **office1-tailscale** | `10.10.1.202` / tailnet **`100.64.0.53`** (`fd7a:115c:a1e0::35`) | **Tailscale 1.98.8**, subnet router advertising `10.10.0.0/22`. Joined the **SELF-HOSTED** control plane `https://tailscale.baldurkeep.com:443` | L3 (LXD VM) | +| **office1-netbox** | `10.10.1.10` | **NetBox 4.6.4** (netbox-docker; Django 6.0.6) on `:8000` | L3 (LXD VM) | +| **office1-tailscale** | `10.10.1.11` / tailnet **`100.64.0.53`** (`fd7a:115c:a1e0::35`) | **Tailscale 1.98.8**, subnet router advertising `10.10.0.0/22`. Joined the **SELF-HOSTED** control plane `https://tailscale.baldurkeep.com:443` | L3 (LXD VM) | | ~~office1-gitbucket~~ | -- | **NOT BUILT -- AND WILL NOT BE (D-116).** `git.baldurkeep.com` remains the git service of record; new repos are created there as deployment tasks require. | -- | `office1-netbox` and `office1-tailscale` are **MAAS-composed LXD VMs** (D-114) -- MAAS enlisted, commissioned, deployed and powers them. The `voffice1` LXD host is `office1-lxd` (vm-host id 1). +**D-120 static-band re-IP (executed 2026-07-15).** Both services were moved from the MAAS +node band (`.201`/`.202`) into the static-services band: `office1-netbox .201 -> .10`, +`office1-tailscale .202 -> .11`. **MAAS-model drift, by design:** MAAS refuses to edit an +interface on a **Deployed** machine (the only lift is Release->redeploy, which would WIPE the +seeded sandbox), so MAAS's interface model still records `.201`/`.202` as `mode=auto`. The LIVE +address is set on each guest via `/etc/netplan/50-cloud-init.yaml` AND +`/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (both edited, so it survives reboot). The drift +reconciles for free at the next teardown/redeploy, when MAAS re-renders from a corrected model. +MAAS auto-assigns nodes from `.201+` upward (observed: netbox/tailscale themselves landed at +`.201`/`.202`), so the low `.2-.49` band is off its allocation path -- no collision. (Being outside +the dynamic range is NOT itself the guarantee; the whole non-dynamic space is MAAS's static pool.) +Guest +control channel during the change was `lxc exec` (LXD socket), immune to the IP swap. + --- ## 3. How to reach each thing @@ -64,8 +78,8 @@ | **OPNsense GUI** | `https://10.10.0.1` | `ssh -L 8443:10.10.0.1:443 jessea123@10.17.11.248` -> `https://localhost:8443` | | **OPNsense shell** | `ssh -i root@10.10.0.1` | via vcloud. **root's shell is `tcsh`** -- feed commands to `sh -s`. | | **MAAS UI** | `http://10.10.0.20:5240/MAAS` | `ssh -L 5240:10.10.0.20:5240 jessea123@10.17.11.248` -> `http://localhost:5240/MAAS` | -| **NetBox UI/API** | `http://10.10.1.201:8000` (needs the route below) | on tailnet: `http://10.10.1.201:8000` direct. Else `ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.201:8000 jessea123@10.10.0.20` -> `http://localhost:8000` | -| **office1-netbox / office1-tailscale** (shell) | `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201` | chain both jumps | +| **NetBox UI/API** | `http://10.10.1.10:8000` (needs the route below) | on tailnet: `http://10.10.1.10:8000` direct. Else `ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.10:8000 jessea123@10.10.0.20` -> `http://localhost:8000` | +| **office1-netbox / office1-tailscale** (shell) | `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.10` | chain both jumps | **Routing to the compose net.** The OPNsense edge has a static route `10.10.1.0/24 -> 10.10.0.20` (gateway object `OFFICE1_LXD_GW`, priority 255, NOT the default gateway). So any `office1-local` @@ -143,7 +157,7 @@ ## 6. IPAM -- what NetBox now says about this site (D-115 carve, imported 2026-07-13) -The Office1 **sandbox** NetBox (`10.10.1.201:8000`) was seeded from the upstream draft (129 objects: +The Office1 **sandbox** NetBox (`10.10.1.10:8000`) was seeded from the upstream draft (129 objects: 1 RIR, 23 roles, 5 regions, 10 sites, 90 prefixes) and then had the D-115 carve applied on top. Site **`vr1-off1` ("VR1 Off1")**, region VR1. diff --git a/netbox/d115-office-carve.py b/netbox/d115-office-carve.py index 72d4cc7..a25e13a 100755 --- a/netbox/d115-office-carve.py +++ b/netbox/d115-office-carve.py @@ -95,7 +95,7 @@ # A sandbox is local, or the known Office1 sandbox address. Anything else is # treated as production and requires the explicit upstream flag. -SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.201"} +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} def die(msg: str): diff --git a/netbox/dc-dc-prefixes-import.py b/netbox/dc-dc-prefixes-import.py index 2f2666f..f6471fd 100644 --- a/netbox/dc-dc-prefixes-import.py +++ b/netbox/dc-dc-prefixes-import.py @@ -159,7 +159,7 @@ # exact off-by-one D-117 closed and D-119 made unrepresentable. # A sandbox is local or the known Office1 sandbox address; anything else is treated # as the production apex and needs --yes-write-upstream (see the guard in main()). -SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.201"} +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} SITES = { "vr1-dc0": {"slug": "vr1-dc0", "name": "VR1 DC0"}, diff --git a/netbox/roles-aggregates-import.py b/netbox/roles-aggregates-import.py index 6e96a34..7243bc4 100644 --- a/netbox/roles-aggregates-import.py +++ b/netbox/roles-aggregates-import.py @@ -102,7 +102,7 @@ # A sandbox is local or the known Office1 sandbox address; anything else is treated # as the production apex and needs --yes-write-upstream (guard in main()). -SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.201"} +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} ROLES = [ ("provider-public", "Provider Public", 100, diff --git a/runbooks/dc-dc-office1-service-reip.md b/runbooks/dc-dc-office1-service-reip.md index d44f835..3992e27 100644 --- a/runbooks/dc-dc-office1-service-reip.md +++ b/runbooks/dc-dc-office1-service-reip.md @@ -1,10 +1,11 @@ # Office1 service re-IP -- move NetBox + Tailscale into the D-120 static band -**Gated mutation on TWO RUNNING services.** Do not run until **D-120 is ADOPTED** (the operator has -ratified the band layout AND the target addresses). Present each step, justify it as minimal, get -individual approval. This is also the reference procedure the DCs reuse for their own compose /24s. +**EXECUTED 2026-07-15** (D-120 ADOPTED, option (a)). Both services re-IP'd, no wipe, verified on the +wire. The PENDING-VERIFICATION markers below are now RESOLVED with the measured method; this is the +reference procedure the DCs reuse for their own compose /24s. It remains a gated mutation on running +services -- present each step, justify it as minimal, get individual approval -- if ever re-run. -## Target (D-120 PROPOSED; confirm against the adopted decision before running) +## Target (EXECUTED) office1-netbox 10.10.1.201 -> 10.10.1.10 (static, band .2-.49) office1-tailscale 10.10.1.202 -> 10.10.1.11 (static, band .2-.49) @@ -20,38 +21,57 @@ 4. Access: `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201` (and `.202`); MAAS on `voffice1` (`10.10.0.20`), secrets in `/root/maas-secrets/` on the box (operator-held). -## Step 1 -- MAAS: static-assign the new address to each VM's interface [MUTATION: gated] +## Step 0 -- Confirm the recovery backstop FIRST (READ-ONLY, blocking) -For each machine, set its interface on the `10.10.1.0/24` subnet to **static-assign** the target -address (a fixed IP outside MAAS's `.100-.200` dynamic range). The `.2-.49` band is already outside -the dynamic range, so NO dynamic-range edit is required. +Before severing any network path, prove `lxc exec`/`console` reaches each guest independent of guest +networking -- it rides the LXD socket, so it survives the re-IP and is the recovery path if the new +address does not come up. On `voffice1`: - Network -> the interface on - 10.10.1.0/24 -> Edit -> Mode = "Static assign" -> IP = 10.10.1.10 (netbox) / .11 (tailscale). - CLI form is roughly `maas $PROFILE interface link-subnet mode=STATIC - subnet= ip_address=10.10.1.10` -- CONFIRM the subcommand/args against the installed - MAAS 3.7 before running; do not run a guessed command against live MAAS.> + sudo -n lxc list # instance names (office1-netbox / -tailscale) + sudo -n lxc exec office1-netbox -- hostname # must return -- proves the backstop -## Step 2 -- Apply the new address on the VM [MUTATION: gated] +## Step 1 -- MAAS: NOT AVAILABLE on a Deployed machine (finding) -A composed VM picks up the MAAS assignment on a network reconfigure or redeploy. +**MAAS refuses interface edits on a Deployed machine:** `interface unlink-subnet ...` returns +"Cannot unlink subnet interface because the machine is not New, Ready, Allocated, or Broken." The +only way to make it Ready is Release, which **WIPES the OS** -- forbidden (it destroys the seeded +sandbox). So the MAAS-model update is skipped and MAAS-model drift is ACCEPTED (MAAS keeps showing +`.201`/`.202` `mode=auto`). Drift reconciles for free at the next teardown/redeploy. `.10`/`.11` are +off MAAS's auto-assign path: MAAS allocates nodes from `.201+` upward (observed -- the two services +themselves landed at `.201`/`.202`), so the low `.2-.49` band is never auto-handed out. (NB: the +whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee -- +the allocation DIRECTION is.) - +> **DC replication:** a DC building fresh should static-assign `.2-.49` in MAAS BEFORE deploy (a +> Ready machine DOES accept `interface link-subnet subnet= mode=STATIC +> ip_address=`), so there is no drift. The Deployed-block only bit Office1 because its services +> were already deployed in the node band. -## Step 3 -- Fix each SERVICE's own address config [MUTATION: gated] +## Step 2 -- Set the address on the GUEST (both files, via lxc exec) [MUTATION: gated] -- **NetBox (`office1-netbox`, netbox-docker):** NetBox binds `0.0.0.0:8000`, so it follows the host - IP -- BUT `ALLOWED_HOSTS` / `CSRF_TRUSTED_ORIGINS` may name the old `10.10.1.201`. - `` -- **Tailscale (`office1-tailscale`):** it advertises `10.10.0.0/22` as a subnet router; the ADVERTISED - routes do not change, but its own LAN interface address does. `` +Drive this over `lxc exec` from `voffice1`, NOT SSH-to-the-guest-IP: the LXD socket is immune to the +`netplan apply` that swaps the address, so your control channel never drops. Edit BOTH files (netplan +takes effect now; the curtin cfg makes it survive a reboot / cloud-init re-render), then apply: + + for f in /etc/netplan/50-cloud-init.yaml /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg; do + sudo -n lxc exec office1-netbox -- cp "$f" "$f.bak-reip201" + sudo -n lxc exec office1-netbox -- sed -i 's#10.10.1.201/24#10.10.1.10/24#' "$f" + done + sudo -n lxc exec office1-netbox -- netplan apply # gateway4-deprecated warnings are benign + sudo -n lxc exec office1-netbox -- ip -4 addr show enp5s0 | grep inet # verify .10, do not assume + +(Same for `office1-tailscale`: `.202/24 -> .11/24`, backups `.bak-reip202`.) + +## Step 3 -- Each SERVICE's own address config [MEASURED] + +- **NetBox (`office1-netbox`, netbox-docker):** binds `0.0.0.0:8000`, follows the host IP. + `ALLOWED_HOSTS` is **`*`** (config default in `/opt/netbox-docker/configuration/configuration.py`; + NOT overridden in `env/netbox.env`), so NO change and NO restart was needed -- it served at `.10` + immediately (HTTP 200 at `http://10.10.1.10:8000/login/`). +- **Tailscale (`office1-tailscale`):** re-asserted automatically after the interface IP change -- + `BackendState=Running`, `PrimaryRoutes=["10.10.0.0/22"]` unchanged, tailnet IP `100.64.0.53` + unchanged. NO `tailscale up` re-assert was needed. (If a future run shows the route dropped, + re-assert with `tailscale up --advertise-routes=10.10.0.0/22`; node identity/key is unchanged.) ## Step 4 -- Verify on the wire (READ-ONLY) @@ -59,18 +79,20 @@ curl -s -o /dev/null -w "%{http_code}\n" http://10.10.1.10:8000/api/ # 403/200 = up ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.11 'tailscale status' # tailscale at .11 -## Step 5 -- Update the 20 repo references + the importer allowlist [repo change; gated] +## Step 5 -- Update the repo references + the importer allowlist [DONE 2026-07-15] -`10.10.1.201 -> 10.10.1.10` and `10.10.1.202 -> 10.10.1.11` across (measured 2026-07-15): +`10.10.1.201 -> 10.10.1.10` and `10.10.1.202 -> 10.10.1.11` applied across: -- **CODE (must, or the guard breaks):** `SANDBOX_HOSTS` in `netbox/roles-aggregates-import.py` and - `netbox/dc-dc-prefixes-import.py` (the `.201` entry -> `.10`); `netbox/d115-office-carve.py`'s - `SANDBOX_HOSTS`; `tests/dc-dc-prefixes-import/test_logic.py` (the sandbox-URL host it sets); - `tests/roles-aggregates-import/run-tests.sh` (the SANDBOX_HOSTS assertion). -- **Docs (forward-looking):** `docs/vr1-office1-as-built.md`, `docs/session-ledger.md`, +- **CODE (done):** `SANDBOX_HOSTS` `.201 -> .10` in `netbox/roles-aggregates-import.py`, + `netbox/dc-dc-prefixes-import.py`, and `netbox/d115-office-carve.py`; + `tests/dc-dc-prefixes-import/test_logic.py` (both sandbox-URL hosts). + `tests/roles-aggregates-import/run-tests.sh` only greps for the literal `SANDBOX_HOSTS` token and + uses `localhost` (still in the allowlist) -- no address change needed there. +- **Docs (done):** `docs/vr1-office1-as-built.md`, `docs/session-ledger.md`, `docs/dc-dc-netbox-buildout-scope.md`, `docs/dc-dc-deployment-workflow.md`, - `runbooks/dc-dc-phase1-office1-standup.md`. Leave dated changelogs as historical snapshots. -- Re-run `bash scripts/run-tests-all.sh` (the SANDBOX_HOSTS harnesses must stay green) + `repo-lint`. + `runbooks/dc-dc-phase1-office1-standup.md`, and D-120 itself. Dated changelogs LEFT as historical + snapshots (they record the deploy-time `.201`/`.202`). +- Gauntlet: `bash scripts/run-tests-all.sh` + `bash scripts/repo-lint.sh` re-run green. ## Step 6 -- Record the D-120 carve in NetBox (via the sandbox loop) @@ -80,7 +102,9 @@ ## Rollback -Re-run Steps 1-3 with the old addresses (`.201`/`.202`), `git revert` the Step-5 repo change. Nothing +Restore each guest's saved netplan + curtin backups (`.bak-reip201` / `.bak-reip202`) via `lxc exec` +and `netplan apply`, then `git revert` the Step-5 repo change. MAAS needs no rollback (its model was +never changed -- see Step 1). Nothing here is destructive -- the VMs and their data are untouched; only the L3 address changes. ## DC replication note diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index dccad26..2cd2f9a 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -703,7 +703,7 @@ | Phase | Host | NetBox | Token | |---|---|---|---| | dump (read upstream) | `vcloud` | `netbox.baldurkeep.com` (apex, READ-ONLY) | upstream, `~/vr1-office1-creds/vr1-netbox.env` | -| seed / carve / import / verify | `office1-netbox` (`10.10.1.201`) | sandbox `localhost:8000` | sandbox (operator-held on the box) | +| seed / carve / import / verify | `office1-netbox` (`10.10.1.10`) | sandbox `localhost:8000` | sandbox (operator-held on the box) | `pynetbox` is on `office1-netbox`, NOT on vcloud -- the two pynetbox importers (`roles-aggregates-import.py`, `dc-dc-prefixes-import.py`) MUST run there. diff --git a/tests/dc-dc-prefixes-import/test_logic.py b/tests/dc-dc-prefixes-import/test_logic.py index 977d61b..329ba5b 100644 --- a/tests/dc-dc-prefixes-import/test_logic.py +++ b/tests/dc-dc-prefixes-import/test_logic.py @@ -211,7 +211,7 @@ sys.argv = old_argv -os.environ["NETBOX_URL"] = "http://10.10.1.201:8000" # the known sandbox (guard passes) +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # the known sandbox (guard passes) os.environ["NETBOX_TOKEN"] = "fake-token" os.environ["ORG_ULA_48"] = "fd12:3456:789a::/48" @@ -440,7 +440,7 @@ # A sandbox host needs NO flag -- the default C2-safe path for the sim. fake_sb = fake_pynetbox.FakeApi() fake_sb.preseed_roles(T.PLANE_ORDER) -os.environ["NETBOX_URL"] = "http://10.10.1.201:8000" # known sandbox +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # known sandbox with captured_stdout(): rc_sb = run_main(["--dc", "vr1-dc0", "--commit"], fake_sb) check(rc_sb == 0, "guard: --commit to a SANDBOX host needs no flag")