diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 277afd2..168fa3a 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -86,27 +86,57 @@ | **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but LESS new than it looks: the MAAS-composed-LXD-VM half is **exactly what VR0 runs today** (its `lxd` KVM host at LXD 5.21.4, with `tailscale` composed into it at 2 cores / 2 GiB / 25 GB) and carries **NO Roosevelt delta**. The containment VM itself IS a deliberate simulation-only delta -- it is what makes a "hard-down an entire facility" DR drill (`virsh destroy vdc1`) executable at all. NetBox-as-IPAM-apex discipline and the `netbox/*-import.py` pattern transfer unchanged. | | **Authoring status** | **Runbook REWRITTEN 2026-07-13 to the D-114 model: `runbooks/dc-dc-phase1-office1-standup.md`.** The old Option A / Option B provisioning fork is **CLOSED** -- OpenTofu (Option A) is the path and `module "voffice1"` is a real, applied `cloudinit-vm` instantiation (real image source, real `user_data`/`meta_data`/`network_config`, plus the new `expose_nested_virt` variable). **PENDING VERIFICATION:** the MAAS/LXD install + LXD-KVM-host registration + VM-compose command lines are NOT yet confirmed against Canonical's current docs -- the runbook marks each unconfirmed flag/channel/subcommand rather than fabricating one, and a separate verification pass owns filling them in. | -**State (2026-07-13): PARTIALLY EXECUTED -- substrate DONE, site half remaining.** +### !!! STAGE 2 IS THE BRANCH MERGE POINT (operator ruling, 2026-07-14) !!! -- **DONE:** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is BUILT and - LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on udp/67). - Config is now REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1` - EXISTS and is RUNNING** (measured 2026-07-13: `virsh list --all`; `module.voffice1` is in - `opentofu/terraform.tfstate`) -- the D-114 containment VM is applied. -- **NOT DONE -- this is what remains of Stage 2:** MAAS-region ON `voffice1`; LXD ON `voffice1` - (5.21 LTS track); the LXD-KVM-host registration back into MAAS; and **every service machine** - (NetBox, Tailscale -- GitBucket dropped, D-116), which are now MAAS-COMPOSED LXD VMs, not standalone libvirt - domains. The NetBox/GitBucket that answer at `baldurkeep.com` are PRE-EXISTING services, NOT - Office1 machines -- do not mistake them for a met gate. Likewise, Tailscale reaching the - jumphost via a pre-existing subnet route is a DIFFERENT node and does not meet Stage 2's - Tailscale bullet. -- **The L3 nesting proof is the load-bearing unknown.** LXD *virtual machines* are qemu/KVM - guests, so composing the service machines inside `voffice1` needs nested KVM at L3 -- the same - depth the DC nodes will need. Office1 is deliberately the CHEAP place to prove it (no - OpenStack, no Ceph, small VMs). If an LXD VM boots inside `voffice1`, DC1 inherits the proof; - if it does not, D-114's model is abandoned early having cost one VM. -- NetBox's "authoritative and populated" bullet remains PARTIAL (mechanism done via DOCFIX-152; - literals still unassigned) -- unchanged by D-114. +Branch `dc-dc-stage1-phase0-first-exec` merges to `main` at **Stage 2 CLOSE** -- when every +gate bullet above is MET, not before. This is NEW policy: `main` has never taken a feature +merge (its only merge commits are inherited from the pre-rename `openstack-caracal-ipv4` +remote, D-110), so the entire VR1 buildout -- D-112 through D-118, the Office1 headend, the +NetBox tooling -- currently exists ONLY on the branch. Until the merge lands, `main` is not +tracking the mission and must not be treated as trunk-of-record for VR1. + +The merge is an operator-gated action, not a session-end habit. See the Stage 2 close-out +checklist below for what must be true first. + +--- + +**State (2026-07-14): SUBSTRATE + SITE BOTH DONE. Stage 2 is in CLOSE-OUT, not in build.** + +> The 2026-07-13 "NOT DONE" list that stood here is **SUPERSEDED and was already stale when +> written** -- MAAS-region, LXD, the LXD-KVM-host registration and both composed service +> machines all landed the same day. This doc's status table has now drifted three times; the +> correction below is measured, not assumed. + +- **DONE (substrate):** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is + BUILT and LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on + udp/67). Config is REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1` + EXISTS and is RUNNING** (`module.voffice1` is in `opentofu/terraform.tfstate`). +- **DONE (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5 + ON `voffice1`; that LXD **registered back into MAAS as an LXD VM host**; and both service + machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.201, NetBox 4.6.4) and + `office1-tailscale` (10.10.1.202, subnet router for `10.10.0.0/22`, route now ENABLED on the + self-hosted control server). GitBucket is out of scope (D-116), so Office1's composed machines + are exactly TWO. Codified in `scripts/site-headend-install.sh` (harness 18/18) -- reusable for + the DCs. +- **The L3 nesting proof is NO LONGER an unknown -- it is DEFINITIVE.** A guest booted, PXE'd and + COMMISSIONED three levels deep. DC1 inherits the proof; D-114's model stands. +- **NetBox is populated in the SANDBOX and proven faithful** (D-115 office carve + the six-plane + roles + both DCs, 90 -> 134 prefixes, field-level fidelity check clean). The apex literals that + blocked it are now assigned: **D-117** (site naming) and **D-118** (`ORG_ULA_48 = + fd50:840e:74e2::/48`). The gate bullet says **authoritative**, and the authoritative NetBox is + `netbox.baldurkeep.com` -- so this bullet is NOT met until the validated carve is fed UPSTREAM. + That write is operator-gated; the tools refuse a non-sandbox target without + `--yes-write-upstream`. + +**Stage 2 close-out checklist (what must be true before the merge):** + +| # | Item | State | +|---|---|---| +| C1 | Office1 LAN IPv6 deployed (`2602:f3e2:f01:100::1/64` + RA on the edge LAN, via the D-113(a2) REST API -- NEVER a config.xml push). WAN v6 leg DEFERRED: v6 does not egress the lab. | **OPEN** (pinned, D-115 q3) | +| C2 | The validated carve fed UPSTREAM to `netbox.baldurkeep.com` -- the gate says *authoritative*, and the sandbox is not it. Operator-gated write. | **OPEN** | +| C3 | The `$DC` shell-selector collision resolved -- `lib-net.sh`/`lib-hosts.sh` `dc0` means **VR0's** DC0 while the D-117 apex `dc0` means **VR1's**. D-117 Option B is only PARTIALLY executed. Blocks Stage 3 (see gap #19). | **OPEN** | +| C4 | The NetBox sandbox loop documented here and in `docs/dc-dc-netbox-buildout-scope.md`. | **OPEN** | +| C5 | This doc's Stage 2 row reconciled against reality (it has drifted three times). | **DONE 2026-07-14** | --- @@ -705,6 +735,39 @@ has no real machines to test against yet -- the runbook states the principle and points at MAAS's own docs rather than guessing flags. +19. **NEW 2026-07-14 -- THE `dc1`/`dc0` NAME NOW MEANS TWO DIFFERENT DATACENTERS. + This BLOCKS Stage 3 and is the same off-by-one D-117 just fixed, one layer down.** + + D-117 (ADOPTED, Option B) moved the repo onto the NetBox apex's 0-indexed + convention: VR1's **first** DC is `vr1-dc0`, its **second** is `vr1-dc1`. + That rename was executed on the IMPORT PATH ONLY (`netbox/dc-dc-prefixes- + import.py` + the scope doc). It was **NOT** executed on the two other + surfaces that speak `$DC`, and they now disagree with the apex: + + | Surface | says `dc1` means | says `dc0` means | + |---|---|---| + | NetBox apex (authority, D-117) | VR1's **SECOND** DC | VR1's **FIRST** DC | + | `opentofu/main.tf` (`dc1_planes`, `dc2_storage`, `mesh_dc1_dc2`) | VR1's **FIRST** DC | -- (no `dc0`) | + | `scripts/lib-net.sh` / `lib-hosts.sh` (DOCFIX-151 selectors) | VR1's first DC | **VR0's** DC0 -- a different REGION | + + So "the next DC we deploy" is, simultaneously: apex `vr1-dc0`, OpenTofu + `dc1_*`, and a `lib-net.sh` `$DC` value that ALSO names VR0's live testcloud. + Three names, one machine, and one of them collides with a **running cloud**. + + **Why a mechanical rename is a trap (D-117 TRAP 1/TRAP 2, now live here):** + `sed s/dc1/dc0/` on `opentofu/main.tf` would hand VR1's first DC the string + `dc0`, which `lib-net.sh` already resolves to **VR0's deployed literals**. The + fix is a SEMANTIC remap through placeholders, and the shell selectors must be + region-qualified (`vr0-dc0` / `vr1-dc0` / `vr1-dc1`), not merely renumbered. + + **Consequence if ignored:** `module "dc1_nodes"` gets authored against the + wrong site's prefixes, and the first VR1 DC deploys with the second DC's + addressing -- discovered, at best, at `juju deploy`. + + **Status: OPEN. It is Stage 2 close-out item C3 and the FIRST step of any + Stage 3 OpenTofu work.** Nothing in `opentofu/` should gain a new DC-keyed + module until it is closed. + --- ## Where things stand today (2026-07-09) -- update this as stages close diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 7eaa38f..b84b35d 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -87,10 +87,14 @@ any "how do I reach X" question. **GitBucket: OUT OF SCOPE (D-116, operator ruling).** No Office1-local GitBucket is built; `git.baldurkeep.com` stays the git service of record. Office1's composed machines are exactly TWO. - **REMAINING on Office1:** (a) OPERATOR: enable the `10.10.0.0/22` route on the Tailscale control - server (`headscale routes enable`) -- it is advertised but not yet enabled, so the subnet is not - reachable from the tailnet yet; (b) import the D-115 carve into the Office1 NetBox; (c) the - pinned Office1 LAN IPv6 (item 1a). + **REMAINING on Office1 (reconciled 2026-07-14):** (a) route enable -- **DONE**, operator enabled + `10.10.0.0/22` on the control server; the edge GUI answers at 10.10.0.1 from the workstation with + no tunnel; (b) import the D-115 carve into the Office1 NetBox -- **DONE** (sandbox seeded from the + upstream draft and PROVEN faithful field-by-field; carve + six-plane roles + BOTH DCs imported, + 90 -> 134 prefixes); (c) the pinned Office1 LAN IPv6 (item 1a) -- **STILL OPEN**, and it is now + the last build item on Office1. + **STAGE 2 IS IN CLOSE-OUT, NOT BUILD. It is also the BRANCH MERGE POINT** (see PINNED). The + close-out checklist C1..C5 is in `docs/dc-dc-deployment-workflow.md`; do not re-derive it here. 1a. **PINNED -- NEXT AFTER NETBOX: deploy Office1's LAN IPv6** (operator ruling 2026-07-13, D-115 question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). Add `2602:f3e2:f01:100::1/64` to the OPNsense edge LAN + RA, and let `voffice1` take its address. @@ -115,13 +119,22 @@ only DC1 *deployment* is gated). Unconditional `cpu = { mode = "host-passthrough" }` -- DC nodes run `nova-compute` and MUST run KVM guests. Without it they would have come up looking healthy and been unable to launch a single instance. -2. **NetBox sandbox loop is undocumented.** Operator-clarified architecture: draft-source - `netbox.baldurkeep.com` (READ ONLY) -> stand up a NEW NetBox on the Office1 VM (sandbox) -> - import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK to - the real NetBox. The sim NEVER `--commit`s upstream. `docs/dc-dc-netbox-buildout-scope.md` - and the Stage 2 runbook still need a note capturing this loop. PENDING. -3. **`pynetbox` / `python3-venv` are not installed** on the vcloud host -- needed for the NetBox - import against the Office1 sandbox; needs operator sudo. +2. **NetBox sandbox loop -- BUILT 2026-07-13/14, still UNDOCUMENTED in the runbook (close-out C4).** + Architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> a NEW NetBox on the Office1 VM + (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements + BACK. The sim NEVER `--commit`s upstream. **The tooling now exists and is proven:** + `netbox/prod-draft-dump.py` (read-only, no write path at all), `netbox/sandbox-seed.py` + (dry-by-default, structurally REFUSES an upstream URL), `netbox/sandbox-fidelity-check.py` + (field-by-field -- counts alone HID a bug where 17 of 90 prefixes silently lost their region + scope). **STILL PENDING:** capture the loop in `docs/dc-dc-netbox-buildout-scope.md` + the Stage 2 + runbook, and **feed the validated carve UPSTREAM** (close-out C2 -- the Stage 2 gate says + *authoritative*, and the sandbox is not it). The upstream write is operator-gated. +3. **`pynetbox` -- CLOSED 2026-07-13.** Installed (7.0.0) on `office1-netbox`, which is where the + imports actually run. The stdlib tools (`prod-draft-dump`/`sandbox-seed`/`sandbox-fidelity-check`) + deliberately need no dependency at all. + **TRAP, recorded in `platform-traps.md`:** the upstream NetBox sits behind a **User-Agent-filtering + WAF**. `Python-urllib/3.12` gets a **403 that looks exactly like an auth failure**; the same token + works from curl. This affects pynetbox too. 4. **The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13.** `scripts/prereqs/install-apparmor-libvirt.sh` (+ `install-all` / `check-prereqs` wiring; harness 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants `/** rwk,` in @@ -130,13 +143,24 @@ Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce libvirtd unless it actually adds the rule). Pool parent defaults to `/var/lib/libvirt/vr1`, override with `VR1_POOL_PARENT`. `docs/changelog-20260713-apparmor-libvirt-prereq.md`. -5. **DC2 planes are DEFERRED (operator ruling, Option B):** do not invent CIDRs -- wait for NetBox - to assign the D-101 supernet/ULA/GUA. DC2 mesh legs and `dc2_storage` are already wired. - Values pending final import: ORG_ULA_48 `fd50:840e:74e2::/48`, GUA dc1 `f02::/48` / dc2 `f03::/48`, - DC2_V4_SUPERNET `10.13.0.0/19`. -6. **G5 NetBox site rename** (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) needs its own D-number. - Also open in NetBox: G2 (aggregates empty -- top blocks are container prefixes), G3 (ULA /48 - not assigned). +5. **The DEFERRED literals are now ALL ASSIGNED -- item CLOSED 2026-07-14.** The operator's Option-B + ruling ("do not invent CIDRs, wait for NetBox") held, and NetBox has now assigned them: + **ORG_ULA_48 = `fd50:840e:74e2::/48`** (**D-118 ADOPTED** -- RFC 4193 valid, and verified NOT to + collide with Tailscale's `fd7a:115c:a1e0::/48`, which matters because `office1-tailscale` runs + `--accept-routes`); GUA per D-117: first DC `2602:f3e2:f02::/48`, second `f03::/48`; the second + DC's v4 = six sequential `/22`s from `10.12.64.0/19` (D-115 moved it INSIDE the Cloud /16). + All are IMPORTED and fidelity-checked in the sandbox. + **CAUTION -- `DC2_V4_SUPERNET` IS A RETIRED NAME (D-117).** The importer now hard-rejects it and + rejects `--dc dc2`. Any doc still saying `10.13.0.0/19` is pre-D-115 and wrong. +6. **G5 NetBox site rename -- CLOSED as D-117 (ADOPTED, Option B).** The repo moved to the apex's + 0-indexed convention (first VR1 DC = `vr1-dc0`), rather than bending the apex to the repo. This + fixed a REAL off-by-one: `dc-dc-prefixes-import.py` bound `--dc dc1` to slug `vr1-dc1` while + treating `f02::/48` as dc1 -- but the apex binds `f02` to `VR1 DC0`. It would have written the + first DC's prefixes onto the second DC's site. The importer is now also **dry-by-default**. + **D-117 IS ONLY PARTIALLY EXECUTED** -- see gap #19 / close-out C3: `opentofu/main.tf` and + `lib-net.sh`/`lib-hosts.sh` still speak the OLD naming, and `lib-net.sh`'s `dc0` means **VR0's** + DC0. That collision BLOCKS Stage 3. + G2 (aggregates) and G3 (ULA /48) are both CLOSED by the same work. 7. **DOCFIX-185 follow-up (operator to rule):** a D-100/D-107 amendment note so the Stage-3 DC edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark fiber is East-West/replication ONLY, never an internet path). @@ -198,6 +222,15 @@ ## Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions) +- **BRANCH MERGE POINT = STAGE 2 CLOSE (operator ruling 2026-07-14).** Branch + `dc-dc-stage1-phase0-first-exec` merges to `main` when every Stage 2 gate bullet is MET -- + not at session end, not "when it feels done." The close-out checklist (C1..C5) lives in + `docs/dc-dc-deployment-workflow.md`'s Stage 2 section; that doc is authoritative for what + "closed" means. NEW policy: `main` has never taken a feature merge (its only merge commits + are inherited from the pre-rename `openstack-caracal-ipv4` remote, D-110), so ALL VR1 work + -- D-112..D-118, the Office1 headend, the NetBox tooling, 52 commits -- lives only on the + branch. `main`'s tip is two ad-hoc operator uploads; it is NOT trunk-of-record for VR1 yet. + The merge itself is a gated action: present it, do not run it unasked. - **D-071** ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning session. Backups now exist (a 902MB controller backup was taken; `juju create-backup` / `download-backup` DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong).