diff --git a/.claude/skills/openstack-cloud-ops/references/operating-discipline.md b/.claude/skills/openstack-cloud-ops/references/operating-discipline.md index f103762..5d3a534 100644 --- a/.claude/skills/openstack-cloud-ops/references/operating-discipline.md +++ b/.claude/skills/openstack-cloud-ops/references/operating-discipline.md @@ -144,3 +144,12 @@ resources: confirm which project/domain scope the shell holds (`openstack token issue`-level certainty) - `server create` and friends use ambient scope silently. Full triage method: references/troubleshooting.md. + +## Guards are load-bearing: reword the content, never weaken the check + +When a repo-lint guard (or any fail-closed check) goes RED on text you +believe is legitimate, the fix is to REWORD the text or add the guard's +own documented opt-out marker -- never to loosen the guard. Precedent: L3 +correctly flagged the config.xml tombstones; the tombstones were reworded +and the guard kept. (Rotated from the session ledger's standing-lessons +block, 2026-07-19.) diff --git a/docs/archive/session-ledger-rotated-20260719.md b/docs/archive/session-ledger-rotated-20260719.md new file mode 100644 index 0000000..4a43d4b --- /dev/null +++ b/docs/archive/session-ledger-rotated-20260719.md @@ -0,0 +1,1090 @@ +# Session-ledger rotated bodies (GA-R4 rule-6 one-time rotation, 2026-07-19) + +Rotated verbatim from docs/session-ledger.md during sweep Batch 4. These are +the pre-rotation body sections (Live state 2026-07-13; OPEN WORK VR1 + VR0; +NetBox write-path bugs; PINNED rulings; standing lessons + state facts -- +ROUTED to durable homes, see the Batch-4 changelog; project-completion; +NetBox import narrative). Status claims herein are HISTORY -- the status +authority is docs/CURRENT-STATE.md. Byte content unmodified below this line. + +--- + + +## Live state (measured 2026-07-13; nothing is mid-flight) + +- **VR1 / Office1 edge `office1-opnsense` is UP and healthy.** LAN 10.10.0.1/24, WAN 172.30.1.2 + (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial + console + getty alive, and `kea-dhcp4` serving DHCP on udp/67 (subnet 10.10.0.0/24, pool + .100-.199). **DHCP is API-MANAGED (D-113(a2)).** +- **OpenTofu repo/state back IN SYNC -- D-119 rename APPLIED (2026-07-14, operator-gated).** + `tofu apply`: **11 added, 0 changed, 11 destroyed**; `tofu plan` now reports "No changes". Verified + on the host: six `vr1-dc0-*` networks, three `mesh-vr1-*` links, `vr1-dc0-pool`/`vr1-dc1-pool`; ZERO + old `dc1-*`/`dc2-*`/`mesh-dc*` names remain. Pre-apply LIVE re-check confirmed all 11 objects empty + (0 leases, 0 volumes, no domain attached). Office1 (`office1-local`/`office1-wan`/`wan`/edge/ + `voffice1`) was NOT in the plan and is untouched -- edge + voffice1 still reachable post-apply. + tofu v1.12.3; tree validates (10/10 modules); lock committed. +- **VR0 / DC0 testcloud:** fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except + item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own + cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts). +- **STAGE 2 CLOSED AND MERGED (2026-07-15).** All close-out gates C1..C5 met; branch + `dc-dc-stage1-phase0-first-exec` merged to `main` via **PR #1 (`e468d2e`, merge commit)** -- the + FIRST VR1 feature merge. `main` is now trunk-of-record for VR1 through Stage 2; the merged branch + was RETIRED (local + remote). **NEXT: Stage 3** (DC0/DC1 substrate, `dc-dc-phase2`->`phase3`) on + branch **`dc-dc-stage3-phase2-dc-substrate`** (off post-merge `main`); read + `docs/dc-dc-deployment-workflow.md`'s tooling-gap register FIRST. Now standing: **branch-per-stage + + merge-at-close** discipline (workflow doc Cross-cutting) -- merging the stage branch is part of every + stage's definition-of-done. In place for Stage 3: region-qualified `$DC` selectors (D-119) + the + creds-audit mint-time discipline (SEC-009 / `scripts/creds-audit.sh`). +- (historical) Branch `dc-dc-stage1-phase0-first-exec` was pushed and level with origin (2026-07-14); + VR1 Stage 1 COMPLETE; Stage 2 was in CLOSE-OUT -- now superseded by the merge above. +- **SESSION CONSOLIDATION (2026-07-14).** Work had fragmented across FOUR concurrent Claude sessions + on this one clone -- including two long-lived `--permission-mode auto` background agents (idle 14-15h + but still holding auto-approve rights against the live cloud), and a second interactive session that + received the SAME "resume last night's work" prompt six minutes before this one and began bootstrapping + in parallel. All three were terminated and the supervising daemon stopped (it had respawned the agents + once). **Standing lesson: the daemon ADOPTS AND RESPAWNS background workers -- killing the child is not + enough; stop the daemon (`claude daemon stop --any`).** The working tree was clean and 3-ahead + throughout; nothing was lost or clobbered. ONE session is now the rule. +- **Last night's three commits (D-113 amendment + the v6 script) were made by the auto-permission + background agent and left UNPUSHED and UNDOCUMENTED.** Both closed 2026-07-14: pushed, and + `docs/changelog-20260714-opnsense-set-interface-v6.md` written. + +--- + +## OPEN WORK -- VR1 DC-DC (the current mission) + +**D-120 ADOPTED + EXECUTED (2026-07-15) -- static-services addressing for the compose /24s.** +Office1's two services were re-IP'd OUT of the MAAS node band into the `.2-.49` static band: +`office1-netbox .201 -> 10.10.1.10`, `office1-tailscale .202 -> 10.10.1.11`. On the wire, no wipe +(NetBox HTTP 200 at .10; tailscale still advertises `10.10.0.0/22`). Band layout ratified: `.2-.49` +static / `.100-.200` dynamic / `.201+` nodes, for every MAAS compose /24; DCs inherit it from day +one. Method finding: MAAS won't edit a Deployed machine's interface (lift = redeploy = WIPE), so the +address is set on the guest (netplan + curtin, over `lxc exec`) and MAAS-model drift is accepted +(reconciles at next teardown/redeploy). Sweep + gauntlet done; `runbooks/dc-dc-office1-service-reip.md` +now records the measured method. **D-120 Step 6 DONE 2026-07-15** -- the /24 child ranges + the two +service IPs are registered in `office1-netbox` (the VR1 IPAM apex) via `netbox/d120-compose-bands.py +--commit` and VERIFIED by the (now-extended) fidelity check -> exit 0. The `ip-ranges`/`ip-addresses` +tooling gap is CLOSED (dumper + checker now cover both classes). NO apex write -- the upstream +write-back stays DEFERRED to end-of-deployment. This closed the last piece of close-out C2. See +`docs/changelog-20260715-d120-load-and-iprange-tooling.md`. + + +**D-126 RULED + tooling delivered (2026-07-16) -- durable rootless vcloud->service-VM access.** +Operator ruled Option A (SSH local-forward via `systemd --user`, "continue [the] ssh proxyjump [we +already use]") over persisting the L3 route (root + races libvirt-managed virbr2). Reachability +RE-established this session by the documented ad-hoc route (operator ran `sudo ip route replace +10.10.1.0/24 via 10.10.0.20 dev virbr2` in a real terminal; verified HTTP 302 at 10.10.1.10:8000). +Delivered `scripts/site-forward.sh` + `tests/site-forward/run-tests.sh` (38/38), D-126 record, +as-built access-table update. **SHELL layer added same day** (operator: "you're going to need shell in +Office and the DCs") -- `scripts/site-ssh-config.sh` + `tests/site-ssh-config/run-tests.sh` (30/30) +generate a `~/.ssh/config.d/vr1-sites` ProxyJump alias set (`ssh office1-netbox` etc.), the FOUNDATION +the forward rides; DC aliases deferred until measured post-apply. **SHELL LAYER EXERCISED 2026-07-16**: +operator installed with `~/vr1-office1-creds/office1_svc_ed25519` (one-key closed env, per +`creds-manifests/vr1-office1.manifest`); agent proved `ssh voffice1`/`office1-netbox`/`office1-tailscale`/ +`office1-opnsense` all return live from this session -- so the agent now HAS durable shell into Office1 +(load-bearing assumption confirmed: key read from config `IdentityFile`, no creds path in command, passes +the guard/classifier). The systemd FORWARD layer stays reviewed-but-unexercised (not installed). +**NetBox apex recovery + edge-WAN import (2026-07-17).** Found the IPAM apex DOWN (netbox-docker cleanly +exited ~3h prior, no auto-restart). Brought it up (`docker compose up -d`, HTTP 302) + added +`restart: unless-stopped` to all 5 services (survives the bookend reboot; override backed up on host). +Both blocks COMMITTED (gated, dry-run-verified; token server-side, sandbox-only). Block A: edge /24s +172.30.2.0/24 (vr1-dc0) + 172.30.3.0/24 (vr1-dc1). Block B: operator chose "write the seeder" -> NEW +`netbox/d124-transit-seed.py` + `tests/d124-transit-seed/` (21/21, incl. a cross-file consistency check vs +dc-rack-mgmt) seeds role `transit` + 172.31.0.0/24 unscoped container; then `dc-rack-mgmt-import` placed +172.31.0.0/30 (transit, vr1-dc0) + 10.12.8.2/22 (rack, dns=vvr1-dc0) -- matches the D-124 tfvars. The full +D-124/edge-WAN IPAM is now in the apex. See +`docs/changelog-20260717-netbox-apex-recovery-edge-wan-import.md`. + +**GUEST UPDATES + D-127 IaC AUTOSTART done (2026-07-17).** Office1 guests patched via agent sudo INSIDE +them: office1-netbox apt upgrade (no reboot, netbox healthy); office1-tailscale full-upgrade -> kernel +6.8.0-136 + reboot (back, re-advertising 10.10.0.0/22); voffice1 0-upgradable. LXD boot.autostart=true set +on both. **D-127 IaC layer DONE** (operator: bake the matrix so DC0/DC1 not missed): `autostart` var added +to cloudinit-vm/opnsense-edge/node-vm, set per matrix (voffice1/office1-opnsense/DC-edge=true; +vvr1-dc0/nodes=FALSE); opentofu T11-T15 pin it; HARD GATE verified (voffice1/office1-opnsense = autostart +in-place, NOT replacement). Outer plan now 5 add / 2 in-place / 6 destroy (collapses to 0-change once the +bookend sets those two autostart live pre-apply). Standup runbook records the LXD boot.autostart step. See +`docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md`. STILL bookend: vcloud + opnsense patch/reboot. + +**vcloud OS-UPDATE + REBOOT window (2026-07-16) -- IN FLIGHT; D-127 autostart policy recorded.** +Operator patching the vcloud host during the open restart window (only Office1 VMs live, before the DC0 +build). D-127 RULED: voffice1 autostarts; office1-opnsense autostarts AFTER one verified post-fix cold boot +(DOCFIX-188 mem-unit fix); vvr1-dc0/dc1 containment stay MANUAL. **A host reboot ENDS the running agent +session** -- so this is recorded before it. **POST-REBOOT TODO (next session):** (1) `virsh start voffice1` ++ `virsh start office1-opnsense`, WATCH the edge cold-boot clean (live-boot verification); (2) verify `ssh +voffice1` / `ssh office1-netbox` answer (D-126 ProxyJump; the direct L3 route to 10.10.1.10 is already gone +-- use SSH); (3) THEN `virsh autostart voffice1` + `virsh autostart office1-opnsense` (operator sequencing: +autostart only AFTER clean-startup confirmed); (4) optionally set `boot.autostart` on the LXD service VMs +inside voffice1. Then resume DC0 readiness (`docs/dc0-deploy-readiness.md`). + +**UPDATE 2026-07-17 (session-scoped health sweep + edge cold boot) -- the "bookend" is PARTIALLY done, +NOT the way the TODO above assumed.** Measured this session: **the vcloud HOST never rebooted** (uptime +8d 18h, boot 2026-07-08) -- so the host-reboot that "ENDS the running agent session" has not happened. +What DID reboot was the guests: **voffice1** (07-16 20:56, kernel 6.8.0-136) and **office1-tailscale** +(07-17 07:25, kernel 6.8.0-136), both healthy. The **office1-opnsense edge was cold-booted THIS session** +(graceful ACPI `virsh shutdown` -> confirmed `shut off` -> `virsh start`) and VERIFIED clean: up-from-0, +physmem 2.0 GiB (DOCFIX-188 fix live, not 2 MiB), pf Enabled/80 rules, `kea-dhcp4` up, both NICs up, LAN +gw + egress 0% loss, netbox 302 + tailscale reachable behind it. That satisfies the D-127 verified-cold-boot +gate, so **`virsh autostart office1-opnsense` is now ENABLED**. **`voffice1` autostart ALSO ENABLED this +session** -- so BOTH foundational VMs now autostart per D-127 (POST-REBOOT TODO item 3 DONE; `vvr1-dc0/dc1` +stay MANUAL and do not exist yet). **LXD service-VM autostart VERIFIED (2026-07-17):** the two LXD VMs +inside voffice1 -- `office1-netbox` (10.10.1.10) + `office1-tailscale` (10.10.1.11) -- both already carry +`boot.autostart=true` (MEASURED via `sudo lxc config get`; confirms the D-127 guest-update note, no change +made). So the autostart chain is now proven END TO END: vcloud->`voffice1`/`office1-opnsense` via `virsh +autostart`; `voffice1`->netbox/tailscale via LXD `boot.autostart` -- a vcloud host reboot self-recovers the +whole Office1 stack with no manual `virsh start`/`lxc start`. **STILL PENDING:** (a) the vcloud HOST patch+reboot -- kernel **6.8.0-136 +is pending in apt but NOT installed** (running -134, newest installed -134, no reboot-required flag), so a +bare host reboot gains nothing: the real step is `sudo apt update && sudo apt full-upgrade` (lands -136 + ~9 +security pkgs) THEN reboot; (b) office1-opnsense itself is still on 14.3-RELEASE-p16 (guest OS not patched, +only cold-booted). + +**BOOKEND ITEM (a) CLOSED 2026-07-18 -- MEASURED, and the autostart chain PROVED ITSELF FOR REAL.** +The vcloud host was patched and rebooted between sessions: `uname -r` = **6.8.0-136-generic**, `apt list +--upgradable` EMPTY, no `/var/run/reboot-required`, and `last -x reboot` shows TWO boots on 2026-07-17 +(17:59 and 23:38; current uptime 5h39m from 23:38, ending the -134 run of 8d20h). Item (a) is DONE. Item +**(b) STAYS OPEN** -- a host reboot does not patch the edge guest; `office1-opnsense` is still on +14.3-RELEASE-p16. **The D-127 autostart chain is now PROVEN BY A REAL HOST REBOOT, not merely configured:** +after an unattended boot, `virsh list --all` shows `voffice1` + `office1-opnsense` both running with +autostart, and `ssh voffice1` / `office1-netbox` / `office1-tailscale` all answer live (all up ~5h38m, +matching the host boot). `site-baseleg.sh check office1` = **ok leg + ok route** (the D-126 oneshot re-added +the isolated-net leg through the reboot). NetBox apex HTTP **302** at 10.10.1.10:8000; edge LAN gw +10.10.0.1 ping **0% loss**. Zero manual `virsh start` / `lxc start` was needed. Office1 is fully self-recovered. + +**STAGE-3 WORKFLOW-DOC BLOCKER IS STALE -- reconciled 2026-07-18 (the ledger was right, the doc is wrong).** +`docs/dc-dc-deployment-workflow.md`'s Stage 3 section still reads "**Blocking to EXECUTE:** the HELD NetBox +addressing -- D-124's rack transit/30 + rack metal-admin IP". MEASURED against the live apex +(read-only API GET, token sourced in-shell and never printed): all three objects are PRESENT -- +`172.31.0.0/30` (role Transit, scope **VR1 DC0**), the `172.31.0.0/24` unscoped Transit container, and +`10.12.8.2/22`. This matches the 2026-07-17 apex-recovery entry above. The NetBox addressing is NOT a +blocker; the workflow doc's Stage-3 "Blocking to EXECUTE" line needs a DOCFIX (next-free 196). + +**!! DC0 OUTER APPLY IS BLOCKED ON NEW DRIFT -- do NOT apply. See +`docs/finding-20260718-voffice1-cloudinit-seed-replace.md`.** Post-reboot re-verification of the outer +root (read-only plan, **OpenTofu v1.12.4**) returns **7 add / 2 change / 7 destroy**, NOT the committed +**5 add / 2 in-place / 6 destroy** in `docs/dc0-deploy-readiness.md`. The entire delta is `voffice1`, the +live headend -- NOT DC0 scope: the libvirt provider staged its cloudinit ISO under +`/tmp/terraform-provider-libvirt-cloudinit/`, **the reboot cleared `/tmp`**, so `create.content.url` reads +changed and **forces replacement of `voffice1`'s seed volume -- which is ATTACHED to the RUNNING domain as +`sda`** (`virsh domblklist voffice1`; the real ISO is intact at +`/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso`). Replacing it risks bouncing the Office1 headend = +NetBox apex + tailscale + the D-128 Plane-2 execution origin + the D-126 ProxyJump path. This is provider- +state drift, not a config change (the volume-replace CONSEQUENCE on a running domain is **UNVERIFIED** -- +do not assert it bounces the guest; trap 1e is about domains, not volumes). +**THE DOMINANT RISK IS THE 2 "in-place" CHANGES, NOT THE VOLUME.** They did NOT collapse to 0 as the D-127 +changelog predicted -- both `voffice1` AND `office1-opnsense` plan as `+ autostart = true`, because `virsh +autostart` was set out-of-band and the provider does not refresh it into state. **The bounce is CONFIRMED +FROM PRIMARY SOURCE** (v0.9.8 `internal/provider/domain_resource.go`, fetched + read directly): `Update`'s +only no-bounce path (L1090) requires `domainConfigUnchanged && runningChanged && autostartUnchanged`, and an +autostart-only diff has `runningChanged`=FALSE, so it falls through to **`stopDomainIfRunning`** -> +undefine -> DefineXML -> SetAutostart. ALSO CONFIRMED (L938): Read only reads `autostart` back +`if isImport || state.Autostart` is non-null -- ours is NULL, so **`-refresh-only` can never populate it** +(matches the empirical `grep -c autostart` = 0 on both domains). NOTE: an earlier draft cited +`platform-traps.md` 1e as the warrant here; 1e was measured on a *memory* change and does NOT transfer to +an autostart-only diff -- the conclusion held but the reasoning was wrong, now replaced by the source above. +1e's operational rule still stands independently: `0 to change` on the domains is the ONLY assurance a guest +will not bounce. So the unscoped apply +would take Office1's headend AND edge down -- to write a value that is ALREADY LIVE on both. An earlier +draft of this finding called these changes "harmless"; that was WRONG and is corrected here and in the +finding doc. Pinned provider = dmacvicar/libvirt **0.9.8**. +Options (a)-(d) are in the finding. **DC0 IS NOT BLOCKED on the unresolved provider question:** the intended +DC0 work (create `vvr1-dc0` + uplink, destroy 6 empty planes) touches NO running guest, so **(a) `-target` +the DC0 addresses** sidesteps both risks and is recommended for the immediate apply, with **(b) reconcile +state first** (`-refresh-only` / re-stage) as the deliberate follow-up to get back to `0 to change`. +**(d) standing hardening:** keep the provider's cloudinit staging OFF `/tmp` so a reboot can never again +force-replace a live VM's seed. **BLAST RADIUS CORRECTED (MEASURED 2026-07-18):** an earlier draft of this +entry claimed this hits "all 9 DC0 node VMs and every DC containment VM" -- **WRONG, retracted.** +`libvirt_cloudinit_disk` appears in exactly ONE module (`modules/cloudinit-vm`), instantiated exactly TWICE +(`main.tf:181` voffice1, `main.tf:361` vvr1_dc0). The 9 nodes use `modules/node-vm` = blank-disk PXE, NO +cloud-init; the edges use REST config (D-113(a2)). Real exposure = **voffice1 + vvr1-dc0 (+vvr1-dc1) = 3**. +Also: on a CLEAN deploy the disk+volume are created in the same apply so there is NO diff -- this only +surfaces on a plan taken after a host reboot, so it does not break a post-teardown redeploy, only ops. +**Mechanism correction:** there is **NO provider knob** for this (VERIFIED via `tofu providers schema +-json` -- provider block exposes `uri` ONLY; `libvirt_cloudinit_disk` has no staging attr); the lever is +almost certainly **`$TMPDIR`** for the tofu process (Go `os.TempDir()`), UNVERIFIED by observation -- +confirm before writing it up. Warrants **D-130** (next-free confirmed 2026-07-18). +**Before ANY apply:** re-run the readiness doc's C1 pre-apply checks THIS session, incl. re-confirming the +6 `vr1-dc0` planes are still empty (0 leases / 0 attached) -- that evidence is from a prior session, and +this finding is itself proof that committed pre-apply evidence goes stale across a reboot. + +**Version drift to log:** OpenTofu on the jumphost is now **v1.12.4**; `README.md:41` and the readiness +evidence say **v1.12.3**. Presumed to have arrived with the host patch (not verified). All committed +plan/validate evidence predates it. + +**SESSION CLOSE 2026-07-18 (bookend). Full record: `docs/finding-20260718-voffice1-cloudinit-seed-replace.md` (committed 43b29d7).** +ESTABLISHED: (1) An autostart-ONLY diff **bounces the guest** -- VERIFIED FROM PRIMARY SOURCE (libvirt +v0.9.8 `internal/provider/domain_resource.go` L1090/L1107): the sole no-bounce path needs +`domainConfigUnchanged && runningChanged && autostartUnchanged`, and runningChanged=FALSE here, so it falls +through to `stopDomainIfRunning` -> undefine -> redefine. (2) `-refresh-only` can NEVER fix it -- Read is +self-gated at L938 (`isImport || state.Autostart` non-null); ours is NULL. (3) The seed force-replace is +STRUCTURAL: the reboot cleared `/tmp`, where the provider stages cloudinit ISOs, so the url changes every +plan. (4) Blast radius CORRECTED to **3 VMs** (voffice1, vvr1-dc0, +vvr1-dc1) -- NOT the 9 nodes +(`node-vm` = blank-disk PXE, no cloud-init); an earlier claim of mine, retracted. +OPEN: **D-130 mechanism ruling** (`ignore_changes` vs `$TMPDIR` -- operator's call, unruled); **verification +7** (what 0.9.8 does replacing a volume attached to a RUNNING domain -- deliberately unasserted); +**verification 8** (that the chosen D-130 mechanism actually suppresses a force-replacement diff); +**doc-integrity list** (D-127 changelog's false "collapse to 0" + "no forces-replacement", readiness-doc +5/0/6-vs-5/2/6, DOCFIX-196 stale Stage-3 NetBox blocker, README tofu 1.12.3->1.12.4, phase-2 runbook). +Gate before any apply: plan must read EXACTLY 5 add / 0 change / 6 destroy. Nothing applied this session. +**SUPERSEDED: this session is closed out by a grounding audit charter arriving via git -- read that first.** + +**DC0 DEPLOY READINESS -- PREP COMPLETE 2026-07-17 (deploy-ready). See `docs/dc0-deploy-readiness.md`.** +Final gate re-checked clean (validate, tfvars, creds-audit vr1-dc0 CLEAN, gauntlet 65, lint 0-fail). +Remaining is NOT prep: the bookend restart (vcloud + voffice1 + opnsense patch/reboot together -- voffice1 +is load-bearing like vcloud) + fresh-session (re-verify plan post-reboot + logged shell + gated outer +apply). Decoupled hygiene (NetBox import; Stage-3 record sweep) is OPTIONAL, does not gate the deploy. +Stage-3 Phase-2 DC0 is EXECUTABLE (D-121..125 ADOPTED, R-1/R-2/R-3 ruled, tfvars in, opentofu-validate +clean). Outer-root `tofu plan` RUN + VERIFIED SAFE: 5 add (`vvr1-dc0` 416GiB/108vcpu + D-125 uplink) / +0 change / 6 destroy (the six EMPTY vr1-dc0 planes relocating to the inner root -- verified 0 leases/0 +attached). RAM gate: 980 GiB avail >= 416. **Nothing applied.** Next = the gated outer `tofu apply`. +Pre-apply per-env-key DECISION now RULED + IMPLEMENTED (operator 2026-07-16, option a): `vvr1-dc0` +gets its OWN key -- minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (creds-audit CLEAN), added +`vr1_dc0_ssh_pubkey_path` var, `vvr1-dc0` cloud-init reads it (voffice1 keeps office1's key), tfvar set; +re-plan UNCHANGED (5/0/6). netem RULED = placeholder for the rehearsal. Full remaining-prep checklist + +fresh-session bootstrap + gated A-E sequence in `docs/dc0-deploy-readiness.md` (the entry point). Inner +apply must use the dc0 private key (wire post-outer-apply). Gauntlet ALL GREEN (65). Nothing applied. + +Original build note: this session had NO ssh key to voffice1 +(none in `~/.ssh`, no agent; key is operator-held in `~/vr1-office1-creds/`, correctly un-enumerable), +so the operator supplies `--key` and runs `install` + first-connect + the one-time +`loginctl enable-linger`. Site table seeded office1-netbox only; DC rows DEFERRED (IPs not measured). +Advisor-conferred (untangle two problems): this does NOT gate `tofu plan` -- the tree has no netbox +provider/data source, so the four operator-ruled `vr1_dc0_rack_*` tfvars alone unblock plan; the +NetBox `--commit` stays a decoupled, operator-run, token-on-netbox-host step. See +`docs/changelog-20260716-d126-durable-forward.md`. + + +**STAGE-3 ADVERSARIAL REVIEW (2026-07-16; advisor + 4-charter debate + Layer-1 gate).** +Full pre-deployment adversarial review of the `a48a60f` batch (D-121..D-124 + substrate/rack/HA +overlay). READ-ONLY; nothing applied/committed/pushed. Deliverable: `docs/stage3-adversarial-review- +20260716.md` (present-only, NOT committed); immutable base patch `docs/stage3-review-base.patch`; +surface frozen at WIP commit `87a7a8a` (local-only). **VERDICT: NO-GO for `tofu apply`** on +hard-gate grounds (substrate design is sound). BLOCKING headline: **2 decisions rest on +inferred/agent-authored rulings** -- D-121 vault-`(v-a)` (no operator utterance; Status `:3264` +contradicts body `:3346`; ENCODED in `overlays/dc-ha-scaleup.yaml:67-80` reversing BUNDLEFIX-002) +AND D-123 Model A ("Do it all now" is not an A/B answer; runbook `:507` already marks D-123 +PROPOSED). Both demote to PROPOSED; region+rack/Option C/D-124 stand. Pre-apply gate list (all +operator-gated, for a post-acceptance sweep): (1) re-rule vault + Model A; (2) ship the SEC-010 +`ip_forward=0`+firewall ARTIFACT + a mechanical gate (currently a ledger promise); (3) assign the +rack transit/30 + rack IP in office1-netbox, feed the 4 tfvars (tofu plan BLOCKED without them), +wire voffice1's transit leg. Record amendments queued: D-122 MAAS-bullet + site-down wording (C1/C3), +D-107 "node artifacts" scoping + DR-claim honesty (C4/C5, permanent-at-Roosevelt), D-121 Option-C +disk record + commit `optc-calc.py` (UNCOMMITTED/absent -- "reproducible" claim is false), D-115 +status-line vs ledger-scan. CONTESTED (surviving dissent, recorded): SEC-011 downgraded to +OBSERVATION -- A2 showed pruning storage NICs INCREASES Roosevelt delta (baremetal trunks all VLANs) +and isolated-L2 planes have nil reachability; reconsider/amend the SEC-011 row. Gauntlet ALL GREEN +(62), repo-lint 0-fail, batch ASCII/LF-clean. See the findings doc for the full R3-F01..F13 register, +minority report, and L1..L5 leads. + +**OPERATOR RULINGS on the review (step-by-step, 2026-07-16; NOT yet swept into design-decisions.md +-- these drive the gated fix-sweep):** +- **[R-1] D-123 node containment = MODEL B (operator ruling 2026-07-16, explicit AskUserQuestion + "Proceed with B, accept all").** Nodes nest INSIDE `vvr1-dc0`; site-down = single + `virsh destroy vvr1-dc0` (restores the operator's original D-122 intent; the inferred Model A is + overturned). **Accepted cascade (operator-confirmed after being shown the consequences):** + (a) supersede ADOPTED **D-103** (OpenTofu-creates-nodes-at-vcloud + MAAS-discovers seam) and + **D-114** (containment scoped to headend + non-stack LXD machines only, NOT the OpenStack nodes); + (b) **D-124 rack sizing (4/8192/80) VOID** -- `vvr1-dc0` must contain one DC's full node fleet + (~360 GiB node RAM + overhead) and set `expose_nested_virt = true`; (c) **depth-4 nested virt + accepted** as a rehearsal cost (vcloud->vvr1-dc0->compute->nova guest = 3x nested KVM; VR0 proved + 2; Phase-5 workload-boot risk flagged + accepted). Substrate reshape: `node-vm` + `maas-vm-host` + + 6 planes + `vr1-dc0-wan` retarget from vcloud libvirt to `vvr1-dc0`'s inner libvirt. STAGED for + the sweep; nothing edited/applied yet. +- **[R-2] D-121 vault-HA backend = RAFT integrated storage (operator ruling 2026-07-16, explicit + AskUserQuestion "Raft integrated storage").** Overturns the agent-authored v-a (MySQL-backed). + Consequences: DROP the `vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster` storage + relation; vault uses its own integrated raft (per-unit persistent storage needed); keep the + `vault-hacluster` VIP. Still 3x manual unseal (SEC-003). **HARD FEASIBILITY GATE -- UNVERIFIED, + not assumed (cardinal-sin discipline):** must confirm the Charmed `vault` charm (1.8/stable) + exposes an integrated-raft storage backend before this ruling is actionable. If the charm does not + support raft, the ruling is moot and returns to the operator (v-a MySQL vs another path). STAGED; + nothing edited/applied. + - **[R-2 FEASIBILITY GATE FAILED -- verified 2026-07-16.]** `bundle.yaml:133-138` (D-068 / + BUNDLEFIX-010): integrated-**Raft storage is a 1.16-charm property**, and the 1.16 vault charm is + RULED INCOMPATIBLE with this reactive Caracal cloud (certs V0->V1, no upgrade path, **broken with + Charmed Ceph**). The deployable charm is **1.8/stable** = MySQL/etcd storage, **no Raft**. So the + operator's Raft ruling is **INFEASIBLE for VR1** without the ruled-out 1.16 charm. Raft is the + correct *Roosevelt/D-068* target (get vault off 1.8.8-EOL), NOT a VR1 choice. **R-2 RETURNED to + the operator:** realistic VR1 options = v-a (MySQL, rec) or v-b (etcd, deprecated). + - **[R-2 FINAL = v-a MySQL-backed (operator ruling 2026-07-16, "v-a: MySQL-backed").]** Vault stays + on the 3-unit mysql-innodb-cluster (via vault-mysql-router), scale to 3, re-add + `vault:ha <-> vault-hacluster`. **NO CODE DELTA** -- the committed `overlays/dc-ha-scaleup.yaml` + already encodes v-a; only the D-121 RECORD needs fixing (attribute to the operator's 2026-07-16 + ruling; reconcile the body's stale "Operator sub-ruling needed"). Charm-verify gate stands (confirm + vault 1.8/stable does HA leader-election on MySQL at deploy). Raft parked as the D-068/Roosevelt + target. 3x manual unseal (SEC-003) stands. +- **[R-3] Option C storage resilience = 4 STORAGE NODES/DC (operator ruling 2026-07-16, "Add a 4th + storage node/DC").** Node layout now **3 control + 2 compute + 4 storage = 9/DC** (18 total, was 16); + `ceph-osd` = 4 units/DC. Restores Ceph rebuild headroom (lose 1 OSD host -> 3 distinct replicas still + placeable -> self-heals). Consequences: (a) whole-host RAM ~838 GiB/1024 (81%, hand-estimate -- FITS + but tighter; the ABSENT `optc-calc.py` must be rebuilt + re-run for the 3+2+4 shape AND Model B + nesting overhead -- R3-F06 now doubly required); (b) under Model B, `vvr1-dc0` holds ~384 GiB node + RAM + overhead (~404 GiB VM); (c) Ceph disk re-checked for 4x500Gi/DC = PASS, 5.31 TiB margin; + (d) `main.tf local.vr1_dc0_nodes` adds `storage-04`, D-121 layout table + sizing updated. +- **[R-4] SEC-011 CLOSED (operator ruling 2026-07-16, "Keep uniform 6-NIC").** A2's Roosevelt-delta + refutation accepted: uniform 6-NIC matches baremetal VLAN-trunking (all planes trunked to every + node); isolated-L2 planes = nil reachability in the sim. **NO CODE CHANGE** (keep uniform + `local.vr1_dc0_node_nics`); amend the SEC-011 ledger row to CLOSED / accepted-non-issue. + +**SWEEP PROGRESS (2026-07-16):** +- **Phase A DONE** -- record amendments (`design-decisions.md` D-103/D-107/D-114/D-115/D-121/D-122/D-123/ + D-124 + `security-ledger.md` SEC-011 close). Decision record now internally coherent; D-115 + ledger-scan false-positive fixed. `docs/changelog-20260716-review-sweep-phaseAB.md`. +- **Phase B DONE** -- `scripts/dc-dc-whole-host-budget.py` + `tests/dc-dc-whole-host-budget/` (13/13) + replace the lost `optc-calc.py` (R3-F06). Re-validation: Model B 3+2+4 = 838 GiB/1024 FIT (186 + headroom). Gauntlet 63 ALL GREEN. +- **Phase C IN PROGRESS** -- Model B OpenTofu reshape. Design cross-checked by a 3-charter agent + workflow (compatibility/best-practice/structure): mechanism = **TWO-ROOT / TWO-STATE split** (forced + by the provider-cannot-depend-on-same-apply-resource invariant; -target + provider-with-unknown-values + are anti-patterns). OUTER root (main.tf, vcloud) = vvr1-dc0 resized + transit + office1; INNER root + (NEW `opentofu/vr1-dc0-substrate/`, own state) = 6 planes + wan + edge + inner pool + 9 nodes, modules + reused VERBATIM. BOOTSTRAP GATE between (extend site-headend-install.sh: nested libvirtd/pool/nested=1/ + base image). Verdict GO-WITH-MITIGATION on depth-4. **[R-5] inner transport = REMOTE qemu+ssh from + Office1 (operator ruling 2026-07-16)** -- tofu on Office1, dials vvr1-dc0 over the D-124 transit; both + states on Office1. **vvr1-dc0 sizing DERIVED (not invented) = 108 vCPU / 416 GiB / ~3000 GiB** + (104 node vCPU + 4 oh; 384 node GiB + 32 oh; 2850 inner pool + OS), whole-host 870/1024 (85%) FIT. + DONE so far: storage-04 (R-3), fallback re-cut R-3-compliant (114d392); **C1 OUTER root** (main.tf: + vvr1_dc0 resized via var.vvr1_dc0_* [108/425984/3221225472000, derived], expose_nested_virt=true, + single transit leg; removed planes/wan/edge/node blocks + the dangling moved block; tofu validate + Success); **C2 INNER root** `opentofu/vr1-dc0-substrate/` (versions/variables/main.tf: qemu+ssh + provider + inner pool + 6 planes + wan + edge + 9 node-vm, modules reused verbatim; init+validate + Success). Both roots validate; repo-lint 0-fail. Cross-check agent running on the two-root impl. + **C3 DONE + REVIEWED** -- `site-headend-install.sh` `--host-nodes` (rack-only) node-host mode: installs + qemu-kvm+libvirt, persists+verifies kvm nested=1, adds ssh user to libvirt group, inner pool dir + + AppArmor grant, verifies/instructs the OPNsense base (delegated to opnsense-prep-image.sh). Harness + 32->45/45. Review-after-step (per operator discipline): gauntlet ALL GREEN (63), repo-lint 0-fail, + ASCII/CR clean, grounding OK (implements the C3 bootstrap the inner-root header names), drift sweep + clean (record/docs/code all reference "node-host mode" consistently). `docs/changelog-20260716-review- + sweep-phaseC.md`. + **PHASE D + REMAINDER DONE + REVIEWED (2026-07-16):** + - **SEC-010 (Phase D)** -- artifact COMMITTED in the node-host bootstrap: a FORWARD-drop across the + transit leg (NOT a global ip_forward=0 -- the inner vr1-dc0-wan NAT forces ip_forward=1), written to + /etc/nftables-sec010.nft + boot-persistent sec010-fw.service; `--host-nodes --check` is the mechanical + pre-apply gate. Harness 45->48/48. SEC-010 ledger row reconciled to the code. Stays OPEN until + applied+verified on vvr1-dc0. + - **maas-vm-host** -- header updated for Model B (register vvr1-dc0's INNER/local virsh to the Office1 + region; no cross-fiber dial). Still DEFERRED (DOCFIX-179; not instantiated). + - **C5 runbook reorg** -- `dc-dc-phase2-tofu-dc-substrate.md` gains the "D-123 MODEL B RESHAPE" section: + outer apply -> bootstrap gate (site-headend --host-nodes) -> inner apply -> maas-register; Step 6's + "8 nodes" superseded by R-3's 9. + - **model-a-fallback-plan.md** synced (~416 GiB; inner-root + node-host mode added to the B->A delta + + revert procedure). + Review-after-each-step applied throughout (harness/validate/lint/ASCII/drift per step). All Phase C+D + items COMPLETE. NOTHING applied/committed since 114d392; Phase C/D edits present-only in the working + tree. REMAINING (operator-gated, deploy-time): the actual outer/bootstrap/inner APPLIES + rack-IP + assignment (Phase E, L1-07) + the depth-4 nested-virt L4 boot gate. + - **Model B cross-check (agent) = FAITHFUL, no BLOCKING**; fixes applied: 32-GiB-overhead RATIFIED in + the record (870/154, vvr1-dc0 416 GiB), false metal-admin-IP comment corrected, SEC-010 updated for the + single-leg-outer/bridges-6-inner-planes reality. **OBS-3: design gap RESOLVED by D-125; egress efficacy + is a deploy-time gate (see the C+D-commit entry below).** Transit IP is duplicated across roots + (`vr1_dc0_rack_transit_ip` outer / `vvr1_dc0_transit_ip` inner) -- keep equal in tfvars. + +- **Stage-3 sweep C+D COMMITTED as `5175365`** (operator "commit but don't push"; NOT pushed). The + adversarial re-run then confirmed **OBS-3 material** once the operator ruled **"DC0 and DC1 have their + own ISP connections"**, resolved by **D-125 (bridge-in, single-NAT)**: + - Model B nesting had severed per-DC ISP egress: `modules/site-wan` makes `vr1-dc0-wan` a libvirt NAT + that egresses via ITS HOST's default route; under Model B that host is `vvr1-dc0` (transit-only, + East-West, SEC-010 FORWARD-dropped) -> dead. Fix: vcloud ISP NAT `vr1-dc0-uplink` + an IP-less uplink + NIC + netplan bridge `br-vr1-dc0-wan` on `vvr1-dc0`; inner `vr1-dc0-wan` becomes a BRIDGE (new + `modules/wan-bridge`) so OPNsense WAN rides to the single vcloud NAT. **Authored present-only; both + roots validate; NOT closed -- the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in + equivalent of the L4 boot gate), fallback = double-NAT.** See D-125 + platform-traps 1g. + - **No new HELD gate (CORRECTED 2026-07-16):** the vcloud ISP /24 is the SAME `172.30.2.0/24` D-115 + already ruled for vr1-dc0 (bridge-in is single-NAT; IPAM identity unchanged from Model A), inlined as + a ruled literal in `module "vr1_dc0_uplink"` -- NOT a NetBox-assigned tfvar. OPNsense WAN stays `.2`; + no re-address. D-125 adds NO NetBox object. (An earlier draft's `vr1_dc0_uplink_cidr` HELD var + + re-address were removed -- see the correction commit below.) + - **br_netfilter constraint:** SEC-010 FORWARD-drop stays interface-scoped, NEVER globalized. `--check` + hardened: fail-open guard now covers the transit-iface existence AND the WAN bridge/uplink enslavement. + - Harness: `site-headend-install` 49->54; `wan-bridge` covered by opentofu-validate S3. Records: D-125, + SEC-010, model-a-fallback B->A delta (item 8 + revert), platform-traps 1g. + - **D-125 COMMITTED `d34ec94` + PUSHED** (operator "commit", then "push"). Branch + `dc-dc-stage3-phase2-dc-substrate` now carries `114d392` (A+B) + `5175365` (C+D) + `d34ec94` (D-125) + on origin. + +- **Doc drift-reconciliation batch (post-Model-B/D-125), all doc-only, COMMITTED + PUSHED:** + - `8496c2e` -- `dc-dc-deployment-workflow.md` Stage-3 section reconciled to D-121..125 + Model B + two-root (Build/Gate/Owns/Authoring-status/rulings-summary); gap-register D-113(a2) OPEN->DONE. + - `87f9d75` -- `opentofu/README.md`: authoritative STATUS-RECONCILED banner (two roots, added + `site-wan`/`wan-bridge`, instantiation reality) + neutralized the DELETED, dangerous OPNsense + config.xml/config-ISO path (D-112/D-113(a2)). + - `788911c` -- `runbooks/dc-dc-teardown-rollback.md`: Model B two-root teardown banner (site-down = + one `virsh destroy vvr1-dc0`; inner-then-outer order; maas-vm-host targets inner virsh). + - Repo-wide drift scan confirmed the deleted-file / old-node-count references elsewhere are all + immutable CHANGELOGS, decision-record supersession context, or deliberate tombstones + (`opentofu/templates/README.md`) -- NOT live drift. Deeper prose in the three reconciled docs + + the Stage-4/5 sections still lag Model B (flagged in-banner as historical), not yet rewritten. + +- **D-125 addressing CORRECTED (`e270325`) + tightened (`922dd29`)**, then **D-124 addressing PINNED** + (operator-ratified 2026-07-16): the vcloud ISP /24 is the ruled `172.30.2.0/24` (not a HELD tfvar); + the D-124 transit supernet is CARVED as **`172.31.0.0/24`** (dedicated `transit` role, scope vr1-dc0), + transit /30 `172.31.0.0/30`, rack IP `10.12.8.2`. `dc-rack-mgmt-import.py` `CONTAINER` pinned to + `172.31.0.0/24`; harness rebased, 96/96. **NetBox `--commit` NOT DONE -- office1-netbox (10.10.1.10) is + UNREACHABLE from the vcloud jumphost (measured) + the apex token is a secret on that host; the write is + an on-host operator step** (edge-WAN import, then rack-mgmt import, then fill the 4 `vr1_dc0_rack_*` + tfvars -> `tofu plan`). Exact commands: `docs/changelog-20260716-d124-addressing-pin.md`. NO apex mutation. +- **Phase D NOT STARTED** -- SEC-010 rack `ip_forward=0` + firewall artifact + mechanical pre-apply gate. +- **Phase E NOT STARTED** -- operator assigns rack transit/30 + rack IP in office1-netbox (apex token), + fill tfvars -> `tofu plan` renders. +- NOTHING committed/pushed; all sweep artifacts present-only in the working tree. +- **[R-1 fallback preserved]** Model A (the not-chosen D-123 option) captured as a revert artifact per + operator request: `docs/model-a-fallback-plan.md` (architecture + the B->A delta + revert + procedure) + git tag **`model-a-fallback`** on `87a7a8a` (local-only). Revert = `git checkout + model-a-fallback -- opentofu/`. Model A already validates (no re-engineering needed). + + +**PLANE-SEGREGATION REVIEW of the Stage-3 commit (2026-07-16; advisor + 2 read-only agents).** +Reviewed the committed networking/plane changes (`a48a60f`, NOT applied) for crosstalk / segregation +breaches. **VERDICT: no crosstalk, no active violation -- committed state is SAFE.** All six `vr1-dc0-*` +planes + all three `mesh-*` legs are isolated L2 (measured: `net-dumpxml` shows no forward/ip/nat, +isolation ENFORCED BY CONSTRUCTION in `dc-planes`/`mesh-link`); the edge is 2-NIC so it structurally +cannot reach the 5 non-LAN planes; `vr1-dc0-wan` NAT touches only the edge WAN. metal-admin DC-LOCAL +(D-052/D-100) is **preserved but not ENFORCED** -- it rests on unpinned host defaults. **Two pre-apply +hardening items logged as SEC-010 (rack `ip_forward=0` + transit-leg firewall -- the one real +plane-bridge) and SEC-011 (node role-scoped NIC sets -- least-connectivity).** Say plainly at handoff: +"committed state is safe; two items to close before `tofu apply`." Plus three notes: +- **FUNCTIONAL (fix before apply, not security):** the rack's committed route `to 10.10.0.0/22 via + ` points at a next-hop with **no VM attached** -- `voffice1` is single-homed + (`network_names = [office1-local]` only, main.tf:184), NOT on `mesh-vr1-dc0-office1`. As committed, + apply yields a rack routing to a nonexistent peer, and the Office1<->DC region path is not closed. + Either wire `voffice1`'s transit leg or gate the rack route until it exists. +- **SEGREGATION CHECK (first boot):** the rack NIC->plane mapping (`enp1s0`->metal-admin, + `enp2s0`->transit) is assumed PCI-order==name; if swapped, the metal-admin /22 lands on the + Office1-facing wire. Verify via MAC (`virsh domiflist` x `network_names` order) before trusting + addressing. This is a segregation check, not cosmetic. +- **DEFERRED (track):** the edge WAN<->provider-public firewall is post-boot REST (opnsense-api), + not in the commit -- "enforced by absence" covers plane REACHABILITY only, not full edge hardening. +When `voffice1`'s transit leg is later wired it becomes an office1-local<->transit bridge and inherits +the SAME `ip_forward=0` pin requirement (a richer surface than the rack: MAAS region + LXD host). + + +**STAGE 3 IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`) -- two rulings landed 2026-07-15.** +Walked the phase2 runbook and surfaced its blocking decisions; the operator ruled both: +- **D-121 (ADOPTED IN PART).** VR1 makes the decorative single-unit control plane REAL: 14 services + scale 1->3 (the 12 hacluster-backed APIs + ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays + 1 per D-108). This is D-009's "mechanical 1->3" pulled forward from Roosevelt -- but it is a PLACEMENT + rewrite (every app's `to:` must spread across 3 distinct hosts), not a num_units tweak. **Node layout + = Option C** (3 control + 2 compute + 3 storage per DC, 16 node VMs), ruled after a whole-host + validation PROVED it fits: measured budget 256 vCPU / 1024 GiB / 10 TiB, Option C allocates + 222 / 790 / 5.4Ti (RAM binding at 77%; model at `scratchpad/optc-calc.py`). Ruled sizing: control + 16/64GiB/150GiB, compute 12/48/100, storage 8/24/550; keep control <=64 GiB (96 GiB -> 96% RAM). + **Vault-HA sub-ruling RESOLVED = (v-a)** (2026-07-15): keep MySQL-backed (on the already-3-unit + mysql-innodb-cluster), scale to 3, re-add `vault:ha`. GATE: verify the `1.8/stable` vault charm does + HA on the MySQL backend with 3 units; if not, evaluate Raft (NOT etcd) via D-068. 3-unit vault + triples the unseal burden (intersects OPEN SEC-003). +- **D-122 (ADOPTED).** VR1 site shape: each site nested in its own containment VM (site-down/recovery + + site-to-site drills); dark fiber DC-DC-Office is East-West ONLY; each site has its own dedicated L3 + ISP uplink (build `vr1-dc0-wan` on the office1-wan pattern); DC edge follows the Office1 pattern + (2048MiB/2vCPU/nano, REST-API config, LAN faces the site network); each site its own MAAS controller. + Subsumes the DOCFIX-185 D-100/D-107 amendment note. +- **Fix-forward DONE (2026-07-15):** the phase2 runbook is reconciled to post-D-119 naming + (`vr1-dc0`/`vr1-dc1`, bare `dcN` rejected) and its obsolete Steps 4-5 rewritten to the D-113(a2) + REST-API bootstrap (the module already dropped `config_iso_path`); the D-121/D-122 rulings folded in + so its Ruling-1/Ruling-2 STOP-gates clear. Honest achievable first-run scope stays **Steps 1-8** -- + Step 9 (MAAS registration) gates on the per-DC MAAS "rack controller", now RESOLVED IN PRINCIPLE by + **D-123** (PROPOSED, recommend Model A): it is just a `vvr1-dc0` site headend VM (the proven Office1 + pattern -- `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which already names + `vvr1-dc0` as a target). Once Model A is ruled it is prepare-able (opentofu block + Step-9 wiring), + which effectively closes the Stage-4 gap for Step 9. D-123 also reconciles the D-122 "one virsh + destroy = site-down" wording (literal for Office1; for a DC, site-down = destroy the `vr1-dc0-*` + domain group). See `docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md`. +- **EDGE NIC MODEL RULED (D-122 refinement, 2026-07-15): 2-NIC, baremetal-matched.** NIC count mirrors + the Roosevelt hardware -- **nodes = 6 NICs** (one per plane, measured on VR0 openstack0-3), **edge = 2** + (WAN + LAN). The six planes are **routed by the fabric** (OVN/OpenStack on the deliberately isolated-L2 + `dc-planes` segments), so the edge is NOT an inter-plane router -- only the external boundary. No + `opnsense-edge` module extension needed. Edge **WAN = `vr1-dc0-wan`** (`172.30.2.0/24`), edge + **LAN = provider-public** (`10.12.4.0/22`; edge is the external gateway per D-100). This SUPERSEDES the + earlier "build an office1-network-analog site LAN" flag -- no new site-internal segment is needed. +- **STAGE-3 PREP BATCH DONE 2026-07-15 (authoring + offline validation; NOTHING applied).** See + `docs/changelog-20260715-stage3-prereqs-prep.md`. Gauntlet ALL GREEN (61), `tofu validate` Success, + repo-lint 0-fail. + - **OpenTofu:** new `modules/site-wan` (NAT /24 uplink; schema confirmed via `tofu providers schema`, + validated) + `main.tf` Stage-3 section wired (NOT APPLIED): `vr1_dc0_wan` (172.30.2.0/24), the 2-NIC + edge (LAN=provider-public, WAN=uplink), 8 Option C node VMs via `for_each` (metal-admin PXE-first), + netem HELD (Step 11; mesh bridge = virbr7 measured), stale `vr1_dc1_planes` comment fixed. **HELD + pending D-123:** the `vvr1-dc0` headend + `maas-vm-host` (Step 9) blocks. + - **NetBox:** `netbox/dc-edge-wan-import.py` + harness (58/58) -- registers the two DC-edge /24s + (172.30.2/3.0/24, role edge, site-scoped), DRY-BY-DEFAULT with whole-plan preflight. READY to + `--commit` (command in the tool header; token operator-held; guard blocked the live read). + - **Bundle HA overlay:** `overlays/dc-ha-scaleup.yaml` (Stage-5 prep) -- 14 services ->3, one-per- + control-node placement (`{{VR1_DC0_CONTROL_0N}}` tokens, Stage-4-bind), + 3 correctness items the + agent caught: rabbitmq `min-cluster-size:3`, hacluster `cluster_count:3`, re-declare `vault-hacluster` + (BUNDLEFIX-002 had commented it out) + re-add `vault:ha`. + - **The failed opentofu agent** (transient API stall, wrote nothing) was covered in-session by the main + loop; the netbox + overlay agents completed clean. +- **D-123/D-124 RULED + `vvr1-dc0` RACK WIRED (2026-07-16).** D-123 Model A + MAAS region-on-Office1 + + rack-per-DC; D-124 Scheme A (office1<->dc0 mesh transit /30) + rack sizing 4/8192/80. `main.tf` now + wires **`module "vvr1_dc0"`** (cloudinit-vm, 4/8192/80, two legs: metal-admin + mesh-office1 transit, + static IPs from NEW tfvars `vr1_dc0_rack_*` -- NetBox-assigned, not invented; interface-naming trap + flagged) + a `mesh-link` `network_name` output. `tofu validate` Success; opentofu-validate 11/11. + **Step-9 `maas-vm-host` STAYS DEFERRED (DOCFIX-179):** adding the `provider "maas"` block would force + every plan (incl. the Steps 1-8 apply) to demand MAAS creds, and vr1-dc0's MAAS doesn't exist until + vvr1-dc0 is up -- wire it as its own step once the rack MAAS is reachable. The rack MAAS INSTALL needs + a rack-only mode of `site-headend-install.sh` (follow-up). +- **D-124 rack-mgmt importer BUILT (2026-07-16):** `netbox/dc-rack-mgmt-import.py` + harness (96/96), + parameterized (`--transit-cidr`/`--rack-ip`, no invented literal), whole-plan preflight. Gauntlet 62. + **Two operator preconditions it flags before `--commit`:** (a) SEED a dedicated `transit` role in + office1-netbox (none exists; `infra` is shared, not reused) -- die-if-absent, like `edge`; (b) DECIDE + the transit supernet -- the importer assumes it nests under `10.12.0.0/16` (Cloud), but D-124 mirrors + the Edge role (which got its OWN `172.30.0.0/16`), so a dedicated mgmt/transit supernet may be wanted + (then a one-line `CONTAINER` const update). Transit prefix scope = site vr1-dc0 (region may be preferred). +- **RESIDUAL for a smooth batch deploy:** (1) NetBox: seed the `transit` role + decide the transit + supernet, then run the D-124 importer (+ the D-115 DC-edge importer) `--commit` with the assigned CIDR/ + IP, then fill the `vr1_dc0_rack_*` tfvars; (2) runtime: `opnsense-prep-image.sh` for the edge nano; + (3) Step-9 maas-vm-host wiring once the rack MAAS is up (**the rack-only `site-headend-install.sh --role + rack` mode is now DONE, 32/32** -- it enrolls a DC rack to Office1's region; `maas init rack` confirmed); + (4) Stage-4: render the HA overlay's control-node machine IDs + `--overlay --dry-run`; (5) D-121 + verifies (vault MySQL-HA at 3; hacluster cluster_count). All operator-gated / Stage-4. +- **`main.tf` stale comment (flagged):** the commented `vr1_dc1_planes` block (~L38-50) still says + "wait for NetBox to assign D-101's supernet" -- STALE per D-115 (assigned `10.12.64.0/19`). Doc-only + pass did not touch `main.tf`; the `main.tf` owner should correct the comment. +- **STALE gate corrected:** the runbook + `main.tf` say `vr1-dc1` is blocked on "NetBox assigning the + supernet" -- D-115 ALREADY assigned it (`10.12.64.0/19`, imported to office1-netbox). vr1-dc1 stays + out of scope this pass by SEQUENCING, not a missing literal. Do NOT uncomment `vr1_dc1_planes`. +- **Logged, NOT executed (adjacent improvement, hard rule 1):** promote `scratchpad/optc-calc.py` into + a committed whole-host capacity calculator with a harness (sibling to `dc-dc-ceph-disk-budget.sh`). + +The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling +gap register). Read it FIRST. This list is only what that doc does not already carry. + +1. **Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13).** + `voffice1`: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, **10.10.0.20** (Kea reservation, outside + the pool), egress through the OPNsense edge. On it: **MAAS 3.7.2** region+rack (PostgreSQL 16.14) + and **LXD 5.21.5**, with that LXD **registered into MAAS as an LXD VM host**. MAAS then + **composed, PXE-booted and COMMISSIONED** `office1-netbox` (LXD VIRTUAL-MACHINE, leased + 10.10.1.100 from MAAS, reached status **Ready**, 2 cores / 4096 MiB detected). + **The L3 nesting proof is now DEFINITIVE** -- a guest booted, PXE'd and commissioned three levels + deep, not merely "`/dev/kvm` exists". VR0's `lxd` + `tailscale` pattern, reproduced. + Codified in `scripts/site-headend-install.sh` (+ harness, 18/18) -- reusable for DC1/DC2. + **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS + dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is + NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. + **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.10) runs **NetBox + 4.6.4**; `office1-tailscale` (10.10.1.11, tailnet 100.64.0.53) is a subnet router advertising + `10.10.0.0/22` on the operator's **SELF-HOSTED** control plane. The edge routes `10.10.1.0/24 -> + 10.10.0.20`. Full connection reference: **`docs/vr1-office1-as-built.md`** -- read that first for + any "how do I reach X" question. + **GitBucket: OUT OF SCOPE (D-116, operator ruling).** No Office1-local GitBucket is built; + `git.baldurkeep.com` stays the git service of record. Office1's composed machines are exactly TWO. + **REMAINING on Office1 (reconciled 2026-07-14):** (a) route enable -- **DONE**, operator enabled + `10.10.0.0/22` on the control server; the edge GUI answers at 10.10.0.1 from the workstation with + no tunnel; (b) import the D-115 carve into the Office1 NetBox -- **DONE** (sandbox seeded from the + upstream draft and PROVEN faithful field-by-field; carve + six-plane roles + BOTH DCs imported, + 90 -> 134 prefixes); (c) the pinned Office1 LAN IPv6 (item 1a) -- **STILL OPEN**, and it is now + the last build item on Office1. + **STAGE 2 IS IN CLOSE-OUT, NOT BUILD. It is also the BRANCH MERGE POINT** (see PINNED). The + close-out checklist C1..C5 is in `docs/dc-dc-deployment-workflow.md`; do not re-derive it here. +1a. **DONE 2026-07-14 -- Office1's LAN IPv6 is LIVE and PROVEN ON THE WIRE. C1 is CLOSED.** + Address `2602:f3e2:f01:100::1/64` on the edge LAN (kernel-verified) + radvd advertising the prefix + with `mode=unmanaged`; `voffice1` autoconfigured `2602:f3e2:f01:100:5054:ff:fe6a:87e5` by SLAAC and + edge->voffice1 GUA ping is 0.0% loss. No collateral damage (v4, Kea DHCPv4, compose-net route all + intact). See `docs/changelog-20260714-office1-lan-ipv6-executed.md`. + **OPERATOR RULING 2026-07-14 -- DO NOT RE-PROPOSE A v6 NAT.** A v4-to-v6 NAT (second OPNsense, or + nginx) was proposed to *simulate* the v6 the lab will not forward, and was WITHDRAWN in favour of + *"configure IPv6 internally as much as possible; configure the IPv6 edge AFTER deployment + completes."* It contradicts **D-101** (*"native GUA routing, no NAT66"*) and **D-115**'s amendment + (upstream's problem, not ours), and a translation box is the one component Roosevelt will NOT have + -- maximum delta, zero proof value. nginx is an L7 proxy, not NAT, and proves nothing about + RA/ICMPv6/Ceph-over-v6. + **TRAP recorded:** `voffice1` has `forwarding=1` + `accept_ra=0` yet STILL autoconfigured -- + netplan/systemd-networkd handles RA in USERSPACE, independent of the kernel sysctl. Never diagnose + "no SLAAC" from the sysctl alone on Ubuntu; read `ip -6 addr` + look for a `proto ra` route. + **The WAN v6 leg remains DEFERRED** -- v6 does not egress the lab (re-measured 2026-07-14: v6 + internet 100% loss, v4 fine). Internet GUA reachability is UNPROVABLE in VR1. + +1a-historical. **(superseded -- the original pinned item)** deploy Office1's LAN IPv6 (operator ruling 2026-07-13, D-115 + question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). + Add `2602:f3e2:f01:100::1/64` to the OPNsense edge LAN + RA, and let `voffice1` take its address. + **CORRECTED 2026-07-14 (D-113 AMENDMENT -- the old instruction here was WRONG):** the ADDRESS half + CANNOT be done via the REST API. Measured on the live edge (OPNsense 26.1): `Interfaces > [LAN]` is + the LEGACY page `/interfaces.php?if=lan`; only Devices and Neighbors were migrated to MVC, and there + is **NO REST endpoint for a base interface's `ipaddrv6`**. D-113(a2)'s "interfaces are API-covered" + was INFERRED, never measured, and is false. + - **Address:** `scripts/opnsense-set-interface-v6.sh` (dry by default; harness 40/40). It is NOT a + config.xml push -- it mutates two leaves of the edge's LIVE config through OPNsense's own model, + so API-managed Kea and the firewall rules cannot be clobbered. **WRITTEN AND TESTED, NEVER RUN + WITH `--commit`.** + - **RA:** that half IS API-native -- `/api/radvd/settings/addEntry` via `scripts/opnsense-api.sh`. + Still forbidden either way: a rendered config.xml push. + **KNOWN LIMIT, do not forget it:** IPv6 DOES NOT EGRESS THE LAB (measured 2026-07-13: v6 gateway + reachable, v6 internet 100% loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding, + which is most of D-101's family matrix -- but NOT internet GUA reachability, which is unprovable + in VR1 until the lab's upstream carries v6. Deploy the LAN leg; DEFER the WAN v6 leg (it would be + a path to nowhere and untestable). See D-115's amendment. +1b. **`10.10.1.0/24` IS NOW A BLESSED ALLOCATION (D-115 ADOPTED).** VR1 Office1 = `10.10.0.0/22` + (Office role, `10.10.0.0/16`): LAN `10.10.0.0/24` + compose `10.10.1.0/24` + 2 spare. Zero + renumber. Also adopted: Edge role `172.30.0.0/16` (legitimises `office1-wan`'s 172.30.1.0/24, + which had been squatting in the Corp OOB block), and **DC2 moved to `10.12.64.0/19`** inside the + Cloud /16 (its old `10.13.0.0/19` sat outside every allocated block). STILL TO DO: write the + carve into NetBox -- production write is operator-gated; the plan is to import into the Office1 + sandbox NetBox first, verify, then feed back. +1c. **(superseded note kept for context)** the original unblessed-allocation finding: Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, + tenant pool=10.20.0.0/16 per D-016 -- the obvious `10.20.x` would have COLLIDED with tenant space), + but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is + one of the VMs this headend composes. +1c. **`modules/node-vm` nested-virt defect: FIXED 2026-07-13** (operator: fixes land as we find them; + only DC1 *deployment* is gated). Unconditional `cpu = { mode = "host-passthrough" }` -- DC nodes + run `nova-compute` and MUST run KVM guests. Without it they would have come up looking healthy and + been unable to launch a single instance. +2. **NetBox sandbox loop -- DOCUMENTED 2026-07-14 (close-out C4 DONE).** scope-doc section 8 + phase1 runbook Step 10b. (Was: BUILT 2026-07-13/14, undocumented.) + Architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> a NEW NetBox on the Office1 VM + (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements + BACK. The sim NEVER `--commit`s upstream. **The tooling now exists and is proven:** + `netbox/prod-draft-dump.py` (read-only, no write path at all), `netbox/sandbox-seed.py` + (dry-by-default, structurally REFUSES an upstream URL), `netbox/sandbox-fidelity-check.py` + (field-by-field -- counts alone HID a bug where 17 of 90 prefixes silently lost their region + scope). **STILL PENDING:** capture the loop in `docs/dc-dc-netbox-buildout-scope.md` + the Stage 2 + runbook, and **ensure `office1-netbox` holds the COMPLETE validated dataset** (close-out C2, + REDEFINED by DOCFIX-195: `office1-netbox` is the VR1 IPAM apex (the working draft every consuming + system reads); `netbox.baldurkeep.com` is the production apex, held as a READ-ONLY v1 reference draft, + and the write-back to it is DEFERRED to end-of-deployment -- NOT a Stage 2 gate). + **C2 sub-task (i) DONE 2026-07-15** -- the hardened fidelity check was re-run against `office1-netbox` + (read-only dump via the consolidated sandbox token + an SSH tunnel), and the sandbox is PROVEN + faithful: exit 0, "upstream draft + EXACTLY the planned delta." Along the way the hardened checker's + own scope rule was found to FALSE-FAIL on the two intentionally-unscoped role-container /16s + (`10.10.0.0/16`, `172.30.0.0/16`); fixed to assert scope == INTENDED (now also catches a WRONG-TARGET + binding -- the D-117 near-miss), harness 10->12, gauntlet ALL GREEN (58). SEC-009 gap closed: the + sandbox token was never consolidated; it now lives at `~/vr1-office1-creds/vr1-netbox-sandbox.env`. + See `docs/changelog-20260715-fidelity-check-scope-correctness.md`. + **C2 sub-task (ii) DONE 2026-07-15 -> C2 FULLY CLOSED -> STAGE 2 CLOSE-OUT COMPLETE (C1..C5 all done).** + The tooling gap was closed (`prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover + `ip-ranges`/`ip-addresses`; harnesses +T11/T12 and a new `netbox/d120-compose-bands.py` +16/16 + harness), and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox + (`--commit`, operator-approved) and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned + delta (D-115/D-117/D-118/D-120)." Gauntlet ALL GREEN (60). D-120 Step 6 CLOSED (no apex write). See + `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. **NEXT: the branch->`main` merge** (Stage 2 + close is the merge point, PINNED) -- an operator-gated action, present it, do not run it unasked. +3. **`pynetbox` -- CLOSED 2026-07-13.** Installed (7.0.0) on `office1-netbox`, which is where the + imports actually run. The stdlib tools (`prod-draft-dump`/`sandbox-seed`/`sandbox-fidelity-check`) + deliberately need no dependency at all. + **TRAP, recorded in `platform-traps.md`:** the upstream NetBox sits behind a **User-Agent-filtering + WAF**. `Python-urllib/3.12` gets a **403 that looks exactly like an auth failure**; the same token + works from curl. This affects pynetbox too. +4. **The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13.** + `scripts/prereqs/install-apparmor-libvirt.sh` (+ `install-all` / `check-prereqs` wiring; harness + 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants `/** rwk,` in + `/etc/apparmor.d/local/abstractions/libvirt-qemu`, which GATES EVERY VR1 VM BOOT: without it a + domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. + Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce + libvirtd unless it actually adds the rule). Pool parent defaults to `/var/lib/libvirt/vr1`, + override with `VR1_POOL_PARENT`. `docs/changelog-20260713-apparmor-libvirt-prereq.md`. +5. **The DEFERRED literals are now ALL ASSIGNED -- item CLOSED 2026-07-14.** The operator's Option-B + ruling ("do not invent CIDRs, wait for NetBox") held, and NetBox has now assigned them: + **ORG_ULA_48 = `fd50:840e:74e2::/48`** (**D-118 ADOPTED** -- RFC 4193 valid, and verified NOT to + collide with Tailscale's `fd7a:115c:a1e0::/48`, which matters because `office1-tailscale` runs + `--accept-routes`); GUA per D-117: first DC `2602:f3e2:f02::/48`, second `f03::/48`; the second + DC's v4 = six sequential `/22`s from `10.12.64.0/19` (D-115 moved it INSIDE the Cloud /16). + All are IMPORTED and fidelity-checked in the sandbox. + **CAUTION -- `DC2_V4_SUPERNET` IS A RETIRED NAME (D-117).** The importer now hard-rejects it and + rejects `--dc dc2`. Any doc still saying `10.13.0.0/19` is pre-D-115 and wrong. +6. **D-119 (ADOPTED 2026-07-14) -- THE DC NAMESPACE IS REGION-QUALIFIED. C3/gap #19 CLOSED.** + `vr0-dc0` (VR0's LIVE testcloud) / `vr1-dc0` (VR1's FIRST DC, GUA f02::/48) / `vr1-dc1` (VR1's + SECOND, f03::/48). Bare `dcN` is REJECTED everywhere -- it meant VR0's live cloud in `lib-net.sh` + but VR1's first DC in the importer: one string, two clouds. D-119 **COMPLETES D-117** (executes + its own amendment); it does not reverse it. **ZERO NetBox writes** -- the apex was already right; + the REPO was the odd one out. The importer's DC->site map is now an IDENTITY, so the off-by-one is + STRUCTURALLY IMPOSSIBLE, not merely defended. Also renamed: `vdc1`/`vdc2` -> `vvr1-dc0`/`vvr1-dc1` + (D-114's own DR primitive is `virsh destroy vdc1` -- it read as "destroy VR1 DC1" while MEANING + "destroy VR1 DC0"). Office KEEPS its number (`vr1-off1`, `Office1`, `voffice1`). + Env var: `DC1_/DC2_V4_SUPERNET` -> **`VR1_DC1_V4_SUPERNET`** (both old names rejected by name). + Gauntlet ALL GREEN (57); dc-selector 21->30, prefixes-import 40->82. + **THE `tofu apply` IS NOT DONE -- IT IS GATED.** Plan: 11 to add, 0 to change, 11 to destroy (the + libvirt object NAMES change, which is ForceNew; `moved{}` makes the plan reviewable but cannot + suppress a replace). MEASURED SAFE: all 11 are EMPTY (no guests, no volumes; Stage 3 hasn't run). + Re-plan at execution and STOP if anything shows ATTACHED. + See `docs/changelog-20260714-d119-region-qualified-dc-namespace.md`. + +6a. **(historical) G5 NetBox site rename -- CLOSED as D-117 (ADOPTED, Option B).** The repo moved to the apex's + 0-indexed convention (first VR1 DC = `vr1-dc0`), rather than bending the apex to the repo. This + fixed a REAL off-by-one: `dc-dc-prefixes-import.py` bound `--dc dc1` to slug `vr1-dc1` while + treating `f02::/48` as dc1 -- but the apex binds `f02` to `VR1 DC0`. It would have written the + first DC's prefixes onto the second DC's site. The importer is now also **dry-by-default**. + **D-117 IS ONLY PARTIALLY EXECUTED** -- see gap #19 / close-out C3: `opentofu/main.tf` and + `lib-net.sh`/`lib-hosts.sh` still speak the OLD naming, and `lib-net.sh`'s `dc0` means **VR0's** + DC0. That collision BLOCKS Stage 3. + G2 (aggregates) and G3 (ULA /48) are both CLOSED by the same work. +7. **DOCFIX-185 follow-up (operator to rule):** a D-100/D-107 amendment note so the Stage-3 DC + edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark + fiber is East-West/replication ONLY, never an internet path). +8. `docs/dc-dc-deployment-workflow.md`'s status table was corrected 2026-07-13, but re-read it + against reality before trusting any row -- it has drifted twice. +9. **DHCP PROVEN END TO END -- CLOSED 2026-07-13.** `voffice1` (the first client ever to sit on + `office1-local`) took a real lease from Kea: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches + its NIC), hostname `voffice1`, state active -- read back through the D-113(a2) REST API. The + service, not just the daemon, now works. +10. **Doc-accuracy finding (measured 2026-07-13, logged not actioned):** `docs/security-ledger.md` + SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web + GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is + correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default + and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE + the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only + path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. + Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local + libvirt bridge, `virbr2`, not routable off-host): `ssh -L 8443:10.10.0.1:443 @10.17.11.248` + -> `https://localhost:8443`. + +**VCLOUD POST-RESTART SWEEP + D-126 BASE-LEG + D-128 OPERATING MODEL (2026-07-17) -- SESSION CLOSED +(committed + pushed; two honest carry-forwards below).** The vcloud host +completed its bookend patch+reboot (running `6.8.0-136`, 0 upgradable, no reboot-required). Full sweep +GREEN: libvirt 12 nets/4 pools active+autostart, both VMs autostart=enable+RUNNING, autostart chain proven +end-to-end (all guests+edge uptime matched host boot). Post-reboot `ssh voffice1` was DOWN -- root cause: +`office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` on the bridge + +compose route `10.10.1.0/24 via 10.10.0.20`) lives nowhere in the XML and drops every reboot; D-126 made the +layers above it durable but never owned this base leg. **DELIVERED:** `scripts/site-baseleg.sh` + +`tests/site-baseleg/` (24/24) -- a root `Type=oneshot` unit `After=libvirtd` that re-adds the leg at boot +(bridge DISCOVERED from the net name, not baked; idempotent apply; `check` = read-only session-verify). +Gauntlet ALL GREEN (67), repo-lint 0-fail. **D-126 amendment** + **NEW D-128** (operating model: Claude on +the vcloud jumphost; Plane-1 substrate on vcloud / Plane-2 executes on voffice1; workstation tailnet = human +path; D-107 untouched) recorded; skill invariant + workflow Cross-cutting bullet added. See +`docs/changelog-20260717-d126-baseleg-oneshot-d128-operating-model.md`. **INSTALLED 2026-07-17; boot-race +UNPROVEN:** `install` run (operator sudo) -- unit written + `enabled` (multi-user.target.wants), `apply` ran +at install -> status=0, `check office1` -> ok. That run was POST-boot so it did NOT exercise the +`After=libvirtd`+poll race; boot durability stays UNPROVEN until an ACTUAL reboot shows `ssh voffice1` +self-heals. Ordering MEASURED (monolithic libvirtd active here). **COMMITTED + PUSHED** on branch +`dc-dc-stage3-phase2-dc-substrate`: `1958063` (tool + records + sweep) and `6f26b55` (tooling-gap register +item **20** CLOSED for Office1 + the skill's per-DC discipline). Gauntlet ALL GREEN (67), repo-lint 0-fail, +next-free D now **129**. `main` untouched (stage-close merge deferred). + +**CARRY-FORWARDS:** +1. **Boot-race proof -- PROVEN 2026-07-18 (RETIRED).** The operator-forced test reboot (host booted + 2026-07-17 23:38:41) exercised the `After=libvirtd`+poll race, and all four checks PASSED: + (a) `site-baseleg-office1.service` active(exited) status=0; (b) the unit journal logs + **`added leg: 10.10.0.10/24 on virbr2`** + `route set: 10.10.1.0/24 via 10.10.0.20 dev virbr2` at + 23:38:53 -- the STRONG proof (leg ABSENT at boot -> the unit CREATED it, not "leg present"); (c) `bash + scripts/site-baseleg.sh check office1` -> ok; (d) `ssh voffice1` returned live with NO manual `ip` + command this session. So the D-126 base-leg oneshot self-heals `office1-local`'s dropped L3 leg across a + real reboot, unattended. (Journal read needed root; jessea123 was added to the `systemd-journal` group + 2026-07-18 + relog so future sessions read unit journals without sudo.) Register item 20 / skill routing + row stand as the durable mechanism. +2. **DC leg rows.** `site-baseleg.sh` generalizes to the DCs but a DC row is DEFERRED-until-measured (hard + rule 2; the harness fails on an un-commented DC supernet). At each DC standup, MEASURE the vcloud->DC + reach and add a MEASURED leg row + install ONLY if that net is isolated (register item 20 / skill invariant). + +**OPERATOR PERSONAL KEY pushed to Office1 service VMs (2026-07-18).** On operator request ("push my +id_ed25519 on the vms"), the operator's GitHub-published personal pubkey (`ssh-ed25519 ...J7Zg +JANeumatrix@github`, gh:JANeumatrix) was APPENDED to `~/.ssh/authorized_keys` on **voffice1** (user +jessea123), **office1-netbox** and **office1-tailscale** (user ubuntu) -- ALONGSIDE the `vr1-office1-svc` +service key (service key untouched; the D-126 ProxyJump path is unchanged). Read-only audit first (key +ABSENT on all three), idempotent grep-guarded append, verified 2 keys each. This is a deliberate EXTENSION +of the D-126 per-site-service-key model: the operator's personal identity now has direct login from any +workstation over the tailnet (`ssh jessea123@10.10.0.20` / `ubuntu@10.10.1.10` / `ubuntu@10.10.1.11`). +Pubkey ONLY -- the personal private key is workstation-held, never on the jumphost; login-with-personal-key +still needs the operator to test from a workstation. **Edge (office1-opnsense): operator installs via the +OPNsense GUI** (System > Access > Users > root > Authorized keys) -- a raw `/root/.ssh/authorized_keys` +append is NON-DURABLE (OPNsense regenerates root's keys from `config.xml`); config store audited, personal +key ABSENT there too. **STANDING (per-DC): at each DC standup, add the same personal pubkey to that DC's +service VMs** ("in case I need to log in directly" -- operator instruction), alongside the base-leg per-DC +discipline above. **REMOVAL/ROTATION OBLIGATION:** strip these personal keys at v1 close / teardown (same +class as the SEC key-rotation rows); track if a security-ledger row is wanted. + +**OPNsense EDGE 26.7 UPDATE + PLUGIN/ADD-ON REVIEW (2026-07-18).** Operator updated `office1-opnsense` +**26.1.11 -> 26.7** (0 outdated pkgs, no pending firmware). As-built version RECORDED in +`docs/vr1-office1-as-built.md:42` (appendix-B is charm-channels only, no edge row). Operator asked for a +forward-looking plugin/add-on review -> `docs/opnsense-edge-addon-review-20260718.md` (all 104 catalog +plugins categorized) + **D-129 (PROPOSED)** captures the base-edge-profile decision. Near-stock edge: +only `os-git-backup` installed, and **UNCONFIGURED** (php-verified no config node) -> **no config backup is +actually running** -- recommend wiring it to `git.baldurkeep.com` (config-as-code) regardless of D-129. +Review recommendations by bucket: metal-hardware profile (`os-smart`/`os-nut`/`os-cpu-microcode`/`os-lldpd`, +record for Roosevelt, inert in VR1 VMs); sim scaffolding (`os-qemu-guest-agent`, VR1 VM edges only, aids +D-126/D-127 reboot work); diagnostics (`os-iperf` for transit validation); observability +(`os-node_exporter` CONDITIONAL on a COS-scope ruling -- D-105 scoped COS per-DC inside the cloud, edge is +outside); DEFER all security hardening per closed-test posture. **UPDATE 2026-07-18 -- operator directive "install the recommendations + pin the rest".** D-129 now +PARTIALLY RULED: VR1 edge profile **{`os-qemu-guest-agent`, `os-iperf`}** adopted. Live install on +`office1-opnsense` via the D-113 firmware API is **operator-GATED** -- the auto-mode classifier blocked the +agent's `core/firmware/install` POST (cloud mutations stay gated even during batch), so the operator runs it +via `!` (or approves); **verification pending** (confirm the `qemu-guest-agent` DAEMON runs, not just the +pkg, + edge DHCP/NAT unbounced). The "rest" is PINNED (not built now, per advisor -- pin = record at the +step): metal profile {`os-smart`/`os-nut`/`os-cpu-microcode`/`os-lldpd`} -> Roosevelt edge build (inert on +VR1 VMs); `os-node_exporter` -> `dc-dc-phase6` Step 10 gated on COS-edge scope; VR1 profile -> `dc-dc-phase2` +edge section (+ build the reusable profile-installer there); security -> deferred (posture). **FOUR +sub-decisions STILL OPEN** (COS-edge scope, `os-frr`, `os-tailscale`, edge-NTP) -- a one-line directive does +not rule them. As-built row updated (FreeBSD base 14.3->15.1 too). +**INSTALL VERIFIED 2026-07-18 (operator ran the firmware POSTs via `!`):** `os-qemu-guest-agent` package +INSTALLED but NON-FUNCTIONAL -- daemon off + `opnsense-edge` module defines no `org.qemu.guest_agent.0` +libvirt channel (grep-confirmed), so `guest-ping` fails. Functional qga = channel-in-module + service-enable ++ edge RESTART -> DEFERRED to next scheduled edge restart (new DC edges born with the channel). `os-iperf` +did NOT install: two back-to-back firmware POSTs collided on the OPNsense firmware LOCK -> re-run pending +(profile-installer must be one-at-a-time + poll). Records reconciled to this reality (D-129 Verification +note, phase2 pin, as-built). Docs committed on branch `dc-dc-stage3-phase2-dc-substrate` (pushed +2026-07-18, `83876c3..c1ef7ab`). +**AUTONOMOUS PROCESSING 2026-07-18 (operator: "process what you can without me running anything").** +Boot-race carry-forward RETIRED (D-126 base-leg PROVEN across the forced reboot -- `added leg` in the +journal; jessea123 added to `systemd-journal` group so unit journals read without sudo). Built the D-129 +profile-installer `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (17/17; gauntlet 68 GREEN) -- +closes the "build the installer" pin; encodes one-at-a-time+poll + the qga-channel warning. Logged the qga +channel design (OPT-IN var, default OFF -> no office1/DC0-plan side-effect; mechanism-to-verify on dmacvicar +0.9.8) + the `os-git-backup` operator-input ask. **Still operator-only:** `os-iperf` re-install (classifier- +gated), qga functional enablement (channel+enable+restart), git-backup wiring, the 4 D-129 sub-decisions. +See `docs/changelog-20260718-opnsense-plugins-installer.md`. + +**PERMISSION POSTURE UPDATED 2026-07-18 (mutations PROMPT, not classifier-deny).** The `auto`-mode +classifier was HARD-DENYING mutations that matched no rule (edge API POST, settings edits, Skill invokes) -- +no approval prompt, so the operator had to run them by hand. claude-code-guide confirmed (doc-cited) that +`ask`/`deny` rules are evaluated BEFORE the classifier, so `ask`-listing the mutation surfaces forces a +prompt. `.claude/settings.json` updated (operator-applied by hand -- editing the permission file is itself +gated; chicken-and-egg): added `ask` for `*opnsense-api.sh POST *`, `*opnsense-plugins.sh apply*`, `*tofu +apply/destroy/import/state rm*`, `*--commit*`, `Edit(.claude/settings*.json)`; added `allow` for the edge +GET + `opnsense-plugins.sh list`. The `Edit(.claude/settings*.json)` ask rules mean the agent can now +PROPOSE settings edits (prompted), no longer hard-blocked. Stayed on `auto`+`ask` (not `default` mode) to +keep read-heavy audits frictionless. See `docs/changelog-20260718-permission-posture-ask-preempts-classifier.md`. +**CARRY-FORWARD (verify after the operator's reload):** the new rules take effect only after a Claude Code +restart. Post-reload, CONFIRM a mutation now PROMPTS (approve/deny/adjust) instead of hard-denying -- the +`os-iperf` install (`OPNSENSE_API_HOST=10.10.0.1 bash scripts/opnsense-plugins.sh apply vr1-edge`, or the raw +`opnsense-api.sh POST core/firmware/install/os-iperf`) is the ready test. If it still hard-denies, the rule +pattern needs adjusting (check the exact command string vs the glob). + +## OPEN WORK -- VR0 / DC0 (the running testcloud) + +1. **d011 batch-3 live-validation queue** -- `scripts/checks/d011-04-octavia-lb.sh` and + `d011-05-magnum-e2e.sh` are DRAFTED and mock-tested; the live PASS was recorded in the + 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: + d011-04 `amphora_headroom()` field parsing (safe default HOLDs on a parse miss -- confirm the + OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant + (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set + `VR_FOIL_APPCRED`. Sequence: non-disruptive half first, then `--include-disruptive`. +2. **Script backlog item 3 (remainder):** fold `scripts/vault-kv-health.sh` into `cloud-assert.sh` + as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script + -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure + runbook. +3. **Script backlog item 5:** `scripts/tenant-cluster-create.sh` -- generalize the stage-6 + + watch + D-066-evidence wrapper. NOT STARTED (verified absent). + (Items 6/7/8 -- `keystone-policy-drift.sh`, `cloud-snapshot.sh`, `tests/tenant-acceptance/` -- + are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.) +4. **S-class script quality (`docs/script-quality-findings-20260707.md`):** S4/S7/S8a/S8b DONE; + S6 no-action; S8c ruled WON'T-DO. **S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ + offboard) are HELD** by the operator until the DOCFIX-126/127 live-verify lands. +5. **DOCFIX-126/127 live-verify is still OPEN** -- the offboard-E0 and onboard-cred-file fixes have + never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard + confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage. +6. **`test1` orphan user -- approved for deletion, staged, NOT run.** Role-less shell in the + devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): + `openstack user delete 9a62f88e331f41adb5c9e41225ec8698`. +7. **The handoff doc is stale:** `docs/handoff-20260703-open-items.md` still lists R-3 (D-063) as + PROPOSED/OPEN. D-063 is `RESOLVED / CLOSED` in `design-decisions.md` (verified). Correct the + handoff doc. +8. **Upstream reports queued (operator):** magnum `can-upgrade-to` anomaly -- Charmhub magnum + 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from + the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN + -- operator judged it site misconfig; the draft is retained.) +9. **Barbican `host_href`:** the charm renders a literal `https://None:9312`, suppressing the + upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A + charm-level fix would need its own D-NNN. Logged, not actioned. +10. **v1-close checklist is QUEUED** (handoff section 6) -- gated on D-011 item 6, which is gated + on the SEC-003 unseal rehearsal. + +## OPEN -- NetBox WRITE-PATH BUGS (found 2026-07-14 by adversarial review; LOGGED, NOT FIXED) + +These are independent of D-119's naming work. **DOCFIX-195 (2026-07-15) reclassified WHEN they matter:** +C2 no longer writes the PRODUCTION apex `netbox.baldurkeep.com` (office1-netbox is the VR1 IPAM apex; +the `netbox.baldurkeep.com` write-back is DEFERRED to end-of-deployment). So these bugs gate the DEFERRED merge-back, NOT Stage 2 +close -- they are what would corrupt the production apex during THAT write. They no longer block C2 or +the Stage 2 merge. + +1. **`roles-aggregates-import.py` still dies HALF-WAY through a production write.** The preflight it + advertises only checks role-NAME collisions. The `ARIN` RIR lookup and `validate_ula_48()` both run + **after** the role-creation loop has already committed 5 roles. A typo'd `ORG_ULA_48`, or an apex + without ARIN, writes 7 objects then dies with **no rollback**. The RIR preflight loop is a literal + no-op (`if ...: continue` + a comment). The harness cannot catch it: it only asserts the STRING + "PREFLIGHT" appears before the first create. **This is the same bug class that already bit us once.** +2. **`sandbox-fidelity-check.py` can return a FALSE GREEN.** It field-compares only the SHARED objects; + the planned delta is checked merely as a SUBSET of `EXPECTED_NEW`. So a prefix created with the + WRONG scope/role/status passes clean -- and if the importer created ZERO of the 36 DC prefixes, + `extra subset-of expected` still holds and it exits 0 saying "nothing lost, nothing stray." + **We have been citing this script as proof the sandbox is faithful. It cannot prove delta + correctness.** It WOULD have caught the historical 17-prefix scope drop (those were shared objects). +3. **`find_role()` dies mid-loop** in `dc-dc-prefixes-import.py` -- roles are resolved lazily INSIDE + the write loop. Against upstream as it stands today (no six-plane roles), a `--commit` creates the + site + 4 prefixes then dies on the 5th. No whole-plan preflight. +4. **Same-dumper blind spot:** `sandbox-fidelity-check` compares two dumps produced by the SAME field + list, so any field the dumper omits is invisible on BOTH sides by construction (prefix `vrf`, + `tenant`, `vlan`, `tags`, `custom_fields`; site `tenant`/`group`/`facility`). Structurally the same + shape as the region-scope bug that dropped 17 prefixes. +5. **Duplicate-CIDR collapse:** prefix-keyed dicts in the seeder and the fidelity check collapse + duplicate prefixes (NetBox permits them, and the importer's own docstring anticipates them vs + `vr0-dc0`). A whole prefix object can vanish with a green check. Latent -- no duplicates upstream today. +6. **The test fake diverges from reality at the failure point:** `fake_pynetbox.get()` returns + `matches[0]` on a multi-match where REAL pynetbox RAISES. So the harness cannot catch #5. + +**Recommended sequence: fix 1-3 BEFORE C2.** 1-3 are DONE (2026-07-14, +`docs/changelog-20260714-netbox-write-path-hardening.md`): the roles-importer preflight now covers +ARIN + the ULA, the prefixes-importer preflights every role, the fidelity check is BOTH-bounds + +delta-scope-aware, and the test fake raises like real pynetbox. Gauntlet 58 ALL GREEN. +**CONSEQUENCE TO ACT ON:** the sandbox was declared "proven faithful" (2026-07-14) using the +fidelity check BEFORE it was hardened -- i.e. under the version that could FALSE-GREEN. RE-RUN +`sandbox-fidelity-check.py` against the sandbox under the hardened check before trusting that +"faithful" verdict -- this re-run IS C2 sub-task (i) (DOCFIX-195), run against `office1-netbox` (the +VR1 IPAM apex), NOT against the production apex `netbox.baldurkeep.com`. (Not done this session: it +needs the office1-netbox token, which is operator-held; folds naturally into the C2 step.) +Items 4-6 remain LOGGED not fixed. + +(Historical rationale: 1 and 3 are partial-write-no-rollback on the production IPAM; 2 is what we would use to prove the write was correct.) + +## Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions) + +- **STAGE 2 MERGED -- DONE 2026-07-15 (PR #1, `e468d2e`).** The Stage 2 close-out (C1..C5 all met) + branch `dc-dc-stage1-phase0-first-exec` merged to `main` via a merge commit -- the first VR1 feature + merge. `main` IS now trunk-of-record for VR1 through Stage 2; the branch was retired. This is no + longer an open ruling; it GRADUATED into the STANDING **branch-per-stage + merge-at-stage-close** + discipline that applies to every stage from here (workflow doc "Cross-cutting discipline"): the merge + is part of each stage's definition-of-done and remains operator-gated (present it, do not run it + unasked). Stage 3 branches off the post-merge `main` as `dc-dc-stage3-phase2-dc-substrate`. +- **UPSTREAM APEX WRITE-BACK -- DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15).** + Seeding `netbox.baldurkeep.com` from the validated `office1-netbox` dataset is a MAYBE at the very + end, NOT a Stage 2 gate. Until then the apex stays a READ-ONLY v1 reference draft (no write during + VR1); `office1-netbox` (10.10.1.10) is the VR1 IPAM apex (the working draft every consuming system + reads). The "NetBox WRITE-PATH BUGS" section gates THAT deferred write, not Stage 2 close. Defers the + write; does NOT revoke D-010. +- **D-071** ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning + session. Backups now exist (a 902MB controller backup was taken; `juju create-backup` / + `download-backup` DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). + Controller HA (single-controller, no-HA) is the unresolved half. +- **D-068** off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714. +- **SEC-001 / SEC-004 / SEC-005 / SEC-007:** rotate / flip-to-private / revoke at v1 close. +- **SEC-003:** custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. + This keeps `d011-06-vault-unseal` MANUAL and gates the FULL close of D-011. +- **SEC-006** (burned NetBox API token) -- **DEFERRED by operator ruling 2026-07-13 to deployment + completion.** Until then the token is LIVE and EXPOSED. +- **D-076 is NOT this repo's to work.** D-076..D-099 are a RESERVED source-project band, never + assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live + in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are + running their own tests. +- **quota-vs-capacity: CLOSED** -- operator ACCEPTS as-documented (no quota trim). + +--- + +## Standing lessons and traps (load-bearing; these live nowhere else) + +1. **THE EDGE TRAP -- DHCP on Office1 is API-MANAGED.** A full rendered `config.xml` push WOULD + CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all + gone -- `opentofu/templates/README.md` is a tombstone). Configure the edge via the REST API + (`scripts/opnsense-api.sh`); mint keys with `scripts/opnsense-bootstrap-apikey.sh`. +2. **"updated in-place" DOES NOT mean "no restart".** A `tofu apply` whose plan says + `libvirt_domain.vm will be updated in-place` **RESTARTED the guest** (measured: uptime 8:36 -> + 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it + says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to + apply a disk-list change and BOUNCES it. **Assume ANY apply touching a `libvirt_domain`'s + devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.** +3. **Do not re-implement vendor internals; call the interface.** OPNsense stores API secrets as + `crypt(secret,'$6$')` (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we + deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep + byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate). +4. **A config ISO can NEVER be read on an OPNsense nano image** (D-112). Any instruction to build + one is a trap; the dc-dc-phase2 runbook directed people into it for days. +5. **root's shell on the OPNsense edge is `tcsh`, not `sh`.** A `$(...)` inside a quoted remote + command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: + a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. + **Always feed remote commands to `sh -s`.** (The `scripts/opnsense-apply-config.sh` that was + once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml + path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.) +6. **Guide changes must sweep the tenant AI skill in the SAME change.** Skill-lag was caught twice + (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 + is pinned, and any guide edit FAILS lint until re-recorded). +7. **The controller model is `admin/controller`,** not `/controller`. +8. **`juju status --format=line` omits workload messages on 3.6** -- this caused a cloud-assert + false negative (fixed by DOCFIX-087). Don't rely on it. +9. **repo-lint's guards are load-bearing; reword rather than weaken.** L3 (docs must not reference + missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, + the guard kept. + +## State facts to remember + +- **The beta cluster and the foil1 tenant are GONE** (both confirmed absent by the 2026-07-08 + read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is + retired here. `capi-test-1` and the `lbtest` LB leftover are also gone; orphan sweep reads + ORPHAN=0, zero unattached floating IPs. +- **devteam is a LIVE tenant** running a self-created cluster (built via Horizon, calico, + keypair `devteam-key`). Their template has `volume_driver` unset but the cluster DOES get a + default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the + box; no template change needed. +- The vcloud host was NOT pristine: 9 orphaned `vr0-*` libvirt nets were torn down; **`wan` was + KEPT** (dual-stack NAT-out-enp1s0, GUA `2602:f3e2:fe:10::/64`) as the working model for the + per-site ISP-uplink network. +- MTU has TWO domains: the ISP uplink `enp1s0` is 1500; the host-internal plane/dark-fiber fabric + is **jumbo, underlay_mtu=9000** (operator ruling; verified on all 10 nets). +- The repo is temporarily **PUBLIC** (SEC-004) -- flip private at v1 close. +- Secrets live jumphost-local and are NEVER committed: `~/vr1-office1-creds/` (edge SSH key, + root password, API key), `~/vr1-office1-creds/vr1-netbox.env` (NetBox token), `~/tenant-devteam/`, `~/vault-init/`. + +## Project-completion (execute after D-011 passes) + +- Consolidate the 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. +- Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials. +- v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. + +## NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13) + +**Done.** The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop +(`netbox/prod-draft-dump.py` READ-ONLY -> `netbox/draft/vr1-draft.json` -> `netbox/sandbox-seed.py`), +seeded 129 objects from the upstream draft, then applied the **D-115 office carve** +(`netbox/d115-office-carve.py`): site `vr1-off1`, roles `office` + **`edge`** (new), and 8 prefixes. +Idempotent, and **verified FIELD-BY-FIELD** against the upstream draft +(`netbox/sandbox-fidelity-check.py`): every shared object is IDENTICAL, and the delta is EXACTLY +`+edge` role, `+vr1-off1` site, `+8` D-115 prefixes. The sandbox is a proven-faithful replica, which +is what makes it a valid baseline for the later feed-back diff. See `docs/vr1-office1-as-built.md` s6. + +**Why that check exists:** counts and idempotency CANNOT prove fidelity. The seeder silently dropped +the scope on 17 of 90 prefixes (it mapped only `dcim.site`; most of the VR1 draft is +`dcim.region`-scoped) -- and every count still matched and re-runs were still idempotent. It was +caught by LUCK on a spot-check. The fidelity check is that catch, made deliberate. + +**D-117 Option B is only PARTIALLY EXECUTED.** Done: the import path (`dc-dc-prefixes-import.py`) and +`docs/dc-dc-netbox-buildout-scope.md`. NOT done: the `$DC` shell selector (below) and the +forward-looking DC prose in the runbooks. Do not read "D-117 ADOPTED" as "fully renamed". + +**D-117 ADOPTED (operator ruling, Option B, DC-only):** VR1's DCs are **dc0/dc1**, matching the apex. +The **office KEEPS its number** (Off1). Fixed the importer's off-by-one site binding and made it +**dry by default**. + +### ~~BLOCKED~~ **UNBLOCKED and DONE** -- D-118 ratified the org ULA; both DCs imported + +**D-118 ADOPTED:** `ORG_ULA_48 = fd50:840e:74e2::/48` (RFC 4193 valid; **no collision with +Tailscale's `fd7a:115c:a1e0::/48`**, which matters because `office1-tailscale` runs `--accept-routes` +-- an overlap would have been a LIVE tailnet routing conflict). Also corrected a **FALSE claim** in +`changelog-20260711-ula-gen-command-fix.md` that called the value "ratified" when no D-number ever +assigned it -- the same document-asserts-authority failure mode as D-117. + +**Imported into the sandbox:** six-plane roles + 2 RIRs + 5 aggregates, then **VR1 DC0 (18 prefixes) +and VR1 DC1 (18)**. The D-117 fix is now **LIVE-PROVEN**, not just offline: `--dc dc0` bound to the +apex site `VR1 DC0` (id=8) and `--dc dc1` to `VR1 DC1` (id=9). ULA reads `DC.NN` in the 4th hextet +exactly as D-111 intends (`fd50:840e:74e2:220::` .. `:350::`). + +**Final state PROVEN by `netbox/sandbox-fidelity-check.py`:** the sandbox is the upstream draft plus +EXACTLY the planned delta (D-115 + D-117 + D-118) -- every shared object field-identical, nothing +lost, nothing stray. 90 -> 134 prefixes. + +### A REAL BUG in `roles-aggregates-import.py`, found by running it for the first time for real + +It checked existence by **slug only**. NetBox also enforces **unique NAMES**. The draft's legacy role +`Replication` (slug `repl`) blocks the six-plane role (slug `replication`, name `Replication`), so it +created 4 roles, **400'd on the 5th, and DIED -- leaving the apex half-populated** with no RIRs and no +aggregates and no rollback. It had **never been run against a NetBox holding the real draft** +(only empty ones) and **shipped with NO harness** -- which is why this survived. + +Fixed: the six-plane role is now named **"Replication Plane"** (slug unchanged -- `lib-net.sh` +SPACES6 and the prefix importer look it up), and the script now **PREFLIGHTS every name/slug and +writes nothing unless the whole plan is viable**. `tests/roles-aggregates-import/` created: 16/16. + +### (historical) the original blocker + +`netbox/dc-dc-prefixes-import.py --dc dc0` cannot run. It requires `ORG_ULA_48`, and that literal is +**still unassigned** -- it is gap **G3** in `docs/dc-dc-netbox-buildout-scope.md`, which *recommends* +`fd50:840e:74e2::/48` (CSPRNG-generated, Global ID `0x50840e74e2`) but **no ADOPTED decision assigns +it** (`grep ORG_ULA docs/design-decisions.md` -> nothing). D-101 explicitly leaves it NetBox-assigned. + +Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. +This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready. + +**To unblock:** ratify an org ULA `/48` under its own D-number. Then: +``` +ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # VR1 DC0 +ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f03::/48 \ + DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # VR1 DC1 +``` +(`DC_GUA_PREFIX` values are MEASURED off the apex; `DC1_V4_SUPERNET` is assigned by D-115. +`DC2_V4_SUPERNET` is RETIRED -- the tool rejects it by name.) + +### Also open (recorded, not blocking) +- **`$DC` shell-selector namespace collision** (D-117 amendment): `lib-net.sh`'s `dc0` means **VR0's** + DC0, while the importer's `dc0` now means **VR1's**. NOT interchangeable. Proposed fix: region- + qualify the shell selector to the apex slugs (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`). +- Feed the D-115 carve upstream to `netbox.baldurkeep.com` (operator-gated; the tool refuses a + non-sandbox target without `--yes-write-upstream`). +- Office1 LAN IPv6 (`2602:f3e2:f01:100::/64` + RA via the OPNsense REST API) -- still pinned. diff --git a/docs/maas-as-built-reference.md b/docs/maas-as-built-reference.md index e645fb5..055a45b 100644 --- a/docs/maas-as-built-reference.md +++ b/docs/maas-as-built-reference.md @@ -189,3 +189,24 @@ resolution; auto-commission-on-create; tag `openstack` re-apply before deploy; `default-space` must resolve to a LIVE space (a stale default-space globally poisons `network-get`). + +## Standing VR0 operational facts (routed from the session ledger, 2026-07-19) + +Rotated here as the durable home during the GA-R4 ledger rotation (measured +facts, as-of their audit dates; re-verify before load-bearing use): +- Beta cluster + foil1 tenant GONE (2026-07-08 read-only audit); capi-test-1 + and the lbtest LB leftover gone; orphan sweep ORPHAN=0, zero unattached + floating IPs. +- devteam is a LIVE tenant with a self-created cluster (Horizon, calico, + keypair devteam-key); cinder-CSI magnum default supplies a StorageClass -- + no template change needed. +- The vcloud host was NOT pristine at VR1 start: 9 orphaned vr0-* libvirt + nets torn down; `wan` KEPT (dual-stack NAT-out-enp1s0) as the per-site + ISP-uplink model. +- MTU has TWO domains: ISP uplink enp1s0 = 1500; host-internal plane/mesh + fabric = jumbo underlay_mtu=9000 (operator ruling; verified on all 10 + nets at the time). +- The controller model is `admin/controller`, not `/controller` + (see also runbooks/ops-update-procedure.md). +- Repo visibility: PUBLIC until v1 close (SEC-004, security-ledger). + Secrets are jumphost-local, never committed (creds-manifests/). diff --git a/docs/session-ledger.md b/docs/session-ledger.md index ab9697b..e00a803 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -61,1084 +61,28 @@ --- -## Live state (measured 2026-07-13; nothing is mid-flight) +## ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable) -- **VR1 / Office1 edge `office1-opnsense` is UP and healthy.** LAN 10.10.0.1/24, WAN 172.30.1.2 - (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial - console + getty alive, and `kea-dhcp4` serving DHCP on udp/67 (subnet 10.10.0.0/24, pool - .100-.199). **DHCP is API-MANAGED (D-113(a2)).** -- **OpenTofu repo/state back IN SYNC -- D-119 rename APPLIED (2026-07-14, operator-gated).** - `tofu apply`: **11 added, 0 changed, 11 destroyed**; `tofu plan` now reports "No changes". Verified - on the host: six `vr1-dc0-*` networks, three `mesh-vr1-*` links, `vr1-dc0-pool`/`vr1-dc1-pool`; ZERO - old `dc1-*`/`dc2-*`/`mesh-dc*` names remain. Pre-apply LIVE re-check confirmed all 11 objects empty - (0 leases, 0 volumes, no domain attached). Office1 (`office1-local`/`office1-wan`/`wan`/edge/ - `voffice1`) was NOT in the plan and is untouched -- edge + voffice1 still reachable post-apply. - tofu v1.12.3; tree validates (10/10 modules); lock committed. -- **VR0 / DC0 testcloud:** fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except - item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own - cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts). -- **STAGE 2 CLOSED AND MERGED (2026-07-15).** All close-out gates C1..C5 met; branch - `dc-dc-stage1-phase0-first-exec` merged to `main` via **PR #1 (`e468d2e`, merge commit)** -- the - FIRST VR1 feature merge. `main` is now trunk-of-record for VR1 through Stage 2; the merged branch - was RETIRED (local + remote). **NEXT: Stage 3** (DC0/DC1 substrate, `dc-dc-phase2`->`phase3`) on - branch **`dc-dc-stage3-phase2-dc-substrate`** (off post-merge `main`); read - `docs/dc-dc-deployment-workflow.md`'s tooling-gap register FIRST. Now standing: **branch-per-stage + - merge-at-close** discipline (workflow doc Cross-cutting) -- merging the stage branch is part of every - stage's definition-of-done. In place for Stage 3: region-qualified `$DC` selectors (D-119) + the - creds-audit mint-time discipline (SEC-009 / `scripts/creds-audit.sh`). -- (historical) Branch `dc-dc-stage1-phase0-first-exec` was pushed and level with origin (2026-07-14); - VR1 Stage 1 COMPLETE; Stage 2 was in CLOSE-OUT -- now superseded by the merge above. -- **SESSION CONSOLIDATION (2026-07-14).** Work had fragmented across FOUR concurrent Claude sessions - on this one clone -- including two long-lived `--permission-mode auto` background agents (idle 14-15h - but still holding auto-approve rights against the live cloud), and a second interactive session that - received the SAME "resume last night's work" prompt six minutes before this one and began bootstrapping - in parallel. All three were terminated and the supervising daemon stopped (it had respawned the agents - once). **Standing lesson: the daemon ADOPTS AND RESPAWNS background workers -- killing the child is not - enough; stop the daemon (`claude daemon stop --any`).** The working tree was clean and 3-ahead - throughout; nothing was lost or clobbered. ONE session is now the rule. -- **Last night's three commits (D-113 amendment + the v6 script) were made by the auto-permission - background agent and left UNPUSHED and UNDOCUMENTED.** Both closed 2026-07-14: pushed, and - `docs/changelog-20260714-opnsense-set-interface-v6.md` written. +The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox +write-path bugs, PINNED rulings, standing lessons, state facts, project- +completion, NetBox import narrative) moved VERBATIM to +`docs/archive/session-ledger-rotated-20260719.md`. Standing lessons + VR0 +state facts were ROUTED to durable homes first (platform-traps already +carried most; juju format=line -> appendix-A; guard discipline -> +operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). +Open work is machine-derived above + CURRENT-STATE.md; do not re-grow +bodies here -- sessions append BOUNDED summaries only (15-line cap). ---- - -## OPEN WORK -- VR1 DC-DC (the current mission) - -**D-120 ADOPTED + EXECUTED (2026-07-15) -- static-services addressing for the compose /24s.** -Office1's two services were re-IP'd OUT of the MAAS node band into the `.2-.49` static band: -`office1-netbox .201 -> 10.10.1.10`, `office1-tailscale .202 -> 10.10.1.11`. On the wire, no wipe -(NetBox HTTP 200 at .10; tailscale still advertises `10.10.0.0/22`). Band layout ratified: `.2-.49` -static / `.100-.200` dynamic / `.201+` nodes, for every MAAS compose /24; DCs inherit it from day -one. Method finding: MAAS won't edit a Deployed machine's interface (lift = redeploy = WIPE), so the -address is set on the guest (netplan + curtin, over `lxc exec`) and MAAS-model drift is accepted -(reconciles at next teardown/redeploy). Sweep + gauntlet done; `runbooks/dc-dc-office1-service-reip.md` -now records the measured method. **D-120 Step 6 DONE 2026-07-15** -- the /24 child ranges + the two -service IPs are registered in `office1-netbox` (the VR1 IPAM apex) via `netbox/d120-compose-bands.py ---commit` and VERIFIED by the (now-extended) fidelity check -> exit 0. The `ip-ranges`/`ip-addresses` -tooling gap is CLOSED (dumper + checker now cover both classes). NO apex write -- the upstream -write-back stays DEFERRED to end-of-deployment. This closed the last piece of close-out C2. See -`docs/changelog-20260715-d120-load-and-iprange-tooling.md`. - - -**D-126 RULED + tooling delivered (2026-07-16) -- durable rootless vcloud->service-VM access.** -Operator ruled Option A (SSH local-forward via `systemd --user`, "continue [the] ssh proxyjump [we -already use]") over persisting the L3 route (root + races libvirt-managed virbr2). Reachability -RE-established this session by the documented ad-hoc route (operator ran `sudo ip route replace -10.10.1.0/24 via 10.10.0.20 dev virbr2` in a real terminal; verified HTTP 302 at 10.10.1.10:8000). -Delivered `scripts/site-forward.sh` + `tests/site-forward/run-tests.sh` (38/38), D-126 record, -as-built access-table update. **SHELL layer added same day** (operator: "you're going to need shell in -Office and the DCs") -- `scripts/site-ssh-config.sh` + `tests/site-ssh-config/run-tests.sh` (30/30) -generate a `~/.ssh/config.d/vr1-sites` ProxyJump alias set (`ssh office1-netbox` etc.), the FOUNDATION -the forward rides; DC aliases deferred until measured post-apply. **SHELL LAYER EXERCISED 2026-07-16**: -operator installed with `~/vr1-office1-creds/office1_svc_ed25519` (one-key closed env, per -`creds-manifests/vr1-office1.manifest`); agent proved `ssh voffice1`/`office1-netbox`/`office1-tailscale`/ -`office1-opnsense` all return live from this session -- so the agent now HAS durable shell into Office1 -(load-bearing assumption confirmed: key read from config `IdentityFile`, no creds path in command, passes -the guard/classifier). The systemd FORWARD layer stays reviewed-but-unexercised (not installed). -**NetBox apex recovery + edge-WAN import (2026-07-17).** Found the IPAM apex DOWN (netbox-docker cleanly -exited ~3h prior, no auto-restart). Brought it up (`docker compose up -d`, HTTP 302) + added -`restart: unless-stopped` to all 5 services (survives the bookend reboot; override backed up on host). -Both blocks COMMITTED (gated, dry-run-verified; token server-side, sandbox-only). Block A: edge /24s -172.30.2.0/24 (vr1-dc0) + 172.30.3.0/24 (vr1-dc1). Block B: operator chose "write the seeder" -> NEW -`netbox/d124-transit-seed.py` + `tests/d124-transit-seed/` (21/21, incl. a cross-file consistency check vs -dc-rack-mgmt) seeds role `transit` + 172.31.0.0/24 unscoped container; then `dc-rack-mgmt-import` placed -172.31.0.0/30 (transit, vr1-dc0) + 10.12.8.2/22 (rack, dns=vvr1-dc0) -- matches the D-124 tfvars. The full -D-124/edge-WAN IPAM is now in the apex. See -`docs/changelog-20260717-netbox-apex-recovery-edge-wan-import.md`. - -**GUEST UPDATES + D-127 IaC AUTOSTART done (2026-07-17).** Office1 guests patched via agent sudo INSIDE -them: office1-netbox apt upgrade (no reboot, netbox healthy); office1-tailscale full-upgrade -> kernel -6.8.0-136 + reboot (back, re-advertising 10.10.0.0/22); voffice1 0-upgradable. LXD boot.autostart=true set -on both. **D-127 IaC layer DONE** (operator: bake the matrix so DC0/DC1 not missed): `autostart` var added -to cloudinit-vm/opnsense-edge/node-vm, set per matrix (voffice1/office1-opnsense/DC-edge=true; -vvr1-dc0/nodes=FALSE); opentofu T11-T15 pin it; HARD GATE verified (voffice1/office1-opnsense = autostart -in-place, NOT replacement). Outer plan now 5 add / 2 in-place / 6 destroy (collapses to 0-change once the -bookend sets those two autostart live pre-apply). Standup runbook records the LXD boot.autostart step. See -`docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md`. STILL bookend: vcloud + opnsense patch/reboot. - -**vcloud OS-UPDATE + REBOOT window (2026-07-16) -- IN FLIGHT; D-127 autostart policy recorded.** -Operator patching the vcloud host during the open restart window (only Office1 VMs live, before the DC0 -build). D-127 RULED: voffice1 autostarts; office1-opnsense autostarts AFTER one verified post-fix cold boot -(DOCFIX-188 mem-unit fix); vvr1-dc0/dc1 containment stay MANUAL. **A host reboot ENDS the running agent -session** -- so this is recorded before it. **POST-REBOOT TODO (next session):** (1) `virsh start voffice1` -+ `virsh start office1-opnsense`, WATCH the edge cold-boot clean (live-boot verification); (2) verify `ssh -voffice1` / `ssh office1-netbox` answer (D-126 ProxyJump; the direct L3 route to 10.10.1.10 is already gone --- use SSH); (3) THEN `virsh autostart voffice1` + `virsh autostart office1-opnsense` (operator sequencing: -autostart only AFTER clean-startup confirmed); (4) optionally set `boot.autostart` on the LXD service VMs -inside voffice1. Then resume DC0 readiness (`docs/dc0-deploy-readiness.md`). - -**UPDATE 2026-07-17 (session-scoped health sweep + edge cold boot) -- the "bookend" is PARTIALLY done, -NOT the way the TODO above assumed.** Measured this session: **the vcloud HOST never rebooted** (uptime -8d 18h, boot 2026-07-08) -- so the host-reboot that "ENDS the running agent session" has not happened. -What DID reboot was the guests: **voffice1** (07-16 20:56, kernel 6.8.0-136) and **office1-tailscale** -(07-17 07:25, kernel 6.8.0-136), both healthy. The **office1-opnsense edge was cold-booted THIS session** -(graceful ACPI `virsh shutdown` -> confirmed `shut off` -> `virsh start`) and VERIFIED clean: up-from-0, -physmem 2.0 GiB (DOCFIX-188 fix live, not 2 MiB), pf Enabled/80 rules, `kea-dhcp4` up, both NICs up, LAN -gw + egress 0% loss, netbox 302 + tailscale reachable behind it. That satisfies the D-127 verified-cold-boot -gate, so **`virsh autostart office1-opnsense` is now ENABLED**. **`voffice1` autostart ALSO ENABLED this -session** -- so BOTH foundational VMs now autostart per D-127 (POST-REBOOT TODO item 3 DONE; `vvr1-dc0/dc1` -stay MANUAL and do not exist yet). **LXD service-VM autostart VERIFIED (2026-07-17):** the two LXD VMs -inside voffice1 -- `office1-netbox` (10.10.1.10) + `office1-tailscale` (10.10.1.11) -- both already carry -`boot.autostart=true` (MEASURED via `sudo lxc config get`; confirms the D-127 guest-update note, no change -made). So the autostart chain is now proven END TO END: vcloud->`voffice1`/`office1-opnsense` via `virsh -autostart`; `voffice1`->netbox/tailscale via LXD `boot.autostart` -- a vcloud host reboot self-recovers the -whole Office1 stack with no manual `virsh start`/`lxc start`. **STILL PENDING:** (a) the vcloud HOST patch+reboot -- kernel **6.8.0-136 -is pending in apt but NOT installed** (running -134, newest installed -134, no reboot-required flag), so a -bare host reboot gains nothing: the real step is `sudo apt update && sudo apt full-upgrade` (lands -136 + ~9 -security pkgs) THEN reboot; (b) office1-opnsense itself is still on 14.3-RELEASE-p16 (guest OS not patched, -only cold-booted). - -**BOOKEND ITEM (a) CLOSED 2026-07-18 -- MEASURED, and the autostart chain PROVED ITSELF FOR REAL.** -The vcloud host was patched and rebooted between sessions: `uname -r` = **6.8.0-136-generic**, `apt list ---upgradable` EMPTY, no `/var/run/reboot-required`, and `last -x reboot` shows TWO boots on 2026-07-17 -(17:59 and 23:38; current uptime 5h39m from 23:38, ending the -134 run of 8d20h). Item (a) is DONE. Item -**(b) STAYS OPEN** -- a host reboot does not patch the edge guest; `office1-opnsense` is still on -14.3-RELEASE-p16. **The D-127 autostart chain is now PROVEN BY A REAL HOST REBOOT, not merely configured:** -after an unattended boot, `virsh list --all` shows `voffice1` + `office1-opnsense` both running with -autostart, and `ssh voffice1` / `office1-netbox` / `office1-tailscale` all answer live (all up ~5h38m, -matching the host boot). `site-baseleg.sh check office1` = **ok leg + ok route** (the D-126 oneshot re-added -the isolated-net leg through the reboot). NetBox apex HTTP **302** at 10.10.1.10:8000; edge LAN gw -10.10.0.1 ping **0% loss**. Zero manual `virsh start` / `lxc start` was needed. Office1 is fully self-recovered. - -**STAGE-3 WORKFLOW-DOC BLOCKER IS STALE -- reconciled 2026-07-18 (the ledger was right, the doc is wrong).** -`docs/dc-dc-deployment-workflow.md`'s Stage 3 section still reads "**Blocking to EXECUTE:** the HELD NetBox -addressing -- D-124's rack transit/30 + rack metal-admin IP". MEASURED against the live apex -(read-only API GET, token sourced in-shell and never printed): all three objects are PRESENT -- -`172.31.0.0/30` (role Transit, scope **VR1 DC0**), the `172.31.0.0/24` unscoped Transit container, and -`10.12.8.2/22`. This matches the 2026-07-17 apex-recovery entry above. The NetBox addressing is NOT a -blocker; the workflow doc's Stage-3 "Blocking to EXECUTE" line needs a DOCFIX (next-free 196). - -**!! DC0 OUTER APPLY IS BLOCKED ON NEW DRIFT -- do NOT apply. See -`docs/finding-20260718-voffice1-cloudinit-seed-replace.md`.** Post-reboot re-verification of the outer -root (read-only plan, **OpenTofu v1.12.4**) returns **7 add / 2 change / 7 destroy**, NOT the committed -**5 add / 2 in-place / 6 destroy** in `docs/dc0-deploy-readiness.md`. The entire delta is `voffice1`, the -live headend -- NOT DC0 scope: the libvirt provider staged its cloudinit ISO under -`/tmp/terraform-provider-libvirt-cloudinit/`, **the reboot cleared `/tmp`**, so `create.content.url` reads -changed and **forces replacement of `voffice1`'s seed volume -- which is ATTACHED to the RUNNING domain as -`sda`** (`virsh domblklist voffice1`; the real ISO is intact at -`/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso`). Replacing it risks bouncing the Office1 headend = -NetBox apex + tailscale + the D-128 Plane-2 execution origin + the D-126 ProxyJump path. This is provider- -state drift, not a config change (the volume-replace CONSEQUENCE on a running domain is **UNVERIFIED** -- -do not assert it bounces the guest; trap 1e is about domains, not volumes). -**THE DOMINANT RISK IS THE 2 "in-place" CHANGES, NOT THE VOLUME.** They did NOT collapse to 0 as the D-127 -changelog predicted -- both `voffice1` AND `office1-opnsense` plan as `+ autostart = true`, because `virsh -autostart` was set out-of-band and the provider does not refresh it into state. **The bounce is CONFIRMED -FROM PRIMARY SOURCE** (v0.9.8 `internal/provider/domain_resource.go`, fetched + read directly): `Update`'s -only no-bounce path (L1090) requires `domainConfigUnchanged && runningChanged && autostartUnchanged`, and an -autostart-only diff has `runningChanged`=FALSE, so it falls through to **`stopDomainIfRunning`** -> -undefine -> DefineXML -> SetAutostart. ALSO CONFIRMED (L938): Read only reads `autostart` back -`if isImport || state.Autostart` is non-null -- ours is NULL, so **`-refresh-only` can never populate it** -(matches the empirical `grep -c autostart` = 0 on both domains). NOTE: an earlier draft cited -`platform-traps.md` 1e as the warrant here; 1e was measured on a *memory* change and does NOT transfer to -an autostart-only diff -- the conclusion held but the reasoning was wrong, now replaced by the source above. -1e's operational rule still stands independently: `0 to change` on the domains is the ONLY assurance a guest -will not bounce. So the unscoped apply -would take Office1's headend AND edge down -- to write a value that is ALREADY LIVE on both. An earlier -draft of this finding called these changes "harmless"; that was WRONG and is corrected here and in the -finding doc. Pinned provider = dmacvicar/libvirt **0.9.8**. -Options (a)-(d) are in the finding. **DC0 IS NOT BLOCKED on the unresolved provider question:** the intended -DC0 work (create `vvr1-dc0` + uplink, destroy 6 empty planes) touches NO running guest, so **(a) `-target` -the DC0 addresses** sidesteps both risks and is recommended for the immediate apply, with **(b) reconcile -state first** (`-refresh-only` / re-stage) as the deliberate follow-up to get back to `0 to change`. -**(d) standing hardening:** keep the provider's cloudinit staging OFF `/tmp` so a reboot can never again -force-replace a live VM's seed. **BLAST RADIUS CORRECTED (MEASURED 2026-07-18):** an earlier draft of this -entry claimed this hits "all 9 DC0 node VMs and every DC containment VM" -- **WRONG, retracted.** -`libvirt_cloudinit_disk` appears in exactly ONE module (`modules/cloudinit-vm`), instantiated exactly TWICE -(`main.tf:181` voffice1, `main.tf:361` vvr1_dc0). The 9 nodes use `modules/node-vm` = blank-disk PXE, NO -cloud-init; the edges use REST config (D-113(a2)). Real exposure = **voffice1 + vvr1-dc0 (+vvr1-dc1) = 3**. -Also: on a CLEAN deploy the disk+volume are created in the same apply so there is NO diff -- this only -surfaces on a plan taken after a host reboot, so it does not break a post-teardown redeploy, only ops. -**Mechanism correction:** there is **NO provider knob** for this (VERIFIED via `tofu providers schema --json` -- provider block exposes `uri` ONLY; `libvirt_cloudinit_disk` has no staging attr); the lever is -almost certainly **`$TMPDIR`** for the tofu process (Go `os.TempDir()`), UNVERIFIED by observation -- -confirm before writing it up. Warrants **D-130** (next-free confirmed 2026-07-18). -**Before ANY apply:** re-run the readiness doc's C1 pre-apply checks THIS session, incl. re-confirming the -6 `vr1-dc0` planes are still empty (0 leases / 0 attached) -- that evidence is from a prior session, and -this finding is itself proof that committed pre-apply evidence goes stale across a reboot. - -**Version drift to log:** OpenTofu on the jumphost is now **v1.12.4**; `README.md:41` and the readiness -evidence say **v1.12.3**. Presumed to have arrived with the host patch (not verified). All committed -plan/validate evidence predates it. - -**SESSION CLOSE 2026-07-18 (bookend). Full record: `docs/finding-20260718-voffice1-cloudinit-seed-replace.md` (committed 43b29d7).** -ESTABLISHED: (1) An autostart-ONLY diff **bounces the guest** -- VERIFIED FROM PRIMARY SOURCE (libvirt -v0.9.8 `internal/provider/domain_resource.go` L1090/L1107): the sole no-bounce path needs -`domainConfigUnchanged && runningChanged && autostartUnchanged`, and runningChanged=FALSE here, so it falls -through to `stopDomainIfRunning` -> undefine -> redefine. (2) `-refresh-only` can NEVER fix it -- Read is -self-gated at L938 (`isImport || state.Autostart` non-null); ours is NULL. (3) The seed force-replace is -STRUCTURAL: the reboot cleared `/tmp`, where the provider stages cloudinit ISOs, so the url changes every -plan. (4) Blast radius CORRECTED to **3 VMs** (voffice1, vvr1-dc0, +vvr1-dc1) -- NOT the 9 nodes -(`node-vm` = blank-disk PXE, no cloud-init); an earlier claim of mine, retracted. -OPEN: **D-130 mechanism ruling** (`ignore_changes` vs `$TMPDIR` -- operator's call, unruled); **verification -7** (what 0.9.8 does replacing a volume attached to a RUNNING domain -- deliberately unasserted); -**verification 8** (that the chosen D-130 mechanism actually suppresses a force-replacement diff); -**doc-integrity list** (D-127 changelog's false "collapse to 0" + "no forces-replacement", readiness-doc -5/0/6-vs-5/2/6, DOCFIX-196 stale Stage-3 NetBox blocker, README tofu 1.12.3->1.12.4, phase-2 runbook). -Gate before any apply: plan must read EXACTLY 5 add / 0 change / 6 destroy. Nothing applied this session. -**SUPERSEDED: this session is closed out by a grounding audit charter arriving via git -- read that first.** - -**DC0 DEPLOY READINESS -- PREP COMPLETE 2026-07-17 (deploy-ready). See `docs/dc0-deploy-readiness.md`.** -Final gate re-checked clean (validate, tfvars, creds-audit vr1-dc0 CLEAN, gauntlet 65, lint 0-fail). -Remaining is NOT prep: the bookend restart (vcloud + voffice1 + opnsense patch/reboot together -- voffice1 -is load-bearing like vcloud) + fresh-session (re-verify plan post-reboot + logged shell + gated outer -apply). Decoupled hygiene (NetBox import; Stage-3 record sweep) is OPTIONAL, does not gate the deploy. -Stage-3 Phase-2 DC0 is EXECUTABLE (D-121..125 ADOPTED, R-1/R-2/R-3 ruled, tfvars in, opentofu-validate -clean). Outer-root `tofu plan` RUN + VERIFIED SAFE: 5 add (`vvr1-dc0` 416GiB/108vcpu + D-125 uplink) / -0 change / 6 destroy (the six EMPTY vr1-dc0 planes relocating to the inner root -- verified 0 leases/0 -attached). RAM gate: 980 GiB avail >= 416. **Nothing applied.** Next = the gated outer `tofu apply`. -Pre-apply per-env-key DECISION now RULED + IMPLEMENTED (operator 2026-07-16, option a): `vvr1-dc0` -gets its OWN key -- minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (creds-audit CLEAN), added -`vr1_dc0_ssh_pubkey_path` var, `vvr1-dc0` cloud-init reads it (voffice1 keeps office1's key), tfvar set; -re-plan UNCHANGED (5/0/6). netem RULED = placeholder for the rehearsal. Full remaining-prep checklist + -fresh-session bootstrap + gated A-E sequence in `docs/dc0-deploy-readiness.md` (the entry point). Inner -apply must use the dc0 private key (wire post-outer-apply). Gauntlet ALL GREEN (65). Nothing applied. - -Original build note: this session had NO ssh key to voffice1 -(none in `~/.ssh`, no agent; key is operator-held in `~/vr1-office1-creds/`, correctly un-enumerable), -so the operator supplies `--key` and runs `install` + first-connect + the one-time -`loginctl enable-linger`. Site table seeded office1-netbox only; DC rows DEFERRED (IPs not measured). -Advisor-conferred (untangle two problems): this does NOT gate `tofu plan` -- the tree has no netbox -provider/data source, so the four operator-ruled `vr1_dc0_rack_*` tfvars alone unblock plan; the -NetBox `--commit` stays a decoupled, operator-run, token-on-netbox-host step. See -`docs/changelog-20260716-d126-durable-forward.md`. - - -**STAGE-3 ADVERSARIAL REVIEW (2026-07-16; advisor + 4-charter debate + Layer-1 gate).** -Full pre-deployment adversarial review of the `a48a60f` batch (D-121..D-124 + substrate/rack/HA -overlay). READ-ONLY; nothing applied/committed/pushed. Deliverable: `docs/stage3-adversarial-review- -20260716.md` (present-only, NOT committed); immutable base patch `docs/stage3-review-base.patch`; -surface frozen at WIP commit `87a7a8a` (local-only). **VERDICT: NO-GO for `tofu apply`** on -hard-gate grounds (substrate design is sound). BLOCKING headline: **2 decisions rest on -inferred/agent-authored rulings** -- D-121 vault-`(v-a)` (no operator utterance; Status `:3264` -contradicts body `:3346`; ENCODED in `overlays/dc-ha-scaleup.yaml:67-80` reversing BUNDLEFIX-002) -AND D-123 Model A ("Do it all now" is not an A/B answer; runbook `:507` already marks D-123 -PROPOSED). Both demote to PROPOSED; region+rack/Option C/D-124 stand. Pre-apply gate list (all -operator-gated, for a post-acceptance sweep): (1) re-rule vault + Model A; (2) ship the SEC-010 -`ip_forward=0`+firewall ARTIFACT + a mechanical gate (currently a ledger promise); (3) assign the -rack transit/30 + rack IP in office1-netbox, feed the 4 tfvars (tofu plan BLOCKED without them), -wire voffice1's transit leg. Record amendments queued: D-122 MAAS-bullet + site-down wording (C1/C3), -D-107 "node artifacts" scoping + DR-claim honesty (C4/C5, permanent-at-Roosevelt), D-121 Option-C -disk record + commit `optc-calc.py` (UNCOMMITTED/absent -- "reproducible" claim is false), D-115 -status-line vs ledger-scan. CONTESTED (surviving dissent, recorded): SEC-011 downgraded to -OBSERVATION -- A2 showed pruning storage NICs INCREASES Roosevelt delta (baremetal trunks all VLANs) -and isolated-L2 planes have nil reachability; reconsider/amend the SEC-011 row. Gauntlet ALL GREEN -(62), repo-lint 0-fail, batch ASCII/LF-clean. See the findings doc for the full R3-F01..F13 register, -minority report, and L1..L5 leads. - -**OPERATOR RULINGS on the review (step-by-step, 2026-07-16; NOT yet swept into design-decisions.md --- these drive the gated fix-sweep):** -- **[R-1] D-123 node containment = MODEL B (operator ruling 2026-07-16, explicit AskUserQuestion - "Proceed with B, accept all").** Nodes nest INSIDE `vvr1-dc0`; site-down = single - `virsh destroy vvr1-dc0` (restores the operator's original D-122 intent; the inferred Model A is - overturned). **Accepted cascade (operator-confirmed after being shown the consequences):** - (a) supersede ADOPTED **D-103** (OpenTofu-creates-nodes-at-vcloud + MAAS-discovers seam) and - **D-114** (containment scoped to headend + non-stack LXD machines only, NOT the OpenStack nodes); - (b) **D-124 rack sizing (4/8192/80) VOID** -- `vvr1-dc0` must contain one DC's full node fleet - (~360 GiB node RAM + overhead) and set `expose_nested_virt = true`; (c) **depth-4 nested virt - accepted** as a rehearsal cost (vcloud->vvr1-dc0->compute->nova guest = 3x nested KVM; VR0 proved - 2; Phase-5 workload-boot risk flagged + accepted). Substrate reshape: `node-vm` + `maas-vm-host` + - 6 planes + `vr1-dc0-wan` retarget from vcloud libvirt to `vvr1-dc0`'s inner libvirt. STAGED for - the sweep; nothing edited/applied yet. -- **[R-2] D-121 vault-HA backend = RAFT integrated storage (operator ruling 2026-07-16, explicit - AskUserQuestion "Raft integrated storage").** Overturns the agent-authored v-a (MySQL-backed). - Consequences: DROP the `vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster` storage - relation; vault uses its own integrated raft (per-unit persistent storage needed); keep the - `vault-hacluster` VIP. Still 3x manual unseal (SEC-003). **HARD FEASIBILITY GATE -- UNVERIFIED, - not assumed (cardinal-sin discipline):** must confirm the Charmed `vault` charm (1.8/stable) - exposes an integrated-raft storage backend before this ruling is actionable. If the charm does not - support raft, the ruling is moot and returns to the operator (v-a MySQL vs another path). STAGED; - nothing edited/applied. - - **[R-2 FEASIBILITY GATE FAILED -- verified 2026-07-16.]** `bundle.yaml:133-138` (D-068 / - BUNDLEFIX-010): integrated-**Raft storage is a 1.16-charm property**, and the 1.16 vault charm is - RULED INCOMPATIBLE with this reactive Caracal cloud (certs V0->V1, no upgrade path, **broken with - Charmed Ceph**). The deployable charm is **1.8/stable** = MySQL/etcd storage, **no Raft**. So the - operator's Raft ruling is **INFEASIBLE for VR1** without the ruled-out 1.16 charm. Raft is the - correct *Roosevelt/D-068* target (get vault off 1.8.8-EOL), NOT a VR1 choice. **R-2 RETURNED to - the operator:** realistic VR1 options = v-a (MySQL, rec) or v-b (etcd, deprecated). - - **[R-2 FINAL = v-a MySQL-backed (operator ruling 2026-07-16, "v-a: MySQL-backed").]** Vault stays - on the 3-unit mysql-innodb-cluster (via vault-mysql-router), scale to 3, re-add - `vault:ha <-> vault-hacluster`. **NO CODE DELTA** -- the committed `overlays/dc-ha-scaleup.yaml` - already encodes v-a; only the D-121 RECORD needs fixing (attribute to the operator's 2026-07-16 - ruling; reconcile the body's stale "Operator sub-ruling needed"). Charm-verify gate stands (confirm - vault 1.8/stable does HA leader-election on MySQL at deploy). Raft parked as the D-068/Roosevelt - target. 3x manual unseal (SEC-003) stands. -- **[R-3] Option C storage resilience = 4 STORAGE NODES/DC (operator ruling 2026-07-16, "Add a 4th - storage node/DC").** Node layout now **3 control + 2 compute + 4 storage = 9/DC** (18 total, was 16); - `ceph-osd` = 4 units/DC. Restores Ceph rebuild headroom (lose 1 OSD host -> 3 distinct replicas still - placeable -> self-heals). Consequences: (a) whole-host RAM ~838 GiB/1024 (81%, hand-estimate -- FITS - but tighter; the ABSENT `optc-calc.py` must be rebuilt + re-run for the 3+2+4 shape AND Model B - nesting overhead -- R3-F06 now doubly required); (b) under Model B, `vvr1-dc0` holds ~384 GiB node - RAM + overhead (~404 GiB VM); (c) Ceph disk re-checked for 4x500Gi/DC = PASS, 5.31 TiB margin; - (d) `main.tf local.vr1_dc0_nodes` adds `storage-04`, D-121 layout table + sizing updated. -- **[R-4] SEC-011 CLOSED (operator ruling 2026-07-16, "Keep uniform 6-NIC").** A2's Roosevelt-delta - refutation accepted: uniform 6-NIC matches baremetal VLAN-trunking (all planes trunked to every - node); isolated-L2 planes = nil reachability in the sim. **NO CODE CHANGE** (keep uniform - `local.vr1_dc0_node_nics`); amend the SEC-011 ledger row to CLOSED / accepted-non-issue. - -**SWEEP PROGRESS (2026-07-16):** -- **Phase A DONE** -- record amendments (`design-decisions.md` D-103/D-107/D-114/D-115/D-121/D-122/D-123/ - D-124 + `security-ledger.md` SEC-011 close). Decision record now internally coherent; D-115 - ledger-scan false-positive fixed. `docs/changelog-20260716-review-sweep-phaseAB.md`. -- **Phase B DONE** -- `scripts/dc-dc-whole-host-budget.py` + `tests/dc-dc-whole-host-budget/` (13/13) - replace the lost `optc-calc.py` (R3-F06). Re-validation: Model B 3+2+4 = 838 GiB/1024 FIT (186 - headroom). Gauntlet 63 ALL GREEN. -- **Phase C IN PROGRESS** -- Model B OpenTofu reshape. Design cross-checked by a 3-charter agent - workflow (compatibility/best-practice/structure): mechanism = **TWO-ROOT / TWO-STATE split** (forced - by the provider-cannot-depend-on-same-apply-resource invariant; -target + provider-with-unknown-values - are anti-patterns). OUTER root (main.tf, vcloud) = vvr1-dc0 resized + transit + office1; INNER root - (NEW `opentofu/vr1-dc0-substrate/`, own state) = 6 planes + wan + edge + inner pool + 9 nodes, modules - reused VERBATIM. BOOTSTRAP GATE between (extend site-headend-install.sh: nested libvirtd/pool/nested=1/ - base image). Verdict GO-WITH-MITIGATION on depth-4. **[R-5] inner transport = REMOTE qemu+ssh from - Office1 (operator ruling 2026-07-16)** -- tofu on Office1, dials vvr1-dc0 over the D-124 transit; both - states on Office1. **vvr1-dc0 sizing DERIVED (not invented) = 108 vCPU / 416 GiB / ~3000 GiB** - (104 node vCPU + 4 oh; 384 node GiB + 32 oh; 2850 inner pool + OS), whole-host 870/1024 (85%) FIT. - DONE so far: storage-04 (R-3), fallback re-cut R-3-compliant (114d392); **C1 OUTER root** (main.tf: - vvr1_dc0 resized via var.vvr1_dc0_* [108/425984/3221225472000, derived], expose_nested_virt=true, - single transit leg; removed planes/wan/edge/node blocks + the dangling moved block; tofu validate - Success); **C2 INNER root** `opentofu/vr1-dc0-substrate/` (versions/variables/main.tf: qemu+ssh - provider + inner pool + 6 planes + wan + edge + 9 node-vm, modules reused verbatim; init+validate - Success). Both roots validate; repo-lint 0-fail. Cross-check agent running on the two-root impl. - **C3 DONE + REVIEWED** -- `site-headend-install.sh` `--host-nodes` (rack-only) node-host mode: installs - qemu-kvm+libvirt, persists+verifies kvm nested=1, adds ssh user to libvirt group, inner pool dir + - AppArmor grant, verifies/instructs the OPNsense base (delegated to opnsense-prep-image.sh). Harness - 32->45/45. Review-after-step (per operator discipline): gauntlet ALL GREEN (63), repo-lint 0-fail, - ASCII/CR clean, grounding OK (implements the C3 bootstrap the inner-root header names), drift sweep - clean (record/docs/code all reference "node-host mode" consistently). `docs/changelog-20260716-review- - sweep-phaseC.md`. - **PHASE D + REMAINDER DONE + REVIEWED (2026-07-16):** - - **SEC-010 (Phase D)** -- artifact COMMITTED in the node-host bootstrap: a FORWARD-drop across the - transit leg (NOT a global ip_forward=0 -- the inner vr1-dc0-wan NAT forces ip_forward=1), written to - /etc/nftables-sec010.nft + boot-persistent sec010-fw.service; `--host-nodes --check` is the mechanical - pre-apply gate. Harness 45->48/48. SEC-010 ledger row reconciled to the code. Stays OPEN until - applied+verified on vvr1-dc0. - - **maas-vm-host** -- header updated for Model B (register vvr1-dc0's INNER/local virsh to the Office1 - region; no cross-fiber dial). Still DEFERRED (DOCFIX-179; not instantiated). - - **C5 runbook reorg** -- `dc-dc-phase2-tofu-dc-substrate.md` gains the "D-123 MODEL B RESHAPE" section: - outer apply -> bootstrap gate (site-headend --host-nodes) -> inner apply -> maas-register; Step 6's - "8 nodes" superseded by R-3's 9. - - **model-a-fallback-plan.md** synced (~416 GiB; inner-root + node-host mode added to the B->A delta + - revert procedure). - Review-after-each-step applied throughout (harness/validate/lint/ASCII/drift per step). All Phase C+D - items COMPLETE. NOTHING applied/committed since 114d392; Phase C/D edits present-only in the working - tree. REMAINING (operator-gated, deploy-time): the actual outer/bootstrap/inner APPLIES + rack-IP - assignment (Phase E, L1-07) + the depth-4 nested-virt L4 boot gate. - - **Model B cross-check (agent) = FAITHFUL, no BLOCKING**; fixes applied: 32-GiB-overhead RATIFIED in - the record (870/154, vvr1-dc0 416 GiB), false metal-admin-IP comment corrected, SEC-010 updated for the - single-leg-outer/bridges-6-inner-planes reality. **OBS-3: design gap RESOLVED by D-125; egress efficacy - is a deploy-time gate (see the C+D-commit entry below).** Transit IP is duplicated across roots - (`vr1_dc0_rack_transit_ip` outer / `vvr1_dc0_transit_ip` inner) -- keep equal in tfvars. - -- **Stage-3 sweep C+D COMMITTED as `5175365`** (operator "commit but don't push"; NOT pushed). The - adversarial re-run then confirmed **OBS-3 material** once the operator ruled **"DC0 and DC1 have their - own ISP connections"**, resolved by **D-125 (bridge-in, single-NAT)**: - - Model B nesting had severed per-DC ISP egress: `modules/site-wan` makes `vr1-dc0-wan` a libvirt NAT - that egresses via ITS HOST's default route; under Model B that host is `vvr1-dc0` (transit-only, - East-West, SEC-010 FORWARD-dropped) -> dead. Fix: vcloud ISP NAT `vr1-dc0-uplink` + an IP-less uplink - NIC + netplan bridge `br-vr1-dc0-wan` on `vvr1-dc0`; inner `vr1-dc0-wan` becomes a BRIDGE (new - `modules/wan-bridge`) so OPNsense WAN rides to the single vcloud NAT. **Authored present-only; both - roots validate; NOT closed -- the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in - equivalent of the L4 boot gate), fallback = double-NAT.** See D-125 + platform-traps 1g. - - **No new HELD gate (CORRECTED 2026-07-16):** the vcloud ISP /24 is the SAME `172.30.2.0/24` D-115 - already ruled for vr1-dc0 (bridge-in is single-NAT; IPAM identity unchanged from Model A), inlined as - a ruled literal in `module "vr1_dc0_uplink"` -- NOT a NetBox-assigned tfvar. OPNsense WAN stays `.2`; - no re-address. D-125 adds NO NetBox object. (An earlier draft's `vr1_dc0_uplink_cidr` HELD var + - re-address were removed -- see the correction commit below.) - - **br_netfilter constraint:** SEC-010 FORWARD-drop stays interface-scoped, NEVER globalized. `--check` - hardened: fail-open guard now covers the transit-iface existence AND the WAN bridge/uplink enslavement. - - Harness: `site-headend-install` 49->54; `wan-bridge` covered by opentofu-validate S3. Records: D-125, - SEC-010, model-a-fallback B->A delta (item 8 + revert), platform-traps 1g. - - **D-125 COMMITTED `d34ec94` + PUSHED** (operator "commit", then "push"). Branch - `dc-dc-stage3-phase2-dc-substrate` now carries `114d392` (A+B) + `5175365` (C+D) + `d34ec94` (D-125) - on origin. - -- **Doc drift-reconciliation batch (post-Model-B/D-125), all doc-only, COMMITTED + PUSHED:** - - `8496c2e` -- `dc-dc-deployment-workflow.md` Stage-3 section reconciled to D-121..125 + Model B - two-root (Build/Gate/Owns/Authoring-status/rulings-summary); gap-register D-113(a2) OPEN->DONE. - - `87f9d75` -- `opentofu/README.md`: authoritative STATUS-RECONCILED banner (two roots, added - `site-wan`/`wan-bridge`, instantiation reality) + neutralized the DELETED, dangerous OPNsense - config.xml/config-ISO path (D-112/D-113(a2)). - - `788911c` -- `runbooks/dc-dc-teardown-rollback.md`: Model B two-root teardown banner (site-down = - one `virsh destroy vvr1-dc0`; inner-then-outer order; maas-vm-host targets inner virsh). - - Repo-wide drift scan confirmed the deleted-file / old-node-count references elsewhere are all - immutable CHANGELOGS, decision-record supersession context, or deliberate tombstones - (`opentofu/templates/README.md`) -- NOT live drift. Deeper prose in the three reconciled docs + - the Stage-4/5 sections still lag Model B (flagged in-banner as historical), not yet rewritten. - -- **D-125 addressing CORRECTED (`e270325`) + tightened (`922dd29`)**, then **D-124 addressing PINNED** - (operator-ratified 2026-07-16): the vcloud ISP /24 is the ruled `172.30.2.0/24` (not a HELD tfvar); - the D-124 transit supernet is CARVED as **`172.31.0.0/24`** (dedicated `transit` role, scope vr1-dc0), - transit /30 `172.31.0.0/30`, rack IP `10.12.8.2`. `dc-rack-mgmt-import.py` `CONTAINER` pinned to - `172.31.0.0/24`; harness rebased, 96/96. **NetBox `--commit` NOT DONE -- office1-netbox (10.10.1.10) is - UNREACHABLE from the vcloud jumphost (measured) + the apex token is a secret on that host; the write is - an on-host operator step** (edge-WAN import, then rack-mgmt import, then fill the 4 `vr1_dc0_rack_*` - tfvars -> `tofu plan`). Exact commands: `docs/changelog-20260716-d124-addressing-pin.md`. NO apex mutation. -- **Phase D NOT STARTED** -- SEC-010 rack `ip_forward=0` + firewall artifact + mechanical pre-apply gate. -- **Phase E NOT STARTED** -- operator assigns rack transit/30 + rack IP in office1-netbox (apex token), - fill tfvars -> `tofu plan` renders. -- NOTHING committed/pushed; all sweep artifacts present-only in the working tree. -- **[R-1 fallback preserved]** Model A (the not-chosen D-123 option) captured as a revert artifact per - operator request: `docs/model-a-fallback-plan.md` (architecture + the B->A delta + revert - procedure) + git tag **`model-a-fallback`** on `87a7a8a` (local-only). Revert = `git checkout - model-a-fallback -- opentofu/`. Model A already validates (no re-engineering needed). - - -**PLANE-SEGREGATION REVIEW of the Stage-3 commit (2026-07-16; advisor + 2 read-only agents).** -Reviewed the committed networking/plane changes (`a48a60f`, NOT applied) for crosstalk / segregation -breaches. **VERDICT: no crosstalk, no active violation -- committed state is SAFE.** All six `vr1-dc0-*` -planes + all three `mesh-*` legs are isolated L2 (measured: `net-dumpxml` shows no forward/ip/nat, -isolation ENFORCED BY CONSTRUCTION in `dc-planes`/`mesh-link`); the edge is 2-NIC so it structurally -cannot reach the 5 non-LAN planes; `vr1-dc0-wan` NAT touches only the edge WAN. metal-admin DC-LOCAL -(D-052/D-100) is **preserved but not ENFORCED** -- it rests on unpinned host defaults. **Two pre-apply -hardening items logged as SEC-010 (rack `ip_forward=0` + transit-leg firewall -- the one real -plane-bridge) and SEC-011 (node role-scoped NIC sets -- least-connectivity).** Say plainly at handoff: -"committed state is safe; two items to close before `tofu apply`." Plus three notes: -- **FUNCTIONAL (fix before apply, not security):** the rack's committed route `to 10.10.0.0/22 via - ` points at a next-hop with **no VM attached** -- `voffice1` is single-homed - (`network_names = [office1-local]` only, main.tf:184), NOT on `mesh-vr1-dc0-office1`. As committed, - apply yields a rack routing to a nonexistent peer, and the Office1<->DC region path is not closed. - Either wire `voffice1`'s transit leg or gate the rack route until it exists. -- **SEGREGATION CHECK (first boot):** the rack NIC->plane mapping (`enp1s0`->metal-admin, - `enp2s0`->transit) is assumed PCI-order==name; if swapped, the metal-admin /22 lands on the - Office1-facing wire. Verify via MAC (`virsh domiflist` x `network_names` order) before trusting - addressing. This is a segregation check, not cosmetic. -- **DEFERRED (track):** the edge WAN<->provider-public firewall is post-boot REST (opnsense-api), - not in the commit -- "enforced by absence" covers plane REACHABILITY only, not full edge hardening. -When `voffice1`'s transit leg is later wired it becomes an office1-local<->transit bridge and inherits -the SAME `ip_forward=0` pin requirement (a richer surface than the rack: MAAS region + LXD host). - - -**STAGE 3 IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`) -- two rulings landed 2026-07-15.** -Walked the phase2 runbook and surfaced its blocking decisions; the operator ruled both: -- **D-121 (ADOPTED IN PART).** VR1 makes the decorative single-unit control plane REAL: 14 services - scale 1->3 (the 12 hacluster-backed APIs + ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays - 1 per D-108). This is D-009's "mechanical 1->3" pulled forward from Roosevelt -- but it is a PLACEMENT - rewrite (every app's `to:` must spread across 3 distinct hosts), not a num_units tweak. **Node layout - = Option C** (3 control + 2 compute + 3 storage per DC, 16 node VMs), ruled after a whole-host - validation PROVED it fits: measured budget 256 vCPU / 1024 GiB / 10 TiB, Option C allocates - 222 / 790 / 5.4Ti (RAM binding at 77%; model at `scratchpad/optc-calc.py`). Ruled sizing: control - 16/64GiB/150GiB, compute 12/48/100, storage 8/24/550; keep control <=64 GiB (96 GiB -> 96% RAM). - **Vault-HA sub-ruling RESOLVED = (v-a)** (2026-07-15): keep MySQL-backed (on the already-3-unit - mysql-innodb-cluster), scale to 3, re-add `vault:ha`. GATE: verify the `1.8/stable` vault charm does - HA on the MySQL backend with 3 units; if not, evaluate Raft (NOT etcd) via D-068. 3-unit vault - triples the unseal burden (intersects OPEN SEC-003). -- **D-122 (ADOPTED).** VR1 site shape: each site nested in its own containment VM (site-down/recovery + - site-to-site drills); dark fiber DC-DC-Office is East-West ONLY; each site has its own dedicated L3 - ISP uplink (build `vr1-dc0-wan` on the office1-wan pattern); DC edge follows the Office1 pattern - (2048MiB/2vCPU/nano, REST-API config, LAN faces the site network); each site its own MAAS controller. - Subsumes the DOCFIX-185 D-100/D-107 amendment note. -- **Fix-forward DONE (2026-07-15):** the phase2 runbook is reconciled to post-D-119 naming - (`vr1-dc0`/`vr1-dc1`, bare `dcN` rejected) and its obsolete Steps 4-5 rewritten to the D-113(a2) - REST-API bootstrap (the module already dropped `config_iso_path`); the D-121/D-122 rulings folded in - so its Ruling-1/Ruling-2 STOP-gates clear. Honest achievable first-run scope stays **Steps 1-8** -- - Step 9 (MAAS registration) gates on the per-DC MAAS "rack controller", now RESOLVED IN PRINCIPLE by - **D-123** (PROPOSED, recommend Model A): it is just a `vvr1-dc0` site headend VM (the proven Office1 - pattern -- `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which already names - `vvr1-dc0` as a target). Once Model A is ruled it is prepare-able (opentofu block + Step-9 wiring), - which effectively closes the Stage-4 gap for Step 9. D-123 also reconciles the D-122 "one virsh - destroy = site-down" wording (literal for Office1; for a DC, site-down = destroy the `vr1-dc0-*` - domain group). See `docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md`. -- **EDGE NIC MODEL RULED (D-122 refinement, 2026-07-15): 2-NIC, baremetal-matched.** NIC count mirrors - the Roosevelt hardware -- **nodes = 6 NICs** (one per plane, measured on VR0 openstack0-3), **edge = 2** - (WAN + LAN). The six planes are **routed by the fabric** (OVN/OpenStack on the deliberately isolated-L2 - `dc-planes` segments), so the edge is NOT an inter-plane router -- only the external boundary. No - `opnsense-edge` module extension needed. Edge **WAN = `vr1-dc0-wan`** (`172.30.2.0/24`), edge - **LAN = provider-public** (`10.12.4.0/22`; edge is the external gateway per D-100). This SUPERSEDES the - earlier "build an office1-network-analog site LAN" flag -- no new site-internal segment is needed. -- **STAGE-3 PREP BATCH DONE 2026-07-15 (authoring + offline validation; NOTHING applied).** See - `docs/changelog-20260715-stage3-prereqs-prep.md`. Gauntlet ALL GREEN (61), `tofu validate` Success, - repo-lint 0-fail. - - **OpenTofu:** new `modules/site-wan` (NAT /24 uplink; schema confirmed via `tofu providers schema`, - validated) + `main.tf` Stage-3 section wired (NOT APPLIED): `vr1_dc0_wan` (172.30.2.0/24), the 2-NIC - edge (LAN=provider-public, WAN=uplink), 8 Option C node VMs via `for_each` (metal-admin PXE-first), - netem HELD (Step 11; mesh bridge = virbr7 measured), stale `vr1_dc1_planes` comment fixed. **HELD - pending D-123:** the `vvr1-dc0` headend + `maas-vm-host` (Step 9) blocks. - - **NetBox:** `netbox/dc-edge-wan-import.py` + harness (58/58) -- registers the two DC-edge /24s - (172.30.2/3.0/24, role edge, site-scoped), DRY-BY-DEFAULT with whole-plan preflight. READY to - `--commit` (command in the tool header; token operator-held; guard blocked the live read). - - **Bundle HA overlay:** `overlays/dc-ha-scaleup.yaml` (Stage-5 prep) -- 14 services ->3, one-per- - control-node placement (`{{VR1_DC0_CONTROL_0N}}` tokens, Stage-4-bind), + 3 correctness items the - agent caught: rabbitmq `min-cluster-size:3`, hacluster `cluster_count:3`, re-declare `vault-hacluster` - (BUNDLEFIX-002 had commented it out) + re-add `vault:ha`. - - **The failed opentofu agent** (transient API stall, wrote nothing) was covered in-session by the main - loop; the netbox + overlay agents completed clean. -- **D-123/D-124 RULED + `vvr1-dc0` RACK WIRED (2026-07-16).** D-123 Model A + MAAS region-on-Office1 + - rack-per-DC; D-124 Scheme A (office1<->dc0 mesh transit /30) + rack sizing 4/8192/80. `main.tf` now - wires **`module "vvr1_dc0"`** (cloudinit-vm, 4/8192/80, two legs: metal-admin + mesh-office1 transit, - static IPs from NEW tfvars `vr1_dc0_rack_*` -- NetBox-assigned, not invented; interface-naming trap - flagged) + a `mesh-link` `network_name` output. `tofu validate` Success; opentofu-validate 11/11. - **Step-9 `maas-vm-host` STAYS DEFERRED (DOCFIX-179):** adding the `provider "maas"` block would force - every plan (incl. the Steps 1-8 apply) to demand MAAS creds, and vr1-dc0's MAAS doesn't exist until - vvr1-dc0 is up -- wire it as its own step once the rack MAAS is reachable. The rack MAAS INSTALL needs - a rack-only mode of `site-headend-install.sh` (follow-up). -- **D-124 rack-mgmt importer BUILT (2026-07-16):** `netbox/dc-rack-mgmt-import.py` + harness (96/96), - parameterized (`--transit-cidr`/`--rack-ip`, no invented literal), whole-plan preflight. Gauntlet 62. - **Two operator preconditions it flags before `--commit`:** (a) SEED a dedicated `transit` role in - office1-netbox (none exists; `infra` is shared, not reused) -- die-if-absent, like `edge`; (b) DECIDE - the transit supernet -- the importer assumes it nests under `10.12.0.0/16` (Cloud), but D-124 mirrors - the Edge role (which got its OWN `172.30.0.0/16`), so a dedicated mgmt/transit supernet may be wanted - (then a one-line `CONTAINER` const update). Transit prefix scope = site vr1-dc0 (region may be preferred). -- **RESIDUAL for a smooth batch deploy:** (1) NetBox: seed the `transit` role + decide the transit - supernet, then run the D-124 importer (+ the D-115 DC-edge importer) `--commit` with the assigned CIDR/ - IP, then fill the `vr1_dc0_rack_*` tfvars; (2) runtime: `opnsense-prep-image.sh` for the edge nano; - (3) Step-9 maas-vm-host wiring once the rack MAAS is up (**the rack-only `site-headend-install.sh --role - rack` mode is now DONE, 32/32** -- it enrolls a DC rack to Office1's region; `maas init rack` confirmed); - (4) Stage-4: render the HA overlay's control-node machine IDs + `--overlay --dry-run`; (5) D-121 - verifies (vault MySQL-HA at 3; hacluster cluster_count). All operator-gated / Stage-4. -- **`main.tf` stale comment (flagged):** the commented `vr1_dc1_planes` block (~L38-50) still says - "wait for NetBox to assign D-101's supernet" -- STALE per D-115 (assigned `10.12.64.0/19`). Doc-only - pass did not touch `main.tf`; the `main.tf` owner should correct the comment. -- **STALE gate corrected:** the runbook + `main.tf` say `vr1-dc1` is blocked on "NetBox assigning the - supernet" -- D-115 ALREADY assigned it (`10.12.64.0/19`, imported to office1-netbox). vr1-dc1 stays - out of scope this pass by SEQUENCING, not a missing literal. Do NOT uncomment `vr1_dc1_planes`. -- **Logged, NOT executed (adjacent improvement, hard rule 1):** promote `scratchpad/optc-calc.py` into - a committed whole-host capacity calculator with a harness (sibling to `dc-dc-ceph-disk-budget.sh`). - -The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling -gap register). Read it FIRST. This list is only what that doc does not already carry. - -1. **Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13).** - `voffice1`: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, **10.10.0.20** (Kea reservation, outside - the pool), egress through the OPNsense edge. On it: **MAAS 3.7.2** region+rack (PostgreSQL 16.14) - and **LXD 5.21.5**, with that LXD **registered into MAAS as an LXD VM host**. MAAS then - **composed, PXE-booted and COMMISSIONED** `office1-netbox` (LXD VIRTUAL-MACHINE, leased - 10.10.1.100 from MAAS, reached status **Ready**, 2 cores / 4096 MiB detected). - **The L3 nesting proof is now DEFINITIVE** -- a guest booted, PXE'd and commissioned three levels - deep, not merely "`/dev/kvm` exists". VR0's `lxd` + `tailscale` pattern, reproduced. - Codified in `scripts/site-headend-install.sh` (+ harness, 18/18) -- reusable for DC1/DC2. - **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS - dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is - NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. - **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.10) runs **NetBox - 4.6.4**; `office1-tailscale` (10.10.1.11, tailnet 100.64.0.53) is a subnet router advertising - `10.10.0.0/22` on the operator's **SELF-HOSTED** control plane. The edge routes `10.10.1.0/24 -> - 10.10.0.20`. Full connection reference: **`docs/vr1-office1-as-built.md`** -- read that first for - any "how do I reach X" question. - **GitBucket: OUT OF SCOPE (D-116, operator ruling).** No Office1-local GitBucket is built; - `git.baldurkeep.com` stays the git service of record. Office1's composed machines are exactly TWO. - **REMAINING on Office1 (reconciled 2026-07-14):** (a) route enable -- **DONE**, operator enabled - `10.10.0.0/22` on the control server; the edge GUI answers at 10.10.0.1 from the workstation with - no tunnel; (b) import the D-115 carve into the Office1 NetBox -- **DONE** (sandbox seeded from the - upstream draft and PROVEN faithful field-by-field; carve + six-plane roles + BOTH DCs imported, - 90 -> 134 prefixes); (c) the pinned Office1 LAN IPv6 (item 1a) -- **STILL OPEN**, and it is now - the last build item on Office1. - **STAGE 2 IS IN CLOSE-OUT, NOT BUILD. It is also the BRANCH MERGE POINT** (see PINNED). The - close-out checklist C1..C5 is in `docs/dc-dc-deployment-workflow.md`; do not re-derive it here. -1a. **DONE 2026-07-14 -- Office1's LAN IPv6 is LIVE and PROVEN ON THE WIRE. C1 is CLOSED.** - Address `2602:f3e2:f01:100::1/64` on the edge LAN (kernel-verified) + radvd advertising the prefix - with `mode=unmanaged`; `voffice1` autoconfigured `2602:f3e2:f01:100:5054:ff:fe6a:87e5` by SLAAC and - edge->voffice1 GUA ping is 0.0% loss. No collateral damage (v4, Kea DHCPv4, compose-net route all - intact). See `docs/changelog-20260714-office1-lan-ipv6-executed.md`. - **OPERATOR RULING 2026-07-14 -- DO NOT RE-PROPOSE A v6 NAT.** A v4-to-v6 NAT (second OPNsense, or - nginx) was proposed to *simulate* the v6 the lab will not forward, and was WITHDRAWN in favour of - *"configure IPv6 internally as much as possible; configure the IPv6 edge AFTER deployment - completes."* It contradicts **D-101** (*"native GUA routing, no NAT66"*) and **D-115**'s amendment - (upstream's problem, not ours), and a translation box is the one component Roosevelt will NOT have - -- maximum delta, zero proof value. nginx is an L7 proxy, not NAT, and proves nothing about - RA/ICMPv6/Ceph-over-v6. - **TRAP recorded:** `voffice1` has `forwarding=1` + `accept_ra=0` yet STILL autoconfigured -- - netplan/systemd-networkd handles RA in USERSPACE, independent of the kernel sysctl. Never diagnose - "no SLAAC" from the sysctl alone on Ubuntu; read `ip -6 addr` + look for a `proto ra` route. - **The WAN v6 leg remains DEFERRED** -- v6 does not egress the lab (re-measured 2026-07-14: v6 - internet 100% loss, v4 fine). Internet GUA reachability is UNPROVABLE in VR1. - -1a-historical. **(superseded -- the original pinned item)** deploy Office1's LAN IPv6 (operator ruling 2026-07-13, D-115 - question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). - Add `2602:f3e2:f01:100::1/64` to the OPNsense edge LAN + RA, and let `voffice1` take its address. - **CORRECTED 2026-07-14 (D-113 AMENDMENT -- the old instruction here was WRONG):** the ADDRESS half - CANNOT be done via the REST API. Measured on the live edge (OPNsense 26.1): `Interfaces > [LAN]` is - the LEGACY page `/interfaces.php?if=lan`; only Devices and Neighbors were migrated to MVC, and there - is **NO REST endpoint for a base interface's `ipaddrv6`**. D-113(a2)'s "interfaces are API-covered" - was INFERRED, never measured, and is false. - - **Address:** `scripts/opnsense-set-interface-v6.sh` (dry by default; harness 40/40). It is NOT a - config.xml push -- it mutates two leaves of the edge's LIVE config through OPNsense's own model, - so API-managed Kea and the firewall rules cannot be clobbered. **WRITTEN AND TESTED, NEVER RUN - WITH `--commit`.** - - **RA:** that half IS API-native -- `/api/radvd/settings/addEntry` via `scripts/opnsense-api.sh`. - Still forbidden either way: a rendered config.xml push. - **KNOWN LIMIT, do not forget it:** IPv6 DOES NOT EGRESS THE LAB (measured 2026-07-13: v6 gateway - reachable, v6 internet 100% loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding, - which is most of D-101's family matrix -- but NOT internet GUA reachability, which is unprovable - in VR1 until the lab's upstream carries v6. Deploy the LAN leg; DEFER the WAN v6 leg (it would be - a path to nowhere and untestable). See D-115's amendment. -1b. **`10.10.1.0/24` IS NOW A BLESSED ALLOCATION (D-115 ADOPTED).** VR1 Office1 = `10.10.0.0/22` - (Office role, `10.10.0.0/16`): LAN `10.10.0.0/24` + compose `10.10.1.0/24` + 2 spare. Zero - renumber. Also adopted: Edge role `172.30.0.0/16` (legitimises `office1-wan`'s 172.30.1.0/24, - which had been squatting in the Corp OOB block), and **DC2 moved to `10.12.64.0/19`** inside the - Cloud /16 (its old `10.13.0.0/19` sat outside every allocated block). STILL TO DO: write the - carve into NetBox -- production write is operator-gated; the plan is to import into the Office1 - sandbox NetBox first, verify, then feed back. -1c. **(superseded note kept for context)** the original unblessed-allocation finding: Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, - tenant pool=10.20.0.0/16 per D-016 -- the obvious `10.20.x` would have COLLIDED with tenant space), - but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is - one of the VMs this headend composes. -1c. **`modules/node-vm` nested-virt defect: FIXED 2026-07-13** (operator: fixes land as we find them; - only DC1 *deployment* is gated). Unconditional `cpu = { mode = "host-passthrough" }` -- DC nodes - run `nova-compute` and MUST run KVM guests. Without it they would have come up looking healthy and - been unable to launch a single instance. -2. **NetBox sandbox loop -- DOCUMENTED 2026-07-14 (close-out C4 DONE).** scope-doc section 8 + phase1 runbook Step 10b. (Was: BUILT 2026-07-13/14, undocumented.) - Architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> a NEW NetBox on the Office1 VM - (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements - BACK. The sim NEVER `--commit`s upstream. **The tooling now exists and is proven:** - `netbox/prod-draft-dump.py` (read-only, no write path at all), `netbox/sandbox-seed.py` - (dry-by-default, structurally REFUSES an upstream URL), `netbox/sandbox-fidelity-check.py` - (field-by-field -- counts alone HID a bug where 17 of 90 prefixes silently lost their region - scope). **STILL PENDING:** capture the loop in `docs/dc-dc-netbox-buildout-scope.md` + the Stage 2 - runbook, and **ensure `office1-netbox` holds the COMPLETE validated dataset** (close-out C2, - REDEFINED by DOCFIX-195: `office1-netbox` is the VR1 IPAM apex (the working draft every consuming - system reads); `netbox.baldurkeep.com` is the production apex, held as a READ-ONLY v1 reference draft, - and the write-back to it is DEFERRED to end-of-deployment -- NOT a Stage 2 gate). - **C2 sub-task (i) DONE 2026-07-15** -- the hardened fidelity check was re-run against `office1-netbox` - (read-only dump via the consolidated sandbox token + an SSH tunnel), and the sandbox is PROVEN - faithful: exit 0, "upstream draft + EXACTLY the planned delta." Along the way the hardened checker's - own scope rule was found to FALSE-FAIL on the two intentionally-unscoped role-container /16s - (`10.10.0.0/16`, `172.30.0.0/16`); fixed to assert scope == INTENDED (now also catches a WRONG-TARGET - binding -- the D-117 near-miss), harness 10->12, gauntlet ALL GREEN (58). SEC-009 gap closed: the - sandbox token was never consolidated; it now lives at `~/vr1-office1-creds/vr1-netbox-sandbox.env`. - See `docs/changelog-20260715-fidelity-check-scope-correctness.md`. - **C2 sub-task (ii) DONE 2026-07-15 -> C2 FULLY CLOSED -> STAGE 2 CLOSE-OUT COMPLETE (C1..C5 all done).** - The tooling gap was closed (`prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover - `ip-ranges`/`ip-addresses`; harnesses +T11/T12 and a new `netbox/d120-compose-bands.py` +16/16 - harness), and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox - (`--commit`, operator-approved) and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned - delta (D-115/D-117/D-118/D-120)." Gauntlet ALL GREEN (60). D-120 Step 6 CLOSED (no apex write). See - `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. **NEXT: the branch->`main` merge** (Stage 2 - close is the merge point, PINNED) -- an operator-gated action, present it, do not run it unasked. -3. **`pynetbox` -- CLOSED 2026-07-13.** Installed (7.0.0) on `office1-netbox`, which is where the - imports actually run. The stdlib tools (`prod-draft-dump`/`sandbox-seed`/`sandbox-fidelity-check`) - deliberately need no dependency at all. - **TRAP, recorded in `platform-traps.md`:** the upstream NetBox sits behind a **User-Agent-filtering - WAF**. `Python-urllib/3.12` gets a **403 that looks exactly like an auth failure**; the same token - works from curl. This affects pynetbox too. -4. **The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13.** - `scripts/prereqs/install-apparmor-libvirt.sh` (+ `install-all` / `check-prereqs` wiring; harness - 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants `/** rwk,` in - `/etc/apparmor.d/local/abstractions/libvirt-qemu`, which GATES EVERY VR1 VM BOOT: without it a - domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. - Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce - libvirtd unless it actually adds the rule). Pool parent defaults to `/var/lib/libvirt/vr1`, - override with `VR1_POOL_PARENT`. `docs/changelog-20260713-apparmor-libvirt-prereq.md`. -5. **The DEFERRED literals are now ALL ASSIGNED -- item CLOSED 2026-07-14.** The operator's Option-B - ruling ("do not invent CIDRs, wait for NetBox") held, and NetBox has now assigned them: - **ORG_ULA_48 = `fd50:840e:74e2::/48`** (**D-118 ADOPTED** -- RFC 4193 valid, and verified NOT to - collide with Tailscale's `fd7a:115c:a1e0::/48`, which matters because `office1-tailscale` runs - `--accept-routes`); GUA per D-117: first DC `2602:f3e2:f02::/48`, second `f03::/48`; the second - DC's v4 = six sequential `/22`s from `10.12.64.0/19` (D-115 moved it INSIDE the Cloud /16). - All are IMPORTED and fidelity-checked in the sandbox. - **CAUTION -- `DC2_V4_SUPERNET` IS A RETIRED NAME (D-117).** The importer now hard-rejects it and - rejects `--dc dc2`. Any doc still saying `10.13.0.0/19` is pre-D-115 and wrong. -6. **D-119 (ADOPTED 2026-07-14) -- THE DC NAMESPACE IS REGION-QUALIFIED. C3/gap #19 CLOSED.** - `vr0-dc0` (VR0's LIVE testcloud) / `vr1-dc0` (VR1's FIRST DC, GUA f02::/48) / `vr1-dc1` (VR1's - SECOND, f03::/48). Bare `dcN` is REJECTED everywhere -- it meant VR0's live cloud in `lib-net.sh` - but VR1's first DC in the importer: one string, two clouds. D-119 **COMPLETES D-117** (executes - its own amendment); it does not reverse it. **ZERO NetBox writes** -- the apex was already right; - the REPO was the odd one out. The importer's DC->site map is now an IDENTITY, so the off-by-one is - STRUCTURALLY IMPOSSIBLE, not merely defended. Also renamed: `vdc1`/`vdc2` -> `vvr1-dc0`/`vvr1-dc1` - (D-114's own DR primitive is `virsh destroy vdc1` -- it read as "destroy VR1 DC1" while MEANING - "destroy VR1 DC0"). Office KEEPS its number (`vr1-off1`, `Office1`, `voffice1`). - Env var: `DC1_/DC2_V4_SUPERNET` -> **`VR1_DC1_V4_SUPERNET`** (both old names rejected by name). - Gauntlet ALL GREEN (57); dc-selector 21->30, prefixes-import 40->82. - **THE `tofu apply` IS NOT DONE -- IT IS GATED.** Plan: 11 to add, 0 to change, 11 to destroy (the - libvirt object NAMES change, which is ForceNew; `moved{}` makes the plan reviewable but cannot - suppress a replace). MEASURED SAFE: all 11 are EMPTY (no guests, no volumes; Stage 3 hasn't run). - Re-plan at execution and STOP if anything shows ATTACHED. - See `docs/changelog-20260714-d119-region-qualified-dc-namespace.md`. - -6a. **(historical) G5 NetBox site rename -- CLOSED as D-117 (ADOPTED, Option B).** The repo moved to the apex's - 0-indexed convention (first VR1 DC = `vr1-dc0`), rather than bending the apex to the repo. This - fixed a REAL off-by-one: `dc-dc-prefixes-import.py` bound `--dc dc1` to slug `vr1-dc1` while - treating `f02::/48` as dc1 -- but the apex binds `f02` to `VR1 DC0`. It would have written the - first DC's prefixes onto the second DC's site. The importer is now also **dry-by-default**. - **D-117 IS ONLY PARTIALLY EXECUTED** -- see gap #19 / close-out C3: `opentofu/main.tf` and - `lib-net.sh`/`lib-hosts.sh` still speak the OLD naming, and `lib-net.sh`'s `dc0` means **VR0's** - DC0. That collision BLOCKS Stage 3. - G2 (aggregates) and G3 (ULA /48) are both CLOSED by the same work. -7. **DOCFIX-185 follow-up (operator to rule):** a D-100/D-107 amendment note so the Stage-3 DC - edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark - fiber is East-West/replication ONLY, never an internet path). -8. `docs/dc-dc-deployment-workflow.md`'s status table was corrected 2026-07-13, but re-read it - against reality before trusting any row -- it has drifted twice. -9. **DHCP PROVEN END TO END -- CLOSED 2026-07-13.** `voffice1` (the first client ever to sit on - `office1-local`) took a real lease from Kea: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches - its NIC), hostname `voffice1`, state active -- read back through the D-113(a2) REST API. The - service, not just the daemon, now works. -10. **Doc-accuracy finding (measured 2026-07-13, logged not actioned):** `docs/security-ledger.md` - SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web - GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is - correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default - and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE - the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only - path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. - Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local - libvirt bridge, `virbr2`, not routable off-host): `ssh -L 8443:10.10.0.1:443 @10.17.11.248` - -> `https://localhost:8443`. - -**VCLOUD POST-RESTART SWEEP + D-126 BASE-LEG + D-128 OPERATING MODEL (2026-07-17) -- SESSION CLOSED -(committed + pushed; two honest carry-forwards below).** The vcloud host -completed its bookend patch+reboot (running `6.8.0-136`, 0 upgradable, no reboot-required). Full sweep -GREEN: libvirt 12 nets/4 pools active+autostart, both VMs autostart=enable+RUNNING, autostart chain proven -end-to-end (all guests+edge uptime matched host boot). Post-reboot `ssh voffice1` was DOWN -- root cause: -`office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` on the bridge + -compose route `10.10.1.0/24 via 10.10.0.20`) lives nowhere in the XML and drops every reboot; D-126 made the -layers above it durable but never owned this base leg. **DELIVERED:** `scripts/site-baseleg.sh` + -`tests/site-baseleg/` (24/24) -- a root `Type=oneshot` unit `After=libvirtd` that re-adds the leg at boot -(bridge DISCOVERED from the net name, not baked; idempotent apply; `check` = read-only session-verify). -Gauntlet ALL GREEN (67), repo-lint 0-fail. **D-126 amendment** + **NEW D-128** (operating model: Claude on -the vcloud jumphost; Plane-1 substrate on vcloud / Plane-2 executes on voffice1; workstation tailnet = human -path; D-107 untouched) recorded; skill invariant + workflow Cross-cutting bullet added. See -`docs/changelog-20260717-d126-baseleg-oneshot-d128-operating-model.md`. **INSTALLED 2026-07-17; boot-race -UNPROVEN:** `install` run (operator sudo) -- unit written + `enabled` (multi-user.target.wants), `apply` ran -at install -> status=0, `check office1` -> ok. That run was POST-boot so it did NOT exercise the -`After=libvirtd`+poll race; boot durability stays UNPROVEN until an ACTUAL reboot shows `ssh voffice1` -self-heals. Ordering MEASURED (monolithic libvirtd active here). **COMMITTED + PUSHED** on branch -`dc-dc-stage3-phase2-dc-substrate`: `1958063` (tool + records + sweep) and `6f26b55` (tooling-gap register -item **20** CLOSED for Office1 + the skill's per-DC discipline). Gauntlet ALL GREEN (67), repo-lint 0-fail, -next-free D now **129**. `main` untouched (stage-close merge deferred). - -**CARRY-FORWARDS:** -1. **Boot-race proof -- PROVEN 2026-07-18 (RETIRED).** The operator-forced test reboot (host booted - 2026-07-17 23:38:41) exercised the `After=libvirtd`+poll race, and all four checks PASSED: - (a) `site-baseleg-office1.service` active(exited) status=0; (b) the unit journal logs - **`added leg: 10.10.0.10/24 on virbr2`** + `route set: 10.10.1.0/24 via 10.10.0.20 dev virbr2` at - 23:38:53 -- the STRONG proof (leg ABSENT at boot -> the unit CREATED it, not "leg present"); (c) `bash - scripts/site-baseleg.sh check office1` -> ok; (d) `ssh voffice1` returned live with NO manual `ip` - command this session. So the D-126 base-leg oneshot self-heals `office1-local`'s dropped L3 leg across a - real reboot, unattended. (Journal read needed root; jessea123 was added to the `systemd-journal` group - 2026-07-18 + relog so future sessions read unit journals without sudo.) Register item 20 / skill routing - row stand as the durable mechanism. -2. **DC leg rows.** `site-baseleg.sh` generalizes to the DCs but a DC row is DEFERRED-until-measured (hard - rule 2; the harness fails on an un-commented DC supernet). At each DC standup, MEASURE the vcloud->DC - reach and add a MEASURED leg row + install ONLY if that net is isolated (register item 20 / skill invariant). - -**OPERATOR PERSONAL KEY pushed to Office1 service VMs (2026-07-18).** On operator request ("push my -id_ed25519 on the vms"), the operator's GitHub-published personal pubkey (`ssh-ed25519 ...J7Zg -JANeumatrix@github`, gh:JANeumatrix) was APPENDED to `~/.ssh/authorized_keys` on **voffice1** (user -jessea123), **office1-netbox** and **office1-tailscale** (user ubuntu) -- ALONGSIDE the `vr1-office1-svc` -service key (service key untouched; the D-126 ProxyJump path is unchanged). Read-only audit first (key -ABSENT on all three), idempotent grep-guarded append, verified 2 keys each. This is a deliberate EXTENSION -of the D-126 per-site-service-key model: the operator's personal identity now has direct login from any -workstation over the tailnet (`ssh jessea123@10.10.0.20` / `ubuntu@10.10.1.10` / `ubuntu@10.10.1.11`). -Pubkey ONLY -- the personal private key is workstation-held, never on the jumphost; login-with-personal-key -still needs the operator to test from a workstation. **Edge (office1-opnsense): operator installs via the -OPNsense GUI** (System > Access > Users > root > Authorized keys) -- a raw `/root/.ssh/authorized_keys` -append is NON-DURABLE (OPNsense regenerates root's keys from `config.xml`); config store audited, personal -key ABSENT there too. **STANDING (per-DC): at each DC standup, add the same personal pubkey to that DC's -service VMs** ("in case I need to log in directly" -- operator instruction), alongside the base-leg per-DC -discipline above. **REMOVAL/ROTATION OBLIGATION:** strip these personal keys at v1 close / teardown (same -class as the SEC key-rotation rows); track if a security-ledger row is wanted. - -**OPNsense EDGE 26.7 UPDATE + PLUGIN/ADD-ON REVIEW (2026-07-18).** Operator updated `office1-opnsense` -**26.1.11 -> 26.7** (0 outdated pkgs, no pending firmware). As-built version RECORDED in -`docs/vr1-office1-as-built.md:42` (appendix-B is charm-channels only, no edge row). Operator asked for a -forward-looking plugin/add-on review -> `docs/opnsense-edge-addon-review-20260718.md` (all 104 catalog -plugins categorized) + **D-129 (PROPOSED)** captures the base-edge-profile decision. Near-stock edge: -only `os-git-backup` installed, and **UNCONFIGURED** (php-verified no config node) -> **no config backup is -actually running** -- recommend wiring it to `git.baldurkeep.com` (config-as-code) regardless of D-129. -Review recommendations by bucket: metal-hardware profile (`os-smart`/`os-nut`/`os-cpu-microcode`/`os-lldpd`, -record for Roosevelt, inert in VR1 VMs); sim scaffolding (`os-qemu-guest-agent`, VR1 VM edges only, aids -D-126/D-127 reboot work); diagnostics (`os-iperf` for transit validation); observability -(`os-node_exporter` CONDITIONAL on a COS-scope ruling -- D-105 scoped COS per-DC inside the cloud, edge is -outside); DEFER all security hardening per closed-test posture. **UPDATE 2026-07-18 -- operator directive "install the recommendations + pin the rest".** D-129 now -PARTIALLY RULED: VR1 edge profile **{`os-qemu-guest-agent`, `os-iperf`}** adopted. Live install on -`office1-opnsense` via the D-113 firmware API is **operator-GATED** -- the auto-mode classifier blocked the -agent's `core/firmware/install` POST (cloud mutations stay gated even during batch), so the operator runs it -via `!` (or approves); **verification pending** (confirm the `qemu-guest-agent` DAEMON runs, not just the -pkg, + edge DHCP/NAT unbounced). The "rest" is PINNED (not built now, per advisor -- pin = record at the -step): metal profile {`os-smart`/`os-nut`/`os-cpu-microcode`/`os-lldpd`} -> Roosevelt edge build (inert on -VR1 VMs); `os-node_exporter` -> `dc-dc-phase6` Step 10 gated on COS-edge scope; VR1 profile -> `dc-dc-phase2` -edge section (+ build the reusable profile-installer there); security -> deferred (posture). **FOUR -sub-decisions STILL OPEN** (COS-edge scope, `os-frr`, `os-tailscale`, edge-NTP) -- a one-line directive does -not rule them. As-built row updated (FreeBSD base 14.3->15.1 too). -**INSTALL VERIFIED 2026-07-18 (operator ran the firmware POSTs via `!`):** `os-qemu-guest-agent` package -INSTALLED but NON-FUNCTIONAL -- daemon off + `opnsense-edge` module defines no `org.qemu.guest_agent.0` -libvirt channel (grep-confirmed), so `guest-ping` fails. Functional qga = channel-in-module + service-enable -+ edge RESTART -> DEFERRED to next scheduled edge restart (new DC edges born with the channel). `os-iperf` -did NOT install: two back-to-back firmware POSTs collided on the OPNsense firmware LOCK -> re-run pending -(profile-installer must be one-at-a-time + poll). Records reconciled to this reality (D-129 Verification -note, phase2 pin, as-built). Docs committed on branch `dc-dc-stage3-phase2-dc-substrate` (pushed -2026-07-18, `83876c3..c1ef7ab`). -**AUTONOMOUS PROCESSING 2026-07-18 (operator: "process what you can without me running anything").** -Boot-race carry-forward RETIRED (D-126 base-leg PROVEN across the forced reboot -- `added leg` in the -journal; jessea123 added to `systemd-journal` group so unit journals read without sudo). Built the D-129 -profile-installer `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (17/17; gauntlet 68 GREEN) -- -closes the "build the installer" pin; encodes one-at-a-time+poll + the qga-channel warning. Logged the qga -channel design (OPT-IN var, default OFF -> no office1/DC0-plan side-effect; mechanism-to-verify on dmacvicar -0.9.8) + the `os-git-backup` operator-input ask. **Still operator-only:** `os-iperf` re-install (classifier- -gated), qga functional enablement (channel+enable+restart), git-backup wiring, the 4 D-129 sub-decisions. -See `docs/changelog-20260718-opnsense-plugins-installer.md`. - -**PERMISSION POSTURE UPDATED 2026-07-18 (mutations PROMPT, not classifier-deny).** The `auto`-mode -classifier was HARD-DENYING mutations that matched no rule (edge API POST, settings edits, Skill invokes) -- -no approval prompt, so the operator had to run them by hand. claude-code-guide confirmed (doc-cited) that -`ask`/`deny` rules are evaluated BEFORE the classifier, so `ask`-listing the mutation surfaces forces a -prompt. `.claude/settings.json` updated (operator-applied by hand -- editing the permission file is itself -gated; chicken-and-egg): added `ask` for `*opnsense-api.sh POST *`, `*opnsense-plugins.sh apply*`, `*tofu -apply/destroy/import/state rm*`, `*--commit*`, `Edit(.claude/settings*.json)`; added `allow` for the edge -GET + `opnsense-plugins.sh list`. The `Edit(.claude/settings*.json)` ask rules mean the agent can now -PROPOSE settings edits (prompted), no longer hard-blocked. Stayed on `auto`+`ask` (not `default` mode) to -keep read-heavy audits frictionless. See `docs/changelog-20260718-permission-posture-ask-preempts-classifier.md`. -**CARRY-FORWARD (verify after the operator's reload):** the new rules take effect only after a Claude Code -restart. Post-reload, CONFIRM a mutation now PROMPTS (approve/deny/adjust) instead of hard-denying -- the -`os-iperf` install (`OPNSENSE_API_HOST=10.10.0.1 bash scripts/opnsense-plugins.sh apply vr1-edge`, or the raw -`opnsense-api.sh POST core/firmware/install/os-iperf`) is the ready test. If it still hard-denies, the rule -pattern needs adjusting (check the exact command string vs the glob). - -## OPEN WORK -- VR0 / DC0 (the running testcloud) - -1. **d011 batch-3 live-validation queue** -- `scripts/checks/d011-04-octavia-lb.sh` and - `d011-05-magnum-e2e.sh` are DRAFTED and mock-tested; the live PASS was recorded in the - 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: - d011-04 `amphora_headroom()` field parsing (safe default HOLDs on a parse miss -- confirm the - OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant - (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set - `VR_FOIL_APPCRED`. Sequence: non-disruptive half first, then `--include-disruptive`. -2. **Script backlog item 3 (remainder):** fold `scripts/vault-kv-health.sh` into `cloud-assert.sh` - as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script - -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure - runbook. -3. **Script backlog item 5:** `scripts/tenant-cluster-create.sh` -- generalize the stage-6 + - watch + D-066-evidence wrapper. NOT STARTED (verified absent). - (Items 6/7/8 -- `keystone-policy-drift.sh`, `cloud-snapshot.sh`, `tests/tenant-acceptance/` -- - are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.) -4. **S-class script quality (`docs/script-quality-findings-20260707.md`):** S4/S7/S8a/S8b DONE; - S6 no-action; S8c ruled WON'T-DO. **S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ - offboard) are HELD** by the operator until the DOCFIX-126/127 live-verify lands. -5. **DOCFIX-126/127 live-verify is still OPEN** -- the offboard-E0 and onboard-cred-file fixes have - never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard - confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage. -6. **`test1` orphan user -- approved for deletion, staged, NOT run.** Role-less shell in the - devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): - `openstack user delete 9a62f88e331f41adb5c9e41225ec8698`. -7. **The handoff doc is stale:** `docs/handoff-20260703-open-items.md` still lists R-3 (D-063) as - PROPOSED/OPEN. D-063 is `RESOLVED / CLOSED` in `design-decisions.md` (verified). Correct the - handoff doc. -8. **Upstream reports queued (operator):** magnum `can-upgrade-to` anomaly -- Charmhub magnum - 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from - the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN - -- operator judged it site misconfig; the draft is retained.) -9. **Barbican `host_href`:** the charm renders a literal `https://None:9312`, suppressing the - upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A - charm-level fix would need its own D-NNN. Logged, not actioned. -10. **v1-close checklist is QUEUED** (handoff section 6) -- gated on D-011 item 6, which is gated - on the SEC-003 unseal rehearsal. - -## OPEN -- NetBox WRITE-PATH BUGS (found 2026-07-14 by adversarial review; LOGGED, NOT FIXED) - -These are independent of D-119's naming work. **DOCFIX-195 (2026-07-15) reclassified WHEN they matter:** -C2 no longer writes the PRODUCTION apex `netbox.baldurkeep.com` (office1-netbox is the VR1 IPAM apex; -the `netbox.baldurkeep.com` write-back is DEFERRED to end-of-deployment). So these bugs gate the DEFERRED merge-back, NOT Stage 2 -close -- they are what would corrupt the production apex during THAT write. They no longer block C2 or -the Stage 2 merge. - -1. **`roles-aggregates-import.py` still dies HALF-WAY through a production write.** The preflight it - advertises only checks role-NAME collisions. The `ARIN` RIR lookup and `validate_ula_48()` both run - **after** the role-creation loop has already committed 5 roles. A typo'd `ORG_ULA_48`, or an apex - without ARIN, writes 7 objects then dies with **no rollback**. The RIR preflight loop is a literal - no-op (`if ...: continue` + a comment). The harness cannot catch it: it only asserts the STRING - "PREFLIGHT" appears before the first create. **This is the same bug class that already bit us once.** -2. **`sandbox-fidelity-check.py` can return a FALSE GREEN.** It field-compares only the SHARED objects; - the planned delta is checked merely as a SUBSET of `EXPECTED_NEW`. So a prefix created with the - WRONG scope/role/status passes clean -- and if the importer created ZERO of the 36 DC prefixes, - `extra subset-of expected` still holds and it exits 0 saying "nothing lost, nothing stray." - **We have been citing this script as proof the sandbox is faithful. It cannot prove delta - correctness.** It WOULD have caught the historical 17-prefix scope drop (those were shared objects). -3. **`find_role()` dies mid-loop** in `dc-dc-prefixes-import.py` -- roles are resolved lazily INSIDE - the write loop. Against upstream as it stands today (no six-plane roles), a `--commit` creates the - site + 4 prefixes then dies on the 5th. No whole-plan preflight. -4. **Same-dumper blind spot:** `sandbox-fidelity-check` compares two dumps produced by the SAME field - list, so any field the dumper omits is invisible on BOTH sides by construction (prefix `vrf`, - `tenant`, `vlan`, `tags`, `custom_fields`; site `tenant`/`group`/`facility`). Structurally the same - shape as the region-scope bug that dropped 17 prefixes. -5. **Duplicate-CIDR collapse:** prefix-keyed dicts in the seeder and the fidelity check collapse - duplicate prefixes (NetBox permits them, and the importer's own docstring anticipates them vs - `vr0-dc0`). A whole prefix object can vanish with a green check. Latent -- no duplicates upstream today. -6. **The test fake diverges from reality at the failure point:** `fake_pynetbox.get()` returns - `matches[0]` on a multi-match where REAL pynetbox RAISES. So the harness cannot catch #5. - -**Recommended sequence: fix 1-3 BEFORE C2.** 1-3 are DONE (2026-07-14, -`docs/changelog-20260714-netbox-write-path-hardening.md`): the roles-importer preflight now covers -ARIN + the ULA, the prefixes-importer preflights every role, the fidelity check is BOTH-bounds + -delta-scope-aware, and the test fake raises like real pynetbox. Gauntlet 58 ALL GREEN. -**CONSEQUENCE TO ACT ON:** the sandbox was declared "proven faithful" (2026-07-14) using the -fidelity check BEFORE it was hardened -- i.e. under the version that could FALSE-GREEN. RE-RUN -`sandbox-fidelity-check.py` against the sandbox under the hardened check before trusting that -"faithful" verdict -- this re-run IS C2 sub-task (i) (DOCFIX-195), run against `office1-netbox` (the -VR1 IPAM apex), NOT against the production apex `netbox.baldurkeep.com`. (Not done this session: it -needs the office1-netbox token, which is operator-held; folds naturally into the C2 step.) -Items 4-6 remain LOGGED not fixed. - -(Historical rationale: 1 and 3 are partial-write-no-rollback on the production IPAM; 2 is what we would use to prove the write was correct.) - -## Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions) - -- **STAGE 2 MERGED -- DONE 2026-07-15 (PR #1, `e468d2e`).** The Stage 2 close-out (C1..C5 all met) - branch `dc-dc-stage1-phase0-first-exec` merged to `main` via a merge commit -- the first VR1 feature - merge. `main` IS now trunk-of-record for VR1 through Stage 2; the branch was retired. This is no - longer an open ruling; it GRADUATED into the STANDING **branch-per-stage + merge-at-stage-close** - discipline that applies to every stage from here (workflow doc "Cross-cutting discipline"): the merge - is part of each stage's definition-of-done and remains operator-gated (present it, do not run it - unasked). Stage 3 branches off the post-merge `main` as `dc-dc-stage3-phase2-dc-substrate`. -- **UPSTREAM APEX WRITE-BACK -- DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15).** - Seeding `netbox.baldurkeep.com` from the validated `office1-netbox` dataset is a MAYBE at the very - end, NOT a Stage 2 gate. Until then the apex stays a READ-ONLY v1 reference draft (no write during - VR1); `office1-netbox` (10.10.1.10) is the VR1 IPAM apex (the working draft every consuming system - reads). The "NetBox WRITE-PATH BUGS" section gates THAT deferred write, not Stage 2 close. Defers the - write; does NOT revoke D-010. -- **D-071** ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning - session. Backups now exist (a 902MB controller backup was taken; `juju create-backup` / - `download-backup` DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). - Controller HA (single-controller, no-HA) is the unresolved half. -- **D-068** off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714. -- **SEC-001 / SEC-004 / SEC-005 / SEC-007:** rotate / flip-to-private / revoke at v1 close. -- **SEC-003:** custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. - This keeps `d011-06-vault-unseal` MANUAL and gates the FULL close of D-011. -- **SEC-006** (burned NetBox API token) -- **DEFERRED by operator ruling 2026-07-13 to deployment - completion.** Until then the token is LIVE and EXPOSED. -- **D-076 is NOT this repo's to work.** D-076..D-099 are a RESERVED source-project band, never - assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live - in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are - running their own tests. -- **quota-vs-capacity: CLOSED** -- operator ACCEPTS as-documented (no quota trim). - ---- - -## Standing lessons and traps (load-bearing; these live nowhere else) - -1. **THE EDGE TRAP -- DHCP on Office1 is API-MANAGED.** A full rendered `config.xml` push WOULD - CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all - gone -- `opentofu/templates/README.md` is a tombstone). Configure the edge via the REST API - (`scripts/opnsense-api.sh`); mint keys with `scripts/opnsense-bootstrap-apikey.sh`. -2. **"updated in-place" DOES NOT mean "no restart".** A `tofu apply` whose plan says - `libvirt_domain.vm will be updated in-place` **RESTARTED the guest** (measured: uptime 8:36 -> - 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it - says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to - apply a disk-list change and BOUNCES it. **Assume ANY apply touching a `libvirt_domain`'s - devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.** -3. **Do not re-implement vendor internals; call the interface.** OPNsense stores API secrets as - `crypt(secret,'$6$')` (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we - deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep - byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate). -4. **A config ISO can NEVER be read on an OPNsense nano image** (D-112). Any instruction to build - one is a trap; the dc-dc-phase2 runbook directed people into it for days. -5. **root's shell on the OPNsense edge is `tcsh`, not `sh`.** A `$(...)` inside a quoted remote - command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: - a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. - **Always feed remote commands to `sh -s`.** (The `scripts/opnsense-apply-config.sh` that was - once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml - path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.) -6. **Guide changes must sweep the tenant AI skill in the SAME change.** Skill-lag was caught twice - (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 - is pinned, and any guide edit FAILS lint until re-recorded). -7. **The controller model is `admin/controller`,** not `/controller`. -8. **`juju status --format=line` omits workload messages on 3.6** -- this caused a cloud-assert - false negative (fixed by DOCFIX-087). Don't rely on it. -9. **repo-lint's guards are load-bearing; reword rather than weaken.** L3 (docs must not reference - missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, - the guard kept. - -## State facts to remember - -- **The beta cluster and the foil1 tenant are GONE** (both confirmed absent by the 2026-07-08 - read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is - retired here. `capi-test-1` and the `lbtest` LB leftover are also gone; orphan sweep reads - ORPHAN=0, zero unattached floating IPs. -- **devteam is a LIVE tenant** running a self-created cluster (built via Horizon, calico, - keypair `devteam-key`). Their template has `volume_driver` unset but the cluster DOES get a - default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the - box; no template change needed. -- The vcloud host was NOT pristine: 9 orphaned `vr0-*` libvirt nets were torn down; **`wan` was - KEPT** (dual-stack NAT-out-enp1s0, GUA `2602:f3e2:fe:10::/64`) as the working model for the - per-site ISP-uplink network. -- MTU has TWO domains: the ISP uplink `enp1s0` is 1500; the host-internal plane/dark-fiber fabric - is **jumbo, underlay_mtu=9000** (operator ruling; verified on all 10 nets). -- The repo is temporarily **PUBLIC** (SEC-004) -- flip private at v1 close. -- Secrets live jumphost-local and are NEVER committed: `~/vr1-office1-creds/` (edge SSH key, - root password, API key), `~/vr1-office1-creds/vr1-netbox.env` (NetBox token), `~/tenant-devteam/`, `~/vault-init/`. - -## Project-completion (execute after D-011 passes) - -- Consolidate the 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. -- Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials. -- v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. - -## NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13) - -**Done.** The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop -(`netbox/prod-draft-dump.py` READ-ONLY -> `netbox/draft/vr1-draft.json` -> `netbox/sandbox-seed.py`), -seeded 129 objects from the upstream draft, then applied the **D-115 office carve** -(`netbox/d115-office-carve.py`): site `vr1-off1`, roles `office` + **`edge`** (new), and 8 prefixes. -Idempotent, and **verified FIELD-BY-FIELD** against the upstream draft -(`netbox/sandbox-fidelity-check.py`): every shared object is IDENTICAL, and the delta is EXACTLY -`+edge` role, `+vr1-off1` site, `+8` D-115 prefixes. The sandbox is a proven-faithful replica, which -is what makes it a valid baseline for the later feed-back diff. See `docs/vr1-office1-as-built.md` s6. - -**Why that check exists:** counts and idempotency CANNOT prove fidelity. The seeder silently dropped -the scope on 17 of 90 prefixes (it mapped only `dcim.site`; most of the VR1 draft is -`dcim.region`-scoped) -- and every count still matched and re-runs were still idempotent. It was -caught by LUCK on a spot-check. The fidelity check is that catch, made deliberate. - -**D-117 Option B is only PARTIALLY EXECUTED.** Done: the import path (`dc-dc-prefixes-import.py`) and -`docs/dc-dc-netbox-buildout-scope.md`. NOT done: the `$DC` shell selector (below) and the -forward-looking DC prose in the runbooks. Do not read "D-117 ADOPTED" as "fully renamed". - -**D-117 ADOPTED (operator ruling, Option B, DC-only):** VR1's DCs are **dc0/dc1**, matching the apex. -The **office KEEPS its number** (Off1). Fixed the importer's off-by-one site binding and made it -**dry by default**. - -### ~~BLOCKED~~ **UNBLOCKED and DONE** -- D-118 ratified the org ULA; both DCs imported - -**D-118 ADOPTED:** `ORG_ULA_48 = fd50:840e:74e2::/48` (RFC 4193 valid; **no collision with -Tailscale's `fd7a:115c:a1e0::/48`**, which matters because `office1-tailscale` runs `--accept-routes` --- an overlap would have been a LIVE tailnet routing conflict). Also corrected a **FALSE claim** in -`changelog-20260711-ula-gen-command-fix.md` that called the value "ratified" when no D-number ever -assigned it -- the same document-asserts-authority failure mode as D-117. - -**Imported into the sandbox:** six-plane roles + 2 RIRs + 5 aggregates, then **VR1 DC0 (18 prefixes) -and VR1 DC1 (18)**. The D-117 fix is now **LIVE-PROVEN**, not just offline: `--dc dc0` bound to the -apex site `VR1 DC0` (id=8) and `--dc dc1` to `VR1 DC1` (id=9). ULA reads `DC.NN` in the 4th hextet -exactly as D-111 intends (`fd50:840e:74e2:220::` .. `:350::`). - -**Final state PROVEN by `netbox/sandbox-fidelity-check.py`:** the sandbox is the upstream draft plus -EXACTLY the planned delta (D-115 + D-117 + D-118) -- every shared object field-identical, nothing -lost, nothing stray. 90 -> 134 prefixes. - -### A REAL BUG in `roles-aggregates-import.py`, found by running it for the first time for real - -It checked existence by **slug only**. NetBox also enforces **unique NAMES**. The draft's legacy role -`Replication` (slug `repl`) blocks the six-plane role (slug `replication`, name `Replication`), so it -created 4 roles, **400'd on the 5th, and DIED -- leaving the apex half-populated** with no RIRs and no -aggregates and no rollback. It had **never been run against a NetBox holding the real draft** -(only empty ones) and **shipped with NO harness** -- which is why this survived. - -Fixed: the six-plane role is now named **"Replication Plane"** (slug unchanged -- `lib-net.sh` -SPACES6 and the prefix importer look it up), and the script now **PREFLIGHTS every name/slug and -writes nothing unless the whole plan is viable**. `tests/roles-aggregates-import/` created: 16/16. - -### (historical) the original blocker - -`netbox/dc-dc-prefixes-import.py --dc dc0` cannot run. It requires `ORG_ULA_48`, and that literal is -**still unassigned** -- it is gap **G3** in `docs/dc-dc-netbox-buildout-scope.md`, which *recommends* -`fd50:840e:74e2::/48` (CSPRNG-generated, Global ID `0x50840e74e2`) but **no ADOPTED decision assigns -it** (`grep ORG_ULA docs/design-decisions.md` -> nothing). D-101 explicitly leaves it NetBox-assigned. - -Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. -This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready. - -**To unblock:** ratify an org ULA `/48` under its own D-number. Then: -``` -ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # VR1 DC0 -ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f03::/48 \ - DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # VR1 DC1 -``` -(`DC_GUA_PREFIX` values are MEASURED off the apex; `DC1_V4_SUPERNET` is assigned by D-115. -`DC2_V4_SUPERNET` is RETIRED -- the tool rejects it by name.) +Retained compact blocks (still-live obligations without another home yet): +- **NetBox write-path bugs (2026-07-14 adversarial review; LOGGED, NOT + FIXED):** importer/write-guard defects recorded in the rotated body + ("OPEN -- NetBox WRITE-PATH BUGS" section) -- route to a findings doc or + fix before the next NetBox write campaign. +- **Project-completion (after D-011 passes):** consolidate the 10 per-phase + do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); + revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 + dual-stack, NetBox import bundle. -### Also open (recorded, not blocking) -- **`$DC` shell-selector namespace collision** (D-117 amendment): `lib-net.sh`'s `dc0` means **VR0's** - DC0, while the importer's `dc0` now means **VR1's**. NOT interchangeable. Proposed fix: region- - qualify the shell selector to the apex slugs (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`). -- Feed the D-115 carve upstream to `netbox.baldurkeep.com` (operator-gated; the tool refuses a - non-sandbox target without `--yes-write-upstream`). -- Office1 LAN IPv6 (`2602:f3e2:f01:100::/64` + RA via the OPNsense REST API) -- still pinned. ## SESSION CLOSE 2026-07-18 -- grounding audit (Phases 1-4); GA-R4 bounded summary diff --git a/runbooks/appendix-A-troubleshooting.md b/runbooks/appendix-A-troubleshooting.md index a0f7100..2893939 100644 --- a/runbooks/appendix-A-troubleshooting.md +++ b/runbooks/appendix-A-troubleshooting.md @@ -658,3 +658,12 @@ defined in bundle.yaml comments beside the lines they fix. Rule going forward: new identifiers get EITHER a changelog/decision entry OR an index line here at introduction time (repo-lint L5 guards duplicates). + +## Symptom: cloud-assert (or any check) reports a unit healthy while its workload message shows an error in `juju status` + +`juju status --format=line` OMITS workload messages on juju 3.6 -- a check +that greps the line format sees agent state only and returns a FALSE +NEGATIVE (this bit cloud-assert once; fixed as DOCFIX-087). Use the tabular +or yaml/json formats when the workload MESSAGE matters. (Rotated here from +the session ledger's standing-lessons block, 2026-07-19 -- this is its +durable home.)