Newer
Older
openstack-caracal-dc-dc / docs / CURRENT-STATE.md

CURRENT-STATE.md -- the single status authority (Omega Cloud / VR1 DC-DC)

Authored 2026-07-18 by the grounding-audit Phase-2 ground-truth agent (charter: docs/audit/grounding-audit-charter.md, section 3) at repo HEAD e999b03 on branch dc-dc-stage3-phase2-dc-substrate. Every claim below carries its evidence (path:line, quoted command output, or commit hash). Claims that could not be evidenced read-only are marked UNKNOWN with what would resolve them. Nothing here is guessed.

STATUS OF THIS DOCUMENT: SIGNED by the operator 2026-07-19 (section 11; charter Phase 6 item 5). STANDING RULE (GA-R1, RATIFIED 2026-07-18 with amendments C1+C2 -- docs/audit/ga-rulings.md): no status claim is hand-written anywhere else; other documents point HERE; this document cites captured command output, and measurement always wins over it (C2). Other status surfaces are pointers or history; where one still carries a claim, it is a defect (Phase 1 proved they contradict: docs/audit/record-inventory.md, 12 groups; findings GA-F01..GA-F15).


1. Where the project IS

  • Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, docs/design-decisions.md:1946).
  • Stage: Stage 3 / Phase 2 -- "OpenTofu builds each DC substrate" (docs/dc-dc-deployment-workflow.md:148; branch name dc-dc-stage3-phase2-dc-substrate; runbook runbooks/dc-dc-phase2-tofu-dc-substrate.md). Stages 0-2 precede it; stages 4-7 are authored, not executed (workflow doc:873).
  • Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- docs/audit/outer-plan-20260719-postA-converged.txt). Deploy step B (bootstrap) COMPLETE 2026-07-20 in the same logged dc0-deploy window: transit reach established (voffice1 holds 172.31.0.1/30; reach = ssh -J voffice1 w/ dc0 key -- no vcloud host leg, item-20 disposition), rack ENROLLED to the Office1 region, node-host ready (libvirt + nested KVM + inner pool), SEC-010 applied+verified BOTH transit ends (row CLOSED), OPNsense 26.7 nano base staged (operator ruling; step-C boot REVALIDATES the D-112/D-113 path on 26.7). Named gate check EXIT 0: docs/audit/stepB-check-20260720-final.txt. Deploy step C (inner apply) COMPLETE 2026-07-20: executed FROM voffice1 (D-128 Plane 2 -- tofu 1.12.4 + repo clone + dc0 key staged there), 28/28 resources, inner plan CONVERGED zero diff (docs/audit/inner-converge-20260720-stepC.txt); 10/10 domains RUNNING inside vvr1-dc0 (9 nodes + edge); edge = fresh 26.7 nano, serial log at the FreeBSD login prompt (D-112 boot path first-datapoint PASS on 26.7). The INNER tfstate lives ON voffice1 (vr1-dc0-substrate/terraform.tfstate -- new state-of-record location; add to the site backup set). ACTIVE gate: G10 remaining. Edge bootstrap DONE 2026-07-20: D-112(c) console bootstrap complete (key-only root SSH proven) and the D-113(a2) API key minted via the vendor model -- GET core/firmware/status 200 with CORE_ABI 26.7, the first proof the API path works on 26.7. Measured: edge vtnet0 = LAN (provider-public), vtnet1 = WAN; edge still on its FACTORY LAN 192.168.1.1/24. Rack legs 10.12.4.2/22 + 10.12.8.2/22 added INTERIM (non-persistent ip addr; script support is a queued finding), plus a temporary 192.168.1.2/22 to reach the factory LAN. D-125 egress isolation gate: PASS / CLOSED 2026-07-20 (executed as written -- throwaway VM on br-vr1-dc0-wan; two identical consecutive runs: gateway ping 0, internet ping 0, curl 1.1.1.1 301, curl archive.ubuntu.com 200). Bridge-in is PROVEN end to end and the double-NAT fallback is NOT needed. Captures: docs/audit/d125-egress-gate-20260720{,-matrix}.txt. One earlier run failed ICMP-to-internet on the same path and is recorded UNEXPLAINED in the session changelog (start there if a DC edge shows first-boot egress failure). Edge ADDRESSED 2026-07-20 via the NEW operator-ruled opnsense-set-interface-v4 pair (D-113 amendment re-measured and still true on 26.7 -- base-iface addressing is not REST-covered): WAN 172.30.2.2/24 + default gw 172.30.2.1 (was dhcp, which could never work on a /24 with no DHCP server), LAN 192.168.1.1/24 -> 10.12.4.1/22 (ruled provider-public gateway). Verified on the kernel; the edge itself egresses to 1.1.1.1 at 0% loss, and the API answers at the new LAN address. Interim bootstrap address removed; virbr5 now carries only the ruled 10.12.4.2/22. D-129 edge profile APPLIED 2026-07-20 on 26.7 (operator-ruled): expose_qga_channel shipped in modules/opnsense-edge (opt-in, default OFF; dc0 true) and applied as an IN-PLACE domain update; os-qemu-guest-agent + os-iperf installed for real and the agent ANSWERS -- guest-ping -> {"return":{}} and domifaddr --source agent reports both legs. Note this run also exposed and fixed a false-success bug: opnsense-plugins.sh apply had ALWAYS dry-run (see session changelog item 12), so any prior "applied" claim from that script is void. Step D part 1 DONE 2026-07-20: rack registered (7chphy, rackd running), metal-admin dynamic range 10.12.8.100-.200 created (operator-ruled D-120 inheritance), VLAN 5005 dhcp_on=true primary_rack=7chphy verified by read-back. INCIDENT RESOLVED 2026-07-20 (operator-approved region restart): dhcpd now RUNNING on both controllers (verified by process, not service status), and all 9 DC0 nodes ENLISTED in MAAS with shapes exactly matching D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB) -- docs/audit/stepD-enlistment-20260720.txt. The G10 depth-4 nested boot gate is therefore PASS: node VMs inside vvr1-dc0 PXE-booted from the Office1 region across the transit and run MAAS's ephemeral kernel. The incident as originally found: MAAS 3.7 drives DHCP via Temporal, and Temporal is wedged on the region ("Not enough hosts to serve the request", 2807 retries), so no dhcpd runs on EITHER controller -- including voffice1 itself, whose compose net reads dhcp=True with no dhcpd process. Predates and is NOT caused by this deploy (almost certainly since the 2026-07-17 host reboot); unnoticed because both Office1 VMs were already Deployed. Any "Office1 MAAS DHCP working" claim is currently FALSE. Proposed gated remedy: restart MAAS on the region -- DONE, and it fixed BOTH sites, confirming a single root cause. Details + the queued detection-gap finding (cloud-assert trusts MAAS's self-report and missed a dead DHCP server): session changelog items 13-14; appendix-A entry queued. Step D part 2 BLOCKED on a ruling (2026-07-20): the nine nodes fell back to New with no power_type -- commissioning cannot finish without power control. New root opentofu/vr1-dc0-maas/ is shipped and its plan is clean, but the apply FAILED: Failed talking to pod: Failed to login to virsh console. MEASURED cause -- the MAAS snap is confined, gets Permission denied on /var/run/libvirt/libvirt-sock, and snap connections maas lists NO libvirt interface, so a LOCAL qemu:///system pod is IMPOSSIBLE with snap MAAS. This refutes the mechanism stated in D-123 Model B and in modules/maas-vm-host's header (intent survives, mechanism does not); both need an amendment once the replacement is ruled. The qemu+ssh replacement was then wired with an operator-ruled DEDICATED key and PROVEN reachable from both snaps -- but the pod apply failed again, finally on domblkinfo ... missing storage backend for 'volume' storage, REPRODUCED LOCALLY on the rack with an active pool. So MAAS virsh pods are incompatible with modules/node-vm's pool+volume disk refs; the pod would require converting node-vm to file-path disks and re-applying all nine domains. The pod is however UNNECESSARY -- its D-103 job was DISCOVERY, already done via PXE -- and per-machine power_type=virsh is MEASURED WORKING (query-power-state -> {"state":"off"} on the canary), which is also the Roosevelt shape (per-node IPMI). RULED 2026-07-20: per-machine virsh power. STEP D IS COMPLETE: shipped scripts/maas-node-power.sh + harness (24/24; gauntlet now 72 ALL GREEN), MAC-matched (MAAS renames machines at enlistment), dry-by-default, each write verified by a real query-power-state. All 9 nodes have power (docs/audit/stepD-power-20260720.txt) and commissioning works end to end -- 3 Ready / 6 Commissioning at time of writing, shapes still exact to D-121 Option C. opentofu/vr1-dc0-maas/ is retained but UNUSED (the pod route is refuted); retire-or-keep is a stage-close question, as is the D-103/D-123 amendment text. Session changelog items 15-17. COMMISSIONING PARTIAL (open, 2026-07-20): 3 nodes Ready, 6 timed out. Established: PXE and the ephemeral handoff WORK, and the ephemeral OS boots with working networking (nodes hold leases and do NTP to the rack) -- it simply never completes. Ruled out by measurement: memory, rack boot-image sync, DHCP, and node shape. The node->region path (SEC-010) is SUSPECTED but UNCONFIRMED (those rules carry no counters). A serial console was added to modules/node-vm and applied in-place to all 9, but the logs stay empty -- firmware writes to VGA, so serial alone does NOT make a PXE-booting node observable (correction queued). Two of the agent's own isolation experiments were INVALID and must not be cited (other nodes were still running; and a re-commission did not restart MAAS's timer) -- so contention remains a LIVE hypothesis, not a refuted one. Failed commissioning is re-runnable; nothing is lost. Session changelog item 19 records the clean experiment to run next. REMAINING IN G10: this diagnosis, then netem (step E, NOT started -- target is the dc0<->dc1 mesh = virbr5 on vcloud, measured; modules/netem-link assumes PASSWORDLESS SUDO which vcloud's operator account lacks). Session changelog item 18.
  • The grounding audit is COMPLETE and EXITED (2026-07-19): Phases 1-6 all closed (charter 148dcef; rulings docs/audit/ga-rulings.md; the Phase-5 sweep ran as six operator-gated batches in one session; exit runs docs/audit/phase6-exit-runs-20260719.md). The FREEZE is LIFTED -- normal change discipline (this document + the GA rulings) governs.
  • The vcloud host bookend (patch + reboot onto kernel -136, docs/dc0-deploy-readiness.md:107) HAS happened: host rebooted ~2026-07-17 23:39, both guests self-recovered via autostart (docs/audit/env-snapshot-20260718.md:10-16; re-measured this session, section 2.2 below).

2. What is APPLIED (from tofu state + live measurement, not from docs)

2.1 OpenTofu outer-root state (command: tofu -chdir=opentofu state list,

run 2026-07-18, 20 resources)

Office1 site (live, load-bearing):

  • module.voffice1.libvirt_domain.vm + .libvirt_volume.disk + .libvirt_volume.seed + .libvirt_cloudinit_disk.seed (BUT see divergence 2.3-i: the cloudinit staging ISO no longer exists live)
  • module.office1_opnsense.libvirt_domain.vm + .libvirt_volume.disk
  • module.office1_network.libvirt_network.office1_local
  • module.office1_storage.libvirt_pool.dc
  • module.ubuntu_noble_base.libvirt_volume.base

Inter-site fabric and DC scaffolding:

  • module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link, module.mesh_vr1_dc0_office1.libvirt_network.link, module.mesh_vr1_dc1_office1.libvirt_network.link (the D-100 mesh triangle)
  • module.vr1_dc0_planes.libvirt_network.plane["data-tenant" | "metal-admin" | "metal-internal" | "provider-public" | "replication" | "storage"] -- applied and live, but REMOVED from config (see 2.3-iii)
  • module.vr1_dc0_storage.libvirt_pool.dc, module.vr1_dc1_storage.libvirt_pool.dc

2.2 Live environment (measured this session, 2026-07-18, read-only)

  • hostname -> vcloud; uname -r -> 6.8.0-136-generic.
  • virsh list --all -> exactly two domains, both running: voffice1 (Id 1), office1-opnsense (Id 2).
  • virsh dominfo -> Autostart: enable on BOTH domains.
  • ssh voffice1 'snap list maas lxd; uname -r' -> maas 3.7.2-17972-g.35e297c4d rev 41649 (3.7/stable), lxd 5.21.5-f2a1a0e rev 40074 (5.21/stable, held), guest kernel 6.8.0-136-generic.
  • ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}" http://localhost:8000/' -> netbox=302 (service up, redirecting to login).
  • ssh office1-tailscale 'tailscale status | head -1' -> 100.64.0.53 office1-tailscale ... linux - (subnet-router VM up).
  • D-126 base leg: scripts/site-baseleg.sh check office1 passed at the Phase-1 snapshot (docs/audit/env-snapshot-20260718.md:16-17); not re-run this session.
  • OPNsense edge version: 26.7 per confirmed as-built (docs/vr1-office1-as-built.md:42, updated 2026-07-18). NOT re-measured this session -- measuring requires the gated API credential path; see section 7.

2.3 State vs live vs config disagreements (REPORTED, not harmonized)

i. module.voffice1.libvirt_cloudinit_disk.seed is IN STATE but its staging ISO was deleted by the reboot -- it still plans as a benign re-create (1 of section 5's 6 adds). The FORCED REPLACEMENT it used to force on libvirt_volume.seed (the GA-F01 defect that stopped the apply) is FIXED: D-130 ADOPTED (a) + implemented 2026-07-19, verified by the v8/v7 captures (gate rows G4/G5). Mechanism history: docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228. ii. Autostart: RESOLVED 2026-07-19 by the G6 state surgery (operator- ruled (ii), gate row G6): state now records autostart = true on both domains (state show | grep -c autostart -> 1 each); the 2 in-place changes are gone from the plan (section 5 capture). Guests were never touched. iii. The six vr1-dc0 plane networks exist live and in state but are REMOVED from config -- the INTENDED Model B relocation (planes get recreated inside vvr1-dc0 by the inner root, opentofu/vr1-dc0-substrate/main.tf:28). Their emptiness (0 leases, 0 attached domains) was verified in a PRIOR session (docs/dc0-deploy-readiness.md:43-45) and must be re-verified in the same session as any apply (finding doc:259-265). iv. RESOLVED 2026-07-19 (Batch 2, GA-F02): the readiness doc's falsified deploy-ready banner and its three contradictory plan counts are demoted -- status and the expected triple point HERE; the fresh- session banner points at the G9 canonical entry doc.

3. What is AUTHORED-BUT-NOT-APPLIED (in the tree, not in state / live)

  • (2026-07-20) The step-B transit-reach work is APPLIED and verified -- voffice1 holds the region end 172.31.0.1/30 on enp2s0 (plus its in-guest drop-in /etc/netplan/60-transit.yaml); vvr1-dc0 answers at 172.31.0.2 (netplan set-name root cause fixed, kernel names enp1s0/enp2s0 kept); ssh -J voffice1 with the dc0 key works; rack->region ping 10.10.0.20 0% loss. Kea reservation re-keyed to the regenerated voffice1 MAC (incident, session changelog item 3). Consequence for the G10 bootstrap: call site-headend-install.sh with --transit-if enp1s0 --uplink-if enp2s0.

  • module "vvr1_dc0" (opentofu/main.tf:360) -- the DC0 containment VM (416 GiB / 108 vCPU, D-121/D-123 sizing) + its disk, seed volume, and cloudinit seed. 4 of the 5 committed DC0 creates in the plan capture.

  • module "vr1_dc0_uplink" (opentofu/main.tf:341) -- the D-125 simulated ISP NAT network 172.30.2.0/24 (capture lines 226-247). The 5th create.
  • The ENTIRE inner root opentofu/vr1-dc0-substrate/ (main/variables/ versions.tf; no state file exists in that directory) -- inner storage pool, the six relocated planes, bridge-in WAN, and the rest of the Model B step-C build.
  • The removal of module "vr1_dc0_planes" from the outer config (the 6 intended destroys; see 2.3-iii).
  • autostart = true on voffice1/office1-opnsense in config (D-127) -- live-true but state-absent (2.3-ii).
  • scripts/opnsense-plugins.sh + tests/opnsense-plugins/ (D-129 profile-installer; commit 4cefa8b). BUILT and green; the live apply against the edge is an operator-gated firmware mutation, NOT run (docs/design-decisions.md:4050-4056).
  • SEC-010 enforcement artifact (scripts/site-headend-install.sh --host-nodes writing the transit FORWARD-drop) -- COMMITTED, applies on vvr1-dc0 at deploy step B; ledger row stays OPEN until applied+verified (bash scripts/ledger-scan.sh output, SEC-010 row).
  • Stage 4-7 runbooks (runbooks/dc-dc-phase3..6-*.md) -- written, not executed (docs/dc-dc-deployment-workflow.md:873).

4. What is RULED-BUT-NOT-BUILT (ADOPTED/RULED with no artifact yet)

  • D-121/D-122/D-123/D-124/D-125 (all ADOPTED/RULED, status lines at docs/design-decisions.md:3291,3474,3543,3635,3713): the DC0 deploy sequence they rule (steps A-E, docs/dc0-deploy-readiness.md:170-179) exists only as config + runbook. Nothing DC0 is built.
  • vr1-dc1: ruled at topology level (D-100/D-101 two-DC design) but HELD -- no ruled dc1 transit/rack addressing, no vr1_dc1_rack_* variables, no dc1 substrate root (docs/dc0-deploy-readiness.md:100-103). Only its storage pool and mesh legs exist (state list, 2.1).
  • D-100 netem: mechanism authored (opentofu/modules/netem-link) but HELD as a comment in the root (opentofu/main.tf:309-319); placeholder parameters ruled for the rehearsal (readiness doc:73-75); final parameters unruled (section 8).
  • D-129 Roosevelt metal-edge plugin profile (os-smart, os-nut | os-apcupsd, microcode, os-lldpd) -- recorded, inert until the Roosevelt edge build (docs/design-decisions.md:4030-4031).
  • D-129 qga enablement: designed (opt-in module variable, default OFF; the guest-agent virtio channel is MISSING from the live edge domain) but not written; retrofit deferred to the edge's next scheduled restart (docs/design-decisions.md:4041-4047,4057-4063).

5. The outer plan count (wording per operator direction, GA-F01)

The true EXPECTED outer plan count is currently NOBODY'S:

  • expected count: UNRESOLVED pending D-130;
  • last captured: 7/2/7 (docs/audit/outer-plan-20260718.txt, line 428: "Plan: 7 to add, 2 to change, 7 to destroy." -- the ONLY citable plan-count source);
  • pre-reboot gate was 5/0/6 (recorded at docs/dc0-deploy-readiness.md:59, docs/session-ledger.md:278).

The EXPECTED outer plan is ZERO DIFF ("no differences"), re-recorded 2026-07-20 with its evidencing capture (docs/audit/outer-plan-20260720-postfix-converged.txt) after two gated applies in the step-B window (both exact: voffice1 transit NIC 1/1/1, vvr1-dc0 seed replace 2/0/2 -- session changelog items 2-4, captures outer-{plan,apply}-20260719-voffice1-transit.txt and outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt). History: 7/2/7 post-reboot symptom -> 6/2/6 post-D-130 -> 6/0/6 post-G6-reconcile -> applied exact -> zero diff -> 1/1/1 (voffice1 transit, ruled+applied) -> 2/0/2 (rack netplan fix, applied) -> zero diff converged (this entry). A future outer plan showing ANY diff is a STOP (investigate drift before touching anything).

6. Open gates

Owner legend: operator (human ruling/approval), session (agent work under gating), external (outside this repo/track). Type legend (GA-R6/E1): [V] = verification-type (closes on its named executable check); [R] = ruling-type (closes per a GA-R5 recorded ruling).

# Gate What closes it Owner Evidence of current state
G1 Audit Phase 3: fresh-agent grounding test [V] 3 clean-context probes score the 7-question set against this doc; holes map made session CLOSED 2026-07-18: 3 probes, 21/21 PASS, holes H1 (amended into G9) + H2 (no action) -- docs/audit/phase3-grounding-test-20260718.md
G2 Audit Phase 4: GA-R1..R7 structural rulings + the stage-status vocabulary A/B [R] ruling-type gate (GA-R6 rule 6): closes when every item carries a GA-R5 Status block operator CLOSED 2026-07-18: all seven GA-R + vocabulary (Option A + H1) RATIFIED, utterances quoted (docs/audit/ga-rulings.md, through commit fe4f1c4 + this one)
G3 Audit Phase 5: repair sweep of GA-F01..F15 (incl. memory hygiene GA-F05..F08, skill sweep) [R] operator-gated fix batches, each commit naming its GA-F operator + session Batch 0 OPENED by operator 2026-07-19; items 0.1 (repo-lint L10, GA-R1/C1), 0.2 (SEC repoint, GA-R4/F3), 0.3 (counter hardening, GA-F15), 0.4 (extractor vocab scan, GA-F10/H1) landed; Batch 0 CLOSED (verification passed 2026-07-19); Batches 0-4 CLOSED 2026-07-19 (Batch 4: GA-R4 ledger rotation 1187->131 lines, F1 cap now enforceable; 96 changelogs + 24 history docs consolidated to docs/archive/ with 4 stage records + per-stage manifest commits; top-level docs/ = 16 files < 25; live-surface refs rewritten); Batch 5 CLOSED 2026-07-19 (skill sweep: GA section added reconciled against ratified text, stale phase/UNVALIDATED claims demoted, bookends + stage-close rewritten; checklist docs/audit/skill-sweep-checklist-20260719.md); Batch 6 OPEN: exit runs 1/2/4/5 PASS (adjudication + captures: docs/audit/phase6-exit-runs-20260719.md); exit item 3 PASS 2026-07-19 (fresh trio 21/21, 7/7 all three -- exit record); PENDING only item 6 (operator re-read + re-sign, replaces section 11); Batches 2-6 await gates; FREEZE holds for un-gated surfaces
G4 The two D-130 verifications [V] run them, capture output session CLOSED 2026-07-19: v8 suppression CONFIRMED (7/2/7 -> 6/2/6, zero forces-replacement; docs/audit/outer-plan-20260719-v8-ignorechanges.txt + -baseline.txt); v7 no-bounce under running domain, zero residue (docs/audit/throwaway-v7-20260719.txt)
G5 D-130 mechanism ruling (seed-volume durable fix) [R] operator rules in Phase 5, quoting G4's captured output operator CLOSED 2026-07-19: D-130 ADOPTED (a) ignore_changes (docs/design-decisions.md D-130, GA-R5 utterance quoted); implemented in modules/cloudinit-vm + tests/cloudinit-vm
G6 State reconcile of autostart + seed WITHOUT bouncing guests [R] gated mechanism, operator-ruled (S3) operator CLOSED 2026-07-19: ruled (ii) state surgery (GA-R5); pull -> inject autostart:true on both domains -> push (serial 22->23, backup terraform.tfstate.pre-G6-20260719); guests never touched (ids 1/2 unchanged, running)
G7 New captured plan == the expected triple recorded in section 5 [V] re-plan to a capture file after G5+G6 session CLOSED 2026-07-19: capture docs/audit/outer-plan-20260719-postG6.txt = 6/0/6, equals section 5 exactly
G8 Same-session pre-apply re-verify: 6 planes still empty [V] run in the SAME session as the apply session CLOSED 2026-07-19: verified in the apply session itself (all six 0 leases; only office1 nets attached) immediately before step A
G9 DC0 outer apply (deploy step A) [V] operator-gated, logged (run-logged.sh), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via site-headend-install.sh --host-nodes --check on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): runbooks/dc-dc-phase2-tofu-dc-substrate.md, with docs/dc0-deploy-readiness.md section E as the step table operator CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched
G10 Deploy steps B-E in-sequence gates: SEC-010 --host-nodes --check on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + TF_VAR_maas_api_key before step D; netem placeholder step E [V] exercised during the gated deploy session (each mutation operator-approved) Step B DONE 2026-07-20 (--check EXIT 0 incl. SEC-010, docs/audit/stepB-check-20260720-final.txt; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (docs/audit/d125-egress-gate-20260720-matrix.txt), and the edge itself now egresses 0% loss after the v4 addressing. REMAINING: D-129 plugin profile on 26.7, MAAS reach + key + region-side metal-admin DHCP (step D), netem (step E)
G11 Operator signs THIS document [R] read top-to-bottom; discrepancies resolved in the document operator CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature)
G12 vr1-dc1 build [R] operator rules dc1 transit/rack addressing; then vars + substrate authored operator HELD (docs/dc0-deploy-readiness.md:100-103)
G13 D-129 residuals [R] operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) operator OPEN / PARTIALLY RULED (docs/design-decisions.md:4017)
G14 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether opentofu/vr1-dc0-maas is retired operator / external docs/security-ledger.md (register of record, GA-R4/F3); count re-verified vs bash scripts/ledger-scan.sh 2026-07-20
G15 D-068 / D-071 rulings [R] operator rules (section 8); neither blocks the VR1 substrate operator PROPOSED/OPEN (status lines, section 8)

7. Version pins (measured; the authority for every pin)

Component Measured value Command (run 2026-07-18) Where measured
OpenTofu v1.12.4 tofu version vcloud (also docs/audit/env-snapshot-20260718.md:26)
libvirt provider dmacvicar/libvirt 0.9.8 (pinned) grep -A2 'provider' opentofu/.terraform.lock.hcl repo lock file
MAAS provider canonical/maas 2.7.2 (pinned) same repo lock file
MAAS 3.7.2-17972-g.35e297c4d (3.7/stable) ssh voffice1 'snap list maas' voffice1
LXD 5.21.5-f2a1a0e (5.21/stable, held) ssh voffice1 'snap list lxd' voffice1
Kernel (host) 6.8.0-136-generic uname -r vcloud
Kernel (voffice1) 6.8.0-136-generic ssh voffice1 'uname -r' voffice1
OPNsense edge 26.7 (FreeBSD base 15.1) not re-measured (gated API creds); per confirmed as-built docs/vr1-office1-as-built.md:42, 2026-07-18 office1-opnsense
NetBox (Office1 apex) 4.6.4 per as-built docs/vr1-office1-as-built.md:44; service UP verified (HTTP 302) this session ssh office1-netbox 'curl ... localhost:8000' office1-netbox
Juju NOT part of VR1 substrate yet (arrives Stage 5). Last recorded VR0 value 3.6.25 -- historical, unverified here n/a n/a

The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so the scan does not count them) were ALL fixed or demoted to pointers in sweep Batches 2-3, 2026-07-19 (session changelog).

8. Open decision queue (the exact questions the operator must answer)

  1. D-130: RULED 2026-07-19, ADOPTED (a) ignore_changes -- rotated to docs/design-decisions.md D-130 (question + utterance + captures).
  2. D-100 netem parameter sub-item (gap #11). Already ruled: same-metro dark-fiber default lean (D-100 sub-items, docs/design-decisions.md: 1962) and placeholder-for-the-rehearsal (readiness doc:73-75, operator ruling 2026-07-16). OPEN QUESTION: the final latency/jitter/loss/rate numbers -- supply the Roosevelt inter-DC target, or confirm the placeholder stands until that target exists. Evidence needed: the Roosevelt link spec (external input; nothing in-repo can answer it).
  3. D-068 -- Vault substrate hardening (PROPOSED/OPEN, docs/design-decisions.md:1457). QUESTION: rule items 2 (Vault listener TLS) and 3 (AppRole secret_id lifecycle) for Roosevelt; item 1 (Vault version) needs a re-scoped migration plan since the 1.16 forward-pin was proven NOT viable (amendment, :1650). Evidence in docs/D-068-vault-1.8-vs-1.16-analysis.md.
  4. D-071 -- update cadence + controller patch policy (PROPOSED, docs/design-decisions.md:1605). QUESTION: adopt policy points 1-4 (monthly-review window trigger; patch-only controller jumps; order; acceptable single-controller risk) as amended by the confirmed backup posture (:1677).
  5. D-129 -- four OPEN sub-decisions (OPEN / PARTIALLY RULED, docs/design-decisions.md:4017): (i) is the boundary edge in-scope for COS scraping (D-105 scope)? (ii) os-frr dynamic routing -- Roosevelt question? (iii) consolidate the Tailscale subnet-router onto the edge vs keep D-107 as-is? (iv) does the edge anchor site NTP? Plus the operator-gated live install of the ruled VR1 profile (scripts/opnsense-plugins.sh apply vr1-edge).
  6. Audit Phase 4 rulings: GA-R1..GA-R6 individually, plus the legal stage-status vocabulary A/B (GA-F10 operator note) -- each gated, one ruling per exchange.

9. Additional defect found while authoring (FIXED in sweep Batch 0.3,

2026-07-19 -- wrap-aware exclusion, GA-F15; history below)

ledger-scan.sh's mention-derived next-free counters (DOCFIX, BUNDLEFIX) are SELF-INFLATED by any doc that quotes a "next-free" value and lets the hyphenated token wrap onto a line without the words "next-free" -- the per-line exclusion filter (scripts/ledger-scan.sh:121-129, the grep -viE 'next[- ]free' at :124) then counts the quote as a real assignment; the script's own CAUTION comment (:112-120) documents exactly this failure class. It happened TWICE inside the audit itself on 2026-07-18: the Phase-1 env snapshot's wrapped next-free line inflated the BUNDLEFIX counter (051 -> reported 052), and this document's own first draft of this very section inflated the DOCFIX counter the same way while asserting DOCFIX was unaffected. Both audit surfaces were reworded token-free the same day and the counters re-verified at their true values (D=130, DOCFIX=197, BUNDLEFIX=052 -- see GA-F15). The D counter is header-authoritative and was never affected. Batch 0.3 hardened the scanner: an excluded line now also suppresses the immediately following line (the wrap case); counters re-verified unchanged post-fix. The authoring discipline stands regardless: never write a hyphenated register-token quote of a next-free value into any doc; state the numbers token-free as this section does.

10. How to verify this document (cold-session re-derivation)

Run read-only, from the repo root:

  • Repo identity: git rev-parse HEAD; git status --short (this doc was authored at e999b03, clean tree).
  • Applied set: tofu -chdir=opentofu state list (expect the 20 resources in section 2.1); virsh list --all; virsh dominfo voffice1 | grep -i autostart (and office1-opnsense).
  • Authored-not-applied: grep -n '^module ' opentofu/main.tf (12 blocks; vvr1_dc0 + vr1_dc0_uplink absent from state list); ls opentofu/vr1-dc0-substrate/ (no *.tfstate).
  • Plan count: read docs/audit/outer-plan-20260718.txt line 428. Do NOT re-run tofu plan casually against live state; if a fresh capture is taken, it must be written to a new dated capture file and cited here.
  • Versions: tofu version; ssh voffice1 'snap list maas lxd; uname -r' </dev/null; uname -r; grep -A2 provider opentofu/.terraform.lock.hcl.
  • Open decisions + SEC + next-free: bash scripts/ledger-scan.sh (BUNDLEFIX caveat: section 9); decision status lines: grep -n '^## D-' docs/design-decisions.md then read each Status line -- a decision's Status line in that file is the ONLY ruling authority.
  • Gates: sections 1a and 4-7 of docs/audit/grounding-audit-charter.md; docs/audit/grounding-audit-20260718.md for GA-F01..F14.
  • Live service probes: ssh office1-netbox 'curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/' </dev/null (expect 302); ssh office1-tailscale 'tailscale status | head -1' </dev/null.

What this document is NOT built from and you must not rebuild it from: the prose of the 95 docs/changelog-*.md files, the docs/session-ledger.md narrative, or auto-memory -- all proven to carry false status (GA-F14, GA-F06..F08).

11. Operator signature (G11)

SIGNED 2026-07-19 (re-signature at audit exit; REPLACES the 2026-07-18 signature per GA-R1 rule 7 -- git history keeps it). Question as presented (Batch 6 item 6, 2026-07-19): read this document top to bottom, then provide the signature statement. Operator answer, exact utterance: "Reviewed, approved, continue." This document is the signed status authority; charter Phase 6 item 5 MET at this baseline (repo HEAD at signing recorded in the close commit).