Authored 2026-07-18 by the grounding-audit Phase-2 ground-truth agent (charter: docs/audit/grounding-audit-charter.md, section 3) at repo HEAD e999b03 on branch dc-dc-stage3-phase2-dc-substrate. Every claim below carries its evidence (path:line, quoted command output, or commit hash). Claims that could not be evidenced read-only are marked UNKNOWN with what would resolve them. Nothing here is guessed.
STATUS OF THIS DOCUMENT: SIGNED by the operator 2026-07-19 (section 11; charter Phase 6 item 5). STANDING RULE (GA-R1, RATIFIED 2026-07-18 with amendments C1+C2 -- docs/audit/ga-rulings.md): no status claim is hand-written anywhere else; other documents point HERE; this document cites captured command output, and measurement always wins over it (C2). Other status surfaces are pointers or history; where one still carries a claim, it is a defect (Phase 1 proved they contradict: docs/audit/record-inventory.md, 12 groups; findings GA-F01..GA-F15).
docs/design-decisions.md:1946).docs/dc-dc-deployment-workflow.md:148; branch name dc-dc-stage3-phase2-dc-substrate; runbook runbooks/dc-dc-phase2-tofu-dc-substrate.md). Stages 0-2 precede it; stages 4-7 are authored, not executed (workflow doc:873).docs/audit/outer-plan-20260719-postA-converged.txt). Deploy step B (bootstrap) COMPLETE 2026-07-20 in the same logged dc0-deploy window: transit reach established (voffice1 holds 172.31.0.1/30; reach = ssh -J voffice1 w/ dc0 key -- no vcloud host leg, item-20 disposition), rack ENROLLED to the Office1 region, node-host ready (libvirt + nested KVM + inner pool), SEC-010 applied+verified BOTH transit ends (row CLOSED), OPNsense 26.7 nano base staged (operator ruling; step-C boot REVALIDATES the D-112/D-113 path on 26.7). Named gate check EXIT 0: docs/audit/stepB-check-20260720-final.txt. Deploy step C (inner apply) COMPLETE 2026-07-20: executed FROM voffice1 (D-128 Plane 2 -- tofu 1.12.4 + repo clone + dc0 key staged there), 28/28 resources, inner plan CONVERGED zero diff (docs/audit/inner-converge-20260720-stepC.txt); 10/10 domains RUNNING inside vvr1-dc0 (9 nodes + edge); edge = fresh 26.7 nano, serial log at the FreeBSD login prompt (D-112 boot path first-datapoint PASS on 26.7). The INNER tfstate lives ON voffice1 (vr1-dc0-substrate/terraform.tfstate -- new state-of-record location; add to the site backup set). ACTIVE gate: G10 remaining. Edge bootstrap DONE 2026-07-20: D-112(c) console bootstrap complete (key-only root SSH proven) and the D-113(a2) API key minted via the vendor model -- GET core/firmware/status 200 with CORE_ABI 26.7, the first proof the API path works on 26.7. Measured: edge vtnet0 = LAN (provider-public), vtnet1 = WAN; edge still on its FACTORY LAN 192.168.1.1/24. Rack legs 10.12.4.2/22 + 10.12.8.2/22 added INTERIM (non-persistent ip addr; script support is a queued finding), plus a temporary 192.168.1.2/22 to reach the factory LAN. D-125 egress isolation gate: PASS / CLOSED 2026-07-20 (executed as written -- throwaway VM on br-vr1-dc0-wan; two identical consecutive runs: gateway ping 0, internet ping 0, curl 1.1.1.1 301, curl archive.ubuntu.com 200). Bridge-in is PROVEN end to end and the double-NAT fallback is NOT needed. Captures: docs/audit/d125-egress-gate-20260720{,-matrix}.txt. One earlier run failed ICMP-to-internet on the same path and is recorded UNEXPLAINED in the session changelog (start there if a DC edge shows first-boot egress failure). Edge ADDRESSED 2026-07-20 via the NEW operator-ruled opnsense-set-interface-v4 pair (D-113 amendment re-measured and still true on 26.7 -- base-iface addressing is not REST-covered): WAN 172.30.2.2/24 + default gw 172.30.2.1 (was dhcp, which could never work on a /24 with no DHCP server), LAN 192.168.1.1/24 -> 10.12.4.1/22 (ruled provider-public gateway). Verified on the kernel; the edge itself egresses to 1.1.1.1 at 0% loss, and the API answers at the new LAN address. Interim bootstrap address removed; virbr5 now carries only the ruled 10.12.4.2/22. D-129 edge profile APPLIED 2026-07-20 on 26.7 (operator-ruled): expose_qga_channel shipped in modules/opnsense-edge (opt-in, default OFF; dc0 true) and applied as an IN-PLACE domain update; os-qemu-guest-agent + os-iperf installed for real and the agent ANSWERS -- guest-ping -> {"return":{}} and domifaddr --source agent reports both legs. Note this run also exposed and fixed a false-success bug: opnsense-plugins.sh apply had ALWAYS dry-run (see session changelog item 12), so any prior "applied" claim from that script is void. Step D part 1 DONE 2026-07-20: rack registered (7chphy, rackd running), metal-admin dynamic range 10.12.8.100-.200 created (operator-ruled D-120 inheritance), VLAN 5005 dhcp_on=true primary_rack=7chphy verified by read-back. INCIDENT RESOLVED 2026-07-20 (operator-approved region restart): dhcpd now RUNNING on both controllers (verified by process, not service status), and all 9 DC0 nodes ENLISTED in MAAS with shapes exactly matching D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB) -- docs/audit/stepD-enlistment-20260720.txt. The G10 depth-4 nested boot gate is therefore PASS: node VMs inside vvr1-dc0 PXE-booted from the Office1 region across the transit and run MAAS's ephemeral kernel. The incident as originally found: MAAS 3.7 drives DHCP via Temporal, and Temporal is wedged on the region ("Not enough hosts to serve the request", 2807 retries), so no dhcpd runs on EITHER controller -- including voffice1 itself, whose compose net reads dhcp=True with no dhcpd process. Predates and is NOT caused by this deploy (almost certainly since the 2026-07-17 host reboot); unnoticed because both Office1 VMs were already Deployed. Any "Office1 MAAS DHCP working" claim is currently FALSE. Proposed gated remedy: restart MAAS on the region -- DONE, and it fixed BOTH sites, confirming a single root cause. Details + the queued detection-gap finding (cloud-assert trusts MAAS's self-report and missed a dead DHCP server): session changelog items 13-14; appendix-A entry queued. Step D part 2 BLOCKED on a ruling (2026-07-20): the nine nodes fell back to New with no power_type -- commissioning cannot finish without power control. New root opentofu/vr1-dc0-maas/ is shipped and its plan is clean, but the apply FAILED: Failed talking to pod: Failed to login to virsh console. MEASURED cause -- the MAAS snap is confined, gets Permission denied on /var/run/libvirt/libvirt-sock, and snap connections maas lists NO libvirt interface, so a LOCAL qemu:///system pod is IMPOSSIBLE with snap MAAS. This refutes the mechanism stated in D-123 Model B and in modules/maas-vm-host's header (intent survives, mechanism does not); both need an amendment once the replacement is ruled. The qemu+ssh replacement was then wired with an operator-ruled DEDICATED key and PROVEN reachable from both snaps -- but the pod apply failed again, finally on domblkinfo ... missing storage backend for 'volume' storage, REPRODUCED LOCALLY on the rack with an active pool. So MAAS virsh pods are incompatible with modules/node-vm's pool+volume disk refs; the pod would require converting node-vm to file-path disks and re-applying all nine domains. The pod is however UNNECESSARY -- its D-103 job was DISCOVERY, already done via PXE -- and per-machine power_type=virsh is MEASURED WORKING (query-power-state -> {"state":"off"} on the canary), which is also the Roosevelt shape (per-node IPMI). RULED 2026-07-20: per-machine virsh power. STEP D IS COMPLETE: shipped scripts/maas-node-power.sh + harness (24/24; gauntlet now 72 ALL GREEN), MAC-matched (MAAS renames machines at enlistment), dry-by-default, each write verified by a real query-power-state. All 9 nodes have power (docs/audit/stepD-power-20260720.txt) and commissioning works end to end -- 3 Ready / 6 Commissioning at time of writing, shapes still exact to D-121 Option C. opentofu/vr1-dc0-maas/ is retained but UNUSED (the pod route is refuted); retire-or-keep is a stage-close question, as is the D-103/D-123 amendment text. Session changelog items 15-17. COMMISSIONING RESOLVED 2026-07-21: all 9 nodes Ready (logged window ops-commissioning-diag; adjudication docs/audit/commissioning-diag-20260721.txt; session changelog 2026-07-21). TWO stacked faults, both measured: (1) the 2026-07-20 in-place serial-console apply REGENERATED all 9 node NIC MACs (tofu-reported 0/9/0 in-place), so MAAS's records went stale and every post-apply boot was an unknown node -- no PXE event, no tag kernel_opts, silent 30-min timeout; repaired operator-ruled via per-machine boot-interface MAC update (mark-broken/update/mark-fixed where needed), read-back verified 9/9. (2) Beneath it, the MAAS 3.7 RACK-ONLY agent resolver SERVFAILs every query on an internet-isolated rack (walks public root hints even for its own authoritative maas-internal zone; ignores resolv.conf), so cloud-init's cloud-config-url never resolved and nodes booted to a login prompt without ever fetching commissioning scripts. Office1/VR0 were immune (co-located region BIND owns node DNS) -- this surface is FIRST EXERCISED in VR1; LP report queued. Operator-ruled workaround, live and proven: dc0-node-dns.service on the rack (dnsmasq on virbr2 alias 10.12.8.3 forwarding to region BIND over the rack's OWN transit connection; SEC-010 re-verified enforced and untouched) + metal-admin subnet dns_servers=10.12.8.3, allow_dns=false. PROOF: canary Ready in ~3 min after seven consecutive 30-min failures, commissioning scripts visible on serial; fleet of 8 re-commissioned concurrently, ALL 9 READY in ~4 min, shapes exact to D-121 Option C. Committee record closed by addendum (its mechanisms were wrong; its instrument found the cause). D-131 PARTIALLY RULED (sub-1 RULED 2026-07-21: the forwarder is the STANDING per-DC pattern, repo-carried + part of DC standup definition-of-done; sub-2..4 OPEN -- status line in design-decisions.md is the authority). SEC-014 OPENED (rack cluster secret exposure during diagnosis). Queued delivery: incident docs SHIPPED 2026-07-21 (two appendix-A entries, platform-traps 1e second corollary + index row, LP draft docs/audit/lp-draft-20260721-maas-agent-resolver.md -- operator to file). Still queued: stale pod object cleanup (stage close, with SEC-013). Forwarder + rack-legs persistence SHIPPED 2026-07-21 as scripts/dc-rack-net.sh (D-131 sub-1 delivery; harness 14 cases; gauntlet 74 ALL GREEN) and INSTALLED on the rack 2026-07-21 (operator-approved): install EXIT 0, self-check PASS 10/10 (docs/audit/dc-rack-net-install-20260721.txt), post-install behavioral probe = forwarder answers authoritative maas-internal SOA. The three rack bridge legs are now reboot-persistent (dc0-rack-legs.service); the hand-placed interim state is fully superseded. MAC pinning SHIPPED 2026-07-21 (54 MACs measured via virsh domiflist + pinned in modules/node-vm + vr1-dc0-substrate; harness 15 cases; gauntlet 73 ALL GREEN) together with an operator-ruled power-ownership guard (ignore_changes = [running] -- MAAS owns node power; the pin-adoption plan had carried 9 out-of-band power-ons). Verification plan captured (docs/audit/inner-plan-20260721-macpin.txt: 0/9/0, 54 mac adoptions, ZERO replaces); guarded re-plan zero power flips (docs/audit/inner-plan-20260721-macpin-guarded.txt); APPLIED 2026-07-21 (operator-approved) from voffice1 via saved plan, exact 0/9/0, convergence zero diff (docs/audit/inner-apply-20260721-macpin.txt); post-apply verified all 9 domains still shut off, MACs unchanged. Node NIC MACs are now config-pinned end to end. History of the diagnosis (superseded; kept for the audit trail): the 2026-07-20 state read "3 nodes Ready, 6 timed out." Established: PXE and the ephemeral handoff WORK, and the ephemeral OS boots with working networking (nodes hold leases and do NTP to the rack) -- it simply never completes. Ruled out by measurement: memory, rack boot-image sync, DHCP, and node shape. The node->region path (SEC-010) is SUSPECTED but UNCONFIRMED (those rules carry no counters). A serial console was added to modules/node-vm and applied in-place to all 9, but the logs stay empty -- firmware writes to VGA, so serial alone does NOT make a PXE-booting node observable (correction queued). Two of the agent's own isolation experiments were INVALID and must not be cited (other nodes were still running; and a re-commission did not restart MAAS's timer) -- so contention remains a LIVE hypothesis, not a refuted one. CLEAN experiment now RUN (item 20): a genuinely isolated node still failed at 1770s (~29.5 of 30 min) -- that refutes CONTENTION but is consistent with INHERENTLY SLOW, and the batch pattern 3-pass/6-fail-at-the-mark is the signature of a MARGINAL 30-min timeout over slow depth-4 nested I/O. 3 nodes reached Ready on this exact rack/subnet/metadata path, so metadata is NOT globally broken (rack :5248 up, rack->region 301). LEADING HYPOTHESIS + cheap decisive test, needing an operator decision (MAAS-wide config): raise node_timeout and commission one node. DIAGNOSTIC COMMITTEE run 2026-07-20 (4 independent reviewers, docs/audit/commissioning-committee-20260720.md) REFUTED that hypothesis 4/4 -- 30 min of SILENCE is a hang, not slow progress; a longer clock cannot fix a hang, and the proposed one-node test was CONFOUNDED (changed timeout + concurrency together). Post-committee reads: MTU branch EXONERATED (metal-admin MAAS VLAN MTU is 1500, so the guest never goes jumbo); region healthy at rest. STILL-LIVE causes, both needing observation DURING a run: region Temporal starvation, and a commissioning-only script hang on nested-virt hardware. Decisive gated test (supersedes node_timeout): one commission with console=ttyS0 on the kernel + a full-window, lease-IP-keyed capture on virbr2 + enp1s0. Failed commissioning is re-runnable; nothing is lost. The committee record (docs/audit/commissioning-committee-20260720.md) is the durable authority for this diagnosis and its ranked live hypotheses. REMAINING IN G10: netem only (step E, NOT started -- target is the dc0<->dc1 mesh = virbr5 on vcloud, re-measured 2026-07-21). The sudo blocker is RESOLVED in mechanism: operator-ruled scoped NOPASSWD 2026-07-21, fragment SHIPPED (scripts/sudoers.d/netem-tc, harness 9 cases, gauntlet 75 ALL GREEN); INSTALL on vcloud is operator-only and PENDING, then the step-E placeholder netem run itself (gated). Session changelog 2026-07-21 item 7.148dcef; rulings docs/audit/ga-rulings.md; the Phase-5 sweep ran as six operator-gated batches in one session; exit runs docs/audit/phase6-exit-runs-20260719.md). The FREEZE is LIFTED -- normal change discipline (this document + the GA rulings) governs.docs/dc0-deploy-readiness.md:107) HAS happened: host rebooted ~2026-07-17 23:39, both guests self-recovered via autostart (docs/audit/env-snapshot-20260718.md:10-16; re-measured this session, section 2.2 below).tofu -chdir=opentofu state list,run 2026-07-18, 20 resources)
Office1 site (live, load-bearing):
module.voffice1.libvirt_domain.vm + .libvirt_volume.disk + .libvirt_volume.seed + .libvirt_cloudinit_disk.seed (BUT see divergence 2.3-i: the cloudinit staging ISO no longer exists live)module.office1_opnsense.libvirt_domain.vm + .libvirt_volume.diskmodule.office1_network.libvirt_network.office1_localmodule.office1_storage.libvirt_pool.dcmodule.ubuntu_noble_base.libvirt_volume.baseInter-site fabric and DC scaffolding:
module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link, module.mesh_vr1_dc0_office1.libvirt_network.link, module.mesh_vr1_dc1_office1.libvirt_network.link (the D-100 mesh triangle)module.vr1_dc0_planes.libvirt_network.plane["data-tenant" | "metal-admin" | "metal-internal" | "provider-public" | "replication" | "storage"] -- applied and live, but REMOVED from config (see 2.3-iii)module.vr1_dc0_storage.libvirt_pool.dc, module.vr1_dc1_storage.libvirt_pool.dchostname -> vcloud; uname -r -> 6.8.0-136-generic.virsh list --all -> exactly two domains, both running: voffice1 (Id 1), office1-opnsense (Id 2).virsh dominfo -> Autostart: enable on BOTH domains.ssh voffice1 'snap list maas lxd; uname -r' -> maas 3.7.2-17972-g.35e297c4d rev 41649 (3.7/stable), lxd 5.21.5-f2a1a0e rev 40074 (5.21/stable, held), guest kernel 6.8.0-136-generic.ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}" http://localhost:8000/' -> netbox=302 (service up, redirecting to login).ssh office1-tailscale 'tailscale status | head -1' -> 100.64.0.53 office1-tailscale ... linux - (subnet-router VM up).scripts/site-baseleg.sh check office1 passed at the Phase-1 snapshot (docs/audit/env-snapshot-20260718.md:16-17); not re-run this session.docs/vr1-office1-as-built.md:42, updated 2026-07-18). NOT re-measured this session -- measuring requires the gated API credential path; see section 7.i. module.voffice1.libvirt_cloudinit_disk.seed is IN STATE but its staging ISO was deleted by the reboot -- it still plans as a benign re-create (1 of section 5's 6 adds). The FORCED REPLACEMENT it used to force on libvirt_volume.seed (the GA-F01 defect that stopped the apply) is FIXED: D-130 ADOPTED (a) + implemented 2026-07-19, verified by the v8/v7 captures (gate rows G4/G5). Mechanism history: docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228. ii. Autostart: RESOLVED 2026-07-19 by the G6 state surgery (operator- ruled (ii), gate row G6): state now records autostart = true on both domains (state show | grep -c autostart -> 1 each); the 2 in-place changes are gone from the plan (section 5 capture). Guests were never touched. iii. The six vr1-dc0 plane networks exist live and in state but are REMOVED from config -- the INTENDED Model B relocation (planes get recreated inside vvr1-dc0 by the inner root, opentofu/vr1-dc0-substrate/main.tf:28). Their emptiness (0 leases, 0 attached domains) was verified in a PRIOR session (docs/dc0-deploy-readiness.md:43-45) and must be re-verified in the same session as any apply (finding doc:259-265). iv. RESOLVED 2026-07-19 (Batch 2, GA-F02): the readiness doc's falsified deploy-ready banner and its three contradictory plan counts are demoted -- status and the expected triple point HERE; the fresh- session banner points at the G9 canonical entry doc.
(2026-07-20) The step-B transit-reach work is APPLIED and verified -- voffice1 holds the region end 172.31.0.1/30 on enp2s0 (plus its in-guest drop-in /etc/netplan/60-transit.yaml); vvr1-dc0 answers at 172.31.0.2 (netplan set-name root cause fixed, kernel names enp1s0/enp2s0 kept); ssh -J voffice1 with the dc0 key works; rack->region ping 10.10.0.20 0% loss. Kea reservation re-keyed to the regenerated voffice1 MAC (incident, session changelog item 3). Consequence for the G10 bootstrap: call site-headend-install.sh with --transit-if enp1s0 --uplink-if enp2s0.
module "vvr1_dc0" (opentofu/main.tf:360) -- the DC0 containment VM (416 GiB / 108 vCPU, D-121/D-123 sizing) + its disk, seed volume, and cloudinit seed. 4 of the 5 committed DC0 creates in the plan capture.
module "vr1_dc0_uplink" (opentofu/main.tf:341) -- the D-125 simulated ISP NAT network 172.30.2.0/24 (capture lines 226-247). The 5th create.opentofu/vr1-dc0-substrate/ (main/variables/ versions.tf; no state file exists in that directory) -- inner storage pool, the six relocated planes, bridge-in WAN, and the rest of the Model B step-C build.module "vr1_dc0_planes" from the outer config (the 6 intended destroys; see 2.3-iii).autostart = true on voffice1/office1-opnsense in config (D-127) -- live-true but state-absent (2.3-ii).scripts/opnsense-plugins.sh + tests/opnsense-plugins/ (D-129 profile-installer; commit 4cefa8b). BUILT and green; the live apply against the edge is an operator-gated firmware mutation, NOT run (docs/design-decisions.md:4050-4056).scripts/site-headend-install.sh --host-nodes writing the transit FORWARD-drop) -- COMMITTED, applies on vvr1-dc0 at deploy step B; ledger row stays OPEN until applied+verified (bash scripts/ledger-scan.sh output, SEC-010 row).runbooks/dc-dc-phase3..6-*.md) -- written, not executed (docs/dc-dc-deployment-workflow.md:873).docs/design-decisions.md:3291,3474,3543,3635,3713): the DC0 deploy sequence they rule (steps A-E, docs/dc0-deploy-readiness.md:170-179) exists only as config + runbook. Nothing DC0 is built.vr1-dc1: ruled at topology level (D-100/D-101 two-DC design) but HELD -- no ruled dc1 transit/rack addressing, no vr1_dc1_rack_* variables, no dc1 substrate root (docs/dc0-deploy-readiness.md:100-103). Only its storage pool and mesh legs exist (state list, 2.1).opentofu/modules/netem-link) but HELD as a comment in the root (opentofu/main.tf:309-319); placeholder parameters ruled for the rehearsal (readiness doc:73-75); final parameters unruled (section 8).os-smart, os-nut | os-apcupsd, microcode, os-lldpd) -- recorded, inert until the Roosevelt edge build (docs/design-decisions.md:4030-4031).docs/design-decisions.md:4041-4047,4057-4063).The true EXPECTED outer plan count is currently NOBODY'S:
docs/audit/outer-plan-20260718.txt, line 428: "Plan: 7 to add, 2 to change, 7 to destroy." -- the ONLY citable plan-count source);docs/dc0-deploy-readiness.md:59, docs/session-ledger.md:278).The EXPECTED outer plan is ZERO DIFF ("no differences"), re-recorded 2026-07-20 with its evidencing capture (docs/audit/outer-plan-20260720-postfix-converged.txt) after two gated applies in the step-B window (both exact: voffice1 transit NIC 1/1/1, vvr1-dc0 seed replace 2/0/2 -- session changelog items 2-4, captures outer-{plan,apply}-20260719-voffice1-transit.txt and outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt). History: 7/2/7 post-reboot symptom -> 6/2/6 post-D-130 -> 6/0/6 post-G6-reconcile -> applied exact -> zero diff -> 1/1/1 (voffice1 transit, ruled+applied) -> 2/0/2 (rack netplan fix, applied) -> zero diff converged (this entry). A future outer plan showing ANY diff is a STOP (investigate drift before touching anything).
Owner legend: operator (human ruling/approval), session (agent work under gating), external (outside this repo/track). Type legend (GA-R6/E1): [V] = verification-type (closes on its named executable check); [R] = ruling-type (closes per a GA-R5 recorded ruling).
| # | Gate | What closes it | Owner | Evidence of current state |
|---|---|---|---|---|
| G1 | Audit Phase 3: fresh-agent grounding test | [V] 3 clean-context probes score the 7-question set against this doc; holes map made | session | CLOSED 2026-07-18: 3 probes, 21/21 PASS, holes H1 (amended into G9) + H2 (no action) -- docs/audit/phase3-grounding-test-20260718.md |
| G2 | Audit Phase 4: GA-R1..R7 structural rulings + the stage-status vocabulary A/B | [R] ruling-type gate (GA-R6 rule 6): closes when every item carries a GA-R5 Status block | operator | CLOSED 2026-07-18: all seven GA-R + vocabulary (Option A + H1) RATIFIED, utterances quoted (docs/audit/ga-rulings.md, through commit fe4f1c4 + this one) |
| G3 | Audit Phase 5: repair sweep of GA-F01..F15 (incl. memory hygiene GA-F05..F08, skill sweep) | [R] operator-gated fix batches, each commit naming its GA-F | operator + session | Batch 0 OPENED by operator 2026-07-19; items 0.1 (repo-lint L10, GA-R1/C1), 0.2 (SEC repoint, GA-R4/F3), 0.3 (counter hardening, GA-F15), 0.4 (extractor vocab scan, GA-F10/H1) landed; Batch 0 CLOSED (verification passed 2026-07-19); Batches 0-4 CLOSED 2026-07-19 (Batch 4: GA-R4 ledger rotation 1187->131 lines, F1 cap now enforceable; 96 changelogs + 24 history docs consolidated to docs/archive/ with 4 stage records + per-stage manifest commits; top-level docs/ = 16 files < 25; live-surface refs rewritten); Batch 5 CLOSED 2026-07-19 (skill sweep: GA section added reconciled against ratified text, stale phase/UNVALIDATED claims demoted, bookends + stage-close rewritten; checklist docs/audit/skill-sweep-checklist-20260719.md); Batch 6 OPEN: exit runs 1/2/4/5 PASS (adjudication + captures: docs/audit/phase6-exit-runs-20260719.md); exit item 3 PASS 2026-07-19 (fresh trio 21/21, 7/7 all three -- exit record); PENDING only item 6 (operator re-read + re-sign, replaces section 11); Batches 2-6 await gates; FREEZE holds for un-gated surfaces |
| G4 | The two D-130 verifications | [V] run them, capture output | session | CLOSED 2026-07-19: v8 suppression CONFIRMED (7/2/7 -> 6/2/6, zero forces-replacement; docs/audit/outer-plan-20260719-v8-ignorechanges.txt + -baseline.txt); v7 no-bounce under running domain, zero residue (docs/audit/throwaway-v7-20260719.txt) |
| G5 | D-130 mechanism ruling (seed-volume durable fix) | [R] operator rules in Phase 5, quoting G4's captured output | operator | CLOSED 2026-07-19: D-130 ADOPTED (a) ignore_changes (docs/design-decisions.md D-130, GA-R5 utterance quoted); implemented in modules/cloudinit-vm + tests/cloudinit-vm |
| G6 | State reconcile of autostart + seed WITHOUT bouncing guests | [R] gated mechanism, operator-ruled (S3) | operator | CLOSED 2026-07-19: ruled (ii) state surgery (GA-R5); pull -> inject autostart:true on both domains -> push (serial 22->23, backup terraform.tfstate.pre-G6-20260719); guests never touched (ids 1/2 unchanged, running) |
| G7 | New captured plan == the expected triple recorded in section 5 | [V] re-plan to a capture file after G5+G6 | session | CLOSED 2026-07-19: capture docs/audit/outer-plan-20260719-postG6.txt = 6/0/6, equals section 5 exactly |
| G8 | Same-session pre-apply re-verify: 6 planes still empty | [V] run in the SAME session as the apply | session | CLOSED 2026-07-19: verified in the apply session itself (all six 0 leases; only office1 nets attached) immediately before step A |
| G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (run-logged.sh), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via site-headend-install.sh --host-nodes --check on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): runbooks/dc-dc-phase2-tofu-dc-substrate.md, with docs/dc0-deploy-readiness.md section E as the step table |
operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched |
| G10 | Deploy steps B-E in-sequence gates: SEC-010 --host-nodes --check on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + TF_VAR_maas_api_key before step D; netem placeholder step E |
[V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (--check EXIT 0 incl. SEC-010, docs/audit/stepB-check-20260720-final.txt; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (docs/audit/d125-egress-gate-20260720-matrix.txt), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- docs/audit/commissioning-diag-20260721.txt; section 1). REMAINING: netem (step E) |
| G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) |
| G12 | vr1-dc1 build |
[R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator | HELD (docs/dc0-deploy-readiness.md:100-103) |
| G13 | D-129 residuals | [R] operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) | operator | OPEN / PARTIALLY RULED (docs/design-decisions.md:4017) |
| G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether opentofu/vr1-dc0-maas is retired |
operator / external | docs/security-ledger.md (register of record, GA-R4/F3); count re-verified vs bash scripts/ledger-scan.sh 2026-07-20 |
| G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | PROPOSED/OPEN (status lines, section 8) |
| Component | Measured value | Command (run 2026-07-18) | Where measured |
|---|---|---|---|
| OpenTofu | v1.12.4 | tofu version |
vcloud (also docs/audit/env-snapshot-20260718.md:26) |
| libvirt provider | dmacvicar/libvirt 0.9.8 (pinned) | grep -A2 'provider' opentofu/.terraform.lock.hcl |
repo lock file |
| MAAS provider | canonical/maas 2.7.2 (pinned) | same | repo lock file |
| MAAS | 3.7.2-17972-g.35e297c4d (3.7/stable) | ssh voffice1 'snap list maas' |
voffice1 |
| LXD | 5.21.5-f2a1a0e (5.21/stable, held) | ssh voffice1 'snap list lxd' |
voffice1 |
| Kernel (host) | 6.8.0-136-generic | uname -r |
vcloud |
| Kernel (voffice1) | 6.8.0-136-generic | ssh voffice1 'uname -r' |
voffice1 |
| OPNsense edge | 26.7 (FreeBSD base 15.1) | not re-measured (gated API creds); per confirmed as-built docs/vr1-office1-as-built.md:42, 2026-07-18 |
office1-opnsense |
| NetBox (Office1 apex) | 4.6.4 per as-built docs/vr1-office1-as-built.md:44; service UP verified (HTTP 302) this session |
ssh office1-netbox 'curl ... localhost:8000' |
office1-netbox |
| Juju | NOT part of VR1 substrate yet (arrives Stage 5). Last recorded VR0 value 3.6.25 -- historical, unverified here | n/a | n/a |
The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so the scan does not count them) were ALL fixed or demoted to pointers in sweep Batches 2-3, 2026-07-19 (session changelog).
docs/design-decisions.md D-130 (question + utterance + captures).docs/design-decisions.md: 1962) and placeholder-for-the-rehearsal (readiness doc:73-75, operator ruling 2026-07-16). OPEN QUESTION: the final latency/jitter/loss/rate numbers -- supply the Roosevelt inter-DC target, or confirm the placeholder stands until that target exists. Evidence needed: the Roosevelt link spec (external input; nothing in-repo can answer it).docs/design-decisions.md:1457). QUESTION: rule items 2 (Vault listener TLS) and 3 (AppRole secret_id lifecycle) for Roosevelt; item 1 (Vault version) needs a re-scoped migration plan since the 1.16 forward-pin was proven NOT viable (amendment, :1650). Evidence in docs/D-068-vault-1.8-vs-1.16-analysis.md.docs/design-decisions.md:1605). QUESTION: adopt policy points 1-4 (monthly-review window trigger; patch-only controller jumps; order; acceptable single-controller risk) as amended by the confirmed backup posture (:1677).docs/design-decisions.md:4017): (i) is the boundary edge in-scope for COS scraping (D-105 scope)? (ii) os-frr dynamic routing -- Roosevelt question? (iii) consolidate the Tailscale subnet-router onto the edge vs keep D-107 as-is? (iv) does the edge anchor site NTP? Plus the operator-gated live install of the ruled VR1 profile (scripts/opnsense-plugins.sh apply vr1-edge).2026-07-19 -- wrap-aware exclusion, GA-F15; history below)
ledger-scan.sh's mention-derived next-free counters (DOCFIX, BUNDLEFIX) are SELF-INFLATED by any doc that quotes a "next-free" value and lets the hyphenated token wrap onto a line without the words "next-free" -- the per-line exclusion filter (scripts/ledger-scan.sh:121-129, the grep -viE 'next[- ]free' at :124) then counts the quote as a real assignment; the script's own CAUTION comment (:112-120) documents exactly this failure class. It happened TWICE inside the audit itself on 2026-07-18: the Phase-1 env snapshot's wrapped next-free line inflated the BUNDLEFIX counter (051 -> reported 052), and this document's own first draft of this very section inflated the DOCFIX counter the same way while asserting DOCFIX was unaffected. Both audit surfaces were reworded token-free the same day and the counters re-verified at their true values (D=130, DOCFIX=197, BUNDLEFIX=052 -- see GA-F15). The D counter is header-authoritative and was never affected. Batch 0.3 hardened the scanner: an excluded line now also suppresses the immediately following line (the wrap case); counters re-verified unchanged post-fix. The authoring discipline stands regardless: never write a hyphenated register-token quote of a next-free value into any doc; state the numbers token-free as this section does.
Run read-only, from the repo root:
git rev-parse HEAD; git status --short (this doc was authored at e999b03, clean tree).tofu -chdir=opentofu state list (expect the 20 resources in section 2.1); virsh list --all; virsh dominfo voffice1 | grep -i autostart (and office1-opnsense).grep -n '^module ' opentofu/main.tf (12 blocks; vvr1_dc0 + vr1_dc0_uplink absent from state list); ls opentofu/vr1-dc0-substrate/ (no *.tfstate).docs/audit/outer-plan-20260718.txt line 428. Do NOT re-run tofu plan casually against live state; if a fresh capture is taken, it must be written to a new dated capture file and cited here.tofu version; ssh voffice1 'snap list maas lxd; uname -r' </dev/null; uname -r; grep -A2 provider opentofu/.terraform.lock.hcl.bash scripts/ledger-scan.sh (BUNDLEFIX caveat: section 9); decision status lines: grep -n '^## D-' docs/design-decisions.md then read each Status line -- a decision's Status line in that file is the ONLY ruling authority.docs/audit/grounding-audit-charter.md; docs/audit/grounding-audit-20260718.md for GA-F01..F14.ssh office1-netbox 'curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/' </dev/null (expect 302); ssh office1-tailscale 'tailscale status | head -1' </dev/null.What this document is NOT built from and you must not rebuild it from: the prose of the 95 docs/changelog-*.md files, the docs/session-ledger.md narrative, or auto-memory -- all proven to carry false status (GA-F14, GA-F06..F08).
SIGNED 2026-07-19 (re-signature at audit exit; REPLACES the 2026-07-18 signature per GA-R1 rule 7 -- git history keeps it). Question as presented (Batch 6 item 6, 2026-07-19): read this document top to bottom, then provide the signature statement. Operator answer, exact utterance: "Reviewed, approved, continue." This document is the signed status authority; charter Phase 6 item 5 MET at this baseline (repo HEAD at signing recorded in the close commit).