Authored 2026-07-18 by the grounding-audit Phase-2 ground-truth agent (charter: docs/audit/grounding-audit-charter.md, section 3) at repo HEAD e999b03 on branch dc-dc-stage3-phase2-dc-substrate. Every claim below carries its evidence (path:line, quoted command output, or commit hash). Claims that could not be evidenced read-only are marked UNKNOWN with what would resolve them. Nothing here is guessed.
STATUS OF THIS DOCUMENT: SIGNED by the operator 2026-07-18 (section 11; charter Phase 6 item 5). Proposed standing rule (GA-R1, NOT yet ratified): no status claim may be hand-written anywhere else; other documents point HERE; this document cites captured command output. Until GA-R1 is ratified, treat every other status surface (readiness doc banners, workflow-doc stage rows, ledger narrative, changelogs, README pins, skill version tables, auto-memory) as UNTRUSTED for status -- Phase 1 proved they contradict each other (docs/audit/record-inventory.md: 12 contradiction groups; findings GA-F01..GA-F14 in docs/audit/grounding-audit-20260718.md).
docs/design-decisions.md:1946).docs/dc-dc-deployment-workflow.md:148; branch name dc-dc-stage3-phase2-dc-substrate; runbook runbooks/dc-dc-phase2-tofu-dc-substrate.md). Stages 0-2 precede it; stages 4-7 are authored, not executed (workflow doc:873).tofu apply (deploy step A) is STOPPED at its own gate. The stop is commit 43b29d7 ("post-reboot outer plan would BOUNCE Office1 headend + edge -- DC0 apply STOPPED", 2026-07-18). Nothing DC0-scoped has been applied (tofu -chdir=opentofu state list shows no vvr1_dc0 or vr1_dc0_uplink resources; output quoted in section 2).148dcef). Phase 1 complete (af2e312, e999b03); this document is the Phase-2 deliverable (ACCEPTED on external review, c49da79); Phase 3 complete -- 3 probes, 21/21 PASS, holes map at docs/audit/phase3-grounding-test-20260718.md. Phases 4-6 have NOT run.docs/dc0-deploy-readiness.md:107) HAS happened: host rebooted ~2026-07-17 23:39, both guests self-recovered via autostart (docs/audit/env-snapshot-20260718.md:10-16; re-measured this session, section 2.2 below).tofu -chdir=opentofu state list,run 2026-07-18, 20 resources)
Office1 site (live, load-bearing):
module.voffice1.libvirt_domain.vm + .libvirt_volume.disk + .libvirt_volume.seed + .libvirt_cloudinit_disk.seed (BUT see divergence 2.3-i: the cloudinit staging ISO no longer exists live)module.office1_opnsense.libvirt_domain.vm + .libvirt_volume.diskmodule.office1_network.libvirt_network.office1_localmodule.office1_storage.libvirt_pool.dcmodule.ubuntu_noble_base.libvirt_volume.baseInter-site fabric and DC scaffolding:
module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link, module.mesh_vr1_dc0_office1.libvirt_network.link, module.mesh_vr1_dc1_office1.libvirt_network.link (the D-100 mesh triangle)module.vr1_dc0_planes.libvirt_network.plane["data-tenant" | "metal-admin" | "metal-internal" | "provider-public" | "replication" | "storage"] -- applied and live, but REMOVED from config (see 2.3-iii)module.vr1_dc0_storage.libvirt_pool.dc, module.vr1_dc1_storage.libvirt_pool.dchostname -> vcloud; uname -r -> 6.8.0-136-generic.virsh list --all -> exactly two domains, both running: voffice1 (Id 1), office1-opnsense (Id 2).virsh dominfo -> Autostart: enable on BOTH domains.ssh voffice1 'snap list maas lxd; uname -r' -> maas 3.7.2-17972-g.35e297c4d rev 41649 (3.7/stable), lxd 5.21.5-f2a1a0e rev 40074 (5.21/stable, held), guest kernel 6.8.0-136-generic.ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}" http://localhost:8000/' -> netbox=302 (service up, redirecting to login).ssh office1-tailscale 'tailscale status | head -1' -> 100.64.0.53 office1-tailscale ... linux - (subnet-router VM up).scripts/site-baseleg.sh check office1 passed at the Phase-1 snapshot (docs/audit/env-snapshot-20260718.md:16-17); not re-run this session.docs/vr1-office1-as-built.md:42, updated 2026-07-18). NOT re-measured this session -- measuring requires the gated API credential path; see section 7.i. module.voffice1.libvirt_cloudinit_disk.seed is IN STATE, but its remote object (the provider's staging ISO under /tmp/terraform-provider-libvirt-cloudinit/) was DELETED by the host reboot. Captured: docs/audit/outer-plan-20260718.txt:34-40 ("has been deleted"). Because the regenerated staging path is random, module.voffice1.libvirt_volume.seed plans as a FORCED REPLACEMENT (create.content.url -- capture lines 119-134), which would bounce the running Office1 headend. This is the defect that stopped the apply (GA-F01) and the subject of the pending D-130 ruling (section 8). Root-cause mechanism, verified against provider v0.9.8 source: docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228. ii. Autostart: LIVE is enable on both domains (measured, 2.2). CONFIG authors autostart = true (D-127, docs/design-decisions.md:3914). But STATE records NO autostart attribute on either domain (empirical: tofu state show ... | grep -c autostart -> 0, finding doc:140-145), so the plan still shows 2 in-place changes -- and a provider-source read confirms an autostart-only update BOUNCES the guest (finding doc:162-184). Live matches intent; state cannot see it; an unscoped apply would still cause an outage. Resolution belongs to the gated reconcile (section 6, gate G6) -- NOT to a quiet re-plan. iii. The six vr1-dc0 plane networks exist live and in state but are REMOVED from config -- the INTENDED Model B relocation (planes get recreated inside vvr1-dc0 by the inner root, opentofu/vr1-dc0-substrate/main.tf:28). Their emptiness (0 leases, 0 attached domains) was verified in a PRIOR session (docs/dc0-deploy-readiness.md:43-45) and must be re-verified in the same session as any apply (finding doc:259-265). iv. docs/dc0-deploy-readiness.md:8 still carries the banner "PREP COMPLETE -- DC0 IS DEPLOY-READY" and internal plan counts 5/2/6 (:37) and 5/0/6 (:59). The captured live plan (7/2/7) falsifies the banner and both counts (GA-F01, GA-F02). Under freeze, the doc is not edited; its checklists remain useful, its status lines do not.
module "vvr1_dc0" (opentofu/main.tf:360) -- the DC0 containment VM (416 GiB / 108 vCPU, D-121/D-123 sizing) + its disk, seed volume, and cloudinit seed. 4 of the 5 committed DC0 creates in the plan capture.module "vr1_dc0_uplink" (opentofu/main.tf:341) -- the D-125 simulated ISP NAT network 172.30.2.0/24 (capture lines 226-247). The 5th create.opentofu/vr1-dc0-substrate/ (main/variables/ versions.tf; no state file exists in that directory) -- inner storage pool, the six relocated planes, bridge-in WAN, and the rest of the Model B step-C build.module "vr1_dc0_planes" from the outer config (the 6 intended destroys; see 2.3-iii).autostart = true on voffice1/office1-opnsense in config (D-127) -- live-true but state-absent (2.3-ii).scripts/opnsense-plugins.sh + tests/opnsense-plugins/ (D-129 profile-installer; commit 4cefa8b). BUILT and green; the live apply against the edge is an operator-gated firmware mutation, NOT run (docs/design-decisions.md:4050-4056).scripts/site-headend-install.sh --host-nodes writing the transit FORWARD-drop) -- COMMITTED, applies on vvr1-dc0 at deploy step B; ledger row stays OPEN until applied+verified (bash scripts/ledger-scan.sh output, SEC-010 row).runbooks/dc-dc-phase3..6-*.md) -- written, not executed (docs/dc-dc-deployment-workflow.md:873).docs/design-decisions.md:3291,3474,3543,3635,3713): the DC0 deploy sequence they rule (steps A-E, docs/dc0-deploy-readiness.md:170-179) exists only as config + runbook. Nothing DC0 is built.vr1-dc1: ruled at topology level (D-100/D-101 two-DC design) but HELD -- no ruled dc1 transit/rack addressing, no vr1_dc1_rack_* variables, no dc1 substrate root (docs/dc0-deploy-readiness.md:100-103). Only its storage pool and mesh legs exist (state list, 2.1).opentofu/modules/netem-link) but HELD as a comment in the root (opentofu/main.tf:309-319); placeholder parameters ruled for the rehearsal (readiness doc:73-75); final parameters unruled (section 8).os-smart, os-nut | os-apcupsd, microcode, os-lldpd) -- recorded, inert until the Roosevelt edge build (docs/design-decisions.md:4030-4031).docs/design-decisions.md:4041-4047,4057-4063).The true EXPECTED outer plan count is currently NOBODY'S:
docs/audit/outer-plan-20260718.txt, line 428: "Plan: 7 to add, 2 to change, 7 to destroy." -- the ONLY citable plan-count source);docs/dc0-deploy-readiness.md:59, docs/session-ledger.md:278).7/2/7 is the post-reboot SYMPTOM of the D-130 seed-volume defect (7 add = 5 committed DC0 creates + voffice1 cloudinit-disk re-create + seed-volume replace-create; 2 change = the state-blind autostart updates; 7 destroy = 6 empty planes + seed-volume replace-destroy -- decomposition at docs/audit/env-snapshot-20260718.md:33-46). 5/0/6 was the pre-reboot gate; 5/2/6 was an intermediate recorded value (GA-F02). The post-fix expected triple DOES NOT EXIST until D-130 is ruled, and this document deliberately does not pick one. When D-130 is ruled, the new expected triple and WHY get recorded HERE before any re-plan (charter Phase 6 item 4). This document may not be used to justify an apply until then.
Owner legend: operator (human ruling/approval), session (agent work under gating), external (outside this repo/track).
| # | Gate | What closes it | Owner | Evidence of current state |
|---|---|---|---|---|
| G1 | Audit Phase 3: fresh-agent grounding test | 3 clean-context probes score the 7-question set against this doc; holes map made | session | CLOSED 2026-07-18: 3 probes, 21/21 PASS, holes H1 (amended into G9) + H2 (no action) -- docs/audit/phase3-grounding-test-20260718.md |
| G2 | Audit Phase 4: GA-R1..R7 structural rulings + the stage-status vocabulary A/B | ruling-type gate (GA-R6 rule 6): closes when every item carries a GA-R5 Status block | operator | CLOSED 2026-07-18: all seven GA-R + vocabulary (Option A + H1) RATIFIED, utterances quoted (docs/audit/ga-rulings.md, through commit fe4f1c4 + this one) |
| G3 | Audit Phase 5: repair sweep of GA-F01..F14 (incl. memory hygiene GA-F05..F08, skill sweep) | operator-gated fix batches, each commit naming its GA-F | operator + session | FREEZE in effect; zero fixes applied (charter section 1) |
| G4 | The two D-130 verifications (charter-exempt, read-only/throwaway): (v7) v0.9.8 volume-replace-under-running-domain on a throwaway domain; (v8) lifecycle ignore_changes suppression of the forced replacement | run them, capture output | session | TO RUN (docs/audit/env-snapshot-20260718.md:47-49) |
| G5 | D-130 mechanism ruling (seed-volume durable fix) | operator rules in Phase 5, quoting G4's captured output | operator | D-130 unassigned; next-free D confirmed 130 (ledger-scan run 2026-07-18; no ^## D-130 in design-decisions.md) |
| G6 | State reconcile of autostart + seed WITHOUT bouncing guests | gated mechanism per the finding: tofu import (reads autostart back) or state surgery, or an operator-scheduled maintenance-window apply; -refresh-only is PROVEN a no-op |
operator | Operator earlier selected reconcile-first (finding doc:136); superseded into audit sequencing by 2b718f5; nothing reconciled yet |
| G7 | New captured plan == the post-D-130 expected triple recorded in section 5 | re-plan to a capture file after G5+G6 | session | Impossible today: expected triple UNRESOLVED (section 5) |
| G8 | Same-session pre-apply re-verify: 6 planes still empty (0 leases / 0 attached domains) | run in the SAME session as the apply | session | Last verified in a prior session only (finding doc:259-265) |
| G9 | DC0 outer apply (deploy step A) | operator-gated, logged (run-logged.sh), after G1-G8; audit exit criteria met (charter Phase 6). CANONICAL ENTRY DOC (probe hole H1): runbooks/dc-dc-phase2-tofu-dc-substrate.md, with docs/dc0-deploy-readiness.md section E as the step table |
operator | STOPPED (43b29d7); state shows nothing DC0 applied |
| G10 | Deploy steps B-E in-sequence gates: SEC-010 --host-nodes --check on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + TF_VAR_maas_api_key before step D; netem placeholder step E |
exercised during the gated deploy | session (each mutation operator-approved) | Not reached (docs/dc0-deploy-readiness.md:67-84,170-179) |
| G11 | Operator signs THIS document | read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: SIGNED 2026-07-18, section 11 (post-probe version, per sequencing directive) |
| G12 | vr1-dc1 build |
operator rules dc1 transit/rack addressing; then vars + substrate authored | operator | HELD (docs/dc0-deploy-readiness.md:100-103) |
| G13 | D-129 residuals | operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) | operator | OPEN / PARTIALLY RULED (docs/design-decisions.md:4017) |
| G14 | 9 OPEN SEC rows (SEC-001, -003..-008, -010) | per-row: rotations/flips at v1 close (external to VR1 track) except SEC-010 = apply+verify at G10 | operator / external | bash scripts/ledger-scan.sh output, 2026-07-18 |
| G15 | D-068 / D-071 rulings | operator rules (section 8); neither blocks the VR1 substrate | operator | PROPOSED/OPEN (status lines, section 8) |
| Component | Measured value | Command (run 2026-07-18) | Where measured |
|---|---|---|---|
| OpenTofu | v1.12.4 | tofu version |
vcloud (also docs/audit/env-snapshot-20260718.md:26) |
| libvirt provider | dmacvicar/libvirt 0.9.8 (pinned) | grep -A2 'provider' opentofu/.terraform.lock.hcl |
repo lock file |
| MAAS provider | canonical/maas 2.7.2 (pinned) | same | repo lock file |
| MAAS | 3.7.2-17972-g.35e297c4d (3.7/stable) | ssh voffice1 'snap list maas' |
voffice1 |
| LXD | 5.21.5-f2a1a0e (5.21/stable, held) | ssh voffice1 'snap list lxd' |
voffice1 |
| Kernel (host) | 6.8.0-136-generic | uname -r |
vcloud |
| Kernel (voffice1) | 6.8.0-136-generic | ssh voffice1 'uname -r' |
voffice1 |
| OPNsense edge | 26.7 (FreeBSD base 15.1) | not re-measured (gated API creds); per confirmed as-built docs/vr1-office1-as-built.md:42, 2026-07-18 |
office1-opnsense |
| NetBox (Office1 apex) | 4.6.4 per as-built docs/vr1-office1-as-built.md:44; service UP verified (HTTP 302) this session |
ssh office1-netbox 'curl ... localhost:8000' |
office1-netbox |
| Juju | NOT part of VR1 substrate yet (arrives Stage 5). Last recorded VR0 value 3.6.25 -- historical, unverified here | n/a | n/a |
Known-stale pins that this table supersedes (do NOT trust them; Phase-5 fixes them): OpenTofu 1.12.3 at opentofu/README.md:41 (historical Stage-1 prose), docs/dc-dc-deployment-workflow.md:47,360,875, .claude/skills/.../platform-traps.md:19 (GA-F03); OPNsense 26.1 across the skill references (GA-F04); jumphost name vopenstack-jesse in CLAUDE.md:4 and the skill -- the host is vcloud, measured (GA-F05).
lifecycle { ignore_changes = [create] } on libvirt_volume.seed in modules/cloudinit-vm -- the operator's recorded lean, EXPLICITLY CONTINGENT on verification G4-v8 confirming it suppresses the forced replacement; if v8 refutes it, the lean is DEAD (do not rescue) and the fallback is (b) a persistent $TMPDIR staging path plus a runbook-enforced env guard, with the fragility concern stated and accepted (charter section 1a). Evidence needed before ruling: the captured outputs of both G4 verifications; the decision text must quote them. Full apply-scoping option set (a)-(d) and the verified provider mechanics: docs/finding-20260718-voffice1-cloudinit-seed-replace.md:104-257. Blast radius (measured): voffice1 + vvr1-dc0 (+ vvr1-dc1 later) = 3 VMs; node VMs and the edge are UNAFFECTED (finding doc:206-223). Sequencing: ruled inside audit Phase 5, AFTER this document (charter section 1a, last bullet).docs/design-decisions.md: 1962) and placeholder-for-the-rehearsal (readiness doc:73-75, operator ruling 2026-07-16). OPEN QUESTION: the final latency/jitter/loss/rate numbers -- supply the Roosevelt inter-DC target, or confirm the placeholder stands until that target exists. Evidence needed: the Roosevelt link spec (external input; nothing in-repo can answer it).docs/design-decisions.md:1457). QUESTION: rule items 2 (Vault listener TLS) and 3 (AppRole secret_id lifecycle) for Roosevelt; item 1 (Vault version) needs a re-scoped migration plan since the 1.16 forward-pin was proven NOT viable (amendment, :1650). Evidence in docs/D-068-vault-1.8-vs-1.16-analysis.md.docs/design-decisions.md:1605). QUESTION: adopt policy points 1-4 (monthly-review window trigger; patch-only controller jumps; order; acceptable single-controller risk) as amended by the confirmed backup posture (:1677).docs/design-decisions.md:4017): (i) is the boundary edge in-scope for COS scraping (D-105 scope)? (ii) os-frr dynamic routing -- Roosevelt question? (iii) consolidate the Tailscale subnet-router onto the edge vs keep D-107 as-is? (iv) does the edge anchor site NTP? Plus the operator-gated live install of the ruled VR1 profile (scripts/opnsense-plugins.sh apply vr1-edge).freeze)
ledger-scan.sh's mention-derived next-free counters (DOCFIX, BUNDLEFIX) are SELF-INFLATED by any doc that quotes a "next-free" value and lets the hyphenated token wrap onto a line without the words "next-free" -- the per-line exclusion filter (scripts/ledger-scan.sh:121-129, the grep -viE 'next[- ]free' at :124) then counts the quote as a real assignment; the script's own CAUTION comment (:112-120) documents exactly this failure class. It happened TWICE inside the audit itself on 2026-07-18: the Phase-1 env snapshot's wrapped next-free line inflated the BUNDLEFIX counter (051 -> reported 052), and this document's own first draft of this very section inflated the DOCFIX counter the same way while asserting DOCFIX was unaffected. Both audit surfaces were reworded token-free the same day and the counters re-verified at their true values (D=130, DOCFIX=197, BUNDLEFIX=052 -- see GA-F15). The D counter is header-authoritative and was never affected. Numbering discipline until the scanner is hardened in Phase 5: never write a hyphenated register-token quote of a next-free value into any doc; state the numbers token-free as this section does.
Run read-only, from the repo root:
git rev-parse HEAD; git status --short (this doc was authored at e999b03, clean tree).tofu -chdir=opentofu state list (expect the 20 resources in section 2.1); virsh list --all; virsh dominfo voffice1 | grep -i autostart (and office1-opnsense).grep -n '^module ' opentofu/main.tf (12 blocks; vvr1_dc0 + vr1_dc0_uplink absent from state list); ls opentofu/vr1-dc0-substrate/ (no *.tfstate).docs/audit/outer-plan-20260718.txt line 428. Do NOT re-run tofu plan casually against live state; if a fresh capture is taken, it must be written to a new dated capture file and cited here.tofu version; ssh voffice1 'snap list maas lxd; uname -r' </dev/null; uname -r; grep -A2 provider opentofu/.terraform.lock.hcl.bash scripts/ledger-scan.sh (BUNDLEFIX caveat: section 9); decision status lines: grep -n '^## D-' docs/design-decisions.md then read each Status line -- a decision's Status line in that file is the ONLY ruling authority.docs/audit/grounding-audit-charter.md; docs/audit/grounding-audit-20260718.md for GA-F01..F14.ssh office1-netbox 'curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/' </dev/null (expect 302); ssh office1-tailscale 'tailscale status | head -1' </dev/null.What this document is NOT built from and you must not rebuild it from: the prose of the 95 docs/changelog-*.md files, the docs/session-ledger.md narrative, or auto-memory -- all proven to carry false status (GA-F14, GA-F06..F08).
SIGNED 2026-07-18. The operator personally reviewed the section-10 verification run (outputs at repo HEAD 8cdbf4a; zero divergences, match table in the session record) and read this document top to bottom. Exact utterance: "SIGNED Reviewed and all signoff conditions are met. I reviewed the section-10 run outputs at 8cdbf4a. Jesse 7/18/2026". This document is the signed status authority (charter Phase 6 item 5 MET at this baseline).