Newer
Older
openstack-caracal-dc-dc / docs / CURRENT-STATE.md

CURRENT-STATE.md -- the single status authority (Omega Cloud / VR1 DC-DC)

Authored 2026-07-18 by the grounding-audit Phase-2 ground-truth agent (charter: docs/audit/grounding-audit-charter.md, section 3) at repo HEAD e999b03 on branch dc-dc-stage3-phase2-dc-substrate. Every claim below carries its evidence (path:line, quoted command output, or commit hash). Claims that could not be evidenced read-only are marked UNKNOWN with what would resolve them. Nothing here is guessed.

STATUS OF THIS DOCUMENT: SIGNED by the operator 2026-07-19 (section 11; charter Phase 6 item 5). STANDING RULE (GA-R1, RATIFIED 2026-07-18 with amendments C1+C2 -- docs/audit/ga-rulings.md): no status claim is hand-written anywhere else; other documents point HERE; this document cites captured command output, and measurement always wins over it (C2). Other status surfaces are pointers or history; where one still carries a claim, it is a defect (Phase 1 proved they contradict: docs/audit/record-inventory.md, 12 groups; findings GA-F01..GA-F15).


1. Where the project IS

>>> STANDING OPERATOR DIRECTIVE, 2026-07-30: THE NEXT SESSION PROCEEDS TO THE JUJU DEPLOYMENT (STAGE 5). NO MATTER WHAT. <<<

Verbatim: "we have to continue to juju deployment next session no matter what".

Precondition work is DONE. Stage-5 preconditions were hardened across 2026-07-27..30 (allocation blocker closed, per-DC Octavia PKI generated AND reissued, gate integrity fixed, P7 wired, identity centralised). Further hardening, auditing, or record repair is OUT OF SCOPE next session unless it directly blocks the deploy.

This is the GA-F06 circuit-breaker made explicit by the operator. This project's measured failure mode is record-churn presenting as progress; the standing rule already says to STOP and escalate after one cycle of fixing records to match records. The directive removes the judgement call: open the stage and deploy.

Entry facts so the next session does not re-derive them:

  • Run FROM voffice1 (D-128 Plane-2 host). juju 3.6.27 is there; the jumphost has none.
  • preflight.sh currently exits FAIL on P5 only -- the credential register is red on PRE-EXISTING defects (SEC-021 declaration gaps, the SEC-020 identity conflation, three power-key asymmetries). Decide accept-or-remediate AT the gate; do not re-audit it. Every other gate is green, and P7 reports [ok] for both DCs on the headend.
  • The deploy input is the bundle + that DC's -vips, -machines and -octavia-pki overlays.
  • G17 arms at first boot -- it is a Stage-5 artefact, not a precondition. Capture it when the nodes come up rather than treating it as a blocker beforehand.
  • Stage-5 runbook: runbooks/dc-dc-phase4-juju-bundle-per-dc.md; Step 4 delegates to phase-01-bundle-deploy.md verbatim.

2. What is APPLIED (from tofu state + live measurement, not from docs)

2.1 OpenTofu outer-root state (command: tofu -chdir=opentofu state list,

run 2026-07-18, 20 resources)

Office1 site (live, load-bearing):

  • module.voffice1.libvirt_domain.vm + .libvirt_volume.disk + .libvirt_volume.seed + .libvirt_cloudinit_disk.seed (BUT see divergence 2.3-i: the cloudinit staging ISO no longer exists live)
  • module.office1_opnsense.libvirt_domain.vm + .libvirt_volume.disk
  • module.office1_network.libvirt_network.office1_local
  • module.office1_storage.libvirt_pool.dc
  • module.ubuntu_noble_base.libvirt_volume.base

Inter-site fabric and DC scaffolding:

  • module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link, module.mesh_vr1_dc0_office1.libvirt_network.link, module.mesh_vr1_dc1_office1.libvirt_network.link (the D-100 mesh triangle)
  • module.vr1_dc0_planes.libvirt_network.plane["data-tenant" | "metal-admin" | "metal-internal" | "provider-public" | "replication" | "storage"] -- applied and live, but REMOVED from config (see 2.3-iii)
  • module.vr1_dc0_storage.libvirt_pool.dc, module.vr1_dc1_storage.libvirt_pool.dc

2.2 Live environment (measured this session, 2026-07-18, read-only)

  • hostname -> vcloud; uname -r -> 6.8.0-136-generic.
  • virsh list --all -> exactly two domains, both running: voffice1 (Id 1), office1-opnsense (Id 2).
  • virsh dominfo -> Autostart: enable on BOTH domains.
  • ssh voffice1 'snap list maas lxd; uname -r' -> maas 3.7.2-17972-g.35e297c4d rev 41649 (3.7/stable), lxd 5.21.5-f2a1a0e rev 40074 (5.21/stable, held), guest kernel 6.8.0-136-generic.
  • ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}" http://localhost:8000/' -> netbox=302 (service up, redirecting to login).
  • ssh office1-tailscale 'tailscale status | head -1' -> 100.64.0.53 office1-tailscale ... linux - (subnet-router VM up).
  • D-126 base leg: scripts/site-baseleg.sh check office1 passed at the Phase-1 snapshot (docs/audit/env-snapshot-20260718.md:16-17); not re-run this session.
  • OPNsense edge version: 26.7 per confirmed as-built (docs/vr1-office1-as-built.md:42, updated 2026-07-18). NOT re-measured this session -- measuring requires the gated API credential path; see section 7.
  • SEC-010 nft files (2026-07-23, operator-approved live re-assert, logged window ops-sec010-reassert): /etc/nftables-sec010.nft on voffice1 and BOTH DC racks now carries the idempotent declare-then-delete preamble -- sec010-fw double-restart converged with no rule duplication (4/2/2 drops; pre/post capture docs/audit/sec010-reassert-20260723.txt; per-host backup .pre-reassert-20260723 on each host). Generator fixed the same day (queue-pass changelog items 3/8).

2.3 State vs live vs config disagreements (REPORTED, not harmonized)

i. module.voffice1.libvirt_cloudinit_disk.seed is IN STATE but its staging ISO was deleted by the reboot -- it still plans as a benign re-create (1 of section 5's 6 adds). The FORCED REPLACEMENT it used to force on libvirt_volume.seed (the GA-F01 defect that stopped the apply) is FIXED: D-130 ADOPTED (a) + implemented 2026-07-19, verified by the v8/v7 captures (gate rows G4/G5). Mechanism history: docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228. ii. Autostart: RESOLVED 2026-07-19 by the G6 state surgery (operator- ruled (ii), gate row G6): state now records autostart = true on both domains (state show | grep -c autostart -> 1 each); the 2 in-place changes are gone from the plan (section 5 capture). Guests were never touched. iii. The six vr1-dc0 plane networks exist live and in state but are REMOVED from config -- the INTENDED Model B relocation (planes get recreated inside vvr1-dc0 by the inner root, opentofu/vr1-dc0-substrate/main.tf:28). Their emptiness (0 leases, 0 attached domains) was verified in a PRIOR session (docs/dc0-deploy-readiness.md:43-45) and must be re-verified in the same session as any apply (finding doc:259-265). iv. RESOLVED 2026-07-19 (Batch 2, GA-F02): the readiness doc's falsified deploy-ready banner and its three contradictory plan counts are demoted -- status and the expected triple point HERE; the fresh- session banner points at the G9 canonical entry doc.

3. What is AUTHORED-BUT-NOT-APPLIED (in the tree, not in state / live)

  • (2026-07-20) The step-B transit-reach work is APPLIED and verified -- voffice1 holds the region end 172.31.0.1/30 on enp2s0 (plus its in-guest drop-in /etc/netplan/60-transit.yaml); vvr1-dc0 answers at 172.31.0.2 (netplan set-name root cause fixed, kernel names enp1s0/enp2s0 kept); ssh -J voffice1 with the dc0 key works; rack->region ping 10.10.0.20 0% loss. Kea reservation re-keyed to the regenerated voffice1 MAC (incident, session changelog item 3). Consequence for the G10 bootstrap: call site-headend-install.sh with --transit-if enp1s0 --uplink-if enp2s0.

  • module "vvr1_dc0" (opentofu/main.tf:360) -- the DC0 containment VM (416 GiB / 108 vCPU, D-121/D-123 sizing) + its disk, seed volume, and cloudinit seed. 4 of the 5 committed DC0 creates in the plan capture.

  • module "vr1_dc0_uplink" (opentofu/main.tf:341) -- the D-125 simulated ISP NAT network 172.30.2.0/24 (capture lines 226-247). The 5th create.
  • The ENTIRE inner root opentofu/vr1-dc0-substrate/ (main/variables/ versions.tf; no state file exists in that directory) -- inner storage pool, the six relocated planes, bridge-in WAN, and the rest of the Model B step-C build.
  • The removal of module "vr1_dc0_planes" from the outer config (the 6 intended destroys; see 2.3-iii).
  • autostart = true on voffice1/office1-opnsense in config (D-127) -- live-true but state-absent (2.3-ii).
  • scripts/opnsense-plugins.sh + tests/opnsense-plugins/ (D-129 profile-installer; commit 4cefa8b). BUILT and green; the live apply against the edge is an operator-gated firmware mutation, NOT run (docs/design-decisions.md:4050-4056).
  • SEC-010 enforcement artifact (scripts/site-headend-install.sh --host-nodes writing the transit FORWARD-drop) -- COMMITTED, applies on vvr1-dc0 at deploy step B; ledger row stays OPEN until applied+verified (bash scripts/ledger-scan.sh output, SEC-010 row).
  • Stage 4-7 runbooks (runbooks/dc-dc-phase3..6-*.md) -- written, not executed (docs/dc-dc-deployment-workflow.md:873).

4. What is RULED-BUT-NOT-BUILT (ADOPTED/RULED with no artifact yet)

  • D-121/D-122/D-123/D-124/D-125 (all ADOPTED/RULED, status lines at docs/design-decisions.md:3291,3474,3543,3635,3713): the DC0 deploy sequence they rule (steps A-E, docs/dc0-deploy-readiness.md:170-179) exists only as config + runbook. Nothing DC0 is built.
  • vr1-dc1: topology ruled (D-100/D-101) and addressing RATIFIED 2026-07-21 (D-124 amendment: planes contiguous in 10.12.64.0/19, transit 172.31.0.4/30, uplink 172.30.3.0/24; apex confirm-free at authoring). Still UNBUILT: no vr1_dc1_rack_* variables, no dc1 substrate root; only its storage pool and mesh legs exist (state list, 2.1). Gate row G12 carries the remaining [V] leg.
  • D-100 netem: mechanism authored (opentofu/modules/netem-link) but HELD as a comment in the root (opentofu/main.tf:309-319); placeholder parameters ruled for the rehearsal (readiness doc:73-75); final parameters unruled (section 8).
  • D-129 Roosevelt metal-edge plugin profile (os-smart, os-nut | os-apcupsd, microcode, os-lldpd) -- recorded, inert until the Roosevelt edge build (docs/design-decisions.md:4030-4031).
  • D-129 qga enablement: designed (opt-in module variable, default OFF; the guest-agent virtio channel is MISSING from the live edge domain) but not written; retrofit deferred to the edge's next scheduled restart (docs/design-decisions.md:4041-4047,4057-4063).

5. The outer plan count (wording per operator direction, GA-F01)

The true EXPECTED outer plan count is currently NOBODY'S:

  • expected count: UNRESOLVED pending D-130;
  • last captured: 7/2/7 (docs/audit/outer-plan-20260718.txt, line 428: "Plan: 7 to add, 2 to change, 7 to destroy." -- the ONLY citable plan-count source);
  • pre-reboot gate was 5/0/6 (recorded at docs/dc0-deploy-readiness.md:59, docs/session-ledger.md:278).

The EXPECTED outer plan is ZERO DIFF ("no differences"), re-recorded 2026-07-22 with its evidencing capture (docs/audit/outer-plan-20260722-postdc1-converged.txt) after the G12 dc1 substrate step-A apply (saved plan 5/0/0 exact -- vvr1-dc1 + vr1-dc1-uplink adds only, zero touches to live resources). A future outer plan showing ANY diff is a STOP (investigate drift before touching anything). History: 7/2/7 post-reboot symptom -> 6/2/6 post-D-130 -> 6/0/6 post-G6-reconcile -> applied exact -> zero diff -> 1/1/1 (voffice1 transit, ruled+applied) -> 2/0/2 (rack netplan fix, applied) -> zero diff converged 2026-07-20 -> 1/1/0 netem-wire STOP -> targeted netem apply 1/0/0 exact -> 0/1/0 office1 residual -> G16 state surgery -> zero diff converged 2026-07-21 -> 5/0/0 dc1 substrate adds (G12 step A, saved-plan applied exact 2026-07-22) -> zero diff converged -> 0/1/0 office1 qga channel (G13 bundle, saved-plan applied exact 2026-07-23) -> zero diff converged (docs/audit/outer-plan-20260723-postqga-converged.txt, this entry) -> RE-CONFIRMED ZERO DIFF 2026-07-27 by the Stage-5 grounding audit, and for the FIRST TIME across ALL THREE roots in one session: the outer root on vcloud (docs/audit/outer-plan-20260727-stage5-audit.txt, "No changes") AND both inner roots on voffice1 (vr1-dc0-substrate and vr1-dc1-substrate, each "No changes", exit 0). Validity of the inner pair was established FIRST by proving the voffice1 clone's inner roots are byte-identical to main despite that clone being 105 commits behind (git diff --stat 61c416e..main -- opentofu/vr1-dc0-substrate/ opentofu/vr1-dc1-substrate/ opentofu/modules/ -> EMPTY); had that diff been non-empty the inner plans would have been UNMEASURED, not green. Full capture: docs/audit/stage5-live-measurement-20260727.txt.

6. Open gates

Owner legend: operator (human ruling/approval), session (agent work under gating), external (outside this repo/track). Type legend (GA-R6/E1): [V] = verification-type (closes on its named executable check); [R] = ruling-type (closes per a GA-R5 recorded ruling).

# Gate What closes it Owner Evidence of current state
G1 Audit Phase 3: fresh-agent grounding test [V] 3 clean-context probes score the 7-question set against this doc; holes map made session CLOSED 2026-07-18: 3 probes, 21/21 PASS, holes H1 (amended into G9) + H2 (no action) -- docs/audit/phase3-grounding-test-20260718.md
G2 Audit Phase 4: GA-R1..R7 structural rulings + the stage-status vocabulary A/B [R] ruling-type gate (GA-R6 rule 6): closes when every item carries a GA-R5 Status block operator CLOSED 2026-07-18: all seven GA-R + vocabulary (Option A + H1) RATIFIED, utterances quoted (docs/audit/ga-rulings.md, through commit fe4f1c4 + this one)
G3 Audit Phase 5: repair sweep of GA-F01..F15 (incl. memory hygiene GA-F05..F08, skill sweep) [R] operator-gated fix batches, each commit naming its GA-F operator + session Batch 0 OPENED by operator 2026-07-19; items 0.1 (repo-lint L10, GA-R1/C1), 0.2 (SEC repoint, GA-R4/F3), 0.3 (counter hardening, GA-F15), 0.4 (extractor vocab scan, GA-F10/H1) landed; Batch 0 CLOSED (verification passed 2026-07-19); Batches 0-4 CLOSED 2026-07-19 (Batch 4: GA-R4 ledger rotation 1187->131 lines, F1 cap now enforceable; 96 changelogs + 24 history docs consolidated to docs/archive/ with 4 stage records + per-stage manifest commits; top-level docs/ = 16 files < 25; live-surface refs rewritten); Batch 5 CLOSED 2026-07-19 (skill sweep: GA section added reconciled against ratified text, stale phase/UNVALIDATED claims demoted, bookends + stage-close rewritten; checklist docs/audit/skill-sweep-checklist-20260719.md); Batch 6 OPEN: exit runs 1/2/4/5 PASS (adjudication + captures: docs/audit/phase6-exit-runs-20260719.md); exit item 3 PASS 2026-07-19 (fresh trio 21/21, 7/7 all three -- exit record); CLOSED 2026-07-19 -- CORRECTED 2026-07-27 (Stage-5 grounding audit, finding L1-7). This cell previously read "Batch 6 OPEN ... PENDING only item 6 (operator re-read + re-sign) ... FREEZE holds for un-gated surfaces". That was contradicted by its OWN cited evidence file: docs/audit/phase6-exit-runs-20260719.md:84 records "## 6. Operator re-read + re-signature -- SIGNED 2026-07-19 ... PASS" and :91-92 states "ALL SIX EXIT RUNS PASS. The grounding audit is EXITED; G3 + G11 CLOSED". Corroborated by row G11 (CLOSED, re-signed 2026-07-19) and by section 11's recorded utterance "Reviewed, approved, continue." No ruling was required to correct this -- two surfaces already declared it closed. The consequential half was the FREEZE clause, not the state token: left standing it would have blocked the very DOCFIX remediation batch the 2026-07-27 audit queues. The freeze was lifted at audit exit 2026-07-19 (section 1) and normal change discipline governs
G4 The two D-130 verifications [V] run them, capture output session CLOSED 2026-07-19: v8 suppression CONFIRMED (7/2/7 -> 6/2/6, zero forces-replacement; docs/audit/outer-plan-20260719-v8-ignorechanges.txt + -baseline.txt); v7 no-bounce under running domain, zero residue (docs/audit/throwaway-v7-20260719.txt)
G5 D-130 mechanism ruling (seed-volume durable fix) [R] operator rules in Phase 5, quoting G4's captured output operator CLOSED 2026-07-19: D-130 ADOPTED (a) ignore_changes (docs/design-decisions.md D-130, GA-R5 utterance quoted); implemented in modules/cloudinit-vm + tests/cloudinit-vm
G6 State reconcile of autostart + seed WITHOUT bouncing guests [R] gated mechanism, operator-ruled (S3) operator CLOSED 2026-07-19: ruled (ii) state surgery (GA-R5); pull -> inject autostart:true on both domains -> push (serial 22->23, backup terraform.tfstate.pre-G6-20260719); guests never touched (ids 1/2 unchanged, running)
G7 New captured plan == the expected triple recorded in section 5 [V] re-plan to a capture file after G5+G6 session CLOSED 2026-07-19: capture docs/audit/outer-plan-20260719-postG6.txt = 6/0/6, equals section 5 exactly
G8 Same-session pre-apply re-verify: 6 planes still empty [V] run in the SAME session as the apply session CLOSED 2026-07-19: verified in the apply session itself (all six 0 leases; only office1 nets attached) immediately before step A
G9 DC0 outer apply (deploy step A) [V] operator-gated, logged (run-logged.sh), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via site-headend-install.sh --host-nodes --check on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): runbooks/dc-dc-phase2-tofu-dc-substrate.md, with docs/dc0-deploy-readiness.md section E as the step table operator CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched
G10 Deploy steps B-E in-sequence gates: SEC-010 --host-nodes --check on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + TF_VAR_maas_api_key before step D; netem placeholder step E [V] exercised during the gated deploy session (each mutation operator-approved) Step B DONE 2026-07-20 (--check EXIT 0 incl. SEC-010, docs/audit/stepB-check-20260720-final.txt; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (docs/audit/d125-egress-gate-20260720-matrix.txt), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- docs/audit/commissioning-diag-20260721.txt; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (docs/audit/stepE-netem-20260721.txt + outer-{plan,apply}-20260721-netem*.txt). G10 CLOSED 2026-07-21
G11 Operator signs THIS document [R] read top-to-bottom; discrepancies resolved in the document operator CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature)
G12 vr1-dc1 build [R] operator rules dc1 transit/rack addressing; then vars + substrate authored operator + session CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on main, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch dc-dc-g12-dc1-substrate): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (docs/audit/dc1-apex-confirm-20260721.txt); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (docs/audit/dc1-rack-import-dryrun-20260721.txt). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet ALL GREEN (76) (docs/audit/gauntlet-20260722-g12-reconcile.txt), repo-lint 0-fail. Apex --commit EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (docs/audit/dc1-rack-import-commit-20260722.txt). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), outer step-A apply DONE 2026-07-22: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). Step B COMPLETE 2026-07-22: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- docs/audit/outer-plan-20260722-voffice1-dc1nic.txt), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 nmpcq4), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 docs/audit/dc1-stepB-check-20260722-final.txt (changelog-20260722 items 5-8, three queued findings). Step C COMPLETE 2026-07-22: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (docs/audit/inner-converge-20260722-dc1-stepC.txt), 10/10 domains RUNNING inside vvr1-dc1, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). D-125 egress gate PASS 2026-07-22 (two identical runs, dc0 criteria exact, isolation confirmed -- docs/audit/d125-egress-gate-20260722-dc1.txt). Edge bootstrap + v4 addressing COMPLETE 2026-07-23 (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed util.inc/shell_safe() -- dc0 lesson iv the .b64 artifact lacked), key-only SSH VERIFIED (15.1-RELEASE-p1); D-113(a2) API key MINTED via the vendor model + smoke test GET core/firmware/status exit 0 product_abi 26.7 (second 26.7 datapoint); edge ADDRESSED -- WAN 172.30.3.2/24 gw 172.30.3.1 (egress 1.1.1.1 0% loss), LAN 192.168.1.1 -> 10.12.64.1/22 (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg 10.12.64.2/22 LIVE on virbr4. Creds consolidated to ~/vr1-dc1-creds/opnsense-api.txt (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (SEC-015 transient, remediated). Two queued findings: bootstrap .b64 missing util.inc; opnsense-bootstrap-apikey.sh scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). Rack standup + region MAAS config DONE 2026-07-23 (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; docs/audit/dc1-rack-net-install-20260723.txt); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); dc1 enlistment PROVEN via canary (machines 11->12 in ~2 min). SEC-016 RULED + WIRED 2026-07-23 (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). COMMISSIONING 9/9 READY 2026-07-23 (docs/audit/dc1-commissioning-verify-20260723.txt): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, power_type=virsh set + verified by real query-power-state (SEC-016 path proven), commissioned to ALL 9 READY in ~3.5 min (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. G12 [V] leg (the dc1 build) is COMPLETE. NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, operator-gated merge of dc-dc-g12-dc1-substrate -> main (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close).
G13 D-129 residuals [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain operator CLOSED 2026-07-23 (operator-approved full maintenance bundle, logged window ops-sec010-reassert): qga channel retrofitted via outer tofu saved-plan apply 0/1/0 exact (docs/audit/outer-plan-20260723-office1-qga.txt; the apply's edge bounce = the ruled "next scheduled restart"; MACs were pinned 07-22 so the in-place-update trap class was closed); edge updated 26.7 -> 26.7.1 via REST (no reboot required; os-iperf had been REFUSED on 26.7 pending exactly this update); both plugins installed=1 by firmware-info read-back, guest-ping -> {"return":{}}, agent reports both legs, egress 0% loss, outer plan re-converged ZERO DIFF (docs/audit/outer-plan-20260723-postqga-converged.txt). Named close capture: docs/audit/g13-close-20260723.txt
G14 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) operator / external docs/security-ledger.md (register of record, GA-R4/F3). COUNT RECONCILED 2026-07-27: 21 open, measured bash scripts/ledger-scan.sh (SEC-001, -003..-008, -012..-025; SEC-024 opened 2026-07-26, SEC-025 opened 2026-07-27 for the consolidated NetBox GUI admin password). Earlier figures in this cell (19 at 2026-07-25) are history. The row's own title text ("12 OPEN SEC rows") is the 2026-07-23 figure and is SUPERSEDED by this cell -- the gate is the ledger, not the count. Since 07-23: SEC-017 (caveman supply-chain), -018/-019 (per-DC MAAS API keys), -020 (MAAS region superuser passwords), and -021/-022/-023 opened 2026-07-25 from the D-137 credential research -- dc0 custody defects (a consolidated credential ABSENT from its recorded location + per-DC power-key divergence), UNAUDITED shadow *-creds/ stores on voffice1 (a scope gap in creds-audit itself, which has no remote capability), and sprawl-glob blind spots incl. a PREDICTED Stage-5 ~/admin-openrc exposure. All three are logged-not-actioned (hard rule 1); remediation is coupled to the unruled D-137 forks. Creation-point research capture: docs/audit/creds-creation-points-20260725.md -- 55 MINT sites inventoried, and 12 declared secrets have NO mint command anywhere in the repo (ssh-keygen returns ZERO hits repo-wide; six SSH keypairs + the OPNsense root password/hash are operator-terminal mints recorded only in a manifest comment, i.e. NOT reproducible if the jumphost is rebuilt -- a Roosevelt-transfer defect, not just hygiene). It also names three credential DIRECTORIES outside the SEC-009 *-creds/ convention and outside creds-audit entirely: ~/vault-init/ (Vault 5 unseal shares + root token), ~/octavia-pki/ (8 PKI artifacts incl. CA private keys), ~/tenant-<client>/; plus overlays/octavia-pki.yaml, which lands a CA key + plaintext passphrase INSIDE the repo clone (gitignored -- and SEC-004 says the repo is still PUBLIC)
G15 D-068 / D-071 rulings [R] operator rules (section 8); neither blocks the VR1 substrate operator D-071 ADOPTED 2026-07-21 (all four points); D-068 items 2-3 RULED 2026-07-21; item 1: plan DRAFTED + Q1/Q2-structure/Q3 ALL RULED 2026-07-23 (three amendments, utterances quoted; monthly-review lines delivered). Sole D-068 remainder: Q2 path selection at Roosevelt Vault design time -- G15 is otherwise decision-complete
G16 office1 edge channels = [] state reconcile (the D-129 module-schema residual) [R] operator rules the mechanism; then [V] the converged re-plan capture operator + session CLOSED 2026-07-21: RULED "State surgery (Recommended)" (GA-R5, session changelog item 16); executed per G6 precedent -- channels null -> [] injected, serial 29 -> 30, backup kept, guests untouched (office1-opnsense Id 2 running throughout); convergence = ZERO DIFF (docs/audit/outer-plan-20260721-postG16-converged.txt); section 5 re-recorded
G17 Per-DC artifact source reachable FROM A NODE -- the node-side half of Stage 4 DoD bullet 5, split out of Stage 4 by operator ruling rather than closed conditionally [V] a NAMED executable check run from a node that has actually booted an OS on its real NICs, per DC. RESHAPED 2026-07-27 by operator ruling (R12) -- the previous wording is superseded because it CARRIED THE WRONG SCOPE AND COULD NOT FAIL. Three assertions per DC, each capturing output: (1) ARTIFACT REACHABILITY, asserted on CONTENT with an exit-code predicate. dc0 -> fetch a real PACKAGE-PATH from the mirror (e.g. curl -fsS -o /dev/null http://10.12.8.4/ubuntu/dists/jammy/Release), NOT the bare root. dc1 -> the NAMED check that already exists, scripts/dc-cache-proxy.sh:210-217's proxied fetch of archive AND UCA Release with -w '%{http_code}', run from the node against 10.12.68.4:3142 (the D-135-AMENDED ruled artifact path -- dc1 has NO node-facing mirror, so checking it as one would fail by design). WHY THE CHANGE: the prior text specified curl -sI http://10.12.8.4/, which cannot fail -- measured, curl -sI exits 0 on 404/403/500 (a planted 404 printed 404 File not found with curl exit 0, while curl -fsI exited 22), and the dc0 URL is an nginx autoindex root created EMPTY by dc-mirror.sh:330 before any sync, so / answers 200 whether or not last-sync.status says OK. It reintroduced the existence-not-content class that dc-mirror.sh check was fixed for on the SAME DAY. The dc1 half named no command at all, though the real one already existed. (2) NODE TIME SOURCE -- folded in here, previously homeless. chronyc sources on the node shows the MAAS-served time source and NOT the DC edge, per D-129(iv). This is the surviving REPLACEMENT for struck DoD bullet 6: DOCFIX-204 struck 'NTP from the DC's own OPNsense edge' because D-129(iv) gave the edge no NTP role -- it did NOT strike time verification, and docs/dc-dc-deployment-workflow.md:206 and runbooks/dc-dc-phase4-juju-bundle-per-dc.md:46 both assign the node time source to G17. Until this reshape, chronyc appeared ZERO times in this document, so the check two surfaces required had no home in the gate meant to carry it. (3) An UNRECOGNISED or UNREACHABLE result REFUSES rather than defaulting to success -- 'could not look' is never 'nothing there'. RULING (GA-R5, 2026-07-27). Question as presented: whether to fold time verification into G17 and fix the check to assert content, give time verification its own gate row, or confirm it struck and remove the conflicting surfaces. Operator answer, exact utterance: "Fold time verification into G17 and fix the check to assert content (Recommended)". Both defects live in the same row and share the same ONE-TIME first-boot window, so they are fixed in one edit; splitting them risked one landing without the other. The natural trigger is Stage 5 first boot, when Juju provisions the nodes and they run apt for real; a gated MAAS rescue-boot is the alternative if it must be answered sooner session (each boot operator-approved) OPEN 2026-07-27. WHY THIS EXISTS: the DoD bullet reads "per-DC mirror reachable from nodes", but the READY-handoff ruling (2026-07-23, DOCFIX-200) leaves all 18 nodes powered off in Ready -- MAAS-deploy is SKIPPED and Juju provisions at Stage 5 -- so no node-side probe can run inside Stage 4 at all. GA-R6 E3 forbids a conditional close, so the remainder splits here. RULING (GA-R5). Question as presented 2026-07-27: "The node-side half of bullet 5. Nodes are powered off by the READY-handoff ruling, so no node-side probe can run as things stand. Either a gated rescue-boot check on one node per DC now (closes it inside Stage 4), or split it into its own gate row targeted at Stage 5 first boot (GA-R6 E3 explicitly permits this; a conditional close is not permitted)." Operator answer, exact utterance: "split it into its own gate row". SCOPE NOTE: what stays in Stage 4 is the RACK-side half -- the artifact source answers on its own address with an attested-current sync -- which is what dc-mirror.sh check / dc-cache-proxy.sh check verify (both fixed this session to stop false-greening; capture docs/audit/stage4-mirror-gate-20260727.txt). G17 is NOT a Stage-5 precondition and must not be conflated with one: Stage 5's own bootstrap needs OPEN edge egress for the juju agent stream + snaps (D-135 items 2-3 unbuilt), which is a different path from the apt artifact source this gate covers.

| G18 | IPAM apex completeness for the Octavia lb-mgmt plane -- does the charm-created lb-mgmt-net prefix get BACK-FILLED into the NetBox apex, or is that plane recorded as deliberately charm-owned and out of apex scope? | [R] ruling-type gate (GA-R6 rule 6): closes ONLY on a GA-R5 recorded ruling with the operator's exact utterance, dated, committed and pushed. DEFERRED BY OPERATOR DIRECTION 2026-07-27 until the cloud is LIVE and IPv6 behaviour has been observed -- it is not answerable from artifacts alone. BLOCKING: the deployment may NOT be declared complete while this is open. ANSWERABLE from Stage 5 onward (the prefix exists once Octavia deploys); BLOCKS the FINAL stage close / project close. | operator | OPEN 2026-07-27. WHY IT EXISTS: R8 ruled that Octavia creates and owns its own IPv6 lb-mgmt network. Measured consequence -- the octavia charm exposes NO CIDR, address-family or router configuration option (create-mgmt-network, default True, is the only related option), so the prefix is CHARM-GENERATED and cannot come from the D-111 carve. NetBox is therefore knowingly INCOMPLETE for exactly one plane. That is the authority-inversion concern the Stage-5 grounding audit's lens 7 raised (the apex being back-filled to match a deploy rather than driving it), and it is adjacent to the UNRULED D-136 NetBox-coupled render pipeline -- so ruling it early would pre-empt D-136. OPERATOR DIRECTION, verbatim: "leave this as an open decision that will need a ruling once we have the cloud live and we have a better read on the network and how everything is functioning with the addition of the new IPv6 configurations. Make this a gated decision so we cannot close the project (or whatever phase you think it best ruled in) without a ruling on this item." RELATED AND ALSO RECORDED: the absence of an lb-mgmt :x80 prefix in the VR1 ULA carve is CORRECT under R8, not a gap -- see the D-101 R8 ruling note; a future session must not "fix" it. Options to present at ruling time: (a) back-fill the charm-created prefix into NetBox post-deploy as a documented record; (b) record the plane as charm-owned and explicitly out of apex scope; (c) fold the decision into D-136's render-pipeline ruling if that is taken first. |

7. Version pins (measured; the authority for every pin)

Component Measured value Command (run 2026-07-18) Where measured
OpenTofu v1.12.4 tofu version vcloud (also docs/audit/env-snapshot-20260718.md:26)
libvirt provider dmacvicar/libvirt 0.9.8 (pinned) grep -A2 'provider' opentofu/.terraform.lock.hcl repo lock file
MAAS provider canonical/maas 2.7.2 (pinned) same repo lock file
MAAS 3.7.2-17972-g.35e297c4d (3.7/stable) ssh voffice1 'snap list maas' voffice1
LXD 5.21.5-f2a1a0e (5.21/stable, held) ssh voffice1 'snap list lxd' voffice1
Kernel (host) 6.8.0-136-generic uname -r vcloud
Kernel (voffice1) 6.8.0-136-generic ssh voffice1 'uname -r' voffice1
OPNsense edge 26.7.1 (FreeBSD base 15.1) MEASURED 2026-07-23 via the gated API (GET core/firmware/status -> product_version 26.7.1, capture docs/audit/g13-close-20260723.txt); updated 26.7 -> 26.7.1 in the G13 bundle. DC edges (vr1-dc0/dc1) remain 26.7 office1-opnsense
NetBox (Office1 apex) 4.6.4 per as-built docs/vr1-office1-as-built.md:44; service UP verified (HTTP 302) this session ssh office1-netbox 'curl ... localhost:8000' office1-netbox
Juju 3.6.27 (rev 35621, 3/stable) MEASURED 2026-07-27 on voffice1 -- the headend is the D-128 Plane-2 execution host and this is the client that will bootstrap the controller. Supersedes the 3.6.25 figure recorded 2026-07-24 (also at line 162, kept there as history): an in-channel patch refresh, which D-071 ADOPTED 2026-07-21 explicitly permits (patch-only jumps, in-channel-only refreshes), so this is policy-compliant drift and NOT an incident. The jumphost has NO juju client (measured ABSENT). ssh voffice1 'snap list' (capture docs/audit/stage5-live-measurement-20260727.txt) voffice1
OpenStack client 6.6.0 (python3-openstackclient 6.6.0-0ubuntu2, noble/main) INSTALLED ON voffice1 2026-07-27 -- Stage-5 Phase 0 precondition 0.2, operator-approved. voffice1 is the D-128 Plane-2 host every Stage-5+ script runs from. Verified behaviourally, not by presence: openstack --version -> openstack 6.6.0, --help exit 0, and server list fails CLEANLY on absent auth config rather than crashing. Companion pins: python3-openstacksdk 3.0.0-0ubuntu2, python3-novaclient 2:18.5.0-0ubuntu1. The snap was REFUTED by measurement, not preference: openstackclients has NO Caracal channel (newest stable zed, 2023-03; latest/stable is xena, 2021), and docs/design-decisions.md:638 already records its home-only confinement trap. 6.6.0 is the Caracal 2024.1 client, verified upstream rather than from memory. noble's native OpenStack release IS Caracal 2024.1, so no UCA is needed on this host. STILL ABSENT ON vcloud -- deliberately: D-128 puts this work on the headend. Capture docs/audit/stage5-phase0-20260727.txt. Supersedes the "ABSENT ON BOTH HOSTS" figure measured earlier the same day (kept here as history, per the Juju row's precedent for in-row supersession): the ten Stage-5/6/7 scripts that invoke it (phase-03-admin-openrc.sh, phase-04-network-{create,verify}.sh, phase-04-internal-cert-san-verify.sh, phase-05-{amphora-pipeline,octavia-verify}.sh, phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh) now have a client on the host D-128 runs them from. ssh voffice1 'openstack --version; dpkg-query -W ...' voffice1

The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so the scan does not count them) were ALL fixed or demoted to pointers in sweep Batches 2-3, 2026-07-19 (session changelog).

8. Open decision queue (the exact questions the operator must answer)

  1. D-130: RULED 2026-07-19, ADOPTED (a) ignore_changes -- rotated to docs/design-decisions.md D-130 (question + utterance + captures).
  2. D-100 netem parameter sub-item (gap #11) -- placeholder CONFIRMED STANDING 2026-07-21 (operator, GA-R5; D-100 sub-items block). No open question remains; the item is AWAITING EXTERNAL INPUT (the Roosevelt inter-DC link spec), trigger defined: re-tune via a gated apply when the spec exists, not before.
  3. D-068 -- Vault substrate hardening. Items 2-3 RULED 2026-07-21; item 1 re-scoped plan DRAFTED 2026-07-23 (docs/D-068-vault-migration-plan-draft.md). Q1 RULED 2026-07-23 (rehearsal-scoped EOL risk-acceptance, posture 1b -- amendment in design-decisions.md is the authority). Q2 structural assumption RULED 2026-07-23 (Roosevelt baselines on 1.8/stable + a FUNDED remediation track; path 2a/2b/2c + deadline stay OPEN to Roosevelt design time on re-verified V1-V5). Q3 RULED 2026-07-23 (monthly-review lines delivered into ops-update-procedure 0c; design-time re-verify trigger). SOLE remainder on item 1: Q2 path selection (2a/2b/2c) + deadline, at Roosevelt Vault design time (item 1 Status line: OPEN on exactly that remainder -- reworded at the 2026-07-23 close for scan attribution).
  4. D-071 -- ADOPTED 2026-07-21: all four policy points ruled (monthly review trigger; patch-only controller jumps; standing order; in-channel-only refreshes), each its own GA-R5 exchange -- status line in design-decisions.md is the authority. No open question remains; ops-update-procedure is the policy vehicle.
  5. D-129 -- ALL FOUR sub-decisions RULED 2026-07-21 ((i) COS scrapes the edge in-scope per-DC; (ii) os-frr pinned to Roosevelt design; (iii) per-site Tailscale = dedicated node on metal-admin, edge excluded; (iv) MAAS hierarchy stays the time authority -- status line in design-decisions.md is the authority). No open question remains; the operator-gated live install of the ruled VR1 profile on office1-opnsense (scripts/opnsense-plugins.sh apply vr1-edge)
    • the qga retrofit at that edge's next restart are EXECUTION items tracked at gate G13.
  6. Audit Phase 4 rulings: GA-R1..GA-R6 individually, plus the legal stage-status vocabulary A/B (GA-F10 operator note) -- each gated, one ruling per exchange.
  7. D-132 -- Roosevelt per-DC MAAS topology (PROPOSED 2026-07-21, operator-pinned to the next deployment; three question groups: HA regions per DC, rack-top rack controllers, cross-site backup custody). Present at Roosevelt MAAS design time, not before.
  8. D-131 sub-4 + pinned DNS architectural review (operator-directed 2026-07-21: stack best practices, forwarder security implications, vendor guidance on the "utility nodes" DC-to-DC configuration) -- executes at next-deployment design time; feeds the sub-4 ruling with the LP outcome.
  9. D-137 -- credential mint-and-consolidate pipeline (PROPOSED 2026-07-25, operator-requested: "a durable rule to make sure when accounts are created there is a consolidation that happens every time"). Diagnosis is MEASURED: the SEC-009 audit is DECLARATION-based, so an undeclared secret is structurally undetectable -- creds-audit vr1-office1 read CLEAN on 2026-07-15 while four region-VM secrets minted 2026-07-13 sat undeclared, and admin.pass surfaced only 2026-07-25 (SEC-020). It is also wired in exactly ONE place and as PROSE (phase-3 runbook:498), not in preflight/cloud-assert/repo-lint/gauntlet -- and that line did not fire at EITHER DC standup (dc0 3 + dc1 1 undeclared at this session's open). THREE forks await a ruling, one exchange each (GA-R5): enforcement strength (advisory / preflight-blocking / plus a PreToolUse guard); --remote discovery scope (declared-directories vs broader sweep, a tenant-isolation concern); policy home (this D as authority with SEC-009 demoted to a pointer, vs policy stays in the ledger). NOT implemented -- PROPOSED means present options, never build. SUB-RULING 1 RULED 2026-07-25 (GA-R5, utterance quoted in the D-137 Status block): "Blocking in preflight" -- the check lands as a new Pn in scripts/preflight.sh and HARD-FAILS on any expected-but-absent / undeclared / per-DC-asymmetric credential. The PreToolUse guard and advisory-only were NOT adopted. SUB-RULING 2 RULED 2026-07-25: "Derive manifests from matrix" -- the matrix is SINGLE SOURCE, --render regenerates creds-manifests/*.manifest, gauntlet fails on rendered-vs-checked-in drift; accepted cost is that manifests become generated (their governance prose must become matrix fields, not be dropped). SUB-RULING 3 RULED 2026-07-26: "Declared locations only" -- a creds-manifests/vm-secret-locations list bounds --remote absolutely (no tenant surface touched, D-069 preserved). To be faithful the list must include the headend shadow stores (SEC-022), the region maas-secrets dir, and the three dirs outside the SEC-009 convention found by the creation-point inventory. SUB-RULING 4 RULED 2026-07-26: "D-137 is the authority" -- D-137 becomes the credential-lifecycle policy authority and the SEC-009 convention block demotes to a pointer (its founding history stays in the ledger as history); gates may then cite a D-number instead of an exposure register. SUB-RULING 5 RULED 2026-07-26: "Fold in as a D-137 invariant" -- the invariant is ONE IDENTITY SERVES ONE PRINCIPAL TYPE, enforced by the matrix principal column, so the SEC-020 conflation becomes machine-detectable and needs no separate D-number. ALL FIVE SUB-RULINGS RULED -> D-137 is ADOPTED 2026-07-26 and implementation is UNBLOCKED, with the build spec at docs/D-137-implementation-plan.md (Status line in design-decisions.md is the authority). Expect the first run to be RED by design: admin serves both a human and a service row, which is the defect the invariant names. Research capture: docs/audit/creds-creation-points-20260725.md -- its APPENDIX now carries the FULL per-row inventory (55 MINT rows with file:line, host, destination, stage and human/service classification), transcribed in-repo 2026-07-26 so the matrix SEED does not depend on a session transcript. TIER 1 BUILT 2026-07-26 (offline/STATIC half only; tiers 2-3 NOT built, and the preflight Pn of ruling 1 is NOT wired -- see the sequencing question below). Shipped: creds-matrix.tsv (72 rows), creds-matrix-notes.md, scripts/creds-matrix.py (S1 schema / S2 manifest coverage both-bounds / S3 render drift / S4 mint-ref resolution / S5 per-DC symmetry / S6 ruling-5 principal invariant / S7 notes integrity), harness tests/creds-matrix/run-tests.sh 24/24; gauntlet ALL GREEN (80), repo-lint 0-fail. SCHEMA AMENDED 10 -> 12 columns (operator-ruled 2026-07-26, "go with the 12-column amendment"): custody + notes-ref added and rows re-keyed to (credential, location), because a read-first round-trip check proved the 10-column form could not carry what ruling 2 forbids dropping -- and because ruling 3 puts four deliberate, reasoned credential copies INSIDE --remote's declared locations, where creds-audit.sh:63-67 would report every one as UNDECLARED. Detail + rationale in docs/D-137-implementation-plan.md; OPS under GA-R3, the five sub-rulings are untouched. RED BY DESIGN AND CORRECTLY SO -- 5 findings, do not "fix" by deleting rows: the ruling-5 identity conflation on maas-region-admin (its own SEC-020 defect), 3x EXPECTED-BUT-ABSENT for SEC-021 (dc0 declares neither an edge API credential nor a jumphost-consolidated power key, both of which dc1 declares), and the S5 asymmetry of dc0's divergently-named headend power key. 27 rows carry mint-ref=operator-terminal = the research FINDING 1 reproducibility debt, admitted and counted, not faulted. Acceptance test PARTIALLY met (corrected from the plan's original text): SEC-021's DECLARATION half is tier-1 detectable and is reproduced; SEC-022/-023 and SEC-021's on-disk half need tier 2. OPEN SEQUENCING QUESTION for the operator: ruling 1 lands the check as a BLOCKING preflight Pn and the first run is red, so wiring it hard-fails preflight.sh until the credential defects are remediated; wiring-as-ruled is the default and deferring until after per-row remediation is the departure. RESOLVED + EXECUTED 2026-07-26. Question as presented: wire the blocking Pn as ruled, or defer it until after per-row remediation. Operator answer, exact utterance: "wire tier 1 as the blocking preflight Pn". TIER 1 IS NOW WIRED as preflight.sh P5, blocking, ahead of the stage-2 reminders block so its verdict participates in the deploy decision; it FAILS CLOSED if the checker is absent (a missing file made python3 exit 2, which note maps to WARN -- so deleting the gate would have downgraded it to a warning; harness T9 encodes this). Tier 2's Pn is NOT wired and remains a separate decision: it needs --remote/--privileged and a caller-supplied --pending-stage, none of which belong in an unattended gate. CONSEQUENCE, STATED PRECISELY: preflight.sh exits 1 and P5 is one of the reasons -- but preflight was ALREADY exiting 1 before this change (P4: overlays/octavia-pki.yaml absent, and MAAS unreachable from the jumphost). P5 adds a fifth reason to an already-red gate; it did NOT flip preflight from pass to fail, and no deploy path that was open is closed by it. TIER 2 TOOLING BUILT 2026-07-26, NOT YET RUN LIVE. Shipped: creds-manifests/vm-secret-locations (ruling 3's absolute bound -- jumphost creds folders, the headend shadow stores of SEC-022, the region secrets dir of SEC-020, the three dirs outside the SEC-009 convention, the in-clone PKI overlay, and the DOCFIX-175 plaintext tfstate; tenant dirs are LOCAL-only so no tenant surface is reachable and D-069 holds by construction); creds-matrix.py --tier2 [--remote] (E1 expected-but- absent / E2 mode / E3 undeclared-at-a-declared-location, with an unreachable host SKIPPED explicitly because "could not look" must never read as "nothing there", and a --pending-stage selector so a not-yet-reached mint stage defers instead of failing -- the CALLER supplies it, the script carries no status claim, GA-R1); creds-audit.sh sprawl globs WIDENED for the SEC-023 blind spots (admin.pass, *.apikey, *.key, *.pem, *_ed25519, *_rsa, *openrc*) -- the old six patterns could not have seen the SEC-020 secret or the predicted Stage-5 ~/admin-openrc. Harnesses creds-matrix 33/33 and creds-audit 13/13 (was 7); gauntlet ALL GREEN (80), repo-lint 0-fail. LIVE TIER-2 SWEEP RUN 2026-07-26, read-only, no sudo (capture docs/audit/d137-tier2-sweep-20260726.txt; jumphost local + voffice1 + office1-netbox, stat over ssh, metadata only). SEC-021's ON-DISK half is now REPRODUCED as a named failure -- vr1-dc0-maas-power_ed25519{,.pub} are genuinely ABSENT from the dc0 jumphost creds folder, not merely undeclared. The sweep ALSO corrected two of its own false-greens, both found by running it: (a) an unprivileged [ -d ] on a root-owned directory is indistinguishable from absent, so /root/maas-secrets and /root/netbox-secrets first reported "does not exist" -- they are now correctly reported UNREADABLE ("could not look" is never "nothing there"), which is a FAIL, not a skip; (b) a role with any unprobed location no longer lets its other locations' listings manufacture false EXPECTED-BUT-ABSENT findings -- 14 headend/netbox rows are explicitly NOT JUDGED instead. STILL OUTSTANDING: the two root-owned directories need a privileged read, so SEC-022's shadow-store verification and the SEC-020 region secrets remain unconfirmed; that run is a remote-sudo shape and is operator-gated. ESCALATION PATH WIRED BUT BLOCKED 2026-07-26: --privileged adds a sudo -n retry attempted ONLY where an unprivileged probe returned unreadable, so the privileged surface stays as small as the ruling-3 bound keeps the search surface (still metadata only, stat, never content). The operator APPROVED the run, but the Claude Code AUTO-MODE CLASSIFIER denied the remote-sudo shape -- the same wall recorded at the 2026-07-23 close, whose noted fix is manual permission mode (a targeted ask rule is the alternative). NOT worked around. PRIVILEGED SWEEP COMPLETED 2026-07-26 (capture docs/audit/d137-tier2-privileged-20260726.txt; supersedes the unprivileged capture). ROOT CAUSE of the block was NOT the classifier overriding a rule: Bash(ssh * sudo *) was ALREADY in the project ask list, but the pattern needs a literal space before sudo and the command was ssh <host> 'sudo ...' -- the quote meant NO rule matched, so it fell through to the classifier. Fixed by adding the quoted variants (ssh *'sudo *, ssh *"sudo *, ssh *sudo -n *) plus a targeted ask rule for the privileged invocation; all are ask, never allow. Bash(ssh * virsh *) carried the IDENTICAL latent gap; it was flagged-not-fixed at discovery (hard rule 1) and then FIXED 2026-07-26 under operator direction (commit 6d43619, quoted variants added). MEASURED RESULTS -- capture docs/audit/d137-location-listing-20260726.txt (a stat listing; the tier-2 capture cited here previously is a checker VERDICT file and contains none of these values -- a GA-R1 rule 2 defect found by the committee and corrected in this commit). Region secrets dir: admin.apikey, admin.pass, db.pass, lxd-trust.pass (0600) -- precisely the SEC-020(i) carve-out list. Netbox dir: admin.pass, api.token, secret_key (0600). SEC-022 shadow stores confirmed. The claim that "ZERO undeclared files remain, so SEC-020/SEC-022 are fully accounted for" is WITHDRAWN -- it rested on a site-blind check (see the committee block below). INFERRED-FILENAME MISS (hard rule 2), corrected count: the session first reported SIX; the committee measured ~21, because only the rows the sweep physically touched were re-measured. .maas.cli was itself never measured and is WRONG -- the MAAS snap CLI stores its profile at ~/snap/maas/current/.maascli.db (measured this session). Matrix 77 rows at that point. The 9 findings then reported were true so far as they went, but the run that produced them is superseded below. COMMITTEE AUDIT 2026-07-26 (6 independent read-only lenses: correctness, coverage, claim-verification, ruling-fidelity, record-integrity, Roosevelt-transfer). VERDICT: the register's DESIGN holds, but TIER 2's VERDICTS ARE NOT TRUSTWORTHY as delivered and this document overstated what was verified. Every defect below was REPRODUCED, and FOUR lenses converged independently on the first one. Superseding figures (these are current; earlier figures in this entry are history): matrix 77 rows, creds-matrix harness 35/35, creds-audit 13/13, gauntlet ALL GREEN (80). The earlier "14 rows NOT JUDGED" was from the unprivileged run; the privileged run reports 2. The widened sprawl glob is *.pass, not admin.pass. Confirmed false greens (a real missing or misplaced credential passes a green sweep): (1) tier 2 is SITE-BLIND -- observed/declared are keyed by host-role only, so a file in the wrong site's folder satisfies the row; this masked SEC-021's opnsense-api.txt on-disk absence, so "SEC-021's on-disk half REPRODUCED" is corrected to the power-key artifacts ONLY; (2) probe_remote reports UNREACHABLE for a location it successfully read (empty-but-readable dir, or absent literal-file path), which gates the whole role and converts every absence FAIL at that role into an [ok]; (3) literal-file locations get no absent/unreadable detection at all -- the T33 fix covered only dir/* patterns; (4) an empty locations file bypasses ruling 3's refusal and affirms existence over zero locations; (5) no non-empty floor -- a 0-row matrix passes every check; (6) a mint-ref pointing at a directory crashes with exit 1, indistinguishable from findings, and S5/S6/S7 never run; (7) E2 MODE is custody-gated so 43 of 77 rows are never mode-checked; (8) S5's dc1->dc0 direction is untested -- deleting it leaves the harness green (the in-repo both-bounds precedent, in the symmetry check itself). Also outstanding: ruling 4's SEC-009 demotion is NOT done and its stated trigger (tier 2 built) has passed; the 12-column amendment is recorded in no D-numbered surface; sec-ref mis-attribution recurs (both juju-maas-user rows cite SEC-020; correct is SEC-018/-019); the libvirt SSH power password (standing rotation obligation, reenroll-hosts.sh:22-25) has no row; and T24 asserts literal finding strings, so remediating the ruling-5 conflation would turn the GAUNTLET red -- the test punishes the fix it exists to protect. Remediation is IN PROGRESS under operator direction ("Do as many as you can autonomously"); this block is the authority on what is fixed. REMEDIATION COMPLETE 2026-07-26 (phases 1-3), capture docs/audit/d137-tier2-postcommittee-20260726.txt (supersedes the pre-committee privileged capture, whose verdicts predate the site-scoping fix and are NOT comparable). ALL EIGHT false greens are FIXED and individually regression-locked; harness 44/44 (was 35), creds-audit 15/15, gauntlet ALL GREEN (80), repo-lint 0-fail. PROOF THE SITE FIX WORKS: E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt' now appears. That on-disk absence was masked in EVERY prior run, so SEC-021's on-disk half is NOW genuinely reproduced in full (the earlier withdrawal stands as history). The ~21 inferred filenames were re-measured against their mint-refs (octavia's 8 real basenames + its three SUBDIRECTORIES, which the bare ~/octavia-pki/* pattern matched none of; vault's init.txt; the tenant rows' <client>- instance prefix, now supported by a placeholder matcher); sec-ref mis-attributions corrected (both juju-maas-user rows SEC-020 -> SEC-018/-019; tfstate SEC-009 -> none); admin.pass's mint-ref moved from the consumer (:452) to the mint (:450); the libvirt power password and vault-ca-root added as rows. Matrix 81 rows. Findings 35 -> 13, all TRUE: SEC-021 (3x S2 + 3x E1), 3x S5 power-key asymmetry, the ruling-5 conflation, one SEC-022 shadow-store gap, SEC-024, and 2 rows disclosed as UNCHECKABLE. NEW EXPOSURE FOUND BY THE FIXED CHECKER -- SEC-024 OPENED: opentofu/terraform.tfstate.backup is mode 0664 (group- and world-readable) and carries the Office1 MAAS API key in plaintext per DOCFIX-175; the live state file is correctly 0600. Invisible to every prior control because the world-readable check was custody-gated and the siblings were undeclared. REMEDIATED 2026-07-26 (mode): operator-approved chmod 600 on terraform.tfstate.backup and .pre-G16-20260721, read-back verified -- all four state files now 0600 and the E2 finding cleared. SEVERITY CORRECTED first: the row's original "group- and world-readable" overstated it -- opentofu/ is 0700 and ~ is 0750, so no other account could traverse to it, and neither file is git-tracked, so SEC-004 was never implicated. It was defence-in-depth, not live exposure. The RETENTION question is RULED 2026-07-27 (GA-R5). Question as presented: whether to delete both pre-* state-surgery snapshots (cleanest, closes SEC-024 fully), keep both and let P5 keep watching them, or keep pre-G6 and delete pre-G16; noting both gates are CLOSED, both carry the MAAS API key in plaintext per DOCFIX-175, and deletion is irreversible with no other copy of that pre-surgery state. Operator answer, exact utterance: "Keep both". So terraform.tfstate.pre-G6-20260719 and .pre-G16-20260721 are RETAINED as the only record of what the state looked like before the two direct tfstate edits; both are 0600 and neither is git-tracked. SEC-024 stays OPEN as a standing WATCH rather than an open remediation: its mode defect is remediated, but the umask CAUSE was out of ruled scope, so a future apply may rewrite 0664 -- P5 is the detection. The key reaching state files in plaintext at all remains DOCFIX-175, rotation owed under SEC-018/-019. STILL OUTSTANDING (not done, not silently dropped): ruling 1's TIER 2 remote Pn (tier 1 + tier-2-local ARE wired -- see the tier-2 gate block below); --render's source-field derivation and the manifest flip to generated output; and the cardinality field remains largely inert with a one-token S5 bypass (per-DC -> singleton). (A dangling fragment here, left by an earlier edit in this same session, was repaired at session close -- noted rather than silently fixed, since CURRENT-STATE is the status authority and its defects are worth seeing.) CONSOLIDATION BATCH EXECUTED 2026-07-27 (operator question: is there a consolidated set of login creds on vcloud for every account that exists; operator direction after the audit: "clear the whole consolidation batch first"). Capture docs/audit/creds-consolidation-audit-20260727.txt; detail docs/archive/changelogs/changelog-20260727-creds-consolidation.md. Superseding figures: matrix 82 rows, creds-matrix harness 60/60 (was 56; V2 had shipped with ZERO cases), creds-audit CLEAN on all three sites, gauntlet ALL GREEN (81), repo-lint 0-fail, findings 13 -> 7. VERIFIED POSITIVE, both previously only asserted: the MAAS account set is COMPLETE -- all 6 accounts enumerated live (maas admin users read) are accounted for (admin + operator passwords on vcloud, juju-vr1-dc0/dc1 random+unstored BY RULING with their API keys present, MAAS/maas-init-node MAAS-internal) -- and tier-3 V1 now MEASURES maas-admin-password byte-identical to the headend source-of-record, so the stale-trap risk SEC-020 records is clear as of this date. DONE: dc0's SEC-012 power key consolidated to vcloud + .pub DERIVED (SEC-021(b) as written; measured first -- the headend maas-virsh_ed25519 and the snap's id_ed25519 are the SAME key, it IS dedicated (distinct from the dc0 service key), and dc0 using the snap default identity is SEC-016's ruled design, so NO re-mint and no live power path touched); dc1 svc .pub backfilled to the headend; NetBox web-GUI admin password consolidated (SEC-025 OPENED for the at-rest exposure the copy creates -- open rows 20 -> 21); V2 taught the ruled-deferral state so SEC-006's standing "revoke at completion of this deployment" ruling is ACKNOWLEDGED (still naming the credential as live and exposed) instead of failing every run -- reissuing it would have CONTRAVENED that ruling. Tier 3 is not in preflight P5, so the blocking gate's behaviour is unchanged. RESIDUAL 7 findings, expected, NOT green: dc0-edge-api x2 (the opnsense-api.txt re-mint is a live edge mutation, deliberately EXCLUDED from the batch -- sole remaining SEC-021(a) item), S5 x3 (RULED by SEC-016, the register needs a ruled-exception mechanism -- operator decision), S6 conflation x1 (the SEC-020 defect), E4 uncheckable x2 (Stage-5/6 rows). NEW FINDINGS LOGGED NOT ACTIONED: (i) NO registered root/console credential at EITHER DC edge -- measured absence of row, manifest entry and SEC row; what those passwords ARE is UNKNOWN and deliberately unprobed (hard rule 2), vector is the LAN-reachable GUI + serial console, not SSH (key-only, proven); (ii) two structural blind spots that let (i) hide -- S5 compares only cardinality=per-DC while all six per-site rows are office1-only, and vm-secret-locations declares no rack/edge/cloud/unit/client location though the checker accepts them (SEC-015 was rack-resident, so the class is real). The lesson generalises D-137's founding argument one level up: absence of a ROW is invisible to the register, so enumerating what EXISTS is a distinct control from auditing what is declared. RE-RUN 2026-07-27 (Stage-5 grounding audit), operator-authorized privileged sweep: python3 scripts/creds-matrix.py --tier2 --remote --privileged -> exit 1, capture docs/audit/stage5-creds-privileged-20260727.txt. This is a RE-CONFIRMATION of the 2026-07-26 privileged run against the current 82-row matrix, NOT a newly-closed item. Result: still exactly 7 findings, the same set (S2 dc0-edge-api, S5 x3 power-key asymmetry, S6 conflation, E1 dc0-edge-api on-disk, E4 x2 uncheckable) -- no drift in a day. All three root-owned locations read successfully via ESCALATION (sudo -n, metadata only): /root/maas-secrets/*, /var/snap/maas/current/root/.ssh/*, /root/netbox-secrets/*. ZERO E3 findings -- no undeclared file at any declared location. Note the checker's own honesty on the V1 arm, worth preserving: "V1 provenance: no identity had two digestible copies -- NOTHING was verified here; this is a skip, not a pass." QUEUED FINDINGS CAPTURED 2026-07-26 at session close: docs/audit/queued-findings-20260726.txt -- an end-of-session sweep for content that existed ONLY in the session transcript. Part A: the three secrets-storage items NOT already repo-carried (Tang/Clevis as the no-HSM unseal mechanism; MAAS 3.7's Vault integration MEASURED status: disabled, with the MAAS/Vault circular dependency that must be designed around before enabling it on bare metal; a Vault SSH CA to retire the static keypairs) plus sequencing advice -- framed so a future session does not re-propose what D-068's analysis and D-137 item 2(a) ALREADY carry. Part B: ten committee findings ACKNOWLEDGED but deliberately NOT acted on, the most consequential being that mint-ref line pins rot SILENTLY (S4 checks existence and EOF, never content, so every pin becomes wrong-but-passing when the runbooks are rewritten for Roosevelt) and that ruling-5 REMEDIATION and ruling-5 EVASION are indistinguishable to S6. Part C: items deferred by ruling, recorded so they are not later mistaken for oversights. NONE of it is ruled or built; cardinality (B6) needs an operator ruling.

9. Additional defect found while authoring (FIXED in sweep Batch 0.3,

2026-07-19 -- wrap-aware exclusion, GA-F15; history below)

ledger-scan.sh's mention-derived next-free counters (DOCFIX, BUNDLEFIX) are SELF-INFLATED by any doc that quotes a "next-free" value and lets the hyphenated token wrap onto a line without the words "next-free" -- the per-line exclusion filter (scripts/ledger-scan.sh:121-129, the grep -viE 'next[- ]free' at :124) then counts the quote as a real assignment; the script's own CAUTION comment (:112-120) documents exactly this failure class. It happened TWICE inside the audit itself on 2026-07-18: the Phase-1 env snapshot's wrapped next-free line inflated the BUNDLEFIX counter (051 -> reported 052), and this document's own first draft of this very section inflated the DOCFIX counter the same way while asserting DOCFIX was unaffected. Both audit surfaces were reworded token-free the same day and the counters re-verified at their true values (D=130, DOCFIX=197, BUNDLEFIX=052 -- see GA-F15). The D counter is header-authoritative and was never affected. Batch 0.3 hardened the scanner: an excluded line now also suppresses the immediately following line (the wrap case); counters re-verified unchanged post-fix. The authoring discipline stands regardless: never write a hyphenated register-token quote of a next-free value into any doc; state the numbers token-free as this section does.

10. How to verify this document (cold-session re-derivation)

Run read-only, from the repo root:

  • Repo identity: git rev-parse HEAD; git status --short (this doc was authored at e999b03, clean tree).
  • Applied set: tofu -chdir=opentofu state list (expect the 20 resources in section 2.1); virsh list --all; virsh dominfo voffice1 | grep -i autostart (and office1-opnsense).
  • Authored-not-applied: grep -n '^module ' opentofu/main.tf (12 blocks; vvr1_dc0 + vr1_dc0_uplink absent from state list); ls opentofu/vr1-dc0-substrate/ (no *.tfstate).
  • Plan count: read docs/audit/outer-plan-20260718.txt line 428. Do NOT re-run tofu plan casually against live state; if a fresh capture is taken, it must be written to a new dated capture file and cited here.
  • Versions: tofu version; ssh voffice1 'snap list maas lxd; uname -r' </dev/null; uname -r; grep -A2 provider opentofu/.terraform.lock.hcl.
  • Open decisions + SEC + next-free: bash scripts/ledger-scan.sh (BUNDLEFIX caveat: section 9); decision status lines: grep -n '^## D-' docs/design-decisions.md then read each Status line -- a decision's Status line in that file is the ONLY ruling authority.
  • Gates: sections 1a and 4-7 of docs/audit/grounding-audit-charter.md; docs/audit/grounding-audit-20260718.md for GA-F01..F14.
  • Live service probes: ssh office1-netbox 'curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/' </dev/null (expect 302); ssh office1-tailscale 'tailscale status | head -1' </dev/null.

What this document is NOT built from and you must not rebuild it from: the prose of the 95 docs/changelog-*.md files, the docs/session-ledger.md narrative, or auto-memory -- all proven to carry false status (GA-F14, GA-F06..F08).

11. Operator signature (G11)

SIGNED 2026-07-19 (re-signature at audit exit; REPLACES the 2026-07-18 signature per GA-R1 rule 7 -- git history keeps it). Question as presented (Batch 6 item 6, 2026-07-19): read this document top to bottom, then provide the signature statement. Operator answer, exact utterance: "Reviewed, approved, continue." This document is the signed status authority; charter Phase 6 item 5 MET at this baseline (repo HEAD at signing recorded in the close commit).