Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / phase3-grounding-test-20260718.md
@JANeumatrix JANeumatrix 1 day ago 21 KB GA Phase 3: grounding test PASSED -- 3 probes, 21/21; scorecard + line-neutral amendments
Raw Blame History

Phase-3 fresh-agent grounding test -- scorecard and holes map (2026-07-18)

Charter section 4. Three clean-context probes (A/B/C), identical prompts, independent runs, no shared scratchpad, repo-only at HEAD c49da79, read-only, live commands forbidden. Each answered the 7-question set with path:line citations. Scored by the audit session against ground truth (docs/CURRENT-STATE.md + the Phase-1 captures); load-bearing probe citations spot-verified against the tree before scoring (opentofu-validate.sh:189 PASS line; platform-traps.md:142 in-place bounce; workflow doc :311-316 config.xml clobber). Raw probe returns are in the appendices (punctuation normalized to ASCII; content verbatim).

Scorecard

Q Probe A Probe B Probe C Verdict
1 stage + next action PASS PASS PASS unanimous, correct (Stage 3/Phase 2, apply STOPPED, next = audit Phase 3 + G4 verifications)
2 applied vs authored (3+3) PASS PASS PASS all from state/measurement citations, none from changelog prose
3 open gates + closers PASS PASS PASS all reproduced G1-G15 with owners
4 pre-apply commands + required reading PASS PASS PASS all three concluded NO APPLY IS JUSTIFIABLE TODAY (expected triple UNRESOLVED pending D-130) -- the correct answer
5 three most dangerous traps PASS PASS PASS seed-replace bounce + KiB-vs-MiB unanimous; third pick varied (no-cpu-block/svm, in-place bounce, config.xml clobber) -- all real, all cited
6 D-NNN ruling authority PASS PASS PASS unanimous: design-decisions.md Status line ONLY; changelogs not trustable (GA-F14)
7 DC0 redeploy entry doc PASS PASS PASS all reached runbooks/dc-dc-phase2-tofu-dc-substrate.md and correctly answered "NOT without leaving the repo" (operator gates + out-of-repo secrets); first-opened doc DIVERGED -> hole H1

Score: 21/21 answers PASS. Zero wrong, zero UNANSWERABLE, zero guessed values. No question was failed by any probe; the double-weight rule (all-three-fail) had nothing to attach to.

Primary success metric (operator-endorsed, charter 1a): a cold session can stand up from the repo alone and know where it is -- MET at this checkpoint. All three probes oriented CLAUDE.md -> SKILL.md routing table -> docs/CURRENT-STATE.md without being steered.

NOTE: this is the Phase-3 baseline, not the exit gate. Charter Phase-6 item 3 requires a NEW trio at audit exit, scoring 7/7 on at least two of three.

Holes map (the Phase-5 documentation queue from this test)

  • H1 (minor, routing): the repo names no SINGLE entry document for the DC0 deploy. Probe A opened the phase-2 runbook (via the workflow gap register), probe B opened dc0-deploy-readiness.md (its own "a FRESH session starts the deployment from here" banner), probe C opened CURRENT-STATE.md then the workflow doc then the runbook. All converged on the same runbook, so this scored PASS, but the readiness doc's falsified DEPLOY-READY banner (GA-F02, frozen) makes the ambiguity live. AMENDED same-day: the G9 row of CURRENT-STATE.md now names the canonical entry path. Residual (banner correction) already queued as GA-F02 in Phase 5.
  • H2 (note, no action): the charter's Phase-6 item 4 still cites the standing 5/0/6 gate; probes B and C both met it and both correctly resolved precedence to CURRENT-STATE.md section 5 (UNRESOLVED pending D-130). The charter's own escape clause ("if the audit legitimately changes the expected counts...") covers this; no amendment.
  • Observations (positive): the stale CLAUDE.md hostname (GA-F05) misled no probe; probe A correctly inferred from committed audit state that it was itself a Phase-3 probe -- the repo carries the audit's own state cold.

Disposition

  • G1 CLOSES with this scorecard (evidence: this file + appendices).
  • CURRENT-STATE.md amended (line-neutral, within the ~350-line budget): G1 evidence cell -> run + result; section-1 phase status; G9 row -> the canonical entry path (H1). The amended version is the G11 review copy.
  • Next per sequencing directive: operator personally runs section 10's commands and reads CURRENT-STATE.md top-to-bottom as the substance of the G11 signature; Phase 4 ruling drafts then go to the operator one per exchange (GA-R5 discipline).

Appendix A -- probe A return (normalized to ASCII)

  1. Stage / next action. Stage 3 / Phase 2 of the VR1 DC-DC track ("OpenTofu builds each DC substrate"), with the DC0 outer tofu apply (deploy step A) STOPPED at its own gate, and a grounding audit with a FULL FREEZE on fixes in effect (docs/CURRENT-STATE.md:27-42; stop commit 43b29d7; charter commit 148dcef). Audit Phase 1 and Phase 2 are done, and Phase 2 was ACCEPTED on external review (commit c49da79; docs/audit/grounding-audit-20260718.md, "Phase-2 review disposition"). The very next action is audit Phase 3 -- the fresh-agent grounding test, three clean-context probes scoring this exact 7-question set (gate G1, docs/CURRENT-STATE.md:210; charter:135-158) -- this session appears to be one of those probes. Then probe holes fold into CURRENT-STATE.md, operator signs it (G11), then Phase-4 rulings (c49da79 sequencing note 1).

  2. Applied vs authored-only. Applied (from tofu state + measurement, docs/CURRENT-STATE.md:49-97): (a) the voffice1 headend VM (MAAS 3.7.2 + LXD 5.21.5), (b) the office1-opnsense edge VM (26.7), (c) the office1-local network, the three D-100 mesh-link networks, the six live vr1-dc0 plane networks, and the dc0/dc1 storage pools -- 20 resources total (2.1). Authored-but-NOT-applied (docs/CURRENT-STATE.md:133-157): (a) module "vvr1_dc0" -- the DC0 containment VM (opentofu/main.tf:360), (b) module "vr1_dc0_uplink" -- the D-125 simulated ISP net (opentofu/main.tf:341), (c) the entire inner root opentofu/vr1-dc0-substrate/ (no state file); also D-129 scripts/opnsense-plugins.sh (built, not live-applied), SEC-010 enforcement, and phase3-6 runbooks.

  3. Open gates. G1-G15, tabulated with owners and closers at docs/CURRENT-STATE.md:206-224. Highlights: G1 Phase-3 probes (this exercise); G2 Phase-4 GA-R1..R6 rulings (operator); G3 Phase-5 repair sweep of GA-F01..F15 (freeze until then); G4 the two D-130 throwaway verifications (v7/v8); G5 the D-130 seed-volume mechanism ruling (operator, quoting G4 output); G6 state reconcile of autostart+seed without bouncing guests; G7 a fresh plan capture equal to the post-D-130 expected triple (impossible today -- the triple is deliberately unrecorded, :199-201); G8 same-session re-verify the 6 planes are empty; G9 the gated DC0 outer apply itself; G10 deploy steps B-E in-sequence gates; G11 operator signature on CURRENT-STATE.md; G12 vr1-dc1 addressing ruling; G13 D-129 residuals; G14 nine OPEN SEC rows; G15 D-068/D-071 rulings.

  4. Before any tofu apply. Check the workflow doc's tooling gap register for open blockers (SKILL.md:141-142), open the logged shell bash scripts/run-logged.sh (CLAUDE.md; docs/dc0-deploy-readiness.md:167), run bash scripts/opentofu-validate.sh -- its output must read "PASS: opentofu-validate (

    )", exit 0 (scripts/opentofu-validate.sh:189-190; runbook checklist runbooks/dc-dc-phase2-tofu-dc-substrate.md:746) -- then a tofu plan captured to a dated file whose count must equal the expected triple recorded in CURRENT-STATE.md section 5, plus the same-session planes-empty check (G7/G8, docs/CURRENT-STATE.md:216-217). Today that reading DOES NOT EXIST: last capture is 7/2/7 (docs/audit/outer-plan-20260718.txt:428), the old 5/0/6 gate is falsified, and "this document may not be used to justify an apply" until D-130 is ruled (docs/CURRENT-STATE.md:183-201). So the correct pre-apply answer today is: no apply is possible.

  5. Three most dangerous traps. (i) The live one: the provider's staging-ISO loss after host reboot forces a seed-volume REPLACEMENT, so an unscoped outer apply would BOUNCE the running Office1 headend and edge -- this is what stopped DC0 (GA-F01/D-130; docs/CURRENT-STATE.md:100-119; docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228; kin to trap 1e '"updated in-place" does not mean the guest stays up', platform-traps.md:142). (ii) Bare memory in libvirt provider 0.9.x is KiB not MiB -- 1024x too little RAM, FreeBSD triple-faults; tofu validate cannot catch it (platform-traps.md:50-79). (iii) Missing cpu block = generic CPU = no svm = nested KVM impossible (platform-traps.md:107) -- fatal here since the whole rehearsal runs nested to depth 4 (docs/dc0-deploy-readiness.md, C3). Runners-up: MAAS 3.6/3.7 incompatible with LXD >= 6.7 (D-114, :245); AppArmor blocking qemu on non-default pools (:216).

  6. D-NNN status authority. docs/design-decisions.md -- grep "^## D-NNN" and read that entry's Status line; "a decision's Status line in that file is the ONLY ruling authority" (docs/CURRENT-STATE.md:335-337; routing at SKILL.md:95-96). No, you cannot trust a changelog's claim: Phase 1 proved status surfaces contradict each other (12 contradiction groups, docs/CURRENT-STATE.md:14-18), changelog prose is explicitly listed as what CURRENT-STATE.md "is NOT built from and you must not rebuild it from" (GA-F14, docs/CURRENT-STATE.md:344-347), and proposed rule GA-R2 states changelogs "are NEVER citable as status authority (Phase 1 proved they lie)" (charter:169-174).

  7. Redeploy DC0 from scratch. Open runbooks/dc-dc-phase2-tofu-dc-substrate.md (the Stage-3 DC0 substrate runbook; routed via docs/dc-dc-deployment-workflow.md's gap register first, SKILL.md:82,141-144 -- note nothing DC0 is applied yet, so "redeploy" is deploy steps A-E, docs/dc0-deploy-readiness.md:170-179; if partial state existed, runbooks/dc-dc-teardown-rollback.md first). It does NOT get you there without leaving the repo: (a) the audit freeze + gates G1-G9 require operator rulings (D-130, G11) that exist in no repo file (docs/CURRENT-STATE.md:199-201,206-224); (b) required secrets live outside the repo -- ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 and TF_VAR_maas_api_key (docs/dc0-deploy-readiness.md:53-58,66-70; CLAUDE.md forbids reading them into context); (c) every mutation is individually operator-approved (CLAUDE.md hard rule 3).

Orientation note: I oriented from CLAUDE.md -> .claude/skills/ openstack-cloud-ops/SKILL.md (the repo's declared routing layer), which routed me to docs/CURRENT-STATE.md -- the self-declared single status authority (docs/CURRENT-STATE.md:1-18) that supersedes all other status surfaces -- cross-checked against the audit charter/register and git log (HEAD c49da79).

Appendix B -- probe B return (normalized to ASCII)

  1. Stage / next action. VR1 DC-DC rehearsal, Stage 3 / Phase 2 ("OpenTofu builds each DC substrate"), branch dc-dc-stage3-phase2-dc-substrate; the DC0 outer tofu apply (deploy step A) is STOPPED at its own gate (commit 43b29d7) and a grounding audit with a FULL FREEZE on fixes is in progress (commit 148dcef). Audit Phase 2 (CURRENT-STATE.md) is ACCEPTED by external review (commit c49da79). Very next action: audit Phase 3 -- the fresh-agent grounding test (gate G1: three clean-context probes scored against the status doc). Evidence: docs/CURRENT-STATE.md:24-42, :210 (G1); docs/audit/grounding-audit-charter.md:135-158; commits 43b29d7, c49da79.

  2. Applied vs authored-only (from tofu state + measurement, per docs/CURRENT-STATE.md:49-73 and :133-157). APPLIED: (a) voffice1 headend VM

  3. its volumes (state + running, module.voffice1.*, CURRENT-STATE.md:55-61); (b) office1-opnsense edge VM (:59); (c) the six vr1-dc0 plane networks + the 3-link mesh triangle + DC storage pools (:64-73 -- planes are live/in-state but already removed from config). AUTHORED-ONLY: (a) module "vvr1_dc0" containment VM (opentofu/main.tf:360; CURRENT-STATE.md:135-138); (b) module "vr1_dc0_uplink" D-125 NAT net (opentofu/main.tf:341; :138-139); (c) the entire inner root opentofu/vr1-dc0-substrate/ -- no state file exists there (:140-143). Also authored-only: SEC-010 enforcement, scripts/opnsense-plugins.sh live apply, Stage 4-7 runbooks (:148-157).

  4. Open gates -- the G1-G15 table at docs/CURRENT-STATE.md:206-224. G1 Phase-3 probe test (closed by 3 clean probes + holes map); G2 Phase-4 GA-R1..R6 rulings (operator rules each individually); G3 Phase-5 repair sweep (operator-gated fix batches, each commit naming its GA-F); G4 two D-130 verifications v7/v8 (run + capture output); G5 D-130 ruling (operator, quoting G4's captured output); G6 state reconcile of autostart+seed without bouncing guests (gated tofu import / state surgery / maintenance-window apply; -refresh-only PROVEN a no-op); G7 new plan == post-D-130 expected triple (impossible until G5); G8 same-session re-verify 6 planes empty; G9 DC0 outer apply (operator-gated, logged, after G1-G8); G10 deploy steps B-E in-sequence gates; G11 operator signs CURRENT-STATE.md; G12 vr1-dc1 addressing ruling; G13 D-129 residuals; G14 nine OPEN SEC rows; G15 D-068/D-071 rulings.

  5. Before any tofu apply. Run bash scripts/opentofu-validate.sh -- must exit 0 with [PASS] per gate and final line "PASS: opentofu-validate (

    )" (scripts/opentofu-validate.sh:189; SKILL.md:147-149 says every opentofu step is reviewed-but-unexercised, validate first). Then open a logged shell (bash scripts/run-logged.sh, CLAUDE.md) and capture a fresh tofu plan to a dated file: its triple must read exactly the expected triple recorded in CURRENT-STATE.md section 5 -- historically "Plan: 5 to add, 0 to change, 6 to destroy" (docs/dc0-deploy-readiness.md:59; charter:228-230) -- but TODAY that expected triple is UNRESOLVED pending D-130 (CURRENT-STATE.md:183-201: "This document may not be used to justify an apply until then"), so no apply is currently justifiable. Same-session plane-empty re-verify (G8) is also required.

  6. Three most dangerous traps. (a) The cloudinit seed-volume staging trap: a host reboot deletes the provider's /tmp staging ISO, so the next outer plan forces a seed-volume REPLACEMENT that would bounce the live Office1 headend

  7. edge -- this is what stopped the apply (GA-F01; docs/CURRENT-STATE.md: 100-110; docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228). (b) "will be updated in-place" does NOT mean the guest stays up -- even an autostart-only update bounces the running domain (platform-traps.md:142; verified against provider source, finding doc:162-184). (c) Bare memory in a libvirt_domain is KiB not MiB -- 1024x too little RAM (platform-traps.md:50, DOCFIX-188). Runners-up: no features block -> acpi=off -> FreeBSD panic (:81); no cpu block -> no svm -> nested KVM impossible (:107); MAAS 3.6/3.7 incompatible with LXD >= 6.7 (:245).

  8. D-NNN ruling authority. docs/design-decisions.md -- grep "^## D-NNN" and read that entry's Status line; "a decision's Status line in that file is the ONLY ruling authority" (docs/CURRENT-STATE.md:335-337; SKILL.md:95-96 -- PROPOSED/OPEN means present options, never pick). A changelog's claim CANNOT be trusted: changelogs are "proven to carry false status" (CURRENT-STATE.md:344-347, GA-F14; charter:23-25 -- the D-127 changelog "asserts plan outcomes that are false as written"), and until GA-R1 is ratified every status surface other than CURRENT-STATE.md is UNTRUSTED for status (CURRENT-STATE.md:14-18).

  9. Redeploy DC0 from scratch. Open docs/dc0-deploy-readiness.md ("A FRESH session starts the deployment from here", :4-5), which routes to runbooks/dc-dc-phase2-tofu-dc-substrate.md (:3, section D step 3) and the gated A-E sequence (:170-179). It does NOT get you there without leaving the repo: (a) its "DEPLOY-READY" banner and plan counts are falsified by the captured 7/2/7 plan (CURRENT-STATE.md:127-131, GA-F01/F02) and the apply is stopped behind gates G1-G9 including the operator-only D-130 ruling and the audit exit criteria (charter:220-235); (b) required inputs live outside the repo: the gitignored opentofu/d124-rack.auto.tfvars (readiness:28-29), per-DC keys in ~/vr1-dc0-creds/ (:26-27,56-62), and the TF_VAR_maas_api_key secret (:68-72); (c) every mutation needs individual operator approval (CLAUDE.md hard rule 3).

Orientation note: I oriented from CLAUDE.md -> .claude/skills/ openstack-cloud-ops/SKILL.md (CLAUDE.md mandates reading it before any nontrivial task; its routing table sent me to the VR1 workflow/status surfaces) -> docs/CURRENT-STATE.md (git log commit be1cd85 and the doc's own header declare it the single status authority; all other status surfaces are declared UNTRUSTED), cross-checked against docs/audit/grounding-audit- charter.md and docs/dc0-deploy-readiness.md.

Appendix C -- probe C return (normalized to ASCII)

  1. Stage and next action. Omega Cloud VR1 DC-DC rehearsal, Stage 3 / Phase 2 ("OpenTofu builds each DC substrate"), branch dc-dc-stage3-phase2-dc- substrate (docs/CURRENT-STATE.md:27-31). The DC0 outer tofu apply (deploy step A) is STOPPED at its own gate (commit 43b29d7); a grounding audit with a FULL FREEZE on fixes is underway, Phases 1-2 complete (docs/CURRENT-STATE.md:32-42; docs/audit/grounding-audit-charter.md:39). Very next action: audit Phase 3, the fresh-agent grounding test (gate G1, "Not run", docs/CURRENT-STATE.md:210; charter section 4, grounding-audit-charter.md:135-149), alongside the charter-exempt read-only D-130 verifications v7/v8 (G4, "TO RUN", docs/CURRENT-STATE.md:213). No status claim outside CURRENT-STATE.md is trusted (docs/CURRENT-STATE.md: 10-18).

  2. Applied vs authored-only. Applied (in tofu state + live, docs/CURRENT-STATE.md:49-96): (a) module.voffice1 containment VM (running, Id 1); (b) module.office1_opnsense edge VM (running, Id 2); (c) the three D-100 mesh-triangle networks mesh_vr1_dc0_vr1_dc1 / _dc0_office1 / _dc1_office1 (also live-but-config-removed: the six vr1-dc0 plane networks, 2.3-iii). Authored-only (docs/CURRENT-STATE.md:133-157): (a) module "vvr1_dc0" at opentofu/main.tf:360; (b) module "vr1_dc0_uplink" (D-125 NAT net) at opentofu/main.tf:341; (c) the entire inner root opentofu/vr1-dc0-substrate/ (no state file). Also authored-only: D-129 scripts/opnsense-plugins.sh, the SEC-010 FORWARD-drop artifact, Stage 4-7 runbooks.

  3. Open gates. G1-G15 at docs/CURRENT-STATE.md:203-224: G1 audit Phase 3 (3 clean probes score the 7-question set); G2 Phase 4 GA-R1..R6 rulings (operator); G3 Phase 5 repair sweep of GA-F01..F14 (freeze lifts, operator-gated batches); G4 D-130 verifications v7/v8 (run + capture); G5 D-130 ruling quoting G4 output (operator); G6 state reconcile of autostart+seed without bouncing guests (import/state surgery/maintenance window); G7 captured plan == post-D-130 expected triple (impossible until G5); G8 same-session re-verify 6 planes empty; G9 DC0 outer apply (operator-gated, after G1-G8 + audit exit criteria); G10 deploy steps B-E in-sequence gates; G11 operator signs CURRENT-STATE.md; G12 vr1-dc1 build (HELD, addressing unruled); G13 D-129 residuals; G14 nine OPEN SEC rows; G15 D-068/D-071 rulings.

  4. Before any tofu apply. Run bash scripts/opentofu-validate.sh (every opentofu step is reviewed-but-unexercised -- skill deploy loop; docs/dc-dc-deployment-workflow.md:875), then tofu plan written to a NEW dated capture file -- never a casual re-plan (docs/CURRENT-STATE.md: 328-330). Its output must read exactly the expected add/change/destroy triple recorded in CURRENT-STATE section 5 -- the standing exit-criteria gate is "5 to add, 0 to change, 6 to destroy" unless CURRENT-STATE records a new triple first (grounding-audit-charter.md:225-228). TODAY the expected triple is UNRESOLVED pending D-130 (last capture 7/2/7, docs/audit/outer-plan-20260718.txt:428), so "no output can legitimize an apply" (docs/CURRENT-STATE.md:183-201, G7 at :216).

  5. Three most dangerous traps. (i) Plan lines that look safe bounce running guests: the reboot-deleted cloudinit staging ISO forces a seed-volume REPLACEMENT that would bounce the live Office1 headend (GA-F01/D-130, docs/CURRENT-STATE.md:100-119), and even "updated in-place" (autostart) bounces the domain (references/platform-traps.md:142-144; docs/finding-20260718-voffice1-cloudinit-seed-replace.md:162-184). (ii) libvirt provider 0.9.x bare memory defaults to KiB not MiB -- 1024x too little RAM (references/platform-traps.md:50-60, DOCFIX-188). (iii) A full rendered config.xml push to the OPNsense edge CLOBBERS the API-managed live config incl. DHCP -- never run the old scp/install/reboot path (docs/dc-dc-deployment-workflow.md:321,877, D-113). Runner-up: MAAS 3.6/3.7 is incompatible with LXD >= 6.7 -- LXD pinned/held at 5.21 (references/platform-traps.md:20,245, D-114).

  6. Decision status authority. Grep docs/design-decisions.md for "^## D-NNN" and read its Status line -- "a decision's Status line in that file is the ONLY ruling authority" (docs/CURRENT-STATE.md:335-337; CLAUDE.md "Decisions" bullet). No, a changelog's claim cannot be trusted: changelog prose is proven to carry false status (GA-F14; "must not rebuild from ... the 95 changelog files", docs/CURRENT-STATE.md:344-347), and until GA-R1 is ratified every non-CURRENT-STATE status surface is UNTRUSTED (docs/CURRENT-STATE.md:13-18).

  7. "Redeploy DC0 from scratch." First correction from evidence: DC0 has never been applied -- nothing DC0-scoped is in state (docs/CURRENT-STATE.md:32-37), so this is a first deploy. Open docs/CURRENT-STATE.md (status authority), then docs/dc-dc-deployment-workflow.md (stage row :148 + gap register), then runbooks/dc-dc-phase2-tofu-dc-substrate.md with docs/dc0-deploy-readiness.md's step table (:170-179). It does NOT get you there without leaving the repo: the apply is gated behind operator rulings and audit exit criteria (G5/G9/G11, docs/CURRENT-STATE.md:214-220; charter :222-233), and requires out-of-repo material -- the dc0 SSH key in ~/vr1-dc0-creds/ and the secret TF_VAR_maas_api_key for step D (docs/dc0-deploy-readiness.md:54-67), plus live measurement and per-mutation human approval.

Orientation note. Oriented from CLAUDE.md -> the openstack-cloud-ops SKILL.md routing table (as CLAUDE.md directs) -> docs/CURRENT-STATE.md, because the skill routes resume-work to the ledger/status surfaces and commit be1cd85 + CURRENT-STATE.md:10-18 declare CURRENT-STATE.md the single status authority superseding all other status surfaces (which Phase 1 proved mutually contradictory).