diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md new file mode 100644 index 0000000..4b36721 --- /dev/null +++ b/docs/CURRENT-STATE.md @@ -0,0 +1,347 @@ +# CURRENT-STATE.md -- the single status authority (Omega Cloud / VR1 DC-DC) + +Authored 2026-07-18 by the grounding-audit Phase-2 ground-truth agent +(charter: `docs/audit/grounding-audit-charter.md`, section 3) at repo HEAD +`e999b03` on branch `dc-dc-stage3-phase2-dc-substrate`. Every claim below +carries its evidence (path:line, quoted command output, or commit hash). +Claims that could not be evidenced read-only are marked UNKNOWN with what +would resolve them. Nothing here is guessed. + +STATUS OF THIS DOCUMENT: DRAFT pending operator signature (charter Phase 6 +item 5). Proposed standing rule (GA-R1, NOT yet ratified): no status claim +may be hand-written anywhere else; other documents point HERE; this document +cites captured command output. Until GA-R1 is ratified, treat every other +status surface (readiness doc banners, workflow-doc stage rows, ledger +narrative, changelogs, README pins, skill version tables, auto-memory) as +UNTRUSTED for status -- Phase 1 proved they contradict each other +(`docs/audit/record-inventory.md`: 12 contradiction groups; findings +GA-F01..GA-F14 in `docs/audit/grounding-audit-20260718.md`). + +--- + +## 1. Where the project IS + +- Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend + virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal + Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). +- Stage: Stage 3 / Phase 2 -- "OpenTofu builds each DC substrate" + (`docs/dc-dc-deployment-workflow.md:148`; branch name + `dc-dc-stage3-phase2-dc-substrate`; runbook + `runbooks/dc-dc-phase2-tofu-dc-substrate.md`). Stages 0-2 precede it; + stages 4-7 are authored, not executed (workflow doc:873). +- Position inside Stage 3: the DC0 outer `tofu apply` (deploy step A) is + STOPPED at its own gate. The stop is commit `43b29d7` ("post-reboot outer + plan would BOUNCE Office1 headend + edge -- DC0 apply STOPPED", + 2026-07-18). Nothing DC0-scoped has been applied + (`tofu -chdir=opentofu state list` shows no `vvr1_dc0` or + `vr1_dc0_uplink` resources; output quoted in section 2). +- A grounding audit is underway with a FULL FREEZE on fixes (charter + section 1, commit `148dcef`). Phase 1 (deterministic inventory + measured + snapshot) is complete (commits `af2e312`, `e999b03`). This document is + the Phase-2 deliverable. Phases 3-6 (probe test, structural rulings, + repair sweep incl. the D-130 ruling, exit criteria) have NOT run. +- The vcloud host bookend (patch + reboot onto kernel -136, + `docs/dc0-deploy-readiness.md:107`) HAS happened: host rebooted + ~2026-07-17 23:39, both guests self-recovered via autostart + (`docs/audit/env-snapshot-20260718.md:10-16`; re-measured this session, + section 2.2 below). + +## 2. What is APPLIED (from tofu state + live measurement, not from docs) + +### 2.1 OpenTofu outer-root state (command: `tofu -chdir=opentofu state list`, +run 2026-07-18, 20 resources) + +Office1 site (live, load-bearing): +- `module.voffice1.libvirt_domain.vm` + `.libvirt_volume.disk` + + `.libvirt_volume.seed` + `.libvirt_cloudinit_disk.seed` + (BUT see divergence 2.3-i: the cloudinit staging ISO no longer exists + live) +- `module.office1_opnsense.libvirt_domain.vm` + `.libvirt_volume.disk` +- `module.office1_network.libvirt_network.office1_local` +- `module.office1_storage.libvirt_pool.dc` +- `module.ubuntu_noble_base.libvirt_volume.base` + +Inter-site fabric and DC scaffolding: +- `module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link`, + `module.mesh_vr1_dc0_office1.libvirt_network.link`, + `module.mesh_vr1_dc1_office1.libvirt_network.link` (the D-100 mesh + triangle) +- `module.vr1_dc0_planes.libvirt_network.plane["data-tenant" | "metal-admin" + | "metal-internal" | "provider-public" | "replication" | "storage"]` + -- applied and live, but REMOVED from config (see 2.3-iii) +- `module.vr1_dc0_storage.libvirt_pool.dc`, + `module.vr1_dc1_storage.libvirt_pool.dc` + +### 2.2 Live environment (measured this session, 2026-07-18, read-only) + +- `hostname` -> `vcloud`; `uname -r` -> `6.8.0-136-generic`. +- `virsh list --all` -> exactly two domains, both running: + `voffice1` (Id 1), `office1-opnsense` (Id 2). +- `virsh dominfo` -> `Autostart: enable` on BOTH domains. +- `ssh voffice1 'snap list maas lxd; uname -r'` -> + `maas 3.7.2-17972-g.35e297c4d` rev 41649 (3.7/stable), + `lxd 5.21.5-f2a1a0e` rev 40074 (5.21/stable, held), + guest kernel `6.8.0-136-generic`. +- `ssh office1-netbox 'curl -s -o /dev/null -w "netbox=%{http_code}" + http://localhost:8000/'` -> `netbox=302` (service up, redirecting to + login). +- `ssh office1-tailscale 'tailscale status | head -1'` -> + `100.64.0.53 office1-tailscale ... linux -` (subnet-router VM up). +- D-126 base leg: `scripts/site-baseleg.sh check office1` passed at the + Phase-1 snapshot (`docs/audit/env-snapshot-20260718.md:16-17`); not + re-run this session. +- OPNsense edge version: 26.7 per confirmed as-built + (`docs/vr1-office1-as-built.md:42`, updated 2026-07-18). NOT re-measured + this session -- measuring requires the gated API credential path; see + section 7. + +### 2.3 State vs live vs config disagreements (REPORTED, not harmonized) + +i. `module.voffice1.libvirt_cloudinit_disk.seed` is IN STATE, but its + remote object (the provider's staging ISO under + `/tmp/terraform-provider-libvirt-cloudinit/`) was DELETED by the host + reboot. Captured: `docs/audit/outer-plan-20260718.txt:34-40` + ("has been deleted"). Because the regenerated staging path is random, + `module.voffice1.libvirt_volume.seed` plans as a FORCED REPLACEMENT + (`create.content.url` -- capture lines 119-134), which would bounce the + running Office1 headend. This is the defect that stopped the apply + (GA-F01) and the subject of the pending D-130 ruling (section 8). + Root-cause mechanism, verified against provider v0.9.8 source: + `docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228`. +ii. Autostart: LIVE is `enable` on both domains (measured, 2.2). CONFIG + authors `autostart = true` (D-127, `docs/design-decisions.md:3914`). + But STATE records NO autostart attribute on either domain (empirical: + `tofu state show ... | grep -c autostart` -> 0, finding doc:140-145), + so the plan still shows 2 in-place changes -- and a provider-source + read confirms an autostart-only update BOUNCES the guest (finding + doc:162-184). Live matches intent; state cannot see it; an unscoped + apply would still cause an outage. Resolution belongs to the gated + reconcile (section 6, gate G6) -- NOT to a quiet re-plan. +iii. The six `vr1-dc0` plane networks exist live and in state but are + REMOVED from config -- the INTENDED Model B relocation (planes get + recreated inside `vvr1-dc0` by the inner root, + `opentofu/vr1-dc0-substrate/main.tf:28`). Their emptiness (0 leases, + 0 attached domains) was verified in a PRIOR session + (`docs/dc0-deploy-readiness.md:43-45`) and must be re-verified in the + same session as any apply (finding doc:259-265). +iv. `docs/dc0-deploy-readiness.md:8` still carries the banner "PREP + COMPLETE -- DC0 IS DEPLOY-READY" and internal plan counts 5/2/6 (:37) + and 5/0/6 (:59). The captured live plan (7/2/7) falsifies the banner + and both counts (GA-F01, GA-F02). Under freeze, the doc is not edited; + its checklists remain useful, its status lines do not. + +## 3. What is AUTHORED-BUT-NOT-APPLIED (in the tree, not in state / live) + +- `module "vvr1_dc0"` (`opentofu/main.tf:360`) -- the DC0 containment VM + (416 GiB / 108 vCPU, D-121/D-123 sizing) + its disk, seed volume, and + cloudinit seed. 4 of the 5 committed DC0 creates in the plan capture. +- `module "vr1_dc0_uplink"` (`opentofu/main.tf:341`) -- the D-125 simulated + ISP NAT network `172.30.2.0/24` (capture lines 226-247). The 5th create. +- The ENTIRE inner root `opentofu/vr1-dc0-substrate/` (main/variables/ + versions.tf; no state file exists in that directory) -- inner storage + pool, the six relocated planes, bridge-in WAN, and the rest of the + Model B step-C build. +- The removal of `module "vr1_dc0_planes"` from the outer config (the 6 + intended destroys; see 2.3-iii). +- `autostart = true` on voffice1/office1-opnsense in config (D-127) -- + live-true but state-absent (2.3-ii). +- `scripts/opnsense-plugins.sh` + `tests/opnsense-plugins/` (D-129 + profile-installer; commit `4cefa8b`). BUILT and green; the live `apply` + against the edge is an operator-gated firmware mutation, NOT run + (`docs/design-decisions.md:4050-4056`). +- SEC-010 enforcement artifact (`scripts/site-headend-install.sh + --host-nodes` writing the transit FORWARD-drop) -- COMMITTED, applies on + `vvr1-dc0` at deploy step B; ledger row stays OPEN until applied+verified + (`bash scripts/ledger-scan.sh` output, SEC-010 row). +- Stage 4-7 runbooks (`runbooks/dc-dc-phase3..6-*.md`) -- written, not + executed (`docs/dc-dc-deployment-workflow.md:873`). + +## 4. What is RULED-BUT-NOT-BUILT (ADOPTED/RULED with no artifact yet) + +- D-121/D-122/D-123/D-124/D-125 (all ADOPTED/RULED, status lines at + `docs/design-decisions.md:3291,3474,3543,3635,3713`): the DC0 deploy + sequence they rule (steps A-E, `docs/dc0-deploy-readiness.md:170-179`) + exists only as config + runbook. Nothing DC0 is built. +- `vr1-dc1`: ruled at topology level (D-100/D-101 two-DC design) but HELD + -- no ruled dc1 transit/rack addressing, no `vr1_dc1_rack_*` variables, + no dc1 substrate root (`docs/dc0-deploy-readiness.md:100-103`). Only its + storage pool and mesh legs exist (state list, 2.1). +- D-100 netem: mechanism authored (`opentofu/modules/netem-link`) but HELD + as a comment in the root (`opentofu/main.tf:309-319`); placeholder + parameters ruled for the rehearsal (readiness doc:73-75); final + parameters unruled (section 8). +- D-129 Roosevelt metal-edge plugin profile (`os-smart`, `os-nut` | + `os-apcupsd`, microcode, `os-lldpd`) -- recorded, inert until the + Roosevelt edge build (`docs/design-decisions.md:4030-4031`). +- D-129 qga enablement: designed (opt-in module variable, default OFF; the + guest-agent virtio channel is MISSING from the live edge domain) but not + written; retrofit deferred to the edge's next scheduled restart + (`docs/design-decisions.md:4041-4047,4057-4063`). + +## 5. The outer plan count (wording per operator direction, GA-F01) + +The true EXPECTED outer plan count is currently NOBODY'S: + +- expected count: UNRESOLVED pending D-130; +- last captured: 7/2/7 (`docs/audit/outer-plan-20260718.txt`, line 428: + "Plan: 7 to add, 2 to change, 7 to destroy." -- the ONLY citable + plan-count source); +- pre-reboot gate was 5/0/6 (recorded at `docs/dc0-deploy-readiness.md:59`, + `docs/session-ledger.md:278`). + +7/2/7 is the post-reboot SYMPTOM of the D-130 seed-volume defect (7 add = +5 committed DC0 creates + voffice1 cloudinit-disk re-create + seed-volume +replace-create; 2 change = the state-blind autostart updates; 7 destroy = +6 empty planes + seed-volume replace-destroy -- decomposition at +`docs/audit/env-snapshot-20260718.md:33-46`). 5/0/6 was the pre-reboot +gate; 5/2/6 was an intermediate recorded value (GA-F02). The post-fix +expected triple DOES NOT EXIST until D-130 is ruled, and this document +deliberately does not pick one. When D-130 is ruled, the new expected +triple and WHY get recorded HERE before any re-plan (charter Phase 6 +item 4). This document may not be used to justify an apply until then. + +## 6. Open gates + +Owner legend: operator (human ruling/approval), session (agent work under +gating), external (outside this repo/track). + +| # | Gate | What closes it | Owner | Evidence of current state | +|---|------|----------------|-------|---------------------------| +| G1 | Audit Phase 3: fresh-agent grounding test | 3 clean-context probes score the 7-question set against this doc; holes map made | session | Not run (no probe outputs exist; charter section 4) | +| G2 | Audit Phase 4: GA-R1..R6 structural rulings + the stage-status vocabulary A/B | operator rules each individually | operator | Drafted only (charter section 5; GA-F10 Phase-4 note) | +| G3 | Audit Phase 5: repair sweep of GA-F01..F14 (incl. memory hygiene GA-F05..F08, skill sweep) | operator-gated fix batches, each commit naming its GA-F | operator + session | FREEZE in effect; zero fixes applied (charter section 1) | +| G4 | The two D-130 verifications (charter-exempt, read-only/throwaway): (v7) v0.9.8 volume-replace-under-running-domain on a throwaway domain; (v8) lifecycle ignore_changes suppression of the forced replacement | run them, capture output | session | TO RUN (`docs/audit/env-snapshot-20260718.md:47-49`) | +| G5 | D-130 mechanism ruling (seed-volume durable fix) | operator rules in Phase 5, quoting G4's captured output | operator | D-130 unassigned; next-free D confirmed 130 (`ledger-scan` run 2026-07-18; no `^## D-130` in design-decisions.md) | +| G6 | State reconcile of autostart + seed WITHOUT bouncing guests | gated mechanism per the finding: `tofu import` (reads autostart back) or state surgery, or an operator-scheduled maintenance-window apply; `-refresh-only` is PROVEN a no-op | operator | Operator earlier selected reconcile-first (finding doc:136); superseded into audit sequencing by `2b718f5`; nothing reconciled yet | +| G7 | New captured plan == the post-D-130 expected triple recorded in section 5 | re-plan to a capture file after G5+G6 | session | Impossible today: expected triple UNRESOLVED (section 5) | +| G8 | Same-session pre-apply re-verify: 6 planes still empty (0 leases / 0 attached domains) | run in the SAME session as the apply | session | Last verified in a prior session only (finding doc:259-265) | +| G9 | DC0 outer apply (deploy step A) | operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6) | operator | STOPPED (`43b29d7`); state shows nothing DC0 applied | +| G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | exercised during the gated deploy | session (each mutation operator-approved) | Not reached (`docs/dc0-deploy-readiness.md:67-84,170-179`) | +| G11 | Operator signs THIS document | read top-to-bottom; discrepancies resolved in the document | operator | Unsigned (this is the draft) | +| G12 | `vr1-dc1` build | operator rules dc1 transit/rack addressing; then vars + substrate authored | operator | HELD (`docs/dc0-deploy-readiness.md:100-103`) | +| G13 | D-129 residuals | operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) | operator | OPEN / PARTIALLY RULED (`docs/design-decisions.md:4017`) | +| G14 | 9 OPEN SEC rows (SEC-001, -003..-008, -010) | per-row: rotations/flips at v1 close (external to VR1 track) except SEC-010 = apply+verify at G10 | operator / external | `bash scripts/ledger-scan.sh` output, 2026-07-18 | +| G15 | D-068 / D-071 rulings | operator rules (section 8); neither blocks the VR1 substrate | operator | PROPOSED/OPEN (status lines, section 8) | + +## 7. Version pins (measured; the authority for every pin) + +| Component | Measured value | Command (run 2026-07-18) | Where measured | +|---|---|---|---| +| OpenTofu | v1.12.4 | `tofu version` | vcloud (also `docs/audit/env-snapshot-20260718.md:26`) | +| libvirt provider | dmacvicar/libvirt 0.9.8 (pinned) | `grep -A2 'provider' opentofu/.terraform.lock.hcl` | repo lock file | +| MAAS provider | canonical/maas 2.7.2 (pinned) | same | repo lock file | +| MAAS | 3.7.2-17972-g.35e297c4d (3.7/stable) | `ssh voffice1 'snap list maas'` | voffice1 | +| LXD | 5.21.5-f2a1a0e (5.21/stable, held) | `ssh voffice1 'snap list lxd'` | voffice1 | +| Kernel (host) | 6.8.0-136-generic | `uname -r` | vcloud | +| Kernel (voffice1) | 6.8.0-136-generic | `ssh voffice1 'uname -r'` | voffice1 | +| OPNsense edge | 26.7 (FreeBSD base 15.1) | not re-measured (gated API creds); per confirmed as-built `docs/vr1-office1-as-built.md:42`, 2026-07-18 | office1-opnsense | +| NetBox (Office1 apex) | 4.6.4 per as-built `docs/vr1-office1-as-built.md:44`; service UP verified (HTTP 302) this session | `ssh office1-netbox 'curl ... localhost:8000'` | office1-netbox | +| Juju | NOT part of VR1 substrate yet (arrives Stage 5). Last recorded VR0 value 3.6.25 -- historical, unverified here | n/a | n/a | + +Known-stale pins that this table supersedes (do NOT trust them; Phase-5 +fixes them): OpenTofu 1.12.3 at `opentofu/README.md:41` (historical Stage-1 +prose), `docs/dc-dc-deployment-workflow.md:47,360,875`, +`.claude/skills/.../platform-traps.md:19` (GA-F03); OPNsense 26.1 across +the skill references (GA-F04); jumphost name `vopenstack-jesse` in +`CLAUDE.md:4` and the skill -- the host is `vcloud`, measured (GA-F05). + +## 8. Open decision queue (the exact questions the operator must answer) + +1. D-130 (to be assigned; next-free D = 130, verified) -- seed-volume + durable mechanism. QUESTION: adopt (a) `lifecycle { ignore_changes = + [create] }` on `libvirt_volume.seed` in `modules/cloudinit-vm` -- the + operator's recorded lean, EXPLICITLY CONTINGENT on verification G4-v8 + confirming it suppresses the forced replacement; if v8 refutes it, the + lean is DEAD (do not rescue) and the fallback is (b) a persistent + `$TMPDIR` staging path plus a runbook-enforced env guard, with the + fragility concern stated and accepted (charter section 1a). Evidence + needed before ruling: the captured outputs of both G4 verifications; + the decision text must quote them. Full apply-scoping option set + (a)-(d) and the verified provider mechanics: + `docs/finding-20260718-voffice1-cloudinit-seed-replace.md:104-257`. + Blast radius (measured): voffice1 + vvr1-dc0 (+ vvr1-dc1 later) = 3 + VMs; node VMs and the edge are UNAFFECTED (finding doc:206-223). + Sequencing: ruled inside audit Phase 5, AFTER this document (charter + section 1a, last bullet). +2. D-100 netem parameter sub-item (gap #11). Already ruled: same-metro + dark-fiber default lean (D-100 sub-items, `docs/design-decisions.md: + 1962`) and placeholder-for-the-rehearsal (readiness doc:73-75, operator + ruling 2026-07-16). OPEN QUESTION: the final latency/jitter/loss/rate + numbers -- supply the Roosevelt inter-DC target, or confirm the + placeholder stands until that target exists. Evidence needed: the + Roosevelt link spec (external input; nothing in-repo can answer it). +3. D-068 -- Vault substrate hardening (PROPOSED/OPEN, + `docs/design-decisions.md:1457`). QUESTION: rule items 2 (Vault + listener TLS) and 3 (AppRole secret_id lifecycle) for Roosevelt; item 1 + (Vault version) needs a re-scoped migration plan since the 1.16 + forward-pin was proven NOT viable (amendment, :1650). Evidence in + `docs/D-068-vault-1.8-vs-1.16-analysis.md`. +4. D-071 -- update cadence + controller patch policy (PROPOSED, + `docs/design-decisions.md:1605`). QUESTION: adopt policy points 1-4 + (monthly-review window trigger; patch-only controller jumps; order; + acceptable single-controller risk) as amended by the confirmed backup + posture (:1677). +5. D-129 -- four OPEN sub-decisions (OPEN / PARTIALLY RULED, + `docs/design-decisions.md:4017`): (i) is the boundary edge in-scope for + COS scraping (D-105 scope)? (ii) `os-frr` dynamic routing -- Roosevelt + question? (iii) consolidate the Tailscale subnet-router onto the edge + vs keep D-107 as-is? (iv) does the edge anchor site NTP? Plus the + operator-gated live install of the ruled VR1 profile + (`scripts/opnsense-plugins.sh apply vr1-edge`). +6. Audit Phase 4 rulings: GA-R1..GA-R6 individually, plus the legal + stage-status vocabulary A/B (GA-F10 operator note) -- each gated, one + ruling per exchange. + +## 9. Additional defect found while authoring (recorded, NOT fixed -- +freeze) + +`ledger-scan.sh`'s mention-derived next-free counters (DOCFIX, BUNDLEFIX) +are SELF-INFLATED by any doc that quotes a "next-free" value and lets the +hyphenated token wrap onto a line without the words "next-free" -- the +per-line exclusion filter (`scripts/ledger-scan.sh:121-129`, the +`grep -viE 'next[- ]free'` at :124) then counts the quote as a real +assignment; the script's own CAUTION comment (:112-120) documents exactly +this failure class. It happened TWICE inside the audit itself on +2026-07-18: the Phase-1 env snapshot's wrapped next-free line inflated the +BUNDLEFIX counter (051 -> reported 052), and this document's own first +draft of this very section inflated the DOCFIX counter the same way while +asserting DOCFIX was unaffected. Both audit surfaces were reworded +token-free the same day and the counters re-verified at their true values +(D=130, DOCFIX=197, BUNDLEFIX=052 -- see GA-F15). The D counter is +header-authoritative and was never affected. Numbering discipline until +the scanner is hardened in Phase 5: never write a hyphenated +register-token quote of a next-free value into any doc; state the numbers +token-free as this section does. + +## 10. How to verify this document (cold-session re-derivation) + +Run read-only, from the repo root: + +- Repo identity: `git rev-parse HEAD; git status --short` + (this doc was authored at `e999b03`, clean tree). +- Applied set: `tofu -chdir=opentofu state list` (expect the 20 resources + in section 2.1); `virsh list --all`; `virsh dominfo voffice1 | grep -i + autostart` (and office1-opnsense). +- Authored-not-applied: `grep -n '^module ' opentofu/main.tf` (12 blocks; + `vvr1_dc0` + `vr1_dc0_uplink` absent from state list); + `ls opentofu/vr1-dc0-substrate/` (no *.tfstate). +- Plan count: read `docs/audit/outer-plan-20260718.txt` line 428. Do NOT + re-run `tofu plan` casually against live state; if a fresh capture is + taken, it must be written to a new dated capture file and cited here. +- Versions: `tofu version`; `ssh voffice1 'snap list maas lxd; uname -r' +