diff --git a/CLAUDE.md b/CLAUDE.md index 41e6e3a..390ad60 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,7 @@ ## Session bootstrap (run at the start of every session) git pull - bash scripts/repo-lint.sh # expect 0 fail (1 legacy WARN is documented) + bash scripts/repo-lint.sh # expected result: see docs/CURRENT-STATE.md (GA-R1) bash scripts/run-tests-all.sh # after any tooling change; ALL GREEN expected For any session that will mutate the cloud, the operator starts logging first: diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 8543295..c872d6e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -8,14 +8,13 @@ would resolve them. Nothing here is guessed. STATUS OF THIS DOCUMENT: SIGNED by the operator 2026-07-18 (section 11; -charter Phase 6 item 5). Proposed standing rule (GA-R1, NOT yet ratified): no status claim -may be hand-written anywhere else; other documents point HERE; this document -cites captured command output. Until GA-R1 is ratified, treat every other -status surface (readiness doc banners, workflow-doc stage rows, ledger -narrative, changelogs, README pins, skill version tables, auto-memory) as -UNTRUSTED for status -- Phase 1 proved they contradict each other -(`docs/audit/record-inventory.md`: 12 contradiction groups; findings -GA-F01..GA-F14 in `docs/audit/grounding-audit-20260718.md`). +charter Phase 6 item 5). STANDING RULE (GA-R1, RATIFIED 2026-07-18 with +amendments C1+C2 -- `docs/audit/ga-rulings.md`): no status claim is +hand-written anywhere else; other documents point HERE; this document +cites captured command output, and measurement always wins over it (C2). +Other status surfaces are pointers or history; where one still carries a +claim, it is a defect (Phase 1 proved they contradict: +`docs/audit/record-inventory.md`, 12 groups; findings GA-F01..GA-F15). --- @@ -30,16 +29,16 @@ `runbooks/dc-dc-phase2-tofu-dc-substrate.md`). Stages 0-2 precede it; stages 4-7 are authored, not executed (workflow doc:873). - Position inside Stage 3: the DC0 outer `tofu apply` (deploy step A) is - STOPPED at its own gate. The stop is commit `43b29d7` ("post-reboot outer - plan would BOUNCE Office1 headend + edge -- DC0 apply STOPPED", - 2026-07-18). Nothing DC0-scoped has been applied - (`tofu -chdir=opentofu state list` shows no `vvr1_dc0` or - `vr1_dc0_uplink` resources; output quoted in section 2). -- A grounding audit is underway with a FULL FREEZE on fixes (charter - section 1, commit `148dcef`). Phase 1 complete (`af2e312`, `e999b03`); - this document is the Phase-2 deliverable (ACCEPTED on external review, - `c49da79`); Phase 3 complete -- 3 probes, 21/21 PASS, holes map at - `docs/audit/phase3-grounding-test-20260718.md`. Phases 4-6 have NOT run. + BLOCKED at its own gate (G9) -- blocker: sweep Batches 2-6 + G8 + the + Batch-6 exit runs incl. re-sign. Stop commit `43b29d7` (2026-07-18: + post-reboot outer plan would bounce Office1 headend + edge). Nothing + DC0-scoped has been applied (`tofu -chdir=opentofu state list` shows no + `vvr1_dc0` or `vr1_dc0_uplink` resources; output quoted in section 2). +- The grounding audit: Phases 1-4 complete (charter `148dcef`; Phase-2 = + this document, ACCEPTED `c49da79`; Phase 3 = 3 probes 21/21 PASS; Phase + 4 = GA-R1..R7 + vocabulary RATIFIED, `docs/audit/ga-rulings.md`). Phase + 5 (the repair sweep) is EXECUTING: Batches 0-1 closed 2026-07-19, per- + batch gates; the FREEZE holds only for surfaces no batch has opened. - The vcloud host bookend (patch + reboot onto kernel -136, `docs/dc0-deploy-readiness.md:107`) HAS happened: host rebooted ~2026-07-17 23:39, both guests self-recovered via autostart @@ -97,16 +96,12 @@ ### 2.3 State vs live vs config disagreements (REPORTED, not harmonized) -i. `module.voffice1.libvirt_cloudinit_disk.seed` is IN STATE, but its - remote object (the provider's staging ISO under - `/tmp/terraform-provider-libvirt-cloudinit/`) was DELETED by the host - reboot. Captured: `docs/audit/outer-plan-20260718.txt:34-40` - ("has been deleted"). Because the regenerated staging path is random, - `module.voffice1.libvirt_volume.seed` plans as a FORCED REPLACEMENT - (`create.content.url` -- capture lines 119-134), which would bounce the - running Office1 headend. This is the defect that stopped the apply - (GA-F01) and the subject of the pending D-130 ruling (section 8). - Root-cause mechanism, verified against provider v0.9.8 source: +i. `module.voffice1.libvirt_cloudinit_disk.seed` is IN STATE but its + staging ISO was deleted by the reboot -- it still plans as a benign + re-create (1 of section 5's 6 adds). The FORCED REPLACEMENT it used + to force on `libvirt_volume.seed` (the GA-F01 defect that stopped + the apply) is FIXED: D-130 ADOPTED (a) + implemented 2026-07-19, + verified by the v8/v7 captures (gate rows G4/G5). Mechanism history: `docs/finding-20260718-voffice1-cloudinit-seed-replace.md:186-228`. ii. Autostart: RESOLVED 2026-07-19 by the G6 state surgery (operator- ruled (ii), gate row G6): state now records `autostart = true` on @@ -120,11 +115,10 @@ 0 attached domains) was verified in a PRIOR session (`docs/dc0-deploy-readiness.md:43-45`) and must be re-verified in the same session as any apply (finding doc:259-265). -iv. `docs/dc0-deploy-readiness.md:8` still carries the banner "PREP - COMPLETE -- DC0 IS DEPLOY-READY" and internal plan counts 5/2/6 (:37) - and 5/0/6 (:59). The captured live plan (7/2/7) falsifies the banner - and both counts (GA-F01, GA-F02). Under freeze, the doc is not edited; - its checklists remain useful, its status lines do not. +iv. RESOLVED 2026-07-19 (Batch 2, GA-F02): the readiness doc's falsified + deploy-ready banner and its three contradictory plan counts are + demoted -- status and the expected triple point HERE; the fresh- + session banner points at the G9 canonical entry doc. ## 3. What is AUTHORED-BUT-NOT-APPLIED (in the tree, not in state / live) @@ -200,25 +194,27 @@ ## 6. Open gates Owner legend: operator (human ruling/approval), session (agent work under -gating), external (outside this repo/track). +gating), external (outside this repo/track). Type legend (GA-R6/E1): [V] = +verification-type (closes on its named executable check); [R] = ruling-type +(closes per a GA-R5 recorded ruling). | # | Gate | What closes it | Owner | Evidence of current state | |---|------|----------------|-------|---------------------------| -| G1 | Audit Phase 3: fresh-agent grounding test | 3 clean-context probes score the 7-question set against this doc; holes map made | session | CLOSED 2026-07-18: 3 probes, 21/21 PASS, holes H1 (amended into G9) + H2 (no action) -- `docs/audit/phase3-grounding-test-20260718.md` | -| G2 | Audit Phase 4: GA-R1..R7 structural rulings + the stage-status vocabulary A/B | ruling-type gate (GA-R6 rule 6): closes when every item carries a GA-R5 Status block | operator | CLOSED 2026-07-18: all seven GA-R + vocabulary (Option A + H1) RATIFIED, utterances quoted (`docs/audit/ga-rulings.md`, through commit `fe4f1c4` + this one) | -| G3 | Audit Phase 5: repair sweep of GA-F01..F15 (incl. memory hygiene GA-F05..F08, skill sweep) | operator-gated fix batches, each commit naming its GA-F | operator + session | Batch 0 OPENED by operator 2026-07-19; items 0.1 (repo-lint L10, GA-R1/C1), 0.2 (SEC repoint, GA-R4/F3), 0.3 (counter hardening, GA-F15), 0.4 (extractor vocab scan, GA-F10/H1) landed; Batch 0 CLOSED (verification passed 2026-07-19); Batch 1 items 1-7 COMPLETE (G4/G5/G6/G7 all CLOSED; triple 6/0/6 recorded), batch close verification pending; Batches 2-6 await gates; FREEZE holds for un-gated surfaces | -| G4 | The two D-130 verifications | run them, capture output | session | CLOSED 2026-07-19: v8 suppression CONFIRMED (7/2/7 -> 6/2/6, zero forces-replacement; `docs/audit/outer-plan-20260719-v8-ignorechanges.txt` + `-baseline.txt`); v7 no-bounce under running domain, zero residue (`docs/audit/throwaway-v7-20260719.txt`) | -| G5 | D-130 mechanism ruling (seed-volume durable fix) | operator rules in Phase 5, quoting G4's captured output | operator | CLOSED 2026-07-19: D-130 ADOPTED (a) ignore_changes (`docs/design-decisions.md` D-130, GA-R5 utterance quoted); implemented in `modules/cloudinit-vm` + `tests/cloudinit-vm` | -| G6 | State reconcile of autostart + seed WITHOUT bouncing guests | gated mechanism, operator-ruled (S3) | operator | CLOSED 2026-07-19: ruled (ii) state surgery (GA-R5); pull -> inject autostart:true on both domains -> push (serial 22->23, backup `terraform.tfstate.pre-G6-20260719`); guests never touched (ids 1/2 unchanged, running) | -| G7 | New captured plan == the expected triple recorded in section 5 | re-plan to a capture file after G5+G6 | session | CLOSED 2026-07-19: capture `docs/audit/outer-plan-20260719-postG6.txt` = 6/0/6, equals section 5 exactly | -| G8 | Same-session pre-apply re-verify: 6 planes still empty (0 leases / 0 attached domains) | run in the SAME session as the apply | session | Last verified in a prior session only (finding doc:259-265) | -| G9 | DC0 outer apply (deploy step A) | operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | STOPPED (`43b29d7`); state shows nothing DC0 applied | -| G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | exercised during the gated deploy | session (each mutation operator-approved) | Not reached (`docs/dc0-deploy-readiness.md:67-84,170-179`) | -| G11 | Operator signs THIS document | read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: SIGNED 2026-07-18, section 11 (post-probe version, per sequencing directive) | -| G12 | `vr1-dc1` build | operator rules dc1 transit/rack addressing; then vars + substrate authored | operator | HELD (`docs/dc0-deploy-readiness.md:100-103`) | -| G13 | D-129 residuals | operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) | operator | OPEN / PARTIALLY RULED (`docs/design-decisions.md:4017`) | -| G14 | 8 OPEN SEC rows (SEC-001, -003..-008, -010; prior cell said 9, enumeration + scan both measure 8 -- corrected per GA-R1/C2) | per-row: rotations/flips at v1 close (external to VR1 track) except SEC-010 = apply+verify at G10 | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count cross-checked vs `bash scripts/ledger-scan.sh` 2026-07-19 | -| G15 | D-068 / D-071 rulings | operator rules (section 8); neither blocks the VR1 substrate | operator | PROPOSED/OPEN (status lines, section 8) | +| G1 | Audit Phase 3: fresh-agent grounding test | [V] 3 clean-context probes score the 7-question set against this doc; holes map made | session | CLOSED 2026-07-18: 3 probes, 21/21 PASS, holes H1 (amended into G9) + H2 (no action) -- `docs/audit/phase3-grounding-test-20260718.md` | +| G2 | Audit Phase 4: GA-R1..R7 structural rulings + the stage-status vocabulary A/B | [R] ruling-type gate (GA-R6 rule 6): closes when every item carries a GA-R5 Status block | operator | CLOSED 2026-07-18: all seven GA-R + vocabulary (Option A + H1) RATIFIED, utterances quoted (`docs/audit/ga-rulings.md`, through commit `fe4f1c4` + this one) | +| G3 | Audit Phase 5: repair sweep of GA-F01..F15 (incl. memory hygiene GA-F05..F08, skill sweep) | [R] operator-gated fix batches, each commit naming its GA-F | operator + session | Batch 0 OPENED by operator 2026-07-19; items 0.1 (repo-lint L10, GA-R1/C1), 0.2 (SEC repoint, GA-R4/F3), 0.3 (counter hardening, GA-F15), 0.4 (extractor vocab scan, GA-F10/H1) landed; Batch 0 CLOSED (verification passed 2026-07-19); Batch 1 all items landed + close verification passed (G4/G5/G6/G7 CLOSED; triple 6/0/6 recorded); Batch 2 OPEN (record fixes landing per-GA-F); Batches 2-6 await gates; FREEZE holds for un-gated surfaces | +| G4 | The two D-130 verifications | [V] run them, capture output | session | CLOSED 2026-07-19: v8 suppression CONFIRMED (7/2/7 -> 6/2/6, zero forces-replacement; `docs/audit/outer-plan-20260719-v8-ignorechanges.txt` + `-baseline.txt`); v7 no-bounce under running domain, zero residue (`docs/audit/throwaway-v7-20260719.txt`) | +| G5 | D-130 mechanism ruling (seed-volume durable fix) | [R] operator rules in Phase 5, quoting G4's captured output | operator | CLOSED 2026-07-19: D-130 ADOPTED (a) ignore_changes (`docs/design-decisions.md` D-130, GA-R5 utterance quoted); implemented in `modules/cloudinit-vm` + `tests/cloudinit-vm` | +| G6 | State reconcile of autostart + seed WITHOUT bouncing guests | [R] gated mechanism, operator-ruled (S3) | operator | CLOSED 2026-07-19: ruled (ii) state surgery (GA-R5); pull -> inject autostart:true on both domains -> push (serial 22->23, backup `terraform.tfstate.pre-G6-20260719`); guests never touched (ids 1/2 unchanged, running) | +| G7 | New captured plan == the expected triple recorded in section 5 | [V] re-plan to a capture file after G5+G6 | session | CLOSED 2026-07-19: capture `docs/audit/outer-plan-20260719-postG6.txt` = 6/0/6, equals section 5 exactly | +| G8 | Same-session pre-apply re-verify: 6 planes still empty (0 leases / 0 attached domains) | [V] run in the SAME session as the apply | session | Last verified in a prior session only (finding doc:259-265) | +| G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | BLOCKED -- blocker: Batches 2-6 + G8 + Batch-6 exit runs incl. re-sign (G11); stop commit `43b29d7`; state shows nothing DC0 applied | +| G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Not reached (`docs/dc0-deploy-readiness.md:67-84,170-179`) | +| G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: SIGNED 2026-07-18, section 11 (post-probe version, per sequencing directive) | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator | HELD (`docs/dc0-deploy-readiness.md:100-103`) | +| G13 | D-129 residuals | [R] operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) | operator | OPEN / PARTIALLY RULED (`docs/design-decisions.md:4017`) | +| G14 | 8 OPEN SEC rows (SEC-001, -003..-008, -010; prior cell said 9, enumeration + scan both measure 8 -- corrected per GA-R1/C2) | [R] per-row: rotations/flips at v1 close (external to VR1 track) except SEC-010 = apply+verify at G10 | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count cross-checked vs `bash scripts/ledger-scan.sh` 2026-07-19 | +| G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | PROPOSED/OPEN (status lines, section 8) | ## 7. Version pins (measured; the authority for every pin) diff --git a/docs/audit/h1-token-map-20260719.md b/docs/audit/h1-token-map-20260719.md new file mode 100644 index 0000000..bb5b92a --- /dev/null +++ b/docs/audit/h1-token-map-20260719.md @@ -0,0 +1,40 @@ +# H1 token migration map -- sweep Batch 2 item 2.5 (2026-07-19) + +Per the ratified stage-status vocabulary (Option A + amendment H1, +`docs/audit/ga-rulings.md`): the extractor's status scan (record-audit.py +section 6, Batch 0.4) listed EVERY distinct status token; this map records +each token's disposition so none survives unmapped. The legal set +(OPEN/BLOCKED/HELD/CLOSED) is used ONLY in CURRENT-STATE.md. + +Scan input: 12 distinct tokens, 163 flagged counted occurrences +(pre-migration run, 2026-07-19). Post-migration verification at the foot. + +| Token | Class | Disposition | +|---|---|---| +| OPEN | legal | retained; CURRENT-STATE rows only as a status; elsewhere prose/history | +| BLOCKED | legal | retained; G9 migrated TO this from STOPPED (blocker named in the row) | +| HELD | legal | retained (vr1-dc1 G12 etc.) | +| CLOSED | legal | retained | +| STOPPED | illegal | -> BLOCKED with the blocker named (G9 + section-1 prose); commit-message quotes cited by hash, not re-quoted | +| DEPLOY-READY | illegal | -> removed (per Option A: readiness is not a state); readiness-doc banner demoted (GA-F02); historical mentions lowercased | +| READY | illegal | -> removed as status; no live row carried it (prose mentions lowercased where flagged) | +| COMPLETE | illegal | -> CLOSED where a gate/stage row carried it (none did on CURRENT-STATE after Batch-1 edits); prose/dated events lowercased | +| COMPLETED | illegal | same as COMPLETE (extractor normalizes) | +| DONE | illegal | -> dated lowercase event wording on live surfaces (workflow doc, runbooks, readiness); close-out instructions RETARGETED to write status into CURRENT-STATE.md, not the workflow tracker | +| TO RUN | illegal | -> OPEN with in-flight evidence (per H1's explicit mapping); the one live row (G4) already CLOSED in Batch 1 | +| RUNNING | illegal | as a STAGE status -> OPEN with in-flight evidence; every flagged occurrence was factual VM-run-state prose -> lowercased | +| VOID / REOPENED / SUPERSEDED | event words | legitimate as transition history in prose; the one status-position occurrence (workflow component table) lowercased into dated event wording | + +History-class surfaces deliberately NOT token-edited (their disposition is +structural, not lexical): `docs/session-ledger.md` (GA-R4 rule-6 one-time +rotation, Batch 4), `docs/changelog-*.md` + `docs/v1-redeploy-changelog.md` +(changelog class, GA-R2 consolidation, Batch 4), +`docs/design-decisions.md` (append-only register, KEPT per GA-R3). +Verbatim quotes (operator utterances, commit messages) are never reworded; +where a quote carried a token, the citation was switched to hash form. + +Post-migration verification (extractor re-run, 2026-07-19): zero flagged +occurrences on live non-history surfaces (CURRENT-STATE, workflow doc, +runbooks, readiness doc); remaining flags sit only on the history-class +files listed above, awaiting their Batch-4 rotation. Gate rows carry +GA-R6/E1 type tags ([V]/[R]) as of this batch. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 2d5d898..99ade64 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -4,8 +4,9 @@ regional) buildout, spanning many sessions. `docs/dc-dc-buildout-design.md` is the WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT sequence, cross-referenced to that plan, to D-100..D-110, and to what already -exists in this repo vs what still needs to be authored. Update the "state" column -as stages complete -- this doc is a tracker, not a one-time snapshot. +exists in this repo vs what still needs to be authored. STATUS AUTHORITY: stage and +gate status live ONLY in `docs/CURRENT-STATE.md` (GA-R1); this doc keeps stage +IDENTITY, links, and dated history -- it is no longer a status tracker. **Governing constraint (unchanged throughout):** minimize delta to Roosevelt. Reuse VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog @@ -28,7 +29,7 @@ appear under "PROPOSED / OPEN decisions." Full ruling record: `docs/changelog-20260709-stage0-ratification.md`. -**State:** DONE. +**State:** cleared 2026-07-09 (history; status authority: docs/CURRENT-STATE.md). --- @@ -100,18 +101,18 @@ --- -**State (2026-07-14): SUBSTRATE + SITE BOTH DONE. Stage 2 is in CLOSE-OUT, not in build.** +**State (history; status authority: docs/CURRENT-STATE.md):** substrate + site both built by 2026-07-14; stage closed and merged to main (PR #1 `e468d2e`, 2026-07-15). -> The 2026-07-13 "NOT DONE" list that stood here is **SUPERSEDED and was already stale when +> The 2026-07-13 "not done" list that stood here is **SUPERSEDED and was already stale when > written** -- MAAS-region, LXD, the LXD-KVM-host registration and both composed service > machines all landed the same day. This doc's status table has now drifted three times; the > correction below is measured, not assumed. -- **DONE (substrate):** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is +- **done (substrate):** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is BUILT and LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on udp/67). Config is REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1` - EXISTS and is RUNNING** (`module.voffice1` is in `opentofu/terraform.tfstate`). -- **DONE (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5 + EXISTS and is running** (`module.voffice1` is in `opentofu/terraform.tfstate`). +- **done (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5 ON `voffice1`; that LXD **registered back into MAAS as an LXD VM host**; and both service machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.10, NetBox 4.6.4) and `office1-tailscale` (10.10.1.11, subnet router for `10.10.0.0/22`, route now ENABLED on the @@ -129,7 +130,7 @@ proceeds), but it is the authoritative NetBox. `netbox.baldurkeep.com` is the production apex, held as a READ-ONLY v1 reference draft and NOT consulted by VR1 systems; the write-back to it is DEFERRED to end-of-deployment (may not happen) -- it is NOT a Stage 2 gate. So the "NetBox authoritative + populated" gate is met when `office1-netbox` holds - the COMPLETE validated dataset, NOT by any upstream write. See C2 for what "complete" and "verified" + the complete validated dataset, NOT by any upstream write. See C2 for what "complete" and "verified" require (the D-120 additions, plus a fidelity-check re-run -- the earlier "fidelity check clean" verdict predates the 2026-07-14 hardening and must be re-established). @@ -137,11 +138,11 @@ | # | Item | State | |---|---|---| -| C1 | Office1 LAN IPv6 deployed (`2602:f3e2:f01:100::1/64` + RA on the edge LAN). WAN v6 leg DEFERRED: v6 does not egress the lab. | **DONE 2026-07-14** -- live and PROVEN ON THE WIRE: address on the kernel, radvd advertising the prefix (`mode=unmanaged`), `voffice1` autoconfigured `...:5054:ff:fe6a:87e5` by SLAAC, edge->voffice1 GUA ping 0.0% loss. NOTE the D-113 AMENDMENT: the ADDRESS half is **not** REST-API-doable (`scripts/opnsense-set-interface-v6.sh`); only the RA half is. `docs/changelog-20260714-office1-lan-ipv6-executed.md` | -| C2 | `office1-netbox` (10.10.1.10) is the VR1 IPAM apex -- a working draft (DOCFIX-195) -- and holds the COMPLETE validated dataset: D-115 carve + six-plane roles + both DCs' prefixes/aggregates/RIRs, PLUS the D-120 additions -- `10.10.1.0/24`'s child ranges (`ip-ranges`: static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11). `netbox.baldurkeep.com` stays a READ-ONLY v1 reference draft; the write-back to it is DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15) -- NOT a Stage 2 gate. | **DONE 2026-07-15.** (i) The HARDENED fidelity check was re-run against office1-netbox -> exit 0 (faithful); it also surfaced + fixed a checker scope false-positive (`docs/changelog-20260715-fidelity-check-scope-correctness.md`). (ii) The tooling gap was CLOSED -- `prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover `ip-ranges`/`ip-addresses` -- and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox via `netbox/d120-compose-bands.py --commit` and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned delta (D-115/D-117/D-118/D-120)." `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. Gauntlet ALL GREEN (60). | -| C3 | The `$DC` shell-selector collision resolved. | **DONE 2026-07-14 (D-119)** -- selectors are REGION-QUALIFIED (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`) across `lib-net.sh`, `lib-hosts.sh`, `opentofu/`, the importer and every runbook call site. Bare `dcN` is REJECTED. The importer's DC->site map is now an IDENTITY, so the off-by-one is structurally impossible. Gap #19 CLOSED. | -| C4 | The NetBox sandbox loop documented. | **DONE 2026-07-14** -- `docs/dc-dc-netbox-buildout-scope.md` section 8 (rationale + write-guard table + WAF trap) and `runbooks/dc-dc-phase1-office1-standup.md` Step 10b (operational sequence). | -| C5 | This doc's Stage 2 row reconciled against reality (it has drifted three times). | **DONE 2026-07-14** | +| C1 | Office1 LAN IPv6 deployed (`2602:f3e2:f01:100::1/64` + RA on the edge LAN). WAN v6 leg DEFERRED: v6 does not egress the lab. | **done 2026-07-14** -- live and PROVEN ON THE WIRE: address on the kernel, radvd advertising the prefix (`mode=unmanaged`), `voffice1` autoconfigured `...:5054:ff:fe6a:87e5` by SLAAC, edge->voffice1 GUA ping 0.0% loss. NOTE the D-113 AMENDMENT: the ADDRESS half is **not** REST-API-doable (`scripts/opnsense-set-interface-v6.sh`); only the RA half is. `docs/changelog-20260714-office1-lan-ipv6-executed.md` | +| C2 | `office1-netbox` (10.10.1.10) is the VR1 IPAM apex -- a working draft (DOCFIX-195) -- and holds the complete validated dataset: D-115 carve + six-plane roles + both DCs' prefixes/aggregates/RIRs, PLUS the D-120 additions -- `10.10.1.0/24`'s child ranges (`ip-ranges`: static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11). `netbox.baldurkeep.com` stays a READ-ONLY v1 reference draft; the write-back to it is DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15) -- NOT a Stage 2 gate. | **done 2026-07-15.** (i) The HARDENED fidelity check was re-run against office1-netbox -> exit 0 (faithful); it also surfaced + fixed a checker scope false-positive (`docs/changelog-20260715-fidelity-check-scope-correctness.md`). (ii) The tooling gap was CLOSED -- `prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover `ip-ranges`/`ip-addresses` -- and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox via `netbox/d120-compose-bands.py --commit` and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned delta (D-115/D-117/D-118/D-120)." `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. Gauntlet ALL GREEN (60). | +| C3 | The `$DC` shell-selector collision resolved. | **done 2026-07-14 (D-119)** -- selectors are REGION-QUALIFIED (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`) across `lib-net.sh`, `lib-hosts.sh`, `opentofu/`, the importer and every runbook call site. Bare `dcN` is REJECTED. The importer's DC->site map is now an IDENTITY, so the off-by-one is structurally impossible. Gap #19 CLOSED. | +| C4 | The NetBox sandbox loop documented. | **done 2026-07-14** -- `docs/dc-dc-netbox-buildout-scope.md` section 8 (rationale + write-guard table + WAF trap) and `runbooks/dc-dc-phase1-office1-standup.md` Step 10b (operational sequence). | +| C5 | This doc's Stage 2 row reconciled against reality (it has drifted three times). | **done 2026-07-14** | --- @@ -156,9 +157,9 @@ | **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Originally walked a set of STOP-gate decisions; most are now RULED (see the rulings summary below): node sizing = D-121/R-3, edge/WAN shape = D-122, site-down/placement = D-123 Model B, rack transit = D-124, ISP egress = D-125, edge-config delivery = D-113(a2) REST (config.xml/ISO retired -- no more `config.xml` tokens). STILL open at execution: MAAS zone/pool + `power_address` (Stage-4 concerns), boot-measured `WAN_IF`/`LAN_IF` (real OPNsense boot), and netem params (D-100 gap #11). NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`; under Model B it targets `vvr1-dc0`'s inner virsh -> Office1 region, DOCFIX-179-deferred) all now exist AND are instantiated across the two Model B roots (D-123): the outer root (`opentofu/`) + the inner root (`opentofu/vr1-dc0-substrate/`), plus the new `modules/wan-bridge` (D-125). Both roots `tofu validate`. Remaining un-instantiated is deploy-time (MAAS zone/pool, `power_address`) not authoring. `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | -**State:** IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`, pushed through `d34ec94`). RUNBOOK -WRITTEN + RESHAPED to Model B, NOT YET EXECUTED. Authoring for this stage is effectively COMPLETE; what -remains is operator-gated (HELD NetBox addressing) + deploy-time gates. +**State:** THE ACTIVE STAGE (identity: branch `dc-dc-stage3-phase2-dc-substrate`); status +authority: docs/CURRENT-STATE.md sections 1 and 6. History: runbook written + reshaped to +Model B; authoring effectively complete; execution position and gates live in CURRENT-STATE. **Rulings landed (through 2026-07-16; supersedes the earlier "two blocking rulings landed 2026-07-15" note):** - **D-121 (ADOPTED) + R-3** -- node layout = **Option C** = 3 control + 2 compute + **4 storage = 9 @@ -203,7 +204,7 @@ | **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing now RULED (D-121/R-3 = 9/DC), so that sub-gap is closed; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. +**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md). --- @@ -218,7 +219,7 @@ | **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`.** Does not duplicate `phase-01..08`'s content -- describes what changes to run that loop twice (once per DC): DC1 needs NO VIP/CIDR edits at all (D-101 inherits DC0 unchanged); DC2 is hard-blocked pending its NetBox literals. Requires every adapted `phase-01..08` command block that reads `lib-net.sh`/`lib-hosts.sh` to call `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` first (DOCFIX-151). Flags a real, unresolved design gap: the D-101 IPv6 family-matrix overlay (dual-stack/ULA-only per-plane charm config) does not exist yet -- the exact charm config option names for v6 addressing have not been confirmed against any charm's real schema, so no overlay file is invented; Step 6 names this as blocking work rather than gliding past it. Also flags D-109's IPv6-SAN cert-issuance support as an unverified charm-behavior assumption. NOT YET EXECUTED. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. **This is where the existing repo tooling (`preflight.sh`, +**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md). **This is where the existing repo tooling (`preflight.sh`, `cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they apply to either DC's clone/session without modification.** @@ -236,7 +237,7 @@ | **Reuse vs new** | Ground truth: `docs/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase5-dr-failover-drill.md`.** Command-level implementation of Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's own requirement, plus the D-104 controller backup/restore drill. Found and flagged three real bundle gaps while writing it (grepped `bundle.yaml`): no `cinder-backup` charm deployed, no `ceph-rbd-mirror` charm deployed, and `ceph-mon`'s existing `rbd-mirror` endpoint binding is on `storage` instead of `replication` (inherited from the VR0/DC0 seed, predates D-108) -- all three become this runbook's own first gated mutation (Step 2), not silently routed around. Every realm/zone/pool/endpoint name is an explicit `` placeholder. Includes a negative-control sub-drill (sever only the replication link first, confirm the split-brain check correctly refuses to promote) before the real hard-down drill. NOT YET EXECUTED -- no live Ceph cluster this session (harness for gap #5's scripts is future work, logged, not built). | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. +**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md). --- @@ -251,7 +252,7 @@ | **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase6-designate-cos-magnum.md`.** Quotes `bundle.yaml`'s and `phase-01-bundle-deploy.md`'s actual current "NO designate (D-019)" text verbatim rather than paraphrasing, then the D-106 reversal against it. Flags that the reversal is bigger than "add designate" -- D-106's bootstrap order also reactivates `os-public-hostname` per API charm, reversing the bundle's separate B5 IP-only posture; proposes (not silently adopts) pushing per-DC hostname literals into new `overlays/dc1-hostnames.yaml`/`dc2-hostnames.yaml` files to keep `bundle.yaml` itself DC-agnostic, mirroring the existing octavia-pki overlay pattern. D-046's domain-setup trap quoted verbatim in a top callout and again inline. COS deployment mechanism explicitly left as an operator decision point (D-105 rules only the scope) rather than invented. Reuses `phase-06/07/08` by filename with concrete DC-parameterization notes (confirmed via tracing which scripts source `lib-net.sh`). NOT YET EXECUTED. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. +**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md). --- @@ -326,7 +327,7 @@ > `docs/changelog-20260713-opnsense-api-proven.md` and > `docs/changelog-20260713-opnsense-api-write-proven.md`. > -> **DONE (D-113(a2)) -- reconciled 2026-07-16; the repo won over this register's stale "OPEN".** The +> **done (D-113(a2)) -- reconciled 2026-07-16; the repo won over this register's stale "OPEN".** The > config-ISO path and the module's `config_seed`/cdrom wiring are RETIRED: `modules/opnsense-edge` no > longer takes `config_iso_path` (verified -- no `config_seed`/`config_iso_path` anywhere in the module), > `opentofu/templates/opnsense-config.xml.tmpl` is DELETED, and Stage 3's edge steps (phase-2 Steps 4-5) @@ -449,13 +450,13 @@ assignment step is unchanged by this delivery; this closes the TOOLING gap, not the DATA gap. Still blocks Stage 2/3 until those literals are assigned and this script is actually run against a live NetBox. -4. **OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) DONE +4. **OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) done 2026-07-09, (d) is the only real gap left.** Each site (Office1, DC1, DC2) gets its own independent OPNsense edge with a simulated ISP uplink (D-100); OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC edges) gates, which require netem parameters "applied and measured." - - **(a) Network substrate -- DONE.** `opentofu/modules/mesh-link` creates + - **(a) Network substrate -- done.** `opentofu/modules/mesh-link` creates the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an OPNsense VM's interfaces would eventually attach to. - **(b)+(c) The OPNsense VM/image AND its config -- BUILT @@ -610,7 +611,7 @@ this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`, already real required inputs elsewhere in the same file. See `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` - (DOCFIX-163) for the full design rationale. **Follow-on work also DONE + (DOCFIX-163) for the full design rationale. **Follow-on work also done 2026-07-13 (D-114):** the "designing the service VMs' actual `cloudinit-vm` `network_config` against this network" tail of this gap is closed, but not the way it was written -- there are no longer three service VMs to attach. @@ -867,17 +868,17 @@ | Layer | Status | |---|---| -| Design (`dc-dc-buildout-design.md`) | DONE -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) | +| Design (`dc-dc-buildout-design.md`) | done -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) | | Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED | -| DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | -| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. Stage 1 EXECUTED 2026-07-10. **Stage 2's runbook REWRITTEN 2026-07-13 to the D-114 model** (one `voffice1` site containment VM + MAAS-composed LXD VMs, replacing the three sibling service VMs and their Option A/B fork); it is PARTIALLY EXECUTED, and its MAAS/LXD command lines are marked PENDING VERIFICATION rather than guessed. Stages 3-7 NOT executed, and D-114 GATES them behind Office1 completing. See gaps #12-15 above for real design gaps the authoring pass surfaced. | -| **D-114 site containment VM (`voffice1`)** | **BUILT + RUNNING 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. | +| DR mechanism seed | done (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | +| Stage 0-7 runbooks | Stage 0 done (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. Stage 1 EXECUTED 2026-07-10. **Stage 2's runbook REWRITTEN 2026-07-13 to the D-114 model** (one `voffice1` site containment VM + MAAS-composed LXD VMs, replacing the three sibling service VMs and their Option A/B fork); it is PARTIALLY EXECUTED, and its MAAS/LXD command lines are marked PENDING VERIFICATION rather than guessed. Stages 3-7 NOT executed, and D-114 GATES them behind Office1 completing. See gaps #12-15 above for real design gaps the authoring pass surfaced. | +| **D-114 site containment VM (`voffice1`)** | **BUILT + running 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. | | `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (tofu binary on the jumphost; pin authority `docs/CURRENT-STATE.md` section 7): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds the DC1 planes, pools, 3 mesh links, `office1-network`, the OPNsense edge, and (2026-07-13, D-114) the `ubuntu_noble_base` image + the **`voffice1` containment VM**. **`modules/cloudinit-vm` is now instantiated for real** and gained `expose_nested_virt`. **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. | -| OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + RUNNING 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. | -| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **SUPERSEDED 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **DONE (reconciled 2026-07-16):** `templates/opnsense-config.xml.tmpl` is DELETED and the module's `config_seed`/cdrom + `config_iso_path` are RETIRED; edge config is the REST-API bootstrap. (Boot-measured `WAN_IF`/`LAN_IF` are still applied over the API at deploy time -- a deploy-time step, not an authoring gap.) | +| OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + running 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. | +| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **superseded 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **done (reconciled 2026-07-16):** `templates/opnsense-config.xml.tmpl` is DELETED and the module's `config_seed`/cdrom + `config_iso_path` are RETIRED; edge config is the REST-API bootstrap. (Boot-measured `WAN_IF`/`LAN_IF` are still applied over the API at deploy time -- a deploy-time step, not an authoring gap.) | | MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` | | `tc netem` mechanism (`modules/netem-link`) | **WAS BROKEN, FIXED 2026-07-13 (DOCFIX-194)** -- its destroy provisioner referenced `var.*`, which OpenTofu rejects at INIT, so the module could not even initialize. Now validates. Real latency/jitter/loss/rate parameters are STILL an unruled D-100 item. Never applied. | -| `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` | +| `opentofu/` (DC2 planes) | not done -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` | | Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` | | Office1 OPNsense edge ownership | **BUILT AND LIVE 2026-07-12/13.** `module "office1_opnsense"` is instantiated (not commented). **Gap #17 is CLOSED FOR OFFICE1**: `office1-wan` (virbr11, NAT, 172.30.1.0/24) is the dedicated per-site ISP uplink, and the edge's WAN sits on it -- the D-100-honouring answer (NOT a mesh leg). **DC1/DC2 still have NO uplink network** -- replicating the `office1-wan` shape per DC is now mechanical, not a decision. | | Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` | diff --git a/docs/dc0-deploy-readiness.md b/docs/dc0-deploy-readiness.md index 484da5c..251024a 100644 --- a/docs/dc0-deploy-readiness.md +++ b/docs/dc0-deploy-readiness.md @@ -8,8 +8,8 @@ > Historical prep record (2026-07-17, this doc's own gate re-check that day): opentofu-validate > clean (both roots + 12 modules), the four `vr1_dc0_rack_*` + `vr1_dc0_ssh_pubkey_path` tfvars in place, > `creds-audit vr1-dc0` CLEAN, gauntlet ALL GREEN (65), repo-lint 0-fail. Per-env-key (D-126 option a) -> DONE; netem = placeholder (deploy-time, Step 11). This blockquote is HISTORY, not a status claim -> (its former DEPLOY-READY banner was falsified post-reboot -- GA-F01/GA-F02). Also noted then: (1) the final +> done; netem = placeholder (deploy-time, Step 11). This blockquote is HISTORY, not a status claim +> (its former deploy-ready banner was falsified post-reboot -- GA-F01/GA-F02). Also noted then: (1) the final > **bookend restart** -- OS patch + reboot of vcloud + voffice1 + office1-opnsense, done together as the > boundary into the new session (voffice1 is load-bearing like vcloud -- it hosts netbox/MAAS + the only > netbox path, so it defers WITH vcloud, not mid-prep); (2) fresh-session start = re-verify the plan @@ -57,7 +57,7 @@ ### C1. Pre-apply (settle BEFORE the outer `tofu apply`) 1. **[RESOLVED 2026-07-16] Per-env key for `vvr1-dc0` = option (a) dedicated dc0 key (operator ruling).** - DONE: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644, `creds-audit vr1-dc0` CLEAN); added + done: minted `~/vr1-dc0-creds/vr1-dc0_svc_ed25519` (600/644, `creds-audit vr1-dc0` CLEAN); added `variable "vr1_dc0_ssh_pubkey_path"`; `vvr1-dc0` cloud-init now installs the dc0 pubkey (voffice1 keeps office1's key); tfvar set. Re-validated + re-planned -> outer plan UNCHANGED at the then-current gate triple (since superseded; the expected triple lives in CURRENT-STATE.md @@ -88,10 +88,10 @@ - MAAS (D): discovers the inner node VMs (needs C2.3). ### C4. IPAM record hygiene (parallel, NON-blocking -- now runnable via `ssh office1-netbox`) -5. **D-124 NetBox import** -- DONE 2026-07-17. `netbox/d124-transit-seed.py` (NEW) seeded role `transit` +5. **D-124 NetBox import** -- done 2026-07-17. `netbox/d124-transit-seed.py` (NEW) seeded role `transit` + `172.31.0.0/24` container; `dc-rack-mgmt-import` placed `172.31.0.0/30` (vr1-dc0) + `10.12.8.2/22` (rack). Ran on the apex host, token server-side, sandbox-only. -6. **Edge-WAN import** -- DONE 2026-07-17. `dc-edge-wan-import` created `172.30.2.0/24` + `172.30.3.0/24` +6. **Edge-WAN import** -- done 2026-07-17. `dc-edge-wan-import` created `172.30.2.0/24` + `172.30.3.0/24` (edge). (Also recovered the apex: netbox-docker was down; brought up + `restart: unless-stopped` added.) ### C5. Record-hygiene sweep (NON-blocking; queued from the Stage-3 adversarial review) diff --git a/runbooks/appendix-A-troubleshooting.md b/runbooks/appendix-A-troubleshooting.md index 8163315..a0f7100 100644 --- a/runbooks/appendix-A-troubleshooting.md +++ b/runbooks/appendix-A-troubleshooting.md @@ -406,7 +406,7 @@ ### octavia-configure-resources -- long-running action; o-hm0 transient is normal (phase-05) - `configure-resources` is long-running: juju's default action wait may time out - ("timed out waiting for results") while the hook KEEPS RUNNING -- do NOT treat the + ("timed out waiting for results") while the hook KEEPS running -- do NOT treat the wait-timeout as failure or re-fire blindly. Use a bound `--wait` and confirm completion via `juju show-operation ` (authoritative), not the streamed log. - NORMAL (not faults) during/after: lb-mgmt-net is IPv6-ULA (fc00::/..) by design; a diff --git a/runbooks/dc-dc-office1-service-reip.md b/runbooks/dc-dc-office1-service-reip.md index 3992e27..87bdc92 100644 --- a/runbooks/dc-dc-office1-service-reip.md +++ b/runbooks/dc-dc-office1-service-reip.md @@ -79,7 +79,7 @@ curl -s -o /dev/null -w "%{http_code}\n" http://10.10.1.10:8000/api/ # 403/200 = up ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.11 'tailscale status' # tailscale at .11 -## Step 5 -- Update the repo references + the importer allowlist [DONE 2026-07-15] +## Step 5 -- Update the repo references + the importer allowlist [done 2026-07-15] `10.10.1.201 -> 10.10.1.10` and `10.10.1.202 -> 10.10.1.11` applied across: diff --git a/runbooks/dc-dc-phase0-vcloud-prep.md b/runbooks/dc-dc-phase0-vcloud-prep.md index 785aab2..a3c4474 100644 --- a/runbooks/dc-dc-phase0-vcloud-prep.md +++ b/runbooks/dc-dc-phase0-vcloud-prep.md @@ -500,9 +500,9 @@ **GATE (buildout-design Section 4, Phase 0):** nested KVM verified (Step 2/4); disk budget computed and Ceph size decision recorded (Step 3); measured MTU recorded (Step 3); virtual networks present and isolated as designed -(this step). All four true -> Stage 1 of `docs/dc-dc-deployment-workflow.md` -moves from NOT STARTED to DONE; update that doc's Stage 1 row and Section -4's tracker table accordingly, and update `opentofu/README.md`'s +(this step). All four true -> Stage 1 is done: record the status change in +`docs/CURRENT-STATE.md` (the ONLY status authority, GA-R1; the workflow doc +keeps identity + history only), and update `opentofu/README.md`'s SCAFFOLD/UNVALIDATED banner to reflect the first real validated run. -> Proceed to Stage 2 (Office1 headend standup): MAAS region controller, @@ -522,6 +522,6 @@ values (host specs, MTU, pool paths -- redact nothing that isn't a secret; these are exactly the kind of as-built facts this repo commits). - [ ] `docs/session-ledger.md` updated with the outcome. -- [ ] `docs/dc-dc-deployment-workflow.md` Stage 1 row and tracker table - updated from NOT STARTED to DONE (or to whatever partial state is - honest, if the run surfaces a blocker). +- [ ] `docs/CURRENT-STATE.md` updated with the honest Stage 1 outcome + (GA-R1: status lives THERE only; partial state named if the run + surfaces a blocker). diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index b9ed878..e45405c 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -22,10 +22,10 @@ **Execution status (measured 2026-07-13, not inferred):** -- Office1's OPNsense edge (`office1-opnsense`) is **BUILT, RUNNING, ROUTING, and +- Office1's OPNsense edge (`office1-opnsense`) is **BUILT, running, ROUTING, and SERVING DHCP** (Kea). Its config is REST-API-managed per D-113(a2). - `office1-local` and `office1-wan` libvirt networks exist and are active. -- **`voffice1` EXISTS and is RUNNING** (`virsh list --all` -> `voffice1 running`; +- **`voffice1` EXISTS and is running** (`virsh list --all` -> `voffice1 running`; `module.voffice1` is in `opentofu/terraform.tfstate`). It was applied by the main session's `tofu apply`, not by this runbook's authoring pass. - **NOT yet done:** everything from Step 5 onward -- MAAS-region on `voffice1`, @@ -144,7 +144,7 @@ CLOSED.** The old fork ("OpenTofu `cloudinit-vm`, blocked pending user_data design" vs "manual `virt-install`, logged as debt") existed because no `cloudinit-vm` instantiation had ever been designed. **OpenTofu (Option A) is - the path, and it is DONE:** `module "voffice1"` in `opentofu/main.tf` is a + the path, and it is done:** `module "voffice1"` in `opentofu/main.tf` is a real, applied `cloudinit-vm` instantiation -- real image source (`modules/base-image`, the official Ubuntu 24.04 noble cloud image), real `user_data` (identity + access + guest agent), real `meta_data`, real @@ -313,7 +313,7 @@ ## Step 3 -- voffice1: the site containment VM [MUTATION: gated] > **STATUS: APPLIED 2026-07-13.** `module.voffice1` is in -> `opentofu/terraform.tfstate` and the domain is RUNNING (measured this session: +> `opentofu/terraform.tfstate` and the domain is running (measured this session: > `virsh list --all` -> `voffice1 running`). The apply was performed by the main > session, not by this runbook's authoring pass. This step is retained as the > procedure -- for a re-standup, and so the gate it satisfies is auditable. @@ -854,13 +854,13 @@ and `netbox/dc-dc-prefixes-import.py` has actually run against the real instance. This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** -- -the substrate half (networks, edge, `voffice1`) is DONE and measured; the SITE +the substrate half (networks, edge, `voffice1`) is done and measured; the SITE half (MAAS, LXD, the LXD KVM host, and every composed service machine) is the remaining work, and its command lines are PENDING VERIFICATION. The NetBox DATA bullet closes its mechanism half only, pending the D-101 literal assignment that -is explicitly out of scope for this stage to invent. Update -`docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever of these -is actually true after a real run, not to "DONE" wholesale. +is explicitly out of scope for this stage to invent. Record whichever of these +is actually true after a real run in `docs/CURRENT-STATE.md` (GA-R1: the only +status authority), never a wholesale rounding-up. -> **Stage 3 (DC1) is GATED behind this stage.** D-114's sequencing is an operator ruling: Office1 FIRST, fully up on this model; DC1 is not started until diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index 685a03c..d29c823 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -168,7 +168,7 @@ D-113 firmware API (`scripts/opnsense-api.sh POST core/firmware/install/`) -- `qemu-guest-agent` gives the inner libvirt host clean guest coordination (graceful shutdown/fsfreeze/IP report; pairs with D-127 autostart), `iperf` validates the transit path. **Verify the `qemu-guest-agent` daemon is - RUNNING and answers `virsh ... guest-ping` -- not just the package present.** USE the built installer: + running and answers `virsh ... guest-ping` -- not just the package present.** USE the built installer: `OPNSENSE_API_HOST= bash scripts/opnsense-plugins.sh apply vr1-edge` (BUILT 2026-07-18, `tests/opnsense-plugins/`) -- it installs **one plugin at a time and polls `core/firmware/upgradestatus` to completion** before the next (back-to-back installs collide on the OPNsense firmware lock and the later @@ -724,7 +724,7 @@ `docs/dc-dc-deployment-workflow.md`'s Stage 3 row to reflect the REAL state after running this (e.g. "PARTIAL -- libvirt objects created, MAAS registration blocked on the per-DC rack controller" or similar), never to a -blanket DONE unless every sub-condition above is genuinely true. +blanket done unless every sub-condition above is genuinely true. **`vr1-dc1`:** none of the above applies to `vr1-dc1` this pass. Its v4 supernet IS assigned (D-115, `10.12.64.0/19`) -- the blocker is sequencing, @@ -755,9 +755,9 @@ sub-steps were blocked and why (per-DC MAAS rack-controller Stage-4 gap, Stage 2 MAAS-region dependency, the edge post-boot bootstrap, D-100 netem sub-item). -- [ ] `docs/dc-dc-deployment-workflow.md` Stage 3 row updated to the REAL, - honest state (PARTIAL/DONE/NOT STARTED per the GATE section above), - never rounded up to DONE with open sub-conditions. +- [ ] `docs/CURRENT-STATE.md` updated to the REAL, honest Stage 3 state + (GA-R1: the only status authority; per the GATE section above), + never rounded up with open sub-conditions. - [ ] `opentofu/README.md` updated to reflect `vr1-dc0`'s OPNsense edge / node-VM / MAAS-vm-host / netem-link modules moving from "not instantiated" to instantiated (or partially so) once this runbook diff --git a/runbooks/dc-dc-phase5-dr-failover-drill.md b/runbooks/dc-dc-phase5-dr-failover-drill.md index b85ce2f..557c11b 100644 --- a/runbooks/dc-dc-phase5-dr-failover-drill.md +++ b/runbooks/dc-dc-phase5-dr-failover-drill.md @@ -609,9 +609,9 @@ Consolidate every timing recorded in Steps 6, 7, 9, 10.5, and 11.5 into this runbook's execution log entry (changelog + `docs/session-ledger.md`, per this repo's standard delivery -discipline). Update `docs/dc-dc-deployment-workflow.md` Stage 6's `**State:**` line from NOT -STARTED to whatever is honestly true after this run (fully DONE only if every GATE above -passed; partial otherwise, with the specific blocking item named). +discipline). Record Stage 6's honest outcome in `docs/CURRENT-STATE.md` (GA-R1: the only status +authority; fully done only if every GATE above passed; partial otherwise, with the +specific blocking item named). --- @@ -630,7 +630,7 @@ not assumed. - All bundle prerequisite gaps from the "Known gaps" section resolved and logged (Step 2). -All true -> Stage 6 of `docs/dc-dc-deployment-workflow.md` moves to DONE; proceed to Stage 7 +All true -> Stage 6 is done: record it in `docs/CURRENT-STATE.md` (GA-R1); proceed to Stage 7 (Designate, COS, Magnum/CAPI, per DC). --- @@ -648,7 +648,7 @@ commits, not secrets. - [ ] `docs/session-ledger.md` updated with the outcome. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 6 row and tracker table updated (Step 12). -- [x] **Extraction DONE 2026-07-09 (DOCFIX-165), not future work anymore -- and Steps 4, 5, 8, +- [x] **Extraction done 2026-07-09 (DOCFIX-165), not future work anymore -- and Steps 4, 5, 8, 10.2, 11.2-11.4 above now each carry a "PREFER `scripts/dc-dc-...`" pointer at their own top (added 2026-07-10 during a full-project sweep, after the first version of this checklist item claimed this was already done when the step bodies themselves still only diff --git a/runbooks/dc-dc-phase6-designate-cos-magnum.md b/runbooks/dc-dc-phase6-designate-cos-magnum.md index 8a347e0..813dc32 100644 --- a/runbooks/dc-dc-phase6-designate-cos-magnum.md +++ b/runbooks/dc-dc-phase6-designate-cos-magnum.md @@ -239,7 +239,7 @@ already uses. This is a proposed pattern, not yet reviewed -- present it at the same review gate as the rest of Step 1, do not silently adopt it. -### 1.5 -- The application/relation shape (DONE 2026-07-10 -- bundle.yaml already carries this) +### 1.5 -- The application/relation shape (done 2026-07-10 -- bundle.yaml already carries this) **This step is CLOSED, not a template anymore.** `bundle.yaml` now carries real `designate`/`designate-bind`/`designate-mysql-router`/`designate-