diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 4b944b4..c80efcc 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -2,9 +2,8 @@ **Purpose.** Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback -is lost at compaction. This ledger is the durable, committed record of what is in flight, -so any session -- after a compaction, or a fresh one, or the parallel Claude Code stream -- -can resume without losing pending work. +is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, +so any session -- after a compaction, or a fresh one -- can resume without losing pending work. **How to use it (standing practice).** 1. At session start: read this ledger AND run `bash scripts/ledger-scan.sh`. Reconcile. @@ -13,2177 +12,212 @@ This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces. 3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN. -4. Numbers and the machine-derived section below are seeded from a scan; re-run the scan and - re-seed rather than editing those by hand. +4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather + than editing it by hand. + +**SINGLE STREAM (collapsed 2026-07-13).** This ledger previously carried three parallel, +separately-owned stream sections (`main-chat`, `jumphost`, `shared`) plus ~30 append-only +session narratives. Those streams are CLOSED and reconciled into the one list below. There is +now ONE stream. Do not re-introduce per-stream sections. + +**Where the history went.** The 2,189-line session-by-session narrative is NOT lost -- it is in +git history (the parent of the collapse commit) and, in durable form, in the 65 `docs/changelog-*.md` +files, `docs/design-decisions.md`, and the incident reports. This ledger deliberately carries only +what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else. --- ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) -_As of 2026-07-09 scan (post-DOCFIX-161, final consolidation pass); -re-run the scan to refresh:_ +_As of the 2026-07-13 scan (the collapse). Re-run the scan to refresh._ -- **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 - ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + - controller patch policy -- operator to rule; gated on the pre-DC-DC HA/backup - session). **D-100..D-110 are ALL ADOPTED as of 2026-07-09** (D-102 MERGED into - D-101; D-109 confirmed standalone) -- Stage 0 of - `docs/dc-dc-deployment-workflow.md` is CLEARED; see - `docs/dc-dc-buildout-design.md` Section 10 for the ruling record. _(D-050 - RESOLVED/CLOSED 2026-07-06 via D-051; D-073 APPLIED live 2026-07-06 -- not - scan-tracked as open. D-076..D-099 are a RESERVED source-project band, never - assigned in this repo, per D-110.)_ -- **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), - SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), - SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 111, DOCFIX = 162, BUNDLEFIX = 012. - (Seeded from the 2026-07-09 scan, post DOCFIX-161.) +- **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT + per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju + controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on + the pre-DC-DC controller HA/backup session). +- **OPEN security rows:** SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007. + (The prior machine block listed only SEC-001/003/004 -- it was STALE by three rows.) +- **Next-free numbers:** D = 114, DOCFIX = 195, BUNDLEFIX = 013. + (The prior machine block said D=111 / DOCFIX=162 / BUNDLEFIX=012 -- STALE.) +- **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) + ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- `ledger-scan` reads + prose and a decoy token inflates the next-free counter. This has bitten twice. + + --- - - -## Active build (this session) +## Live state (measured 2026-07-13; nothing is mid-flight) -- **validate.sh D-011 runner -- modular.** Foundation committed: `scripts/lib-validate.sh` - (exit contract 0/1/2/3/4, emit, vr_json stderr-safe, run, env hygiene, disruptive gate) + - `scripts/validate.sh` orchestrator (profiles, verdict, --stop-on-fail, --include-disruptive). - - Batch 1 committed: `checks/d011-01-charms`, `checks/d011-06-vault-unseal` (MANUAL until - the SEC-003 rehearsal closes). - - Batch 2 committed: `checks/d011-02-vip-jumphost` (catalog-derive public origins, TLS+HTTP), - `checks/d011-03-vip-tenant` (ephemeral agnhost pod egress -> keystone VIP, D-035 proof). - `post-restart` profile now fully populated. - - Batch 3 DRAFTED (mock-tested only, NEEDS LIVE VALIDATION): `d011-04-octavia-lb` - (RR+member additive; amphora failover --disruptive, N+1 headroom-guarded) + - `d011-05-magnum-e2e` (wraps tenant-acceptance + timing). full-d011 profile now COMPLETE. - - **Next:** live-validate batch 3 (see Verify-live queue), then item-3/#5-#8 backlog. - - D-011 AMENDED bar: 1 charms; 2 VIP jumphost; 3 VIP tenant; 4 octavia RR/failover/recovery; - 5 magnum e2e + OCCM; 6 second-person manual unseal (attestation, D-069); 7 DROPPED - (D-070 supersedes D-012 snapshots); 8 DROPPED (D-019, no Designate). +- **VR1 / Office1 edge `office1-opnsense` is UP and healthy.** LAN 10.10.0.1/24, WAN 172.30.1.2 + (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial + console + getty alive, and `kea-dhcp4` serving DHCP on udp/67 (subnet 10.10.0.0/24, pool + .100-.199). **DHCP is API-MANAGED (D-113(a2)).** +- **OpenTofu repo/state are IN SYNC** -- `tofu plan` reports "No changes". tofu v1.12.3; the + tree validates (10/10 modules); `opentofu/.terraform.lock.hcl` is committed. +- **VR0 / DC0 testcloud:** fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except + item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own + cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts). +- Branch `dc-dc-stage1-phase0-first-exec`, pushed. VR1 Stage 1 COMPLETE; Stage 2 partially done. -## Script backlog (changelog-tracked) +--- -- DONE: tenant-offboard.sh, vault-kv-health.sh, cloud-assert.sh (item 3 largely), validate - foundation + batch 1. -- **item 3 remainder:** fold `vault-kv-health.sh` into cloud-assert as a section; use the - combined gate to regenerate the stale restart-procedure runbook. -- **item 5:** `scripts/tenant-cluster-create.sh` -- generalize the t2-02 stage6 + watch + - D-066-evidence wrapper. Not started. -- **item 6:** `scripts/keystone-policy-drift.sh` -- D-051 base_* alignment (LIVE-READ PENDING). - Not started. -- **item 7:** `scripts/cloud-snapshot.sh` -- juju-export baseline capture (D-070 retired KVM - snapshots but KEPT the export/inventory baseline). Not started. -- **item 8:** `tests/tenant-acceptance/` harness -- pending (tenant-onboard harness landed). +## OPEN WORK -- VR1 DC-DC (the current mission) -## Register / operator-gated (see docs/handoff-20260703-open-items.md) +The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling +gap register). Read it FIRST. This list is only what that doc does not already carry. -- **R-1 / SEC-003:** Vault unseal-key custodian assignment (OPEN). Gates V-2 and closes - `d011-06-vault-unseal` (currently MANUAL). Operator input, recorded off-repo. -- **V-2:** second-person unseal rehearsal (blocked on R-1). -- **R-2 (D-043 caveat):** capi-mgmt-v2 auto-resume exclusion -- rule or accept. -- **R-3:** was D-063 -- now CLOSED. The handoff doc still lists R-3 OPEN; update it. -- **D-1:** Pattern-A full redeploy (VR0 DC0). +1. **Stage 2 is the critical path and is NOT finished: the three Office1 headend service VMs** + (MAAS-region, NetBox, GitBucket) do not exist. Blocked on two undesigned things, both repo-side: + (a) no base image source chosen; (b) no real `user_data`/`meta_data`/`network_config` written + for the `cloudinit-vm` module. The Stage 2 runbook offers an explicit fork -- Option A (OpenTofu + `cloudinit-vm`) vs Option B (manual `virt-install`, logged as debt). **UNRULED: the operator has + not picked.** Everything downstream (Stage 3+) waits on this. +2. **NetBox sandbox loop is undocumented.** Operator-clarified architecture: draft-source + `netbox.baldurkeep.com` (READ ONLY) -> stand up a NEW NetBox on the Office1 VM (sandbox) -> + import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK to + the real NetBox. The sim NEVER `--commit`s upstream. `docs/dc-dc-netbox-buildout-scope.md` + and the Stage 2 runbook still need a note capturing this loop. PENDING. +3. **`pynetbox` / `python3-venv` are not installed** on the vcloud host -- needed for the NetBox + import against the Office1 sandbox; needs operator sudo. +4. **The apparmor rule is applied on THIS host but is NOT in the repo.** `/var/lib/libvirt/vr1/** rwk,` + in `/etc/apparmor.d/local/abstractions/libvirt-qemu` GATES EVERY VR1 VM BOOT (custom pool path is + outside libvirt's abstraction; without it a domain defines but qemu fails Permission-denied). + It is a one-time operator sudo that exists only as host state -- a rebuild or a second host hits + the identical foundational failure. **Fold it into `scripts/prereqs/`.** (Was logged as a PREREQ + candidate and never actioned.) +5. **DC2 planes are DEFERRED (operator ruling, Option B):** do not invent CIDRs -- wait for NetBox + to assign the D-101 supernet/ULA/GUA. DC2 mesh legs and `dc2_storage` are already wired. + Values pending final import: ORG_ULA_48 `fd50:840e:74e2::/48`, GUA dc1 `f02::/48` / dc2 `f03::/48`, + DC2_V4_SUPERNET `10.13.0.0/19`. +6. **G5 NetBox site rename** (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) needs its own D-number. + Also open in NetBox: G2 (aggregates empty -- top blocks are container prefixes), G3 (ULA /48 + not assigned). +7. **DOCFIX-185 follow-up (operator to rule):** a D-100/D-107 amendment note so the Stage-3 DC + edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark + fiber is East-West/replication ONLY, never an internet path). +8. `docs/dc-dc-deployment-workflow.md`'s status table was corrected 2026-07-13, but re-read it + against reality before trusting any row -- it has drifted twice. +9. **Doc-accuracy finding (measured 2026-07-13, logged not actioned):** `docs/security-ledger.md` + SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web + GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is + correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default + and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE + the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only + path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. + Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local + libvirt bridge, `virbr2`, not routable off-host): `ssh -L 8443:10.10.0.1:443 @10.17.11.248` + -> `https://localhost:8443`. -## Verify-live queue (batch 3 drafts -- confirm on real cloud before trusting) +## OPEN WORK -- VR0 / DC0 (the running testcloud) -- d011-04 amphora_headroom() field parsing (`server show` .flavor; `hypervisor list --long` - vcpu/ram) -- safe default HOLDs on any parse miss, but confirm the OK path live. -- d011-04 OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior. -- d011-05 needs a FOIL tenant (2nd tenant) for P3 isolation -- onboard one or set VR_FOIL_APPCRED - (acme, the former foil, was offboarded). -- Sequence: run d011-04 non-disruptively first (RR+member), then --include-disruptive once - headroom confirmed. +1. **d011 batch-3 live-validation queue** -- `scripts/checks/d011-04-octavia-lb.sh` and + `d011-05-magnum-e2e.sh` are DRAFTED and mock-tested; the live PASS was recorded in the + 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: + d011-04 `amphora_headroom()` field parsing (safe default HOLDs on a parse miss -- confirm the + OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant + (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set + `VR_FOIL_APPCRED`. Sequence: non-disruptive half first, then `--include-disruptive`. +2. **Script backlog item 3 (remainder):** fold `scripts/vault-kv-health.sh` into `cloud-assert.sh` + as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script + -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure + runbook. +3. **Script backlog item 5:** `scripts/tenant-cluster-create.sh` -- generalize the stage-6 + + watch + D-066-evidence wrapper. NOT STARTED (verified absent). + (Items 6/7/8 -- `keystone-policy-drift.sh`, `cloud-snapshot.sh`, `tests/tenant-acceptance/` -- + are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.) +4. **S-class script quality (`docs/script-quality-findings-20260707.md`):** S4/S7/S8a/S8b DONE; + S6 no-action; S8c ruled WON'T-DO. **S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ + offboard) are HELD** by the operator until the DOCFIX-126/127 live-verify lands. +5. **DOCFIX-126/127 live-verify is still OPEN** -- the offboard-E0 and onboard-cred-file fixes have + never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard + confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage. +6. **`test1` orphan user -- approved for deletion, staged, NOT run.** Role-less shell in the + devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): + `openstack user delete 9a62f88e331f41adb5c9e41225ec8698`. +7. **The handoff doc is stale:** `docs/handoff-20260703-open-items.md` still lists R-3 (D-063) as + PROPOSED/OPEN. D-063 is `RESOLVED / CLOSED` in `design-decisions.md` (verified). Correct the + handoff doc. +8. **Upstream reports queued (operator):** magnum `can-upgrade-to` anomaly -- Charmhub magnum + 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from + the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN + -- operator judged it site misconfig; the draft is retained.) +9. **Barbican `host_href`:** the charm renders a literal `https://None:9312`, suppressing the + upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A + charm-level fix would need its own D-NNN. Logged, not actioned. +10. **v1-close checklist is QUEUED** (handoff section 6) -- gated on D-011 item 6, which is gated + on the SEC-003 unseal rehearsal. -## Logged-not-actioned (small; would vanish at compaction) +## Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions) -- offboard v2 `--sweep-magnum-orphans` mode (orphan per-cluster trustee in the magnum domain). -- offboard stage-3 app-cred idempotency re-run guard. -- `host_href=None` barbican observation (exercised fine in stage 6; probably closeable). -- DOCFIX candidate: sweep repo for other `-f json ... 2>&1` stderr-merge hazards (DOCFIX-085 class). -- D-050 (keystone policyd override) and the D-063-adjacent `identity:list_trusts=""` hardening - remain PROPOSED/OPEN, not actioned. +- **D-071** ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning + session. Backups now exist (a 902MB controller backup was taken; `juju create-backup` / + `download-backup` DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). + Controller HA (single-controller, no-HA) is the unresolved half. +- **D-068** off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714. +- **SEC-001 / SEC-004 / SEC-005 / SEC-007:** rotate / flip-to-private / revoke at v1 close. +- **SEC-003:** custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. + This keeps `d011-06-vault-unseal` MANUAL and gates the FULL close of D-011. +- **SEC-006** (burned NetBox API token) -- **DEFERRED by operator ruling 2026-07-13 to deployment + completion.** Until then the token is LIVE and EXPOSED. +- **D-076 is NOT this repo's to work.** D-076..D-099 are a RESERVED source-project band, never + assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live + in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are + running their own tests. +- **quota-vs-capacity: CLOSED** -- operator ACCEPTS as-documented (no quota trim). - - -## Jumphost stream -- ops-update-20260705: WINDOW CLOSED 2026-07-05 (addenda 13-16) +--- -First execution of runbooks/ops-update-procedure.md (DOCFIX-086; addendum 13). -Logged session ops-update-20260705; checkpointed here at each milestone for -cross-stream sync. State as of the Section 2 entry checkpoint: +## Standing lessons and traps (load-bearing; these live nowhere else) -- **DONE Section 1 (pre-flight):** client 3.6.25 / controller 3.6.24 / all 91 agents - uniformly 3.6.24 -> target 3.6.25. Worklist = 17 apps. Quiesce clean. Pre-change BOM - asbuilt/20260705-102951 committed on a TRUE cloud-assert PASS (required the DOCFIX-087 - A7 false-negative fix, addendum 14 -- window was HELD on the operator's Option B - ruling until it landed). -- **DONE Section 2.1:** controller state backup via `juju create-backup -m - admin/controller` (902MB, checksummed, ~/openstack-baseline/, jumphost-only). - NOTE: controller model is admin/controller, NOT /controller. -- **DONE Section 2.2/2.3:** controller upgraded 3.6.24 -> 3.6.25 (blind window ~1 min, - as expected); post-controller cloud-assert PASS. -- **DONE Section 3:** all 91 openstack-model agents + controller machine at 3.6.25, - states clean (63 idle / 28 started), nothing stuck. -- **DONE Section 4 G0:** keystone 778->817; token probe OK; cloud-assert PASS. -- **DONE Section 4 G1 (11 apps, standing group approval, serial+gated):** - placement 125->154, nova-cloud-controller 795->823, neutron-api 650->710, - neutron-api-plugin-ovn 178->215, glance 642->681, glance-simplestreams-sync - 124->152, octavia-diskimage-retrofit 196->232, cinder 733->820, cinder-ceph - 533->568, barbican 209->265, barbican-vault 75->99. All probes OK; - boundary cloud-assert PASS. -- **DONE Section 4 G2:** octavia 441->542; LBs 2/2 ACTIVE/ONLINE; cloud-assert PASS. -- **DONE Section 4 G3:** dashboards 728->750 / 59->122 / 120->168; D-044 override - INTACT; HTTP+login healthy. RCA: VIP HTTPS dead SINCE DEPLOY (haproxy 443 backend - targets vhost-less internal addr; L4 check masks; phase-03 3.3 check fails open) -- - NOT a G3 regression; G3 stands (operator ruling; addendum 15). Upstream bug + - 2 DOCFIX candidates queued post-window. -- **DONE (coordinated):** vault bundle revert 1.16->1.8/stable (BUNDLEFIX-010) + - D-068 AMENDMENT recorded; 1.16 ruled out (certs V0/V1, Raft-only, Ceph, BUSL); - "off EOL 1.8" remains OPEN. Evidence: docs/D-068-vault-1.8-vs-1.16-analysis.md. -- **DONE Section 4 G4:** nova-compute 827->894; hypervisors up; guests - byte-identical pre/post. -- **DONE Sections 5-6:** post cloud-assert PASS; post BOM asbuilt/20260705-194617 - committed; version coherence exact (91 agents @ 3.6.25); BOM diff = 17 expected - rev pairs + 1 explained metadata delta (cinder secrets-storage endpoint); - appendix-B B.1 fully re-baselined; runbook as-executed corrections DOCFIX-088; - D-071 amended (backup exists). WINDOW COMPLETE. -- **DONE follow-on (addendum 17):** magnum 70->96 + vault 372->714 refreshed after - download-and-diff analysis (mass-rebuild campaign upstream; vault stayed reactive/V0, - handlers byte-identical, never sealed). Fleet FULLY CURRENT (zero can-upgrade-to); - appendix-B re-baselined from asbuilt/20260705-201118. -- **DONE (addendum 18):** D-072 ADOPTED+EXECUTED -- dashboard VIP TLS repaired - (cluster binding -> metal-admin; BUNDLEFIX-011 durable; https 200 first time on - this build). DOCFIX-089 phase-03 3.3 fail-closed + appendix-A entry. Upstream bug - DRAFTED: docs/upstream-bug-draft-dashboard-tls.md. -- **POST-SESSION QUEUE (operator):** submit the upstream bug + record LP number in - D-072; rule on D-071 (still PROPOSED); Vault off-EOL path (Roosevelt-scale, 1.16 - ruled out per D-068 amendment; in-channel 714 applied). -- **Window findings logged for close-out (do not action mid-window):** - 1. magnum can-upgrade-to anomaly REPRODUCED (points at magnum-dashboard-122; - evidence ~/openstack-baseline/magnum-can-upgrade-anomaly-20260705.json) AND - Charmhub magnum 2024.1/stable latest release (r101, 2026-07-01) is s390x-ONLY -- - no valid amd64 target; magnum EXCLUDED; report upstream to OpenStack Charmers. - 2. vault NEWLY offers in-channel 372->714 on 1.8/stable -- NOT refreshed (D-068); - logged only. - 3. `juju create-backup`/`download-backup` EXIST on juju 3.6 -- authoring assumption - ("removed in 3.0, expected absent") is WRONG; correct ops-update-procedure 2.1 - and the D-071 risk section at window close (DOCFIX candidate). - 4. juju status --format=line omits workload messages on 3.6 (root cause of the A7 - false negative; fixed by DOCFIX-087). - -## Jumphost stream -- session 2026-07-05 (post-window resume) - -- Orientation + reconcile DONE: machine-derived block re-seeded from today's scan - (DOCFIX next-free 090 -> 091 was stale); shared-section stale-facts correction - appended per the handoff reconcile item. Fences validate 4/0/0. -- FINDING (logged, not actioned): `repo_lint.py` L5 "next-free identifiers" info - line counts ANY textual mention of D-/DOCFIX-/BUNDLEFIX-NNN across all files, so - it is inflated by the deliberate high-numbered decoys in the tests/ledger-scan - fixture ("must NOT inflate") and by next-free pointer lines themselves. ledger-scan - (heading-based, pointer-excluding) is the numbering authority and is correct. - DOCFIX candidate: make L5 reuse ledger-scan's exclusion rules or drop the line. - No number consumed. NOTE (2026-07-06 correction): the first version of THIS bullet - quoted the decoy tokens literally and thereby inflated ledger-scan's own - DOCFIX/BUNDLEFIX next-free (docs/ prose is scan-scanned; only "next-free" lines - are excluded). Tokens de-fanged; standing rule: never write identifier-shaped - tokens above the real high-water mark in docs/ or runbooks/ prose. -- Standing by for batch-3 live validation support (verify-live queue). - -## Jumphost stream -- session 2026-07-06 (operator ruling sweep; addendum 20) - -- RECORDED: D-050 RESOLVED/CLOSED via D-051 (operator ruling); D-073 ADOPTED - (identity:list_trusts tightened to modern keystone default -- IMPLEMENTATION + - TESTING PENDING, nothing live yet; proposed sequencing: before batch-3 d011-05); - D-072 amendment (upstream dashboard-TLS bug WITHDRAWN -- operator judges site - misconfig, draft retained); SEC-001 -> rotate at v1 close; SEC-003/004 deferred. -- PINNED (operator): D-071 ratification and D-068 off-EOL path choice held until - A1/A2 (batch-3 entry) complete. Magnum can-upgrade-to upstream report pinned. -- A1 foil-tenant onboarding APPROVED in principle; tenant-structure model under - discussion first (SCS Domain Manager persona confirmation + operator/tenant - responsibility split). A2 (d011-04 disruptive gating) held behind A1. -- IN DISCUSSION (B2): OpenBao evaluation; D-068 item 2 (vault_url cleartext http); - D-068 item 3 (AppRole TTL audit + proactive probe) scoping. -- DELIVERED (addendum 21): DOCFIX-091 docs/tenant-onboarding-contract.md per A1 - rulings 1-4 (boundary, roles, intake, dangers, hardening register H1-H5 -- - proposals logged, none implemented). A1.4 interpretation pending operator - confirm: "SSH access into their domain" read as canary access-proof (H3). - Verify-live additions: Horizon manager-role GUI identity probe (contract 5.3). -- RULED (2026-07-06 cont.): A1.3 H1-H5 ADOPTED as pre-v1-close tasks; A1.4 canary - interpretation CONFIRMED + adopted as pre-deployment task. Full onboarding - completion package (scripts + internal docs + client-facing drafts + live - end-to-end validation) added to handoff section 6 v1-close checklist. -- RULED (2026-07-06 cont.): clientdocs/ TOP-LEVEL for client-facing drafts; - build INTERLEAVES with batch-3 (foil onboarding = live workflow validation); - A2 disruptive half PRE-APPROVED conditional on live N+1 headroom PASS. -- DELIVERED (addendum 22): H1 scripts/tenant-assert.sh + 7-case harness (green); - clientdocs/ package DOCFIX-092 (README/intake/welcome/self-service drafts). - Remaining in the interleave before foil onboarding: H2 idempotency guard, - H3 canary stage, D-073 list_trusts zip implementation (then gated live apply). -- DELIVERED (operator-away batch, addenda 23-25): H2 re-run guards + H3 canary - stage7 (harness 12/12); D-073 implementation STAGED (39-rule zip, drift check - PASS -- live apply GATED, block in the D-073 amendment); DOCFIX-093 - runbooks/d011-batch3-window-DRAFT.md (the one-window do-doc: D-073 apply + - foil stages 0-4 + tenant-assert + canary + d011-04/05 + close-out); H5 embedded; - docs/D-068-openbao-assessment-DRAFT.md (B2 research input -- no OpenBao charm - exists, MySQL backend removed upstream, V1 goes to Sunbeam not legacy charms; - research rec: deadline-bound risk acceptance). Stderr-merge sweep DONE: - 10-site inventory logged in addendum 25 (fixes = DOCFIX candidate, not actioned). -- REPO-SIDE QUEUE now EMPTY except: H4 policy-drift script (main-chat backlog - item 6 -- coordinate before building) and the client-docs sweep-on-change rule. - EVERYTHING ELSE waits on the operator-present live window (the do-doc). -- DELIVERED (operator-away batch part 2, addendum 26): DOCFIX-094 stderr-separation - fixes (cloud-assert A5 FAIL-OPEN closed + harness proof; onboard resolvers; - offboard blind-sweep guard exit 20; 3 runbook paste blocks; acceptance:43 rides - its future harness). Vault measurements: charm `channel` + ssl-* options - verified; snap has NO 1.13/1.14 track -- max MPL payload in-charm = 1.12.11; - OpenBao memo updated. Orphan-trustee sweep draft NOT started (queued). -- BLOCKER: juju controller macaroon EXPIRED ~17:10 2026-07-06 -- all juju cmds - prompt for password. OPERATOR: `juju login -u jessea123` at next presence. - Blocked reads queued: keystone policy.json PENDING-LIVE-READ trio; barbican - host_href look. (Vault reads above completed BEFORE expiry.) - CLEARED later 2026-07-06: auth verified working; both queued reads DONE - (see addendum 27 closures). - -## Jumphost stream -- session 2026-07-06 (cont.): d011-batch3 WINDOW EXECUTED (addendum 27) - -- **WINDOW COMPLETE, all gates green** (logged d011-batch3, per-command wrap -- - operator-ruled Claude Code lane, now in the convention doc as DOCFIX-097): - D-073 APPLIED live (verify pair + H4-live PASS proof); foil1 onboarded - 10.20.29.0/24 (stages 0-4 first-try, tenant-assert PASS 15/15 after fix, - H2 double-SKIP, H3 canary PASS FIP 10.12.7.177/ubuntu); d011-04 PASS both - halves (A2 conditional honored; headroom OK on all 3 hypervisors); d011-05 - PASS (P1/P2/P3 with foil1); baseline + close-out cloud-assert PASS. -- **In-window checker fixes (each harness-proven then re-run live):** - DOCFIX-095 tenant-assert owner-context app-cred probe (admin-side list is - always-403 by base policy admin_or_owner -- measured, NOT a D-073 effect); - DOCFIX-096 d011-04 amphora_headroom --long + display-name hypervisor keys. -- **D-011: batch-3 CLOSED; D-011 CLOSED-except-item-6** (SEC-003 manual unseal - rehearsal). v1-close checklist (handoff section 6) now QUEUED. -- **Queued-read closures:** keystone trio CONFIRMED (D-064 values explicit in - live base policy; overlay comment tags dischargeable -- DOCFIX candidate); - barbican host_href CLOSED benign-with-corrected-mechanism (charm renders - literal https://None:9312, suppressing the upstream request-derived - fallback; catalog unaffected; charm-level fix = future item needing D-NNN). -- **Operator rulings this session:** devteam pinned as next tenant, onboarding - HELD until a cloud snapshot exists (item-7 cloud-snapshot.sh draft launched); - orphan-sweep audit stays rc 0 report-only; clientdocs L7 receipt keeps all - 14 files; H4 design defaults accepted as delivered. -- **Worktree drafts DELIVERED (uncommitted, integration queued):** orphan-trustee - sweep (14/14), clientdocs L7 (17/17), tenant-acceptance harness (16 checks; - closes the last stderr-inventory site), H4 keystone-policy-drift (10/10 + - first live PASS), clientdocs handover pack (lint clean), cloud-snapshot.sh - (in flight). Integration order: after this window commit; tenant-acceptance - + L7 receipt last (receipt must hash post-integration file states). -- **Open (small):** Horizon manual checks on foil1 (operator, browser, - non-blocking); capi-test-1 CREATE_FAILED leftover (orphan-sweep candidate); - beta-cluster lbtest leftover; repo-lint worktree-exclusion DOCFIX candidate. -- **SCRIPT-QUALITY BATCH LANDED (addendum 33, 2026-07-07):** repo-wide sweep - (110 scripts) -> operator-ruled fix scope shipped as DOCFIX-112..117: - stage2 anti-escalation guard fail-closed (SECURITY -- was fail-open both - ways); tenant-offboard hardening cluster (stderr-as-argv, Phase B counter - CLOSED after 2 days logged, LB-wait fails loud, cred-file auth_url); - validate.sh nonstd-exit HOLD; juju-spaces-check macaroon-vs-absent; pipefail - sweep + vault-kv-health rc-test; guard-hook blocklist extension. Gauntlet - 38/38. Deferred S-class + minor R items in - docs/script-quality-findings-20260707.md (operator has NOT ruled S1-S5/S7/S8). - -## Jumphost stream -- session 2026-07-08: devteam cluster-template incident (addendum 39) - -- **DIAGNOSED + DELIVERED:** devteam "Unable to create cluster template" = - Public/Hidden flag -> `clustertemplate:publish` (admin_api) refusal, BY - DESIGN; read-only diagnosis, no cloud change. DOCFIX-121 (clientdocs - template-creation guidance + Public/Hidden warning) + DOCFIX-122 - (appendix-A symptom entry); L7 receipt re-recorded; ~/devteam-package - re-rendered (kubernetes.md, self-service-guide.md) for tonight's push. -- **OPEN (operator option, unruled): stage5 for devteam** -- `devteam-k8s` - was never delivered (window ran stages 0-4); one gated command - (`bash scripts/tenant-onboard.sh devteam 5`) closes the docs' delivered- - template promise; else devteam self-creates per new docs. -- **OPEN (operator, small):** role-less GUI test user in the devteam - domain (grant member or delete); Role-dropdown question on the - dashboard-override test (why load-balancer_admin was submitted) -- - evidence folded into the D-076 draft (worktree, uncommitted). -- **TestCluster NoValidHost + decommission audit (addendum 40):** devteam - 1+3 cluster stalled on RAM capacity (NOT quota); operator deleted it, - devteam re-created 1+1 (TestCluster2). QUEUED RULINGS from the audit: - (a) beta-cluster teardown (~17GB reclaim) -- cross-stream, main-chat - owns the resize-acceptance role; (b) lbtest LB leftover reclaim via - kubectl (beta kubeconfig, main-chat asset); (c) foil1 offboard - (RECOMMENDED -- dormant shell); (d) quota-vs-capacity expectation - (trim testcloud quotas or document in clientdocs) -- unruled. - -## Jumphost stream -- session 2026-07-08 (cont.): Jenkins+K8s guide overnight (addendum 41) - -- **DELIVERED (DOCFIX-123):** new clientdocs/jenkins-kubernetes-guide.md -- - the single end-to-end Jenkins-to-Kubernetes workflow HUB (Pattern A deploy- - into-cluster + Pattern B build-agents; self-create cluster path; kubeconfig- - as-Jenkins-credential; NoValidHost sizing; flannel/Public-Hidden/keypair - traps; troubleshooting; glossary; cross-links). Flannel hole also closed in - tenant-skill kubernetes.md. Guide added to LEAK_FILES (harness change), - README updated, L7 receipt 30 files, gauntlet 38/38. Instantiated to - ~/devteam-package (devteam values). Operator pushes tonight. -- **OPERATOR RULINGS (on return, 2026-07-08 PM) + status:** - 1. devteam cluster: DELETE + DELIVER TEMPLATE ruled. DONE (addendum 42): - TestCluster2 confirmed gone + devteam-k8s calico template delivered. - 2b. Handover package RE-consolidation (operator re-ruling, addenda 46-47): - glossary folded into handover-pack + glossary.md removed + account-table - hard de-dup (DOCFIX-128); tenant AI skill aligned to the consolidated - package -- 3 missing failure signatures added, capacity caveat, Jenkins- - guide integration (DOCFIX-129). Top-level docs 9->8. Package re-instantiated - (glossary appendix in handover-pack, skill refreshed). DONE. - 2. Consolidation review M1-M7: ALL APPROVED. DONE (addendum 43, DOCFIX-124): - M5 error-holes in self-service-guide; M3 entry point + goal-map in welcome; - M1/M2 single-source-of-truth + pointers (softer than hard-delete -- flagged, - protects the AI skill + P2 test); M6 Jenkinsfile path/stage fixes; M4 new - glossary.md. Package re-instantiation for the changed set = next step. - FLAGGED: package handover-pack.md has unfilled {{CUSTODIAN_1/2}} (operator - value; pre-existing, devteam placeholder-custodian test). - 3. Gated script fixes: BOTH DONE (addendum 45). DOCFIX-126 offboard E0 - always-403 no longer FATALs before Phase F (harness case added, 21/21); - DOCFIX-127 onboard cred files now carry dashboard_url (env-overridable, - as-built https://10.12.4.58/horizon). LIVE VERIFY STILL GATED: next real - onboard/offboard confirms end-to-end (onboard cred-write stages 1/2 have - no unit coverage -- pre-existing; logged future harness item). - 4. DOCFIX list CLOSED (operator); handover-pack.md fine as-is (already carries - {{DASHBOARD_URL}}/{{AUTH_URL}}). - 5. HANDOVER READY (operator confirmed 2026-07-08). Two flags ACCEPTED for the - devteam FIRST INTERNAL TEST: (a) package handover-pack.md keeps literal - {{CUSTODIAN_1/2}} -- fine for the test, fill with real names for a real - client handover; (b) existing ~/tenant-devteam cred files predate the - DOCFIX-127 dashboard_url field (written 2026-07-07) -- fine for today, the - dashboard URL is in handover-pack section 3 which the team receives. Repo - fully pushed (HEAD db284fc). REMAINING = operator-manual delivery only: - copy ~/devteam-package + ~/tenant-devteam to the two custodians in person - (guard blocks agent access to cred material). -- **Offboard E0 defect (for the fix):** admin-side - identity:list_application_credentials is always-403 (base admin_or_owner; - measured), so offboard v2 --apply counts 3 false failures and FATALs at exit - 23 BEFORE Phase F (cred-dir retirement). foil1 offboard hit this live; cred - dir retired by hand. Fix: owner-context-or-skip E0 inventory, do not count - known-403 as failure; + harness case. - -## Jumphost stream -- session 2026-07-08 (cont.): devteam self-service cluster live (post-handover) - -- **devteam self-created their first cluster (read-only audit, no cloud change).** - `TestCluster` (uuid 92b92eed-afa3-4aaf-ab33-13d3ddeea869) CREATE_COMPLETE + - HEALTHY (cluster / infrastructure / controlplane / nodegroup all Ready), created - 2026-07-08 17:18 UTC in devteam-prod. 1 master + 1 worker, keypair devteam-key, - api https://10.12.7.130:6443. calico + small-first + correct keypair + no - Public/Hidden flag -- the DOCFIX-121 dashboard symptom resolved IN PRACTICE on - their first real self-service create. (Note: this fresh TestCluster reuses the - name of the earlier deleted TestCluster/TestCluster2, addenda 40/42 -- cosmetic.) -- **Built via HORIZON, not the CLI.** They created their own template - `TestTemplate` (calico, public/hidden False, keypair baked in = devteam-key) in - the dashboard and launched TestCluster from it, rather than the delivered CLI - template devteam-k8s (keypair_id None; measured, correct-by-design, still unused). - Both are valid Magnum objects. -- **What the Horizon path changes (LOGGED, not actioned; DOCFIX candidate, no - number consumed):** - 1. Nothing about the running cluster -- same Magnum API object either way. - 2. Explains the keypair-on-template confusion this session: Horizon's Create - Cluster Template dialog exposes an OPTIONAL Keypair field, so their - GUI-built TestTemplate shows a key while the delivered devteam-k8s (CLI - stage5 omits it by design) shows none. Comparing the two in-GUI drove the - "no keypair -> must be required" read. It is not; the key is supplied at - cluster-create time and their live cluster already carries it. - 3. Doc-gap: jenkins-kubernetes-guide.md is CLI-centric. Their next guide step - (Section 3, kubeconfig fetch via `openstack coe cluster config`) has NO - Horizon path -- the supported retrieval is the CLI. (Earlier draft of this - bullet claimed magnum-ui "offers a cluster-config download" -- UNVERIFIED, - retracted; do not assert it.) The Jenkins integration requires the - CLI/app-cred path regardless. -- **ADDRESSED: DOCFIX-130 (addendum 48, COMMITTED f0f11ae; ~/devteam-package - guide RE-INSTANTIATED 2026-07-08, 34937 bytes, tokens filled from live-measured - AUTH_URL/REGION).** Grew from 2 gaps to a full guide pass on operator - instruction ("fold and add 1-7... make GUI/CLI paths better defined"). - clientdocs/jenkins-kubernetes-guide.md, additive only, no token touched: - - STRUCTURE: new Section 0 dashboard-vs-CLI selector + **On the command line** / - **In the dashboard** markers; dashboard equivalents for create (2.3), resize - (2.5 new), teardown (8). Rule: lifecycle = either path; everything Kubernetes - (kubeconfig onward) = CLI only. - - GAPS 1-7 closed: registry/imagePullSecret (5.0; VERIFIED no registry service - in catalog), ingress-nginx how-to + no-DNS (5.4 new), cluster resize - (2.5 new; VERIFIED syntax), persistent storage self-verify (5.5 new), cert - expiry read one-liner (3), tooling install + kubectl-skew (1); + ImagePullBackOff - and PVC-Pending troubleshooting rows (9). - - Leak harness CAUGHT two self-inflicted client-doc leaks ("Horizon", "magnum") - on first pass -- reworded, now PASS 8/0; gauntlet 38/38. - - ~/devteam-package NOT re-instantiated (refresh at next package sweep). - See the changelog DOCFIX-130 entry. Numbers: DOCFIX next-free now 131. -- **STORAGE OPERATOR FOLLOW-UP (logged, not actioned):** delivered cluster - template has volume_driver unset + no docker volume (MEASURED). Whether the - built cluster gets a default StorageClass from the cinder-CSI default is - UNVERIFIED (verifying = kubectl into devteam's live cluster; deferred for tenant - isolation). If none, persistent volumes are unavailable out of the box -- - candidate template improvement (would need a D-NNN). 5.5 is written self-verifying - so the doc is correct either way. -- **CLEANUP QUEUE RECONCILED against live (2026-07-08 read-only audit) -- LIVE IS - AHEAD OF LEDGER, several "open" items already resolved:** - - foil1 offboard (was RECOMMENDED, addendum-40 c): DONE -- no foil project/user, - no orphan trustee. (Closes the dormant-shell item; also means the gated - DOCFIX-126/127 offboard live-verify was NOT exercised by us -- still unverified - end-to-end, next real offboard confirms.) - - capi-test-1 CREATE_FAILED delete (REMAINING-OPEN item 3, was operator-executes): - DONE -- no cluster record, trustee reaped. - - beta-cluster teardown + lbtest LB (addendum-40 a/b, cross-stream/main-chat): - DONE -- no beta cluster, no stray LB. (Main-chat to log its side.) - - Audit state: only devteam TestCluster (CREATE_COMPLETE) + its kubeapi LB + - its protected trustee remain; orphan sweep ORPHAN=0 PROTECTED=2 UNPROVEN=0; - zero unattached floating IPs. - - STILL OPEN from the addendum-40 audit: only (d) quota-vs-capacity expectation - -- trim testcloud quotas or document the capacity ceiling in clientdocs - (unruled, operator). Everything else in that audit is closed. - -## Jumphost stream -- session 2026-07-08 (cont.): S-class script-quality batch (operator-greenlit) - -Operator greenlit the recommended sequence: (1) Phase A stderr fix, (2) S7+S4, -(3) S8; HOLD S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/offboard) until -the DOCFIX-126/127 live-verify lands. Source: docs/script-quality-findings-20260707.md. - -- **DONE -- DOCFIX-131 (addendum 49): tenant-offboard.sh Phase A stdout-only.** - New `tinv()` (tenant-scoped `inv()` twin) for the 3 Phase A `coe cluster list` - captures; wait loop retries on list-error, final check fails closed. Latent bug: - merged stderr could false-trip exit 21 on an already-clean tenant. Harness un- - excludes coe from warnstderr + new coe-stderr-not-argv assertion (22/22). - Gauntlet 38/38, lint 0 fail, L7 re-recorded. Committed + pushed. -- **DONE -- S7 (DOCFIX-132) + S4 (DOCFIX-133), addendum 50.** S7: removed dead - `CF` (d011-04) + subsumed `vr_is_ipv4` guard (d011-03). S4: three inline trust - filters -> one fixture-tested scripts/trust_filter.py (+tests/trust_filter/ 9/9); - SDIR added to offboard + acceptance. Gauntlet 39 ALL GREEN. Committed + pushed. -- **S8 RE-SCOPED (bigger than its "S" estimate; operator decision needed before I - wire it):** the KEYSTONE_VIP literal (10.12.4.50) lives in 6 files, and 3 of - them (tenant-onboard/offboard/acceptance) do NOT source lib-net.sh -- so - centralizing means adding `. lib-net.sh` sourcing into client-critical scripts I - just modified (DOCFIX-131 + S4), which stacks risk. Breakdown: - - S8a (FIP pool FIP_START/END -> lib-net.sh): DONE (DOCFIX-134). FIP_POOL_START/END - added to lib-net.sh; phase-04 create/verify reference them. Gauntlet 39/39. - - S8b (KEYSTONE_VIP -> lib-net.sh): DEFERRED -- needs sourcing plumbing into 6 - scripts incl. the 3 tenant scripts (onboard/offboard/acceptance) that do NOT - source lib-net. Operator decision needed before wiring client-critical scripts. - - S8c (provider-bundle-check.py restates plane CIDRs): DEFERRED -- design change - (python can't source bash -- needs a generated/parsed form). -- **DONE -- DOCFIX-135 (addendum 51): tenant AI skill registry gap.** devteam asked - (via the assistant) if there's a registry setup step; the DOCFIX-130 registry - content was not mirrored into the skill (grep found zero). Added "Images: bring - your own registry" to references/kubernetes.md + ImagePullBackOff signature to - references/troubleshooting.md; skill harness 8/8, gauntlet 39/39; package skill - files re-instantiated. Committed + pushed. (Skill-lag class; watch for it on - future guide changes.) -- **DONE -- DOCFIX-136 (addendum 52): tenant AI skill FULL SWEEP vs DOCFIX-130.** - Operator asked for a full sweep after the registry gap. Grep-audit found 6 more - guide topics absent; all added concise (pointer to guide for detail): resize - command, kubeconfig cert-expiry (+openssl read), dashboard-vs-CLI split, no-DNS, - NEW Persistent storage section, ServiceAccount pointer; + PVC-Pending and - cert-expired troubleshooting signatures. Skill now covers the full DOCFIX-130 set - (ingress stays recommendation+pointer by design). Harness 8/8, gauntlet 39/39, - package re-instantiated. Committed + pushed. STANDING LESSON: guide changes must - sweep the skill in the same change -- skill-lag caught twice (DOCFIX-135/136). -- **DONE -- DOCFIX-137 (addendum 53): the standing lesson is now ENFORCEABLE.** - repo-lint L8 guide<->skill coupling guard: clientdocs/guide-skill-coupling.txt - pins the guide's sha256; any guide change FAILS lint (naming the skill files to - review) until re-recorded with `bash scripts/repo-lint.sh - --record-guide-skill-coupling`. Harness +5 (28/28), gauntlet 39/39. So the next - guide edit CANNOT land without a conscious skill review. Skill-lag closed at the - source. -- **S-CLASS BATCH STATUS (operator ruled 2026-07-08):** DONE = S7, S4, S8a - (DOCFIX-132/133/134). **S8b APPROVED -- DO IT (next actionable):** centralize - KEYSTONE_VIP into lib-net.sh. **DONE (DOCFIX-138):** KEYSTONE_VIP_DEFAULT in - lib-net.sh; all 6 sites reference it (3 forms: KEYSTONE_VIP, KEYSTONE_HOSTPORT, - acceptance inline); lib-net source added where absent. Gauntlet 39/39, lint 0 fail. - **S8 FULLY DONE (S8a+S8b).** **S8c RULED WON'T-DO** (operator: marginal value for as-built - constants vs bash<->python emit/parse machinery). S1/S2/S3/S5 (lib-validate) - STILL HELD -- operator keeps DOCFIX-126/127 live-verify open for an upcoming - opportunity (devteam actively using resources; not now). S6 no-action. -- **quota-vs-capacity (addendum-40 d): CLOSED -- operator ACCEPTS as-documented** - (DOCFIX-130 sec 2.3/5.5 + skill cover NoValidHost/capacity thoroughly; no quota trim). -- **StorageClass follow-up (DOCFIX-130 sec 5.5): RESOLVED 2026-07-08.** Operator - authorized full read of devteam's cluster (fully internal domain); scoped via - ~/devteam-cluster-openrc (snap needs $HOME for --dir; kubeconfig fetched under - $HOME, used, DELETED). devteam TestCluster HAS a default StorageClass: - `default (default)` provisioner cinder.csi.openstack.org, Retain, - WaitForFirstConsumer, expansion=true. So the delivered template (volume_driver - unset) DOES yield PV support via the cinder-CSI magnum default -- persistent - storage works out of the box; no template change needed. sec 5.5 doc is correct - and self-verifying; OPTIONAL enhancement: state the expected `default` cinder - class (would trigger L8 skill sweep). Cleanup: kubeconfig removed, no residue. -- **D-076 assessment (operator #6: "tested working w/ devteam domain-admin? check - for additional work"):** NOT fully working -- optimistic recollection. Revision 1 - FAILED (full-page 403s + forced logout), reverted. Revision 2 PARTIALLY passed: - identity panels render for the manager persona + session intact, BUT the - create-user GUI flow submits a NON-managed role (load-balancer_admin) whose grant - 403s, stranding a role-less user (flow not atomic). ADDITIONAL WORK before it can - advance: (1) root-cause why the Role dropdown submitted load-balancer_admin - (form default-role preselection vs mis-select); (2) add a per-view SEMANTIC harness - test asserting create-user grants exactly `member` (invisible to the existing 50 - policy pass/fail assertions); (3) clean up the orphaned test user - 9a62f88e331f41adb5c9e41225ec8698 (name `test1`, CONFIRMED still present in devteam, - ZERO roles -- delete or grant member); (4) operator RULING: adopt rev2 (a) vs - close GUI/CLI-only (b) + accept/reject the charmhelpers blacklist-hole reliance; - (5) another operator-present gated window to re-test after rework. NOTE: live - override state UNVERIFIED -- juju macaroon expired (needs `juju login`); confirm - use-policyd-override is reverted to baseline. Draft in worktree agent-a73ec798 - (uncommitted). Clients unaffected: CLI identity path is the documented behavior. -- **D-076 PINNED (operator 2026-07-08):** do NOT work D-076 while devteam are - running their own tests in the env. Deferred until devteam testing completes; - no further D-076 investigation/window until then. Draft stays in worktree. -- **test1 orphan SCHEDULED FOR REMOVAL (operator 2026-07-08):** approved to delete; - staged, NOT run now. Exact gated command (re-verify role-less at execution): - `openstack user delete 9a62f88e331f41adb5c9e41225ec8698` (name test1, devteam - domain, 0 roles measured 2026-07-08). Run at the next convenient window; it is a - role-less shell with no resources, so removal does not affect devteam's tests. - -## REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window) - -1. **Devteam delivery -- last two steps are OPERATOR-MANUAL (everything - else DONE, addenda 34-35):** intake collected 2026-07-07 (internal - test; placeholder custodians operator-accepted; filed jumphost-local - ~/devteam-intake-record.md, NEVER commit); quota applied+verified - (10/20/51200, runbook standard); pack instantiated to - ~/devteam-package (17 files, tokens verified zero, banners stripped; - templates in clientdocs/ untouched -- L7 receipt intact). REMAINING - (operator): (a) copy the CA into the package: `cp - ~/vault-init/vault-ca-root.pem ~/devteam-package/omega-cloud-ca.pem` - (guard correctly blocks agent access to ~/vault-init); (b) deliver - ~/devteam-package + the ~/tenant-devteam credential files in-person - to the two named custodians. -2. **Horizon contract checks: CLOSED for devteam (2026-07-07 full - three-account walkthrough, addendum 38).** 5.1 VERIFIED (login + - Identity-only view + expected popups); 5.3 verdict: GUI identity probe - FAILS as the contract anticipated -- CLOSED via the CLI-path clause - (DOCFIX-120 contract amendment); D-075 TLS-leg redirect CONFIRMED - (operator browser). Isolation, keypair asymmetry, per-user app-cred - visibility, and OBJECT STORAGE (container create/delete -- first live - proof, closes the acceptance-checklist caveat) all verified in-GUI. - foil1's own 5.1/5.3 rows: surface now proven on devteam; operator may - run or waive foil1's pass (non-blocking either way). -2c. **Dashboard policy override (GUI identity for manager persona): - first live apply FAILED acceptance + REVERTED 2026-07-07; rework IN - FLIGHT (background agent, worktree).** Zip mechanics + PO: activation - worked; panels rendered but page-level checks refused (Domains/ - Projects/Users/Groups "not authorized"); Roles panel popped two errors - then FORCED LOGOUT. Reverted to baseline in one config cycle - (operator-confirmed). Decision draft stays PROPOSED (uncommitted, - agent worktree; next-free D claimed at commit). Next: agent rework - (per-view rule/target mapping, logout root cause, scope-down to - Users+Projects, offline per-view semantic tests), then a second gated - apply in a later window. Clients are unaffected: CLI identity path is - the documented behavior. -2b. **D-075 RULED (option c) + EXECUTED 2026-07-07** (was: root-autoindex - finding): redirect / -> /horizon + Options -Indexes via - conf-enabled/99-omega-root.conf on the dashboard unit. First attempt - (RedirectMatch) caught leaking the backend port in verify and replaced - with per-dir mod_rewrite. Delivered as D-075 entry + DOCFIX-118 - (phase-03 PER-REBUILD block + appendix-A symptom entry). PER-REBUILD - like the D-044 override. -3. **capi-test-1 CREATE_FAILED record delete: RULED YES (2026-07-07), - execution handed to operator** -- the Claude Code permission classifier - (correctly) refuses agent-side deletes of resources it did not create; - exact command staged in-session (uuid fae7ec70-712c-4e58-88dd-da0d7b8c89b2, - measured this window). After it runs: next `tenant-offboard.sh - --sweep-magnum-orphans --audit` reaps its trustee. -4. **S-class consolidation + minor R items:** proposals in - docs/script-quality-findings-20260707.md -- present options to operator - before building (S1-S5/S7/S8 unruled; S6 ruled no-action). -5. **Pinned decisions (do NOT re-ask):** D-071 ratification (pre-DC-DC HA/ - backup session); D-068 off-EOL path (Roosevelt-scale); SEC-001/003/004 - at v1 close; d011-06 stays MANUAL until SEC-003 rehearsal (gates D-011 - FULL close; v1-close checklist QUEUED per handoff section 6). -6. **Cross-stream:** main-chat to reconcile its ledger section against - addenda 27-33 (its backlog items 6/7/8 delivered here; its Verify-live - queue fully cleared; D-050/D-073/D-074 all resolved). - -- **Horizon three-account walkthrough + docs alignment (2026-07-07 cont., - addendum 38):** operator live-tested all three devteam accounts in the - GUI; every observation contract-correct (details in item 2 above). - DOCFIX-119: clientdocs aligned to observed GUI reality (5 files by - background docs agent -- first-login expectations, CLI user-creation - commands, popup signatures, object-storage row de-hedged + acceptance - script aligned; harnesses 8/8 + 47/47; L7 receipt re-recorded) + README - instantiation guidance (DASHBOARD_URL includes webroot). DOCFIX-120: - contract section 5 amended with live verdicts (5.1 verified; 5.3 - closed-by-CLI-clause; stage-7 canary marked IMPLEMENTED). Package - RE-INSTANTIATED to ~/devteam-package from updated templates (incl. - /horizon URL); OPERATOR: re-copy to local + deliver. Dashboard - override attempt/revert logged in item 2c. -- **devteam intake + quota + pack instantiation (2026-07-07 cont., - addendum 35):** intake walkthrough with operator (7 sections; CIDR + - short name pre-settled by rulings); quota set devteam-prod 10/20/51200 - (values equal nova class defaults -- set explicitly per runbook "documents - the record"); first-ever clientdocs pack instantiation: 17 files to - ~/devteam-package via scratchpad script (9 tokens: endpoints MEASURED - in-session -- AUTH_URL https://10.12.4.50:5000/v3, RegionOne; dashboard - https://10.12.4.58 carried from D-072 as-executed verification). - Guard/classifier blocks honored: no ~/vault-init access (CA copy handed - to operator), no curl -k probe. LOGGED-NOT-ACTIONED (DOCFIX candidate): - tenant-skill/references/day2-operations.md line ~7 explains the {{THIS}} - notation in DELIVERED prose, so the literal token survives instantiation; - reword at next clientdocs sweep (L7 receipt re-record required). -- **devteam ONBOARDED (2026-07-07 window ops-devteam-onboard, addendum 34):** - stages 0-4 + tenant-assert (15/15 PASS) + stage7 canary (SSH via FIP - 10.12.7.43, teardown clean) ALL FIRST-TRY on TENANT_CIDR=10.100.0.0/24 - (D-074 ruled fallback -- first live use of the fallback path and first - client-facing workflow execution; ZERO as-executed corrections needed). - Stage-2 anti-escalation guard (DOCFIX-112 batch) first live client run: - admin grant denied as designed. Baseline reused per ruling - (asbuilt/20260706-224851, not stale). Open cloud-assert PASS (A5-A7 held - on first run -- one-shot shell lacked admin scope, known jumphost trait, - re-run sourced) and close cloud-assert PASS. Handover creds staged in - ~/tenant-devteam (0600, jumphost-only). Pack instantiation HELD on intake - answers (item 1 above). capi-test-1 delete handed to operator (item 3). -- **CLIENT PACKAGE RELEASED (addendum 31, 2026-07-07):** operator-requested - polish sweep integrated (DOCFIX-110) + banner ruling applied (DRAFT - removed; the three per-client templates keep TEMPLATE; README says - RELEASED-for-handover). Package is handover-ready pending only per-client - instantiation. In flight: repo-wide script quality sweep (report-only - agent; findings to operator triage next). -- **CLIENT PACKAGE COMPLETE (addendum 30):** starter kit (DOCFIX-108, 6 - deliverables incl. tenancy-audit from the skill delivery; pipeline-smoke + - leftover-check deduped away) + tenant AI skill omega-cloud-tenant - (DOCFIX-109, leakage-gated) integrated; L7 coverage recursive; receipt 29 - files; gauntlet 37/37. REMAINING before handover: operator-requested - polish pass over the final clientdocs set (in flight), then devteam - onboarding next session instantiates the pack. -- **D-074 EXECUTED (addendum 29):** operator ADOPTED the main-chat CIDR - correction PLAN same-day (overlap-allowed tenant CIDRs; client-choice + - 10.100.0.0/24 fallback; capi-mgmt stays reserved; L5 print removed). - Phase 0 gates honored: allow_overlapping_ips=True measured; overlap+Magnum - proof via foil1 (EXACT 10.20.28.0/24 overlap with beta; cluster ~14.5 min; - 2 Ready nodes via API-LB while beta served the same CIDR); teardown clean; - cloud-assert PASS. Guard rewritten (DOCFIX-106, net_overlap.py, harness - 12->17), docs aligned (contract/runbook/intake), D-016 amended, repo_lint - L5 next-free removed (DOCFIX-107). FINDING-doc BUNDLEFIX token de-fanged - (counter was inflated to 052; restored 012). Phase 3 (devteam onboarding, - fallback CIDR) remains NEXT SESSION. Corrections fed back to main-chat via - addendum 29: northwind example fictional; runbook filename wrong in PLAN. -- **INTEGRATED (addendum 28):** all six worktree drafts -> main as - DOCFIX-098 (H4 drift detector), 099/100 (offboard orphan sweep + Phase E0 - guard), 101 (clientdocs handover pack), 102 (cloud-snapshot.sh -- unblocks - the devteam snapshot gate), 103 (tenant-acceptance harness + the last - stderr-inventory fix), 104 (lint L7 sweep-on-change ENFORCED; receipt live, - 17 files). Gauntlet 35/35 on the integrated tree. Main-chat backlog items - 6/7/8 are now DELIVERED (by this stream, operator-assigned) -- main chat to - reconcile its section. -- **devteam gate SATISFIED (2026-07-06):** cloud-snapshot first live runs both - green -- asbuilt/20260706-224851 committed (12 redactions verified; scrubber - caught a real embedded CA private key) + 1.1GB controller backup - jumphost-local (~/openstack-baseline/, NEVER commit). Operator ruled: devteam - onboarding NOT this session -- next session runs stages 0-4 + tenant-assert - + canary on 10.20.30.0/24 (recommended, unruled), then instantiates the - handover pack with dev-team intake answers (7 questions in the DOCFIX-101 - entry). Also still open: foil1 Horizon manual checks (contract 5.1/5.3). - -## Jumphost stream -- session 2026-07-09: new repo `openstack-caracal-dc-dc` first load (repo-rename sweep) - -- **Environment note:** this session's shell is Windows (win32), not the Linux - jumphost (`vopenstack-jesse`) CLAUDE.md describes -- flagged to operator. - `run-tests-all.sh` gauntlet on this box shows 23/39 pre-existing failures - (missing `jq`, tty/exit-code quirks) unrelated to any change made this - session; `repo-lint` (the only suite touched) is ALL PASS standalone and in - the gauntlet. Confirm actual jumphost behavior at next Linux session. -- **DONE -- DOCFIX-139 + DOCFIX-140 (changelog - `docs/changelog-20260709-dc-dc-repo-rename-sweep.md`):** repo-rename sweep - (stale `openstack-caracal-ipv4` path/URL references repaired to - `openstack-caracal-dc-dc` across the skill, 4 runbook RUN blocks, and the - repo_lint.py docstring; clientdocs L7 receipt re-recorded after confirming no - client-facing drift) + a real repo_lint.py bug fix (L1 Windows - path-separator: the `docs/design-decisions.md` legacy-carve-out comparison - used `str(p.relative_to(R))`, which never matches on Windows, so the - documented WARN silently became a FAIL -- fixed to `.as_posix()` at all 3 - affected sites; harness gained T3b covering the carve-out path directly, - 29/29 green). NOT actioned (flagged, larger scope): README.md and - environment.md's body content still describe VR0 DC0/v1 single-DC scope - wholesale -- awaiting operator direction on whether/when to rewrite for VR1 - DC-DC. Uncommitted -- operator has not yet asked to commit. -- **DONE -- DOCFIX-141 (changelog `docs/changelog-20260709-repo-agnostic-sweep.md`):** - operator asked to scope the "make scripts repo-agnostic" idea as its own - sweep + a lint rule, following directly from the DOCFIX-139/140 experience. - New repo-lint check L9 (self-reference guard) FAILS any script/RUN-block/ - skill file that hardcodes the current repo's own directory name; added with - a 5-case harness (34/34 green). Along the way, narrowed `all_text()`'s - `.claude` exclusion from "skip all of `.claude/`" to "skip only - `.claude/worktrees/`" -- the blanket exclusion meant checked-in - `.claude/skills/` and `.claude/hooks/` content was invisible to lint - entirely (why the SKILL.md hardcode from DOCFIX-139 wasn't caught). - Established a `$REPO` session-variable convention (documented in - `runbooks/README.md` Conventions) and converted the 8 remaining hardcoded - clone-path references (SKILL.md + 4 runbooks) to require it explicitly - (fail loud if unset, no inferred fallback, per hard rule 2). environment.md's - repo URL is KEPT (documented anchor fact, same as its other facts) but - tagged with the new `repo-lint: allow-selfref` opt-out. LOGGED, NOT ACTIONED: - L3/L4/L6/L9 only recognize triple-backtick fences for "in code" detection; - `d011-batch3-window-DRAFT.md` (indented pseudo-code) and - `tenant-onboarding-v2-DRAFT.md` (BEGIN/END block comments) use different - conventions and are invisible to these checks -- a pre-existing gap, not - introduced here, DOCFIX candidate (teach the checks the other conventions, - or standardize runbooks onto fences). Uncommitted -- operator has not yet - asked to commit. -- **DONE -- workflow doc + gap register (2026-07-09):** `docs/dc-dc-deployment-workflow.md` - added (Stage 0-7 followable checklist + tooling gap register audited - directly against the repo) + a companion visual Artifact - (https://claude.ai/code/artifact/e752f6dd-23ff-4c59-9ce5-1dedb917e5ea). - COMMITTED + PUSHED (374aef5). -- **DONE -- DOCFIX-142 (changelog `docs/changelog-20260709-opentofu-scaffold.md`):** - first OpenTofu content in the repo (gap register item 2). `opentofu/` - network + storage-pool scaffold: `dc-planes` (six per-DC planes), - `mesh-link` (D-100 dark-fiber triangle), `dc-storage-pool` modules; DC1 + - Office1 wired, DC2 held back pending its CIDR assignment (D-101). Provider - schema (`dmacvicar/libvirt` v0.9.8) verified against its actual current docs - this session, not memory -- differs materially from older common examples - (network isolation is `forward.mode`, nested, not top-level `mode`; domain - resource restructured to ~40 args mirroring raw libvirt XML). Node-VM/volume - resources, OPNsense, DC2 planes, and `tc netem` application DELIBERATELY - DEFERRED -- schema too unfamiliar to author safely without a real - `tofu providers schema` pass; see `opentofu/README.md`. Added - `scripts/opentofu-validate.sh` + `tests/opentofu-validate/` (2/2 green -- - only the guard-clause paths testable without a `tofu` binary, which is not - available anywhere this repo has been worked in this session; the actual - fmt/init/validate path is UNTESTED, logged not hidden). Extended - `repo_lint.py`'s L1 file-extension set to include `.tf` (previously - uncovered since no Terraform/OpenTofu files existed). repo-lint 34/34 - harness green, 0 fail/1 documented warn. Uncommitted -- operator has not yet - asked to commit this piece. -- **NEXT ACTIONABLE STEP (logged):** on a machine with the `tofu` binary + - registry network access, run `scripts/opentofu-validate.sh`, then - `tofu providers schema -json` against the initialized provider, and read the - real `libvirt_domain`/`libvirt_volume` schema directly before authoring the - node-VM module. -- **DONE -- OPNsense gap detail (2026-07-09):** gap register item 4 split into - its 4 separable sub-gaps (network substrate done; VM/image blocked on the - domain module; OPNsense's own routing config not started, no mechanism - chosen; `tc netem` a separate OS-level thing blocked on an unruled D-100 - parameter). Mirrored into `opentofu/README.md` + the visual tracker. - COMMITTED + PUSHED (0bb6cff). -- **DONE -- DOCFIX-143 (changelog `docs/changelog-20260709-repo-lint-crlf-fix.md`):** - fixed the CRLF `write_text` bug flagged during the DOCFIX-141 commit -- - `--record-clientdocs-sweep`/`--record-guide-skill-coupling` now - `write_bytes()` with an explicit ascii-encode instead of relying on - `.gitattributes` eol=lf to clean up CRLF after the fact. Verified 0 CR bytes - on a live re-record; harness 34/34 unchanged; repo-lint 0 fail/1 warn. - COMMITTED + PUSHED (f93dd0d). -- **DONE -- DOCFIX-144 (changelog `docs/changelog-20260709-opentofu-node-vm.md`):** - moved into the domain module as requested. Built `opentofu/modules/node-vm` - (blank-disk PXE-boot MAAS-managed node VM, D-103's pattern for Stage 3) -- - this time from the provider's own real example `.tf` files, not - doc-summarization alone, after last session deferred `libvirt_domain` for - exactly that confidence gap. **Found and fixed a real bug in the - already-committed DOCFIX-142 delivery in the process:** `dc-planes`/ - `mesh-link`/`dc-storage-pool` used classic HCL block syntax - (`domain { ... }`) where the provider's current schema actually needs - attribute-style objects (`domain = { ... }`) -- confirmed once real - provider examples surfaced the convention; doc-summarization alone had NOT - caught this. All three corrected; `opentofu/README.md`'s schema notes now - state the attribute-object convention as a provider-wide assumption. - UNVERIFIED, flagged: the per-device `boot`-order attribute's exact shape - (an inference by pattern). Still deferred: the cloud-init/base-image VM - pattern (Office1 services + OPNsense), DC2 planes, netem. - COMMITTED + PUSHED (3324387). -- **DONE -- DOCFIX-145 (changelog `docs/changelog-20260709-opentofu-cloudinit-vm.md`):** - built `opentofu/modules/base-image` + `modules/cloudinit-vm` (the OTHER VM - pattern, for Office1's own service VMs -- MAAS-region, NetBox, GitBucket). - Same high-confidence source as DOCFIX-144 (`examples/alpine_cloudinit.tf`, - fetched as real code). Threaded the base image's `.path` output across the - module boundary as a real attribute reference rather than reconstructing a - volume path from pool+name strings -- a cleaner design than the earlier - draft, avoiding an unconfirmed-convention guess entirely. - `user_data`/`meta_data`/`network_config` are required inputs with no - default (Office1 VM configs aren't designed yet; a plausible DHCP default - would also silently fail since these planes carry no libvirt DHCP). - **OPNsense applicability explicitly NOT confirmed** -- FreeBSD-based, - cloud-init/NoCloud support not the same guarantee as Linux images, flagged - prominently rather than assumed. Neither module instantiated (no image - source chosen for any service VM). Workflow doc + visual tracker updated. - Uncommitted -- operator has not yet asked to commit this piece. -- **DONE -- OPNsense deployment research (changelog - `docs/changelog-20260709-opnsense-research.md`, no DOCFIX -- research - delivery, matches the design-doc precedent):** operator asked to check - OPNsense's actual documentation. Confirmed cloud-init is genuinely - unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing - the DOCFIX-145 open question. Found the real mechanism instead: the - Configuration Importer (official docs) scans an attached volume for - `/conf/config.xml` in a 2-3s boot window; ISO9660 support was added to it - specifically for VM/cloud automation (a closed, milestone-targeted, - core-dev-assigned GitHub issue) -- mechanically identical to - `cloudinit-vm`'s cdrom-attach shape, different payload. Use the nano image - for KVM (pre-installed, serial-ready), not the installer images. Fully - sourced in `opentofu/README.md`. NOT built yet: a decompress/convert - preprocessing script for the nano image, an ISO9660 config.xml-seed - builder, and the real config.xml content -- itemized as the next - actionable step. COMMITTED + PUSHED (75ca823). -- **DONE -- DOCFIX-146 (changelog `docs/changelog-20260709-opnsense-edge-module.md`):** - operator asked to continue with OPNsense while the research was fresh. - Built all three itemized next-steps: `scripts/opnsense-prep-image.sh` - (nano image decompress+convert, no hardcoded mirror -- `OPNSENSE_MIRROR_BASE` - required), `scripts/opnsense-build-config-iso.sh` (ISO9660 config.xml - seed), and `opentofu/modules/opnsense-edge` wiring both together -- - mechanically identical to `cloudinit-vm`'s cdrom-attach shape, no - `libvirt_cloudinit_disk` involved. Explicit `lan_network_name`/ - `wan_network_name` variables (not an ordered list) to prevent a silent - LAN/WAN swap. Both harnesses green (4/4, 3/3) exercising REAL missing-tool - guards (`qemu-img`/`genisoimage`/`xorriso` are all genuinely absent this - session). Two things flagged, not presented as fact: whether the - Configuration Importer's ISO9660 behavior holds on a real boot, and - whether `devices.interfaces` list order reliably maps to LAN/WAN NIC - enumeration for this guest. `config_iso_path` has no default -- real - config.xml content per site is still undesigned. Not instantiated in root - `main.tf`. `opentofu/README.md` + workflow doc + visual tracker updated to - reflect BUILT (not just researched). COMMITTED + PUSHED (a3ff07f). -- **DONE -- DOCFIX-147 (changelog `docs/changelog-20260709-opentofu-maas-netem.md`):** - operator asked to pull docs and validate before drafting the next modules - -- the two remaining unbuilt mechanisms from the OpenTofu scope - (MAAS VM-host registration, tc netem application). Researched a real - official MAAS provider (`canonical/maas` v2.7.2) and OpenTofu's own - `local-exec`/`terraform_data` docs BEFORE writing any code, per - instruction. Caught a real design mistake before it happened: read both - `maas_vm_host` (register) and `maas_vm_host_machine` (compose -- takes - cores/memory/disks as INPUTS, MAAS's "Compose machine" pod feature) - schemas, cross-checked against D-103's explicit "MAAS does NOT compose new - ones" ruling, and built `modules/maas-vm-host` using ONLY `maas_vm_host` -- - the other resource would have had MAAS and `node-vm` fighting over VM - creation. Also confirmed `local-exec` runs on the machine invoking `tofu - apply` (Office1 per D-103), not the vcloud host where bridges live -- - `modules/netem-link` wraps its `tc qdisc` command in an explicit SSH hop - because of this, using `terraform_data` (OpenTofu's own recommended - replacement for `null_resource`) with a destroy-time provisioner to clean - up. Neither module instantiated (MAAS zone/pool, vcloud SSH target, and - real netem parameters all still pending). `opentofu/README.md` gained a - new sourced research section; workflow doc + visual tracker updated. - COMMITTED + PUSHED (ea276c5). -- **DONE -- DOCFIX-148 (changelog `docs/changelog-20260709-opentofu-audit.md`):** - operator asked to check back over all current modules, pull docs for - anything drafted without it, and review for errors/assumptions. Grepped - every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules and - researched each further rather than re-stating hedges unchanged. **Found - and fixed a genuine documentation error, not just a hedge:** - `opnsense-edge`'s comments claimed interface list-order directly sets - OPNsense's LAN/WAN role -- researching OPNsense's own interface-assignment - model showed role assignment is a SEPARATE, explicit config.xml mapping, - not automatic from device order ("vtnet0=WAN" is a convention some guides - use, not an enforced rule). Comments + both LAN/WAN variable descriptions - corrected; no config.xml has been written yet so this fixes documentation - accuracy for whoever writes it next, not current runtime behavior. - Confidence UPGRADED (not fully resolved) on three items: `node-vm`'s - `boot`-order shape (confirmed the provider 1:1-mirrors libvirt's own XML, - and libvirt's own docs confirm ``'s single-attribute - shape), `create.content.url` accepting local paths (second independent - source), and the `genisoimage` flags (confirmed standard, cleanly - separated from the still-open "does OPNsense's Importer actually read it" - question). Confirmed safe: `maas_vm_host`'s `zone`/`pool` computed-if-unset - behavior. `opentofu/README.md` gained a new "Audit pass" section. Deliberately - did NOT re-research already-well-sourced findings (Importer mechanism, - vm_host-vs-vm_host_machine, the syntax-bug fix) -- targeted only real gaps. - Also added (same delivery): `.claude/skills/openstack-cloud-ops/references/ - opentofu-provider-docs.md` -- indexes every provider/doc source used - building `opentofu/` with exact URLs, confirmed versions, default - branches, AND the fetch methodology that actually worked this session - (Registry doc pages are JS-rendered, use GitHub instead; branch names vary, - check `default_branch` via the API; real example `.tf` files beat - doc-summarized prose for syntax questions). New SKILL.md routing entry; - cross-referenced from `opentofu/README.md` in both directions, no content - duplicated (README.md stays the single source of truth for per-module - findings). COMMITTED + PUSHED (96ecbad). -- **DONE -- DOCFIX-149 (changelog `docs/changelog-20260709-opnsense-config-design.md`):** - operator asked to move into designing the real config.xml content. - Fetched OPNsense's own real, currently-shipped `config.xml.sample` - (`opnsense/core` repo) and its `Route.xml` static-routes model before - drafting. Built `opentofu/templates/opnsense-config.xml.tmpl` - (`{{TOKEN}}`-parameterized, reusing this repo's existing clientdocs - convention) + `scripts/opnsense-render-config.sh`. The real sample fetch - DIRECTLY CONFIRMED the DOCFIX-148 audit finding with an actual example: - ships literal placeholder device names inside each interface's own `` - element, proving LAN/WAN role assignment really is the explicit per-block - mapping DOCFIX-148 concluded. **The renderer is tested END-TO-END, not - just guard clauses** (8/8 -- it needs no external tool, a first among the - opnsense-* scripts) -- and the harness caught a REAL bug before it shipped: - the token `HOSTNAME` collides with bash's own built-in `$HOSTNAME` - variable (`unset` doesn't actually clear it), silently passing a test that - should have failed. Renamed to `OPNSENSE_HOSTNAME` throughout. Full token - legend in `opentofu/templates/README.md`, marking exactly which values - are real (NTP pool, D-106 naming) vs. still pending Stage 0 ratification - (D-100/D-101/D-107) vs. only measurable on a real boot (`vtnetN` - assignment) vs. a security requirement (root password hash must be - freshly generated, never the stock sample's own shipped default). - `opentofu/README.md` + workflow doc + visual tracker updated. Uncommitted - -- operator has not yet asked to commit this piece. -- **DONE -- Stage 0 (decision ratification) RATIFIED 2026-07-09 (changelog - `docs/changelog-20260709-stage0-ratification.md`, DOCFIX-150):** operator - walked through the 6 buildout-design redline items live (AskUserQuestion, - 2 rounds): Ceph size=3-by-default (size=2 only as an explicit Phase-0 - fallback); netem same-metro lean; D-102 folds into D-101, D-109 stays - standalone; metal-admin GAINS a ULA leg (reverses the stated lean -- a - real deviation, not a rubber-stamp); COS per-DC-only confirmed; mirror - sync topology independent-per-DC confirmed. All 11 of D-100..D-110 - individually flipped PROPOSED -> ADOPTED in `docs/design-decisions.md` - (D-102's content preserved under a MERGED-INTO-D-101 status line, per the - append-only discipline -- not deleted). `docs/dc-dc-buildout-design.md` - Section 10 rewritten from "open items" to a ruling record; Section 3's - Ceph gate and the stray "Section D-102" cross-reference corrected to - D-101. **Verified via `ledger-scan.sh`: D-100..D-110 no longer appear - under PROPOSED/OPEN** -- only D-068/D-071 remain (separate, pre-existing, - unrelated to this gate). repo-lint 0 fail/1 documented warn, harness - 34/34. Stage 0 of `docs/dc-dc-deployment-workflow.md` is CLEARED. -- **Operator context, 2026-07-09 evening:** operator stepped away from the - workstation for the night, granted blanket git permission for the - session, and authorized autonomous subagent use to make maximum progress - toward an end-to-end-ready project by morning. Operator clarified this - session runs from the WORKSTATION, not the vopenstack-jesse jumphost -- - tonight's scope is PREP ONLY (docs, scripts, tests, hardened runbooks); - no live infra (juju/MAAS/OpenStack APIs) is reachable or in scope tonight. - Operator will switch to the vcloud host in the morning to run actual - installs/deployments. This explains (and validates) why - `run-tests-all.sh` shows 23 pre-existing failures in this environment -- - all either "jq required" (not installed on the workstation) or - live-cloud-dependent suites (`cloud-assert`, `preflight`, `tenant-*`, - etc.) -- confirmed via `git stash` A/B to be identical with and without - this session's own changes, i.e. an environment gap, not a regression. -- **DONE -- Tooling gap register #1 CLOSED 2026-07-09 ($DC selector - convention, changelog `docs/changelog-20260709-dc-selector-convention.md`, - DOCFIX-151):** added `lib_net_select_dc()` to `scripts/lib-net.sh` - (dc0/dc1 no-op per D-101's inherited-layout ruling; dc2 fails loud -- - gap #3 still open) and `lib_hosts_select_dc()` to `scripts/lib-hosts.sh` - (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment - exists for either yet, an intentional asymmetry vs. the net selector, - documented in-line in both files). Backward compatible by construction -- - no existing caller of either library is affected. New harness - `tests/dc-selector/run-tests.sh`: 21/21 PASS (caught and fixed a real - `pipefail`-vs-nonzero-exit-code bug in the harness itself while writing - it). repo-lint 0 fail/1 documented warn (unchanged); full gauntlet - confirmed via `git stash` A/B to introduce zero new failures. Stage 5's - runbook (not yet authored) still needs to actually CALL - `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` -- the mechanism - exists, the call sites don't yet. -- **DONE -- Tooling gap register #3 mechanism CLOSED 2026-07-09 (NetBox - DC-DC pipeline, changelog `docs/changelog-20260709-netbox-dc-dc-pipeline.md`, - DOCFIX-152):** `netbox/dc-dc-prefixes-import.py` extends the v1 single-site - IPv4-only NetBox import to VR1's two-DC dual-stack model (D-101). DC1's - v4 planes hardcoded (D-101: inherited unchanged, this is decision text, - not an inference); DC2's v4 supernet, the org ULA /48, and the per-DC GUA - carve are all REQUIRED env vars with no defaults -- fails loud rather than - inventing any of them. The per-plane subdivision scheme for the ULA/GUA - blocks is flagged in the script's own docstring as a proposed, unratified - convention pending operator review. New harness - `tests/dc-dc-prefixes-import/` (a fake in-memory NetBox client): 37/37 - PASS at the time (later corrected to 40/40 by DOCFIX-160 -- a dead-code - bug in the harness itself had left 3 happy-path assertions silently - uncounted; see that entry below). Caught and fixed two real authoring - bugs at delivery time: (1) example GUA addresses - in the first draft had host bits set beyond the /40 boundary, caught by - Python's own `ipaddress.ip_network(strict=True)`; (2) the test's own - "count CREATED lines" assertion was inflated by the script's unrelated - site-creation message also containing the word CREATED -- tightened to - match prefix-specific lines. Flagged, not fixed (out of scope): the - existing v1 `ipv4-prefixes-import.py`'s role slugs predate the current - six-plane model in `scripts/lib-net.sh`; this new script uses the CURRENT - slugs instead. repo-lint 0 fail/1 documented warn (also caught and fixed - a CRLF regression introduced by a Python find/replace during test-data - cleanup). Full gauntlet 23/45 FAILED -- confirmed identical pre-existing - environment-gap failures (jq missing / no live juju-cloud access on this - workstation), zero regressions. UNVALIDATED against a real NetBox (none - reachable this session). This closes the TOOLING half of gap #3 only -- - the literals themselves (ULA /48, GUA carve, DC2 supernet) still need - real assignment before this script can be run for real. -- **DONE -- Stage 1 runbook WRITTEN 2026-07-09 (changelog - `docs/changelog-20260709-dc-dc-phase0-runbook.md`, DOCFIX-153):** - `runbooks/dc-dc-phase0-vcloud-prep.md` -- command-level Phase 0 (vcloud - host prep) steps, matching the phase-00 runbook's style (sequence table, - read-only CHECK before every mutation, individually gated mutations, an - explicit gate section, delivery checklist). Surfaces a genuine open - structural question rather than silently resolving it: `opentofu/main.tf`'s - unconditional `provider "maas"` block may force MAAS credentials to be set - at `tofu plan` time even though this stage's plan touches only libvirt - resources -- documented both possible outcomes and what to do for each. - Also flags the VR0/DC0-pool-path-collision risk if the vcloud host is the - same physical machine being repurposed (buildout design's own framing). - NOT YET EXECUTED -- this is the runbook the operator will run tomorrow - morning after switching to the vcloud host; nothing in this delivery - claims a real run happened. repo-lint 0 fail/1 documented warn (unchanged, - prose-only). -- **DONE -- Stages 2, 3, and 5 runbooks WRITTEN 2026-07-09**, three of five - remaining runbooks delegated to parallel subagents (Stages 2, 3, 4, 6, 7) - while Stage 5 was authored directly (the ground-truth/copy-point stage): - - **Stage 2** (changelog `docs/changelog-20260709-dc-dc-phase1-runbook.md`, - DOCFIX-154): `runbooks/dc-dc-phase1-office1-standup.md`. Every VM- - creation step forks explicit Option A (OpenTofu, blocked pending - cloud-init content design)/Option B (manual, logged debt). Found a real, - previously-unnoticed gap: no Office1-local virtual network is modeled - anywhere in `opentofu/`. Caught and fixed a repo-lint L2 false-flag - during review (a correctly-sourced historical D-014 path reference, - reworded onto one line so the rule's own EXEMPT pattern applies, not - deleted). - - **Stage 3** (changelog `docs/changelog-20260709-dc-dc-phase2-runbook.md`, - DOCFIX-155): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`. DC1-first; - DC2 hard-gated (not "later"). Five `!!!` callouts naming real - constraints (Stage 2 dependency, DC2 gate, no-Roosevelt-analog shim, - undecided node sizing, unruled netem params) plus OPNsense tool- - presence check. GATE section concludes "CONDITIONALLY MET AT BEST," - refusing to claim more than is actually true. - - **Stage 5** (changelog `docs/changelog-20260709-dc-dc-phase4-runbook.md`, - DOCFIX-156): `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. Describes - adapting (not duplicating) `phase-01..08` to run twice, once per DC. - DC1 needs zero VIP/CIDR edits (D-101 inheritance); DC2 blocked on the - same NetBox literals as Stages 3/4. Mandates `lib_net_select_dc "$DC"`/ - `lib_hosts_select_dc "$DC"` before every adapted command block touching - those libraries (DOCFIX-151's first real consumer). Surfaces two - genuine open design gaps rather than inventing around them: the D-101 - IPv6 family-matrix overlay (exact charm config option names - unconfirmed) and D-109's IPv6-SAN cert-issuance charm support - (unverified assumption). - All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED (prep-only - session, no live infrastructure reachable). -- **DONE -- Stages 4, 6, 7 runbooks WRITTEN 2026-07-09 -- ALL SEVEN Stage - 1-7 runbooks now complete, tooling gap register item #9 CLOSED:** - - **Stage 4** (changelog `docs/changelog-20260709-dc-dc-phase3-runbook.md`, - DOCFIX-157): `runbooks/dc-dc-phase3-maas-enlist-deploy.md`. Mandates - `lib_net_select_dc "$DC"` then HONESTLY documents `lib_hosts_select_dc - "$DC"` as expected to fail for both DCs until this stage's own - enrollment populates real host data -- a deliberate chicken-and-egg. - Names 7 real gaps in its own section (no OpenTofu module stands up a - per-DC MAAS rack controller VM; three scripts not `$DC`-parameterized; - node sizing undecided; DC2 blocked at the selector layer; no owner for - the per-DC mirror; VM-host-discovery MAAS status unconfirmed; DC0- - specific tag/fabric literals). - - **Stage 6** (changelog `docs/changelog-20260709-dc-dc-phase5-runbook.md`, - DOCFIX-158): `runbooks/dc-dc-phase5-dr-failover-drill.md`. Found three - real `bundle.yaml` gaps by grepping it directly (no `cinder-backup` - charm, no `ceph-rbd-mirror` charm, `ceph-mon`'s `rbd-mirror` binding on - the wrong plane) and makes fixing them the runbook's own first gated - mutation. Includes a negative-control drill (partition-only, expected - to correctly refuse promotion) before the real hard-down drill. Every - Ceph object name is an explicit placeholder. - - **Stage 7** (changelog `docs/changelog-20260709-dc-dc-phase6-runbook.md`, - DOCFIX-159): `runbooks/dc-dc-phase6-designate-cos-magnum.md`. Quotes - the bundle's actual current "NO designate" text verbatim before - describing the reversal; correctly identifies D-106's reactivation - also reverses the bundle's separate B5 os-public-hostname-dropped - posture (two coupled changes, not one); proposes per-DC hostname - overlays to keep `bundle.yaml` DC-agnostic; quotes D-046's domain-setup - trap verbatim twice; leaves COS's deployment mechanism as an explicit - operator decision point rather than inventing one. - All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED. Four new - tooling-gap-register items (#12-15) added for the cross-cutting findings - this whole runbook-authoring pass surfaced (Office1-local network - missing, IPv6 family-matrix overlay missing, D-108 bundle gaps, Stage-4 - MAAS/multi-rack gaps) -- see `docs/dc-dc-deployment-workflow.md`. -- **DONE -- Adversarial review pass across the whole evening's work - (changelog `docs/changelog-20260709-adversarial-review-fixes.md`, - DOCFIX-160):** ran a dedicated fresh-eyes review subagent (no authoring - context) across `lib-net.sh`/`lib-hosts.sh`'s new selectors, - `netbox/dc-dc-prefixes-import.py`, and all seven Stage 1-7 runbooks, - instructed to find real defects and cross-check every claim against - ground truth directly (not trust the runbooks' own self-report) -- - including running both new test harnesses live. Found and fixed 3 real - issues: (1) a genuine Python control-flow bug in `tests/dc-dc-prefixes- - import/test_logic.py`'s `expect_ok()` -- a `return` inside `try` made - the `else` clause unreachable dead code, silently leaving 3 happy-path - assertions uncounted (37/37 was true for what ran, but overstated what - was verified; now 40/40 with the fix, independently re-verified via a - live `python3 -c` reproduction before trusting the agent's claim); (2) a - self-contradictory pair of adjacent comments in `dc-dc-prefixes- - import.py` about /19-vs-/22 arithmetic (no functional bug, just wrong - prose); (3) stale "as of this writing" self-references in the Stage 4 - runbook, written before Stage 3 existed and never reconciled once it - did. Propagated the corrected 40/40 count into `docs/dc-dc-deployment- - workflow.md` and this ledger's own earlier entry (left the original - delivery changelog un-edited as historical record of what was believed - at the time). The review explicitly found NOTHING wrong with: the $DC - selector asymmetry description (consistent everywhere), the hand- - verified ULA/GUA carve arithmetic, Stage 6's `bundle.yaml` claims - (confirmed byte-for-byte), 3+ direct D-NNN quotes (verbatim-accurate), - and DOCFIX/D-NNN numbering consistency across every file -- and found no - inferred/invented literal value anywhere in the reviewed material. - repo-lint 0 fail/1 documented warn; both affected test suites re-run - clean (40/40 and 21/21) after the fixes. -- **DONE -- Final consolidation pass (changelog - `docs/changelog-20260709-final-consolidation.md`, DOCFIX-161), closing - out tonight's autonomous session:** a last full read-through of - `docs/dc-dc-deployment-workflow.md` for internal cross-stage - consistency found ONE MORE real gap -- no stage explicitly owns creating - Office1's own OPNsense edge VM, despite the topology being explicit that - the design is per-site (three sites, three edges: DC1, DC2, AND - Office1) -- logged as gap #16, not silently decided. Also fixed two - stale cross-references left over from earlier in the same evening (the - design-doc summary row's confusing "DONE, PROPOSED" phrasing; the - lib-net/lib-hosts row's "Stage 5 runbook still needs to call it," now - true of both Stage 4 and Stage 5). Companion Artifact redeployed twice - more: all seven stage pills moved to a new "Runbook written" state (a - 4th click-cycle state added, distinct from not-started/in-progress/ - done), gap register items 1/3/9 marked closed and 12-16 added, and the - 37/37-to-40/40 test-count correction propagated in. Final full gauntlet - re-run: same 23 pre-existing environment-gap failures as every earlier - check tonight (jq missing / no live juju-cloud access on this - workstation) -- zero regressions across the entire evening's cumulative - changes. repo-lint 0 fail/1 documented warn. - **Session summary for the operator's morning read:** 11 commits - (DOCFIX-151 through DOCFIX-161) delivered gap #1 (closed), gap #3's - tooling half (closed, data half open), and all seven Stage 1-7 runbooks - (gap #9 closed), plus an adversarial review pass and this consolidation. - Nothing was executed against real infrastructure -- prep-only session, - as the operator specified. The morning's first real action is Stage 1's - runbook (`runbooks/dc-dc-phase0-vcloud-prep.md`) on the actual vcloud - host. Genuinely open blockers, roughly in the order they'll be hit: gap - #7 (MTU/Ceph disk-budget arithmetic), gap #16 (Office1 OPNsense edge - ownership), gap #3's data half (NetBox literals), gap #2's UNVALIDATED - status (no `tofu` binary has run yet), gaps #12-15 (Office1-local - network, IPv6 family-matrix overlay, bundle.yaml DR gaps, MAAS/ - multi-rack gaps). -- **DONE -- Tooling gap register item #7 CLOSED 2026-07-09 (MTU/geneve + - Ceph disk-budget calculators, changelog `docs/changelog-20260709-mtu- - ceph-budget-calculators.md`, DOCFIX-162 -- numbering deferred to the - orchestrating session to avoid a collision with other parallel - in-flight agents tonight):** `scripts/dc-dc-mtu-geneve-budget.sh` - (required `--underlay-mtu`, no default) reproduces D-101's own worked - example exactly (1500 - 56 = 1444), leaves tenant MTU at 1500 for a - measured jumbo (>=9000) underlay, and adds a clearly-separated - IPv6-minimum-link-MTU (1280, RFC 8200) sanity floor beyond D-101's own - text. `scripts/dc-dc-ceph-disk-budget.sh` (required `--total-disk`, - per-DC `--dcN-nodes`/`--dcN-per-node-osd`, and required - `--backup-overhead-fraction` with deliberately NO default -- not a hard - number anywhere in this repo, must come from real DC1 Ceph/radosgw-admin - measurement) computes whether size=3 across DC1+DC2 plus overhead fits - the measured total disk, and if not, names size=2 as the ONLY documented - fallback (D-101/Section 3) without silently applying it -- an explicit - operator-logged decision. New harnesses: `tests/dc-dc-mtu-geneve-budget/ - run-tests.sh` 19/19 PASS, `tests/dc-dc-ceph-disk-budget/run-tests.sh` - 16/16 PASS -- both cover the exact worked/constructed numeric examples, - the jumbo/fits vs fallback branches, and every required-arg/bad-format - fail-loud path. `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy - warn (unchanged). Both scripts are read-only calculators (mutate - nothing) and have not yet been run against the real vcloud host -- that - first real run is part of executing `runbooks/dc-dc-phase0-vcloud- - prep.md` Step 3. Uncommitted -- operator has not yet asked to commit - this piece; the orchestrating session will assign the real DOCFIX number - and find-replace `DOCFIX-162` at integration time. -- **DONE -- Tooling gap register items #12 and #16 CLOSED 2026-07-09 - (changelog `docs/changelog-20260709-office1-network-edge.md`, - DOCFIX-163 -- numbering deferred to the orchestrating session, same - reason as the #7 item above):** built `opentofu/modules/office1-network` - (a new, dc-planes-shaped isolated `libvirt_network` sized for ONE network, - not six planes -- closes gap #12) and INSTANTIATED it for real in root - `main.tf` (`module "office1_network"`, needs no unmeasured value beyond - already-real `domain_suffix`/`underlay_mtu`). Design decision: a new - module, deliberately NOT a reused host bridge -- a bridge would be the one - network in this topology living outside OpenTofu's D-103-mandated - inventory and would silently break this repo's own `virsh net-list --all` - completeness assumption; full reasoning in the module's own header and in - `runbooks/dc-dc-phase1-office1-standup.md`'s Open question #1 (now marked - RESOLVED). Also decided gap #16 (Office1 OPNsense edge ownership): YES, - Office1 gets its own `modules/opnsense-edge` call, a fourth site alongside - DC1/DC2, and Stage 2's own runbook owns creating it (new Step 4b, - CHECK-before-MUTATION shape, its own GATE bullet) -- Stage 2's whole scope - is already "Office1 headend standup" and Stage 1 already builds this - edge's network neighbors, so Stage 3 has no reason to reach back into - Office1's own infrastructure. A commented-out `module "office1_opnsense"` - skeleton now exists in `main.tf`, mirroring Stage 3's `dc1_opnsense` - template exactly; its `lan_network_name` resolves cleanly to the new - network, but its `wan_network_name` does not -- investigated rather than - invented, and found to be a genuinely DEEPER, cross-site gap: Stage 3's - own `dc1_opnsense` template carries the identical unresolved - `wan_network_name` placeholder, confirming no dedicated per-site - ISP-uplink/WAN network exists yet for ANY site (DC1/DC2 included). Logged - as NEW tooling gap register item #17 rather than defaulted to a mesh-link - network, which would have directly contradicted D-100's own sub-item - ruling that the three mesh legs carry management traffic only. `bash - scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn (unchanged); - `opentofu/` remains UNVALIDATED overall (no `tofu` binary available this - session, same standing status as every other module here -- not - newly introduced by this delivery). Uncommitted -- operator has not yet - asked to commit; the orchestrating session will assign the real DOCFIX - number and find-replace `DOCFIX-163` at integration time. -- **DONE (uncommitted) -- Gap #13 (D-101 IPv6 family-matrix overlay) - DRAFTED, one real risk left open, researched directly by the - orchestrating session (not delegated) given the accuracy stakes:** - `overlays/dc-dc-ipv6-family-matrix.yaml` + `docs/dc-dc-ipv6-charm- - research.md` (full sourcing). Fetched REAL charm config/source, not - memory: `charm-keystone/config.yaml` confirms `prefer-ipv6` (bool) is a - real, shared charms.openstack option; `charm-nova-cloud-controller`'s - actual "Dual Stack VIPs" commit confirms it's genuinely ADDITIVE (HAProxy - binds both v4 and v6 wildcard addresses simultaneously), not an - either/or switch -- resolving the biggest open question about whether - D-101's dual-stack ambition is even mechanically possible. `charm-ceph- - mon/config.yaml` confirms its OWN separate `prefer-ipv6` + - `ceph-public-network`/`ceph-cluster-network`, which per the charm's own - docs IS a straight switch (not additive) -- the correct shape for the - ULA-only storage/replication planes. Fetched the FULL `charm-layer- - ovn/config.yaml` (25 options) and confirmed NO IPv6/encapsulation option - exists at all -- OVN needs no overlay entry, geneve family follows the - bound interface once the plane is ULA-only. Fetched `charm-vault`'s - actual `vault_pki.py` `sort_sans()` code and confirmed no IPv4/IPv6 - distinction in cert-issuance -- D-109's IPv6-SAN requirement is - code-confirmed, not assumed. **Found a real, open upstream risk while - researching, not invented or glossed over:** Octavia's `lb-mgmt-net` - IPv6 support has two real, still-referenced Launchpad bugs (#1911788, - #1913409) describing failures -- #1911788's root cause is an OVN/LXD/ - MAAS hostname-resolution mismatch, the SAME CLASS of problem this repo's - own D-008 bootstrap order already hardens against, so not necessarily - fatal, but genuinely unresolved. The overlay deliberately excludes an - Octavia entry rather than silently forcing the risk through. Updated - Stage 5's runbook (`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`) Step 6 - and the tooling gap register item #13 to reflect all of this. `bash - scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. Changelog: - `docs/changelog-20260709-ipv6-family-matrix-overlay.md`, **DOCFIX-164** - (number confirmed and closed out here -- this line previously ended - with unresolved "will assign at integration time" boilerplate; caught by - the full-project-sweep's DC-DC-runbook-consistency agent as an - integration miss, fixed 2026-07-10). -- **DONE -- `$DC` parameterization for the 3 MAAS scripts (changelog - `docs/changelog-20260709-maas-scripts-dc-param.md`, DOCFIX-166), - closing gap #15 sub-item 2's CLI-invocation half:** `scripts/reenroll- - hosts.sh`, `scripts/carve-host-interfaces.sh`, and `scripts/phase-00-maas- - standup.sh` now accept an opt-in `$DC` env var and call `lib_net_select_dc` - / `lib_hosts_select_dc` immediately after sourcing, matching the same - convention (and the same dc1-network-no-op/dc1-hosts-fail asymmetry) - DOCFIX-151 gave the two library files the night before. Unset `$DC` is - provably byte-for-byte unchanged (verified by direct, jq-independent - invocation of all three scripts across the unset/dc0/dc1/dc2/bogus - matrix, since the selector calls sit before any MAAS/jq touch-point in - every script). Added `$DC` test cases to both existing harnesses - (`carve-host-interfaces`, `phase-00-maas-standup`) and created a new one - (`reenroll-hosts` had none before). Did NOT invent any per-DC host/CIDR - literal, and did NOT make `phase-00-maas-standup.sh`'s hardcoded `PLANES` - table dc2-aware -- both remain honestly open (dc1 only "works" there - because D-101 makes its real target identical to dc0's). Did not run a - `git stash` A/B this time since the working tree carries other parallel - agents' in-flight uncommitted work tonight; verified directly instead. - repo-lint 0 fail/1 documented warn. `docs/dc-dc-deployment-workflow.md` - item #15 and `runbooks/dc-dc-phase3-maas-enlist-deploy.md` (gap #2, Steps - 5 and 8, delivery checklist) updated to reflect the partial closure -- - six of the seven gap-#15 sub-findings remain open, said plainly rather - than overclaiming. No live infrastructure touched. (DOCFIX-166, already - cited above -- this closing line's stale "will assign at integration - time" boilerplate corrected 2026-07-10, flagged by the same sweep pass.) -- **DONE (uncommitted) -- Gaps #6 + #14 (bundle.yaml: Designate reactivation - + D-108 DR mechanism), done directly by the orchestrating session given - the stakes (this is the SAME `bundle.yaml` the jumphost operates the - live VR0/DC0 rehearsal cloud with, not a DC-DC-only file):** - `bundle.yaml` gained `designate`/`designate-bind`/`designate-mysql- - router`/`designate-hacluster` (D-106) and `cinder-backup`/`ceph-rbd- - mirror` (D-108) + `ceph-mon`'s `rbd-mirror` binding corrected from - `storage` to `replication`. **Caught and fixed three real mistakes - during self-review, by fetching each new charm's REAL metadata.yaml - rather than trusting the standard-pattern draft or the Stage 7 runbook's - own earlier "template" section:** (1) designate's real provides-side - endpoint is `dnsaas`, not `public` like the other API charms; (2) - `cinder-backup` is a SUBORDINATE charm (confirmed `subordinate: true`) - -- the first draft wrongly gave it `num_units`/`to`/an `amqp` relation - that doesn't exist on this charm at all; (3) `ceph-rbd-mirror` is a - principal with `ceph-local`/`ceph-remote` requires endpoints (confirmed - `subordinate: false`) -- the first draft invented a single wrong binding - key. **A genuine, unresolved cross-DC wiring limit documented rather - than worked around:** `ceph-remote` (the peer DC's ceph-mon) cannot be - a same-bundle relation given D-104's per-DC-independent-Juju-controller - design -- only `ceph-local` (this DC's own leg) is wired; the cross-DC - peer relationship needs `juju offer`/`consume` or the manual `rbd - mirror pool peer bootstrap` CLI flow Stage 6's runbook already - documents. Deliberately did NOT reactivate `os-public-hostname` on the - 11 API charms in this shared bundle (D-106's OTHER coupled change) -- - pushed to a proposed per-DC overlay instead, to keep `bundle.yaml` - itself DC-agnostic. Verified: `python3 -c "import yaml; yaml.safe_load( - open('bundle.yaml'))"` parses clean (56 apps, 108 relations); `python3 - scripts/provider-bundle-check.py` PASS, all 6 existing invariants - unaffected (`tests/provider-bundle-check/`: 8/8 unaffected); `bash - scripts/repo-lint.sh` 0 fail/1 documented warn. Reconciled Stage 6's and - Stage 7's runbooks to point at the now-real bundle content instead of - their earlier "template"/"known gap" framing. NOT applied to any live - model this session -- no live cloud reachable; this is a repo-source - change only, to be reviewed via `juju deploy --dry-run` before any real - apply. Changelog `docs/changelog-20260709-designate-cinderbackup- - rbdmirror.md`, **DOCFIX-167** (number confirmed and closed out here -- - this line previously ended with unresolved "will assign at integration - time" boilerplate; caught as an integration miss by the full-project - sweep's DC-DC-runbook-consistency agent, fixed 2026-07-10). -- **DONE (uncommitted) -- Tooling gap register item #5 MECHANISM CLOSED - 2026-07-09 (changelog `docs/changelog-20260709-ceph-replication- - tooling.md`, DOCFIX-165 -- numbering deferred to the orchestrating - session, same reason as the other parallel deliveries tonight):** the - radosgw multisite / rbd-mirror command sequences already written out in - `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and - 11.2-11.4 (verbatim upstream Ceph administration, not invented) are now - real, callable scripts: `scripts/dc-dc-radosgw-multisite.sh` - (`master-init`/`join-readonly`/`enable-two-way`), `scripts/dc-dc-rbd- - mirror.sh` (`bootstrap-primary`/`bootstrap-secondary --direction - rx-only|rx-tx`), and `scripts/dc-dc-dr-drill.sh` (`failover`/`failback`). - Every realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument - -- no invented defaults, matching the runbook's own `` - placeholder discipline. Default mode is `--dry-run` (prints the exact - `juju ssh`/`juju run` command sequence, secrets redacted); `--apply` - executes one command at a time, aborting on first failure. **The - single most safety-critical property in the whole drill -- demote the - CURRENT PRIMARY before promoting the RECOVERING side during failback -- - is HARD-CODED as fixed call order in `dc-dc-dr-drill.sh`'s `failback` - subcommand**, not left to documentation or an operator-orderable flag; - in apply mode the promote step is only ever reached if the preceding - demote succeeded. Step 10.2's per-image Glance re-registration is always - printed as a flagged MANUAL reminder and NEVER auto-executed (the - runbook is explicit the real invocation depends on unconfirmed Glance - driver behavior); Step 11.3's Cinder reconciliation is likewise not - scripted (tenant/volume-specific). Each script sources `scripts/lib- - net.sh` and calls `lib_net_select_dc` per DC token touched (DOCFIX-151 - convention) -- informational in dry-run (a plan preview is not a - mutation), hard-blocking before any `--apply` (dc2 --apply refused - today, exit 3, since D-101/gap #3 leaves DC2 with no assigned network - literals). New harnesses `tests/dc-dc-radosgw-multisite/`, - `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23 - PASS) test ONLY argument parsing, guard clauses, and dry-run output -- - including a line-order grep proving the failback demote-before-promote - property -- NOT real radosgw/rbd-mirror behavior; no live Ceph cluster - exists this session to test against (same UNVALIDATED posture as every - other tooling delivery tonight). Stage 6 runbook's own "Follow-up, NOT - built now" delivery-checklist item and `docs/dc-dc-deployment- - workflow.md` item #5 both updated to point at these scripts instead of - describing them as future work. `bash scripts/repo-lint.sh`: 0 fail, 1 - documented legacy warn (unchanged). No live infrastructure touched. - Uncommitted -- orchestrating session will assign the real DOCFIX number - and find-replace `DOCFIX-165` at integration time. -- **DONE (uncommitted) -- Gaps #4(d)/#11: proposed netem parameters + ULA/ - GUA/DC2-supernet generation guidance (changelog `docs/changelog- - 20260709-netem-ula-gua-proposal.md`, DOCFIX-168), `docs/dc-dc-netem-and-ula-gua- - proposal.md`:** does NOT rule the netem decision or generate any of the - three real NetBox literals itself -- presents a concrete, reasoned - proposal (1ms delay / 0.2ms jitter / 0.01% loss / uncapped rate, each - with rationale grounded in the buildout design's own "same-metro dark - fiber, low single-digit ms" lean) for OPERATOR ratification, and gives - the exact safe command (`openssl rand -hex 5`, RFC 4193-compliant) for - the operator to generate the real org ULA /48 themselves. Explicitly - does NOT attempt to generate the per-DC GUA carve or DC2's supernet -- - both require real coordination (the ARIN-block administrator; NetBox's - full address inventory) this session has no authority or visibility to - perform; the section instead explains precisely what each requires and - how to close it once ready. `docs/dc-dc-deployment-workflow.md` gap - items #4(d)/#11 updated to reference it; item #10 (harness debt) also - corrected -- LARGELY CLOSED given how many new test suites landed - tonight (dc-selector, dc-dc-prefixes-import, dc-dc-mtu-geneve-budget, - dc-dc-ceph-disk-budget, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, - dc-dc-dr-drill, plus MAAS script additions). `bash scripts/repo-lint.sh`: - 0 fail, 1 documented legacy warn. No live infrastructure touched. - Uncommitted -- orchestrating session will assign the real DOCFIX number - at integration time. -- **DONE -- Full-project sweep for errors/architectural problems, requested - after the seven gap-closure commits: FIRST finding, fixed (changelog - `docs/changelog-20260710-phase01-bundle-staleness-fix.md`, DOCFIX-169):** - `runbooks/phase-01-bundle-deploy.md` -- an EXISTING, LIVE-CLOUD-relevant - VR0 runbook, not a DC-DC artifact -- had gone stale the moment DOCFIX-167 - edited the SAME shared `bundle.yaml` it deploys. Its "expected plan: 50 - apps, 97 relations" (Constants section + Step 1.2's dry-run GATE) was - wrong (real count now 56/108); its GATE's "NO designate (D-019)" text - flatly contradicted the new reality. Fixed both LIVE-gate references; - deliberately left the historical "As-built reference (2026-06-03)" - section's own `50 apps/97 relations` line UNCHANGED (append-only audit- - trail discipline -- it documents a real past deploy, not something to - retroactively edit). Flagged a real, unresolved architectural tension - rather than silently picking a side: this bundle now serves BOTH a plain - VR0-only redeploy AND standing up DC1, and reactivating Designate/the DR - mechanism in the shared file means any future `phase-01` run gets them - regardless of intent -- a future plain-VR0 redeploy needing to exclude - them would need an explicit override, not an assumption either way. - Extended the EXIT GATE's "waiting on vault certs" list to include - designate/designate-bind (inferred from their real `certificates` - relation to vault, clearly labeled NOT YET OBSERVED against a live - deploy). `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. - Sweep continues across the rest of the repo (scripts, other runbooks, - OpenTofu modules, test quality, security/secrets posture). -- **DONE -- Full-project sweep COMPLETE: 8 parallel agents (bundle.yaml, - VR0 runbooks, scripts, OpenTofu modules, DC-DC runbook consistency, test - harness quality, security/secrets, NetBox/IPAM), all findings triaged - and real ones fixed, changelogs DOCFIX-170/171/172/173:** - - **DOCFIX-170** (`docs/changelog-20260710-sweep-vr0-runbook-opentofu- - fixes.md`): `phase-08-workload-cluster-acceptance.md`'s Designate - framing (same cascading-staleness class as DOCFIX-169), nuanced not a - blanket flip -- charms now active/idle-checkable, but D-011.8's real - resolution criterion still needs Stage 7's os-public-hostname overlay, - not automatic from deployment alone. Plus a stale `opentofu/modules/ - base-image/outputs.tf` comment (cosmetic, no functional effect). - - **DOCFIX-171** (`docs/changelog-20260710-sweep-dcdc-runbook-fixes.md`): - Stage 1's calculator scripts (DOCFIX-162) never wired into its own - Step 3 -- fixed to actually invoke them; Stage 5's Sequence-list still - said `[NOT YET WRITTEN]` for the IPv6 overlay after the Step 6 body - was updated -- fixed; Stage 6's checklist claimed its steps call the - new Ceph scripts when the step bodies still only showed raw commands - -- added real "PREFER `scripts/dc-dc-...`" pointers to Steps 4, 5, 8, - 10.2, 11.2 themselves, closing the gap between claim and text. Also - closed out two orphaned ledger entries (DOCFIX-164, 167) whose closing - lines never got their real number written in during the earlier - integration pass -- an integration miss the sweep itself caught. - - **DOCFIX-172** (`docs/changelog-20260710-sweep-bundle-fixes.md`): - `magnum-dashboard` had NO `bindings:` block AT ALL -- a real, - pre-existing bug (confirmed via `git log`, present since the D-052 - explicit-bindings commit, not tonight's regression) violating this - bundle's own stated "explicit per-application blocks" discipline. - Fixed with a minimal, metadata-confirmed block (subordinate, only a - real `dashboard` endpoint -- NOT a copy of `octavia-dashboard`'s - fuller shape, which has a `certificates` endpoint this charm doesn't). - Also relocated two section-header comments that sat before the WRONG - app's bindings block (functionally harmless, confirmed via - `yaml.safe_load` -- purely a future-edit-risk fix) and corrected the - header's stale "11 API charms" VIP-chain count to 12 (with the - VIP-triple count of 11 correctly left alone, since designate doesn't - carry a `vip:` option). `provider-bundle-check.py` PASS throughout, - 56 apps/108 relations unchanged. - - **DOCFIX-173** (`docs/changelog-20260710-sweep-script-fixes.md`): 4 - confirmed logic bugs + 1 security hardening, ALL independently - reproduced in isolation before being trusted (the same discipline - that caught the dc-dc-prefixes-import bug earlier tonight) -- - including catching that a FIRST proposed fix for one of them - (`scripts/phase-06-capi-stack.sh`) was itself wrong: `|| true` after a - failing pipeline under `pipefail` silently discards `PIPESTATUS`, - which would have made the "GATE FAIL" check never fire again, WORSE - than the original bug. The correct `if pipeline; then :; fi` idiom was - verified via reproduction before committing. Full list: `carve-host- - interfaces.sh`'s `emit()` returning 1 unguarded under `set -e` (killed - apply-mode runs on the first MAAS error, before the FATAL-accumulator - summary could print); `phase-06-capi-stack.sh`'s `PIPESTATUS` read - (as above); `juju-spaces-check.sh` and `osd-blank-check.sh`'s one - unguarded display-only pipe each (both safely `|| true`-able since - nothing downstream needed their PIPESTATUS); `scripts/ledger-scan.sh`'s - DOCFIX/BUNDLEFIX regex using a fixed `{3}` instead of `{3,}` -- the - SAME bug class this repo already got bitten by once for D-numbers - (fixed there via `{3,}`, never mirrored here) -- a dormant landmine, - not yet reachable (repo is at DOCFIX-170ish) but real; and - `dc-dc-rbd-mirror.sh`'s bootstrap token using a predictable `/tmp` - path with no permission hardening -- hardened to a random-suffixed - path + immediate `chmod 600`. - Full test gauntlet re-run after all fixes: 23/51 FAILED, identical to - every prior check tonight (same pre-existing jq-missing environment - gaps) -- zero regressions across the entire sweep-and-fix pass. - **Also flagged, not a repo issue:** the scripts-audit agent reported - that a skill's tool output contained injected text resembling a system - reminder, instructing it to conceal a date change from the operator -- - it correctly disregarded the concealment instruction and reported the - anomaly rather than act on it. The main session observed an - identically-worded, identically-formatted message earlier this same - evening (a genuine date-rollover notice, consistent with the session - crossing midnight) and treated it as benign/expected rather than a live - attack -- noted here for the operator's own awareness per this - environment's prompt-injection disclosure norm, not because either - session took any unsafe action in response. -- **DOCFIX-174** (`docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md`): - caught before assigning the next real DOCFIX number for follow-up work -- - `bash scripts/ledger-scan.sh` reported `DOCFIX next-free=1005`, wildly - implausible against a repo at DOCFIX-173. Root cause: DOCFIX-173's OWN - changelog narrated its `{3}`->`{3,}` regex fix using a literal - illustrative example (the real prefix plus one-zero-zero-four), which the - just-widened regex then matched as a real assignment -- a false-positive - one commit after fixing - a false-negative of the opposite kind. Fixed: reworded the illustrative - example in prose (both the DOCFIX-173 changelog and this file's own - mirrored comment), added a `reproduc`-line exclusion to - `nextfree_mentions()` as defense-in-depth (this repo's delivery style - narrates bug reproductions constantly, so this will recur), and added a - regression fixture to `tests/ledger-scan/run-tests.sh` (37/37 PASS, up - from 33). True next-free after this delivery: D-111, DOCFIX-175, - BUNDLEFIX-012. Caught the same way as everything else tonight: by - running the tool and checking whether its own output was plausible - before trusting it, not by code review. -- **DOCFIX-177** (`docs/changelog-20260710-skill-vr1-awareness.md`): - operator was heading to the vcloud host to begin real execution; asked - "is the skill ready?" -- it wasn't fully. `.claude/skills/openstack-cloud- - ops/SKILL.md` and `runbooks/README.md` (what its routing table points to) - had ZERO mentions of the entire VR1 DC-DC track -- a fresh session reading - only the skill would be routed to VR0's phase-01..08 runbooks with no - indication the DC-DC work (today's actual mission) exists. Fixed: - `runbooks/README.md` gained a full "VR1 DC-DC track" index table (7 Stage - runbooks + `dc-dc-teardown-rollback.md`, explicit that its numbering is a - separate sequence from VR0's), `SKILL.md`'s routing table + standard loops - gained VR1-specific rows, `references/environment.md` gained VR1 pointers - (a third "deployment" entry + a repo-map addition), `references/ - troubleshooting.md` gained a note that appendix-A is VR0-only and VR1 has - no incident index yet. Also found and fixed first: `main` was 2 commits - ahead of `origin/main` (DOCFIX-174, 175/176) -- pushed before this - delivery, since the vcloud host's session bootstrap starts with `git - pull` and would otherwise have missed them this morning. -- **DOCFIX-176** (`docs/changelog-20260710-vr1-teardown-rollback-runbook.md`): - closes gap register item #19. New `runbooks/dc-dc-teardown-rollback.md` -- - the OpenTofu/libvirt/MAAS-vm_host layer had no teardown runbook (D-061's - scripts cover a DIFFERENT layer, the juju/MAAS-machine one). Carries - D-061's PRINCIPLE (MAAS-side cleanup before destroying the libvirt - resource underneath) but not its specific mechanism (this repo doesn't use - MAAS pod-composition for OpenTofu-created VMs, so `--keep-instance` - doesn't directly transfer) -- flagged as untested/residual since no VR1 DC - has reached Stage 4 yet. Two paths (scoped `-target` teardown with - HashiCorp's own targeting-risk warning quoted, or full VR1 destroy), a - rollback decision tree favoring fix-forward over destroy for partial - apply failures, explicit mesh-link shared-infrastructure handling, and - confirmed (by reading the actual `.tf` files) that `netem-link`'s - destroy-time provisioner already works correctly. **Caught and fixed - mid-delivery:** re-running `ledger-scan.sh` before assigning this number - found DOCFIX-174's OWN changelog/ledger narration had reintroduced the - exact false-positive class it had just fixed (quoting the illustrative - example literally again) -- fixed by rewording, folded into DOCFIX-174's - existing changelog rather than a new number, since it corrects that - delivery before commit. Also confirmed DOCFIX-175's Stage-1-not-Stage-3 - correction (see its own entry above) was fully propagated before this - runbook's cross-references were written. -- **DOCFIX-175** (`docs/changelog-20260710-opentofu-state-file-security.md`): - answering the operator's own "what have we missed" question surfaced a - real, previously-unlogged gap -- `terraform.tfstate` stores - `var.maas_api_key` in plaintext regardless of `sensitive = true` (that flag - only suppresses CLI output), and nothing gitignored the state file, - documented backup/permission handling, or noted local state's lack of - locking. Fixed: `.gitignore` excludes `opentofu/**/*.tfstate*` + related - local artifacts, `opentofu/README.md` gained a full "State file handling" - section, `variables.tf`'s `maas_api_key` description carries the caveat, - and `dc-dc-phase2-tofu-dc-substrate.md` Step 8 (the actual first real - `tofu apply` in this repo) gained a "SECURE THE STATE FILE" callout right - after its GATE line -- closing the loop rather than leaving a "someone - should add this" note dangling. Gap register gained item #18 (this, CLOSED) - and #19 (VR1 teardown/rollback runbook -- OPEN, tracked as the next - delivery). Remote backend + real locking explicitly flagged as - Roosevelt/production-scope, not invented here. - -## Jumphost stream -- session 2026-07-10 (cont.): VR1 Stage 1 FIRST REAL EXECUTION (dc-dc-phase0) - -First-ever execution of a DC-DC runbook against real infrastructure. Logged -session `dc-dc-phase0-vcloud-prep` (~/as-executed/2026-07-10-dc-dc-phase0-vcloud-prep.log). -On the ACTUAL vcloud host this time (hostname `vcloud`, AMD, 256 vCPU / 1 TiB / -10 TiB, Ubuntu 24.04) -- NOT the Windows box the 2026-07-09 session flagged. - -- **Steps 1-3 DONE (read-only audit + recorded decisions):** - - Step 1: on-host, libvirt reachable `qemu:///system`, user in libvirt group. - `libvirt_uri="qemu:///system"`. - - Step 2: nested KVM ALREADY ON (`kvm_amd/nested=1`) -> **Step 4 SKIPPED**. - No running domains. - - Step 3: uplink `enp1s0` MTU 1500. **Operator ruling: underlay_mtu=9000 - (jumbo)** for the host-internal dark-fiber/plane fabric (the module planes - are addressless isolated virtio bridges, jumbo-capable regardless of the - 1500 ISP uplink; 1500 governs future WAN-edge nets only). Ceph size=3 - disk-budget PLAUSIBILITY PASS (4.60 TiB margin @ conservative 9 TiB total, - 3 OSD nodes/DC x 500G, 0.5 overhead) -- real budget deferred to Stage 5. -- **FINDING: host was NOT pristine** -- carried a prior hand-built VR0 topology - (7 `vr0-dc0-*` planes + `vr0-off0-1_lan` + `vr0-ptp-dc0-off0` + a NAT `wan` - net) and the `default` pool (/var/lib/libvirt/images). **Operator ruled: tear - down the 9 orphaned vr0 nets, KEEP `wan`** (dual-stack NAT-out-enp1s0, GUA - 2602:f3e2:fe:10::/64 within the org ARIN block -- a ready model for gap #17's - ISP-uplink network; Stage-2 finding). Teardown DONE, gated (net 1 individually, - nets 2-9 after durable approval, stop-on-anomaly). Only `wan` + `default` - remain. -- **Step 5 DONE:** pool dirs `/var/lib/libvirt/vr1/{dc1,office1,dc2}`. Operator - granted a self-owned parent `/var/lib/libvirt/vr1` (setgid libvirt) so agent - creates subdirs without sudo. Non-secret session env `~/vr1-stage1.env` - created (TF_VAR_* + log path; NEVER holds maas_api_key). -- **Step 7 DONE (repo wiring):** `opentofu/dc-dc-phase0.auto.tfvars` (gitignored, - DOCFIX-175) from measured values; **`module "dc2_storage"` ACTIVATED** + - `dc2_pool_path` var added (operator: "wire dc2 while we're here"). **DC2 PLANES - DEFERRED (Option B, operator ruling)** -- wait for NetBox to assign the D-101 - supernet/ULA/GUA; do not invent CIDRs. DC2 mesh legs already wired. -- **DELIVERED -- DOCFIX-178:** `scripts/prereqs/` idempotent workstation prereq - installers (opentofu/virtualization/jq/image-tools + check-prereqs + install-all - + lib + README) + `tests/prereqs/run-tests.sh` (24/24) + a Prerequisites - section in the phase0 runbook + Step 6 concretized. Operator ruling: runbooks - must not assume prereq runtimes; give new operators install scripts. repo-lint - 0 fail. Changelog `docs/changelog-20260710-prereq-install-scripts.md`. -- **Steps 6-11 DONE -- STAGE 1 COMPLETE (2026-07-10).** tofu installed via - scripts/prereqs/install-opentofu.sh (OpenTofu v1.12.3). Step 8 init/validate - needed DOCFIX-179 fixes: per-module `required_providers` (child modules do NOT - inherit provider SOURCE, OpenTofu inferred nonexistent hashicorp/libvirt) + - `tofu fmt` of the never-fmt'd tree. Step 9 plan: the provider "maas" Known Gap - demanded BOTH maas_api_url + the SENSITIVE maas_api_key for a zero-MAAS plan - -> operator ruled STRUCTURAL FIX (Option B): deferred the provider "maas" - block + both vars to Stage 3 (DOCFIX-179); maas stays in root required_providers - (lock file complete). Step 10 apply: 13 objects, 0 changed/0 destroyed; state - chmod 600 (no secret in it now). Step 11 gate: all 4 bullets PASS, MTU 9000 - verified on all 10 nets. .terraform.lock.hcl to be committed. -- **NOC redesign (mid-session): RAISED then RETRACTED by operator** -- a - 4-site regional model (Office0/NOC0/DC0/DC1, NOC-hub fiber) was floated, then - withdrawn ("misunderstood an outside conversation; nothing changes"). No design - change; existing VR1 (DC1/DC2/Office1 triangle) stands. Logged so the pivot - isn't re-litigated. -- **DOCFIX candidates ACCRUING for the stage-completion commit (draft runbook - updates per operator's standing instruction):** (1) Step-3 MTU-domain - clarification (uplink 1500 vs jumbo internal underlay); (2) non-pristine-host + - `wan`-as-gap-#17 note for Steps 5/11; (3) Step-9 expected-resource list is - stale (omits `office1_network`) and now also gains `dc2_storage` -> real plan - is 13 resources: 6 dc1 planes + 3 pools (dc1/office1/dc2) + 4 nets - (office1-local + 3 mesh); (4) Step-7 tfvars-vs-gitignore (DOCFIX-175 gitignores - what Step 7 says to commit); (5) other dc-dc-phaseN runbooks want a - `scripts/prereqs/` pointer (per DOCFIX-178 follow-up). -- **DELIVERED -- DOCFIX-180:** dc-dc-phase0 runbook as-executed corrections - folded in at Stage-1 close (operator model: DOCFIX-log through a phase, update - the runbook at phase end). Top banner + Known-gap RESOLVED (maas deferral) + - Step-3 two-MTU-domains + Step-5 non-pristine-host/wan/pool-parent + Step-8 - init/required_providers + Step-9 -input=false and 13-resource plan. Changelog - `docs/changelog-20260710-phase0-runbook-as-executed.md`. repo-lint 0 fail. -- **Numbers after DOCFIX-180:** DOCFIX next-free 181 (re-scan before assigning). -- **STAGE 1 CLOSED + COMMITTED.** Branch `dc-dc-stage1-phase0-first-exec` - (340ac58): DOCFIX-178/179/180. NOT pushed. Git identity set repo-local - (JANeumatrix/jesse.austin@neumatrix.com, matching prior commits) -- this host - had none. -- **DELIVERED -- DOCFIX-181: `carve_gua` subnet-enumeration hang FIXED.** The full - `run-tests-all.sh` gauntlet was HANGING on `tests/dc-dc-prefixes-import` (its - long-claimed "40/40" was never actually reached -- it hung). Root cause: - `netbox/dc-dc-prefixes-import.py::carve_gua()` did - `list(prefix.subnets(new_prefix=64))` -- ~16.7M /64 objects for a /40 GUA -- - when it only uses the first. Fixed with `next()` (O(1)); added a - `faulthandler.dump_traceback_later(30, exit=True)` watchdog to `test_logic.py` - so any future hang self-aborts with a traceback instead of stalling the - gauntlet. Harness 40/40 in 0.05s; full gauntlet now completes green. - Changelog `docs/changelog-20260710-carve-gua-hang-fix.md`. Next-free -> 182. -- **Next: Stage 2** (`dc-dc-phase1-office1-standup`) when directed -- watch gap - #17 (per-site ISP-uplink/WAN net; `wan` is now a candidate model). - -## Jumphost stream -- session 2026-07-11: VR1 NetBox buildout (Phase 1) -- IPv6 sync + D-111 - -- **SYNCED to the Chat/planning stream.** Merged origin/main (057c31b "netbox - uploads": `docs/dc-dc-netbox-buildout-scope.md` [AUTHORITATIVE], - `netbox/roles-aggregates-import.py`, changelog) into the stage1 branch (clean; - keeps the DOCFIX-181 carve_gua fix). Read the scope doc + tool fully. -- **ARCHITECTURE CLARIFIED (operator):** workflow is draft-source - `netbox.baldurkeep.com` -> spin up a NEW NetBox on the Office1 VM (sandbox) -> - import draft + planned changes -> SIMULATE VR1 against the Office1 NetBox -> - feed validated refinements BACK to `netbox.baldurkeep.com`. The sim NEVER - `--commit`s to the real NetBox (read=source, write=feedback-only). Scope doc + - Stage 2 runbook need a note capturing this loop (pending). -- **Read-only draft export DONE** (`netbox.baldurkeep.com` v4.5.8; session token - stored 0600 at `~/.vr1-netbox.env`, NEVER printed -- REVOKE + purge at session - end). Real modeling: VR1 = `2602:f3e2:f00::/40`, DC0=`f02::/48`, DC1=`f03::/48` - (empty containers). VR0-DC0 (`e02::/48`) is the fully-built TEMPLATE: NN - mnemonic (provider 10/VIP 11, metal 20, data 30, storage 40, repl 50, lbaas 80, - vpn e0, oob f0), `/60`+`/64` per plane, all-GUA (old model). Edge Networks - `2602:f3e2:fe::/48` exists (gap-#17 WAN home; `wan` uses `fe:10`). ULA /48 not - assigned (G3); aggregates empty (G2, top blocks are container PREFIXES). -- **D-111 ADOPTED + IMPLEMENTED (DOCFIX-182):** `dc-dc-prefixes-import.py` v6 - subcarve aligned to the deployed NN mnemonic (`/60`+`/64` per plane; metal `/60` - shared by admin `:20`/internal `:21`; ULA `/56` indexed to GUA nibble so 4th - hextet reads `DC.NN`). `carve_ula`+`carve_gua` -> `carve_v6` via direct-math - `_sub_at` (no enumeration; DOCFIX-181 property kept). Harness 40 -> 53 PASS, - repo-lint 0 fail. -- **FIXED (DOCFIX-183):** `docs/dc-dc-netem-and-ula-gua-proposal.md`'s ULA-gen - `sed` emitted INVALID IPv6 (4+6 split); corrected to 2+4+4. Ratified VR1 ULA is - `fd50:840e:74e2::/48` (from the scope doc), so the command is reference-only. -- **Address values now come from REAL NetBox** (supersede earlier invented - proposals): ORG_ULA_48 recommend `fd50:840e:74e2::/48`, GUA dc1 `f02::/48` / - dc2 `f03::/48`, DC2_V4_SUPERNET `10.13.0.0/19` -- pending final import against - the Office1 NetBox. -- **NEXT:** (a) fix the proposal-doc ULA-gen command; (b) stand up the Office1 - NetBox VM (Stage 2: Office1-local addressing + Option-A cloud-init + NetBox - Docker; `pynetbox`/`python3-venv` NOT installed -- needs operator sudo); - (c) G5 site rename (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) as its own - D-number; (d) run roles-aggregates + prefixes import against the Office1 NetBox. -- **Numbers:** D next-free 112, DOCFIX next-free 184. Branch - `dc-dc-stage1-phase0-first-exec` now also carries the merge + D-111 + DOCFIX-183. - -## Jumphost stream -- session 2026-07-12: Office1 OPNsense edge (bring the site router+DHCP online) - -- **Operator ruling:** bring the Office1 OPNsense edge online FIRST (router + DHCP for - `office1-local`) before the Office1 NetBox VM. Edges are the real router; do not hang - service VMs on a `wan` shortcut. -- **TRANSPORT MODEL CLARIFIED (operator, corrects an earlier misread):** each site (both - DCs + Office) simulates its OWN real ISP connection -- the OPNsense edge is a normal - internet-facing router. Dark fiber (mesh links) = East-West / replication ONLY, not an - internet path. D-107 node-airgap (per-DC mirror) is a SEPARATE DC concern, not the edge - WAN egress. -> **DOCFIX-185:** stripped the egress-airgap rules (seq 20/21/99 + MIRROR_*) - from `opnsense-config.xml.tmpl` + render script + harness (8/8). FOLLOW-UP: a D-100/D-107 - amendment note so the Stage-3 DC edges are built to this model (operator to rule). -- **Creds (closed test, agent-generated, 0600 at `~/vr1-office1-creds/`, NEVER commit; - operator to REVOKE the NetBox token + these at session end):** SSH ed25519 keypair + - OPNsense root password + bcrypt hash. -- **Build DONE except boot:** `office1-wan` NAT net `172.30.1.0/24`->enp1s0 (virsh; D-103 - debt); OPNsense 26.1 nano image prepped (DOCFIX-184: bz2 -> python3 fallback, bzip2 absent); - config rendered (office1-opnsense, LAN 10.10.0.1/24, WAN static 172.30.1.2 -- NOTE: the "DHCP .100-.199" this line originally claimed was FALSE; no DHCP was ever in the template, see the 2026-07-12 D-112(c) entry - gw .1) + ISO built; `module "office1_opnsense"` instantiated in main.tf + validated/planned - (first real validation of modules/opnsense-edge); disk + config_seed volumes created. -- **TWO first-boot findings (DOCFIX-186):** (1) `config_iso_path` must be OUTSIDE the pool - (create.content.url collides with its own target volume) -> staged at - `/var/lib/libvirt/vr1/staging/`. (2) **apparmor blocks `/var/lib/libvirt/vr1/`** (custom - pool path not in libvirt's abstraction) -> domain defined-but-failed-to-start, qemu - Permission-denied despite good POSIX perms. FOUNDATIONAL (all VR1 VMs). Fix = operator - one-time sudo: `echo '/var/lib/libvirt/vr1/** rwk,' | sudo tee - /etc/apparmor.d/local/abstractions/libvirt-qemu; sudo systemctl reload apparmor && restart - libvirtd`. PREREQ candidate (gates every VR1 VM boot). -- **NEXT:** operator applies the apparmor rule -> clean up the partial apply (domain defined - but not in tofu state) -> re-apply to boot -> verify WAN 172.30.1.2 + LAN DHCP/routing -> - then the Office1 NetBox VM. -- **Numbers:** D next-free 112, DOCFIX next-free 187 (184/185/186 committed this session). - -## Jumphost stream -- session 2026-07-12 (cont.): OPNsense edge BOOT INCIDENT (OPEN) + context-limit checkpoint - -- **Office1 OPNsense edge boots but OPNsense TRIPLE-FAULTS at the BTX loader** (262 bytes: - `/boot.config: -S115200 -h -D` then dies loading /boot/loader). Double-nested virt - (OPNsense -> vcloud VM -> outer hypervisor), AMD Opteron_G3. **INCIDENT REPORT (read - first on resume): `docs/incident-20260712-opnsense-edge-boot-triplefault.md`.** -- **DOCFIX-187 (committed): opnsense-edge module fixes** -- serial console (nano is - serial-only; was the module's biggest gap), q35->i440fx, COW-overlay->direct-copy disk, - host-passthrough CPU + AMD `svm` disabled. All correct + kept; NONE resolved the boot. -- **KEY GOTCHA:** `machine`/`cpu` are create-time; the provider plans them in-place and - libvirt ignores that -- must `virsh destroy; virsh undefine; tofu apply` to recreate. - Provider CPU-feature key is `features` (plural); `feature` silently drops. -- **NEXT (ranked, in the incident report):** full CPU flags (+kvm_pv_eoi,+kvm_pv_unhalt via - qemu_commandline), add a video device (`-D` dual-console + no VGA -> boot2 may fault), - mem 2G->4G, UEFI/OVMF, or outer-hypervisor CPU investigation. -- **Live state:** `office1-opnsense` domain paused/faulted; `office1-wan` NAT net up; - Stage-1 13 objects + edge disk/config-seed volumes exist. Creds `~/vr1-office1-creds/` - (revoke at close, + the pasted NetBox token). apparmor rule for /var/lib/libvirt/vr1/** - applied by operator (needed for any VR1 VM). -- **CONTEXT-LIMIT CHECKPOINT.** Session near max; resume in a fresh session from the incident - report + this ledger. **Numbers:** D next-free 112, DOCFIX next-free 188. - -## Jumphost stream -- session 2026-07-12 (cont.): BOOT INCIDENT ROOT-CAUSED (DOCFIX-188) - -- **The triple-fault was NEVER a CPU/nesting/machine/console problem. The guest had 2 MiB - of RAM.** `dmacvicar/libvirt` >=0.9 changed `memory` from MiB (0.8-era meaning, which the - modules were authored against) to raw libvirt units defaulting to **KiB**. No - `memory_unit` => `memory = 2048` rendered `2048` => QEMU - **`-m size=2048k`** = 2 MiB. Measured every hop: module input -> tofu state -> domain XML - -> `virsh dominfo` (`Max memory: 2048 KiB`) -> live QEMU cmdline. `boot2` fits in 2 MiB - and echoes `/boot.config`; `/boot/loader` does not fit -> deterministic 262-byte fault, - immune to every knob the prior window turned. -- **FOUNDATIONAL, not local to OPNsense:** the identical one-line defect was latent in ALL - THREE VM modules -- `opnsense-edge` (live blocker), `cloudinit-vm` (MAAS/NetBox/GitBucket, - the literal NEXT step), `node-vm` (Stage 3 DC nodes). Every VR1 VM would have gotten 1024x - too little RAM. Only opnsense-edge had been instantiated, which is why it alone detonated. - Same class as the apparmor finding: it gates every VR1 VM. -- **FIXED (DOCFIX-188):** `memory_unit = "MiB"` on all three domains; `memory_mib = 2048` - value untouched (it now means the 2 GiB always intended). Disproven `svm`-is-root-cause - comment in opnsense-edge CORRECTED (setting retained as legitimate hardening; the false - claim would have misled the next session). Incident report gains a READ-THIS-FIRST box so - nobody re-chases the CPU theories. -- **GUARDED (DOCFIX-188):** `tofu validate` CANNOT catch this (`memory_unit` is optional => - omitting it is schema-valid -- exactly how it shipped). Added `opentofu-validate.sh` **S1** - static guard + `--check-memory-unit` flag (no tofu binary/network needed) + harness T3/T4/T5. - **T3 is a true negative test:** the fixture IS the shipped defect and S1 rejects it. - Harness 1 -> 4 PASS. Gauntlet ALL GREEN (52). repo-lint 0 fail. -- **NEXT (operator-gated live step -- the ONLY live mutation pending):** recreate the edge. - `tofu plan` says `0 add / 1 change / 0 destroy` (disk volume safe) BUT **do not trust - "update in-place"** -- max-boot-memory is create-time to libvirt, the same trap DOCFIX-187 - logged for `machine`/`cpu`. Must be: `virsh destroy office1-opnsense; virsh undefine - office1-opnsense; tofu apply`. Re-run `plan` AFTER the undefine -- the captured in-place - plan is NOT what executes; post-undefine it becomes `1 to add`. **Pass criteria (NOT - "apply exited 0"):** (1) AUTHORITATIVE -- `virsh dominfo` shows `Max memory: 2097152 KiB`. - Do NOT anchor on the QEMU `-m` string: libvirt may normalize `MiB` to `-m size=2097152k`, - which is SUCCESS. (2) serial log grows past 262 bytes; (3) router actually works -- WAN - 172.30.1.2 up, LAN DHCP serving `office1-local`. FALLBACKS: wrong size => provider is not - honouring `memory_unit`, use `memory = var.memory_mib * 1024`; boots but no working router - => SIZING question (verify OPNsense 26.1's real minimum -- the incident report's "3 GB" is - UNVERIFIED), bump to 4096; that is not a regression of this fix. - Then: the Office1 NetBox VM (which `cloudinit-vm` now no longer breaks on). -- **STANDING LESSON:** a bootloader dying at a FIXED byte offset, immune to every knob, is a - RESOURCE problem, not a CPU-feature problem. Read the domain XML + QEMU cmdline (ground - truth) before theorising about nested virt. -- **LOGGED-NOT-ACTIONED (DOCFIX candidate, no number consumed):** `run-tests-all.sh` leaves - the working tree dirty -- it `chmod +x`'s the `tests/*/fakebin/*` stubs, so a gauntlet run - produces **37 spurious `100644 -> 100755` mode-change diffs** (zero content change). Had to - `git checkout` them out of this session's commit. Fix is either commit the stubs 0755 or - have the harness restore the mode; either way a gauntlet run should not dirty the tree. -- **RE-AUDIT of the max-context session (DOCFIX-189, operator-instructed).** That session - authored docs+module comments *while working from a wrong diagnosis*, so its reasoning was - re-checked, not just its one-line bug. Findings: - - **4 FALSE CAUSAL CLAIMS corrected at the source** (opnsense-edge/main.tf): the disk shape, - q35->i440fx, the serial console, and svm-disable were EACH written up as a contributing - cause. **None changed the symptom** -- the fault sat at a deterministic 262 bytes through - all of them. All four SETTINGS are RETAINED (each defensible on its own merits); only the - false justifications are gone. The serial console genuinely earned its keep -- it made the - fault legible -- but it revealed the fault, it did not change it. - - **ONE SILENTLY DEAD KNOB (real functional defect):** `disk_size_bytes = 16 GiB` in main.tf - did NOTHING once DOCFIX-187 made the disk a direct copy -- unused variables are legal HCL, - so validate/plan never complained. MEASURED: live disk is **11.00 GiB**, not 16. Removed - from main.tf + module variables; sizing is really `opnsense-prep-image.sh`'s GROW (+8G). - Proof it was dead: `tofu plan` is byte-identical before/after removal. - - **`Opteron_G3` is a RED HERRING:** the host is really an **AMD EPYC 9965 (Zen 5, fam 26)**; - libvirt's CPU-model DB doesn't know fam 26 and falls back to the oldest name. The prior - session partly built its nested-virt theory on this artifact. Read /proc/cpuinfo. - - **"OPNsense 3 GB min" is UNVERIFIED** (no source) -- annotated, do not propagate. - - DOCFIX-186's `backing_store` parenthetical is STALE (disk is a copy now) -- same - conclusion, different mechanism; annotated. DOCFIX-187 changelog gains a - SUPERSEDED-IN-PART header. DOCFIX-186's apparmor + config-iso-staging findings are REAL - and STAND. DOCFIX-185 (operator transport ruling) untouched. - - **STANDING LESSON:** when a change does NOT fix the symptom, do not write it up as though - it did. Four successive non-fixes were each recorded as a cause, and that accumulated - fiction is what a fresh session would have inherited. A fault that is deterministic and - immune to every knob means you have not yet touched the cause. -## Jumphost stream -- session 2026-07-12 (cont.): edge BOOTS. Both boot bugs closed (DOCFIX-190) - -- **LIVE APPLY EXECUTED (operator-gated, 3 recreates).** DOCFIX-188's memory fix went live: - `Max memory: 2048 KiB -> 2097152 KiB`, QEMU `-m size=2097152k`, domain `running (booted)` - not `paused`, serial 262 -> 3882 bytes. **Triple-fault CONFIRMED DEAD.** -- **SECOND BUG, found underneath it (DOCFIX-190): ACPI was OFF.** With RAM finally correct - the kernel reached interrupt init and PANICKED: `panic: running without device atpic - requires a local APIC` -> parked at ddb `db>` = the 100%-of-one-core spin, while `virsh` - still cheerfully reported `running`. Cause: NO `features` block in ANY of the 3 VM modules - -> libvirt renders `-machine ...,acpi=off`. FreeBSD gets its local APIC from ACPI's MADT - and OPNsense has no `atpic` fallback. FIXED: `features = { acpi = true, apic = {} }` in - opnsense-edge + cloudinit-vm + node-vm. **node-vm matters most long-term: MAAS drives - power via ACPI, so without it a graceful shutdown/MAAS power-off would NEVER work.** -- **GUARD S2** added to opentofu-validate (same reason as S1: `features` is optional, so - `tofu validate` structurally cannot see its absence). Flag renamed `--static-only` - (`--check-memory-unit` kept as alias). Harness 4 -> **6 PASS**; `fixtures/s2-bad` PASSES S1 - and still FAILS S2, proving the guards catch independent classes. -- **RESULT: OPNSENSE BOOTS AND RUNS.** acpi=on; kernel to userland; CPU idle (~19s, no spin); - **DHCP lease 172.30.1.126, hostname `OPNsense`** on office1-wan. `BIOS 639kB/2096108kB - available memory` re-confirms the 2 GiB. -- **STILL OPEN -- Configuration Importer did NOT apply.** OPNsense came up on FACTORY - DEFAULTS (WAN=DHCP) instead of static 172.30.1.2 / LAN 10.10.0.1. The ISO is VERIFIED - well-formed (ISO9660 `OPNSENSE_CFG`, contains `CONF/CONFIG.XML` with `` + - `10.10.0.1`) and attached as SATA cdrom -- so the payload is right, the Importer just did - not consume it. This is exactly the mechanism the module header has ALWAYS flagged - UNVERIFIED. **Do NOT guess the fix** -- read the Importer's own early-boot console output. - Untested hypothesis only: ISO9660 8.3 UPPERCASE vs the lowercase `/conf/config.xml` it - looks for. **The unanswered WAN ping is NOT a fault** -- OPNsense blocks inbound on WAN by - default; do not chase it. -- **OPS GAP (contributed to the original misdiagnosis):** libvirt creates the serial log - `root:0600` and RECREATES it on every domain create, so the most valuable boot-incident - artifact needs an interactive `sudo` every retry. A default ACL does NOT work: the 0600 - creation mode sets the ACL **mask** to `---`, nullifying the named-user entry (measured: - `user:jessea123:r-- #effective:---`). Proposed (NOT yet verified): pre-create the log 0644 - before `tofu apply` so libvirt appends to an already-readable file. -- **BUG 3 ROOT-CAUSED -> D-112 PROPOSED (operator ruling required; NOT implemented).** The - Configuration Importer **can never fire on a pre-installed nano image, BY DESIGN.** Upstream - `opnsense-importer -b` does `touch /.probe.for.readonly` -- a PROBE FOR A READ-ONLY ROOT -- - and `if [ -f ${INSTALL} -a -f /conf/config.xml ]; then bootstrap_and_exit 0; fi`. On INSTALLER - media root is read-only so the touch fails and it scans media (cd9660 supported). On our nano, - root is WRITABLE and a factory `/conf/config.xml` ALREADY EXISTS -> both true -> it exits - immediately, never enumerating a device. Boot log confirms: `>>> Invoking import script - 'importer'` then NOTHING. Our ISO was fine all along (verified: ISO9660 `OPNSENSE_CFG`, - `CONF/CONFIG.XML` with 10.10.0.1; guest sees it as cd0, exact byte size) -- nothing was ever - going to read it. - - The README research (items 2-3) correctly established the Importer supports ISO9660; it NEVER - established the Importer RUNS on a nano. The module's own UNVERIFIED flag was right, and is - now discharged NEGATIVE. - - **D-112 RULED: option (c)** -- post-boot network provisioning (operator: "C since it is the way - people automate opnsense"). The rendered config (`opnsense-config.xml.tmpl`) is REUSED - unchanged; only the DELIVERY changes. IMPLEMENTATION NOT STARTED. - - **SELF-CORRECTION (flagged to operator before the ruling):** I had described option (b) as - making the ISO mechanism "work as designed, unchanged". That was OVER-CLAIMED. Even with a - read-only root the importer prompts "Press any key..." with a 7s timeout and then asks for a - device name -- i.e. it appears INTERACTIVE BY DESIGN, so (b) may have been (d) in disguise. Not - verified either way; recorded so nobody revives (b) on my bad summary. - - **BOOTSTRAP PROBLEM -- UNSOLVED, gates (c), sub-choices B1/B2 UNRULED (see D-112).** "Push the - config over the API" is NOT reachable from a factory-default box: (1) NO ROUTE -- default LAN is - 192.168.1.1/24 on `office1-local`, which is an ISOLATED libvirt net with NO host leg on virbr2 - (measured); WAN is reachable at 172.30.1.126 but OPNsense BLOCKS ALL INBOUND ON WAN by default, - so that is not a way in. (2) NO CREDENTIALED API -- the REST API needs an API key+secret (not - the root password), minted in the GUI; and SSH is DISABLED by default. (3) OUR SERIAL IS - WRITE-ONLY -- the module attaches `` (one-way capture); it cannot be typed - into. Making it interactive = `type='pty'` + `virsh console` (module change + recreate). - NOTE: a console *login shell* is NOT the rejected option (d) -- (d) was rejected for racing the - importer's 7s timeout; a login shell has no timing race. Do not conflate them. - - Impacts if not (b): `opnsense-build-config-iso.sh` + harness, the module's config_seed/cdrom, - the README research section. ALSO GATES STAGE 3 (per-DC edges use this same module). -## Jumphost stream -- 2026-07-12 (cont.): D-112(c) EXECUTED. THE OFFICE1 EDGE IS UP AND ROUTING. - -- **END STATE (measured over SSH):** `office1-opnsense.office1.vr1.cloud.neumatrix.local`, - OPNsense 26.1. WAN `172.30.1.2/24` static, LAN `10.10.0.1/24`, default route via - `172.30.1.1`, NAT active, **egress to the internet WORKS** (`ping 1.1.1.1` 0% loss through - the simulated ISP). Serial console restored (`kern.console=ttyu0`, getty running). sshd up - with the service key (we manage it over SSH now). - (NOTE: the `sshd: DOWN` line in the verify sweep was MY CHECK BEING WRONG -- it ran over an - SSH session, so sshd was obviously up; the pgrep pattern just did not match. Not a fault.) -- **Bootstrap executed:** console (pexpect + `virsh console`) -> factory login -> config - streamed as base64, SHA-256 verified byte-exact, factory config backed up -> reboot. Then - the DOCFIX-192 fix was pushed **over SSH** -- i.e. the D-112(c) steady-state path proving - itself. B1 host leg `10.10.0.10/24` on virbr2 is IN PLACE (non-persistent; re-add after a - host reboot: `sudo ip addr add 10.10.0.10/24 dev virbr2`). -- **DOCFIX-191: the config would have LOCKED MANAGEMENT OUT.** Template had - `admins` = OPNsense's DISABLED-sshd shape, and NO authorized key. - Applying it would have left the edge with sshd off and no key. Fixed (`enabled` - + `` + `` base64). Names VERIFIED upstream, not guessed. -- **DOCFIX-192: the config SILENCED ITS OWN ONLY CONSOLE.** No ``/``, - and the edge VM has **NO VIDEO DEVICE**, so serial is the only console. After the first reboot - the console went permanently silent. **The box survived ONLY because DOCFIX-191 had just turned - SSH on -- one lockout bug saved us from the other.** Fixed; console restored. - - **KEY GOTCHA (cost a confused cycle):** console settings take TWO reboots. OPNsense - regenerates `/boot.config` + `/boot/loader.conf` FROM config.xml *during* boot, so they only - take effect on the NEXT boot. After reboot #1 the loader files were already correct while the - running kernel was still on video. Do not conclude "the fix failed" after one reboot. - - `/etc/ttys` uses `onifconsole`, so getty starts on ttyu0 ONLY if ttyu0 is the kernel console. - `sysctl kern.console` is the ground truth. -- **GUEST SHELL IS CSH.** Both `virsh console` and `ssh root@` land in csh, which rejects - `$(...)` ("Illegal variable name."). Pipe POSIX scripts to `sh -s`. This silently broke two - scripts before it was spotted. -- **PROVIDER GOTCHA:** `virsh console` REFUSES a unix-socket chardev ("character device serial0 - is not using a PTY"). The module's serial must be a **pty** + a `log` file. A unix socket is - also unreachable in practice (libvirt-qemu:kvm, and the staging default ACL stamps a named-user - entry that outranks group perms -> r-- only). `virsh console` via libvirtd needs no extra privs. - The provider's `source.pty` requires a `path`, which libvirt allocates at start -- so OMIT - `source` entirely to get libvirt's default ``. -- **OPEN GAP -- DHCP WAS NEVER IMPLEMENTED (the ledger claim below was FALSE).** The 2026-07-12 - entry says "LAN 10.10.0.1/24 + DHCP .100-.199". **There is NO DHCP configuration in the template - at all** -- no ``, no `` (measured: template top-level sections are filter, gateways, - interfaces, nat, ntpd, OPNsense, rrd, staticroutes, system, theme, unbound). Nothing serves DHCP - on `office1-local` (measured: no kea/dhcpd/dnsmasq process, nothing on udp/67). Another optimistic - claim from the max-context session. **The operator's original goal was "router + DHCP" -- the - ROUTER half is done, the DHCP half is NOT.** OPNsense 26.1 uses **Kea** (ISC dhcpd is gone), so - this needs the Kea schema verified upstream (do NOT invent it) + a harness assertion. -- **~/vr1-office1.env created (0600, jumphost-only, NEVER COMMIT).** Contains NO secrets: it READS - `~/vr1-office1-creds/*` at source time, so there is exactly one copy of key material on disk. - Carries the as-built values + `$OPNSENSE_SSH_KEY` / `$OPNSENSE_CONFIG_OUT` handles. -- **Numbers:** D next-free 113, DOCFIX next-free 193. - - - - -## Jumphost stream -- 2026-07-13: DHCP APPLIED. The Office1 edge is a ROUTER + DHCP SERVER. - -- **DOCFIX-193 APPLIED to the live edge** (was built-but-not-applied at last session's close). - `kea-dhcp4` is running and **bound udp4 10.10.0.1:67**; subnet `10.10.0.0/24`, pool - `.100-.199`, routers/DNS `10.10.0.1`. Router unaffected: WAN 172.30.1.2, LAN 10.10.0.1, - default route 172.30.1.1, NAT automatic, egress 0.0% loss, serial console + getty alive. - Full record: `docs/changelog-20260713-office1-dhcp-apply.md`. -- **KEY AS-BUILT FACT (do not re-derive):** a full-config push to this edge drops ~667 - migration-populated elements, INCLUDING the only 2 firewall pass rules on the box - ("Default allow LAN to any" + IPv6; legacy `/filter/rule` count is 0 in 26.1). OPNsense - **regenerates them on boot** -- proven from `/conf/backup/` across 4 push+boot cycles, and - re-confirmed post-apply (791 elements, both rules back, TLS cert back, our Kea subnet4 - survived). The overwrite is self-healing. Do NOT hand-merge to "preserve" them. -- **OPEN (logged, not executed):** the apply path is still ad-hoc shell, not a harnessed - script. Wanted: `scripts/opnsense-apply-config.sh` (snapshot -> scp -> SHA verify -> install - -> reboot -> assert kea/udp67 + 2 LAN rules + egress + console). The ad-hoc path already - shipped one defect: root's shell on OPNsense is **tcsh**, so a `$(...)` in a quoted remote - command dies with "Illegal variable name" -- the pre-install snapshot silently no-op'd and - the install ran without it. Always feed remote commands to `sh -s`. -- **NOT verified end-to-end:** no client has taken a lease -- office1-local has no host yet. - First Stage-2 host on that LAN is the real proof. -- Platform question (operator asked): recommendation is **stay on OPNsense**, not VyOS -- - reasoning in the prior session's close-out; would be D-113 if it needs a formal ruling. - NOT raised as a decision; no D-number assigned. - -- **Unattended git push now works (SEC-005).** `credential.helper store` + a dedicated - GitBucket PAT at `~/.git-credentials` (mode 600). The branch had been stuck 12 commits - ahead of origin for two days; now pushed (`425d434`, ls-remote confirmed). Two gotchas, - both measured, in `docs/changelog-20260713-git-credential-store.md`: - (a) the git username is **`jesse.austin`** -- NOT the web-UI login email, NOT `JANeumatrix`; - (b) GitBucket 401s PAT basic-auth on `/api/v3/` but ACCEPTS it for git transport, so - validate creds against `info/refs?service=git-receive-pack`, never the API; - (c) `store` DELETES the credential on a 401, so a rejected token silently empties the file. - SEC-005 is OPEN: plaintext account-wide token on the jumphost -- revoke/rotate at v1 close. - -- **D-113 PROPOSED (2026-07-13, UNRULED):** is OPNsense the right edge platform? Written up - with alternatives (a1 OPNsense+config.xml = status quo, a2 OPNsense+REST API, b VyOS, - c plain-Linux router, d pfSense=reject). **Key finding while drafting: D-112 adopted (c) on - the rationale "the way people automate opnsense" = the REST API, but what we BUILT is - full-config.xml templating over scp -- which satisfies (c)'s letter, not its rationale. - All three OPNsense-specific bugs (DOCFIX-191/192/193) plus the 667-element self-heal - dependency are artifacts of hand-authoring XML -- exactly what the API exists to avoid.** - So the real question is a1 vs a2, not OPNsense vs VyOS. My lean: stay on OPNsense, move to - (a2). NOT RULED -- operator decides. Rule BEFORE Stage 3 (it builds 2 more edges from this - same module; ruling after means migrating 3 edges instead of 1). - -- **D-113 RULED 2026-07-13: option (a2) -- stay on OPNsense, MOVE CONFIG TO THE REST API.** - Two operator constraints narrowed the field decisively: (1) **GUI access is a REQUIREMENT** - -- this eliminates VyOS and plain-Linux, which had been the leading candidates; (2) VyOS LTS - binaries are subscription-gated (verified), so free VyOS = rolling, fighting appendix-B - version pinning. OpenWrt was the credible alternative (UCI is text-first by design) but costs - a new pipeline + bootstrap + dnsmasq to escape a problem fixable in place. (a1) config.xml - templating is now explicitly REJECTED. - **Next: retire hand-authored config.xml.** Bootstrap config keeps ONLY what the API cannot - set on a factory box (sshd + root key + console + the API key); everything else -- DHCP, - firewall, interfaces -- moves to the API/Ansible collection. Live edge recon (2026-07-13): - lighttpd IS listening on 443/80, the API answers 401 (alive), root has 0 API keys -> minting - one via the GUI is the bootstrap step (the supported path; hand-editing apikeys into - config.xml would be the very anti-pattern being retired). - -- **D-113(a2) step 1 SHIPPED:** `scripts/opnsense-api.sh` + `tests/opnsense-api/run-tests.sh` - (21 PASS; gauntlet 53 ALL GREEN). Thin REST client; secret passed to curl via `--config` on - STDIN so it never lands in argv/`ps` (harness T9 enforces this -- a future `curl -u` - "simplification" turns it red). Host never inferred (T3). `--insecure` is deliberate: the - edge's self-signed cert is REGENERATED EVERY BOOT, so pinning is pointless; scoped to the - private lab LAN leg only. - **BLOCKED ON:** an API key. Measured: API is alive (401 from lighttpd on 443) but root has - **0 apikeys**. Minting is a GUI action (System > Access > Users > root > API keys > "+") -- - deliberately NOT automated, since hand-writing apikeys into config.xml is the exact - anti-pattern D-113 retired. Key file -> `~/vr1-office1-creds/opnsense-api.txt` (operator-only, - never into agent context; SEC-007 covers that dir). - **Next after the key:** drive yesterday's DHCP subnet through the API as the proof-of-path. - If it round-trips, the template retires to a MINIMAL bootstrap (sshd + key + console + apikey) - and DHCP/firewall/interfaces move to the API. If it cannot express it, D-113 gets revisited. - -- **D-113(a2) PROVEN 2026-07-13 against the LIVE edge.** API key minted; `GET - core/firmware/status` -> 200; `POST kea/dhcpv4/search_subnet` reads back EXACTLY yesterday's - hand-pushed DHCP (10.10.0.0/24, pool .100-.199, routers/DNS 10.10.0.1) as TYPED fields. - **GUI gotcha (26.1):** API keys are NOT created in the user-edit dialog nor the ApiKeys tab - (that tab is search+delete ONLY). They are a ROW ACTION on the Users list -- the `fa-ticket` - icon. The secret is shown ONCE (stored `crypt($6$)`). - **API gotchas:** response root key is `dhcpv4` (not `dhcp4`); camelCase actions map to - snake_case URLs (`searchSubnetAction` -> `search_subnet`). - **NOT done:** no WRITE through the API yet (gated live mutation). **The config.xml template is - still live -- DO NOT push a full rendered config.xml now; it would clobber API-managed state.** - Next: reduce the template to a MINIMAL bootstrap (sshd + key + console) and move - DHCP/firewall/interfaces to the API. - -- **D-113(a2) WRITE PATH PROVEN 2026-07-13 (live, gated).** `POST kea/dhcpv4/set_subnet/` - -> `{"result":"saved"}`; `POST kea/service/reconfigure` -> `{"status":"ok"}`. GROUND TRUTH - (not the API's self-report): kea-dhcp4 RESTARTED (pid 30027), bound udp4 10.10.0.1:67, and - Kea's OWN generated `/usr/local/etc/kea/kea-dhcp4.conf` carries subnet 10.10.0.0/24, pool - .100-.199, routers/DNS 10.10.0.1. Router unaffected (egress 0.0% loss, 8 LAN pass rules). - **Full chain works: API write -> OPNsense renders -> Kea loads -> daemon serves.** - **PAYLOAD GOTCHA (measured):** a GET response is NOT directly re-POSTable. GET returns select - fields as `{value:{selected:0|1}}`; SET expects comma-joined selected values -- must be - flattened. `kea/service/reconfigure` is the apply verb; without it the config saves but the - daemon never reloads. - **CONSEQUENCE -- the config.xml push path is RETIRED IN PRACTICE for Office1.** DHCP is now - API-managed; a full rendered config.xml push WOULD CLOBBER IT. Do NOT run the old - scp/install/reboot path against this edge. Template reduction to a minimal bootstrap (sshd + - key + console) is the next step and is NOT yet done. - -## D-113(a2) COMPLETE 2026-07-13: the config.xml path is DELETED - -- Operator ruling ("I'll defer to your lean"): the template is **DELETED, not reduced**. Reducing - it to a minimal bootstrap proved UNNECESSARY -- console bootstrap gives sshd+key, - `opnsense-bootstrap-apikey.sh` gives the API key, the REST API gives everything else. **The - provisioning chain has NO config.xml in it anywhere.** -- **DELETED:** `opentofu/templates/opnsense-config.xml.tmpl`, `scripts/opnsense-render-config.sh`, - `scripts/opnsense-build-config-iso.sh`, both their harnesses (gauntlet 54 -> 52, expected), the - `opnsense-edge` module's `config_seed` volume + cdrom disk + `config_iso_path` var, and the - xorriso/genisoimage prereq (it existed ONLY for the ISO builder). All in git history. - `opentofu/templates/README.md` is now a TOMBSTONE -- people will go looking for that template. -- **THE LIVE CHANGE (measured BEFORE applying, with `tofu plan`):** - `libvirt_domain.vm will be updated IN-PLACE` + `config_seed will be destroyed`; - `Plan: 0 to add, 1 to change, 1 to destroy`. **NO replacement of the running domain** -- a - replacement would have DESTROYED the live edge. Applying now is the SAFE choice: leaving the - repo/state divergence would mean the next `tofu apply` (e.g. Stage 3's DC1 build) sweeps it up - as a SURPRISE mutation on the running Office1 edge, mid-stage. -- **A guard did its job:** repo-lint's L3 (runbooks must not reference missing scripts) went RED on - the tombstone notes. It was RIGHT and has no opt-out. Rather than weaken it, the tombstones were - reworded to name the bare filename instead of the `scripts/` path. Guard intact. -- **APPLIED 2026-07-13. AS-BUILT FACT, LEARNED THE HARD WAY -- "updated in-place" DOES NOT MEAN - "no restart".** The apply **RESTARTED the guest** (uptime 8:36 -> 6 secs); it was predicted not - to, and that prediction was WRONG. The libvirt provider redefines the domain to apply a disk-list - change and BOUNCES it: a ~30s outage with no routing and no DHCP. - `libvirt_domain ... will be updated in-place` means the RESOURCE is not replaced -- it says - NOTHING about the guest staying up. **STAGE 3 CONSEQUENCE:** assume ANY `tofu apply` touching a - `libvirt_domain`'s devices (DC edges, node VMs) BOUNCES THE GUEST. Schedule/gate it as an outage, - never as a no-op tidy-up. - Recovery was clean, no intervention: kea-dhcp4 back on udp/67 with subnet/pool intact, routing + - egress 0.0% loss, 8 pass rules, console + getty. `tofu plan` -> "No changes" (state in sync). -- `docs/changelog-20260713-config-xml-path-deleted.md`. D-113 AMENDED in design-decisions. - -## D-113(a2): API-KEY BOOTSTRAP SOLVED 2026-07-13 (no GUI click, no re-implemented crypto) - -- **The last unknown in edge provisioning is closed.** `scripts/opnsense-bootstrap-apikey.sh` + - `scripts/opnsense-mint-apikey.php` (harness 8 PASS; gauntlet 54 ALL GREEN). Ships the minter to - the edge over SSH and calls **OPNsense's OWN model** (`Auth\User -> apikeys->add()`) -- the exact - code path the GUI ticket button invokes. -- **VERIFIED END TO END on the live edge:** minted a 2nd key -> `GET core/firmware/status` HTTP 200 - (it AUTHENTICATES) -> `del_api_key` -> 1 key remaining (operator's original, untouched) -> the - retired key is REJECTED (negative test). Temp creds shredded. Edge back to prior state. -- **DESIGN NOTE (do not "optimize" this away):** OPNsense stores API secrets as - `crypt(secret,'$6$')` (SHA-512, EMPTY salt -- verified: `$6$$` + 86 chars). We COULD mint keys - offline and seed them into a bootstrap config.xml. **We deliberately DO NOT** -- it would couple - provisioning to a vendor hash format we must keep byte-compatible forever, and a mismatch FAILS - SILENTLY (keys that never authenticate). Call the interface; do not re-implement the internals. - (An offline-hash attempt was blocked by the permission classifier mid-session. The block was - RIGHT and produced the better design.) -- **GUARD (harness T5):** the script REFUSES to overwrite an existing creds file. An existing file - means a LIVE key; overwriting the local copy STRANDS it on the edge forever (the secret is - hashed and can never be read back) = unrecoverable credential loss. -- **THE PROVISIONING CHAIN IS NOW HUMAN-FREE:** boot factory nano -> D-112(c) console bootstrap - (SSH + key) -> `opnsense-bootstrap-apikey.sh` -> `opnsense-api.sh` for DHCP/firewall/interfaces. - **There is no config.xml anywhere in that chain.** -- **OPEN QUESTION THIS RAISES:** the template may not need REDUCING -- it may need DELETING. The - console bootstrap + the key minter cover first contact; nothing else needs a config.xml. NOT - ruled; do not delete unilaterally (the `opnsense-edge` module's `config_seed` volume is - INSTANTIATED -- in tfstate -- so removing it is a LIVE change, not a docs change). -- `docs/changelog-20260713-opnsense-apikey-bootstrap.md`. - -## WORKFLOW-DOC STATUS CORRECTED 2026-07-13 (docs only) - -- `docs/dc-dc-deployment-workflow.md` (the doc a fresh session reads FIRST) had 2026-07-09 status - claims that were now materially FALSE. Corrected against measured state: gap #2 (tofu binary - EXISTS, v1.12.3; tree validates 10/10 and is partly APPLIED), gap #17 (**CLOSED for Office1** -- - `office1-wan` virbr11 NAT 172.30.1.0/24 carries the live edge WAN; **DC1/DC2 still have none**), - the opnsense-edge / config.xml / netem-link status rows, and Stage 2's gate + state. -- **THE TRAP REMOVED:** Stage 2's gate lists "NetBox authoritative" + "GitBucket serving" -- and - both DO answer at baldurkeep.com. **But those are PRE-EXISTING services, NOT Office1 headend - VMs.** Measured: `office1-opnsense` is the ONLY VM on the vcloud host. It would be easy to tick - that gate as met and walk into Stage 3 with no headend. The corrected State section says so. -- **What actually remains of Stage 2:** the three headend VMs (MAAS-region, NetBox, GitBucket) - DO NOT EXIST. Blocked on a real decision (cloudinit-vm user_data/meta_data/network_config + - image source, vs manual virt-install as debt) -- not on typing. -- `docs/changelog-20260713-workflow-doc-status-correction.md`. - -## DOCFIX-194 2026-07-13: the OpenTofu gate was GREEN over two BROKEN modules - -- **`scripts/opentofu-validate.sh` printed PASS while `node-vm` and `netem-link` were flatly - broken.** Root cause: `tofu validate` on the ROOT module never parses a module the root does - not instantiate. Root calls dc-planes/dc-storage-pool/mesh-link/office1-network/opnsense-edge. - It does NOT call **base-image, cloudinit-vm, maas-vm-host, netem-link, node-vm** -- exactly the - set Stage 2 and Stage 3 are built on. Never parsed by any tool, ever. -- **node-vm:** `libvirt_volume` had a top-level `format = {...}`. Schema-verified - (`tofu providers schema -json`, dmacvicar/libvirt v0.9.8): **there is no top-level `format`** -- - it nests under `target`. base-image had it right all along. **node-vm builds EVERY DC node**; - Stage 3/4 would have failed on first use. -- **netem-link:** a destroy provisioner referenced `var.*`. OpenTofu rejects that at INIT -- the - module could not even initialize. It is the DR/latency mechanism for Stage 3 + the Stage 6 - failover drill. Fixed with the canonical `input` + `self.input.*` pattern. -- **Durable fix -- S3:** the gate now validates EVERY module standalone (in a temp copy, so it - never pollutes the repo). Harness 9 PASS; **T8 is the load-bearing case** (root VALID, uncalled - module BROKEN -> gate must FAIL). If T8 ever goes green, the blind spot is back. -- Only findable because a **tofu binary now exists** on the jumphost (v1.12.3). The tree was - authored with no binary to self-check -- precisely the risk `opentofu/README.md` flagged. -- **A gate that reports green over unparsed code is worse than no gate.** -- `docs/changelog-20260713-docfix194-opentofu-module-validation.md`. - -**STALE DOC NOTE (not yet fixed):** `docs/dc-dc-deployment-workflow.md`'s status table still says -gap #2 is open ("no tofu binary"), the tree is UNVALIDATED, and gap #17 (no ISP-uplink network) -is open. All three are now FALSE: tofu v1.12.3 is installed, the tree validates (10/10 modules), -and `office1-wan` exists. Fixing that table is queued, NOT done. - -## SAFETY SWEEP 2026-07-13 (done after the checkpoint, operator away, repo-only) - -- Marked every instruction that still said "render config.xml and push it to the edge" as - DANGEROUS -- it would now CLOBBER Office1's API-managed DHCP. 5 files: both dc-dc runbooks, - both opnsense config scripts (point-of-use headers), and the workflow doc's gap register. -- **Caught a PRE-EXISTING landmine:** `dc-dc-phase2` Step 4 still told you to build a **config - ISO** -- which D-112 proved can NEVER be read on a nano image. That runbook has been directing - people straight into the failure that cost 2026-07-12. Now marked STOP at the source. Its - WAN_IF/LAN_IF "chicken-and-egg" note is moot for the same reason. -- Changelogs deliberately NOT touched (history, not instructions). -- Harnesses green (24/2/21 PASS), lint 0 fail. No live system touched. -- `docs/changelog-20260713-config-xml-danger-sweep.md`. - -## CHECKPOINT 2026-07-13 (session close; operator away, resumes in the morning) - -**LIVE STATE (measured, end of session):** the Office1 edge `office1-opnsense` (10.10.0.1 LAN / -172.30.1.2 WAN) is UP: routing, NAT (automatic), egress 0.0% loss, serial console + getty, -SSH key management, and **kea-dhcp4 serving DHCP on udp/67** (pool .100-.199). 8 LAN pass rules -loaded in pf. Nothing is broken; nothing is mid-flight. - -**WHAT LANDED TODAY** -1. DOCFIX-193 APPLIED -- DHCP live on the edge (was built-but-unapplied at last close). -2. SEC-005 -- GitBucket credential stored; unattended `git push` now WORKS (was the 2-day - data-loss exposure: the branch sat 12 commits ahead of origin). -3. SEC-006/007 -- the two known credential exposures put ON the ledger (they had been prose only). -4. **D-113 RULED (a2)**: stay on OPNsense, move config to the REST API. Driven by two operator - constraints: GUI is a REQUIREMENT (kills VyOS + plain-Linux) and VyOS LTS is subscription-gated. -5. **D-113(a2) PROVEN END TO END on the live edge** -- READ and WRITE. API -> OPNsense -> Kea -> - daemon. `scripts/opnsense-api.sh` + harness (21 PASS) shipped; gauntlet 53 ALL GREEN. - -**THE ONE TRAP -- READ THIS BEFORE TOUCHING THE EDGE** -DHCP on Office1 is now **API-MANAGED**. A full rendered `config.xml` push WOULD CLOBBER IT. -**DO NOT** run the old scp/install/reboot path against this edge. The template has NOT yet been -reduced to a minimal bootstrap -- that is the next step. - -**NEXT STEP (the actual D-113(a2) migration, NOT started):** reduce -`opentofu/templates/opnsense-config.xml.tmpl` to a MINIMAL bootstrap (sshd + root key + console -+ nothing else) and move DHCP/firewall/interfaces to the API. This is what makes Stage 3's two -edges nearly free. Repo work; no live mutation required to land it. - -**Open, unruled:** nothing blocking. SEC-006 (burned NetBox token) is DEFERRED by operator ruling -to deployment completion. +1. **THE EDGE TRAP -- DHCP on Office1 is API-MANAGED.** A full rendered `config.xml` push WOULD + CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all + gone -- `opentofu/templates/README.md` is a tombstone). Configure the edge via the REST API + (`scripts/opnsense-api.sh`); mint keys with `scripts/opnsense-bootstrap-apikey.sh`. +2. **"updated in-place" DOES NOT mean "no restart".** A `tofu apply` whose plan says + `libvirt_domain.vm will be updated in-place` **RESTARTED the guest** (measured: uptime 8:36 -> + 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it + says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to + apply a disk-list change and BOUNCES it. **Assume ANY apply touching a `libvirt_domain`'s + devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.** +3. **Do not re-implement vendor internals; call the interface.** OPNsense stores API secrets as + `crypt(secret,'$6$')` (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we + deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep + byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate). +4. **A config ISO can NEVER be read on an OPNsense nano image** (D-112). Any instruction to build + one is a trap; the dc-dc-phase2 runbook directed people into it for days. +5. **Guide changes must sweep the tenant AI skill in the SAME change.** Skill-lag was caught twice + (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 + is pinned, and any guide edit FAILS lint until re-recorded). +6. **The controller model is `admin/controller`,** not `/controller`. +7. **`juju status --format=line` omits workload messages on 3.6** -- this caused a cloud-assert + false negative (fixed by DOCFIX-087). Don't rely on it. +8. **repo-lint's guards are load-bearing; reword rather than weaken.** L3 (docs must not reference + missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, + the guard kept. ## State facts to remember -- beta cluster left at **node_count=2** (deliberate; bonus resize acceptance coverage). -- repo is temporarily **PUBLIC** for Claude web_fetch (SEC-004) -- flip private at v1 close. -- Jumphost stream: see "Active window" section above (ops-update-20260705 in flight). - Vault stays 1.8/stable (D-068 unruled). -- CORRECTION (2026-07-05, jumphost stream, per handoff reconcile item): the two bullets - above are stale -- ops-update-20260705 is CLOSED (fleet fully current, juju 3.6.25); - D-068 is RULED on the 1.16 question (1.16 ruled out; vault stays 1.8/stable rev 714; - the off-EOL-1.8.8 path remains OPEN under D-068). +- **The beta cluster and the foil1 tenant are GONE** (both confirmed absent by the 2026-07-08 + read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is + retired here. `capi-test-1` and the `lbtest` LB leftover are also gone; orphan sweep reads + ORPHAN=0, zero unattached floating IPs. +- **devteam is a LIVE tenant** running a self-created cluster (built via Horizon, calico, + keypair `devteam-key`). Their template has `volume_driver` unset but the cluster DOES get a + default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the + box; no template change needed. +- The vcloud host was NOT pristine: 9 orphaned `vr0-*` libvirt nets were torn down; **`wan` was + KEPT** (dual-stack NAT-out-enp1s0, GUA `2602:f3e2:fe:10::/64`) as the working model for the + per-site ISP-uplink network. +- MTU has TWO domains: the ISP uplink `enp1s0` is 1500; the host-internal plane/dark-fiber fabric + is **jumbo, underlay_mtu=9000** (operator ruling; verified on all 10 nets). +- The repo is temporarily **PUBLIC** (SEC-004) -- flip private at v1 close. +- Secrets live jumphost-local and are NEVER committed: `~/vr1-office1-creds/` (edge SSH key, + root password, API key), `~/.vr1-netbox.env` (NetBox token), `~/tenant-devteam/`, `~/vault-init/`. ## Project-completion (execute after D-011 passes) -- Consolidate 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. -- Set repo visibility PRIVATE (SEC-004). +- Consolidate the 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. +- Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials. - v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. - -### Cross-session pins (append-only) - -- D-071 completion sweep (2026-07-05, DONE): the ops-update window closed; D-071 was AMENDED - (create-backup premise corrected -- `juju create-backup`/`download-backup` DO exist on juju 3.6; - a 902MB controller backup was taken). D-071 REMAINS PROPOSED -- not ratified to ADOPTED. Operator - ratification is pending; the single-controller (no-HA) half of the risk is unresolved by backups - alone. See D-071 + its 2026-07-05 amendment. -- Pre-DC-DC controller HA/backup planning (PINNED): before the Roosevelt (DC-DC) phase, hold a - dedicated planning session on Juju controller High-Availability / backup architecture. Backups - now exist (per D-071 amendment); controller HA is the open design item. This resolves the - single-controller risk recorded under D-071 for production. - -