Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is in flight, so any session -- after a compaction, or a fresh one, or the parallel Claude Code stream -- can resume without losing pending work.

How to use it (standing practice).

1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
4. Numbers and the machine-derived section below are seeded from a scan; re-run the scan and re-seed rather than editing those by hand.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

As of 2026-07-09 scan (post-DOCFIX-161, final consolidation pass); re-run the scan to refresh:

• PROPOSED / OPEN decisions: D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + controller patch policy -- operator to rule; gated on the pre-DC-DC HA/backup session). D-100..D-110 are ALL ADOPTED as of 2026-07-09 (D-102 MERGED into D-101; D-109 confirmed standalone) -- Stage 0 of docs/dc-dc-deployment-workflow.md is CLEARED; see docs/dc-dc-buildout-design.md Section 10 for the ruling record. (D-050 RESOLVED/CLOSED 2026-07-06 via D-051; D-073 APPLIED live 2026-07-06 -- not scan-tracked as open. D-076..D-099 are a RESERVED source-project band, never assigned in this repo, per D-110.)
• OPEN security rows: SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed).
• Next-free numbers: D = 111, DOCFIX = 162, BUNDLEFIX = 012. (Seeded from the 2026-07-09 scan, post DOCFIX-161.)

Active build (this session)

• validate.sh D-011 runner -- modular. Foundation committed: scripts/lib-validate.sh (exit contract 0/1/2/3/4, emit, vr_json stderr-safe, run, env hygiene, disruptive gate) + scripts/validate.sh orchestrator (profiles, verdict, --stop-on-fail, --include-disruptive).
  • Batch 1 committed: checks/d011-01-charms, checks/d011-06-vault-unseal (MANUAL until the SEC-003 rehearsal closes).
  • Batch 2 committed: checks/d011-02-vip-jumphost (catalog-derive public origins, TLS+HTTP), checks/d011-03-vip-tenant (ephemeral agnhost pod egress -> keystone VIP, D-035 proof). post-restart profile now fully populated.
  • Batch 3 DRAFTED (mock-tested only, NEEDS LIVE VALIDATION): d011-04-octavia-lb (RR+member additive; amphora failover --disruptive, N+1 headroom-guarded) + d011-05-magnum-e2e (wraps tenant-acceptance + timing). full-d011 profile now COMPLETE.
  • Next: live-validate batch 3 (see Verify-live queue), then item-3/#5-#8 backlog.
  • D-011 AMENDED bar: 1 charms; 2 VIP jumphost; 3 VIP tenant; 4 octavia RR/failover/recovery; 5 magnum e2e + OCCM; 6 second-person manual unseal (attestation, D-069); 7 DROPPED (D-070 supersedes D-012 snapshots); 8 DROPPED (D-019, no Designate).

Script backlog (changelog-tracked)

• DONE: tenant-offboard.sh, vault-kv-health.sh, cloud-assert.sh (item 3 largely), validate foundation + batch 1.
• item 3 remainder: fold vault-kv-health.sh into cloud-assert as a section; use the combined gate to regenerate the stale restart-procedure runbook.
• item 5: scripts/tenant-cluster-create.sh -- generalize the t2-02 stage6 + watch + D-066-evidence wrapper. Not started.
• item 6: scripts/keystone-policy-drift.sh -- D-051 base_* alignment (LIVE-READ PENDING). Not started.
• item 7: scripts/cloud-snapshot.sh -- juju-export baseline capture (D-070 retired KVM snapshots but KEPT the export/inventory baseline). Not started.
• item 8: tests/tenant-acceptance/ harness -- pending (tenant-onboard harness landed).

Register / operator-gated (see docs/handoff-20260703-open-items.md)

• R-1 / SEC-003: Vault unseal-key custodian assignment (OPEN). Gates V-2 and closes d011-06-vault-unseal (currently MANUAL). Operator input, recorded off-repo.
• V-2: second-person unseal rehearsal (blocked on R-1).
• R-2 (D-043 caveat): capi-mgmt-v2 auto-resume exclusion -- rule or accept.
• R-3: was D-063 -- now CLOSED. The handoff doc still lists R-3 OPEN; update it.
• D-1: Pattern-A full redeploy (VR0 DC0).

Verify-live queue (batch 3 drafts -- confirm on real cloud before trusting)

• d011-04 amphora_headroom() field parsing (server show .flavor; hypervisor list --long vcpu/ram) -- safe default HOLDs on any parse miss, but confirm the OK path live.
• d011-04 OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior.
• d011-05 needs a FOIL tenant (2nd tenant) for P3 isolation -- onboard one or set VR_FOIL_APPCRED (acme, the former foil, was offboarded).
• Sequence: run d011-04 non-disruptively first (RR+member), then --include-disruptive once headroom confirmed.

Logged-not-actioned (small; would vanish at compaction)

• offboard v2 --sweep-magnum-orphans mode (orphan per-cluster trustee in the magnum domain).
• offboard stage-3 app-cred idempotency re-run guard.
• host_href=None barbican observation (exercised fine in stage 6; probably closeable).
• DOCFIX candidate: sweep repo for other -f json ... 2>&1 stderr-merge hazards (DOCFIX-085 class).
• D-050 (keystone policyd override) and the D-063-adjacent identity:list_trusts="" hardening remain PROPOSED/OPEN, not actioned.

Jumphost stream -- ops-update-20260705: WINDOW CLOSED 2026-07-05 (addenda 13-16)

First execution of runbooks/ops-update-procedure.md (DOCFIX-086; addendum 13). Logged session ops-update-20260705; checkpointed here at each milestone for cross-stream sync. State as of the Section 2 entry checkpoint:

• DONE Section 1 (pre-flight): client 3.6.25 / controller 3.6.24 / all 91 agents uniformly 3.6.24 -> target 3.6.25. Worklist = 17 apps. Quiesce clean. Pre-change BOM asbuilt/20260705-102951 committed on a TRUE cloud-assert PASS (required the DOCFIX-087 A7 false-negative fix, addendum 14 -- window was HELD on the operator's Option B ruling until it landed).
• DONE Section 2.1: controller state backup via juju create-backup -m admin/controller (902MB, checksummed, ~/openstack-baseline/, jumphost-only). NOTE: controller model is admin/controller, NOT /controller.
• DONE Section 2.2/2.3: controller upgraded 3.6.24 -> 3.6.25 (blind window ~1 min, as expected); post-controller cloud-assert PASS.
• DONE Section 3: all 91 openstack-model agents + controller machine at 3.6.25, states clean (63 idle / 28 started), nothing stuck.
• DONE Section 4 G0: keystone 778->817; token probe OK; cloud-assert PASS.
• DONE Section 4 G1 (11 apps, standing group approval, serial+gated): placement 125->154, nova-cloud-controller 795->823, neutron-api 650->710, neutron-api-plugin-ovn 178->215, glance 642->681, glance-simplestreams-sync 124->152, octavia-diskimage-retrofit 196->232, cinder 733->820, cinder-ceph 533->568, barbican 209->265, barbican-vault 75->99. All probes OK; boundary cloud-assert PASS.
• DONE Section 4 G2: octavia 441->542; LBs 2/2 ACTIVE/ONLINE; cloud-assert PASS.
• DONE Section 4 G3: dashboards 728->750 / 59->122 / 120->168; D-044 override INTACT; HTTP+login healthy. RCA: VIP HTTPS dead SINCE DEPLOY (haproxy 443 backend targets vhost-less internal addr; L4 check masks; phase-03 3.3 check fails open) -- NOT a G3 regression; G3 stands (operator ruling; addendum 15). Upstream bug + 2 DOCFIX candidates queued post-window.
• DONE (coordinated): vault bundle revert 1.16->1.8/stable (BUNDLEFIX-010) + D-068 AMENDMENT recorded; 1.16 ruled out (certs V0/V1, Raft-only, Ceph, BUSL); "off EOL 1.8" remains OPEN. Evidence: docs/D-068-vault-1.8-vs-1.16-analysis.md.
• DONE Section 4 G4: nova-compute 827->894; hypervisors up; guests byte-identical pre/post.
• DONE Sections 5-6: post cloud-assert PASS; post BOM asbuilt/20260705-194617 committed; version coherence exact (91 agents @ 3.6.25); BOM diff = 17 expected rev pairs + 1 explained metadata delta (cinder secrets-storage endpoint); appendix-B B.1 fully re-baselined; runbook as-executed corrections DOCFIX-088; D-071 amended (backup exists). WINDOW COMPLETE.
• DONE follow-on (addendum 17): magnum 70->96 + vault 372->714 refreshed after download-and-diff analysis (mass-rebuild campaign upstream; vault stayed reactive/V0, handlers byte-identical, never sealed). Fleet FULLY CURRENT (zero can-upgrade-to); appendix-B re-baselined from asbuilt/20260705-201118.
• DONE (addendum 18): D-072 ADOPTED+EXECUTED -- dashboard VIP TLS repaired (cluster binding -> metal-admin; BUNDLEFIX-011 durable; https 200 first time on this build). DOCFIX-089 phase-03 3.3 fail-closed + appendix-A entry. Upstream bug DRAFTED: docs/upstream-bug-draft-dashboard-tls.md.
• POST-SESSION QUEUE (operator): submit the upstream bug + record LP number in D-072; rule on D-071 (still PROPOSED); Vault off-EOL path (Roosevelt-scale, 1.16 ruled out per D-068 amendment; in-channel 714 applied).
• Window findings logged for close-out (do not action mid-window):
  1. magnum can-upgrade-to anomaly REPRODUCED (points at magnum-dashboard-122; evidence ~/openstack-baseline/magnum-can-upgrade-anomaly-20260705.json) AND Charmhub magnum 2024.1/stable latest release (r101, 2026-07-01) is s390x-ONLY -- no valid amd64 target; magnum EXCLUDED; report upstream to OpenStack Charmers.
  2. vault NEWLY offers in-channel 372->714 on 1.8/stable -- NOT refreshed (D-068); logged only.
  3. juju create-backup/download-backup EXIST on juju 3.6 -- authoring assumption ("removed in 3.0, expected absent") is WRONG; correct ops-update-procedure 2.1 and the D-071 risk section at window close (DOCFIX candidate).
  4. juju status --format=line omits workload messages on 3.6 (root cause of the A7 false negative; fixed by DOCFIX-087).

Jumphost stream -- session 2026-07-05 (post-window resume)

• Orientation + reconcile DONE: machine-derived block re-seeded from today's scan (DOCFIX next-free 090 -> 091 was stale); shared-section stale-facts correction appended per the handoff reconcile item. Fences validate 4/0/0.
• FINDING (logged, not actioned): repo_lint.py L5 "next-free identifiers" info line counts ANY textual mention of D-/DOCFIX-/BUNDLEFIX-NNN across all files, so it is inflated by the deliberate high-numbered decoys in the tests/ledger-scan fixture ("must NOT inflate") and by next-free pointer lines themselves. ledger-scan (heading-based, pointer-excluding) is the numbering authority and is correct. DOCFIX candidate: make L5 reuse ledger-scan's exclusion rules or drop the line. No number consumed. NOTE (2026-07-06 correction): the first version of THIS bullet quoted the decoy tokens literally and thereby inflated ledger-scan's own DOCFIX/BUNDLEFIX next-free (docs/ prose is scan-scanned; only "next-free" lines are excluded). Tokens de-fanged; standing rule: never write identifier-shaped tokens above the real high-water mark in docs/ or runbooks/ prose.
• Standing by for batch-3 live validation support (verify-live queue).

Jumphost stream -- session 2026-07-06 (operator ruling sweep; addendum 20)

• RECORDED: D-050 RESOLVED/CLOSED via D-051 (operator ruling); D-073 ADOPTED (identity:list_trusts tightened to modern keystone default -- IMPLEMENTATION + TESTING PENDING, nothing live yet; proposed sequencing: before batch-3 d011-05); D-072 amendment (upstream dashboard-TLS bug WITHDRAWN -- operator judges site misconfig, draft retained); SEC-001 -> rotate at v1 close; SEC-003/004 deferred.
• PINNED (operator): D-071 ratification and D-068 off-EOL path choice held until A1/A2 (batch-3 entry) complete. Magnum can-upgrade-to upstream report pinned.
• A1 foil-tenant onboarding APPROVED in principle; tenant-structure model under discussion first (SCS Domain Manager persona confirmation + operator/tenant responsibility split). A2 (d011-04 disruptive gating) held behind A1.
• IN DISCUSSION (B2): OpenBao evaluation; D-068 item 2 (vault_url cleartext http); D-068 item 3 (AppRole TTL audit + proactive probe) scoping.
• DELIVERED (addendum 21): DOCFIX-091 docs/tenant-onboarding-contract.md per A1 rulings 1-4 (boundary, roles, intake, dangers, hardening register H1-H5 -- proposals logged, none implemented). A1.4 interpretation pending operator confirm: "SSH access into their domain" read as canary access-proof (H3). Verify-live additions: Horizon manager-role GUI identity probe (contract 5.3).
• RULED (2026-07-06 cont.): A1.3 H1-H5 ADOPTED as pre-v1-close tasks; A1.4 canary interpretation CONFIRMED + adopted as pre-deployment task. Full onboarding completion package (scripts + internal docs + client-facing drafts + live end-to-end validation) added to handoff section 6 v1-close checklist.
• RULED (2026-07-06 cont.): clientdocs/ TOP-LEVEL for client-facing drafts; build INTERLEAVES with batch-3 (foil onboarding = live workflow validation); A2 disruptive half PRE-APPROVED conditional on live N+1 headroom PASS.
• DELIVERED (addendum 22): H1 scripts/tenant-assert.sh + 7-case harness (green); clientdocs/ package DOCFIX-092 (README/intake/welcome/self-service drafts). Remaining in the interleave before foil onboarding: H2 idempotency guard, H3 canary stage, D-073 list_trusts zip implementation (then gated live apply).
• DELIVERED (operator-away batch, addenda 23-25): H2 re-run guards + H3 canary stage7 (harness 12/12); D-073 implementation STAGED (39-rule zip, drift check PASS -- live apply GATED, block in the D-073 amendment); DOCFIX-093 runbooks/d011-batch3-window-DRAFT.md (the one-window do-doc: D-073 apply + foil stages 0-4 + tenant-assert + canary + d011-04/05 + close-out); H5 embedded; docs/D-068-openbao-assessment-DRAFT.md (B2 research input -- no OpenBao charm exists, MySQL backend removed upstream, V1 goes to Sunbeam not legacy charms; research rec: deadline-bound risk acceptance). Stderr-merge sweep DONE: 10-site inventory logged in addendum 25 (fixes = DOCFIX candidate, not actioned).
• REPO-SIDE QUEUE now EMPTY except: H4 policy-drift script (main-chat backlog item 6 -- coordinate before building) and the client-docs sweep-on-change rule. EVERYTHING ELSE waits on the operator-present live window (the do-doc).
• DELIVERED (operator-away batch part 2, addendum 26): DOCFIX-094 stderr-separation fixes (cloud-assert A5 FAIL-OPEN closed + harness proof; onboard resolvers; offboard blind-sweep guard exit 20; 3 runbook paste blocks; acceptance:43 rides its future harness). Vault measurements: charm channel + ssl-* options verified; snap has NO 1.13/1.14 track -- max MPL payload in-charm = 1.12.11; OpenBao memo updated. Orphan-trustee sweep draft NOT started (queued).
• BLOCKER: juju controller macaroon EXPIRED ~17:10 2026-07-06 -- all juju cmds prompt for password. OPERATOR: juju login -u jessea123 at next presence. Blocked reads queued: keystone policy.json PENDING-LIVE-READ trio; barbican host_href look. (Vault reads above completed BEFORE expiry.) CLEARED later 2026-07-06: auth verified working; both queued reads DONE (see addendum 27 closures).

Jumphost stream -- session 2026-07-06 (cont.): d011-batch3 WINDOW EXECUTED (addendum 27)

• WINDOW COMPLETE, all gates green (logged d011-batch3, per-command wrap -- operator-ruled Claude Code lane, now in the convention doc as DOCFIX-097): D-073 APPLIED live (verify pair + H4-live PASS proof); foil1 onboarded 10.20.29.0/24 (stages 0-4 first-try, tenant-assert PASS 15/15 after fix, H2 double-SKIP, H3 canary PASS FIP 10.12.7.177/ubuntu); d011-04 PASS both halves (A2 conditional honored; headroom OK on all 3 hypervisors); d011-05 PASS (P1/P2/P3 with foil1); baseline + close-out cloud-assert PASS.
• In-window checker fixes (each harness-proven then re-run live): DOCFIX-095 tenant-assert owner-context app-cred probe (admin-side list is always-403 by base policy admin_or_owner -- measured, NOT a D-073 effect); DOCFIX-096 d011-04 amphora_headroom --long + display-name hypervisor keys.
• D-011: batch-3 CLOSED; D-011 CLOSED-except-item-6 (SEC-003 manual unseal rehearsal). v1-close checklist (handoff section 6) now QUEUED.
• Queued-read closures: keystone trio CONFIRMED (D-064 values explicit in live base policy; overlay comment tags dischargeable -- DOCFIX candidate); barbican host_href CLOSED benign-with-corrected-mechanism (charm renders literal https://None:9312, suppressing the upstream request-derived fallback; catalog unaffected; charm-level fix = future item needing D-NNN).
• Operator rulings this session: devteam pinned as next tenant, onboarding HELD until a cloud snapshot exists (item-7 cloud-snapshot.sh draft launched); orphan-sweep audit stays rc 0 report-only; clientdocs L7 receipt keeps all 14 files; H4 design defaults accepted as delivered.
• Worktree drafts DELIVERED (uncommitted, integration queued): orphan-trustee sweep (14/14), clientdocs L7 (17/17), tenant-acceptance harness (16 checks; closes the last stderr-inventory site), H4 keystone-policy-drift (10/10 + first live PASS), clientdocs handover pack (lint clean), cloud-snapshot.sh (in flight). Integration order: after this window commit; tenant-acceptance
  • L7 receipt last (receipt must hash post-integration file states).
• Open (small): Horizon manual checks on foil1 (operator, browser, non-blocking); capi-test-1 CREATE_FAILED leftover (orphan-sweep candidate); beta-cluster lbtest leftover; repo-lint worktree-exclusion DOCFIX candidate.
• SCRIPT-QUALITY BATCH LANDED (addendum 33, 2026-07-07): repo-wide sweep (110 scripts) -> operator-ruled fix scope shipped as DOCFIX-112..117: stage2 anti-escalation guard fail-closed (SECURITY -- was fail-open both ways); tenant-offboard hardening cluster (stderr-as-argv, Phase B counter CLOSED after 2 days logged, LB-wait fails loud, cred-file auth_url); validate.sh nonstd-exit HOLD; juju-spaces-check macaroon-vs-absent; pipefail sweep + vault-kv-health rc-test; guard-hook blocklist extension. Gauntlet 38/38. Deferred S-class + minor R items in docs/script-quality-findings-20260707.md (operator has NOT ruled S1-S5/S7/S8).

Jumphost stream -- session 2026-07-08: devteam cluster-template incident (addendum 39)

• DIAGNOSED + DELIVERED: devteam "Unable to create cluster template" = Public/Hidden flag -> clustertemplate:publish (admin_api) refusal, BY DESIGN; read-only diagnosis, no cloud change. DOCFIX-121 (clientdocs template-creation guidance + Public/Hidden warning) + DOCFIX-122 (appendix-A symptom entry); L7 receipt re-recorded; ~/devteam-package re-rendered (kubernetes.md, self-service-guide.md) for tonight's push.
• OPEN (operator option, unruled): stage5 for devteam -- devteam-k8s was never delivered (window ran stages 0-4); one gated command (bash scripts/tenant-onboard.sh devteam 5) closes the docs' delivered- template promise; else devteam self-creates per new docs.
• OPEN (operator, small): role-less GUI test user in the devteam domain (grant member or delete); Role-dropdown question on the dashboard-override test (why load-balancer_admin was submitted) -- evidence folded into the D-076 draft (worktree, uncommitted).
• TestCluster NoValidHost + decommission audit (addendum 40): devteam 1+3 cluster stalled on RAM capacity (NOT quota); operator deleted it, devteam re-created 1+1 (TestCluster2). QUEUED RULINGS from the audit: (a) beta-cluster teardown (~17GB reclaim) -- cross-stream, main-chat owns the resize-acceptance role; (b) lbtest LB leftover reclaim via kubectl (beta kubeconfig, main-chat asset); (c) foil1 offboard (RECOMMENDED -- dormant shell); (d) quota-vs-capacity expectation (trim testcloud quotas or document in clientdocs) -- unruled.

Jumphost stream -- session 2026-07-08 (cont.): Jenkins+K8s guide overnight (addendum 41)

• DELIVERED (DOCFIX-123): new clientdocs/jenkins-kubernetes-guide.md -- the single end-to-end Jenkins-to-Kubernetes workflow HUB (Pattern A deploy- into-cluster + Pattern B build-agents; self-create cluster path; kubeconfig- as-Jenkins-credential; NoValidHost sizing; flannel/Public-Hidden/keypair traps; troubleshooting; glossary; cross-links). Flannel hole also closed in tenant-skill kubernetes.md. Guide added to LEAK_FILES (harness change), README updated, L7 receipt 30 files, gauntlet 38/38. Instantiated to ~/devteam-package (devteam values). Operator pushes tonight.
• OPERATOR RULINGS (on return, 2026-07-08 PM) + status:
  1. devteam cluster: DELETE + DELIVER TEMPLATE ruled. DONE (addendum 42): TestCluster2 confirmed gone + devteam-k8s calico template delivered. 2b. Handover package RE-consolidation (operator re-ruling, addenda 46-47): glossary folded into handover-pack + glossary.md removed + account-table hard de-dup (DOCFIX-128); tenant AI skill aligned to the consolidated package -- 3 missing failure signatures added, capacity caveat, Jenkins- guide integration (DOCFIX-129). Top-level docs 9->8. Package re-instantiated (glossary appendix in handover-pack, skill refreshed). DONE.
  2. Consolidation review M1-M7: ALL APPROVED. DONE (addendum 43, DOCFIX-124): M5 error-holes in self-service-guide; M3 entry point + goal-map in welcome; M1/M2 single-source-of-truth + pointers (softer than hard-delete -- flagged, protects the AI skill + P2 test); M6 Jenkinsfile path/stage fixes; M4 new glossary.md. Package re-instantiation for the changed set = next step. FLAGGED: package handover-pack.md has unfilled {{CUSTODIAN_1/2}} (operator value; pre-existing, devteam placeholder-custodian test).
  3. Gated script fixes: BOTH DONE (addendum 45). DOCFIX-126 offboard E0 always-403 no longer FATALs before Phase F (harness case added, 21/21); DOCFIX-127 onboard cred files now carry dashboard_url (env-overridable, as-built https://10.12.4.58/horizon). LIVE VERIFY STILL GATED: next real onboard/offboard confirms end-to-end (onboard cred-write stages 1/2 have no unit coverage -- pre-existing; logged future harness item).
  4. DOCFIX list CLOSED (operator); handover-pack.md fine as-is (already carries {{DASHBOARD_URL}}/{{AUTH_URL}}).
  5. HANDOVER READY (operator confirmed 2026-07-08). Two flags ACCEPTED for the devteam FIRST INTERNAL TEST: (a) package handover-pack.md keeps literal {{CUSTODIAN_1/2}} -- fine for the test, fill with real names for a real client handover; (b) existing ~/tenant-devteam cred files predate the DOCFIX-127 dashboard_url field (written 2026-07-07) -- fine for today, the dashboard URL is in handover-pack section 3 which the team receives. Repo fully pushed (HEAD db284fc). REMAINING = operator-manual delivery only: copy ~/devteam-package + ~/tenant-devteam to the two custodians in person (guard blocks agent access to cred material).
• Offboard E0 defect (for the fix): admin-side identity:list_application_credentials is always-403 (base admin_or_owner; measured), so offboard v2 --apply counts 3 false failures and FATALs at exit 23 BEFORE Phase F (cred-dir retirement). foil1 offboard hit this live; cred dir retired by hand. Fix: owner-context-or-skip E0 inventory, do not count known-403 as failure; + harness case.

Jumphost stream -- session 2026-07-08 (cont.): devteam self-service cluster live (post-handover)

• devteam self-created their first cluster (read-only audit, no cloud change). TestCluster (uuid 92b92eed-afa3-4aaf-ab33-13d3ddeea869) CREATE_COMPLETE + HEALTHY (cluster / infrastructure / controlplane / nodegroup all Ready), created 2026-07-08 17:18 UTC in devteam-prod. 1 master + 1 worker, keypair devteam-key, api https://10.12.7.130:6443. calico + small-first + correct keypair + no Public/Hidden flag -- the DOCFIX-121 dashboard symptom resolved IN PRACTICE on their first real self-service create. (Note: this fresh TestCluster reuses the name of the earlier deleted TestCluster/TestCluster2, addenda 40/42 -- cosmetic.)
• Built via HORIZON, not the CLI. They created their own template TestTemplate (calico, public/hidden False, keypair baked in = devteam-key) in the dashboard and launched TestCluster from it, rather than the delivered CLI template devteam-k8s (keypair_id None; measured, correct-by-design, still unused). Both are valid Magnum objects.
• What the Horizon path changes (LOGGED, not actioned; DOCFIX candidate, no number consumed):
  1. Nothing about the running cluster -- same Magnum API object either way.
  2. Explains the keypair-on-template confusion this session: Horizon's Create Cluster Template dialog exposes an OPTIONAL Keypair field, so their GUI-built TestTemplate shows a key while the delivered devteam-k8s (CLI stage5 omits it by design) shows none. Comparing the two in-GUI drove the "no keypair -> must be required" read. It is not; the key is supplied at cluster-create time and their live cluster already carries it.
  3. Doc-gap: jenkins-kubernetes-guide.md is CLI-centric. Their next guide step (Section 3, kubeconfig fetch via openstack coe cluster config) has NO Horizon path -- the supported retrieval is the CLI. (Earlier draft of this bullet claimed magnum-ui "offers a cluster-config download" -- UNVERIFIED, retracted; do not assert it.) The Jenkins integration requires the CLI/app-cred path regardless.
• ADDRESSED: DOCFIX-130 (addendum 48, COMMITTED f0f11ae; ~/devteam-package guide RE-INSTANTIATED 2026-07-08, 34937 bytes, tokens filled from live-measured AUTH_URL/REGION). Grew from 2 gaps to a full guide pass on operator instruction ("fold and add 1-7... make GUI/CLI paths better defined"). clientdocs/jenkins-kubernetes-guide.md, additive only, no token touched:
  • STRUCTURE: new Section 0 dashboard-vs-CLI selector + On the command line / In the dashboard markers; dashboard equivalents for create (2.3), resize (2.5 new), teardown (8). Rule: lifecycle = either path; everything Kubernetes (kubeconfig onward) = CLI only.
  • GAPS 1-7 closed: registry/imagePullSecret (5.0; VERIFIED no registry service in catalog), ingress-nginx how-to + no-DNS (5.4 new), cluster resize (2.5 new; VERIFIED syntax), persistent storage self-verify (5.5 new), cert expiry read one-liner (3), tooling install + kubectl-skew (1); + ImagePullBackOff and PVC-Pending troubleshooting rows (9).
  • Leak harness CAUGHT two self-inflicted client-doc leaks ("Horizon", "magnum") on first pass -- reworded, now PASS 8/0; gauntlet 38/38.
  • ~/devteam-package NOT re-instantiated (refresh at next package sweep). See the changelog DOCFIX-130 entry. Numbers: DOCFIX next-free now 131.
• STORAGE OPERATOR FOLLOW-UP (logged, not actioned): delivered cluster template has volume_driver unset + no docker volume (MEASURED). Whether the built cluster gets a default StorageClass from the cinder-CSI default is UNVERIFIED (verifying = kubectl into devteam's live cluster; deferred for tenant isolation). If none, persistent volumes are unavailable out of the box -- candidate template improvement (would need a D-NNN). 5.5 is written self-verifying so the doc is correct either way.
• CLEANUP QUEUE RECONCILED against live (2026-07-08 read-only audit) -- LIVE IS AHEAD OF LEDGER, several "open" items already resolved:
  • foil1 offboard (was RECOMMENDED, addendum-40 c): DONE -- no foil project/user, no orphan trustee. (Closes the dormant-shell item; also means the gated DOCFIX-126/127 offboard live-verify was NOT exercised by us -- still unverified end-to-end, next real offboard confirms.)
  • capi-test-1 CREATE_FAILED delete (REMAINING-OPEN item 3, was operator-executes): DONE -- no cluster record, trustee reaped.
  • beta-cluster teardown + lbtest LB (addendum-40 a/b, cross-stream/main-chat): DONE -- no beta cluster, no stray LB. (Main-chat to log its side.)
  • Audit state: only devteam TestCluster (CREATE_COMPLETE) + its kubeapi LB + its protected trustee remain; orphan sweep ORPHAN=0 PROTECTED=2 UNPROVEN=0; zero unattached floating IPs.
  • STILL OPEN from the addendum-40 audit: only (d) quota-vs-capacity expectation -- trim testcloud quotas or document the capacity ceiling in clientdocs (unruled, operator). Everything else in that audit is closed.

Jumphost stream -- session 2026-07-08 (cont.): S-class script-quality batch (operator-greenlit)

Operator greenlit the recommended sequence: (1) Phase A stderr fix, (2) S7+S4, (3) S8; HOLD S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/offboard) until the DOCFIX-126/127 live-verify lands. Source: docs/script-quality-findings-20260707.md.

• DONE -- DOCFIX-131 (addendum 49): tenant-offboard.sh Phase A stdout-only. New tinv() (tenant-scoped inv() twin) for the 3 Phase A coe cluster list captures; wait loop retries on list-error, final check fails closed. Latent bug: merged stderr could false-trip exit 21 on an already-clean tenant. Harness un- excludes coe from warnstderr + new coe-stderr-not-argv assertion (22/22). Gauntlet 38/38, lint 0 fail, L7 re-recorded. Committed + pushed.
• DONE -- S7 (DOCFIX-132) + S4 (DOCFIX-133), addendum 50. S7: removed dead CF (d011-04) + subsumed vr_is_ipv4 guard (d011-03). S4: three inline trust filters -> one fixture-tested scripts/trust_filter.py (+tests/trust_filter/ 9/9); SDIR added to offboard + acceptance. Gauntlet 39 ALL GREEN. Committed + pushed.
• S8 RE-SCOPED (bigger than its "S" estimate; operator decision needed before I wire it): the KEYSTONE_VIP literal (10.12.4.50) lives in 6 files, and 3 of them (tenant-onboard/offboard/acceptance) do NOT source lib-net.sh -- so centralizing means adding . lib-net.sh sourcing into client-critical scripts I just modified (DOCFIX-131 + S4), which stacks risk. Breakdown:
  • S8a (FIP pool FIP_START/END -> lib-net.sh): DONE (DOCFIX-134). FIP_POOL_START/END added to lib-net.sh; phase-04 create/verify reference them. Gauntlet 39/39.
  • S8b (KEYSTONE_VIP -> lib-net.sh): DEFERRED -- needs sourcing plumbing into 6 scripts incl. the 3 tenant scripts (onboard/offboard/acceptance) that do NOT source lib-net. Operator decision needed before wiring client-critical scripts.
  • S8c (provider-bundle-check.py restates plane CIDRs): DEFERRED -- design change (python can't source bash -- needs a generated/parsed form).
• DONE -- DOCFIX-135 (addendum 51): tenant AI skill registry gap. devteam asked (via the assistant) if there's a registry setup step; the DOCFIX-130 registry content was not mirrored into the skill (grep found zero). Added "Images: bring your own registry" to references/kubernetes.md + ImagePullBackOff signature to references/troubleshooting.md; skill harness 8/8, gauntlet 39/39; package skill files re-instantiated. Committed + pushed. (Skill-lag class; watch for it on future guide changes.)
• DONE -- DOCFIX-136 (addendum 52): tenant AI skill FULL SWEEP vs DOCFIX-130. Operator asked for a full sweep after the registry gap. Grep-audit found 6 more guide topics absent; all added concise (pointer to guide for detail): resize command, kubeconfig cert-expiry (+openssl read), dashboard-vs-CLI split, no-DNS, NEW Persistent storage section, ServiceAccount pointer; + PVC-Pending and cert-expired troubleshooting signatures. Skill now covers the full DOCFIX-130 set (ingress stays recommendation+pointer by design). Harness 8/8, gauntlet 39/39, package re-instantiated. Committed + pushed. STANDING LESSON: guide changes must sweep the skill in the same change -- skill-lag caught twice (DOCFIX-135/136).
• DONE -- DOCFIX-137 (addendum 53): the standing lesson is now ENFORCEABLE. repo-lint L8 guide<->skill coupling guard: clientdocs/guide-skill-coupling.txt pins the guide's sha256; any guide change FAILS lint (naming the skill files to review) until re-recorded with bash scripts/repo-lint.sh --record-guide-skill-coupling. Harness +5 (28/28), gauntlet 39/39. So the next guide edit CANNOT land without a conscious skill review. Skill-lag closed at the source.
• S-CLASS BATCH STATUS (operator ruled 2026-07-08): DONE = S7, S4, S8a (DOCFIX-132/133/134). S8b APPROVED -- DO IT (next actionable): centralize KEYSTONE_VIP into lib-net.sh. DONE (DOCFIX-138): KEYSTONE_VIP_DEFAULT in lib-net.sh; all 6 sites reference it (3 forms: KEYSTONE_VIP, KEYSTONE_HOSTPORT, acceptance inline); lib-net source added where absent. Gauntlet 39/39, lint 0 fail. S8 FULLY DONE (S8a+S8b). S8c RULED WON'T-DO (operator: marginal value for as-built constants vs bash<->python emit/parse machinery). S1/S2/S3/S5 (lib-validate) STILL HELD -- operator keeps DOCFIX-126/127 live-verify open for an upcoming opportunity (devteam actively using resources; not now). S6 no-action.
• quota-vs-capacity (addendum-40 d): CLOSED -- operator ACCEPTS as-documented (DOCFIX-130 sec 2.3/5.5 + skill cover NoValidHost/capacity thoroughly; no quota trim).
• StorageClass follow-up (DOCFIX-130 sec 5.5): RESOLVED 2026-07-08. Operator authorized full read of devteam's cluster (fully internal domain); scoped via ~/devteam-cluster-openrc (snap needs $HOME for --dir; kubeconfig fetched under $HOME, used, DELETED). devteam TestCluster HAS a default StorageClass: default (default) provisioner cinder.csi.openstack.org, Retain, WaitForFirstConsumer, expansion=true. So the delivered template (volume_driver unset) DOES yield PV support via the cinder-CSI magnum default -- persistent storage works out of the box; no template change needed. sec 5.5 doc is correct and self-verifying; OPTIONAL enhancement: state the expected default cinder class (would trigger L8 skill sweep). Cleanup: kubeconfig removed, no residue.
• D-076 assessment (operator #6: "tested working w/ devteam domain-admin? check for additional work"): NOT fully working -- optimistic recollection. Revision 1 FAILED (full-page 403s + forced logout), reverted. Revision 2 PARTIALLY passed: identity panels render for the manager persona + session intact, BUT the create-user GUI flow submits a NON-managed role (load-balancer_admin) whose grant 403s, stranding a role-less user (flow not atomic). ADDITIONAL WORK before it can advance: (1) root-cause why the Role dropdown submitted load-balancer_admin (form default-role preselection vs mis-select); (2) add a per-view SEMANTIC harness test asserting create-user grants exactly member (invisible to the existing 50 policy pass/fail assertions); (3) clean up the orphaned test user 9a62f88e331f41adb5c9e41225ec8698 (name test1, CONFIRMED still present in devteam, ZERO roles -- delete or grant member); (4) operator RULING: adopt rev2 (a) vs close GUI/CLI-only (b) + accept/reject the charmhelpers blacklist-hole reliance; (5) another operator-present gated window to re-test after rework. NOTE: live override state UNVERIFIED -- juju macaroon expired (needs juju login); confirm use-policyd-override is reverted to baseline. Draft in worktree agent-a73ec798 (uncommitted). Clients unaffected: CLI identity path is the documented behavior.
• D-076 PINNED (operator 2026-07-08): do NOT work D-076 while devteam are running their own tests in the env. Deferred until devteam testing completes; no further D-076 investigation/window until then. Draft stays in worktree.
• test1 orphan SCHEDULED FOR REMOVAL (operator 2026-07-08): approved to delete; staged, NOT run now. Exact gated command (re-verify role-less at execution): openstack user delete 9a62f88e331f41adb5c9e41225ec8698 (name test1, devteam domain, 0 roles measured 2026-07-08). Run at the next convenient window; it is a role-less shell with no resources, so removal does not affect devteam's tests.

REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window)

1. Devteam delivery -- last two steps are OPERATOR-MANUAL (everything else DONE, addenda 34-35): intake collected 2026-07-07 (internal test; placeholder custodians operator-accepted; filed jumphost-local ~/devteam-intake-record.md, NEVER commit); quota applied+verified (10/20/51200, runbook standard); pack instantiated to ~/devteam-package (17 files, tokens verified zero, banners stripped; templates in clientdocs/ untouched -- L7 receipt intact). REMAINING (operator): (a) copy the CA into the package: cp ~/vault-init/vault-ca-root.pem ~/devteam-package/omega-cloud-ca.pem (guard correctly blocks agent access to ~/vault-init); (b) deliver ~/devteam-package + the ~/tenant-devteam credential files in-person to the two named custodians.
2. Horizon contract checks: CLOSED for devteam (2026-07-07 full three-account walkthrough, addendum 38). 5.1 VERIFIED (login + Identity-only view + expected popups); 5.3 verdict: GUI identity probe FAILS as the contract anticipated -- CLOSED via the CLI-path clause (DOCFIX-120 contract amendment); D-075 TLS-leg redirect CONFIRMED (operator browser). Isolation, keypair asymmetry, per-user app-cred visibility, and OBJECT STORAGE (container create/delete -- first live proof, closes the acceptance-checklist caveat) all verified in-GUI. foil1's own 5.1/5.3 rows: surface now proven on devteam; operator may run or waive foil1's pass (non-blocking either way). 2c. Dashboard policy override (GUI identity for manager persona): first live apply FAILED acceptance + REVERTED 2026-07-07; rework IN FLIGHT (background agent, worktree). Zip mechanics + PO: activation worked; panels rendered but page-level checks refused (Domains/ Projects/Users/Groups "not authorized"); Roles panel popped two errors then FORCED LOGOUT. Reverted to baseline in one config cycle (operator-confirmed). Decision draft stays PROPOSED (uncommitted, agent worktree; next-free D claimed at commit). Next: agent rework (per-view rule/target mapping, logout root cause, scope-down to Users+Projects, offline per-view semantic tests), then a second gated apply in a later window. Clients are unaffected: CLI identity path is the documented behavior. 2b. D-075 RULED (option c) + EXECUTED 2026-07-07 (was: root-autoindex finding): redirect / -> /horizon + Options -Indexes via conf-enabled/99-omega-root.conf on the dashboard unit. First attempt (RedirectMatch) caught leaking the backend port in verify and replaced with per-dir mod_rewrite. Delivered as D-075 entry + DOCFIX-118 (phase-03 PER-REBUILD block + appendix-A symptom entry). PER-REBUILD like the D-044 override.
3. capi-test-1 CREATE_FAILED record delete: RULED YES (2026-07-07), execution handed to operator -- the Claude Code permission classifier (correctly) refuses agent-side deletes of resources it did not create; exact command staged in-session (uuid fae7ec70-712c-4e58-88dd-da0d7b8c89b2, measured this window). After it runs: next tenant-offboard.sh --sweep-magnum-orphans --audit reaps its trustee.
4. S-class consolidation + minor R items: proposals in docs/script-quality-findings-20260707.md -- present options to operator before building (S1-S5/S7/S8 unruled; S6 ruled no-action).
5. Pinned decisions (do NOT re-ask): D-071 ratification (pre-DC-DC HA/ backup session); D-068 off-EOL path (Roosevelt-scale); SEC-001/003/004 at v1 close; d011-06 stays MANUAL until SEC-003 rehearsal (gates D-011 FULL close; v1-close checklist QUEUED per handoff section 6).

6. Cross-stream: main-chat to reconcile its ledger section against addenda 27-33 (its backlog items 6/7/8 delivered here; its Verify-live queue fully cleared; D-050/D-073/D-074 all resolved).

7. Horizon three-account walkthrough + docs alignment (2026-07-07 cont., addendum 38): operator live-tested all three devteam accounts in the GUI; every observation contract-correct (details in item 2 above). DOCFIX-119: clientdocs aligned to observed GUI reality (5 files by background docs agent -- first-login expectations, CLI user-creation commands, popup signatures, object-storage row de-hedged + acceptance script aligned; harnesses 8/8 + 47/47; L7 receipt re-recorded) + README instantiation guidance (DASHBOARD_URL includes webroot). DOCFIX-120: contract section 5 amended with live verdicts (5.1 verified; 5.3 closed-by-CLI-clause; stage-7 canary marked IMPLEMENTED). Package RE-INSTANTIATED to ~/devteam-package from updated templates (incl. /horizon URL); OPERATOR: re-copy to local + deliver. Dashboard override attempt/revert logged in item 2c.

8. devteam intake + quota + pack instantiation (2026-07-07 cont., addendum 35): intake walkthrough with operator (7 sections; CIDR + short name pre-settled by rulings); quota set devteam-prod 10/20/51200 (values equal nova class defaults -- set explicitly per runbook "documents the record"); first-ever clientdocs pack instantiation: 17 files to ~/devteam-package via scratchpad script (9 tokens: endpoints MEASURED in-session -- AUTH_URL https://10.12.4.50:5000/v3, RegionOne; dashboard https://10.12.4.58 carried from D-072 as-executed verification). Guard/classifier blocks honored: no ~/vault-init access (CA copy handed to operator), no curl -k probe. LOGGED-NOT-ACTIONED (DOCFIX candidate): tenant-skill/references/day2-operations.md line ~7 explains the {{THIS}} notation in DELIVERED prose, so the literal token survives instantiation; reword at next clientdocs sweep (L7 receipt re-record required).
9. devteam ONBOARDED (2026-07-07 window ops-devteam-onboard, addendum 34): stages 0-4 + tenant-assert (15/15 PASS) + stage7 canary (SSH via FIP 10.12.7.43, teardown clean) ALL FIRST-TRY on TENANT_CIDR=10.100.0.0/24 (D-074 ruled fallback -- first live use of the fallback path and first client-facing workflow execution; ZERO as-executed corrections needed). Stage-2 anti-escalation guard (DOCFIX-112 batch) first live client run: admin grant denied as designed. Baseline reused per ruling (asbuilt/20260706-224851, not stale). Open cloud-assert PASS (A5-A7 held on first run -- one-shot shell lacked admin scope, known jumphost trait, re-run sourced) and close cloud-assert PASS. Handover creds staged in ~/tenant-devteam (0600, jumphost-only). Pack instantiation HELD on intake answers (item 1 above). capi-test-1 delete handed to operator (item 3).
10. CLIENT PACKAGE RELEASED (addendum 31, 2026-07-07): operator-requested polish sweep integrated (DOCFIX-110) + banner ruling applied (DRAFT removed; the three per-client templates keep TEMPLATE; README says RELEASED-for-handover). Package is handover-ready pending only per-client instantiation. In flight: repo-wide script quality sweep (report-only agent; findings to operator triage next).
11. CLIENT PACKAGE COMPLETE (addendum 30): starter kit (DOCFIX-108, 6 deliverables incl. tenancy-audit from the skill delivery; pipeline-smoke + leftover-check deduped away) + tenant AI skill omega-cloud-tenant (DOCFIX-109, leakage-gated) integrated; L7 coverage recursive; receipt 29 files; gauntlet 37/37. REMAINING before handover: operator-requested polish pass over the final clientdocs set (in flight), then devteam onboarding next session instantiates the pack.
12. D-074 EXECUTED (addendum 29): operator ADOPTED the main-chat CIDR correction PLAN same-day (overlap-allowed tenant CIDRs; client-choice + 10.100.0.0/24 fallback; capi-mgmt stays reserved; L5 print removed). Phase 0 gates honored: allow_overlapping_ips=True measured; overlap+Magnum proof via foil1 (EXACT 10.20.28.0/24 overlap with beta; cluster ~14.5 min; 2 Ready nodes via API-LB while beta served the same CIDR); teardown clean; cloud-assert PASS. Guard rewritten (DOCFIX-106, net_overlap.py, harness 12->17), docs aligned (contract/runbook/intake), D-016 amended, repo_lint L5 next-free removed (DOCFIX-107). FINDING-doc BUNDLEFIX token de-fanged (counter was inflated to 052; restored 012). Phase 3 (devteam onboarding, fallback CIDR) remains NEXT SESSION. Corrections fed back to main-chat via addendum 29: northwind example fictional; runbook filename wrong in PLAN.
13. INTEGRATED (addendum 28): all six worktree drafts -> main as DOCFIX-098 (H4 drift detector), 099/100 (offboard orphan sweep + Phase E0 guard), 101 (clientdocs handover pack), 102 (cloud-snapshot.sh -- unblocks the devteam snapshot gate), 103 (tenant-acceptance harness + the last stderr-inventory fix), 104 (lint L7 sweep-on-change ENFORCED; receipt live, 17 files). Gauntlet 35/35 on the integrated tree. Main-chat backlog items 6/7/8 are now DELIVERED (by this stream, operator-assigned) -- main chat to reconcile its section.
14. devteam gate SATISFIED (2026-07-06): cloud-snapshot first live runs both green -- asbuilt/20260706-224851 committed (12 redactions verified; scrubber caught a real embedded CA private key) + 1.1GB controller backup jumphost-local (~/openstack-baseline/, NEVER commit). Operator ruled: devteam onboarding NOT this session -- next session runs stages 0-4 + tenant-assert
    • canary on 10.20.30.0/24 (recommended, unruled), then instantiates the handover pack with dev-team intake answers (7 questions in the DOCFIX-101 entry). Also still open: foil1 Horizon manual checks (contract 5.1/5.3).

Jumphost stream -- session 2026-07-09: new repo openstack-caracal-dc-dc first load (repo-rename sweep)

• Environment note: this session's shell is Windows (win32), not the Linux jumphost (vopenstack-jesse) CLAUDE.md describes -- flagged to operator. run-tests-all.sh gauntlet on this box shows 23/39 pre-existing failures (missing jq, tty/exit-code quirks) unrelated to any change made this session; repo-lint (the only suite touched) is ALL PASS standalone and in the gauntlet. Confirm actual jumphost behavior at next Linux session.
• DONE -- DOCFIX-139 + DOCFIX-140 (changelog docs/changelog-20260709-dc-dc-repo-rename-sweep.md): repo-rename sweep (stale openstack-caracal-ipv4 path/URL references repaired to openstack-caracal-dc-dc across the skill, 4 runbook RUN blocks, and the repo_lint.py docstring; clientdocs L7 receipt re-recorded after confirming no client-facing drift) + a real repo_lint.py bug fix (L1 Windows path-separator: the docs/design-decisions.md legacy-carve-out comparison used str(p.relative_to(R)), which never matches on Windows, so the documented WARN silently became a FAIL -- fixed to .as_posix() at all 3 affected sites; harness gained T3b covering the carve-out path directly, 29/29 green). NOT actioned (flagged, larger scope): README.md and environment.md's body content still describe VR0 DC0/v1 single-DC scope wholesale -- awaiting operator direction on whether/when to rewrite for VR1 DC-DC. Uncommitted -- operator has not yet asked to commit.
• DONE -- DOCFIX-141 (changelog docs/changelog-20260709-repo-agnostic-sweep.md): operator asked to scope the "make scripts repo-agnostic" idea as its own sweep + a lint rule, following directly from the DOCFIX-139/140 experience. New repo-lint check L9 (self-reference guard) FAILS any script/RUN-block/ skill file that hardcodes the current repo's own directory name; added with a 5-case harness (34/34 green). Along the way, narrowed all_text()'s .claude exclusion from "skip all of .claude/" to "skip only .claude/worktrees/" -- the blanket exclusion meant checked-in .claude/skills/ and .claude/hooks/ content was invisible to lint entirely (why the SKILL.md hardcode from DOCFIX-139 wasn't caught). Established a $REPO session-variable convention (documented in runbooks/README.md Conventions) and converted the 8 remaining hardcoded clone-path references (SKILL.md + 4 runbooks) to require it explicitly (fail loud if unset, no inferred fallback, per hard rule 2). environment.md's repo URL is KEPT (documented anchor fact, same as its other facts) but tagged with the new repo-lint: allow-selfref opt-out. LOGGED, NOT ACTIONED: L3/L4/L6/L9 only recognize triple-backtick fences for "in code" detection; d011-batch3-window-DRAFT.md (indented pseudo-code) and tenant-onboarding-v2-DRAFT.md (BEGIN/END block comments) use different conventions and are invisible to these checks -- a pre-existing gap, not introduced here, DOCFIX candidate (teach the checks the other conventions, or standardize runbooks onto fences). Uncommitted -- operator has not yet asked to commit.
• DONE -- workflow doc + gap register (2026-07-09): docs/dc-dc-deployment-workflow.md added (Stage 0-7 followable checklist + tooling gap register audited directly against the repo) + a companion visual Artifact (https://claude.ai/code/artifact/e752f6dd-23ff-4c59-9ce5-1dedb917e5ea). COMMITTED + PUSHED (374aef5).
• DONE -- DOCFIX-142 (changelog docs/changelog-20260709-opentofu-scaffold.md): first OpenTofu content in the repo (gap register item 2). opentofu/ network + storage-pool scaffold: dc-planes (six per-DC planes), mesh-link (D-100 dark-fiber triangle), dc-storage-pool modules; DC1 + Office1 wired, DC2 held back pending its CIDR assignment (D-101). Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current docs this session, not memory -- differs materially from older common examples (network isolation is forward.mode, nested, not top-level mode; domain resource restructured to ~40 args mirroring raw libvirt XML). Node-VM/volume resources, OPNsense, DC2 planes, and tc netem application DELIBERATELY DEFERRED -- schema too unfamiliar to author safely without a real tofu providers schema pass; see opentofu/README.md. Added scripts/opentofu-validate.sh + tests/opentofu-validate/ (2/2 green -- only the guard-clause paths testable without a tofu binary, which is not available anywhere this repo has been worked in this session; the actual fmt/init/validate path is UNTESTED, logged not hidden). Extended repo_lint.py's L1 file-extension set to include .tf (previously uncovered since no Terraform/OpenTofu files existed). repo-lint 34/34 harness green, 0 fail/1 documented warn. Uncommitted -- operator has not yet asked to commit this piece.
• NEXT ACTIONABLE STEP (logged): on a machine with the tofu binary + registry network access, run scripts/opentofu-validate.sh, then tofu providers schema -json against the initialized provider, and read the real libvirt_domain/libvirt_volume schema directly before authoring the node-VM module.
• DONE -- OPNsense gap detail (2026-07-09): gap register item 4 split into its 4 separable sub-gaps (network substrate done; VM/image blocked on the domain module; OPNsense's own routing config not started, no mechanism chosen; tc netem a separate OS-level thing blocked on an unruled D-100 parameter). Mirrored into opentofu/README.md + the visual tracker. COMMITTED + PUSHED (0bb6cff).
• DONE -- DOCFIX-143 (changelog docs/changelog-20260709-repo-lint-crlf-fix.md): fixed the CRLF write_text bug flagged during the DOCFIX-141 commit -- --record-clientdocs-sweep/--record-guide-skill-coupling now write_bytes() with an explicit ascii-encode instead of relying on .gitattributes eol=lf to clean up CRLF after the fact. Verified 0 CR bytes on a live re-record; harness 34/34 unchanged; repo-lint 0 fail/1 warn. COMMITTED + PUSHED (f93dd0d).
• DONE -- DOCFIX-144 (changelog docs/changelog-20260709-opentofu-node-vm.md): moved into the domain module as requested. Built opentofu/modules/node-vm (blank-disk PXE-boot MAAS-managed node VM, D-103's pattern for Stage 3) -- this time from the provider's own real example .tf files, not doc-summarization alone, after last session deferred libvirt_domain for exactly that confidence gap. Found and fixed a real bug in the already-committed DOCFIX-142 delivery in the process: dc-planes/ mesh-link/dc-storage-pool used classic HCL block syntax (domain { ... }) where the provider's current schema actually needs attribute-style objects (domain = { ... }) -- confirmed once real provider examples surfaced the convention; doc-summarization alone had NOT caught this. All three corrected; opentofu/README.md's schema notes now state the attribute-object convention as a provider-wide assumption. UNVERIFIED, flagged: the per-device boot-order attribute's exact shape (an inference by pattern). Still deferred: the cloud-init/base-image VM pattern (Office1 services + OPNsense), DC2 planes, netem. COMMITTED + PUSHED (3324387).
• DONE -- DOCFIX-145 (changelog docs/changelog-20260709-opentofu-cloudinit-vm.md): built opentofu/modules/base-image + modules/cloudinit-vm (the OTHER VM pattern, for Office1's own service VMs -- MAAS-region, NetBox, GitBucket). Same high-confidence source as DOCFIX-144 (examples/alpine_cloudinit.tf, fetched as real code). Threaded the base image's .path output across the module boundary as a real attribute reference rather than reconstructing a volume path from pool+name strings -- a cleaner design than the earlier draft, avoiding an unconfirmed-convention guess entirely. user_data/meta_data/network_config are required inputs with no default (Office1 VM configs aren't designed yet; a plausible DHCP default would also silently fail since these planes carry no libvirt DHCP). OPNsense applicability explicitly NOT confirmed -- FreeBSD-based, cloud-init/NoCloud support not the same guarantee as Linux images, flagged prominently rather than assumed. Neither module instantiated (no image source chosen for any service VM). Workflow doc + visual tracker updated. Uncommitted -- operator has not yet asked to commit this piece.
• DONE -- OPNsense deployment research (changelog docs/changelog-20260709-opnsense-research.md, no DOCFIX -- research delivery, matches the design-doc precedent): operator asked to check OPNsense's actual documentation. Confirmed cloud-init is genuinely unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing the DOCFIX-145 open question. Found the real mechanism instead: the Configuration Importer (official docs) scans an attached volume for /conf/config.xml in a 2-3s boot window; ISO9660 support was added to it specifically for VM/cloud automation (a closed, milestone-targeted, core-dev-assigned GitHub issue) -- mechanically identical to cloudinit-vm's cdrom-attach shape, different payload. Use the nano image for KVM (pre-installed, serial-ready), not the installer images. Fully sourced in opentofu/README.md. NOT built yet: a decompress/convert preprocessing script for the nano image, an ISO9660 config.xml-seed builder, and the real config.xml content -- itemized as the next actionable step. COMMITTED + PUSHED (75ca823).
• DONE -- DOCFIX-146 (changelog docs/changelog-20260709-opnsense-edge-module.md): operator asked to continue with OPNsense while the research was fresh. Built all three itemized next-steps: scripts/opnsense-prep-image.sh (nano image decompress+convert, no hardcoded mirror -- OPNSENSE_MIRROR_BASE required), scripts/opnsense-build-config-iso.sh (ISO9660 config.xml seed), and opentofu/modules/opnsense-edge wiring both together -- mechanically identical to cloudinit-vm's cdrom-attach shape, no libvirt_cloudinit_disk involved. Explicit lan_network_name/ wan_network_name variables (not an ordered list) to prevent a silent LAN/WAN swap. Both harnesses green (4/4, 3/3) exercising REAL missing-tool guards (qemu-img/genisoimage/xorriso are all genuinely absent this session). Two things flagged, not presented as fact: whether the Configuration Importer's ISO9660 behavior holds on a real boot, and whether devices.interfaces list order reliably maps to LAN/WAN NIC enumeration for this guest. config_iso_path has no default -- real config.xml content per site is still undesigned. Not instantiated in root main.tf. opentofu/README.md + workflow doc + visual tracker updated to reflect BUILT (not just researched). COMMITTED + PUSHED (a3ff07f).
• DONE -- DOCFIX-147 (changelog docs/changelog-20260709-opentofu-maas-netem.md): operator asked to pull docs and validate before drafting the next modules -- the two remaining unbuilt mechanisms from the OpenTofu scope (MAAS VM-host registration, tc netem application). Researched a real official MAAS provider (canonical/maas v2.7.2) and OpenTofu's own local-exec/terraform_data docs BEFORE writing any code, per instruction. Caught a real design mistake before it happened: read both maas_vm_host (register) and maas_vm_host_machine (compose -- takes cores/memory/disks as INPUTS, MAAS's "Compose machine" pod feature) schemas, cross-checked against D-103's explicit "MAAS does NOT compose new ones" ruling, and built modules/maas-vm-host using ONLY maas_vm_host -- the other resource would have had MAAS and node-vm fighting over VM creation. Also confirmed local-exec runs on the machine invoking tofu apply (Office1 per D-103), not the vcloud host where bridges live -- modules/netem-link wraps its tc qdisc command in an explicit SSH hop because of this, using terraform_data (OpenTofu's own recommended replacement for null_resource) with a destroy-time provisioner to clean up. Neither module instantiated (MAAS zone/pool, vcloud SSH target, and real netem parameters all still pending). opentofu/README.md gained a new sourced research section; workflow doc + visual tracker updated. COMMITTED + PUSHED (ea276c5).
• DONE -- DOCFIX-148 (changelog docs/changelog-20260709-opentofu-audit.md): operator asked to check back over all current modules, pull docs for anything drafted without it, and review for errors/assumptions. Grepped every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules and researched each further rather than re-stating hedges unchanged. Found and fixed a genuine documentation error, not just a hedge: opnsense-edge's comments claimed interface list-order directly sets OPNsense's LAN/WAN role -- researching OPNsense's own interface-assignment model showed role assignment is a SEPARATE, explicit config.xml mapping, not automatic from device order ("vtnet0=WAN" is a convention some guides use, not an enforced rule). Comments + both LAN/WAN variable descriptions corrected; no config.xml has been written yet so this fixes documentation accuracy for whoever writes it next, not current runtime behavior. Confidence UPGRADED (not fully resolved) on three items: node-vm's boot-order shape (confirmed the provider 1:1-mirrors libvirt's own XML, and libvirt's own docs confirm <boot order='N'/>'s single-attribute shape), create.content.url accepting local paths (second independent source), and the genisoimage flags (confirmed standard, cleanly separated from the still-open "does OPNsense's Importer actually read it" question). Confirmed safe: maas_vm_host's zone/pool computed-if-unset behavior. opentofu/README.md gained a new "Audit pass" section. Deliberately did NOT re-research already-well-sourced findings (Importer mechanism, vm_host-vs-vm_host_machine, the syntax-bug fix) -- targeted only real gaps. Also added (same delivery): .claude/skills/openstack-cloud-ops/references/ opentofu-provider-docs.md -- indexes every provider/doc source used building opentofu/ with exact URLs, confirmed versions, default branches, AND the fetch methodology that actually worked this session (Registry doc pages are JS-rendered, use GitHub instead; branch names vary, check default_branch via the API; real example .tf files beat doc-summarized prose for syntax questions). New SKILL.md routing entry; cross-referenced from opentofu/README.md in both directions, no content duplicated (README.md stays the single source of truth for per-module findings). COMMITTED + PUSHED (96ecbad).
• DONE -- DOCFIX-149 (changelog docs/changelog-20260709-opnsense-config-design.md): operator asked to move into designing the real config.xml content. Fetched OPNsense's own real, currently-shipped config.xml.sample (opnsense/core repo) and its Route.xml static-routes model before drafting. Built opentofu/templates/opnsense-config.xml.tmpl ({{TOKEN}}-parameterized, reusing this repo's existing clientdocs convention) + scripts/opnsense-render-config.sh. The real sample fetch DIRECTLY CONFIRMED the DOCFIX-148 audit finding with an actual example: ships literal placeholder device names inside each interface's own <if> element, proving LAN/WAN role assignment really is the explicit per-block mapping DOCFIX-148 concluded. The renderer is tested END-TO-END, not just guard clauses (8/8 -- it needs no external tool, a first among the opnsense-* scripts) -- and the harness caught a REAL bug before it shipped: the token HOSTNAME collides with bash's own built-in $HOSTNAME variable (unset doesn't actually clear it), silently passing a test that should have failed. Renamed to OPNSENSE_HOSTNAME throughout. Full token legend in opentofu/templates/README.md, marking exactly which values are real (NTP pool, D-106 naming) vs. still pending Stage 0 ratification (D-100/D-101/D-107) vs. only measurable on a real boot (vtnetN assignment) vs. a security requirement (root password hash must be freshly generated, never the stock sample's own shipped default). opentofu/README.md + workflow doc + visual tracker updated. Uncommitted -- operator has not yet asked to commit this piece.
• DONE -- Stage 0 (decision ratification) RATIFIED 2026-07-09 (changelog docs/changelog-20260709-stage0-ratification.md, DOCFIX-150): operator walked through the 6 buildout-design redline items live (AskUserQuestion, 2 rounds): Ceph size=3-by-default (size=2 only as an explicit Phase-0 fallback); netem same-metro lean; D-102 folds into D-101, D-109 stays standalone; metal-admin GAINS a ULA leg (reverses the stated lean -- a real deviation, not a rubber-stamp); COS per-DC-only confirmed; mirror sync topology independent-per-DC confirmed. All 11 of D-100..D-110 individually flipped PROPOSED -> ADOPTED in docs/design-decisions.md (D-102's content preserved under a MERGED-INTO-D-101 status line, per the append-only discipline -- not deleted). docs/dc-dc-buildout-design.md Section 10 rewritten from "open items" to a ruling record; Section 3's Ceph gate and the stray "Section D-102" cross-reference corrected to D-101. Verified via ledger-scan.sh: D-100..D-110 no longer appear under PROPOSED/OPEN -- only D-068/D-071 remain (separate, pre-existing, unrelated to this gate). repo-lint 0 fail/1 documented warn, harness 34/34. Stage 0 of docs/dc-dc-deployment-workflow.md is CLEARED.
• Operator context, 2026-07-09 evening: operator stepped away from the workstation for the night, granted blanket git permission for the session, and authorized autonomous subagent use to make maximum progress toward an end-to-end-ready project by morning. Operator clarified this session runs from the WORKSTATION, not the vopenstack-jesse jumphost -- tonight's scope is PREP ONLY (docs, scripts, tests, hardened runbooks); no live infra (juju/MAAS/OpenStack APIs) is reachable or in scope tonight. Operator will switch to the vcloud host in the morning to run actual installs/deployments. This explains (and validates) why run-tests-all.sh shows 23 pre-existing failures in this environment -- all either "jq required" (not installed on the workstation) or live-cloud-dependent suites (cloud-assert, preflight, tenant-*, etc.) -- confirmed via git stash A/B to be identical with and without this session's own changes, i.e. an environment gap, not a regression.
• DONE -- Tooling gap register #1 CLOSED 2026-07-09 ($DC selector convention, changelog docs/changelog-20260709-dc-selector-convention.md, DOCFIX-151): added lib_net_select_dc() to scripts/lib-net.sh (dc0/dc1 no-op per D-101's inherited-layout ruling; dc2 fails loud -- gap #3 still open) and lib_hosts_select_dc() to scripts/lib-hosts.sh (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment exists for either yet, an intentional asymmetry vs. the net selector, documented in-line in both files). Backward compatible by construction -- no existing caller of either library is affected. New harness tests/dc-selector/run-tests.sh: 21/21 PASS (caught and fixed a real pipefail-vs-nonzero-exit-code bug in the harness itself while writing it). repo-lint 0 fail/1 documented warn (unchanged); full gauntlet confirmed via git stash A/B to introduce zero new failures. Stage 5's runbook (not yet authored) still needs to actually CALL lib_net_select_dc "$DC" / lib_hosts_select_dc "$DC" -- the mechanism exists, the call sites don't yet.
• DONE -- Tooling gap register #3 mechanism CLOSED 2026-07-09 (NetBox DC-DC pipeline, changelog docs/changelog-20260709-netbox-dc-dc-pipeline.md, DOCFIX-152): netbox/dc-dc-prefixes-import.py extends the v1 single-site IPv4-only NetBox import to VR1's two-DC dual-stack model (D-101). DC1's v4 planes hardcoded (D-101: inherited unchanged, this is decision text, not an inference); DC2's v4 supernet, the org ULA /48, and the per-DC GUA carve are all REQUIRED env vars with no defaults -- fails loud rather than inventing any of them. The per-plane subdivision scheme for the ULA/GUA blocks is flagged in the script's own docstring as a proposed, unratified convention pending operator review. New harness tests/dc-dc-prefixes-import/ (a fake in-memory NetBox client): 37/37 PASS at the time (later corrected to 40/40 by DOCFIX-160 -- a dead-code bug in the harness itself had left 3 happy-path assertions silently uncounted; see that entry below). Caught and fixed two real authoring bugs at delivery time: (1) example GUA addresses in the first draft had host bits set beyond the /40 boundary, caught by Python's own ipaddress.ip_network(strict=True); (2) the test's own "count CREATED lines" assertion was inflated by the script's unrelated site-creation message also containing the word CREATED -- tightened to match prefix-specific lines. Flagged, not fixed (out of scope): the existing v1 ipv4-prefixes-import.py's role slugs predate the current six-plane model in scripts/lib-net.sh; this new script uses the CURRENT slugs instead. repo-lint 0 fail/1 documented warn (also caught and fixed a CRLF regression introduced by a Python find/replace during test-data cleanup). Full gauntlet 23/45 FAILED -- confirmed identical pre-existing environment-gap failures (jq missing / no live juju-cloud access on this workstation), zero regressions. UNVALIDATED against a real NetBox (none reachable this session). This closes the TOOLING half of gap #3 only -- the literals themselves (ULA /48, GUA carve, DC2 supernet) still need real assignment before this script can be run for real.
• DONE -- Stage 1 runbook WRITTEN 2026-07-09 (changelog docs/changelog-20260709-dc-dc-phase0-runbook.md, DOCFIX-153): runbooks/dc-dc-phase0-vcloud-prep.md -- command-level Phase 0 (vcloud host prep) steps, matching the phase-00 runbook's style (sequence table, read-only CHECK before every mutation, individually gated mutations, an explicit gate section, delivery checklist). Surfaces a genuine open structural question rather than silently resolving it: opentofu/main.tf's unconditional provider "maas" block may force MAAS credentials to be set at tofu plan time even though this stage's plan touches only libvirt resources -- documented both possible outcomes and what to do for each. Also flags the VR0/DC0-pool-path-collision risk if the vcloud host is the same physical machine being repurposed (buildout design's own framing). NOT YET EXECUTED -- this is the runbook the operator will run tomorrow morning after switching to the vcloud host; nothing in this delivery claims a real run happened. repo-lint 0 fail/1 documented warn (unchanged, prose-only).
• DONE -- Stages 2, 3, and 5 runbooks WRITTEN 2026-07-09, three of five remaining runbooks delegated to parallel subagents (Stages 2, 3, 4, 6, 7) while Stage 5 was authored directly (the ground-truth/copy-point stage):
  • Stage 2 (changelog docs/changelog-20260709-dc-dc-phase1-runbook.md, DOCFIX-154): runbooks/dc-dc-phase1-office1-standup.md. Every VM- creation step forks explicit Option A (OpenTofu, blocked pending cloud-init content design)/Option B (manual, logged debt). Found a real, previously-unnoticed gap: no Office1-local virtual network is modeled anywhere in opentofu/. Caught and fixed a repo-lint L2 false-flag during review (a correctly-sourced historical D-014 path reference, reworded onto one line so the rule's own EXEMPT pattern applies, not deleted).
  • Stage 3 (changelog docs/changelog-20260709-dc-dc-phase2-runbook.md, DOCFIX-155): runbooks/dc-dc-phase2-tofu-dc-substrate.md. DC1-first; DC2 hard-gated (not "later"). Five !!! callouts naming real constraints (Stage 2 dependency, DC2 gate, no-Roosevelt-analog shim, undecided node sizing, unruled netem params) plus OPNsense tool- presence check. GATE section concludes "CONDITIONALLY MET AT BEST," refusing to claim more than is actually true.
  • Stage 5 (changelog docs/changelog-20260709-dc-dc-phase4-runbook.md, DOCFIX-156): runbooks/dc-dc-phase4-juju-bundle-per-dc.md. Describes adapting (not duplicating) phase-01..08 to run twice, once per DC. DC1 needs zero VIP/CIDR edits (D-101 inheritance); DC2 blocked on the same NetBox literals as Stages 3/4. Mandates lib_net_select_dc "$DC"/ lib_hosts_select_dc "$DC" before every adapted command block touching those libraries (DOCFIX-151's first real consumer). Surfaces two genuine open design gaps rather than inventing around them: the D-101 IPv6 family-matrix overlay (exact charm config option names unconfirmed) and D-109's IPv6-SAN cert-issuance charm support (unverified assumption). All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED (prep-only session, no live infrastructure reachable).
• DONE -- Stages 4, 6, 7 runbooks WRITTEN 2026-07-09 -- ALL SEVEN Stage 1-7 runbooks now complete, tooling gap register item #9 CLOSED:
  • Stage 4 (changelog docs/changelog-20260709-dc-dc-phase3-runbook.md, DOCFIX-157): runbooks/dc-dc-phase3-maas-enlist-deploy.md. Mandates lib_net_select_dc "$DC" then HONESTLY documents lib_hosts_select_dc "$DC" as expected to fail for both DCs until this stage's own enrollment populates real host data -- a deliberate chicken-and-egg. Names 7 real gaps in its own section (no OpenTofu module stands up a per-DC MAAS rack controller VM; three scripts not $DC-parameterized; node sizing undecided; DC2 blocked at the selector layer; no owner for the per-DC mirror; VM-host-discovery MAAS status unconfirmed; DC0- specific tag/fabric literals).
  • Stage 6 (changelog docs/changelog-20260709-dc-dc-phase5-runbook.md, DOCFIX-158): runbooks/dc-dc-phase5-dr-failover-drill.md. Found three real bundle.yaml gaps by grepping it directly (no cinder-backup charm, no ceph-rbd-mirror charm, ceph-mon's rbd-mirror binding on the wrong plane) and makes fixing them the runbook's own first gated mutation. Includes a negative-control drill (partition-only, expected to correctly refuse promotion) before the real hard-down drill. Every Ceph object name is an explicit placeholder.
  • Stage 7 (changelog docs/changelog-20260709-dc-dc-phase6-runbook.md, DOCFIX-159): runbooks/dc-dc-phase6-designate-cos-magnum.md. Quotes the bundle's actual current "NO designate" text verbatim before describing the reversal; correctly identifies D-106's reactivation also reverses the bundle's separate B5 os-public-hostname-dropped posture (two coupled changes, not one); proposes per-DC hostname overlays to keep bundle.yaml DC-agnostic; quotes D-046's domain-setup trap verbatim twice; leaves COS's deployment mechanism as an explicit operator decision point rather than inventing one. All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED. Four new tooling-gap-register items (#12-15) added for the cross-cutting findings this whole runbook-authoring pass surfaced (Office1-local network missing, IPv6 family-matrix overlay missing, D-108 bundle gaps, Stage-4 MAAS/multi-rack gaps) -- see docs/dc-dc-deployment-workflow.md.
• DONE -- Adversarial review pass across the whole evening's work (changelog docs/changelog-20260709-adversarial-review-fixes.md, DOCFIX-160): ran a dedicated fresh-eyes review subagent (no authoring context) across lib-net.sh/lib-hosts.sh's new selectors, netbox/dc-dc-prefixes-import.py, and all seven Stage 1-7 runbooks, instructed to find real defects and cross-check every claim against ground truth directly (not trust the runbooks' own self-report) -- including running both new test harnesses live. Found and fixed 3 real issues: (1) a genuine Python control-flow bug in tests/dc-dc-prefixes- import/test_logic.py's expect_ok() -- a return inside try made the else clause unreachable dead code, silently leaving 3 happy-path assertions uncounted (37/37 was true for what ran, but overstated what was verified; now 40/40 with the fix, independently re-verified via a live python3 -c reproduction before trusting the agent's claim); (2) a self-contradictory pair of adjacent comments in dc-dc-prefixes- import.py about /19-vs-/22 arithmetic (no functional bug, just wrong prose); (3) stale "as of this writing" self-references in the Stage 4 runbook, written before Stage 3 existed and never reconciled once it did. Propagated the corrected 40/40 count into docs/dc-dc-deployment- workflow.md and this ledger's own earlier entry (left the original delivery changelog un-edited as historical record of what was believed at the time). The review explicitly found NOTHING wrong with: the $DC selector asymmetry description (consistent everywhere), the hand- verified ULA/GUA carve arithmetic, Stage 6's bundle.yaml claims (confirmed byte-for-byte), 3+ direct D-NNN quotes (verbatim-accurate), and DOCFIX/D-NNN numbering consistency across every file -- and found no inferred/invented literal value anywhere in the reviewed material. repo-lint 0 fail/1 documented warn; both affected test suites re-run clean (40/40 and 21/21) after the fixes.
• DONE -- Final consolidation pass (changelog docs/changelog-20260709-final-consolidation.md, DOCFIX-161), closing out tonight's autonomous session: a last full read-through of docs/dc-dc-deployment-workflow.md for internal cross-stage consistency found ONE MORE real gap -- no stage explicitly owns creating Office1's own OPNsense edge VM, despite the topology being explicit that the design is per-site (three sites, three edges: DC1, DC2, AND Office1) -- logged as gap #16, not silently decided. Also fixed two stale cross-references left over from earlier in the same evening (the design-doc summary row's confusing "DONE, PROPOSED" phrasing; the lib-net/lib-hosts row's "Stage 5 runbook still needs to call it," now true of both Stage 4 and Stage 5). Companion Artifact redeployed twice more: all seven stage pills moved to a new "Runbook written" state (a 4th click-cycle state added, distinct from not-started/in-progress/ done), gap register items 1/3/9 marked closed and 12-16 added, and the 37/37-to-40/40 test-count correction propagated in. Final full gauntlet re-run: same 23 pre-existing environment-gap failures as every earlier check tonight (jq missing / no live juju-cloud access on this workstation) -- zero regressions across the entire evening's cumulative changes. repo-lint 0 fail/1 documented warn. Session summary for the operator's morning read: 11 commits (DOCFIX-151 through DOCFIX-161) delivered gap #1 (closed), gap #3's tooling half (closed, data half open), and all seven Stage 1-7 runbooks (gap #9 closed), plus an adversarial review pass and this consolidation. Nothing was executed against real infrastructure -- prep-only session, as the operator specified. The morning's first real action is Stage 1's runbook (runbooks/dc-dc-phase0-vcloud-prep.md) on the actual vcloud host. Genuinely open blockers, roughly in the order they'll be hit: gap #7 (MTU/Ceph disk-budget arithmetic), gap #16 (Office1 OPNsense edge ownership), gap #3's data half (NetBox literals), gap #2's UNVALIDATED status (no tofu binary has run yet), gaps #12-15 (Office1-local network, IPv6 family-matrix overlay, bundle.yaml DR gaps, MAAS/ multi-rack gaps).
• DONE -- Tooling gap register item #7 CLOSED 2026-07-09 (MTU/geneve + Ceph disk-budget calculators, changelog docs/changelog-20260709-mtu- ceph-budget-calculators.md, DOCFIX-162 -- numbering deferred to the orchestrating session to avoid a collision with other parallel in-flight agents tonight): scripts/dc-dc-mtu-geneve-budget.sh (required --underlay-mtu, no default) reproduces D-101's own worked example exactly (1500 - 56 = 1444), leaves tenant MTU at 1500 for a measured jumbo (>=9000) underlay, and adds a clearly-separated IPv6-minimum-link-MTU (1280, RFC 8200) sanity floor beyond D-101's own text. scripts/dc-dc-ceph-disk-budget.sh (required --total-disk, per-DC --dcN-nodes/--dcN-per-node-osd, and required --backup-overhead-fraction with deliberately NO default -- not a hard number anywhere in this repo, must come from real DC1 Ceph/radosgw-admin measurement) computes whether size=3 across DC1+DC2 plus overhead fits the measured total disk, and if not, names size=2 as the ONLY documented fallback (D-101/Section 3) without silently applying it -- an explicit operator-logged decision. New harnesses: tests/dc-dc-mtu-geneve-budget/ run-tests.sh 19/19 PASS, tests/dc-dc-ceph-disk-budget/run-tests.sh 16/16 PASS -- both cover the exact worked/constructed numeric examples, the jumbo/fits vs fallback branches, and every required-arg/bad-format fail-loud path. bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn (unchanged). Both scripts are read-only calculators (mutate nothing) and have not yet been run against the real vcloud host -- that first real run is part of executing runbooks/dc-dc-phase0-vcloud- prep.md Step 3. Uncommitted -- operator has not yet asked to commit this piece; the orchestrating session will assign the real DOCFIX number and find-replace DOCFIX-162 at integration time.
• DONE -- Tooling gap register items #12 and #16 CLOSED 2026-07-09 (changelog docs/changelog-20260709-office1-network-edge.md, DOCFIX-163 -- numbering deferred to the orchestrating session, same reason as the #7 item above): built opentofu/modules/office1-network (a new, dc-planes-shaped isolated libvirt_network sized for ONE network, not six planes -- closes gap #12) and INSTANTIATED it for real in root main.tf (module "office1_network", needs no unmeasured value beyond already-real domain_suffix/underlay_mtu). Design decision: a new module, deliberately NOT a reused host bridge -- a bridge would be the one network in this topology living outside OpenTofu's D-103-mandated inventory and would silently break this repo's own virsh net-list --all completeness assumption; full reasoning in the module's own header and in runbooks/dc-dc-phase1-office1-standup.md's Open question #1 (now marked RESOLVED). Also decided gap #16 (Office1 OPNsense edge ownership): YES, Office1 gets its own modules/opnsense-edge call, a fourth site alongside DC1/DC2, and Stage 2's own runbook owns creating it (new Step 4b, CHECK-before-MUTATION shape, its own GATE bullet) -- Stage 2's whole scope is already "Office1 headend standup" and Stage 1 already builds this edge's network neighbors, so Stage 3 has no reason to reach back into Office1's own infrastructure. A commented-out module "office1_opnsense" skeleton now exists in main.tf, mirroring Stage 3's dc1_opnsense template exactly; its lan_network_name resolves cleanly to the new network, but its wan_network_name does not -- investigated rather than invented, and found to be a genuinely DEEPER, cross-site gap: Stage 3's own dc1_opnsense template carries the identical unresolved wan_network_name placeholder, confirming no dedicated per-site ISP-uplink/WAN network exists yet for ANY site (DC1/DC2 included). Logged as NEW tooling gap register item #17 rather than defaulted to a mesh-link network, which would have directly contradicted D-100's own sub-item ruling that the three mesh legs carry management traffic only. bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn (unchanged); opentofu/ remains UNVALIDATED overall (no tofu binary available this session, same standing status as every other module here -- not newly introduced by this delivery). Uncommitted -- operator has not yet asked to commit; the orchestrating session will assign the real DOCFIX number and find-replace DOCFIX-163 at integration time.
• DONE (uncommitted) -- Gap #13 (D-101 IPv6 family-matrix overlay) DRAFTED, one real risk left open, researched directly by the orchestrating session (not delegated) given the accuracy stakes: overlays/dc-dc-ipv6-family-matrix.yaml + docs/dc-dc-ipv6-charm- research.md (full sourcing). Fetched REAL charm config/source, not memory: charm-keystone/config.yaml confirms prefer-ipv6 (bool) is a real, shared charms.openstack option; charm-nova-cloud-controller's actual "Dual Stack VIPs" commit confirms it's genuinely ADDITIVE (HAProxy binds both v4 and v6 wildcard addresses simultaneously), not an either/or switch -- resolving the biggest open question about whether D-101's dual-stack ambition is even mechanically possible. charm-ceph- mon/config.yaml confirms its OWN separate prefer-ipv6 + ceph-public-network/ceph-cluster-network, which per the charm's own docs IS a straight switch (not additive) -- the correct shape for the ULA-only storage/replication planes. Fetched the FULL charm-layer- ovn/config.yaml (25 options) and confirmed NO IPv6/encapsulation option exists at all -- OVN needs no overlay entry, geneve family follows the bound interface once the plane is ULA-only. Fetched charm-vault's actual vault_pki.py sort_sans() code and confirmed no IPv4/IPv6 distinction in cert-issuance -- D-109's IPv6-SAN requirement is code-confirmed, not assumed. Found a real, open upstream risk while researching, not invented or glossed over: Octavia's lb-mgmt-net IPv6 support has two real, still-referenced Launchpad bugs (#1911788, #1913409) describing failures -- #1911788's root cause is an OVN/LXD/ MAAS hostname-resolution mismatch, the SAME CLASS of problem this repo's own D-008 bootstrap order already hardens against, so not necessarily fatal, but genuinely unresolved. The overlay deliberately excludes an Octavia entry rather than silently forcing the risk through. Updated Stage 5's runbook (runbooks/dc-dc-phase4-juju-bundle-per-dc.md) Step 6 and the tooling gap register item #13 to reflect all of this. bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn. Changelog: docs/changelog-20260709-ipv6-family-matrix-overlay.md, DOCFIX-164 (number confirmed and closed out here -- this line previously ended with unresolved "will assign at integration time" boilerplate; caught by the full-project-sweep's DC-DC-runbook-consistency agent as an integration miss, fixed 2026-07-10).
• DONE -- $DC parameterization for the 3 MAAS scripts (changelog docs/changelog-20260709-maas-scripts-dc-param.md, DOCFIX-166), closing gap #15 sub-item 2's CLI-invocation half: scripts/reenroll- hosts.sh, scripts/carve-host-interfaces.sh, and scripts/phase-00-maas- standup.sh now accept an opt-in $DC env var and call lib_net_select_dc / lib_hosts_select_dc immediately after sourcing, matching the same convention (and the same dc1-network-no-op/dc1-hosts-fail asymmetry) DOCFIX-151 gave the two library files the night before. Unset $DC is provably byte-for-byte unchanged (verified by direct, jq-independent invocation of all three scripts across the unset/dc0/dc1/dc2/bogus matrix, since the selector calls sit before any MAAS/jq touch-point in every script). Added $DC test cases to both existing harnesses (carve-host-interfaces, phase-00-maas-standup) and created a new one (reenroll-hosts had none before). Did NOT invent any per-DC host/CIDR literal, and did NOT make phase-00-maas-standup.sh's hardcoded PLANES table dc2-aware -- both remain honestly open (dc1 only "works" there because D-101 makes its real target identical to dc0's). Did not run a git stash A/B this time since the working tree carries other parallel agents' in-flight uncommitted work tonight; verified directly instead. repo-lint 0 fail/1 documented warn. docs/dc-dc-deployment-workflow.md item #15 and runbooks/dc-dc-phase3-maas-enlist-deploy.md (gap #2, Steps 5 and 8, delivery checklist) updated to reflect the partial closure -- six of the seven gap-#15 sub-findings remain open, said plainly rather than overclaiming. No live infrastructure touched. (DOCFIX-166, already cited above -- this closing line's stale "will assign at integration time" boilerplate corrected 2026-07-10, flagged by the same sweep pass.)
• **DONE (uncommitted) -- Gaps #6 + #14 (bundle.yaml: Designate reactivation
  • D-108 DR mechanism), done directly by the orchestrating session given the stakes (this is the SAME bundle.yaml the jumphost operates the live VR0/DC0 rehearsal cloud with, not a DC-DC-only file): bundle.yaml gained designate/designate-bind/designate-mysql- router/designate-hacluster (D-106) and cinder-backup/ceph-rbd- mirror (D-108) + ceph-mon's rbd-mirror binding corrected from storage to replication. Caught and fixed three real mistakes during self-review, by fetching each new charm's REAL metadata.yaml rather than trusting the standard-pattern draft or the Stage 7 runbook's own earlier "template" section: (1) designate's real provides-side endpoint is dnsaas, not public like the other API charms; (2) cinder-backup is a SUBORDINATE charm (confirmed subordinate: true) -- the first draft wrongly gave it num_units/to/an amqp relation that doesn't exist on this charm at all; (3) ceph-rbd-mirror is a principal with ceph-local/ceph-remote requires endpoints (confirmed subordinate: false) -- the first draft invented a single wrong binding key. A genuine, unresolved cross-DC wiring limit documented rather than worked around: ceph-remote (the peer DC's ceph-mon) cannot be a same-bundle relation given D-104's per-DC-independent-Juju-controller design -- only ceph-local (this DC's own leg) is wired; the cross-DC peer relationship needs juju offer/consume or the manual rbd mirror pool peer bootstrap CLI flow Stage 6's runbook already documents. Deliberately did NOT reactivate os-public-hostname on the 11 API charms in this shared bundle (D-106's OTHER coupled change) -- pushed to a proposed per-DC overlay instead, to keep bundle.yaml itself DC-agnostic. Verified: python3 -c "import yaml; yaml.safe_load( open('bundle.yaml'))" parses clean (56 apps, 108 relations); python3 scripts/provider-bundle-check.py PASS, all 6 existing invariants unaffected (tests/provider-bundle-check/: 8/8 unaffected); bash scripts/repo-lint.sh 0 fail/1 documented warn. Reconciled Stage 6's and Stage 7's runbooks to point at the now-real bundle content instead of their earlier "template"/"known gap" framing. NOT applied to any live model this session -- no live cloud reachable; this is a repo-source change only, to be reviewed via juju deploy --dry-run before any real apply. Changelog docs/changelog-20260709-designate-cinderbackup- rbdmirror.md, DOCFIX-167** (number confirmed and closed out here -- this line previously ended with unresolved "will assign at integration time" boilerplate; caught as an integration miss by the full-project sweep's DC-DC-runbook-consistency agent, fixed 2026-07-10).
• DONE (uncommitted) -- Tooling gap register item #5 MECHANISM CLOSED 2026-07-09 (changelog docs/changelog-20260709-ceph-replication- tooling.md, DOCFIX-165 -- numbering deferred to the orchestrating session, same reason as the other parallel deliveries tonight): the radosgw multisite / rbd-mirror command sequences already written out in runbooks/dc-dc-phase5-dr-failover-drill.md Steps 4, 5, 8, 10.2, and 11.2-11.4 (verbatim upstream Ceph administration, not invented) are now real, callable scripts: scripts/dc-dc-radosgw-multisite.sh (master-init/join-readonly/enable-two-way), scripts/dc-dc-rbd- mirror.sh (bootstrap-primary/bootstrap-secondary --direction rx-only|rx-tx), and scripts/dc-dc-dr-drill.sh (failover/failback). Every realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument -- no invented defaults, matching the runbook's own <UPPER_SNAKE_TBD> placeholder discipline. Default mode is --dry-run (prints the exact juju ssh/juju run command sequence, secrets redacted); --apply executes one command at a time, aborting on first failure. The single most safety-critical property in the whole drill -- demote the CURRENT PRIMARY before promoting the RECOVERING side during failback -- is HARD-CODED as fixed call order in dc-dc-dr-drill.sh's failback subcommand, not left to documentation or an operator-orderable flag; in apply mode the promote step is only ever reached if the preceding demote succeeded. Step 10.2's per-image Glance re-registration is always printed as a flagged MANUAL reminder and NEVER auto-executed (the runbook is explicit the real invocation depends on unconfirmed Glance driver behavior); Step 11.3's Cinder reconciliation is likewise not scripted (tenant/volume-specific). Each script sources scripts/lib- net.sh and calls lib_net_select_dc per DC token touched (DOCFIX-151 convention) -- informational in dry-run (a plan preview is not a mutation), hard-blocking before any --apply (dc2 --apply refused today, exit 3, since D-101/gap #3 leaves DC2 with no assigned network literals). New harnesses tests/dc-dc-radosgw-multisite/, tests/dc-dc-rbd-mirror/, tests/dc-dc-dr-drill/ (19/19, 19/19, 23/23 PASS) test ONLY argument parsing, guard clauses, and dry-run output -- including a line-order grep proving the failback demote-before-promote property -- NOT real radosgw/rbd-mirror behavior; no live Ceph cluster exists this session to test against (same UNVALIDATED posture as every other tooling delivery tonight). Stage 6 runbook's own "Follow-up, NOT built now" delivery-checklist item and docs/dc-dc-deployment- workflow.md item #5 both updated to point at these scripts instead of describing them as future work. bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn (unchanged). No live infrastructure touched. Uncommitted -- orchestrating session will assign the real DOCFIX number and find-replace DOCFIX-165 at integration time.
• DONE (uncommitted) -- Gaps #4(d)/#11: proposed netem parameters + ULA/ GUA/DC2-supernet generation guidance (changelog docs/changelog- 20260709-netem-ula-gua-proposal.md, DOCFIX-168), docs/dc-dc-netem-and-ula-gua- proposal.md: does NOT rule the netem decision or generate any of the three real NetBox literals itself -- presents a concrete, reasoned proposal (1ms delay / 0.2ms jitter / 0.01% loss / uncapped rate, each with rationale grounded in the buildout design's own "same-metro dark fiber, low single-digit ms" lean) for OPERATOR ratification, and gives the exact safe command (openssl rand -hex 5, RFC 4193-compliant) for the operator to generate the real org ULA /48 themselves. Explicitly does NOT attempt to generate the per-DC GUA carve or DC2's supernet -- both require real coordination (the ARIN-block administrator; NetBox's full address inventory) this session has no authority or visibility to perform; the section instead explains precisely what each requires and how to close it once ready. docs/dc-dc-deployment-workflow.md gap items #4(d)/#11 updated to reference it; item #10 (harness debt) also corrected -- LARGELY CLOSED given how many new test suites landed tonight (dc-selector, dc-dc-prefixes-import, dc-dc-mtu-geneve-budget, dc-dc-ceph-disk-budget, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, dc-dc-dr-drill, plus MAAS script additions). bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn. No live infrastructure touched. Uncommitted -- orchestrating session will assign the real DOCFIX number at integration time.
• DONE -- Full-project sweep for errors/architectural problems, requested after the seven gap-closure commits: FIRST finding, fixed (changelog docs/changelog-20260710-phase01-bundle-staleness-fix.md, DOCFIX-169): runbooks/phase-01-bundle-deploy.md -- an EXISTING, LIVE-CLOUD-relevant VR0 runbook, not a DC-DC artifact -- had gone stale the moment DOCFIX-167 edited the SAME shared bundle.yaml it deploys. Its "expected plan: 50 apps, 97 relations" (Constants section + Step 1.2's dry-run GATE) was wrong (real count now 56/108); its GATE's "NO designate (D-019)" text flatly contradicted the new reality. Fixed both LIVE-gate references; deliberately left the historical "As-built reference (2026-06-03)" section's own 50 apps/97 relations line UNCHANGED (append-only audit- trail discipline -- it documents a real past deploy, not something to retroactively edit). Flagged a real, unresolved architectural tension rather than silently picking a side: this bundle now serves BOTH a plain VR0-only redeploy AND standing up DC1, and reactivating Designate/the DR mechanism in the shared file means any future phase-01 run gets them regardless of intent -- a future plain-VR0 redeploy needing to exclude them would need an explicit override, not an assumption either way. Extended the EXIT GATE's "waiting on vault certs" list to include designate/designate-bind (inferred from their real certificates relation to vault, clearly labeled NOT YET OBSERVED against a live deploy). bash scripts/repo-lint.sh: 0 fail, 1 documented legacy warn. Sweep continues across the rest of the repo (scripts, other runbooks, OpenTofu modules, test quality, security/secrets posture).
• DONE -- Full-project sweep COMPLETE: 8 parallel agents (bundle.yaml, VR0 runbooks, scripts, OpenTofu modules, DC-DC runbook consistency, test harness quality, security/secrets, NetBox/IPAM), all findings triaged and real ones fixed, changelogs DOCFIX-170/171/172/173:
  • DOCFIX-170 (docs/changelog-20260710-sweep-vr0-runbook-opentofu- fixes.md): phase-08-workload-cluster-acceptance.md's Designate framing (same cascading-staleness class as DOCFIX-169), nuanced not a blanket flip -- charms now active/idle-checkable, but D-011.8's real resolution criterion still needs Stage 7's os-public-hostname overlay, not automatic from deployment alone. Plus a stale opentofu/modules/ base-image/outputs.tf comment (cosmetic, no functional effect).
  • DOCFIX-171 (docs/changelog-20260710-sweep-dcdc-runbook-fixes.md): Stage 1's calculator scripts (DOCFIX-162) never wired into its own Step 3 -- fixed to actually invoke them; Stage 5's Sequence-list still said [NOT YET WRITTEN] for the IPv6 overlay after the Step 6 body was updated -- fixed; Stage 6's checklist claimed its steps call the new Ceph scripts when the step bodies still only showed raw commands -- added real "PREFER scripts/dc-dc-..." pointers to Steps 4, 5, 8, 10.2, 11.2 themselves, closing the gap between claim and text. Also closed out two orphaned ledger entries (DOCFIX-164, 167) whose closing lines never got their real number written in during the earlier integration pass -- an integration miss the sweep itself caught.
  • DOCFIX-172 (docs/changelog-20260710-sweep-bundle-fixes.md): magnum-dashboard had NO bindings: block AT ALL -- a real, pre-existing bug (confirmed via git log, present since the D-052 explicit-bindings commit, not tonight's regression) violating this bundle's own stated "explicit per-application blocks" discipline. Fixed with a minimal, metadata-confirmed block (subordinate, only a real dashboard endpoint -- NOT a copy of octavia-dashboard's fuller shape, which has a certificates endpoint this charm doesn't). Also relocated two section-header comments that sat before the WRONG app's bindings block (functionally harmless, confirmed via yaml.safe_load -- purely a future-edit-risk fix) and corrected the header's stale "11 API charms" VIP-chain count to 12 (with the VIP-triple count of 11 correctly left alone, since designate doesn't carry a vip: option). provider-bundle-check.py PASS throughout, 56 apps/108 relations unchanged.
  • DOCFIX-173 (docs/changelog-20260710-sweep-script-fixes.md): 4 confirmed logic bugs + 1 security hardening, ALL independently reproduced in isolation before being trusted (the same discipline that caught the dc-dc-prefixes-import bug earlier tonight) -- including catching that a FIRST proposed fix for one of them (scripts/phase-06-capi-stack.sh) was itself wrong: || true after a failing pipeline under pipefail silently discards PIPESTATUS, which would have made the "GATE FAIL" check never fire again, WORSE than the original bug. The correct if pipeline; then :; fi idiom was verified via reproduction before committing. Full list: carve-host- interfaces.sh's emit() returning 1 unguarded under set -e (killed apply-mode runs on the first MAAS error, before the FATAL-accumulator summary could print); phase-06-capi-stack.sh's PIPESTATUS read (as above); juju-spaces-check.sh and osd-blank-check.sh's one unguarded display-only pipe each (both safely || true-able since nothing downstream needed their PIPESTATUS); scripts/ledger-scan.sh's DOCFIX/BUNDLEFIX regex using a fixed {3} instead of {3,} -- the SAME bug class this repo already got bitten by once for D-numbers (fixed there via {3,}, never mirrored here) -- a dormant landmine, not yet reachable (repo is at DOCFIX-170ish) but real; and dc-dc-rbd-mirror.sh's bootstrap token using a predictable /tmp path with no permission hardening -- hardened to a random-suffixed path + immediate chmod 600. Full test gauntlet re-run after all fixes: 23/51 FAILED, identical to every prior check tonight (same pre-existing jq-missing environment gaps) -- zero regressions across the entire sweep-and-fix pass. Also flagged, not a repo issue: the scripts-audit agent reported that a skill's tool output contained injected text resembling a system reminder, instructing it to conceal a date change from the operator -- it correctly disregarded the concealment instruction and reported the anomaly rather than act on it. The main session observed an identically-worded, identically-formatted message earlier this same evening (a genuine date-rollover notice, consistent with the session crossing midnight) and treated it as benign/expected rather than a live attack -- noted here for the operator's own awareness per this environment's prompt-injection disclosure norm, not because either session took any unsafe action in response.
• DOCFIX-174 (docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md): caught before assigning the next real DOCFIX number for follow-up work -- bash scripts/ledger-scan.sh reported DOCFIX next-free=1005, wildly implausible against a repo at DOCFIX-173. Root cause: DOCFIX-173's OWN changelog narrated its {3}->{3,} regex fix using a literal illustrative example (the real prefix plus one-zero-zero-four), which the just-widened regex then matched as a real assignment -- a false-positive one commit after fixing a false-negative of the opposite kind. Fixed: reworded the illustrative example in prose (both the DOCFIX-173 changelog and this file's own mirrored comment), added a reproduc-line exclusion to nextfree_mentions() as defense-in-depth (this repo's delivery style narrates bug reproductions constantly, so this will recur), and added a regression fixture to tests/ledger-scan/run-tests.sh (37/37 PASS, up from 33). True next-free after this delivery: D-111, DOCFIX-175, BUNDLEFIX-012. Caught the same way as everything else tonight: by running the tool and checking whether its own output was plausible before trusting it, not by code review.
• DOCFIX-177 (docs/changelog-20260710-skill-vr1-awareness.md): operator was heading to the vcloud host to begin real execution; asked "is the skill ready?" -- it wasn't fully. .claude/skills/openstack-cloud- ops/SKILL.md and runbooks/README.md (what its routing table points to) had ZERO mentions of the entire VR1 DC-DC track -- a fresh session reading only the skill would be routed to VR0's phase-01..08 runbooks with no indication the DC-DC work (today's actual mission) exists. Fixed: runbooks/README.md gained a full "VR1 DC-DC track" index table (7 Stage runbooks + dc-dc-teardown-rollback.md, explicit that its numbering is a separate sequence from VR0's), SKILL.md's routing table + standard loops gained VR1-specific rows, references/environment.md gained VR1 pointers (a third "deployment" entry + a repo-map addition), references/ troubleshooting.md gained a note that appendix-A is VR0-only and VR1 has no incident index yet. Also found and fixed first: main was 2 commits ahead of origin/main (DOCFIX-174, 175/176) -- pushed before this delivery, since the vcloud host's session bootstrap starts with git pull and would otherwise have missed them this morning.
• DOCFIX-176 (docs/changelog-20260710-vr1-teardown-rollback-runbook.md): closes gap register item #19. New runbooks/dc-dc-teardown-rollback.md -- the OpenTofu/libvirt/MAAS-vm_host layer had no teardown runbook (D-061's scripts cover a DIFFERENT layer, the juju/MAAS-machine one). Carries D-061's PRINCIPLE (MAAS-side cleanup before destroying the libvirt resource underneath) but not its specific mechanism (this repo doesn't use MAAS pod-composition for OpenTofu-created VMs, so --keep-instance doesn't directly transfer) -- flagged as untested/residual since no VR1 DC has reached Stage 4 yet. Two paths (scoped -target teardown with HashiCorp's own targeting-risk warning quoted, or full VR1 destroy), a rollback decision tree favoring fix-forward over destroy for partial apply failures, explicit mesh-link shared-infrastructure handling, and confirmed (by reading the actual .tf files) that netem-link's destroy-time provisioner already works correctly. Caught and fixed mid-delivery: re-running ledger-scan.sh before assigning this number found DOCFIX-174's OWN changelog/ledger narration had reintroduced the exact false-positive class it had just fixed (quoting the illustrative example literally again) -- fixed by rewording, folded into DOCFIX-174's existing changelog rather than a new number, since it corrects that delivery before commit. Also confirmed DOCFIX-175's Stage-1-not-Stage-3 correction (see its own entry above) was fully propagated before this runbook's cross-references were written.
• DOCFIX-175 (docs/changelog-20260710-opentofu-state-file-security.md): answering the operator's own "what have we missed" question surfaced a real, previously-unlogged gap -- terraform.tfstate stores var.maas_api_key in plaintext regardless of sensitive = true (that flag only suppresses CLI output), and nothing gitignored the state file, documented backup/permission handling, or noted local state's lack of locking. Fixed: .gitignore excludes opentofu/**/*.tfstate* + related local artifacts, opentofu/README.md gained a full "State file handling" section, variables.tf's maas_api_key description carries the caveat, and dc-dc-phase2-tofu-dc-substrate.md Step 8 (the actual first real tofu apply in this repo) gained a "SECURE THE STATE FILE" callout right after its GATE line -- closing the loop rather than leaving a "someone should add this" note dangling. Gap register gained item #18 (this, CLOSED) and #19 (VR1 teardown/rollback runbook -- OPEN, tracked as the next delivery). Remote backend + real locking explicitly flagged as Roosevelt/production-scope, not invented here.

Jumphost stream -- session 2026-07-10 (cont.): VR1 Stage 1 FIRST REAL EXECUTION (dc-dc-phase0)

First-ever execution of a DC-DC runbook against real infrastructure. Logged session dc-dc-phase0-vcloud-prep (~/as-executed/2026-07-10-dc-dc-phase0-vcloud-prep.log). On the ACTUAL vcloud host this time (hostname vcloud, AMD, 256 vCPU / 1 TiB / 10 TiB, Ubuntu 24.04) -- NOT the Windows box the 2026-07-09 session flagged.

• Steps 1-3 DONE (read-only audit + recorded decisions):
  • Step 1: on-host, libvirt reachable qemu:///system, user in libvirt group. libvirt_uri="qemu:///system".
  • Step 2: nested KVM ALREADY ON (kvm_amd/nested=1) -> Step 4 SKIPPED. No running domains.
  • Step 3: uplink enp1s0 MTU 1500. Operator ruling: underlay_mtu=9000 (jumbo) for the host-internal dark-fiber/plane fabric (the module planes are addressless isolated virtio bridges, jumbo-capable regardless of the 1500 ISP uplink; 1500 governs future WAN-edge nets only). Ceph size=3 disk-budget PLAUSIBILITY PASS (4.60 TiB margin @ conservative 9 TiB total, 3 OSD nodes/DC x 500G, 0.5 overhead) -- real budget deferred to Stage 5.
• FINDING: host was NOT pristine -- carried a prior hand-built VR0 topology (7 vr0-dc0-* planes + vr0-off0-1_lan + vr0-ptp-dc0-off0 + a NAT wan net) and the default pool (/var/lib/libvirt/images). Operator ruled: tear down the 9 orphaned vr0 nets, KEEP wan (dual-stack NAT-out-enp1s0, GUA 2602:f3e2:fe:10::/64 within the org ARIN block -- a ready model for gap #17's ISP-uplink network; Stage-2 finding). Teardown DONE, gated (net 1 individually, nets 2-9 after durable approval, stop-on-anomaly). Only wan + default remain.
• Step 5 DONE: pool dirs /var/lib/libvirt/vr1/{dc1,office1,dc2}. Operator granted a self-owned parent /var/lib/libvirt/vr1 (setgid libvirt) so agent creates subdirs without sudo. Non-secret session env ~/vr1-stage1.env created (TFVAR* + log path; NEVER holds maas_api_key).
• Step 7 DONE (repo wiring): opentofu/dc-dc-phase0.auto.tfvars (gitignored, DOCFIX-175) from measured values; module "dc2_storage" ACTIVATED + dc2_pool_path var added (operator: "wire dc2 while we're here"). DC2 PLANES DEFERRED (Option B, operator ruling) -- wait for NetBox to assign the D-101 supernet/ULA/GUA; do not invent CIDRs. DC2 mesh legs already wired.
• DELIVERED -- DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers (opentofu/virtualization/jq/image-tools + check-prereqs + install-all
  • lib + README) + tests/prereqs/run-tests.sh (24/24) + a Prerequisites section in the phase0 runbook + Step 6 concretized. Operator ruling: runbooks must not assume prereq runtimes; give new operators install scripts. repo-lint 0 fail. Changelog docs/changelog-20260710-prereq-install-scripts.md.
• Steps 6-11 DONE -- STAGE 1 COMPLETE (2026-07-10). tofu installed via scripts/prereqs/install-opentofu.sh (OpenTofu v1.12.3). Step 8 init/validate needed DOCFIX-179 fixes: per-module required_providers (child modules do NOT inherit provider SOURCE, OpenTofu inferred nonexistent hashicorp/libvirt) + tofu fmt of the never-fmt'd tree. Step 9 plan: the provider "maas" Known Gap demanded BOTH maas_api_url + the SENSITIVE maas_api_key for a zero-MAAS plan -> operator ruled STRUCTURAL FIX (Option B): deferred the provider "maas" block + both vars to Stage 3 (DOCFIX-179); maas stays in root required_providers (lock file complete). Step 10 apply: 13 objects, 0 changed/0 destroyed; state chmod 600 (no secret in it now). Step 11 gate: all 4 bullets PASS, MTU 9000 verified on all 10 nets. .terraform.lock.hcl to be committed.
• NOC redesign (mid-session): RAISED then RETRACTED by operator -- a 4-site regional model (Office0/NOC0/DC0/DC1, NOC-hub fiber) was floated, then withdrawn ("misunderstood an outside conversation; nothing changes"). No design change; existing VR1 (DC1/DC2/Office1 triangle) stands. Logged so the pivot isn't re-litigated.
• DOCFIX candidates ACCRUING for the stage-completion commit (draft runbook updates per operator's standing instruction): (1) Step-3 MTU-domain clarification (uplink 1500 vs jumbo internal underlay); (2) non-pristine-host + wan-as-gap-#17 note for Steps 5/11; (3) Step-9 expected-resource list is stale (omits office1_network) and now also gains dc2_storage -> real plan is 13 resources: 6 dc1 planes + 3 pools (dc1/office1/dc2) + 4 nets (office1-local + 3 mesh); (4) Step-7 tfvars-vs-gitignore (DOCFIX-175 gitignores what Step 7 says to commit); (5) other dc-dc-phaseN runbooks want a scripts/prereqs/ pointer (per DOCFIX-178 follow-up).
• DELIVERED -- DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at Stage-1 close (operator model: DOCFIX-log through a phase, update the runbook at phase end). Top banner + Known-gap RESOLVED (maas deferral) + Step-3 two-MTU-domains + Step-5 non-pristine-host/wan/pool-parent + Step-8 init/required_providers + Step-9 -input=false and 13-resource plan. Changelog docs/changelog-20260710-phase0-runbook-as-executed.md. repo-lint 0 fail.
• Numbers after DOCFIX-180: DOCFIX next-free 181 (re-scan before assigning).
• STAGE 1 CLOSED + COMMITTED. Branch dc-dc-stage1-phase0-first-exec (340ac58): DOCFIX-178/179/180. NOT pushed. Git identity set repo-local (JANeumatrix/jesse.austin@neumatrix.com, matching prior commits) -- this host had none.
• DELIVERED -- DOCFIX-181: carve_gua subnet-enumeration hang FIXED. The full run-tests-all.sh gauntlet was HANGING on tests/dc-dc-prefixes-import (its long-claimed "40/40" was never actually reached -- it hung). Root cause: netbox/dc-dc-prefixes-import.py::carve_gua() did list(prefix.subnets(new_prefix=64)) -- ~16.7M /64 objects for a /40 GUA -- when it only uses the first. Fixed with next() (O(1)); added a faulthandler.dump_traceback_later(30, exit=True) watchdog to test_logic.py so any future hang self-aborts with a traceback instead of stalling the gauntlet. Harness 40/40 in 0.05s; full gauntlet now completes green. Changelog docs/changelog-20260710-carve-gua-hang-fix.md. Next-free -> 182.
• Next: Stage 2 (dc-dc-phase1-office1-standup) when directed -- watch gap #17 (per-site ISP-uplink/WAN net; wan is now a candidate model).

Jumphost stream -- session 2026-07-11: VR1 NetBox buildout (Phase 1) -- IPv6 sync + D-111

• SYNCED to the Chat/planning stream. Merged origin/main (057c31b "netbox uploads": docs/dc-dc-netbox-buildout-scope.md [AUTHORITATIVE], netbox/roles-aggregates-import.py, changelog) into the stage1 branch (clean; keeps the DOCFIX-181 carve_gua fix). Read the scope doc + tool fully.
• ARCHITECTURE CLARIFIED (operator): workflow is draft-source netbox.baldurkeep.com -> spin up a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> SIMULATE VR1 against the Office1 NetBox -> feed validated refinements BACK to netbox.baldurkeep.com. The sim NEVER --commits to the real NetBox (read=source, write=feedback-only). Scope doc + Stage 2 runbook need a note capturing this loop (pending).
• Read-only draft export DONE (netbox.baldurkeep.com v4.5.8; session token stored 0600 at ~/.vr1-netbox.env, NEVER printed -- REVOKE + purge at session end). Real modeling: VR1 = 2602:f3e2:f00::/40, DC0=f02::/48, DC1=f03::/48 (empty containers). VR0-DC0 (e02::/48) is the fully-built TEMPLATE: NN mnemonic (provider 10/VIP 11, metal 20, data 30, storage 40, repl 50, lbaas 80, vpn e0, oob f0), /60+/64 per plane, all-GUA (old model). Edge Networks 2602:f3e2:fe::/48 exists (gap-#17 WAN home; wan uses fe:10). ULA /48 not assigned (G3); aggregates empty (G2, top blocks are container PREFIXES).
• D-111 ADOPTED + IMPLEMENTED (DOCFIX-182): dc-dc-prefixes-import.py v6 subcarve aligned to the deployed NN mnemonic (/60+/64 per plane; metal /60 shared by admin :20/internal :21; ULA /56 indexed to GUA nibble so 4th hextet reads DC.NN). carve_ula+carve_gua -> carve_v6 via direct-math _sub_at (no enumeration; DOCFIX-181 property kept). Harness 40 -> 53 PASS, repo-lint 0 fail.
• FIXED (DOCFIX-183): docs/dc-dc-netem-and-ula-gua-proposal.md's ULA-gen sed emitted INVALID IPv6 (4+6 split); corrected to 2+4+4. Ratified VR1 ULA is fd50:840e:74e2::/48 (from the scope doc), so the command is reference-only.
• Address values now come from REAL NetBox (supersede earlier invented proposals): ORG_ULA_48 recommend fd50:840e:74e2::/48, GUA dc1 f02::/48 / dc2 f03::/48, DC2_V4_SUPERNET 10.13.0.0/19 -- pending final import against the Office1 NetBox.
• NEXT: (a) fix the proposal-doc ULA-gen command; (b) stand up the Office1 NetBox VM (Stage 2: Office1-local addressing + Option-A cloud-init + NetBox Docker; pynetbox/python3-venv NOT installed -- needs operator sudo); (c) G5 site rename (vr1-dc0/vr1-dc1 -> vr1-dc1/vr1-dc2) as its own D-number; (d) run roles-aggregates + prefixes import against the Office1 NetBox.
• Numbers: D next-free 112, DOCFIX next-free 184. Branch dc-dc-stage1-phase0-first-exec now also carries the merge + D-111 + DOCFIX-183.

Jumphost stream -- session 2026-07-12: Office1 OPNsense edge (bring the site router+DHCP online)

• Operator ruling: bring the Office1 OPNsense edge online FIRST (router + DHCP for office1-local) before the Office1 NetBox VM. Edges are the real router; do not hang service VMs on a wan shortcut.
• TRANSPORT MODEL CLARIFIED (operator, corrects an earlier misread): each site (both DCs + Office) simulates its OWN real ISP connection -- the OPNsense edge is a normal internet-facing router. Dark fiber (mesh links) = East-West / replication ONLY, not an internet path. D-107 node-airgap (per-DC mirror) is a SEPARATE DC concern, not the edge WAN egress. -> DOCFIX-185: stripped the egress-airgap rules (seq 20/21/99 + MIRROR_*) from opnsense-config.xml.tmpl + render script + harness (8/8). FOLLOW-UP: a D-100/D-107 amendment note so the Stage-3 DC edges are built to this model (operator to rule).
• Creds (closed test, agent-generated, 0600 at ~/vr1-office1-creds/, NEVER commit; operator to REVOKE the NetBox token + these at session end): SSH ed25519 keypair + OPNsense root password + bcrypt hash.
• Build DONE except boot: office1-wan NAT net 172.30.1.0/24->enp1s0 (virsh; D-103 debt); OPNsense 26.1 nano image prepped (DOCFIX-184: bz2 -> python3 fallback, bzip2 absent); config rendered (office1-opnsense, LAN 10.10.0.1/24, WAN static 172.30.1.2 -- NOTE: the "DHCP .100-.199" this line originally claimed was FALSE; no DHCP was ever in the template, see the 2026-07-12 D-112(c) entry gw .1) + ISO built; module "office1_opnsense" instantiated in main.tf + validated/planned (first real validation of modules/opnsense-edge); disk + config_seed volumes created.
• TWO first-boot findings (DOCFIX-186): (1) config_iso_path must be OUTSIDE the pool (create.content.url collides with its own target volume) -> staged at /var/lib/libvirt/vr1/staging/. (2) apparmor blocks /var/lib/libvirt/vr1/ (custom pool path not in libvirt's abstraction) -> domain defined-but-failed-to-start, qemu Permission-denied despite good POSIX perms. FOUNDATIONAL (all VR1 VMs). Fix = operator one-time sudo: echo '/var/lib/libvirt/vr1/** rwk,' | sudo tee /etc/apparmor.d/local/abstractions/libvirt-qemu; sudo systemctl reload apparmor && restart libvirtd. PREREQ candidate (gates every VR1 VM boot).
• NEXT: operator applies the apparmor rule -> clean up the partial apply (domain defined but not in tofu state) -> re-apply to boot -> verify WAN 172.30.1.2 + LAN DHCP/routing -> then the Office1 NetBox VM.
• Numbers: D next-free 112, DOCFIX next-free 187 (184/185/186 committed this session).

Jumphost stream -- session 2026-07-12 (cont.): OPNsense edge BOOT INCIDENT (OPEN) + context-limit checkpoint

• Office1 OPNsense edge boots but OPNsense TRIPLE-FAULTS at the BTX loader (262 bytes: /boot.config: -S115200 -h -D then dies loading /boot/loader). Double-nested virt (OPNsense -> vcloud VM -> outer hypervisor), AMD Opteron_G3. INCIDENT REPORT (read first on resume): docs/incident-20260712-opnsense-edge-boot-triplefault.md.
• DOCFIX-187 (committed): opnsense-edge module fixes -- serial console (nano is serial-only; was the module's biggest gap), q35->i440fx, COW-overlay->direct-copy disk, host-passthrough CPU + AMD svm disabled. All correct + kept; NONE resolved the boot.
• KEY GOTCHA: machine/cpu are create-time; the provider plans them in-place and libvirt ignores that -- must virsh destroy; virsh undefine; tofu apply to recreate. Provider CPU-feature key is features (plural); feature silently drops.
• NEXT (ranked, in the incident report): full CPU flags (+kvm_pv_eoi,+kvm_pv_unhalt via qemu_commandline), add a video device (-D dual-console + no VGA -> boot2 may fault), mem 2G->4G, UEFI/OVMF, or outer-hypervisor CPU investigation.
• Live state: office1-opnsense domain paused/faulted; office1-wan NAT net up; Stage-1 13 objects + edge disk/config-seed volumes exist. Creds ~/vr1-office1-creds/ (revoke at close, + the pasted NetBox token). apparmor rule for /var/lib/libvirt/vr1/** applied by operator (needed for any VR1 VM).
• CONTEXT-LIMIT CHECKPOINT. Session near max; resume in a fresh session from the incident report + this ledger. Numbers: D next-free 112, DOCFIX next-free 188.

Jumphost stream -- session 2026-07-12 (cont.): BOOT INCIDENT ROOT-CAUSED (DOCFIX-188)

• The triple-fault was NEVER a CPU/nesting/machine/console problem. The guest had 2 MiB of RAM. dmacvicar/libvirt >=0.9 changed memory from MiB (0.8-era meaning, which the modules were authored against) to raw libvirt units defaulting to KiB. No memory_unit => memory = 2048 rendered <memory unit='KiB'>2048</memory> => QEMU -m size=2048k = 2 MiB. Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo (Max memory: 2048 KiB) -> live QEMU cmdline. boot2 fits in 2 MiB and echoes /boot.config; /boot/loader does not fit -> deterministic 262-byte fault, immune to every knob the prior window turned.
• FOUNDATIONAL, not local to OPNsense: the identical one-line defect was latent in ALL THREE VM modules -- opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the literal NEXT step), node-vm (Stage 3 DC nodes). Every VR1 VM would have gotten 1024x too little RAM. Only opnsense-edge had been instantiated, which is why it alone detonated. Same class as the apparmor finding: it gates every VR1 VM.
• FIXED (DOCFIX-188): memory_unit = "MiB" on all three domains; memory_mib = 2048 value untouched (it now means the 2 GiB always intended). Disproven svm-is-root-cause comment in opnsense-edge CORRECTED (setting retained as legitimate hardening; the false claim would have misled the next session). Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
• GUARDED (DOCFIX-188): tofu validate CANNOT catch this (memory_unit is optional => omitting it is schema-valid -- exactly how it shipped). Added opentofu-validate.sh S1 static guard + --check-memory-unit flag (no tofu binary/network needed) + harness T3/T4/T5. T3 is a true negative test: the fixture IS the shipped defect and S1 rejects it. Harness 1 -> 4 PASS. Gauntlet ALL GREEN (52). repo-lint 0 fail.
• NEXT (operator-gated live step -- the ONLY live mutation pending): recreate the edge. tofu plan says 0 add / 1 change / 0 destroy (disk volume safe) BUT do not trust "update in-place" -- max-boot-memory is create-time to libvirt, the same trap DOCFIX-187 logged for machine/cpu. Must be: virsh destroy office1-opnsense; virsh undefine office1-opnsense; tofu apply. Re-run plan AFTER the undefine -- the captured in-place plan is NOT what executes; post-undefine it becomes 1 to add. Pass criteria (NOT "apply exited 0"): (1) AUTHORITATIVE -- virsh dominfo shows Max memory: 2097152 KiB. Do NOT anchor on the QEMU -m string: libvirt may normalize MiB to -m size=2097152k, which is SUCCESS. (2) serial log grows past 262 bytes; (3) router actually works -- WAN 172.30.1.2 up, LAN DHCP serving office1-local. FALLBACKS: wrong size => provider is not honouring memory_unit, use memory = var.memory_mib * 1024; boots but no working router => SIZING question (verify OPNsense 26.1's real minimum -- the incident report's "3 GB" is UNVERIFIED), bump to 4096; that is not a regression of this fix. Then: the Office1 NetBox VM (which cloudinit-vm now no longer breaks on).
• STANDING LESSON: a bootloader dying at a FIXED byte offset, immune to every knob, is a RESOURCE problem, not a CPU-feature problem. Read the domain XML + QEMU cmdline (ground truth) before theorising about nested virt.
• LOGGED-NOT-ACTIONED (DOCFIX candidate, no number consumed): run-tests-all.sh leaves the working tree dirty -- it chmod +x's the tests/*/fakebin/* stubs, so a gauntlet run produces 37 spurious 100644 -> 100755 mode-change diffs (zero content change). Had to git checkout them out of this session's commit. Fix is either commit the stubs 0755 or have the harness restore the mode; either way a gauntlet run should not dirty the tree.

• RE-AUDIT of the max-context session (DOCFIX-189, operator-instructed). That session authored docs+module comments while working from a wrong diagnosis, so its reasoning was re-checked, not just its one-line bug. Findings:

  • 4 FALSE CAUSAL CLAIMS corrected at the source (opnsense-edge/main.tf): the disk shape, q35->i440fx, the serial console, and svm-disable were EACH written up as a contributing cause. None changed the symptom -- the fault sat at a deterministic 262 bytes through all of them. All four SETTINGS are RETAINED (each defensible on its own merits); only the false justifications are gone. The serial console genuinely earned its keep -- it made the fault legible -- but it revealed the fault, it did not change it.
  • ONE SILENTLY DEAD KNOB (real functional defect): disk_size_bytes = 16 GiB in main.tf did NOTHING once DOCFIX-187 made the disk a direct copy -- unused variables are legal HCL, so validate/plan never complained. MEASURED: live disk is 11.00 GiB, not 16. Removed from main.tf + module variables; sizing is really opnsense-prep-image.sh's GROW (+8G). Proof it was dead: tofu plan is byte-identical before/after removal.
  • Opteron_G3 is a RED HERRING: the host is really an AMD EPYC 9965 (Zen 5, fam 26); libvirt's CPU-model DB doesn't know fam 26 and falls back to the oldest name. The prior session partly built its nested-virt theory on this artifact. Read /proc/cpuinfo.
  • "OPNsense 3 GB min" is UNVERIFIED (no source) -- annotated, do not propagate.
  • DOCFIX-186's backing_store parenthetical is STALE (disk is a copy now) -- same conclusion, different mechanism; annotated. DOCFIX-187 changelog gains a SUPERSEDED-IN-PART header. DOCFIX-186's apparmor + config-iso-staging findings are REAL and STAND. DOCFIX-185 (operator transport ruling) untouched.
  • STANDING LESSON: when a change does NOT fix the symptom, do not write it up as though it did. Four successive non-fixes were each recorded as a cause, and that accumulated fiction is what a fresh session would have inherited. A fault that is deterministic and immune to every knob means you have not yet touched the cause.

    Jumphost stream -- session 2026-07-12 (cont.): edge BOOTS. Both boot bugs closed (DOCFIX-190)

• LIVE APPLY EXECUTED (operator-gated, 3 recreates). DOCFIX-188's memory fix went live: Max memory: 2048 KiB -> 2097152 KiB, QEMU -m size=2097152k, domain running (booted) not paused, serial 262 -> 3882 bytes. Triple-fault CONFIRMED DEAD.

• SECOND BUG, found underneath it (DOCFIX-190): ACPI was OFF. With RAM finally correct the kernel reached interrupt init and PANICKED: panic: running without device atpic requires a local APIC -> parked at ddb db> = the 100%-of-one-core spin, while virsh still cheerfully reported running. Cause: NO features block in ANY of the 3 VM modules -> libvirt renders -machine ...,acpi=off. FreeBSD gets its local APIC from ACPI's MADT and OPNsense has no atpic fallback. FIXED: features = { acpi = true, apic = {} } in opnsense-edge + cloudinit-vm + node-vm. node-vm matters most long-term: MAAS drives power via ACPI, so without it a graceful shutdown/MAAS power-off would NEVER work.
• GUARD S2 added to opentofu-validate (same reason as S1: features is optional, so tofu validate structurally cannot see its absence). Flag renamed --static-only (--check-memory-unit kept as alias). Harness 4 -> 6 PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch independent classes.
• RESULT: OPNSENSE BOOTS AND RUNS. acpi=on; kernel to userland; CPU idle (~19s, no spin); DHCP lease 172.30.1.126, hostname OPNsense on office1-wan. BIOS 639kB/2096108kB available memory re-confirms the 2 GiB.
• STILL OPEN -- Configuration Importer did NOT apply. OPNsense came up on FACTORY DEFAULTS (WAN=DHCP) instead of static 172.30.1.2 / LAN 10.10.0.1. The ISO is VERIFIED well-formed (ISO9660 OPNSENSE_CFG, contains CONF/CONFIG.XML with <opnsense> + 10.10.0.1) and attached as SATA cdrom -- so the payload is right, the Importer just did not consume it. This is exactly the mechanism the module header has ALWAYS flagged UNVERIFIED. Do NOT guess the fix -- read the Importer's own early-boot console output. Untested hypothesis only: ISO9660 8.3 UPPERCASE vs the lowercase /conf/config.xml it looks for. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN by default; do not chase it.
• OPS GAP (contributed to the original misdiagnosis): libvirt creates the serial log root:0600 and RECREATES it on every domain create, so the most valuable boot-incident artifact needs an interactive sudo every retry. A default ACL does NOT work: the 0600 creation mode sets the ACL mask to ---, nullifying the named-user entry (measured: user:jessea123:r-- #effective:---). Proposed (NOT yet verified): pre-create the log 0644 before tofu apply so libvirt appends to an already-readable file.

• BUG 3 ROOT-CAUSED -> D-112 PROPOSED (operator ruling required; NOT implemented). The Configuration Importer can never fire on a pre-installed nano image, BY DESIGN. Upstream opnsense-importer -b does touch /.probe.for.readonly -- a PROBE FOR A READ-ONLY ROOT -- and if [ -f ${INSTALL} -a -f /conf/config.xml ]; then bootstrap_and_exit 0; fi. On INSTALLER media root is read-only so the touch fails and it scans media (cd9660 supported). On our nano, root is WRITABLE and a factory /conf/config.xml ALREADY EXISTS -> both true -> it exits immediately, never enumerating a device. Boot log confirms: >>> Invoking import script 'importer' then NOTHING. Our ISO was fine all along (verified: ISO9660 OPNSENSE_CFG, CONF/CONFIG.XML with 10.10.0.1; guest sees it as cd0, exact byte size) -- nothing was ever going to read it.

  • The README research (items 2-3) correctly established the Importer supports ISO9660; it NEVER established the Importer RUNS on a nano. The module's own UNVERIFIED flag was right, and is now discharged NEGATIVE.
  • D-112 RULED: option (c) -- post-boot network provisioning (operator: "C since it is the way people automate opnsense"). The rendered config (opnsense-config.xml.tmpl) is REUSED unchanged; only the DELIVERY changes. IMPLEMENTATION NOT STARTED.
  • SELF-CORRECTION (flagged to operator before the ruling): I had described option (b) as making the ISO mechanism "work as designed, unchanged". That was OVER-CLAIMED. Even with a read-only root the importer prompts "Press any key..." with a 7s timeout and then asks for a device name -- i.e. it appears INTERACTIVE BY DESIGN, so (b) may have been (d) in disguise. Not verified either way; recorded so nobody revives (b) on my bad summary.
  • BOOTSTRAP PROBLEM -- UNSOLVED, gates (c), sub-choices B1/B2 UNRULED (see D-112). "Push the config over the API" is NOT reachable from a factory-default box: (1) NO ROUTE -- default LAN is 192.168.1.1/24 on office1-local, which is an ISOLATED libvirt net with NO host leg on virbr2 (measured); WAN is reachable at 172.30.1.126 but OPNsense BLOCKS ALL INBOUND ON WAN by default, so that is not a way in. (2) NO CREDENTIALED API -- the REST API needs an API key+secret (not the root password), minted in the GUI; and SSH is DISABLED by default. (3) OUR SERIAL IS WRITE-ONLY -- the module attaches <serial type='file'> (one-way capture); it cannot be typed into. Making it interactive = type='pty' + virsh console (module change + recreate). NOTE: a console login shell is NOT the rejected option (d) -- (d) was rejected for racing the importer's 7s timeout; a login shell has no timing race. Do not conflate them.
  • Impacts if not (b): opnsense-build-config-iso.sh + harness, the module's config_seed/cdrom, the README research section. ALSO GATES STAGE 3 (per-DC edges use this same module).

    Jumphost stream -- 2026-07-12 (cont.): D-112(c) EXECUTED. THE OFFICE1 EDGE IS UP AND ROUTING.

• END STATE (measured over SSH): office1-opnsense.office1.vr1.cloud.neumatrix.local, OPNsense 26.1. WAN 172.30.1.2/24 static, LAN 10.10.0.1/24, default route via 172.30.1.1, NAT active, egress to the internet WORKS (ping 1.1.1.1 0% loss through the simulated ISP). Serial console restored (kern.console=ttyu0, getty running). sshd up with the service key (we manage it over SSH now). (NOTE: the sshd: DOWN line in the verify sweep was MY CHECK BEING WRONG -- it ran over an SSH session, so sshd was obviously up; the pgrep pattern just did not match. Not a fault.)

• Bootstrap executed: console (pexpect + virsh console) -> factory login -> config streamed as base64, SHA-256 verified byte-exact, factory config backed up -> reboot. Then the DOCFIX-192 fix was pushed over SSH -- i.e. the D-112(c) steady-state path proving itself. B1 host leg 10.10.0.10/24 on virbr2 is IN PLACE (non-persistent; re-add after a host reboot: sudo ip addr add 10.10.0.10/24 dev virbr2).
• DOCFIX-191: the config would have LOCKED MANAGEMENT OUT. Template had <ssh><group>admins</group></ssh> = OPNsense's DISABLED-sshd shape, and NO authorized key. Applying it would have left the edge with sshd off and no key. Fixed (<enabled>enabled</enabled>
  • <permitrootlogin> + <authorizedkeys> base64). Names VERIFIED upstream, not guessed.
• DOCFIX-192: the config SILENCED ITS OWN ONLY CONSOLE. No <primaryconsole>/<serialspeed>, and the edge VM has NO VIDEO DEVICE, so serial is the only console. After the first reboot the console went permanently silent. The box survived ONLY because DOCFIX-191 had just turned SSH on -- one lockout bug saved us from the other. Fixed; console restored.
  • KEY GOTCHA (cost a confused cycle): console settings take TWO reboots. OPNsense regenerates /boot.config + /boot/loader.conf FROM config.xml during boot, so they only take effect on the NEXT boot. After reboot #1 the loader files were already correct while the running kernel was still on video. Do not conclude "the fix failed" after one reboot.
  • /etc/ttys uses onifconsole, so getty starts on ttyu0 ONLY if ttyu0 is the kernel console. sysctl kern.console is the ground truth.
• GUEST SHELL IS CSH. Both virsh console and ssh root@ land in csh, which rejects $(...) ("Illegal variable name."). Pipe POSIX scripts to sh -s. This silently broke two scripts before it was spotted.
• PROVIDER GOTCHA: virsh console REFUSES a unix-socket chardev ("character device serial0 is not using a PTY"). The module's serial must be a pty + a log file. A unix socket is also unreachable in practice (libvirt-qemu:kvm, and the staging default ACL stamps a named-user entry that outranks group perms -> r-- only). virsh console via libvirtd needs no extra privs. The provider's source.pty requires a path, which libvirt allocates at start -- so OMIT source entirely to get libvirt's default <serial type='pty'>.
• OPEN GAP -- DHCP WAS NEVER IMPLEMENTED (the ledger claim below was FALSE). The 2026-07-12 entry says "LAN 10.10.0.1/24 + DHCP .100-.199". There is NO DHCP configuration in the template at all -- no <dhcpd>, no <Kea> (measured: template top-level sections are filter, gateways, interfaces, nat, ntpd, OPNsense, rrd, staticroutes, system, theme, unbound). Nothing serves DHCP on office1-local (measured: no kea/dhcpd/dnsmasq process, nothing on udp/67). Another optimistic claim from the max-context session. The operator's original goal was "router + DHCP" -- the ROUTER half is done, the DHCP half is NOT. OPNsense 26.1 uses Kea (ISC dhcpd is gone), so this needs the Kea schema verified upstream (do NOT invent it) + a harness assertion.
• ~/vr1-office1.env created (0600, jumphost-only, NEVER COMMIT). Contains NO secrets: it READS ~/vr1-office1-creds/* at source time, so there is exactly one copy of key material on disk. Carries the as-built values + $OPNSENSE_SSH_KEY / $OPNSENSE_CONFIG_OUT handles.
• Numbers: D next-free 113, DOCFIX next-free 193.

Jumphost stream -- 2026-07-13: DHCP APPLIED. The Office1 edge is a ROUTER + DHCP SERVER.

• DOCFIX-193 APPLIED to the live edge (was built-but-not-applied at last session's close). kea-dhcp4 is running and bound udp4 10.10.0.1:67; subnet 10.10.0.0/24, pool .100-.199, routers/DNS 10.10.0.1. Router unaffected: WAN 172.30.1.2, LAN 10.10.0.1, default route 172.30.1.1, NAT automatic, egress 0.0% loss, serial console + getty alive. Full record: docs/changelog-20260713-office1-dhcp-apply.md.
• KEY AS-BUILT FACT (do not re-derive): a full-config push to this edge drops ~667 migration-populated elements, INCLUDING the only 2 firewall pass rules on the box ("Default allow LAN to any" + IPv6; legacy /filter/rule count is 0 in 26.1). OPNsense regenerates them on boot -- proven from /conf/backup/ across 4 push+boot cycles, and re-confirmed post-apply (791 elements, both rules back, TLS cert back, our Kea subnet4 survived). The overwrite is self-healing. Do NOT hand-merge to "preserve" them.
• OPEN (logged, not executed): the apply path is still ad-hoc shell, not a harnessed script. Wanted: scripts/opnsense-apply-config.sh (snapshot -> scp -> SHA verify -> install -> reboot -> assert kea/udp67 + 2 LAN rules + egress + console). The ad-hoc path already shipped one defect: root's shell on OPNsense is tcsh, so a $(...) in a quoted remote command dies with "Illegal variable name" -- the pre-install snapshot silently no-op'd and the install ran without it. Always feed remote commands to sh -s.
• NOT verified end-to-end: no client has taken a lease -- office1-local has no host yet. First Stage-2 host on that LAN is the real proof.

• Platform question (operator asked): recommendation is stay on OPNsense, not VyOS -- reasoning in the prior session's close-out; would be D-113 if it needs a formal ruling. NOT raised as a decision; no D-number assigned.

• Unattended git push now works (SEC-005). credential.helper store + a dedicated GitBucket PAT at ~/.git-credentials (mode 600). The branch had been stuck 12 commits ahead of origin for two days; now pushed (425d434, ls-remote confirmed). Two gotchas, both measured, in docs/changelog-20260713-git-credential-store.md: (a) the git username is jesse.austin -- NOT the web-UI login email, NOT JANeumatrix; (b) GitBucket 401s PAT basic-auth on /api/v3/ but ACCEPTS it for git transport, so validate creds against info/refs?service=git-receive-pack, never the API; (c) store DELETES the credential on a 401, so a rejected token silently empties the file. SEC-005 is OPEN: plaintext account-wide token on the jumphost -- revoke/rotate at v1 close.

• D-113 PROPOSED (2026-07-13, UNRULED): is OPNsense the right edge platform? Written up with alternatives (a1 OPNsense+config.xml = status quo, a2 OPNsense+REST API, b VyOS, c plain-Linux router, d pfSense=reject). Key finding while drafting: D-112 adopted (c) on the rationale "the way people automate opnsense" = the REST API, but what we BUILT is full-config.xml templating over scp -- which satisfies (c)'s letter, not its rationale. All three OPNsense-specific bugs (DOCFIX-191/192/193) plus the 667-element self-heal dependency are artifacts of hand-authoring XML -- exactly what the API exists to avoid. So the real question is a1 vs a2, not OPNsense vs VyOS. My lean: stay on OPNsense, move to (a2). NOT RULED -- operator decides. Rule BEFORE Stage 3 (it builds 2 more edges from this same module; ruling after means migrating 3 edges instead of 1).

• D-113 RULED 2026-07-13: option (a2) -- stay on OPNsense, MOVE CONFIG TO THE REST API. Two operator constraints narrowed the field decisively: (1) GUI access is a REQUIREMENT -- this eliminates VyOS and plain-Linux, which had been the leading candidates; (2) VyOS LTS binaries are subscription-gated (verified), so free VyOS = rolling, fighting appendix-B version pinning. OpenWrt was the credible alternative (UCI is text-first by design) but costs a new pipeline + bootstrap + dnsmasq to escape a problem fixable in place. (a1) config.xml templating is now explicitly REJECTED. Next: retire hand-authored config.xml. Bootstrap config keeps ONLY what the API cannot set on a factory box (sshd + root key + console + the API key); everything else -- DHCP, firewall, interfaces -- moves to the API/Ansible collection. Live edge recon (2026-07-13): lighttpd IS listening on 443/80, the API answers 401 (alive), root has 0 API keys -> minting one via the GUI is the bootstrap step (the supported path; hand-editing apikeys into config.xml would be the very anti-pattern being retired).

• D-113(a2) step 1 SHIPPED: scripts/opnsense-api.sh + tests/opnsense-api/run-tests.sh (21 PASS; gauntlet 53 ALL GREEN). Thin REST client; secret passed to curl via --config on STDIN so it never lands in argv/ps (harness T9 enforces this -- a future curl -u "simplification" turns it red). Host never inferred (T3). --insecure is deliberate: the edge's self-signed cert is REGENERATED EVERY BOOT, so pinning is pointless; scoped to the private lab LAN leg only. BLOCKED ON: an API key. Measured: API is alive (401 from lighttpd on 443) but root has 0 apikeys. Minting is a GUI action (System > Access > Users > root > API keys > "+") -- deliberately NOT automated, since hand-writing apikeys into config.xml is the exact anti-pattern D-113 retired. Key file -> ~/vr1-office1-creds/opnsense-api.txt (operator-only, never into agent context; SEC-007 covers that dir). Next after the key: drive yesterday's DHCP subnet through the API as the proof-of-path. If it round-trips, the template retires to a MINIMAL bootstrap (sshd + key + console + apikey) and DHCP/firewall/interfaces move to the API. If it cannot express it, D-113 gets revisited.

• D-113(a2) PROVEN 2026-07-13 against the LIVE edge. API key minted; GET core/firmware/status -> 200; POST kea/dhcpv4/search_subnet reads back EXACTLY yesterday's hand-pushed DHCP (10.10.0.0/24, pool .100-.199, routers/DNS 10.10.0.1) as TYPED fields. GUI gotcha (26.1): API keys are NOT created in the user-edit dialog nor the ApiKeys tab (that tab is search+delete ONLY). They are a ROW ACTION on the Users list -- the fa-ticket icon. The secret is shown ONCE (stored crypt($6$)). API gotchas: response root key is dhcpv4 (not dhcp4); camelCase actions map to snake_case URLs (searchSubnetAction -> search_subnet). NOT done: no WRITE through the API yet (gated live mutation). The config.xml template is still live -- DO NOT push a full rendered config.xml now; it would clobber API-managed state. Next: reduce the template to a MINIMAL bootstrap (sshd + key + console) and move DHCP/firewall/interfaces to the API.

State facts to remember

• beta cluster left at node_count=2 (deliberate; bonus resize acceptance coverage).
• repo is temporarily PUBLIC for Claude web_fetch (SEC-004) -- flip private at v1 close.
• Jumphost stream: see "Active window" section above (ops-update-20260705 in flight). Vault stays 1.8/stable (D-068 unruled).
• CORRECTION (2026-07-05, jumphost stream, per handoff reconcile item): the two bullets above are stale -- ops-update-20260705 is CLOSED (fleet fully current, juju 3.6.25); D-068 is RULED on the 1.16 question (1.16 ruled out; vault stays 1.8/stable rev 714; the off-EOL-1.8.8 path remains OPEN under D-068).

Project-completion (execute after D-011 passes)

• Consolidate 10 per-phase do-documents into docs/v1-deploy-runbook.md.
• Set repo visibility PRIVATE (SEC-004).
• v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle.

Cross-session pins (append-only)

• D-071 completion sweep (2026-07-05, DONE): the ops-update window closed; D-071 was AMENDED (create-backup premise corrected -- juju create-backup/download-backup DO exist on juju 3.6; a 902MB controller backup was taken). D-071 REMAINS PROPOSED -- not ratified to ADOPTED. Operator ratification is pending; the single-controller (no-HA) half of the risk is unresolved by backups alone. See D-071 + its 2026-07-05 amendment.
• Pre-DC-DC controller HA/backup planning (PINNED): before the Roosevelt (DC-DC) phase, hold a dedicated planning session on Juju controller High-Availability / backup architecture. Backups now exist (per D-071 amendment); controller HA is the open design item. This resolves the single-controller risk recorded under D-071 for production.