diff --git a/docs/archive/changelogs/changelog-20260703-process-hardening.md b/docs/archive/changelogs/changelog-20260703-process-hardening.md new file mode 100644 index 0000000..b2af03d --- /dev/null +++ b/docs/archive/changelogs/changelog-20260703-process-hardening.md @@ -0,0 +1,371 @@ +# Change log -- 2026-07-02/03 redeploy-readiness + process-hardening patchset + +CUMULATIVE: this covers BOTH work blocks (the DOCFIX-066..072 sweep patches and +the process-improvement build). One ZIP, extract over repo root. Every item +lists its revert. Executed under blanket approval; nothing here touched live +infrastructure -- all changes are repo content. Validation state at packaging: +merged-tree repo lint PASS (1 documented WARN), bundle gate PASS, four test +harnesses 33/33 cases, all python compiles. Identifier numbering verified +next-free at HEAD 690779a; RE-GREP at commit (per discipline). + +================================================================================ +## Block 1 -- redeploy-readiness fixes (DOCFIX-066..072) +================================================================================ + +### 1. DOCFIX-066 -- teardown runbook rebuilt around the D-061 fork +FILE: runbooks/phase-00-teardown-maas-reset.md (REWRITE) +WHY: old spine drove the DEPRECATED phase-00-teardown.sh ("releases to Ready" + -- the premise that decomposed machines 3x). New spine: destroy path as the + validated redeploy route (with the previously MISSING reenroll step), release + path documented with the honest re-acquisition caveat, all invocations + bash-prefixed. +REVERT: git checkout HEAD~ -- runbooks/phase-00-teardown-maas-reset.md + (reverting re-arms the decompose trap; recommend not). + +### 2. DOCFIX-067 -- octavia PKI SAN derived, not baked +FILE: runbooks/phase-01-bundle-deploy.md (CNF block + one prose line) +WHY: IP.1 was the pre-R14 literal 10.12.4.233; current VIP is .57. Now derived + from the bundle at generation time, echoed, regex-gated. VERIFY-LIVE still + queued: read the DEPLOYED cert SAN before deciding whether any live action + is ever needed (none recommended -- regenerates next redeploy). +REVERT: restore the literal block from git history. + +### 3. DOCFIX-068 -- phase-01 constants de-staled +FILE: runbooks/phase-01-bundle-deploy.md ("Constants and env-literals") +WHY: carried the pre-D-052 plane map (incl. retired lbaas), hardcoded MAAS + subnet ids (violates PATTERN-1) and system_ids (violates DOCFIX-040). Now + points at lib-net.sh / lib-hosts.sh; only verified-current literals remain. +REVERT: git history (not recommended; values were wrong). + +### 4. DOCFIX-069 -- exec-bit reality encoded +FILES: phase-00 runbook (folded into #1); apply-notes optional + `git update-index --chmod=+x scripts/*.sh`. +WHY: all 37 scripts are mode 100644 (Windows commit path); bare invocation + fails on a fresh clone. bash-prefix is the durable invariant; repo-lint L6 + now guards regressions. +REVERT: n/a (wording only) -- to reject the rule, remove lint check L6. + +### 5. DOCFIX-070 -- one bundle gate instead of two disagreeing ones +FILES: scripts/provider-bundle-check.py (checks 4-7 added); + scripts/review-bundle.py (git rm -- manual action, see apply notes); + tests/provider-bundle-check/ (NEW: 8-case harness). +WHY: review-bundle.py was pre-D-052; FAIL=71 pure noise against HEAD (alarm + fatigue). Its still-valid checks (relation syntax/existence) are absorbed; + added: D-062 unit count, VIP-octet uniqueness, DOCFIX-071 policy wiring + + zip/source content drift guard. +REVERT: git checkout the old provider-bundle-check.py; git restore + review-bundle.py. (Harness T-cases will then fail -- delete tests/ dir too.) + +### 6. DOCFIX-071 -- keystone policy ships IN the bundle +FILES: bundle.yaml (keystone resources: stanza, +6 lines); + policies/overrides.zip (NEW, binary; built from committed source); + .gitattributes (+ *.zip binary); runbooks/phase-03-core-verify.md + (NEW Step 3.4: PO: gate + behavioral G3 gate); + runbooks/appendix-C-identity-rbac.md (attach block: subshell-wrapped, + /tmp -> repo path [snap trap], reframed as live-update path). +WHY: the D-064 attach was unreachable from the schedule -- a clean redeploy + shipped WITHOUT the SCS Domain Manager RBAC; the only written procedure used + the snap-blocked /tmp form. +REVERT: remove the 6 bundle lines + the zip; restore appendix-C block from + history. (Reverting re-opens the blocker; the phase-03 gate would then FAIL + by design -- that is the guard working.) + +### 7. DOCFIX-072 / D-043 -- decision brought level with the bundle +FILE: docs/design-decisions.md (D-043 RESOLVED -> ADOPTED option (a)). +WHY: bundle set resume-guests-state-on-host-boot=True while D-043 was still + PROPOSED. Adopted: auto-resume is the tenant-VM norm; D-041 unchanged for + control-plane. JUDGMENT CALL FLAGGED EARLIER: capi-mgmt-v2 WILL auto-resume; + manual-start policy now governs deliberate stops only. If you want a real + exclusion, that needs a mechanism -- say so and I will draft options. +REVERT: append a reversal entry (append-only doc) + remove the bundle option. + +### 8. D-doc alignment amendments (appended, append-only discipline) +FILE: docs/design-decisions.md -- D-002 (channel matrix reconciled: etcd/easyrsa + dead, memcached latest-only, vault 1.16 per D-068), D-009 (rabbitmq + min-cluster-size Roosevelt delta), D-011 (item 6 = manual unseal, matching + phase-08), D-051 (bundle-native delivery), D-061 (validation scope: + survival vs re-acquisition; destroy = validated spine). +REVERT: each is a discrete appended section -- delete the section. + +================================================================================ +## Block 2 -- process improvements (DOCFIX-073..078, D-069, D-070) +================================================================================ + +### 9. DOCFIX-073 -- preflight.sh: THE single pre-deploy gate +FILES: scripts/preflight.sh (NEW), scripts/channel_assert.py (NEW), + tests/preflight/ (NEW, 7 cases), runbooks/phase-01-bundle-deploy.md + (prerequisites GATE added -- pre-flight was previously invoked by NO runbook). +WHY: gate surface was six artifacts remembered independently. Orchestrates + repo-lint -> bundle gate -> channel assert -> live pre-flight, worst-exit + aggregation, stage-2 reminders. channel_assert fulfils D-002's "verify + against Charmhub each deploy" claim (previously unimplemented): every pinned + channel must exist on Charmhub; typo/retired-track = FAIL; offline = WARN. +REVERT: delete the three new paths + the phase-01 GATE paragraph. + +### 10. DOCFIX-074 -- repo-lint: the drift sweep, productized +FILES: scripts/repo_lint.py + scripts/repo-lint.sh (NEW), + tests/repo-lint/ (NEW, 9 cases). Plus SANITIZE remediation of every ASCII-rule + violation it found: docs/v1-pre-deploy-fixes.md, docs/netbox-vip-queue.md, + .gitignore, netbox/README.md, netbox/ipv4-prefixes-import.py, + netbox/ipv6-mark-reserved.py (punctuation transliteration only; python files + re-compiled OK). Plus two catches it made against MY OWN morning patch and + bundle HEAD, both fixed: a missed .233 prose line in phase-01; a stale + storage/replication CIDR comment in bundle.yaml. +CHECKS: L1 encoding, L2 stale tokens (with explicit per-file opt-out marker for + guard scripts), L3 ghost script refs, L4 deprecated refs, L5 numbering + collisions + next-free report, L6 bare invocations. +REVERT: delete scripts + tests; restore pre-sanitize files from history. + +### 11. DOCFIX-075 -- cloud-assert.sh: the behavioral verifier + BOM capture +FILES: scripts/cloud-assert.sh (NEW), tests/cloud-assert/ (NEW, 9 fakebin + cases), runbooks/ops-restart-procedure.md (NEW -- the restart procedure, + previously operator-local, adapted to current references with [REVALIDATE] + markers and committed). +WHY: the D-045/D-046/D-051/D-042 family = "juju green, service broken". One + idempotent read-only sweep of every service-own-verdict gate (vault seal, + mysql 1xR/W, OVN unity, chassis, hypervisors, LBs, PO:, trustee domain, coe + 403, conductor LIVE args). Missing admin scope = HELD exit 2, never a silent + pass. --capture writes a committed asbuilt// BOM (Roosevelt drift + baseline). Replaces the never-committed post-maintenance-health-check.sh. +REVERT: delete the three paths. + +### 12. DOCFIX-076 -- as-executed log convention +FILES: scripts/run-logged.sh (NEW), docs/as-executed-log-convention.md (NEW), + logs/as-executed-index.md (NEW, committed index; content stays jumphost-only). +WHY: the verbatim-retrieval rule for one-shot steps depended on an artifact + with no defined location/format (DOCFIX-006 is the cautionary tale). +REVERT: delete the three paths. + +### 13. DOCFIX-077 -- appendix-A completions +FILE: runbooks/appendix-A-troubleshooting.md (two appended sections): + mysql-innodb-cluster recovery (D-062 signatures + the destructive-action + guard) and the point-of-use identifier index (DOCFIX-027/028/029/034/037, + BUNDLEFIX-001..006) closing the dangling-reference gap. +REVERT: delete the two appended sections. + +### 14. DOCFIX-078 -- security exposure ledger +FILE: docs/security-ledger.md (NEW). Seeded: SEC-001 libvirt credential + exposure (was living only in a script header), SEC-002 juju action-log + token rule, SEC-003 vault custody, SEC-004 repo-public flag. +REVERT: delete the file. + +### 15. D-069 -- vault unseal-key custody (ADOPTED, policy) +FILE: docs/design-decisions.md (appended). +WHAT: split custody (no individual holds threshold), second-person unseal + rehearsal as an acceptance item, custody re-cut review at every re-init. + Custodian ASSIGNMENT deliberately left as operator input (SEC-003). +REVERT: append a reversal entry. + +### 16. D-070 -- supersedes D-012 (no KVM snapshot restore path) +FILE: docs/design-decisions.md (appended). +WHAT: D-012 was never exercised (no virsh snapshot step exists anywhere); + rebuild-from-runbooks (D-017/D-018) is THE restore path; baseline-capture + role moves to cloud-assert --capture. Counter-argument recorded in the entry + (mid-rehearsal rollback convenience) with the revisit condition. +REVERT: append a reversal entry (this one is the most opinion-weighted item + in the set -- if you disagree anywhere, it is probably here). + +================================================================================ +## Manual git actions (cannot ship in a zip) +================================================================================ + git rm scripts/review-bundle.py # DOCFIX-070 + git rm scripts/phase-00-teardown.sh # deprecated; runbook no longer + # references it by path (repo-lint + # L3/L4 assume it gone) + # optional: git update-index --chmod=+x scripts/*.sh + +## Pre-commit verify (jumphost or Windows+python3) + python3 scripts/repo_lint.py . # expect: 0 fail, 1 legacy WARN + python3 scripts/provider-bundle-check.py bundle.yaml # expect: PASS, 6 [ok] + for t in tests/*/run-tests.sh; do bash "$t"; done # expect: 4x ALL PASS (33 cases) + +## Still queued (unchanged by this patchset) + Verify-live: octavia deployed-cert SAN read; `juju info` channel probe on the + jumphost (channel_assert now automates it at preflight); second-person unseal + rehearsal (D-069). Operator inputs: vault custodian assignment (SEC-003); + ruling if a real capi-mgmt auto-resume exclusion is wanted (item 7). + +================================================================================ +## Block 3 -- code hardening sweep (all repo scripts + harness estate) +================================================================================ +Method: automated audit of every script and runbook paste block against the +house hardening rules (SIGPIPE, capture-die, inner-ssh stdin, /tmp-snap, gawk, +sed -i, column-order, juju-run capture, bare-exit-in-paste), then MANUAL +verification of every hit before any change. Audit honesty notes: H3 (inner +ssh) and H7 (paste-block exit) closed at ZERO true positives -- the rc/rcap/J +wrapper convention and the subshell/remote-quote discipline held everywhere; +H2/H4/H6/H8 clean. + +### 17. DOCFIX-079 -- seven SIGPIPE (H1) fixes: capture-then-test conversions +FILES: scripts/phase-02-vault-preflight.sh, phase-03-core-verify.sh (model- + presence gate: race read a PRESENT model as absent -> false HOLD); + phase-06-bootstrap.sh (role verify-first: race re-ran grants); + phase-06-net-setup.sh (SG-rule verify-first: race -> duplicate create -> 409 + aborts the run under set -e); phase-07-conductor-graft.sh + its runbook twin + (helm version check: converted to substring param-expansion, no pipe at all); + tenant-onboard.sh x2 -- the serious pair: the manager-grant verify could + FALSE-DIE on a successful grant, and the duplicate-CIDR guard FAILED OPEN + (match -> SIGPIPE 141 -> '&& die' skipped -> colliding subnet proceeds). +NEW TEST: tests/tenant-onboard/run-tests.sh proves the CIDR guard fires on a + collision and stays quiet on a clear CIDR. +REVERT: each is a commented, anchored block -- git history per file. + +### 18. DOCFIX-080 -- tenant-onboard image-ID resolution off column-order luck +FILE: scripts/tenant-onboard.sh (2 sites) + a jq presence gate added. +WHY: `-f value -c ID -c Name | awk '{print $1}'` rode alphabetical column + ordering (ID br-ex OVS + static; br-metal/.103/ +br-internal chain; raw data/storage/replication statics; enp11s0 idle), +missing-subnet and wrong-VID hard gates, all-SKIP idempotent re-run; forbids +br-prov-api/enp1s0.104/provider-vip forever. REVERT: git history (red harness). + +### 26. scripts/validate.sh -- placeholder TODO aligned to amended decisions +The D-011 runner's drafting list still targeted DROPPED features (Designate +resolution -- D-019; snapshot existence -- D-012, superseded by D-070) and +predated the manual-unseal ruling. List now points at the existing building +blocks (cloud-assert / tenant-acceptance / run-tests-all) and the genuinely +missing items, so the eventual implementation is built to the real spec. + +### 27. skills/openstack-cloud-ops/ -- unpacked skill SOURCE committed +The repo carried only the packaged .skill zip (opaque to grep/diff/repo-lint). +The unpacked SKILL.md + references/ (v1.2) now live beside it as the reviewable +source of truth; regenerate the .skill from this folder on any change (or drop +the zip -- operator's call). Divergence rule applies: repo source wins over any +installed copy. + +### Review attestations (Block 4 sweep) +Runbook->script contract cross-check: CLEAN (both automated hits verified +false -- reenroll --check and carve --apply exist under parser styles the +checker missed). Loop-efficiency scan: clean (flagged sites are watchers/polls +by design). Exit-code audit: clean (tenant-acceptance's 11-14 are a declared +per-phase contract). deploy-watch, maas-fabric-prune, juju-spaces-check, +osd-blank-check: read; disciplined; no findings. + +================================================================================ +## Block 5 -- Claude Code on the jumphost (guardrails + session contract) +================================================================================ +Context: Claude Code now runs on the jumphost with this repo as the project +dir -- a live shell with real cloud credentials. Mechanics verified against +the Claude Code permissions docs: rules evaluate deny -> ask -> allow; deny is +absolute in every mode; a PreToolUse hook exit-2 blocks BEFORE permission +evaluation; Read() deny rules do not stop arbitrary subprocess file reads. + +### 28. NEW -- CLAUDE.md (always-loaded session contract) +The three hard rules, session bootstrap, secrets norms, and routing pointers. +Deliberately short: always-in-context essentials only; depth lives in the skill. + +### 29. NEW -- .claude/settings.json (committed, team-shared permission rules) +allow = the CHECK surface (read-only juju/maas/openstack, the gate scripts, +harnesses). ask = the ENTIRE mutation surface (juju/maas/openstack mutations, +juju ssh/run/exec, anything --apply, sudo, virsh, git commit/push) -- this IS +the gated-mutation model, enforced by the tool rather than by convention. +deny = the never class: vault one-shots, destroy-controller, maas list +(DOCFIX-016), force-push, and Read/Edit on ~/vault-init, ~/as-executed, +~/tenant-*, *-cred.txt, *appcred*. Design choice (debate): destructive ops are +ASK not DENY -- they legitimately run under runbooks with human approval; deny +is reserved for operations NO session should ever perform. + +### 30. NEW -- .claude/hooks/guard-destructive.py + tests/claude-guard/ (13 cases) +PreToolUse belt-and-suspenders: hard-blocks (exit 2, all modes, even bypass) +the never class plus secret-file SHELL reads that Read() rules cannot see. +Exists because hook blocks precede permission evaluation and upstream Bash +rule enforcement has a documented reliability history. Fail-open contract on +malformed input is deliberate (permission rules still apply) and tested. + +### 31. Skill relocated to .claude/skills/openstack-cloud-ops/ +Claude Code auto-discovers project skills there; skills/ top-level now carries +only the packaged .skill artifact (for claude.ai upload). Single source: edit +under .claude/skills/, repackage the .skill from it, reinstall in claude.ai. +MANUAL GIT ACTION if the move shows as delete+add: none needed (git mv staged). + +### Not done, deliberately (revisit triggers noted) +- OS sandboxing (bubblewrap): additive value once Claude Code usage matures; + the permission+hook layer is the v1 boundary. Revisit at Roosevelt. +- disableBypassPermissionsMode: managed-settings scope (system paths, admin) -- + worth doing when a second operator joins the jumphost. + +### 32. Block 5.1 -- hermetic-harness fix (caught by the FIRST jumphost gauntlet run) +tests/preflight T7 simulated "charmhub unreachable" by DELETING the fake juju +-- which un-shadowed the REAL juju on the jumphost, fell through to LIVE +Charmhub (a hermeticity violation: a test made a network call), found the pin +valid, and correctly returned 0 against the test's expectation of 2. FIX: T7 +now installs a FAILING fake (realistic connection error, exit 1) -- identical +behavior on any host. Same-class sweep: the carve and standup fakebin maas +unmatched arms returned '{}' exit 0 (silent-tolerate); flipped to fail-loud +(harnesses stay green, proving no uncovered calls exist today and none can +sneak in silently tomorrow). Rule encoded in the skill (script-authoring): +fakes SHADOW and fail loud; never simulate absence by deletion. The .skill +artifact regenerated from .claude/skills/ source. Credit: the gauntlet-on- +real-host practice worked exactly as designed on its first run. diff --git a/docs/archive/changelogs/changelog-20260709-adversarial-review-fixes.md b/docs/archive/changelogs/changelog-20260709-adversarial-review-fixes.md new file mode 100644 index 0000000..6d1d461 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-adversarial-review-fixes.md @@ -0,0 +1,110 @@ +# Changelog 2026-07-09 -- Adversarial review pass: 3 findings fixed (DOCFIX-160) + +No live infrastructure touched. Operator's standing instruction ("check +back over current work, review for errors/assumptions") applied to +everything built this evening: `scripts/lib-net.sh`/`lib-hosts.sh`'s +selector functions, `netbox/dc-dc-prefixes-import.py`, and all seven +Stage 1-7 runbooks. Ran a dedicated adversarial-review subagent (fresh +eyes, no authoring context) instructed to find real defects, not summarize +-- it read every file in full, cross-checked claims against +`design-decisions.md`/`bundle.yaml`/`opentofu/*.tf` directly, and ran both +new test harnesses live rather than trusting their reported pass counts. + +## Items + +### 1. DOCFIX-160a -- dead-code bug in `tests/dc-dc-prefixes-import/test_logic.py` +FILE: `tests/dc-dc-prefixes-import/test_logic.py`. + +WHAT: `expect_ok()`'s `try: return fn(...) / except SystemExit: ... / else: +ok(label)` structure has a real Python control-flow bug -- a `return` +statement inside a `try` block causes the function to exit immediately, +so the `else` clause (which only runs when the `try` block completes +WITHOUT a `return`/exception) is UNREACHABLE dead code whenever `fn()` +succeeds. Verified empirically (`python3 -c` reproduction): the `else` +branch's `print`/`ok()` call never executes on the success path. + +CONCRETE IMPACT: three happy-path assertions -- +`"ula48 valid /48 within fc00::/7"`, `"gua valid /40 within +2602:f3e2::/32"`, `"dc2 supernet valid /19 no overlap"` -- were silently +never counted as PASS *or* FAIL. The harness's "ALL PASS (37 checks)" +banner was true for the 37 checks that DID run, but overstated what was +actually verified: the safety property (rejecting bad input) was still +exercised correctly (that path lives in `except`, unaffected by the bug), +but "valid input is accepted, not spuriously rejected" was unverified and +uncounted. This "37/37" figure was quoted in `docs/dc-dc-deployment- +workflow.md`, `docs/changelog-20260709-netbox-dc-dc-pipeline.md`, and +`docs/session-ledger.md` -- all now stale by 3 checks (see item 3 below). + +FIX: restructured to assign the result inside `try`, call `ok(label)` +unconditionally after the `except` block (which itself still returns +early on failure), and return the result: +```python +def expect_ok(label, fn, *a, **kw): + try: + result = fn(*a, **kw) + except SystemExit as exc: + no(label, f"unexpectedly raised SystemExit({exc})") + return None + ok(label) + return result +``` + +VERIFIED: re-ran the harness -- the three previously-silent labels now +print `PASS`, and the total moved from 37 to **40 checks, all passing**. + +### 2. DOCFIX-160b -- self-contradictory comment in `netbox/dc-dc-prefixes-import.py` +FILE: `netbox/dc-dc-prefixes-import.py` (lines 165-168). + +WHAT: two adjacent comments about the same arithmetic fact contradicted +each other -- one claimed a /19 "exactly fits six /22s with zero slack," +the other correctly stated "a /19 = exactly 8x /22." 2^(22-19) = 8, so a +/19 holds eight /22s (six used, two spare), not six with no slack. No +functional bug -- the code only requires prefixlen <= 19 and slices the +first six subnets, which was always correct -- but a genuine, verifiable +self-contradiction a few lines apart, exactly the class of thing an +adversarial pass should catch. + +FIX: corrected the wrong comment to match the arithmetic the other +comment (and the code itself) already had right. + +### 3. DOCFIX-160c -- stale self-referential text in Stage 4 runbook +FILE: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`. + +WHAT: written mid-session when Stage 3's runbook didn't exist yet, two +spots said so explicitly ("Stage 3's own runbook does not exist yet as of +this writing... state NOT STARTED"). By the time the full session +finished, Stage 3 (and Stage 4 itself) were both written and reflected as +such in `docs/dc-dc-deployment-workflow.md` -- these two spots were never +reconciled against that final state. Not misleading in practice (the +runbook already hedged with "if it has been written and run by the time +you execute this..."), but stale phrasing left behind. + +FIX: reworded both spots to state Stage 3's runbook is now written but not +yet EXECUTED, and to check its actual execution state (not just its +existence) before proceeding. + +VERIFIED (all three items): `bash scripts/repo-lint.sh` 0 fail, 1 +documented legacy warn. `bash tests/dc-dc-prefixes-import/run-tests.sh`: +40/40 PASS (up from 37, item 1's fix). `bash tests/dc-selector/run- +tests.sh`: 21/21 PASS (unaffected, re-run to confirm no regression from +touching a neighboring file in the same review pass). + +**What the review found nothing wrong with, despite genuinely trying:** +the `$DC` selector asymmetry description (identical and correct across +every file that mentions it); the ULA/GUA carve arithmetic (hand-verified +deterministic, no cross-DC collision); Stage 6's `bundle.yaml` claims +(confirmed byte-for-byte); 3+ direct D-NNN quotes spot-checked verbatim; +DOCFIX numbering and D-100..D-110 ratification consistency across every +changelog/tracker/ledger entry; every `opentofu/main.tf`/`variables.tf` +claim in the Stage 1/3 runbooks. No inferred/invented literal value was +found anywhere in the reviewed material. + +REVERT: not applicable -- all three fixes are themselves corrections to +today's own deliverables, reverting them would reintroduce the bugs. + +## Next actionable step + +Adversarial review pass complete (this session's task list item). Final +consolidation (workflow doc cross-check, gap register, companion artifact, +changelog roll-up, session ledger) is the last remaining item before this +evening's work is fully wrapped for the operator's return in the morning. diff --git a/docs/archive/changelogs/changelog-20260709-ceph-replication-tooling.md b/docs/archive/changelogs/changelog-20260709-ceph-replication-tooling.md new file mode 100644 index 0000000..4725ac1 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-ceph-replication-tooling.md @@ -0,0 +1,144 @@ +# Changelog 2026-07-09 -- Ceph replication tooling: radosgw multisite + rbd-mirror scripts (DOCFIX-165) + +No live infrastructure touched -- new scripts + test harnesses only. Closes the +MECHANISM half of tooling gap register item #5 in `docs/dc-dc-deployment- +workflow.md` ("Ceph replication tooling (radosgw multisite + rbd-mirror). No +script in the repo touches either ... D-108's entire mechanism ... is +net-new. Blocks Stage 6."). The DATA half (no live Ceph cluster exists this +session to run any of this against) stays open -- this is a prep-only +session, same as every other tooling delivery tonight. + +Primary source: `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, +10.2, and 11.2-11.4 already carried the exact `radosgw-admin`/`rbd mirror` +command sequences (verbatim upstream Ceph administration, not invented) -- +this delivery extracts them into parameterized, callable scripts instead of +hand-typed runbook prose, per that runbook's own "Follow-up, NOT built now" +delivery-checklist item. Style/posture template: `scripts/opnsense-prep- +image.sh` / `scripts/opnsense-build-config-iso.sh` (real commands, no live +system to test against tonight, same posture). + +## Items + +### 1. DOCFIX-165 -- `scripts/dc-dc-radosgw-multisite.sh` +FILES: `scripts/dc-dc-radosgw-multisite.sh`, new +`tests/dc-dc-radosgw-multisite/run-tests.sh`. + +WHAT: subcommands `master-init` (Step 4 DC1 master realm/zonegroup/zone +create), `join-readonly` (Step 4 DC2 read-only zone join), `enable-two-way` +(Step 8's radosgw half -- zone modify --read-only=false). Every name (realm/ +zonegroup/zone), endpoint URL, juju model/unit, and the charm's restart- +action name is a REQUIRED argument -- no invented defaults, matching the +runbook's own `` placeholder discipline. `--restart-action` +is the sole optional flag (unknown action name is not guessed; omit it and +the script reminds the operator to confirm+restart manually instead). +Default mode is a dry-run PLAN print (secrets redacted); `--apply` executes +one command at a time via `juju ssh`, aborting on the first failure (per the +runbook's own "verify each command before proceeding -- do not chain +unattended" caution). + +### 2. DOCFIX-165 -- `scripts/dc-dc-rbd-mirror.sh` +FILES: `scripts/dc-dc-rbd-mirror.sh`, new `tests/dc-dc-rbd-mirror/run- +tests.sh`. + +WHAT: subcommands `bootstrap-primary` (Step 5 DC1: pool enable + peer +bootstrap create, token written to a file ON THE REMOTE UNIT, never +captured/printed into this session -- secret-adjacent, per the runbook's own +out-of-band-transfer instruction) and `bootstrap-secondary` (Step 5 DC2 / +Step 8 two-way: pool enable + peer bootstrap import, `--direction` REQUIRED +with no default -- `rx-only` for one-way staging, `rx-tx` for two-way, a +conscious choice every time, never assumed). `--pool` is required and its +guard message explicitly echoes the runbook's "do not assume glance -- +confirm against `ceph osd pool ls`" caution. + +### 3. DOCFIX-165 -- `scripts/dc-dc-dr-drill.sh` +FILES: `scripts/dc-dc-dr-drill.sh`, new `tests/dc-dc-dr-drill/run-tests.sh`. + +WHAT: subcommand `failover` (Step 10.2: `rbd mirror pool promote --force`, +gated on an explicit `--confirmed-down` flag acknowledging the runbook's +Step 10.1 independent-path split-brain check was already done elsewhere -- +this script does not perform that check itself; the per-image Glance +re-registration step is ALWAYS printed as a flagged MANUAL reminder and is +NEVER executed by this script even under `--apply`, because the runbook is +explicit the exact invocation depends on unconfirmed Glance driver +behavior). Subcommand `failback` (Steps 11.2-11.4): demote the recovering +side (11.2), a Step 11.3 reminder (Cinder backup/restore reconciliation -- +tenant/volume-specific, intentionally NOT scripted here), then -- THE +SAFETY-CRITICAL PART -- demote the CURRENT PRIMARY before promoting the +RECOVERING side (11.4a then 11.4b), hard-coded as fixed call order in the +script (not a flag an operator could reorder, not just documentation): in +apply mode, 11.4b is only ever invoked if 11.4a's demote succeeded first. +`--skip-11-4` matches the runbook's own "optional, may stay on the current +primary for a while" clause. `--apply` additionally requires a typed +`FAILBACK` confirmation read from `/dev/tty` (matching `scripts/phase-00- +teardown-destroy.sh`'s typed-approval gate pattern; `--no-prompt` bypasses it +for tested automation only). + +### Shared `$DC` gate design (all three scripts) +Each script sources `scripts/lib-net.sh` and calls `lib_net_select_dc` +(DOCFIX-151 convention) for every DC token it touches. The gate is +INFORMATIONAL in dry-run mode (a plan preview is not a mutation -- the whole +point of `--dry-run` is to stay reviewable even for a DC whose network +literals aren't assigned yet, which is exactly today's DC2 state) and HARD +BLOCKING immediately before any `--apply` execution (refuses to mutate an +unassigned DC, exit 3). Each script also prints, purely as a reminder, the +replication-plane CIDR key from `lib-net.sh`'s `PLANE_CIDRS`/`PLANE_NAME` +(the "10.12.36.0/22" -> "replication" mapping, per D-101/D-108's correction +of the DR-seed's original v4 carrier language) -- this value is NEVER +substituted into a command; the real ULA peer/endpoint address is always a +separately measured, required argument. + +## What is STANDARD vs REPO-SPECIFIC (stated in each script's own header) + +STANDARD, trust these: every `radosgw-admin`/`rbd mirror pool` subcommand +and flag used is copied verbatim from the runbook (itself copied from +upstream Ceph multisite/rbd-mirror docs) -- realm/zonegroup/zone create, +zone modify --read-only=false, period update --commit, pool enable image, +peer bootstrap create/import --direction, pool promote --force, pool +demote/promote. Nothing here is invented Ceph administration. + +REPO-SPECIFIC, needs review before trusting: every name (realm/zonegroup/ +zone/pool/site), every endpoint URL, every juju model/unit, and the charm +restart-action name -- none of these exist yet (Stage 6's own "Known gaps" +section: no `cinder-backup` charm, no `ceph-rbd-mirror` charm, `rbd-mirror` +binding still on `storage` not `replication`). All required arguments, zero +invented defaults. + +## VERIFIED + +- New harnesses, offline only (no live juju/Ceph reachable or attempted): + - `tests/dc-dc-radosgw-multisite/run-tests.sh`: 19/19 PASS. + - `tests/dc-dc-rbd-mirror/run-tests.sh`: 19/19 PASS. + - `tests/dc-dc-dr-drill/run-tests.sh`: 23/23 PASS -- including the + single most important assertion in this delivery: the failback PLAN's + printed line order is 11.2 < 11.4a(demote CURRENT PRIMARY) < + 11.4b(promote RECOVERING), verified by grepping actual line numbers in + the dry-run output, plus a check that 11.4a is a `demote` and 11.4b is a + `promote` (not swapped). + - Each harness explicitly tests ONLY this script's own logic (argument + parsing, guard clauses, dry-run plan text/order, the `$DC` gate's + dry-run-informational/apply-blocking split, the `--apply`/typed- + confirmation gates) -- none validate real radosgw/rbd-mirror daemon + behavior, stated in each harness's own header, matching + `tests/opnsense-prep-image/run-tests.sh`'s disclosure style. + - `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN + (unchanged). + - `bash scripts/run-tests-all.sh`: full gauntlet re-run after this + delivery; no new failures introduced by these three scripts/harnesses. + +REVERT: `rm -f scripts/dc-dc-radosgw-multisite.sh scripts/dc-dc-rbd-mirror.sh +scripts/dc-dc-dr-drill.sh && rm -rf tests/dc-dc-radosgw-multisite tests/dc-dc- +rbd-mirror tests/dc-dc-dr-drill` (safe -- nothing else in the repo calls any +of these three scripts yet). + +## Next actionable step + +Tooling gap register item #5's MECHANISM is CLOSED; its DATA half (a real +DC1/DC2 Ceph cluster, real realm/zonegroup/zone/pool names, real endpoints, +the three `bundle.yaml` gaps from Stage 6's own "Known gaps" section) is +unchanged and still blocks Stage 6 for real. The Stage 6 runbook +(`runbooks/dc-dc-phase5-dr-failover-drill.md`) Steps 4/5/8/10.2/11.2-11.4 and +its delivery checklist now point at these scripts instead of describing them +as future work -- the runbook's own command sequences remain the reference +if a script's behavior is ever in question. `docs/dc-dc-deployment- +workflow.md` tooling gap register item #5 updated to CLOSED (mechanism), +mirroring the phrasing pattern used for items #1/#3/#9. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-design.md b/docs/archive/changelogs/changelog-20260709-dc-dc-design.md new file mode 100644 index 0000000..bf46878 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-design.md @@ -0,0 +1,45 @@ +# Changelog 2026-07-09 -- DC-DC (VR1) originating design + +Delivery from the main-chat DC-DC planning stream. Grounded on +`openstack-caracal-dc-dc` HEAD 8813efc (seed-verified). All decisions PROPOSED -- +this is a design/plan deliverable, not an implementation. Under blanket approval this +changelog is the review surface: what, why (evidence), and how to revert, per item. + +## Items + +1. ADD `docs/dc-dc-buildout-design.md` -- originating buildout design for the virtual + regional two-DC test env (VR1), Phase 0-6 spine, gates, D-refs, shim register, DR drill + skeleton, host/disk/MTU plan. + Why: the mission deliverable; no prior DC-DC design doc existed. + Revert: delete the file. + +2. ADD `docs/dc-dc-D100-D110.append.md` -- append fragment carrying PROPOSED decisions + D-100..D-110 for concatenation onto `docs/design-decisions.md`. + Why: encodes the settled DC-DC inputs as append-only D-entries; fragment (not direct + edit of design-decisions.md) keeps the overlay a clean documented append. + Revert: delete the fragment; if already appended to design-decisions.md, remove the + D-100..D-110 block (append-only, so a tail trim) and re-run ledger-scan. + +## Grounded corrections captured (not silent) + +- D-052 "metal-internal may-cross-fiber" refined to DC-local for the independent-cloud + topology (only replication crosses). Evidence: D-052 text (stretched-cloud assumption) + + independent-Keystones topology. +- D-057 provider-vip plane NOT carried -- reverted by D-060 (Pattern A). Only the + root-cause br-ex discipline is carried. Evidence: D-057 live status header. +- DR-seed v4 replication carrier (10.12.36.0/22) superseded -> IPv6-only ULA. Evidence: + operator ruling + D-101 family matrix. +- No IPv6 literals hardcoded -- ULA /48, per-DC GUA carve, DC2 v4 supernet are + NetBox-authoritative. Evidence: repo grep found no committed authoritative literals. + +## Landing (operator) + +Per `docs/dc-dc-buildout-design.md` Section 11: append the fragment to design-decisions.md, +run `ledger-scan.sh` (expect D next-free 111), add a main-chat ledger-fence line, +`repo-lint.sh` (ignore >099 next-free noise), commit. + +## Open for redline + +Ceph size=3 vs documented size=2 fallback (Phase-0 disk math); netem link profile; +D-102/D-101 merge; D-109 standalone; metal-admin ULA leg; COS roll-up; mirror sync topology. +See design doc Section 10. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase0-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase0-runbook.md new file mode 100644 index 0000000..ced3740 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase0-runbook.md @@ -0,0 +1,75 @@ +# Changelog 2026-07-09 -- Stage 1 runbook: Phase 0 vcloud host prep (DOCFIX-153) + +No live infrastructure touched -- new runbook document only. Operator +clarified this session runs from the workstation (not the vopenstack-jesse +jumphost or the vcloud host); tonight's scope is prep so tomorrow morning's +switch to the vcloud host can execute against real infrastructure +immediately. This is the FIRST runbook that will be, since Stage 0 +(ratification) is a decision-record exercise, not an infrastructure step. + +## Item + +### 1. DOCFIX-153 -- `runbooks/dc-dc-phase0-vcloud-prep.md` +FILES: `runbooks/dc-dc-phase0-vcloud-prep.md` (new), `docs/dc-dc-deployment- +workflow.md` (Stage 1 row + tracker table updated). + +WHAT: command-level runbook for buildout-design Section 4's Phase 0 (turn +the bare vcloud host into a substrate Office1 can deploy from), matching +this repo's established runbook conventions (see `runbooks/phase-00- +teardown-maas-reset.md` as the style template): a sequence table, read-only +CHECK blocks before every mutation, each MUTATION individually gated, an +explicit gate section at the end, and a delivery checklist. + +Eleven steps: identify the target host + confirm libvirt reach; measure +CPU/RAM/disk and CPU virtualization-extension support; measure host L2 MTU +and record the D-101 MTU decision (jumbo vs. 1500-with-reduced-tenant-MTU) +plus the Ceph size=3-vs-size=2 disk-budget plausibility check (Section 3's +three Phase-0 gates); enable nested KVM if not already on; prepare libvirt +storage pool paths for DC1 and Office1 (explicitly NOT DC2 -- no CIDRs +assigned yet, gap #3's data half); install/confirm OpenTofu and registry +reach; write a `dc-dc-phase0.auto.tfvars` file from the session's measured +values (with `maas_api_key` explicitly kept OUT of any committed file, via +`TF_VAR_maas_api_key` instead); `tofu init`/`validate`/`fmt`; `tofu plan` +reviewed line-by-line against an explicit expected-resource list before +`tofu apply`; post-apply verification against the Phase-0 gate criteria. + +**Genuine open question surfaced and flagged, not silently resolved:** +`opentofu/main.tf` declares the MAAS provider block unconditionally, even +though this Phase-0 stage's plan only touches libvirt resources (planes, +pools, mesh links) -- no MAAS resource is instantiated until Stage 3. It is +not yet known whether `tofu init`/`plan` will demand real +`maas_api_url`/`maas_api_key` values before Stage 2 has even stood up MAAS. +The runbook documents both possible outcomes and what to do for each +(placeholder values IF plan-time validation only checks non-null, vs. a +logged structural finding -- splitting the MAAS provider into a Stage-3-only +root module -- if OpenTofu actually attempts real connectivity at plan +time with zero MAAS resources). This is exactly the kind of thing this +runbook is supposed to catch by being read carefully before tomorrow's +execution, rather than being discovered live. + +**Also flagged, not silently assumed:** if the vcloud host tomorrow is the +SAME physical host that ran the VR0/DC0 single-DC testcloud (buildout-design's +own Ground Truth section implies this -- the "~256 vCPU/1 TiB/1 TiB disk +dedicated to VR1" host is a repurposing, not a new machine), Step 5 tells the +operator to check for an existing libvirt storage pool at any candidate path +before creating DC1's pool there, to avoid a silent path collision with +VR0's own pool. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn +(unchanged -- prose-only addition, no script logic touched). This runbook +itself is NOT YET EXECUTED -- that is tomorrow morning's work, per the +operator's own plan; nothing in this delivery claims otherwise. + +REVERT: `rm runbooks/dc-dc-phase0-vcloud-prep.md && git checkout HEAD~ -- +docs/dc-dc-deployment-workflow.md` (safe -- no infrastructure has been +touched by this delivery, purely a document). + +## Next actionable step + +This runbook is the hand-off artifact for tomorrow morning's switch to the +vcloud host. Stage 2 (Office1 headend standup) is the next runbook to draft +tonight, continuing while the operator is away -- its content depends on +Stage 1 actually having run (real libvirt reach, pool paths, MTU decision) +only loosely; it can be drafted in parallel since MAAS-region/NetBox/ +GitBucket/Tailscale standup on Office1 is a mostly independent set of +steps once OpenTofu itself is confirmed working. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase1-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase1-runbook.md new file mode 100644 index 0000000..c363f61 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase1-runbook.md @@ -0,0 +1,66 @@ +# Changelog 2026-07-09 -- Stage 2 runbook: Office1 headend standup (DOCFIX-154) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +the buildout design, D-103/D-106/D-107, the tooling gap register, and the +Stage 1 runbook as a style template), reviewed and lint-fixed before this +commit. + +## Item + +### 1. DOCFIX-154 -- `runbooks/dc-dc-phase1-office1-standup.md` +FILES: `runbooks/dc-dc-phase1-office1-standup.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 2 row updated). + +WHAT: command-level runbook for buildout-design Section 4's Phase 1 +(Office1 headend standup: MAAS region controller, OpenTofu reach +re-verification, NetBox, GitBucket, Tailscale), matching this repo's +runbook conventions. + +**Structural blocker surfaced, not silently resolved:** none of the three +Office1 service VMs (MAAS-region, NetBox, GitBucket) have an OpenTofu +`cloudinit-vm` instantiation yet -- no image source chosen, no +`user_data`/`meta_data`/`network_config` designed. Rather than inventing +that content or silently defaulting to a manual path, every VM-creation +step presents an explicit Option A (OpenTofu, blocked pending design) vs. +Option B (manual `virt-install`, logged as debt) fork. + +**Genuine, previously-unnoticed gap found and flagged:** no Office1-local +virtual network is modeled anywhere in `opentofu/` -- `main.tf` wires +`office1_pool_path` (storage only) and the three inter-site mesh links +(management-traffic-only per D-100's own sub-item ruling), but nothing the +three service VMs (or the vcloud host's own management interface) could +actually attach to as a shared local network. Flagged as an open question +for whoever designs the `cloudinit-vm` content next, not resolved here. + +**Fixed during review (this session, not the subagent):** the subagent's +first draft cited the historical D-014 GitBucket path (`jesse.austin/ +openstack-caracal-ipv4`) as context for why THIS stage's GitBucket instance +is separate -- correctly sourced, but repo-lint's L2 rule flags that literal +string as a "dead repo path" regardless of context (a blunt static check). +Reworded onto a single line with the word "historical" so the rule's own +existing EXEMPT pattern applies, rather than deleting a legitimate, +correctly-attributed historical reference. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +NOT YET EXECUTED -- authored with no live vcloud host, MAAS, NetBox, +GitBucket, or Tailscale reachable this session; every command is written +for the operator to run and individually gate later. + +**Honest exit state, per the runbook's own GATE section:** MAAS/GitBucket/ +Tailscale/OpenTofu-reach bullets are closeable in one real run; NetBox's +"authoritative and populated" bullet is only PARTIAL -- the import +mechanism (DOCFIX-152) is runnable, but the underlying literals (org ULA +/48, per-DC GUA carve, DC2 supernet) are not assigned this session, so this +runbook cannot close that half of Stage 2's gate. This is stated plainly in +the runbook rather than rounded up to "done." + +REVERT: `rm runbooks/dc-dc-phase1-office1-standup.md && git checkout HEAD~ +-- docs/dc-dc-deployment-workflow.md` (safe -- no infrastructure touched). + +## Next actionable step + +Stage 3's runbook (OpenTofu builds each DC substrate) depends on this +stage's MAAS reachability for its own MAAS-registration step -- both were +drafted in parallel tonight; Stage 3's runbook correctly states that +dependency rather than assuming Stage 2 is done. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase2-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase2-runbook.md new file mode 100644 index 0000000..1ceaf19 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase2-runbook.md @@ -0,0 +1,63 @@ +# Changelog 2026-07-09 -- Stage 3 runbook: OpenTofu builds each DC substrate (DOCFIX-155) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +the buildout design, D-103/D-100, the tooling gap register, `opentofu/ +README.md` and `templates/README.md`, and the Stage 1 runbook as a style +template), reviewed before this commit -- no changes needed, repo-lint was +clean on the first pass. + +## Item + +### 1. DOCFIX-155 -- `runbooks/dc-dc-phase2-tofu-dc-substrate.md` +FILES: `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 3 row updated). + +WHAT: command-level runbook for buildout-design Section 4's Phase 2 +(OpenTofu builds each DC's node-VM domains, plane networks, and OPNsense +edge, then registers the DC's libvirt host with its MAAS rack controller). +DC1-first; DC2 is treated as a HARD gate (D-101's supernet still +unassigned, gap #3's data half), not merely "do it later" -- the runbook +explicitly instructs not to uncomment `main.tf`'s commented DC2 blocks or +invent placeholder DC2 CIDRs. + +**Five `!!!` callouts up front**, each naming a real constraint rather than +glossing over it: (1) hard dependency on Stage 2's MAAS reachability -- +Steps 1-8 (libvirt-layer only) can proceed without it, Step 9 onward +cannot; (2) DC2's hard gate; (3) node-VM creation has no Roosevelt analog +(Section 9 shim register); (4) node sizing (vCPU/RAM/disk) is undecided -- +the runbook uses `` placeholders in its `.tf` block shapes, +never invented numbers; (5) `tc netem` parameters remain an unruled D-100 +sub-item (gap #11) -- the runbook applies ONLY the buildout design's +qualitative "same-metro lean" language as an explicitly-labeled placeholder, +never a specific number presented as measured or ruled. + +**A sixth callout** confirms the OPNsense prep tools (`bunzip2`, +`qemu-img`, `genisoimage`/`xorriso`) are not yet confirmed present on the +vcloud host -- Step 2 checks for all of them before anything else proceeds. + +**Config.xml token gap named precisely:** `{{WAN_IF}}`/`{{LAN_IF}}` are the +one token pair that genuinely cannot be filled before the OPNsense domain +exists (only measurable via a real boot's `ifconfig` output) -- the runbook +states this plainly as a same-session blocker for the config render step, +rather than guessing a `vtnetN` name. + +**The GATE section is deliberately honest**, walking each of Stage 3's four +exit-condition bullets and stating which are achievable this session vs. +conditional on Stage 2, node sizing, a real boot measurement, or D-100's +netem ruling -- concluding "this stage's exit gate is therefore +CONDITIONALLY MET AT BEST," with explicit instruction to update the +tracker doc to the real state, never a blanket DONE. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed for this file). NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase2-tofu-dc-substrate.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +Stage 4 (MAAS enlist/commission/deploy) is the direct successor; its +runbook was drafted in parallel tonight and should be reviewed for +consistency with this stage's own MAAS-discovery verification step (Step +10 here hands off to Stage 4's commissioning work). diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase3-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase3-runbook.md new file mode 100644 index 0000000..2ff2f3a --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase3-runbook.md @@ -0,0 +1,73 @@ +# Changelog 2026-07-09 -- Stage 4 runbook: MAAS enlist/commission/deploy per DC (DOCFIX-157) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +D-103/D-100/D-107, the tooling gap register, `phase-00-teardown-maas- +reset.md` and `scripts/lib-net.sh`/`lib-hosts.sh` as primary sources), +reviewed before this commit -- repo-lint was clean on the first pass. + +## Item + +### 1. DOCFIX-157 -- `runbooks/dc-dc-phase3-maas-enlist-deploy.md` +FILES: `runbooks/dc-dc-phase3-maas-enlist-deploy.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 4 row updated, tooling gap register items +11/12/13/14/15 added or corrected). + +WHAT: adapts `phase-00-teardown-maas-reset.md`'s validated single-rack +MAAS commission/deploy/carve sequence to VR1's multi-rack, per-DC shape. +Run once per `$DC` (dc1/dc2), explicitly not mixed in one shell session. + +**Mandates the `$DC` selector convention as its first real step** -- +`lib_net_select_dc "$DC"` (a documented no-op for dc1, fail-loud for dc2) +followed HONESTLY by `lib_hosts_select_dc "$DC"`, which the runbook states +is EXPECTED TO FAIL for both DCs right now: no per-DC host data exists in +`scripts/lib-hosts.sh` yet, and populating it from THIS stage's own real +enrollment is exactly the follow-up delivery that lets the selector start +passing. This is documented as a deliberate chicken-and-egg, not a bug to +route around -- every step in the runbook resolves host identity LIVE from +MAAS instead. + +**Seven real gaps found and named in a dedicated section, each logged as a +finding rather than silently worked around:** +1. No OpenTofu module stands up a per-DC MAAS rack controller VM itself -- + only registration (`modules/maas-vm-host`) exists. +2. `reenroll-hosts.sh`, `carve-host-interfaces.sh`, and `phase-00-maas- + standup.sh` do not accept a `$DC` argument or call the new selector + functions -- only `lib-net.sh`/`lib-hosts.sh` themselves gained them + (DOCFIX-151). The runbook describes adapting each script's MECHANICS by + hand rather than inventing a `--dc` flag those scripts don't have. +3. Per-DC node count/sizing is an unmade Phase-0 decision -- the runbook + discovers however many nodes actually exist live (Step 2), never + assuming DC0's four hosts carry over. +4. DC2 is additionally blocked at the `lib_net_select_dc` layer (no NetBox + literals yet). +5. The per-DC artifact mirror (D-107) has no identified owning stage/ + module anywhere in this repo as surveyed -- flagged before Step 7's + reachability check assumes one exists. +6. The MAAS status a VM-host-DISCOVERED (vs. PXE-enlisted) pre-existing + domain lands in has never been observed in this repo -- Step 2 measures + it live rather than assuming `New` or an auto-`Ready` per phase-00's + different code path. +7. `lib-hosts.sh`'s `HOST_TAG`/boot-fabric-name literals (`openstack`, + `2_metal`) are DC0-specific -- confirmed live per DC, not assumed reused. + +**A real sequencing subtlety flagged, not glossed over:** phase-00's own +precedent carves interfaces BEFORE deploy (Ready-only); if Stage 3's +discovery+commission flow for a DC leaves nodes at `Ready` before this +stage's carve step, the runbook instructs reordering Steps 4/5 to match +phase-00's actual order and recording which order was used, rather than +blindly following this document's own numbering against contradicting live +machine states. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed). NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase3-maas-enlist-deploy.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +This stage's own delivery checklist requires populating `scripts/lib- +hosts.sh` with real per-DC host data once it actually runs -- that follow- +up unblocks `lib_hosts_select_dc` for the DC it ran against and is a +prerequisite named explicitly in Stage 5's own runbook. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase4-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase4-runbook.md new file mode 100644 index 0000000..11ada28 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase4-runbook.md @@ -0,0 +1,73 @@ +# Changelog 2026-07-09 -- Stage 5 runbook: Juju controller + bundle, per DC (DOCFIX-156) + +No live infrastructure touched -- new runbook document only. Authored +directly (not delegated), since this is the ground-truth/copy-point stage +(`docs/dc-dc-buildout-design.md` line 12: this whole repo was seeded from +`openstack-caracal-ipv4` specifically so `bundle.yaml` and `phase-01..08` +could be reused here) and the most consequential adaptation to get right. + +## Item + +### 1. DOCFIX-156 -- `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` +FILES: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 5 row updated). + +WHAT: does NOT duplicate `phase-01..08`'s command-level content. Describes +what changes to run that existing, validated deploy loop TWICE (once per +DC): a dedicated per-DC single-unit Juju controller bootstrap (D-104), the +existing bundle deploy/vault-bringup/core-verify/network-carve/octavia +sequence referenced by filename, and the D-109 independent-per-DC-Vault- +root implication (separate `vault operator init` runs, separate unseal +shares/root tokens, stored separately per DC). + +**DC1 needs literally NO VIP/CIDR changes** -- D-101 inherits DC0's layout +unchanged, so the existing `bundle.yaml` deploys into DC1's own Juju model +verbatim. **DC2 is explicitly blocked** on the same still-unassigned NetBox +literals as Stages 3/4 (gap #3's data half) -- the runbook states plainly +that DC2's octavia SAN IP, VIP band, and everything else address-derived +cannot be decided until those literals exist, and instructs stopping there +for DC2 rather than inventing a "test range." + +**Every `phase-01..08` command block that reads `scripts/lib-net.sh`/ +`lib-hosts.sh` values must be preceded by an explicit `lib_net_select_dc +"$DC"` / `lib_hosts_select_dc "$DC"` call (DOCFIX-151)** -- the runbook +calls out that for DC1 both are a documented no-op (so skipping them +"happens to work" by accident) while for DC2 skipping them would silently +reuse DC1's values, exactly the bug the selector convention exists to +prevent. This is the stage tooling gap register item #1 was built to +unblock, and this is the first runbook to actually exercise it. + +**Two real, unresolved design gaps surfaced rather than papered over:** +1. **The D-101 IPv6 family-matrix overlay does not exist.** Which planes + need ULA-only vs. dual-stack vs. GUA addressing is fully specified by + D-101 (Section 2 of the buildout design), but the actual Juju charm + config option names needed to configure that per application (`ceph- + mon`, `ovn-central`, `octavia`, etc.) have not been confirmed against + any charm's real current schema this session. Rather than inventing + plausible-looking option names, Step 6 names this as blocking work with + a three-part path to close it (confirm real charm options -> author the + overlay -> apply it) and explicitly forbids fabricating option names. +2. **D-109's IPv6-SAN cert-issuance requirement is an unverified charm- + behavior assumption** -- whether the vault/certificates-relation charms + in `bundle.yaml` actually support issuing v6 SANs today has not been + confirmed; flagged as a real finding to log if it turns out they don't, + not something to silently route around. + +Also documents the metal-admin ULA amendment explicitly (the 2026-07-09 +operator ruling reversing D-101's own original v4-only lean for that +plane), warning against assuming older documentation/training-data +patterns. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase4-juju-bundle-per-dc.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +The Step 6 IPv6 overlay design gap is the single biggest piece of real, +undone work surfaced by tonight's runbook-authoring pass -- it blocks this +stage's own "Ceph-over-v6 and geneve-over-v6 verified" gate criterion and +should be logged as its own tooling-gap-register item in the final +consolidation pass, alongside gaps #1-#11 already tracked. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase5-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase5-runbook.md new file mode 100644 index 0000000..801792b --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase5-runbook.md @@ -0,0 +1,66 @@ +# Changelog 2026-07-09 -- Stage 6 runbook: DR wiring and failover drill (DOCFIX-158) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +the DR seed doc, D-108/D-104, Section 8's failover skeleton, and the +tooling gap register), reviewed before this commit -- repo-lint was clean +on the first pass. + +## Item + +### 1. DOCFIX-158 -- `runbooks/dc-dc-phase5-dr-failover-drill.md` +FILES: `runbooks/dc-dc-phase5-dr-failover-drill.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 6 row updated, tooling gap register item 14 +added). + +WHAT: command-level implementation of `docs/dc-dc-buildout-design.md` +Section 8's failover/failback skeleton, staged one-way-then-two-way per +D-108's explicit requirement, plus the D-104 per-DC controller backup/ +restore drill (exercised here, not deferred). + +**Three real bundle gaps found and flagged before any mutation is +proposed** (grepped `bundle.yaml` directly, not assumed): no `cinder- +backup` charm is deployed in either DC's bundle (the DR seed's own text +assumes cinder already exposes a backup-backend endpoint, which is true, +but the separate charm that runs the backup driver isn't there); no +`ceph-rbd-mirror` charm is deployed either; and `ceph-mon`'s existing +`rbd-mirror` endpoint binding is on `storage`, not `replication` +(inherited from the VR0/DC0 seed at commit 8813efc, predates D-108's ULA- +carrier requirement). All three become the runbook's own Step 2 (a gated, +reviewed bundle-relation-binding change with its own changelog/revert +requirement) rather than being silently routed around or left unmentioned. + +**Every realm/zonegroup/zone/pool/endpoint name is an explicit +`` placeholder** -- the runbook is explicit that the +underlying Ceph commands (`radosgw-admin realm/zonegroup/zone create`, +`rbd mirror pool enable/peer bootstrap/promote/demote`) are standard, +well-documented upstream Ceph administration, not invented for this repo, +while the specific names/endpoints/addresses genuinely don't exist yet and +must never be guessed. + +**A negative-control drill precedes the real one:** Step 10.0 runs a +replication-plane-ONLY partition first (DC1 stays otherwise reachable) and +explicitly states the split-brain-avoidance check in Step 10.1 SHOULD +REFUSE to treat that as "DC1 down" -- if it promotes DC2 anyway under a +mere partition, that's flagged as a finding against the drill design, not +a pass. Only after that negative control does Step 10 run the real +hard-down drill. + +RTO/RPO are treated strictly as MEASURED DRILL OUTPUTS throughout (Steps +6, 7, 9, 10.5, 11.5), never asserted as a pre-committed SLA, matching +D-108's own explicit language. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed). NOT YET EXECUTED -- no live Ceph cluster this session. + +REVERT: `rm runbooks/dc-dc-phase5-dr-failover-drill.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +Once this drill has been run manually end-to-end at least once, the +radosgw multisite/rbd-mirror command sequences it documents are strong +candidates for extraction into real `scripts/` with a test harness, +closing tooling gap register item #5 for real -- explicitly logged as +future work in the runbook's own delivery checklist, not built here (no +live Ceph cluster exists this session to test a script against). diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-phase6-runbook.md b/docs/archive/changelogs/changelog-20260709-dc-dc-phase6-runbook.md new file mode 100644 index 0000000..604d18d --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-phase6-runbook.md @@ -0,0 +1,69 @@ +# Changelog 2026-07-09 -- Stage 7 runbook: Designate, COS, Magnum/CAPI (DOCFIX-159) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +D-106/D-008/D-019/D-021/D-105/D-046, `bundle.yaml`, `phase-01-bundle- +deploy.md`, and `phase-06/07/08` as primary sources), reviewed before this +commit -- repo-lint was clean on the first pass. + +## Item + +### 1. DOCFIX-159 -- `runbooks/dc-dc-phase6-designate-cos-magnum.md` +FILES: `runbooks/dc-dc-phase6-designate-cos-magnum.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 7 row updated). + +WHAT: per-DC Designate reactivation (D-106, superseding D-019/reactivating +D-008), COS observability, and Magnum/CAPI, reusing `phase-06/07/08` by +filename rather than duplicating their content. + +**Quotes, rather than paraphrases, the exact current repo state being +reversed:** `bundle.yaml`'s own D-019 comment line and `phase-01-bundle- +deploy.md`'s "NO designate (D-019)" gate text are both reproduced verbatim +before describing the reversal, so the diff is traceable to real text, not +a vague "add designate" gloss. + +**Correctly identifies the reversal is bigger than one application:** +D-106's own reactivated bootstrap order includes `os-public-hostname` per +API charm as its second step -- directly reversing `bundle.yaml`'s +separate B5 posture ("IP-ONLY -- os-public-hostname dropped on all API +charms... No control-plane DNS dependency"). The runbook treats this as +two coupled changes under one D-106 umbrella (add designate; re-add +os-public-hostname fleet-wide) rather than quietly bundling it into a +one-line description, and proposes -- without silently adopting -- +pushing the new per-DC FQDN literals into new `overlays/dc1-hostnames.yaml` +/`dc2-hostnames.yaml` files (mirroring the existing `octavia-pki.yaml` +overlay pattern) so `bundle.yaml` itself stays DC-agnostic. + +**D-046's domain-setup trap is quoted verbatim twice** (a top-of-file +`!!!` callout and again inline at the Magnum step) -- "magnum reports +active / 'Unit is ready' regardless of whether it has run" -- with an +explicit instruction that this re-run/re-assertion applies even when this +whole Stage 7 pass is itself a redo after an earlier Stage 4-6 rebuild. + +**COS deployment mechanism explicitly NOT invented:** D-105 rules only the +SCOPE (per-DC, in-scope, no Office1 roll-up); no decision in this repo +specifies whether COS means the `cos-lite` k8s bundle or machine-charm- +native observability. The runbook presents both real options with their +tradeoffs and defers the choice to the operator at execution time, logged +as a real architectural decision rather than picked for them. + +**DC-parameterization notes are traced, not assumed:** confirmed by +reading which phase-06/07/08 scripts actually source `scripts/lib-net.sh` +(`phase-06-k8s-bootstrap.sh`, `phase-06-kubeconfig-gate.sh`, both via +`KEYSTONE_VIP_DEFAULT`) and require `lib_net_select_dc "$DC"` first, versus +which literals those runbooks already resolve dynamically per-run and need +no further changes. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed). NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase6-designate-cos-magnum.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +This is the last of the seven Stage 1-7 runbooks (tooling gap register +item #9 now CLOSED for authoring). Remaining work is the adversarial +review pass across everything built this session and the final +consolidation of the workflow doc, gap register, companion Artifact, +changelogs, and session ledger. diff --git a/docs/archive/changelogs/changelog-20260709-dc-dc-repo-rename-sweep.md b/docs/archive/changelogs/changelog-20260709-dc-dc-repo-rename-sweep.md new file mode 100644 index 0000000..904331e --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-dc-repo-rename-sweep.md @@ -0,0 +1,87 @@ +# Changelog 2026-07-09 -- DC-DC repo-rename sweep + repo-lint L1 Windows fix + +Delivered on the freshly-seeded `openstack-caracal-dc-dc` repo (seeded from +`openstack-caracal-ipv4` at 8813efc). Executed under blanket approval; no live +cloud mutation -- repo content only. Validation at packaging: repo-lint 0 fail +/ 1 documented WARN; `tests/repo-lint` harness 29/29 (was 28/28; one case +added); full `run-tests-all.sh` gauntlet shows pre-existing Windows-session +environment gaps (missing `jq`, tty/exit-code quirks) unrelated to these +changes -- `repo-lint`, the only touched suite, is ALL PASS standalone and in +the gauntlet. + +## Items + +### 1. DOCFIX-139 -- stale `openstack-caracal-ipv4` references repaired to `openstack-caracal-dc-dc` +FILES: +- `.claude/skills/openstack-cloud-ops/SKILL.md` (repo name x2, clone path, clone + URL, session-bootstrap command) +- `.claude/skills/openstack-cloud-ops/references/environment.md` (repo web + + clone URL) +- `scripts/repo_lint.py` (module docstring only, cosmetic) +- `runbooks/ops-update-procedure.md`, `runbooks/d011-batch3-window-DRAFT.md`, + `runbooks/phase-01-bundle-deploy.md`, `runbooks/tenant-onboarding-v2-DRAFT.md` + (hardcoded `~/openstack-caracal-ipv4` jumphost clone paths inside RUN blocks) +- `clientdocs/sweep-receipt.txt` (L7 re-record; reviewed clientdocs/ for + drift from the tenant-onboarding-v2-DRAFT.md path edit -- no client-facing + file mirrors those internal jumphost lines, so no content change needed + there, only the pinned hash) + +WHY: repo was seeded under a new name for the DC-DC (VR1) phase +(`docs/design-decisions.md` D-110); operational path/URL references still +pointed at the old `openstack-caracal-ipv4` clone, which would 404 clones and +break `source`/`cp`/`BUNDLE=` defaults in runbook RUN blocks on first live use. + +ASSUMPTION FLAGGED: the jumphost clone directory is assumed to be +`~/openstack-caracal-dc-dc` (dir-name-equals-repo-name, matching the existing +convention for the sibling repo). Not measured this session (no jumphost shell +in this Windows working session) -- confirm on first live jumphost session. + +LEFT UNCHANGED (correct as historical/lineage record, not stale): +- `docs/design-decisions.md` legacy D-014 entries + the new D-110 lineage note + ("seeded from openstack-caracal-ipv4") +- `docs/dc-dc-buildout-design.md` lineage note (same commit-seed record) +- `docs/v1-pre-deploy-fixes.md` (historical record of the *sibling* repo's own + 2026-05-27 GitBucket-group move) +- `runbooks/phase-08-workload-cluster-acceptance.md:340` (correctly refers to + flipping the *sibling* ipv4 repo private at ITS v1 close, SEC-004) +- `scripts/repo_lint.py` STALE-token regex for `jesse\.austin/openstack-caracal-ipv4` + (an unrelated dead-path class from the pre-05-27 org move, still a live check) + +NOT actioned (flagged to operator, larger scope): `README.md` and +`references/environment.md`'s body content still describe VR0 DC0/v1 single-DC +scope wholesale (title, six-plane table, phase list) -- a content rewrite tied +to the VR1 DC-DC design, not a name-swap. Awaiting direction. + +REVERT: `git checkout HEAD~ -- ` per file above (dropping +`clientdocs/sweep-receipt.txt` back requires re-running +`bash scripts/repo-lint.sh --record-clientdocs-sweep` afterward against the +reverted tree, or L7 will FAIL on the mismatch). + +### 2. DOCFIX-140 -- repo_lint.py L1 Windows path-separator bug (WARN carve-out silently became FAIL) +FILE: `scripts/repo_lint.py` (3 sites: L1 encoding check, L2 stale-token check, +L3/L4/L6 runbook check -- all used `str(p.relative_to(R))`) + +WHY: found during session bootstrap -- `bash scripts/repo-lint.sh` reported +`[FAIL] L1 docs\design-decisions.md: 239 non-ASCII byte(s)` where CLAUDE.md's +documented bootstrap expectation is "0 fail (1 legacy WARN)". Root cause: the +L1 carve-out compares `rel == "docs/design-decisions.md"` (forward slash +literal) against `rel = str(p.relative_to(R))`, which yields backslashes on +Windows (`docs\design-decisions.md`). The comparison never matched on this +platform, so the documented legacy WARN silently escalated to FAIL. Confirmed +this was NOT caused by the DC-DC design-doc commits (diffed: the 200 lines +`docs/dc-dc-buildout-design.md`/2b2fa3f added contain zero non-ASCII bytes; +the flagged 239 bytes are the pre-existing legacy D-001..018 content, unchanged +today). Fixed by switching all three sites to `p.relative_to(R).as_posix()`, +matching the convention already used elsewhere in the same file (L7/L8 checks). +Also fixes cosmetic backslash-vs-forward-slash inconsistency in L2/L3/L4/L6 +message output on Windows. + +HARNESS: added T3b to `tests/repo-lint/run-tests.sh` -- plants an em-dash in a +fixture's `docs/design-decisions.md` and asserts rc=2 (WARN) with the +carve-out message, not rc=1 (FAIL). This exact carve-out path had no direct +test before (only the non-carved-out FAIL case, T2, was covered), which is how +the platform bug went unnoticed. 29/29 green after the change. + +REVERT: `git checkout HEAD~ -- scripts/repo_lint.py tests/repo-lint/run-tests.sh` +(re-arms the Windows false-FAIL; not recommended for any Windows-based +session). diff --git a/docs/archive/changelogs/changelog-20260709-dc-selector-convention.md b/docs/archive/changelogs/changelog-20260709-dc-selector-convention.md new file mode 100644 index 0000000..7a02b53 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-dc-selector-convention.md @@ -0,0 +1,78 @@ +# Changelog 2026-07-09 -- $DC selector convention for lib-net.sh/lib-hosts.sh (DOCFIX-151) + +No live infrastructure touched -- library-script additions + test harness only. +Closes tooling gap register item #1 in `docs/dc-dc-deployment-workflow.md`, +which explicitly BLOCKED the Stage 5 "high reuse" claim (`phase-01..08` +runbooks reused per-DC) from being operationally true. + +## Item + +### 1. DOCFIX-151 -- `lib_net_select_dc()` / `lib_hosts_select_dc()` +FILES: `scripts/lib-net.sh`, `scripts/lib-hosts.sh`, new +`tests/dc-selector/run-tests.sh`. + +WHAT: added one opt-in selector function to each library, following the same +pattern as the existing `$REPO` convention (DOCFIX-139/141) -- an explicit, +measured selector, never an inferred default. + +- `lib_net_select_dc()` in `lib-net.sh`: `dc0`/`dc1` are both a no-op (D-101, + ADOPTED 2026-07-09, rules DC1 inherits DC0's v4 plane layout UNCHANGED -- + so there is deliberately no second copy of the CIDR literals to drift out + of sync with the first); `dc2` FAILS LOUD (no NetBox-assigned literals + exist yet -- gap #3, still open); an unknown token FAILS LOUD. +- `lib_hosts_select_dc()` in `lib-hosts.sh`: `dc0` is a no-op (the real, + enrolled VR0 four-host set); `dc1` AND `dc2` BOTH fail loud -- unlike the + network side, DC1's host *inventory* does not exist yet either (its node + VMs haven't been created by `opentofu/modules/node-vm` or enrolled in + MAAS), so aliasing dc1 to dc0's real hosts would be a silent-reuse bug, + not a valid inheritance. This asymmetry (`lib_net_select_dc dc1` no-ops, + `lib_hosts_select_dc dc1` fails) is intentional and is called out in both + the changelog and the source comments so a future reader isn't confused by + the two libraries behaving differently for the same DC token. + +Backward compatible by construction: sourcing either file with no further +action is completely unaffected -- every existing script that sources +`lib-net.sh`/`lib-hosts.sh` today never calls the new functions, so nothing +about its behavior changes. This is an additive, opt-in mechanism. + +VERIFIED: +- New harness `tests/dc-selector/run-tests.sh`: 21/21 PASS. Covers: backward + compatibility (flat vars unaffected by sourcing alone), dc0/dc1 no-op + + dc2/unknown-token/missing-arg fail-loud for the net selector, dc0 no-op + + dc1/dc2/unknown-token/missing-arg fail-loud for the hosts selector, error + messages cite the correct blocking reason (NetBox gap vs. no-enrollment), + and an explicit assertion that the dc1 asymmetry between the two libraries + is real and intentional (not a bug). +- Caught and fixed a real bug in the harness itself while writing it: the + first draft piped `lib_net_select_dc dc2 2>&1 >/dev/null | grep -q + "NetBox"` under `set -o pipefail` -- since the selector function's own + exit code is 1 (by design, for the fail-loud case), `pipefail` made the + whole pipeline report failure even when `grep` matched correctly. Fixed by + capturing stderr into a variable via command substitution instead of + piping through grep, so the assertion checks message content independent + of the (expected-nonzero) selector exit code. +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged). +- `bash scripts/run-tests-all.sh`: confirmed via `git stash` A/B that the + SAME 23 pre-existing failures (all either "jq required" -- not installed + on this Windows workstation -- or live juju/cloud-dependent suites like + `cloud-assert`/`preflight`/`tenant-*`, none of which touch + `lib-net.sh`/`lib-hosts.sh`) exist identically with and without this + change. `dc-selector` itself and `vault-kv-health` (the only other harness + that sources either library) both PASS cleanly. This session is running + from the operator's workstation, not the vopenstack-jesse jumphost -- no + live cloud/juju/MAAS access is expected or required for tonight's prep + work (operator instruction, 2026-07-09 evening). + +REVERT: `git checkout HEAD~ -- scripts/lib-net.sh scripts/lib-hosts.sh && +rm -rf tests/dc-selector` (safe -- no other script calls either new +function yet, so nothing downstream depends on their presence). + +## Next actionable step + +Tooling gap register item #1 CLOSED. Stage 5's runbook (Juju controller + +OpenStack bundle, per DC) can now honestly claim the `phase-01..08` reuse +path IS DC-safe once it's written to call `lib_net_select_dc "$DC"` / +`lib_hosts_select_dc "$DC"` explicitly -- that call-site wiring is part of +authoring the Stage 5 runbook itself (task list item), not this delivery. +Gap #3 (NetBox multi-DC import pipeline) is the next dependency: `dc2` +remains fail-loud in both libraries until it closes. diff --git a/docs/archive/changelogs/changelog-20260709-designate-cinderbackup-rbdmirror.md b/docs/archive/changelogs/changelog-20260709-designate-cinderbackup-rbdmirror.md new file mode 100644 index 0000000..8cbbc82 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-designate-cinderbackup-rbdmirror.md @@ -0,0 +1,134 @@ +# Changelog 2026-07-10 -- bundle.yaml: Designate reactivation + D-108 DR mechanism (DOCFIX-167) + +No live infrastructure touched -- `bundle.yaml` is a repo source file; no +`juju deploy`/`juju add-relation` was run against any live model tonight +(no live cloud reachable this session). This closes tooling gap register +items #6 (Designate reactivation is a bundle change) and #14 (three real +`bundle.yaml` gaps for D-108's DR mechanism). + +**Why this file, specifically, is high-stakes and was handled with extra +care:** `bundle.yaml` is not a DC-DC-only artifact -- it is the SAME file +this repo's jumphost operations use against the live VR0/DC0 rehearsal +cloud (per `CLAUDE.md`: "Commands here reach the real cloud"). Editing it +changes what a future `juju deploy ./bundle.yaml` would do against ANY +model, not just a not-yet-created DC1/DC2. D-106 and D-108 are both +ADOPTED (Stage 0, 2026-07-09), so this reactivation is authorized in +principle -- exactly what tooling gap #6 said still needed doing ("editing +`bundle.yaml` itself... in addition to writing the reactivation runbook"). +This delivery is a REPO-level source change only; before any live +`juju deploy`/`juju add-relation` against an actual model (existing VR0/DC0 +or a future DC1/DC2), run `bash scripts/preflight.sh` and review the diff +with `juju deploy ./bundle.yaml --dry-run` first, per this repo's own +standing discipline -- do not treat this changelog as authorization to +apply live. + +## Items + +### 1. Designate reactivation (D-106, supersedes D-019) +Added `designate`, `designate-bind`, `designate-mysql-router`, and +`designate-hacluster` application blocks + 8 relations, all researched +against REAL charm metadata (`raw.githubusercontent.com/openstack/charm- +designate/master/src/metadata.yaml`, fetched directly), not guessed: + +- **Found and corrected a real mistake in the first draft during self- + review**: Designate's actual provides-side endpoint is named `dnsaas` + (interface `designate`), NOT `public` like every other API charm in this + bundle. The first draft used `public: provider-public` by pattern- + matching the other 10 charms; caught by checking designate's own + metadata.yaml directly before finalizing, corrected to `dnsaas: + provider-public`. +- Confirmed via source: `dns-backend` (interface `bind-rndc`, relates to + `designate-bind`), `ha` (interface `hacluster`), `coordinator-memcached` + (interface `memcache`, relates to the existing `memcached` principal). +- `shared-db`/`identity-service`/`amqp`/`certificates` follow the SAME + standard names every other charms.openstack-family charm in this bundle + already uses successfully -- pattern-matched (very high confidence given + 10 other charms confirm the pattern), not independently re-fetched for + designate specifically. Flagged in the bundle's own inline comment; + verify with `juju deploy --dry-run` before a real apply. +- Updated the bundle's top header: the `D-019 ... Designate deferred to + v2` line now states it is SUPERSEDED 2026-07-10 by D-106 (D-019's text + retained per this repo's append-only decision discipline -- not deleted). + The `HA Cluster Subordinates` count comment corrected from 11 to 12. +- **Deliberately NOT done here**: `os-public-hostname` re-activation on + the 11 clustered API charms (D-106's OTHER coupled change, reversing the + bundle's B5 IP-only posture) -- per `runbooks/dc-dc-phase6-designate- + cos-magnum.md`'s own proposal, this is DC-specific (embeds `dc1`/`dc2` + in the hostname) and belongs in a new per-DC overlay + (`overlays/${DC}-hostnames.yaml`), NOT this shared, DC-agnostic bundle. + A new header comment states this explicitly so a future reader isn't + confused about why B5's own line still says "IP-ONLY." + +### 2. D-108 DR mechanism: cinder-backup + ceph-rbd-mirror +Added `cinder-backup` and `ceph-rbd-mirror` application blocks + 2 +relations, and corrected `ceph-mon`'s `rbd-mirror` binding. + +- **Found and corrected two real mistakes in the first draft via source + research, not guessed:** + 1. `cinder-backup` is a SUBORDINATE charm (confirmed: + `charm-cinder-backup`'s real `metadata.yaml` carries `subordinate: + true`, provides only `backup-backend` with `scope: container`, + requires `ceph` + `juju-info` -- NO `amqp` relation exists on this + charm at all). The first draft gave it `num_units`/`to`/`constraints` + and an `amqp` binding as if it were a principal -- all removed; + corrected to a bare subordinate block matching this bundle's existing + `cinder-ceph` shape. + 2. `ceph-rbd-mirror` IS a principal (confirmed: real `metadata.yaml` + carries `subordinate: false`) with TWO requires endpoints on the SAME + `ceph-rbd-mirror` interface: `ceph-local` and `ceph-remote`. The + first draft used a single invented `ceph-rbd-mirror:` binding key + that does not exist on the real charm; corrected to the real + `ceph-local`/`ceph-remote` endpoint names. +- **A genuine, unresolved cross-DC wiring limit, documented rather than + worked around**: `ceph-remote` is meant to relate to the PEER DC's + ceph-mon -- but D-104 gives each DC its own independent Juju controller, + so a same-bundle relation line cannot express this (Juju relations are + intra-model). Only `ceph-rbd-mirror:ceph-local -> ceph-mon:rbd-mirror` + (this DC's own leg) is wired. The cross-DC peer relationship is EITHER a + `juju offer`/`juju consume` cross-model relation OR (the mechanism + `runbooks/dc-dc-phase5-dr-failover-drill.md` already documents in full) + the manual `rbd mirror pool peer bootstrap create/import` CLI flow -- + documented in the bundle's own comment rather than invented as a fake + relation line. +- `ceph-mon`'s `rbd-mirror` binding corrected from `storage` to + `replication` (D-108 requires the cross-DC rbd-mirror daemon traffic + ride the IPv6-only ULA replication plane; the old binding predates D-108, + inherited unchanged from the VR0/DC0 seed at commit 8813efc -- this is + the exact finding `runbooks/dc-dc-phase5-dr-failover-drill.md`'s "Known + gaps" section already surfaced, now actually fixed). + +## Verification + +- `python3 -c "import yaml; yaml.safe_load(open('bundle.yaml'))"` -- + parses cleanly. 56 applications (was 52), 108 relations (was up from the + baseline; net +10 for the two new service groups). Confirmed all five + new application names present and `ceph-mon`'s `rbd-mirror` binding now + reads `replication`. +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn (unchanged). +- `python3 scripts/provider-bundle-check.py`: **PASS**, all six existing + invariants still hold unchanged (11 charms bind public->provider-public, + none on provider-vip; 11 clustered VIP triples, octet 50-60; all 4 ovn- + chassis chassis MACs present; 108 relations well-formed; mysql-innodb- + cluster num_units=3; keystone policyd-override wired) -- confirms the + new applications did NOT disturb any existing invariant this repo's own + gate already checks. Designate correctly does NOT appear in the + VIP-triple count (it uses `dnsaas`/DNS-native addressing, not an + HAProxy VIP, so 11 stays 11 -- expected, not a gap in the new work). + `tests/provider-bundle-check/run-tests.sh`: 8/8 PASS (unaffected). + +REVERT: `git checkout HEAD~ -- bundle.yaml` (safe -- no live model has this +bundle applied yet this session; reverts to the pre-reactivation state +exactly, including B5's original IP-only posture and D-019's original +un-superseded text). + +## Next actionable step + +The `os-public-hostname` per-DC overlay (`overlays/${DC}-hostnames.yaml`) +this reactivation's OTHER coupled change depends on is not yet authored -- +that is Stage 5's own runbook's job at real execution time, not this +delivery's. Before any live apply: independently verify the pattern- +matched Designate relations (`shared-db`/`identity-service`/`amqp`/ +`certificates`) via `juju deploy ./bundle.yaml --dry-run` against a real +MAAS/Juju environment, since they were not individually re-confirmed from +designate's own metadata.yaml the way the DNS-specific and D-108 endpoints +were. diff --git a/docs/archive/changelogs/changelog-20260709-final-consolidation.md b/docs/archive/changelogs/changelog-20260709-final-consolidation.md new file mode 100644 index 0000000..fef30fb --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-final-consolidation.md @@ -0,0 +1,98 @@ +# Changelog 2026-07-09 -- Final consolidation pass (DOCFIX-161) + +No live infrastructure touched -- doc/tracker consistency pass only, the +last delivery of tonight's session before handing off to the operator's +morning switch to the vcloud host. + +## Items + +### 1. DOCFIX-161a -- one more real gap found during final consistency check +FILE: `docs/dc-dc-deployment-workflow.md` (tooling gap register item #16, +new). + +WHAT: while doing a final full read-through of the tracker doc for +internal consistency (cross-checking every stage's Build/Gate list against +every other stage's), found that NO stage explicitly owns creating +Office1's own OPNsense edge VM. The topology (buildout design Section 1) +is explicit that the design is "per-site" -- three sites, three simulated- +ISP-edge OPNsense instances (DC1, DC2, AND Office1). Stage 3's Build list +explicitly covers DC1/DC2's edges; Stage 2's Build list (MAAS region, +OpenTofu, NetBox, GitBucket, Tailscale) does not mention an Office1 edge +at all, and neither runbook instantiates one. Logged as gap #16 rather +than silently deciding which stage should own it or inventing an +instantiation to close the gap. + +### 2. DOCFIX-161b -- stale cross-reference cleanup +FILE: `docs/dc-dc-deployment-workflow.md` ("Where things stand today" +table). + +WHAT: two rows had gone stale as later deliveries overtook them within +the same evening and were never reconciled: +- The Design-doc row read "DONE, PROPOSED" -- confusing, since it read as + if the whole design were still proposal-stage even though Stage 0 + ratified its embedded decisions hours earlier. Reworded to distinguish + the doc's own header (still literally reads "DESIGN / PLAN" -- a status + line, not a blocker) from its embedded decisions (ADOPTED). +- The `lib-net.sh`/`lib-hosts.sh` row said "Stage 5 runbook still needs to + call it" -- true when gap #1 first closed (DOCFIX-151), but Stage 4 AND + Stage 5's runbooks (both written later the same evening) both now + mandate calling the selector. Updated to reflect both runbooks actually + requiring it, not yet executed for real. + +### 3. DOCFIX-161c -- companion Artifact redeployed twice more +FILE: (external) `https://claude.ai/code/artifact/e752f6dd-23ff-4c59-9ce5-1dedb917e5ea`. + +WHAT: (a) all seven stage pills moved from "Not started" to a new +"Runbook written" state (a 4th state added to the tracker's click-to-cycle +model, distinct from "Not started"/"In progress"/"Done," styled with the +page's existing accent color rather than good/warn/block); gap-register +items #1/#3/#9 marked closed, items #12-16 added, summary table rows +updated. (b) a second redeploy corrected the NetBox pipeline's test count +from 37/37 to 40/40 (DOCFIX-160's fix) and added a one-line note to the +runbook-debt gap item about the adversarial review pass finding 3 issues. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +Full gauntlet re-run clean (same pre-existing environment-gap failures as +every prior delivery tonight, confirmed via the session's established A/B +methodology; no new failures introduced by tonight's cumulative changes). + +REVERT: `git checkout HEAD~ -- docs/dc-dc-deployment-workflow.md` (the +Artifact redeploys are not repo-tracked and have no revert mechanism +beyond redeploying an earlier saved version). + +## Summary of tonight's full session (for the operator's morning read) + +Starting point: blanket git permission granted, subagent use authorized, +operator stepping away for the evening with instructions to make maximum +progress toward an end-to-end-ready project by morning, then clarifying +this session runs from the workstation (not the vcloud host) so tonight +is PREP ONLY. + +Delivered (DOCFIX-151 through DOCFIX-161, 11 commits): +- `$DC` selector convention for `lib-net.sh`/`lib-hosts.sh` (gap #1 closed). +- NetBox multi-DC/dual-stack import pipeline mechanism (gap #3's tooling + half closed; data half remains open pending real literal assignment). +- All seven Stage 1-7 runbooks (gap #9 closed) -- command-level, + individually-gated, each honest about what it can and cannot close given + today's real state. Five new tooling gaps (#12-16) surfaced and logged + during authoring, not silently routed around. +- An adversarial review pass that found and fixed 3 real issues (a + dead-code test bug, a self-contradictory comment, stale cross-refs). +- This final consolidation pass, catching one more real gap (#16) and two + more stale cross-references during a last full read-through. + +NOT done tonight, by design: nothing was executed against real +infrastructure (no vcloud host, MAAS, NetBox, or live cloud reachable from +the workstation). The operator's morning switch to the vcloud host is +where Stage 1's runbook (`runbooks/dc-dc-phase0-vcloud-prep.md`) becomes +the first thing actually run for real. + +Genuinely open items still blocking real progress, in rough execution +order: gap #7 (MTU/Ceph disk-budget arithmetic, blocks Stage 1's gate), +gap #16 (Office1 OPNsense edge ownership, blocks Stage 2/3), gap #3's data +half (NetBox literals: org ULA /48, per-DC GUA carve, DC2 supernet, blocks +Stage 2's full closure and everything DC2-side), gap #2's remaining +UNVALIDATED status (no `tofu` binary has run against any OpenTofu module +yet -- the vcloud host tomorrow is where that changes), gap #12 (Office1- +local network), gap #13 (IPv6 family-matrix overlay), gap #14 (three +`bundle.yaml` DR gaps), gap #15 (seven MAAS/multi-rack gaps). diff --git a/docs/archive/changelogs/changelog-20260709-ipv6-family-matrix-overlay.md b/docs/archive/changelogs/changelog-20260709-ipv6-family-matrix-overlay.md new file mode 100644 index 0000000..f4b5db4 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-ipv6-family-matrix-overlay.md @@ -0,0 +1,71 @@ +# Changelog 2026-07-10 -- D-101 IPv6 family-matrix overlay drafted (DOCFIX-164) + +No live infrastructure touched -- new overlay + research memo, and +existing-doc updates. Closes the "overlay doesn't exist" half of tooling +gap register item #13; researched and authored directly (not delegated), +given the accuracy stakes of getting charm config option names right. + +## Item + +### 1. DOCFIX-164 -- `overlays/dc-dc-ipv6-family-matrix.yaml` + research memo +FILES: `overlays/dc-dc-ipv6-family-matrix.yaml` (new), `docs/dc-dc-ipv6- +charm-research.md` (new, full sourcing), `runbooks/dc-dc-phase4-juju- +bundle-per-dc.md` (Step 6 updated from "NOT YET WRITTEN" to "DRAFTED"), +`docs/dc-dc-deployment-workflow.md` (gap #13 updated). + +WHAT: researched real charm config/source directly (WebFetch/WebSearch +against `openstack/charm-*` GitHub mirrors and Launchpad, not memory) to +build a sourced IPv6/dual-stack overlay for the nine dual-stack API +charms plus `ceph-mon`'s ULA-only switch, rather than guessing plausible +option names. + +**Key findings, each independently confirmed by reading real source:** +- `prefer-ipv6` is a real, shared charms.openstack-family config option + (confirmed in `charm-keystone/config.yaml`). +- It is genuinely ADDITIVE dual-stack, not either/or -- confirmed by + reading `charm-nova-cloud-controller`'s actual "Dual Stack VIPs" commit: + HAProxy binds both `*:port` (v4) and `:::port` (v6) simultaneously when + the flag is set. +- `ceph-mon` has its OWN separate `prefer-ipv6` + `ceph-public-network`/ + `ceph-cluster-network`, which per the charm's own docs IS a straight + either/or switch -- the correct shape for D-101's ULA-only storage/ + replication planes. +- OVN (`ovn-central`/`ovn-chassis`, via `charm-layer-ovn`) has NO IPv6/ + encapsulation config option at all (confirmed: fetched and read the + full 25-option `config.yaml`) -- geneve family follows the bound + interface automatically once the plane is ULA-only; no overlay entry + needed or written for it. +- `charm-vault`'s cert-issuance code (`vault_pki.py`'s `sort_sans()`) has + no IPv4/IPv6 distinction -- D-109's IPv6-SAN requirement is code- + confirmed, not merely assumed. + +**A real, open risk found and deliberately NOT worked around:** Octavia's +`lb-mgmt-net` (the amphora management network D-101 wants ULA-only) has +two real, still-referenced Launchpad bug reports describing IPv6 failures +(#1911788, #1913409). #1911788's root cause is an OVN/LXD/MAAS hostname- +resolution mismatch -- the SAME CLASS of problem this repo's own D-008 +bootstrap order already hardens against, so not necessarily fatal, but +genuinely unresolved. The overlay deliberately excludes an Octavia entry +rather than silently forcing the risk through; both paths forward +(accept and test, or keep Octavia's lb-mgmt-net as a deliberate v4-only +exception) are presented for an explicit operator decision. + +**Also flagged, not silently assumed:** 8 of the 9 dual-stack charms' +`prefer-ipv6` presence is inferred by shared-framework pattern (very high +confidence, but not each individually fetched) -- confirm via `juju +config ` once any is actually deployed. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +UNVALIDATED against any live model -- no Juju session this evening. + +REVERT: `rm overlays/dc-dc-ipv6-family-matrix.yaml docs/dc-dc-ipv6-charm- +research.md && git checkout HEAD~ -- runbooks/dc-dc-phase4-juju-bundle- +per-dc.md docs/dc-dc-deployment-workflow.md` (safe -- nothing applied +live). + +## Next actionable step + +Before Stage 5's runbook can honestly close its "Ceph-over-v6 and geneve- +over-v6 verified" gate: (a) confirm the 8 inferred-by-pattern charms' +`prefer-ipv6` option live, once any is deployed; (b) get an explicit +operator decision on the Octavia lb-mgmt-net IPv6 risk. diff --git a/docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md b/docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md new file mode 100644 index 0000000..506122f --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md @@ -0,0 +1,153 @@ +# Changelog 2026-07-10 -- $DC parameterization for the 3 MAAS scripts (DOCFIX-166) + +No live infrastructure touched -- script + test-harness changes only (prep +session; no jumphost/juju/MAAS access this session). Closes the CLI-wiring +half of tooling gap register item #15 sub-item 2 in +`docs/dc-dc-deployment-workflow.md` (full detail in +`runbooks/dc-dc-phase3-maas-enlist-deploy.md`'s "Known gaps" section, item +2), which blocked `scripts/reenroll-hosts.sh`, `scripts/carve-host- +interfaces.sh`, and `scripts/phase-00-maas-standup.sh` from being invoked +with a `$DC` argument at all -- DOCFIX-151 (2026-07-09) only gave the +selector *functions* to `lib-net.sh`/`lib-hosts.sh`; no consumer script +called them yet. + +## Item + +### 1. DOCFIX-166 -- `$DC` env var wiring in the 3 MAAS scripts +FILES: `scripts/reenroll-hosts.sh`, `scripts/carve-host-interfaces.sh`, +`scripts/phase-00-maas-standup.sh`, `tests/carve-host-interfaces/run- +tests.sh`, `tests/phase-00-maas-standup/run-tests.sh`, new `tests/reenroll- +hosts/run-tests.sh`. + +WHAT: each script now reads an opt-in `$DC` env var immediately after +sourcing `lib-net.sh`/`lib-hosts.sh` (the earliest possible point, matching +the pattern `runbooks/dc-dc-phase3-maas-enlist-deploy.md`'s Step 1 already +documents) and, only if `$DC` is non-empty, calls `lib_net_select_dc "$DC"` +(all three scripts) and `lib_hosts_select_dc "$DC"` (`reenroll-hosts.sh` and +`carve-host-interfaces.sh` only -- `phase-00-maas-standup.sh` never sources +`lib-hosts.sh`, it has no host-identity dependency to select). This matches +the convention the selector functions and the Stage 4 runbook already +established (an explicit `$DC` env var passed positionally into each +function), rather than inventing a new `--dc` CLI-flag convention that does +not exist anywhere else in this repo. + +```bash +DC="${DC:-}" +if [ -n "$DC" ]; then + lib_net_select_dc "$DC" + lib_hosts_select_dc "$DC" # reenroll-hosts.sh / carve-host-interfaces.sh only +fi +``` + +Backward compatible by construction: unset `$DC` skips the `if` body +entirely, so every existing invocation (`bash scripts/reenroll-hosts.sh +--check`, `bash scripts/carve-host-interfaces.sh openstack0 --apply`, `bash +scripts/phase-00-maas-standup.sh`) runs byte-for-byte as it did before this +change -- confirmed by re-invoking each script with no `$DC` set and diffing +against pre-change behavior descriptions in `runbooks/phase-00-teardown- +maas-reset.md` (see VERIFIED below). + +No second validation layer was added: because all three scripts already run +under `set -euo pipefail`, a selector function returning non-zero (dc2 at +the network layer; dc1 AND dc2 at the host layer) causes the script to exit +immediately with that same exit code -- which happens to line up with each +script's own "1 = fatal" convention, so the propagation is free. + +VERIFIED (this workstation; no jq installed, no live jumphost/MAAS/juju +access -- same environment gap DOCFIX-151's changelog documented): + +- Manual, jq-independent invocation of all three scripts confirms the exact + intended matrix (the selector calls happen before `need_jq`/any MAAS call, + so these do not need jq or a fake `maas` at all): + - `DC` unset -> unaffected (rc identical to before this change in every + script). + - `DC=dc0` -> explicit no-op, byte-for-byte same as unset. + - `DC=dc1` -> `reenroll-hosts.sh` and `carve-host-interfaces.sh` FAIL LOUD + at `lib_hosts_select_dc` ("no enrolled hosts yet..."), before touching + MAAS; `phase-00-maas-standup.sh` (network-selector only) no-ops (D-101: + dc1 inherits dc0's plane literals) and proceeds to its own MAAS calls. + - `DC=dc2` -> all three FAIL LOUD at `lib_net_select_dc` first ("no + assigned network literals yet..."), before `lib_hosts_select_dc` is ever + reached in the two scripts that also call it. + - `DC=bogus` -> all three FAIL LOUD ("unknown DC 'bogus'"). +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged). +- `bash scripts/run-tests-all.sh`: new `reenroll-hosts` suite added (none + existed before); it reports `FAIL: jq required`, identically to its + sibling suites `carve-host-interfaces` and `phase-00-maas-standup` (both + already jq-gated, unrelated to this change) and to roughly twenty other + suites in the same gauntlet run that all fail for the same reason on this + jq-less Windows workstation. Did NOT run a `git stash` A/B comparison this + time (unlike DOCFIX-151's changelog) because the working tree currently + carries OTHER in-flight, uncommitted changes from parallel agent sessions + tonight (per the operator's own instruction that multiple agents are + working concurrently) -- stashing would have touched their work, not just + this delivery's. Instead, correctness was verified directly (the + jq-independent invocation matrix above), and the only new gauntlet + observation is the addition of the `reenroll-hosts` suite itself + (previously absent, so it could not have been part of any prior baseline) + failing for the same pre-existing, already-documented reason as its + siblings -- not a regression this delivery introduced. +- Added `$DC` test cases to both existing harnesses + (`tests/carve-host-interfaces/run-tests.sh`, + `tests/phase-00-maas-standup/run-tests.sh`): a `DC=dc0` explicit no-op + case (must match the existing default-case assertions), and fail-loud + cases for the blocked DC values, using a new `run_dc()` helper that skips + fixture setup entirely (these calls exit at the selector, before the + script ever reaches `need_jq`/a MAAS call, so they do not depend on the + fixtures OR on jq being installed -- confirmed by invoking them directly, + bypassing the harness's own jq gate, per the VERIFIED bullet above). +- Created `tests/reenroll-hosts/run-tests.sh` (no harness existed for this + script before). Scoped narrowly to what this delivery actually changed: + the `$DC` fail-loud/no-op matrix, plus a smoke test that `--check`'s + read-only reporting is unaffected when `$DC` is unset. Does not attempt to + cover the create/poll/tag mutation paths (those need an interactive + password prompt and a much heavier MAAS fixture set -- out of scope for a + targeted parameterization change). Uses a minimal fake `maas` returning an + empty machines list plus `< /dev/null` on stdin so a bug that ever reached + the real interactive password prompt would fail loudly (EOF) instead of + hanging the harness. + +REVERT: `git checkout HEAD~ -- scripts/reenroll-hosts.sh scripts/carve- +host-interfaces.sh scripts/phase-00-maas-standup.sh +tests/carve-host-interfaces/run-tests.sh +tests/phase-00-maas-standup/run-tests.sh && rm -rf tests/reenroll-hosts` +(safe -- purely additive/opt-in; unset `$DC` behavior is provably unchanged, +so no other script or runbook step that does not set `$DC` depends on any of +this). + +## What this delivery does NOT close (be precise) + +Tooling gap register item #15 has SEVEN sub-findings (enumerated in +`runbooks/dc-dc-phase3-maas-enlist-deploy.md`'s "Known gaps" section). This +delivery closes ONLY the CLI-invocation half of sub-item 2 (the three +scripts not accepting `$DC` at all). It does NOT: + +- Invent any per-DC literal. No dc1/dc2 hostname, octet, boot MAC, or + VIRSH_POWER_ADDRESS was added to `lib-hosts.sh`. `lib_hosts_select_dc` + still fails loud for both dc1 and dc2, exactly as DOCFIX-151 left it -- + that data gap is a separate, later delivery item (the Stage 4 runbook's + own delivery checklist), not something this change works around. +- Make `phase-00-maas-standup.sh`'s `PLANES` table dc2-aware. That table is + a hardcoded D-052/D-053 literal, not derived from `lib-net.sh`'s flat + vars -- selecting `dc1` happens to be correct today only because D-101 + makes dc1's real target identical to dc0's, not because the script reads + dc1-specific data. Once dc2 gets real NetBox CIDRs (gap #3), this table + will need its own parameterization pass. +- Close sub-items 1 (no OpenTofu rack-controller module), 3 (node + count/sizing undecided), 4 (dc2 blocked at the network-selector layer -- + unrelated to this delivery, pre-existing), 5 (per-DC mirror ownership + unconfirmed), 6 (VM-host-discovered MAAS status unconfirmed), or 7 + (`HOST_TAG`/boot-fabric-name literals are DC0-specific). All six remain + OPEN, unchanged by this delivery. + +## Next actionable step + +`docs/dc-dc-deployment-workflow.md` item #15 and `runbooks/dc-dc-phase3- +maas-enlist-deploy.md`'s gap #2 / Steps 5 and 8 updated to reflect the +partial closure (see those files' diffs). The next real dependency for +sub-item 2's FULL closure is Stage 4's own execution: once it measures real +per-DC hostnames/octets/boot MACs and a real per-DC `PLANES` table update +gets authored, `lib-hosts.sh` and `phase-00-maas-standup.sh`'s literal table +go from "selector wired but fails loud" to "selector wired and populated" -- +no further change to the three scripts themselves should be required for +that. diff --git a/docs/archive/changelogs/changelog-20260709-mtu-ceph-budget-calculators.md b/docs/archive/changelogs/changelog-20260709-mtu-ceph-budget-calculators.md new file mode 100644 index 0000000..e9538b7 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-mtu-ceph-budget-calculators.md @@ -0,0 +1,138 @@ +# Changelog 2026-07-09 -- MTU/geneve budget + Ceph disk-budget calculators (DOCFIX-162) + +No live infrastructure touched -- read-only, arithmetic-only shell scripts +plus test harnesses only. Closes tooling gap register item #7 in +`docs/dc-dc-deployment-workflow.md`, which named the measurement-then- +computation gates in D-101 and buildout-design Section 3 as manual +arithmetic in prose today, not a script -- blocking the Stage 1 Phase-0 +gates in `runbooks/dc-dc-phase0-vcloud-prep.md` Step 3. + +**Numbering note:** several other agents are working the tooling gap +register in parallel this session, all uncommitted -- to avoid a +collision on the next-free DOCFIX number, this delivery uses the literal +placeholder `DOCFIX-162` everywhere a real number would go (this file, +the gap-register update, the session-ledger entry). The orchestrating +session assigns the real sequential number and find-replaces it at +integration/commit time. + +## Item + +### 1. DOCFIX-162 -- `scripts/dc-dc-mtu-geneve-budget.sh` +FILES: `scripts/dc-dc-mtu-geneve-budget.sh`, new +`tests/dc-dc-mtu-geneve-budget/run-tests.sh`. + +WHAT: turns D-101's ("Tenant/MTU sub-policy", folded in from D-102) +geneve-over-v6 MTU arithmetic into a runnable, read-only calculator. +Required `--underlay-mtu ` (no default -- the measured host L2 MTU +per Step 3's own `ip -o link show | awk '{print $2, $0}' | grep -i mtu` +check); optional `--jumbo-threshold` (default 9000, overridable for policy +exploration only). Computes: `>= 9000` -> jumbo, tenant MTU stays 1500; +otherwise `tenant_mtu = underlay_mtu - 56` (56 = 40 IPv6 outer header + 8 +UDP + 8 Geneve base, quoted verbatim from D-101), matching D-101's own +worked example exactly (1500 - 56 = 1444). Added one sanity floor NOT +itself part of D-101's text: if the computed tenant MTU falls below 1280 +(the IPv6 protocol's own minimum link MTU, RFC 8200 Sec 5), the verdict is +FAIL rather than a silent PASS -- clearly labeled in the script's header as +a separate, defensible addition, not D-101 arithmetic. Prints a +human-readable computation, a PASS/FAIL verdict, and a single +`MTU-BUDGET: ...` summary line for pasting into a changelog/as-executed +log, matching what Step 3 asks the operator to record. + +VERIFIED: +- New harness `tests/dc-dc-mtu-geneve-budget/run-tests.sh`: 19/19 PASS. + Covers the exact D-101 worked example (1500 -> 1444, both the + human-readable line and the summary line reproduce verbatim), jumbo + input (9000 and above) correctly leaves tenant MTU at 1500, the jumbo + threshold boundary (exactly-9000 vs 8999), missing `--underlay-mtu` + fails loud (rc 2, clear message), non-numeric/zero/negative + `--underlay-mtu` and bad `--jumbo-threshold` all fail loud (rc 2), the + IPv6-minimum-MTU sanity floor triggers FAIL (rc 1) at 1300 and 1280 + underlay, `--help` exits 0 and cites both D-101 and the 56-byte + overhead, and an unknown flag fails loud (rc 2). +- Manually re-ran the worked example, jumbo case, and floor case directly + (not just through the harness) to confirm output text before writing the + harness's regexes against it. +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged + D-001..018 non-ASCII carve-out; nothing this delivery touched). + +REVERT: `rm scripts/dc-dc-mtu-geneve-budget.sh && rm -rf +tests/dc-dc-mtu-geneve-budget` (safe -- no other script calls this one +yet; it is a standalone calculator, not sourced by anything). + +### 2. DOCFIX-162 -- `scripts/dc-dc-ceph-disk-budget.sh` +FILES: `scripts/dc-dc-ceph-disk-budget.sh`, new +`tests/dc-dc-ceph-disk-budget/run-tests.sh`. + +WHAT: turns buildout-design Section 3 / D-101's Ceph size=3-vs-size=2 +disk-budget arithmetic into a runnable, read-only calculator. Required +inputs, ALL with no default (fails loud if any is missing): `--total-disk +` (measured total disk, Step 2's `df`/`lsblk` output), `--dc1-nodes +` / `--dc1-per-node-osd ` and `--dc2-nodes ` / +`--dc2-per-node-osd ` (measured or firmly planned per-DC Ceph OSD +node count and per-node raw footprint -- already reflecting each +cluster's own size=3 replication as it would actually consume disk), and +`--backup-overhead-fraction ` (a decimal fraction of the +combined DC1+DC2 Ceph total representing radosgw backup pools + cross-DC +multisite duplication + Glance rbd-mirror targets + images + per-DC +mirrors -- quoted directly from Section 3's own itemization). This +fraction is NOT specified as a hard number anywhere in the repo (checked +`docs/design-decisions.md` and `docs/dc-dc-buildout-design.md` +directly) -- so it has NO default and the script refuses to run without +it explicitly passed; its header documents that it must be estimated from +real Ceph/radosgw-admin usage measurements once DC1's cluster exists. +Sizes accept plain byte integers or 1024-based `Ki/Mi/Gi/Ti`/`K/M/G/T` +suffixes (matching `df -h` binary units). + +Computes `required = (dc1_total + dc2_total) * (1 + overhead_fraction)` +against the measured total disk. If it fits: PASS, prints the margin. If +not: FAIL, prints the shortfall, and states -- but does NOT compute or +silently apply -- that size=2 is the ONLY documented fallback per D-101 / +Section 3, and that adopting it is a decision the OPERATOR must explicitly +log (its own decision note / changelog entry). Deliberately does NOT +invent a size=2 disk-requirement number: the real per-node footprint under +size=2 depends on a distinct measurement this repo does not have yet, and +fabricating a scaling factor would be exactly the unmeasured-arithmetic +this repo's discipline forbids. Prints a `CEPH-BUDGET: ...` summary line +for the same changelog/as-executed-log use as the MTU script. + +VERIFIED: +- New harness `tests/dc-dc-ceph-disk-budget/run-tests.sh`: 16/16 PASS. + Covers a constructed size=3-fits numeric example (3 nodes x 500G per DC, + 30% overhead, 10TiB measured total -> ~3.80TiB required, PASS with + ~6.19TiB margin, exact summary line asserted) and a constructed + size=3-does-not-fit example (same Ceph footprint against a 2TiB measured + total -> FAIL, exact shortfall asserted, size=2 fallback named, + explicit-logged-decision language asserted, never applied silently); + each of the six required arguments fails loud (rc 2) when missing, + specifically confirming `--backup-overhead-fraction` has no default and + blocks the run alone; bad numeric/size formats (non-integer node count, + non-decimal fraction, bad size suffix) all fail loud (rc 2); a + plain-byte total-disk value produces the same PASS verdict as its + 1024-based-suffix equivalent (10995116277760 bytes == 10Ti); `--help` + exits 0 and cites Section 3 and the no-default-fraction rule; an unknown + flag fails loud (rc 2). +- Manually re-ran both constructed examples directly to confirm output + text before writing the harness's regexes against it. +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged). + +REVERT: `rm scripts/dc-dc-ceph-disk-budget.sh && rm -rf +tests/dc-dc-ceph-disk-budget` (safe -- standalone, nothing else calls it +yet). + +## Next actionable step + +Tooling gap register item #7 CLOSED (mechanism). Both calculators are +read-only and take every varying input as a required, explicitly-measured +argument -- no invented defaults anywhere, matching hard rule 2. Neither +has been run against the real vcloud host yet (no live infrastructure this +session, prep-only per the operator's own instruction) -- that first real +run is part of executing `runbooks/dc-dc-phase0-vcloud-prep.md` Step 3 +tomorrow morning: measure the real interface MTU and feed it to +`dc-dc-mtu-geneve-budget.sh`, and once Step 2's `df`/`lsblk` numbers and a +real (or firmly planned) per-node OSD footprint exist, feed those to +`dc-dc-ceph-disk-budget.sh`. The `--backup-overhead-fraction` input +specifically remains a genuine open item until DC1's Ceph cluster exists +and reports real radosgw/rbd-mirror pool usage (Section 3's own point: +"the reserve split ... is set in Phase 0 from the measured DC0 per-node +OSD footprint") -- this delivery gives the arithmetic a home, not a real +number to plug into it yet. diff --git a/docs/archive/changelogs/changelog-20260709-netbox-dc-dc-pipeline.md b/docs/archive/changelogs/changelog-20260709-netbox-dc-dc-pipeline.md new file mode 100644 index 0000000..53a2d68 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-netbox-dc-dc-pipeline.md @@ -0,0 +1,110 @@ +# Changelog 2026-07-09 -- NetBox multi-DC/dual-stack import pipeline (DOCFIX-152) + +No live infrastructure touched -- new script + test harness only. Closes +tooling gap register item #3 in `docs/dc-dc-deployment-workflow.md`. + +## Item + +### 1. DOCFIX-152 -- `netbox/dc-dc-prefixes-import.py` +FILES: `netbox/dc-dc-prefixes-import.py` (new), `netbox/README.md` (new VR1 +section), `tests/dc-dc-prefixes-import/` (new: `run-tests.sh`, +`test_logic.py`, `fake_pynetbox.py`, `stub_syspath/pynetbox.py`). + +WHAT: extends the v1 single-site IPv4-only NetBox import pattern +(`netbox/ipv4-prefixes-import.py`) to VR1's two-DC, dual-stack model per +D-101. Per DC (`--dc dc1`/`--dc dc2`), creates the six-plane IPv4 structure +plus the D-101 family-matrix IPv6 legs (ULA for +metal-admin/metal-internal/data-tenant/storage/replication, GUA for +provider-public). + +**Nothing invented, consistent with this repo's hard rule:** D-101 names +three literals as NetBox-assigned, explicitly "NOT hardcoded in this +decision" -- the org ULA /48, the per-DC GUA carve out of the real ARIN +2602:f3e2::/32 block, and DC2's v4 supernet. All three are REQUIRED +environment variables (`ORG_ULA_48`, `DC_GUA_PREFIX`, and, for dc2 only, +`DC2_V4_SUPERNET`) with NO defaults -- the script validates format and +containment and FAILS LOUD if any is missing, malformed, out of range, or +(for DC2's supernet) overlapping DC1. DC1's six v4 planes ARE hardcoded, +deliberately: D-101 rules DC1 inherits DC0's layout UNCHANGED, so those +CIDRs are the ADOPTED decision text itself, not an inference. + +The exact per-plane subdivision of the ULA /48 and the GUA carve (which /64 +goes to which plane) is this script's OWN PROPOSED scheme, documented in the +module docstring as NOT itself an operator-ratified D-NNN sub-decision -- +flagged for review before trusting real NetBox writes, the same posture +D-102 had before its own ratification pass folded it into D-101. + +FOUND AND FIXED while writing the test harness: the first draft's example +GUA addresses (`2602:f3e2:0:100::/40` style) had host bits set beyond the +/40 boundary (the fourth hextet exceeds bit 40) -- Python's +`ipaddress.ip_network(..., strict=True)` correctly rejected them. Corrected +to `2602:f3e2:1000::/40` / `2602:f3e2:2000::/40` in both the script's own +docstring examples and the test data. + +Also found and fixed a real test-harness bug (not a script bug): the +end-to-end test counted the substring "CREATED" in captured stdout to +verify prefix-creation counts, but the script's own site-creation message +also contains the word "CREATED" (`'... -- CREATED.'` when a new NetBox +site is created), inflating the count by one on first run. Tightened the +assertion to count "CREATED prefix" / "EXISTS prefix" specifically. + +FLAGGED, not fixed (out of scope): `netbox/ipv4-prefixes-import.py`'s +existing IPAM role slugs (`provider`/`metal`/`lbaas-management`/ +`openstack-tenant`) predate the D-052/D-053 six-plane cutover that +`scripts/lib-net.sh` reflects. This new script uses NEW role slugs matching +the CURRENT six-plane names -- a real, pre-existing staleness in the v1 +script, called out in both the new script's docstring and `netbox/README.md` +rather than silently perpetuated or silently fixed mid-step in an unrelated +v1 artifact. + +FLAGGED as a genuine open call, not silently decided: DC1's import may +create prefixes numerically identical to the existing `vr0-dc0` NetBox +site's entries. NetBox permits duplicate global prefixes (soft warning, +not a hard block) but whether to retire the `vr0-dc0` entries once DC1 +supersedes that rehearsal environment, or keep both for historical record, +is an operator IPAM-hygiene decision -- the script prints a note and does +not decide it. + +VERIFIED, not just asserted: +- New harness `tests/dc-dc-prefixes-import/`: 37/37 PASS. Pure-logic tests + for every validator (`validate_ula_48`, `validate_gua_prefix`, + `validate_dc2_supernet`) and carve function (`carve_dc2_v4`, `carve_ula`, + `carve_gua`), plus a full fake-NetBox end-to-end run of `main()` for both + DCs covering: happy path, idempotency (re-run creates nothing, reports + all EXISTS), DC2's supernet is genuinely required (fails loud when unset), + and argparse's `--dc` requirement/choice validation. The fake NetBox + client (`fake_pynetbox.py`) implements only the specific pynetbox calls + this script makes -- not a general mock. +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN. Caught and + fixed a real CRLF regression during authoring: a Python-based find/replace + used during test-data cleanup wrote the file back in Windows text mode + (`\r\n`), which repo-lint's L1 check correctly flagged; fixed by + rewriting the file in binary mode with LF-only line endings. +- `bash scripts/run-tests-all.sh`: full gauntlet 23/45 FAILED -- the SAME 23 + pre-existing environment-gap failures already documented in + `docs/changelog-20260709-dc-selector-convention.md` (jq missing on this + workstation, or live juju/cloud-dependent suites); the new + `dc-dc-prefixes-import` suite itself is clean (37/37), confirming zero + regressions. + +UNVALIDATED regardless of the above: no live NetBox / `pynetbox`-reachable +instance exists this session (operator's workstation, not the +vopenstack-jesse jumphost) to run this end-to-end against a real server. +The fake-NetBox tests confirm the script's OWN logic is internally +consistent; they cannot confirm pynetbox's real API surface matches what +this script assumes (which mirrors the same assumption the existing v1 +scripts already make, uncontested, in this same directory). + +REVERT: `rm -f netbox/dc-dc-prefixes-import.py && rm -rf +tests/dc-dc-prefixes-import && git checkout HEAD~ -- netbox/README.md` +(safe -- no other script calls this one yet). + +## Next actionable step + +Tooling gap register #3 CLOSED (mechanism). Before a real run: the operator +must generate the org ULA /48 (RFC 4193 random allocation) and obtain the +per-DC GUA carve from whoever manages 2602:f3e2::/32 region-0 /36 -- neither +exists yet (this was already true; this script doesn't create that gap, it +surfaces it explicitly via fail-loud env-var requirements instead of a +silent default). Stage 2/3 runbooks (not yet authored) are the first real +consumers of this pipeline's output. diff --git a/docs/archive/changelogs/changelog-20260709-netem-ula-gua-proposal.md b/docs/archive/changelogs/changelog-20260709-netem-ula-gua-proposal.md new file mode 100644 index 0000000..caaae22 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-netem-ula-gua-proposal.md @@ -0,0 +1,52 @@ +# Changelog 2026-07-10 -- Proposed netem parameters + ULA/GUA generation guidance (DOCFIX-168) + +No live infrastructure touched -- documentation only, no repo mutation +beyond a new doc and gap-register/ledger pointers. Addresses tooling gap +register items #4(d) and #11. + +## Item + +### 1. DOCFIX-168 -- `docs/dc-dc-netem-and-ula-gua-proposal.md` +FILES: `docs/dc-dc-netem-and-ula-gua-proposal.md` (new), `docs/dc-dc- +deployment-workflow.md` (gap items #4(d) and #11 updated to reference it). + +WHAT: two independent parts, neither of which invents or decides a value +the operator/NetBox must own -- both make the NEXT step fast once the +operator is ready, per this repo's discipline of presenting options +rather than silently deciding anything not yet ADOPTED. + +**Part 1 -- proposed `tc netem` parameters (D-100 gap #4(d)):** a concrete +recommendation (1ms delay, 0.2ms jitter, 0.01% loss, uncapped rate), each +with reasoning grounded directly in the buildout design's own stated +"same-metro dark fiber, low single-digit ms" lean (not invented from +nothing) -- presented as a table for explicit operator ratification, with +instructions to record the ruling as a D-100 sub-item amendment in +`docs/design-decisions.md` (mirroring how every other Stage-0 redline item +was recorded) once accepted or adjusted. + +**Part 2 -- ULA/GUA/DC2-supernet generation guidance (D-101 gap #11 + gap +#3's data half):** explicitly does NOT generate the org ULA /48, obtain +the real per-DC GUA carve, or assign DC2's supernet -- gives the operator +the exact, safe command to generate a real RFC 4193-compliant ULA /48 +themselves (`openssl rand -hex 5`, confirmed via research this evening +that a cryptographically random 40 bits satisfies RFC 4193's statistical- +uniqueness requirement, matching what every independent ULA-generator +tool checked tonight actually does), and explains precisely why the GUA +carve and DC2 supernet are NOT generation tasks at all but real IPAM/ARIN +coordination requiring visibility and authority this session does not +have. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. + +REVERT: `rm docs/dc-dc-netem-and-ula-gua-proposal.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe -- no infrastructure +touched, no value generated or applied). + +## Next actionable step + +Both parts wait on the operator: Part 1 needs an explicit accept/adjust +ruling (recordable as a D-100 amendment in one sitting); Part 2's ULA +command takes two minutes to run once the operator is ready, after which +`netbox/dc-dc-prefixes-import.py --dc dc1` becomes runnable for real (only +`ORG_ULA_48`/`DC_GUA_PREFIX` are needed for dc1; `DC2_V4_SUPERNET` +additionally for dc2). diff --git a/docs/archive/changelogs/changelog-20260709-office1-network-edge.md b/docs/archive/changelogs/changelog-20260709-office1-network-edge.md new file mode 100644 index 0000000..663c1da --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-office1-network-edge.md @@ -0,0 +1,134 @@ +# Changelog 2026-07-09 -- Office1-local network + Office1 OPNsense edge ownership (DOCFIX-163) + +No live infrastructure touched -- OpenTofu module/wiring additions, runbook +updates, and documentation only (prep-only session; no `tofu` binary +available, same UNVALIDATED status as every other OpenTofu module in this +repo). Closes tooling gap register items #12 and #16 in +`docs/dc-dc-deployment-workflow.md`; surfaces a new item, #17, found while +closing #16. + +## Items + +### 1. DOCFIX-163 -- `opentofu/modules/office1-network` (closes gap #12) + +FILES: `opentofu/modules/office1-network/main.tf`, +`opentofu/modules/office1-network/variables.tf`, +`opentofu/modules/office1-network/outputs.tf`, `opentofu/main.tf` (new +`module "office1_network"` block, instantiated for real), `opentofu/README.md`. + +WHAT: a new module, shaped exactly like `modules/dc-planes` (isolated +`libvirt_network`, no `forward`/DHCP block, attribute-style nested-object +syntax per this provider's confirmed schema convention) but sized for ONE +network instead of six planes -- Office1 is a headend, not a per-DC +OpenStack cloud, so the six-plane D-052/D-100 template does not apply to it. +This is the shared L2 segment the three Office1 service VMs (MAAS-region, +NetBox, GitBucket), the vcloud host's own management interface, and Office1's +own OPNsense edge's LAN side all attach to -- previously modeled nowhere in +`opentofu/` (gap #12, found 2026-07-09 while drafting +`runbooks/dc-dc-phase1-office1-standup.md`). + +DESIGN DECISION: a NEW module, not a reused host bridge (the runbook's own +"Open questions" section had left this as a genuine, undecided fork). +Reasoning, in full in the module's own header comment and in +`runbooks/dc-dc-phase1-office1-standup.md`'s Open question #1: +- D-103 already establishes that OpenTofu owns create/destroy of every + virtual network in this topology. A hand-configured host bridge would be + the one network living outside that inventory -- exactly the + not-reproducible, not-versioned debt D-103 exists to eliminate, and the + same class of debt this repo already logs explicitly (as a DOCFIX + candidate) whenever Option B (manual VM creation) is used elsewhere in the + same runbook. +- This repo's own CHECK steps (`virsh net-list --all`, used throughout the + Stage 1/2 runbooks) implicitly assume that command is a COMPLETE inventory + of what exists on vcloud. A pre-existing, OpenTofu-invisible bridge would + silently break that assumption, and Stage 0's teardown path (D-061) would + never reach it. +Instantiated for REAL in root `main.tf` (`module "office1_network"`), unlike +most of the other still-scaffolded modules in this repo -- it needs no +unmeasured value beyond `domain_suffix`/`underlay_mtu`, both already real, +already-required inputs elsewhere in the same file (the identical reasoning +that already lets the three `mesh-link` legs be instantiated today). + +VERIFIED: +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged). +- **UNVALIDATED against a real provider** -- no `tofu` binary was available + this session (same status as every other module in `opentofu/`; see + `opentofu/README.md`'s top-level status line). Run + `bash scripts/opentofu-validate.sh` on a connected machine before trusting + the schema. + +REVERT: `git checkout HEAD~ -- opentofu/main.tf opentofu/README.md && rm -rf +opentofu/modules/office1-network` (safe -- nothing else references this +module yet). + +### 2. DOCFIX-163 -- Office1 OPNsense edge ownership decision (closes gap #16) + +FILES: `opentofu/main.tf` (new, commented-out `module "office1_opnsense"` +skeleton), `opentofu/README.md`, +`runbooks/dc-dc-phase1-office1-standup.md` (new Step 4b, updated Open +questions/Sequence/GATE/delivery-checklist sections). + +WHAT: the buildout design's topology (Section 1) is explicit -- "full +dark-fiber triangle... plus a **per-site** OPNsense simulated ISP edge," +three sites, three edges. Stage 3's Build list already covered DC1/DC2's +edges (`modules/opnsense-edge` per DC); neither Stage 2's nor Stage 3's +runbook instantiated one for Office1 (gap #16, found 2026-07-09 during final +consolidation of the DC-DC runbook set). + +DECISION: **YES, Office1 gets its own `modules/opnsense-edge` call, a fourth +site alongside DC1/DC2 -- and Stage 2's own runbook +(`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it**, via a new +Step 4b (CHECK-before-MUTATION shape, its own GATE bullet, matching the +runbook's existing style exactly). Rationale: Stage 2's entire scope is +already "Office1 headend standup" -- the OPNsense edge is part of that +headend's own boundary, not DC substrate -- and Stage 1 already creates the +mesh-link networks this edge's neighbors depend on; Stage 3 has no reason to +reach back into Office1's own infrastructure to build a piece of it. + +The new `module "office1_opnsense"` skeleton in `main.tf` mirrors Stage 3's +`dc1_opnsense` template shape exactly (same module, same variable names). +Its `lan_network_name` resolves cleanly to Item 1's new network +(`module.office1_network.network_name`). Its `wan_network_name` does NOT -- +see Item 3 below. The block stays commented out, same as DC1/DC2's own +not-yet-instantiated edges (neither is real either, per +`opentofu/README.md`), pending real `memory_mib`/`vcpu`/`disk_size_bytes`/ +`base_volume_path`/`config_iso_path` values plus gap #17's resolution. + +VERIFIED: `bash scripts/repo-lint.sh`: 0 fail (module block is commented +out; nothing new to `tofu validate` here that Item 1 didn't already cover). + +REVERT: `git checkout HEAD~ -- opentofu/main.tf opentofu/README.md +runbooks/dc-dc-phase1-office1-standup.md` (safe -- Step 4b's module call was +never uncommented/applied). + +### 3. Gap register item #17 (NEW) -- no per-site ISP-uplink/WAN network exists yet, for any site + +Investigated per this task's own instruction to check before inventing a +WAN-side network for Office1's edge: read +`runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 5 directly. Its own +`dc1_opnsense` template carries an IDENTICAL, still-unresolved placeholder: +``` +wan_network_name = module.mesh_dc1_office1... # or a dedicated per-site ISP-uplink segment -- confirm which, do not assume +``` +This confirms the WAN-side network question is NOT Office1-specific -- it is +a genuinely deeper, cross-site gap that also blocks DC1's and DC2's own +edges, not something this delivery could resolve alone without inventing a +value or (worse) defaulting Office1's WAN side to a mesh-link network, which +would directly contradict D-100's own sub-item ruling that the three +dark-fiber mesh legs carry MANAGEMENT TRAFFIC ONLY (Office1<->DC = MAAS/ +Juju/operator), not a simulated ISP uplink. Logged as new tooling gap +register item #17 in `docs/dc-dc-deployment-workflow.md`, cross-referenced +from gaps #4/#12/#16 and from both new module headers -- flagged rather than +invented, per this task's own instruction. + +## Next actionable step + +Tooling gap register items #12 and #16 CLOSED. Gap #17 (NEW) is now the +blocking dependency for instantiating ANY of the three sites' +`opnsense-edge` calls end-to-end -- needs a decision on the WAN-side network +shape (most likely a fourth per-site `libvirt_network`, analogous to +`office1-network`/`mesh-link`/`dc-planes`) before `dc1_opnsense`, +`dc2_opnsense`, or `office1_opnsense` can be uncommented for real. Designing +the three Office1 service VMs' actual `cloudinit-vm` `network_config` +content against the now-real `office1-local` network remains separate, +un-actioned work (unchanged by this delivery). diff --git a/docs/archive/changelogs/changelog-20260709-opentofu-audit.md b/docs/archive/changelogs/changelog-20260709-opentofu-audit.md new file mode 100644 index 0000000..88f5e63 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opentofu-audit.md @@ -0,0 +1,106 @@ +# Changelog 2026-07-09 -- OpenTofu module audit pass (DOCFIX-148) + +No live infrastructure touched -- documentation/comment corrections only, +plus confirmatory research. Operator asked to pull documentation for +anything drafted without it and review all 9 modules built this session for +errors/assumptions, after DOCFIX-142/144/145/146/147 had already shipped. + +## Item + +### 1. DOCFIX-148 -- audited every flagged UNVERIFIED/inferred/assumed note +FILES: `opentofu/modules/opnsense-edge/{main,variables}.tf`, +`opentofu/modules/node-vm/main.tf`, `opentofu/modules/maas-vm-host/variables.tf`, +`scripts/opnsense-build-config-iso.sh`, `opentofu/README.md`. + +METHOD: grepped the entire `opentofu/` tree for every "UNVERIFIED"/"NOT +confirmed"/"inference"/"assumed"/"not independently" marker left by prior +deliveries, then researched each one further (libvirt's own official XML +docs, a second independent source on the provider's volume-fetch mechanism, +FreeBSD/OPNsense interface-naming documentation, `genisoimage` reference +docs, MAAS provider zone/pool behavior) rather than re-stating the same +hedge unchanged. + +**One genuine error found and fixed**, not just a hedge re-confirmed: +`opnsense-edge`'s comments claimed `devices.interfaces` list order +("index 0 = LAN") directly determines OPNsense's LAN/WAN role. Researching +FreeBSD/OPNsense's interface-assignment model showed this conflates two +separate things: list order plausibly controls which `vtnetN` device number +a NIC gets (libvirt's own PCI-addressing docs: auto-assigned addresses +"usually match" XML order for a simple topology), but OPNsense's actual +LAN/WAN designation is a SEPARATE, explicit mapping set inside `config.xml` +itself (or the interactive/API assignment step) -- "`vtnet0` = WAN" is a +convention some guides choose, not an enforced rule. `main.tf`'s comments +and both `lan_network_name`/`wan_network_name` variable descriptions +rewritten to state this correctly: the variables express intent for which +network should end up as which role, but whoever writes the still-undesigned +`config.xml` must independently confirm the real vtnetN<->network mapping on +a boot and set ``/`` accordingly. No `config.xml` has been written +yet in this repo, so this corrects documentation accuracy for a future +author, not current runtime behavior. + +**Confidence upgraded (not resolved outright) on three other items:** +- `node-vm`'s `boot = { order = N }` shape: confirmed the provider + code-generates its schema as a 1:1 mirror of libvirt's own domain XML + (an architectural fact not visible in the resource doc pages alone), and + confirmed libvirt's own docs show the native element is exactly + `` -- consistent with the established single-attribute + mapping pattern already seen elsewhere in this schema. Still recommend a + real `tofu providers schema -json` check before trusting it, but this is + reasoned support now, not a bare guess. +- `create.content.url` accepting local filesystem paths: corroborated by a + second, independent source describing the underlying fetch mechanism as + supporting local paths alongside remote URLs. +- `genisoimage`/`xorriso` invocation flags (`-V`/`-J`/`-R`): confirmed as + standard, correct usage for a plain cross-platform ISO9660 image -- + separated cleanly from the genuinely still-open question (whether + OPNsense's Importer actually reads the result on a real boot), which + remains unverified. + +**Confirmed safe, previously assumed:** `maas_vm_host`'s `zone`/`pool` being +computed by MAAS when left unset -- validates the `default = null` design +choice in `modules/maas-vm-host` as a supported no-op. + +**Deliberately NOT re-researched:** findings already backed by a primary, +authoritative source on the first pass (the OPNsense Configuration Importer +mechanism itself, the `maas_vm_host` vs. `maas_vm_host_machine` distinction, +the provider's attribute-vs-block syntax fix) -- re-verifying already-solid +findings on every audit would be its own kind of waste; the audit targeted +specifically the notes marked as gaps. + +`opentofu/README.md` gained a new "Audit pass (2026-07-09)" section +documenting all of the above with reasoning, positioned as its own dated +entry alongside the existing OPNsense and MAAS/netem research sections +rather than silently editing history. + +### 2. Documentation references added to the `openstack-cloud-ops` skill +FILE (new): `.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md`. +FILES (modified): `.claude/skills/openstack-cloud-ops/SKILL.md` (new routing +entry), `opentofu/README.md` (cross-reference back to the skill file). + +WHAT: operator asked for documentation links/access to be added to the skill +so future sessions can reference the direct sources rather than +rediscovering them. The new file indexes every provider/doc source used +building `opentofu/` (dmacvicar/libvirt, canonical/maas, OpenTofu's own +`local-exec`/`terraform_data` docs, OPNsense's docs/forum/GitHub issues) with +their exact URLs, confirmed versions, and default branches -- plus, more +valuably than the links alone, the FETCH METHODOLOGY that actually worked +this session: the Registry's own doc pages are JS-rendered and return empty +shells on a plain fetch (go to the provider's GitHub repo instead); GitHub +raw-file branch names vary and should be checked via the API's +`default_branch` field, not assumed; a general web search's own summary can +itself be wrong (encountered once); asking for "bare field names only" gets +past copyright-caution refusals that "reproduce this page" trips; real +example `.tf` files are consistently more reliable than doc-summarized prose +for syntax questions (the root cause of the DOCFIX-142 syntax bug this +whole delivery arc has referenced). + +Deliberately does NOT duplicate the per-module findings themselves +(schema notes, what's confirmed vs. assumed per attribute) -- that stays in +`opentofu/README.md` as the single source of truth, cross-referenced from +both directions, so it can't drift into two disagreeing copies. + +REVERT: `git rm .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md`; +revert the two cross-reference edits if reverting fully. + +REVERT: `git checkout HEAD~ -- opentofu/modules/opnsense-edge opentofu/modules/node-vm/main.tf opentofu/modules/maas-vm-host/variables.tf scripts/opnsense-build-config-iso.sh opentofu/README.md` +(re-arms the LAN/WAN documentation error -- not recommended). diff --git a/docs/archive/changelogs/changelog-20260709-opentofu-cloudinit-vm.md b/docs/archive/changelogs/changelog-20260709-opentofu-cloudinit-vm.md new file mode 100644 index 0000000..0876602 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opentofu-cloudinit-vm.md @@ -0,0 +1,67 @@ +# Changelog 2026-07-09 -- OpenTofu cloud-init/base-image VM modules (DOCFIX-145) + +No live infrastructure touched -- unapplied IaC source only. Continues the +DOCFIX-142/144 OpenTofu scaffold. + +## Item + +### 1. DOCFIX-145 -- `opentofu/modules/base-image` + `opentofu/modules/cloudinit-vm` +FILES (new): `opentofu/modules/base-image/{main,variables,outputs}.tf`, +`opentofu/modules/cloudinit-vm/{main,variables,outputs}.tf`. + +WHAT: the second VM-creation pattern this repo needs, for Office1's own +service VMs (MAAS-region, NetBox, GitBucket -- D-103), genuinely distinct +from `modules/node-vm`'s blank-disk PXE pattern (that one is for MAAS-managed +OpenStack nodes MAAS itself images after enlistment; these are self-managed +VMs that need a real bootable OS + cloud-init). Split into two modules +matching how the provider's own real example separates concerns: +`base-image` downloads one shared cloud image once (`libvirt_volume` with +`create.content.url`); `cloudinit-vm` creates a per-instance copy-on-write +overlay disk (`backing_store`), a `libvirt_cloudinit_disk` NoCloud seed, and +the domain referencing both. + +VERIFICATION: every resource/attribute shape used is taken directly from +`examples/alpine_cloudinit.tf` (fetched and read as real, runnable code last +turn while building the node-VM module) -- the same high-confidence source +that surfaced the syntax-bug fix in DOCFIX-144. One design choice improved +during authoring: the base image's `.path` is threaded across the module +boundary as a real output/input (`module..path` -> +`cloudinit-vm`'s `base_volume_path` variable) rather than reconstructed from +a pool name + volume name string, avoiding an unconfirmed path-convention +guess entirely. + +DELIBERATELY NOT DONE: +- **`user_data`/`meta_data`/`network_config` have no defaults and no generic + fallback content.** What Office1's three service VMs actually need + installed/configured is real design work that hasn't happened; a + plausible-looking default (e.g. the common `dhcp4: true` netplan stanza) + would also silently fail here regardless, since this repo's plane networks + carry no libvirt-managed DHCP (NetBox is the IPAM apex). +- **OPNsense applicability is explicitly NOT confirmed.** This whole + cloud-init/NoCloud mechanism is standard for Linux cloud images; OPNsense + is FreeBSD-based, and its support for the same mechanism was not checked + this session. Flagged prominently in both modules' headers and + `opentofu/README.md` -- do not assume `cloudinit-vm` covers OPNsense + without verifying that specifically first. +- **Neither module instantiated in root `main.tf`.** No image source has + been chosen for any Office1 service VM. + +Workflow doc (`docs/dc-dc-deployment-workflow.md`) Stage 2 authoring status, +gap register items 2 and 4, and the status summary table updated to match; +companion visual tracker (same Artifact URL) redeployed. + +HARNESS: none added -- pure `.tf` source, same as the prior OpenTofu +deliveries; `scripts/opentofu-validate.sh` covers all of `opentofu/` +generically once a `tofu` binary is available to run it. + +REVERT: `git rm -r opentofu/modules/base-image opentofu/modules/cloudinit-vm`; +revert the workflow-doc sections listed above if reverting fully. + +## Next actionable step (unchanged, still logged not actioned) + +Same as DOCFIX-144: on a machine with the `tofu` binary and registry network +access, run `bash scripts/opentofu-validate.sh` against all of `opentofu/`, +then `tofu providers schema -json` to confirm the remaining unverified +details (the `boot`-order shape in `node-vm`; whether `libvirt_cloudinit_disk` ++ this whole pattern actually produces a bootable OPNsense image, which needs +a real test boot, not just a schema check). diff --git a/docs/archive/changelogs/changelog-20260709-opentofu-maas-netem.md b/docs/archive/changelogs/changelog-20260709-opentofu-maas-netem.md new file mode 100644 index 0000000..8180ab6 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opentofu-maas-netem.md @@ -0,0 +1,84 @@ +# Changelog 2026-07-09 -- MAAS VM-host registration + tc netem modules (DOCFIX-147) + +No live infrastructure touched -- unapplied IaC only. Researched against +official sources before drafting (operator instruction this turn), not +inferred from memory -- continues the DOCFIX-142/144/145/146 OpenTofu build. + +## Items + +### 1. DOCFIX-147 -- `opentofu/modules/maas-vm-host` +FILES (new): `opentofu/modules/maas-vm-host/{main,variables,outputs}.tf`; +root `versions.tf`/`variables.tf`/`main.tf` gained the `canonical/maas` +provider (v2.7.2, confirmed from the registry's own metadata, not guessed) +plus `maas_api_url`/`maas_api_key` variables (`maas_api_key` marked +`sensitive = true`). + +WHAT: registers the vcloud host's virsh/libvirt connection with MAAS as a +VM host (D-103: "OpenTofu registers each DC's libvirt host to that DC's MAAS +rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created +node VMs"). + +VERIFICATION, the point of this delivery: read both `maas_vm_host` and +`maas_vm_host_machine`'s real schemas (fetched from +`canonical/terraform-provider-maas`'s own `docs/resources/*.md`) before +picking one. `maas_vm_host_machine` takes `cores`/`memory`/`storage_disks`/ +`network_interfaces` as INPUT arguments -- MAAS's own "Compose machine" pod +feature, where you specify desired specs and MAAS creates a new VM itself +via the virsh connection. D-103 explicitly rules this out ("MAAS ... does +NOT compose new ones"). `maas_vm_host` (register the chassis; MAAS's own +discovery then enlists whatever domains already exist) is the one that +matches D-103's actual described flow, where `modules/node-vm` already +pre-creates the VMs. Using `vm_host_machine` here would have MAAS and +OpenTofu fighting over VM creation -- a real design mistake this research +step caught before any code was written, not after. + +Not instantiated: needs a real MAAS zone/pool and the vcloud host's real +`power_address` (confirmed format `qemu+ssh://user@host/system` from the +provider's own docs example -- same shape as this module's existing +`libvirt_uri` variable, kept as an independent input rather than assumed +identical). + +### 2. DOCFIX-147 (cont.) -- `opentofu/modules/netem-link` +FILES (new): `opentofu/modules/netem-link/{main,variables,outputs}.tf`. + +WHAT: applies `tc qdisc ... netem` WAN-simulation parameters (D-100) to a +`modules/mesh-link` bridge -- the mechanism flagged as missing in every +prior OpenTofu changelog this session ("needs either a null_resource +provisioner or a small companion script -- not yet written"). + +VERIFICATION: fetched OpenTofu's own official docs for `local-exec` and +`terraform_data` before drafting. Two findings that materially shaped the +design: (a) OpenTofu's own docs recommend `terraform_data` over the older +`null_resource` for exactly this "provisioner with no logical resource to +attach to" case -- used here, not `null_resource`. (b) `local-exec` runs on +the machine invoking `tofu apply` (Office1, per D-103), NOT on the vcloud +host where the bridge interfaces actually exist -- a bare local command +would silently do nothing useful. The module wraps the command in an +explicit SSH hop to the vcloud host instead. `tc qdisc replace` (not `add`) +is used so a re-apply is idempotent. A destroy-time provisioner +(`when = destroy`) removes the qdisc on teardown. + +Not instantiated: needs the real bridge name (reading a `libvirt_network`'s +`bridge` attribute back was already flagged unconfirmed in `mesh-link`'s +outputs.tf, so this is a required, explicit input, not auto-derived), the +vcloud host's SSH target, and the actual netem parameters -- still an +unruled D-100 open item. + +`opentofu/README.md` gained a new "MAAS registration + tc netem research" +section (mirroring the OPNsense research section's structure) with full +sourcing, and updated schema notes for `maas_vm_host`, the `maas` provider +block, and `terraform_data`. Workflow doc (Stage 3/4 authoring notes, gap +register items 2/4, summary table) and the companion visual tracker updated +to match. + +REVERT: `git rm -r opentofu/modules/maas-vm-host opentofu/modules/netem-link`; +revert `opentofu/versions.tf`/`variables.tf`/`main.tf`'s maas-provider +additions and the doc sections if reverting fully. + +## Next actionable step + +Same underlying need as every prior OpenTofu delivery this session: a +machine with the `tofu` binary to run `scripts/opentofu-validate.sh` against +all of `opentofu/` -- this would catch any remaining schema mistakes at +once, the same way it would have caught the DOCFIX-142 syntax bug +immediately had it been available then. diff --git a/docs/archive/changelogs/changelog-20260709-opentofu-node-vm.md b/docs/archive/changelogs/changelog-20260709-opentofu-node-vm.md new file mode 100644 index 0000000..bd01b15 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opentofu-node-vm.md @@ -0,0 +1,81 @@ +# Changelog 2026-07-09 -- OpenTofu node-VM module + syntax-bug fix (DOCFIX-144) + +No live infrastructure touched -- unapplied IaC source only. Continues the +DOCFIX-142 OpenTofu scaffold. + +## Items + +### 1. DOCFIX-144 -- `opentofu/modules/node-vm` (D-103's PXE-boot node-VM pattern) +FILES (new): `opentofu/modules/node-vm/variables.tf`, `main.tf`, `outputs.tf`. + +WHAT: one MAAS-managed OpenStack node VM per call -- a blank `libvirt_volume` +boot disk (no image, no cloud-init) plus a `libvirt_domain` with PXE boot +priority on its first network interface. This is the D-103 "OpenTofu creates +the node-VM libvirt domains (SHIM)" need for buildout-design Stage 3 / +Phase 2. Deliberately NOT the cloud-init/pre-built-image pattern (confirmed +as a separate, distinct provider mechanism via a real example file -- see +item 2 -- but not built as its own module this session; needed for Office1 +service VMs and OPNsense instead). + +VERIFICATION: this time, built from the provider's own real, runnable example +files (`examples/domain_with_network.tf`, `examples/alpine_cloudinit.tf` -- +fetched and read directly, not summarized-and-guessed) rather than +doc-summarization passes alone, after the previous session's attempt at +`libvirt_domain` was deferred for exactly that reason. Every attribute in +`main.tf` is used in the same shape as a real example, EXCEPT one, flagged +explicitly in the file and in `opentofu/README.md`: the per-device `boot` +attribute's internal shape (`{ order = N }`) is an inference by pattern +(confirmed to exist as a field name on both `interfaces[]` and `disks[]`, +matching native libvirt's per-device ``, but not +independently confirmed from a real example this session). Flagged as the +first thing to check with `tofu providers schema -json` before relying on +PXE-first boot working as written. + +NOT instantiated in root `main.tf`: node count, memory, vcpu, and disk sizing +are a Phase-0 host/disk-budget decision (buildout-design Section 3) that +hasn't been made. Call the module once real values exist -- no placeholder +specs invented to wire it in sooner. + +### 2. DOCFIX-144 (cont.) -- real syntax bug found and fixed in the DOCFIX-142 delivery +FILES: `opentofu/modules/dc-planes/main.tf`, `opentofu/modules/mesh-link/main.tf`, +`opentofu/modules/dc-storage-pool/main.tf`. + +WHAT: all three originally used classic HCL block syntax for nested +structures (`domain { name = ... }`, `mtu { size = ... }`, `target { path = +... }`). Building the node-VM module required fetching real example `.tf` +files from the provider's repo, which showed those examples consistently use +attribute-style nested OBJECTS instead (`os = { ... }`, `devices = { ... }`, +`target = { format = { ... } }`) -- a provider-wide schema convention (this +provider's current major version appears to use a terraform-plugin-framework +object-typed-attribute schema mirroring raw libvirt XML, not classic blocks). +All three files corrected to match; each carries a syntax-note comment +explaining the change and its verification source. + +WHY THIS MATTERS: this was a real correctness bug in already-committed, +already-pushed code (DOCFIX-142, commit 14d7382) -- not caught by +`repo-lint` (it checks encoding/staleness, not HCL semantics) and not +catchable without either a real `tofu validate` pass (unavailable this +session, still) or, as it turned out, real example code from the provider +itself. Logged plainly rather than glossed over: the doc-summarization +verification process used for DOCFIX-142 was insufficient on its own: it +correctly identified WHICH fields exist, but could not reliably convey HOW +they're called (block vs. attribute syntax). Real example code closed that +gap; doc-summarization alone did not. + +DOCUMENTED (not fixed, same class of remaining risk): `opentofu/README.md`'s +"Schema notes" section now states this attribute-object convention as a +provider-wide assumption to apply to ANY nested field not personally +confirmed otherwise -- since the reasoning that surfaced this bug (a handful +of real examples, not an exhaustive schema dump) could plausibly have missed +a resource-specific exception. + +## Next actionable step (unchanged, still logged not actioned) + +On a machine with the `tofu` binary and registry network access: run +`bash scripts/opentofu-validate.sh` against all of `opentofu/` (this would +have caught the DOCFIX-142 syntax bug immediately), then +`tofu providers schema -json` to confirm the `boot`-order attribute shape +before trusting PXE-first boot in `modules/node-vm`. + +REVERT: `git checkout HEAD~ -- opentofu/` (drops the node-vm module and +reverts the three modules to their pre-fix, WRONG syntax -- not recommended). diff --git a/docs/archive/changelogs/changelog-20260709-opentofu-scaffold.md b/docs/archive/changelogs/changelog-20260709-opentofu-scaffold.md new file mode 100644 index 0000000..7f550b9 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opentofu-scaffold.md @@ -0,0 +1,86 @@ +# Changelog 2026-07-09 -- OpenTofu network/pool scaffold (DOCFIX-142) + +First OpenTofu content in the repo, addressing the largest item in the +tooling gap register (`docs/dc-dc-deployment-workflow.md` gap #2). Executed +under blanket approval; no live infrastructure touched -- this is unapplied +IaC source only, and no live cloud/vcloud-host mutation occurred or was +attempted. Ratification note: D-100..D-110 (which this code encodes the +mechanics of) remain ALL PROPOSED (Stage 0 of the workflow doc); nothing here +bakes in a specific unratified value as fact -- CIDRs/MTU/paths are either +required inputs with no default, or DC1's already-inherited-from-DC0 values +(D-101), never an invented DC2 or netem specific. + +## Items + +### 1. DOCFIX-142 -- `opentofu/` network + storage-pool scaffold +FILES (new): `opentofu/versions.tf`, `opentofu/variables.tf`, `opentofu/main.tf`, +`opentofu/README.md`, `opentofu/modules/dc-planes/*`, +`opentofu/modules/mesh-link/*`, `opentofu/modules/dc-storage-pool/*`. + +WHAT: the six per-DC plane segments (D-052/D-100) and the D-100 dark-fiber +mesh triangle (DC1<->DC2, DC1<->Office1, DC2<->Office1), each as an isolated +`libvirt_network` resource (no `forward`/`ips`/`dhcp` block -- NetBox is the +IPAM apex per skill hard rule 3, not libvirt DHCP); plus a directory-backed +`libvirt_pool` per DC/Office1 for future node-VM disks. Root module wires DC1 +(using its DC0-inherited CIDRs, D-101) and Office1; DC2 is written but +commented out pending its NetBox CIDR assignment (D-101 open sub-item). + +WHY: `opentofu/` had zero files before this -- the single largest item in the +2026-07-09 tooling gap register, and a hard blocker for buildout-design +Stage 1-2 (vcloud host prep, Office1 headend, per-DC substrate creation). + +SCHEMA VERIFICATION (read `opentofu/README.md` for the full account): the +`dmacvicar/libvirt` provider's current schema (confirmed against its actual +`docs/resources/*.md`, tag v0.9.8, fetched this session -- NOT taken from +training-data memory of older provider versions, which materially differs) +turned out to nest network isolation under `forward.mode` rather than a +top-level `mode` argument, and to have restructured `libvirt_domain` into a +~40-argument schema mirroring raw libvirt XML. The `forward.mode` accepted +value enum could not be confirmed from the reachable docs this session; this +scaffold sidesteps the ambiguity by omitting the `forward` block entirely +(documented native-libvirt isolated-network behavior) rather than guessing a +value. Flagged prominently in `opentofu/README.md` as the first thing to +re-check if `tofu validate` objects. + +DELIBERATELY NOT BUILT, and why (see `opentofu/README.md` "Scope of this +delivery" for the full list): `libvirt_domain` (node VMs) and `libvirt_volume` +(disk images) -- the domain schema's nested `devices` block (disks, +interfaces, filesystems, graphics, etc.) could not be verified with enough +confidence this session to author safely; DC2's plane networks -- CIDRs not +yet assigned; `tc netem` WAN-simulation application -- an OS-level `tc qdisc` +setting with no equivalent `libvirt_network` argument, needing a provisioner +or companion script not yet written, and whose parameters are themselves a +Phase-0 open item (D-100); OPNsense/MAAS-region/NetBox/GitBucket service VMs +-- all depend on the deferred domain module. + +HARNESS: `scripts/opentofu-validate.sh` (fmt-check + init + validate wrapper, +BASH_SOURCE-self-locating, no hardcoded repo name) + `tests/opentofu-validate/` +(2/2 green -- only the guard-clause paths this environment can exercise +without a `tofu` binary; the fmt/init/validate path itself is UNTESTED here, +logged not hidden). Also extended `scripts/repo_lint.py`'s L1 (ASCII/LF) file +extension set to include `.tf` (it did not cover Terraform/OpenTofu files at +all before this, since none existed) -- verified this introduces no new +findings (the new `.tf` files are plain ASCII/LF) and the 34/34 repo-lint +harness still passes. + +REVERT: `git rm -r opentofu/ scripts/opentofu-validate.sh tests/opentofu-validate/`; +revert the one-line `exts` change in `scripts/repo_lint.py` and its harness +counterpart if reverting fully (not recommended -- `.tf` coverage is a real +gap otherwise, independent of whether this specific scaffold survives). + +### 2. Workflow doc + visual tracker updated to match +FILES: `docs/dc-dc-deployment-workflow.md`, the companion Artifact (same URL, +redeployed) -- Stage 1/3 authoring-status notes, gap register item 2, and the +status summary table all updated to reflect the scaffold's existence and its +UNVALIDATED state, per the "update this as stages close" standing instruction +on that doc. + +REVERT: `git checkout HEAD~ -- docs/dc-dc-deployment-workflow.md`. + +## Next actionable step (logged, not actioned) + +On a machine with the `tofu` binary and registry network access: run +`bash scripts/opentofu-validate.sh`, then `tofu providers schema -json` against +the initialized provider, and read the actual `libvirt_domain`/`libvirt_volume` +schema directly before authoring the node-VM module -- do not proceed from +memory or from a doc-summarization pass a second time. diff --git a/docs/archive/changelogs/changelog-20260709-opnsense-config-design.md b/docs/archive/changelogs/changelog-20260709-opnsense-config-design.md new file mode 100644 index 0000000..ca15128 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opnsense-config-design.md @@ -0,0 +1,80 @@ +# Changelog 2026-07-09 -- OPNsense config.xml design (DOCFIX-149) + +No live infrastructure touched. Real design work, not a placeholder -- +researched OPNsense's own real, currently-shipped config.xml before +drafting, per this session's established discipline. + +## Item + +### 1. DOCFIX-149 -- `opentofu/templates/opnsense-config.xml.tmpl` + renderer +FILES (new): `opentofu/templates/opnsense-config.xml.tmpl`, +`opentofu/templates/README.md`, `scripts/opnsense-render-config.sh`, +`tests/opnsense-render-config/run-tests.sh`. + +WHAT: a `{{TOKEN}}`-parameterized OPNsense `config.xml` template (this +repo's existing clientdocs token convention, reused rather than inventing a +new one), covering system/interfaces/gateway/staticroutes/DNS/NTP and +D-107-shaped firewall rules (default-deny WAN egress except NTP + the +per-DC mirror's upstream sync -- the two exceptions D-107 names). Paired +with `scripts/opnsense-render-config.sh`, which substitutes tokens from +required environment variables (no invented defaults except the NTP pool, +which defaults to the exact same public pool OPNsense's own sample ships -- +a real, confirmed default, not an invention) and feeds the result to the +already-existing `scripts/opnsense-build-config-iso.sh` -> ISO9660 -> +`modules/opnsense-edge` pipeline. + +VERIFICATION: built directly from OPNsense's own real `config.xml.sample` +(`opnsense/core`'s repo, `src/etc/config.xml.sample`) fetched and read as +real content, not summarized -- the same discipline that caught the +DOCFIX-142 syntax bug and the DOCFIX-148 LAN/WAN documentation error. This +fetch also DIRECTLY CONFIRMED the DOCFIX-148 audit finding: the real sample +ships literal placeholder device names (`mismatch0`/`mismatch1`) inside each +interface's own `` element, proving LAN/WAN role assignment is exactly +the per-block explicit mapping DOCFIX-148 concluded it was, now with a real +example rather than research-by-inference. The `staticroutes` schema was +confirmed from `opnsense/core`'s own `Route.xml` MVC model file (the actual +schema-defining source, not a doc page) -- `network`/`gateway`/`descr`/ +`enabled` fields, and `gateway` must reference a named gateway object +(hence the template's own `` addition, +which the stock sample doesn't need since it uses DHCP). + +TESTED END-TO-END, not just guard clauses: unlike every other opnsense-* +script this session, `opnsense-render-config.sh` needs no external tool +(pure bash + template substitution), so its harness (8/8) exercises the +real behavior -- happy path, well-formed-XML output (verified with +`python3 -m xml.dom.minidom`), the NTP real-default fallback, one static +route rendering while an unset second slot correctly emits nothing, and +three failure paths. + +**A real bug the harness itself caught:** the token `HOSTNAME` collides +with bash's own built-in `$HOSTNAME` variable -- `unset HOSTNAME` does not +actually clear it (bash repopulates it), so a "missing required token" +test that should have failed instead silently passed with the shell's own +system hostname. Renamed to `OPNSENSE_HOSTNAME` throughout (template, +script, docs, tests) before this shipped, not after. + +WHAT'S STILL NOT REAL: `opentofu/templates/README.md`'s token legend marks +each token's status precisely -- several cannot be filled with a genuine +value yet because the decision they depend on is still PROPOSED +(D-100/D-101/D-107's WAN addressing, LAN plane assignment, mirror +software choice), and two (`WAN_IF`/`LAN_IF`) are only measurable on a real +boot (which `vtnetN` a network becomes). `ROOT_PASSWORD_HASH` must be +freshly generated per deployment -- the stock sample's own shipped hash is +a well-known default and must never be reused verbatim. + +`opentofu/README.md`'s OPNsense research section and gap register item 4 +in `docs/dc-dc-deployment-workflow.md` (+ its summary table, + the +companion visual tracker) updated to reflect this as BUILT (the mechanism +and the tested renderer), not just the still-pending final values. + +REVERT: `git rm -r opentofu/templates scripts/opnsense-render-config.sh +tests/opnsense-render-config`; revert the `opentofu/README.md` and +workflow-doc sections if reverting fully. + +## Next actionable step + +Two independent things still block a real per-site config.xml: (1) Stage 0 +ratification of D-100/D-101/D-107 (WAN addressing, LAN plane assignment, +mirror software choice) -- a decision, not a build task; (2) a real boot to +measure `vtnetN` device assignment before setting `WAN_IF`/`LAN_IF`. Neither +needs more OpenTofu/script work -- the mechanism is done and tested. diff --git a/docs/archive/changelogs/changelog-20260709-opnsense-edge-module.md b/docs/archive/changelogs/changelog-20260709-opnsense-edge-module.md new file mode 100644 index 0000000..f5cefe6 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opnsense-edge-module.md @@ -0,0 +1,91 @@ +# Changelog 2026-07-09 -- opnsense-edge module + prep scripts (DOCFIX-146) + +No live infrastructure touched -- unapplied IaC/scripts only. Implements the +mechanism identified in the same-day OPNsense deployment research +(`docs/changelog-20260709-opnsense-research.md`). + +## Items + +### 1. DOCFIX-146 -- `scripts/opnsense-prep-image.sh` +FILE (new): `scripts/opnsense-prep-image.sh` + `tests/opnsense-prep-image/run-tests.sh`. + +WHAT: downloads the OPNsense nano image, decompresses it, converts raw -> +qcow2, and grows it -- the preprocessing `libvirt_volume`'s `create.content.url` +can't do itself (it almost certainly performs a plain fetch, no +decompression). No mirror URL is hardcoded -- `OPNSENSE_MIRROR_BASE` is a +required env var (hard rule 2: never use an inferred value; OPNsense +publishes multiple mirrors and a silently-hardcoded one risks going stale +unnoticed). + +HARNESS: 4/4 green, but only the guard-clause paths this environment can +exercise -- `qemu-img` is genuinely absent here, so that failure path is +real, not synthetic; the actual download/convert/resize behavior is +UNTESTED (would need `qemu-img` + network access to a real mirror). + +### 2. DOCFIX-146 (cont.) -- `scripts/opnsense-build-config-iso.sh` +FILE (new): `scripts/opnsense-build-config-iso.sh` + `tests/opnsense-build-config-iso/run-tests.sh`. + +WHAT: builds a plain ISO9660 image containing `/conf/config.xml`, for +OPNsense's Configuration Importer (see the OPNsense research changelog -- +ISO9660 support was added to the Importer specifically for VM/cloud +automation). Uses `genisoimage`/`xorriso`, whichever is present. + +HARNESS: 3/3 green, same caveat -- neither ISO tool is present in this +environment, so the missing-tool guard is exercised for real, but the actual +ISO construction and whether OPNsense's Importer picks it up on a real boot +are UNTESTED. The exact genisoimage/xorriso invocation is a reasonable, +standard construction for a plain ISO9660 volume with one file at a fixed +path -- not independently verified. + +### 3. DOCFIX-146 (cont.) -- `opentofu/modules/opnsense-edge` +FILES (new): `opentofu/modules/opnsense-edge/{main,variables,outputs}.tf`. + +WHAT: one OPNsense edge VM per call. Mechanically identical to +`modules/cloudinit-vm`'s shape (a `backing_store` overlay disk + a secondary +volume attached as `cdrom`), but with the config-seed volume's +`create.content.url` pointing directly at the ISO9660 file built by script 2 +above -- no `libvirt_cloudinit_disk` resource involved (that's NoCloud-format, +wrong for this). Two explicit variables, `lan_network_name` and +`wan_network_name`, rather than a generic ordered list -- OPNsense's own +interface-1-is-LAN/interface-2-is-WAN convention makes list-ordering a real +silent-mistake risk, so the names are explicit instead. No PXE `boot`-order +attribute used (unlike `node-vm`): this VM boots from its own pre-installed +disk, not over the network, so the field flagged UNVERIFIED in +`node-vm/main.tf` doesn't need touching here at all. + +TWO THINGS FLAGGED, NOT PRESENTED AS FACT: (a) whether the Configuration +Importer's ISO9660 support behaves as described on a real boot -- the +research is well-sourced (a closed, milestone-targeted, core-developer- +assigned GitHub issue plus a corroborating later bug report) but not +independently tested this session; (b) whether `devices.interfaces` list +order reliably maps to which NIC FreeBSD/OPNsense enumerates first vs. +second -- assumed consistent with general libvirt XML ordering, not +independently confirmed for this specific guest OS. + +`config_iso_path` has no default and no fallback: the real `config.xml` +content for any site's LAN/WAN/routing role is design work that hasn't +happened. Not instantiated in root `main.tf`. + +`opentofu/README.md` updated: the "Scope of this delivery" and OPNsense +research sections both now describe this as BUILT (not just researched), +with the schema notes section gaining a new entry on `create.content.url` +accepting local filesystem paths (the mechanism both new scripts' outputs +rely on, inferred from `examples/alpine_cloudinit.tf`'s +`libvirt_cloudinit_disk.path` usage, not independently named in the docs +pages fetched this session). Workflow doc gap register item 4 and its +summary-table row updated to match; companion visual tracker (same Artifact +URL) redeployed. + +REVERT: `git rm -r opentofu/modules/opnsense-edge scripts/opnsense-prep-image.sh +scripts/opnsense-build-config-iso.sh tests/opnsense-prep-image +tests/opnsense-build-config-iso`; revert the `opentofu/README.md` and +workflow-doc sections if reverting fully. + +## Next actionable step + +Same underlying need as DOCFIX-144/145: a machine with the `tofu` binary, +`qemu-img`, and `genisoimage`/`xorriso` to actually run the full chain -- +`opnsense-prep-image.sh` for real, `opnsense-build-config-iso.sh` for real, +then `tofu apply` the module and boot it, to confirm the Configuration +Importer's ISO9660 behavior and the LAN/WAN interface-ordering assumption +against a real OPNsense instance rather than sourced-but-unverified research. diff --git a/docs/archive/changelogs/changelog-20260709-opnsense-research.md b/docs/archive/changelogs/changelog-20260709-opnsense-research.md new file mode 100644 index 0000000..5896173 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-opnsense-research.md @@ -0,0 +1,68 @@ +# Changelog 2026-07-09 -- OPNsense deployment research + +Research delivery, not a script/code change -- no live infrastructure touched, +no DOCFIX number assigned (matches the `changelog-20260709-dc-dc-design.md` +precedent for design/research-only deliveries). + +## Item + +### 1. OPNsense deployment mechanism researched against OPNsense's own docs/forum/GitHub +FILES: `opentofu/README.md` (new dedicated section, sourced), +`docs/dc-dc-deployment-workflow.md` (gap register item 4, summary table). + +WHAT: closes the open question from DOCFIX-145 -- whether `modules/ +cloudinit-vm` applies to OPNsense. It does not (confirmed, not just hedged): +OPNsense's own forum consensus is that FreeBSD's cloud-init support "is not +great yet." But there IS a real, officially documented alternative that uses +the exact same mechanical shape already built: + +- The **Configuration Importer** (official docs) runs in a 2-3 second + boot-time window, scanning an attached volume for `/conf/config.xml`. + Supported filesystems: GPT, MBR, ZFS pool, msdosfs (FAT). +- **ISO9660 support was added to the Importer specifically for VM/cloud + automation** -- a closed, milestone-targeted, core-developer-assigned + GitHub issue requested it explicitly so a config ISO could be attached as + a secondary CD-ROM ("shared one to many"). A later bug report confirms the + behavior is live in current code (exact shipped version not pinned from + the 22.7 changelog text directly -- the later bug report is the stronger + signal; confirm on a real boot before relying on it). + This means: same mechanical pattern as `cloudinit-vm` (a secondary volume + attached as a `cdrom` disk), different payload (a plain ISO9660 image with + `/conf/config.xml`, not a NoCloud seed). +- A deeper `autorun` hook (`/usr/local/etc/rc.syshook.d/import/`) exists for + more advanced provisioning beyond a static config.xml. +- **Use the nano image for KVM**, not the vga/serial/dvd installer images -- + pre-installed, serial-console-ready, auto-expands on first boot. Confirmed + real-world workflow: download `.img.bz2` -> `bunzip2` -> `qemu-img convert + -f raw -O qcow2` -> `qemu-img resize` -> boot via libvirt with a FreeBSD + os-variant, serial console, no graphics. +- Network convention: interface 1 = LAN (default 192.168.1.1), interface 2 = + WAN -- order matters, same instinct as `node-vm`'s PXE-boot-priority design. + +All findings sourced with URLs in `opentofu/README.md`'s dedicated section +(docs.opnsense.org, the OPNsense forum, opnsense/core GitHub issues, and one +independent KVM how-to + one community VM-image-building project used for +cross-corroboration). + +WHAT STILL NEEDS BUILDING (not done this delivery, itemized so it's a +punch-list not a vague "TODO"): +1. A preprocessing script for the nano image's decompress+convert step -- + `libvirt_volume`'s `create.content.url` almost certainly does a plain HTTP + fetch with no decompression, so the `.img.bz2` can't be consumed directly; + the qcow2 needs to exist before OpenTofu sees it. +2. A way to build a plain ISO9660 image containing `/conf/config.xml` + (`genisoimage`/`xorriso` -- exact tool/flags not yet verified against a + real build) to feed a `libvirt_volume`'s `create.content.url`, the same + way `libvirt_cloudinit_disk.path` feeds `cloudinit-vm`'s seed volume today. +3. The actual `config.xml` content for OPNsense's LAN/WAN/routing role at + each site -- real design work, not started. + +REVERT: `git checkout HEAD~ -- opentofu/README.md docs/dc-dc-deployment-workflow.md`. + +## Next actionable step + +Build the OPNsense-specific module (working name: `modules/opnsense-edge`) +once items 1-3 above are ready, following the now-confirmed pattern. Verify +the ISO9660 Importer behavior on a real boot as part of that work, not before +-- this research pins down the mechanism with reasonable confidence but the +module build is where it gets tested for real. diff --git a/docs/archive/changelogs/changelog-20260709-repo-agnostic-sweep.md b/docs/archive/changelogs/changelog-20260709-repo-agnostic-sweep.md new file mode 100644 index 0000000..6dd3e09 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-repo-agnostic-sweep.md @@ -0,0 +1,90 @@ +# Changelog 2026-07-09 -- repo-agnostic sweep + repo-lint L9 self-reference guard + +Follow-on to DOCFIX-139/140 (the mechanical ipv4->dc-dc rename sweep). This repo +has now been renamed/forked twice in its short life (per D-110); rather than +sweep by hand again next time, this delivery removes the remaining hardcoded +self-references and adds a lint rule so a NEW hardcode can't land silently. +Executed under blanket approval; no live cloud mutation -- repo content only. +Validation: repo-lint 0 fail / 1 documented WARN; `tests/repo-lint` harness +34/34 (was 29/29; 5 L9 cases added); full repo-wide grep for the current repo +name across scripts/runbooks/skill confirms only the two deliberately-exempted +environment.md fact lines remain. + +## Items + +### 1. DOCFIX-141 -- new repo-lint rule L9: self-reference (repo-agnostic guard) +FILE: `scripts/repo_lint.py` + +WHAT: a new check that FAILS when a script/RUN-block/skill file hardcodes the +CURRENT repo's own directory name (`R.name`) as a literal path/URL token. +Scope: whole-file for `scripts/**/*.{sh,py}` and `.claude/hooks/*.{sh,py}`; +fenced-code-block-only for `runbooks/*.md` (prose naming a repo -- including a +sibling repo by name -- is harmless; only an executed literal is the hazard); +whole-file for `.claude/skills/**/*.md` (agent-operational instructions, not +incidental prose). Opt-out marker `repo-lint: allow-selfref` for a genuine, +reviewed exception (mirrors the existing L2 `allow-stale-tokens` convention). + +WHY: this repo was seeded from `openstack-caracal-ipv4` this same week (D-110) +and DOCFIX-139 was a manual, one-time grep-and-fix sweep for the resulting +stale references. Nothing stopped a NEW hardcode from being written today, or +stopped this same class of bug from recurring on the next fork/rename. A lint +rule converts "remember to grep for this" into an enforced gate. + +SIDE FIX: `all_text()`'s `.claude` exclusion was narrowed from "skip all of +`.claude/`" to "skip only `.claude/worktrees/`" (matching `.gitattributes`' +own scoping of the worktree gitignore rule). The blanket exclusion meant +`.claude/skills/` and `.claude/hooks/` -- both real, checked-in repo content, +not session-local worktree copies -- were invisible to L1 (and now L9) lint +entirely. Verified no new L1 findings from widening scope (checked non-ASCII/ +CR byte counts in all 6 newly-in-scope files before narrowing: all clean). + +HARNESS: `tests/repo-lint/run-tests.sh` T29-T33 (script hardcode FAILS, opt-out +marker suppresses it, runbook prose outside a code block does NOT fire, runbook +code-block hardcode FAILS, skill file hardcode FAILS). 34/34 green. + +KNOWN LIMITATION (logged, not actioned -- pre-existing, not introduced here): +L9's runbook check inherits L3/L4/L6's `in_code` detection, which only +recognizes triple-backtick fences. `runbooks/d011-batch3-window-DRAFT.md` +(4-space-indented pseudo-code) and `runbooks/tenant-onboarding-v2-DRAFT.md` +(`--- BEGIN block ... --- END block ---` comment delimiters) use different +conventions and are NOT reached by L3/L4/L6/L9 at all -- confirmed by manually +grepping both for the DOCFIX-139 hardcodes, which lint did not flag. This is a +real, separate lint-coverage gap (affects the existing L3/L4/L6 checks too, not +just the new L9), logged here rather than fixed, since generalizing in-code +detection to arbitrary block conventions is its own scoped piece of work. +DOCFIX candidate: either teach the checks to recognize indented/BEGIN-END +blocks, or standardize all runbooks onto triple-backtick fences. + +REVERT: `git checkout HEAD~ -- scripts/repo_lint.py tests/repo-lint/run-tests.sh` +(re-arms both the self-reference blind spot and the `.claude/skills` / +`.claude/hooks` lint-invisibility; not recommended). + +### 2. DOCFIX-141 (cont.) -- self-reference removal across skill + 4 runbooks +FILES: +- `.claude/skills/openstack-cloud-ops/SKILL.md` -- frontmatter description, + Step 0 identity/discovery text (now: confirm identity via `git remote -v` + rather than guess a clone path; ask the operator for the clone URL rather + than hardcode one), session-bootstrap line (now references `$REPO`). +- `.claude/skills/openstack-cloud-ops/references/environment.md` -- the repo + URL is KEPT (this file's stated purpose is documenting anchor facts, exactly + like its VIP/version/hostname entries -- re-verify before command-use, same + discipline as everything else here) but tagged `repo-lint: allow-selfref` + with an explicit rationale; added a `$REPO` convention pointer bullet. +- `runbooks/README.md` -- new Conventions bullet establishing `$REPO`: set once + per session (`export REPO=$(git rev-parse --show-toplevel)` if already `cd`'d + into the clone, else `export REPO=~/`); every RUN + block that needs the repo path references `$REPO`, never a literal. +- `runbooks/ops-update-procedure.md`, `runbooks/d011-batch3-window-DRAFT.md`, + `runbooks/phase-01-bundle-deploy.md`, `runbooks/tenant-onboarding-v2-DRAFT.md` + -- the same 8 hardcoded paths fixed mechanically in DOCFIX-139 (rename) are + now converted to `$REPO` (required, fails loud if unset -- no hardcoded + fallback guess, consistent with hard rule 2 "never use an inferred value"). +- `clientdocs/sweep-receipt.txt` (L7 re-record; reviewed for drift from the + tenant-onboarding-v2-DRAFT.md edit -- no client-facing mirror, hash-only). + +WHY: converts "the clone dir must be named exactly X" from an assumption baked +into 8 command sites into one explicit, operator-stated session input -- +exactly the "measured or carried from confirmed as-built" standard hard rule 2 +already requires for every other value in this repo. + +REVERT: `git checkout HEAD~ -- ` per file above. diff --git a/docs/archive/changelogs/changelog-20260709-repo-lint-crlf-fix.md b/docs/archive/changelogs/changelog-20260709-repo-lint-crlf-fix.md new file mode 100644 index 0000000..b576c6a --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-repo-lint-crlf-fix.md @@ -0,0 +1,31 @@ +# Changelog 2026-07-09 -- repo_lint.py CRLF write_text bug (DOCFIX-143) + +## Item + +### 1. DOCFIX-143 -- `--record-clientdocs-sweep`/`--record-guide-skill-coupling` no longer emit CRLF +FILE: `scripts/repo_lint.py` (2 sites: the receipt writer, the coupling writer) + +WHY: flagged during the 2026-07-09 commit-and-push of the repo-agnostic sweep +(DOCFIX-141) -- `receipt.write_text(...)` on Windows translates `\n` to the +platform newline (`\r\n`) before writing, so `clientdocs/sweep-receipt.txt` +came out CRLF on disk every time `--record-clientdocs-sweep` ran in this +session. `.gitattributes`' `eol=lf` normalizes it away at `git add` time, so +no CRLF ever reached a commit -- but the script was silently depending on git +to clean up after its own sloppy output, the same class of "works here only +because something downstream bails us out" issue as the DOCFIX-140 L1 +path-separator bug. + +FIX: both writers now build the exact string, `.encode("ascii")` it, and +`write_bytes()` it -- no platform-dependent newline translation happens at +all. `.encode("ascii")` (not `"utf-8"`) deliberately: this repo's own L1 rule +is ASCII-only, so a future path/hash that somehow isn't ASCII now raises +loudly here instead of writing content L1 would fail on anyway. + +VERIFIED: re-ran `--record-clientdocs-sweep` after the fix; `sweep-receipt.txt` +byte-inspected directly (0 CR, 41 LF, was 41/41 before). `tests/repo-lint` +harness 34/34 (unchanged -- the fix doesn't change any file's logical +content, only its byte-level newline encoding). Full `repo-lint.sh` still +0 fail / 1 documented legacy warn. + +REVERT: `git checkout HEAD~ -- scripts/repo_lint.py` (re-arms the +CRLF-then-git-cleans-it-up dependency; not recommended). diff --git a/docs/archive/changelogs/changelog-20260709-stage0-ratification.md b/docs/archive/changelogs/changelog-20260709-stage0-ratification.md new file mode 100644 index 0000000..ff47956 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260709-stage0-ratification.md @@ -0,0 +1,86 @@ +# Changelog 2026-07-09 -- Stage 0 ratification: D-100..D-110 ADOPTED (DOCFIX-150) + +No live infrastructure touched -- decision-record edits only. Closes Stage 0 +of `docs/dc-dc-deployment-workflow.md`, the gate blocking real Phase-0/1/2 +work since the DC-DC design was drafted this same day. + +## Item + +### 1. DOCFIX-150 -- Stage 0 decision ratification +FILES: `docs/design-decisions.md` (11 status-line updates + D-101 +amendment + D-102 merge), `docs/dc-dc-buildout-design.md` (Section 10 +rewritten as a ruling record, Section 3's Ceph gate annotated, stray +"Section D-102" cross-reference corrected to D-101), `docs/session-ledger.md` +(machine-derived block re-seeded, narrative entry). + +WHAT: operator ruled on the buildout design's 6 redline items (via two +AskUserQuestion rounds, live) and on each of D-100 through D-110 +individually, per this repo's own discipline ("present options, do not +implement" until ruled -- this delivery is the ruling, not a unilateral +implementation). + +**Redline rulings:** +- Ceph replication factor: size=3 by default; size=2 only as an explicit, + logged Phase-0 fallback if the measured disk budget doesn't fit -- never a + silent choice. +- `tc netem` link profile: same-metro dark-fiber lean, confirmed; re-tune + when a specific Roosevelt inter-DC target is known. +- D-102 (IPv6 tenant/MTU sub-policy) folds into D-101 (address plan); D-109 + (Vault posture) stays standalone, NOT merged. +- metal-admin: GAINS a ULA (IPv6) leg alongside its existing v4 -- this + REVERSES the buildout design's own stated lean ("keep metal-admin + v4-only"). A genuine ruling, not a rubber-stamp of the recommended default. +- COS (observability): per-DC only for VR1, confirmed (no Office1 roll-up). +- Per-DC artifact mirror sync topology: independent upstream pull per DC, + confirmed (not one DC seeding the peer). + +**Per-decision disposition, `docs/design-decisions.md`:** +- D-100, D-103, D-104, D-105, D-106, D-107, D-108, D-110: status flipped + PROPOSED -> ADOPTED 2026-07-09 (operator ruling). D-100's netem/Office1- + fiber sub-items, D-105's COS sub-item, and D-107's mirror-topology + sub-item rewritten from "Open sub-item(s)" to "Sub-item(s) RULED," with + the operator's specific answer recorded inline. +- D-101: status ADOPTED + AMENDED same day -- the family matrix's + metal-admin line moved from "IPv4 retained" to the dual-stack group (the + ULA-leg ruling); D-102's full decision body folded in as a new "Tenant/MTU + sub-policy" section; the "Open sub-items" line narrowed to only the + genuinely still-open NetBox literals (a NetBox-assignment task, not a + ratification question -- already tracked as gap #3 in the deployment + workflow's tooling gap register). +- D-102: status changed to `MERGED INTO D-101 (2026-07-09, operator + ruling)`. Body text LEFT IN PLACE, per this repo's append-only + decision-record discipline -- the entry is retained for numbering/history, + not deleted; D-101 is now the current, authoritative text for this + sub-policy. +- D-109: status ADOPTED, explicitly noting it was confirmed to stand alone + despite D-102 folding into D-101 the same day (a real, considered + distinction, not an oversight -- Vault/security posture is different + enough from addressing to keep independently amendable). +- D-106's own open sub-item (cross-DC name resolution, not needed for VR1) + was NOT one of the six items the buildout doc flagged for explicit + redline -- adopted per its own stated lean along with the rest of D-106, + noted plainly in the decision text rather than silently assumed. + +VERIFIED, not just asserted: `bash scripts/ledger-scan.sh`'s "PROPOSED / OPEN +decisions" list no longer contains any of D-100 through D-110 -- only D-068 +and D-071 remain (pre-existing, unrelated to this gate). `repo-lint.sh`: +0 fail / 1 documented legacy warn (unchanged). `tests/repo-lint` harness: +34/34 (unchanged -- no script logic touched, decision-record content only). + +`docs/dc-dc-deployment-workflow.md`'s Stage 0 banner and summary table, +plus the companion visual tracker, updated to reflect Stage 0 CLEARED (see +the companion Artifact redeploy). + +REVERT: `git checkout HEAD~ -- docs/design-decisions.md +docs/dc-dc-buildout-design.md docs/session-ledger.md` (re-opens Stage 0 -- +not recommended; if a specific ruling needs reconsidering, amend that one +decision's entry instead of reverting the whole ratification). + +## Next actionable step + +Stage 0 cleared -- Stage 1 (Phase 0: vcloud host prep) and onward can now +proceed on runbook/tooling authorship without an open-decision blocker. +Real values still pending NetBox assignment (DC2's supernet, the org ULA +/48, per-DC GUA carve) and infrastructure access (a `tofu`/`qemu-img`/ +ISO-tool-capable machine, a real vcloud host) remain separate, non-decision +blockers -- tracked in the tooling gap register, not this ratification. diff --git a/docs/archive/changelogs/changelog-20260710-carve-gua-hang-fix.md b/docs/archive/changelogs/changelog-20260710-carve-gua-hang-fix.md new file mode 100644 index 0000000..595ddf4 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-carve-gua-hang-fix.md @@ -0,0 +1,48 @@ +# Changelog 2026-07-10 -- DOCFIX-181: carve_gua subnet-enumeration hang + harness watchdog + +**Context.** After the Stage-1 commit, the full `scripts/run-tests-all.sh` +gauntlet was seen to HANG on `tests/dc-dc-prefixes-import/`. Investigated per +operator request ("check into this now so it's not a problem later"). Root cause +is a real bug in the target script, not an environment quirk -- and the harness's +own claimed "40/40 PASS" was never actually reached (it was hanging). + +## Root cause + +`netbox/dc-dc-prefixes-import.py::carve_gua()` built +`list(dc_gua_prefix.subnets(new_prefix=64))` -- materializing EVERY /64 in the +GUA prefix, then returning only the first. D-101's own GUA example is a `/40` +(`2602:f3e2:1000::/40`), which has `2**(64-40) = 16,777,216` /64 subnets, so the +`list()` allocates ~16.7M `IPv6Network` objects -> effective hang / OOM. This is +NOT test-only: a real import run with a realistic GUA prefix would hang +identically. (`carve_ula` is safe -- it carves /48->/56->/64, bounded at 256.) + +Diagnosed with `faulthandler.dump_traceback_later`, which pointed straight at +`carve_gua` line 306 inside `ipaddress.subnets()`. + +## Fix + +- **`netbox/dc-dc-prefixes-import.py::carve_gua`** -- take the first /64 LAZILY + with `next(dc_gua_prefix.subnets(new_prefix=64))` (O(1)) instead of + `list(...)[0]`. Same result, no enumeration. +- **`tests/dc-dc-prefixes-import/test_logic.py`** -- added a + `faulthandler.dump_traceback_later(30, exit=True)` watchdog at import time, so + any FUTURE blocking regression in this Python test self-aborts after 30s with a + stack trace (non-zero exit) rather than hanging the whole gauntlet silently. + +## Verification +- `bash tests/dc-dc-prefixes-import/run-tests.sh` -> `ALL PASS (40 checks)` in + ~0.05s (previously: hung indefinitely, never completing). +- `bash scripts/repo-lint.sh` -> 0 fail. +- Full `scripts/run-tests-all.sh` gauntlet now runs to completion. + +## Revert +- `git checkout netbox/dc-dc-prefixes-import.py tests/dc-dc-prefixes-import/test_logic.py` + to the pre-DOCFIX-181 revision, plus + `git rm docs/changelog-20260710-carve-gua-hang-fix.md`. + (Reverting reinstates the hang -- do not, without replacing the fix.) + +## Note +- The pre-existing ledger claim of "dc-dc-prefixes-import 40/40" was optimistic: + the end-to-end path had been hanging, so those checks never actually ran to a + green count until this fix. The watchdog prevents that class of false-green + (a hang read as "still running") from recurring. diff --git a/docs/archive/changelogs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md b/docs/archive/changelogs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md new file mode 100644 index 0000000..d94d80e --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md @@ -0,0 +1,101 @@ +# Changelog 2026-07-10 -- ledger-scan.sh: self-inflicted next-free false positive (DOCFIX-174) + +No live infrastructure touched. Found while preparing to assign the next real +DOCFIX number for follow-up work: `bash scripts/ledger-scan.sh` reported +`DOCFIX: highest=1004 next-free=1005` -- clearly wrong, since the repo is at +DOCFIX-173 (confirmed via `git log`). + +## Root cause + +Ironic: DOCFIX-173's OWN changelog +(`docs/changelog-20260710-sweep-script-fixes.md`) narrated a bug reproduction +for the `{3}` -> `{3,}` regex widening it was making, and used a literal +illustrative example (this repo's real prefix, followed by the digits one- +zero-zero-four) to demonstrate the OLD bug's truncation behavior. +`nextfree_mentions()` in `scripts/ledger-scan.sh` greps ALL prose under +`docs/` and `runbooks/` for the prefix-plus-3-or-more-digits pattern +(excluding only lines that look like "next-free" pointers) and takes the +highest number found. The just-widened regex now matched that illustrative +example as if it were a real assignment, inflating next-free from 174 to +1005. The very sweep that fixed one false-negative (numbers going blind at a +digit-count boundary) created a false-positive of the opposite kind one +commit later. + +Confirmed via reproduction: scanning `docs/` and `runbooks/` for the real +prefix-plus-digits pattern (excluding "next-free" pointer lines) lists +DOCFIX-173 as the true highest real mention, plus one spurious high-looking +match traced directly to the changelog's own illustrative prose described +above. + +## Fix + +1. Reworded the illustrative example in + `docs/changelog-20260710-sweep-script-fixes.md` to describe the + reproduction in words instead of spelling out a literal + `DOCFIX-`-shaped string, removing the false match at its source. +2. Made the identical edit to `scripts/ledger-scan.sh`'s own comment (which + had the same literal string in a code comment -- not itself scanned since + `scripts/` isn't a scanned directory, but worth fixing for hygiene so it + isn't copy-pasted into a doc later). +3. Hardened `nextfree_mentions()` itself: consolidated the (accidentally + duplicated, see Verification below) function definition into one, and + added a second exclusion alongside the existing "next-free" pointer-line + filter: lines matching `reproduc` (case-insensitive) are now also + excluded from the mention scan. This is defense-in-depth, not a + substitute for #1/#2 -- this repo's own delivery discipline narrates bug + reproductions constantly ("Reproduced:", "confirmed-via-reproduction"), + so a future sweep changelog doing exactly what DOCFIX-173's did is a + real, recurring risk, not a one-off. +4. Added a regression fixture to `tests/ledger-scan/run-tests.sh`: a + `docs/repro-example.md` file containing a "Reproduced: ... DOCFIX-1004 + ..." line, asserting the DOCFIX next-free assertion (`next-free=086`) + still holds with that fixture present -- so this exact class can't + regress silently again. + +**Process note on how this was caught:** not by review -- by simply running +`bash scripts/ledger-scan.sh` before assigning the next real DOCFIX number for +unrelated follow-up work, and noticing the output was implausible (a jump from +173 to 1004) rather than trusting it. Same "verify before trusting a tool's +own output" discipline this session used repeatedly elsewhere (the +`expect_ok` dead-code bug, the `phase-06-capi-stack.sh` `PIPESTATUS` fix). + +**It recurred immediately, within this same delivery, before commit.** This +changelog's own first draft, and its mirrored bullet in +`docs/session-ledger.md`, ALSO quoted the illustrative example as a literal +string while narrating the incident -- reinflating next-free right back to +1005 on the next `ledger-scan.sh` run, caught by re-running the scan again +before assigning DOCFIX-175/176 rather than assuming the fix from this same +delivery was airtight. Fixed by rewording both (spelled-out digits, same +convention `scripts/ledger-scan.sh`'s own comment now documents) rather than +relying solely on the `reproduc`-line exclusion, which only catches the +word appearing on the SAME line as the example -- a narrower guarantee than +it first appears, worth remembering next time this bug class comes up. + +## Verification + +`bash tests/ledger-scan/run-tests.sh`: 37/37 PASS (up from 33/33 -- one new +regression fixture + its assertion). `bash scripts/ledger-scan.sh`: now +correctly reports `DOCFIX: highest=173 next-free=174`. `bash +scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. Full gauntlet +(`bash scripts/run-tests-all.sh`): re-run after this fix, same pre-existing +environment gaps as every prior check tonight, zero regressions (see this +delivery's companion session-ledger entry for the exact count). + +While editing, also caught and fixed my own mistake mid-fix: the first pass +at hardening `nextfree_mentions()` left TWO definitions of the function in +the file (the original, unedited one at its original location, plus a new +one with the fix inlined just before the call sites) -- a duplicate that +would still have worked (bash uses the last definition) but was sloppy and +would have confused the next reader. Caught by re-reading the file before +running the harness, consolidated to one definition in place. + +REVERT: `git checkout HEAD~ -- scripts/ledger-scan.sh +tests/ledger-scan/run-tests.sh +docs/changelog-20260710-sweep-script-fixes.md` (safe -- no live infrastructure +depends on this script's output; it is a reporting/reconciliation tool only). + +## Next actionable step + +None required by this fix alone. The true next-free numbers going into any +follow-up work tonight are: D-111, DOCFIX-175, BUNDLEFIX-012 (this delivery +consumes DOCFIX-174). diff --git a/docs/archive/changelogs/changelog-20260710-netbox-roles-aggregates.md b/docs/archive/changelogs/changelog-20260710-netbox-roles-aggregates.md new file mode 100644 index 0000000..4d838a2 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-netbox-roles-aggregates.md @@ -0,0 +1,50 @@ +# Changelog 2026-07-10 -- NetBox roles + aggregates prerequisite (VR1 DC-DC) + +## Added + +- **`netbox/roles-aggregates-import.py`** -- idempotent, preview-by-default script that creates + the five missing six-plane IPAM roles and the aggregates that `netbox/dc-dc-prefixes-import.py` + assumes already exist. Closes gaps G1 + G2 (below). +- **`docs/dc-dc-netbox-buildout-scope.md`** -- consolidated planning/decision scope for the VR1 + NetBox buildout, reconciling the repo (D-100..D-110), a live read of `netbox.baldurkeep.com` + (v4.5.8, 2026-07-10), and the parallel IPAM-design chats. The authoritative reference the Code + stream builds from. + +## Why -- reconciliation findings (live NetBox vs repo vs design chats) + +A live NetBox read surfaced gaps and divergences that the prefixes-import tool would trip on: + +- **G1 -- six-plane roles:** only `storage` exists; `provider-public`, `metal-admin`, + `metal-internal`, `data-tenant`, `replication` were **missing**. The prefixes-import tool + `die()`s on a missing role by design (does not create roles). +- **G2 -- aggregates:** **0 aggregates** despite ARIN being the RIR and NetBox being the IPAM + apex (D-010). +- **Authoritative model = D-101** (six-plane, ULA-internal, GUA-external, `/60` tenant). The + byte-aligned `/32` / `/64`-per-tenant carve from the parallel IPAM-design project **diverged and + is superseded for the DC-DC buildout** (it used the old role set, `/64` tenants, and a + nibble-1 VR0/VR1 placement; deployed NetBox + D-101 use the six-plane set, `/60` tenants from a + separate delegation pool, and nibble-2 VR placement). Details in the scope doc, section 3. +- **Still open (not this delivery):** G5 DC-naming mismatch (`vr1-dc0`/`vr1-dc1` in NetBox vs + DC1/DC2 in decisions + the prefixes tool's `vr1-dc1`/`vr1-dc2`) blocks the prefixes import until + resolved; VR1 VLAN groups absent (G4); the three D-101 literals (`ORG_ULA_48`, per-DC + `DC_GUA_PREFIX`, `DC2_V4_SUPERNET`) still need assignment. + +## Aggregate detail + +`2602:f3e2::/36` (the HELD allocation = region-0 of the planned `/32`; NOT `/32` -- declaring +space not yet allocated is dishonest for an IPAM apex; update to `/32` after the expand), +`23.157.124.0/24`, `10.0.0.0/8`, `172.16.0.0/12`, and -- only if `ORG_ULA_48` is passed -- the +org ULA `/48`. New RIRs `RFC 4193 ULA` and `RFC 1918` are created for the private aggregates; +`ARIN` is assumed pre-existing (it is). + +## Validation + +Script previewed (read-only) against live NetBox 4.5.8: correctly reports 12 objects to create +(5 roles + 2 RIRs + 5 aggregates) and 0 pre-existing, `storage` detected as existing, `ARIN` +detected (id=1). Not committed -- Code runs `--commit` after review. + +## Sub-decisions requiring ratification (candidate D-111+, assign via ledger-scan) + +Per the scope doc section 5: the prefixes tool's per-plane subcarve (align to the deployed `NN` +role mnemonic); DC naming (G5); DC2 v4 supernet value; tenant `/60` confirmation; VR0-DC0 +old-model retirement. None block this delivery (roles + aggregates are model-independent). diff --git a/docs/archive/changelogs/changelog-20260710-opentofu-scaffold-first-validation.md b/docs/archive/changelogs/changelog-20260710-opentofu-scaffold-first-validation.md new file mode 100644 index 0000000..31e8474 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-opentofu-scaffold-first-validation.md @@ -0,0 +1,74 @@ +# Changelog 2026-07-10 -- DOCFIX-179: OpenTofu scaffold, first real validation + apply + +**Context.** First-ever execution of the `opentofu/` tree against a real `tofu` +binary (OpenTofu v1.12.3), on the vcloud host, executing +`runbooks/dc-dc-phase0-vcloud-prep.md` Steps 8-11. The tree had carried a +**SCAFFOLD, UNVALIDATED** banner since authoring (no `tofu` had ever run it). +Two real bugs blocked `tofu init`/`plan`; both fixed here, then the Stage-1 +module slice was validated, planned, and applied clean (13 libvirt objects). + +## What changed + +### FIX 1 -- per-module `required_providers` (the `tofu init` blocker) +- `tofu init` failed: child modules use `libvirt_*`/`maas_*` resources but did + not declare their own `required_providers`, so OpenTofu inferred the + `libvirt_` prefix as **`hashicorp/libvirt`** (nonexistent) instead of + `dmacvicar/libvirt`. Provider SOURCE mapping is NOT inherited by child + modules -- only provider CONFIGURATION is. +- Added `versions.tf` (source-only; version stays pinned at the root) to every + module using a provider resource: 8 libvirt modules (`dc-planes`, `mesh-link`, + `dc-storage-pool`, `office1-network`, `node-vm`, `base-image`, `cloudinit-vm`, + `opnsense-edge`) + 1 maas module (`maas-vm-host`). `netem-link` uses only the + built-in `terraform_data`, so it needs none. Fixed comprehensively (all + modules, not just Stage 1's four) so Stage 2/3 do not re-hit it. + +### FIX 2 -- `tofu fmt` (the whole never-fmt'd scaffold) +- `tofu fmt -recursive opentofu/` normalized 11 files (attribute alignment, + comment reflow) that had drifted because no `tofu` had ever formatted them. + +### CHANGE 3 -- `provider "maas"` deferred to Stage 3 +- The root `provider "maas"` block forced `tofu plan` to demand `maas_api_url` + AND the **sensitive** `maas_api_key` for a Stage-1 plan that creates ZERO + MAAS resources -- which would bake a fake key into `phase0.tfplan` and + `terraform.tfstate` (the DOCFIX-175 plaintext-secret surface). Operator ruled + (2026-07-10) to fix structurally rather than use the runbook's placeholder + workaround. Removed the `provider "maas"` block from `main.tf` and the + `maas_api_url`/`maas_api_key` variables from `variables.tf`, with in-file + notes to re-add them at Stage 3 (when `maas-vm-host` is instantiated). `maas` + stays in the ROOT `versions.tf` `required_providers` (version pinned, + `.terraform.lock.hcl` complete for Stage 3). + +### FIX 4 -- repo-lint must skip OpenTofu's `.terraform/` provider cache +- After the first real `tofu init`, `repo-lint` FAILED L1 (ASCII) on a + downloaded provider plugin's README inside `opentofu/.terraform/`. DOCFIX-175 + gitignored `.terraform/` for git, but `repo_lint.py`'s `all_text()` scans the + filesystem, not git. Added `".terraform" in p.parts` to `all_text()`'s skip + set (alongside `.git`/`__pycache__`) -- third-party, gitignored, not repo + content. `tests/repo-lint/run-tests.sh` -> 34/34 PASS. + +### Result +- `tofu validate` -> Success. `tofu plan` -> 13 to add / 0 / 0, with NO MAAS + vars required. `tofu apply` -> **13 added, 0 changed, 0 destroyed** (6 DC1 + planes + dc1/office1/dc2 pools + office1-local + 3 mesh legs, all MTU 9000). +- `.terraform.lock.hcl` is now generated -- **commit it** (pins provider + hashes: `dmacvicar/libvirt` 0.9.8, `canonical/maas` 2.7.2). +- `opentofu/README.md` status banner updated: Stage-1 module slice VALIDATED + + APPLIED; the not-yet-instantiated Stage 2/3 modules remain UNVALIDATED. + +## Verification +- `bash scripts/opentofu-validate.sh` (== `tofu fmt -check` + `init` + `validate`) + -> clean after the fixes. +- `bash scripts/repo-lint.sh` -> 0 fail. + +## Revert +- `git rm opentofu/modules/*/versions.tf` (the 9 added files) -- restores the + original `tofu init` inference failure. +- Re-add the `provider "maas"` block to `main.tf` and the two `maas_*` variables + to `variables.tf` from this changelog / git history. +- `tofu fmt` reverts are cosmetic; `git checkout` the 11 files to undo. +- (Live infra: `runbooks/dc-dc-teardown-rollback.md` / `tofu destroy` -- not part + of a repo revert.) + +## Follow-up (logged) +- Stage 3's runbook must re-add the `provider "maas"` block + the two variables + when it wires `maas-vm-host` -- noted in `main.tf`/`variables.tf` in-file. diff --git a/docs/archive/changelogs/changelog-20260710-opentofu-state-file-security.md b/docs/archive/changelogs/changelog-20260710-opentofu-state-file-security.md new file mode 100644 index 0000000..e12d34d --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-opentofu-state-file-security.md @@ -0,0 +1,111 @@ +# Changelog 2026-07-10 -- OpenTofu state file security posture (DOCFIX-175) + +No live infrastructure touched -- no `tofu` binary has ever been run in any +session that authored this repo clone, so no `terraform.tfstate` exists yet. +This is a preventive fix, found while reviewing what's genuinely missing from +the VR1 buildout ahead of Stage 1's own first real `tofu apply` +(`runbooks/dc-dc-phase0-vcloud-prep.md` Step 10 -- DC1 planes + Office1 pool ++ mesh links), not a live incident. + +**Correction made mid-delivery, worth recording:** the first draft of this +work assumed Stage 3 (`dc-dc-phase2-tofu-dc-substrate.md`) was the first real +apply, based on that file's name suggesting it was "the OpenTofu stage." +Reading the actual runbooks directly (not inferring from filenames) showed +Stage 1's own Step 10 applies DC1's planes, Office1's storage pool, and the +three mesh links -- exactly what's already uncommented in `opentofu/main.tf` +today -- well before Stage 3 touches anything. Caught and fixed before +committing, same "verify against the actual file, not the name" discipline +this repo already uses everywhere else. + +## The gap + +Nothing in this repo documented how OpenTofu's own state file is protected. +No `backend` block is configured anywhere (`versions.tf`/`main.tf` are silent +on it; every stage's own `tofu init` runs `-backend=false`), so state +defaults to a local `terraform.tfstate` in whatever directory `tofu` is +invoked from (the Office1 operator VM, per `opentofu/README.md`'s own +framing). Two real problems follow from that: + +1. **Plaintext secrets in state.** `var.maas_api_key` is marked + `sensitive = true`, but that flag only redacts the value from CLI/plan + OUTPUT -- verified against HashiCorp's own current documentation (not + assumed): *"Terraform stores values with the `sensitive` argument in both + state and plan files, and anyone who can access those files can access + your sensitive values"* (developer.hashicorp.com/terraform/language/ + manage-sensitive-data, fetched 2026-07-10). This applies regardless of + whether the value is consumed by a resource argument or, as here, only by + a `provider {}` block -- the docs draw no such distinction. At Stage 1 + time `maas_api_key` may still be a placeholder (MAAS doesn't exist for + real until Stage 2 -- see Stage 1's own "Known gap" section), but the + moment a REAL `TF_VAR_maas_api_key` is exported for any later plan/apply, + it lands in state in the clear, with no protection beyond filesystem + permissions -- and nothing documented what those permissions should be. +2. **State was not gitignored.** Harmless only by accident (no state file has + ever existed in a repo clone), not by design -- a real gap that would have + let a future `git add -A` commit a plaintext-credential-bearing file to + history. + +Neither problem was previously logged anywhere -- not the gap register, not +`opentofu/README.md`, not `.gitignore`. + +## Fix + +1. **`.gitignore`**: added `opentofu/**/*.tfstate`, `*.tfstate.*`, + `.terraform/`, `*.tfplan`, `*.auto.tfvars*`, and the standard Terraform + override/crash-log patterns. `.terraform.lock.hcl` deliberately NOT + excluded (provider-version pin; matches this repo's own exact-release-not- + range discipline already established in `versions.tf`). +2. **`opentofu/variables.tf`**: `maas_api_key`'s description now states the + plaintext-in-state caveat inline and directs to `TF_VAR_maas_api_key` + (never `-var`, which lands in shell history) -- consistent with, and + reinforcing, `dc-dc-phase0-vcloud-prep.md` Step 7's own pre-existing + "never commit `maas_api_key`" guidance for the *input* side; this delivery + covers the *output* (state) side that guidance didn't. +3. **`opentofu/README.md`**: new "State file handling" section covering the + risk and five concrete mitigations for VR1's rehearsal scope (never + commit state, never pass secrets via `-var`, restrict file permissions, + back up out-of-band per the existing `~/vault-init/`-class secret + discipline, and a flagged-not-decided note that local state has no + locking -- a real Roosevelt/production gap, not solved here). +4. **`runbooks/dc-dc-phase0-vcloud-prep.md`**: Step 10 (the actual first real + `tofu apply` in this whole repo) gained a "SECURE THE STATE FILE" callout + immediately after it -- `chmod 700`/`chmod 600`, a `git status` check that + the state file shows ignored not staged, out-of-band backup, and an + explicit note that this is a standing check repeated after every + subsequent stage's apply, not a one-time Stage 1 step. +5. **`runbooks/dc-dc-phase2-tofu-dc-substrate.md`**: Step 8 gained a short + pointer back to Stage 1's callout (not a full duplicate) plus a note that + Step 9 next is where this stage first needs a REAL MAAS credential to + succeed. +6. **`docs/dc-dc-deployment-workflow.md`**: added gap-register item #18 + (this gap, CLOSED) and #19 (a related, still-OPEN gap found in the same + review pass: no teardown/rollback runbook exists for the VR1 OpenTofu + layer itself -- tracked separately, not invented as a rushed addendum + here). + +## What this deliberately does NOT do + +Migrate to a remote backend (S3-compatible + lock table, Terraform Cloud, +Consul, etc.) or add real state locking. Both are production-grade +infrastructure decisions appropriate for Roosevelt planning, not something to +invent unilaterally for a rehearsal repo that has never run `tofu` for real. +Flagged explicitly in the README rather than silently deferred. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. No script +logic changed -- `.gitignore`, one `.tf` file (description-only edit, no +resource/variable type change), one README, two runbooks (prose additions), +one tracker doc. No test harness applies (nothing here is executable script +logic); `git diff --stat` confirms no `.sh`/`.py` file in this delivery. + +REVERT: `git checkout HEAD~ -- .gitignore opentofu/variables.tf +opentofu/README.md runbooks/dc-dc-phase0-vcloud-prep.md +runbooks/dc-dc-phase2-tofu-dc-substrate.md +docs/dc-dc-deployment-workflow.md` (safe -- no live infrastructure depends on +any of these; nothing here has ever been applied against real state). + +## Next actionable step + +Gap #19 (VR1 teardown/rollback runbook) is a separate, tracked follow-up -- +see its own changelog once delivered. diff --git a/docs/archive/changelogs/changelog-20260710-phase0-runbook-as-executed.md b/docs/archive/changelogs/changelog-20260710-phase0-runbook-as-executed.md new file mode 100644 index 0000000..1fbf07a --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-phase0-runbook-as-executed.md @@ -0,0 +1,49 @@ +# Changelog 2026-07-10 -- DOCFIX-180: dc-dc-phase0 runbook as-executed corrections + +**Context.** End-of-Stage-1 runbook update. Per the operator's model (keep a +DOCFIX log through a whole phase, then update the runbook at phase close), the +corrections accrued during the first real execution of +`runbooks/dc-dc-phase0-vcloud-prep.md` (2026-07-10) are now folded back into the +runbook as draft updates. The live execution itself is DOCFIX-179 (scaffold) + +DOCFIX-178 (prereq installers); this entry is the runbook-text reconciliation. + +## What changed in `runbooks/dc-dc-phase0-vcloud-prep.md` + +1. **Top banner** -- marks the runbook FIRST EXECUTED 2026-07-10 and points at + the DOCFIX-178/179/180 changelogs. +2. **Known-gap section -> RESOLVED.** The `provider "maas"` block DID force + `tofu plan` to demand both `maas_api_url` AND the sensitive `maas_api_key` + for a zero-MAAS plan. Recorded the operator's structural resolution (defer the + provider block + both vars to Stage 3; DOCFIX-179) instead of the placeholder + workaround. Also corrected a stale reference ("Step 4" -> it surfaces at + Step 8/9). +3. **Step 3 (MTU).** Added the two-MTU-domains clarification: the measured host + *uplink* MTU (1500) is the ISP/WAN-edge value (Stage 2/3), while `underlay_mtu` + drives host-internal jumbo-capable virtio bridges -> set it to jumbo (9000) + for the internal fabric. First run: measured 1500, ruled `underlay_mtu=9000`. +4. **Step 5 (pools).** Added the non-pristine-host reality (a prior VR0 topology + + `wan` were present): inventory first, tear down leftovers gated, KEEP `wan` + as a gap-#17 ISP-uplink model. Replaced the bare `sudo mkdir` with the + self-owned-parent pattern (`install -d -o $USER -g libvirt /var/lib/libvirt/vr1` + once, then sudo-free subdirs). Recorded that DC2's storage pool IS wired now + (planes still deferred). +5. **Step 8 (init/validate).** Recorded the first-`init` failure + its two fixes + (per-module `required_providers`; `tofu fmt -recursive`) and the README banner + flip to STAGE-1-VALIDATED. +6. **Step 9 (plan).** Added `-input=false` (non-interactive safety) and corrected + the expected plan from 11 to the real **13 resources** (added the omitted + `office1_network` and the newly-wired `dc2_storage`). + +## Verification +- `bash scripts/repo-lint.sh` -> 0 fail (1 documented legacy WARN). +- No script changed here (doc-only), so no harness; the tooling changes these + notes describe are covered by DOCFIX-178/179's harnesses. + +## Revert +- `git checkout runbooks/dc-dc-phase0-vcloud-prep.md` to the pre-DOCFIX-180 + revision (removes the 6 as-executed edits), and + `git rm docs/changelog-20260710-phase0-runbook-as-executed.md`. + +## Follow-up (logged) +- The other `dc-dc-phaseN` runbooks will get the same end-of-phase as-executed + treatment as each is first-executed (their own DOCFIX numbers). diff --git a/docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md b/docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md new file mode 100644 index 0000000..d37f5c6 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-phase01-bundle-staleness-fix.md @@ -0,0 +1,59 @@ +# Changelog 2026-07-10 -- phase-01-bundle-deploy.md: stale bundle expectations fixed (DOCFIX-169) + +No live infrastructure touched -- runbook doc fix only. Found during a +full-project sweep for errors/architectural problems, requested +immediately after the seven gap-closure commits (DOCFIX-162 through 168). + +## Item + +### 1. DOCFIX-169 -- `runbooks/phase-01-bundle-deploy.md` stale post-DOCFIX-167 expectations +FILE: `runbooks/phase-01-bundle-deploy.md`. + +WHAT: `bundle.yaml`'s DOCFIX-167 edit (Designate reactivation + D-108 DR +mechanism, earlier tonight) changed the SAME shared bundle file this +existing, LIVE-CLOUD-relevant VR0 runbook deploys -- but that edit did not +update this runbook's own documented expectations at the time. Found while +sweeping for cascading effects of tonight's changes: + +- **"expected plan: 50 apps, 97 relations"** (Step 1.2's dry-run GATE, the + Constants section, and the historical "As-built reference") was stale -- + the real current count is 56 apps / 108 relations (confirmed via + `python3 -c "import yaml; ..."`). Fixed the two LIVE-GATE references + (Constants section, Step 1.2's GATE); the "As-built reference (2026-06-03 + second redeploy)" section's `50 apps / 97 relations` line was LEFT + UNCHANGED -- it is a historical audit-trail record of a specific past + deploy, and this repo's discipline treats as-built records as append-only + history, not something to retroactively edit. +- **"NO designate (D-019)"** (Step 1.2's GATE) was flatly wrong post- + DOCFIX-167 -- corrected to state designate/designate-bind/designate- + mysql-router/designate-hacluster and cinder-backup/ceph-rbd-mirror are + now present, while `vault:ha` remains correctly absent (unchanged). +- **A real, unresolved architectural tension flagged, not decided:** this + bundle is used for BOTH a plain VR0/DC0 single-DC rehearsal redeploy AND + standing up VR1's DC1 (per the buildout design's own framing -- VR0's + testcloud IS becoming DC1, not staying separate). Reactivating Designate/ + the DR mechanism in the shared file means ANY future `phase-01` run gets + these new charms now, regardless of which intent the operator has. Added + a `!!!` callout stating this plainly -- if a plain VR0-only redeploy + without these charms is ever needed, that requires an explicit decision + (a bundle fragment/override), not a silent assumption either way. +- **EXIT GATE's "waiting on vault certs" list extended, clearly labeled + unconfirmed:** `designate`/`designate-bind` have a `certificates: + metal-internal` relation to vault, so by the same pattern as the + existing list entries they are LIKELY also waiting on vault certs + pre-init -- inferred from the relation graph, NOT confirmed against a + real deploy (no live cloud reachable this session). Flagged explicitly + as not-yet-observed rather than asserted as fact. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +This is a doc-only fix; no script/bundle logic changed (`bundle.yaml` +itself is unchanged by this delivery, only this runbook's prose). + +REVERT: `git checkout HEAD~ -- runbooks/phase-01-bundle-deploy.md` (safe). + +## Next actionable step + +This finding is the first result of a broader full-project sweep now +underway (looking for other cascading effects of tonight's bundle.yaml +edit, and any other latent architectural/consistency problems across the +whole repo, not just tonight's own deliverables). diff --git a/docs/archive/changelogs/changelog-20260710-prereq-install-scripts.md b/docs/archive/changelogs/changelog-20260710-prereq-install-scripts.md new file mode 100644 index 0000000..31cd57f --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-prereq-install-scripts.md @@ -0,0 +1,60 @@ +# Changelog 2026-07-10 -- DOCFIX-178: workstation prerequisite install scripts + +**Context.** First real execution of `runbooks/dc-dc-phase0-vcloud-prep.md` on the +vcloud host surfaced that `tofu` was not installed, and the runbook's Step 6 only +hand-waved ("install per OpenTofu's official instructions ... not prescribed +here"). Operator ruling (2026-07-10): a runbook must not assume prereq runtimes +are present, and new operators should have individual, repo-provided install +scripts to get a workstation/workspace ready. This delivers that. + +## What changed + +### NEW: `scripts/prereqs/` (idempotent Debian/Ubuntu installers) +- `lib-prereq.sh` -- shared helpers (sourced): `pq_have`, `pq_require_apt`, + `pq_parse_mode` (--check/--dry-run/--help), `pq_run` (dry-run-aware runner). +- `install-opentofu.sh` -- OpenTofu `tofu` >= 1.6.0 via the official **deb** + method (lands `/usr/bin/tofu`, the command name every runbook step + + `scripts/opentofu-validate.sh` expect; the `opentofu` snap may not expose it). + Version-checks and skips if already new enough. +- `install-virtualization.sh` -- libvirt-daemon-system + qemu-kvm + libvirt-clients + + virtinst, and adds the invoking user to the `libvirt` group (re-login noted). +- `install-jq.sh` -- jq (a known cause of harness-gauntlet failures when absent). +- `install-image-tools.sh` -- qemu-utils (`qemu-img`) + `xorriso`, for the Stage + 2/3 OPNsense build path (`opnsense-prep-image.sh`, `opnsense-build-config-iso.sh`). +- `check-prereqs.sh` -- read-only report (all of the above + curl/python3/sudo), + exit 0 all-required-present / 1 missing; each miss names its fixer script. +- `install-all.sh` -- orchestrator (virtualization -> tofu -> jq -> image-tools) + + final check; passes `--dry-run` through. +- `README.md` -- quick start, per-script table, scope/conventions. +- Each installer supports `--check` (read-only) and `--dry-run` (mutate nothing); + `sudo` is used only for the actual package/group mutation, never in check/dry-run. + Non-Debian/Ubuntu OS -> fail loud (exit 3), no package-manager guessing. + +### NEW: `tests/prereqs/run-tests.sh` +- Exercises the parse/guard/`--check`/`--dry-run` paths only (never a real apt + install -- same posture as the opnsense-* harnesses). Includes the safety + assertion that `pq_run` under dry-run does NOT execute its command. **24/24 PASS.** + +### MODIFIED: `runbooks/dc-dc-phase0-vcloud-prep.md` +- New **Prerequisites** section (before "Known gap") with the one-pass + `check-prereqs`/`install-all` flow + a per-prereq table pointing at the installers. +- Step 6's vague "install per official instructions" replaced with a concrete + `bash scripts/prereqs/install-opentofu.sh` pointer + the >=1.6.0 floor. + +## Verification +- `bash tests/prereqs/run-tests.sh` -> `prereqs: 24/24 PASS`. +- `bash scripts/repo-lint.sh` -> 0 fail (1 documented legacy WARN). +- `bash scripts/ledger-scan.sh` -> DOCFIX next-free was 178 before this entry. + +## Revert +- `git rm -r scripts/prereqs tests/prereqs docs/changelog-20260710-prereq-install-scripts.md` +- Revert the two `runbooks/dc-dc-phase0-vcloud-prep.md` edits (Prerequisites + section + Step 6 paragraph) back to their pre-DOCFIX-178 text. + +## Follow-up (logged, not actioned) +- The other `dc-dc-phaseN` runbooks should gain the same short "see + `scripts/prereqs/`" pointer in their intros (their prereq sets overlap: tofu, + libvirt, and Stage 2/3 additionally need qemu-img/xorriso). One-line-per-runbook + sweep, its own DOCFIX. `scripts/prereqs/` itself is already stage-agnostic. +- macOS/RHEL install paths (currently exit-3 fail-loud) if a non-Ubuntu control + point ever appears. diff --git a/docs/archive/changelogs/changelog-20260710-skill-vr1-awareness.md b/docs/archive/changelogs/changelog-20260710-skill-vr1-awareness.md new file mode 100644 index 0000000..703e6f6 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-skill-vr1-awareness.md @@ -0,0 +1,74 @@ +# Changelog 2026-07-10 -- skill/reference VR1 DC-DC awareness (DOCFIX-177) + +No live infrastructure touched. Found while the operator was heading to the +vcloud host to begin real execution: `.claude/skills/openstack-cloud-ops/` +(loaded before any nontrivial task, per this repo's own `CLAUDE.md`) and +`runbooks/README.md` (what the skill's routing table points to for "Deploy / +redeploy / teardown") were both SILENT on the entire VR1 DC-DC track -- +zero mentions of `docs/dc-dc-deployment-workflow.md`, the seven +`dc-dc-phase*.md` runbooks, `dc-dc-teardown-rollback.md` (DOCFIX-176), or +`opentofu/`. A fresh session reading only the skill (which is exactly what +happens at session start / after compaction) would have been routed straight +to VR0's `phase-01..08` runbooks with no indication the DC-DC track -- the +actual live mission for today -- exists at all. + +## Fix + +1. **`runbooks/README.md`**: new "VR1 DC-DC track" section -- a table + mirroring the existing VR0 "Phases" table's format, listing all 7 Stage + runbooks + `dc-dc-teardown-rollback.md`, explicit that Stage numbering + here is a SEPARATE sequence from the VR0 phase-NN numbers above it (they + share a repo, not a numbering scheme), and a pointer to + `docs/dc-dc-deployment-workflow.md` as the actual master tracker (stage + status + tooling gap register) rather than duplicating that content here. +2. **`.claude/skills/openstack-cloud-ops/SKILL.md`**: routing table gained a + dedicated VR1 DC-DC row (previously the "Deploy / redeploy / teardown" + row only pointed at the VR0 sequence); "Standard loops" gained a second + "Deploy loop (VR1 DC-DC)" entry alongside the existing VR0 one, including + the fix-forward-before-teardown reminder from DOCFIX-176's rollback + decision tree and the opentofu UNVALIDATED caveat. +3. **`references/environment.md`**: "The two deployments" section (VR0 + + Roosevelt) became "The three deployments" -- VR1 DC-DC added as a + pointer-only entry (this file stays VR0's own detailed ground truth; + VR1's facts live in the master tracker, not duplicated here). "Repo map" + section gained a VR1-specific addition listing where the new docs/ + runbooks/opentofu/scripts actually live. +4. **`references/troubleshooting.md`**: one paragraph clarifying that + `appendix-A-troubleshooting.md` is VR0-specific and the VR1 track has no + incident index yet (nothing has been executed) -- so a future + troubleshooting session doesn't waste time searching appendix-A for a + VR1-specific symptom that was never logged there, and knows to log a + genuinely new VR1 finding as a DOCFIX/D-NNN candidate instead of assuming + it must already be documented. + +**Checked, not changed:** `references/operating-discipline.md`'s "Session +continuity" section was re-read against tonight's actual practice (ledger +updates at every deliverable, `ledger-scan.sh` reconciliation) and found +accurate -- no drift to fix. `references/script-authoring.md` is generic +command-block/gating discipline that applies equally to both tracks -- +correctly track-agnostic, left as-is. `references/opentofu-provider-docs.md` +is provider-research content (already VR1-authored this session), not +topology/routing content -- out of scope for this pass. + +## Also fixed before this delivery: unpushed commits + +Found `git status` showing `main` 2 commits ahead of `origin/main` +(DOCFIX-174 and DOCFIX-175/176) -- since the session bootstrap the operator +runs on the vcloud host starts with `git pull`, these would have been +invisible there this morning. Pushed to `origin/main` before starting this +delivery. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn (ASCII-only +convention held across all edits). No script logic changed -- doc/runbook/ +skill prose only, no test harness applies. Full gauntlet re-run: same +pre-existing environment gaps as every prior check tonight, zero +regressions (see this delivery's session-ledger entry for the exact count). + +REVERT: `git checkout HEAD~ -- runbooks/README.md +.claude/skills/openstack-cloud-ops/SKILL.md +.claude/skills/openstack-cloud-ops/references/environment.md +.claude/skills/openstack-cloud-ops/references/troubleshooting.md` (safe -- +routing/reference content only, no live infrastructure or script behavior +depends on any of it). diff --git a/docs/archive/changelogs/changelog-20260710-sweep-bundle-fixes.md b/docs/archive/changelogs/changelog-20260710-sweep-bundle-fixes.md new file mode 100644 index 0000000..5ead97f --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-sweep-bundle-fixes.md @@ -0,0 +1,81 @@ +# Changelog 2026-07-10 -- Full-project sweep: bundle.yaml fixes (DOCFIX-172) + +No live infrastructure touched -- `bundle.yaml` is a repo source file; no +`juju deploy`/`juju add-relation` was run against any live model. Part of +the full-project sweep (see DOCFIX-170 for overall scope). A dedicated +agent audited the ~50 applications NOT touched by tonight's earlier +DOCFIX-167 edit, since that edit only added/verified the 5 new charms. + +## Items + +### 1. DOCFIX-172a -- `magnum-dashboard` had NO `bindings:` block at all +FILE: `bundle.yaml`. + +WHAT: a real, CONFIRMED pre-existing bug -- `git log -p` shows this +application has had no `bindings:` block since the D-052 explicit-bindings +commit, and every commit since (NOT introduced by tonight's DOCFIX-167 +work). Every other application in this file, including its closest sibling +`octavia-dashboard`, gets an explicit bindings block, per the file's own +stated top-of-file discipline ("explicit per-application blocks, no +anchors"). Without one, `magnum-dashboard`'s endpoints fall to the model's +implicit default space instead of a deliberate plane -- a real risk on a +model whose default space isn't routable to the app's LXD container. + +Fixed with a MINIMAL block, confirmed against the real `charm-magnum- +dashboard` metadata.yaml (fetched directly, not assumed from +`octavia-dashboard`'s shape): this charm is a subordinate with ONLY a +`dashboard` (interface `dashboard-plugin`) requires endpoint -- no +`certificates` endpoint exists on it (unlike `octavia-dashboard`, which +does have one). The fix is `'': metal-admin, dashboard: metal-internal`, +not a copy-paste of `octavia-dashboard`'s fuller block. + +### 2. DOCFIX-172b -- two section-header comments misplaced (cosmetic, confirmed harmless) +FILE: `bundle.yaml`. + +WHAT: the "# Secrets: Barbican" and "# Kubernetes-as-a-Service: Magnum" +section-header comments each sat BEFORE a `bindings:` block that YAML +actually attaches to the PRECEDING application (`octavia-diskimage- +retrofit` and `barbican-vault` respectively), not the app the comment +announces. Confirmed via `yaml.safe_load` that both bindings blocks +attached to the correct (preceding) app both before and after this fix -- +this was purely a visual/formatting issue, not a functional bug. Fixed by +relocating each comment block to immediately precede the app it actually +describes, removing the future-edit risk (a reader could plausibly +mistake the bindings block for belonging to the announced app and edit the +wrong one). + +### 3. DOCFIX-172c -- header VIP/HA-chain count staleness +FILE: `bundle.yaml`. + +WHAT: the top-of-file "HA chain" comment still read "ACTIVE for 11 API +charms (10 prior + ceph-radosgw, un-deferred)" -- stale since DOCFIX-167 +added `designate-hacluster` as a 12th hacluster-subordinate charm (a +different, newer comment block elsewhere in the file already correctly +said 12). Fixed to state 12 charms carry `:ha` relations, of which 11 +still carry the dual/triple-VIP pattern (designate uses its own +`dnsaas`/DNS-native addressing, not an HAProxy VIP, so it doesn't add to +that specific count) -- both the corrected the 12-count and the +still-accurate 11-VIP-count are now stated together, avoiding a NEW +staleness by conflating the two different numbers. + +## Verification + +`python3 -c "import yaml; yaml.safe_load(open('bundle.yaml'))"`: parses +clean, 56 applications / 108 relations unchanged from before this delivery +(these fixes don't add/remove any application or relation, only fix an +existing one's bindings + move comments + correct prose). `python3 +scripts/provider-bundle-check.py`: PASS, all 6 invariants unaffected +(confirmed the corrected `magnum-dashboard` bindings don't disturb the +"11 charms bind public->provider-public" or "11 clustered VIP triples" +counts, since `dashboard`/`''` aren't `public`/`vip`). `bash scripts/ +repo-lint.sh`: 0 fail, 1 documented legacy warn. + +REVERT: `git checkout HEAD~ -- bundle.yaml` (safe -- no live model has +this bundle applied yet). + +## Next actionable step + +None of these three fixes require any downstream runbook update -- +`magnum-dashboard`'s corrected bindings are exactly what a fresh deploy +would need going forward; no runbook asserted a specific (wrong) binding +for it that would now need correcting to match. diff --git a/docs/archive/changelogs/changelog-20260710-sweep-dcdc-runbook-fixes.md b/docs/archive/changelogs/changelog-20260710-sweep-dcdc-runbook-fixes.md new file mode 100644 index 0000000..b22894c --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-sweep-dcdc-runbook-fixes.md @@ -0,0 +1,77 @@ +# Changelog 2026-07-10 -- Full-project sweep: DC-DC runbook consistency fixes (DOCFIX-171) + +No live infrastructure touched. Part of the full-project sweep (see +`docs/changelog-20260710-sweep-vr0-runbook-opentofu-fixes.md`, DOCFIX-170, +for the sweep's overall scope). This changelog covers findings from a +dedicated agent that re-checked all seven DC-DC Stage runbooks and the +master tracker against the actual repo state after the second gap-closure +wave (DOCFIX-162 through 168). + +## Items + +### 1. DOCFIX-171a -- Stage 1's calculator scripts never wired into Step 3 +FILE: `runbooks/dc-dc-phase0-vcloud-prep.md`. + +WHAT: `docs/dc-dc-deployment-workflow.md`'s gap register item #7 and +`scripts/dc-dc-mtu-geneve-budget.sh`'s own header both asserted "the first +real run is part of executing Step 3" of this runbook -- but Step 3 still +contained only the old manual back-of-envelope arithmetic, never actually +invoking either `scripts/dc-dc-mtu-geneve-budget.sh` or `scripts/dc-dc- +ceph-disk-budget.sh`. Same class as DOCFIX-169: a gap-closure delivery +never got cross-referenced back into the runbook that was supposed to +consume it. Fixed: Step 3 now shows the real invocations (with their exact +flags) in place of the hand-arithmetic instructions, explicitly noting the +calculator's OUTPUT (a downstream tenant-MTU consequence) is distinct from +what actually goes into `underlay_mtu` (the measured value itself). + +### 2. DOCFIX-171b -- Stage 5's stale Sequence-list bracket tag +FILE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. + +WHAT: the top-level Sequence list still read `6. Write + apply the IPv6 +family-matrix overlay [NOT YET WRITTEN -- see below]`, even though Step 6's +own body had already been correctly updated (during DOCFIX-164) to +`[OVERLAY DRAFTED 2026-07-10, NOT YET APPLIED]`. The Sequence-list's own +bracket tag was simply never touched when the Step 6 body was -- a direct, +within-file self-contradiction. Fixed to match. + +### 3. DOCFIX-171c -- Stage 6's checklist overstated its own step content +FILE: `runbooks/dc-dc-phase5-dr-failover-drill.md`. + +WHAT: the delivery checklist claimed Steps 4/5/8/10.2/11.2-11.4 "call +these scripts... instead of typing the command sequences by hand," but +those step BODIES (verified by reading them) still showed only raw +`radosgw-admin`/`rbd mirror pool ...` commands -- no invocation of +`scripts/dc-dc-radosgw-multisite.sh`, `dc-dc-rbd-mirror.sh`, or +`dc-dc-dr-drill.sh` appeared anywhere in Steps 1-12. Fixed by adding a +"PREFER `scripts/dc-dc-...`" pointer at the top of each of the five +affected steps, directing to the corresponding script (with its exact +subcommand) as the safer, tested alternative to typing the raw commands, +while keeping the raw commands as the reference/fallback. Updated the +checklist item itself to describe what's now actually true rather than +what was aspirationally claimed before. + +### 4. DOCFIX-171d -- two DOCFIX numbers never closed out in the session ledger +FILE: `docs/session-ledger.md`. + +WHAT: comparing final in-use numbers (from changelog filenames and +cross-references elsewhere) against the ledger's own tail: DOCFIX-164 (the +IPv6 overlay) and DOCFIX-167 (the bundle.yaml Designate/D-108 change) both +had ledger bullets ending in unresolved "Uncommitted -- orchestrating +session will assign the real DOCFIX number at integration time" boilerplate +-- even though both deliveries HAD been assigned real numbers and committed, +correctly cited everywhere else (the tracker doc, the runbooks, the +changelog file titles themselves). This is an integration-pass omission, +not a numbering collision (no number was reused or double-assigned) -- +fixed by writing the real numbers into both closing lines. A third, lower- +severity instance (gap #15/DOCFIX-166's entry, which already cited the +correct number up front but still carried the same stale closing +boilerplate) was fixed for consistency too. + +## Verification + +`bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. No script +logic touched by this changelog -- doc/runbook prose only. + +REVERT: `git checkout HEAD~ -- runbooks/dc-dc-phase0-vcloud-prep.md +runbooks/dc-dc-phase4-juju-bundle-per-dc.md runbooks/dc-dc-phase5-dr- +failover-drill.md docs/session-ledger.md` (safe). diff --git a/docs/archive/changelogs/changelog-20260710-sweep-script-fixes.md b/docs/archive/changelogs/changelog-20260710-sweep-script-fixes.md new file mode 100644 index 0000000..273b643 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260710-sweep-script-fixes.md @@ -0,0 +1,118 @@ +# Changelog 2026-07-10 -- Full-project sweep: script logic bugs + security hardening (DOCFIX-173) + +No live infrastructure touched. Part of the full-project sweep (see +DOCFIX-170 for overall scope). A dedicated agent read every `.sh` file +under `scripts/` and found 4 confirmed logic bugs, all independently +reproduced in isolation before being trusted -- the same discipline that +caught the `dc-dc-prefixes-import` test bug earlier tonight. A separate +security-audit agent found 1 real hardening gap in a script built +earlier tonight. + +**Notable process point:** the FIRST proposed fix for one of these bugs +(`scripts/phase-06-capi-stack.sh`) was itself WRONG, caught before +committing by reproducing it in isolation: `... | cmd || true` after a +failing pipeline, under `pipefail`, silently DISCARDS the pipeline's real +`PIPESTATUS` array and replaces it with `true`'s own trivial exit status +(0) -- meaning the fix would have made the script's own "GATE FAIL" check +never fire again, silently treating every remote failure as success. That +is WORSE than the original bug (a loud-but-blind abort vs. a silent +false-pass). The correct fix uses the `if pipeline; then :; fi` idiom +instead, verified via the same reproduction technique before committing. + +## Items (all confirmed via isolated `bash -c` reproduction, not asserted) + +### 1. `scripts/carve-host-interfaces.sh` -- `emit()`'s `return 1` killed apply-mode runs on the first MAAS error +Every call site invokes `emit` as a bare, unguarded statement under this +file's `set -euo pipefail`. `emit()`'s own `fail()` call already does the +FATAL-count accounting its caller needs; the subsequent `return 1` was +never consumed by any caller, and under `set -e` it terminated the WHOLE +SCRIPT immediately on the first real MAAS mutation failure -- before +`carve_raw`'s later plane carves, the "resulting interface tree" +verification, or the final "Summary: N fatal" line could run, even though +the script's own design (a FATAL accumulator + end-of-run summary) clearly +intends to report the full picture after a partial failure. Fixed by +removing the redundant `return 1` (with a comment explaining why, to +prevent reintroduction). + +### 2. `scripts/phase-06-capi-stack.sh` -- `run_step`'s `PIPESTATUS` read was dead code +`ssh ... | sed 's/^/ /'` as a bare statement under `set -euo pipefail`: +a real ssh failure makes the pipeline itself fail (via `pipefail`), which +triggers `set -e` and kills the script at THAT line, before +`local rc=${PIPESTATUS[0]}` on the next line is ever reached -- so the +function's own "GATE FAIL: `