Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260724-stage5-committee-fork1.md
@JANeumatrix JANeumatrix 13 hours ago 18 KB DOCFIX-202: demote buildout-scope section 5 to a D-111 pointer
Raw Blame History

Session changelog 2026-07-24 -- Stage-5 bundle-render committee + Fork-1 ruling

Branch dc-dc-stage4-phase3-maas-deploy. Session resumed the DC1 Stage-5 bundle-rework blocker under a committee review (operator-requested). Status lives ONLY in docs/CURRENT-STATE.md (GA-R1); this changelog is the review surface -- each item states what / why / revert. No cloud state was mutated this session.

Item 1 -- Committee record (Track 2 bundle render)

What: new docs/audit/committee-20260724-track2-bundle-render.md -- a 4-lens (Roosevelt-delta / HA-DR / operational-risk / Juju-MAAS-mechanism) read-only committee review of the vr1-dc1 deploy-artifact render, with the adjudication and open operator decisions. Advisory/audit record; adopts nothing (GA-R5). Why: operator: "Deploy a committee to review Track 2 and provide recommendations based on overall end project goals and decisions already researched." Durable, citable authority (commissioning-committee precedent). Revert: git rm docs/audit/committee-20260724-track2-bundle-render.md.

Item 2 -- Capacity gate capture (Fork-1 option a)

What: new docs/audit/stage5-controller-capacity-20260724.txt -- captured scripts/dc-dc-whole-host-budget.py runs: baseline built layout 3+2+4 x2 DC = 838/1024 GiB (82%, FIT); +controller (+8 GiB/+4 vCPU per-DC containment overhead) = 854/1024 GiB (83%, FIT, 170 GiB headroom). Read-only calculator; nothing queried live. Why: GA-R1 -- the Fork-1 (a) capacity claim must be MEASURED, not inferred (the signed doc's 790/77% is the OLD 8-node layout). Gate = PASS. Revert: git rm docs/audit/stage5-controller-capacity-20260724.txt.

Item 3 -- D-104 amendment (Fork-1 ruling: dedicated 10th controller VM)

What: appended a D-104 amendment (2026-07-24) in docs/design-decisions.md recording the operator's GA-R5 ruling -- the per-DC Juju controller is a DEDICATED 10th node VM (distinct tag juju-controller-vr1-dc1, no role tag), separate from the 9 Option C role nodes; additive (for_each-keyed); capacity gate PASS (cites item 2); the 9-node D-121 layout is unchanged. Why: Fork 1 was the blocking architectural decision; operator selected "Dedicated 10th VM (Recommended)" over shrink-a-role and co-locate. Controller placement is D-104's subject; recorded as its amendment (not a new D-number). Revert: delete the "### D-104 -- AMENDMENT (2026-07-24)" block in docs/design-decisions.md (git range-diff of this commit).

Item 4 -- CURRENT-STATE Stage-4 bullet update (GA-R1 / C1)

What: rewrote the Stage-4 bullet's render sub-bullets: committee-adjudicated; Fork 1 RULED (a) + capacity gate; Fork 2/3 as OPS committee-unanimous recommendations (gated at execution); BUNDLEFIX-052 fixed; a corrected NEXT list. Also corrected an earlier inaccurate characterization (dc-ha-scaleup is not "stale ceph-osd 3" -- it deliberately excludes ceph-osd) and the "consumes 1 of 9" arithmetic (now a dedicated 10th VM). Why: GA-R1/C1 -- a status-changing commit updates CURRENT-STATE in the SAME commit (repo-lint L10). Status authority stays single-sourced. Revert: restore the prior Stage-4 bullet text (git range-diff of this commit).

Item 5 -- BUNDLEFIX-052: ceph-rbd-mirror duplicate bindings:

What: removed an orphaned 4-space-indented bindings: block in bundle.yaml (old line ~907) that yaml.safe_load take-last had absorbed into ceph-rbd-mirror, clobbering its real ceph-local/ceph-remote (replication) + nrpe-external-master bindings (D-108/D-052) and injecting a phantom dashboard binding (a charm with no such endpoint) -> juju deploy validation reject. Why: confirmed pre-existing latent defect (committee mechanism lens; verified via yaml.safe_load). Fork-independent; fix now so the dc1 render starts clean. Validation: yaml.safe_load confirms ceph-rbd-mirror.bindings restored to {'': metal-admin, ceph-local: replication, ceph-remote: replication, nrpe-external-master: metal-internal}; provider-bundle-check.py bundle.yaml PASS. Revert: re-insert the orphan block after the vault-hacluster comment (not advised -- it is the bug).

Item 6 -- provider-bundle-check.py: overlay merge + per-DC VIP bands + placement invariants

What: extended scripts/provider-bundle-check.py with (a) --overlay FILE (juju-like merge, so the checker validates the EFFECTIVE deploy input), (b) --dc vr1-dc0|vr1-dc1 selecting expected VIP bands (default dc0 = base bundle, so an un-parameterized run is byte-identical to before), and (c) placement invariants that activate ONLY on a role-separated bundle (machines tagged control/compute/storage) and self-skip on the base/un-rendered bundle: no dangling to:; LXD containers on control, ceph-osd on storage, nova-compute on compute; each num_units>=2 app spread across that many DISTINCT machines (anti-affinity); ceph-osd == storage count, nova-compute == compute count. Dropped the "controller-tag" idea (the controller is the bootstrap machine, not a bundle machine -- verified at the MAAS-tagging step). Why: the committee's non-negotiable "spine" -- the checker was placement-blind, so any render could silently ship decorative HA. Makes the hand-render machine-checked; also closes the committee's dc1-VIP-outside-the-check gap. Caveat (VERIFY-LIVE): the checker's machines-block overlay merge MIRRORS juju's documented behavior; the actual machines-retag-via-overlay is confirmed at juju deploy --dry-run before deploy (no offline juju merger exists). Fallback if it does not take: inline the dc1 tags into a rendered bundle. Validation: harness tests/provider-bundle-check/run-tests.sh 15/15 (7 new behavioral cases). Gauntlet ALL GREEN (78). repo-lint 0 fail. Revert: git checkout <this-commit>~ -- scripts/provider-bundle-check.py tests/provider-bundle-check/run-tests.sh.

Item 7 -- vr1-dc1 bundle render (9-node role-separated) + retag overlay

What: rendered bundle.yaml from VR0's 4-node hyperconverged layout to VR1 dc1's 9-node role-separated one: machines block "8"-"11" (tags=openstack) -> "0"-"8" with role+DC tags (block style -- flow {...} would split the comma-tag); every app to: re-homed (mysql/ovn-central/ceph-mon -> control 0/1/2; nova-compute 3->2 on compute 3/4; ceph-osd -> storage 5/6/7/8; the 14 scaled apps -> control placeholder that the overlay spreads; utilities -> control); ovn-chassis MACs -> the 2 dc1 compute provider NICs. Rendered overlays/dc-ha-scaleup.yaml tokens -> logical control ids 0/1/2 (now DC-neutral) + updated its header + fixed the stale ceph-osd-3 comment. New overlays/vr1-dc1-machines.yaml (per-DC retag; machines-overlay merge is VERIFY-LIVE at juju deploy --dry-run). Why: the committee-adjudicated render (Fork 2 = hand-render now); unblocks the dc1 Stage-5 deploy. Done as an asserted one-shot line-transform (preserves all D-NNN comments) reviewed via git diff + the validator. Validation: provider-bundle-check.py bundle.yaml (base standalone) PASS; the FULL dc1 deploy input (--overlay vr1-dc1-machines --overlay vr1-dc1-vips --overlay dc-ha-scaleup --dc vr1-dc1) PASS; harness 15/15; gauntlet ALL GREEN (78); repo-lint 0-fail. The validator CAUGHT a real render bug (YAML flow-comma dropping the role tag) -- exactly its purpose. Revert: git checkout <this-commit>~ -- bundle.yaml overlays/dc-ha-scaleup.yaml + git rm overlays/vr1-dc1-machines.yaml.

Item 8 -- provider-bundle-check ovn-chassis role-sep branch + expansion review record

What: (a) provider-bundle-check.py -- computes role_sep early; the ovn-chassis MAC check now skips the VR0 openstack0-3 Pattern-A set on a role-separated bundle (verifies present + well-formed instead), since dc chassis are the compute-node provider MACs. (b) new docs/audit/stage5-expansion-review-20260724.md -- three read-only reviewer verdicts (relations/bindings/cross-DC; config/pins; deploy-scripts), dispositioned (handled / follow-up / VERIFY-LIVE / resolve-before-deploy / no-change). Why: the operator-requested deployment-script expansion review; captures the cross-connection/pinning/gate findings durably so the Chat design session (NetBox-deploy coupling) reads them from the repo. Validation: harness 15/15 (T9's role-sep fixture exercises the ovn role-sep branch); gauntlet ALL GREEN (78). Revert: git checkout <this-commit>~ -- scripts/provider-bundle-check.py + git rm docs/audit/stage5-expansion-review-20260724.md.

Item 9 -- office1-netbox RECORD dumper (source separation) + current snapshot

What: new netbox/office1-record-dump.py -- a dedicated READ-ONLY dumper for the office1-netbox RECORD (the VR1 working apex, 10.10.1.10:8000, 4.6.4, v2 token), separate from netbox/prod-draft-dump.py (which pulls the FROZEN v1 reference netbox.baldurkeep.com). Operator-directed source separation ("one script per source" -- each holds only its own credential/URL; the token never travels, read from env). Added a source-clarity note to prod-draft-dump.py's docstring. New offline harness tests/office1-record-dump/run-tests.sh (5/5: slim / prefix-scope rewrite / range key / source-target). Committed the current snapshot netbox/draft/vr1-office1-current-20260725.json. Why: a Chat design session (NetBox->deploy coupling) has no live infra; the existing vr1-draft.json is the FROZEN reference (dated 07-15), NOT the current in-cloud state. This captures the live apex so Chat designs against current data. Discovery: the standard dump targets baldurkeep, not office1-netbox -- corrected before committing wrong-source data. Validation: dump succeeded READ-ONLY (139 prefixes / 11 sites incl. vr1-dc0/dc1/off1; dc1 bands 10.12.64/68/72.0/22 present -- confirms current buildout; differs from the 90-prefix frozen reference). Harness 5/5; nothing written to NetBox. Revert: git rm netbox/office1-record-dump.py netbox/draft/vr1-office1-current-20260725.json tests/office1-record-dump/run-tests.sh + restore prod-draft-dump.py docstring.

Item 10 -- dc-mirror upstream-host preference note (mirror.uoregon.edu)

What: added an UPSTREAM HOST NOTE to scripts/dc-mirror.sh's header (operator, 2026-07-25): PREFER http://mirror.uoregon.edu/ubuntu/ (closest/fastest for this deployment) over archive.ubuntu.com for the MAIN archive debmirror -- future change, NOT yet applied. Scoped: main archive only (--host=mirror.uoregon.edu, root stays ubuntu); the UCA cloud-archive uses a separate host uoregon does not mirror; applies to dc-cache-proxy.sh + the Roosevelt dedicated mirror too; verify jammy coverage before flipping. Header comment only -- does NOT change the generated <site>-mirror-sync unit. Why: operator-provided performance preference, logged for a future mirror-tuning session (findings LOGGED, not executed mid-step). Revert: delete the UPSTREAM HOST NOTE block in scripts/dc-mirror.sh.

Item 11 -- full decision reconciliation (record)

What: new docs/audit/decision-recon-20260725.md -- a 4-lens read-only recon of every ruled IP/VLAN/node decision against the authoritative surfaces (office1-netbox apex snapshot, lib-net/lib-hosts, bundle/overlays, MAAS carve captures). CURRENT-STATE Stage-4 bullet updated (GA-R1/C1) with a pointer + the key findings + a WARNING that this doc's section 4 is stale. Why: operator-requested full decision recon + "what other reconciliations to queue." Feeds the Chat NetBox->deploy coupling design and the stage close. Findings (headline): IP prefix-layer COMPLETE + drift-free; VLANs correctly absent (untagged-per-fabric, D-133); GAP = sub-prefix (D-134 bands + 33 VIP addresses, in prose/overlays only) -- blocks the coupling. Node/MAC/power drift-free; Fork-3 tags + 10th VM pending gated. Decision-status: STALE surfaces flagged (CURRENT-STATE section 4 HIGH; G14 SEC 12->15; phase-4 designate) -- fixes QUEUED (operator-deferred). Recon backlog P0-P2 in the record; extend sandbox-fidelity-check.py first. Revert: git rm docs/audit/decision-recon-20260725.md + restore the CURRENT-STATE Stage-4 recon bullet.

Item 12 -- Ruling 1 recorded (GA-R5): address-family policy

What: recorded the operator's 2026-07-25 address-family ruling as a dated RULING NOTE under ## D-101 in docs/design-decisions.md (verbatim question + utterance) + resolved the CURRENT-STATE resolve-before-deploy bullet (b). Confirms D-101's matrix; NO amendment. GA-R5 record. Question: v4-only vs dual-stack for dc1 phase-4. Operator utterance, verbatim: "Dual stack to be used where IPv4 is required, IPv6 where IPv6 only makes sense." Clarifying, verbatim: "Dual stack deployment for DC0 and DC1. This is the deployment when the dual-stack is added". Why: GA-R5 -- the VIP reconciliation (dependent work) is invalid until this is committed+pushed. Closes the v4-only phasing option for both DCs. From the Chat handoff pack (_handoff-not-committed/20-ruling-records.md, ruling 1); the verbatim was transcribed into durable homes (D-101 note + here) rather than left in the delete-after-use pack file (committee BLOCKER-1 fix). Revert: git revert <commit> -- deletes the D-101 ruling note + restores bullet (b).

Item 13 -- Ruling 2 recorded (GA-R5): keystone policy-override VERIFY-LIVE gate

What: added a Stage-4 VERIFY-LIVE gate to CURRENT-STATE -- after dc-ha-scaleup scales keystone to 3, juju status must show PO: on EVERY keystone unit, never PO (broken):. GA-R5 record. Question: whether to add a VERIFY-LIVE gate confirming the policyd-override resource lands on every keystone unit after scale-up (a broken override is atomic -- the whole policy is discarded, silently reverting every tenant domain-manager to a plain user). Operator utterance, verbatim: "dc-ha-scaleup: Yes to the verify-live". Why: highest-consequence single check in the deploy -- it protects the SCS Domain Manager persona (D-051/D-064) the commercial tenant self-service model rests on, and failure has no other symptom. Mechanism intact at HEAD (bundle.yaml use-policyd-override: true + resources: policyd-override -> policies/overrides.zip 3419 bytes; keystone-policy-drift.sh byte-matches source) -- the gate confirms it SURVIVES scale-up. From the Chat handoff pack ruling 2; verbatim transcribed to a durable home (here), not left in the delete-after-use pack file. Revert: git revert <commit> -- removes the gate line.

Item 14 -- D-136 PROPOSED (NetBox-coupled render pipeline) + Ruling 3 recorded (GA-R5)

What: appended D-136 (PROPOSED, [ARCH]) to docs/design-decisions.md register tail; CURRENT-STATE Stage-4 bullet updated SAME COMMIT (C1) -- bullet (c) artifact-shape, the coupling bullet -> "PROPOSED as D-136 / option (C)", the FOLLOW-UP gate blockers. Ruling 3 is recorded INLINE in D-136's "Artifact shape (RULED 2026-07-25)" paragraph with the verbatim question + utterance. GA-R5 (Ruling 3). Question: symmetric (extract dc0 VIPs to an overlay, base VIP-free) vs asymmetric (dc0 v6 into base, two renderer modes)? Operator utterance, verbatim: "Your question about 1 or 2, go with 1. Base bundle and everything else is applied via overlay." Effect: symmetric per-DC overlays; bundle.yaml VIP-free; gates the dc0 VIP extraction. Pack-review committee corrections folded in (2026-07-25): verbatim inlined into durable homes (D-136 paragraph + here), not the delete-after-use pack file (BLOCKER-1); added the VIP_OCTET_MAX=60 sibling constant + the 2nd stale VIP_COUNT_EXPECT site; mis-citation count corrected to SEVEN. Chat correction (2026-07-25) folded in: open question 5 rewritten -- Jenkins IS the runner (gap VOID); real requirements are webhooks + status-back (unverified vs GitBucket); placement OPEN (voffice1/VR1, per-DC agents Roosevelt, D-132/D-105 caveats); carry-forward handoff-20260703-open-items.md:159. Why: escalates the Fork-2 renderer deferral; scopes the sub-prefix gap fix. PROPOSED -- operator rules the mechanism. Revert: git revert <commit> -- deletes D-136 + restores the CURRENT-STATE bullets.

Item 15 -- DOCFIX-201: the dual-VIP decision is D-020, not D-036 (SEVEN sites)

What: corrected SEVEN VIP-context citations of D-036 where D-020 is meant (## D-020: Dual provider + metal API VIPs; ## D-036 is the magnum-capi-helm decision): docs/design-decisions.md (3: the D-121 "problem this fixes" per D-036, the D-009, D-036 HA-scaleup table row, the References D-036 (dual VIPs)), docs/CURRENT-STATE.md (the vault-VIP bullet), docs/audit/decision-recon-20260725.md (2 backlog rows), and docs/audit/stage5-expansion-review-20260724.md (in-place with the bracketed marker [corrected DOCFIX-201: D-020, not D-036], operator-chosen). Why: the mis-citation had spread to SIX sites (2 created by the recon itself); the 7th (the D-009, D-036 table row) was caught by the pack-review committee -- the pack listed only 6 and its own verify count would have failed at 9. Reading the correct D-020 also surfaced that D-020's ratified set includes vault + excludes ceph-radosgw while the deployed set does the opposite (fed into D-136). Verify: git grep -n "D-036" -- ':!docs/archive' ':(exclude)_handoff-not-committed/*' -> 8 hits, all driver/chart/CAPO, zero VIP-context. Revert: git revert <commit> -- text-only across four files, no behavioural effect.

Item 16 -- DOCFIX-202: buildout-scope section 5 demoted to a D-111 pointer

What: docs/dc-dc-netbox-buildout-scope.md section 5 presented the per-DC v6 subcarve (sub-decision 1) as open/un-ratified at lines 5, 132, 138 -- contradicting the ratified pointer already present at line 146 ("RATIFIED as D-111 (2026-07-11)"). Demoted the three open-framing sites to acknowledge D-111 (ADOPTED); kept the reasoning below as history. Why: ## D-111 is ADOPTED 2026-07-11 and names this exact sub-decision; the stale open framing misled the Chat seat 2026-07-25 (it treated the subcarve as a blocking trust gate, withdrawn only after reading the register). Committee-confirmed the targets; the rest of section 5 has no other adopted-decision contradiction. Verify: bash scripts/ledger-scan.sh -- D-111 not under PROPOSED/OPEN; grep -n "un-ratified\|candidate D-111" docs/dc-dc-netbox-buildout-scope.md -- no hit presenting the subcarve as open. Revert: git revert <commit> -- single-file text change, no behavioural effect.

Next (not done this session; gated)

Commit+push this batch (GA-R5: rulings land before dependent work). Then, each operator-gated: per-role MAAS tags; author + apply the 10th controller VM (egress OPEN); render the 9-node bundle.yaml (re-home base placements, re-scope tokens) + extend provider-bundle-check.py with placement assertions + wire overlays + de-stale phase-4 Step 4; gated juju bootstrap (by controller tag, egress OPEN) -> deploy. Deploy-time VERIFY-LIVE gates: vault-3-unit HA-on-MySQL, hacluster cluster_count:3, dc1 VIPs, CRUSH failure-domain=host.