Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260723-stage4-open.md
@JANeumatrix JANeumatrix 2 hours ago 10 KB Deploy-model ruled (Ready+jammy, DOCFIX-200), per-DC placement tags applied, mirror ruled BUILD-NOW
Raw Blame History

Changelog -- 2026-07-23 -- Stage 4 open: discovery + D-133 carve ruling

Session scope: open Stage 4 (Phase 3, MAAS enlist/commission/deploy per DC) on branch dc-dc-stage4-phase3-maas-deploy off post-merge main. Bootstrap ran clean (pull up-to-date at bff71d4; repo-lint 0 fail / 1 legacy warn; ledger-scan reconciled -- 12 open SEC rows, D next-free 133). No cloud mutation this session yet; everything below is read-only measurement + repo records.

Item 1 -- Stage 4 OPENED: precondition + discovery (read-only), captured

  • Branch dc-dc-stage4-phase3-maas-deploy created off main (bff71d4).
  • Runbook Step 0 precondition PASS: both DC rack controllers enrolled (vvr1-dc0 7chphy, vvr1-dc1 nmpcq4).
  • Step 2 discovery complete for BOTH DCs, all read-only, captured to docs/audit/stage4-discovery-20260723.txt: 9 nodes per DC, ALL Ready, power=virsh; dc0 = boot fabric-4 / MACs 52:54:00:, dc1 = fabric-142 / pinned 52:54:01:d1:; zone/pool default everywhere (no zone/pool convention exists -- fabric + MAC prefix is the grouping signal). Also captured: one dc0 node's full physical NIC list (6 flat per-plane NICs), default_distro_series=noble, and the synced boot-resources list (noble ONLY -- no jammy images on the region).
  • Runbook-vs-reality deltas recorded in CURRENT-STATE (the runbook predates Stage 3's execution): Step 3 commissioning already satisfied; carve comes BEFORE deploy per the runbook's own Ready-state sequencing note; gap #6 (pod-discovery landing status) is moot -- the pod path was refuted 2026-07-20.
  • Revert: git branch -D dc-dc-stage4-phase3-maas-deploy (after moving off it); delete docs/audit/stage4-discovery-20260723.txt; revert the CURRENT-STATE Stage-4 bullet. No live state was touched.

Item 2 -- D-133 ADOPTED: flat per-NIC carve (VR1); hardware-faithful NIC topology next deployment

  • New entry docs/design-decisions.md D-133 [ARCH], Status ADOPTED 2026-07-23, operator utterance quoted verbatim per GA-R5 ("Fine, keep it flat but on the next deployment we need to build the VMs with exact hardware duplicated networking so we can work out any bonding/trunking kinks.").
  • Part 1: VR1 DC nodes carve every plane FLAT on its dedicated NIC; no VLAN-103 stack in VR1 (supersedes the D-052-era Pattern A shape for VR1 DC nodes only; VR0 untouched).
  • Part 2 (pinned to next deployment): node VMs duplicate target hardware's exact NIC topology (bond members, trunks) -- composes with D-132.
  • Affected-surfaces list recorded IN the entry (buildout-design VID-103 text, workflow Stage 4 row, phase3 runbook Step 5 tree, lib-net vr1-dc0 arm's inherited METAL_INTERNAL_VID) -- queued as this stage's DOCFIX/delivery work, not silently edited.
  • CURRENT-STATE section 1 updated in the same commit (GA-R1 C1): Stage 4 OPEN bullet with discovery facts + the open decision points (deploy OS series, octet plan, mirror gap #5, tag shape).
  • Revert: remove the D-133 block from design-decisions.md and the Stage-4 bullet from CURRENT-STATE.md (single commit; git revert <sha>).

Item 3 -- D-134 ADOPTED: per-role octet bands for VR1 DC node statics

  • New entry docs/design-decisions.md D-134 [ARCH], ADOPTED 2026-07-23. Sequence per GA-R5: the flat .40-.48 proposal was presented and REJECTED ("Different band"); operator directed "Per role bands with headroom. Give me a recommendation"; the recommended band table was presented with measured collision checks (bundle VIPs .50-.60 grep, 10.12.12.4, FIP pool, D-120 dynamic range) and adopted ("Adopt as recommended").
  • Bands: .1-.9 infra (.4-.9 utility-reserved), .10-.19 control, .20-.49 compute, .50-.99 formal VIP band, .100-.200 dynamic (D-120), .201-.254 storage. One octet per node across all six planes, both DCs; VR1 assignments control .10-.12 / compute .20-.21 / storage .201-.204 keyed by tofu node name via pinned boot MAC.
  • CURRENT-STATE Stage-4 bullet updated in the same commit (octet plan RULED; carve now unblocked on D-133 + D-134).
  • Revert: remove the D-134 block + the CURRENT-STATE bullet edit (git revert <sha>).

Item 4 -- D-134 AMENDED: contiguous band table (Option B swap)

  • Operator rejected the original table's interleaving (node roles split by the VIP + dynamic blocks); movability analysis presented (pinned: .1/.2/.3 live, .50-.60 bundle VIPs, FIP rows 1-3 on provider; movable: the D-120-inherited dynamic range). Option B ruled: nodes get the contiguous .100-.200 run (control .100-.119 / compute .120-.149 / storage .150-.200), dynamic moves to .201-.254, utility .4-.49, VIP .50-.99.
  • VR1 assignments now: control .100-.102, compute .120-.121, storage .150-.153 (identity table unchanged -- tofu name via pinned boot MAC).
  • Consequence queued: the dc0/dc1 metal-admin dynamic-range MOVE is the carve's first gated live mutation per DC (before static assignment; nodes powered off). Office1's own D-120 ranges untouched.
  • CURRENT-STATE Stage-4 bullet updated same-commit.
  • Revert: remove the amendment block + CURRENT-STATE edit (git revert <sha>); no live state touched yet.

Item 5 -- Approvals workflow: session-log-keyed permission rules (operator-directed)

  • Operator directed session-length approvals mid-carve (manual mode was prompting per command; the earlier auto-mode classifier BLOCKED the first mutation outright -- the known dc1-session behavior, ledger 2026-07-23).
  • .claude/settings.local.json (uncommitted, personal): ADDED allow Bash(script -aqe ~/as-executed/2026-07-23-stage4-carve.log -c *) (the logged-window wrapper -- inherently session-scoped, the rule names this window's log file) + ask rule keeping maas admin machine * shapes (commission/deploy) prompting inside that same wrapper. PRUNED seven dead rules keyed to retired log files (2026-07-21-ops-commissioning-diag, 2026-07-22-dc1-deploy).
  • FLAGGED for stage close, not acted on: local allow list carries a broad Bash(ssh *) from a prior session -- prune candidate.
  • Review surface stays the as-executed log + this changelog (GA-R2 blanket-approval precedent).
  • Revert: remove the two added rules from settings.local.json (file is local-only, not committed).

Item 6 -- Stage 4 carve EXECUTED, both DCs (logged window stage4-carve)

  • All mutations in ~/as-executed/2026-07-23-stage4-carve.log (per-command wrap; helper scripts' full content logged verbatim before first use). Verification capture: docs/audit/stage4-carve-verify-20260723.txt.
  • Set 0: dc0+dc1 metal-admin dynamic ranges MOVED to .201-.254 in place (iprange ids 2/3; office1 id 1 untouched) -- the D-134-amendment consequence, done before statics.
  • Set 1: 10 named plane fabrics created (vr1-dc0- ids 188-192 / vlans 5189-5193; vr1-dc1- ids 193-197 / vlans 5194-5198).
  • Set 2: 90 NIC re-homes, one node per approval-unit, read-back each: every non-boot NIC moved from its private auto-fabric to its plane fabric; boot NICs stay fabric-4 (dc0) / fabric-142 (dc1). MACs matched config-of-record pins throughout.
  • Set 3: provider subnets 10.12.4.0/22 + 10.12.64.0/22 MOVED off shared fabric-5 to their per-DC provider fabrics (+ ruled gateways .1); 8 plane subnets CREATED (dc0 12/16/32/36, dc1 72/76/80/84).
  • Set 4: 6 spaces created (bundle binding names verified against bundle.yaml first); 12 untagged VLANs assigned (metal-admin = the two boot fabrics). Office1/transit subnets remain in undefined untouched.
  • Set 5: 18 nodes' statics per D-134-amended assignments (control .100-.102, compute .120-.121, storage .150-.153, same octet all six planes) -- five raw-NIC statics + br-ex (OVS, Pattern A) carrying the provider static, provider NIC left raw (D-100). All nodes remain Ready.
  • Residue NOT touched (flagged): fabric-5 still holds 192.168.1.0/24 (factory-LAN residue, both edges re-addressed) and the now-empty ex-provider fabric-5 itself; the ~90 empty auto-created fabrics (fabric-98..187 class) also remain -- cleanup candidates, gated separately at stage close, zero functional impact.
  • Revert: MAAS-record-level; reverse order: unlink statics + delete br-ex per node, delete the 8 created subnets, move the 2 provider subnets back to fabric-5, unassign spaces + delete the 6 spaces, re-home NICs to auto-fabrics (or delete the 10 named fabrics after), move dynamic ranges back to .100-.200. Nodes were powered off throughout; no OS-level state exists to revert.

Item 7 -- Deploy-model ruling + DOCFIX-200 + jammy sync + placement tags

  • RULING (operator: "Ready + jammy images (Recommended)"): Stage 4 ends with nodes READY + carved + tagged; MAAS-deploy is SKIPPED -- Juju provisions Ready machines at Stage 5 and requests the bundle's default-base (ubuntu@22.04). The phase3 runbook's Step 4 as written (maas machine deploy per node) contradicted phase-01's own precondition ("machines Ready/power=off") and would have made all 18 nodes un-allocatable to Juju. [OPS -- encoded in the proven phase-01 precedent; no D-number.]
  • DOCFIX-200: runbooks/dc-dc-phase3-maas-enlist-deploy.md Step 4 rewritten (tag + base-image sync, nodes STAY Ready; ruling quoted), sequence table line + GATE "Nodes deployed" bullet corrected against the buildout-design wording.
  • Jammy images: boot-source selection id=2 (ubuntu/jammy amd64) created on source 1 + import triggered; ubuntu/jammy reads Synced (re-verify at close while sub-resources finish).
  • RULING (operator: "Adopt per-DC tags (Recommended)"): deploy-placement tags are region-qualified per-DC -- openstack-vr1-dc0 / openstack-vr1-dc1 (D-119 naming), applied 9+9, add=9/remove=0 both. Stage 5's per-DC bundle parameterization constrains on them.
  • RULING (operator: "Build the mirror now"): the D-107 per-DC artifact mirror gets BUILT in-stage (rehearsal exception REJECTED); Stage 4's mirror gate bullet stays OPEN until it exists and nodes' reachability is verified. Design presented as its own decision (D-135 candidate).
  • Queued findings: phase-00-maas-standup.sh's hardcoded PLANES table asserts the VR0 VLAN-103 shape and would false-DRIFT against the ruled D-133 flat carve (Step 8 satisfied instead by the spaces cross-check in the carve capture); Step 7's edge-NTP check text is superseded by the D-129(iv) ruling (MAAS hierarchy stays time authority; nodes measured NTP-ing to their rack during Stage 3 commissioning).
  • Revert: runbook edits git revert; tags maas admin tag update-nodes <tag> remove=... x9 + tags delete; boot-source-selection delete id=2 (images age out).