diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b558dfd..1c55625 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -552,6 +552,49 @@ region's sole rack is `hot-kid`. All 10 dc0 machines are `Ready` and POWERED OFF, so the DHCP gap is free to take -- two servers on one segment is the hazard the off-then-on ordering avoids, which is why it is ONE operation. + **THE EXACT UNBLOCK SEQUENCE is in `docs/changelog-20260730-dc0-region-migration.md` + item 10, with every parameter RE-RESOLVED LIVE at session end** (Office1 fabric 4 / vid 0; + new region fabric 0 / vid 0; rack `hot-kid` = **`c3aqh8` IN THE NEW REGION**, `tw7ptw` in + Office1 -- the same machine with two ids, one more proof that ids do not cross regions). + **BOTH PRE-CUTOVER CHECKS WERE DONE AND BOTH MATTERED:** `hot-kid` IS eligible as + `primary_rack` for fabric-0 (its `enp1s0` is on that VLAN, static `10.12.8.6`) -- had it + not been, step 1 would succeed, step 3 would fail, and the segment would have NO DHCP with + ten machines about to need PXE. + **A DEFECT IN THIS SESSION'S OWN APPLY, found by reading interfaces back, now fixed three + ways.** The first `dc-region-topology.sh apply` CREATED the provider-public fabric and + MOVED `10.12.4.0/22` onto it. **MAAS does not bring interface links along on a subnet + move:** the region VM's `enp2s0` stayed on the old VLAN while its subnet left, leaving + `hot-kid` holding a static `10.12.4.6` on a subnet-less VLAN -- the leg it needs for image + sync, and the SAME under-carve class that cost three bootstrap attempts on the Juju + controller. **Live connectivity was UNAFFECTED** (the OS netplan is independent of MAAS's + model), and the check read subnets/fabrics/spaces/tags but never an interface link, so it + **passed 39/39 with the model inconsistent**. Fixed LIVE (subnet returned, wrong fabric + deleted, `fabric-1` RENAMED, space re-bound -- `enp2s0` now reads + `fabric=vr1-dc0-provider-public` with its static intact); fixed IN THE TOOL (RENAME an auto + fabric that already carries the subnet -- renaming moves nothing; a MOVE now warns that + links do not follow); and fixed IN THE GATE (new assertion that every interface link sits + on its subnet's VLAN). **Live re-check now 40 passed / 0 failed, EXIT 0.** + **`dc-node-v6-carve.py` FIXED, not merely recorded:** it now honours `MAAS_PROFILE` + (`--profile` still overrides) and PRINTS which profile it targets and where that came + from; an absent `maas` CLI now REFUSES with exit 2 instead of exiting 1 on a traceback. + Harness 14/14 (was 9), both mutation-proven. + **THE VR1 NODE NIC -> PLANE ORDER IS NOW MEASURED AND GUARDED (`lib-hosts.sh` + `NIC_PLANE_ORDER` + `BREX_PARENT_NIC`).** `macs[i]` in the substrate root IS `enps0`, + verified on `vr1-dc0-control-01` against the live carve: `enp1s0` metal-admin (PXE), + `enp2s0` provider-public (the `br-ex` parent), then metal-internal, data-tenant, storage, + replication. **THIS IS NOT `lib-net.sh`'s `PLANE_CIDRS` ORDER**, which starts with + provider-public -- walking `PLANE_CIDRS` positionally to place NICs would swap `enp1s0` + and `enp2s0` and strand commissioning on a plane with no DHCP. `dc-selector` (88 checks) + asserts the two orders differ at position 0 so the guard cannot go vacuous. + **STILL OWED, enumerated in the changelog's closing section:** the cutover itself; + `scripts/dc-node-carve.sh` (60 NIC re-homes + 9 `br-ex` + 54 statics -- DELIBERATELY not + written blind, since it cannot be exercised until the nodes are re-enrolled and BOTH + defects found today were caught by LIVE runs rather than fixtures); wiring + `maas-profile-assert.sh` into `dc-plane-ipam.sh` / `maas-role-tags.sh` / + `maas-node-power.sh`, which still default to `admin` and could therefore mutate OFFICE1, + where dc1's nine nodes still live; the rack DNS forwarder's upstream; and the two + credential removals (the `vr1-dc0-region` profile on voffice1, SEC-026; Office1's copy of + the dc0 power key). - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/audit/dc0-region-topology-20260730.txt b/docs/audit/dc0-region-topology-20260730.txt index a2b2f7e..0539156 100644 --- a/docs/audit/dc0-region-topology-20260730.txt +++ b/docs/audit/dc0-region-topology-20260730.txt @@ -51,3 +51,13 @@ [ok] tag 'openstack-vr1-dc0' exists dc-region-topology: 39 passed, 0 failed EXIT = 0 + +== RE-CHECKED after the fabric-rename remediation (see changelog item 12) == +$ dc-region-topology.sh check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid + [ok] metal-admin carries node-facing dns_servers (got '10.12.8.6') -- D-131 + [ok] metal-admin allow_dns is False (got 'False') -- D-131 + [ok] metal-admin has a DHCP dynamic range (got 1) + [ok] tag 'openstack-vr1-dc0' exists + [ok] no stranded interface links (every link is on its subnet's VLAN) +dc-region-topology: 40 passed, 0 failed + EXIT = 0 diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index 2e7273d..fa90982 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -400,6 +400,120 @@ **No revert needed** -- nothing was changed. +**THE EXACT UNBLOCK SEQUENCE, every parameter RE-RESOLVED LIVE at session end.** +Run these on the jumphost with the `!` prefix so the output lands in-session. The +`vr1-dc0-region` profile needs the tunnel, which was verified UP at capture time +(`pgrep -f '5241:10.12.8.6'`); if it has since died, re-open it with +`ssh voffice1 "ssh -f -N -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2"`. + + # 1. OFF at Office1 (fabric-4 / vid 0 -- dc0 metal-admin, dhcp_on=True, primary_rack=7chphy) + ssh voffice1 'maas admin vlan update 4 0 dhcp_on=false' + + # 2. VERIFY BY PROCESS, not by MAAS self-report (the 2026-07-20 Temporal incident class) + ssh vr1-dc0-rack 'pgrep -c dhcpd; ps -ef | grep "[d]hcpd"' # expect 0 dhcpd + + # 3. ON at the new region (fabric-0 / vid 0, rack hot-kid = c3aqh8 IN THAT REGION) + ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8' + + # 4. VERIFY BY PROCESS again -- expect exactly one dhcpd, now owned by the region VM + ssh vr1-dc0-maas 'pgrep -c dhcpd; ps -ef | grep "[d]hcpd"' + + # 5. Re-run the named gate; it should stay 40/0 + ssh voffice1 'cd ~/openstack-caracal-dc-dc && bash scripts/dc-region-topology.sh \ + check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid' + +**PRE-CUTOVER CHECKS, both done, both were genuinely load-bearing:** +- **`hot-kid` IS eligible as `primary_rack` for fabric-0/vid 0** -- its `enp1s0` is on that + VLAN (`vlan_row=5001`, static `10.12.8.6`). If it were not, step 1 would succeed, step 3 + would fail, and the segment would have NO DHCP with ten machines about to need PXE. That + is the one-way risk in the sequence and it is answerable read-only, so it was. +- **`hot-kid`'s system_id is `c3aqh8` in the NEW region and `tw7ptw` in Office1** -- the same + machine, two different ids. One more proof that ids do not cross regions. + +--- + +## Item 12 -- DEFECT IN MY OWN APPLY, found by reading interfaces back, and fixed + +**What went wrong.** `dc-region-topology.sh apply` CREATED `vr1-dc0-provider-public` and +MOVED `10.12.4.0/22` onto it. **MAAS does not bring interface links along on a subnet +move.** The region VM's own `enp2s0` stayed on the old auto fabric's VLAN while its subnet +left, so `hot-kid` was left holding a static `10.12.4.6` on a VLAN with no subnet -- the +region's provider-public leg, which it needs for image sync, and the **identical under-carve +class that cost three bootstrap attempts on the Juju controller**. + +**Why nothing caught it.** Live connectivity was UNAFFECTED (the OS netplan is independent +of MAAS's model -- `default via 10.12.4.1`, egress 200), and the check read subnets, +fabrics, spaces and tags but never an interface link. **It passed 39/39 with the model +inconsistent.** + +**Fixed three ways:** +1. **Live:** subnet returned to vlan 5002, the wrongly-created fabric deleted, `fabric-1` + RENAMED to `vr1-dc0-provider-public`, space re-bound. `enp2s0` now reads + `fabric=vr1-dc0-provider-public` with its static intact. +2. **In the tool:** it now RENAMES an auto fabric that already carries only this plane's + subnet -- renaming moves nothing. A MOVE survives only for the case where the fabric + carries other planes, and now prints a warning that links do not follow. +3. **In the gate:** a new assertion that every interface link sits on its subnet's VLAN. + Live re-check is now **40 passed / 0 failed, exit 0**. + +Harness 47/47; the rename path and the stranded-link check each proven able to fail. One +older assertion asserted the CREATE behaviour and was RE-POINTED rather than deleted, per +the standing rule. + +--- + +## Item 13 -- `dc-node-v6-carve.py`: env-blindness and traceback fixed + +Item 6(b) recorded this as a foot-gun; it is now fixed rather than only noted, because this +migration will invoke it. `PROFILE_DEFAULT` now reads `MAAS_PROFILE` (with `--profile` still +overriding), and every run PRINTS which profile it targets and where that came from -- a +capture that does not name the profile cannot be audited for wrong-region work. Separately, +an absent `maas` CLI raised `FileNotFoundError` and exited **1 with a traceback**, +indistinguishable from a real failure and contrary to this script's own REFUSE contract; it +now exits **2**. Harness 14/14 (was 9); both fixes mutation-proven. + +--- + +## Item 14 -- the MEASURED VR1 node NIC -> plane order (`lib-hosts.sh`) + +The per-node v4 carve has no repo tool, so the next step was about to re-derive this by +hand. MEASURED instead, end to end: the substrate roots pin six MACs per node and the module +attaches them in list order, so `macs[i]` IS `enps0`. Verified on +`vr1-dc0-control-01` against the live carve -- +`enp1s0` metal-admin (PXE), `enp2s0` provider-public (the `br-ex` parent, Pattern A), +`enp3s0` metal-internal, `enp4s0` data-tenant, `enp5s0` storage, `enp6s0` replication. + +**THE TRAP, now guarded: this is NOT `lib-net.sh`'s `PLANE_CIDRS` order**, which starts with +provider-public. Anything walking `PLANE_CIDRS` positionally to place NICs swaps `enp1s0` +and `enp2s0` and puts the PXE leg on provider-public -- stranding commissioning on a plane +with no DHCP. `NIC_PLANE_ORDER` + `BREX_PARENT_NIC` are recorded with the evidence, and +`dc-selector` (88 checks) asserts the two orders differ at position 0, so the guard cannot +go vacuous. + +--- + +## STILL OWED (explicit, so nothing is lost) + +1. **The DHCP cutover** -- item 10, blocked on the permission wall. +2. **`scripts/dc-node-carve.sh` does not exist** -- 60 NIC re-homes + 9 `br-ex` bridges + 54 + v4 statics, the largest remaining no-tool gap. The two hard inputs are now settled + (item 14's ordering, and `carve-host-interfaces.sh:211-236`'s `build_ovs_brex()` call + sequence, which the recon flagged as the correct MAAS ordering: unlink commissioning + links -> move the parent NIC onto the provider VLAN -> `interfaces create-bridge + bridge_type=ovs` -> `link-subnet mode=STATIC` on the bridge). **Deliberately NOT written + blind:** it cannot be exercised until the nodes are re-enrolled, and both defects found + today (the stranding bug, the fixture mismatch) were caught by LIVE runs, not by + fixtures. Write it against a region it can actually run against. +3. **`maas-profile-assert.sh` is wired into `dc-region-topology.sh` only.** + `dc-plane-ipam.sh`, `maas-role-tags.sh` and `maas-node-power.sh` still default to + `MAAS_PROFILE=admin`. A forgotten `export` on any of them mutates OFFICE1 -- which still + owns dc1's nine live nodes, so a mis-targeted `maas-role-tags.sh apply` would retag the + wrong DC. Until they are wired, run the assert in the same command immediately before + each and put it in the capture. +4. **The rack DNS forwarder's upstream** -- item 9's follow-up. +5. **Remove the `vr1-dc0-region` profile from voffice1** once dc0's control host settles + (SEC-026), and **remove Office1's copy of the dc0 power key** once the migration lands. + --- ## Item 11 -- as-executed log NOT used for this window, and why