diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index a55a019..98a8065 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -214,6 +214,39 @@ would have hurt most -- a missing `-days 730` (openssl defaults to **30 days**) and a dropped EKU (one-directional mTLS failure at LB-create time, every gate reading PASS). +**A PER-DC MAAS REGION MIGRATION HAS AN ORDERING THAT IS NOT THE OBVIOUS ONE, AND FOUR OF ITS +STEPS HAVE NO REPO TOOL (RULED D-132 q1; executed for dc0 2026-07-30).** MAAS cannot move +machines between regions, so the carve is RECREATED and the machines RE-ENROLLED. Hard-won +shape: **(i) prove WHICH REGION a profile resolves to before every command** -- +`scripts/maas-profile-assert.sh` asserts by RACK-CONTROLLER IDENTITY, because a machine COUNT +is not proof and against the wrong profile an apply is an idempotent no-op that prints PASS +while `machine delete` destroys the real record. Repo-wide `MAAS_PROFILE` defaults to `admin` += the OFFICE1 region. **(ii) Numeric MAAS ids do NOT cross regions** -- VLAN row 5005 is dc0 +metal-admin in one region and `vr1-dc0-data-tenant` in the other; the same host is `tw7ptw` +there and `c3aqh8` here. Stable keys are fabric NAME, space NAME, subnet CIDR, tag NAME, BOOT +MAC, interface NAME. A pre-migration capture is a REBUILD REFERENCE, never a diff target. +**(iii) Per-node ordering: `delete` (old region) -> `virsh start` BY HAND (nothing else can +start a machine the new region has never seen) -> self-enlist -> SET POWER -> `abort` -> +`commission`. NEVER `commission` a machine whose `power_type` is unset** -- it passes every +script and drops back to `New` with zero events. **(iv) Enlist in BATCHES OF THREE**; nine at +once produced 15 minutes of silence and zero records while egress, the MAAS proxy, region load +and node liveness all measured healthy. **(v) A fresh region has ZERO SSH KEYS** -- MAAS +injects them into deployed machines and `juju bootstrap` needs SSH, so this reproduces the +"cannot SSH the machine" bootstrap failure exactly; import before deploying. **(vi) The node +carve is ROLE-DEPENDENT** -- role nodes take all six planes plus an OVS `br-ex` parented on +`enp2s0` (the static lands on br-ex, the NIC holds no L3 link); the juju controller takes only +metal-admin + provider-public raw WITH the gateway, no br-ex, `enp3-6` left on auto VLANs. +`lib-hosts.sh` carries `NIC_PLANE_ORDER`, which is **NOT** `lib-net.sh`'s `PLANE_CIDRS` order: +walking PLANE_CIDRS positionally swaps `enp1s0`/`enp2s0` and strands PXE on provider-public. +**(vii) `maas-role-tags.sh` PASSES while the `openstack-` PLACEMENT tag is still absent** +-- it is not in its ROLES set and has no creator in the repo, yet `bundle.yaml` places on it +and `dc-node-v6-carve.py` dies without it. **(viii) `phase-00-maas-standup.sh` must NOT be +pointed at a new region** -- unlike `carve-host-interfaces.sh` and `reenroll-hosts.sh`, which +REFUSE `vr1-*` outright, it does not refuse and would create space-named fabrics and a VID-103 +metal-internal that D-133 abolished. Tools built for this: `maas-profile-assert.sh`, +`maas-region-power-key.sh`, `dc-region-topology.sh`; `dc-plane-ipam.sh` and `maas-role-tags.sh` +already worked once given `MAAS_PROFILE`. + **A FINDING IS AN OBSERVATION, NOT A CONCLUSION -- MEASURE BEFORE YOU PUT IT TO THE OPERATOR (2026-07-27 Stage-5 grounding audit; the second-most productive move after the one below).** A seven-lens read-only committee produced findings that were reliable AS OBSERVATIONS and diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 8cf2c95..060b556 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -682,6 +682,16 @@ MAAS vlan as its v4 twin and READ BACK. The v6 half needs the NetBox apex, which is the reason the voffice1->rack tunnel architecture exists. Its own forced sequencing held: the `reserve` run SKIPPED every v6 band with "not in MAAS yet -- run carve-v6 first". + **SESSION SWEEP AT CLOSE: `docs/audit/queued-findings-20260730-dc0-region-migration.txt` + (F1-F10).** THREE items lived ONLY in the transcript and would have been lost: **F1** this + session's permission rules exist only in the GITIGNORED `.claude/settings.local.json`, so + their verbatim text is now recorded (including a broad `Bash(ssh vr1-dc0-rack *)` that an + interactive approval auto-added and which deserves review); **F2** the Office1 region STILL + registers a rack controller on the dc0 rack (`vvr1-dc0`/`7chphy`) and still holds the dc0 + region VM's machine record (`hot-kid`/`tw7ptw`) -- neither is harmful (Office1's DHCP for + that VLAN is off, verified by process) but both are dual-registration cleanup the operator + should rule on; **F10** the five instrument errors of this session as ONE pattern, with the + rule that catches them. F9 carries the next session's first commands verbatim. **WHAT REMAINS FOR THE CARVE, and it is now the ONLY thing between here and the deploy:** `enp1s0` is correctly on `fabric-0` (metal-admin -- it PXE'd there) with an AUTO link on `10.12.8.0/22`, while `enp2s0..enp6s0` each landed on a FRESH auto-created fabric (`fabric-7..11`) with `link_up` diff --git a/docs/audit/queued-findings-20260730-dc0-region-migration.txt b/docs/audit/queued-findings-20260730-dc0-region-migration.txt new file mode 100644 index 0000000..9f99191 --- /dev/null +++ b/docs/audit/queued-findings-20260730-dc0-region-migration.txt @@ -0,0 +1,140 @@ +QUEUED FINDINGS -- dc0 per-DC MAAS region migration (2026-07-30, session part 5) +================================================================================ +Swept at session close. Items are LOGGED, NOT EXECUTED (hard rule 1). Each says where +it already lives, or that this file is its FIRST surface. Precedent: queued-findings +-20260727, -20260729, -20260730, -20260730-stage5. + +THE POINT OF THIS SWEEP is the items marked FIRST SURFACE -- they existed only in the +session transcript and would have been lost on a context clear. + + +F1. THE SESSION'S PERMISSION RULES LIVE ONLY IN A GITIGNORED FILE. + FIRST SURFACE for the verbatim rule text (the changelog describes them; it does not + record them reproducibly). + `.claude/settings.local.json` is gitignored (`~/.config/git/ignore:1`), so if this + clone or the jumphost is rebuilt, every rule added this session is GONE and the next + session hits the same classifier walls with no record of what unblocked them. + Added to "ask" (operator-ruled ask, NOT allow -- hard rule 3 stays intact): + Bash(ssh *'maas * update*) + Bash(ssh *"maas * update*) + Bash(ssh *'maas * ipranges *) + Bash(ssh *"maas * ipranges *) + Added to "allow" (read-only process inspection): + Bash(ssh *'pgrep *) Bash(ssh *"pgrep *) + Bash(ssh *'ps -ef*) Bash(ssh *"ps -ef*) + Bash(timeout * ssh *'pgrep *) Bash(timeout * ssh *"pgrep *) + Bash(timeout * ssh *'ps -ef*) Bash(timeout * ssh *"ps -ef*) + ALSO PRESENT and NOT added deliberately by me: `Bash(ssh vr1-dc0-rack *)` in "allow", + auto-recorded from an interactive approval. That is BROAD -- it permits any command on + the dc0 rack, including destructive ones -- and is worth an explicit review. + WHY THE RULES WERE NEEDED AT ALL: existing rules pin the DOUBLE-quote form + (`ssh voffice1 "maas admin subnet *`) and the `maas admin` profile; the migration used + SINGLE quotes and `maas vr1-dc0-region`. Neither matched, so calls fell through to the + classifier. Quoting style and the profile token are BOTH part of the match. + +F2. OFFICE1 STILL RUNS A RACK CONTROLLER ON THE dc0 RACK, AND STILL HOLDS THE REGION VM'S + MACHINE RECORD. FIRST SURFACE for the rack half. + MEASURED at close: `maas admin rack-controllers read` -> voffice1 (mtstwf), + **vvr1-dc0 (7chphy)**, vvr1-dc1 (nmpcq4). So the dc0 rack host is STILL registered as + an Office1 rack controller even though the per-DC region now owns all ten dc0 machines + and serves their DHCP. + NOT CURRENTLY HARMFUL: Office1's DHCP for that VLAN is OFF (verified by process, + `pgrep -c dhcpd` = 0 on the rack), so there is no two-server hazard. But it is a + dual-registration that a later session will find confusing, and the rackd is dead + weight on the rack host. + ALSO: `hot-kid`/`tw7ptw` (the dc0 region VM) remains a Deployed machine record in + OFFICE1 -- deliberately left, since deleting it would leave the region VM + power-unmanaged for no gain during the migration. Now that the migration is complete + that reasoning has expired and the decision should be revisited. + NEITHER IS DEPLOY-BLOCKING. Both are cleanup decisions for the operator. + +F3. THE SSH TUNNEL IS A LIVE PROCESS WITH NO SUPERVISION. + ALREADY ON SURFACE (changelog item 2, CURRENT-STATE) including the re-open command, + but restated here because it is the single most likely thing to be broken at the start + of the next session: + ssh voffice1 'ssh -f -N -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2' + The `vr1-dc0-region` MAAS profile points at `127.0.0.1:5241`, so with the tunnel down + EVERY tool targeting the new region refuses. FAILURE DIRECTION IS SAFE (refuse, never + a silent fall-back to Office1), but it will look like a broken region until the tunnel + is checked. `pgrep -f '5241:10.12.8.6'` on voffice1 is the check. + +F4. THE dc0 RACK'S NODE-DNS FORWARDER STILL POINTS AT THE OFFICE1 REGION. + ALREADY ON SURFACE (changelog item 9). Restated because the migration has now made it + WRONG rather than merely stale: `/etc/dnsmasq-dc0-node.conf` is `no-resolv` + + `server=10.10.0.20`, so anything on the rack resolving `maas-internal` gets the OLD + region's zone, which no longer holds dc0's node records. `dc-rack-net.sh` carries + `DNS_UPSTREAM="10.10.0.20"` for BOTH sites (`:66`, `:82`); dc1's is still correct, so + this is a per-site change. NODE DNS IS UNAFFECTED -- the metal-admin subnet now serves + `dns_servers=10.12.8.6` (the DC-local region), which is what nodes actually use. + Also noted: the live unit carries the D-119-RETIRED bare token (`dc0-node-dns.service`, + `/etc/dnsmasq-dc0-node.conf`) while the current script generates `vr1-dc0-*` names. + +F5. ENLISTMENT/COMMISSIONING apt GOES TO THE INTERNET, NOT THE DC MIRROR. + ALREADY ON SURFACE (changelog item 20 / CURRENT-STATE), restated for the D-135 file. + The ephemeral is NOBLE; the dc0 mirror carries JAMMY + UCA caracal only, so + `archive.ubuntu.com` / `security.ubuntu.com` are fetched through MAAS's own squid proxy + on the region (`10.12.8.6:8000`) -- which is the ONLY egress path the nodes have, + because metal-admin carries NO gateway. This is the practical reason nine concurrent + enlistments stalled and three did not. It also means the D-135/D-107 egress narrowing + would break ENLISTMENT as well as image sync and bootstrap -- one more consumer for + the F10 sequencing question already queued on 2026-07-30. + +F6. THE PER-NODE SERIAL LOG STOPPED CAPTURING ON 2026-07-21. + ALREADY ON SURFACE (changelog item 17). Restated because it is a DIAGNOSTIC BLIND SPOT + that cost real time this session: the file still contains a complete, plausible boot + and `tail` returns it confidently. Verify by MTIME before trusting any serial log. + +F7. NO AS-EXECUTED LOG FOR THIS WINDOW. + ALREADY ON SURFACE. `run-logged.sh` opens an INTERACTIVE `script(1)` subshell and is + unusable from a non-interactive session. All evidence went to `docs/audit/*` captures + instead, which is what GA-R6 actually requires of a gate. A `logs/as-executed-index.md` + row declaring the gap is OWED. + +F8. `maas-role-tags.sh` REPORTS PASS ON AN INCOMPLETE TAG SET. + ALREADY ON SURFACE (changelog item 23) but flagged here as a TOOLING defect rather + than a migration event: its `ROLES` set is control/compute/storage + + `juju-controller-`, so it returns `0 missing / 0 needing / 0 not-in-MAAS` while + every machine still lacks `openstack-` -- the tag `bundle.yaml` places on and + `dc-node-v6-carve.py:89-93` DIES without. A green gate on an incomplete state. + CANDIDATE FIX (not made -- hard rule 1): have it assert the site placement tag too, or + have `dc-region-topology.sh check` assert that every role node carries it. + +F9. THE NEXT SESSION'S FIRST COMMANDS, so they are not re-derived. + # 0. tunnel (F3), then prove the region before anything else + ssh voffice1 "cd ~/openstack-caracal-dc-dc && bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid" + # 1. the four green gates, to confirm nothing drifted + MAAS_PROFILE=vr1-dc0-region bash scripts/dc-region-topology.sh check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid + MAAS_PROFILE=vr1-dc0-region bash scripts/maas-role-tags.sh check vr1-dc0 + MAAS_PROFILE=vr1-dc0-region bash scripts/dc-plane-ipam.sh check vr1-dc0 + # 2. then the ONLY remaining blocker: write scripts/dc-node-carve.sh + Its hard inputs are already MEASURED and test-guarded: + - `NIC_PLANE_ORDER` in lib-hosts.sh -- NOT `PLANE_CIDRS` order. Walking PLANE_CIDRS + positionally swaps enp1s0/enp2s0 and strands PXE on provider-public. + - role asymmetry: role nodes = 6 planes + OVS `br-ex` parented on enp2s0 (the static + lands on br-ex, enp2s0 holds no L3 link); juju controller = 2 planes only + (metal-admin + provider-public raw, WITH gateway), no br-ex, enp3-6 left on auto. + - `carve-host-interfaces.sh:211-236` `build_ovs_brex()` is the correct MAAS call + ORDER: unlink commissioning links -> move parent NIC onto the provider VLAN -> + `interfaces create-bridge bridge_type=ovs` -> `link-subnet mode=STATIC` on br-ex. + - `carve-host-interfaces.sh` itself REFUSES `vr1-*` by design; do not remove that + guard, and do NOT point `phase-00-maas-standup.sh` at the new region (it does not + refuse, and would create space-named fabrics + a VID-103 metal-internal that D-133 + abolished). + THEN: `dc-node-v6-carve.py apply vr1-dc0 --commit --profile vr1-dc0-region` (it now + honours MAAS_PROFILE too), then re-point Juju at `10.12.8.6:5240` -- which needs a + region-scoped MAAS API key (the Office1 key will NOT authenticate), an updated cloud + + credential, and the still-owed `juju unregister vr1-dc0-controller` on the rack. + +F10. FIVE INSTRUMENT ERRORS IN ONE SESSION, ALL THE SAME SHAPE. + FIRST SURFACE as a consolidated item (each is recorded individually). + A confident NEGATIVE reported from an under-checked instrument: (a) a DNS probe at + `+time=3 +tries=1` called a capability gap -- it was a cold cache; (b) a listener list + truncated by `head -5` called a missing TFTP bind -- it was there; (c) a serial log + nine days stale read as current; (d) a bad elapsed-time estimate called enlistment + stalled when the last line was 90 seconds old; (e) counting 104 "enlist" lines as + progress when they were DHCP renewals -- filtering for `cloud-init` returned 0. + THE RULE THAT CATCHES ALL FIVE, and which this repo already states for inherited + findings but not for one's own live reads: before reporting a negative from a log, + listener list or probe, STATE HOW YOU KNOW THE INSTRUMENT WAS CURRENT -- mtime, + unfiltered output, adequate timeout, and a filter that matches the thing you actually + care about. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index ae6242b..657c77a 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -275,4 +275,5 @@ - **A GREEN GATE ON AN INCOMPLETE STATE:** `maas-role-tags.sh` PASSed while all ten still lacked `openstack-vr1-dc0` -- it is not in its `ROLES` set and has no creator in the repo. `bundle.yaml` places on it and `dc-node-v6-carve.py` dies without it. Caught ONLY by diffing against the pre-migration snapshot, which earned its keep. - **OWNED -- five instrument errors, all the same shape** (a confident negative from an under-checked instrument): a cold-cache DNS probe, a `head -5`-truncated listener list, a nine-day-stale serial log read as current, a bad elapsed-time estimate, and counting DHCP renewals as cloud-init progress. Also batched nine destructive deletes into a loop (the guard stopped me; hard rule 3 forbids it), and twice over-claimed in the records -- writing the re-enrolment ordering as proven before any machine reached `Ready`, and "no `New` detour" when I had aborted them there myself. All corrected on-surface. - **NEXT, and it is the ONLY thing between here and the deploy:** `scripts/dc-node-carve.sh` does not exist -- 60 NIC re-homes + 9 `br-ex` + 54 v4 statics. Its hard inputs are now MEASURED and guarded: `NIC_PLANE_ORDER` (NOT `PLANE_CIDRS` order -- walking that positionally strands PXE on provider-public), the role asymmetry, and `carve-host-interfaces.sh:211-236`'s `build_ovs_brex()` call sequence. Then `dc-node-v6-carve.py --profile vr1-dc0-region`, then re-point Juju at `10.12.8.6:5240` (needs a region-scoped API key + `juju unregister vr1-dc0-controller` on the rack). +- **CLOSE SWEEP: `docs/audit/queued-findings-20260730-dc0-region-migration.txt` (F1-F10).** THREE items lived ONLY in the transcript: **F1** this session's permission rules exist only in GITIGNORED `settings.local.json` (verbatim text now recorded, incl. a broad `Bash(ssh vr1-dc0-rack *)` auto-added by an approval and worth review); **F2** Office1 STILL registers a rack controller on the dc0 rack (`vvr1-dc0`/`7chphy`) and still holds the region VM's machine record; **F10** the five instrument errors as one pattern. F9 carries the next session's first commands and every measured input the carve tool needs. - Gauntlet **ALL GREEN (92)**; repo-lint 0 fail. Body: `docs/changelog-20260730-dc0-region-migration.md` (24 items). Status ONLY in CURRENT-STATE.md.