diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1c55625..8f2a276 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -538,7 +538,31 @@ (`dc-rack-net.sh` carries `DNS_UPSTREAM="10.10.0.20"` for BOTH sites at `:66`/`:82`; dc1's is still correct, so this is a per-site cutover change). The live unit also carries the D-119-retired BARE token (`dc0-node-dns.service`) while the script now generates `vr1-dc0-*`. - **>>> STAGE 5 IS BLOCKED ON A PERMISSION WALL, NOT ON A DECISION OR A DEFECT. <<<** + **DHCP HANDOVER COMPLETE 2026-07-30 -- the dc0 metal-admin segment is now served by the + DC-LOCAL region.** Capture `docs/audit/dc0-dhcp-handover-20260730.txt`. Executed in the + ruled order with the region identity proven first: Office1 `vlan update 4 0 dhcp_on=false` + -> read back `dhcp_on=False`; **verified STOPPED BY PROCESS** (`pgrep -c dhcpd` = **0** on + the dc0 rack -- MAAS's self-report is not evidence, per the 2026-07-20 Temporal incident + class); then new region `vlan update 0 0 dhcp_on=true primary_rack=c3aqh8` -> read back + `dhcp_on=True primary_rack=c3aqh8 fabric=fabric-0 space=metal-admin`; **verified RUNNING BY + PROCESS** on `hot-kid` with the rack re-checked at **0**, so exactly ONE DHCP server is on + the segment. All 10 machines were `Ready` and POWERED OFF throughout, so the gap was free. + **RECORDED SO IT IS NOT LATER READ AS A FAULT: the new region starts `dhcpd6` as well** + (the Office1 rack ran v4 only). Its v6 metal-admin subnet carries NO dynamic range -- only + `rfc-4291-2.6.1` and `assigned-ip,reserved`, identical to the old region -- so it has + nothing to hand out and nodes keep their ruled MAAS statics. + **THE PERMISSION WALL WAS A RULE THAT FAILED TO MATCH, NOT A CLASSIFIER FAULT.** + `.claude/settings.json` already carried `Bash(maas admin * update*)` in **ask**, and + `settings.local.json` carried `Bash(ssh voffice1 "maas admin subnet *)` and friends -- but + those pin the DOUBLE-quote form while the commands used SINGLE quotes, and they pin the + `maas admin` profile while the new region needs `maas vr1-dc0-region`. Neither matched, so + the call fell through to the classifier. Operator-approved targeted **ask** rules were + added to `.claude/settings.local.json` (gitignored, so team policy in `settings.json` is + unchanged): `Bash(ssh *'maas * update*)` + the double-quote twin, profile-agnostic so both + regions match. Read-only `pgrep`/`ps -ef` over ssh were added to **allow**. This is the + standing lesson restated: CHECK WHETHER A RULE FAILED TO MATCH BEFORE BLAMING THE + CLASSIFIER, and remember that quoting style and the profile token are both part of the match. + **>>> PREVIOUSLY BLOCKED ON A PERMISSION WALL (now cleared -- kept as history) <<<** The next step is the one-way DHCP cutover -- `vlan update 4 0 dhcp_on=false` at Office1, verify `dhcpd` STOPPED **by process** (MAAS self-report is not evidence -- the 2026-07-20 Temporal incident class), then `vlan update 0 0 dhcp_on=true primary_rack=` on the diff --git a/docs/audit/dc0-dhcp-handover-20260730.txt b/docs/audit/dc0-dhcp-handover-20260730.txt new file mode 100644 index 0000000..5b0e3e9 --- /dev/null +++ b/docs/audit/dc0-dhcp-handover-20260730.txt @@ -0,0 +1,29 @@ +dc0 METAL-ADMIN DHCP HANDOVER -- Office1 region -> per-DC region (D-132 q1) +Captured 2026-07-30. Nodes were all Ready and POWERED OFF, so the gap was free. +====================================================================== + +-- region identity proven before the mutating step -- +OK profile 'vr1-dc0-region' -> region with racks [hot-kid] (as expected) + +STEP 1 $ ssh voffice1 'maas admin vlan update 4 0 dhcp_on=false' + read back: + OFFICE1 10.12.8.0/22 dhcp_on=False primary_rack=7chphy fabric=fabric-4 + +STEP 2 verify STOPPED by PROCESS (MAAS self-report is not evidence -- + the 2026-07-20 Temporal incident class) + dc0 rack pgrep -c dhcpd = 0 + +STEP 3 $ ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8' + NEW REGION 10.12.8.0/22 dhcp_on=True primary_rack=c3aqh8 fabric=fabric-0 space=metal-admin + +STEP 4 verify RUNNING by PROCESS, and that the rack did NOT come back + region VM (hot-kid): + root 20565 5585 0 21:28 ? 00:00:00 /snap/maas/41649/usr/sbin/dhcpd -f -4 -pf /var/snap/maas/common/maas/dhcp/dhcpd.pid -cf /var/snap/maas/common/maas/dhcpd.conf -lf /var/snap/maas/common/maas/dhcp/dhcpd.leases enp1s0 + root 20571 5585 0 21:28 ? 00:00:00 /snap/maas/41649/usr/sbin/dhcpd -f -6 -pf /var/snap/maas/common/maas/dhcp/dhcpd6.pid -cf /var/snap/maas/common/maas/dhcpd6.conf -lf /var/snap/maas/common/maas/dhcp/dhcpd6.leases enp1s0 + dc0 rack pgrep -c dhcpd = 0 + +RESULT: exactly ONE DHCP server on the dc0 metal-admin segment, owned by the +DC-LOCAL region. dhcpd6 also starts (the old region ran v4 only) but the v6 +metal-admin subnet carries NO dynamic range -- only rfc-4291-2.6.1 and +assigned-ip/reserved -- so it has nothing to hand out and nodes keep their +ruled MAAS statics. Recorded so a later session does not read it as a fault. diff --git a/docs/changelog-20260730-dc0-region-migration.md b/docs/changelog-20260730-dc0-region-migration.md index fa90982..cb16228 100644 --- a/docs/changelog-20260730-dc0-region-migration.md +++ b/docs/changelog-20260730-dc0-region-migration.md @@ -364,7 +364,7 @@ --- -## Item 10 -- BLOCKED: the DHCP handover (task 5) needs an operator decision +## Item 10 -- the DHCP handover (task 5): initially BLOCKED, then EXECUTED **What was attempted.** With the topology verified complete (39/0), the next step is the one-way DHCP cutover: @@ -398,7 +398,8 @@ node is mid-DHCP. Two DHCP servers on one segment is the hazard the ordering avoids, which is why off-then-on is one operation and not two independent ones. -**No revert needed** -- nothing was changed. +**RESOLVED AND EXECUTED 2026-07-30** -- see item 15. Everything above stands as the record +of the blocked attempt; nothing was changed at the time of the refusal. **THE EXACT UNBLOCK SEQUENCE, every parameter RE-RESOLVED LIVE at session end.** Run these on the jumphost with the `!` prefix so the output lands in-session. The @@ -524,3 +525,52 @@ complete and is not, this session captures to `docs/audit/*` files -- which is what GA-R6 actually requires of a gate -- and says so here. A `logs/as-executed-index.md` row is owed at close. + + +--- + +## Item 15 -- DHCP HANDOVER EXECUTED, and the permission wall diagnosed + +**Executed** in the ruled order, region identity proven first (capture +`docs/audit/dc0-dhcp-handover-20260730.txt`): + +| step | command | read-back | +|---|---|---| +| 1 | `maas admin vlan update 4 0 dhcp_on=false` | `dhcp_on=False` | +| 2 | `pgrep -c dhcpd` on the dc0 rack | **0** | +| 3 | `maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8` | `dhcp_on=True primary_rack=c3aqh8 fabric=fabric-0 space=metal-admin` | +| 4 | `ps -ef` on `hot-kid`; rack re-checked | 2 dhcpd on the region VM; rack still **0** | + +Step 2 is the one that matters: MAAS's self-report is not evidence that dhcpd stopped (the +2026-07-20 Temporal incident class). Exactly ONE DHCP server is now on the segment, owned by +the DC-local region. All 10 machines were `Ready` and POWERED OFF throughout. + +**Recorded so it is not later read as a fault:** the new region also starts `dhcpd6`, where +the Office1 rack ran v4 only. Its v6 metal-admin subnet carries NO dynamic range -- only +`rfc-4291-2.6.1` and `assigned-ip,reserved`, identical to the old region -- so it has nothing +to offer and nodes keep their ruled MAAS statics. + +**THE WALL WAS A RULE THAT FAILED TO MATCH, NOT A CLASSIFIER FAULT** -- the standing lesson, +restated with two new specifics. `.claude/settings.json` already had +`Bash(maas admin * update*)` in **ask**, and `settings.local.json` already had +`Bash(ssh voffice1 "maas admin subnet *)` and siblings. Neither matched, because: +1. those entries pin the **double-quote** form (`ssh voffice1 "maas ...`) while the commands + used **single** quotes; and +2. they pin the **`maas admin`** profile, while the new region needs `maas vr1-dc0-region`. + +So the call fell through to the classifier, which blocked it as unmatched. Operator-approved +rules were added to `.claude/settings.local.json` (gitignored -- committed team policy in +`settings.json` is unchanged): profile-agnostic **ask** rules `Bash(ssh *'maas * update*)` +plus the double-quote twin and the `ipranges` pair, and read-only **allow** rules for +`pgrep` / `ps -ef` over ssh. The operator chose **ask** over **allow** deliberately, keeping +hard rule 3 intact for every MAAS mutation. + +**A NOTE ON THE RULE I ALMOST WROTE:** the shape first proposed pinned `maas admin * update*`, +mirroring the existing project rule. It would have matched step 1 and **silently failed to +match step 3**, which runs against `maas vr1-dc0-region` -- leaving the cutover half done at +exactly the two-servers boundary. Widened to `maas * update*` before use. An `ask` rule that +is too broad only costs extra prompts; one that is too narrow costs a stall mid-cutover. + +**Revert.** `ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=false'` then +`ssh voffice1 'maas admin vlan update 4 0 dhcp_on=true primary_rack=7chphy'`, verifying by +process at each step. Remove the added rules from `.claude/settings.local.json`.