diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 9e97cf8..51341cd 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -821,11 +821,48 @@ being fully populated against an all-empty Old side -- an implausible shape, not a plausible one. **G17 and the node's actual routing table remain FIRST-BOOT facts**; this gate proves the MAAS model that produces them, and says so in its own output. - **STILL OWED before bootstrap:** `dc-node-v6-carve.py apply vr1-dc0 --commit --profile - vr1-dc0-region` (it needs `br-ex` to exist, which it now does), then re-point Juju at - `10.12.8.6:5240` -- which needs a region-scoped MAAS API key (the Office1 key will NOT - authenticate), an updated juju cloud + credential, and `juju unregister - vr1-dc0-controller` on the rack. + **THE v6 CARVE IS ALSO COMPLETE -- `dc-node-v6-carve.py check vr1-dc0` = 54 link(s) + correct, 0 missing, 0 errors, PASS.** Applied with the existing tool, no new code: + `applied=54 skipped=0 errors=0` with its own `READ-BACK: 54/54 link(s) verified live`. + Prefixes match the pre-migration capture exactly (`:220` metal-admin, `:221` + metal-internal, `:230` data-tenant, `:240` storage, `:250` replication ULAs, plus the + `2602:f3e2:f02:10::` GUA on `br-ex`), host part mirroring each node's v4 octet. + **SCOPE, stated because the tool's own scope is narrower than "the fleet": it walks the + NINE nodes tagged `openstack-vr1-dc0` and NOT the Juju controller**, which is + deliberately untagged. The controller's metal-admin v6 `fd50:840e:74e2:220::5` was + therefore applied separately and read back (`enp1s0 static:10.12.8.5, + static:fd50:840e:74e2:220::5`) -- restoring ruled-and-previously-built state from the + D-134 amendment ("Fix now: static .5 + v6, re-bootstrap"), which the re-enrolment + dropped. **The `2602:f3e2:f02:10::5` GUA on the controller's `enp2s0` remains + DELIBERATELY ABSENT** -- that was an explicit operator hold, verified still absent, and + it was not silently "fixed" for symmetry. + **GAUNTLET ALL GREEN (93 harnesses)** on vcloud, up from 92 with `dc-node-carve` + included; repo-lint 0 fail. + **ONE OWED ITEM IS DISCHARGED BY MEASUREMENT, NOT BY WORK: `juju unregister + vr1-dc0-controller` is NOT needed.** `juju controllers` on the dc0 rack returns + `ERROR No controllers registered.` The third bootstrap never persisted a controller + record. Recorded so the next session does not re-derive it. + **STILL OWED before bootstrap, and the FIRST is now the blocker:** re-point Juju at + `10.12.8.6:5240`. MEASURED on the rack (the D-138 client host): `juju` 3.6.27 present; + cloud `vr1-maas` still reads `endpoint: http://10.10.0.20:5240/MAAS` -- **the OFFICE1 + region, which is exactly what killed bootstrap attempt 3**; credential `vr1-dc0-cred` + present but carrying the Office1 oauth key, which will NOT authenticate against the new + region's separate database. The rack reaches BOTH regions (200/200), so this is a + configuration gap and not a reachability one. + **THE NEW REGION HAS NO `juju-vr1-dc0` MAAS USER.** Measured: it holds `MAAS`, `admin`, + `maas-init-node` only, while Office1 holds `juju-vr1-dc0` and `juju-vr1-dc1`, both + `is_superuser: true`. This is the SAME CLASS as the two standup gaps this migration has + already hit -- a fresh region has zero SSH keys, and its snap ssh dir did not exist -- and + it belongs beside them as a standing DC-standup item. **BLOCKED ON OPERATOR APPROVAL of a + credential mint** (see the session's queued question): re-establishing `juju-vr1-dc0` in + the new region necessarily mints a NEW API key, because the regions hold separate + databases. The alternative -- pointing juju at the region ADMIN key already consolidated + at `~/vr1-dc0-creds/maas-region-api-key.txt` on vcloud -- is strictly worse: it puts an + admin-scoped credential on the rack, which is the residency SEC-026 exists to constrain. + **NOTHING WAS MINTED.** The call was refused by the harness and NOT retried in an altered + shape; the refusal was a permission rule failing to MATCH (`Bash(ssh vr1-dc0-rack *)` is + allowed, there is no `vr1-dc0-maas` rule) rather than a classifier fault -- the same + standing lesson as the 2026-07-30 DHCP cutover. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/changelog-20260730-dc0-node-carve.md b/docs/changelog-20260730-dc0-node-carve.md index b0b437c..5b228be 100644 --- a/docs/changelog-20260730-dc0-node-carve.md +++ b/docs/changelog-20260730-dc0-node-carve.md @@ -155,3 +155,59 @@ **Revert.** There is no single-command revert, and none is wanted: the target state IS the pre-migration carve, now re-established. To undo a single leg, `maas vr1-dc0-region interface unlink-subnet id=`. + +## Item 4 -- the v6 node carve, and the controller's v6 leg + +**What.** `dc-node-v6-carve.py apply vr1-dc0 --commit --profile vr1-dc0-region`: +`applied=54 skipped=0 errors=0`, with the tool's own `READ-BACK: 54/54 link(s) verified +live`. `check` then reads 54 correct / 0 missing / 0 errors, PASS. Existing tool, no new +code -- it was already fixed last session to honour `MAAS_PROFILE`. + +**Ordering that mattered:** it had to run AFTER the v4 carve, because the provider v6 +static lands on `br-ex` and `br-ex` did not exist until item 3. + +**SCOPE, stated rather than glossed.** The tool walks the NINE nodes tagged +`openstack-vr1-dc0`. The Juju controller is deliberately untagged and is therefore OUTSIDE +it -- so its metal-admin v6 was applied separately and read back. That restores +ruled-and-previously-built state (the D-134 amendment, operator utterance "Fix now: static +.5 + v6, re-bootstrap") which the re-enrolment dropped. + +**The `2602:f3e2:f02:10::5` GUA on the controller's `enp2s0` remains DELIBERATELY ABSENT.** +Verified still absent. It was an explicit operator hold and was NOT silently "fixed" for +symmetry against the role nodes -- the pre-migration capture confirms the controller never +carried it. + +**Revert.** Per link: +`maas vr1-dc0-region interface unlink-subnet id=`. + +## Item 5 -- Juju re-point: surveyed, and BLOCKED on a credential mint + +**What (read-only survey, on the dc0 rack = the D-138 client host).** `juju` 3.6.27 +present. Cloud `vr1-maas` still reads `endpoint: http://10.10.0.20:5240/MAAS` -- the +OFFICE1 region. Credential `vr1-dc0-cred` present, carrying the Office1 oauth key. The rack +reaches BOTH regions (200/200), so this is configuration, not reachability. + +**ONE OWED ITEM DISCHARGED BY MEASUREMENT.** `juju unregister vr1-dc0-controller` is NOT +needed: `juju controllers` returns `ERROR No controllers registered.` The third bootstrap +never persisted a controller record. + +**THE BLOCKER.** The new region has no `juju-vr1-dc0` MAAS user -- only `MAAS`, `admin`, +`maas-init-node`, where Office1 has `juju-vr1-dc0` and `juju-vr1-dc1`, both +`is_superuser: true`. Same class as "a fresh region has ZERO SSH keys" and "the snap ssh +dir did not exist": a standing DC-standup item nobody had enumerated. + +Re-establishing it necessarily MINTS A NEW API KEY (separate databases), so it is an +operator-gated credential decision and NOTHING WAS MINTED. The mint script was written to +never print key material -- it writes to `0600` files and reports only byte count and +part count -- but the call was refused by the harness and was NOT retried in an altered +shape. + +**The refusal was a permission rule failing to MATCH, not a classifier fault** -- +`Bash(ssh vr1-dc0-rack *)` is allowed and there is no `vr1-dc0-maas` equivalent. Same +standing lesson as the DHCP cutover earlier in the migration. + +**Registration owed BEFORE bootstrap if the mint is approved** (SEC-022's lesson: an +unlisted location is not audited): a `creds-matrix.tsv` row per DC, a `vm-secret-locations` +entry for the rack, the dc0 manifest declaration, and a new SEC row (next free SEC-028). + +**Revert.** Nothing to revert -- no mutation was made.