# Changelog 2026-07-30 -- dc0 node interface carve (`dc-node-carve.sh`) Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. Status claims live ONLY in `docs/CURRENT-STATE.md`. Context: the dc0 machine migration to the per-DC MAAS region is COMPLETE -- all ten machines `Ready`, four named gates green. The last thing between here and the Juju deploy is the per-node v4 interface carve: 60 NIC re-homes, 9 OVS `br-ex` bridges and 54 statics, deliberately NOT written blind last session because it could not be exercised until the nodes were re-enrolled. --- ## Item 1 -- `scripts/dc-node-carve.sh` + `tests/dc-node-carve/` (NEW) **What.** The v4 node interface carve for a VR1 DC, run against that DC's own MAAS region. `check` is the named GA-R6 gate; `apply` is dry-run by default and needs `--commit` to mutate. `--host` restricts to one machine so the work can go canary-first then in batches. **Why it did not already exist.** The equivalent work at Stage 4 was done ad-hoc through the MAAS API and logged only to `~/as-executed/2026-07-23-stage4-carve.log`, which is NOT in the repo. When D-132 q1 moved dc0 to its own region and every machine had to be re-enrolled, there was nothing to re-run. A read-only survey last session identified this as the largest of three no-tool gaps. **The four-call cycle was PROVEN ON ONE INTERFACE BEFORE THE TOOL WAS WRITTEN.** Nothing in the repo had exercised `interface update vlan=` against an interface whose only link is `link_up` on an auto-created fabric, which is the state all sixty NICs are in. Run on the migration canary (`6q4syf`/civil-bug = `vr1-dc0-storage-04`, the least central node): interface update 6q4syf 10 vlan=5004 -> read back vlan=5004 / vr1-dc0-metal-internal interface link-subnet ... mode=STATIC ... -> read back static:10.12.12.153 Both bit. MEASURED IN PASSING, and it simplified the tool: `link-subnet` REPLACES a `link_up` link, so plane NICs need no explicit unlink. Only the `br-ex` member does, because there the L3 has to move off the member onto the bridge. **Properties that are structural, not stylistic:** - **No default profile, and `maas-profile-assert.sh` runs before anything is read or written.** The repo-wide default `MAAS_PROFILE=admin` resolves to the OFFICE1 region, where dc1's nine nodes still live. A 60-NIC re-home against the wrong region is either an idempotent no-op that prints PASS or damage to the wrong DC. - **Site refusal is the mirror image of `carve-host-interfaces.sh`.** That script refuses `vr1-*` because it builds the VR0 Pattern-A VID-103 stack; this one refuses everything that is not `vr1-dc0|vr1-dc1` because it builds the D-133 flat per-NIC carve. - **Every id is resolved live and keyed by a stable identity** -- machines by PINNED BOOT MAC (system_id AND hostname are both re-minted on re-enlistment), subnets and VLANs by CIDR, interfaces by name. The pre-migration capture is a rebuild reference for ADDRESSES AND INTERFACE NAMES ONLY: its `vlan_id=5005` is dc0 metal-admin in Office1 and `vr1-dc0-data-tenant` in the new region. - **Every mutation is READ BACK and compared, and a mismatch is fatal.** MAAS has been measured returning a full JSON object while changing nothing, surfacing only later as `{"subnet": ["None found with id=N."]}`. - **NIC order comes from `lib-hosts.sh NIC_PLANE_ORDER`, never `lib-net.sh PLANE_CIDRS`.** The plane->CIDR map is INVERTED from lib-net's own `PLANE_NAME` rather than retyped, so a plane re-address cannot silently disagree. - **Role asymmetry is DERIVED from the host token** (`JUJU_HOST_SUFFIX`), never typed. Role nodes take six planes plus an OVS `br-ex` parented on `enp2s0`; the Juju controller takes metal-admin + provider-public RAW, no `br-ex`, and `enp3s0..enp6s0` deliberately left on their auto VLANs. The gate asserts that deliberate ABSENCE, so a later session cannot "helpfully" carve them and have everything stay green. - **`build_ovs_brex`'s guard structure is preserved verbatim** from `carve-host-interfaces.sh:211-236`. The VLAN move sits INSIDE the "br-ex does not exist yet" branch on purpose: a MAAS bridge inherits its parent's VLAN at create, and once br-ex exists the parent is enslaved and a VLAN update is invalid. Restructuring it for symmetry with `carve_raw` would break re-runnability. - **A link on the right subnet with the wrong address FAILS; it is never skipped.** **What this gate does NOT prove, stated in its own output.** The node's actual routing table is a first-boot observation (G17). The carve proves the MAAS MODEL that produces a default route -- a static on provider-public plus `gateway_ip` on that subnet -- and the absence of that leg is exactly the under-carve that cost three bootstrap attempts on 2026-07-30. A green carve is not a proven default route, and the script says so rather than letting a reader infer it. **Harness: 40/40, and every assertion is mutation-proven.** Eight fixtures each supply a failing direction: `raw` (the measured uncarved state), `wrongip`, `patternb` (provider static on the raw NIC with no bridge -- the D-057 shape that darkened the floating-IP plane), `memberl3`, `brexip`, `jujucarved`, `jujubrex`, `nogw`. **A MUTATION PASS FOUND ONE ASSERTION THAT COULD NOT FAIL, and a first pass of mutations that proved less than it appeared to.** Neutering three assertions by deleting their MESSAGE turned the suite red only via the tests that grep for that message -- which proves the assertion EXISTS, not that its predicate works. Re-run neutering only the PREDICATE and keeping the message, three killed properly and **the `br-ex` static-address compare SURVIVED**: `patternb` was being caught by the type/parent assertions, so the address compare was never exercised. Added the `brexip` fixture (a correctly-parented OVS bridge carrying the WRONG static) and re-proved the kill. **Revert.** `git rm scripts/dc-node-carve.sh tests/dc-node-carve/run-tests.sh` and `bash scripts/run-tests-all.sh --record-manifest`. Nothing else depends on it yet. ## Item 1b -- the LIVE DRY-RUN found a defect the whole fixture suite had missed **What.** The first live `apply` (plan only) against the canary refused `enp1s0`: [FAIL] enp1s0: already linked to 10.12.8.0/22 as 'auto:-' but the ruled address is '10.12.8.153' -- refusing to silently accept it **Why it matters.** Every PXE leg comes out of enlistment holding an `auto` link on metal-admin -- MEASURED on all ten dc0 machines. So the tool could not carve the metal-admin leg on ANY machine. The "wrong address on the right subnet is a FAILURE" rule was correct in intent and too broad in fact: a commissioning link is the EXPECTED STARTING STATE, not a conflict. **Fix.** The conversion is narrowed BY MODE. An `auto`/`dhcp`/`link_up` link is MAAS's own default and gets unlinked and re-created as STATIC (MAAS will not convert a link's mode in place). A `static` link with the wrong address is still REFUSED -- something deliberate put it there, and moving it silently could strand whatever depends on it. Both directions are now fixtures (T26, T27) and both are mutation-proven: dropping the unlink turns T26c red, widening the conversion to include `static:` turns T27 red. **THIS IS THE THIRD TIME IN TWO DAYS A LIVE RUN CAUGHT WHAT FIXTURES DID NOT** -- the same reason this tool was deliberately not written blind last session. The `raw` fixture DID model the auto link correctly; what the harness lacked was an assertion that `apply` over the raw state SUCCEEDS. Tests T21-T24 only grepped for planned strings and never checked the exit code, so a refusal on the very first NIC read as green. **Revert.** Same as item 1. ## Item 2 -- `tests/HARNESS-MANIFEST` re-recorded (92 -> 93) **What.** Re-recorded to pin the new harness. Required: the gauntlet's R15(2) drift gate FAILS on a harness that is present but unpinned. **Revert.** `git checkout ^ -- tests/HARNESS-MANIFEST`. ## Item 3 -- the dc0 v4 node carve EXECUTED, all ten machines **What.** `dc-node-carve.sh apply` run canary-first then node by node, never batched. 11 mutations on the canary + 13 on each of the other eight role nodes + 4 on the Juju controller. Every one read back and compared. **Named gate (GA-R6):** `dc-node-carve.sh check vr1-dc0` = **pass 134 / fail 0, exit 0**. Capture `docs/audit/dc0-node-carve-20260730.txt` (195 lines). Before the apply the same gate read 25/109. **Verified independently by diffing against the pre-migration capture, and the diff is EMPTY** -- every node's six v4 legs match the Office1 carve address for address. Compared by LEG SET rather than by name, because system_id AND hostname are both re-minted at re-enlistment (`moral-salmon` -> `civil-bug`, and so on for all ten). **INSTRUMENT NOTE, recorded because it nearly became a false negative.** The first extraction read field 8 of the capture (`parents=`) instead of field 9, and returned `-` for every OLD address. Presented as-is that reads "the old region had no statics" -- a confident, well-formed, wrong result. It was caught only because the NEW side was fully populated against an all-empty OLD side, which is an implausible shape rather than a plausible one. This is the same class as the five instrument errors swept on 2026-07-30. **What this does NOT establish.** The node's actual routing table. G17 and the default route are first-boot observations; the carve proves the MAAS model that produces them. **Revert.** There is no single-command revert, and none is wanted: the target state IS the pre-migration carve, now re-established. To undo a single leg, `maas vr1-dc0-region interface unlink-subnet id=`. ## Item 4 -- the v6 node carve, and the controller's v6 leg **What.** `dc-node-v6-carve.py apply vr1-dc0 --commit --profile vr1-dc0-region`: `applied=54 skipped=0 errors=0`, with the tool's own `READ-BACK: 54/54 link(s) verified live`. `check` then reads 54 correct / 0 missing / 0 errors, PASS. Existing tool, no new code -- it was already fixed last session to honour `MAAS_PROFILE`. **Ordering that mattered:** it had to run AFTER the v4 carve, because the provider v6 static lands on `br-ex` and `br-ex` did not exist until item 3. **SCOPE, stated rather than glossed.** The tool walks the NINE nodes tagged `openstack-vr1-dc0`. The Juju controller is deliberately untagged and is therefore OUTSIDE it -- so its metal-admin v6 was applied separately and read back. That restores ruled-and-previously-built state (the D-134 amendment, operator utterance "Fix now: static .5 + v6, re-bootstrap") which the re-enrolment dropped. **The `2602:f3e2:f02:10::5` GUA on the controller's `enp2s0` remains DELIBERATELY ABSENT.** Verified still absent. It was an explicit operator hold and was NOT silently "fixed" for symmetry against the role nodes -- the pre-migration capture confirms the controller never carried it. **Revert.** Per link: `maas vr1-dc0-region interface unlink-subnet id=`. ## Item 5 -- Juju re-point: surveyed, and BLOCKED on a credential mint **What (read-only survey, on the dc0 rack = the D-138 client host).** `juju` 3.6.27 present. Cloud `vr1-maas` still reads `endpoint: http://10.10.0.20:5240/MAAS` -- the OFFICE1 region. Credential `vr1-dc0-cred` present, carrying the Office1 oauth key. The rack reaches BOTH regions (200/200), so this is configuration, not reachability. **ONE OWED ITEM DISCHARGED BY MEASUREMENT.** `juju unregister vr1-dc0-controller` is NOT needed: `juju controllers` returns `ERROR No controllers registered.` The third bootstrap never persisted a controller record. **THE BLOCKER.** The new region has no `juju-vr1-dc0` MAAS user -- only `MAAS`, `admin`, `maas-init-node`, where Office1 has `juju-vr1-dc0` and `juju-vr1-dc1`, both `is_superuser: true`. Same class as "a fresh region has ZERO SSH keys" and "the snap ssh dir did not exist": a standing DC-standup item nobody had enumerated. Re-establishing it necessarily MINTS A NEW API KEY (separate databases), so it is an operator-gated credential decision and NOTHING WAS MINTED. The mint script was written to never print key material -- it writes to `0600` files and reports only byte count and part count -- but the call was refused by the harness and was NOT retried in an altered shape. **The refusal was a permission rule failing to MATCH, not a classifier fault** -- `Bash(ssh vr1-dc0-rack *)` is allowed and there is no `vr1-dc0-maas` equivalent. Same standing lesson as the DHCP cutover earlier in the migration. **Registration owed BEFORE bootstrap if the mint is approved** (SEC-022's lesson: an unlisted location is not audited): a `creds-matrix.tsv` row per DC, a `vm-secret-locations` entry for the rack, the dc0 manifest declaration, and a new SEC row (next free SEC-028). **Revert.** Nothing to revert -- no mutation was made. ## Item 6 -- per-DC Juju service credential MINTED, cloud re-pointed (SEC-028) **Operator rulings (GA-R5, one exchange each, quoted).** Credential shape: *"Mint juju-vr1-dc0 on the new region (Recommended)"*. Permission wall: *"Add it to allow"*. **What.** `juju-vr1-dc0` created in the dc0 region as a SUPERUSER -- mirroring the MEASURED Office1 shape, since MAAS 3.7 needs it for machine allocate/deploy. It was NOT "improved" to a lesser role; this is re-establishing an as-built shape, not designing a new one. **Custody.** Minted ON the region VM, never printed. Moved region-VM -> vcloud -> rack with all three sha256 digests compared and equal, staging `shred`ded, `0600` inside a `0700` folder read back at the destination. Only byte counts and part counts ever reached the transcript. **PROVEN TO AUTHENTICATE BEFORE ANY BOOTSTRAP.** Bootstrap is one-shot with no `--dry-run`, so a scoped `maas login` + `users read` + `machines read` (10 total, 10 Ready) + `rack-controllers read` -> `[hot-kid]` + logout proved authentication, superuser scope, AND DC-local region identity in four read-only calls. A credential that EXISTS is not a credential that WORKS. **Cloud re-pointed and read back:** `endpoint: http://10.12.8.6:5240/MAAS`. The stale Office1-keyed `vr1-dc0-cred` was REMOVED before the new one was added under the same name -- leaving both gives the cloud `credential-count: 2` and juju may pick either. **TWO SNAP-CONFINEMENT TRAPS, each cost an attempt, neither error names confinement.** (a) `juju` is a snap with a PRIVATE `/tmp`: a YAML written there by the shell is INVISIBLE to it, reported as `no such file or directory` for a path that demonstrably exists. (b) The snap `home` interface grants access to NON-HIDDEN files only, so `$HOME/.stage/...` fails `permission denied` on a directory the caller owns. Both now in the phase-4 runbook. **Revert.** `juju remove-credential vr1-maas vr1-dc0-cred --client`; restore the endpoint with `juju update-cloud`; `maas vr1-dc0-region users delete juju-vr1-dc0`; remove the two files from both creds folders and their matrix/manifest rows. ## Item 7 -- DOCFIX-206: the Step 2.0 credential gate was not region-scoped **What.** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 2.0 says "credential LISTED and MAAS user exists -> SKIP the mint". Under D-132 q1 that is WRONG, and I hit it: `juju credentials --client` listed `vr1-dc0-cred`, `~/vr1-dc0-creds` existed, and `juju-vr1-dc0` existed -- **in OFFICE1**. The new region held only `MAAS`, `admin`, `maas-init-node`. **Why it matters.** The SKIP branch leads straight to a bootstrap that fails on authentication, which presents as a network fault -- the same misleading signature that consumed three attempts on 2026-07-30. The regions hold SEPARATE DATABASES: a user in one is not a user in the other, and a credential's NAME says nothing about which region it authenticates against. **Fixed.** Every check in the block must name the region and prove it with `maas-profile-assert.sh` first; a FOURTH gate outcome was added for "listed but the user is in another region" (STALE -> remove and re-mint, never "make it consistent"); the mint location was corrected from `voffice1` to the DC's OWN region VM; and the endpoint re-point, the two snap traps, the printf-not-heredoc rule and the authenticate-before-bootstrap proof were all added. **Revert.** `git checkout ^ -- runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. ## Item 8 -- JUJU CONTROLLER BOOTSTRAPPED (fourth attempt, first success) **What.** `juju bootstrap vr1-maas vr1-dc0-controller --constraints "tags=juju-controller-vr1-dc0" --bootstrap-constraints "tags=juju-controller-vr1-dc0" --bootstrap-base ubuntu@22.04`, run FROM THE dc0 RACK per D-138. Capture `docs/audit/stage5-bootstrap-dc0-20260730.txt`. **Result, measured.** `juju controllers` -> `vr1-dc0-controller* admin superuser vr1-maas, 1 model, 1 node, HA none, 3.6.27`. `juju status -m controller` -> app `controller` ACTIVE 1/1 on `juju-controller` rev 311 (`3.6/stable`), unit `controller/0*` ACTIVE/IDLE, machine 0 STARTED `ubuntu@22.04`, inst id `subtle-grouse`. MAAS reads `arfr7p` as Deployed ubuntu/jammy. **Why this one worked where three failed the same day.** Each earlier failure was a real defect one layer deeper, and all three are now closed by a different fix: 1. no L3 path from the juju client to any node plane -> **D-138** moved the client into the DC; 2. the controller VM had NO DEFAULT ROUTE (added after both Stage-4 carves) -> the under-carve fix, now re-applied by `dc-node-carve.sh` as part of the standard carve; 3. `jujud` could not reach the Office1 MAAS region API -> **D-132 q1**, per-DC regions. **What the transcript proves beyond "it worked":** - Agent binaries resolved on the FIRST attempt from `streams.canonical.com`. The 339-retry failure class stays closed. - Machine targeting bit again: it took the tagged controller VM, not one of the nine role nodes. (As before, this run cannot show WHICH of the two ruled flags did the selecting.) - Juju tried `10.12.8.5`, `[fd50:840e:74e2:220::5]` and `10.12.4.5` before `Connected to 10.12.8.5`. So the controller v6 leg restored in item 4 is live in MAAS and offered to juju, and metal-admin is what actually carried the SSH. - `Public address` is `10.12.4.5` -- the provider-public leg that supplies the default route. **Revert.** `juju kill-controller vr1-dc0-controller` (the controller API is up this time, so unlike attempt 3 this WILL work), then release `arfr7p` in MAAS. ## Item 9 -- FINDING (logged, not fixed): `preflight.sh` is REGION-BLIND **What.** `DC=vr1-dc0 bash scripts/preflight.sh` on voffice1 reports: PASS: MAAS reachable (profile=admin) FAIL: vr1-dc0-control-01 not enrolled in MAAS (x10, every machine) FAIL: host vr1-dc0-control-01 not found in MAAS (x9) **All nineteen are FALSE NEGATIVES from a wrong-region instrument.** `admin` is the OFFICE1 profile; dc0's ten machines were migrated to the per-DC region and are all `Ready` there -- measured minutes earlier, and confirmed by re-running with `MAAS_PROFILE=vr1-dc0-region`, which drops the node failures to **zero**. **Why this matters more than a wrong flag.** The message says "not enrolled in MAAS", which reads as a deployment-state problem. The true statement is "you are asking the wrong region". A session that trusted it would go looking for a broken enrolment that does not exist -- the same confident-well-formed-wrong-evidence class as the stale serial log. **`pre-flight-checks.sh:64` is `MAAS_PROFILE="${MAAS_PROFILE:-admin}"`.** This is the FOURTH consumer of that default; CURRENT-STATE already lists `dc-plane-ipam.sh`, `maas-role-tags.sh` and `maas-node-power.sh` as owing the same wiring, and preflight was not among them. **NOT FIXED, and the fix is not the obvious one (hard rule 1).** A blanket "refuse `admin` for `vr1-*`" would be WRONG: **dc1's nine nodes still live in Office1**, so `admin` is currently the CORRECT profile for `DC=vr1-dc1`. The right shape is to resolve the profile per-DC from a mapping and assert region identity (rack identity, per `maas-profile-assert.sh`) rather than to refuse a token. That needs its own change with a harness, and it does not block this deploy now that the profile is passed explicitly. **Workaround, and it must be carried forward:** every preflight run for dc0 needs `MAAS_PROFILE=vr1-dc0-region` until this is fixed. With it, preflight is RED on **P5 only** -- the ruled-accepted credential register -- and P1-P4 + P7 all PASS.