Changelog 2026-07-30 -- dc0 node interface carve (dc-node-carve.sh)

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Context: the dc0 machine migration to the per-DC MAAS region is COMPLETE -- all ten machines Ready, four named gates green. The last thing between here and the Juju deploy is the per-node v4 interface carve: 60 NIC re-homes, 9 OVS br-ex bridges and 54 statics, deliberately NOT written blind last session because it could not be exercised until the nodes were re-enrolled.

Item 1 -- scripts/dc-node-carve.sh + tests/dc-node-carve/ (NEW)

What. The v4 node interface carve for a VR1 DC, run against that DC's own MAAS region. check is the named GA-R6 gate; apply is dry-run by default and needs --commit to mutate. --host restricts to one machine so the work can go canary-first then in batches.

Why it did not already exist. The equivalent work at Stage 4 was done ad-hoc through the MAAS API and logged only to ~/as-executed/2026-07-23-stage4-carve.log, which is NOT in the repo. When D-132 q1 moved dc0 to its own region and every machine had to be re-enrolled, there was nothing to re-run. A read-only survey last session identified this as the largest of three no-tool gaps.

The four-call cycle was PROVEN ON ONE INTERFACE BEFORE THE TOOL WAS WRITTEN. Nothing in the repo had exercised interface update vlan= against an interface whose only link is link_up on an auto-created fabric, which is the state all sixty NICs are in. Run on the migration canary (6q4syf/civil-bug = vr1-dc0-storage-04, the least central node):

interface update 6q4syf 10 vlan=5004        -> read back vlan=5004 / vr1-dc0-metal-internal
interface link-subnet ... mode=STATIC ...   -> read back static:10.12.12.153

Both bit. MEASURED IN PASSING, and it simplified the tool: link-subnet REPLACES a link_up link, so plane NICs need no explicit unlink. Only the br-ex member does, because there the L3 has to move off the member onto the bridge.

Properties that are structural, not stylistic:

• No default profile, and maas-profile-assert.sh runs before anything is read or written. The repo-wide default MAAS_PROFILE=admin resolves to the OFFICE1 region, where dc1's nine nodes still live. A 60-NIC re-home against the wrong region is either an idempotent no-op that prints PASS or damage to the wrong DC.
• Site refusal is the mirror image of carve-host-interfaces.sh. That script refuses vr1-* because it builds the VR0 Pattern-A VID-103 stack; this one refuses everything that is not vr1-dc0|vr1-dc1 because it builds the D-133 flat per-NIC carve.
• Every id is resolved live and keyed by a stable identity -- machines by PINNED BOOT MAC (system_id AND hostname are both re-minted on re-enlistment), subnets and VLANs by CIDR, interfaces by name. The pre-migration capture is a rebuild reference for ADDRESSES AND INTERFACE NAMES ONLY: its vlan_id=5005 is dc0 metal-admin in Office1 and vr1-dc0-data-tenant in the new region.
• Every mutation is READ BACK and compared, and a mismatch is fatal. MAAS has been measured returning a full JSON object while changing nothing, surfacing only later as {"subnet": ["None found with id=N."]}.
• NIC order comes from lib-hosts.sh NIC_PLANE_ORDER, never lib-net.sh PLANE_CIDRS. The plane->CIDR map is INVERTED from lib-net's own PLANE_NAME rather than retyped, so a plane re-address cannot silently disagree.
• Role asymmetry is DERIVED from the host token (JUJU_HOST_SUFFIX), never typed. Role nodes take six planes plus an OVS br-ex parented on enp2s0; the Juju controller takes metal-admin + provider-public RAW, no br-ex, and enp3s0..enp6s0 deliberately left on their auto VLANs. The gate asserts that deliberate ABSENCE, so a later session cannot "helpfully" carve them and have everything stay green.
• build_ovs_brex's guard structure is preserved verbatim from carve-host-interfaces.sh:211-236. The VLAN move sits INSIDE the "br-ex does not exist yet" branch on purpose: a MAAS bridge inherits its parent's VLAN at create, and once br-ex exists the parent is enslaved and a VLAN update is invalid. Restructuring it for symmetry with carve_raw would break re-runnability.
• A link on the right subnet with the wrong address FAILS; it is never skipped.

What this gate does NOT prove, stated in its own output. The node's actual routing table is a first-boot observation (G17). The carve proves the MAAS MODEL that produces a default route -- a static on provider-public plus gateway_ip on that subnet -- and the absence of that leg is exactly the under-carve that cost three bootstrap attempts on 2026-07-30. A green carve is not a proven default route, and the script says so rather than letting a reader infer it.

Harness: 40/40, and every assertion is mutation-proven. Eight fixtures each supply a failing direction: raw (the measured uncarved state), wrongip, patternb (provider static on the raw NIC with no bridge -- the D-057 shape that darkened the floating-IP plane), memberl3, brexip, jujucarved, jujubrex, nogw.

A MUTATION PASS FOUND ONE ASSERTION THAT COULD NOT FAIL, and a first pass of mutations that proved less than it appeared to. Neutering three assertions by deleting their MESSAGE turned the suite red only via the tests that grep for that message -- which proves the assertion EXISTS, not that its predicate works. Re-run neutering only the PREDICATE and keeping the message, three killed properly and the br-ex static-address compare SURVIVED: patternb was being caught by the type/parent assertions, so the address compare was never exercised. Added the brexip fixture (a correctly-parented OVS bridge carrying the WRONG static) and re-proved the kill.

Revert. git rm scripts/dc-node-carve.sh tests/dc-node-carve/run-tests.sh and bash scripts/run-tests-all.sh --record-manifest. Nothing else depends on it yet.

Item 1b -- the LIVE DRY-RUN found a defect the whole fixture suite had missed

What. The first live apply (plan only) against the canary refused enp1s0:

[FAIL] enp1s0: already linked to 10.12.8.0/22 as 'auto:-' but the ruled address is
       '10.12.8.153' -- refusing to silently accept it

Why it matters. Every PXE leg comes out of enlistment holding an auto link on metal-admin -- MEASURED on all ten dc0 machines. So the tool could not carve the metal-admin leg on ANY machine. The "wrong address on the right subnet is a FAILURE" rule was correct in intent and too broad in fact: a commissioning link is the EXPECTED STARTING STATE, not a conflict.

Fix. The conversion is narrowed BY MODE. An auto/dhcp/link_up link is MAAS's own default and gets unlinked and re-created as STATIC (MAAS will not convert a link's mode in place). A static link with the wrong address is still REFUSED -- something deliberate put it there, and moving it silently could strand whatever depends on it. Both directions are now fixtures (T26, T27) and both are mutation-proven: dropping the unlink turns T26c red, widening the conversion to include static: turns T27 red.

THIS IS THE THIRD TIME IN TWO DAYS A LIVE RUN CAUGHT WHAT FIXTURES DID NOT -- the same reason this tool was deliberately not written blind last session. The raw fixture DID model the auto link correctly; what the harness lacked was an assertion that apply over the raw state SUCCEEDS. Tests T21-T24 only grepped for planned strings and never checked the exit code, so a refusal on the very first NIC read as green.

Revert. Same as item 1.

Item 2 -- tests/HARNESS-MANIFEST re-recorded (92 -> 93)

What. Re-recorded to pin the new harness. Required: the gauntlet's R15(2) drift gate FAILS on a harness that is present but unpinned.

Revert. git checkout <this-commit>^ -- tests/HARNESS-MANIFEST.

Item 3 -- the dc0 v4 node carve EXECUTED, all ten machines

What. dc-node-carve.sh apply run canary-first then node by node, never batched. 11 mutations on the canary + 13 on each of the other eight role nodes + 4 on the Juju controller. Every one read back and compared.

Named gate (GA-R6): dc-node-carve.sh check vr1-dc0 = pass 134 / fail 0, exit 0. Capture docs/audit/dc0-node-carve-20260730.txt (195 lines). Before the apply the same gate read 25/109.

Verified independently by diffing against the pre-migration capture, and the diff is EMPTY -- every node's six v4 legs match the Office1 carve address for address. Compared by LEG SET rather than by name, because system_id AND hostname are both re-minted at re-enlistment (moral-salmon -> civil-bug, and so on for all ten).

INSTRUMENT NOTE, recorded because it nearly became a false negative. The first extraction read field 8 of the capture (parents=) instead of field 9, and returned - for every OLD address. Presented as-is that reads "the old region had no statics" -- a confident, well-formed, wrong result. It was caught only because the NEW side was fully populated against an all-empty OLD side, which is an implausible shape rather than a plausible one. This is the same class as the five instrument errors swept on 2026-07-30.

What this does NOT establish. The node's actual routing table. G17 and the default route are first-boot observations; the carve proves the MAAS model that produces them.

Revert. There is no single-command revert, and none is wanted: the target state IS the pre-migration carve, now re-established. To undo a single leg, maas vr1-dc0-region interface unlink-subnet <sysid> <ifid> id=<linkid>.

Item 4 -- the v6 node carve, and the controller's v6 leg

What. dc-node-v6-carve.py apply vr1-dc0 --commit --profile vr1-dc0-region: applied=54 skipped=0 errors=0, with the tool's own READ-BACK: 54/54 link(s) verified live. check then reads 54 correct / 0 missing / 0 errors, PASS. Existing tool, no new code -- it was already fixed last session to honour MAAS_PROFILE.

Ordering that mattered: it had to run AFTER the v4 carve, because the provider v6 static lands on br-ex and br-ex did not exist until item 3.

SCOPE, stated rather than glossed. The tool walks the NINE nodes tagged openstack-vr1-dc0. The Juju controller is deliberately untagged and is therefore OUTSIDE it -- so its metal-admin v6 was applied separately and read back. That restores ruled-and-previously-built state (the D-134 amendment, operator utterance "Fix now: static .5 + v6, re-bootstrap") which the re-enrolment dropped.

The 2602:f3e2:f02:10::5 GUA on the controller's enp2s0 remains DELIBERATELY ABSENT. Verified still absent. It was an explicit operator hold and was NOT silently "fixed" for symmetry against the role nodes -- the pre-migration capture confirms the controller never carried it.

Revert. Per link: maas vr1-dc0-region interface unlink-subnet <sysid> <ifid> id=<linkid>.

Item 5 -- Juju re-point: surveyed, and BLOCKED on a credential mint

What (read-only survey, on the dc0 rack = the D-138 client host). juju 3.6.27 present. Cloud vr1-maas still reads endpoint: http://10.10.0.20:5240/MAAS -- the OFFICE1 region. Credential vr1-dc0-cred present, carrying the Office1 oauth key. The rack reaches BOTH regions (200/200), so this is configuration, not reachability.

ONE OWED ITEM DISCHARGED BY MEASUREMENT. juju unregister vr1-dc0-controller is NOT needed: juju controllers returns ERROR No controllers registered. The third bootstrap never persisted a controller record.

THE BLOCKER. The new region has no juju-vr1-dc0 MAAS user -- only MAAS, admin, maas-init-node, where Office1 has juju-vr1-dc0 and juju-vr1-dc1, both is_superuser: true. Same class as "a fresh region has ZERO SSH keys" and "the snap ssh dir did not exist": a standing DC-standup item nobody had enumerated.

Re-establishing it necessarily MINTS A NEW API KEY (separate databases), so it is an operator-gated credential decision and NOTHING WAS MINTED. The mint script was written to never print key material -- it writes to 0600 files and reports only byte count and part count -- but the call was refused by the harness and was NOT retried in an altered shape.

The refusal was a permission rule failing to MATCH, not a classifier fault -- Bash(ssh vr1-dc0-rack *) is allowed and there is no vr1-dc0-maas equivalent. Same standing lesson as the DHCP cutover earlier in the migration.

Registration owed BEFORE bootstrap if the mint is approved (SEC-022's lesson: an unlisted location is not audited): a creds-matrix.tsv row per DC, a vm-secret-locations entry for the rack, the dc0 manifest declaration, and a new SEC row (next free SEC-028).

Revert. Nothing to revert -- no mutation was made.

Item 6 -- per-DC Juju service credential MINTED, cloud re-pointed (SEC-028)

Operator rulings (GA-R5, one exchange each, quoted). Credential shape: "Mint juju-vr1-dc0 on the new region (Recommended)". Permission wall: "Add it to allow".

What. juju-vr1-dc0 created in the dc0 region as a SUPERUSER -- mirroring the MEASURED Office1 shape, since MAAS 3.7 needs it for machine allocate/deploy. It was NOT "improved" to a lesser role; this is re-establishing an as-built shape, not designing a new one.

Custody. Minted ON the region VM, never printed. Moved region-VM -> vcloud -> rack with all three sha256 digests compared and equal, staging shredded, 0600 inside a 0700 folder read back at the destination. Only byte counts and part counts ever reached the transcript.

PROVEN TO AUTHENTICATE BEFORE ANY BOOTSTRAP. Bootstrap is one-shot with no --dry-run, so a scoped maas login + users read + machines read (10 total, 10 Ready) + rack-controllers read -> [hot-kid] + logout proved authentication, superuser scope, AND DC-local region identity in four read-only calls. A credential that EXISTS is not a credential that WORKS.

Cloud re-pointed and read back: endpoint: http://10.12.8.6:5240/MAAS. The stale Office1-keyed vr1-dc0-cred was REMOVED before the new one was added under the same name -- leaving both gives the cloud credential-count: 2 and juju may pick either.

TWO SNAP-CONFINEMENT TRAPS, each cost an attempt, neither error names confinement. (a) juju is a snap with a PRIVATE /tmp: a YAML written there by the shell is INVISIBLE to it, reported as no such file or directory for a path that demonstrably exists. (b) The snap home interface grants access to NON-HIDDEN files only, so $HOME/.stage/... fails permission denied on a directory the caller owns. Both now in the phase-4 runbook.

Revert. juju remove-credential vr1-maas vr1-dc0-cred --client; restore the endpoint with juju update-cloud; maas vr1-dc0-region users delete juju-vr1-dc0; remove the two files from both creds folders and their matrix/manifest rows.

Item 7 -- DOCFIX-206: the Step 2.0 credential gate was not region-scoped

What. runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 2.0 says "credential LISTED and MAAS user exists -> SKIP the mint". Under D-132 q1 that is WRONG, and I hit it: juju credentials --client listed vr1-dc0-cred, ~/vr1-dc0-creds existed, and juju-vr1-dc0 existed -- in OFFICE1. The new region held only MAAS, admin, maas-init-node.

Why it matters. The SKIP branch leads straight to a bootstrap that fails on authentication, which presents as a network fault -- the same misleading signature that consumed three attempts on 2026-07-30. The regions hold SEPARATE DATABASES: a user in one is not a user in the other, and a credential's NAME says nothing about which region it authenticates against.

Fixed. Every check in the block must name the region and prove it with maas-profile-assert.sh first; a FOURTH gate outcome was added for "listed but the user is in another region" (STALE -> remove and re-mint, never "make it consistent"); the mint location was corrected from voffice1 to the DC's OWN region VM; and the endpoint re-point, the two snap traps, the printf-not-heredoc rule and the authenticate-before-bootstrap proof were all added.

Revert. git checkout <this-commit>^ -- runbooks/dc-dc-phase4-juju-bundle-per-dc.md.