Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260730-dc0-node-carve.md

Changelog 2026-07-30 -- dc0 node interface carve (dc-node-carve.sh)

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Context: the dc0 machine migration to the per-DC MAAS region is COMPLETE -- all ten machines Ready, four named gates green. The last thing between here and the Juju deploy is the per-node v4 interface carve: 60 NIC re-homes, 9 OVS br-ex bridges and 54 statics, deliberately NOT written blind last session because it could not be exercised until the nodes were re-enrolled.


Item 1 -- scripts/dc-node-carve.sh + tests/dc-node-carve/ (NEW)

What. The v4 node interface carve for a VR1 DC, run against that DC's own MAAS region. check is the named GA-R6 gate; apply is dry-run by default and needs --commit to mutate. --host restricts to one machine so the work can go canary-first then in batches.

Why it did not already exist. The equivalent work at Stage 4 was done ad-hoc through the MAAS API and logged only to ~/as-executed/2026-07-23-stage4-carve.log, which is NOT in the repo. When D-132 q1 moved dc0 to its own region and every machine had to be re-enrolled, there was nothing to re-run. A read-only survey last session identified this as the largest of three no-tool gaps.

The four-call cycle was PROVEN ON ONE INTERFACE BEFORE THE TOOL WAS WRITTEN. Nothing in the repo had exercised interface update vlan= against an interface whose only link is link_up on an auto-created fabric, which is the state all sixty NICs are in. Run on the migration canary (6q4syf/civil-bug = vr1-dc0-storage-04, the least central node):

interface update 6q4syf 10 vlan=5004        -> read back vlan=5004 / vr1-dc0-metal-internal
interface link-subnet ... mode=STATIC ...   -> read back static:10.12.12.153

Both bit. MEASURED IN PASSING, and it simplified the tool: link-subnet REPLACES a link_up link, so plane NICs need no explicit unlink. Only the br-ex member does, because there the L3 has to move off the member onto the bridge.

Properties that are structural, not stylistic:

  • No default profile, and maas-profile-assert.sh runs before anything is read or written. The repo-wide default MAAS_PROFILE=admin resolves to the OFFICE1 region, where dc1's nine nodes still live. A 60-NIC re-home against the wrong region is either an idempotent no-op that prints PASS or damage to the wrong DC.
  • Site refusal is the mirror image of carve-host-interfaces.sh. That script refuses vr1-* because it builds the VR0 Pattern-A VID-103 stack; this one refuses everything that is not vr1-dc0|vr1-dc1 because it builds the D-133 flat per-NIC carve.
  • Every id is resolved live and keyed by a stable identity -- machines by PINNED BOOT MAC (system_id AND hostname are both re-minted on re-enlistment), subnets and VLANs by CIDR, interfaces by name. The pre-migration capture is a rebuild reference for ADDRESSES AND INTERFACE NAMES ONLY: its vlan_id=5005 is dc0 metal-admin in Office1 and vr1-dc0-data-tenant in the new region.
  • Every mutation is READ BACK and compared, and a mismatch is fatal. MAAS has been measured returning a full JSON object while changing nothing, surfacing only later as {"subnet": ["None found with id=N."]}.
  • NIC order comes from lib-hosts.sh NIC_PLANE_ORDER, never lib-net.sh PLANE_CIDRS. The plane->CIDR map is INVERTED from lib-net's own PLANE_NAME rather than retyped, so a plane re-address cannot silently disagree.
  • Role asymmetry is DERIVED from the host token (JUJU_HOST_SUFFIX), never typed. Role nodes take six planes plus an OVS br-ex parented on enp2s0; the Juju controller takes metal-admin + provider-public RAW, no br-ex, and enp3s0..enp6s0 deliberately left on their auto VLANs. The gate asserts that deliberate ABSENCE, so a later session cannot "helpfully" carve them and have everything stay green.
  • build_ovs_brex's guard structure is preserved verbatim from carve-host-interfaces.sh:211-236. The VLAN move sits INSIDE the "br-ex does not exist yet" branch on purpose: a MAAS bridge inherits its parent's VLAN at create, and once br-ex exists the parent is enslaved and a VLAN update is invalid. Restructuring it for symmetry with carve_raw would break re-runnability.
  • A link on the right subnet with the wrong address FAILS; it is never skipped.

What this gate does NOT prove, stated in its own output. The node's actual routing table is a first-boot observation (G17). The carve proves the MAAS MODEL that produces a default route -- a static on provider-public plus gateway_ip on that subnet -- and the absence of that leg is exactly the under-carve that cost three bootstrap attempts on 2026-07-30. A green carve is not a proven default route, and the script says so rather than letting a reader infer it.

Harness: 40/40, and every assertion is mutation-proven. Eight fixtures each supply a failing direction: raw (the measured uncarved state), wrongip, patternb (provider static on the raw NIC with no bridge -- the D-057 shape that darkened the floating-IP plane), memberl3, brexip, jujucarved, jujubrex, nogw.

A MUTATION PASS FOUND ONE ASSERTION THAT COULD NOT FAIL, and a first pass of mutations that proved less than it appeared to. Neutering three assertions by deleting their MESSAGE turned the suite red only via the tests that grep for that message -- which proves the assertion EXISTS, not that its predicate works. Re-run neutering only the PREDICATE and keeping the message, three killed properly and the br-ex static-address compare SURVIVED: patternb was being caught by the type/parent assertions, so the address compare was never exercised. Added the brexip fixture (a correctly-parented OVS bridge carrying the WRONG static) and re-proved the kill.

Revert. git rm scripts/dc-node-carve.sh tests/dc-node-carve/run-tests.sh and bash scripts/run-tests-all.sh --record-manifest. Nothing else depends on it yet.

Item 2 -- tests/HARNESS-MANIFEST re-recorded (92 -> 93)

What. Re-recorded to pin the new harness. Required: the gauntlet's R15(2) drift gate FAILS on a harness that is present but unpinned.

Revert. git checkout <this-commit>^ -- tests/HARNESS-MANIFEST.