Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260807-dc0-tailscale-provisioning.md

Changelog 2026-08-07 -- dc0 Tailscale .7 subnet-router: MAAS provisioning + aux-carve tooling

Session: pick up the 2026-08-06/07 close -- provision the per-DC Tailscale .7 subnet-router toward closing phase-03 Step 3.3 (Horizon over the tailnet). Branch dc-dc-stage5-preconditions. Under blanket approval the changelog is the review surface: each item states WHAT, WHY (evidence), and HOW TO REVERT. Live cloud mutations are individually operator-gated.

Operator rulings this session (GA-R5; verbatim, dated 2026-08-07):

  • dc1 approach: "No migration. Build region on DC1 correctly." -- dc1's .7 is gated behind building the dedicated vr1-dc1-region on vr1-dc1-maas-01 (N3); do NOT enlist it into the Office1 admin region. Consistent with the standing region-per-DC invariant (D-132 q1).
  • Tailscale join key: operator has a plain (not tag-scoped) pre-auth key. The ruled design (D-129(iii) amdt note 1) is TAGGED identity; a plain key cannot authorise --advertise-tags=tag:subnet-router, so the JOIN stays blocked pending a tagged key + the Headscale-side autoApprovers/star-ACL (N1). Provisioning to carved-and-ready is unblocked.

As-executed log (O3, repeated): run-logged.sh opens an interactive script -aqe subshell, which a background agent cannot drive; the log was NOT opened. Every mutation below is gated and captured here with its read-back. Flagged, not silent.

Measured corrections to the record (GA-R1 C2 -- measurement wins)

  • The dc0 .7 VM had already self-enlisted into vr1-dc0-region as known-marten (system_id n4parh), status New, power_type UNSET, powered off -- its six NICs carry the pinned tailscale MACs (opentofu/vr1-dc0-substrate/main.tf:243-245). The 2026-08-06/07 close said "powered off"; the enlisted-already part was not recorded. virsh domstate = shut off.
  • subtle-grouse is the dc0 juju controller (power_id=vr1-dc0-juju-01), never renamed in MAAS; it carries the 2-plane carve because dc-node-carve.sh resolves ROLE from the stable lib-hosts LOGICAL name and machine IDENTITY from the pinned boot MAC -- the MAAS random name is irrelevant. This is the idiom the tailscale aux-carve reuses.
  • vr1-dc1-region is NOT registered; dc1's rack (nmpcq4) + 9 nodes are in the Office1 admin region. dc1 .7 has no dedicated region -> the operator ruling above.

Item 1 -- dc0 .7 VM: MAAS power control set + commissioned to Ready (LIVE, gated)

WHAT: on known-marten (n4parh) in vr1-dc0-region: (a) maas-node-power.sh --commit set power_type=virsh, power_address=qemu+ssh://jessea123@10.12.8.2/system (the DC-region-reachable metal-admin form, read off the working juju controller's power-parameters), power_id=vr1-dc0-tailscale-01. Read-back: [ok] known-marten -> vr1-dc0-tailscale-01 (power state: off) -- MAAS proved power control by a live query-power-state. (b) maas vr1-dc0-region machine commission n4parh -> polled Commissioning -> Testing -> Ready at 02:55:33Z.

WHY: the .7 router must be MAAS-deployed to get an OS (it is NOT in the juju model, so nothing else deploys it -- DOCFIX-200's "stay Ready" applies only to juju-managed role nodes). Power MUST be set before commission (the unset-power silent drop-to-New trap). The power address is the region-reachable 10.12.8.2 (measured OPEN from the DC-local region; the Office1-form 172.31.0.2 is CLOSED from there); listed domains over the transit 172.31.0.2 via VIRSH_URI.

REVERT: maas vr1-dc0-region machine release n4parh (back to Ready) or ... machine delete n4parh + power off the domain; the substrate VM is untouched.

Item 2 -- dc-node-carve.sh + lib-hosts.sh: AUX carve target for the Tailscale .7 router

WHAT: taught the v4 node carve to carve the per-DC Tailscale .7 VM, which is NOT an OpenStack role node:

  • scripts/lib-hosts.sh: new CARVE_AUX_HOSTS array (default empty; ( vr1-dc0-tailscale-01 ) / ( vr1-dc1-tailscale-01 ) per DC) so the VM is a carve target WITHOUT entering HOSTS (every HOSTS consumer -- maas-role-tags, pre-flight-checks ${#HOSTS[@]}, phase-00-teardown, reenroll-hosts, dc-node-v6-carve -- would otherwise treat it as a role node). Inert HOST_OCTET[..]=7 + HOST_BOOT_MAC entries (map keys are looked-up, never iterated). New NODE_CARVE_TAILSCALE_PLANES=2 + TAILSCALE_HOST_SUFFIX.
  • scripts/dc-node-carve.sh: is_tailscale_host + is_two_plane_host (juju OR tailscale); planes_for_host and BOTH br-ex branches + the "spare NICs / NO br-ex" verifier now key on is_two_plane_host -- so the tailscale VM carves metal-admin + provider-public RAW with NO OVS br-ex (identical to the juju controller, octet 7). --host accepts CARVE_AUX_HOSTS members; a bare apply still iterates HOSTS only (aux = opt-in by --host).
  • tests/dc-node-carve/run-tests.sh: t01 machine fixture + emit_tailscale + T28..T33 -- passes on the 2-plane no-br-ex end state at 10.12.8.7/10.12.4.7, and FAILS (failing- direction fixtures) on a tailscale VM grown a br-ex (tsbrex) or with a spare NIC carved onto a plane (tscarved); T31 proves the aux --host allowance did not widen unknown hosts.

WHY: hard rule 4 -- no tested artifact carved this 2-leg shape (the existing tool is role/juju shaped; a mis-key silently falls through to a 6-plane br-ex role carve). Implements the ruled D-129(iii) amendment (2026-08-07) + D-134 octet map; no new D-number. The carve must precede MAAS-deploy (MAAS refuses interface changes on a Deployed machine).

GATES: tests/dc-node-carve/run-tests.sh 58 passed / 0 failed; repo-lint 0 fail (1 legacy warn); full gauntlet ALL GREEN (101 harnesses) -- docs/audit/gauntlet-20260807-tailscale-carve.txt.

REVERT: git revert this commit -- removes CARVE_AUX_HOSTS, the tailscale predicates, and the harness cases; dc-node-carve.sh returns to role+juju only. lib-hosts map keys become inert. No live-cloud effect (this item is repo tooling; the carve APPLY is a separate gated mutation).

Item 3 -- dc0 .7 VM: 2-leg carve applied + verified (LIVE, gated)

WHAT: dc-node-carve.sh apply vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid --host vr1-dc0-tailscale-01 --commit -> 4 mutations, each read back:

  • enp1s0: unlinked the commissioning auto link (id 1219) -> STATIC 10.12.8.7 on metal-admin (VLAN 5001, already correct -- move skipped);
  • enp2s0: -> VLAN 5002 (provider-public) -> STATIC 10.12.4.7. Post-apply check = pass=8 fail=0: both legs static on the right VLANs, enp3s0..enp6s0 carry NO plane link, NO br-ex. Dry-run check beforehand FAILED (uncarved) and the dry-run plan matched exactly -- verify-before-mutate.

WHY: MAAS renders netplan from these links at deploy time; the carve must land while Ready (MAAS refuses interface changes on a Deployed machine). metal-admin is gatewayless (advertised plane); provider-public carries the subnet gateway 10.12.4.1 (the node default route -> egress to Headscale). br-ex is deliberately absent (not an OVN node).

REVERT: re-link enp1s0/enp2s0 to auto/link_up via maas vr1-dc0-region interface link-subnet/unlink-subnet, or release+recommission the machine (returns interfaces to the commissioning state). No downstream consumer depends on the .7 statics yet.

Item 4 -- dc0 .7 VM: MAAS-deployed jammy (LIVE, gated)

WHAT: maas vr1-dc0-region machine deploy n4parh distro_series=jammy -> polled Deploying -> Deployed at 03:27:05Z, power on. The .7 router is NOT in the juju model, so it must be MAAS-deployed to carry an OS (DOCFIX-200's "stay Ready" applies only to juju-managed nodes). jammy chosen = the established DC image (role nodes) and confirmed deployable (boot-resources read: ubuntu/jammy + ubuntu/noble). Set distro_series explicitly (hard rule 2).

READ-BACK: the VM's sshd answers on metal-admin 10.12.8.7. BOTH carved legs rendered and are live -- from the dc0 rack (provider-public leg MEASURED virbr5 10.12.4.2/22, not inferred): ping 10.12.8.7 0% loss AND ping 10.12.4.7 0% loss. So netplan rendered both legs; the provider-public egress leg (the reason the leg exists) is up. The in-VM DEFAULT ROUTE via 10.12.4.1 is the only unproven piece (needs in-VM access; a G17-class first-boot fact, exercised at join time by site-tailscale.sh install). See Item 5 for the access gap.

REVERT: maas vr1-dc0-region machine release n4parh -> back to Ready (the carve persists).

Item 5 -- carved-and-ready state + the two remaining JOIN prerequisites (FINDING)

STATE reached this session: dc0 .7 router = Deployed (jammy) + carved (verified) + sshd live on 10.12.8.7. This is "carved-and-ready": the provisioning is complete; only the Tailscale join remains. Two prerequisites, BOTH outside this session:

  1. Tagged pre-auth key + Headscale-side policy (N1). The operator's key is PLAIN (not tag-scoped); site-tailscale.sh install advertises --advertise-tags=tag:subnet-router, which a plain key cannot authorise, and the route needs autoApprovers/manual approval + the star ACL on tailscale.baldurkeep.com. Deliberately NOT attempted with the plain key: a botched first advertise joins the node UNTAGGED (the office1 key-expiry defect, R5) and burns the autoApprovers window (Headscale does not approve retroactively). N4 SEC row (per-DC Tailscale key custody) opens at tagged-key mint.

  2. SSH access to the .7 VM for the install. MEASURED: vr1-dc0-region injected only the vr1-office1-svc key (region sshkeys read); neither voffice1's default identity nor the per-DC ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 authenticate as ubuntu@10.12.8.7 (both Permission denied (publickey)). The working path for site-tailscale.sh install is the holder of vr1-office1-svc over ssh -J voffice1,<rack> ubuntu@10.12.8.7 (the operator has this Office1 key; I did not fish for it -- secrets discipline). OBSERVATION (logged, not actioned -- hard rule 1): the region injecting the Office1 service key rather than a per-DC key is a cross-DC key-reuse pattern in tension with SEC-012/SEC-016 (per-DC key isolation); it affects ALL dc0-region deploys, is pre-existing, and is a custody-hygiene item for the D-137 fork, not a deploy blocker.

WHY carved-and-ready is the right stop: the join cannot complete without (1), which is the operator's Headscale work. The session leaves the VM one tailscale up from the Horizon close.

REVERT: n/a (finding + state record).