QUEUED FINDINGS -- dc0 per-DC MAAS region migration (2026-07-30, session part 5)
================================================================================
Swept at session close. Items are LOGGED, NOT EXECUTED (hard rule 1). Each says where
it already lives, or that this file is its FIRST surface. Precedent: queued-findings
-20260727, -20260729, -20260730, -20260730-stage5.
THE POINT OF THIS SWEEP is the items marked FIRST SURFACE -- they existed only in the
session transcript and would have been lost on a context clear.
F1. THE SESSION'S PERMISSION RULES LIVE ONLY IN A GITIGNORED FILE.
FIRST SURFACE for the verbatim rule text (the changelog describes them; it does not
record them reproducibly).
`.claude/settings.local.json` is gitignored (`~/.config/git/ignore:1`), so if this
clone or the jumphost is rebuilt, every rule added this session is GONE and the next
session hits the same classifier walls with no record of what unblocked them.
Added to "ask" (operator-ruled ask, NOT allow -- hard rule 3 stays intact):
Bash(ssh *'maas * update*)
Bash(ssh *"maas * update*)
Bash(ssh *'maas * ipranges *)
Bash(ssh *"maas * ipranges *)
Added to "allow" (read-only process inspection):
Bash(ssh *'pgrep *) Bash(ssh *"pgrep *)
Bash(ssh *'ps -ef*) Bash(ssh *"ps -ef*)
Bash(timeout * ssh *'pgrep *) Bash(timeout * ssh *"pgrep *)
Bash(timeout * ssh *'ps -ef*) Bash(timeout * ssh *"ps -ef*)
ALSO PRESENT and NOT added deliberately by me: `Bash(ssh vr1-dc0-rack *)` in "allow",
auto-recorded from an interactive approval. That is BROAD -- it permits any command on
the dc0 rack, including destructive ones -- and is worth an explicit review.
WHY THE RULES WERE NEEDED AT ALL: existing rules pin the DOUBLE-quote form
(`ssh voffice1 "maas admin subnet *`) and the `maas admin` profile; the migration used
SINGLE quotes and `maas vr1-dc0-region`. Neither matched, so calls fell through to the
classifier. Quoting style and the profile token are BOTH part of the match.
F2. OFFICE1 STILL RUNS A RACK CONTROLLER ON THE dc0 RACK, AND STILL HOLDS THE REGION VM'S
MACHINE RECORD. FIRST SURFACE for the rack half.
MEASURED at close: `maas admin rack-controllers read` -> voffice1 (mtstwf),
**vvr1-dc0 (7chphy)**, vvr1-dc1 (nmpcq4). So the dc0 rack host is STILL registered as
an Office1 rack controller even though the per-DC region now owns all ten dc0 machines
and serves their DHCP.
NOT CURRENTLY HARMFUL: Office1's DHCP for that VLAN is OFF (verified by process,
`pgrep -c dhcpd` = 0 on the rack), so there is no two-server hazard. But it is a
dual-registration that a later session will find confusing, and the rackd is dead
weight on the rack host.
ALSO: `hot-kid`/`tw7ptw` (the dc0 region VM) remains a Deployed machine record in
OFFICE1 -- deliberately left, since deleting it would leave the region VM
power-unmanaged for no gain during the migration. Now that the migration is complete
that reasoning has expired and the decision should be revisited.
NEITHER IS DEPLOY-BLOCKING. Both are cleanup decisions for the operator.
F3. THE SSH TUNNEL IS A LIVE PROCESS WITH NO SUPERVISION.
ALREADY ON SURFACE (changelog item 2, CURRENT-STATE) including the re-open command,
but restated here because it is the single most likely thing to be broken at the start
of the next session:
ssh voffice1 'ssh -f -N -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2'
The `vr1-dc0-region` MAAS profile points at `127.0.0.1:5241`, so with the tunnel down
EVERY tool targeting the new region refuses. FAILURE DIRECTION IS SAFE (refuse, never
a silent fall-back to Office1), but it will look like a broken region until the tunnel
is checked. `pgrep -f '5241:10.12.8.6'` on voffice1 is the check.
F4. THE dc0 RACK'S NODE-DNS FORWARDER STILL POINTS AT THE OFFICE1 REGION.
ALREADY ON SURFACE (changelog item 9). Restated because the migration has now made it
WRONG rather than merely stale: `/etc/dnsmasq-dc0-node.conf` is `no-resolv` +
`server=10.10.0.20`, so anything on the rack resolving `maas-internal` gets the OLD
region's zone, which no longer holds dc0's node records. `dc-rack-net.sh` carries
`DNS_UPSTREAM="10.10.0.20"` for BOTH sites (`:66`, `:82`); dc1's is still correct, so
this is a per-site change. NODE DNS IS UNAFFECTED -- the metal-admin subnet now serves
`dns_servers=10.12.8.6` (the DC-local region), which is what nodes actually use.
Also noted: the live unit carries the D-119-RETIRED bare token (`dc0-node-dns.service`,
`/etc/dnsmasq-dc0-node.conf`) while the current script generates `vr1-dc0-*` names.
F5. ENLISTMENT/COMMISSIONING apt GOES TO THE INTERNET, NOT THE DC MIRROR.
ALREADY ON SURFACE (changelog item 20 / CURRENT-STATE), restated for the D-135 file.
The ephemeral is NOBLE; the dc0 mirror carries JAMMY + UCA caracal only, so
`archive.ubuntu.com` / `security.ubuntu.com` are fetched through MAAS's own squid proxy
on the region (`10.12.8.6:8000`) -- which is the ONLY egress path the nodes have,
because metal-admin carries NO gateway. This is the practical reason nine concurrent
enlistments stalled and three did not. It also means the D-135/D-107 egress narrowing
would break ENLISTMENT as well as image sync and bootstrap -- one more consumer for
the F10 sequencing question already queued on 2026-07-30.
F6. THE PER-NODE SERIAL LOG STOPPED CAPTURING ON 2026-07-21.
ALREADY ON SURFACE (changelog item 17). Restated because it is a DIAGNOSTIC BLIND SPOT
that cost real time this session: the file still contains a complete, plausible boot
and `tail` returns it confidently. Verify by MTIME before trusting any serial log.
F7. NO AS-EXECUTED LOG FOR THIS WINDOW.
ALREADY ON SURFACE. `run-logged.sh` opens an INTERACTIVE `script(1)` subshell and is
unusable from a non-interactive session. All evidence went to `docs/audit/*` captures
instead, which is what GA-R6 actually requires of a gate. A `logs/as-executed-index.md`
row declaring the gap is OWED.
F8. `maas-role-tags.sh` REPORTS PASS ON AN INCOMPLETE TAG SET.
ALREADY ON SURFACE (changelog item 23) but flagged here as a TOOLING defect rather
than a migration event: its `ROLES` set is control/compute/storage +
`juju-controller-<site>`, so it returns `0 missing / 0 needing / 0 not-in-MAAS` while
every machine still lacks `openstack-<site>` -- the tag `bundle.yaml` places on and
`dc-node-v6-carve.py:89-93` DIES without. A green gate on an incomplete state.
CANDIDATE FIX (not made -- hard rule 1): have it assert the site placement tag too, or
have `dc-region-topology.sh check` assert that every role node carries it.
F9. THE NEXT SESSION'S FIRST COMMANDS, so they are not re-derived.
# 0. tunnel (F3), then prove the region before anything else
ssh voffice1 "cd ~/openstack-caracal-dc-dc && bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid"
# 1. the four green gates, to confirm nothing drifted
MAAS_PROFILE=vr1-dc0-region bash scripts/dc-region-topology.sh check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid
MAAS_PROFILE=vr1-dc0-region bash scripts/maas-role-tags.sh check vr1-dc0
MAAS_PROFILE=vr1-dc0-region bash scripts/dc-plane-ipam.sh check vr1-dc0
# 2. then the ONLY remaining blocker: write scripts/dc-node-carve.sh
Its hard inputs are already MEASURED and test-guarded:
- `NIC_PLANE_ORDER` in lib-hosts.sh -- NOT `PLANE_CIDRS` order. Walking PLANE_CIDRS
positionally swaps enp1s0/enp2s0 and strands PXE on provider-public.
- role asymmetry: role nodes = 6 planes + OVS `br-ex` parented on enp2s0 (the static
lands on br-ex, enp2s0 holds no L3 link); juju controller = 2 planes only
(metal-admin + provider-public raw, WITH gateway), no br-ex, enp3-6 left on auto.
- `carve-host-interfaces.sh:211-236` `build_ovs_brex()` is the correct MAAS call
ORDER: unlink commissioning links -> move parent NIC onto the provider VLAN ->
`interfaces create-bridge bridge_type=ovs` -> `link-subnet mode=STATIC` on br-ex.
- `carve-host-interfaces.sh` itself REFUSES `vr1-*` by design; do not remove that
guard, and do NOT point `phase-00-maas-standup.sh` at the new region (it does not
refuse, and would create space-named fabrics + a VID-103 metal-internal that D-133
abolished).
THEN: `dc-node-v6-carve.py apply vr1-dc0 --commit --profile vr1-dc0-region` (it now
honours MAAS_PROFILE too), then re-point Juju at `10.12.8.6:5240` -- which needs a
region-scoped MAAS API key (the Office1 key will NOT authenticate), an updated cloud +
credential, and the still-owed `juju unregister vr1-dc0-controller` on the rack.
F10. FIVE INSTRUMENT ERRORS IN ONE SESSION, ALL THE SAME SHAPE.
FIRST SURFACE as a consolidated item (each is recorded individually).
A confident NEGATIVE reported from an under-checked instrument: (a) a DNS probe at
`+time=3 +tries=1` called a capability gap -- it was a cold cache; (b) a listener list
truncated by `head -5` called a missing TFTP bind -- it was there; (c) a serial log
nine days stale read as current; (d) a bad elapsed-time estimate called enlistment
stalled when the last line was 90 seconds old; (e) counting 104 "enlist" lines as
progress when they were DHCP renewals -- filtering for `cloud-init` returned 0.
THE RULE THAT CATCHES ALL FIVE, and which this repo already states for inherited
findings but not for one's own live reads: before reporting a negative from a log,
listener list or probe, STATE HOW YOU KNOW THE INSTRUMENT WAS CURRENT -- mtime,
unfiltered output, adequate timeout, and a filter that matches the thing you actually
care about.