QUEUED FINDINGS -- Stage-5 open + per-DC region session (2026-07-30)
====================================================================
Swept at session close. Items are LOGGED, NOT EXECUTED (hard rule 1). Each says where it
already lives, or that this file is its first surface. Precedent: queued-findings-20260727,
-20260729, -20260730.
F1. THE `openstack` CLI DOES NOT EXIST ON THE DC CLIENT HOST -- blocks Stage 5 Step 7+.
FIRST SURFACE: this file (was transcript + task list only).
D-138 moved cloud-facing tools INTO the DC, but `python3-openstackclient 6.6.0` was
installed on voffice1 on 2026-07-27 (CURRENT-STATE section 7) and is NOT on the dc0
rack or the region VM. Ten scripts invoke it -- phase-03-admin-openrc.sh,
phase-04-network-{create,verify}.sh, phase-04-internal-cert-san-verify.sh,
phase-05-{amphora-pipeline,octavia-verify}.sh, phase-06-{bootstrap,capi-stack,mgmt-vm,
net-setup}.sh -- all against keystone's VIP 10.12.4.50 on provider-public, which
voffice1 cannot reach. MEASURED PIN, do not re-derive: the snap is REFUTED (no Caracal
channel; newest stable `zed` 2023-03), and noble's native OpenStack release IS Caracal
2024.1, so the deb is correct and needs no UCA. Install on the DC client host before
Step 7 and add the host to CURRENT-STATE section 7.
F2. metal-admin v6 HAS NODE ADDRESSES BUT NO RACK-SIDE LEG.
FIRST SURFACE: this file (was transcript + task list only).
MEASURED on the dc0 rack: `ip -6 -o addr show dev virbr2` returns EMPTY and ping6 to
the controller gives "Network is unreachable", while v4 10.12.8.5 answers 0% loss. The
2026-07-27 IPv6 carve assigned v6 statics to NODES in MAAS but `dc-rack-net.sh` adds v4
legs only (10.12.8.2/.3/.4). NOT BLOCKING: juju fell back to v4 unaided, and
node-to-node / agent-to-controller v6 on the same L2 needs no rack leg because every
node holds a v6 static on the same /64. CONSEQUENCE TO CHECK LATER: anything
RACK-ORIGINATED toward a node over v6 fails, and D-101 makes metal-admin dual-stack.
Verify per plane per DC before concluding scope -- likely true on dc1 and on the other
five planes.
F3. PLANE FABRICS READ mtu=1500 WHILE THE LIBVIRT NETWORKS ARE mtu 9000.
FIRST SURFACE: this file.
From docs/audit/dc0-maas-carve-premigration-20260730.txt: all five named dc0 plane
fabrics report `mtu=1500`. `virsh net-dumpxml vr1-dc0-metal-admin` reports `<mtu
size='9000'/>`. PRE-EXISTING -- not introduced by this session's work, and not chased
here (hard rule 1). Why it matters: Ceph and geneve are the classic jumbo-frame
consumers, phase-4 Step 12 carries a jumbo gate, and CURRENT-STATE already records "two
lagging MTU segments" from the 2026-07-27 audit. Establish which layer is authoritative
before the Stage-5 storage/network gates run.
F4. `ceph-osd` CARRIES THE STALE VR0 CONSTRAINT `tags=openstack` (bundle.yaml:592).
ALREADY ON SURFACE: docs/CURRENT-STATE.md section 1 + changelog item 2. Repeated here
only so the sweep is complete. Decision is owed at Step 4.2's dry-run, where its impact
is first observable. Measured absent from the VR1 region; only app of 56 with a tag
constraint.
F5. DOCFIX OWED (next-free 206; assign at delivery, never write the token above the
high-water mark in prose first). TWO corrections, one DOCFIX:
(a) `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 1's RUN LOCATION banner says
every juju and maas command runs on voffice1. D-138 amends that for CLOUD-FACING
tools; maas/NetBox/inner-tofu stay on voffice1. Steps 2, 3.5, 4, 4b and 5-12 all
carry run-location text now wrong for the juju half.
(b) Step 2's bootstrap command reads `--constraints` only; juju 3.6.27 assigns
machine-targeting to `--bootstrap-constraints`. Operator ruled "Use both flags".
D-104's amendment bullet carries the same single-flag wording.
ALREADY ON SURFACE: changelog item 3 (the flag half). The run-location half is NEW here.
NO GATE READS PROSE, so nothing in this repo catches either.
F6. THE AS-EXECUTED LOG FOR THIS SESSION IS PARTIAL, AND THAT IS RECORDED NOT HIDDEN.
FIRST SURFACE: this file (+ the index row added this session).
The DOCFIX-097 Claude-Code lane wraps each live command as
`script -aqe ~/as-executed/<date>-<label>.log -c '<cmd>'`. MEASURED: the harness
classifier REFUSED several wrapped forms (`script ... | tail`, `script ... | python3`,
and two bare `script -aqe ... -c 'ssh voffice1 ...'` MAAS calls), while the plain
`ssh voffice1 'maas admin ...'` form passed. Those calls therefore ran UNWRAPPED and are
NOT in the log. Everything they did is in the changelog and CURRENT-STATE with
read-backs, but a future session reading the as-executed log must not treat it as a
complete record of this window. Candidate fix: a targeted permission rule for the
wrapped form, so discipline and tooling stop fighting.
F7. THE dc0 MANIFEST CARRIES A LITERAL `<stamp>` PLACEHOLDER THAT CAN NEVER MATCH.
PRE-EXISTING (from the 2026-07-30 part-2 reissue work), surfaced by this session's
`creds-audit vr1-dc0` run: 2 problems -- MISSING
`vr1-dc0-octavia-pki-<stamp>.tar.gz` and UNDECLARED
`vr1-dc0-octavia-pki-20260730T070146Z-1520866.tar.gz`. The manifest declares a
placeholder filename rather than a glob, so the real timestamped backup is permanently
both missing and undeclared. NOT this session's defect and NOT fixed here.
F8. D-104's AMENDMENT PROSE SAYS "1 add / 0 change / 0 destroy"; MEASURED IT IS 2.
ALREADY ON SURFACE: CURRENT-STATE + changelog item 7. `modules/node-vm` creates a
`libvirt_domain` AND a `libvirt_volume` per node. The half that protects you --
`0 change / 0 destroy` -- is correct. D-104's own text is wrong at resource level for
the same reason; flagged, not edited (not this item's surface).
F9. TOOL TRAPS MEASURED THIS SESSION -- candidates for `references/platform-traps.md`.
FIRST SURFACE for most: this file.
(a) `ssh -n` REDIRECTS STDIN FROM /dev/null, so `ssh -n host bash -s <<'EOF'` silently
runs NOTHING. Cost this session: a credential shred that never executed and a DB
step that never ran. Conversely, an inner `ssh` WITHOUT `-n` EATS the remaining
heredoc from the outer script -- which is what made that shred silently skip. Both
directions bite; pick per call and verify by reading state back.
(b) `printf ... | grep -q` UNDER `set -o pipefail` RETURNS FAILURE. grep short-circuits,
printf takes SIGPIPE. Already recorded once this repo-day; it false-FAILED the G17
mirror assertion. Use a here-string: `grep -q PATTERN <<<"$BODY"`.
(c) SNAP CONFINEMENT: the juju snap cannot read `/tmp` (private mount), so
`juju add-cloud -f /tmp/x.yaml` fails with "no such file". Use `$HOME`. The repo
already records the sibling trap for `openstackclients`.
(d) A MAAS WRITE CAN RETURN A FULL 200-SHAPED JSON OBJECT AND CHANGE NOTHING.
`interface update <sys> <iface> vlan=` on a `Deployed` machine returned the whole
interface object while leaving the VLAN untouched; the refusal only surfaced later
and indirectly as `{"subnet": ["None found with id=7."]}`. ALWAYS read the
interface back. MAAS accepts interface changes only in New/Ready/Allocated/Broken.
(e) `creds-matrix.tsv` IS SPACE-ALIGNED, NOT TAB-SEPARATED. `awk -F'\t'` returns zero
rows and looks exactly like a real result. Already a standing lesson; it recurred.
(f) `boot-resources import` RACES a just-created selection: the first import returned
noble-only with `is-importing=false`. Assert on `Synced` ROWS, never on the import
call returning.
(g) PINNING MACs BOUNCES THE GUEST AND INTERRUPTS THE AUTO-COMMISSION THE FIRST APPLY
TRIGGERED. Already promoted to CURRENT-STATE as a standing DC-standup rule.
F10. THE D-135/D-107 EGRESS NARROWING WOULD BREAK IMAGE SYNC AND JUJU BOOTSTRAP OUTRIGHT.
ALREADY ON SURFACE: CURRENT-STATE (this session). Repeated here because it is the
highest-consequence unruled item left. MEASURED: three artifact classes, ONE local --
apt packages from 10.12.8.4 (local); MAAS boot images from images.maas.io (the mirror
404s on every simplestreams path); juju agent stream + juju/juju-db snaps from
streams.canonical.com / api.snapcraft.io. The narrowing is described as "the closing
mutation" and NOTHING currently sequences it after these. Needs either a ruling that
sequences it explicitly, or D-135 items 2-3 built first.
F11. THE JUJU CONTROLLER'S v6 GUA LEG ON provider-public WAS DELIBERATELY NOT APPLIED.
ALREADY ON SURFACE: CURRENT-STATE. Role nodes carry `2602:f3e2:f02:10::NNN` on br-ex;
the controller has v4 `10.12.4.5` only. A default route needs only the v4 leg, and
handing a controller a public GUA is an operator call. Recorded so a later session does
not silently "fix" the asymmetry.
F12. TWO RULINGS TODAY REST ON WORK THAT IS AUTHORED BUT NOT APPLIED AT dc1.
`vr1-dc1-maas-01` is in the substrate root and NOT applied; dc1's three region
credentials are EXPECTED by the matrix and correctly FAIL S2. Both are deliberate --
one DC proves the sequence first -- but neither should be read as an oversight.