Newer
Older
openstack-caracal-dc-dc / docs / audit / queued-findings-20260730-stage5.txt
QUEUED FINDINGS -- Stage-5 open + per-DC region session (2026-07-30)
====================================================================
Swept at session close. Items are LOGGED, NOT EXECUTED (hard rule 1). Each says where it
already lives, or that this file is its first surface. Precedent: queued-findings-20260727,
-20260729, -20260730.

F1. THE `openstack` CLI DOES NOT EXIST ON THE DC CLIENT HOST -- blocks Stage 5 Step 7+.
    FIRST SURFACE: this file (was transcript + task list only).
    D-138 moved cloud-facing tools INTO the DC, but `python3-openstackclient 6.6.0` was
    installed on voffice1 on 2026-07-27 (CURRENT-STATE section 7) and is NOT on the dc0
    rack or the region VM. Ten scripts invoke it -- phase-03-admin-openrc.sh,
    phase-04-network-{create,verify}.sh, phase-04-internal-cert-san-verify.sh,
    phase-05-{amphora-pipeline,octavia-verify}.sh, phase-06-{bootstrap,capi-stack,mgmt-vm,
    net-setup}.sh -- all against keystone's VIP 10.12.4.50 on provider-public, which
    voffice1 cannot reach. MEASURED PIN, do not re-derive: the snap is REFUTED (no Caracal
    channel; newest stable `zed` 2023-03), and noble's native OpenStack release IS Caracal
    2024.1, so the deb is correct and needs no UCA. Install on the DC client host before
    Step 7 and add the host to CURRENT-STATE section 7.

F2. metal-admin v6 HAS NODE ADDRESSES BUT NO RACK-SIDE LEG.
    FIRST SURFACE: this file (was transcript + task list only).
    MEASURED on the dc0 rack: `ip -6 -o addr show dev virbr2` returns EMPTY and ping6 to
    the controller gives "Network is unreachable", while v4 10.12.8.5 answers 0% loss. The
    2026-07-27 IPv6 carve assigned v6 statics to NODES in MAAS but `dc-rack-net.sh` adds v4
    legs only (10.12.8.2/.3/.4). NOT BLOCKING: juju fell back to v4 unaided, and
    node-to-node / agent-to-controller v6 on the same L2 needs no rack leg because every
    node holds a v6 static on the same /64. CONSEQUENCE TO CHECK LATER: anything
    RACK-ORIGINATED toward a node over v6 fails, and D-101 makes metal-admin dual-stack.
    Verify per plane per DC before concluding scope -- likely true on dc1 and on the other
    five planes.

F3. PLANE FABRICS READ mtu=1500 WHILE THE LIBVIRT NETWORKS ARE mtu 9000.
    FIRST SURFACE: this file.
    From docs/audit/dc0-maas-carve-premigration-20260730.txt: all five named dc0 plane
    fabrics report `mtu=1500`. `virsh net-dumpxml vr1-dc0-metal-admin` reports `<mtu
    size='9000'/>`. PRE-EXISTING -- not introduced by this session's work, and not chased
    here (hard rule 1). Why it matters: Ceph and geneve are the classic jumbo-frame
    consumers, phase-4 Step 12 carries a jumbo gate, and CURRENT-STATE already records "two
    lagging MTU segments" from the 2026-07-27 audit. Establish which layer is authoritative
    before the Stage-5 storage/network gates run.

F4. `ceph-osd` CARRIES THE STALE VR0 CONSTRAINT `tags=openstack` (bundle.yaml:592).
    ALREADY ON SURFACE: docs/CURRENT-STATE.md section 1 + changelog item 2. Repeated here
    only so the sweep is complete. Decision is owed at Step 4.2's dry-run, where its impact
    is first observable. Measured absent from the VR1 region; only app of 56 with a tag
    constraint.

F5. DOCFIX OWED (next-free 206; assign at delivery, never write the token above the
    high-water mark in prose first). TWO corrections, one DOCFIX:
    (a) `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 1's RUN LOCATION banner says
        every juju and maas command runs on voffice1. D-138 amends that for CLOUD-FACING
        tools; maas/NetBox/inner-tofu stay on voffice1. Steps 2, 3.5, 4, 4b and 5-12 all
        carry run-location text now wrong for the juju half.
    (b) Step 2's bootstrap command reads `--constraints` only; juju 3.6.27 assigns
        machine-targeting to `--bootstrap-constraints`. Operator ruled "Use both flags".
        D-104's amendment bullet carries the same single-flag wording.
    ALREADY ON SURFACE: changelog item 3 (the flag half). The run-location half is NEW here.
    NO GATE READS PROSE, so nothing in this repo catches either.

F6. THE AS-EXECUTED LOG FOR THIS SESSION IS PARTIAL, AND THAT IS RECORDED NOT HIDDEN.
    FIRST SURFACE: this file (+ the index row added this session).
    The DOCFIX-097 Claude-Code lane wraps each live command as
    `script -aqe ~/as-executed/<date>-<label>.log -c '<cmd>'`. MEASURED: the harness
    classifier REFUSED several wrapped forms (`script ... | tail`, `script ... | python3`,
    and two bare `script -aqe ... -c 'ssh voffice1 ...'` MAAS calls), while the plain
    `ssh voffice1 'maas admin ...'` form passed. Those calls therefore ran UNWRAPPED and are
    NOT in the log. Everything they did is in the changelog and CURRENT-STATE with
    read-backs, but a future session reading the as-executed log must not treat it as a
    complete record of this window. Candidate fix: a targeted permission rule for the
    wrapped form, so discipline and tooling stop fighting.

F7. THE dc0 MANIFEST CARRIES A LITERAL `<stamp>` PLACEHOLDER THAT CAN NEVER MATCH.
    PRE-EXISTING (from the 2026-07-30 part-2 reissue work), surfaced by this session's
    `creds-audit vr1-dc0` run: 2 problems -- MISSING
    `vr1-dc0-octavia-pki-<stamp>.tar.gz` and UNDECLARED
    `vr1-dc0-octavia-pki-20260730T070146Z-1520866.tar.gz`. The manifest declares a
    placeholder filename rather than a glob, so the real timestamped backup is permanently
    both missing and undeclared. NOT this session's defect and NOT fixed here.

F8. D-104's AMENDMENT PROSE SAYS "1 add / 0 change / 0 destroy"; MEASURED IT IS 2.
    ALREADY ON SURFACE: CURRENT-STATE + changelog item 7. `modules/node-vm` creates a
    `libvirt_domain` AND a `libvirt_volume` per node. The half that protects you --
    `0 change / 0 destroy` -- is correct. D-104's own text is wrong at resource level for
    the same reason; flagged, not edited (not this item's surface).

F9. TOOL TRAPS MEASURED THIS SESSION -- candidates for `references/platform-traps.md`.
    FIRST SURFACE for most: this file.
    (a) `ssh -n` REDIRECTS STDIN FROM /dev/null, so `ssh -n host bash -s <<'EOF'` silently
        runs NOTHING. Cost this session: a credential shred that never executed and a DB
        step that never ran. Conversely, an inner `ssh` WITHOUT `-n` EATS the remaining
        heredoc from the outer script -- which is what made that shred silently skip. Both
        directions bite; pick per call and verify by reading state back.
    (b) `printf ... | grep -q` UNDER `set -o pipefail` RETURNS FAILURE. grep short-circuits,
        printf takes SIGPIPE. Already recorded once this repo-day; it false-FAILED the G17
        mirror assertion. Use a here-string: `grep -q PATTERN <<<"$BODY"`.
    (c) SNAP CONFINEMENT: the juju snap cannot read `/tmp` (private mount), so
        `juju add-cloud -f /tmp/x.yaml` fails with "no such file". Use `$HOME`. The repo
        already records the sibling trap for `openstackclients`.
    (d) A MAAS WRITE CAN RETURN A FULL 200-SHAPED JSON OBJECT AND CHANGE NOTHING.
        `interface update <sys> <iface> vlan=` on a `Deployed` machine returned the whole
        interface object while leaving the VLAN untouched; the refusal only surfaced later
        and indirectly as `{"subnet": ["None found with id=7."]}`. ALWAYS read the
        interface back. MAAS accepts interface changes only in New/Ready/Allocated/Broken.
    (e) `creds-matrix.tsv` IS SPACE-ALIGNED, NOT TAB-SEPARATED. `awk -F'\t'` returns zero
        rows and looks exactly like a real result. Already a standing lesson; it recurred.
    (f) `boot-resources import` RACES a just-created selection: the first import returned
        noble-only with `is-importing=false`. Assert on `Synced` ROWS, never on the import
        call returning.
    (g) PINNING MACs BOUNCES THE GUEST AND INTERRUPTS THE AUTO-COMMISSION THE FIRST APPLY
        TRIGGERED. Already promoted to CURRENT-STATE as a standing DC-standup rule.

F10. THE D-135/D-107 EGRESS NARROWING WOULD BREAK IMAGE SYNC AND JUJU BOOTSTRAP OUTRIGHT.
    ALREADY ON SURFACE: CURRENT-STATE (this session). Repeated here because it is the
    highest-consequence unruled item left. MEASURED: three artifact classes, ONE local --
    apt packages from 10.12.8.4 (local); MAAS boot images from images.maas.io (the mirror
    404s on every simplestreams path); juju agent stream + juju/juju-db snaps from
    streams.canonical.com / api.snapcraft.io. The narrowing is described as "the closing
    mutation" and NOTHING currently sequences it after these. Needs either a ruling that
    sequences it explicitly, or D-135 items 2-3 built first.

F11. THE JUJU CONTROLLER'S v6 GUA LEG ON provider-public WAS DELIBERATELY NOT APPLIED.
    ALREADY ON SURFACE: CURRENT-STATE. Role nodes carry `2602:f3e2:f02:10::NNN` on br-ex;
    the controller has v4 `10.12.4.5` only. A default route needs only the v4 leg, and
    handing a controller a public GUA is an operator call. Recorded so a later session does
    not silently "fix" the asymmetry.

F12. TWO RULINGS TODAY REST ON WORK THAT IS AUTHORED BUT NOT APPLIED AT dc1.
    `vr1-dc1-maas-01` is in the substrate root and NOT applied; dc1's three region
    credentials are EXPECTED by the matrix and correctly FAIL S2. Both are deliberate --
    one DC proves the sequence first -- but neither should be read as an oversight.