Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260723-queue-pass.md
@JANeumatrix JANeumatrix 15 hours ago 11 KB G13 CLOSED: office1 edge bundle -- qga channel live, 26.7.1, os-iperf in (operator-approved)
Raw Blame History

Changelog -- 2026-07-23 session: queue pass (post-G12 queued findings)

Session scope: process the queued-findings backlog left open at the G12/dc1 close (ledger 2026-07-23 close entry: "3 findings"; plus the four tool gaps queued in changelog-20260722-g12-dc1-build.md items 7/9). Repo deliverables only -- NO cloud mutation this session. Each item carries its revert.

Verified up front: the netem runbook Step-11 divergence flagged at step E is ALREADY fixed (runbook targets the dc0<->dc1 leg, 07-21 captures cited) -- off the queue, nothing to do.

Item 1 -- opnsense-prep-image.sh survives being piped (finding 07-22 c)

  • WHAT: removed the unused HERE="$(dirname "${BASH_SOURCE[0]}")" line (with a comment stating why it must not come back). Under ssh 'bash -s' < script BASH_SOURCE is unset and set -u killed the script at line 20, before its first guard (measured on the dc1 build, 2026-07-22 changelog item 7c). The variable was referenced nowhere -- deletion IS the minimal fix.
  • HARNESS: new T3b pipes the script via bash -s and asserts it reaches the mirror guard (no "unbound variable"). 4/4 + 1 SKIP (environmental).
  • REVERT: restore the HERE line; drop T3b.

Item 2 -- site-headend-install.sh emitted hints made version/site-neutral (finding 07-22 a)

  • WHAT: the --check/setup hint prose hardcoded opnsense-prep-image.sh 26.1 (while 26.7 is the ruled version) and cd opentofu/vr1-dc0-substrate (while parameterized for any DC). Hints now say <ruled-version> pointing at the version authority (CURRENT-STATE section 7) and vr1-<dc>-substrate. NOT changed (value, not prose -- out of the finding's cosmetic scope): the OPNSENSE_BASE default filename still embeds 26.1; every real standup passes --opnsense-base explicitly. Flagged here, not fixed.
  • HARNESS: new cases assert the emitted hints carry no 26.1, no vr1-dc0-substrate, and do point at CURRENT-STATE.
  • REVERT: restore the two echo lines; drop the three assertions.

Item 3 -- nftables-sec010.nft generated idempotent (finding 07-22 b)

  • WHAT: nft -f on a live table APPENDS, so a sec010-fw.service restart (or install re-run) DUPLICATED the FORWARD-drop rules -- observed live on voffice1 2026-07-22, cleaned by hand. The generated file now opens with the declare-then-delete preamble (table inet sec010 {} + delete table inet sec010) so any reload converges to exactly one rule pair. No destroy keyword (older-nft-safe).
  • LIVE STATE: re-asserted same session under operator approval -- see Item 8 (all three hosts converged, pre/post captured).
  • HARNESS: asserts the preamble is present in the generator.
  • REVERT: restore the old heredoc; drop the assertion.

Item 4 -- serial-log staging dir joins --host-nodes standup (finding 07-22 item 9)

  • WHAT: site-headend-install.sh --host-nodes now creates /var/lib/libvirt/vr1/staging (0755; literal keyed to modules/opnsense-edge's hardcoded serial-log path) and --check verifies it. Hand-created on dc0; its absence failed the dc1 inner apply at 18/28.
  • HARNESS: 'staging' added to the node-host step assertions. 59/59 PASS.
  • REVERT: drop the run/check lines + the assertion.

Item 5 -- opnsense-bootstrap-apikey.sh: sshd readiness wait + first-scp retry (finding 07-23)

  • WHAT (all three measured on the dc1 mint, 2026-07-23): (a) bounded readiness probe (6 x 5s) before shipping anything -- the edge sshd is restarting right after the console bootstrap's configctl openssh restart and first contact got scp: Connection closed; fails LOUD exit 2 if the edge never answers. (b) one retry on the first scp. (c) -n on the base ssh opts -- an inner ssh without it EATS a piped bash -s driver's stdin (the swallowed retrieve/cleanup lines). Plus the dc0-queued usage note: the minter PHP must sit BESIDE the script when staged to a rack. OPNSENSE_BOOTSTRAP_RETRY_DELAY env exists ONLY so the harness need not sleep for real.
  • HARNESS: new T9 (stateful scp stub -- first ship fails, retry completes the mint) + T10 (dead edge -> bounded loud exit 2, no creds file). 10/10 PASS.
  • REVERT: restore the pre-item SSH/SCP block and scp line; drop T9/T10.

Item 6 -- DOCFIX-198: phase2 runbook Steps 9-10 superseded + DC standup definition-of-done

  • WHAT: runbooks/dc-dc-phase2-tofu-dc-substrate.md Steps 9-10 still prescribed the modules/maas-vm-host virsh-POD route and called D-123 "PROPOSED, recommend Model A" -- refuted/retired by the D-103/D-123 amendments (2026-07-20) but never re-pointed. Added SUPERSEDED banners on both steps (per-machine power_type=virsh via maas-node-power.sh is the ruled path; pod-delete cascade trap cited) and a new "DC standup definition-of-done" section after Step 12 -- the 7-row as-executed checklist (measured reach/site-baseleg; dc-rack-net twin; staging dir; region MAAS subnet config; per-DC power key incl. the SEC-016 snap-refresh RE-ASSERT fragility; per-machine power; commissioning 9/9 gate). This lands the two "queued to DC standup DoD" obligations (staging dir; snap-key re-assert) in their runbook home.
  • REVERT: remove the two banners + the DoD section.

Item 7 -- DOCFIX-199: D-112(c) console-bootstrap payload requirements folded into the runbook

  • WHAT: the replacement-chain "Console bootstrap" step now carries the measured payload requirements (dc0 lessons i-v + dc1 confirmation): <=160-char b64 chunks; require_once("util.inc") (shell_safe -- the dc1 abort; any surviving dc0 .b64 rack artifact LACKS it, do not replay); local_user_set() to materialize authorized_keys; tcsh no-quoted-one-liners; ssh -n when driven via bash -s; the post-restart sshd transient (now handled by Item 5). This closes the "fold util.inc into the canonical console bootstrap payload" queue item -- the runbook checklist IS the canonical payload spec (no repo-carried payload artifact exists; authoring one is a possible future deliverable, logged here, not started).
  • REVERT: remove the requirements block.

Item 8 -- LIVE re-assert of /etc/nftables-sec010.nft on all three hosts (operator-approved)

  • RULING (GA-R5): question "Re-assert the idempotent nftables-sec010 file on the three live hosts now?" -- operator answer, exact utterance (option selected): "Approve -- run now (Recommended)". Logged window ops-sec010-reassert (index row added); each host an individually gated mutation.
  • WHAT: on vvr1-dc0 (172.31.0.2), vvr1-dc1 (172.31.0.6), voffice1 -- backup to /etc/nftables-sec010.nft.pre-reassert-20260723, inserted the declare-then-delete preamble after the shebang IN PLACE (voffice1's hand-extended two-leg file preserved verbatim), then restarted sec010-fw TWICE per host: rule counts stable at 2/2, 2/2, 4/4 drops -- the double restart is the proof the append-duplication defect is gone. Pre+post capture: docs/audit/sec010-reassert-20260723.txt. SEC-010 enforcement never gapped (delete+recreate inside one atomic nft -f).
  • REVERT (per host): sudo cp -a /etc/nftables-sec010.nft.pre-reassert-20260723 /etc/nftables-sec010.nft && sudo systemctl restart sec010-fw.

Item 9 -- G13 plugin leg on office1-opnsense: PARTIAL (operator-approved)

  • RULING (GA-R5): question "Run the D-129 vr1-edge plugin install on office1-opnsense now (dry-run, then gated apply)?" -- operator answer, exact utterance (option selected): "Approve -- run now (Recommended)".
  • WHAT RAN (logged window ops-sec010-reassert): dry-run clean (exactly the two ruled install POSTs); live apply via the D-113 REST firmware API. RESULT MEASURED by read-back (core/firmware/info): os-qemu-guest-agent 1.3 installed=1; os-iperf installed=0 -- REFUSED by the edge, upgradestatus log verbatim: "Installation out of date. The update to opnsense-26.7.1 is required."
  • FINDING (logged, not executed): new blocker for the os-iperf half -- the edge wants the 26.7.1 minor update before further package installs. That update is a SEPARATE gated maintenance decision (restart-bearing; natural bundle with the D-129 qga channel retrofit, which waits on exactly such a restart). NOTHING further was mutated after the refusal.
  • G13 row updated in CURRENT-STATE (same commit). qga package present but agent non-functional until the channel retrofit (known, by design).
  • REVERT (installed half): remove os-qemu-guest-agent via the same firmware API (POST core/firmware/remove/os-qemu-guest-agent).

Item 10 -- G13 CLOSED: office1 edge maintenance bundle (operator-approved)

  • RULING (GA-R5): question "Run the office1 edge maintenance bundle now (qga channel + 26.7.1 update + finish os-iperf)?" -- operator answer, exact utterance (option selected): "Approve full bundle (Recommended)".
  • EXECUTED (logged window ops-sec010-reassert; each mutation gated):
    1. expose_qga_channel = true on module.office1_opnsense (opentofu/main.tf); validate PASS; saved plan 0/1/0 exact -- only the channel block, in-place (docs/audit/outer-plan-20260723-office1-qga.txt); applied; edge bounced and returned in ~5s answering ping. Channel in the live domain XML, state=connected; guest-ping -> {"return":{}}; domifaddr --source agent reports vtnet0/vtnet1 -- the dc0-equivalent D-129 proof, first time on office1.
    2. Firmware update via POST core/firmware/update: 26.7 -> 26.7.1, package-level only (no reboot required; upgradestatus DONE, web GUI restarted). Egress re-verified 0% loss.
    3. opnsense-plugins.sh apply vr1-edge re-run: os-iperf installed=1 (the 26.7 refusal from Item 9 resolved by the update); os-qemu-guest-agent still installed=1.
    4. Outer plan re-converged ZERO DIFF (docs/audit/outer-plan-20260723-postqga-converged.txt); section-5 history chain re-recorded.
  • Named close capture: docs/audit/g13-close-20260723.txt (plugins read-back, product_version 26.7.1, channel count, guest-ping, egress). G13 row CLOSED in CURRENT-STATE; as-built edge row updated (same commit).
  • VERSION-DRIFT NOTE (logged, no action): office1 edge now 26.7.1; the two DC edges remain 26.7 -- whether to roll 26.7.1 to them is a future gated maintenance question (they have no current install-blocker forcing it).
  • REVERT: channel -- remove expose_qga_channel from main.tf + gated 0/1/0 apply (bounces the edge). Plugins -- POST core/firmware/remove/<pkg>. Firmware -- effectively one-way (no API downgrade); the operator accepted this in the bundle approval.

Non-items (verified/logged, nothing executed)

  • Netem runbook Step-11 DOCFIX flagged at step E: ALREADY delivered at stage close (runbook targets virbr5 with 07-21 captures cited). Off the queue.
  • opentofu/vr1-dc0-maas/ on-disk residue: git tracks NOTHING there (retire ruling fully executed in the repo); what remains locally is only the untracked .terraform/ provider cache. Local cleanup candidate (safe rm -rf), left untouched this session.
  • Pinned queue items NOT processed (their triggers have not fired): D-131 sub-4 + DNS review (next-deployment design time), D-132 (Roosevelt MAAS design time), tenant-onboarding review (deployment close), D-068 item 1 migration-plan draft (untouched -- needs its own session), G13 execution (gated live mutations), NetBox write-path bugs (before next NetBox write campaign), SEC rows (v1 close / external).