diff --git a/docs/changelog-20260723-queue-pass.md b/docs/changelog-20260723-queue-pass.md new file mode 100644 index 0000000..1fe9576 --- /dev/null +++ b/docs/changelog-20260723-queue-pass.md @@ -0,0 +1,120 @@ +# Changelog -- 2026-07-23 session: queue pass (post-G12 queued findings) + +Session scope: process the queued-findings backlog left open at the G12/dc1 +close (ledger 2026-07-23 close entry: "3 findings"; plus the four tool gaps +queued in changelog-20260722-g12-dc1-build.md items 7/9). Repo deliverables +only -- NO cloud mutation this session. Each item carries its revert. + +Verified up front: the netem runbook Step-11 divergence flagged at step E is +ALREADY fixed (runbook targets the dc0<->dc1 leg, 07-21 captures cited) -- +off the queue, nothing to do. + +## Item 1 -- opnsense-prep-image.sh survives being piped (finding 07-22 c) + +- WHAT: removed the unused `HERE="$(dirname "${BASH_SOURCE[0]}")"` line (with + a comment stating why it must not come back). Under `ssh 'bash -s' < script` + BASH_SOURCE is unset and `set -u` killed the script at line 20, before its + first guard (measured on the dc1 build, 2026-07-22 changelog item 7c). The + variable was referenced nowhere -- deletion IS the minimal fix. +- HARNESS: new T3b pipes the script via `bash -s` and asserts it reaches the + mirror guard (no "unbound variable"). 4/4 + 1 SKIP (environmental). +- REVERT: restore the HERE line; drop T3b. + +## Item 2 -- site-headend-install.sh emitted hints made version/site-neutral (finding 07-22 a) + +- WHAT: the `--check`/setup hint prose hardcoded `opnsense-prep-image.sh 26.1` + (while 26.7 is the ruled version) and `cd opentofu/vr1-dc0-substrate` (while + parameterized for any DC). Hints now say `` pointing at the + version authority (CURRENT-STATE section 7) and `vr1--substrate`. + NOT changed (value, not prose -- out of the finding's cosmetic scope): the + `OPNSENSE_BASE` default filename still embeds `26.1`; every real standup + passes `--opnsense-base` explicitly. Flagged here, not fixed. +- HARNESS: new cases assert the emitted hints carry no `26.1`, no + `vr1-dc0-substrate`, and do point at CURRENT-STATE. +- REVERT: restore the two echo lines; drop the three assertions. + +## Item 3 -- nftables-sec010.nft generated idempotent (finding 07-22 b) + +- WHAT: `nft -f` on a live table APPENDS, so a sec010-fw.service restart (or + install re-run) DUPLICATED the FORWARD-drop rules -- observed live on + voffice1 2026-07-22, cleaned by hand. The generated file now opens with the + declare-then-delete preamble (`table inet sec010 {}` + `delete table inet + sec010`) so any reload converges to exactly one rule pair. No `destroy` + keyword (older-nft-safe). +- LIVE STATE NOT TOUCHED: the files already installed on vvr1-dc0, vvr1-dc1 + and voffice1 keep the old shape until a gated re-run of the installer (or a + hand re-write) -- QUEUED as a gated live re-assert, NOT done this session. + Until then a sec010-fw restart on those hosts duplicates rules (benign to + enforcement -- same verdict, duplicated -- but wrong). +- HARNESS: asserts the preamble is present in the generator. +- REVERT: restore the old heredoc; drop the assertion. + +## Item 4 -- serial-log staging dir joins --host-nodes standup (finding 07-22 item 9) + +- WHAT: `site-headend-install.sh --host-nodes` now creates + `/var/lib/libvirt/vr1/staging` (0755; literal keyed to + modules/opnsense-edge's hardcoded serial-log path) and `--check` verifies + it. Hand-created on dc0; its absence failed the dc1 inner apply at 18/28. +- HARNESS: 'staging' added to the node-host step assertions. 59/59 PASS. +- REVERT: drop the run/check lines + the assertion. + +## Item 5 -- opnsense-bootstrap-apikey.sh: sshd readiness wait + first-scp retry (finding 07-23) + +- WHAT (all three measured on the dc1 mint, 2026-07-23): (a) bounded readiness + probe (6 x 5s) before shipping anything -- the edge sshd is restarting right + after the console bootstrap's `configctl openssh restart` and first contact + got `scp: Connection closed`; fails LOUD exit 2 if the edge never answers. + (b) one retry on the first scp. (c) `-n` on the base ssh opts -- an inner + ssh without it EATS a piped `bash -s` driver's stdin (the swallowed + retrieve/cleanup lines). Plus the dc0-queued usage note: the minter PHP must + sit BESIDE the script when staged to a rack. + `OPNSENSE_BOOTSTRAP_RETRY_DELAY` env exists ONLY so the harness need not + sleep for real. +- HARNESS: new T9 (stateful scp stub -- first ship fails, retry completes the + mint) + T10 (dead edge -> bounded loud exit 2, no creds file). 10/10 PASS. +- REVERT: restore the pre-item SSH/SCP block and scp line; drop T9/T10. + +## Item 6 -- DOCFIX-198: phase2 runbook Steps 9-10 superseded + DC standup definition-of-done + +- WHAT: `runbooks/dc-dc-phase2-tofu-dc-substrate.md` Steps 9-10 still + prescribed the `modules/maas-vm-host` virsh-POD route and called D-123 + "PROPOSED, recommend Model A" -- refuted/retired by the D-103/D-123 + amendments (2026-07-20) but never re-pointed. Added SUPERSEDED banners on + both steps (per-machine `power_type=virsh` via maas-node-power.sh is the + ruled path; pod-delete cascade trap cited) and a new "DC standup + definition-of-done" section after Step 12 -- the 7-row as-executed checklist + (measured reach/site-baseleg; dc-rack-net twin; staging dir; region MAAS + subnet config; per-DC power key incl. the SEC-016 snap-refresh RE-ASSERT + fragility; per-machine power; commissioning 9/9 gate). This lands the two + "queued to DC standup DoD" obligations (staging dir; snap-key re-assert) + in their runbook home. +- REVERT: remove the two banners + the DoD section. + +## Item 7 -- DOCFIX-199: D-112(c) console-bootstrap payload requirements folded into the runbook + +- WHAT: the replacement-chain "Console bootstrap" step now carries the + measured payload requirements (dc0 lessons i-v + dc1 confirmation): + <=160-char b64 chunks; `require_once("util.inc")` (shell_safe -- the dc1 + abort; any surviving dc0 `.b64` rack artifact LACKS it, do not replay); + `local_user_set()` to materialize authorized_keys; tcsh no-quoted-one-liners; + `ssh -n` when driven via `bash -s`; the post-restart sshd transient (now + handled by Item 5). This closes the "fold util.inc into the canonical + console bootstrap payload" queue item -- the runbook checklist IS the + canonical payload spec (no repo-carried payload artifact exists; authoring + one is a possible future deliverable, logged here, not started). +- REVERT: remove the requirements block. + +## Non-items (verified/logged, nothing executed) + +- Netem runbook Step-11 DOCFIX flagged at step E: ALREADY delivered at stage + close (runbook targets virbr5 with 07-21 captures cited). Off the queue. +- `opentofu/vr1-dc0-maas/` on-disk residue: git tracks NOTHING there (retire + ruling fully executed in the repo); what remains locally is only the + untracked `.terraform/` provider cache. Local cleanup candidate (safe + `rm -rf`), left untouched this session. +- Pinned queue items NOT processed (their triggers have not fired): D-131 + sub-4 + DNS review (next-deployment design time), D-132 (Roosevelt MAAS + design time), tenant-onboarding review (deployment close), D-068 item 1 + migration-plan draft (untouched -- needs its own session), G13 execution + (gated live mutations), NetBox write-path bugs (before next NetBox write + campaign), SEC rows (v1 close / external). diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index ae87d26..4013fc3 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -370,6 +370,29 @@ 1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service public key. This is the only step that needs the console; everything after it is scripted. + + **Payload requirements (DOCFIX-199 -- every one MEASURED, dc0 2026-07-20 lessons i-v + + dc1 2026-07-23 confirmation; skip one and the bootstrap fails exactly the way it already + failed once):** + - Ship the PHP payload as **base64 chunks <= 160 chars** reassembled on the edge + (`openssl base64 -d`) -- the serial console WRAPS long lines (lesson i). + - The payload MUST `require_once("config.inc")` **AND `require_once("util.inc")`** -- + `write_config()` calls `shell_safe()`, which lives in util.inc (measured: + `grep -l shell_safe /usr/local/etc/inc/*.inc`; lesson iv). A payload without it dies at + `Call to undefined function shell_safe()` BEFORE writing. Any surviving dc0 `.b64` + artifact on a rack predates this line and is missing the include -- do not replay it + verbatim. + - Writing config.xml does NOT materialize `/root/.ssh/authorized_keys` -- the payload must + call the vendor's **`local_user_set()`** (auth.inc) after the config write (lesson v; the + measured reason key auth "mysteriously" failed after a clean write). + - The edge root shell is **tcsh**: never patch files via nested-quoted one-liners + (`Unmatched '"'"'`; `2>&1` = "Ambiguous output redirect") -- re-ship the WHOLE payload + instead (lesson iii; platform-traps carries the shell trap). + - When driving the rack over a piped `bash -s`, every inner `ssh` takes **`-n`** or it EATS + the remaining driver stdin (measured on the dc1 mint: retrieve/cleanup lines swallowed). + - Immediately after the bootstrap's `configctl openssh restart`, first contact can get a + transient `Connection closed` -- `opnsense-bootstrap-apikey.sh` now waits for sshd and + retries its first scp (2026-07-23). 2. **Mint the API key** -- no GUI click: ```bash export OPNSENSE_SSH_KEY=... # the service private key @@ -564,6 +587,22 @@ ## Step 9 -- Wire + apply `modules/maas-vm-host` -- register `vr1-dc0`'s libvirt host with `vr1-dc0`'s MAAS rack controller [MUTATION: gated] +> **SUPERSEDED (DOCFIX-198, 2026-07-23). Do NOT execute Steps 9-10 as written.** +> The virsh-POD mechanism this step prescribes (`modules/maas-vm-host`) was +> REFUTED BY MEASUREMENT on the dc0 build and the module's root retired: +> snap-confined MAAS cannot reach a local libvirt socket, and a `qemu+ssh` pod +> fails on `modules/node-vm`'s pool+volume disk refs (D-123 AMENDMENT +> 2026-07-20 -- the ruling authority; D-103 amendment likewise). The pod's +> discovery job is done by PXE enlistment; the RULED replacement is +> **per-machine `power_type=virsh` via `scripts/maas-node-power.sh`** +> (MAC-matched, dry-by-default, verified by a real `query-power-state`) -- +> the Roosevelt shape (per-node IPMI). Also note the appendix-A trap: a MAAS +> pod delete CASCADES to its linked machine records -- read the pod's machine +> list BEFORE any `vm-host delete`; non-empty = STOP (2026-07-21 incident). +> The as-executed sequence both DCs actually used is the "DC standup +> definition-of-done" section after Step 12. D-123's "PROPOSED, recommend +> Model A" framing below is history (Model B ADOPTED 2026-07-16). + **Hard dependency -- `vr1-dc0`'s MAAS controller.** This step requires `vr1-dc0`'s own MAAS to exist and be reachable. **D-123 (PROPOSED, recommend Model A) resolves what that is:** a `vvr1-dc0` **site headend VM** built the @@ -601,6 +640,12 @@ ## Step 10 -- Verify MAAS discovers the node VMs (READ-ONLY) +> **SUPERSEDED with Step 9 (DOCFIX-198):** discovery happens by PXE +> enlistment against the pinned node MACs, not via a VM host; the machines +> appear with MAAS-random hostnames, matched by MAC (see the DoD section +> after Step 12). The `vm-hosts read` check below has no object to read on +> the ruled path. + **CHECK** ```bash maas admin vm-hosts read | jq -r '.[] | select(.name=="vr1-dc0-vcloud") | {id, name, resources}' @@ -693,6 +738,55 @@ --- +## DC standup definition-of-done (DOCFIX-198 -- as executed on dc0 + dc1; the dc2+/Roosevelt checklist) + +Every item below was established the hard way on the dc0 build (2026-07-19..21) +and pre-empted the same failure on dc1 (2026-07-22/23). A DC standup is NOT +done until each row is verified; the citations are the ruling/lesson +authorities. Every mutation stays individually operator-gated as usual. + +1. **Host->site reach measured, never inferred (D-126 / register item 20).** + At standup, MEASURE how vcloud reaches the DC's containment/service net. + An ISOLATED host-local net gets a measured leg row + `site-baseleg.sh` + unit install; a routed/NAT path needs none. DCs keep the `-J`-through- + containment ssh shape. +2. **Rack network twin (D-131 sub-1, RULED).** `scripts/dc-rack-net.sh + install ` on the rack: reboot-persistent rack bridge legs + the + node-DNS dnsmasq forwarder on metal-admin, forwarding to region BIND over + the rack's own transit. WITHOUT it, MAAS 3.7's rack-only agent resolver + SERVFAILs on an isolated rack and commissioning silently times out + (appendix-A carries the symptom pair). Verify: `dc-rack-net.sh check + ` PASS + forwarder answers an authoritative `maas-internal` SOA. +3. **Serial-log staging dir** `/var/lib/libvirt/vr1/staging/` on the rack -- + created by `site-headend-install.sh --host-nodes` (since 2026-07-23; its + absence failed the dc1 inner apply at 18/28). `--check` verifies it. +4. **Region-side MAAS subnet config on the DC's metal-admin subnet:** D-120 + dynamic range (.100-.200), D-131 `dns_servers=` + + `allow_dns=false`, then DHCP `dhcp_on=true primary_rack=` + -- verify dhcpd RUNNING by process on the rack, not by MAAS self-report + (the 2026-07-20 Temporal incident class). +5. **Per-DC MAAS->libvirt power key (SEC-012 dc0 / SEC-016 dc1 -- standing + invariant, RULED).** Each DC gets its OWN dedicated power keypair, never + cross-DC reuse: private half in the region MAAS snap + (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh config` block so + each rack transit IP uses its own key); public half authorizes the + libvirt-group user on that rack. **FRAGILITY: the snap-side key + config + live under per-revision `/var/snap/maas/current/` -- RE-ASSERT after any + MAAS snap refresh** (queued rotation/scope questions live in + `docs/security-ledger.md`). Roosevelt analog: per-DC IPMI/BMC creds. +6. **Per-machine node power, no pods (D-103/D-123 amendments, RULED).** + `scripts/maas-node-power.sh` dry -> commit -> each write verified by a + real `query-power-state`. Node identity rides the config-pinned MACs + (modules/node-vm; 54 per DC), which is what makes PXE re-enlistment + deterministic. +7. **Commissioning gate:** all 9 nodes READY, shapes exact to D-121 Option C + (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), captured to a dated + audit file (dc0: `commissioning-diag-20260721`; dc1: + `dc1-commissioning-verify-20260723` -- READY in ~3.5 min when 1-6 are in + place). + +--- + ## GATE -- Stage 3 exit condition (honest state, not aspirational) Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as: diff --git a/scripts/opnsense-bootstrap-apikey.sh b/scripts/opnsense-bootstrap-apikey.sh index 97e11e0..3a90d86 100755 --- a/scripts/opnsense-bootstrap-apikey.sh +++ b/scripts/opnsense-bootstrap-apikey.sh @@ -22,6 +22,15 @@ # bootstrap has already run). This script does NOT solve first-contact; it solves # "we have SSH, now get an API key so everything else can be REST". # +# STAGING NOTE: the minter (opnsense-mint-apikey.php) must sit BESIDE this script +# in the SAME directory -- when staging to a rack, copy BOTH files (measured on the +# dc0 build; the script fails loud but only at run time). +# +# TIMING: right after the console bootstrap the edge sshd is RESTARTING +# (`configctl openssh restart`) and the first contact can get "Connection closed" +# (measured transient, dc1 2026-07-23). This script therefore WAITS for sshd to +# answer before shipping anything, and retries the first scp once. +# # THE SECRET IS NEVER PRINTED. OPNsense hashes it on save, so creation is the ONLY # moment it exists in cleartext. It goes straight to (0600) in the # `key=`/`secret=` form scripts/opnsense-api.sh expects. Stdout gets lengths only. @@ -43,6 +52,7 @@ echo "usage: opnsense-bootstrap-apikey.sh [--dry-run] " >&2 echo " e.g. opnsense-bootstrap-apikey.sh 10.10.0.1 ~/vr1-office1-creds/opnsense-api.txt" >&2 echo " env: OPNSENSE_SSH_KEY (required) OPNSENSE_API_USER (default: root)" >&2 + echo " NOTE: opnsense-mint-apikey.php must sit BESIDE this script -- copy BOTH when staging to a rack" >&2 exit 1 } @@ -68,8 +78,13 @@ [ -n "${OPNSENSE_SSH_KEY:-}" ] || { echo "FAIL: \$OPNSENSE_SSH_KEY not set" >&2; exit 1; } -SSH=(ssh -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15) +# -n on every ssh: none of these calls needs stdin, and an ssh that reads stdin EATS the +# rest of a piped `bash -s` driver script (measured on the dc1 mint, 2026-07-23 -- the +# retrieve/cleanup lines were swallowed). Same class as the lxc/tcsh stdin traps. +SSH=(ssh -n -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15) SCP=(scp -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15 -q) +# Retry pacing: overridable ONLY so the offline harness need not sleep for real. +RETRY_DELAY="${OPNSENSE_BOOTSTRAP_RETRY_DELAY:-5}" REMOTE_PHP="/tmp/opnsense-mint-apikey.$$.php" REMOTE_OUT="/tmp/opnsense-apikey.$$.txt" @@ -78,7 +93,22 @@ cleanup() { "${SSH[@]}" "root@${EDGE}" "rm -f '$REMOTE_PHP' '$REMOTE_OUT'" >/dev/null 2>&1 || true; } trap cleanup EXIT -"${SCP[@]}" "$MINTER" "root@${EDGE}:${REMOTE_PHP}" || { echo "FAIL: could not ship the minter to $EDGE" >&2; exit 2; } +# Post-bootstrap sshd readiness wait (see TIMING in the header): probe until the edge +# answers before shipping anything. Bounded -- never spins forever on a dead edge. +ready=0 +for attempt in 1 2 3 4 5 6; do + if "${SSH[@]}" "root@${EDGE}" true >/dev/null 2>&1; then ready=1; break; fi + echo " ... edge sshd not answering (attempt $attempt/6); retrying in ${RETRY_DELAY}s" + sleep "$RETRY_DELAY" +done +[ "$ready" = "1" ] || { echo "FAIL: edge sshd at $EDGE never became ready (still restarting after the console bootstrap? wrong host/key?)" >&2; exit 2; } + +# First contact after the readiness probe can still hit the restart transient -- one retry. +if ! "${SCP[@]}" "$MINTER" "root@${EDGE}:${REMOTE_PHP}"; then + echo " ... first scp failed (post-restart transient?); retrying once in ${RETRY_DELAY}s" + sleep "$RETRY_DELAY" + "${SCP[@]}" "$MINTER" "root@${EDGE}:${REMOTE_PHP}" || { echo "FAIL: could not ship the minter to $EDGE" >&2; exit 2; } +fi # load_phalcon.php is resolved RELATIVE to /usr/local/opnsense/mvc -- must cd there. "${SSH[@]}" "root@${EDGE}" \ diff --git a/scripts/opnsense-prep-image.sh b/scripts/opnsense-prep-image.sh index a39753a..711012d 100644 --- a/scripts/opnsense-prep-image.sh +++ b/scripts/opnsense-prep-image.sh @@ -17,7 +17,9 @@ # writes only to and a self-cleaning temp dir. # Exit: 0 prepared | 1 bad input | 2 required tool missing. set -euo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# NO ${BASH_SOURCE[0]} here: this script is also run PIPED (`ssh host 'bash -s' -- ... < script`), +# where BASH_SOURCE is unset and `set -u` kills it before the first guard (measured, dc1 2026-07-22). +# It needs no self-location anyway -- everything is argument- or env-driven. VERSION="${1:?usage: opnsense-prep-image.sh []}" OUT="${2:?usage: opnsense-prep-image.sh []}" diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh index 18ddb68..738cd21 100644 --- a/scripts/site-headend-install.sh +++ b/scripts/site-headend-install.sh @@ -213,7 +213,8 @@ else echo " [--] nested KVM OFF (kmod=${kmod:-none}) -- inner node VMs get no /dev/kvm"; ok=1; fi id -nG "$HOST_USER" 2>/dev/null | grep -qw libvirt && echo " [ok] $HOST_USER in libvirt group" || { echo " [--] $HOST_USER NOT in libvirt group"; ok=1; } [ -d "$INNER_POOL_PATH" ] && echo " [ok] inner pool dir $INNER_POOL_PATH" || { echo " [--] inner pool dir $INNER_POOL_PATH missing"; ok=1; } - [ -s "$OPNSENSE_BASE" ] && echo " [ok] opnsense base $OPNSENSE_BASE" || { echo " [--] opnsense base missing -- run: opnsense-prep-image.sh 26.1 $OPNSENSE_BASE"; ok=1; } + [ -d /var/lib/libvirt/vr1/staging ] && echo " [ok] serial-log staging dir /var/lib/libvirt/vr1/staging" || { echo " [--] serial-log staging dir /var/lib/libvirt/vr1/staging missing -- inner apply FAILS at the edge domain (modules/opnsense-edge serial log)"; ok=1; } + [ -s "$OPNSENSE_BASE" ] && echo " [ok] opnsense base $OPNSENSE_BASE" || { echo " [--] opnsense base missing -- run: opnsense-prep-image.sh $OPNSENSE_BASE (version authority: CURRENT-STATE section 7)"; ok=1; } # SEC-010 mechanical pre-apply gate: the transit FORWARD-drop must be present (DC-LOCAL, D-052/D-100) # AND must key on an interface that ACTUALLY EXISTS -- nftables oifname/iifname with an absent name # loads clean but matches NOTHING (fail-open), which would silently leave DC-LOCAL unenforced. @@ -263,6 +264,11 @@ # inner pool dir + AppArmor grant (platform-traps: a non-default libvirt pool path otherwise # yields qemu 'Permission denied' define-then-fail on the inner domains -- a silent dead guest). run "create inner pool dir $INNER_POOL_PATH" -- mkdir -p "$INNER_POOL_PATH" + # Serial-log staging dir: modules/opnsense-edge hardcodes the edge domain's serial log at + # /var/lib/libvirt/vr1/staging/-serial.log (main.tf, LITERAL keyed to that module). It was + # hand-created on dc0 (07-20 serial work) and its absence FAILED the dc1 inner apply at 18/28 + # (2026-07-22 queued finding) -- standup owns creating it so dc2+ does not repeat that. + run "create serial-log staging dir /var/lib/libvirt/vr1/staging (modules/opnsense-edge serial log)" -- install -d -m 0755 /var/lib/libvirt/vr1/staging run "AppArmor: allow inner pool path" -- bash -c "install -d /etc/apparmor.d/local/abstractions 2>/dev/null; grep -qF '$INNER_POOL_PATH' /etc/apparmor.d/local/abstractions/libvirt-qemu 2>/dev/null || echo ' \"$INNER_POOL_PATH/**\" rwk,' >> /etc/apparmor.d/local/abstractions/libvirt-qemu; systemctl reload apparmor 2>/dev/null || true" # SEC-010 (D-052/D-100 DC-LOCAL under Model B): enforce DC-local with a FORWARD-drop across the # TRANSIT leg -- nothing routes ACROSS the fiber THROUGH vvr1-dc0 (the rack proxies MAAS at the app @@ -281,8 +287,15 @@ if [ "$DRYRUN" = "1" ]; then echo " [dry-run] write /etc/nftables-sec010.nft (FORWARD-drop in+out '$TRANSIT_IF') + enable sec010-fw.service (boot-persistent)" else + # IDEMPOTENT reload (measured 2026-07-22 queued finding): `nft -f` on a live table APPENDS, + # so a service restart or re-run used to DUPLICATE the drop rules. The declare-then-delete + # idiom below makes the file safe to load any number of times (the bare `table` line is a + # no-op create if absent, so the `delete` can never fail on first load; works on any nft, + # no `destroy` keyword needed). cat > /etc/nftables-sec010.nft < opnsense base present: $OPNSENSE_BASE" - else echo " NOTE: opnsense base MISSING at $OPNSENSE_BASE -- run on this host BEFORE the inner apply:"; echo " bash scripts/opnsense-prep-image.sh 26.1 $OPNSENSE_BASE"; fi - echo " -> node-host ready. Inner apply next (from Office1): cd opentofu/vr1-dc0-substrate && tofu apply." + else echo " NOTE: opnsense base MISSING at $OPNSENSE_BASE -- run on this host BEFORE the inner apply:"; echo " bash scripts/opnsense-prep-image.sh $OPNSENSE_BASE # version authority: docs/CURRENT-STATE.md section 7"; fi + echo " -> node-host ready. Inner apply next (from Office1): cd opentofu/vr1--substrate && tofu apply." } do_report() { diff --git a/tests/opnsense-bootstrap-apikey/run-tests.sh b/tests/opnsense-bootstrap-apikey/run-tests.sh index 9748a5a..f047ae5 100755 --- a/tests/opnsense-bootstrap-apikey/run-tests.sh +++ b/tests/opnsense-bootstrap-apikey/run-tests.sh @@ -102,6 +102,50 @@ *) ok "T8 secret absent from output";; esac +# T9/T10 cover the post-bootstrap sshd restart transient (measured dc1 2026-07-23): +# the edge's `configctl openssh restart` can close the FIRST connection after the +# console bootstrap. The script must wait for sshd and retry the first scp once -- +# and must FAIL LOUD (not hang) when the edge never comes back. +# OPNSENSE_BOOTSTRAP_RETRY_DELAY=0 exists for exactly this harness: no real sleeps. + +echo "T9: a TRANSIENT first-scp failure is retried and succeeds" +rm -f "$TMP/scp-attempts" +cat > "$TMP/bin/scp" < "$TMP/scp-attempts" + [ "\$n" -eq 1 ] && { echo "scp: Connection closed" >&2; exit 1; } + exit 0 ;; + *) printf 'key=%s\nsecret=%s\n' "$FAKE_KEY" "$FAKE_SECRET" > "\$dest"; exit 0 ;; +esac +STUB +chmod +x "$TMP/bin/scp" +out="$(OPNSENSE_SSH_KEY=/dev/null OPNSENSE_BOOTSTRAP_RETRY_DELAY=0 PATH="$TMP/bin:$PATH" bash "$SCRIPT" 10.10.0.1 "$TMP/out9.txt" 2>&1)"; rc=$? +if [ "$rc" = "0" ] && [ -f "$TMP/out9.txt" ] && grep -q "retrying once" <<<"$out" && [ "$(cat "$TMP/scp-attempts")" = "2" ]; then + ok "T9 transient first scp retried once, mint completed" +else + bad "T9 transient-scp retry (rc=$rc attempts=$(cat "$TMP/scp-attempts" 2>/dev/null)): $out" +fi +mk_stubs # restore the plain stubs + +echo "T10: edge sshd NEVER ready -> bounded loud failure (exit 2), nothing shipped" +cat > "$TMP/bin/ssh" <<'STUB' +#!/usr/bin/env bash +echo "ssh: connect to host 10.10.0.1 port 22: Connection refused" >&2 +exit 255 +STUB +chmod +x "$TMP/bin/ssh" +out="$(OPNSENSE_SSH_KEY=/dev/null OPNSENSE_BOOTSTRAP_RETRY_DELAY=0 PATH="$TMP/bin:$PATH" bash "$SCRIPT" 10.10.0.1 "$TMP/out10.txt" 2>&1)"; rc=$? +if [ "$rc" = "2" ] && grep -q "never became ready" <<<"$out" && [ ! -e "$TMP/out10.txt" ]; then + ok "T10 dead edge fails loud after bounded probes, no creds file created" +else + bad "T10 dead-edge readiness guard (rc=$rc): $out" +fi +mk_stubs # restore for any later case + echo echo "RESULT: PASS=$PASS FAIL=$FAIL" [ "$FAIL" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "FAILURES"; exit 1; } diff --git a/tests/opnsense-prep-image/run-tests.sh b/tests/opnsense-prep-image/run-tests.sh index e4b8bc7..e61b8b0 100644 --- a/tests/opnsense-prep-image/run-tests.sh +++ b/tests/opnsense-prep-image/run-tests.sh @@ -29,6 +29,17 @@ unset OPNSENSE_MIRROR_BASE run 1 'OPNSENSE_MIRROR_BASE' "T3 OPNSENSE_MIRROR_BASE unset FAILS (rc 1)" "26.1" "$TMPOUT/out.qcow2" +# T3b: the script must survive being PIPED (`bash -s`), where BASH_SOURCE is unset under +# `set -u` -- it died on `BASH_SOURCE[0]: unbound variable` before its first guard when +# shipped via `ssh 'bash -s'` on the dc1 build (2026-07-22 queued finding). The correct +# behavior piped is IDENTICAL to T3: reach the mirror guard, not an unbound-variable death. +out="$(bash -s -- "26.1" "$TMPOUT/out.qcow2" < "$SCRIPT" 2>&1)"; rc=$? +if [[ "$rc" == "1" ]] && grep -q 'OPNSENSE_MIRROR_BASE' <<<"$out" && ! grep -q 'unbound variable' <<<"$out"; then + echo " PASS T3b piped (bash -s) reaches the mirror guard, no BASH_SOURCE death"; PASS=$((PASS+1)) +else + echo " FAIL T3b piped (bash -s) run (rc=$rc)"; echo "$out" | sed 's/^/ /'; FAIL=$((FAIL+1)) +fi + if command -v qemu-img >/dev/null 2>&1; then echo " SKIP T4 qemu-img-missing case (qemu-img IS present in this environment -- can't exercise the missing-binary guard here)" else diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh index 22dd60b..c615801 100644 --- a/tests/site-headend-install/run-tests.sh +++ b/tests/site-headend-install/run-tests.sh @@ -105,7 +105,7 @@ t "rack + --host-nodes --dry-run -> 0" 0 bash "$S" --role rack --host-nodes --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" nhout="$(bash "$S" --role rack --host-nodes --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" # the node-host setup steps must appear (incl. the SEC-010 DC-LOCAL forward-drop, R3-F02/Phase D) -for step in 'node-host' 'nested=1' 'qemu-kvm' 'inner pool' 'AppArmor' 'libvirt group' 'SEC-010' 'FORWARD-drop'; do +for step in 'node-host' 'nested=1' 'qemu-kvm' 'inner pool' 'AppArmor' 'libvirt group' 'SEC-010' 'FORWARD-drop' 'staging'; do printf '%s' "$nhout" | grep -qi "$step" && ok || bad "node-host --dry-run missing step: '$step'" done # SEC-010 forward-drop must key on the transit interface, overridable via --transit-if @@ -137,6 +137,17 @@ printf '%s' "$rackout" | grep -qi 'node-host' && bad "plain rack regressed: printed node-host steps without --host-nodes" || ok # opnsense base is staged by opnsense-prep-image.sh, not fetched inline by this script printf '%s' "$nhout" | grep -q 'opnsense-prep-image.sh' && ok || bad "node-host must delegate the opnsense base to opnsense-prep-image.sh" +# the base-image hints must be VERSION-NEUTRAL: a hardcoded OPNsense version in emitted prose +# misled operators on the dc1 build (26.1 quoted while 26.7 was the ruled version -- 2026-07-22 +# queued finding a). The version authority is CURRENT-STATE section 7, and the hint must say so. +vout="$(bash "$S" --role rack --host-nodes --opnsense-base /nonexistent/base.qcow2 --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" +printf '%s' "$vout" | grep -q '26\.1' && bad "emitted hint hardcodes an OPNsense version (26.1) -- must be version-neutral" || ok +printf '%s' "$vout" | grep -q 'CURRENT-STATE' && ok || bad "base-image hint must point at the version authority (CURRENT-STATE section 7)" +# the inner-apply hint must not hardcode dc0's inner root (finding a, same class) +printf '%s' "$vout" | grep -q 'vr1-dc0-substrate' && bad "emitted hint hardcodes opentofu/vr1-dc0-substrate -- must be site-neutral" || ok +# SEC-010 nft file must be IDEMPOTENT to reload: `nft -f` on a live table APPENDS, so without a +# declare-then-delete preamble a service restart DUPLICATES the drop rules (2026-07-22 finding b). +grep -q 'delete table inet sec010' "$S" && ok || bad "nftables-sec010.nft lacks the declare-then-delete preamble -- reload duplicates rules" echo total=$((pass+fail))