diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 89846f1..b9a2352 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2199,6 +2199,32 @@ note). **NEXT: STEP 3 -- the clean redeploy** (Path M model teardown -> preflight -> phase-01 with Step 1.2b -> phase-02 vault init -> converge). Body: `docs/changelog-20260804-ovn-central-cert-fix.md`. + **>>> 2026-08-04 (CONTINUED) -- STEP 3 REDEPLOY IN PROGRESS. Stage 5 remains OPEN. <<<** + Full as-exec: `docs/audit/stage5-dc0-redeploy-asexec-20260804.txt`; changelog + `docs/changelog-20260804-stage5-dc0-redeploy.md`. **(a) PATH M TEARDOWN COMPLETE + CLEAN.** + `juju destroy-model vr1-dc0` drained 36->15 graceful (no --force, agents alive), wedged on 4 + error units (magnum/octavia/ovn-central/placement, failed teardown hooks); `juju resolved + --no-retry` (operator-ruled) cleared 15->3; last 3 records went machine-state STOPPED (the + M.2 agents-stopped stall) and `--force --no-wait` (operator-ruled) finished 3->0. `Model + destroyed.` rc=0. **NO ORPHAN this time** (controller API healthy, no life:dead residue -- + the 08-03 failure did not recur). M.5 cascade CLEAN: MAAS profile dc0 = 10 machines UNCHANGED + (9 Ready + 1 Deployed controller VM `subtle-grouse`); dc-cache-proxy + dc-snap-proxy check dc0 + both PASS. **(b) THE 08-04 "PROVEN + WIRED" ovn-central FIX HAD A DELIVERY BUG, NOW FIXED.** + Step 1.2b's live `juju model-config cloudinit-userdata` was REJECTED -- "runcmd not allowed, + use preruncmd or postruncmd instead": juju owns top-level `runcmd`. The 08-04 session proved + the CONCEPT live but never the DELIVERY (harness graded cloud-init YAML, which accepts runcmd, + not juju acceptance). FIXED: `scripts/dc-node-etchosts.sh` render `runcmd:`->`postruncmd:` + (sha 0f79de90->107fc197); harness gains T10 (asserts a juju-accepted key, rejects bare runcmd) + MUTATION-PROVEN, 10/10; re-staged to rack (sha verified). Step 1.2b RE-RUN live: juju ACCEPTED + (`OK: set`), `check` OK carries 10.12.12.0/22, rc=0. On-node /etc/hosts is verify-at-provision. + **(c) M.6 REBUILD to the deploy step:** `add-model vr1-dc0` (cred vr1-dc0-cred); artifact + source inherited from model-defaults (apt-http-proxy :3142 + snap :3129, apt-mirror empty -- + D-135 amendment (b) proxy shape; phase-4 Step 3.5's mirror text is STALE, runbook-fold gap + flagged); spaces gate PASS 0 fatal; egress gate 8/8 PASS; dry-run gate PASS (9 machines, + tags=openstack-vr1-dc0,). **NEXT: Step 4.3 juju deploy (operator-gated), then Step 4.4, + phase-02 vault init (operator-only), converge.** QUEUED (operator, needs a ruling before + execution): Tailscale on DC0/DC1 -- intersects D-107 / D-128 / the 08-02 "VPN deferred to + Roosevelt" ruling. **>>> SESSION CLOSE 2026-08-03 (GA-R4 bookend). Stage 5 remains OPEN; this is a session bookend, NOT a stage close. <<<** DEPLOY IS MID-STAGE-5: bundle deployed + mostly converged (9 machines started, mysql ONLINE, vault init+unseal+root-CA, ~25 units active, 0 error), diff --git a/docs/audit/stage5-dc0-redeploy-asexec-20260804.txt b/docs/audit/stage5-dc0-redeploy-asexec-20260804.txt new file mode 100644 index 0000000..b2e37db --- /dev/null +++ b/docs/audit/stage5-dc0-redeploy-asexec-20260804.txt @@ -0,0 +1,147 @@ +============================================================================== +AS-EXECUTED CAPTURE -- Stage 5 dc0 clean redeploy (ovn-central cert fix) +Session 2026-08-04. Branch dc-dc-stage5-preconditions. +============================================================================== +WHY THIS FILE: run-logged.sh opens an interactive script(1) subshell that only +logs a live terminal; a Claude-driven session executes mutations as separate +tool-call processes that such a subshell cannot capture. Per operator ruling +2026-08-04 ("Option 3, capture as you go"), each gated mutation's exact command +and output is appended here instead -- the tool-call-friendly substitute that +keeps the irreversible-step record (vault init, destroy-model) EXISTING and +FINDABLE (docs/as-executed-log-convention.md purpose). + +CONVENTION per entry: [UTC] :: -> . +Read-only measurements are NOT logged here (they are in the transcript); only +mutations are. Status claims live in docs/CURRENT-STATE.md ONLY. + +Known coverage gap declared: read-only census/measurements this session live in +the session transcript, not in this file. This file covers MUTATIONS only. +------------------------------------------------------------------------------ + +## M-0 F3 redeploy prerequisite -- stage dc-node-etchosts.sh to the dc0 rack +[2026-08-04] vcloud -> vr1-dc0-rack :: + scp -p scripts/dc-node-etchosts.sh \ + vr1-dc0-rack:/home/jessea123/repo-stage/scripts/dc-node-etchosts.sh + -> rc=0 +VERIFY (vr1-dc0-rack, sha256 first 16): + dc-node-etchosts.sh staged=0f79de90c19e9f5a repo=0f79de90c19e9f5a MATCH mode=664 + lib-net.sh (dep) staged=3e465ae7de3bf92d repo=3e465ae7de3bf92d MATCH +RESULT: F3 CLOSED -- redeploy prerequisite satisfied. +REVERT: rm ~/repo-stage/scripts/dc-node-etchosts.sh on the rack (restores prior + staged set; the script is a redeploy input, not live cloud state). + +## M.2 Path M teardown of the vr1-dc0 model [MUTATION, GA-R5 operator-ruled 2026-08-04] +Operator ruling (verbatim selection): "Proceed with Path M.2 teardown now". +Pre-teardown MAAS baseline (profile dc0, rack hot-kid): 10 machines, all Deployed, +owner juju-vr1-dc0. juju storage empty (no storage flag). juju offers empty. +[2026-08-04 ~18:2x] vr1-dc0-rack :: + juju destroy-model vr1-dc0 --no-prompt + (WITHOUT --force -- agents alive; --force orphaned the model + downed the + controller on 08-03. WITHOUT --no-wait -- reserved for the agents-stopped stall.) +DRAIN (progressive, hooks executing -- the proven path, NOT the 07-31 flat stall): + 36 -> 35 -> 34 -> 30 -> 28 -> 27 -> 25 -> 24 -> 23 -> 22 -> 21 machines ... (in progress) + juju models: vr1-dc0 state=destroying. +DRAIN OUTCOME: graceful drain reached the tail then WEDGED flat at 15 machines + (1 control host + 14 LXD containers) for ~12 min. Retry worker ALIVE (attempt + 25 -> 28 -> 37 climbing) but count would not drop: 4 units stuck in workload + error with failed teardown hooks their idle agents could not complete -- + magnum/0 (identity-service-relation-departed), octavia/0 + (certificates-relation-changed), ovn-central/0 (ovsdb-cms-relation-broken), + placement/0 (identity-service-relation-departed). This is M.2's documented + agents-can't-run-stop-hooks stall. Agents were EXECUTING (not stopped), so per + M.2 --force was NOT auto-warranted; presented as a gated decision. +INSTRUMENT NOTE (owned): first watcher counted juju status --format=json + .machines and read 1 (collapses to top-level host during destroy, hiding the + 14 containers) -> false STALL alarm. Real total = the "found N machines" retry + line. Corrected. + +## M.2b Clear the stall via juju resolved [MUTATION, GA-R5 operator-ruled 2026-08-04] +Operator ruling (verbatim selection): "Try `juju resolved` on the 4 error units first". +[2026-08-04] vr1-dc0-rack :: + juju resolved --no-retry -m vr1-dc0 magnum/0 octavia/0 ovn-central/0 placement/0 + -> rc=0 + (--no-retry: skip the failing teardown hook rather than re-run it into the same + failure; safe in a destroy -- MAAS re-images the machines.) +RESULT: unblocked -- applications 53 -> 41 within ~30s; attempts climbing (37). + New error units surface as the drain proceeds (barbican/0, designate/0 next), + so continuing with a bounded resolve-loop (juju resolved --no-retry --all every + ~25s) until the model is GONE (M.5 positive-signal verified). +REVERT: n/a -- teardown is intended. + +## M.2c Finish the stall via --force --no-wait [MUTATION, GA-R5 operator-ruled 2026-08-04] +Operator ruling (verbatim selection): "Escalate: --force --no-wait now". +CONDITION AT ESCALATION: resolve loop took the drain 15 -> 3 machines, then the +last 3 records (control-01 + 0/lxd/2 + 0/lxd/18) went to machine-state STOPPED -- +agents down, no teardown hook can run (the genuine M.2 agents-stopped stall that +juju resolved cannot address). Model was 'destroying' (NOT 'dead'), so re-issuing +is the documented escalation, not the forbidden dead-model re-issue. +[2026-08-04 ~19:29] vr1-dc0-rack :: + juju destroy-model vr1-dc0 --force --no-wait --no-prompt + -> "Destroying model" ... 3 -> 1 -> 0 machines ... "Model destroyed." rc=0 + +RESULT (M.2 overall): TEARDOWN COMPLETE, CLEAN, NO ORPHAN. + ORPHAN CHECK (the 08-03 failure -- did NOT recur): + - juju models: only 'controller' (available); vr1-dc0 fully gone, not life:dead. + - controller model API responds (juju status -m controller OK) -- 17070 up. + - models now = [('admin/controller','alive')] -- no dead-model residue; + add-model will not be refused. + M.5(b) CASCADE CHECK (profile dc0): 10 machines, count UNCHANGED from baseline + -> NO cascade. States: Ready 9 (role nodes, owner=None) + Deployed 1 + (controller VM subtle-grouse). Exactly the expected post-teardown state. +REVERT: n/a -- teardown is intended; rebuild is M.6 (add-model + rebuild). + +------------------------------------------------------------------------------ +## M.5(c) proxy-survival post-checks (read-only, logged for the teardown record) + dc-cache-proxy check dc0: PASS (apt-cacher-ng active :3142, serves archive+UCA 200) + dc-snap-proxy check dc0: PASS (:3129, store API 200, CDN tunnel allowed, allowlist enforced) + => M.5 ALL GREEN. Path M teardown complete + clean. Proceeding to M.6 rebuild. + +## M.6 REBUILD +## Step 3.5 -- add-model + spaces + artifact source [MUTATION, gated] +[2026-08-04] vr1-dc0-rack (D-138; runbook says voffice1 but that predates D-138) :: + juju add-model vr1-dc0 + -> "Added 'vr1-dc0' model with credential 'vr1-dc0-cred' for user 'admin'" rc=0 +GATES (read-only): + (a) artifact source inherited from model-defaults: apt-http-proxy=http://10.12.8.4:3142, + snap-http[s]-proxy=http://10.12.8.4:3129, apt-mirror EMPTY. Correct per D-135 + amendment (b) (dc0 on the caching PROXY, not the retired mirror). apt-mirror + NOT set (Step 3.5's mirror text is STALE -- runbook-fold gap, flagged). + (b) juju-spaces-check.sh vr1-dc0: PASS 0 fatal -- all six spaces present, none stale. +REVERT: juju destroy-model vr1-dc0 (empty model; trivial). + +## QUEUED -- NOT EXECUTED THIS SESSION (operator request 2026-08-04, mid-redeploy) +Q1. "Queue a Tailscale installation for DC0 and DC1." DEFERRED (hard rule 1 -- new + scope is logged mid-step, not run). NEEDS A RULING BEFORE EXECUTION, not just an + ops task: it intersects ruled surfaces -- + - D-107: Tailscale is the operator WORKSTATION / human-GUI path, NOT the DC nodes. + - D-128: durable DC reach is the base-leg oneshot, explicitly NOT tailscale-on-host. + - 2026-08-02 ruling: "VPN deferred to Roosevelt" (VPN class deferred). + ACTION OWED: grep the exact D-107/D-128 text + the 08-02 VPN-deferred ruling, then + put the DC-Tailscale question to the operator as a GA-R5 exchange (likely a new + D-number, GA-R3 A1: decides DC access at every future standup). Do NOT install until ruled. + +## Step 1.2b -- metal-internal /etc/hosts cloudinit (the ovn-central cert fix) [MUTATION, gated] +FIRST ATTEMPT FAILED -- delivery-mechanism defect found: + juju model-config REJECTED the config: "cloudinit-userdata: runcmd not allowed, + use preruncmd or postruncmd instead" (rc=1, nothing set). The 08-04 fix proved the + CONCEPT (manual /etc/hosts -> cert) but its DELIVERY (cloudinit-userdata) was never + live-tested; the harness graded rendered YAML (cloud-init accepts runcmd) not juju + acceptance (juju forbids it). Classic can't-see-the-live-apply green harness. +FIX (repo change, delivered with discipline): + scripts/dc-node-etchosts.sh: render key runcmd: -> postruncmd: (repo sha + 0f79de90.. -> 107fc197..); + inline comment forbidding revert. + tests/dc-node-etchosts: switched to postruncmd; NEW T10 asserts a juju-accepted + key + rejects bare runcmd (the missing gate). Harness 10/10; T10 MUTATION-PROVEN + to fail (revert to runcmd -> 5 FAIL). Re-staged to rack (sha 107fc197 verified). +RE-RUN (live delivery test that was missing): +[2026-08-04] vr1-dc0-rack :: + JUJU_MODEL=vr1-dc0 bash scripts/dc-node-etchosts.sh apply vr1-dc0 -> "OK: set" (juju ACCEPTED postruncmd) + JUJU_MODEL=vr1-dc0 bash scripts/dc-node-etchosts.sh check vr1-dc0 -> OK, carries 10.12.12.0/22 mapping, rc=0 +GATE: check exit 0. On-node /etc/hosts delivery VERIFY-AT-PROVISION after Step 1.4. +REVERT (model-config): juju model-config -m vr1-dc0 --reset cloudinit-userdata. + +## Step 4.2 dry-run gate (read-only, logged for the deploy record) + 9 machines "0"-"8", every machine tags=openstack-vr1-dc0, (3 control/2 compute/4 storage); + ceph-osd->5-8, nova-compute->3-4; no overlay error; rc=0. GATE PASS. + +## (below appended as each further mutation executes -- NEXT: Step 4.3 deploy, operator-gated) diff --git a/docs/changelog-20260804-stage5-dc0-redeploy.md b/docs/changelog-20260804-stage5-dc0-redeploy.md new file mode 100644 index 0000000..7754139 --- /dev/null +++ b/docs/changelog-20260804-stage5-dc0-redeploy.md @@ -0,0 +1,52 @@ +# Changelog 2026-08-04 -- Stage 5 dc0 clean redeploy (Path M teardown + ovn-central cert delivery fix) + +Session-scoped (GA-R2). Status claims live in docs/CURRENT-STATE.md ONLY; this file +is the review surface for the changes, each with evidence + a revert. + +## 1. Path M model teardown of vr1-dc0 (operator-ruled, executed clean) + +- **What:** tore down the ovn-central-degraded vr1-dc0 model to redeploy clean with the + Step 1.2b cert fix. Graceful `juju destroy-model vr1-dc0 --no-prompt` (no --force, + agents alive) drained 36->15; wedged on 4 error units with failed teardown hooks; + `juju resolved --no-retry` (operator-ruled) cleared them 15->3; the last 3 records went + to machine-state STOPPED (the genuine M.2 agents-stopped stall) and + `juju destroy-model vr1-dc0 --force --no-wait --no-prompt` (operator-ruled) finished 3->0. +- **Evidence:** `Model destroyed.` rc=0. NO orphan (controller API healthy, no life:dead + residue -- the 08-03 failure did not recur). M.5 cascade: MAAS profile dc0 shows 10 + machines UNCHANGED (9 Ready + 1 Deployed controller VM). Proxies survived (dc-cache-proxy + + dc-snap-proxy check dc0 both PASS). Full as-exec: `docs/audit/stage5-dc0-redeploy-asexec-20260804.txt`. +- **Revert:** n/a (teardown intended); the model is rebuilt in this same work (add-model done). + +## 2. scripts/dc-node-etchosts.sh -- DELIVERY BUG fixed: runcmd -> postruncmd + +- **What & why:** Step 1.2b's live `juju model-config cloudinit-userdata=...` was REJECTED: + "runcmd not allowed, use preruncmd or postruncmd instead". juju reserves top-level `runcmd` + for its own machine-agent bootstrap. The render emitted `runcmd:`; changed to `postruncmd:` + (runs after juju's bootstrap, still long before the ovn-central cert hook). The 08-04 fix + proved the CONCEPT live but its DELIVERY was never live-tested -- the harness graded + cloud-init YAML validity (which accepts runcmd), not juju acceptance. +- **Harness reconcile:** `tests/dc-node-etchosts` switched to postruncmd + NEW **T10** asserts + a juju-accepted key (postruncmd/preruncmd) and REJECTS bare runcmd -- the missing gate. + 10/10 PASS; T10 MUTATION-PROVEN (revert to runcmd -> 5 FAIL incl T10). +- **Evidence:** re-run live -> `OK: set` (juju accepted), `check` OK carries 10.12.12.0/22, rc=0. + Re-staged to the dc0 rack, sha 107fc197 verified. repo-lint 0 fail. +- **Revert:** `git checkout ~ -- scripts/dc-node-etchosts.sh tests/dc-node-etchosts/run-tests.sh`; + live: `juju model-config -m vr1-dc0 --reset cloudinit-userdata`. + +## 3. M.6 rebuild progress (add-model, gates) + +- `juju add-model vr1-dc0` -> Added with credential vr1-dc0-cred. Artifact source inherited + from model-defaults (apt-http-proxy :3142 + snap proxies :3129; apt-mirror empty -- D-135 + amendment (b): dc0 on the caching PROXY, NOT the retired mirror; Step 3.5's mirror text is + STALE -- runbook-fold gap, flagged). Spaces gate PASS 0 fatal. Egress gate 8/8 PASS. + Dry-run gate PASS (9 machines, correct per-DC tags). Step 1.2b applied+checked (item 2). +- **Owed (logged, not done):** DOCFIX/runbook-fold to correct phase-4 Step 3.5's mirror text + to the D-135-(b) proxy shape; the dc-node-etchosts.sh delivery bug is a correction to the + 08-04 "PROVEN + WIRED" record (CURRENT-STATE updated this commit). +- **Revert:** `juju destroy-model vr1-dc0` (empty/partial model). + +## Queued (operator request, NOT executed -- needs a ruling) + +- Tailscale install on DC0/DC1: intersects D-107 (workstation path), D-128 (base-leg is the + DC reach), and the 2026-08-02 "VPN deferred to Roosevelt" ruling. Put as a GA-R5 exchange + (likely a D-number) before any install. See the as-exec file's QUEUED section. diff --git a/scripts/dc-node-etchosts.sh b/scripts/dc-node-etchosts.sh index d327df1..24d5495 100644 --- a/scripts/dc-node-etchosts.sh +++ b/scripts/dc-node-etchosts.sh @@ -33,7 +33,7 @@ # Usage: dc-node-etchosts.sh # render print the cloudinit-userdata YAML (pure; no juju) -- what the harness grades # apply set it on the DC's juju model (juju client; run on the DC rack, D-138) -# check read it back and verify the runcmd + this DC's metal-internal CIDR are present +# check read it back and verify the postruncmd + this DC's metal-internal CIDR are present # env: JUJU_MODEL (default: derived , e.g. vr1-dc0) ; MODEL_ARG override set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -57,7 +57,7 @@ # --- render the cloudinit-userdata YAML --- # Built with printf, one line per statement, the CIDR passed as a %s ARGUMENT (the -# repo's heredoc-whitespace rule). The runcmd is deterministic + idempotent: for each +# repo's heredoc-whitespace rule). The postruncmd is deterministic + idempotent: for each # of the node's own IPv4 addresses that lies IN this DC's metal-internal CIDR, append # ` .maas ` to /etc/hosts unless already present. The CIDR # membership test uses python3 (present on every Ubuntu node image) so a /22 is handled @@ -67,7 +67,12 @@ printf '%s\n' '# GENERATED by scripts/dc-node-etchosts.sh -- do not hand-edit the deployed value.' printf '%s\n' '# Gives each node a local /etc/hosts reverse mapping for its metal-internal' printf '%s\n' '# address so cert-CN-from-get_hostname resolves on the isolated plane (D-052).' - printf '%s\n' 'runcmd:' + # postruncmd, NOT runcmd: juju cloudinit-userdata REJECTS a top-level `runcmd` + # (it owns runcmd for the machine-agent bootstrap: "runcmd not allowed, use + # preruncmd or postruncmd instead", measured live 2026-08-04). postruncmd runs + # AFTER juju's bootstrap, still long before the ovn-central cert hook. Do NOT + # revert this to runcmd -- the harness T10 fails if you do. + printf '%s\n' 'postruncmd:' printf '%s\n' '- - /bin/bash' printf '%s\n' ' - -c' printf '%s\n' " - 'for ip in \$(hostname -I); do if python3 -c \"import ipaddress,sys; sys.exit(0 if ipaddress.ip_address(sys.argv[1]) in ipaddress.ip_network(sys.argv[2]) else 1)\" \"\$ip\" \"${MI_CIDR}\" 2>/dev/null; then grep -qw \"\$ip\" /etc/hosts || echo \"\$ip \$(hostname).maas \$(hostname)\" >> /etc/hosts; fi; done'" @@ -90,7 +95,8 @@ GOT="$(juju model-config -m "$MODEL" cloudinit-userdata 2>/dev/null)" fail=0 printf '%s' "$GOT" | grep -q "$MI_CIDR" || { echo "FAIL: model cloudinit-userdata missing this DC's metal-internal CIDR $MI_CIDR"; fail=1; } - printf '%s' "$GOT" | grep -q '/etc/hosts' || { echo "FAIL: model cloudinit-userdata has no /etc/hosts runcmd"; fail=1; } + printf '%s' "$GOT" | grep -q 'postruncmd' || { echo "FAIL: model cloudinit-userdata missing postruncmd (juju rejects bare runcmd)"; fail=1; } + printf '%s' "$GOT" | grep -q '/etc/hosts' || { echo "FAIL: model cloudinit-userdata has no /etc/hosts postruncmd"; fail=1; } printf '%s' "$GOT" | grep -q 'ip_network' || { echo "FAIL: model cloudinit-userdata missing the CIDR-membership guard"; fail=1; } [ "$fail" -eq 0 ] && { echo "OK: model '$MODEL' cloudinit-userdata carries the metal-internal /etc/hosts mapping for $MI_CIDR"; exit 0; } || exit 1 ;; diff --git a/tests/dc-node-etchosts/run-tests.sh b/tests/dc-node-etchosts/run-tests.sh index 63ca41f..c5b089c 100644 --- a/tests/dc-node-etchosts/run-tests.sh +++ b/tests/dc-node-etchosts/run-tests.sh @@ -9,7 +9,11 @@ # an address IN the metal-internal CIDR, SKIPS one outside it (blast-radius), and # is IDEMPOTENT. We run the real rendered command against a temp hosts file + a # stub `hostname`, so behaviour is proven, not just "it parsed". -# 3. every assertion can FAIL (wrong CIDR, unscoped, non-idempotent, refuse). +# 3. the render uses a JUJU-ACCEPTED key (postruncmd/preruncmd), NOT a bare `runcmd` +# which juju model-config rejects -- cloud-init accepts runcmd as valid YAML, so YAML +# validity alone cannot catch it (T10; the 2026-08-04 live-apply failure this harness +# was green through). +# 4. every assertion can FAIL (wrong CIDR, unscoped, non-idempotent, refuse, bare runcmd). # ASCII + LF. Exit 0 all pass | 1 any fail. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -25,19 +29,19 @@ bash "$SCRIPT" render "$1" | python3 -c " import sys, yaml d = yaml.safe_load(sys.stdin) -cmd = d['runcmd'][0][2] +cmd = d['postruncmd'][0][2] print(cmd.replace('/etc/hosts', '$TMP/hosts')) " } -# T1 render dc0 parses as cloud-init YAML with the runcmd in the /bin/bash -c list form +# T1 render dc0 parses as cloud-init YAML with the postruncmd in the /bin/bash -c list form if bash "$SCRIPT" render vr1-dc0 | python3 -c " import sys, yaml d = yaml.safe_load(sys.stdin) -rc = d['runcmd'][0] -assert isinstance(rc, list) and rc[0:2]==['/bin/bash','-c'], 'runcmd not /bin/bash -c list form' -assert d.get('runcmd'), 'no runcmd' -" 2>/dev/null; then ok "T1 render vr1-dc0 is valid cloud-init YAML (/bin/bash -c runcmd)"; else no "T1 render vr1-dc0 valid YAML"; fi +rc = d['postruncmd'][0] +assert isinstance(rc, list) and rc[0:2]==['/bin/bash','-c'], 'postruncmd not /bin/bash -c list form' +assert d.get('postruncmd'), 'no postruncmd' +" 2>/dev/null; then ok "T1 render vr1-dc0 is valid cloud-init YAML (/bin/bash -c postruncmd)"; else no "T1 render vr1-dc0 valid YAML"; fi # T2 dc0 carries dc0's metal-internal CIDR bash "$SCRIPT" render vr1-dc0 | grep -q '10.12.12.0/22' && ok "T2 dc0 render pins metal-internal 10.12.12.0/22" || no "T2 dc0 CIDR" @@ -83,6 +87,18 @@ bash "$SCRIPT" frobnicate vr1-dc0 >/dev/null 2>&1; rc=$? [ "$rc" -eq 2 ] && ok "T9 REFUSES an unknown mode (rc=2)" || no "T9 refuse mode (rc=$rc)" +# T10 JUJU-ACCEPTANCE PROXY (the gap that let the live apply fail on 2026-08-04 while +# this harness was green): juju model-config REJECTS a top-level `runcmd` in +# cloudinit-userdata ("runcmd not allowed, use preruncmd or postruncmd instead"), +# so the render MUST use postruncmd/preruncmd and MUST NOT emit a bare `runcmd:`. +# cloud-init itself accepts runcmd as valid YAML, so YAML validity (T1) cannot catch +# this -- it is a juju-specific constraint. Proves the failing direction: revert the +# script to `runcmd:` and T10 goes red. +r10="$(bash "$SCRIPT" render vr1-dc0 2>/dev/null)" +if grep -qE '^(post|pre)runcmd:' <<<"$r10" && ! grep -qE '^runcmd:' <<<"$r10"; then + ok "T10 render uses juju-accepted postruncmd/preruncmd, not a bare runcmd (juju rejects runcmd)" +else no "T10 juju-accepted key (top-level keys: $(grep -oE '^[a-z]+runcmd:|^runcmd:' <<<"$r10" | tr '\n' ' '))"; fi + echo echo "RESULT: PASS=$PASS FAIL=$FAIL" [ "$FAIL" -eq 0 ] && { echo "ALL PASS"; exit 0; } || exit 1