==============================================================================
AS-EXECUTED CAPTURE -- Stage 5 dc0 clean redeploy (ovn-central cert fix)
Session 2026-08-04. Branch dc-dc-stage5-preconditions.
==============================================================================
WHY THIS FILE: run-logged.sh opens an interactive script(1) subshell that only
logs a live terminal; a Claude-driven session executes mutations as separate
tool-call processes that such a subshell cannot capture. Per operator ruling
2026-08-04 ("Option 3, capture as you go"), each gated mutation's exact command
and output is appended here instead -- the tool-call-friendly substitute that
keeps the irreversible-step record (vault init, destroy-model) EXISTING and
FINDABLE (docs/as-executed-log-convention.md purpose).

CONVENTION per entry: [UTC] <where-run> :: <exact command> -> <result/rc>.
Read-only measurements are NOT logged here (they are in the transcript); only
mutations are. Status claims live in docs/CURRENT-STATE.md ONLY.

Known coverage gap declared: read-only census/measurements this session live in
the session transcript, not in this file. This file covers MUTATIONS only.
------------------------------------------------------------------------------

## M-0  F3 redeploy prerequisite -- stage dc-node-etchosts.sh to the dc0 rack
[2026-08-04] vcloud -> vr1-dc0-rack ::
  scp -p scripts/dc-node-etchosts.sh \
    vr1-dc0-rack:/home/jessea123/repo-stage/scripts/dc-node-etchosts.sh
  -> rc=0
VERIFY (vr1-dc0-rack, sha256 first 16):
  dc-node-etchosts.sh  staged=0f79de90c19e9f5a  repo=0f79de90c19e9f5a  MATCH  mode=664
  lib-net.sh (dep)     staged=3e465ae7de3bf92d  repo=3e465ae7de3bf92d  MATCH
RESULT: F3 CLOSED -- redeploy prerequisite satisfied.
REVERT: rm ~/repo-stage/scripts/dc-node-etchosts.sh on the rack (restores prior
        staged set; the script is a redeploy input, not live cloud state).

## M.2  Path M teardown of the vr1-dc0 model  [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Proceed with Path M.2 teardown now".
Pre-teardown MAAS baseline (profile dc0, rack hot-kid): 10 machines, all Deployed,
owner juju-vr1-dc0. juju storage empty (no storage flag). juju offers empty.
[2026-08-04 ~18:2x] vr1-dc0-rack ::
  juju destroy-model vr1-dc0 --no-prompt
  (WITHOUT --force -- agents alive; --force orphaned the model + downed the
   controller on 08-03. WITHOUT --no-wait -- reserved for the agents-stopped stall.)
DRAIN (progressive, hooks executing -- the proven path, NOT the 07-31 flat stall):
  36 -> 35 -> 34 -> 30 -> 28 -> 27 -> 25 -> 24 -> 23 -> 22 -> 21 machines ... (in progress)
  juju models: vr1-dc0 state=destroying.
DRAIN OUTCOME: graceful drain reached the tail then WEDGED flat at 15 machines
  (1 control host + 14 LXD containers) for ~12 min. Retry worker ALIVE (attempt
  25 -> 28 -> 37 climbing) but count would not drop: 4 units stuck in workload
  error with failed teardown hooks their idle agents could not complete --
  magnum/0 (identity-service-relation-departed), octavia/0
  (certificates-relation-changed), ovn-central/0 (ovsdb-cms-relation-broken),
  placement/0 (identity-service-relation-departed). This is M.2's documented
  agents-can't-run-stop-hooks stall. Agents were EXECUTING (not stopped), so per
  M.2 --force was NOT auto-warranted; presented as a gated decision.
INSTRUMENT NOTE (owned): first watcher counted juju status --format=json
  .machines and read 1 (collapses to top-level host during destroy, hiding the
  14 containers) -> false STALL alarm. Real total = the "found N machines" retry
  line. Corrected.

## M.2b  Clear the stall via juju resolved  [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Try `juju resolved` on the 4 error units first".
[2026-08-04] vr1-dc0-rack ::
  juju resolved --no-retry -m vr1-dc0 magnum/0 octavia/0 ovn-central/0 placement/0
  -> rc=0
  (--no-retry: skip the failing teardown hook rather than re-run it into the same
   failure; safe in a destroy -- MAAS re-images the machines.)
RESULT: unblocked -- applications 53 -> 41 within ~30s; attempts climbing (37).
  New error units surface as the drain proceeds (barbican/0, designate/0 next),
  so continuing with a bounded resolve-loop (juju resolved --no-retry --all every
  ~25s) until the model is GONE (M.5 positive-signal verified). <loop in progress>
REVERT: n/a -- teardown is intended.

## M.2c  Finish the stall via --force --no-wait  [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Escalate: --force --no-wait now".
CONDITION AT ESCALATION: resolve loop took the drain 15 -> 3 machines, then the
last 3 records (control-01 + 0/lxd/2 + 0/lxd/18) went to machine-state STOPPED --
agents down, no teardown hook can run (the genuine M.2 agents-stopped stall that
juju resolved cannot address). Model was 'destroying' (NOT 'dead'), so re-issuing
is the documented escalation, not the forbidden dead-model re-issue.
[2026-08-04 ~19:29] vr1-dc0-rack ::
  juju destroy-model vr1-dc0 --force --no-wait --no-prompt
  -> "Destroying model" ... 3 -> 1 -> 0 machines ... "Model destroyed." rc=0

RESULT (M.2 overall): TEARDOWN COMPLETE, CLEAN, NO ORPHAN.
  ORPHAN CHECK (the 08-03 failure -- did NOT recur):
    - juju models: only 'controller' (available); vr1-dc0 fully gone, not life:dead.
    - controller model API responds (juju status -m controller OK) -- 17070 up.
    - models now = [('admin/controller','alive')] -- no dead-model residue;
      add-model will not be refused.
  M.5(b) CASCADE CHECK (profile dc0): 10 machines, count UNCHANGED from baseline
    -> NO cascade. States: Ready 9 (role nodes, owner=None) + Deployed 1
    (controller VM subtle-grouse). Exactly the expected post-teardown state.
REVERT: n/a -- teardown is intended; rebuild is M.6 (add-model + rebuild).

------------------------------------------------------------------------------
## M.5(c) proxy-survival post-checks (read-only, logged for the teardown record)
  dc-cache-proxy check dc0: PASS (apt-cacher-ng active :3142, serves archive+UCA 200)
  dc-snap-proxy  check dc0: PASS (:3129, store API 200, CDN tunnel allowed, allowlist enforced)
  => M.5 ALL GREEN. Path M teardown complete + clean. Proceeding to M.6 rebuild.

## M.6 REBUILD
## Step 3.5 -- add-model + spaces + artifact source  [MUTATION, gated]
[2026-08-04] vr1-dc0-rack (D-138; runbook says voffice1 but that predates D-138) ::
  juju add-model vr1-dc0
  -> "Added 'vr1-dc0' model with credential 'vr1-dc0-cred' for user 'admin'" rc=0
GATES (read-only):
  (a) artifact source inherited from model-defaults: apt-http-proxy=http://10.12.8.4:3142,
      snap-http[s]-proxy=http://10.12.8.4:3129, apt-mirror EMPTY. Correct per D-135
      amendment (b) (dc0 on the caching PROXY, not the retired mirror). apt-mirror
      NOT set (Step 3.5's mirror text is STALE -- runbook-fold gap, flagged).
  (b) juju-spaces-check.sh vr1-dc0: PASS 0 fatal -- all six spaces present, none stale.
REVERT: juju destroy-model vr1-dc0 (empty model; trivial).

## QUEUED -- NOT EXECUTED THIS SESSION (operator request 2026-08-04, mid-redeploy)
Q1. "Queue a Tailscale installation for DC0 and DC1." DEFERRED (hard rule 1 -- new
    scope is logged mid-step, not run). NEEDS A RULING BEFORE EXECUTION, not just an
    ops task: it intersects ruled surfaces --
      - D-107: Tailscale is the operator WORKSTATION / human-GUI path, NOT the DC nodes.
      - D-128: durable DC reach is the base-leg oneshot, explicitly NOT tailscale-on-host.
      - 2026-08-02 ruling: "VPN deferred to Roosevelt" (VPN class deferred).
    ACTION OWED: grep the exact D-107/D-128 text + the 08-02 VPN-deferred ruling, then
    put the DC-Tailscale question to the operator as a GA-R5 exchange (likely a new
    D-number, GA-R3 A1: decides DC access at every future standup). Do NOT install until ruled.

## Step 1.2b -- metal-internal /etc/hosts cloudinit (the ovn-central cert fix)  [MUTATION, gated]
FIRST ATTEMPT FAILED -- delivery-mechanism defect found:
  juju model-config REJECTED the config: "cloudinit-userdata: runcmd not allowed,
  use preruncmd or postruncmd instead" (rc=1, nothing set). The 08-04 fix proved the
  CONCEPT (manual /etc/hosts -> cert) but its DELIVERY (cloudinit-userdata) was never
  live-tested; the harness graded rendered YAML (cloud-init accepts runcmd) not juju
  acceptance (juju forbids it). Classic can't-see-the-live-apply green harness.
FIX (repo change, delivered with discipline):
  scripts/dc-node-etchosts.sh: render key runcmd: -> postruncmd: (repo sha
    0f79de90.. -> 107fc197..); + inline comment forbidding revert.
  tests/dc-node-etchosts: switched to postruncmd; NEW T10 asserts a juju-accepted
    key + rejects bare runcmd (the missing gate). Harness 10/10; T10 MUTATION-PROVEN
    to fail (revert to runcmd -> 5 FAIL). Re-staged to rack (sha 107fc197 verified).
RE-RUN (live delivery test that was missing):
[2026-08-04] vr1-dc0-rack ::
  JUJU_MODEL=vr1-dc0 bash scripts/dc-node-etchosts.sh apply vr1-dc0  -> "OK: set" (juju ACCEPTED postruncmd)
  JUJU_MODEL=vr1-dc0 bash scripts/dc-node-etchosts.sh check vr1-dc0  -> OK, carries 10.12.12.0/22 mapping, rc=0
GATE: check exit 0. On-node /etc/hosts delivery VERIFY-AT-PROVISION after Step 1.4.
REVERT (model-config): juju model-config -m vr1-dc0 --reset cloudinit-userdata.

## Step 4.2 dry-run gate (read-only, logged for the deploy record)
  9 machines "0"-"8", every machine tags=openstack-vr1-dc0,<role> (3 control/2 compute/4 storage);
  ceph-osd->5-8, nova-compute->3-4; no overlay error; rc=0. GATE PASS.

## (below appended as each further mutation executes -- NEXT: Step 4.3 deploy, operator-gated)