==============================================================================
AS-EXECUTED CAPTURE -- Stage 5 dc0 clean redeploy (ovn-central cert fix)
Session 2026-08-04. Branch dc-dc-stage5-preconditions.
==============================================================================
WHY THIS FILE: run-logged.sh opens an interactive script(1) subshell that only
logs a live terminal; a Claude-driven session executes mutations as separate
tool-call processes that such a subshell cannot capture. Per operator ruling
2026-08-04 ("Option 3, capture as you go"), each gated mutation's exact command
and output is appended here instead -- the tool-call-friendly substitute that
keeps the irreversible-step record (vault init, destroy-model) EXISTING and
FINDABLE (docs/as-executed-log-convention.md purpose).
CONVENTION per entry: [UTC] <where-run> :: <exact command> -> <result/rc>.
Read-only measurements are NOT logged here (they are in the transcript); only
mutations are. Status claims live in docs/CURRENT-STATE.md ONLY.
Known coverage gap declared: read-only census/measurements this session live in
the session transcript, not in this file. This file covers MUTATIONS only.
------------------------------------------------------------------------------
## M-0 F3 redeploy prerequisite -- stage dc-node-etchosts.sh to the dc0 rack
[2026-08-04] vcloud -> vr1-dc0-rack ::
scp -p scripts/dc-node-etchosts.sh \
vr1-dc0-rack:/home/jessea123/repo-stage/scripts/dc-node-etchosts.sh
-> rc=0
VERIFY (vr1-dc0-rack, sha256 first 16):
dc-node-etchosts.sh staged=0f79de90c19e9f5a repo=0f79de90c19e9f5a MATCH mode=664
lib-net.sh (dep) staged=3e465ae7de3bf92d repo=3e465ae7de3bf92d MATCH
RESULT: F3 CLOSED -- redeploy prerequisite satisfied.
REVERT: rm ~/repo-stage/scripts/dc-node-etchosts.sh on the rack (restores prior
staged set; the script is a redeploy input, not live cloud state).
## M.2 Path M teardown of the vr1-dc0 model [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Proceed with Path M.2 teardown now".
Pre-teardown MAAS baseline (profile dc0, rack hot-kid): 10 machines, all Deployed,
owner juju-vr1-dc0. juju storage empty (no storage flag). juju offers empty.
[2026-08-04 ~18:2x] vr1-dc0-rack ::
juju destroy-model vr1-dc0 --no-prompt
(WITHOUT --force -- agents alive; --force orphaned the model + downed the
controller on 08-03. WITHOUT --no-wait -- reserved for the agents-stopped stall.)
DRAIN (progressive, hooks executing -- the proven path, NOT the 07-31 flat stall):
36 -> 35 -> 34 -> 30 -> 28 -> 27 -> 25 -> 24 -> 23 -> 22 -> 21 machines ... (in progress)
juju models: vr1-dc0 state=destroying.
DRAIN OUTCOME: graceful drain reached the tail then WEDGED flat at 15 machines
(1 control host + 14 LXD containers) for ~12 min. Retry worker ALIVE (attempt
25 -> 28 -> 37 climbing) but count would not drop: 4 units stuck in workload
error with failed teardown hooks their idle agents could not complete --
magnum/0 (identity-service-relation-departed), octavia/0
(certificates-relation-changed), ovn-central/0 (ovsdb-cms-relation-broken),
placement/0 (identity-service-relation-departed). This is M.2's documented
agents-can't-run-stop-hooks stall. Agents were EXECUTING (not stopped), so per
M.2 --force was NOT auto-warranted; presented as a gated decision.
INSTRUMENT NOTE (owned): first watcher counted juju status --format=json
.machines and read 1 (collapses to top-level host during destroy, hiding the
14 containers) -> false STALL alarm. Real total = the "found N machines" retry
line. Corrected.
## M.2b Clear the stall via juju resolved [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Try `juju resolved` on the 4 error units first".
[2026-08-04] vr1-dc0-rack ::
juju resolved --no-retry -m vr1-dc0 magnum/0 octavia/0 ovn-central/0 placement/0
-> rc=0
(--no-retry: skip the failing teardown hook rather than re-run it into the same
failure; safe in a destroy -- MAAS re-images the machines.)
RESULT: unblocked -- applications 53 -> 41 within ~30s; attempts climbing (37).
New error units surface as the drain proceeds (barbican/0, designate/0 next),
so continuing with a bounded resolve-loop (juju resolved --no-retry --all every
~25s) until the model is GONE (M.5 positive-signal verified). <loop in progress>
REVERT: n/a -- teardown is intended.
## M.2c Finish the stall via --force --no-wait [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Escalate: --force --no-wait now".
CONDITION AT ESCALATION: resolve loop took the drain 15 -> 3 machines, then the
last 3 records (control-01 + 0/lxd/2 + 0/lxd/18) went to machine-state STOPPED --
agents down, no teardown hook can run (the genuine M.2 agents-stopped stall that
juju resolved cannot address). Model was 'destroying' (NOT 'dead'), so re-issuing
is the documented escalation, not the forbidden dead-model re-issue.
[2026-08-04 ~19:29] vr1-dc0-rack ::
juju destroy-model vr1-dc0 --force --no-wait --no-prompt
-> "Destroying model" ... 3 -> 1 -> 0 machines ... "Model destroyed." rc=0
RESULT (M.2 overall): TEARDOWN COMPLETE, CLEAN, NO ORPHAN.
ORPHAN CHECK (the 08-03 failure -- did NOT recur):
- juju models: only 'controller' (available); vr1-dc0 fully gone, not life:dead.
- controller model API responds (juju status -m controller OK) -- 17070 up.
- models now = [('admin/controller','alive')] -- no dead-model residue;
add-model will not be refused.
M.5(b) CASCADE CHECK (profile dc0): 10 machines, count UNCHANGED from baseline
-> NO cascade. States: Ready 9 (role nodes, owner=None) + Deployed 1
(controller VM subtle-grouse). Exactly the expected post-teardown state.
REVERT: n/a -- teardown is intended; rebuild is M.6 (add-model + rebuild).
------------------------------------------------------------------------------
## M.5(c) proxy-survival post-checks (read-only, logged for the teardown record)
dc-cache-proxy check dc0: PASS (apt-cacher-ng active :3142, serves archive+UCA 200)
dc-snap-proxy check dc0: PASS (:3129, store API 200, CDN tunnel allowed, allowlist enforced)
=> M.5 ALL GREEN. Path M teardown complete + clean. Proceeding to M.6 rebuild.
## M.6 REBUILD
## Step 3.5 -- add-model + spaces + artifact source [MUTATION, gated]
[2026-08-04] vr1-dc0-rack (D-138; runbook says voffice1 but that predates D-138) ::
juju add-model vr1-dc0
-> "Added 'vr1-dc0' model with credential 'vr1-dc0-cred' for user 'admin'" rc=0
GATES (read-only):
(a) artifact source inherited from model-defaults: apt-http-proxy=http://10.12.8.4:3142,
snap-http[s]-proxy=http://10.12.8.4:3129, apt-mirror EMPTY. Correct per D-135
amendment (b) (dc0 on the caching PROXY, not the retired mirror). apt-mirror
NOT set (Step 3.5's mirror text is STALE -- runbook-fold gap, flagged).
(b) juju-spaces-check.sh vr1-dc0: PASS 0 fatal -- all six spaces present, none stale.
REVERT: juju destroy-model vr1-dc0 (empty model; trivial).
## QUEUED -- NOT EXECUTED THIS SESSION (operator request 2026-08-04, mid-redeploy)
Q1. "Queue a Tailscale installation for DC0 and DC1." DEFERRED (hard rule 1 -- new
scope is logged mid-step, not run). NEEDS A RULING BEFORE EXECUTION, not just an
ops task: it intersects ruled surfaces --
- D-107: Tailscale is the operator WORKSTATION / human-GUI path, NOT the DC nodes.
- D-128: durable DC reach is the base-leg oneshot, explicitly NOT tailscale-on-host.
- 2026-08-02 ruling: "VPN deferred to Roosevelt" (VPN class deferred).
ACTION OWED: grep the exact D-107/D-128 text + the 08-02 VPN-deferred ruling, then
put the DC-Tailscale question to the operator as a GA-R5 exchange (likely a new
D-number, GA-R3 A1: decides DC access at every future standup). Do NOT install until ruled.
## Step 1.2b -- metal-internal /etc/hosts cloudinit (the ovn-central cert fix) [MUTATION, gated]
FIRST ATTEMPT FAILED -- delivery-mechanism defect found:
juju model-config REJECTED the config: "cloudinit-userdata: runcmd not allowed,
use preruncmd or postruncmd instead" (rc=1, nothing set). The 08-04 fix proved the
CONCEPT (manual /etc/hosts -> cert) but its DELIVERY (cloudinit-userdata) was never
live-tested; the harness graded rendered YAML (cloud-init accepts runcmd) not juju
acceptance (juju forbids it). Classic can't-see-the-live-apply green harness.
FIX (repo change, delivered with discipline):
scripts/dc-node-etchosts.sh: render key runcmd: -> postruncmd: (repo sha
0f79de90.. -> 107fc197..); + inline comment forbidding revert.
tests/dc-node-etchosts: switched to postruncmd; NEW T10 asserts a juju-accepted
key + rejects bare runcmd (the missing gate). Harness 10/10; T10 MUTATION-PROVEN
to fail (revert to runcmd -> 5 FAIL). Re-staged to rack (sha 107fc197 verified).
RE-RUN (live delivery test that was missing):
[2026-08-04] vr1-dc0-rack ::
JUJU_MODEL=vr1-dc0 bash scripts/dc-node-etchosts.sh apply vr1-dc0 -> "OK: set" (juju ACCEPTED postruncmd)
JUJU_MODEL=vr1-dc0 bash scripts/dc-node-etchosts.sh check vr1-dc0 -> OK, carries 10.12.12.0/22 mapping, rc=0
GATE: check exit 0. On-node /etc/hosts delivery VERIFY-AT-PROVISION after Step 1.4.
REVERT (model-config): juju model-config -m vr1-dc0 --reset cloudinit-userdata.
## Step 4.2 dry-run gate (read-only, logged for the deploy record)
9 machines "0"-"8", every machine tags=openstack-vr1-dc0,<role> (3 control/2 compute/4 storage);
ceph-osd->5-8, nova-compute->3-4; no overlay error; rc=0. GATE PASS.
## Step 4.3 -- the real deploy [MUTATION, GA-R5 operator-ruled 2026-08-04]
Operator ruling (verbatim selection): "Proceed with the deploy".
[2026-08-04] vr1-dc0-rack ::
cd ~/repo-stage && juju deploy ./bundle.yaml \
--overlay overlays/vr1-dc0-vips.yaml --overlay overlays/vr1-dc0-machines.yaml \
--overlay overlays/vr1-dc0-octavia-pki.yaml -m vr1-dc0
-> "Deploy of bundle completed." rc=0. 9 machines + ~24 containers/units added.
ovn-central 0/lxd/15, 1/lxd/3, 2/lxd/3 (will provision WITH the postruncmd /etc/hosts fix).
REVERT: juju destroy-model vr1-dc0 (Path M).
## Step 4.4 -- merged-config gate (read-only, run immediately) GATE PASS
ovn-chassis all THREE options present (source=user): bridge-interface-mappings=
'br-ex:52:54:00:8c:2a:8c br-ex:52:54:00:50:48:88', ovn-bridge-mappings='physnet1:br-ex',
prefer-chassis-as-gw=True -- juju merged the options map key-by-key correctly.
openstack-origin/source: all set, none UNSET (mix of cloud:jammy-caracal + caracal,
matching the prior converged deploy / same bundle sha -- observation, not a defect).
First execution of this gate ran clean (no shape correction needed).
## CONVERGE INCIDENT -- mysql-innodb-cluster/1 wedged (08-03 apt recurrence) [operator-ruled]
SYMPTOM: mysql/1 (container 1/lxd/2) stuck agent=allocating "waiting for machine" ~90min.
cloud-init blocked since 20:24:54 on `eatmydata apt-get update` (capture=False) that
infinitely retried a hanging index (jammy-backports InRelease via the proxy) and never
completed -> jujud never installed (/var/lib/juju/tools empty). The 08-03 apt-wedge class;
appendix-A has NO entry for it (DOC GAP owed -- fold the symptom+fix into appendix-A).
REMEDY 1 (operator ruling: "Restart apt-cacher-ng proxy first") [MUTATION]:
vr1-dc0-rack :: sudo systemctl restart apt-cacher-ng -> active, :3142. Proxy PROVEN
healthy after (curl -x proxy jammy-backports InRelease = 200 in ~1s). But the container's
apt stayed independently wedged (stale looping process); restart alone did not clear it.
REMEDY 2 (operator ruling: "Remove + re-add the unit") [MUTATION]:
vr1-dc0-rack :: juju remove-unit mysql-innodb-cluster/1 --force --no-prompt -> rc=0
(--force: agent never came up, no hooks to run; unit never joined the cluster).
Verified: /1 gone, container 1/lxd/2 GONE.
vr1-dc0-rack :: juju add-unit mysql-innodb-cluster -n 1 --to lxd:1 -> rc=0
-> mysql-innodb-cluster/3 on fresh container 1/lxd/4 (provisioning against the healthy proxy).
REVERT: n/a (converge remediation; the fresh unit replaces the wedged one).
## >>> MAJOR FINDING (verify-at-provision gate CAUGHT IT): the D-052 amendment binding
## (53e046e) makes ovn-central UNREACHABLE from the controller. NEVER DEPLOYED BEFORE. <<<
MEASURED: all 3 ovn-central containers have a SINGLE leg on the ISOLATED metal-internal
plane (/0=10.12.12.106, /1=10.12.12.3, /2=10.12.12.1), NO default route, NO metal-admin
leg. -> cannot reach the controller (10.12.8.5/10.12.4.5 :17070 UNREACHABLE) -> "download
agent binaries" retries forever (attempt 547+) -> jujud installed but agent stuck
allocating -> ovn-central can NEVER come up. mysql containers have TWO legs (metal-admin +
metal-internal); ovn-central got metal-internal ONLY because the bundle binds EVERY
ovn-central endpoint incl the DEFAULT '' to metal-internal.
ROOT: commit 53e046e (2026-08-03 20:49) "apply ovn-central default -> metal-internal":
'': metal-admin -> '': metal-internal. juju uses the DEFAULT binding for the machine's
controller/management connection, so moving it to the isolated plane cut controller reach.
This amendment was NEVER DEPLOYED: the 08-03 deploy PREDATED it (its ovn-central units were
idle/up, degraded on cert only -- which REQUIRES controller reachability, impossible on a
metal-internal-only leg). This fresh deploy is its first exposure; it fails.
CROSS-CUT: the 08-04 /etc/hosts postruncmd fix (this session, PROVEN logic -- manual
WOULD-ADD test) makes get_hostname resolve INDEPENDENTLY of the binding. So the amendment's
stated purpose (single-space cert-CN resolution) is now covered by a different mechanism,
and the amendment's remaining effect is purely harmful (unreachable ovn-central).
=> This is a RULED-BUT-NOT-BUILT amendment whose deciding reason is refuted by measurement
(the D-139 shape). Needs a GA-R5 D-052 RE-AMENDMENT ruling. HOLD vault init until ruled
(settle gate PROCEED is true-but-irrelevant: it never checks ovn-central).
LOGGED (not fixed, hard rule 1): (a) do NOT edit bundle bindings mid-step -- present options;
(b) the postruncmd mechanism is STRUCTURALLY FRAGILE -- it runs at the END of juju's runcmd,
so ANY earlier blocker (apt wedge, agent-binary retry) prevents the /etc/hosts entry from
ever being written. Delivery is gated behind juju's entire runcmd completing. Record in the
changelog/appendix-A regardless of the binding ruling.
Model state PRESERVED (converged-but-ovn-blocked) for whichever way the operator rules.
## D-052 RE-AMENDMENT APPLIED -- ovn-central re-homed to metal-admin [MUTATION, GA-R5 2026-08-05]
Operator ruling (exact utterance): "Once that is done then move on to the proposed move to metal-admin".
Repo half committed d6975fb (bundle '' -> metal-admin; D-052 re-amendment; reference doc). Live half:
[2026-08-05] vr1-dc0-rack ::
scp -p bundle.yaml -> rack ~/repo-stage (sha 213b51baea23a3f9 verified)
juju remove-application ovn-central --force --no-prompt -> rc=0 (3 stuck units removed; agents
never came up, so --force clean -- containers were single-legged on metal-internal, unremovable in place)
DRY-RUN re-deploy: plan = add ovn-central + 4 relations (certificates, ovsdb-cms, ovsdb x2) + 3
units ONLY; no other app touched. Confirmed before applying.
juju deploy ./bundle.yaml --overlay {vips,machines,octavia-pki} -m vr1-dc0 -> "Deploy of bundle
completed" rc=0. ovn-central/3,4,5 on fresh 0/lxd/0, 1/lxd/0, 2/lxd/0 with '' = metal-admin.
GATE (pending verify): new containers must have a metal-admin leg (10.12.8.x) -> controller
reachable -> agent starts (NOT stuck allocating). Then /etc/hosts postruncmd at cloud-init end.
REVERT: n/a (this IS the fix; the prior binding is superseded by the D-052 re-amendment).
## >>> D-052 RE-AMENDMENT VALIDATED LIVE -- BOTH ovn-central FIXES CONFIRMED. <<<
(1) REACHABILITY: ovn-central/3,4,5 agents went allocating -> executing (poll 5). New
containers are TWO-LEGGED: ovn-central/3 (0/lxd/19) eth0 10.12.9.14 (metal-admin) +
eth1 10.12.12.129 (metal-internal). The metal-admin leg = controller reachable = agent
starts. The single-leg failure is CLOSED by the '' -> metal-admin re-amendment.
(2) CERT-CN DELIVERY: /etc/hosts on ovn-central/3 carries "10.12.12.129
juju-d4f45a-0-lxd-19.maas juju-d4f45a-0-lxd-19" and getent hosts reverse-RESOLVES it ->
get_hostname(metal-internal addr) will return a non-empty CN -> vault will issue the
server cert. This is the postruncmd DELIVERY PROOF that was impossible on 08-04 (the
containers were wedged/mis-bound). BOTH fixes work end-to-end pre-cert.
PRE-VAULT-INIT SETTLE RE-CONFIRMED after the re-home: phase-02-vault-preflight vr1-dc0 ->
PROCEED (mysql 3/3 active+ONLINE 1 R/W, vault fresh, census 66 units 0 workload-error
0 agent-error, 40 active). Model ready for vault init.
## HANDED OFF (operator-only one-shot): phase-02 vault init.
runbooks/phase-02-vault-bringup.md Step 2.1 (init/unseal/root-CA), run VERBATIM by the
operator on the DC rack (D-138 juju client); guard-hook blocks the agent (DOCFIX-006/D-069).
On completion: vault issues certs -> ovn-central requests its server cert with the resolved
CN -> OVN NB/SB cluster forms (6641/6642 listen) -> the redeploy's purpose is achieved.
## (delivery proof) -> phase-02 Step 2.1 vault init (OPERATOR-ONLY one-shot) -> ovn cert
## issues -> converge. (below appended as each further mutation executes)
## >>> VAULT INIT COMPLETE (operator-run one-shot) + ovn-central RESOLVED end-to-end. 2026-08-05 <<<
Operator ran phase-02-vault-bringup.md Steps 2.1-2.3 VERBATIM on the dc0 rack (D-138),
model -m vr1-dc0 (VR0 -m openstack head-substituted -- fold item F13 / D-142 R1). The agent
did NOT run these (guard-hook, secret custody); recorded here from the operator's pasted
transcript with ALL secret material excluded.
2.1 init: PROBE_OK (pre-init writability probe, D-142 safety add, run manually); pre-check
Initialized=false; init one-shot with `2>&1 | tee ~/vault-init/init.txt` -> GATE: 5 unseal
keys + TOKEN_OK. Off-host save handled by the operator by hand (R2 transport gap surfaced
verbally at hand-off; still the D-142 open item).
2.2 unseal: 1/3 -> 2/3 -> 3/3 -> Sealed=false, Initialized=true, Storage=mysql, HA=false;
Cluster vault-cluster-12969e82.
2.3 authorize + root CA: authorize-charm op1/task2 completed; generate-root-ca op3/task4
returned the root CA PEM. vault/0 -> active/idle "Unit is ready (active: true, mlock:
disabled)"; vault-mysql-router/0 active. EXIT GATE MET.
ROOT CA (PUBLIC -- no secret custody; as-built metadata only; openssl-verified on the rack
from the pasted PEM, NOT a self-decode): subject "CN = Vault Root Certificate Authority
(charm-pki-local)"; notBefore Aug 5 02:05:57 2026 GMT; notAfter Aug 2 01:06:27 2036 GMT
(~10y anchor); sha256 75:DF:33:97:43:48:17:FE:85:97:1C:EC:60:DB:D5:4B:0D:FB:60:4A:0B:F4:
60:08:92:E8:12:E5:29:69:35:A1. PEM body NOT stored (public but unnecessary).
SECURITY HYGIENE NOTE: the child token printed by `vault token create -field=token` appeared
in plaintext in the operator's paste (init keys + root token WERE redacted). It was
ttl=10m, unset after use, and long expired by paste time -> dead/benign, no rotation.
Lesson: redact the token-create output too. NOT stored/echoed/committed anywhere.
## >>> ovn-central CERT CASCADE -- the redeploy's PURPOSE -- ACHIEVED (measured, read-only). <<<
Post-init `juju status -m vr1-dc0`: ovn-central/3,4,5 ALL active/idle "Unit is ready";
/4 leader (ovnnb_db, ovnsb_db); /5 northd active. The OVN NB/SB cluster FORMED. This closes
the multi-session ovn-central saga: the CN-from-reverse-DNS-on-isolated-plane root cause
(08-04) is defeated by BOTH fixes landed this cycle -- the /etc/hosts postruncmd (CN delivery,
dc-node-etchosts.sh Step 1.2b) AND the D-052 re-amendment '' -> metal-admin (reachability).
Census moved 40 -> 47 active, 0 workload-error, 0 agent-error.
## REMAINING (separate item, NOT vault/ovn, logged not fixed -- hard rule 1):
ceph-mon/2 + ceph-radosgw/0 still `allocating`/"waiting for machine" (container provisioning,
likely the 08-03 apt-cacher-ng/cloud-init wedge class). Cascade: ceph-mon can't reach 3-peer
quorum -> ceph-osd x4 "Incomplete relation: monitor" -> glance + nova-compute x2 "Incomplete
storage-backend". To be triaged AFTER the bookend. Expected-waiting (not stuck): designate
(Stage-7 DNS), octavia (end-user configure-resources), ceph-rbd-mirror (two-DC).
