# Rotated session-ledger summaries -- moved 2026-08-07 (GA-R4 rule 3 / F1)

Moved VERBATIM from `docs/session-ledger.md` at the 2026-08-06/07 close to keep the live
ledger under the 300-line cap (rule 3). Oldest-first.

---

## SESSION CLOSE 2026-08-03 -- Stage 5 dc0: bundle DEPLOYED, controller rebuilt, vault up; ovn-central cert DEFERRED (bounded, GA-R4)

- Branch `dc-dc-stage5-preconditions`, ~23 commits pushed. NO stage opened/closed. Scan: 3 decisions, **SEC 28**, **D 142 / DOCFIX 209 / BUNDLEFIX 053** (D-141 + DOCFIX-208 assigned this session).
- **THE dc0 BUNDLE IS DEPLOYED and mostly converged:** 9 machines started, mysql cluster ONLINE, vault init+unseal+root-CA (operator-run), ~25 units active, 0 in error. DOCFIX-208 fixed the machines-overlay omission; UCA signing-key failure root-caused; **D-135 amendment (b)** converged dc0 onto the apt caching proxy; the **v4 VIP revert (D-141)** cleared keystone's `Invalid vips`.
- **CONTROLLER REBUILT (Path C):** a `--force` model destroy orphaned the model (missing status doc) and took the controller API down; rebuilt fresh from MAAS end to end. New runbook **Paths M + C** (juju model + controller teardown/rebuild), plus the measured `kill-controller`-fails-when-API-down + `--no-wait` guidance.
- **RULINGS:** **D-141** (IPAM dual-stack, v4 `active` / v6 `reserved`-until-capable); **D-052 AMENDED** (ovn-central `""` default -> metal-internal, architecturally correct -- OVN NB/SB DB is a metal-internal service).
- **>>> ovn-central x3 DEGRADED, DEFERRED:** charm-ovn-central **LP #2044324** -- cert request carries NO `common_name` -> vault issues no server cert -> OVN NB/SB cluster not formed. Three remedies exhausted (reissue-certificates / rebind / relation bounce). Only OVN/tenant-networking is gated. <<<
- **GATE RED AT CLOSE:** gauntlet **1/98 FAIL (provider-bundle-check)** -- the D-141 v4 revert broke 4 dual-family harness cases; LOGGED, harness owes a reconcile to the v4-only input. repo-lint 0 fail.
- **CONFIRMED (operator Q):** a dual-stack deploy overlay with unpopulated container v6 causes the `Invalid vips` block on all 11 API charms (the charm hard-errors on an unplaceable VIP) -- so v4-only-deploy + v6-reserved-in-apex (D-141) is the correct split until the container-v6 gate clears.
- **OWNED:** twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update.
- **DURABILITY:** vcloud 0/0; voffice1 was **36 behind, PULLED to sync** (new savegame **Step 1b**, operator-directed); dc0 rack `~/repo-stage` matches HEAD.
- **NEXT:** escalate LP #2044324 + decide accept-degraded vs the unverified `os-*-network` avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: `docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt` (**4 FIRST SURFACE**). Status ONLY in CURRENT-STATE.md.

## SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4)

- Branch `dc-dc-stage5-preconditions`, **5 commits** pushed (`bd7a1d0..9b49b84`; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, **SEC 29** (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS).
- **>>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<<** charm-ovn-central derives its TLS common_name from `get_hostname(its metal-internal address)`; metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane.
- **The prior diagnosis was WRONG on every count, corrected in-record:** LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding).
- **App STAYS on metal-internal** (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it.
- **Fix PROVEN end-to-end (controlled single-unit LIVE test):** an /etc/hosts reverse entry -> CN populated -> vault issued `ovn-central_0.server.cert` -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN.
- **WIRED for the redeploy:** NEW `scripts/dc-node-etchosts.sh` + `tests/dc-node-etchosts` (9/9) render a per-DC `cloudinit-userdata` adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated **Step 1.2b** of `phase-01-bundle-deploy.md`. A NEW mechanism borrowing D-008's shape, NOT D-008.
- **F2/F3 resolve in the clean redeploy:** the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only.
- Gauntlet **ALL GREEN (99)**, repo-lint 0 fail. voffice1 PULLED to sync (`9b49b84`); dc0 rack `~/repo-stage` lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2).
- **OWNED:** called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking).
- **NEXT:** we are at **STEP 3** -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: `docs/audit/queued-findings-20260804-ovn-cert-fix.txt` (4 FIRST SURFACE). Body: `docs/changelog-20260804-ovn-central-cert-fix.md`. Status ONLY in CURRENT-STATE.md.
# Rotated session-ledger summaries -- moved 2026-08-07 (GA-R4 rule 3, <300 cap)

## SESSION CLOSE 2026-08-05 -- vault init DONE + ovn-central RESOLVED; D-142 vault-init QoL saved (bounded, GA-R4)

- Branch `dc-dc-stage5-preconditions`, pushed `d6975fb..ca3c0d3`. NO stage opened/closed; Stage 5 remains OPEN. Scan: **4** PROPOSED/OPEN decisions (D-142 added), **SEC 29**, next-free **D-143 / DOCFIX-209 / BUNDLEFIX-053**.
- **>>> VAULT INIT COMPLETE (operator-run one-shot) + ovn-central RESOLVED end-to-end. <<<** Fresh `phase-02-vault-preflight vr1-dc0` -> PROCEED; operator ran phase-02-vault-bringup 2.1-2.3 VERBATIM on the dc0 rack (`-m vr1-dc0`): init (5 shares + TOKEN_OK, `2>&1|tee`), unseal 3/3, authorize + generate-root-ca -> vault active/idle. Root CA (public): valid 2026-08-05 02:05:57 -> 2036-08-02 01:06:27 UTC.
- **ovn-central/3,4,5 ALL active** (/4 leader ovnnb_db+ovnsb_db, /5 northd) -- OVN NB/SB cluster FORMED. Closes the multi-session saga via BOTH cycle fixes: dc-node-etchosts Step 1.2b (CN delivery) + D-052 re-amendment `''`->metal-admin (reachability). Census 40->47 active, 0 error.
- **D-142 PROPOSED -- vault-init workflow QoL sweep SAVED (approved-in-principle, IMPL DEFERRED per operator "run current commands now, test QoL next opportunity"; R2 off-host transport UNRESOLVED).** Full R1-R5 + hidden-prompt safety analysis + tee-write residual + pre-init writability probe + R3 harness constraints: `docs/audit/vault-init-qol-proposal-20260805.md`. Fold-register **F13** (D-138 `-m openstack`->`-m vr1-dc0`, scope-stretch stated). Distinct from D-068 (substrate) / D-011.6 (unseal bar).
- **Hung-apps directive (fulfilled):** the "hung" set was mostly correctly-waiting (ovn certs, designate/octavia/rbd-mirror). GENUINELY stuck: **ceph-mon/2 + ceph-radosgw/0** `allocating` (apt-wedge class) -> cascade to ceph-osd/glance/nova-compute. Logged, not fixed (hard rule 1).
- **Security hygiene:** child token printed in the operator's paste (init keys + root token WERE redacted) was ttl=10m/unset/expired -> benign, not stored/echoed/committed. Lesson: redact token-create output too.
- **Durability:** vcloud 0 uncommitted / 0 unpushed at close; repo-lint 0 fail / 1 legacy warn; **voffice1 PULLED to sync** (was at `3321c57`, 4 behind -> `8305ceb`); dc0 rack `~/repo-stage` UNAFFECTED (docs-only session, no staged script touched; `phase-02-vault-preflight.sh` sha `90910dfb` verified == repo today). Gauntlet not owed (docs-only, no script/harness change). Root CA + vault init recorded no-secrets (root CA openssl-verified on the rack) in `docs/audit/stage5-dc0-redeploy-asexec-20260804.txt`.
- **NEXT:** triage the ceph-mon/2 + ceph-radosgw/0 apt-wedge (appendix-A) -> ceph/storage-backend convergence -> continue Stage 5 (phase-03 core verify). Sweep: `docs/audit/queued-findings-20260805-vault-init-close.txt` (3 items). **Body: `docs/changelog-20260805-vault-init-ovn-resolved.md`.** Status ONLY in CURRENT-STATE.md.

## SESSION CLOSE 2026-08-05 (part 2) -- D-121 HA scale-up 14/14 LIVE + vault METAL-ONLY (D-020 amdt); harness + amendment UNCOMMITTED (Task #2) (bounded, GA-R4)

- Branch dc-dc-stage5-preconditions; 5 commits pushed today (33f9158 71c5b97 c1111a3 ae06c72 4392118). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-210 / BUNDLEFIX-054.
- **CEPH:** ceph-mon/2 + ceph-radosgw/0 "allocating" ROOT-CAUSED to apt-cacher-ng CLOSE-WAIT (NOT proxy-down; proxies PASS), remediated (remove-unit -> remove-machine --force -> re-add); appendix-A entry added. Body: `changelog-20260805-stage5-dc0-ceph-remediation.md`.
- **D-121 EXECUTED LIVE -- all 14 control-plane apps scaled 1->3 HA** (8 Wave-1 + placement pilot + nova-cc + rabbitmq + barbican + keystone + vault). barbican-vault RESOLVED. Findings: apt-concurrency forces 1-app/2-container batches; PROACTIVE cluster_count=3 avoids the keystone VIP auth-blip (F5).
- **BUNDLE:** BUNDLEFIX-053 -- num_units/cluster_count 1->3, vault-hacluster + vault:ha, rabbitmq min-cluster-size; every change charm-doc-verified; provider-bundle-check PASS. Body: `changelog-20260805-d121-ha-scaleup.md`.
- **VAULT:** 3-unit HA on MySQL backend (operator unsealed vault/1,2). D-020 AMENDMENT ratified 2026-08-05 -> vault METAL-ONLY (PKI/CA root not tenant-facing; provider VIP unhostable -- no provider leg, findif). Live metal-only DONE, vault-hacluster UNBLOCKED.
- **SKILL:** DOCFIX-209 (session-close bookend text). Body: `changelog-20260805-skill-close-convention-docfix.md`.
- **OWNED:** shipped BUNDLEFIX-053 (ae06c72) WITHOUT running its harness -> harness now RED (Task #2, F1); twice mis-framed vault init-state before measuring the settling mysql-router; nearly concluded MySQL backend un-shared during a relation-settle.
- **RED AT CLOSE (both = Task #2):** repo-lint L10 (held amendment) + provider-bundle-check harness 6/? FAIL. Amendment + dc0 overlay + check-exception are UNCOMMITTED (on disk).
- **DURABILITY:** vcloud 0 unpushed; **3 files UNCOMMITTED** (Task #2, held); voffice1 PULLED to 4392118; dc0 rack `~/repo-stage/bundle.yaml` may be STALE post-BUNDLEFIX-053 (F9).
- **NEXT:** Task #2 (harness reconcile + amendment/overlay/check commit + CURRENT-STATE 14/14); Task #1 (post-wave bundle/overlay review); verify vault `ha_enabled`/(v-a) (F4) + ceph-radosgw radosgw-service (F8). Sweep: `docs/audit/queued-findings-20260805-d121-ha-vault.txt`. Status ONLY in CURRENT-STATE.md.