Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / archive / session-ledger-rotated-20260807.md
@JANeumatrix JANeumatrix 8 hours ago 5 KB Session bookend 2026-08-07 (dc0 tailscale .7 carved-and-ready): GA-R4 + sweep
Raw Blame History

Rotated session-ledger summaries -- moved 2026-08-07 (GA-R4 rule 3 / F1)

Moved VERBATIM from docs/session-ledger.md at the 2026-08-06/07 close to keep the live ledger under the 300-line cap (rule 3). Oldest-first.

SESSION CLOSE 2026-08-03 -- Stage 5 dc0: bundle DEPLOYED, controller rebuilt, vault up; ovn-central cert DEFERRED (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, ~23 commits pushed. NO stage opened/closed. Scan: 3 decisions, SEC 28, D 142 / DOCFIX 209 / BUNDLEFIX 053 (D-141 + DOCFIX-208 assigned this session).
  • THE dc0 BUNDLE IS DEPLOYED and mostly converged: 9 machines started, mysql cluster ONLINE, vault init+unseal+root-CA (operator-run), ~25 units active, 0 in error. DOCFIX-208 fixed the machines-overlay omission; UCA signing-key failure root-caused; D-135 amendment (b) converged dc0 onto the apt caching proxy; the v4 VIP revert (D-141) cleared keystone's Invalid vips.
  • CONTROLLER REBUILT (Path C): a --force model destroy orphaned the model (missing status doc) and took the controller API down; rebuilt fresh from MAAS end to end. New runbook Paths M + C (juju model + controller teardown/rebuild), plus the measured kill-controller-fails-when-API-down + --no-wait guidance.
  • RULINGS: D-141 (IPAM dual-stack, v4 active / v6 reserved-until-capable); D-052 AMENDED (ovn-central "" default -> metal-internal, architecturally correct -- OVN NB/SB DB is a metal-internal service).
  • >>> ovn-central x3 DEGRADED, DEFERRED: charm-ovn-central LP #2044324 -- cert request carries NO common_name -> vault issues no server cert -> OVN NB/SB cluster not formed. Three remedies exhausted (reissue-certificates / rebind / relation bounce). Only OVN/tenant-networking is gated. <<<
  • GATE RED AT CLOSE: gauntlet 1/98 FAIL (provider-bundle-check) -- the D-141 v4 revert broke 4 dual-family harness cases; LOGGED, harness owes a reconcile to the v4-only input. repo-lint 0 fail.
  • CONFIRMED (operator Q): a dual-stack deploy overlay with unpopulated container v6 causes the Invalid vips block on all 11 API charms (the charm hard-errors on an unplaceable VIP) -- so v4-only-deploy + v6-reserved-in-apex (D-141) is the correct split until the container-v6 gate clears.
  • OWNED: twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update.
  • DURABILITY: vcloud 0/0; voffice1 was 36 behind, PULLED to sync (new savegame Step 1b, operator-directed); dc0 rack ~/repo-stage matches HEAD.
  • NEXT: escalate LP #2044324 + decide accept-degraded vs the unverified os-*-network avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt (4 FIRST SURFACE). Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 5 commits pushed (bd7a1d0..9b49b84; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, SEC 29 (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS).
  • >>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<< charm-ovn-central derives its TLS common_name from get_hostname(its metal-internal address); metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane.
  • The prior diagnosis was WRONG on every count, corrected in-record: LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding).
  • App STAYS on metal-internal (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it.
  • Fix PROVEN end-to-end (controlled single-unit LIVE test): an /etc/hosts reverse entry -> CN populated -> vault issued ovn-central_0.server.cert -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN.
  • WIRED for the redeploy: NEW scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (9/9) render a per-DC cloudinit-userdata adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated Step 1.2b of phase-01-bundle-deploy.md. A NEW mechanism borrowing D-008's shape, NOT D-008.
  • F2/F3 resolve in the clean redeploy: the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only.
  • Gauntlet ALL GREEN (99), repo-lint 0 fail. voffice1 PULLED to sync (9b49b84); dc0 rack ~/repo-stage lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2).
  • OWNED: called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking).
  • NEXT: we are at STEP 3 -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: docs/audit/queued-findings-20260804-ovn-cert-fix.txt (4 FIRST SURFACE). Body: docs/changelog-20260804-ovn-central-cert-fix.md. Status ONLY in CURRENT-STATE.md.