Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / archive / session-ledger-rotated-20260807.md
@JANeumatrix JANeumatrix 11 hours ago 11 KB savegame 2026-08-07 (part 2): bookend + sweep -- dc1 region standup + SEC-031 edge rebuilt
Raw Blame History

Rotated session-ledger summaries -- moved 2026-08-07 (GA-R4 rule 3 / F1)

Moved VERBATIM from docs/session-ledger.md at the 2026-08-06/07 close to keep the live ledger under the 300-line cap (rule 3). Oldest-first.

SESSION CLOSE 2026-08-03 -- Stage 5 dc0: bundle DEPLOYED, controller rebuilt, vault up; ovn-central cert DEFERRED (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, ~23 commits pushed. NO stage opened/closed. Scan: 3 decisions, SEC 28, D 142 / DOCFIX 209 / BUNDLEFIX 053 (D-141 + DOCFIX-208 assigned this session).
  • THE dc0 BUNDLE IS DEPLOYED and mostly converged: 9 machines started, mysql cluster ONLINE, vault init+unseal+root-CA (operator-run), ~25 units active, 0 in error. DOCFIX-208 fixed the machines-overlay omission; UCA signing-key failure root-caused; D-135 amendment (b) converged dc0 onto the apt caching proxy; the v4 VIP revert (D-141) cleared keystone's Invalid vips.
  • CONTROLLER REBUILT (Path C): a --force model destroy orphaned the model (missing status doc) and took the controller API down; rebuilt fresh from MAAS end to end. New runbook Paths M + C (juju model + controller teardown/rebuild), plus the measured kill-controller-fails-when-API-down + --no-wait guidance.
  • RULINGS: D-141 (IPAM dual-stack, v4 active / v6 reserved-until-capable); D-052 AMENDED (ovn-central "" default -> metal-internal, architecturally correct -- OVN NB/SB DB is a metal-internal service).
  • >>> ovn-central x3 DEGRADED, DEFERRED: charm-ovn-central LP #2044324 -- cert request carries NO common_name -> vault issues no server cert -> OVN NB/SB cluster not formed. Three remedies exhausted (reissue-certificates / rebind / relation bounce). Only OVN/tenant-networking is gated. <<<
  • GATE RED AT CLOSE: gauntlet 1/98 FAIL (provider-bundle-check) -- the D-141 v4 revert broke 4 dual-family harness cases; LOGGED, harness owes a reconcile to the v4-only input. repo-lint 0 fail.
  • CONFIRMED (operator Q): a dual-stack deploy overlay with unpopulated container v6 causes the Invalid vips block on all 11 API charms (the charm hard-errors on an unplaceable VIP) -- so v4-only-deploy + v6-reserved-in-apex (D-141) is the correct split until the container-v6 gate clears.
  • OWNED: twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update.
  • DURABILITY: vcloud 0/0; voffice1 was 36 behind, PULLED to sync (new savegame Step 1b, operator-directed); dc0 rack ~/repo-stage matches HEAD.
  • NEXT: escalate LP #2044324 + decide accept-degraded vs the unverified os-*-network avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt (4 FIRST SURFACE). Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, 5 commits pushed (bd7a1d0..9b49b84; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, SEC 29 (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS).
  • >>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<< charm-ovn-central derives its TLS common_name from get_hostname(its metal-internal address); metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane.
  • The prior diagnosis was WRONG on every count, corrected in-record: LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding).
  • App STAYS on metal-internal (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it.
  • Fix PROVEN end-to-end (controlled single-unit LIVE test): an /etc/hosts reverse entry -> CN populated -> vault issued ovn-central_0.server.cert -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN.
  • WIRED for the redeploy: NEW scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (9/9) render a per-DC cloudinit-userdata adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated Step 1.2b of phase-01-bundle-deploy.md. A NEW mechanism borrowing D-008's shape, NOT D-008.
  • F2/F3 resolve in the clean redeploy: the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only.
  • Gauntlet ALL GREEN (99), repo-lint 0 fail. voffice1 PULLED to sync (9b49b84); dc0 rack ~/repo-stage lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2).
  • OWNED: called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking).
  • NEXT: we are at STEP 3 -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: docs/audit/queued-findings-20260804-ovn-cert-fix.txt (4 FIRST SURFACE). Body: docs/changelog-20260804-ovn-central-cert-fix.md. Status ONLY in CURRENT-STATE.md.

    Rotated session-ledger summaries -- moved 2026-08-07 (GA-R4 rule 3, <300 cap)

SESSION CLOSE 2026-08-05 -- vault init DONE + ovn-central RESOLVED; D-142 vault-init QoL saved (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, pushed d6975fb..ca3c0d3. NO stage opened/closed; Stage 5 remains OPEN. Scan: 4 PROPOSED/OPEN decisions (D-142 added), SEC 29, next-free D-143 / DOCFIX-209 / BUNDLEFIX-053.
  • >>> VAULT INIT COMPLETE (operator-run one-shot) + ovn-central RESOLVED end-to-end. <<< Fresh phase-02-vault-preflight vr1-dc0 -> PROCEED; operator ran phase-02-vault-bringup 2.1-2.3 VERBATIM on the dc0 rack (-m vr1-dc0): init (5 shares + TOKEN_OK, 2>&1|tee), unseal 3/3, authorize + generate-root-ca -> vault active/idle. Root CA (public): valid 2026-08-05 02:05:57 -> 2036-08-02 01:06:27 UTC.
  • ovn-central/3,4,5 ALL active (/4 leader ovnnb_db+ovnsb_db, /5 northd) -- OVN NB/SB cluster FORMED. Closes the multi-session saga via BOTH cycle fixes: dc-node-etchosts Step 1.2b (CN delivery) + D-052 re-amendment ''->metal-admin (reachability). Census 40->47 active, 0 error.
  • D-142 PROPOSED -- vault-init workflow QoL sweep SAVED (approved-in-principle, IMPL DEFERRED per operator "run current commands now, test QoL next opportunity"; R2 off-host transport UNRESOLVED). Full R1-R5 + hidden-prompt safety analysis + tee-write residual + pre-init writability probe + R3 harness constraints: docs/audit/vault-init-qol-proposal-20260805.md. Fold-register F13 (D-138 -m openstack->-m vr1-dc0, scope-stretch stated). Distinct from D-068 (substrate) / D-011.6 (unseal bar).
  • Hung-apps directive (fulfilled): the "hung" set was mostly correctly-waiting (ovn certs, designate/octavia/rbd-mirror). GENUINELY stuck: ceph-mon/2 + ceph-radosgw/0 allocating (apt-wedge class) -> cascade to ceph-osd/glance/nova-compute. Logged, not fixed (hard rule 1).
  • Security hygiene: child token printed in the operator's paste (init keys + root token WERE redacted) was ttl=10m/unset/expired -> benign, not stored/echoed/committed. Lesson: redact token-create output too.
  • Durability: vcloud 0 uncommitted / 0 unpushed at close; repo-lint 0 fail / 1 legacy warn; voffice1 PULLED to sync (was at 3321c57, 4 behind -> 8305ceb); dc0 rack ~/repo-stage UNAFFECTED (docs-only session, no staged script touched; phase-02-vault-preflight.sh sha 90910dfb verified == repo today). Gauntlet not owed (docs-only, no script/harness change). Root CA + vault init recorded no-secrets (root CA openssl-verified on the rack) in docs/audit/stage5-dc0-redeploy-asexec-20260804.txt.
  • NEXT: triage the ceph-mon/2 + ceph-radosgw/0 apt-wedge (appendix-A) -> ceph/storage-backend convergence -> continue Stage 5 (phase-03 core verify). Sweep: docs/audit/queued-findings-20260805-vault-init-close.txt (3 items). Body: docs/changelog-20260805-vault-init-ovn-resolved.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-05 (part 2) -- D-121 HA scale-up 14/14 LIVE + vault METAL-ONLY (D-020 amdt); harness + amendment UNCOMMITTED (Task #2) (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 5 commits pushed today (33f9158 71c5b97 c1111a3 ae06c72 4392118). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-210 / BUNDLEFIX-054.
  • CEPH: ceph-mon/2 + ceph-radosgw/0 "allocating" ROOT-CAUSED to apt-cacher-ng CLOSE-WAIT (NOT proxy-down; proxies PASS), remediated (remove-unit -> remove-machine --force -> re-add); appendix-A entry added. Body: changelog-20260805-stage5-dc0-ceph-remediation.md.
  • D-121 EXECUTED LIVE -- all 14 control-plane apps scaled 1->3 HA (8 Wave-1 + placement pilot + nova-cc + rabbitmq + barbican + keystone + vault). barbican-vault RESOLVED. Findings: apt-concurrency forces 1-app/2-container batches; PROACTIVE cluster_count=3 avoids the keystone VIP auth-blip (F5).
  • BUNDLE: BUNDLEFIX-053 -- num_units/cluster_count 1->3, vault-hacluster + vault:ha, rabbitmq min-cluster-size; every change charm-doc-verified; provider-bundle-check PASS. Body: changelog-20260805-d121-ha-scaleup.md.
  • VAULT: 3-unit HA on MySQL backend (operator unsealed vault/1,2). D-020 AMENDMENT ratified 2026-08-05 -> vault METAL-ONLY (PKI/CA root not tenant-facing; provider VIP unhostable -- no provider leg, findif). Live metal-only DONE, vault-hacluster UNBLOCKED.
  • SKILL: DOCFIX-209 (session-close bookend text). Body: changelog-20260805-skill-close-convention-docfix.md.
  • OWNED: shipped BUNDLEFIX-053 (ae06c72) WITHOUT running its harness -> harness now RED (Task #2, F1); twice mis-framed vault init-state before measuring the settling mysql-router; nearly concluded MySQL backend un-shared during a relation-settle.
  • RED AT CLOSE (both = Task #2): repo-lint L10 (held amendment) + provider-bundle-check harness 6/? FAIL. Amendment + dc0 overlay + check-exception are UNCOMMITTED (on disk).
  • DURABILITY: vcloud 0 unpushed; 3 files UNCOMMITTED (Task #2, held); voffice1 PULLED to 4392118; dc0 rack ~/repo-stage/bundle.yaml may be STALE post-BUNDLEFIX-053 (F9).
  • NEXT: Task #2 (harness reconcile + amendment/overlay/check commit + CURRENT-STATE 14/14); Task #1 (post-wave bundle/overlay review); verify vault ha_enabled/(v-a) (F4) + ceph-radosgw radosgw-service (F8). Sweep: docs/audit/queued-findings-20260805-d121-ha-vault.txt. Status ONLY in CURRENT-STATE.md.