Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / archive / session-ledger-rotated-20260809.md
@JANeumatrix JANeumatrix 20 hours ago 10 KB savegame 2026-08-09 (part 2): open-items review bookend -- GA-R4 (sweep + ledger close + rotation + memory #25)
Raw Blame History

Session ledger -- rotated summaries (2026-08-09 pass, GA-R4 rule 3)

The two oldest closed-session summaries below were moved VERBATIM out of docs/session-ledger.md to restore the 300-line cap at the 2026-08-08 (part 3) close (amphora build FIXED + geneve-over-v6 root-caused). Oldest-first.

SESSION CLOSE 2026-08-06 (part 3) -- phase-03 core-verify: F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 9 commits (b8d2b25..edba699 pushed; cd13baa ec8a1d3 ace0e16 UNPUSHED). Scan: 29 SEC, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (used 056/057/058).
  • STEP 7 (phase-03 core-verify, adapted vr1-dc0, run from the dc0 rack): CORE-API VERIFIED (falsifiable settle walk, haproxy sweep, admin-openrc scoped token, IP-only endpoints, two-sourced VIP). F-CV2 RESOLVED: openstack CLI installed on the dc0 rack (D-138; was voffice1-only).
  • F-CV1 (designate _admin backend DOWN) RESOLVED (BUNDLEFIX-056): bundle omitted public+internal -> orphaned VIP triple; fixed + live-rebound. Governing = D-052 + D-020 amdt, NOT D-141.
  • F-CV3 (dashboard VIP plaintext) RESOLVED via D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057: VR1 split-metal INVERTS D-072 (charm serves metal-internal, no admin/internal binding/os-*-network lever) -> cluster->metal-internal. Proven LIVE before ratifying (operator prove-then-rule method).
  • BUNDLEFIX-058: designate-stack conformance (designate amqp+cluster, designate-bind cluster -> metal-internal). Binding conformance now CLEAN cloud-wide (no other active deviations).
  • D-134 Roosevelt-delta annotation (LXD container addrs auto-picked, not carved); gap-21 access-model context (tailnet -> metal-admin dashboards).
  • OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule.
  • DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16.
  • NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06/07 -- phase-03 Step 3.4 G3 DONE + per-DC Tailscale rulings + .7 VMs stood up both DCs (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; pushed afbaed2..9e74f43 (7 commits); the node-vm harness reconcile is UNCOMMITTED in this bookend. voffice1 synced 9e74f43. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- all D-129/D-134 AMENDMENTS).
  • STEP 3.4 (phase-03) DONE: scripts/g3-domain-manager-probe.sh (+harness 12/12) as the GA-R6 named check; live G3 PASS on dc0 (7 ok/0 fail, teardown verified clean); stage-1 PO: verified per-UNIT (app-aggregate hid it).
  • DECISION C (Horizon): reconciled to VR1 -- both dashboard VIPs serve HTTPS login 200+csrftoken; D-044 + the VR0 nginx-repoint are plain-HTTP-leg artifacts that would only WEAKEN the cookie. phase-03 does NOT close; Step 3.3 SPLITS to its own gate (tailnet-access-gated).
  • PER-DC TAILSCALE (D-129(iii) AMENDMENT, rulings a-d, BOTH DCs): (a) dedicated .7 VM; (b) STAR operator->DC; (c) SINGLE, HA scale-up PINNED; (d) SNAT on. D-134 octet map ->.7; D-107 citation DOCFIX-in-amendment. site-tailscale.sh (+harness 15/15).
  • SUBSTRATE: .7 VMs APPLIED (tofu, gated) + MACs pinned + tofu No-changes on BOTH DCs -- dc0 tailscale; dc1 region VM + tailscale (operator "dc1 full"). Capacity re-gated FIT 874/1024=85%.
  • DEFERRED: Headscale-side join (tagged authkey / autoApprovers / star ACL / Office1-untagged-fix) BLOCKED on control-plane access (operator lacks it); per-VM MAAS commission/deploy/carve/install; dc1 region SETUP workstream.
  • OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint.
  • Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md.
  • NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-07 (dc0 tailscale + dc1 region) -- dc0 .7 carved-and-ready; naming convention; vr1-dc1-region LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; commits d36d815..9216bdf (+ a final close) all pushed; voffice1 synced. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers). Body: docs/changelog-20260807-dc0-tailscale-provisioning.md (Items 1-10). Sweep: docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt (F1-F4).
  • dc0 .7 DRIVEN TO CARVED-AND-READY (gated): power->commission->carve pass=8/0 (metal-admin 10.12.8.7 + provider-public 10.12.4.7, no br-ex)->deploy jammy, both legs live. C2 correction: it had already self-enlisted (known-marten). Join still blocked (PLAIN key + no Headscale ACL; .7 reachable only via vr1-office1-svc).
  • NEW aux-carve tooling: dc-node-carve.sh is_tailscale/is_region/is_two_plane_host + lib-hosts CARVE_AUX_HOSTS (isolates aux VMs from HOSTS consumers); harness 63/0; gauntlet 101.
  • NAMING convention corrected + RULED standing: renamed dc0 machines to vr1-dc0-*; D-134 AMENDMENT ("Record that as the preferred naming convention going forward") + lib-hosts comment + memory.
  • dc1 RULED "No migration. Build region on DC1 correctly." + "Rebuild fresh into dc1-region"; then BUILT it: .6 (vr1-dc1-maas-01) power->recommission->carve (10.12.68.6/10.12.64.6, --profile admin)->deploy jammy; vr1-dc1-region MAAS LIVE (maas 3.7.2 + postgresql 16.14; the shared vr1-office1-svc key is ON VCLOUD ~/vr1-office1-creds/office1_svc_ed25519; reached the .6 from vcloud; snap egress via snapd proxy; operator-authorised credential one-shot, creds 0600 on the .6, never in context; 10.12.68.6:5240/MAAS/->301). PROFILE registered+verified (voffice1 tunnel -L 5243; rack vr1-dc1-maas-01, 0 machines).
  • OWNED: nearly read "powered off" as "not enlisted" (measured first); advisor caught the unverified provider-public leg + the missing bookend; a false-negative curl -x 400 vs snap-egress-works (tested definitively); run-logged.sh NOT opened (O3, 3rd consecutive).
  • OWED: consolidate the .6 MAAS creds to ~/vr1-dc1-creds/ (SEC-020); RE-OPEN the voffice1:5243 tunnel after any reboot before maas vr1-dc1-region ....
  • NEXT (dc1, --profile vr1-dc1-region --expect-rack vr1-dc1-maas-01, tunnel up): dc-region-topology.sh apply (6-plane) -> maas-region-power-key.sh (.6 snap) + dc-plane-ipam.sh + DHCP + jammy image sync -> import office1_svc pubkey -> REBUILD 9 nodes+juju+.7 fresh in (delete from Office1 admin -> enlist/commission/deploy/carve, name vr1-dc1--NN). Separately dc0 Step 3.3 on a TAGGED key + Headscale.
  • NEXT (NetBox, operator-flagged -- do not miss): record this session's new IPAM objects into office1-netbox (live apex, DOCFIX-195) -- dc0 .7 + dc1 .6 region VM + new vr1-dc1-region + dc1 .7 (when live) + D-134 utility .4-.9 assignments + dc0 renames. Enumerated: changelog-20260807 Item 11. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-07 (part 2) -- dc1 region STANDUP (topology/IPAM/DHCP cutover/power-key) + SEC-031 edge REBUILT + NetBox util-host importer (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 2 commits (b6798df, 2b476e4) pushed at close; voffice1 pulled to sync. Scan: 4 open decisions, SEC 29->28 (SEC-031 CLOSED), next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (no new numbers assigned).
  • F: built netbox/dc-util-hosts-import.py (+harness 19/19) closing the D-134 utility-HOST IP gap; landed 8 ip-addresses into office1-netbox (186->194, ids 187-194). .4 artifact + dc1 .5/.7 DEFERRED (S3).
  • dc1 region STANDUP (--profile vr1-dc1-region): topology apply 40/0; IPAM carve-v6 6/6 + reserve 13/13 (check 24/0); DHCP CUTOVER off-admin->on-region, .6 sole server (no dual-serve, false-alarm resolved); power-key install 9/9 (SEC-016). Captures docs/audit/dc1-*-20260807.
  • SEC-031 CLOSED: dc1 edge REBUILT via the proven dc0 procedure (operator "look back at dc0"), dc-egress-check dc1 8/8, .6 egress restored -> now a RUNBOOK (runbooks/dc-edge-rebuild.md) + tool (scripts/opnsense-console-rebuild.py) per operator "log accurately so we don't re-derive". jammy image sync TRIGGERED + downloading at close.
  • OWNED: pgrep -c dhcpd self-match false-alarm (instrument #21); recalled DHCP range .100-.200, MEASURED .201-.254; shipped opnsense-console-rebuild.py WITHOUT a harness (F1 owed); reserve skipped=6 glance owed; run-logged NOT opened (F2, 4th+ consecutive).
  • Gates: repo-lint 0 fail/1 legacy warn; gauntlet ALL GREEN (102); ledger-scan reconciled. Ledger rotated (2026-08-05 x2 -> archive/session-ledger-rotated-20260807.md).
  • NEXT: verify jammy sync completes; D (import ssh keys into vr1-dc1-region) -> E (rebuild 9 nodes+juju+.7 FRESH, destructive batches of 3, SEC-028 in-region juju mint + SEC-026 residency) -> G (dc1 .6 creds SEC-020); the 2 NetBox findings (S3); the console-driver harness (F1). Sweep: docs/audit/queued-findings-20260807-dc1-region-sequence.txt (F1-F3 FIRST SURFACE). Body: docs/changelog-20260807-dc1-region-sequence.md. Status ONLY in CURRENT-STATE.md.