Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 5 hours ago 30 KB Session bookend 2026-08-07 (dc0 tailscale .7 carved-and-ready): GA-R4 + sweep
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

RE-SEEDED 2026-08-05 from bash scripts/ledger-scan.sh (this commit). The prior seed was 2026-08-02; it had gone STALE by one decision (D-142), one SEC row (SEC-033), and the D / DOCFIX next-free (D 141 -> 143, DOCFIX 208 -> 210) -- the per-session summaries below carried the current figures while this block did not, which is the one thing this block exists to prevent. Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (4): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment), D-142 (vault-init workflow QoL sweep -- APPROVED-IN-PRINCIPLE / IMPL-DEFERRED, R2 off-host transport unresolved). D-136/D-137 dropped off the scan when ADOPTED (correct). Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 29 per bash scripts/ledger-scan.sh (was 28 at the last seed). The one added since, verified against the register: SEC-033 2026-08-04 (the tls-certificates relation databag exposes vault's global-client private key to any juju model reader; interface-level, mitigate via juju RBAC). SEC-031 remains the live-exposure one: the dc1 edge lost its FreeBSD user database to fsck and runs with NO pf ruleset, i.e. an open router serving its GUI. The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 143, DOCFIX = 210, BUNDLEFIX = 053. D moved 141 -> 143 as D-141 (IPAM allocations dual-stack, status-distinguished) and D-142 (vault-init QoL, PROPOSED) were assigned. DOCFIX moved 208 -> 210: DOCFIX-208 (phase-4 dc0 machines-overlay omission, 2026-08-02) and DOCFIX-209 (this session's SKILL.md close-bookend correction) are both assigned.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

ROTATED 2026-07-27 (second pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The oldest remaining live summary (2026-07-24, caveman guardrails + DC1 proxy-build start) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 305 lines against the 300-line cap. Sessions from 2026-07-25 onward remain live below.

ROTATED 2026-07-27 (third pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The TWO oldest remaining live summaries -- 2026-07-25 handoff-pack execution + recon + Chat D-136 coupling, and 2026-07-25 MAAS admin-account recovery (SEC-020) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 321 lines against the 300-line cap; one pass left it under. (This paragraph was left MANGLED by an earlier edit -- a sentence broke off mid-number and ran into a stray fragment of the 2026-07-25 MAAS-admin-recovery summary. Found by a 2026-07-29 sweep and repaired here; that summary's full body is at docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.)

ROTATED 2026-07-27 (fourth pass, GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest remaining live summary ("POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)") moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 316 lines against the 300-line cap.

ROTATED 2026-07-29 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- 2026-07-26 (D-137 build + committee audit + remediation) and 2026-07-27 (creds consolidation + the STAGE 4 CLOSE-OUT) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260729.md. The live ledger would otherwise have been 321 lines against the 300-line cap once this session's bookend was appended. Sessions from the 2026-07-27 grounding audit onward remain live below.

ROTATED 2026-07-30 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- the 2026-07-27 STAGE-5 GROUNDING AUDIT close and its 14-RULINGS post-close addendum -- moved VERBATIM to docs/archive/session-ledger-rotated-20260730.md. The live ledger stood at exactly 300 lines, so the next append would have breached the cap. Rotated HERE rather than leaving it owed, because the next session is directed straight at the juju deployment. Sessions from the 2026-07-27 Phase-0 close onward remain live below.

ROTATED 2026-08-02 (b) (GA-R4 rule 3 / F1 -- cap restored at this close)

The TWO oldest live summaries (2026-07-30 part 3 -- Stage 5 opened, three bootstraps, D-138 + D-132 ruled; and part 4 -- dc0 region topology built, cutover blocked on a permission wall) moved VERBATIM to docs/archive/session-ledger-rotated-20260802b.md. The live ledger stood at 294 lines and this close's summary would have breached the cap.

ROTATED 2026-08-03 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest live summary (2026-07-30 part 5 -- dc0 migrated to its own MAAS region, 10/10 Ready, four named gates green) moved VERBATIM to docs/archive/session-ledger-rotated-20260803.md. The live ledger would have breached the 300-line cap with this close's summary.

ROTATED 2026-08-04 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary (2026-07-31 -- dc0 node carve COMPLETE, controller bootstrapped) moved VERBATIM to docs/archive/session-ledger-rotated-20260804.md. The live ledger stood at 292 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-07-31 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary moved VERBATIM to docs/archive/session-ledger-rotated-20260731.md. The live ledger stood at 295 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-08-05 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The two oldest closed-session summaries (2026-07-31 prefer-ipv6/deploy-then-teardown, and 2026-08-01 snap-proxy/D-139/IPv6) moved VERBATIM to docs/archive/session-ledger-rotated-20260805.md. The live ledger stood at 315 lines and the 2026-08-05 (part 2) close breached the 300-line cap. (The 2026-08-01 rotation's own pointer -> docs/archive/session-ledger-rotated-20260801.md is subsumed here; that archive is unchanged.)

ROTATED 2026-08-02 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary moved VERBATIM to docs/archive/session-ledger-rotated-20260802.md. The live ledger stood at 283 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-08-07 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The oldest closed-session summaries (2026-08-03 -- Stage 5 dc0 bundle DEPLOYED, controller rebuilt Path C; AND 2026-08-04 -- ovn-central cert root-caused + wired for redeploy) moved VERBATIM to docs/archive/session-ledger-rotated-20260807.md. Rotated oldest-first across the 2026-08-07 closes to keep the live ledger under the 300-line cap (GA-R4 rule 3).

SESSION CLOSE 2026-08-05 -- vault init DONE + ovn-central RESOLVED; D-142 vault-init QoL saved (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions, pushed d6975fb..ca3c0d3. NO stage opened/closed; Stage 5 remains OPEN. Scan: 4 PROPOSED/OPEN decisions (D-142 added), SEC 29, next-free D-143 / DOCFIX-209 / BUNDLEFIX-053.
  • >>> VAULT INIT COMPLETE (operator-run one-shot) + ovn-central RESOLVED end-to-end. <<< Fresh phase-02-vault-preflight vr1-dc0 -> PROCEED; operator ran phase-02-vault-bringup 2.1-2.3 VERBATIM on the dc0 rack (-m vr1-dc0): init (5 shares + TOKEN_OK, 2>&1|tee), unseal 3/3, authorize + generate-root-ca -> vault active/idle. Root CA (public): valid 2026-08-05 02:05:57 -> 2036-08-02 01:06:27 UTC.
  • ovn-central/3,4,5 ALL active (/4 leader ovnnb_db+ovnsb_db, /5 northd) -- OVN NB/SB cluster FORMED. Closes the multi-session saga via BOTH cycle fixes: dc-node-etchosts Step 1.2b (CN delivery) + D-052 re-amendment ''->metal-admin (reachability). Census 40->47 active, 0 error.
  • D-142 PROPOSED -- vault-init workflow QoL sweep SAVED (approved-in-principle, IMPL DEFERRED per operator "run current commands now, test QoL next opportunity"; R2 off-host transport UNRESOLVED). Full R1-R5 + hidden-prompt safety analysis + tee-write residual + pre-init writability probe + R3 harness constraints: docs/audit/vault-init-qol-proposal-20260805.md. Fold-register F13 (D-138 -m openstack->-m vr1-dc0, scope-stretch stated). Distinct from D-068 (substrate) / D-011.6 (unseal bar).
  • Hung-apps directive (fulfilled): the "hung" set was mostly correctly-waiting (ovn certs, designate/octavia/rbd-mirror). GENUINELY stuck: ceph-mon/2 + ceph-radosgw/0 allocating (apt-wedge class) -> cascade to ceph-osd/glance/nova-compute. Logged, not fixed (hard rule 1).
  • Security hygiene: child token printed in the operator's paste (init keys + root token WERE redacted) was ttl=10m/unset/expired -> benign, not stored/echoed/committed. Lesson: redact token-create output too.
  • Durability: vcloud 0 uncommitted / 0 unpushed at close; repo-lint 0 fail / 1 legacy warn; voffice1 PULLED to sync (was at 3321c57, 4 behind -> 8305ceb); dc0 rack ~/repo-stage UNAFFECTED (docs-only session, no staged script touched; phase-02-vault-preflight.sh sha 90910dfb verified == repo today). Gauntlet not owed (docs-only, no script/harness change). Root CA + vault init recorded no-secrets (root CA openssl-verified on the rack) in docs/audit/stage5-dc0-redeploy-asexec-20260804.txt.
  • NEXT: triage the ceph-mon/2 + ceph-radosgw/0 apt-wedge (appendix-A) -> ceph/storage-backend convergence -> continue Stage 5 (phase-03 core verify). Sweep: docs/audit/queued-findings-20260805-vault-init-close.txt (3 items). Body: docs/changelog-20260805-vault-init-ovn-resolved.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-05 (part 2) -- D-121 HA scale-up 14/14 LIVE + vault METAL-ONLY (D-020 amdt); harness + amendment UNCOMMITTED (Task #2) (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 5 commits pushed today (33f9158 71c5b97 c1111a3 ae06c72 4392118). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-210 / BUNDLEFIX-054.
  • CEPH: ceph-mon/2 + ceph-radosgw/0 "allocating" ROOT-CAUSED to apt-cacher-ng CLOSE-WAIT (NOT proxy-down; proxies PASS), remediated (remove-unit -> remove-machine --force -> re-add); appendix-A entry added. Body: changelog-20260805-stage5-dc0-ceph-remediation.md.
  • D-121 EXECUTED LIVE -- all 14 control-plane apps scaled 1->3 HA (8 Wave-1 + placement pilot + nova-cc + rabbitmq + barbican + keystone + vault). barbican-vault RESOLVED. Findings: apt-concurrency forces 1-app/2-container batches; PROACTIVE cluster_count=3 avoids the keystone VIP auth-blip (F5).
  • BUNDLE: BUNDLEFIX-053 -- num_units/cluster_count 1->3, vault-hacluster + vault:ha, rabbitmq min-cluster-size; every change charm-doc-verified; provider-bundle-check PASS. Body: changelog-20260805-d121-ha-scaleup.md.
  • VAULT: 3-unit HA on MySQL backend (operator unsealed vault/1,2). D-020 AMENDMENT ratified 2026-08-05 -> vault METAL-ONLY (PKI/CA root not tenant-facing; provider VIP unhostable -- no provider leg, findif). Live metal-only DONE, vault-hacluster UNBLOCKED.
  • SKILL: DOCFIX-209 (session-close bookend text). Body: changelog-20260805-skill-close-convention-docfix.md.
  • OWNED: shipped BUNDLEFIX-053 (ae06c72) WITHOUT running its harness -> harness now RED (Task #2, F1); twice mis-framed vault init-state before measuring the settling mysql-router; nearly concluded MySQL backend un-shared during a relation-settle.
  • RED AT CLOSE (both = Task #2): repo-lint L10 (held amendment) + provider-bundle-check harness 6/? FAIL. Amendment + dc0 overlay + check-exception are UNCOMMITTED (on disk).
  • DURABILITY: vcloud 0 unpushed; 3 files UNCOMMITTED (Task #2, held); voffice1 PULLED to 4392118; dc0 rack ~/repo-stage/bundle.yaml may be STALE post-BUNDLEFIX-053 (F9).
  • NEXT: Task #2 (harness reconcile + amendment/overlay/check commit + CURRENT-STATE 14/14); Task #1 (post-wave bundle/overlay review); verify vault ha_enabled/(v-a) (F4) + ceph-radosgw radosgw-service (F8). Sweep: docs/audit/queued-findings-20260805-d121-ha-vault.txt. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 -- Task #2 DONE: D-020 vault-metal-only COMMITTED + enforced across 4 gates; renderer taught (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 2 commits pushed (07e2d9d Task #2, a08783c awk hardening). voffice1 SYNCED to a08783c (ff-only). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-211 / BUNDLEFIX-054.
  • Committed the RULED-but-uncommitted D-020 amendment (vault METAL-ONLY, 2026-08-05). Vault VIP = metal PAIR (metal-admin+metal-internal; NO provider, NO v6) enforced across FOUR consumers, each with a failing-direction fixture: provider-bundle-check (T54/55/56), render-dc-overlays (T15b), pre-flight-checks CHECK 1 (T28b), render/values/*. DOCFIX-210.
  • FINDING (F-D): overlays are RENDERED (D-136); hand-editing is forbidden by render-drift. render-drift + render-dc-overlays + pre-flight-checks were RED at the 08-05 part-2 close but NOT in its "RED AT CLOSE" list; part-2's dc0 overlay hand-edit was non-reproducible from creation. Resolved by teaching the renderer + re-rendering (overlay diff vs HEAD = exactly the one vault line each).
  • FINDING (F-C): Task #2's own guidance ("edit overlays/...") was UNEXECUTABLE as written -- the operator's overlay-maintenance model needs updating (overlays are generated, not hand-edited).
  • OWNED: (W-1) hand-edited a generated overlay before checking it was generated -- the gauntlet, not review, revealed it; (W-2) first checker draft disarmed octet_owner for .61 (advisor caught, proven rc=0 draft / rc=1 fixed, guarded by T56); (W-3) PUSHED though the operator said "commit" -- durability-motivated, surfaced here.
  • Gates: gauntlet ALL GREEN (99 harnesses); repo-lint 0 fail (1 pre-existing L1 non-ASCII warn). Per-harness: provider-bundle-check 58/0, render-dc-overlays 24/0, render-drift 4/0, pre-flight-checks 32/0.
  • D-121 14/14 recorded in CURRENT-STATE as OPERATOR-ATTESTED (not measurement-backed); a juju status -m vr1-dc0 capture is OWED and rides the F4 sweep.
  • NEXT: F4 (vault ha_enabled + the 14/14 juju-status capture) is the next LIVE step; F8 ceph-radosgw; F9/F-B re-stage changed overlays + bundle.yaml to both racks (sha256-verify); Task #1 post-wave review (incl. whether dc-ha-scaleup.yaml is now redundant; F-A bundle.yaml:23 stale "12 charms" -> 13). Sweep: docs/audit/queued-findings-20260806-task2-vault-metal-only.txt. Body: docs/changelog-20260805-task2-vault-metal-only-commit.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 (part 2) -- F4 measured + dc-ha-scaleup RETIRED (R6 superseded) + memcached 1->3 LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 5 commits, UNPUSHED (push HELD by operator): 7555479 90f15d7 5c2f335 ef47213 667252a. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-056 (used DOCFIX-211/212, BUNDLEFIX-054/055).
  • F4: 14/14 HA now MEASUREMENT-backed (12 active/idle + 2 known-blocked octavia/designate); vault ha_enabled MEASURED FALSE x3 -- charm has no ha_enabled option, HA is VIP-model not vault-native, CONFIRMS D-121 (v-a), Raft stays D-068. F8 ceph-radosgw resolved. F-A/BUNDLEFIX-054 bundle HA-chain header (13 subs; 12 triple + vault metal-pair). F9 rack re-stage (operator "Approved").
  • dc-ha-scaleup.yaml RETIRED + archived; R6 SUPERSEDED (GA-R5, "Retire the redundancy and archive") -- DOCFIX-211. Harness re-pointed off the retired fixture (T17 dropped; T17b/T32/T33/T34 onto the base HA chain via mutate(); provider-bundle-check enumerates per-offender, MEASURED, so single-sub == all-13); runbook two-phase deploy model retired; vips comment fixed via render SOURCE + re-render.
  • memcached VALUE drift caught + resolved: overlay carried memcached=3 (2026-07-31 direction) that BUNDLEFIX-053 never folded (bundle+live=1). Operator ruled "3 units (restore intent)" -> BUNDLEFIX-055 folds it + DOCFIX-212 (D-121). Operator "Both approved" -> scaled LIVE add-unit -n 2 --to lxd:1,lxd:2 to 3/3 active/idle; nova-cc sees all 3 servers, designate coordination sees 1 (Stage-7 re-check, F-1). Rack bundle.yaml re-staged 42845edb.
  • OWNED: told operator "memcached=3 in bundle.yaml" -- WRONG (was 1); retirement first claimed "wholly redundant / all-keys no-op" -- overstated, only a merge-diff (run after a downstream question) showed memcached diverged; earlier F4 draft mis-blamed the mysql backend (advisor-caught). All corrected in-record (instrument-currency memory #17).
  • Gates: gauntlet ALL GREEN (99), repo-lint 0 fail / 1 legacy warn, ledger-scan reconciled (decisions + SEC unchanged; numbers moved as assigned). Ledger rotated (08-02 x2 -> archive/session-ledger-rotated-20260806.md), 299->under-300.
  • Durability: vcloud 0 uncommitted / 5 UNPUSHED (operator hold); voffice1 LAGS until push (Step 1b pull blocked on push); dc0 rack bundle.yaml current (42845edb), vips STALE-by-comment (re-sync at next deploy).
  • NEXT: operator PUSH the 5 commits (then sync voffice1); designate coordination Stage-7 re-check (F-1); pre-existing Stage-6/7 activation blocks (octavia/designate/ceph-rbd-mirror). Sweep: docs/audit/queued-findings-20260806-postwave-retire-memcached.txt (5 FIRST SURFACE, F-1 leads). Body: docs/changelog-20260806-stage5-dc0-f4-postwave.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 (part 3) -- phase-03 core-verify: F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 9 commits (b8d2b25..edba699 pushed; cd13baa ec8a1d3 ace0e16 UNPUSHED). Scan: 29 SEC, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (used 056/057/058).
  • STEP 7 (phase-03 core-verify, adapted vr1-dc0, run from the dc0 rack): CORE-API VERIFIED (falsifiable settle walk, haproxy sweep, admin-openrc scoped token, IP-only endpoints, two-sourced VIP). F-CV2 RESOLVED: openstack CLI installed on the dc0 rack (D-138; was voffice1-only).
  • F-CV1 (designate _admin backend DOWN) RESOLVED (BUNDLEFIX-056): bundle omitted public+internal -> orphaned VIP triple; fixed + live-rebound. Governing = D-052 + D-020 amdt, NOT D-141.
  • F-CV3 (dashboard VIP plaintext) RESOLVED via D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057: VR1 split-metal INVERTS D-072 (charm serves metal-internal, no admin/internal binding/os-*-network lever) -> cluster->metal-internal. Proven LIVE before ratifying (operator prove-then-rule method).
  • BUNDLEFIX-058: designate-stack conformance (designate amqp+cluster, designate-bind cluster -> metal-internal). Binding conformance now CLEAN cloud-wide (no other active deviations).
  • D-134 Roosevelt-delta annotation (LXD container addrs auto-picked, not carved); gap-21 access-model context (tailnet -> metal-admin dashboards).
  • OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule.
  • DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16.
  • NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06/07 -- phase-03 Step 3.4 G3 DONE + per-DC Tailscale rulings + .7 VMs stood up both DCs (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; pushed afbaed2..9e74f43 (7 commits); the node-vm harness reconcile is UNCOMMITTED in this bookend. voffice1 synced 9e74f43. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- all D-129/D-134 AMENDMENTS).
  • STEP 3.4 (phase-03) DONE: scripts/g3-domain-manager-probe.sh (+harness 12/12) as the GA-R6 named check; live G3 PASS on dc0 (7 ok/0 fail, teardown verified clean); stage-1 PO: verified per-UNIT (app-aggregate hid it).
  • DECISION C (Horizon): reconciled to VR1 -- both dashboard VIPs serve HTTPS login 200+csrftoken; D-044 + the VR0 nginx-repoint are plain-HTTP-leg artifacts that would only WEAKEN the cookie. phase-03 does NOT close; Step 3.3 SPLITS to its own gate (tailnet-access-gated).
  • PER-DC TAILSCALE (D-129(iii) AMENDMENT, rulings a-d, BOTH DCs): (a) dedicated .7 VM; (b) STAR operator->DC; (c) SINGLE, HA scale-up PINNED; (d) SNAT on. D-134 octet map ->.7; D-107 citation DOCFIX-in-amendment. site-tailscale.sh (+harness 15/15).
  • SUBSTRATE: .7 VMs APPLIED (tofu, gated) + MACs pinned + tofu No-changes on BOTH DCs -- dc0 tailscale; dc1 region VM + tailscale (operator "dc1 full"). Capacity re-gated FIT 874/1024=85%.
  • DEFERRED: Headscale-side join (tagged authkey / autoApprovers / star ACL / Office1-untagged-fix) BLOCKED on control-plane access (operator lacks it); per-VM MAAS commission/deploy/carve/install; dc1 region SETUP workstream.
  • OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint.
  • Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md.
  • NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-07 (dc0 tailscale) -- dc0 Tailscale .7 driven to CARVED-AND-READY + aux-carve tooling (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; pushed d36d815..c8ddfb6 (2 commits) + this bookend. voffice1 synced. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- implements the D-129(iii) amendment).
  • dc0 .7 subnet-router DRIVEN TO CARVED-AND-READY (all gated, read back): power set (10.12.8.2) -> commissioned Ready -> carve pass=8/0 (metal-admin 10.12.8.7 + provider-public 10.12.4.7, no br-ex) -> MAAS-deployed jammy; BOTH legs live (ping 0% from the rack). MEASURED correction: the .7 had ALREADY self-enlisted as known-marten (prior close's "powered off" omitted the enlisted part).
  • NEW aux-carve tooling: dc-node-carve.sh is_tailscale_host/is_two_plane_host + lib-hosts CARVE_AUX_HOSTS (isolates the .7 from every HOSTS consumer); harness 58/0; gauntlet ALL GREEN 101.
  • RULINGS (verbatim): dc1 = "No migration. Build region on DC1 correctly." (dc1 .7 GATED behind building vr1-dc1-region -- MEASURED not registered); the join key on hand is PLAIN (not tag-scoped) -> the Tailscale JOIN stays blocked.
  • TWO JOIN PREREQS remain (off-session): a TAGGED pre-auth key + Headscale autoApprovers/ACL; SSH access via vr1-office1-svc (region injects only that key -- operator holds it).
  • OWNED: nearly read the prior close's "powered off" as "not enlisted" (measured known-marten first); the advisor caught the unverified provider-public leg (then measured both legs live) and the missing bookend; run-logged.sh NOT opened (O3, 2nd consecutive -- a background agent cannot drive its interactive subshell).
  • Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 (docs-only edits since). Sweep: docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt (F1-F4 FIRST SURFACE: no per-DC MAAS-region-build runbook; the vr1-office1-svc inject vs SEC-012/016; phase-3 aux-deploy DOCFIX). Body: docs/changelog-20260807-dc0-tailscale-provisioning.md.
  • NEXT: build vr1-dc1-region (no runbook -- F1) for dc1; and once a TAGGED key + Headscale access -> site-tailscale.sh install on dc0 .7 -> browser login https://10.12.8.58 closes Step 3.3. Status ONLY in CURRENT-STATE.md.