Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 4 hours ago 30 KB savegame 2026-08-08: dc0 .7 tailscale FIXED (advertise-only) + 10.13 re-IP PIVOT + dc0 checkpoint plan
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

RE-SEEDED 2026-08-05 from bash scripts/ledger-scan.sh (this commit). The prior seed was 2026-08-02; it had gone STALE by one decision (D-142), one SEC row (SEC-033), and the D / DOCFIX next-free (D 141 -> 143, DOCFIX 208 -> 210) -- the per-session summaries below carried the current figures while this block did not, which is the one thing this block exists to prevent. Re-run the scan to refresh.

  • PROPOSED / OPEN decisions (4): D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for rack-only controllers [ARCH] -- sub-4 open + the pinned DNS architectural review), D-132 (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment), D-142 (vault-init workflow QoL sweep -- APPROVED-IN-PRINCIPLE / IMPL-DEFERRED, R2 off-host transport unresolved). D-136/D-137 dropped off the scan when ADOPTED (correct). Status lines in docs/design-decisions.md are the only ruling authority.
  • OPEN security rows: 29 per bash scripts/ledger-scan.sh (was 28 at the last seed). The one added since, verified against the register: SEC-033 2026-08-04 (the tls-certificates relation databag exposes vault's global-client private key to any juju model reader; interface-level, mitigate via juju RBAC). SEC-031 remains the live-exposure one: the dc1 edge lost its FreeBSD user database to fsck and runs with NO pf ruleset, i.e. an open router serving its GUI. The SEC register of record is docs/security-ledger.md; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows.
  • Next-free numbers: D = 143, DOCFIX = 210, BUNDLEFIX = 053. D moved 141 -> 143 as D-141 (IPAM allocations dual-stack, status-distinguished) and D-142 (vault-init QoL, PROPOSED) were assigned. DOCFIX moved 208 -> 210: DOCFIX-208 (phase-4 dc0 machines-overlay omission, 2026-08-02) and DOCFIX-209 (this session's SKILL.md close-bookend correction) are both assigned.
  • Gates: G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in docs/CURRENT-STATE.md is the authority).
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174).
  • ledger-scan D-115 false-positive: RESOLVED. The scan keys on the LAST **Status:** line per ## D-NNN: block, not arbitrary body prose. The general hygiene -- trust the Status line over body text -- still holds.

ROTATED 2026-07-19 (GA-R4 rule 6 -- the one-time rotation; F1 cap now enforceable)

The pre-rotation bodies (~1080 lines: Live state, OPEN WORK VR1/VR0, NetBox write-path bugs, PINNED rulings, standing lessons, state facts, project- completion, NetBox import narrative) moved VERBATIM to docs/archive/session-ledger-rotated-20260719.md. Standing lessons + VR0 state facts were ROUTED to durable homes first (platform-traps already carried most; juju format=line -> appendix-A; guard discipline -> operating-discipline; VR0 cloud facts -> maas-as-built-reference.md). Open work is machine-derived above + CURRENT-STATE.md; do not re-grow bodies here -- sessions append BOUNDED summaries only (15-line cap).

Retained compact blocks (still-live obligations without another home yet):

  • NetBox write-path bugs: ROUTED 2026-07-23 to docs/netbox-write-path-findings.md (the durable home; the rotated-body section is history). Open residue: dumper blind spot + duplicate-CIDR collapse unfixed, hardened fidelity re-run + fix re-verification owed -- ALL gated on the next NetBox WRITE campaign, none current.
  • Project-completion (after D-011 passes): consolidate the 10 per-phase do-documents into a v1 deploy runbook; flip repo PRIVATE (SEC-004); revoke/rotate SEC-005/006/007; v2-deferred: GitBucket SSH, IPv6 dual-stack, NetBox import bundle.
  • Deployment-close consideration review -- tenant-onboarding items (PINNED 2026-07-21, operator-directed): the committee review of Chat's three tenant-onboarding inserts refuted each as written but identified corrected residue; the operator DEFERRED all of it to an end-of-deployment review (posture and end goals are shifting this deployment -- reconsider fit then; nothing lands before that review). Items: (1) appendix-A symptom entry for sub-floor OR nonexistent flavor -> cluster-create rejection (capi-helm thresholds are CONFIG OPTIONS, defaults 2 vCPU / 2048 MB RAM, as-built sets neither; the RAM error also fires for mistyped flavors; command shape --public -f json, never the deprecated --long -- archive R9); (2) appendix-A entry for tenant-net-cannot-reach-public-keystone -> nodes register but stall uninitialized (auth_url in the per-cluster clouds secret is hardcoded PUBLIC at capi-helm 1.4.0; as-built carries no interface key), pointing at the phase-08 D-011.3 probe, plus a per-DC precondition line in dc-dc-phase6 Step 12; (3) tenant contract hardening list: PROPOSED -svc-only password neutralization (admin-rotate to discarded value; keystone lock_password blocks password CHANGE, not auth; NEVER lock -cluster -- it keeps PASSWORD auth permanently per D-066); (4) flavor catalog marking = commercial catalog policy, operator prioritization only. RE-VERIFY every citation at review time -- the full verdict lives in the 2026-07-21 disconnected-session transcript, not in a repo surface.

ROTATED 2026-07-26 (GA-R4 rule 3 / F1 -- oldest-first, cap restored)

The eight oldest closed-session summaries (2026-07-18 through the fifth 2026-07-21 session) moved VERBATIM to docs/archive/session-ledger-rotated-20260726.md. The live ledger was 413 lines against the 300-line cap -- a breach the 2026-07-25 close recorded as OWED. Each rotated summary still points at its own archived full body; only the summaries moved. Sessions from 2026-07-22 onward remain live below.

ROTATED 2026-07-27 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at the Stage 4 close)

The three oldest live summaries (all 2026-07-23: G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 317 lines against the 300-line cap. Sessions from 2026-07-24 onward remain live below.

ROTATED 2026-07-27 (second pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The oldest remaining live summary (2026-07-24, caveman guardrails + DC1 proxy-build start) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 305 lines against the 300-line cap. Sessions from 2026-07-25 onward remain live below.

ROTATED 2026-07-27 (third pass, GA-R4 rule 3 / F1 -- cap restored at the grounding-audit close)

The TWO oldest remaining live summaries -- 2026-07-25 handoff-pack execution + recon + Chat D-136 coupling, and 2026-07-25 MAAS admin-account recovery (SEC-020) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 321 lines against the 300-line cap; one pass left it under. (This paragraph was left MANGLED by an earlier edit -- a sentence broke off mid-number and ran into a stray fragment of the 2026-07-25 MAAS-admin-recovery summary. Found by a 2026-07-29 sweep and repaired here; that summary's full body is at docs/archive/changelogs/changelog-20260725-maas-admin-recovery.md.)

ROTATED 2026-07-27 (fourth pass, GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest remaining live summary ("POST-CLOSE ADDENDUM 2026-07-26 -- D-137 ADOPTED (GA-R4; 07-18/07-21 addendum precedent)") moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. The live ledger was 316 lines against the 300-line cap.

ROTATED 2026-07-29 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- 2026-07-26 (D-137 build + committee audit + remediation) and 2026-07-27 (creds consolidation + the STAGE 4 CLOSE-OUT) -- moved VERBATIM to docs/archive/session-ledger-rotated-20260729.md. The live ledger would otherwise have been 321 lines against the 300-line cap once this session's bookend was appended. Sessions from the 2026-07-27 grounding audit onward remain live below.

ROTATED 2026-07-30 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The TWO oldest live summaries -- the 2026-07-27 STAGE-5 GROUNDING AUDIT close and its 14-RULINGS post-close addendum -- moved VERBATIM to docs/archive/session-ledger-rotated-20260730.md. The live ledger stood at exactly 300 lines, so the next append would have breached the cap. Rotated HERE rather than leaving it owed, because the next session is directed straight at the juju deployment. Sessions from the 2026-07-27 Phase-0 close onward remain live below.

ROTATED 2026-08-02 (b) (GA-R4 rule 3 / F1 -- cap restored at this close)

The TWO oldest live summaries (2026-07-30 part 3 -- Stage 5 opened, three bootstraps, D-138 + D-132 ruled; and part 4 -- dc0 region topology built, cutover blocked on a permission wall) moved VERBATIM to docs/archive/session-ledger-rotated-20260802b.md. The live ledger stood at 294 lines and this close's summary would have breached the cap.

ROTATED 2026-08-03 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest live summary (2026-07-30 part 5 -- dc0 migrated to its own MAAS region, 10/10 Ready, four named gates green) moved VERBATIM to docs/archive/session-ledger-rotated-20260803.md. The live ledger would have breached the 300-line cap with this close's summary.

ROTATED 2026-08-04 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary (2026-07-31 -- dc0 node carve COMPLETE, controller bootstrapped) moved VERBATIM to docs/archive/session-ledger-rotated-20260804.md. The live ledger stood at 292 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-07-31 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary moved VERBATIM to docs/archive/session-ledger-rotated-20260731.md. The live ledger stood at 295 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-08-05 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The two oldest closed-session summaries (2026-07-31 prefer-ipv6/deploy-then-teardown, and 2026-08-01 snap-proxy/D-139/IPv6) moved VERBATIM to docs/archive/session-ledger-rotated-20260805.md. The live ledger stood at 315 lines and the 2026-08-05 (part 2) close breached the 300-line cap. (The 2026-08-01 rotation's own pointer -> docs/archive/session-ledger-rotated-20260801.md is subsumed here; that archive is unchanged.)

ROTATED 2026-08-02 (GA-R4 rule 3 / F1 -- cap restored at this close)

The oldest closed-session summary moved VERBATIM to docs/archive/session-ledger-rotated-20260802.md. The live ledger stood at 283 lines and this close's summary would have breached the 300-line cap.

ROTATED 2026-08-07 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close)

The oldest closed-session summaries (2026-08-03 -- Stage 5 dc0 bundle DEPLOYED, controller rebuilt Path C; AND 2026-08-04 -- ovn-central cert root-caused + wired for redeploy) moved VERBATIM to docs/archive/session-ledger-rotated-20260807.md. Rotated oldest-first across the 2026-08-07 closes to keep the live ledger under the 300-line cap (GA-R4 rule 3).

SESSION CLOSE 2026-08-06 -- Task #2 DONE: D-020 vault-metal-only COMMITTED + enforced across 4 gates; renderer taught (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 2 commits pushed (07e2d9d Task #2, a08783c awk hardening). voffice1 SYNCED to a08783c (ff-only). Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-211 / BUNDLEFIX-054.
  • Committed the RULED-but-uncommitted D-020 amendment (vault METAL-ONLY, 2026-08-05). Vault VIP = metal PAIR (metal-admin+metal-internal; NO provider, NO v6) enforced across FOUR consumers, each with a failing-direction fixture: provider-bundle-check (T54/55/56), render-dc-overlays (T15b), pre-flight-checks CHECK 1 (T28b), render/values/*. DOCFIX-210.
  • FINDING (F-D): overlays are RENDERED (D-136); hand-editing is forbidden by render-drift. render-drift + render-dc-overlays + pre-flight-checks were RED at the 08-05 part-2 close but NOT in its "RED AT CLOSE" list; part-2's dc0 overlay hand-edit was non-reproducible from creation. Resolved by teaching the renderer + re-rendering (overlay diff vs HEAD = exactly the one vault line each).
  • FINDING (F-C): Task #2's own guidance ("edit overlays/...") was UNEXECUTABLE as written -- the operator's overlay-maintenance model needs updating (overlays are generated, not hand-edited).
  • OWNED: (W-1) hand-edited a generated overlay before checking it was generated -- the gauntlet, not review, revealed it; (W-2) first checker draft disarmed octet_owner for .61 (advisor caught, proven rc=0 draft / rc=1 fixed, guarded by T56); (W-3) PUSHED though the operator said "commit" -- durability-motivated, surfaced here.
  • Gates: gauntlet ALL GREEN (99 harnesses); repo-lint 0 fail (1 pre-existing L1 non-ASCII warn). Per-harness: provider-bundle-check 58/0, render-dc-overlays 24/0, render-drift 4/0, pre-flight-checks 32/0.
  • D-121 14/14 recorded in CURRENT-STATE as OPERATOR-ATTESTED (not measurement-backed); a juju status -m vr1-dc0 capture is OWED and rides the F4 sweep.
  • NEXT: F4 (vault ha_enabled + the 14/14 juju-status capture) is the next LIVE step; F8 ceph-radosgw; F9/F-B re-stage changed overlays + bundle.yaml to both racks (sha256-verify); Task #1 post-wave review (incl. whether dc-ha-scaleup.yaml is now redundant; F-A bundle.yaml:23 stale "12 charms" -> 13). Sweep: docs/audit/queued-findings-20260806-task2-vault-metal-only.txt. Body: docs/changelog-20260805-task2-vault-metal-only-commit.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 (part 2) -- F4 measured + dc-ha-scaleup RETIRED (R6 superseded) + memcached 1->3 LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 5 commits, UNPUSHED (push HELD by operator): 7555479 90f15d7 5c2f335 ef47213 667252a. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-056 (used DOCFIX-211/212, BUNDLEFIX-054/055).
  • F4: 14/14 HA now MEASUREMENT-backed (12 active/idle + 2 known-blocked octavia/designate); vault ha_enabled MEASURED FALSE x3 -- charm has no ha_enabled option, HA is VIP-model not vault-native, CONFIRMS D-121 (v-a), Raft stays D-068. F8 ceph-radosgw resolved. F-A/BUNDLEFIX-054 bundle HA-chain header (13 subs; 12 triple + vault metal-pair). F9 rack re-stage (operator "Approved").
  • dc-ha-scaleup.yaml RETIRED + archived; R6 SUPERSEDED (GA-R5, "Retire the redundancy and archive") -- DOCFIX-211. Harness re-pointed off the retired fixture (T17 dropped; T17b/T32/T33/T34 onto the base HA chain via mutate(); provider-bundle-check enumerates per-offender, MEASURED, so single-sub == all-13); runbook two-phase deploy model retired; vips comment fixed via render SOURCE + re-render.
  • memcached VALUE drift caught + resolved: overlay carried memcached=3 (2026-07-31 direction) that BUNDLEFIX-053 never folded (bundle+live=1). Operator ruled "3 units (restore intent)" -> BUNDLEFIX-055 folds it + DOCFIX-212 (D-121). Operator "Both approved" -> scaled LIVE add-unit -n 2 --to lxd:1,lxd:2 to 3/3 active/idle; nova-cc sees all 3 servers, designate coordination sees 1 (Stage-7 re-check, F-1). Rack bundle.yaml re-staged 42845edb.
  • OWNED: told operator "memcached=3 in bundle.yaml" -- WRONG (was 1); retirement first claimed "wholly redundant / all-keys no-op" -- overstated, only a merge-diff (run after a downstream question) showed memcached diverged; earlier F4 draft mis-blamed the mysql backend (advisor-caught). All corrected in-record (instrument-currency memory #17).
  • Gates: gauntlet ALL GREEN (99), repo-lint 0 fail / 1 legacy warn, ledger-scan reconciled (decisions + SEC unchanged; numbers moved as assigned). Ledger rotated (08-02 x2 -> archive/session-ledger-rotated-20260806.md), 299->under-300.
  • Durability: vcloud 0 uncommitted / 5 UNPUSHED (operator hold); voffice1 LAGS until push (Step 1b pull blocked on push); dc0 rack bundle.yaml current (42845edb), vips STALE-by-comment (re-sync at next deploy).
  • NEXT: operator PUSH the 5 commits (then sync voffice1); designate coordination Stage-7 re-check (F-1); pre-existing Stage-6/7 activation blocks (octavia/designate/ceph-rbd-mirror). Sweep: docs/audit/queued-findings-20260806-postwave-retire-memcached.txt (5 FIRST SURFACE, F-1 leads). Body: docs/changelog-20260806-stage5-dc0-f4-postwave.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06 (part 3) -- phase-03 core-verify: F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 9 commits (b8d2b25..edba699 pushed; cd13baa ec8a1d3 ace0e16 UNPUSHED). Scan: 29 SEC, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (used 056/057/058).
  • STEP 7 (phase-03 core-verify, adapted vr1-dc0, run from the dc0 rack): CORE-API VERIFIED (falsifiable settle walk, haproxy sweep, admin-openrc scoped token, IP-only endpoints, two-sourced VIP). F-CV2 RESOLVED: openstack CLI installed on the dc0 rack (D-138; was voffice1-only).
  • F-CV1 (designate _admin backend DOWN) RESOLVED (BUNDLEFIX-056): bundle omitted public+internal -> orphaned VIP triple; fixed + live-rebound. Governing = D-052 + D-020 amdt, NOT D-141.
  • F-CV3 (dashboard VIP plaintext) RESOLVED via D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057: VR1 split-metal INVERTS D-072 (charm serves metal-internal, no admin/internal binding/os-*-network lever) -> cluster->metal-internal. Proven LIVE before ratifying (operator prove-then-rule method).
  • BUNDLEFIX-058: designate-stack conformance (designate amqp+cluster, designate-bind cluster -> metal-internal). Binding conformance now CLEAN cloud-wide (no other active deviations).
  • D-134 Roosevelt-delta annotation (LXD container addrs auto-picked, not carved); gap-21 access-model context (tailnet -> metal-admin dashboards).
  • OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule.
  • DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16.
  • NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-06/07 -- phase-03 Step 3.4 G3 DONE + per-DC Tailscale rulings + .7 VMs stood up both DCs (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; pushed afbaed2..9e74f43 (7 commits); the node-vm harness reconcile is UNCOMMITTED in this bookend. voffice1 synced 9e74f43. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- all D-129/D-134 AMENDMENTS).
  • STEP 3.4 (phase-03) DONE: scripts/g3-domain-manager-probe.sh (+harness 12/12) as the GA-R6 named check; live G3 PASS on dc0 (7 ok/0 fail, teardown verified clean); stage-1 PO: verified per-UNIT (app-aggregate hid it).
  • DECISION C (Horizon): reconciled to VR1 -- both dashboard VIPs serve HTTPS login 200+csrftoken; D-044 + the VR0 nginx-repoint are plain-HTTP-leg artifacts that would only WEAKEN the cookie. phase-03 does NOT close; Step 3.3 SPLITS to its own gate (tailnet-access-gated).
  • PER-DC TAILSCALE (D-129(iii) AMENDMENT, rulings a-d, BOTH DCs): (a) dedicated .7 VM; (b) STAR operator->DC; (c) SINGLE, HA scale-up PINNED; (d) SNAT on. D-134 octet map ->.7; D-107 citation DOCFIX-in-amendment. site-tailscale.sh (+harness 15/15).
  • SUBSTRATE: .7 VMs APPLIED (tofu, gated) + MACs pinned + tofu No-changes on BOTH DCs -- dc0 tailscale; dc1 region VM + tailscale (operator "dc1 full"). Capacity re-gated FIT 874/1024=85%.
  • DEFERRED: Headscale-side join (tagged authkey / autoApprovers / star ACL / Office1-untagged-fix) BLOCKED on control-plane access (operator lacks it); per-VM MAAS commission/deploy/carve/install; dc1 region SETUP workstream.
  • OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint.
  • Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md.
  • NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-07 (dc0 tailscale + dc1 region) -- dc0 .7 carved-and-ready; naming convention; vr1-dc1-region LIVE (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; commits d36d815..9216bdf (+ a final close) all pushed; voffice1 synced. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers). Body: docs/changelog-20260807-dc0-tailscale-provisioning.md (Items 1-10). Sweep: docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt (F1-F4).
  • dc0 .7 DRIVEN TO CARVED-AND-READY (gated): power->commission->carve pass=8/0 (metal-admin 10.12.8.7 + provider-public 10.12.4.7, no br-ex)->deploy jammy, both legs live. C2 correction: it had already self-enlisted (known-marten). Join still blocked (PLAIN key + no Headscale ACL; .7 reachable only via vr1-office1-svc).
  • NEW aux-carve tooling: dc-node-carve.sh is_tailscale/is_region/is_two_plane_host + lib-hosts CARVE_AUX_HOSTS (isolates aux VMs from HOSTS consumers); harness 63/0; gauntlet 101.
  • NAMING convention corrected + RULED standing: renamed dc0 machines to vr1-dc0-*; D-134 AMENDMENT ("Record that as the preferred naming convention going forward") + lib-hosts comment + memory.
  • dc1 RULED "No migration. Build region on DC1 correctly." + "Rebuild fresh into dc1-region"; then BUILT it: .6 (vr1-dc1-maas-01) power->recommission->carve (10.12.68.6/10.12.64.6, --profile admin)->deploy jammy; vr1-dc1-region MAAS LIVE (maas 3.7.2 + postgresql 16.14; the shared vr1-office1-svc key is ON VCLOUD ~/vr1-office1-creds/office1_svc_ed25519; reached the .6 from vcloud; snap egress via snapd proxy; operator-authorised credential one-shot, creds 0600 on the .6, never in context; 10.12.68.6:5240/MAAS/->301). PROFILE registered+verified (voffice1 tunnel -L 5243; rack vr1-dc1-maas-01, 0 machines).
  • OWNED: nearly read "powered off" as "not enlisted" (measured first); advisor caught the unverified provider-public leg + the missing bookend; a false-negative curl -x 400 vs snap-egress-works (tested definitively); run-logged.sh NOT opened (O3, 3rd consecutive).
  • OWED: consolidate the .6 MAAS creds to ~/vr1-dc1-creds/ (SEC-020); RE-OPEN the voffice1:5243 tunnel after any reboot before maas vr1-dc1-region ....
  • NEXT (dc1, --profile vr1-dc1-region --expect-rack vr1-dc1-maas-01, tunnel up): dc-region-topology.sh apply (6-plane) -> maas-region-power-key.sh (.6 snap) + dc-plane-ipam.sh + DHCP + jammy image sync -> import office1_svc pubkey -> REBUILD 9 nodes+juju+.7 fresh in (delete from Office1 admin -> enlist/commission/deploy/carve, name vr1-dc1--NN). Separately dc0 Step 3.3 on a TAGGED key + Headscale.
  • NEXT (NetBox, operator-flagged -- do not miss): record this session's new IPAM objects into office1-netbox (live apex, DOCFIX-195) -- dc0 .7 + dc1 .6 region VM + new vr1-dc1-region + dc1 .7 (when live) + D-134 utility .4-.9 assignments + dc0 renames. Enumerated: changelog-20260807 Item 11. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-07 (part 2) -- dc1 region STANDUP (topology/IPAM/DHCP cutover/power-key) + SEC-031 edge REBUILT + NetBox util-host importer (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 2 commits (b6798df, 2b476e4) pushed at close; voffice1 pulled to sync. Scan: 4 open decisions, SEC 29->28 (SEC-031 CLOSED), next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (no new numbers assigned).
  • F: built netbox/dc-util-hosts-import.py (+harness 19/19) closing the D-134 utility-HOST IP gap; landed 8 ip-addresses into office1-netbox (186->194, ids 187-194). .4 artifact + dc1 .5/.7 DEFERRED (S3).
  • dc1 region STANDUP (--profile vr1-dc1-region): topology apply 40/0; IPAM carve-v6 6/6 + reserve 13/13 (check 24/0); DHCP CUTOVER off-admin->on-region, .6 sole server (no dual-serve, false-alarm resolved); power-key install 9/9 (SEC-016). Captures docs/audit/dc1-*-20260807.
  • SEC-031 CLOSED: dc1 edge REBUILT via the proven dc0 procedure (operator "look back at dc0"), dc-egress-check dc1 8/8, .6 egress restored -> now a RUNBOOK (runbooks/dc-edge-rebuild.md) + tool (scripts/opnsense-console-rebuild.py) per operator "log accurately so we don't re-derive". jammy image sync TRIGGERED + downloading at close.
  • OWNED: pgrep -c dhcpd self-match false-alarm (instrument #21); recalled DHCP range .100-.200, MEASURED .201-.254; shipped opnsense-console-rebuild.py WITHOUT a harness (F1 owed); reserve skipped=6 glance owed; run-logged NOT opened (F2, 4th+ consecutive).
  • Gates: repo-lint 0 fail/1 legacy warn; gauntlet ALL GREEN (102); ledger-scan reconciled. Ledger rotated (2026-08-05 x2 -> archive/session-ledger-rotated-20260807.md).
  • NEXT: verify jammy sync completes; D (import ssh keys into vr1-dc1-region) -> E (rebuild 9 nodes+juju+.7 FRESH, destructive batches of 3, SEC-028 in-region juju mint + SEC-026 residency) -> G (dc1 .6 creds SEC-020); the 2 NetBox findings (S3); the console-driver harness (F1). Sweep: docs/audit/queued-findings-20260807-dc1-region-sequence.txt (F1-F3 FIRST SURFACE). Body: docs/changelog-20260807-dc1-region-sequence.md. Status ONLY in CURRENT-STATE.md.

SESSION CLOSE 2026-08-08 -- dc0 .7 tailscale FIXED (advertise-only) + MAJOR 10.13 re-IP PIVOT + dc0-full-deployment checkpoint plan (bounded, GA-R4)

  • Branch dc-dc-stage5-preconditions; 2 commits (02e0b12, faef662) UNPUSHED (operator push decision pending) + this bookend uncommitted (savegame prepares, does not commit). voffice1 at a62e4b9 (=origin; lags this host by 2 -- pull after push). Scan: 4 open decisions, SEC 28, next-free D-143/DOCFIX-213/BUNDLEFIX-059.
  • PIVOT (operator): 10.12.0.0/16 collides with the still-online IPv4 cloud (surfaced at Headscale). Drive dc0 to FULL deployment as a checkpoint, then TEARDOWN+REDEPLOY on 10.13.0.0/16 (never edit live infra). OWED: a GA-R5 ruling + CURRENT-STATE pivot update. Background agent drafting the 10.13 subnetting (Task #2).
  • dc0 .7 tailscale: first join with --accept-routes BLACKHOLED its own L3 (locked out) -> recovered via qemu-nbd offline-mask -> purged + REBUILT advertise-only (TSIP 100.64.0.57, Running, route UNAPPROVED -- operator STOPPED the tailscale workstream; do NOT approve 10.12.8.0/22). Tool fixed advertise-only + --authkey=file: + check guards (own-subnet, control-reach); harness 27/0, repo-lint 0-fail.
  • dc1 HELD (0 machines in vr1-dc1-region; all nodes powered off). dc0 scope RULED: "activate + smoke-test" (networks + Octavia 1-LB + Designate 1-zone + wrap gates; Magnum DEFERRED). dc0 measured 66 machines/162 units, all core services active; only octavia/designate/ceph-rbd-mirror blocked; tenant-resource-empty.
  • OWNED: a wrong "profile wiped" NEGATIVE rode into a changelog (profiles were in the snap db path; RETRACTED + corrected); pkill self-match killed my own ssh; read a 90s timeout as a failed join that had SUCCEEDED at ~7min. Instrument-currency #22. run-logged NOT opened (Nth).
  • Pinned tasks #1-#4 (dc0 checkpoint; 10.13 NetBox scope; NetBox on vcloud; Chat repo-consolidation).
  • NEXT: operator PUSH 02e0b12+faef662 -> voffice1 pull; re-IP GA-R5 ruling + CURRENT-STATE pivot; MAAS-profile fix on the racks (existing+rebuild); rebuild MAAS nest DC0>regional>rack, NO migration; dc0 activation resolving the D-138 co-location gap. Sweep: docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt (F1-F11). Body: docs/changelog-20260807-dc0-tailscale-install.md. Status ONLY in CURRENT-STATE.md.