Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 8 hours ago 22 KB D-115 office carve IMPORTED into the Office1 NetBox; DC import blocked on ORG_ULA_48
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

As of the 2026-07-13 scan (the collapse). Re-run the scan to refresh.

  • PROPOSED / OPEN decisions: D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on the pre-DC-DC controller HA/backup session).
  • OPEN security rows: SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007. (The prior machine block listed only SEC-001/003/004 -- it was STALE by three rows.)
  • Next-free numbers: D = 114, DOCFIX = 195, BUNDLEFIX = 013. (The prior machine block said D=111 / DOCFIX=162 / BUNDLEFIX=012 -- STALE.)
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- ledger-scan reads prose and a decoy token inflates the next-free counter. This has bitten twice.

Live state (measured 2026-07-13; nothing is mid-flight)

  • VR1 / Office1 edge office1-opnsense is UP and healthy. LAN 10.10.0.1/24, WAN 172.30.1.2 (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial console + getty alive, and kea-dhcp4 serving DHCP on udp/67 (subnet 10.10.0.0/24, pool .100-.199). DHCP is API-MANAGED (D-113(a2)).
  • OpenTofu repo/state are IN SYNC -- tofu plan reports "No changes". tofu v1.12.3; the tree validates (10/10 modules); opentofu/.terraform.lock.hcl is committed.
  • VR0 / DC0 testcloud: fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts).
  • Branch dc-dc-stage1-phase0-first-exec, pushed. VR1 Stage 1 COMPLETE; Stage 2 partially done.

OPEN WORK -- VR1 DC-DC (the current mission)

The authoritative per-stage tracker is docs/dc-dc-deployment-workflow.md (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry.

  1. Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13). voffice1: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, 10.10.0.20 (Kea reservation, outside the pool), egress through the OPNsense edge. On it: MAAS 3.7.2 region+rack (PostgreSQL 16.14) and LXD 5.21.5, with that LXD registered into MAAS as an LXD VM host. MAAS then composed, PXE-booted and COMMISSIONED office1-netbox (LXD VIRTUAL-MACHINE, leased 10.10.1.100 from MAAS, reached status Ready, 2 cores / 4096 MiB detected). The L3 nesting proof is now DEFINITIVE -- a guest booted, PXE'd and commissioned three levels deep, not merely "/dev/kvm exists". VR0's lxd + tailscale pattern, reproduced. Codified in scripts/site-headend-install.sh (+ harness, 18/18) -- reusable for DC1/DC2. THE DHCP SPLIT (structural, not conventional): 10.10.0.0/24 -> Kea on the edge (MAAS dhcp_on=False); 10.10.1.0/24 (lxdbr0) -> MAAS (dhcp_on=True). Separate L2s; lxdbr0 is NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13). office1-netbox (10.10.1.201) runs NetBox 4.6.4; office1-tailscale (10.10.1.202, tailnet 100.64.0.53) is a subnet router advertising 10.10.0.0/22 on the operator's SELF-HOSTED control plane. The edge routes 10.10.1.0/24 -> 10.10.0.20. Full connection reference: docs/vr1-office1-as-built.md -- read that first for any "how do I reach X" question. GitBucket: OUT OF SCOPE (D-116, operator ruling). No Office1-local GitBucket is built; git.baldurkeep.com stays the git service of record. Office1's composed machines are exactly TWO. REMAINING on Office1: (a) OPERATOR: enable the 10.10.0.0/22 route on the Tailscale control server (headscale routes enable) -- it is advertised but not yet enabled, so the subnet is not reachable from the tailnet yet; (b) import the D-115 carve into the Office1 NetBox; (c) the pinned Office1 LAN IPv6 (item 1a). 1a. PINNED -- NEXT AFTER NETBOX: deploy Office1's LAN IPv6 (operator ruling 2026-07-13, D-115 question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). Add 2602:f3e2:f01:100::1/64 to the OPNsense edge LAN + RA, and let voffice1 take its address. Do it via the D-113(a2) REST API -- NEVER a config.xml push. KNOWN LIMIT, do not forget it: IPv6 DOES NOT EGRESS THE LAB (measured 2026-07-13: v6 gateway reachable, v6 internet 100% loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding, which is most of D-101's family matrix -- but NOT internet GUA reachability, which is unprovable in VR1 until the lab's upstream carries v6. Deploy the LAN leg; DEFER the WAN v6 leg (it would be a path to nowhere and untestable). See D-115's amendment. 1b. 10.10.1.0/24 IS NOW A BLESSED ALLOCATION (D-115 ADOPTED). VR1 Office1 = 10.10.0.0/22 (Office role, 10.10.0.0/16): LAN 10.10.0.0/24 + compose 10.10.1.0/24 + 2 spare. Zero renumber. Also adopted: Edge role 172.30.0.0/16 (legitimises office1-wan's 172.30.1.0/24, which had been squatting in the Corp OOB block), and DC2 moved to 10.12.64.0/19 inside the Cloud /16 (its old 10.13.0.0/19 sat outside every allocated block). STILL TO DO: write the carve into NetBox -- production write is operator-gated; the plan is to import into the Office1 sandbox NetBox first, verify, then feed back. 1c. (superseded note kept for context) the original unblessed-allocation finding: Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant pool=10.20.0.0/16 per D-016 -- the obvious 10.20.x would have COLLIDED with tenant space), but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is one of the VMs this headend composes. 1c. modules/node-vm nested-virt defect: FIXED 2026-07-13 (operator: fixes land as we find them; only DC1 deployment is gated). Unconditional cpu = { mode = "host-passthrough" } -- DC nodes run nova-compute and MUST run KVM guests. Without it they would have come up looking healthy and been unable to launch a single instance.
  2. NetBox sandbox loop is undocumented. Operator-clarified architecture: draft-source netbox.baldurkeep.com (READ ONLY) -> stand up a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK to the real NetBox. The sim NEVER --commits upstream. docs/dc-dc-netbox-buildout-scope.md and the Stage 2 runbook still need a note capturing this loop. PENDING.
  3. pynetbox / python3-venv are not installed on the vcloud host -- needed for the NetBox import against the Office1 sandbox; needs operator sudo.
  4. The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13. scripts/prereqs/install-apparmor-libvirt.sh (+ install-all / check-prereqs wiring; harness 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants <pool-parent>/** rwk, in /etc/apparmor.d/local/abstractions/libvirt-qemu, which GATES EVERY VR1 VM BOOT: without it a domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce libvirtd unless it actually adds the rule). Pool parent defaults to /var/lib/libvirt/vr1, override with VR1_POOL_PARENT. docs/changelog-20260713-apparmor-libvirt-prereq.md.
  5. DC2 planes are DEFERRED (operator ruling, Option B): do not invent CIDRs -- wait for NetBox to assign the D-101 supernet/ULA/GUA. DC2 mesh legs and dc2_storage are already wired. Values pending final import: ORG_ULA_48 fd50:840e:74e2::/48, GUA dc1 f02::/48 / dc2 f03::/48, DC2_V4_SUPERNET 10.13.0.0/19.
  6. G5 NetBox site rename (vr1-dc0/vr1-dc1 -> vr1-dc1/vr1-dc2) needs its own D-number. Also open in NetBox: G2 (aggregates empty -- top blocks are container prefixes), G3 (ULA /48 not assigned).
  7. DOCFIX-185 follow-up (operator to rule): a D-100/D-107 amendment note so the Stage-3 DC edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark fiber is East-West/replication ONLY, never an internet path).
  8. docs/dc-dc-deployment-workflow.md's status table was corrected 2026-07-13, but re-read it against reality before trusting any row -- it has drifted twice.
  9. DHCP PROVEN END TO END -- CLOSED 2026-07-13. voffice1 (the first client ever to sit on office1-local) took a real lease from Kea: 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state active -- read back through the D-113(a2) REST API. The service, not just the daemon, now works.
  10. Doc-accuracy finding (measured 2026-07-13, logged not actioned): docs/security-ledger.md SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local libvirt bridge, virbr2, not routable off-host): ssh -L 8443:10.10.0.1:443 <user>@10.17.11.248 -> https://localhost:8443.

OPEN WORK -- VR0 / DC0 (the running testcloud)

  1. d011 batch-3 live-validation queue -- scripts/checks/d011-04-octavia-lb.sh and d011-05-magnum-e2e.sh are DRAFTED and mock-tested; the live PASS was recorded in the 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: d011-04 amphora_headroom() field parsing (safe default HOLDs on a parse miss -- confirm the OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set VR_FOIL_APPCRED. Sequence: non-disruptive half first, then --include-disruptive.
  2. Script backlog item 3 (remainder): fold scripts/vault-kv-health.sh into cloud-assert.sh as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure runbook.
  3. Script backlog item 5: scripts/tenant-cluster-create.sh -- generalize the stage-6 + watch + D-066-evidence wrapper. NOT STARTED (verified absent). (Items 6/7/8 -- keystone-policy-drift.sh, cloud-snapshot.sh, tests/tenant-acceptance/ -- are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.)
  4. S-class script quality (docs/script-quality-findings-20260707.md): S4/S7/S8a/S8b DONE; S6 no-action; S8c ruled WON'T-DO. S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ offboard) are HELD by the operator until the DOCFIX-126/127 live-verify lands.
  5. DOCFIX-126/127 live-verify is still OPEN -- the offboard-E0 and onboard-cred-file fixes have never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage.
  6. test1 orphan user -- approved for deletion, staged, NOT run. Role-less shell in the devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): openstack user delete 9a62f88e331f41adb5c9e41225ec8698.
  7. The handoff doc is stale: docs/handoff-20260703-open-items.md still lists R-3 (D-063) as PROPOSED/OPEN. D-063 is RESOLVED / CLOSED in design-decisions.md (verified). Correct the handoff doc.
  8. Upstream reports queued (operator): magnum can-upgrade-to anomaly -- Charmhub magnum 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN -- operator judged it site misconfig; the draft is retained.)
  9. Barbican host_href: the charm renders a literal https://None:9312, suppressing the upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A charm-level fix would need its own D-NNN. Logged, not actioned.
  10. v1-close checklist is QUEUED (handoff section 6) -- gated on D-011 item 6, which is gated on the SEC-003 unseal rehearsal.

Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions)

  • D-071 ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning session. Backups now exist (a 902MB controller backup was taken; juju create-backup / download-backup DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). Controller HA (single-controller, no-HA) is the unresolved half.
  • D-068 off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714.
  • SEC-001 / SEC-004 / SEC-005 / SEC-007: rotate / flip-to-private / revoke at v1 close.
  • SEC-003: custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. This keeps d011-06-vault-unseal MANUAL and gates the FULL close of D-011.
  • SEC-006 (burned NetBox API token) -- DEFERRED by operator ruling 2026-07-13 to deployment completion. Until then the token is LIVE and EXPOSED.
  • D-076 is NOT this repo's to work. D-076..D-099 are a RESERVED source-project band, never assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are running their own tests.
  • quota-vs-capacity: CLOSED -- operator ACCEPTS as-documented (no quota trim).

Standing lessons and traps (load-bearing; these live nowhere else)

  1. THE EDGE TRAP -- DHCP on Office1 is API-MANAGED. A full rendered config.xml push WOULD CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all gone -- opentofu/templates/README.md is a tombstone). Configure the edge via the REST API (scripts/opnsense-api.sh); mint keys with scripts/opnsense-bootstrap-apikey.sh.
  2. "updated in-place" DOES NOT mean "no restart". A tofu apply whose plan says libvirt_domain.vm will be updated in-place RESTARTED the guest (measured: uptime 8:36 -> 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to apply a disk-list change and BOUNCES it. Assume ANY apply touching a libvirt_domain's devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.
  3. Do not re-implement vendor internals; call the interface. OPNsense stores API secrets as crypt(secret,'$6$') (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate).
  4. A config ISO can NEVER be read on an OPNsense nano image (D-112). Any instruction to build one is a trap; the dc-dc-phase2 runbook directed people into it for days.
  5. root's shell on the OPNsense edge is tcsh, not sh. A $(...) inside a quoted remote command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. Always feed remote commands to sh -s. (The scripts/opnsense-apply-config.sh that was once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.)
  6. Guide changes must sweep the tenant AI skill in the SAME change. Skill-lag was caught twice (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 is pinned, and any guide edit FAILS lint until re-recorded).
  7. The controller model is admin/controller, not <user>/controller.
  8. juju status --format=line omits workload messages on 3.6 -- this caused a cloud-assert false negative (fixed by DOCFIX-087). Don't rely on it.
  9. repo-lint's guards are load-bearing; reword rather than weaken. L3 (docs must not reference missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, the guard kept.

State facts to remember

  • The beta cluster and the foil1 tenant are GONE (both confirmed absent by the 2026-07-08 read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is retired here. capi-test-1 and the lbtest LB leftover are also gone; orphan sweep reads ORPHAN=0, zero unattached floating IPs.
  • devteam is a LIVE tenant running a self-created cluster (built via Horizon, calico, keypair devteam-key). Their template has volume_driver unset but the cluster DOES get a default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the box; no template change needed.
  • The vcloud host was NOT pristine: 9 orphaned vr0-* libvirt nets were torn down; wan was KEPT (dual-stack NAT-out-enp1s0, GUA 2602:f3e2:fe:10::/64) as the working model for the per-site ISP-uplink network.
  • MTU has TWO domains: the ISP uplink enp1s0 is 1500; the host-internal plane/dark-fiber fabric is jumbo, underlay_mtu=9000 (operator ruling; verified on all 10 nets).
  • The repo is temporarily PUBLIC (SEC-004) -- flip private at v1 close.
  • Secrets live jumphost-local and are NEVER committed: ~/vr1-office1-creds/ (edge SSH key, root password, API key), ~/.vr1-netbox.env (NetBox token), ~/tenant-devteam/, ~/vault-init/.

Project-completion (execute after D-011 passes)

  • Consolidate the 10 per-phase do-documents into docs/v1-deploy-runbook.md.
  • Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials.
  • v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle.

NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13)

Done. The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop (netbox/prod-draft-dump.py READ-ONLY -> netbox/draft/vr1-draft.json -> netbox/sandbox-seed.py), seeded 129 objects from the upstream draft, then applied the D-115 office carve (netbox/d115-office-carve.py): site vr1-off1, roles office + edge (new), and 8 prefixes. Idempotent, verified. See docs/vr1-office1-as-built.md section 6.

D-117 ADOPTED (operator ruling, Option B, DC-only): VR1's DCs are dc0/dc1, matching the apex. The office KEEPS its number (Off1). Fixed the importer's off-by-one site binding and made it dry by default.

BLOCKED: the VR1 DC six-plane import needs ORG_ULA_48 RATIFIED

netbox/dc-dc-prefixes-import.py --dc dc0 cannot run. It requires ORG_ULA_48, and that literal is still unassigned -- it is gap G3 in docs/dc-dc-netbox-buildout-scope.md, which recommends fd50:840e:74e2::/48 (CSPRNG-generated, Global ID 0x50840e74e2) but no ADOPTED decision assigns it (grep ORG_ULA docs/design-decisions.md -> nothing). D-101 explicitly leaves it NetBox-assigned.

Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready.

To unblock: ratify an org ULA /48 under its own D-number. Then:

ORG_ULA_48=<ratified /48>  DC_GUA_PREFIX=2602:f3e2:f02::/48  --dc dc0     # VR1 DC0
ORG_ULA_48=<ratified /48>  DC_GUA_PREFIX=2602:f3e2:f03::/48  \
    DC1_V4_SUPERNET=10.12.64.0/19  --dc dc1                               # VR1 DC1

(DC_GUA_PREFIX values are MEASURED off the apex; DC1_V4_SUPERNET is assigned by D-115. DC2_V4_SUPERNET is RETIRED -- the tool rejects it by name.)

Also open (recorded, not blocking)

  • $DC shell-selector namespace collision (D-117 amendment): lib-net.sh's dc0 means VR0's DC0, while the importer's dc0 now means VR1's. NOT interchangeable. Proposed fix: region- qualify the shell selector to the apex slugs (vr0-dc0/vr1-dc0/vr1-dc1).
  • Feed the D-115 carve upstream to netbox.baldurkeep.com (operator-gated; the tool refuses a non-sandbox target without --yes-write-upstream).
  • Office1 LAN IPv6 (2602:f3e2:f01:100::/64 + RA via the OPNsense REST API) -- still pinned.