Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 8 hours ago 17 KB D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

As of the 2026-07-13 scan (the collapse). Re-run the scan to refresh.

  • PROPOSED / OPEN decisions: D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on the pre-DC-DC controller HA/backup session).
  • OPEN security rows: SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007. (The prior machine block listed only SEC-001/003/004 -- it was STALE by three rows.)
  • Next-free numbers: D = 114, DOCFIX = 195, BUNDLEFIX = 013. (The prior machine block said D=111 / DOCFIX=162 / BUNDLEFIX=012 -- STALE.)
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- ledger-scan reads prose and a decoy token inflates the next-free counter. This has bitten twice.

Live state (measured 2026-07-13; nothing is mid-flight)

  • VR1 / Office1 edge office1-opnsense is UP and healthy. LAN 10.10.0.1/24, WAN 172.30.1.2 (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial console + getty alive, and kea-dhcp4 serving DHCP on udp/67 (subnet 10.10.0.0/24, pool .100-.199). DHCP is API-MANAGED (D-113(a2)).
  • OpenTofu repo/state are IN SYNC -- tofu plan reports "No changes". tofu v1.12.3; the tree validates (10/10 modules); opentofu/.terraform.lock.hcl is committed.
  • VR0 / DC0 testcloud: fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts).
  • Branch dc-dc-stage1-phase0-first-exec, pushed. VR1 Stage 1 COMPLETE; Stage 2 partially done.

OPEN WORK -- VR1 DC-DC (the current mission)

The authoritative per-stage tracker is docs/dc-dc-deployment-workflow.md (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry.

  1. Stage 2 -- IN FLIGHT under D-114 (the three-service-VM model is SUPERSEDED). voffice1 (the Office1 site containment VM) is BUILT AND LIVE: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, 10.10.0.100 by DHCP, egress through the edge, /dev/kvm + svm present (D-114's L3 nesting probe PASSES). The Option A/B fork is RESOLVED -- OpenTofu owns it; base-image + cloudinit-vm are instantiated for real. REMAINING on voffice1, in order: (a) install MAAS-region; (b) install LXD pinned to 5.21 LTS (MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7); (c) register that LXD to MAAS as an LXD KVM host; (d) COMPOSE NetBox / GitBucket / Tailscale into it as MAAS-managed VMs. The first composed VM is also the DEFINITIVE L3 proof -- until one boots, nested KVM is proven AVAILABLE, not proven WORKING. HARD CONSTRAINT for step (b): LXD's lxdbr0 runs its own dnsmasq DHCP. Kea on the edge is authoritative on office1-local. lxdbr0 must NEVER be bridged onto that LAN -- two DHCP servers on one L2 is an intermittent failure that is genuinely unpleasant to diagnose. 1b. BLOCKS STAGE 3 (logged, not actioned -- DC1 is gated): modules/node-vm has the SAME missing-cpu-block defect cloudinit-vm just had. DC nodes run nova-compute, which needs working KVM. Without host-passthrough + svm they will come up unable to run one instance. Fix before Stage 3.
  2. NetBox sandbox loop is undocumented. Operator-clarified architecture: draft-source netbox.baldurkeep.com (READ ONLY) -> stand up a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK to the real NetBox. The sim NEVER --commits upstream. docs/dc-dc-netbox-buildout-scope.md and the Stage 2 runbook still need a note capturing this loop. PENDING.
  3. pynetbox / python3-venv are not installed on the vcloud host -- needed for the NetBox import against the Office1 sandbox; needs operator sudo.
  4. The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13. scripts/prereqs/install-apparmor-libvirt.sh (+ install-all / check-prereqs wiring; harness 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants <pool-parent>/** rwk, in /etc/apparmor.d/local/abstractions/libvirt-qemu, which GATES EVERY VR1 VM BOOT: without it a domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce libvirtd unless it actually adds the rule). Pool parent defaults to /var/lib/libvirt/vr1, override with VR1_POOL_PARENT. docs/changelog-20260713-apparmor-libvirt-prereq.md.
  5. DC2 planes are DEFERRED (operator ruling, Option B): do not invent CIDRs -- wait for NetBox to assign the D-101 supernet/ULA/GUA. DC2 mesh legs and dc2_storage are already wired. Values pending final import: ORG_ULA_48 fd50:840e:74e2::/48, GUA dc1 f02::/48 / dc2 f03::/48, DC2_V4_SUPERNET 10.13.0.0/19.
  6. G5 NetBox site rename (vr1-dc0/vr1-dc1 -> vr1-dc1/vr1-dc2) needs its own D-number. Also open in NetBox: G2 (aggregates empty -- top blocks are container prefixes), G3 (ULA /48 not assigned).
  7. DOCFIX-185 follow-up (operator to rule): a D-100/D-107 amendment note so the Stage-3 DC edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark fiber is East-West/replication ONLY, never an internet path).
  8. docs/dc-dc-deployment-workflow.md's status table was corrected 2026-07-13, but re-read it against reality before trusting any row -- it has drifted twice.
  9. DHCP PROVEN END TO END -- CLOSED 2026-07-13. voffice1 (the first client ever to sit on office1-local) took a real lease from Kea: 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state active -- read back through the D-113(a2) REST API. The service, not just the daemon, now works.
  10. Doc-accuracy finding (measured 2026-07-13, logged not actioned): docs/security-ledger.md SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local libvirt bridge, virbr2, not routable off-host): ssh -L 8443:10.10.0.1:443 <user>@10.17.11.248 -> https://localhost:8443.

OPEN WORK -- VR0 / DC0 (the running testcloud)

  1. d011 batch-3 live-validation queue -- scripts/checks/d011-04-octavia-lb.sh and d011-05-magnum-e2e.sh are DRAFTED and mock-tested; the live PASS was recorded in the 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: d011-04 amphora_headroom() field parsing (safe default HOLDs on a parse miss -- confirm the OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set VR_FOIL_APPCRED. Sequence: non-disruptive half first, then --include-disruptive.
  2. Script backlog item 3 (remainder): fold scripts/vault-kv-health.sh into cloud-assert.sh as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure runbook.
  3. Script backlog item 5: scripts/tenant-cluster-create.sh -- generalize the stage-6 + watch + D-066-evidence wrapper. NOT STARTED (verified absent). (Items 6/7/8 -- keystone-policy-drift.sh, cloud-snapshot.sh, tests/tenant-acceptance/ -- are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.)
  4. S-class script quality (docs/script-quality-findings-20260707.md): S4/S7/S8a/S8b DONE; S6 no-action; S8c ruled WON'T-DO. S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ offboard) are HELD by the operator until the DOCFIX-126/127 live-verify lands.
  5. DOCFIX-126/127 live-verify is still OPEN -- the offboard-E0 and onboard-cred-file fixes have never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage.
  6. test1 orphan user -- approved for deletion, staged, NOT run. Role-less shell in the devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): openstack user delete 9a62f88e331f41adb5c9e41225ec8698.
  7. The handoff doc is stale: docs/handoff-20260703-open-items.md still lists R-3 (D-063) as PROPOSED/OPEN. D-063 is RESOLVED / CLOSED in design-decisions.md (verified). Correct the handoff doc.
  8. Upstream reports queued (operator): magnum can-upgrade-to anomaly -- Charmhub magnum 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN -- operator judged it site misconfig; the draft is retained.)
  9. Barbican host_href: the charm renders a literal https://None:9312, suppressing the upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A charm-level fix would need its own D-NNN. Logged, not actioned.
  10. v1-close checklist is QUEUED (handoff section 6) -- gated on D-011 item 6, which is gated on the SEC-003 unseal rehearsal.

Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions)

  • D-071 ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning session. Backups now exist (a 902MB controller backup was taken; juju create-backup / download-backup DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). Controller HA (single-controller, no-HA) is the unresolved half.
  • D-068 off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714.
  • SEC-001 / SEC-004 / SEC-005 / SEC-007: rotate / flip-to-private / revoke at v1 close.
  • SEC-003: custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. This keeps d011-06-vault-unseal MANUAL and gates the FULL close of D-011.
  • SEC-006 (burned NetBox API token) -- DEFERRED by operator ruling 2026-07-13 to deployment completion. Until then the token is LIVE and EXPOSED.
  • D-076 is NOT this repo's to work. D-076..D-099 are a RESERVED source-project band, never assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are running their own tests.
  • quota-vs-capacity: CLOSED -- operator ACCEPTS as-documented (no quota trim).

Standing lessons and traps (load-bearing; these live nowhere else)

  1. THE EDGE TRAP -- DHCP on Office1 is API-MANAGED. A full rendered config.xml push WOULD CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all gone -- opentofu/templates/README.md is a tombstone). Configure the edge via the REST API (scripts/opnsense-api.sh); mint keys with scripts/opnsense-bootstrap-apikey.sh.
  2. "updated in-place" DOES NOT mean "no restart". A tofu apply whose plan says libvirt_domain.vm will be updated in-place RESTARTED the guest (measured: uptime 8:36 -> 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to apply a disk-list change and BOUNCES it. Assume ANY apply touching a libvirt_domain's devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.
  3. Do not re-implement vendor internals; call the interface. OPNsense stores API secrets as crypt(secret,'$6$') (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate).
  4. A config ISO can NEVER be read on an OPNsense nano image (D-112). Any instruction to build one is a trap; the dc-dc-phase2 runbook directed people into it for days.
  5. root's shell on the OPNsense edge is tcsh, not sh. A $(...) inside a quoted remote command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. Always feed remote commands to sh -s. (The scripts/opnsense-apply-config.sh that was once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.)
  6. Guide changes must sweep the tenant AI skill in the SAME change. Skill-lag was caught twice (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 is pinned, and any guide edit FAILS lint until re-recorded).
  7. The controller model is admin/controller, not <user>/controller.
  8. juju status --format=line omits workload messages on 3.6 -- this caused a cloud-assert false negative (fixed by DOCFIX-087). Don't rely on it.
  9. repo-lint's guards are load-bearing; reword rather than weaken. L3 (docs must not reference missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, the guard kept.

State facts to remember

  • The beta cluster and the foil1 tenant are GONE (both confirmed absent by the 2026-07-08 read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is retired here. capi-test-1 and the lbtest LB leftover are also gone; orphan sweep reads ORPHAN=0, zero unattached floating IPs.
  • devteam is a LIVE tenant running a self-created cluster (built via Horizon, calico, keypair devteam-key). Their template has volume_driver unset but the cluster DOES get a default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the box; no template change needed.
  • The vcloud host was NOT pristine: 9 orphaned vr0-* libvirt nets were torn down; wan was KEPT (dual-stack NAT-out-enp1s0, GUA 2602:f3e2:fe:10::/64) as the working model for the per-site ISP-uplink network.
  • MTU has TWO domains: the ISP uplink enp1s0 is 1500; the host-internal plane/dark-fiber fabric is jumbo, underlay_mtu=9000 (operator ruling; verified on all 10 nets).
  • The repo is temporarily PUBLIC (SEC-004) -- flip private at v1 close.
  • Secrets live jumphost-local and are NEVER committed: ~/vr1-office1-creds/ (edge SSH key, root password, API key), ~/.vr1-netbox.env (NetBox token), ~/tenant-devteam/, ~/vault-init/.

Project-completion (execute after D-011 passes)

  • Consolidate the 10 per-phase do-documents into docs/v1-deploy-runbook.md.
  • Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials.
  • v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle.