# Session ledger -- in-flight work continuity record **Purpose.** Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work. **How to use it (standing practice).** 1. At session start: read this ledger AND run `bash scripts/ledger-scan.sh`. Reconcile. 2. `ledger-scan.sh` is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces. 3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN. 4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand. **SINGLE STREAM (collapsed 2026-07-13).** This ledger previously carried three parallel, separately-owned stream sections (`main-chat`, `jumphost`, `shared`) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections. **Where the history went.** The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 `docs/changelog-*.md` files, `docs/design-decisions.md`, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else. --- ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) _Re-seeded from the 2026-07-14 scan. Re-run `bash scripts/ledger-scan.sh` to refresh._ - **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on the pre-DC-DC controller HA/backup session). - **OPEN security rows:** SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007, **SEC-008**. (SEC-008 -- the vcloud re-enrolment key -- was MISSING from the prior block.) - **Next-free numbers:** D = **119**, DOCFIX = 195, BUNDLEFIX = 013. (The prior block said D=114 -- STALE by five; D-114..D-118 are all assigned and ADOPTED.) - **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- `ledger-scan` reads prose and a decoy token inflates the next-free counter. This has bitten twice. - **KNOWN `ledger-scan` DEFECT (found 2026-07-14, logged not fixed):** the scan reports **D-115 as PROPOSED/OPEN. It is not -- D-115 is ADOPTED.** The scan matches the words "PROPOSED/OPEN" anywhere in an entry's prose, and D-115's Status line reads "ADOPTED ... Originally PROPOSED/OPEN the same day". It should read the `**Status:**` line, not the body. This is the *status* twin of the decoy-token rule above. **Until it is fixed, do not trust the scan's PROPOSED/OPEN list without checking the Status line.** D-115 is the only current false positive. --- ## Live state (measured 2026-07-13; nothing is mid-flight) - **VR1 / Office1 edge `office1-opnsense` is UP and healthy.** LAN 10.10.0.1/24, WAN 172.30.1.2 (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial console + getty alive, and `kea-dhcp4` serving DHCP on udp/67 (subnet 10.10.0.0/24, pool .100-.199). **DHCP is API-MANAGED (D-113(a2)).** - **OpenTofu repo/state back IN SYNC -- D-119 rename APPLIED (2026-07-14, operator-gated).** `tofu apply`: **11 added, 0 changed, 11 destroyed**; `tofu plan` now reports "No changes". Verified on the host: six `vr1-dc0-*` networks, three `mesh-vr1-*` links, `vr1-dc0-pool`/`vr1-dc1-pool`; ZERO old `dc1-*`/`dc2-*`/`mesh-dc*` names remain. Pre-apply LIVE re-check confirmed all 11 objects empty (0 leases, 0 volumes, no domain attached). Office1 (`office1-local`/`office1-wan`/`wan`/edge/ `voffice1`) was NOT in the plan and is untouched -- edge + voffice1 still reachable post-apply. tofu v1.12.3; tree validates (10/10 modules); lock committed. - **VR0 / DC0 testcloud:** fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts). - Branch `dc-dc-stage1-phase0-first-exec`, **pushed and level with origin (2026-07-14)**. VR1 Stage 1 COMPLETE; Stage 2 in CLOSE-OUT (C1..C4 open, C5 done). - **SESSION CONSOLIDATION (2026-07-14).** Work had fragmented across FOUR concurrent Claude sessions on this one clone -- including two long-lived `--permission-mode auto` background agents (idle 14-15h but still holding auto-approve rights against the live cloud), and a second interactive session that received the SAME "resume last night's work" prompt six minutes before this one and began bootstrapping in parallel. All three were terminated and the supervising daemon stopped (it had respawned the agents once). **Standing lesson: the daemon ADOPTS AND RESPAWNS background workers -- killing the child is not enough; stop the daemon (`claude daemon stop --any`).** The working tree was clean and 3-ahead throughout; nothing was lost or clobbered. ONE session is now the rule. - **Last night's three commits (D-113 amendment + the v6 script) were made by the auto-permission background agent and left UNPUSHED and UNDOCUMENTED.** Both closed 2026-07-14: pushed, and `docs/changelog-20260714-opnsense-set-interface-v6.md` written. --- ## OPEN WORK -- VR1 DC-DC (the current mission) The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry. 1. **Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13).** `voffice1`: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, **10.10.0.20** (Kea reservation, outside the pool), egress through the OPNsense edge. On it: **MAAS 3.7.2** region+rack (PostgreSQL 16.14) and **LXD 5.21.5**, with that LXD **registered into MAAS as an LXD VM host**. MAAS then **composed, PXE-booted and COMMISSIONED** `office1-netbox` (LXD VIRTUAL-MACHINE, leased 10.10.1.100 from MAAS, reached status **Ready**, 2 cores / 4096 MiB detected). **The L3 nesting proof is now DEFINITIVE** -- a guest booted, PXE'd and commissioned three levels deep, not merely "`/dev/kvm` exists". VR0's `lxd` + `tailscale` pattern, reproduced. Codified in `scripts/site-headend-install.sh` (+ harness, 18/18) -- reusable for DC1/DC2. **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.201) runs **NetBox 4.6.4**; `office1-tailscale` (10.10.1.202, tailnet 100.64.0.53) is a subnet router advertising `10.10.0.0/22` on the operator's **SELF-HOSTED** control plane. The edge routes `10.10.1.0/24 -> 10.10.0.20`. Full connection reference: **`docs/vr1-office1-as-built.md`** -- read that first for any "how do I reach X" question. **GitBucket: OUT OF SCOPE (D-116, operator ruling).** No Office1-local GitBucket is built; `git.baldurkeep.com` stays the git service of record. Office1's composed machines are exactly TWO. **REMAINING on Office1 (reconciled 2026-07-14):** (a) route enable -- **DONE**, operator enabled `10.10.0.0/22` on the control server; the edge GUI answers at 10.10.0.1 from the workstation with no tunnel; (b) import the D-115 carve into the Office1 NetBox -- **DONE** (sandbox seeded from the upstream draft and PROVEN faithful field-by-field; carve + six-plane roles + BOTH DCs imported, 90 -> 134 prefixes); (c) the pinned Office1 LAN IPv6 (item 1a) -- **STILL OPEN**, and it is now the last build item on Office1. **STAGE 2 IS IN CLOSE-OUT, NOT BUILD. It is also the BRANCH MERGE POINT** (see PINNED). The close-out checklist C1..C5 is in `docs/dc-dc-deployment-workflow.md`; do not re-derive it here. 1a. **DONE 2026-07-14 -- Office1's LAN IPv6 is LIVE and PROVEN ON THE WIRE. C1 is CLOSED.** Address `2602:f3e2:f01:100::1/64` on the edge LAN (kernel-verified) + radvd advertising the prefix with `mode=unmanaged`; `voffice1` autoconfigured `2602:f3e2:f01:100:5054:ff:fe6a:87e5` by SLAAC and edge->voffice1 GUA ping is 0.0% loss. No collateral damage (v4, Kea DHCPv4, compose-net route all intact). See `docs/changelog-20260714-office1-lan-ipv6-executed.md`. **OPERATOR RULING 2026-07-14 -- DO NOT RE-PROPOSE A v6 NAT.** A v4-to-v6 NAT (second OPNsense, or nginx) was proposed to *simulate* the v6 the lab will not forward, and was WITHDRAWN in favour of *"configure IPv6 internally as much as possible; configure the IPv6 edge AFTER deployment completes."* It contradicts **D-101** (*"native GUA routing, no NAT66"*) and **D-115**'s amendment (upstream's problem, not ours), and a translation box is the one component Roosevelt will NOT have -- maximum delta, zero proof value. nginx is an L7 proxy, not NAT, and proves nothing about RA/ICMPv6/Ceph-over-v6. **TRAP recorded:** `voffice1` has `forwarding=1` + `accept_ra=0` yet STILL autoconfigured -- netplan/systemd-networkd handles RA in USERSPACE, independent of the kernel sysctl. Never diagnose "no SLAAC" from the sysctl alone on Ubuntu; read `ip -6 addr` + look for a `proto ra` route. **The WAN v6 leg remains DEFERRED** -- v6 does not egress the lab (re-measured 2026-07-14: v6 internet 100% loss, v4 fine). Internet GUA reachability is UNPROVABLE in VR1. 1a-historical. **(superseded -- the original pinned item)** deploy Office1's LAN IPv6 (operator ruling 2026-07-13, D-115 question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). Add `2602:f3e2:f01:100::1/64` to the OPNsense edge LAN + RA, and let `voffice1` take its address. **CORRECTED 2026-07-14 (D-113 AMENDMENT -- the old instruction here was WRONG):** the ADDRESS half CANNOT be done via the REST API. Measured on the live edge (OPNsense 26.1): `Interfaces > [LAN]` is the LEGACY page `/interfaces.php?if=lan`; only Devices and Neighbors were migrated to MVC, and there is **NO REST endpoint for a base interface's `ipaddrv6`**. D-113(a2)'s "interfaces are API-covered" was INFERRED, never measured, and is false. - **Address:** `scripts/opnsense-set-interface-v6.sh` (dry by default; harness 40/40). It is NOT a config.xml push -- it mutates two leaves of the edge's LIVE config through OPNsense's own model, so API-managed Kea and the firewall rules cannot be clobbered. **WRITTEN AND TESTED, NEVER RUN WITH `--commit`.** - **RA:** that half IS API-native -- `/api/radvd/settings/addEntry` via `scripts/opnsense-api.sh`. Still forbidden either way: a rendered config.xml push. **KNOWN LIMIT, do not forget it:** IPv6 DOES NOT EGRESS THE LAB (measured 2026-07-13: v6 gateway reachable, v6 internet 100% loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding, which is most of D-101's family matrix -- but NOT internet GUA reachability, which is unprovable in VR1 until the lab's upstream carries v6. Deploy the LAN leg; DEFER the WAN v6 leg (it would be a path to nowhere and untestable). See D-115's amendment. 1b. **`10.10.1.0/24` IS NOW A BLESSED ALLOCATION (D-115 ADOPTED).** VR1 Office1 = `10.10.0.0/22` (Office role, `10.10.0.0/16`): LAN `10.10.0.0/24` + compose `10.10.1.0/24` + 2 spare. Zero renumber. Also adopted: Edge role `172.30.0.0/16` (legitimises `office1-wan`'s 172.30.1.0/24, which had been squatting in the Corp OOB block), and **DC2 moved to `10.12.64.0/19`** inside the Cloud /16 (its old `10.13.0.0/19` sat outside every allocated block). STILL TO DO: write the carve into NetBox -- production write is operator-gated; the plan is to import into the Office1 sandbox NetBox first, verify, then feed back. 1c. **(superseded note kept for context)** the original unblessed-allocation finding: Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant pool=10.20.0.0/16 per D-016 -- the obvious `10.20.x` would have COLLIDED with tenant space), but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is one of the VMs this headend composes. 1c. **`modules/node-vm` nested-virt defect: FIXED 2026-07-13** (operator: fixes land as we find them; only DC1 *deployment* is gated). Unconditional `cpu = { mode = "host-passthrough" }` -- DC nodes run `nova-compute` and MUST run KVM guests. Without it they would have come up looking healthy and been unable to launch a single instance. 2. **NetBox sandbox loop -- DOCUMENTED 2026-07-14 (close-out C4 DONE).** scope-doc section 8 + phase1 runbook Step 10b. (Was: BUILT 2026-07-13/14, undocumented.) Architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK. The sim NEVER `--commit`s upstream. **The tooling now exists and is proven:** `netbox/prod-draft-dump.py` (read-only, no write path at all), `netbox/sandbox-seed.py` (dry-by-default, structurally REFUSES an upstream URL), `netbox/sandbox-fidelity-check.py` (field-by-field -- counts alone HID a bug where 17 of 90 prefixes silently lost their region scope). **STILL PENDING:** capture the loop in `docs/dc-dc-netbox-buildout-scope.md` + the Stage 2 runbook, and **feed the validated carve UPSTREAM** (close-out C2 -- the Stage 2 gate says *authoritative*, and the sandbox is not it). The upstream write is operator-gated. 3. **`pynetbox` -- CLOSED 2026-07-13.** Installed (7.0.0) on `office1-netbox`, which is where the imports actually run. The stdlib tools (`prod-draft-dump`/`sandbox-seed`/`sandbox-fidelity-check`) deliberately need no dependency at all. **TRAP, recorded in `platform-traps.md`:** the upstream NetBox sits behind a **User-Agent-filtering WAF**. `Python-urllib/3.12` gets a **403 that looks exactly like an auth failure**; the same token works from curl. This affects pynetbox too. 4. **The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13.** `scripts/prereqs/install-apparmor-libvirt.sh` (+ `install-all` / `check-prereqs` wiring; harness 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants `/** rwk,` in `/etc/apparmor.d/local/abstractions/libvirt-qemu`, which GATES EVERY VR1 VM BOOT: without it a domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce libvirtd unless it actually adds the rule). Pool parent defaults to `/var/lib/libvirt/vr1`, override with `VR1_POOL_PARENT`. `docs/changelog-20260713-apparmor-libvirt-prereq.md`. 5. **The DEFERRED literals are now ALL ASSIGNED -- item CLOSED 2026-07-14.** The operator's Option-B ruling ("do not invent CIDRs, wait for NetBox") held, and NetBox has now assigned them: **ORG_ULA_48 = `fd50:840e:74e2::/48`** (**D-118 ADOPTED** -- RFC 4193 valid, and verified NOT to collide with Tailscale's `fd7a:115c:a1e0::/48`, which matters because `office1-tailscale` runs `--accept-routes`); GUA per D-117: first DC `2602:f3e2:f02::/48`, second `f03::/48`; the second DC's v4 = six sequential `/22`s from `10.12.64.0/19` (D-115 moved it INSIDE the Cloud /16). All are IMPORTED and fidelity-checked in the sandbox. **CAUTION -- `DC2_V4_SUPERNET` IS A RETIRED NAME (D-117).** The importer now hard-rejects it and rejects `--dc dc2`. Any doc still saying `10.13.0.0/19` is pre-D-115 and wrong. 6. **D-119 (ADOPTED 2026-07-14) -- THE DC NAMESPACE IS REGION-QUALIFIED. C3/gap #19 CLOSED.** `vr0-dc0` (VR0's LIVE testcloud) / `vr1-dc0` (VR1's FIRST DC, GUA f02::/48) / `vr1-dc1` (VR1's SECOND, f03::/48). Bare `dcN` is REJECTED everywhere -- it meant VR0's live cloud in `lib-net.sh` but VR1's first DC in the importer: one string, two clouds. D-119 **COMPLETES D-117** (executes its own amendment); it does not reverse it. **ZERO NetBox writes** -- the apex was already right; the REPO was the odd one out. The importer's DC->site map is now an IDENTITY, so the off-by-one is STRUCTURALLY IMPOSSIBLE, not merely defended. Also renamed: `vdc1`/`vdc2` -> `vvr1-dc0`/`vvr1-dc1` (D-114's own DR primitive is `virsh destroy vdc1` -- it read as "destroy VR1 DC1" while MEANING "destroy VR1 DC0"). Office KEEPS its number (`vr1-off1`, `Office1`, `voffice1`). Env var: `DC1_/DC2_V4_SUPERNET` -> **`VR1_DC1_V4_SUPERNET`** (both old names rejected by name). Gauntlet ALL GREEN (57); dc-selector 21->30, prefixes-import 40->82. **THE `tofu apply` IS NOT DONE -- IT IS GATED.** Plan: 11 to add, 0 to change, 11 to destroy (the libvirt object NAMES change, which is ForceNew; `moved{}` makes the plan reviewable but cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY (no guests, no volumes; Stage 3 hasn't run). Re-plan at execution and STOP if anything shows ATTACHED. See `docs/changelog-20260714-d119-region-qualified-dc-namespace.md`. 6a. **(historical) G5 NetBox site rename -- CLOSED as D-117 (ADOPTED, Option B).** The repo moved to the apex's 0-indexed convention (first VR1 DC = `vr1-dc0`), rather than bending the apex to the repo. This fixed a REAL off-by-one: `dc-dc-prefixes-import.py` bound `--dc dc1` to slug `vr1-dc1` while treating `f02::/48` as dc1 -- but the apex binds `f02` to `VR1 DC0`. It would have written the first DC's prefixes onto the second DC's site. The importer is now also **dry-by-default**. **D-117 IS ONLY PARTIALLY EXECUTED** -- see gap #19 / close-out C3: `opentofu/main.tf` and `lib-net.sh`/`lib-hosts.sh` still speak the OLD naming, and `lib-net.sh`'s `dc0` means **VR0's** DC0. That collision BLOCKS Stage 3. G2 (aggregates) and G3 (ULA /48) are both CLOSED by the same work. 7. **DOCFIX-185 follow-up (operator to rule):** a D-100/D-107 amendment note so the Stage-3 DC edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark fiber is East-West/replication ONLY, never an internet path). 8. `docs/dc-dc-deployment-workflow.md`'s status table was corrected 2026-07-13, but re-read it against reality before trusting any row -- it has drifted twice. 9. **DHCP PROVEN END TO END -- CLOSED 2026-07-13.** `voffice1` (the first client ever to sit on `office1-local`) took a real lease from Kea: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches its NIC), hostname `voffice1`, state active -- read back through the D-113(a2) REST API. The service, not just the daemon, now works. 10. **Doc-accuracy finding (measured 2026-07-13, logged not actioned):** `docs/security-ledger.md` SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local libvirt bridge, `virbr2`, not routable off-host): `ssh -L 8443:10.10.0.1:443 @10.17.11.248` -> `https://localhost:8443`. ## OPEN WORK -- VR0 / DC0 (the running testcloud) 1. **d011 batch-3 live-validation queue** -- `scripts/checks/d011-04-octavia-lb.sh` and `d011-05-magnum-e2e.sh` are DRAFTED and mock-tested; the live PASS was recorded in the 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: d011-04 `amphora_headroom()` field parsing (safe default HOLDs on a parse miss -- confirm the OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set `VR_FOIL_APPCRED`. Sequence: non-disruptive half first, then `--include-disruptive`. 2. **Script backlog item 3 (remainder):** fold `scripts/vault-kv-health.sh` into `cloud-assert.sh` as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure runbook. 3. **Script backlog item 5:** `scripts/tenant-cluster-create.sh` -- generalize the stage-6 + watch + D-066-evidence wrapper. NOT STARTED (verified absent). (Items 6/7/8 -- `keystone-policy-drift.sh`, `cloud-snapshot.sh`, `tests/tenant-acceptance/` -- are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.) 4. **S-class script quality (`docs/script-quality-findings-20260707.md`):** S4/S7/S8a/S8b DONE; S6 no-action; S8c ruled WON'T-DO. **S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ offboard) are HELD** by the operator until the DOCFIX-126/127 live-verify lands. 5. **DOCFIX-126/127 live-verify is still OPEN** -- the offboard-E0 and onboard-cred-file fixes have never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage. 6. **`test1` orphan user -- approved for deletion, staged, NOT run.** Role-less shell in the devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): `openstack user delete 9a62f88e331f41adb5c9e41225ec8698`. 7. **The handoff doc is stale:** `docs/handoff-20260703-open-items.md` still lists R-3 (D-063) as PROPOSED/OPEN. D-063 is `RESOLVED / CLOSED` in `design-decisions.md` (verified). Correct the handoff doc. 8. **Upstream reports queued (operator):** magnum `can-upgrade-to` anomaly -- Charmhub magnum 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN -- operator judged it site misconfig; the draft is retained.) 9. **Barbican `host_href`:** the charm renders a literal `https://None:9312`, suppressing the upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A charm-level fix would need its own D-NNN. Logged, not actioned. 10. **v1-close checklist is QUEUED** (handoff section 6) -- gated on D-011 item 6, which is gated on the SEC-003 unseal rehearsal. ## OPEN -- NetBox WRITE-PATH BUGS (found 2026-07-14 by adversarial review; LOGGED, NOT FIXED) These are independent of D-119's naming work. **They matter BEFORE C2** (feeding the carve upstream), because they are what would corrupt the production apex during that write. 1. **`roles-aggregates-import.py` still dies HALF-WAY through a production write.** The preflight it advertises only checks role-NAME collisions. The `ARIN` RIR lookup and `validate_ula_48()` both run **after** the role-creation loop has already committed 5 roles. A typo'd `ORG_ULA_48`, or an apex without ARIN, writes 7 objects then dies with **no rollback**. The RIR preflight loop is a literal no-op (`if ...: continue` + a comment). The harness cannot catch it: it only asserts the STRING "PREFLIGHT" appears before the first create. **This is the same bug class that already bit us once.** 2. **`sandbox-fidelity-check.py` can return a FALSE GREEN.** It field-compares only the SHARED objects; the planned delta is checked merely as a SUBSET of `EXPECTED_NEW`. So a prefix created with the WRONG scope/role/status passes clean -- and if the importer created ZERO of the 36 DC prefixes, `extra subset-of expected` still holds and it exits 0 saying "nothing lost, nothing stray." **We have been citing this script as proof the sandbox is faithful. It cannot prove delta correctness.** It WOULD have caught the historical 17-prefix scope drop (those were shared objects). 3. **`find_role()` dies mid-loop** in `dc-dc-prefixes-import.py` -- roles are resolved lazily INSIDE the write loop. Against upstream as it stands today (no six-plane roles), a `--commit` creates the site + 4 prefixes then dies on the 5th. No whole-plan preflight. 4. **Same-dumper blind spot:** `sandbox-fidelity-check` compares two dumps produced by the SAME field list, so any field the dumper omits is invisible on BOTH sides by construction (prefix `vrf`, `tenant`, `vlan`, `tags`, `custom_fields`; site `tenant`/`group`/`facility`). Structurally the same shape as the region-scope bug that dropped 17 prefixes. 5. **Duplicate-CIDR collapse:** prefix-keyed dicts in the seeder and the fidelity check collapse duplicate prefixes (NetBox permits them, and the importer's own docstring anticipates them vs `vr0-dc0`). A whole prefix object can vanish with a green check. Latent -- no duplicates upstream today. 6. **The test fake diverges from reality at the failure point:** `fake_pynetbox.get()` returns `matches[0]` on a multi-match where REAL pynetbox RAISES. So the harness cannot catch #5. **Recommended sequence: fix 1-3 BEFORE C2.** 1-3 are DONE (2026-07-14, `docs/changelog-20260714-netbox-write-path-hardening.md`): the roles-importer preflight now covers ARIN + the ULA, the prefixes-importer preflights every role, the fidelity check is BOTH-bounds + delta-scope-aware, and the test fake raises like real pynetbox. Gauntlet 58 ALL GREEN. **CONSEQUENCE TO ACT ON:** the sandbox was declared "proven faithful" (2026-07-14) using the fidelity check BEFORE it was hardened -- i.e. under the version that could FALSE-GREEN. RE-RUN `sandbox-fidelity-check.py` against the sandbox under the hardened check before trusting that "faithful" verdict, and certainly before C2 feeds the carve upstream. (Not done this session: it needs the sandbox NetBox token, which is operator-held; folds naturally into the C2 step.) Items 4-6 remain LOGGED not fixed. (Historical rationale: 1 and 3 are partial-write-no-rollback on the production IPAM; 2 is what we would use to prove the write was correct.) ## Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions) - **BRANCH MERGE POINT = STAGE 2 CLOSE (operator ruling 2026-07-14).** Branch `dc-dc-stage1-phase0-first-exec` merges to `main` when every Stage 2 gate bullet is MET -- not at session end, not "when it feels done." The close-out checklist (C1..C5) lives in `docs/dc-dc-deployment-workflow.md`'s Stage 2 section; that doc is authoritative for what "closed" means. NEW policy: `main` has never taken a feature merge (its only merge commits are inherited from the pre-rename `openstack-caracal-ipv4` remote, D-110), so ALL VR1 work -- D-112..D-118, the Office1 headend, the NetBox tooling, 52 commits -- lives only on the branch. `main`'s tip is two ad-hoc operator uploads; it is NOT trunk-of-record for VR1 yet. The merge itself is a gated action: present it, do not run it unasked. - **D-071** ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning session. Backups now exist (a 902MB controller backup was taken; `juju create-backup` / `download-backup` DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). Controller HA (single-controller, no-HA) is the unresolved half. - **D-068** off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714. - **SEC-001 / SEC-004 / SEC-005 / SEC-007:** rotate / flip-to-private / revoke at v1 close. - **SEC-003:** custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. This keeps `d011-06-vault-unseal` MANUAL and gates the FULL close of D-011. - **SEC-006** (burned NetBox API token) -- **DEFERRED by operator ruling 2026-07-13 to deployment completion.** Until then the token is LIVE and EXPOSED. - **D-076 is NOT this repo's to work.** D-076..D-099 are a RESERVED source-project band, never assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are running their own tests. - **quota-vs-capacity: CLOSED** -- operator ACCEPTS as-documented (no quota trim). --- ## Standing lessons and traps (load-bearing; these live nowhere else) 1. **THE EDGE TRAP -- DHCP on Office1 is API-MANAGED.** A full rendered `config.xml` push WOULD CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all gone -- `opentofu/templates/README.md` is a tombstone). Configure the edge via the REST API (`scripts/opnsense-api.sh`); mint keys with `scripts/opnsense-bootstrap-apikey.sh`. 2. **"updated in-place" DOES NOT mean "no restart".** A `tofu apply` whose plan says `libvirt_domain.vm will be updated in-place` **RESTARTED the guest** (measured: uptime 8:36 -> 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to apply a disk-list change and BOUNCES it. **Assume ANY apply touching a `libvirt_domain`'s devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.** 3. **Do not re-implement vendor internals; call the interface.** OPNsense stores API secrets as `crypt(secret,'$6$')` (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate). 4. **A config ISO can NEVER be read on an OPNsense nano image** (D-112). Any instruction to build one is a trap; the dc-dc-phase2 runbook directed people into it for days. 5. **root's shell on the OPNsense edge is `tcsh`, not `sh`.** A `$(...)` inside a quoted remote command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. **Always feed remote commands to `sh -s`.** (The `scripts/opnsense-apply-config.sh` that was once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.) 6. **Guide changes must sweep the tenant AI skill in the SAME change.** Skill-lag was caught twice (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 is pinned, and any guide edit FAILS lint until re-recorded). 7. **The controller model is `admin/controller`,** not `/controller`. 8. **`juju status --format=line` omits workload messages on 3.6** -- this caused a cloud-assert false negative (fixed by DOCFIX-087). Don't rely on it. 9. **repo-lint's guards are load-bearing; reword rather than weaken.** L3 (docs must not reference missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, the guard kept. ## State facts to remember - **The beta cluster and the foil1 tenant are GONE** (both confirmed absent by the 2026-07-08 read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is retired here. `capi-test-1` and the `lbtest` LB leftover are also gone; orphan sweep reads ORPHAN=0, zero unattached floating IPs. - **devteam is a LIVE tenant** running a self-created cluster (built via Horizon, calico, keypair `devteam-key`). Their template has `volume_driver` unset but the cluster DOES get a default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the box; no template change needed. - The vcloud host was NOT pristine: 9 orphaned `vr0-*` libvirt nets were torn down; **`wan` was KEPT** (dual-stack NAT-out-enp1s0, GUA `2602:f3e2:fe:10::/64`) as the working model for the per-site ISP-uplink network. - MTU has TWO domains: the ISP uplink `enp1s0` is 1500; the host-internal plane/dark-fiber fabric is **jumbo, underlay_mtu=9000** (operator ruling; verified on all 10 nets). - The repo is temporarily **PUBLIC** (SEC-004) -- flip private at v1 close. - Secrets live jumphost-local and are NEVER committed: `~/vr1-office1-creds/` (edge SSH key, root password, API key), `~/.vr1-netbox.env` (NetBox token), `~/tenant-devteam/`, `~/vault-init/`. ## Project-completion (execute after D-011 passes) - Consolidate the 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. - Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials. - v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. ## NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13) **Done.** The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop (`netbox/prod-draft-dump.py` READ-ONLY -> `netbox/draft/vr1-draft.json` -> `netbox/sandbox-seed.py`), seeded 129 objects from the upstream draft, then applied the **D-115 office carve** (`netbox/d115-office-carve.py`): site `vr1-off1`, roles `office` + **`edge`** (new), and 8 prefixes. Idempotent, and **verified FIELD-BY-FIELD** against the upstream draft (`netbox/sandbox-fidelity-check.py`): every shared object is IDENTICAL, and the delta is EXACTLY `+edge` role, `+vr1-off1` site, `+8` D-115 prefixes. The sandbox is a proven-faithful replica, which is what makes it a valid baseline for the later feed-back diff. See `docs/vr1-office1-as-built.md` s6. **Why that check exists:** counts and idempotency CANNOT prove fidelity. The seeder silently dropped the scope on 17 of 90 prefixes (it mapped only `dcim.site`; most of the VR1 draft is `dcim.region`-scoped) -- and every count still matched and re-runs were still idempotent. It was caught by LUCK on a spot-check. The fidelity check is that catch, made deliberate. **D-117 Option B is only PARTIALLY EXECUTED.** Done: the import path (`dc-dc-prefixes-import.py`) and `docs/dc-dc-netbox-buildout-scope.md`. NOT done: the `$DC` shell selector (below) and the forward-looking DC prose in the runbooks. Do not read "D-117 ADOPTED" as "fully renamed". **D-117 ADOPTED (operator ruling, Option B, DC-only):** VR1's DCs are **dc0/dc1**, matching the apex. The **office KEEPS its number** (Off1). Fixed the importer's off-by-one site binding and made it **dry by default**. ### ~~BLOCKED~~ **UNBLOCKED and DONE** -- D-118 ratified the org ULA; both DCs imported **D-118 ADOPTED:** `ORG_ULA_48 = fd50:840e:74e2::/48` (RFC 4193 valid; **no collision with Tailscale's `fd7a:115c:a1e0::/48`**, which matters because `office1-tailscale` runs `--accept-routes` -- an overlap would have been a LIVE tailnet routing conflict). Also corrected a **FALSE claim** in `changelog-20260711-ula-gen-command-fix.md` that called the value "ratified" when no D-number ever assigned it -- the same document-asserts-authority failure mode as D-117. **Imported into the sandbox:** six-plane roles + 2 RIRs + 5 aggregates, then **VR1 DC0 (18 prefixes) and VR1 DC1 (18)**. The D-117 fix is now **LIVE-PROVEN**, not just offline: `--dc dc0` bound to the apex site `VR1 DC0` (id=8) and `--dc dc1` to `VR1 DC1` (id=9). ULA reads `DC.NN` in the 4th hextet exactly as D-111 intends (`fd50:840e:74e2:220::` .. `:350::`). **Final state PROVEN by `netbox/sandbox-fidelity-check.py`:** the sandbox is the upstream draft plus EXACTLY the planned delta (D-115 + D-117 + D-118) -- every shared object field-identical, nothing lost, nothing stray. 90 -> 134 prefixes. ### A REAL BUG in `roles-aggregates-import.py`, found by running it for the first time for real It checked existence by **slug only**. NetBox also enforces **unique NAMES**. The draft's legacy role `Replication` (slug `repl`) blocks the six-plane role (slug `replication`, name `Replication`), so it created 4 roles, **400'd on the 5th, and DIED -- leaving the apex half-populated** with no RIRs and no aggregates and no rollback. It had **never been run against a NetBox holding the real draft** (only empty ones) and **shipped with NO harness** -- which is why this survived. Fixed: the six-plane role is now named **"Replication Plane"** (slug unchanged -- `lib-net.sh` SPACES6 and the prefix importer look it up), and the script now **PREFLIGHTS every name/slug and writes nothing unless the whole plan is viable**. `tests/roles-aggregates-import/` created: 16/16. ### (historical) the original blocker `netbox/dc-dc-prefixes-import.py --dc dc0` cannot run. It requires `ORG_ULA_48`, and that literal is **still unassigned** -- it is gap **G3** in `docs/dc-dc-netbox-buildout-scope.md`, which *recommends* `fd50:840e:74e2::/48` (CSPRNG-generated, Global ID `0x50840e74e2`) but **no ADOPTED decision assigns it** (`grep ORG_ULA docs/design-decisions.md` -> nothing). D-101 explicitly leaves it NetBox-assigned. Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready. **To unblock:** ratify an org ULA `/48` under its own D-number. Then: ``` ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # VR1 DC0 ORG_ULA_48= DC_GUA_PREFIX=2602:f3e2:f03::/48 \ DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # VR1 DC1 ``` (`DC_GUA_PREFIX` values are MEASURED off the apex; `DC1_V4_SUPERNET` is assigned by D-115. `DC2_V4_SUPERNET` is RETIRED -- the tool rejects it by name.) ### Also open (recorded, not blocking) - **`$DC` shell-selector namespace collision** (D-117 amendment): `lib-net.sh`'s `dc0` means **VR0's** DC0, while the importer's `dc0` now means **VR1's**. NOT interchangeable. Proposed fix: region- qualify the shell selector to the apex slugs (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`). - Feed the D-115 carve upstream to `netbox.baldurkeep.com` (operator-gated; the tool refuses a non-sandbox target without `--yes-write-upstream`). - Office1 LAN IPv6 (`2602:f3e2:f01:100::/64` + RA via the OPNsense REST API) -- still pinned.