Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / session-ledger.md
@JANeumatrix JANeumatrix 22 hours ago 38 KB Add the missing preflight-failure test; reconcile the stale tofu-sync ledger line
Raw Blame History

Session ledger -- in-flight work continuity record

Purpose. Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, so any session -- after a compaction, or a fresh one -- can resume without losing pending work.

How to use it (standing practice).

  1. At session start: read this ledger AND run bash scripts/ledger-scan.sh. Reconcile.
  2. ledger-scan.sh is the DRIFT CHECK -- it derives the open-work it reliably can (PROPOSED/OPEN decisions, OPEN security rows, next-free numbers) straight from the repo. This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces.
  3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN.
  4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather than editing it by hand.

SINGLE STREAM (collapsed 2026-07-13). This ledger previously carried three parallel, separately-owned stream sections (main-chat, jumphost, shared) plus ~30 append-only session narratives. Those streams are CLOSED and reconciled into the one list below. There is now ONE stream. Do not re-introduce per-stream sections.

Where the history went. The 2,189-line session-by-session narrative is NOT lost -- it is in git history (the parent of the collapse commit) and, in durable form, in the 65 docs/changelog-*.md files, docs/design-decisions.md, and the incident reports. This ledger deliberately carries only what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else.

Machine-derived (re-seed from scripts/ledger-scan.sh; do not hand-edit)

Re-seeded from the 2026-07-14 scan. Re-run bash scripts/ledger-scan.sh to refresh.

  • PROPOSED / OPEN decisions: D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on the pre-DC-DC controller HA/backup session).
  • OPEN security rows: SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007, SEC-008. (SEC-008 -- the vcloud re-enrolment key -- was MISSING from the prior block.)
  • Next-free numbers: D = 119, DOCFIX = 195, BUNDLEFIX = 013. (The prior block said D=114 -- STALE by five; D-114..D-118 are all assigned and ADOPTED.)
  • Standing numbering rule: never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in docs/ or runbooks/ prose -- ledger-scan reads prose and a decoy token inflates the next-free counter. This has bitten twice.
  • KNOWN ledger-scan DEFECT (found 2026-07-14, logged not fixed): the scan reports D-115 as PROPOSED/OPEN. It is not -- D-115 is ADOPTED. The scan matches the words "PROPOSED/OPEN" anywhere in an entry's prose, and D-115's Status line reads "ADOPTED ... Originally PROPOSED/OPEN the same day". It should read the **Status:** line, not the body. This is the status twin of the decoy-token rule above. Until it is fixed, do not trust the scan's PROPOSED/OPEN list without checking the Status line. D-115 is the only current false positive.

Live state (measured 2026-07-13; nothing is mid-flight)

  • VR1 / Office1 edge office1-opnsense is UP and healthy. LAN 10.10.0.1/24, WAN 172.30.1.2 (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial console + getty alive, and kea-dhcp4 serving DHCP on udp/67 (subnet 10.10.0.0/24, pool .100-.199). DHCP is API-MANAGED (D-113(a2)).
  • OpenTofu repo/state DIVERGED BY D-119 (2026-07-14), APPLY GATED. tofu plan now shows 11 to add, 0 to change, 11 to destroy -- the DC module/object renames (dc1_*->vr1_dc0_*, dc2_*->vr1_dc1_*). Every replaced object is EMPTY (measured; Stage 3 hasn't run), and Office1's live objects are NOT in the plan. moved{} blocks make it read as clean replacements. This is a DESTROYING apply and is operator-gated -- do it BEFORE any Stage 3 tofu apply, so Stage 3 does not silently bundle the rename. tofu v1.12.3; tree validates (10/10 modules); lock committed.
  • VR0 / DC0 testcloud: fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts).
  • Branch dc-dc-stage1-phase0-first-exec, pushed and level with origin (2026-07-14). VR1 Stage 1 COMPLETE; Stage 2 in CLOSE-OUT (C1..C4 open, C5 done).
  • SESSION CONSOLIDATION (2026-07-14). Work had fragmented across FOUR concurrent Claude sessions on this one clone -- including two long-lived --permission-mode auto background agents (idle 14-15h but still holding auto-approve rights against the live cloud), and a second interactive session that received the SAME "resume last night's work" prompt six minutes before this one and began bootstrapping in parallel. All three were terminated and the supervising daemon stopped (it had respawned the agents once). Standing lesson: the daemon ADOPTS AND RESPAWNS background workers -- killing the child is not enough; stop the daemon (claude daemon stop --any). The working tree was clean and 3-ahead throughout; nothing was lost or clobbered. ONE session is now the rule.
  • Last night's three commits (D-113 amendment + the v6 script) were made by the auto-permission background agent and left UNPUSHED and UNDOCUMENTED. Both closed 2026-07-14: pushed, and docs/changelog-20260714-opnsense-set-interface-v6.md written.

OPEN WORK -- VR1 DC-DC (the current mission)

The authoritative per-stage tracker is docs/dc-dc-deployment-workflow.md (stage table + tooling gap register). Read it FIRST. This list is only what that doc does not already carry.

  1. Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13). voffice1: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, 10.10.0.20 (Kea reservation, outside the pool), egress through the OPNsense edge. On it: MAAS 3.7.2 region+rack (PostgreSQL 16.14) and LXD 5.21.5, with that LXD registered into MAAS as an LXD VM host. MAAS then composed, PXE-booted and COMMISSIONED office1-netbox (LXD VIRTUAL-MACHINE, leased 10.10.1.100 from MAAS, reached status Ready, 2 cores / 4096 MiB detected). The L3 nesting proof is now DEFINITIVE -- a guest booted, PXE'd and commissioned three levels deep, not merely "/dev/kvm exists". VR0's lxd + tailscale pattern, reproduced. Codified in scripts/site-headend-install.sh (+ harness, 18/18) -- reusable for DC1/DC2. THE DHCP SPLIT (structural, not conventional): 10.10.0.0/24 -> Kea on the edge (MAAS dhcp_on=False); 10.10.1.0/24 (lxdbr0) -> MAAS (dhcp_on=True). Separate L2s; lxdbr0 is NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13). office1-netbox (10.10.1.201) runs NetBox 4.6.4; office1-tailscale (10.10.1.202, tailnet 100.64.0.53) is a subnet router advertising 10.10.0.0/22 on the operator's SELF-HOSTED control plane. The edge routes 10.10.1.0/24 -> 10.10.0.20. Full connection reference: docs/vr1-office1-as-built.md -- read that first for any "how do I reach X" question. GitBucket: OUT OF SCOPE (D-116, operator ruling). No Office1-local GitBucket is built; git.baldurkeep.com stays the git service of record. Office1's composed machines are exactly TWO. REMAINING on Office1 (reconciled 2026-07-14): (a) route enable -- DONE, operator enabled 10.10.0.0/22 on the control server; the edge GUI answers at 10.10.0.1 from the workstation with no tunnel; (b) import the D-115 carve into the Office1 NetBox -- DONE (sandbox seeded from the upstream draft and PROVEN faithful field-by-field; carve + six-plane roles + BOTH DCs imported, 90 -> 134 prefixes); (c) the pinned Office1 LAN IPv6 (item 1a) -- STILL OPEN, and it is now the last build item on Office1. STAGE 2 IS IN CLOSE-OUT, NOT BUILD. It is also the BRANCH MERGE POINT (see PINNED). The close-out checklist C1..C5 is in docs/dc-dc-deployment-workflow.md; do not re-derive it here. 1a. DONE 2026-07-14 -- Office1's LAN IPv6 is LIVE and PROVEN ON THE WIRE. C1 is CLOSED. Address 2602:f3e2:f01:100::1/64 on the edge LAN (kernel-verified) + radvd advertising the prefix with mode=unmanaged; voffice1 autoconfigured 2602:f3e2:f01:100:5054:ff:fe6a:87e5 by SLAAC and edge->voffice1 GUA ping is 0.0% loss. No collateral damage (v4, Kea DHCPv4, compose-net route all intact). See docs/changelog-20260714-office1-lan-ipv6-executed.md. OPERATOR RULING 2026-07-14 -- DO NOT RE-PROPOSE A v6 NAT. A v4-to-v6 NAT (second OPNsense, or nginx) was proposed to simulate the v6 the lab will not forward, and was WITHDRAWN in favour of "configure IPv6 internally as much as possible; configure the IPv6 edge AFTER deployment completes." It contradicts D-101 ("native GUA routing, no NAT66") and D-115's amendment (upstream's problem, not ours), and a translation box is the one component Roosevelt will NOT have -- maximum delta, zero proof value. nginx is an L7 proxy, not NAT, and proves nothing about RA/ICMPv6/Ceph-over-v6. TRAP recorded: voffice1 has forwarding=1 + accept_ra=0 yet STILL autoconfigured -- netplan/systemd-networkd handles RA in USERSPACE, independent of the kernel sysctl. Never diagnose "no SLAAC" from the sysctl alone on Ubuntu; read ip -6 addr + look for a proto ra route. The WAN v6 leg remains DEFERRED -- v6 does not egress the lab (re-measured 2026-07-14: v6 internet 100% loss, v4 fine). Internet GUA reachability is UNPROVABLE in VR1.

1a-historical. (superseded -- the original pinned item) deploy Office1's LAN IPv6 (operator ruling 2026-07-13, D-115 question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). Add 2602:f3e2:f01:100::1/64 to the OPNsense edge LAN + RA, and let voffice1 take its address. CORRECTED 2026-07-14 (D-113 AMENDMENT -- the old instruction here was WRONG): the ADDRESS half CANNOT be done via the REST API. Measured on the live edge (OPNsense 26.1): Interfaces > [LAN] is the LEGACY page /interfaces.php?if=lan; only Devices and Neighbors were migrated to MVC, and there is NO REST endpoint for a base interface's ipaddrv6. D-113(a2)'s "interfaces are API-covered" was INFERRED, never measured, and is false.

  • Address: scripts/opnsense-set-interface-v6.sh (dry by default; harness 40/40). It is NOT a config.xml push -- it mutates two leaves of the edge's LIVE config through OPNsense's own model, so API-managed Kea and the firewall rules cannot be clobbered. WRITTEN AND TESTED, NEVER RUN WITH --commit.
  • RA: that half IS API-native -- /api/radvd/settings/addEntry via scripts/opnsense-api.sh. Still forbidden either way: a rendered config.xml push. KNOWN LIMIT, do not forget it: IPv6 DOES NOT EGRESS THE LAB (measured 2026-07-13: v6 gateway reachable, v6 internet 100% loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding, which is most of D-101's family matrix -- but NOT internet GUA reachability, which is unprovable in VR1 until the lab's upstream carries v6. Deploy the LAN leg; DEFER the WAN v6 leg (it would be a path to nowhere and untestable). See D-115's amendment. 1b. 10.10.1.0/24 IS NOW A BLESSED ALLOCATION (D-115 ADOPTED). VR1 Office1 = 10.10.0.0/22 (Office role, 10.10.0.0/16): LAN 10.10.0.0/24 + compose 10.10.1.0/24 + 2 spare. Zero renumber. Also adopted: Edge role 172.30.0.0/16 (legitimises office1-wan's 172.30.1.0/24, which had been squatting in the Corp OOB block), and DC2 moved to 10.12.64.0/19 inside the Cloud /16 (its old 10.13.0.0/19 sat outside every allocated block). STILL TO DO: write the carve into NetBox -- production write is operator-gated; the plan is to import into the Office1 sandbox NetBox first, verify, then feed back. 1c. (superseded note kept for context) the original unblessed-allocation finding: Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant pool=10.20.0.0/16 per D-016 -- the obvious 10.20.x would have COLLIDED with tenant space), but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is one of the VMs this headend composes. 1c. modules/node-vm nested-virt defect: FIXED 2026-07-13 (operator: fixes land as we find them; only DC1 deployment is gated). Unconditional cpu = { mode = "host-passthrough" } -- DC nodes run nova-compute and MUST run KVM guests. Without it they would have come up looking healthy and been unable to launch a single instance.
    1. NetBox sandbox loop -- BUILT 2026-07-13/14, still UNDOCUMENTED in the runbook (close-out C4). Architecture: draft-source netbox.baldurkeep.com (READ ONLY) -> a NEW NetBox on the Office1 VM (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements BACK. The sim NEVER --commits upstream. The tooling now exists and is proven: netbox/prod-draft-dump.py (read-only, no write path at all), netbox/sandbox-seed.py (dry-by-default, structurally REFUSES an upstream URL), netbox/sandbox-fidelity-check.py (field-by-field -- counts alone HID a bug where 17 of 90 prefixes silently lost their region scope). STILL PENDING: capture the loop in docs/dc-dc-netbox-buildout-scope.md + the Stage 2 runbook, and feed the validated carve UPSTREAM (close-out C2 -- the Stage 2 gate says authoritative, and the sandbox is not it). The upstream write is operator-gated.
    2. pynetbox -- CLOSED 2026-07-13. Installed (7.0.0) on office1-netbox, which is where the imports actually run. The stdlib tools (prod-draft-dump/sandbox-seed/sandbox-fidelity-check) deliberately need no dependency at all. TRAP, recorded in platform-traps.md: the upstream NetBox sits behind a User-Agent-filtering WAF. Python-urllib/3.12 gets a 403 that looks exactly like an auth failure; the same token works from curl. This affects pynetbox too.
    3. The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13. scripts/prereqs/install-apparmor-libvirt.sh (+ install-all / check-prereqs wiring; harness 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants <pool-parent>/** rwk, in /etc/apparmor.d/local/abstractions/libvirt-qemu, which GATES EVERY VR1 VM BOOT: without it a domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce libvirtd unless it actually adds the rule). Pool parent defaults to /var/lib/libvirt/vr1, override with VR1_POOL_PARENT. docs/changelog-20260713-apparmor-libvirt-prereq.md.
    4. The DEFERRED literals are now ALL ASSIGNED -- item CLOSED 2026-07-14. The operator's Option-B ruling ("do not invent CIDRs, wait for NetBox") held, and NetBox has now assigned them: ORG_ULA_48 = fd50:840e:74e2::/48 (D-118 ADOPTED -- RFC 4193 valid, and verified NOT to collide with Tailscale's fd7a:115c:a1e0::/48, which matters because office1-tailscale runs --accept-routes); GUA per D-117: first DC 2602:f3e2:f02::/48, second f03::/48; the second DC's v4 = six sequential /22s from 10.12.64.0/19 (D-115 moved it INSIDE the Cloud /16). All are IMPORTED and fidelity-checked in the sandbox. CAUTION -- DC2_V4_SUPERNET IS A RETIRED NAME (D-117). The importer now hard-rejects it and rejects --dc dc2. Any doc still saying 10.13.0.0/19 is pre-D-115 and wrong.
    5. D-119 (ADOPTED 2026-07-14) -- THE DC NAMESPACE IS REGION-QUALIFIED. C3/gap #19 CLOSED. vr0-dc0 (VR0's LIVE testcloud) / vr1-dc0 (VR1's FIRST DC, GUA f02::/48) / vr1-dc1 (VR1's SECOND, f03::/48). Bare dcN is REJECTED everywhere -- it meant VR0's live cloud in lib-net.sh but VR1's first DC in the importer: one string, two clouds. D-119 COMPLETES D-117 (executes its own amendment); it does not reverse it. ZERO NetBox writes -- the apex was already right; the REPO was the odd one out. The importer's DC->site map is now an IDENTITY, so the off-by-one is STRUCTURALLY IMPOSSIBLE, not merely defended. Also renamed: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114's own DR primitive is virsh destroy vdc1 -- it read as "destroy VR1 DC1" while MEANING "destroy VR1 DC0"). Office KEEPS its number (vr1-off1, Office1, voffice1). Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected by name). Gauntlet ALL GREEN (57); dc-selector 21->30, prefixes-import 40->82. THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to destroy (the libvirt object NAMES change, which is ForceNew; moved{} makes the plan reviewable but cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY (no guests, no volumes; Stage 3 hasn't run). Re-plan at execution and STOP if anything shows ATTACHED. See docs/changelog-20260714-d119-region-qualified-dc-namespace.md.

6a. (historical) G5 NetBox site rename -- CLOSED as D-117 (ADOPTED, Option B). The repo moved to the apex's 0-indexed convention (first VR1 DC = vr1-dc0), rather than bending the apex to the repo. This fixed a REAL off-by-one: dc-dc-prefixes-import.py bound --dc dc1 to slug vr1-dc1 while treating f02::/48 as dc1 -- but the apex binds f02 to VR1 DC0. It would have written the first DC's prefixes onto the second DC's site. The importer is now also dry-by-default. D-117 IS ONLY PARTIALLY EXECUTED -- see gap #19 / close-out C3: opentofu/main.tf and lib-net.sh/lib-hosts.sh still speak the OLD naming, and lib-net.sh's dc0 means VR0's DC0. That collision BLOCKS Stage 3. G2 (aggregates) and G3 (ULA /48) are both CLOSED by the same work.

  1. DOCFIX-185 follow-up (operator to rule): a D-100/D-107 amendment note so the Stage-3 DC edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark fiber is East-West/replication ONLY, never an internet path).
  2. docs/dc-dc-deployment-workflow.md's status table was corrected 2026-07-13, but re-read it against reality before trusting any row -- it has drifted twice.
  3. DHCP PROVEN END TO END -- CLOSED 2026-07-13. voffice1 (the first client ever to sit on office1-local) took a real lease from Kea: 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state active -- read back through the D-113(a2) REST API. The service, not just the daemon, now works.
  4. Doc-accuracy finding (measured 2026-07-13, logged not actioned): docs/security-ledger.md SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local libvirt bridge, virbr2, not routable off-host): ssh -L 8443:10.10.0.1:443 <user>@10.17.11.248 -> https://localhost:8443.

OPEN WORK -- VR0 / DC0 (the running testcloud)

  1. d011 batch-3 live-validation queue -- scripts/checks/d011-04-octavia-lb.sh and d011-05-magnum-e2e.sh are DRAFTED and mock-tested; the live PASS was recorded in the 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: d011-04 amphora_headroom() field parsing (safe default HOLDs on a parse miss -- confirm the OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set VR_FOIL_APPCRED. Sequence: non-disruptive half first, then --include-disruptive.
  2. Script backlog item 3 (remainder): fold scripts/vault-kv-health.sh into cloud-assert.sh as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure runbook.
  3. Script backlog item 5: scripts/tenant-cluster-create.sh -- generalize the stage-6 + watch + D-066-evidence wrapper. NOT STARTED (verified absent). (Items 6/7/8 -- keystone-policy-drift.sh, cloud-snapshot.sh, tests/tenant-acceptance/ -- are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.)
  4. S-class script quality (docs/script-quality-findings-20260707.md): S4/S7/S8a/S8b DONE; S6 no-action; S8c ruled WON'T-DO. S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ offboard) are HELD by the operator until the DOCFIX-126/127 live-verify lands.
  5. DOCFIX-126/127 live-verify is still OPEN -- the offboard-E0 and onboard-cred-file fixes have never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage.
  6. test1 orphan user -- approved for deletion, staged, NOT run. Role-less shell in the devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): openstack user delete 9a62f88e331f41adb5c9e41225ec8698.
  7. The handoff doc is stale: docs/handoff-20260703-open-items.md still lists R-3 (D-063) as PROPOSED/OPEN. D-063 is RESOLVED / CLOSED in design-decisions.md (verified). Correct the handoff doc.
  8. Upstream reports queued (operator): magnum can-upgrade-to anomaly -- Charmhub magnum 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN -- operator judged it site misconfig; the draft is retained.)
  9. Barbican host_href: the charm renders a literal https://None:9312, suppressing the upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A charm-level fix would need its own D-NNN. Logged, not actioned.
  10. v1-close checklist is QUEUED (handoff section 6) -- gated on D-011 item 6, which is gated on the SEC-003 unseal rehearsal.

OPEN -- NetBox WRITE-PATH BUGS (found 2026-07-14 by adversarial review; LOGGED, NOT FIXED)

These are independent of D-119's naming work. They matter BEFORE C2 (feeding the carve upstream), because they are what would corrupt the production apex during that write.

  1. roles-aggregates-import.py still dies HALF-WAY through a production write. The preflight it advertises only checks role-NAME collisions. The ARIN RIR lookup and validate_ula_48() both run after the role-creation loop has already committed 5 roles. A typo'd ORG_ULA_48, or an apex without ARIN, writes 7 objects then dies with no rollback. The RIR preflight loop is a literal no-op (if ...: continue + a comment). The harness cannot catch it: it only asserts the STRING "PREFLIGHT" appears before the first create. This is the same bug class that already bit us once.
  2. sandbox-fidelity-check.py can return a FALSE GREEN. It field-compares only the SHARED objects; the planned delta is checked merely as a SUBSET of EXPECTED_NEW. So a prefix created with the WRONG scope/role/status passes clean -- and if the importer created ZERO of the 36 DC prefixes, extra subset-of expected still holds and it exits 0 saying "nothing lost, nothing stray." We have been citing this script as proof the sandbox is faithful. It cannot prove delta correctness. It WOULD have caught the historical 17-prefix scope drop (those were shared objects).
  3. find_role() dies mid-loop in dc-dc-prefixes-import.py -- roles are resolved lazily INSIDE the write loop. Against upstream as it stands today (no six-plane roles), a --commit creates the site + 4 prefixes then dies on the 5th. No whole-plan preflight.
  4. Same-dumper blind spot: sandbox-fidelity-check compares two dumps produced by the SAME field list, so any field the dumper omits is invisible on BOTH sides by construction (prefix vrf, tenant, vlan, tags, custom_fields; site tenant/group/facility). Structurally the same shape as the region-scope bug that dropped 17 prefixes.
  5. Duplicate-CIDR collapse: prefix-keyed dicts in the seeder and the fidelity check collapse duplicate prefixes (NetBox permits them, and the importer's own docstring anticipates them vs vr0-dc0). A whole prefix object can vanish with a green check. Latent -- no duplicates upstream today.
  6. The test fake diverges from reality at the failure point: fake_pynetbox.get() returns matches[0] on a multi-match where REAL pynetbox RAISES. So the harness cannot catch #5.

Recommended sequence: fix 1-3 BEFORE C2. 1-3 are DONE (2026-07-14, docs/changelog-20260714-netbox-write-path-hardening.md): the roles-importer preflight now covers ARIN + the ULA, the prefixes-importer preflights every role, the fidelity check is BOTH-bounds + delta-scope-aware, and the test fake raises like real pynetbox. Gauntlet 58 ALL GREEN. CONSEQUENCE TO ACT ON: the sandbox was declared "proven faithful" (2026-07-14) using the fidelity check BEFORE it was hardened -- i.e. under the version that could FALSE-GREEN. RE-RUN sandbox-fidelity-check.py against the sandbox under the hardened check before trusting that "faithful" verdict, and certainly before C2 feeds the carve upstream. (Not done this session: it needs the sandbox NetBox token, which is operator-held; folds naturally into the C2 step.) Items 4-6 remain LOGGED not fixed.

(Historical rationale: 1 and 3 are partial-write-no-rollback on the production IPAM; 2 is what we would use to prove the write was correct.)

Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions)

  • BRANCH MERGE POINT = STAGE 2 CLOSE (operator ruling 2026-07-14). Branch dc-dc-stage1-phase0-first-exec merges to main when every Stage 2 gate bullet is MET -- not at session end, not "when it feels done." The close-out checklist (C1..C5) lives in docs/dc-dc-deployment-workflow.md's Stage 2 section; that doc is authoritative for what "closed" means. NEW policy: main has never taken a feature merge (its only merge commits are inherited from the pre-rename openstack-caracal-ipv4 remote, D-110), so ALL VR1 work -- D-112..D-118, the Office1 headend, the NetBox tooling, 52 commits -- lives only on the branch. main's tip is two ad-hoc operator uploads; it is NOT trunk-of-record for VR1 yet. The merge itself is a gated action: present it, do not run it unasked.
  • D-071 ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning session. Backups now exist (a 902MB controller backup was taken; juju create-backup / download-backup DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). Controller HA (single-controller, no-HA) is the unresolved half.
  • D-068 off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714.
  • SEC-001 / SEC-004 / SEC-005 / SEC-007: rotate / flip-to-private / revoke at v1 close.
  • SEC-003: custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. This keeps d011-06-vault-unseal MANUAL and gates the FULL close of D-011.
  • SEC-006 (burned NetBox API token) -- DEFERRED by operator ruling 2026-07-13 to deployment completion. Until then the token is LIVE and EXPOSED.
  • D-076 is NOT this repo's to work. D-076..D-099 are a RESERVED source-project band, never assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are running their own tests.
  • quota-vs-capacity: CLOSED -- operator ACCEPTS as-documented (no quota trim).

Standing lessons and traps (load-bearing; these live nowhere else)

  1. THE EDGE TRAP -- DHCP on Office1 is API-MANAGED. A full rendered config.xml push WOULD CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all gone -- opentofu/templates/README.md is a tombstone). Configure the edge via the REST API (scripts/opnsense-api.sh); mint keys with scripts/opnsense-bootstrap-apikey.sh.
  2. "updated in-place" DOES NOT mean "no restart". A tofu apply whose plan says libvirt_domain.vm will be updated in-place RESTARTED the guest (measured: uptime 8:36 -> 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to apply a disk-list change and BOUNCES it. Assume ANY apply touching a libvirt_domain's devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.
  3. Do not re-implement vendor internals; call the interface. OPNsense stores API secrets as crypt(secret,'$6$') (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate).
  4. A config ISO can NEVER be read on an OPNsense nano image (D-112). Any instruction to build one is a trap; the dc-dc-phase2 runbook directed people into it for days.
  5. root's shell on the OPNsense edge is tcsh, not sh. A $(...) inside a quoted remote command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. Always feed remote commands to sh -s. (The scripts/opnsense-apply-config.sh that was once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.)
  6. Guide changes must sweep the tenant AI skill in the SAME change. Skill-lag was caught twice (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 is pinned, and any guide edit FAILS lint until re-recorded).
  7. The controller model is admin/controller, not <user>/controller.
  8. juju status --format=line omits workload messages on 3.6 -- this caused a cloud-assert false negative (fixed by DOCFIX-087). Don't rely on it.
  9. repo-lint's guards are load-bearing; reword rather than weaken. L3 (docs must not reference missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, the guard kept.

State facts to remember

  • The beta cluster and the foil1 tenant are GONE (both confirmed absent by the 2026-07-08 read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is retired here. capi-test-1 and the lbtest LB leftover are also gone; orphan sweep reads ORPHAN=0, zero unattached floating IPs.
  • devteam is a LIVE tenant running a self-created cluster (built via Horizon, calico, keypair devteam-key). Their template has volume_driver unset but the cluster DOES get a default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the box; no template change needed.
  • The vcloud host was NOT pristine: 9 orphaned vr0-* libvirt nets were torn down; wan was KEPT (dual-stack NAT-out-enp1s0, GUA 2602:f3e2:fe:10::/64) as the working model for the per-site ISP-uplink network.
  • MTU has TWO domains: the ISP uplink enp1s0 is 1500; the host-internal plane/dark-fiber fabric is jumbo, underlay_mtu=9000 (operator ruling; verified on all 10 nets).
  • The repo is temporarily PUBLIC (SEC-004) -- flip private at v1 close.
  • Secrets live jumphost-local and are NEVER committed: ~/vr1-office1-creds/ (edge SSH key, root password, API key), ~/.vr1-netbox.env (NetBox token), ~/tenant-devteam/, ~/vault-init/.

Project-completion (execute after D-011 passes)

  • Consolidate the 10 per-phase do-documents into docs/v1-deploy-runbook.md.
  • Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials.
  • v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle.

NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13)

Done. The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop (netbox/prod-draft-dump.py READ-ONLY -> netbox/draft/vr1-draft.json -> netbox/sandbox-seed.py), seeded 129 objects from the upstream draft, then applied the D-115 office carve (netbox/d115-office-carve.py): site vr1-off1, roles office + edge (new), and 8 prefixes. Idempotent, and verified FIELD-BY-FIELD against the upstream draft (netbox/sandbox-fidelity-check.py): every shared object is IDENTICAL, and the delta is EXACTLY +edge role, +vr1-off1 site, +8 D-115 prefixes. The sandbox is a proven-faithful replica, which is what makes it a valid baseline for the later feed-back diff. See docs/vr1-office1-as-built.md s6.

Why that check exists: counts and idempotency CANNOT prove fidelity. The seeder silently dropped the scope on 17 of 90 prefixes (it mapped only dcim.site; most of the VR1 draft is dcim.region-scoped) -- and every count still matched and re-runs were still idempotent. It was caught by LUCK on a spot-check. The fidelity check is that catch, made deliberate.

D-117 Option B is only PARTIALLY EXECUTED. Done: the import path (dc-dc-prefixes-import.py) and docs/dc-dc-netbox-buildout-scope.md. NOT done: the $DC shell selector (below) and the forward-looking DC prose in the runbooks. Do not read "D-117 ADOPTED" as "fully renamed".

D-117 ADOPTED (operator ruling, Option B, DC-only): VR1's DCs are dc0/dc1, matching the apex. The office KEEPS its number (Off1). Fixed the importer's off-by-one site binding and made it dry by default.

BLOCKEDUNBLOCKED and DONE -- D-118 ratified the org ULA; both DCs imported

D-118 ADOPTED: ORG_ULA_48 = fd50:840e:74e2::/48 (RFC 4193 valid; no collision with Tailscale's fd7a:115c:a1e0::/48, which matters because office1-tailscale runs --accept-routes -- an overlap would have been a LIVE tailnet routing conflict). Also corrected a FALSE claim in changelog-20260711-ula-gen-command-fix.md that called the value "ratified" when no D-number ever assigned it -- the same document-asserts-authority failure mode as D-117.

Imported into the sandbox: six-plane roles + 2 RIRs + 5 aggregates, then VR1 DC0 (18 prefixes) and VR1 DC1 (18). The D-117 fix is now LIVE-PROVEN, not just offline: --dc dc0 bound to the apex site VR1 DC0 (id=8) and --dc dc1 to VR1 DC1 (id=9). ULA reads DC.NN in the 4th hextet exactly as D-111 intends (fd50:840e:74e2:220:: .. :350::).

Final state PROVEN by netbox/sandbox-fidelity-check.py: the sandbox is the upstream draft plus EXACTLY the planned delta (D-115 + D-117 + D-118) -- every shared object field-identical, nothing lost, nothing stray. 90 -> 134 prefixes.

A REAL BUG in roles-aggregates-import.py, found by running it for the first time for real

It checked existence by slug only. NetBox also enforces unique NAMES. The draft's legacy role Replication (slug repl) blocks the six-plane role (slug replication, name Replication), so it created 4 roles, 400'd on the 5th, and DIED -- leaving the apex half-populated with no RIRs and no aggregates and no rollback. It had never been run against a NetBox holding the real draft (only empty ones) and shipped with NO harness -- which is why this survived.

Fixed: the six-plane role is now named "Replication Plane" (slug unchanged -- lib-net.sh SPACES6 and the prefix importer look it up), and the script now PREFLIGHTS every name/slug and writes nothing unless the whole plan is viable. tests/roles-aggregates-import/ created: 16/16.

(historical) the original blocker

netbox/dc-dc-prefixes-import.py --dc dc0 cannot run. It requires ORG_ULA_48, and that literal is still unassigned -- it is gap G3 in docs/dc-dc-netbox-buildout-scope.md, which recommends fd50:840e:74e2::/48 (CSPRNG-generated, Global ID 0x50840e74e2) but no ADOPTED decision assigns it (grep ORG_ULA docs/design-decisions.md -> nothing). D-101 explicitly leaves it NetBox-assigned.

Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready.

To unblock: ratify an org ULA /48 under its own D-number. Then:

ORG_ULA_48=<ratified /48>  DC_GUA_PREFIX=2602:f3e2:f02::/48  --dc dc0     # VR1 DC0
ORG_ULA_48=<ratified /48>  DC_GUA_PREFIX=2602:f3e2:f03::/48  \
    DC1_V4_SUPERNET=10.12.64.0/19  --dc dc1                               # VR1 DC1

(DC_GUA_PREFIX values are MEASURED off the apex; DC1_V4_SUPERNET is assigned by D-115. DC2_V4_SUPERNET is RETIRED -- the tool rejects it by name.)

Also open (recorded, not blocking)

  • $DC shell-selector namespace collision (D-117 amendment): lib-net.sh's dc0 means VR0's DC0, while the importer's dc0 now means VR1's. NOT interchangeable. Proposed fix: region- qualify the shell selector to the apex slugs (vr0-dc0/vr1-dc0/vr1-dc1).
  • Feed the D-115 carve upstream to netbox.baldurkeep.com (operator-gated; the tool refuses a non-sandbox target without --yes-write-upstream).
  • Office1 LAN IPv6 (2602:f3e2:f01:100::/64 + RA via the OPNsense REST API) -- still pinned.