diff --git a/docs/changelog-20260709-netem-ula-gua-proposal.md b/docs/changelog-20260709-netem-ula-gua-proposal.md new file mode 100644 index 0000000..caaae22 --- /dev/null +++ b/docs/changelog-20260709-netem-ula-gua-proposal.md @@ -0,0 +1,52 @@ +# Changelog 2026-07-10 -- Proposed netem parameters + ULA/GUA generation guidance (DOCFIX-168) + +No live infrastructure touched -- documentation only, no repo mutation +beyond a new doc and gap-register/ledger pointers. Addresses tooling gap +register items #4(d) and #11. + +## Item + +### 1. DOCFIX-168 -- `docs/dc-dc-netem-and-ula-gua-proposal.md` +FILES: `docs/dc-dc-netem-and-ula-gua-proposal.md` (new), `docs/dc-dc- +deployment-workflow.md` (gap items #4(d) and #11 updated to reference it). + +WHAT: two independent parts, neither of which invents or decides a value +the operator/NetBox must own -- both make the NEXT step fast once the +operator is ready, per this repo's discipline of presenting options +rather than silently deciding anything not yet ADOPTED. + +**Part 1 -- proposed `tc netem` parameters (D-100 gap #4(d)):** a concrete +recommendation (1ms delay, 0.2ms jitter, 0.01% loss, uncapped rate), each +with reasoning grounded directly in the buildout design's own stated +"same-metro dark fiber, low single-digit ms" lean (not invented from +nothing) -- presented as a table for explicit operator ratification, with +instructions to record the ruling as a D-100 sub-item amendment in +`docs/design-decisions.md` (mirroring how every other Stage-0 redline item +was recorded) once accepted or adjusted. + +**Part 2 -- ULA/GUA/DC2-supernet generation guidance (D-101 gap #11 + gap +#3's data half):** explicitly does NOT generate the org ULA /48, obtain +the real per-DC GUA carve, or assign DC2's supernet -- gives the operator +the exact, safe command to generate a real RFC 4193-compliant ULA /48 +themselves (`openssl rand -hex 5`, confirmed via research this evening +that a cryptographically random 40 bits satisfies RFC 4193's statistical- +uniqueness requirement, matching what every independent ULA-generator +tool checked tonight actually does), and explains precisely why the GUA +carve and DC2 supernet are NOT generation tasks at all but real IPAM/ARIN +coordination requiring visibility and authority this session does not +have. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. + +REVERT: `rm docs/dc-dc-netem-and-ula-gua-proposal.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe -- no infrastructure +touched, no value generated or applied). + +## Next actionable step + +Both parts wait on the operator: Part 1 needs an explicit accept/adjust +ruling (recordable as a D-100 amendment in one sitting); Part 2's ULA +command takes two minutes to run once the operator is ready, after which +`netbox/dc-dc-prefixes-import.py --dc dc1` becomes runnable for real (only +`ORG_ULA_48`/`DC_GUA_PREFIX` are needed for dc1; `DC2_V4_SUPERNET` +additionally for dc2). diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 387855d..3b35946 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -52,11 +52,11 @@ | | | |---|---| | **Goal** | Stand up the operator's deployment headend first, before any DC. | -| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only). | -| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only. | +| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only); Office1's own OPNsense simulated-ISP edge (gap #16, DECIDED 2026-07-09: Stage 2 owns it). | +| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only; Office1-local network created (gap #12 CLOSED); Office1 OPNsense edge module wired (instantiation still blocked on real specs + gap #17's WAN-side network). | | **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | | **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | -| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. Flags a genuine, previously-unnoticed gap: no Office1-local virtual network is modeled anywhere in `opentofu/` for these VMs to attach to. NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09, UPDATED 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. The Office1-local-network gap it originally flagged (gap #12) is now CLOSED (`opentofu/modules/office1-network`, instantiated for real), and a new Step 4b now creates Office1's own OPNsense edge (gap #16, CLOSED -- ownership decided). NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | **State:** RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned). @@ -280,22 +280,76 @@ provisioner (OpenTofu runs from Office1, not the vcloud host where the bridge lives -- a bare local command can't reach it). UNVALIDATED, not instantiated. The exact parameters (latency/jitter/loss/rate) are - themselves still an unruled D-100 open sub-item (gap #11), so this - needs a decision before it can be applied for real, not just scripting. -5. **Ceph replication tooling (radosgw multisite + rbd-mirror).** No script in - the repo touches either (checked; `osd-blank-check.sh` is unrelated). D-108's - entire mechanism -- zonegroup setup, rbd-mirror daemon peering, the - failover/failback drill commands -- is net-new. Blocks Stage 6. -6. **Designate reactivation is a bundle change, not just a runbook.** - `runbooks/phase-01-bundle-deploy.md` states the bundle explicitly ships - "NO designate (D-019)" -- reactivating it per D-106 means editing - `bundle.yaml` itself (a built-surface change, gated behind D-106 leaving - PROPOSED), in addition to writing the reactivation runbook. Blocks Stage 7. -7. **MTU/geneve budget + Ceph disk-budget calculators.** D-102 and buildout-design - Section 3 both describe measurement-then-computation gates (host L2 MTU -> - geneve-over-v6 overhead; measured per-node OSD footprint -> size=3 vs size=2 - fallback) that are manual arithmetic in prose today, not a script. Blocks the - Stage 1 Phase-0 gates. + themselves still an unruled D-100 open sub-item (gap #11) -- **a + concrete PROPOSAL now exists** (`docs/dc-dc-netem-and-ula-gua- + proposal.md`, DOCFIX-168, Part 1: 1ms/0.2ms jitter/0.01% loss/uncapped + rate, with reasoning, presented for operator ratification, not + silently applied) -- still needs an actual ruling before it can be + applied for real. +5. **Ceph replication tooling (radosgw multisite + rbd-mirror) -- MECHANISM + CLOSED 2026-07-09 (DOCFIX-165), data half still open.** `runbooks/dc- + dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and 11.2-11.4's + `radosgw-admin`/`rbd mirror` command sequences (verbatim upstream Ceph + administration, not invented) are now callable, parameterized scripts: + `scripts/dc-dc-radosgw-multisite.sh` (subcommands `master-init`/`join- + readonly`/`enable-two-way` -- Step 4 one-way staging + Step 8's radosgw + half), `scripts/dc-dc-rbd-mirror.sh` (subcommands `bootstrap-primary`/ + `bootstrap-secondary --direction rx-only|rx-tx` -- Step 5 one-way staging + + Step 8's rbd-mirror half), and `scripts/dc-dc-dr-drill.sh` (subcommands + `failover` -- Step 10.2 force-promote + a flagged, never-auto-executed + Glance re-registration reminder -- and `failback` -- Steps 11.2-11.4, with + the demote-CURRENT-PRIMARY-before-promote-RECOVERING safety ordering + HARD-CODED as fixed call order, not left to documentation). Every + realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument, no + invented defaults; default mode is a `--dry-run` plan print, `--apply` + executes one command at a time via `juju ssh`/`juju run`, aborting on + first failure. New harnesses `tests/dc-dc-radosgw-multisite/`, + `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23 + PASS) test ONLY these scripts' own argument parsing, guard clauses, + dry-run output, and (critically) the failback step ORDER -- NOT real + radosgw/rbd-mirror behavior, since no live Ceph cluster exists this + session to test against (same UNVALIDATED posture as gap #3's NetBox + pipeline). The MECHANISM exists; the DATA half (real DC1/DC2 Ceph + cluster, real names/endpoints, the three `bundle.yaml` gaps in the Stage + 6 runbook's own "Known gaps" section) is unchanged and still blocks Stage + 6 for real. See `docs/changelog-20260709-ceph-replication-tooling.md`. +6. **Designate reactivation is a bundle change -- CLOSED 2026-07-10 + (DOCFIX-167).** `bundle.yaml` now carries `designate`/`designate- + bind`/`designate-mysql-router`/`designate-hacluster` + 8 relations, + researched against designate's real metadata.yaml (a real mistake + caught during authoring: its provides-side endpoint is `dnsaas`, not + `public` like the other API charms -- corrected). `os-public-hostname` + re-activation (D-106's OTHER coupled change) deliberately NOT done in + the shared bundle -- pushed to a proposed per-DC overlay instead, to + keep `bundle.yaml` DC-agnostic. `python3 scripts/provider-bundle- + check.py` PASS, all 6 existing invariants unaffected. See + `docs/changelog-20260709-designate-cinderbackup-rbdmirror.md`. NOT + applied to any live model -- repo-source change only. +7. **MTU/geneve budget + Ceph disk-budget calculators -- CLOSED 2026-07-09 + (DOCFIX-162).** D-102 and buildout-design Section 3 both describe + measurement-then-computation gates (host L2 MTU -> geneve-over-v6 + overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback) + that were manual arithmetic in prose, not a script, blocking the Stage 1 + Phase-0 gates. `scripts/dc-dc-mtu-geneve-budget.sh` takes the MEASURED + underlay MTU via required `--underlay-mtu` (no default) and reproduces + D-101's own worked example exactly (1500 - 56 = 1444), with a jumbo + (>=9000) path leaving tenant MTU at 1500, plus an added IPv6-minimum-MTU + (1280, RFC 8200) sanity floor clearly separated from D-101's own text. + `scripts/dc-dc-ceph-disk-budget.sh` takes MEASURED total disk, per-DC + OSD node counts and per-node footprints, and a required + `--backup-overhead-fraction` (deliberately NO default -- not specified + as a hard number anywhere in this repo, must come from real + Ceph/radosgw-admin measurement once DC1 exists) and computes whether + size=3 across both DCs plus overhead fits the measured total disk; + if not, it names size=2 as the ONLY documented fallback (D-101 / + Section 3) without silently applying it -- explicitly an operator-logged + decision. New harnesses `tests/dc-dc-mtu-geneve-budget/run-tests.sh` + (19/19 PASS) and `tests/dc-dc-ceph-disk-budget/run-tests.sh` (16/16 + PASS). Both scripts are read-only/mutate nothing and have not yet been + run against the real vcloud host (prep-only session, no live + infrastructure reachable) -- that first real run is part of executing + `runbooks/dc-dc-phase0-vcloud-prep.md` Step 3. Changelog: + `docs/changelog-20260709-mtu-ceph-budget-calculators.md`. 8. **The "developed IPv6 plan" D-101 cites does not exist in this repo.** D-101's context line references "the developed family-follows-reachability IPv6 plan (in-project, not yet committed)" as prior art -- grepped the whole repo, it is @@ -313,65 +367,152 @@ what it can and cannot close given today's real state (NetBox literals, node sizing, and several other decisions remain genuinely open -- see each runbook's own GATE section for the honest per-stage breakdown). -10. **Harness debt.** This repo's own delivery rule -- no script change ships - without its `tests//run-tests.sh` harness -- applies to everything in - #2-7 above. None exist yet because none of those scripts exist yet; expect - roughly 5-6 new test suites alongside whatever gets built. +10. **Harness debt -- LARGELY CLOSED 2026-07-09/10.** This repo's own delivery + rule -- no script change ships without its `tests//run-tests.sh` + harness -- has been honored for every new script tonight: `dc-selector` + (21/21), `dc-dc-prefixes-import` (40/40), `dc-dc-mtu-geneve-budget` + (19/19), `dc-dc-ceph-disk-budget` (16/16), `dc-dc-radosgw-multisite` + (19/19), `dc-dc-rbd-mirror` (19/19), `dc-dc-dr-drill` (23/23), plus new + `$DC` test cases added to `carve-host-interfaces`/`phase-00-maas- + standup` and a new `reenroll-hosts` harness. Gap #2 (OpenTofu modules) + remains the one area still lacking a harness beyond `scripts/opentofu- + validate.sh` itself (no `tofu` binary this session to validate against). 11. **Decision sub-items still genuinely open** (the big items are RULED per - Stage 0; these specific literals/params are not): exact netem parameters + Stage 0; these specific literals/params are not -- but see + `docs/dc-dc-netem-and-ula-gua-proposal.md` (DOCFIX-168) for a concrete + netem proposal and exact, safe ULA-generation guidance, drafted + 2026-07-10): exact netem parameters (D-100), exact NetBox literals for DC2/ULA/GUA (D-101). -12. **No Office1-local virtual network is modeled anywhere in `opentofu/`** - (found 2026-07-09 while drafting the Stage 2 runbook). `main.tf` wires - `office1_pool_path` (storage only) and the three inter-site mesh links - (management-traffic-only per D-100's own sub-item ruling) -- nothing the - three Office1 service VMs (MAAS-region, NetBox, GitBucket), or the vcloud - host's own management interface, could actually attach to as a shared - local network. Blocks designing real `cloudinit-vm` `network_config` - content for any of the three VMs. Needs either a new `modules/dc-planes`- - shaped Office1-local network, or a deliberate, documented reuse of an - existing host bridge -- not decided yet. -13. **The D-101 IPv6 family-matrix overlay does not exist** (found 2026-07-09 - while drafting the Stage 5 runbook). Which planes need ULA-only vs. - dual-stack vs. GUA addressing is fully specified by D-101 (Section 2 of - the buildout design), but the actual Juju charm config option names - needed to configure that per application (`ceph-mon`, `ovn-central`, - `octavia`, the HA-cluster/VIP charms, etc.) have not been confirmed - against any charm's real current schema. Blocks Stage 5's "Ceph-over-v6 - and geneve-over-v6 verified" gate criterion. D-109's IPv6-SAN cert- - issuance charm support is a related, separately-unverified assumption. -14. **Three real `bundle.yaml` gaps for D-108's DR mechanism** (found - 2026-07-09 while drafting the Stage 6 runbook, by grepping the actual - bundle): no `cinder-backup` charm deployed; no `ceph-rbd-mirror` charm - deployed; `ceph-mon`'s existing `rbd-mirror` endpoint binding is on - `storage`, not `replication` (inherited from the VR0/DC0 seed, predates - D-108's ULA-carrier requirement). All three are Stage 6's own first - gated mutation (its Step 2), not fixed by this delivery -- no live Ceph - cluster exists this session to test the fix against. +12. **No Office1-local virtual network is modeled anywhere in `opentofu/` -- + CLOSED 2026-07-09.** `main.tf` previously wired only `office1_pool_path` + (storage only) and the three inter-site mesh links (management-traffic- + only per D-100's own sub-item ruling) -- nothing the three Office1 + service VMs (MAAS-region, NetBox, GitBucket), or the vcloud host's own + management interface, could actually attach to as a shared local network. + Resolved by a NEW `opentofu/modules/office1-network` module (a + `dc-planes`-shaped isolated `libvirt_network`, sized for ONE network + rather than six planes since Office1 is a headend, not an OpenStack DC), + deliberately NOT a reused host bridge -- a hand-configured bridge would + be the one network in this topology living outside OpenTofu's D-103- + mandated inventory, silently breaking this repo's own `virsh net-list + --all` completeness assumption. INSTANTIATED for real in root `main.tf` + (`module "office1_network"`) -- unlike most of gap #2's other scaffolding, + this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`, + already real required inputs elsewhere in the same file. See + `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` + (DOCFIX-163) for the full design rationale. Designing the three + service VMs' actual `cloudinit-vm` `network_config` content against this + network remains separate, un-actioned work. +13. **The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real + risk still open.** `overlays/dc-dc-ipv6-family-matrix.yaml` now exists, + sourced against real charm config/source (`docs/dc-dc-ipv6-charm- + research.md` has full citations): `prefer-ipv6` (confirmed real option, + genuinely additive dual-stack HAProxy binding per `charm-nova-cloud- + controller`'s actual commit, not an either/or switch) + appended v6/GUA + VIPs for the API charms; `ceph-mon`'s own separate `prefer-ipv6` + + `ceph-public-network`/`ceph-cluster-network` for the ULA-only planes; + confirmed NO charm-config option exists for OVN (geneve family follows + the bound interface automatically). D-109's IPv6-SAN cert support is + CONFIRMED (read `charm-vault`'s actual `sort_sans()` code -- no v4/v6 + distinction). **Genuinely unresolved:** Octavia's `lb-mgmt-net` IPv6 + support is excluded from the overlay on purpose -- two real, still- + referenced upstream Launchpad bugs (#1911788, #1913409) document IPv6 + lb-mgmt-net failures; needs an explicit operator decision (accept the + risk and test for real, or keep Octavia's lb-mgmt-net as a deliberate + v4-only D-101 exception), not silently resolved here. Also unconfirmed: + 8 of the 9 dual-stack charms' `prefer-ipv6` presence is inferred by + shared-framework pattern, not individually fetched. UNVALIDATED against + any live model. +14. **Three real `bundle.yaml` gaps for D-108's DR mechanism -- CLOSED + 2026-07-10 (DOCFIX-167).** `cinder-backup` (a SUBORDINATE charm, + confirmed via real metadata.yaml -- the first draft wrongly gave it + principal-style `num_units`/`to`/`amqp`, corrected) and `ceph-rbd- + mirror` (a principal with `ceph-local`/`ceph-remote` requires + endpoints, confirmed via real metadata.yaml -- the first draft + invented a wrong binding key, corrected) are now in `bundle.yaml`; + `ceph-mon`'s `rbd-mirror` binding corrected from `storage` to + `replication`. **Genuinely unresolved, documented not invented:** + `ceph-remote` (the peer DC's ceph-mon) is a cross-Juju-controller + relation D-104's per-DC-independent-controller design means cannot be + a same-bundle relation line -- only this DC's own `ceph-local` leg is + wired; the cross-DC peer relationship needs a `juju offer`/`consume` + or the manual `rbd mirror pool peer bootstrap` CLI flow (Stage 6's + runbook already documents the latter). `python3 scripts/provider- + bundle-check.py` PASS. See `docs/changelog-20260709-designate- + cinderbackup-rbdmirror.md`. NOT applied to any live model. 15. **Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the Stage 4 runbook** (full detail in that runbook's own "Known gaps" - section): no OpenTofu module stands up a per-DC MAAS rack controller VM - itself (only registration, `modules/maas-vm-host`, exists); `reenroll- - hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not - `$DC`-parameterized (DOCFIX-151 only touched the two library files); - per-DC node count/sizing is an unmade Phase-0 decision; the per-DC - artifact mirror (D-107) has no identified owning stage/module; the MAAS - status a VM-host-discovered (vs. PXE-enlisted) machine lands in is - unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific. -16. **No stage explicitly owns creating Office1's own OPNsense edge VM** - (found 2026-07-09 during final consolidation, cross-checking Stage 2 - against Stage 3). The topology (buildout design Section 1) is + section). **Sub-item 2 (script `$DC`-parameterization) CLOSED + 2026-07-10 (DOCFIX-166)** -- `reenroll-hosts.sh`, + `carve-host-interfaces.sh`, and `phase-00-maas-standup.sh` now accept an + opt-in `$DC` env var and call `lib_net_select_dc`/`lib_hosts_select_dc` + immediately after sourcing (unset `$DC` is byte-for-byte unchanged + behavior; see `docs/changelog-20260709-maas-scripts-dc-param.md`). This + closes the CLI-invocation half of the gap only -- it does NOT create any + new per-DC data (no dc1/dc2 hostnames, octets, boot MACs, or node-count + figures were invented), and `phase-00-maas-standup.sh`'s own `PLANES` + table remains hardcoded to D-052/D-053's literals rather than driven by + `lib-net.sh`'s flat vars, so it is not truly dc2-topology-aware yet even + though it now gates dc2 out cleanly. The other six sub-items remain + OPEN: no OpenTofu module stands up a per-DC MAAS rack controller VM + itself (only registration, `modules/maas-vm-host`, exists); per-DC node + count/sizing is an unmade Phase-0 decision; DC2 is still blocked at the + network-selector layer (gap #3, NetBox literals not assigned); the + per-DC artifact mirror (D-107) has no identified owning stage/module; + the MAAS status a VM-host-discovered (vs. PXE-enlisted) machine lands in + is unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific. +16. **No stage explicitly owns creating Office1's own OPNsense edge VM -- + CLOSED 2026-07-09.** The topology (buildout design Section 1) is explicit: "full dark-fiber triangle... plus a **per-site** OPNsense - simulated ISP edge" -- three sites, three edges (DC1, DC2, AND - Office1). Stage 3's Build list explicitly covers DC1/DC2's edges + simulated ISP edge" -- three sites, three edges (DC1, DC2, AND Office1). + Stage 3's Build list already covered DC1/DC2's edges (`modules/opnsense-edge` per DC); Stage 2's Build list (MAAS region, - OpenTofu, NetBox, GitBucket, Tailscale) does NOT mention an Office1 - edge at all, and neither Stage 2's nor Stage 3's runbook instantiates - one. Related to, but distinct from, gap #12 (no Office1-local - network) -- that gap is about the service VMs' own attach point; this - one is about whether Office1 gets a simulated-ISP-edge boundary at - all. Needs a decision: does Office1 get its own `modules/opnsense- - edge` call (a fourth site alongside DC1/DC2), and if so, which - stage's runbook creates it? Not decided, not invented here. + OpenTofu, NetBox, GitBucket, Tailscale) did not mention an Office1 edge + at all, and neither runbook instantiated one. **Decision: YES, Office1 + gets its own `modules/opnsense-edge` call, a fourth site alongside + DC1/DC2; Stage 2's runbook + (`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it.** + Rationale: Stage 2's entire scope is already "Office1 headend standup" + (the OPNsense edge IS part of that headend, not DC substrate), and + Stage 1 already creates the mesh-link networks this edge's neighbors + depend on -- Stage 3 has no reason to reach back and create a piece of + Office1's own infrastructure. Wired as a commented-out `module + "office1_opnsense"` skeleton in root `main.tf` (mirroring the identical + NOT-YET-REAL status of DC1/DC2's own edges -- neither is instantiated + either) plus a new gated step in + `runbooks/dc-dc-phase1-office1-standup.md` (Step 4b) and its own GATE + bullet. Related to, but distinct from, gap #12 (Office1-local network, + now also CLOSED) -- that gap was the service VMs' own attach point; + this one was whether Office1 gets a simulated-ISP-edge boundary at all, + and who builds it. The edge's LAN side now resolves cleanly to gap + #12's new network; its WAN side does not -- see NEW gap #17 + immediately below, a genuinely deeper, cross-site gap surfaced while + closing this one, not invented as a substitute for a real answer. +17. **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in + this repo, for ANY site** (found 2026-07-09 while closing gap #16 for + Office1). Every `modules/opnsense-edge` caller needs a + `wan_network_name` distinct from BOTH its LAN-side network (a + `dc-planes` plane, or gap #12's new `office1-network`) AND the three + D-100 mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC- + ONLY by D-100's own sub-item ruling (Office1<->DC fiber carries MAAS/ + Juju/operator traffic only), not a simulated ISP uplink, so wiring an + edge's WAN side to a mesh leg would misrepresent the topology rather + than merely placeholder it. This is NOT an Office1-specific gap: + `runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 5's own `dc1_opnsense` + template carries the IDENTICAL unresolved placeholder + (`wan_network_name = module.mesh_dc1_office1... # or a dedicated + per-site ISP-uplink segment -- confirm which, do not assume`) -- + confirmed by reading that runbook directly, not assumed. Deliberately + NOT invented here for Office1 alone (that would contradict D-100 and + leave DC1/DC2 with the same unresolved gap under a different name). + Needs a decision: a fourth network shape per site (one more + `dc-planes`/`mesh-link`/`office1-network`-style dedicated + `libvirt_network` standing in for that site's own "internet" uplink), + or some other mechanism not yet considered. Blocks instantiating ANY of + the three sites' `opnsense-edge` calls (`dc1_opnsense`, `dc2_opnsense`, + `office1_opnsense`) end-to-end -- all three are otherwise ready + (image/config mechanism per gap #4, LAN-side network per gap #12/this + stage's plane modules) but for this one missing piece. --- @@ -389,6 +530,8 @@ | MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` | | `tc netem` mechanism (`modules/netem-link`) | BUILT 2026-07-09, UNVALIDATED -- real latency/jitter/loss/rate parameters still an unruled D-100 item, see `opentofu/README.md` | | `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` | +| Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` | +| Office1 OPNsense edge ownership | DECIDED 2026-07-09 (DOCFIX-163, gap #16 CLOSED) -- Stage 2 owns it; `module "office1_opnsense"` skeleton in `main.tf` (commented, same as DC1/DC2's own not-yet-instantiated edges); WAN-side network still blocked on NEW gap #17 (no per-site ISP-uplink network exists anywhere yet, DC1/DC2 included) | | Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` | | `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 4 and Stage 5's runbooks (both written 2026-07-09) mandate calling it, not yet executed for real | | NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned | diff --git a/docs/dc-dc-netem-and-ula-gua-proposal.md b/docs/dc-dc-netem-and-ula-gua-proposal.md new file mode 100644 index 0000000..c5b4e0c --- /dev/null +++ b/docs/dc-dc-netem-and-ula-gua-proposal.md @@ -0,0 +1,131 @@ +# Proposed netem parameters + ULA/GUA generation guidance (2026-07-10) + +Addresses tooling gap register items #4(d) and #11: the two D-100/D-101 +sub-items still genuinely "leaning," not ruled, even after Stage 0 closed +the big decisions. This document does NOT rule either -- it presents a +concrete recommendation for #4(d) (present options, get a ruling, per this +repo's own discipline for anything touching an ADOPTED-but-not-fully- +specified decision) and, for #11's literal-generation half, exact commands +the OPERATOR runs to produce a real value (this document does not generate +or invent the actual ULA/48 itself -- that is explicitly the operator's/ +NetBox's job per D-101's own text). + +--- + +## Part 1 -- Proposed `tc netem` parameters (D-100 sub-item #4(d)) + +**Current state:** `docs/dc-dc-buildout-design.md` Section 6 states only a +qualitative lean: "same-metro dark fiber (low single-digit ms, jumbo- +capable)." No specific latency/jitter/loss/rate numbers are ruled. +`opentofu/modules/netem-link` (built 2026-07-09) is ready to apply real +parameters the moment they're decided -- this is the one missing input. + +**Proposal (for operator ratification, not silently applied):** + +| Parameter | Proposed value | Rationale | +|---|---|---| +| Latency | 1ms (each direction) | Same-metro dark fiber circuits (single city, no long-haul) typically show sub-millisecond one-way propagation delay for distances under ~50km (fiber propagation is ~5us/km) -- 1ms per direction (2ms RTT) is a deliberately conservative round number that accounts for real-world switching/equipment overhead beyond pure propagation, while staying solidly within the buildout design's own "low single-digit ms" lean. | +| Jitter | 0.2ms | A small, non-zero jitter keeps the simulation honest (a perfectly flat-latency link is unrealistic even on dedicated fiber) without dominating the 1ms base latency -- roughly 20%, a common rule-of-thumb ratio for stable dedicated circuits (as opposed to shared/internet-transit links, which would warrant a much larger jitter fraction). | +| Loss | 0.01% | Dark fiber with modern optical equipment is very low-loss; this is a nonzero-but-negligible value so packet-loss-handling code paths are still technically exercised during the drill, without meaningfully affecting throughput-sensitive tests (Ceph replication, radosgw sync). | +| Rate | Not capped (no `rate` netem param) | The buildout design's own lean explicitly says "jumbo-capable" -- same-metro dark fiber circuits are typically provisioned well above what this test environment's actual traffic will generate (VR1 is a virtual rehearsal on one physical host, not a real multi-site deployment) -- an artificial rate cap would constrain the TEST more than a real Roosevelt link would constrain PRODUCTION, which is backwards for a rehearsal whose value is proving the failover mechanism works, not proving it works under bandwidth starvation. If the operator wants to also rehearse a bandwidth-constrained scenario, that is a SEPARATE, deliberate test configuration, not this default. | + +**Resulting `tc netem` invocation shape** (for reference; `modules/netem- +link`'s `netem_args` variable takes this as a string): +``` +delay 1ms 0.2ms loss 0.01% +``` + +**This is a recommendation, not a ruling.** Per this repo's standing +discipline (present options for anything not yet ADOPTED, never silently +decide), the operator should explicitly ratify this table (accept as-is, +or adjust) before `modules/netem-link` is actually instantiated with real +values in Stage 3's runbook. Once ratified, this should be recorded as a +D-100 amendment (a new "Sub-item RULED" line in `docs/design-decisions.md` +D-100's entry, following the exact pattern D-100's other redline items +already got at Stage 0 ratification) -- not left as a standalone doc. +**Re-tune when a specific Roosevelt inter-DC target is known** (buildout +design's own stated intent) -- this proposal is a VR1 rehearsal default, +not a permanent value. + +--- + +## Part 2 -- ULA/GUA/DC2-supernet generation guidance (D-101, gap #11 + gap #3's data half) + +**What this section does NOT do:** generate the org ULA /48, obtain the +real per-DC GUA carve, or assign DC2's supernet. D-101's own text is +explicit that these are NetBox-authoritative, populated via the extended +import pipeline -- not hardcoded in any decision, and not something this +session (or any Claude session) should invent, guess, or pre-select on the +operator's behalf. What follows is the exact, safe MECHANISM for the +operator to produce each real value themselves, so that step is a two- +minute copy-paste rather than a research task when they're ready to do it. + +### 2.1 -- Org ULA /48 (RFC 4193) + +RFC 4193 requires the 40-bit Global ID be "randomly generated" for +statistical uniqueness across organizations -- a cryptographically random +40 bits satisfies this directly (this is also what every independent +RFC-4193-compliant ULA generator tool checked during tonight's research +actually does under the hood, not a shortcut invented here): + +```bash +printf 'fd%s::/48\n' "$(openssl rand -hex 5 | sed 's/\(..\)\(..\)\(..\)\(..\)\(..\)/\1\2:\3\4\5/')" +``` + +Run this ONCE, by the operator, on a real machine -- the output is the +real `ORG_ULA_48` value `netbox/dc-dc-prefixes-import.py` (DOCFIX-152) +requires. Record it in NetBox as the IPAM apex (D-101's own requirement), +not just as an environment variable -- this is a permanent organizational +identifier, not a per-session value. + +### 2.2 -- Per-DC GUA carve (from ARIN 2602:f3e2::/32 region-0 /36) + +This is NOT something to generate randomly -- D-101's own text names this +as ALREADY a real, existing ARIN allocation ("GUA (ARIN 2602:f3e2::/32, +region-0 /36)"), meaning the org already holds this block from a real +Internet numbering authority. The per-DC carve within it (which /40 or /44 +goes to DC1 vs. DC2) is an INTERNAL allocation decision within an already- +owned block, not a new external assignment -- this can be decided by +whoever administers that ARIN allocation for this organization, following +whatever internal IPAM convention they already use for carving GUA space +(e.g. sequential /40s: DC1 = `2602:f3e2:1000::/40`, DC2 = +`2602:f3e2:2000::/40`, matching the example addresses already used in +`netbox/dc-dc-prefixes-import.py`'s own docstring and README -- those are +ILLUSTRATIVE examples in this repo, not yet a real assignment; confirm +with whoever holds the ARIN allocation before treating any specific /40 as +final). + +### 2.3 -- DC2's v4 supernet + +This is a NetBox IPAM assignment task, not a generation task -- DC2 needs +a real, non-overlapping v4 supernet (at least a /19, per `netbox/dc-dc- +prefixes-import.py`'s own `DC2_MIN_SUPERNET_PREFIXLEN` check) chosen from +whatever address space this organization has available and not already in +use by DC1, Office1, client VPN ranges (D-074), or any other real +allocation. This is a real IPAM decision requiring visibility into the +FULL address inventory, which this session does not have -- assign it via +NetBox directly, following whatever process this organization already +uses for carving new supernets. + +### 2.4 -- Once all three exist + +Run `netbox/dc-dc-prefixes-import.py --dc dc1` (needs only `ORG_ULA_48` +and `DC_GUA_PREFIX` from 2.1/2.2) and `--dc dc2` (additionally needs +`DC2_V4_SUPERNET` from 2.3) for real, closing the DATA half of tooling gap +#3 -- the mechanism has been ready since DOCFIX-152; only these three real +inputs were ever missing. + +--- + +## Summary: what this closes vs. leaves open + +**CLOSES:** gap #4(d) now has a concrete, reasoned proposal ready for a +one-line operator ratification (not a re-research task); gap #11's ULA- +generation half now has an exact, safe, two-minute command instead of +"go figure out RFC 4193." + +**STAYS OPEN, by design:** the actual netem ratification (a decision only +the operator can make); the actual ULA-48/GUA-carve/DC2-supernet values +(real-world IPAM/ARIN coordination this session has no authority or +visibility to perform) -- this document makes producing them fast and +low-risk once the operator is ready, it does not produce them. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 0ccff4e..ae293cf 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1125,6 +1125,231 @@ status (no `tofu` binary has run yet), gaps #12-15 (Office1-local network, IPv6 family-matrix overlay, bundle.yaml DR gaps, MAAS/ multi-rack gaps). +- **DONE -- Tooling gap register item #7 CLOSED 2026-07-09 (MTU/geneve + + Ceph disk-budget calculators, changelog `docs/changelog-20260709-mtu- + ceph-budget-calculators.md`, DOCFIX-162 -- numbering deferred to the + orchestrating session to avoid a collision with other parallel + in-flight agents tonight):** `scripts/dc-dc-mtu-geneve-budget.sh` + (required `--underlay-mtu`, no default) reproduces D-101's own worked + example exactly (1500 - 56 = 1444), leaves tenant MTU at 1500 for a + measured jumbo (>=9000) underlay, and adds a clearly-separated + IPv6-minimum-link-MTU (1280, RFC 8200) sanity floor beyond D-101's own + text. `scripts/dc-dc-ceph-disk-budget.sh` (required `--total-disk`, + per-DC `--dcN-nodes`/`--dcN-per-node-osd`, and required + `--backup-overhead-fraction` with deliberately NO default -- not a hard + number anywhere in this repo, must come from real DC1 Ceph/radosgw-admin + measurement) computes whether size=3 across DC1+DC2 plus overhead fits + the measured total disk, and if not, names size=2 as the ONLY documented + fallback (D-101/Section 3) without silently applying it -- an explicit + operator-logged decision. New harnesses: `tests/dc-dc-mtu-geneve-budget/ + run-tests.sh` 19/19 PASS, `tests/dc-dc-ceph-disk-budget/run-tests.sh` + 16/16 PASS -- both cover the exact worked/constructed numeric examples, + the jumbo/fits vs fallback branches, and every required-arg/bad-format + fail-loud path. `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy + warn (unchanged). Both scripts are read-only calculators (mutate + nothing) and have not yet been run against the real vcloud host -- that + first real run is part of executing `runbooks/dc-dc-phase0-vcloud- + prep.md` Step 3. Uncommitted -- operator has not yet asked to commit + this piece; the orchestrating session will assign the real DOCFIX number + and find-replace `DOCFIX-162` at integration time. +- **DONE -- Tooling gap register items #12 and #16 CLOSED 2026-07-09 + (changelog `docs/changelog-20260709-office1-network-edge.md`, + DOCFIX-163 -- numbering deferred to the orchestrating session, same + reason as the #7 item above):** built `opentofu/modules/office1-network` + (a new, dc-planes-shaped isolated `libvirt_network` sized for ONE network, + not six planes -- closes gap #12) and INSTANTIATED it for real in root + `main.tf` (`module "office1_network"`, needs no unmeasured value beyond + already-real `domain_suffix`/`underlay_mtu`). Design decision: a new + module, deliberately NOT a reused host bridge -- a bridge would be the one + network in this topology living outside OpenTofu's D-103-mandated + inventory and would silently break this repo's own `virsh net-list --all` + completeness assumption; full reasoning in the module's own header and in + `runbooks/dc-dc-phase1-office1-standup.md`'s Open question #1 (now marked + RESOLVED). Also decided gap #16 (Office1 OPNsense edge ownership): YES, + Office1 gets its own `modules/opnsense-edge` call, a fourth site alongside + DC1/DC2, and Stage 2's own runbook owns creating it (new Step 4b, + CHECK-before-MUTATION shape, its own GATE bullet) -- Stage 2's whole scope + is already "Office1 headend standup" and Stage 1 already builds this + edge's network neighbors, so Stage 3 has no reason to reach back into + Office1's own infrastructure. A commented-out `module "office1_opnsense"` + skeleton now exists in `main.tf`, mirroring Stage 3's `dc1_opnsense` + template exactly; its `lan_network_name` resolves cleanly to the new + network, but its `wan_network_name` does not -- investigated rather than + invented, and found to be a genuinely DEEPER, cross-site gap: Stage 3's + own `dc1_opnsense` template carries the identical unresolved + `wan_network_name` placeholder, confirming no dedicated per-site + ISP-uplink/WAN network exists yet for ANY site (DC1/DC2 included). Logged + as NEW tooling gap register item #17 rather than defaulted to a mesh-link + network, which would have directly contradicted D-100's own sub-item + ruling that the three mesh legs carry management traffic only. `bash + scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn (unchanged); + `opentofu/` remains UNVALIDATED overall (no `tofu` binary available this + session, same standing status as every other module here -- not + newly introduced by this delivery). Uncommitted -- operator has not yet + asked to commit; the orchestrating session will assign the real DOCFIX + number and find-replace `DOCFIX-163` at integration time. +- **DONE (uncommitted) -- Gap #13 (D-101 IPv6 family-matrix overlay) + DRAFTED, one real risk left open, researched directly by the + orchestrating session (not delegated) given the accuracy stakes:** + `overlays/dc-dc-ipv6-family-matrix.yaml` + `docs/dc-dc-ipv6-charm- + research.md` (full sourcing). Fetched REAL charm config/source, not + memory: `charm-keystone/config.yaml` confirms `prefer-ipv6` (bool) is a + real, shared charms.openstack option; `charm-nova-cloud-controller`'s + actual "Dual Stack VIPs" commit confirms it's genuinely ADDITIVE (HAProxy + binds both v4 and v6 wildcard addresses simultaneously), not an + either/or switch -- resolving the biggest open question about whether + D-101's dual-stack ambition is even mechanically possible. `charm-ceph- + mon/config.yaml` confirms its OWN separate `prefer-ipv6` + + `ceph-public-network`/`ceph-cluster-network`, which per the charm's own + docs IS a straight switch (not additive) -- the correct shape for the + ULA-only storage/replication planes. Fetched the FULL `charm-layer- + ovn/config.yaml` (25 options) and confirmed NO IPv6/encapsulation option + exists at all -- OVN needs no overlay entry, geneve family follows the + bound interface once the plane is ULA-only. Fetched `charm-vault`'s + actual `vault_pki.py` `sort_sans()` code and confirmed no IPv4/IPv6 + distinction in cert-issuance -- D-109's IPv6-SAN requirement is + code-confirmed, not assumed. **Found a real, open upstream risk while + researching, not invented or glossed over:** Octavia's `lb-mgmt-net` + IPv6 support has two real, still-referenced Launchpad bugs (#1911788, + #1913409) describing failures -- #1911788's root cause is an OVN/LXD/ + MAAS hostname-resolution mismatch, the SAME CLASS of problem this repo's + own D-008 bootstrap order already hardens against, so not necessarily + fatal, but genuinely unresolved. The overlay deliberately excludes an + Octavia entry rather than silently forcing the risk through. Updated + Stage 5's runbook (`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`) Step 6 + and the tooling gap register item #13 to reflect all of this. `bash + scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. Uncommitted -- + orchestrating session will assign the real DOCFIX number at integration + time alongside the other parallel gap-closure work tonight. +- **DONE -- `$DC` parameterization for the 3 MAAS scripts (changelog + `docs/changelog-20260709-maas-scripts-dc-param.md`, DOCFIX-166), + closing gap #15 sub-item 2's CLI-invocation half:** `scripts/reenroll- + hosts.sh`, `scripts/carve-host-interfaces.sh`, and `scripts/phase-00-maas- + standup.sh` now accept an opt-in `$DC` env var and call `lib_net_select_dc` + / `lib_hosts_select_dc` immediately after sourcing, matching the same + convention (and the same dc1-network-no-op/dc1-hosts-fail asymmetry) + DOCFIX-151 gave the two library files the night before. Unset `$DC` is + provably byte-for-byte unchanged (verified by direct, jq-independent + invocation of all three scripts across the unset/dc0/dc1/dc2/bogus + matrix, since the selector calls sit before any MAAS/jq touch-point in + every script). Added `$DC` test cases to both existing harnesses + (`carve-host-interfaces`, `phase-00-maas-standup`) and created a new one + (`reenroll-hosts` had none before). Did NOT invent any per-DC host/CIDR + literal, and did NOT make `phase-00-maas-standup.sh`'s hardcoded `PLANES` + table dc2-aware -- both remain honestly open (dc1 only "works" there + because D-101 makes its real target identical to dc0's). Did not run a + `git stash` A/B this time since the working tree carries other parallel + agents' in-flight uncommitted work tonight; verified directly instead. + repo-lint 0 fail/1 documented warn. `docs/dc-dc-deployment-workflow.md` + item #15 and `runbooks/dc-dc-phase3-maas-enlist-deploy.md` (gap #2, Steps + 5 and 8, delivery checklist) updated to reflect the partial closure -- + six of the seven gap-#15 sub-findings remain open, said plainly rather + than overclaiming. No live infrastructure touched. Uncommitted -- + orchestrating session will assign the real DOCFIX number at integration + time. +- **DONE (uncommitted) -- Gaps #6 + #14 (bundle.yaml: Designate reactivation + + D-108 DR mechanism), done directly by the orchestrating session given + the stakes (this is the SAME `bundle.yaml` the jumphost operates the + live VR0/DC0 rehearsal cloud with, not a DC-DC-only file):** + `bundle.yaml` gained `designate`/`designate-bind`/`designate-mysql- + router`/`designate-hacluster` (D-106) and `cinder-backup`/`ceph-rbd- + mirror` (D-108) + `ceph-mon`'s `rbd-mirror` binding corrected from + `storage` to `replication`. **Caught and fixed three real mistakes + during self-review, by fetching each new charm's REAL metadata.yaml + rather than trusting the standard-pattern draft or the Stage 7 runbook's + own earlier "template" section:** (1) designate's real provides-side + endpoint is `dnsaas`, not `public` like the other API charms; (2) + `cinder-backup` is a SUBORDINATE charm (confirmed `subordinate: true`) + -- the first draft wrongly gave it `num_units`/`to`/an `amqp` relation + that doesn't exist on this charm at all; (3) `ceph-rbd-mirror` is a + principal with `ceph-local`/`ceph-remote` requires endpoints (confirmed + `subordinate: false`) -- the first draft invented a single wrong binding + key. **A genuine, unresolved cross-DC wiring limit documented rather + than worked around:** `ceph-remote` (the peer DC's ceph-mon) cannot be + a same-bundle relation given D-104's per-DC-independent-Juju-controller + design -- only `ceph-local` (this DC's own leg) is wired; the cross-DC + peer relationship needs `juju offer`/`consume` or the manual `rbd + mirror pool peer bootstrap` CLI flow Stage 6's runbook already + documents. Deliberately did NOT reactivate `os-public-hostname` on the + 11 API charms in this shared bundle (D-106's OTHER coupled change) -- + pushed to a proposed per-DC overlay instead, to keep `bundle.yaml` + itself DC-agnostic. Verified: `python3 -c "import yaml; yaml.safe_load( + open('bundle.yaml'))"` parses clean (56 apps, 108 relations); `python3 + scripts/provider-bundle-check.py` PASS, all 6 existing invariants + unaffected (`tests/provider-bundle-check/`: 8/8 unaffected); `bash + scripts/repo-lint.sh` 0 fail/1 documented warn. Reconciled Stage 6's and + Stage 7's runbooks to point at the now-real bundle content instead of + their earlier "template"/"known gap" framing. NOT applied to any live + model this session -- no live cloud reachable; this is a repo-source + change only, to be reviewed via `juju deploy --dry-run` before any real + apply. Uncommitted -- orchestrating session will assign the real DOCFIX + number at integration time. +- **DONE (uncommitted) -- Tooling gap register item #5 MECHANISM CLOSED + 2026-07-09 (changelog `docs/changelog-20260709-ceph-replication- + tooling.md`, DOCFIX-165 -- numbering deferred to the orchestrating + session, same reason as the other parallel deliveries tonight):** the + radosgw multisite / rbd-mirror command sequences already written out in + `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and + 11.2-11.4 (verbatim upstream Ceph administration, not invented) are now + real, callable scripts: `scripts/dc-dc-radosgw-multisite.sh` + (`master-init`/`join-readonly`/`enable-two-way`), `scripts/dc-dc-rbd- + mirror.sh` (`bootstrap-primary`/`bootstrap-secondary --direction + rx-only|rx-tx`), and `scripts/dc-dc-dr-drill.sh` (`failover`/`failback`). + Every realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument + -- no invented defaults, matching the runbook's own `` + placeholder discipline. Default mode is `--dry-run` (prints the exact + `juju ssh`/`juju run` command sequence, secrets redacted); `--apply` + executes one command at a time, aborting on first failure. **The + single most safety-critical property in the whole drill -- demote the + CURRENT PRIMARY before promoting the RECOVERING side during failback -- + is HARD-CODED as fixed call order in `dc-dc-dr-drill.sh`'s `failback` + subcommand**, not left to documentation or an operator-orderable flag; + in apply mode the promote step is only ever reached if the preceding + demote succeeded. Step 10.2's per-image Glance re-registration is always + printed as a flagged MANUAL reminder and NEVER auto-executed (the + runbook is explicit the real invocation depends on unconfirmed Glance + driver behavior); Step 11.3's Cinder reconciliation is likewise not + scripted (tenant/volume-specific). Each script sources `scripts/lib- + net.sh` and calls `lib_net_select_dc` per DC token touched (DOCFIX-151 + convention) -- informational in dry-run (a plan preview is not a + mutation), hard-blocking before any `--apply` (dc2 --apply refused + today, exit 3, since D-101/gap #3 leaves DC2 with no assigned network + literals). New harnesses `tests/dc-dc-radosgw-multisite/`, + `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23 + PASS) test ONLY argument parsing, guard clauses, and dry-run output -- + including a line-order grep proving the failback demote-before-promote + property -- NOT real radosgw/rbd-mirror behavior; no live Ceph cluster + exists this session to test against (same UNVALIDATED posture as every + other tooling delivery tonight). Stage 6 runbook's own "Follow-up, NOT + built now" delivery-checklist item and `docs/dc-dc-deployment- + workflow.md` item #5 both updated to point at these scripts instead of + describing them as future work. `bash scripts/repo-lint.sh`: 0 fail, 1 + documented legacy warn (unchanged). No live infrastructure touched. + Uncommitted -- orchestrating session will assign the real DOCFIX number + and find-replace `DOCFIX-165` at integration time. +- **DONE (uncommitted) -- Gaps #4(d)/#11: proposed netem parameters + ULA/ + GUA/DC2-supernet generation guidance (changelog `docs/changelog- + 20260709-netem-ula-gua-proposal.md`, DOCFIX-168), `docs/dc-dc-netem-and-ula-gua- + proposal.md`:** does NOT rule the netem decision or generate any of the + three real NetBox literals itself -- presents a concrete, reasoned + proposal (1ms delay / 0.2ms jitter / 0.01% loss / uncapped rate, each + with rationale grounded in the buildout design's own "same-metro dark + fiber, low single-digit ms" lean) for OPERATOR ratification, and gives + the exact safe command (`openssl rand -hex 5`, RFC 4193-compliant) for + the operator to generate the real org ULA /48 themselves. Explicitly + does NOT attempt to generate the per-DC GUA carve or DC2's supernet -- + both require real coordination (the ARIN-block administrator; NetBox's + full address inventory) this session has no authority or visibility to + perform; the section instead explains precisely what each requires and + how to close it once ready. `docs/dc-dc-deployment-workflow.md` gap + items #4(d)/#11 updated to reference it; item #10 (harness debt) also + corrected -- LARGELY CLOSED given how many new test suites landed + tonight (dc-selector, dc-dc-prefixes-import, dc-dc-mtu-geneve-budget, + dc-dc-ceph-disk-budget, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, + dc-dc-dr-drill, plus MAAS script additions). `bash scripts/repo-lint.sh`: + 0 fail, 1 documented legacy warn. No live infrastructure touched. + Uncommitted -- orchestrating session will assign the real DOCFIX number + at integration time.