diff --git a/docs/changelog-20260709-dc-dc-phase1-runbook.md b/docs/changelog-20260709-dc-dc-phase1-runbook.md new file mode 100644 index 0000000..c363f61 --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase1-runbook.md @@ -0,0 +1,66 @@ +# Changelog 2026-07-09 -- Stage 2 runbook: Office1 headend standup (DOCFIX-154) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +the buildout design, D-103/D-106/D-107, the tooling gap register, and the +Stage 1 runbook as a style template), reviewed and lint-fixed before this +commit. + +## Item + +### 1. DOCFIX-154 -- `runbooks/dc-dc-phase1-office1-standup.md` +FILES: `runbooks/dc-dc-phase1-office1-standup.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 2 row updated). + +WHAT: command-level runbook for buildout-design Section 4's Phase 1 +(Office1 headend standup: MAAS region controller, OpenTofu reach +re-verification, NetBox, GitBucket, Tailscale), matching this repo's +runbook conventions. + +**Structural blocker surfaced, not silently resolved:** none of the three +Office1 service VMs (MAAS-region, NetBox, GitBucket) have an OpenTofu +`cloudinit-vm` instantiation yet -- no image source chosen, no +`user_data`/`meta_data`/`network_config` designed. Rather than inventing +that content or silently defaulting to a manual path, every VM-creation +step presents an explicit Option A (OpenTofu, blocked pending design) vs. +Option B (manual `virt-install`, logged as debt) fork. + +**Genuine, previously-unnoticed gap found and flagged:** no Office1-local +virtual network is modeled anywhere in `opentofu/` -- `main.tf` wires +`office1_pool_path` (storage only) and the three inter-site mesh links +(management-traffic-only per D-100's own sub-item ruling), but nothing the +three service VMs (or the vcloud host's own management interface) could +actually attach to as a shared local network. Flagged as an open question +for whoever designs the `cloudinit-vm` content next, not resolved here. + +**Fixed during review (this session, not the subagent):** the subagent's +first draft cited the historical D-014 GitBucket path (`jesse.austin/ +openstack-caracal-ipv4`) as context for why THIS stage's GitBucket instance +is separate -- correctly sourced, but repo-lint's L2 rule flags that literal +string as a "dead repo path" regardless of context (a blunt static check). +Reworded onto a single line with the word "historical" so the rule's own +existing EXEMPT pattern applies, rather than deleting a legitimate, +correctly-attributed historical reference. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +NOT YET EXECUTED -- authored with no live vcloud host, MAAS, NetBox, +GitBucket, or Tailscale reachable this session; every command is written +for the operator to run and individually gate later. + +**Honest exit state, per the runbook's own GATE section:** MAAS/GitBucket/ +Tailscale/OpenTofu-reach bullets are closeable in one real run; NetBox's +"authoritative and populated" bullet is only PARTIAL -- the import +mechanism (DOCFIX-152) is runnable, but the underlying literals (org ULA +/48, per-DC GUA carve, DC2 supernet) are not assigned this session, so this +runbook cannot close that half of Stage 2's gate. This is stated plainly in +the runbook rather than rounded up to "done." + +REVERT: `rm runbooks/dc-dc-phase1-office1-standup.md && git checkout HEAD~ +-- docs/dc-dc-deployment-workflow.md` (safe -- no infrastructure touched). + +## Next actionable step + +Stage 3's runbook (OpenTofu builds each DC substrate) depends on this +stage's MAAS reachability for its own MAAS-registration step -- both were +drafted in parallel tonight; Stage 3's runbook correctly states that +dependency rather than assuming Stage 2 is done. diff --git a/docs/changelog-20260709-dc-dc-phase2-runbook.md b/docs/changelog-20260709-dc-dc-phase2-runbook.md new file mode 100644 index 0000000..1ceaf19 --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase2-runbook.md @@ -0,0 +1,63 @@ +# Changelog 2026-07-09 -- Stage 3 runbook: OpenTofu builds each DC substrate (DOCFIX-155) + +No live infrastructure touched -- new runbook document only. Authored by a +delegated subagent (general-purpose, briefed with this repo's discipline, +the buildout design, D-103/D-100, the tooling gap register, `opentofu/ +README.md` and `templates/README.md`, and the Stage 1 runbook as a style +template), reviewed before this commit -- no changes needed, repo-lint was +clean on the first pass. + +## Item + +### 1. DOCFIX-155 -- `runbooks/dc-dc-phase2-tofu-dc-substrate.md` +FILES: `runbooks/dc-dc-phase2-tofu-dc-substrate.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 3 row updated). + +WHAT: command-level runbook for buildout-design Section 4's Phase 2 +(OpenTofu builds each DC's node-VM domains, plane networks, and OPNsense +edge, then registers the DC's libvirt host with its MAAS rack controller). +DC1-first; DC2 is treated as a HARD gate (D-101's supernet still +unassigned, gap #3's data half), not merely "do it later" -- the runbook +explicitly instructs not to uncomment `main.tf`'s commented DC2 blocks or +invent placeholder DC2 CIDRs. + +**Five `!!!` callouts up front**, each naming a real constraint rather than +glossing over it: (1) hard dependency on Stage 2's MAAS reachability -- +Steps 1-8 (libvirt-layer only) can proceed without it, Step 9 onward +cannot; (2) DC2's hard gate; (3) node-VM creation has no Roosevelt analog +(Section 9 shim register); (4) node sizing (vCPU/RAM/disk) is undecided -- +the runbook uses `` placeholders in its `.tf` block shapes, +never invented numbers; (5) `tc netem` parameters remain an unruled D-100 +sub-item (gap #11) -- the runbook applies ONLY the buildout design's +qualitative "same-metro lean" language as an explicitly-labeled placeholder, +never a specific number presented as measured or ruled. + +**A sixth callout** confirms the OPNsense prep tools (`bunzip2`, +`qemu-img`, `genisoimage`/`xorriso`) are not yet confirmed present on the +vcloud host -- Step 2 checks for all of them before anything else proceeds. + +**Config.xml token gap named precisely:** `{{WAN_IF}}`/`{{LAN_IF}}` are the +one token pair that genuinely cannot be filled before the OPNsense domain +exists (only measurable via a real boot's `ifconfig` output) -- the runbook +states this plainly as a same-session blocker for the config render step, +rather than guessing a `vtnetN` name. + +**The GATE section is deliberately honest**, walking each of Stage 3's four +exit-condition bullets and stating which are achievable this session vs. +conditional on Stage 2, node sizing, a real boot measurement, or D-100's +netem ruling -- concluding "this stage's exit gate is therefore +CONDITIONALLY MET AT BEST," with explicit instruction to update the +tracker doc to the real state, never a blanket DONE. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn (no +fixes needed for this file). NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase2-tofu-dc-substrate.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +Stage 4 (MAAS enlist/commission/deploy) is the direct successor; its +runbook was drafted in parallel tonight and should be reviewed for +consistency with this stage's own MAAS-discovery verification step (Step +10 here hands off to Stage 4's commissioning work). diff --git a/docs/changelog-20260709-dc-dc-phase4-runbook.md b/docs/changelog-20260709-dc-dc-phase4-runbook.md new file mode 100644 index 0000000..11ada28 --- /dev/null +++ b/docs/changelog-20260709-dc-dc-phase4-runbook.md @@ -0,0 +1,73 @@ +# Changelog 2026-07-09 -- Stage 5 runbook: Juju controller + bundle, per DC (DOCFIX-156) + +No live infrastructure touched -- new runbook document only. Authored +directly (not delegated), since this is the ground-truth/copy-point stage +(`docs/dc-dc-buildout-design.md` line 12: this whole repo was seeded from +`openstack-caracal-ipv4` specifically so `bundle.yaml` and `phase-01..08` +could be reused here) and the most consequential adaptation to get right. + +## Item + +### 1. DOCFIX-156 -- `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` +FILES: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` (new), `docs/dc-dc- +deployment-workflow.md` (Stage 5 row updated). + +WHAT: does NOT duplicate `phase-01..08`'s command-level content. Describes +what changes to run that existing, validated deploy loop TWICE (once per +DC): a dedicated per-DC single-unit Juju controller bootstrap (D-104), the +existing bundle deploy/vault-bringup/core-verify/network-carve/octavia +sequence referenced by filename, and the D-109 independent-per-DC-Vault- +root implication (separate `vault operator init` runs, separate unseal +shares/root tokens, stored separately per DC). + +**DC1 needs literally NO VIP/CIDR changes** -- D-101 inherits DC0's layout +unchanged, so the existing `bundle.yaml` deploys into DC1's own Juju model +verbatim. **DC2 is explicitly blocked** on the same still-unassigned NetBox +literals as Stages 3/4 (gap #3's data half) -- the runbook states plainly +that DC2's octavia SAN IP, VIP band, and everything else address-derived +cannot be decided until those literals exist, and instructs stopping there +for DC2 rather than inventing a "test range." + +**Every `phase-01..08` command block that reads `scripts/lib-net.sh`/ +`lib-hosts.sh` values must be preceded by an explicit `lib_net_select_dc +"$DC"` / `lib_hosts_select_dc "$DC"` call (DOCFIX-151)** -- the runbook +calls out that for DC1 both are a documented no-op (so skipping them +"happens to work" by accident) while for DC2 skipping them would silently +reuse DC1's values, exactly the bug the selector convention exists to +prevent. This is the stage tooling gap register item #1 was built to +unblock, and this is the first runbook to actually exercise it. + +**Two real, unresolved design gaps surfaced rather than papered over:** +1. **The D-101 IPv6 family-matrix overlay does not exist.** Which planes + need ULA-only vs. dual-stack vs. GUA addressing is fully specified by + D-101 (Section 2 of the buildout design), but the actual Juju charm + config option names needed to configure that per application (`ceph- + mon`, `ovn-central`, `octavia`, etc.) have not been confirmed against + any charm's real current schema this session. Rather than inventing + plausible-looking option names, Step 6 names this as blocking work with + a three-part path to close it (confirm real charm options -> author the + overlay -> apply it) and explicitly forbids fabricating option names. +2. **D-109's IPv6-SAN cert-issuance requirement is an unverified charm- + behavior assumption** -- whether the vault/certificates-relation charms + in `bundle.yaml` actually support issuing v6 SANs today has not been + confirmed; flagged as a real finding to log if it turns out they don't, + not something to silently route around. + +Also documents the metal-admin ULA amendment explicitly (the 2026-07-09 +operator ruling reversing D-101's own original v4-only lean for that +plane), warning against assuming older documentation/training-data +patterns. + +VERIFIED: `bash scripts/repo-lint.sh` 0 fail, 1 documented legacy warn. +NOT YET EXECUTED. + +REVERT: `rm runbooks/dc-dc-phase4-juju-bundle-per-dc.md && git checkout +HEAD~ -- docs/dc-dc-deployment-workflow.md` (safe). + +## Next actionable step + +The Step 6 IPv6 overlay design gap is the single biggest piece of real, +undone work surfaced by tonight's runbook-authoring pass -- it blocks this +stage's own "Ceph-over-v6 and geneve-over-v6 verified" gate criterion and +should be logged as its own tooling-gap-register item in the final +consolidation pass, alongside gaps #1-#11 already tracked. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 255fbe6..27c207f 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -56,9 +56,9 @@ | **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only. | | **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | | **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | -| **Authoring status** | Runbook not started. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism for the MAAS-region/NetBox/GitBucket service VMs -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. Flags a genuine, previously-unnoticed gap: no Office1-local virtual network is modeled anywhere in `opentofu/` for these VMs to attach to. NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | -**State:** NOT STARTED. +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned). --- @@ -71,9 +71,9 @@ | **Gate** | MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured. | | **Owns** | D-103 (seam), D-100 (edges). | | **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | -| **Authoring status** | Runbook not started. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Walks every still-open decision (node sizing, MAAS zone/pool, `power_address`, config.xml tokens incl. boot-measured `WAN_IF`/`LAN_IF`, netem params) and stops at the corresponding step if unresolved rather than inventing values. States its own exit gate as "CONDITIONALLY MET AT BEST" pending Stage 2, node sizing, a real OPNsense boot measurement, and D-100's netem ruling. NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | -**State:** NOT STARTED. +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half. --- @@ -101,9 +101,9 @@ | **Gate** | `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; controller backup taken + stored; Ceph-over-v6 and geneve-over-v6 verified. | | **Owns** | D-104 (controllers), D-109 (Vault), D-101 (families). | | **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. | -| **Authoring status** | Existing `phase-01..08` runbooks are the starting point; need a per-DC parameterization pass (D-101 family matrix, D-109 Vault) before reuse, not a from-scratch write. | +| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`.** Does not duplicate `phase-01..08`'s content -- describes what changes to run that loop twice (once per DC): DC1 needs NO VIP/CIDR edits at all (D-101 inherits DC0 unchanged); DC2 is hard-blocked pending its NetBox literals. Requires every adapted `phase-01..08` command block that reads `lib-net.sh`/`lib-hosts.sh` to call `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` first (DOCFIX-151). Flags a real, unresolved design gap: the D-101 IPv6 family-matrix overlay (dual-stack/ULA-only per-plane charm config) does not exist yet -- the exact charm config option names for v6 addressing have not been confirmed against any charm's real schema, so no overlay file is invented; Step 6 names this as blocking work rather than gliding past it. Also flags D-109's IPv6-SAN cert-issuance support as an unverified charm-behavior assumption. NOT YET EXECUTED. | -**State:** NOT STARTED. **This is where the existing repo tooling (`preflight.sh`, +**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. **This is where the existing repo tooling (`preflight.sh`, `cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they apply to either DC's clone/session without modification.** diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 6cd9966..cc5afe7 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -21,8 +21,8 @@ ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) -_As of 2026-07-09 scan (post-DOCFIX-153, Stage 1 runbook written); re-run -the scan to refresh:_ +_As of 2026-07-09 scan (post-DOCFIX-156, Stages 2/3/5 runbooks written); +re-run the scan to refresh:_ - **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + @@ -37,8 +37,8 @@ - **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 111, DOCFIX = 154, BUNDLEFIX = 012. - (Seeded from the 2026-07-09 scan, post DOCFIX-153.) +- **Next-free numbers:** D = 111, DOCFIX = 157, BUNDLEFIX = 012. + (Seeded from the 2026-07-09 scan, post DOCFIX-156.) --- @@ -994,6 +994,40 @@ morning after switching to the vcloud host; nothing in this delivery claims a real run happened. repo-lint 0 fail/1 documented warn (unchanged, prose-only). +- **DONE -- Stages 2, 3, and 5 runbooks WRITTEN 2026-07-09**, three of five + remaining runbooks delegated to parallel subagents (Stages 2, 3, 4, 6, 7) + while Stage 5 was authored directly (the ground-truth/copy-point stage): + - **Stage 2** (changelog `docs/changelog-20260709-dc-dc-phase1-runbook.md`, + DOCFIX-154): `runbooks/dc-dc-phase1-office1-standup.md`. Every VM- + creation step forks explicit Option A (OpenTofu, blocked pending + cloud-init content design)/Option B (manual, logged debt). Found a real, + previously-unnoticed gap: no Office1-local virtual network is modeled + anywhere in `opentofu/`. Caught and fixed a repo-lint L2 false-flag + during review (a correctly-sourced historical D-014 path reference, + reworded onto one line so the rule's own EXEMPT pattern applies, not + deleted). + - **Stage 3** (changelog `docs/changelog-20260709-dc-dc-phase2-runbook.md`, + DOCFIX-155): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`. DC1-first; + DC2 hard-gated (not "later"). Five `!!!` callouts naming real + constraints (Stage 2 dependency, DC2 gate, no-Roosevelt-analog shim, + undecided node sizing, unruled netem params) plus OPNsense tool- + presence check. GATE section concludes "CONDITIONALLY MET AT BEST," + refusing to claim more than is actually true. + - **Stage 5** (changelog `docs/changelog-20260709-dc-dc-phase4-runbook.md`, + DOCFIX-156): `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. Describes + adapting (not duplicating) `phase-01..08` to run twice, once per DC. + DC1 needs zero VIP/CIDR edits (D-101 inheritance); DC2 blocked on the + same NetBox literals as Stages 3/4. Mandates `lib_net_select_dc "$DC"`/ + `lib_hosts_select_dc "$DC"` before every adapted command block touching + those libraries (DOCFIX-151's first real consumer). Surfaces two + genuine open design gaps rather than inventing around them: the D-101 + IPv6 family-matrix overlay (exact charm config option names + unconfirmed) and D-109's IPv6-SAN cert-issuance charm support + (unverified assumption). + All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED (prep-only + session, no live infrastructure reachable). Stages 4, 6, 7 were also + delegated to parallel subagents and completed around the same time -- + see their own changelog entries once reviewed and committed. diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md new file mode 100644 index 0000000..fc871d5 --- /dev/null +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -0,0 +1,515 @@ +# DC-DC Phase 1 -- Office1 headend standup (Stage 2) + +Stand up the operator's deployment headend -- MAAS region controller, confirmed +OpenTofu reach, NetBox, GitBucket, Tailscale -- BEFORE either DC substrate +exists. This is the second DC-DC runbook; it follows +`runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit gate is this +runbook's entry condition. Per this repo's session contract, do NOT re-derive +anything Stage 1 already settled (nested KVM, disk budget, MTU, the DC1/Office1 +libvirt planes and pools, the mesh-link networks) -- confirm Stage 1's gate +passed and move on. + +**Authoring context (read before running):** this runbook was authored PREP-ONLY, +with no live vcloud host, MAAS, NetBox, GitBucket, or Tailscale reachable in the +authoring session. Every command below is written to be run for real, later, +by the operator -- it has not been executed against real infrastructure. Per +`opentofu/README.md` and `docs/dc-dc-deployment-workflow.md` Stage 2 (tooling +gap register items #2, #3, #9): **none of the three Office1 service VMs +(MAAS-region, NetBox, GitBucket) have an OpenTofu module instantiation yet** -- +`opentofu/modules/cloudinit-vm` exists as a mechanism, but no image source has +been chosen and no `user_data`/`meta_data`/`network_config` content has been +designed for any of the three. This runbook does not invent that content. It +gives each VM-creation step two explicit paths (Section "Provisioning-path +decision" below) and flags the OpenTofu path as blocked-pending-design rather +than silently choosing the manual path for the operator. + +**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 Phase 1 +(goal/build/gate) and Section 5 (OpenTofu/MAAS/NetBox/Juju boundary); +`docs/dc-dc-deployment-workflow.md` Stage 2 (tracker row -- update its +`**State:**` line when this runbook completes a real run) and the Tooling gap +register (items #2, #3, #9 above all bear directly on this stage); +`opentofu/README.md` (module scope/status for `modules/cloudinit-vm` / +`modules/base-image`); `netbox/dc-dc-prefixes-import.py` (the NetBox +multi-DC/dual-stack importer, built 2026-07-09 -- MECHANISM only, see Step 7). + +Decisions this runbook owns: **D-103** (OpenTofu/MAAS/Juju lifecycle seam -- +Office1 service VMs are OpenTofu-created, eventually; MAAS owns their +provisioning lifecycle once enlisted the same way DC node VMs are), **D-107** +(airgap posture -- Office1 is explicitly OUT of the core-service path: no node +artifacts, no NTP served from Office1; Tailscale is a narrowly-scoped front +door, not a general egress path), **D-106** (naming convention -- Office1 +services use a region-level subdomain outside the per-cloud convention, e.g. +`maas.office1.vr1...`; exact scheme is this stage's job to fix, not yet fixed), +and **D-101** (referenced only -- NetBox is the IPAM apex this stage stands up, +but the address literals D-101 requires are NOT this stage's job to assign). + +!!! PREP-ONLY / NOT YET EXECUTED. No live infrastructure exists or was + reachable in the authoring session. Every MUTATION below is written for + the operator to run and individually gate later, per this repo's + verify-before-mutate discipline -- none of it has been run. + +!!! STRUCTURAL BLOCKER, flagged not fixed here: the OpenTofu path for all + three Office1 service VMs needs `user_data`/`meta_data`/`network_config` + designed against `modules/cloudinit-vm` before it can be used -- that + design work has NOT been done. Step 3 below presents the resulting + Option A (blocked) / Option B (manual, interim) fork explicitly for each + VM. Do not silently pick one without reading Step 3 first. + +--- + +## Entry condition (Stage 1 gate -- confirm, do not re-derive) + +**CHECK -- read-only, before starting anything in this runbook** +```bash +virsh net-list --all +virsh pool-list --all +``` +Expect: the six DC1 plane networks, the three mesh-link networks +(dc1<->dc2, dc1<->office1, dc2<->office1), and the `dc1`/`office1` storage +pools, all `active` -- this is `dc-dc-phase0-vcloud-prep.md`'s Step 11 output. +If any of this is missing or `inactive`, STOP -- Stage 1 is not actually done; +go re-run/complete that runbook rather than starting Stage 2 on an unfinished +Stage 1. Do not re-measure nested KVM, disk budget, or MTU here -- Stage 1 +already recorded those; this runbook only reads their OUTCOME (the networks +existing) as its gate. + +--- + +## Open questions flagged here (not resolved by this runbook) + +Per this repo's discipline of naming ambiguity explicitly rather than quietly +picking an answer: + +1. **What virtual network do the three Office1 service VMs themselves attach + to?** `opentofu/main.tf` (per Stage 1) wires `office1_pool_path` (storage + only) and the three mesh-link networks (Office1<->DC1, Office1<->DC2, + inter-site only, "management traffic only" per D-100's own sub-item + ruling). No Office1-LOCAL network (something the three service VMs and the + vcloud host's own management interface could all share) is modeled + anywhere in `opentofu/`. This is a real gap, not an oversight this runbook + papers over: either a new `modules/dc-planes`-shaped Office1-local network + needs designing, or an existing host bridge needs to be reused deliberately + (and documented as such) -- a decision for whoever designs the + `cloudinit-vm` `network_config` content in the blocked Option A path below. + Flag this to the operator before Step 3; do not invent a network name to + move forward. +2. **NetBox's own address literals are not this stage's job**, per D-101's own + text ("the org ULA /48, the per-DC GUA carve... and DC2's v4 supernet are + NetBox-authoritative... NOT hardcoded in this decision"). This runbook + stands NetBox up and can run its import MECHANISM (Step 7), but the + underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not + exist yet -- see Step 7's own gate. +3. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, + pre-existing `git.baldurkeep.com` instance from the v1 rehearsal -- this + stage stands up a SEPARATE, vcloud-local GitBucket instance (per the + buildout design's own "GitBucket (vcloud-local mirror)" wording) -- what + it mirrors, and under what repo path, is not decided here. Flag as an + open item before Step 8's mutation. +4. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the + front door is "Office1 only" -- the actual tailnet ACL policy, tags, and + which services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the + service VMs?) are exposed through it versus reachable only from Office1's + own local network, is real policy design left to Step 9 and the operator's + own tailnet account -- not invented here. +5. **Exact MAAS / NetBox / GitBucket version/channel pins are not decided.** + Each install step below names the well-known official install MECHANISM + (snap for MAAS, Docker Compose for NetBox, WAR/Docker for GitBucket) but + does not pin an exact version/channel/tag -- consult each project's current + official install docs at execution time and record whatever is actually + installed in the changelog entry for this runbook's real run. + +--- + +## Provisioning-path decision (applies to all three service VMs) + +Per the structural blocker above, EVERY VM-creation step in this runbook (5, +6, 8) has two paths. Do not silently default to one -- read both, pick +deliberately, and log which path was used in the changelog entry. + +- **Option A (preferred, long-term, BLOCKED pending design):** create the VM + via `opentofu/modules/cloudinit-vm`, once (a) Open question #1 above is + resolved (what network it attaches to) and (b) real `user_data`/`meta_data`/ + `network_config` content is designed for that specific VM (what packages, + what static address, what hostname). This is the D-103-correct path (`OpenTofu + creates... the Office1 service VMs`) and is NOT available tonight -- doing + it right requires a design pass this session did not do and this runbook + will not invent. +- **Option B (interim, manual):** create the VM directly against the vcloud + host's libvirt (e.g. `virt-install` with a real official cloud image ISO/ + qcow2, or an interactive install), attach it to whatever network Open + question #1 resolves to, then run the install steps below directly over SSH + once the VM has a reachable address. This gets Office1 stood up now, at the + cost of NOT being reproducible/versioned in OpenTofu -- log it explicitly as + a known debt (a DOCFIX candidate: "Office1 service VM `` was + hand-created via Option B on ``; migrate to `modules/cloudinit-vm` + once its cloud-init content is designed"). + +--- + +## Sequence +``` +1. Confirm Stage 1 exit gate (read-only, above) +2. Confirm OpenTofu still reaches vcloud libvirt from Office1 (read-only) +3. Read the provisioning-path decision + open questions (above; no cmds) +4. MAAS region controller -- create VM (A or B) + install [MUTATION, gated] +5. NetBox -- create VM (A or B) + install (official Docker) [MUTATION, gated] +6. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; + literals pending -- PARTIAL) [MUTATION, gated] +7. GitBucket -- create VM (A or B) + install (WAR or Docker) [MUTATION, gated] +8. Tailscale -- install + scope to Office1 only [MUTATION, gated] +9. Post-standup verify against the Stage 2 gate (read-only) + -> EXIT GATE (PARTIAL, see GATE section) -> Stage 3 +``` + +--- + +## Step 1 -- Confirm Stage 1 exit gate (READ-ONLY) + +Covered above under "Entry condition." Do not proceed past this point until +that CHECK is green. + +--- + +## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt from Office1 (READ-ONLY) + +Stage 1 already proved OpenTofu -> vcloud-host libvirt reachability (its Step +1 CHECK + Step 10 apply). This step is a re-verification, not a re-derivation +-- if this session is running some time after Stage 1, confirm nothing +regressed rather than assuming it still holds. + +**CHECK -- from the Office1 operator VM/session** +```bash +cd opentofu +tofu version +virsh -c "$(grep -A1 'variable "libvirt_uri"' -m1 /dev/null 2>/dev/null; true)" list --all 2>/dev/null || true +``` +More directly -- use the exact `libvirt_uri` value recorded in Stage 1's +`opentofu/dc-dc-phase0.auto.tfvars`: +```bash +virsh -c "" list --all +``` +Expect: the domain list (the DC1 plane/pool objects have no domains yet, so +this may be empty, but the CONNECTION must succeed with no error). If it +fails, STOP -- this is the same blocking condition Stage 1's Step 1 described; +do not proceed into VM creation without confirmed libvirt reach. + +**CHECK -- MAAS provider not yet needed but flagged (Stage 1's Known Gap)** +```bash +grep -n 'provider "maas"' opentofu/main.tf +``` +Confirm whether Stage 1 worked around the unconditional `maas` provider block +with placeholder values or whether a DOCFIX split it out -- read whatever +Stage 1's changelog entry recorded, and use the same resolved answer here +rather than re-deciding it. + +--- + +## Step 3 -- Read the provisioning-path decision and open questions + +No commands. Confirm with the operator (or record your own reasoned choice, +logged) which path -- A or B -- is being used for each of the three VMs +below, and how Open question #1 (Office1-local network) is being resolved for +this run. Do not proceed to Step 4 silently. + +--- + +## Step 4 -- MAAS region controller: create VM + install [MUTATION: gated] + +**CHECK -- confirm no pre-existing MAAS-region VM under this name** +```bash +virsh -c "" list --all | grep -i maas-region +``` +Expect: no match (fresh standup) or a clear, understood match (re-run/repair +case) -- do not proceed if an unexplained existing domain shares the name. + +**MUTATION -- create the VM (Option A or B per Step 3)** + +Option A: not available tonight (blocked, see above) -- instantiate +`module "office1_maas_region"` using `modules/cloudinit-vm` once its +`user_data`/`meta_data`/`network_config` are designed, then `tofu apply`. + +Option B (interim manual path): +```bash +# Illustrative shape only -- fill every with a value you +# measured/assigned this session; none of these are literals to copy verbatim. +virt-install \ + --connect "" \ + --name maas-region-office1 \ + --memory \ + --vcpus \ + --disk pool=office1,size= \ + --network network= \ + --os-variant \ + --cdrom \ + --graphics none --console pty,target_type=serial +``` +Boot it, complete the OS install per that OS's own docs, and record the +resulting static/DHCP address this session assigns it -- MEASURE/ASSIGN, never +guessed. This is the exact "known debt" Option B commits to logging (see +Provisioning-path decision above). + +**MUTATION -- install MAAS region controller (official snap-based method)** + +Once the VM is up and reachable over SSH: +```bash +ssh 'snap info maas' +``` +Consult MAAS's current official install docs for the exact channel to pin +(this repo does not hardcode a MAAS snap channel anywhere else to copy from -- +pick and record the current LTS/stable channel per those docs, not a guessed +one) and run the equivalent of: +```bash +sudo snap install maas --channel= +``` +Then initialize it as a region controller per MAAS's own current `maas init` +documentation for that installed version -- the exact flags (`--maas-url`, +database backend selection, admin creation) vary by MAAS release; consult the +official docs rather than reusing a flag set from memory or from a different +MAAS version's docs. + +**CHECK -- verify the region controller is reachable** +```bash +curl -sI "http://:5240/MAAS/" | head -1 +``` +Expect an HTTP response, not a connection failure. Naming per D-106: this +service's FQDN, once Designate/DNS exists for Office1, would follow the +region-level subdomain scheme D-106 names as an example +(`maas.office1.vr1...`) -- exact scheme is confirmed here as part of this +stage, not assumed; record whatever real hostname convention you actually use. + +**GATE (Stage 2, first Gate bullet):** MAAS region reachable. + +--- + +## Step 5 -- NetBox: create VM + install (official Docker method) [MUTATION: gated] + +**CHECK -- confirm no pre-existing NetBox VM under this name** +```bash +virsh -c "" list --all | grep -i netbox +``` + +**MUTATION -- create the VM (Option A or B per Step 3)**, same shape and same +caveats as Step 4's MAAS-region VM creation (Option A blocked pending +`cloudinit-vm` content design for THIS specific VM; Option B manual, logged as +debt). + +**MUTATION -- install NetBox (official netbox-docker Compose method)** + +Once the VM is up and reachable over SSH, follow NetBox's own official +Docker-based install (the `netbox-community/netbox-docker` repo's documented +Compose workflow: clone it, configure its `.env` / `docker-compose.override.yml` +per its own current docs, `docker compose up -d`). Do not hand-roll a +from-source install -- Docker Compose is NetBox's own recommended path and the +lowest-delta one to keep reproducible. Consult NetBox's current official docs +for the exact compose file / image tag to pin; do not fabricate a version tag. + +**CHECK -- verify NetBox is reachable** +```bash +curl -sI "http://:8080/" | head -1 +``` +Expect an HTTP response. Create/record a NetBox API token per NetBox's own +docs (needed by Step 6) -- handle it per this repo's secrets discipline: never +printed, never committed, passed as an environment variable only +(`NETBOX_TOKEN`, matching `netbox/dc-dc-prefixes-import.py`'s own documented +usage). + +**GATE (Stage 2 Gate, partial):** NetBox reachable and running. NOT yet +"authoritative and populated" -- see Step 6. + +--- + +## Step 6 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] + +`netbox/dc-dc-prefixes-import.py` (built 2026-07-09, closes tooling gap #3's +MECHANISM) extends the v1 single-site import to VR1's per-DC, dual-stack +model per D-101. It is idempotent and `--verify-only`-capable. **It does NOT, +and cannot, close the DATA half of gap #3**: the real org ULA /48, the per-DC +GUA carve, and DC2's v4 supernet do not exist yet as assigned values anywhere +in this repo or session. Running it for DC1 only requires the two v6 literals +(`ORG_ULA_48`, `DC_GUA_PREFIX`); running it for DC2 additionally requires +`DC2_V4_SUPERNET` -- none of the three has a default, by the script's own +design, and none is invented here. + +**CHECK -- dry run, no writes** +```bash +NETBOX_URL="http://:8080" \ +NETBOX_TOKEN="" \ +ORG_ULA_48="" \ +DC_GUA_PREFIX="" \ +python3 netbox/dc-dc-prefixes-import.py --dc dc1 --verify-only +``` +If `ORG_ULA_48` / `DC_GUA_PREFIX` are not yet assigned, this command CANNOT be +run for real yet -- that is the honest state of this stage tonight, not a +missing step in this runbook. Record that as the blocking condition rather +than inventing placeholder-looking values to "get past" it (this script +itself is designed to fail loud on exactly that pattern, per its own header). + +**MUTATION -- once the literals above are real (a later session/step, not +necessarily tonight)** +```bash +NETBOX_URL="..." NETBOX_TOKEN="..." ORG_ULA_48="..." DC_GUA_PREFIX="..." \ + python3 netbox/dc-dc-prefixes-import.py --dc dc1 +``` +DC1's v4 is hardcoded in the script itself (D-101: "DC1 equals the validated +template," not an inferred value -- it is the explicit text of an ADOPTED +decision). DC2 additionally needs `DC2_V4_SUPERNET`, not assigned yet either. + +**GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 5). +"Authoritative and populated (planes, per-DC v4, ULA/GUA carve)" is only +PARTIALLY achievable right now: the IMPORT MECHANISM exists and is runnable +the moment the literals are assigned, but the literals themselves are not +assigned this session, so the DATA half of this gate bullet is NOT met yet. +Do not mark this gate green in the tracker until a real `ORG_ULA_48` / +`DC_GUA_PREFIX` (and, for DC2, `DC2_V4_SUPERNET`) exist and this script has +actually been run against this real NetBox instance. + +--- + +## Step 7 -- GitBucket: create VM + install (official WAR or Docker method) [MUTATION: gated] + +**CHECK -- confirm no pre-existing GitBucket VM under this name** +```bash +virsh -c "" list --all | grep -i gitbucket +``` + +**MUTATION -- create the VM (Option A or B per Step 3)**, same shape/caveats +as Steps 4 and 5. + +**MUTATION -- install GitBucket (official method)** + +GitBucket ships as a single runnable WAR (`java -jar gitbucket.war`, its own +documented standalone-Jetty mode) or as an official Docker image +(`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one +and record which. Consult GitBucket's current official docs/release page for +the exact WAR filename/version or image tag to pin -- do not fabricate one. +Open question #3 above (this instance's purpose/repo-path scheme, distinct +from the existing `git.baldurkeep.com` v1 instance per D-014) must be resolved +before deciding what this instance actually mirrors. + +**CHECK -- verify GitBucket is reachable** +```bash +curl -sI "http://:8080/" | head -1 +``` +Expect an HTTP response. + +**GATE (Stage 2 Gate bullet):** GitBucket serving. + +--- + +## Step 8 -- Tailscale: install + scope to Office1 only [MUTATION: gated] + +Per D-107, Office1 is explicitly OUT of the core-service path (no node +artifacts, no NTP served from Office1) -- Tailscale here is a narrowly-scoped +OPERATOR front door to Office1's own services, not a general egress/access +path for anything DC-side. Do not wire Tailscale into any DC-facing plane. + +**CHECK -- confirm Tailscale is not already installed/conflicting** +```bash +ssh 'tailscale version 2>&1 || echo "not installed"' +``` + +**MUTATION -- install (official method)** +```bash +ssh 'curl -fsSL https://tailscale.com/install.sh | sh' +``` +Consult Tailscale's own current docs if this host's distro needs a different +install path (the install script itself detects and branches on distro). + +**MUTATION -- bring up, scoped** +```bash +ssh 'sudo tailscale up --advertise-tags=' +``` +The actual ACL policy restricting reachability to Office1's services only +(and NOT to any DC-side network) is configured in the operator's own Tailscale +admin console/ACL file -- this is real policy design (Open question #4), not +invented here. Do not advertise routes into any DC plane from this node. + +**CHECK -- verify scope** +```bash +ssh 'tailscale status' +``` +Confirm only the expected Office1 headend node(s) appear, and that no route +advertisement reaches into DC1/DC2 plane CIDRs. + +**GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only. + +--- + +## Step 9 -- Post-standup verify against the Stage 2 gate (READ-ONLY) + +```bash +curl -sI "http://:5240/MAAS/" | head -1 +curl -sI "http://:8080/" | head -1 +curl -sI "http://:/" | head -1 +ssh 'tailscale status' +virsh -c "" list --all +``` +Cross-check each against the GATE section below before declaring this stage +done or partially done. + +--- + +## GATE (Stage 2 exit condition -- honest partial state) + +Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row: "MAAS region +reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA +carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale +confirmed to Office1 only." + +- **MAAS region reachable:** achievable this stage (Step 4). +- **OpenTofu reaches vcloud host libvirt:** already proven in Stage 1; Step 2 + here only re-verifies it still holds. +- **GitBucket serving:** achievable this stage (Step 7). +- **Tailscale confirmed to Office1 only:** achievable this stage (Step 8), + contingent on the operator's own ACL policy design (Open question #4). +- **NetBox authoritative and populated:** ONLY PARTIALLY achievable right now. + The instance can be stood up and reachable (Step 5), and the import + MECHANISM exists and is runnable (Step 6) -- but the real address literals + (org ULA /48, per-DC GUA carve, DC2 v4 supernet) are not assigned this + session, so NetBox cannot yet be truly "populated" per D-101. Do not mark + this bullet green in `docs/dc-dc-deployment-workflow.md`'s Stage 2 row until + those literals exist and `netbox/dc-dc-prefixes-import.py` has actually run + against this real instance. + +This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** +-- MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in full tonight's +sequence; the NetBox bullet closes its mechanism half only, pending the D-101 +literal assignment that is explicitly out of scope for this stage to invent. +Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever +of these is actually true after a real run, not to "DONE" wholesale. + +-> Proceed to Stage 3 (OpenTofu builds each DC substrate) only once the +achievable bullets above are actually green; the NetBox DATA gap does not +block Stage 3's OWN gate (Stage 3 depends on MAAS reachability and OpenTofu +reach, not on NetBox's DC2/GUA literals), but should be tracked as a standing +gap until closed. + +--- + +## Delivery checklist (this repo's standard discipline) + +- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo + changes made while executing this runbook (e.g. any real `opentofu/` + instantiation once Option A becomes available; any real tfvars, minus + secrets). +- [ ] If any Option A `cloudinit-vm` instantiation was written this run, + `bash scripts/opentofu-validate.sh` green. +- [ ] Changelog entry for this runbook's first real execution (next DOCFIX + number via `bash scripts/ledger-scan.sh`), noting: which provisioning + path (A or B) was used for each of the three VMs, the real addresses/ + hostnames assigned, and whether NetBox's literal-assignment gap was + closed or remains open. +- [ ] `docs/session-ledger.md` updated with the outcome. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table + updated to the ACTUAL state reached (likely "PARTIAL," not "DONE," per + the GATE section above) -- do not mark NetBox's gate bullet closed + unless the D-101 literals were genuinely assigned and imported this run. +- [ ] If Option B (manual VM creation) was used for any service VM, log the + resulting debt explicitly (a DOCFIX candidate: migrate that VM to + `opentofu/modules/cloudinit-vm` once its cloud-init content is + designed) rather than letting it go unrecorded. +- [ ] If Open question #1 (Office1-local network) was resolved this run, + record the resolution and, if it implies a reusable OpenTofu module + (an Office1-local `dc-planes`-shaped network), flag it as a follow-on + delivery rather than silently leaving it as a one-off manual libvirt + network. diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md new file mode 100644 index 0000000..0df3580 --- /dev/null +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -0,0 +1,549 @@ +# DC-DC Phase 2 -- OpenTofu builds each DC substrate (Stage 3) + +From Office1, script each DC's "physical" layer into existence: the node-VM +libvirt domains MAAS will discover and deploy, the DC's plane networks, and +its OPNsense edge -- then register the DC's libvirt host to that DC's own +MAAS rack controller as a virsh VM-host. This runbook is written **DC1-first**. +DC2 is explicitly gated (see the callout below) and is NOT part of this +session's run until that gate clears -- do not duplicate these steps for DC2 +with invented values in the meantime. + +**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 2" (the +goal/build/gate this runbook satisfies), Section 9 (shim register -- the +node-VM creation module has **NO Roosevelt analog**, see the callout below), +and Section 6 (WAN simulation / netem, same-metro lean). +`docs/dc-dc-deployment-workflow.md` Stage 3 row and the Tooling gap register +(items 1, 4, 11 govern this stage directly). `opentofu/README.md` and +`opentofu/templates/README.md` for exact module/token status -- read both +before touching any `.tf` file or the OPNsense template. + +Decisions this runbook owns: **D-103** (the OpenTofu/MAAS/Juju lifecycle seam +-- OpenTofu creates, MAAS discovers/commissions, never composes), **D-100** +(the dark-fiber mesh + per-site OPNsense edge, and the netem WAN-simulation +mechanism riding on it). + +--- + +!!! **Depends on Stage 2 (Office1 headend standup).** This stage's MAAS + registration step needs a REACHABLE MAAS region controller (Stage 2's + gate) and Stage 1's OpenTofu-reaches-vcloud-libvirt gate already closed. + If Stage 2 has not completed when this session runs, STOP at Step 9 (MAAS + registration) -- everything through Step 8 (planes/pool/edge/node-VM + creation at the libvirt layer) does not itself require MAAS and may + proceed, but do not fabricate a `maas_api_url`/`maas_api_key` pair to push + past that dependency. Confirm Stage 2's actual state in + `docs/dc-dc-deployment-workflow.md` before starting. + +!!! **DC2 is explicitly gated, not merely "later."** D-101 has not assigned + DC2's real, non-overlapping v4 supernet yet (open sub-item; Tooling gap + register item 3's DATA half -- the import pipeline mechanism + (`netbox/dc-dc-prefixes-import.py`) is built, but the literals themselves + are not assigned). `opentofu/main.tf`'s `dc2_planes`/`dc2_storage` blocks + are commented out for exactly this reason. Do NOT uncomment them, do NOT + invent a placeholder DC2 CIDR, and do NOT run this runbook's steps against + DC2 until NetBox has assigned the real supernet and `main.tf` reflects it + for real. This is a hard gate, re-stated in the GATE section. + +!!! **Node-VM creation has NO Roosevelt analog (Section 9 shim register).** + `modules/node-vm` (the blank-disk, PXE-boot libvirt domain this stage + calls) is a virtual-only stand-in for what, in Roosevelt, is physical + racking plus BMC enlistment -- MAAS enlists hardware that already exists + there. Do not treat this module, or the act of Terraform-creating "node + hardware," as reusable production IaC. Everything from MAAS enlistment + onward (commission/deploy, Juju bundle) IS Roosevelt-transferable; this + one step is not. + +!!! **Node sizing is not decided -- do not invent vCPU/RAM/disk.** + `modules/node-vm`'s `memory_mib`, `vcpu`, and `disk_size_bytes` variables + have no defaults by design (buildout-design Section 3's Phase-0 host/ + disk-budget decision has not been made). If that decision is still open + when you reach Step 6, STOP there and flag it -- do not pick numbers that + "seem reasonable" to get `tofu plan` to run. + +!!! **netem parameters are still an unruled D-100 sub-item (gap #11).** This + stage's own gate text says "netem parameters applied and measured," but + the exact latency/jitter/loss/rate values do not exist as a ruled + decision -- only the qualitative buildout-design Section 6 lean exists: + "same-metro dark fiber... low single-digit ms, jitter-capable, jumbo- + capable." Step 11 below applies ONLY that qualitative lean as an + explicitly-labeled PLACEHOLDER profile (recorded as such, with a citation + to Section 6), pending a firmer numeric ruling. Do not write a specific + number (e.g. "delay 3ms") into `netem_args` and represent it as measured + or ruled -- if you pick a number to instantiate the mechanism for real, + log it as exactly what it is: a placeholder pending D-100's still-open + sub-item, not a final value. + +!!! **OPNsense prep tools not confirmed present.** `scripts/opnsense-prep- + image.sh` requires `bunzip2` + `qemu-img` on PATH; `scripts/opnsense- + build-config-iso.sh` requires `genisoimage` OR `xorriso`. Neither has + been confirmed present on the vcloud host as of this writing + (`opentofu/README.md`). Step 2 below checks for all four before anything + else in this runbook proceeds -- treat a missing tool as a blocking + finding (install it, or escalate), not something to route around. + +--- + +## Sequence (DC1; DC2 remains gated -- see callout above) + +``` +1. Pre-flight: confirm Stage 1 + Stage 2 gates closed (read-only) +2. Confirm OPNsense prep-tool prereqs on the vcloud host (read-only) +3. Confirm/record still-open decisions this stage depends + on (node sizing, config.xml tokens, MAAS zone/pool, + netem parameters) -- STOP on any that are unresolved (read-only) +4. Prepare DC1's OPNsense base image + render config.xml + + build the config ISO [MUTATION: host-local files, gated] +5. Wire modules/opnsense-edge for DC1 into main.tf [repo change, gated] +6. Wire modules/node-vm calls for DC1's node VMs into + main.tf (ONLY once sizing is real) [repo change, gated] +7. tofu init / validate / plan -- review before apply (read-only) +8. tofu apply -- DC1 OPNsense edge + DC1 node-VM domains [MUTATION: creates libvirt objects, gated] +9. Wire + apply modules/maas-vm-host -- register DC1's + libvirt host with DC1's MAAS rack controller [MUTATION: MAAS + repo change, gated] +10. Verify MAAS discovers the node VMs (read-only) +11. Wire + apply modules/netem-link on DC1's mesh legs, + using the Section-6 placeholder lean, explicitly flagged [MUTATION: gated] +12. Verify edge routing + simulated ISP uplink + netem applied (read-only) + -> EXIT GATE -> Stage 4 (per-DC, once DC2 clears its own gate) +``` + +--- + +## Step 1 -- Pre-flight: confirm Stage 1 + Stage 2 gates closed (READ-ONLY) + +**CHECK -- Stage 1 (vcloud host prep) closed?** +```bash +virsh net-list --all +virsh pool-list --all +``` +Expect: DC1's six plane networks, the three mesh-link networks (`dc1-dc2`, +`dc1-office1`, `dc2-office1`), and the `dc1`/`office1` storage pools, all +`active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is +not actually done regardless of what `docs/dc-dc-deployment-workflow.md` +says -- stop and reconcile there first. + +**CHECK -- Stage 2 (Office1 headend) closed?** +Confirm against `docs/dc-dc-deployment-workflow.md`'s Stage 2 row and, if a +Stage 2 runbook exists by the time you run this, its own exit gate. At +minimum, confirm: +```bash +# from Office1 (or wherever this session's tofu apply will run from) +tofu -chdir=opentofu version +curl -sI "$MAAS_API_URL" 2>/dev/null | head -1 # MAAS_API_URL: this session's real value, not invented +``` +Expect a real MAAS region controller reachable at a real URL. If Stage 2 is +not done, everything through Step 8 below (libvirt-layer objects only) may +still proceed since it needs no MAAS reach; Step 9 onward (MAAS registration) +is a hard stop until Stage 2 closes -- do not substitute a placeholder MAAS +endpoint to push past it. + +**GATE:** Stage 1 objects present and active; Stage 2's MAAS reachability +either confirmed, or explicitly recorded as NOT YET (in which case this +session's scope for today is Steps 1-8 only). + +--- + +## Step 2 -- Confirm OPNsense prep-tool prerequisites on the vcloud host (READ-ONLY) + +**CHECK** +```bash +for bin in bunzip2 qemu-img genisoimage xorriso curl wget; do + command -v "$bin" >/dev/null 2>&1 && echo "present: $bin" || echo "absent: $bin" +done +``` +Required: `bunzip2` + `qemu-img` (for `scripts/opnsense-prep-image.sh`), and +EITHER `genisoimage` OR `xorriso` (for `scripts/opnsense-build-config-iso.sh`), +plus `curl` or `wget` for the image fetch. If any required tool is absent, +install it on the vcloud host (an OS-specific package install, out of scope +to prescribe here) before Step 4 -- do not skip the tool and hand-build the +ISO/qcow2 by some improvised method not covered by this repo's tested +scripts. + +**GATE:** all required tools present (or installed and re-checked). + +--- + +## Step 3 -- Confirm/record still-open decisions this stage depends on (READ-ONLY) + +Before wiring anything into `main.tf`, walk this list. Any "NOT YET" item is a +STOP for the corresponding later step -- record it plainly rather than +inventing a value to keep moving: + +- **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` for `modules/node-vm`, + and how many node VMs DC1 needs) -- buildout-design Section 3's Phase-0 + host/disk-budget decision. Blocks Step 6. +- **MAAS zone/pool for DC1's VM-host registration** (`modules/maas-vm-host`'s + `zone`/`pool` variables) -- both are optional (MAAS computes them if left + `null`, confirmed safe per `opentofu/README.md`'s audit pass), so this is + NOT necessarily a blocker; record whichever real zone/pool name the + operator wants, or confirm the `null` no-op is intentional. Affects Step 9. +- **DC1's real `power_address`** (the virsh URI MAAS uses to reach the vcloud + host) -- must be measured this session (or carried from Stage 1's + `libvirt_uri` tfvar, if it is the SAME endpoint -- confirm, do not assume + identical per `modules/maas-vm-host`'s own variable note). Blocks Step 9. +- **OPNsense `config.xml` tokens** -- per `opentofu/templates/README.md`'s + token legend, several tokens are pending Stage-0-adjacent decisions or a + real boot measurement: `{{WAN_IF}}`/`{{LAN_IF}}` (only measurable by + booting the domain and running `ifconfig` -- genuinely cannot be filled in + before Step 8 creates the domain), `{{WAN_IPADDR}}`/`{{WAN_SUBNET_BITS}}`/ + `{{WAN_GATEWAY}}` (pending NetBox), `{{LAN_IPADDR}}`/`{{LAN_SUBNET_BITS}}` + (pending which plane(s) OPNsense's LAN interface actually serves -- + flagged in the template README as not fully specified in the buildout + design), `{{MIRROR_SYNC_PROTOCOL}}`/`{{MIRROR_UPSTREAM_NET}}`/ + `{{MIRROR_SYNC_PORT}}` (pending mirror-software choice, gap #3-adjacent), + and `{{ROOT_PASSWORD_HASH}}` (must be freshly generated per site, never + reused from the shipped sample). Blocks Step 4's render sub-step for + everything except `{{OPNSENSE_HOSTNAME}}`/`{{DOMAIN}}`/NTP defaults. +- **netem parameters** -- still an unruled D-100 sub-item (gap #11); Step 11 + uses the Section-6 qualitative placeholder only, explicitly labeled as + such. Not a hard blocker for Step 11 (the mechanism can be exercised with a + labeled placeholder), but IS a blocker for calling this stage's netem gate + "final" -- see the GATE section. + +**GATE:** every item above is either resolved with a real, measured/ruled +value, or explicitly recorded as NOT YET with the corresponding downstream +step named as blocked. Do not proceed past a step whose input is still +"NOT YET" by inventing one. + +--- + +## Step 4 -- Prepare DC1's OPNsense base image + config.xml + config ISO [MUTATION: host-local files, gated] + +This step produces plain files on the vcloud host filesystem -- no libvirt or +MAAS object is created yet. Run from the vcloud host (or wherever these +scripts execute with reach to write there). + +**MUTATION -- prep the base image (needs Step 2's tools)** +```bash +bash scripts/opnsense-prep-image.sh # see the script's own header for its exact args/output path +``` +**Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a +real output path -- record that path, it feeds Step 5's `base_volume_path`. + +**MUTATION -- render config.xml (only if every non-boot-measured token from +Step 3 is real)** +```bash +export OPNSENSE_HOSTNAME=... DOMAIN=... ROOT_PASSWORD_HASH=... +export WAN_IPADDR=... WAN_SUBNET_BITS=... WAN_GATEWAY=... +export LAN_IPADDR=... LAN_SUBNET_BITS=... +export MIRROR_SYNC_PROTOCOL=... MIRROR_UPSTREAM_NET=... MIRROR_SYNC_PORT=... +# WAN_IF / LAN_IF: cannot be set for real yet -- see the note below. +bash scripts/opnsense-render-config.sh dc1-opnsense-config.xml +``` +`WAN_IF`/`LAN_IF` are the one pair the script's own `REQUIRED_VARS` list +demands that genuinely cannot be measured before the domain exists (Step 8 +creates it). If Step 3 flagged these as not yet measurable, this render +sub-step cannot complete for real this session -- STOP here for DC1's config +render, record it as blocked pending a real boot, and do not substitute a +guessed `vtnetN` name. (One documented path forward once the domain exists: +boot the domain with a MINIMAL/placeholder config good enough to reach a +console, measure `ifconfig`'s real `vtnetN` mapping, then re-render and +re-seed -- that re-seed mechanic is not designed in this delivery; treat it +as a real follow-up need if it comes to that, not an invented shortcut here.) + +**MUTATION -- build the config ISO (only once config.xml render succeeded)** +```bash +bash scripts/opnsense-build-config-iso.sh dc1-opnsense-config.xml dc1-opnsense-config.iso +``` +**Expect:** a plain ISO9660 image containing `/conf/config.xml` at a real +output path -- record it, it feeds Step 5's `config_iso_path`. + +**GATE:** base image path and (if unblocked) config ISO path recorded as real +host-local file paths. If the config.xml render step is blocked on +`WAN_IF`/`LAN_IF`, record that explicitly and treat Step 5's `config_iso_path` +input as still pending -- do not wire Step 5 with a placeholder ISO. + +--- + +## Step 5 -- Wire `modules/opnsense-edge` for DC1 into `main.tf` [repo change, gated] + +Only proceed once Step 4 produced real `base_volume_path` and +`config_iso_path` values (or an explicit decision to proceed with a +genuinely-final, if minimal, config.xml -- not a placeholder). + +**MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like: +```hcl +module "dc1_opnsense" { + source = "./modules/opnsense-edge" + vm_name = "dc1-opnsense" + memory_mib = + vcpu = + pool_name = module.dc1_storage.pool_name # confirm real output name in modules/dc-storage-pool + disk_size_bytes = + base_volume_path = "" + config_iso_path = "" + lan_network_name = + wan_network_name = module.mesh_dc1_office1... # or a dedicated per-site ISP-uplink segment -- confirm which, do not assume +} +``` +Every placeholder above (`<...>`) must be a real, measured/decided value +before this is committed -- this template line exists to show shape, not to +be copy-pasted with invented numbers. `memory_mib`/`vcpu`/`disk_size_bytes` +here have no ratified default in the module either (same "no invented specs" +discipline as `modules/node-vm`). + +**GATE:** `opentofu/main.tf` diff shows exactly one new `module "dc1_opnsense"` +block, every argument a real value, nothing else changed. + +--- + +## Step 6 -- Wire `modules/node-vm` calls for DC1's node VMs into `main.tf` [repo change, gated] + +**ONLY if Step 3 confirmed node sizing is real.** If sizing is still "NOT +YET," STOP here -- this step does not proceed, and this session's DC1 pass +ends at whatever the last unblocked step was (Step 5 or wherever Step 4 left +off). Re-run from here once sizing exists; do not invent numbers to close +this out early. + +**MUTATION -- edit `opentofu/main.tf`**, one `module "dc1_node_XX"` block per +node VM, shaped like: +```hcl +module "dc1_node_01" { + source = "./modules/node-vm" + vm_name = "dc1-node-01" + memory_mib = + vcpu = + pool_name = module.dc1_storage.pool_name + disk_size_bytes = + network_names = [ ] +} +``` +Repeat per node, per the real node count from the sizing decision. + +**GATE:** every `module "dc1_node_*"` block uses only real, decided values; +node count matches the recorded sizing decision exactly. + +--- + +## Step 7 -- `tofu init` / `validate` / `plan` (READ-ONLY against providers) + +```bash +cd opentofu +tofu fmt -check -recursive -diff . +tofu init -backend=false -input=false +tofu validate +tofu plan -out=phase2-dc1.tfplan +``` +Review the plan line by line: expect creates for `module.dc1_opnsense` +(the edge VM's boot volume, config-seed volume, and domain) and each +`module.dc1_node_*` (a blank boot volume + domain per node). Confirm nothing +else is planned -- in particular, confirm no DC2 resource appears (its +module blocks should still be commented out per the DC2 gate above). + +**GATE:** plan matches this expectation exactly; no unexpected creates, +updates, or destroys. Do not apply a plan you have not read. + +--- + +## Step 8 -- `tofu apply` -- DC1 OPNsense edge + node-VM domains [MUTATION: gated] + +```bash +cd opentofu +tofu apply phase2-dc1.tfplan +``` +Confirm this is the exact reviewed plan file from Step 7 (not re-planned) +before running. This is the first live mutation THIS stage performs. + +**VERIFY** +```bash +virsh list --all +virsh domblklist dc1-opnsense +virsh domblklist dc1-node-01 # per real node name +``` +Expect the new domains present (likely `shut off` until first boot/PXE), the +OPNsense domain showing both the base overlay disk and the config ISO +attached as a secondary cdrom, each node VM showing a single blank boot disk. + +**GATE:** all new domains defined as planned; disk attachments match Step +5/6's wiring. + +--- + +## Step 9 -- Wire + apply `modules/maas-vm-host` -- register DC1's libvirt host with DC1's MAAS rack controller [MUTATION: gated] + +**Hard dependency on Stage 2.** This step requires DC1's MAAS rack controller +(under the Stage-2-stood-up MAAS region) to actually exist and be reachable. +If Stage 2 (or the DC1 rack-controller enrollment specifically, however that +ends up sequenced relative to Stage 2's own runbook) is not done, STOP here +-- this stage's remaining steps (9-12) do not proceed, and DC1's Phase 2 work +is INCOMPLETE, not silently skippable. + +**MUTATION -- edit `opentofu/main.tf`** +```hcl +module "dc1_maas_vm_host" { + source = "./modules/maas-vm-host" + vm_host_name = "dc1-vcloud" + power_address = "" + zone = + pool = +} +``` + +```bash +cd opentofu +tofu plan -out=phase2-dc1-maashost.tfplan # review: exactly one new maas_vm_host resource +tofu apply phase2-dc1-maashost.tfplan +``` + +**GATE:** `maas_vm_host` resource created; no other resource touched by this +plan. + +--- + +## Step 10 -- Verify MAAS discovers the node VMs (READ-ONLY) + +**CHECK** +```bash +maas admin vm-hosts read | jq -r '.[] | select(.name=="dc1-vcloud") | {id, name, resources}' +maas admin machines read | jq -r '.[] | select(.hostname|test("^dc1-node")) | "\(.hostname)\t\(.status_name)"' +``` +Expect DC1's node VMs listed as MAAS machines (status likely `New` -- +enlistment discovered them; commissioning is Stage 4's job, not this +stage's). If the node VMs do not appear, do not proceed to declare this +stage's MAAS-discovery gate met -- troubleshoot the VM-host registration +(power_address correctness, virsh reachability from the MAAS rack +controller) before moving on. + +**GATE:** DC1's OpenTofu-created node VMs visible in MAAS, associated with +the `dc1-vcloud` VM host. + +--- + +## Step 11 -- Wire + apply `modules/netem-link` on DC1's mesh legs, placeholder profile [MUTATION: gated] + +Per the netem callout above: this step applies ONLY the buildout design's +qualitative Section 6 lean ("same-metro dark fiber... low single-digit ms, +jitter-capable, jumbo-capable"), explicitly recorded as a PLACEHOLDER, not a +ruled numeric value. If the operator has since ruled a specific numeric +profile, use that instead and cite the ruling; do not use this placeholder +language once a real ruling exists. + +**MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg +(at minimum DC1<->DC2 and DC1<->Office1; DC2<->Office1 is out of scope while +DC2 remains gated): +```hcl +module "netem_dc1_office1" { + source = "./modules/netem-link" + link_name = "dc1-office1" + vcloud_host_ssh_target = "" + bridge_name = "" + netem_args = "" +} +``` +(A DC1<->DC2 block follows the same shape once DC2 itself is ungated; until +then, applying netem on that specific leg has no DC2-side substrate to +matter for, so it is naturally deferred alongside the rest of DC2's work.) + +```bash +cd opentofu +tofu plan -out=phase2-dc1-netem.tfplan +tofu apply phase2-dc1-netem.tfplan +``` + +**GATE:** the `terraform_data`/`local-exec` resource applies without error; +record the exact `netem_args` string used and its PLACEHOLDER status in the +as-executed log, alongside a note that D-100's exact parameters remain open +(gap #11). + +--- + +## Step 12 -- Verify edge routing + simulated ISP uplink + netem applied (READ-ONLY) + +**CHECK -- from the vcloud host** +```bash +sudo tc qdisc show dev # per Step 11's real bridge name +``` +Expect a `netem` qdisc present with the applied (placeholder) parameters. + +**CHECK -- OPNsense edge reachability** (only meaningful once WAN_IF/LAN_IF +have been measured on a real boot per Step 4's note, and the domain has +actually been booted/configured) +```bash +virsh domstate dc1-opnsense +``` +If the config.xml render was blocked in Step 4 (pending `WAN_IF`/`LAN_IF`), +this verification is necessarily partial this session -- record that +honestly rather than declaring edge routing verified when it is not. + +**GATE (this step):** netem qdisc present and matching Step 11's recorded +placeholder profile; OPNsense domain running IF its config.xml was +successfully completed, otherwise recorded as blocked pending a real-boot +`vtnetN` measurement and re-seed. + +--- + +## GATE -- Stage 3 exit condition (honest state, not aspirational) + +Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as: +"MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node +VMs; edge routing + simulated ISP uplink up; netem parameters applied and +measured." As of what this runbook can actually close this session: + +- **MAAS rack controller per DC up:** achievable ONLY once Stage 2 (or + whatever runbook stands up DC1's specific rack controller) is done -- + Step 1/9 depend on it. Record actual state, do not assume it. +- **MAAS discovers the OpenTofu-created node VMs:** achievable through Step + 10, CONDITIONAL on node sizing being real (Step 3/6) and Stage 2's MAAS + region being reachable (Step 9's dependency). +- **Edge routing + simulated ISP uplink up:** PARTIALLY achievable. The + OPNsense domain and its network attachments can be created (Step 8), but + full edge routing verification depends on `WAN_IF`/`LAN_IF` being measured + on a real boot (Step 4's documented blocker) and on the WAN/LAN addressing + tokens (pending NetBox) -- do not mark this sub-gate fully closed unless + those are genuinely resolved. +- **netem parameters applied and measured:** applied, YES (Step 11), using + an explicitly-labeled PLACEHOLDER same-metro-lean profile per + buildout-design Section 6 -- MEASURED in the sense of "the qdisc is + present and its effect can be observed" (e.g. via `ping`/`iperf` once the + domains are reachable), but NOT measured against a ruled, final numeric + target, because that target does not exist yet (D-100 gap #11). Do not + represent this sub-gate as "done" in the same sense as a stage with no + open decisions -- it is done AS FAR AS A PLACEHOLDER ALLOWS, with the + remaining gap named. + +**This stage's exit gate is therefore CONDITIONALLY MET at best** on any run +before (a) Stage 2 is complete, (b) node sizing is decided, (c) DC1's +OPNsense WAN/LAN addressing and `vtnetN` mapping are measured, and (d) D-100's +exact netem parameters are ruled. Update `docs/dc-dc-deployment-workflow.md`'s +Stage 3 row to reflect the REAL state after running this (e.g. "PARTIAL -- +libvirt objects created, MAAS registration blocked on Stage 2" or similar), +never to a blanket DONE unless every sub-condition above is genuinely true. + +**DC2:** none of the above applies to DC2 until NetBox assigns its real +supernet (D-101 open item) and `main.tf`'s DC2 blocks are uncommented with +real values -- re-run this entire runbook for DC2 at that point, do not +attempt a partial DC2 pass now. + +-> Proceed to Stage 4 (MAAS enlist/commission/deploy) for DC1 once this +gate's applicable sub-conditions are genuinely met; DC2 remains blocked at +Stage 3 until its own gate (NetBox supernet) clears. + +--- + +## Delivery checklist (this repo's standard discipline) + +- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo + changes made while executing this runbook (`main.tf` edits, any new + tfvars, minus secrets). +- [ ] `bash scripts/opentofu-validate.sh` green (the harness for `opentofu/`, + per its own README) -- re-run after every `main.tf` change in Steps + 5/6/9/11, not just once at the end. +- [ ] Changelog entry for this runbook's first real execution (next DOCFIX + number via `bash scripts/ledger-scan.sh`), noting the ACTUAL measured/ + decided values used (node sizing, MAAS zone/pool, power_address, the + netem placeholder profile and its provisional status) -- redact + nothing that isn't a secret. +- [ ] `docs/session-ledger.md` updated with the outcome, including which + sub-steps were blocked and why (Stage 2 dependency, node sizing, + WAN/LAN token gaps, D-100 netem sub-item). +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 3 row updated to the REAL, + honest state (PARTIAL/DONE/NOT STARTED per the GATE section above), + never rounded up to DONE with open sub-conditions. +- [ ] `opentofu/README.md` updated to reflect DC1's OPNsense edge / node-VM / + MAAS-vm-host / netem-link modules moving from "not instantiated" to + instantiated (or partially so) once this runbook actually applies + them, including any real bug found while doing so (matching this + repo's practice of logging schema surprises as they're found). +- [ ] DC2 gate re-confirmed as still closed (NetBox supernet not yet + assigned) if this session did not clear it -- do not let this + runbook's DC1 completion be misread as covering DC2. diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md new file mode 100644 index 0000000..7c14320 --- /dev/null +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -0,0 +1,323 @@ +# DC-DC Phase 4 -- Juju controller + OpenStack bundle, per DC (Stage 5) + +Stand up a running, independent OpenStack cloud in DC1, then repeat the same +loop for DC2. This is the ground-truth/copy-point stage: `docs/dc-dc-buildout- +design.md` line 12 states this repo was seeded from `openstack-caracal-ipv4` +at commit 8813efc specifically so `bundle.yaml` and `runbooks/phase-01..08- +*.md` could be reused here almost unchanged. **This runbook does not +duplicate those eight runbooks' command-level content.** It describes what +changes to run them TWICE, independently, in the VR1 two-DC dual-stack +context, and calls out the real adaptation work plainly rather than +asserting "just re-run it" glosses over anything. + +**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 4" entry +and Section 2 (address-family table); `docs/design-decisions.md` D-104 +(controller topology), D-109 (per-DC independent Vault roots + IPv6-SAN +certs), D-101 (address plan / family matrix); `docs/dc-dc-deployment- +workflow.md` Stage 5 row (**"This is where the existing repo tooling +(preflight.sh, cloud-assert.sh, lib-net.sh/lib-hosts.sh, run-logged.sh) does +the most direct work"**) and tooling gap register item #1 (the `$DC` +selector convention, CLOSED 2026-07-09 DOCFIX-151 -- this is the stage that +actually calls it). + +**Prerequisites (must be true entering this stage, per DC):** Stage 4 +(`runbooks/dc-dc-phase3-maas-enlist-deploy.md`) done for that DC -- nodes +Deployed; six planes present with correct fabrics/VLANs; provider NIC raw; +PXE (v4) working; that DC's own artifact mirror reachable; NTP from that +DC's edge working; `scripts/lib-hosts.sh`'s per-DC host data populated (that +stage's own follow-up delivery) so `lib_hosts_select_dc "$dc"` no longer +fails loud for it. + +!!! Every command block in the referenced `phase-01..08` runbooks that reads + `scripts/lib-net.sh` or `scripts/lib-hosts.sh` values (VIP bands, host + octets, `KEYSTONE_VIP_DEFAULT`, etc.) MUST be preceded, in THIS stage's + session, by: + ```bash + source scripts/lib-net.sh; lib_net_select_dc "$DC" # $DC = dc1 or dc2 + source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" + ``` + Do not skip this because it "worked before" on DC0/DC1 -- for DC1 both + calls are a documented no-op (D-101: DC1 inherits DC0 unchanged), so + skipping them happens to produce the same values by accident; for DC2 it + is NOT a no-op and skipping it silently reuses DC1's values, exactly the + bug DOCFIX-151 exists to prevent. Always call both, for both DCs, + unconditionally. + +--- + +## What is genuinely IDENTICAL to VR0/DC0 for DC1, and what is not + +**DC1 needs NO VIP/CIDR changes at all.** D-101: DC1 inherits DC0's six-plane +v4 layout UNCHANGED. `bundle.yaml`'s existing VIP values (`.50-.60` band, +per phase-01's own text) are ALREADY correct for DC1 -- deploy the SAME +`bundle.yaml` this repo already carries, in DC1's own separate Juju +controller/model, with no v4-address edits. `phase-01..08`'s command blocks +that reference `lib-net.sh`/`lib-hosts.sh` values will resolve to the exact +same numbers for DC1 as they did for the original DC0 rehearsal -- this is +expected, not a bug (see the `$DC` selector callout above). + +**DC2 needs a distinct, non-overlapping v4 supernet + all-new VIPs -- NOT +YET ASSIGNED.** Per D-101 and tooling gap register item #3: DC2's supernet, +the org ULA /48, and the per-DC GUA carve are NetBox-authoritative and +still genuinely unassigned (the import PIPELINE closed 2026-07-09, DOCFIX- +152, but the literals themselves have not been generated/assigned). **DC2's +bundle deploy is BLOCKED on this** -- you cannot pick DC2's VIP band, the +octavia SAN IP (per phase-01's own `1.0-GEN.c` step, which derives the SAN +IP FROM the bundle's real VIP value, not a literal), or any other DC2 +address until NetBox has real values. Do not invent a "DC2 test range" to +get moving faster -- that is exactly the inference this repo's hard rule +forbids. If DC2's literals still don't exist when you reach this stage, +STOP here for DC2 and only proceed with DC1. + +--- + +## Sequence (repeat entire sequence once per DC, DC1 first) + +``` +1. Confirm Stage 4 gate + call the $DC selectors (read-only) +2. Bootstrap a DEDICATED single-unit Juju controller, IN-DC [MUTATION, gated] +3. preflight.sh PASS (read-only gate) +4. Run phase-01 (bundle deploy) against THIS DC's model [adapted, gated -- see below] +5. Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] +6. Write + apply the IPv6 family-matrix overlay [NOT YET WRITTEN -- see below] +7. Run phase-03 (core verify) [adapted, gated] +8. Run phase-04 (network carve) [adapted, gated] +9. Run phase-05 (octavia enablement) [adapted, gated] +10. cloud-assert.sh --capture (read-only + committed BOM) +11. Controller backup: juju create-backup / download-backup [MUTATION, gated] +12. Ceph-over-v6 + geneve-over-v6 verification (read-only gate) + -> EXIT GATE (this DC) -> repeat 1-12 for the other DC -> Stage 6 +``` + +--- + +## Step 1 -- Confirm Stage 4 gate + call the $DC selectors (READ-ONLY) + +**CHECK -- jumphost / wherever this session runs juju against this DC** +```bash +export DC=dc1 # or dc2 -- explicit, per this repo's $DC convention (never omitted) +source scripts/lib-net.sh; lib_net_select_dc "$DC" +source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" +echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}" +echo "HOSTS: ${HOSTS[*]}" +``` +**GATE:** both selector calls exit 0 (for DC2, this REQUIRES Stage 4 having +already populated real host data into `scripts/lib-hosts.sh` for dc2 -- if +`lib_hosts_select_dc dc2` still fails loud here, Stage 4 for DC2 is not +actually done yet, stop and go back). + +--- + +## Step 2 -- Bootstrap a dedicated, in-DC, single-unit Juju controller [MUTATION, gated] + +Per D-104: ONE Juju controller PER DC, deployed IN that DC (on a MAAS- +deployed machine in that DC, from Stage 4), single-unit (not HA) for VR1, +managing ONLY its own cloud. This is explicitly NOT the same controller +model as VR0/DC0's original bootstrap -- a fresh `juju bootstrap` targeting +THIS DC's MAAS cloud/region, not a reused controller. + +**CHECK (read-only) -- confirm the target MAAS cloud is registered to Juju** +```bash +juju clouds --client | grep -i maas +``` + +**MUTATION** +```bash +juju bootstrap +``` +The exact MAAS cloud name and the controller name are session-specific +identifiers you choose/confirm at execution time (e.g. reflecting `dc1`/ +`dc2` in the controller name for operator clarity) -- not invented here. + +Per D-104's rationale: this keeps DR-independence honest (a controller loss +affects only its own DC) and is a deliberate VR1 test-only economy (HA +deferred to Roosevelt) -- do not "upgrade" this to HA mid-stream without a +new D-NNN, that would silently reverse an ADOPTED decision. + +--- + +## Step 3 -- `preflight.sh` PASS (read-only gate) + +```bash +bash scripts/preflight.sh +``` +Confirmed repo-agnostic as of the 2026-07-09 sweep (`docs/dc-dc-deployment- +workflow.md` Stage 5 row) -- runs unmodified against either DC's clone/ +session. Expect `PREFLIGHT: PASS`. + +--- + +## Step 4 -- Run phase-01 (bundle deploy) against this DC's model [adapted, gated] + +Follow `runbooks/phase-01-bundle-deploy.md` verbatim, with these DC-DC- +specific notes, not a rewrite: + +- **DC1:** deploy the EXISTING `bundle.yaml` with NO VIP edits (see the + "identical to DC0" section above). The Octavia PKI overlay (Step 1.0) + can reuse the existing `overlays/octavia-pki.yaml` if its CAs are still + valid (10y validity, per that runbook), OR regenerate fresh per-DC PKI via + Step 1.0-GEN if independent per-DC Octavia trust domains are desired -- + this is a call the operator should make explicitly (log it as a decision + note either way), not silently default to reuse. +- **DC2:** BLOCKED until its VIP band exists (see above). Once NetBox + assigns DC2's real supernet and it's imported via `netbox/dc-dc-prefixes- + import.py --dc dc2` (DOCFIX-152), `bundle.yaml`'s VIP lines need editing + to DC2's real `.50-.60`-equivalent band within DC2's own provider-public + plane -- a genuine, reviewed bundle edit, not a copy-paste of DC1's values. +- Both DCs: `bundle.yaml` explicitly ships "NO designate" per phase-01's own + gate text (D-019) -- this is UNCHANGED here; Designate reactivation is + Stage 7's job (D-106), do not reactivate it mid-Stage-5. +- Run phase-01's own EXIT GATE checks exactly as written, against THIS DC's + model. + +--- + +## Step 5 -- Run phase-02 (vault bring-up) -- INDEPENDENT root per DC [adapted, SECRET-HANDLING] + +Follow `runbooks/phase-02-vault-bringup.md` verbatim per DC. Per D-109: each +DC's Vault is its OWN independent root CA -- no regional root-of-trust this +phase. This means: +- The `vault operator init` one-shot step (Step 2.1) runs ONCE PER DC, + producing DISTINCT unseal-key shares and a DISTINCT root token for DC1 vs. + DC2. Store them separately, clearly labeled by DC (this repo's existing + secrets discipline -- `~/vault-init/`-style off-host capture -- applies + identically per DC, just not shared between them). +- D-109 also requires each DC's Vault to issue certs with BOTH IPv4 and + IPv6 (ULA) SANs for the dual-stacked metal-internal API endpoints. **This + is a real charm-behavior question not yet confirmed this session:** + whether the `vault`/`vault-pki`-relation charms used in `bundle.yaml` + actually support requesting/issuing IPv6 SANs today (vs. only IPv4) is + UNVERIFIED -- check the actual charm config/relation-data schema (e.g. + `openstack-loadbalancer`/certificates interface docs, or the vault charm's + own config options) before assuming this "just works" once IPv6 addresses + exist on the plane. If it does not support v6 SANs natively, that is a + real finding to log (a D-NNN amendment or a new decision), not something + to route around silently. + +--- + +## Step 6 -- Write + apply the IPv6 family-matrix overlay [NOT YET WRITTEN] + +**This is real, undone design work -- flagged here rather than glossed +over.** D-101's family matrix (Section 2 of the buildout design) requires, +per DC: +- `data-tenant`, `storage`, `replication`: IPv6-ONLY ULA (no v4 config at + all on these planes for the relevant charms -- e.g. `ceph-mon`'s + `public-network`/`cluster-network`, OVN's geneve encapsulation address). +- `metal-internal`, `metal-admin`: DUAL-STACK (v4 retained + a ULA leg + added -- the metal-admin ULA leg is the 2026-07-09 operator AMENDMENT to + D-101, a reversal of that decision's own original lean, so do not assume + older documentation/training-data patterns for "metal-admin stays v4-only" + -- this repo's ADOPTED text says otherwise). +- `provider-public`: DUAL-STACK v4 + GUA (native, no NAT66 -- D-101's + tenant/MTU sub-policy folded in from D-102). + +**No overlay file implementing this exists yet in this repo.** The exact +Juju charm config option names needed to add v6 addressing to each affected +application (`ceph-mon`, `ovn-central`, `ovn-chassis`, `neutron-api-plugin- +ovn`, `octavia`, the HA-cluster/vip-carrying API charms, etc.) have NOT been +confirmed against those charms' real current config schemas this session -- +inventing plausible-looking option names here would violate this repo's +"never fabricate a flag/option name you haven't verified" norm. Before this +step can actually run: +1. Confirm each affected charm's real config options for v6/dual-stack + addressing (charm's own `config.yaml` in charmhub, or `juju config + ` against a live unit once deployed) -- do not assume symmetry with + the v4 option names. +2. Author a new `overlays/dc-dc-ipv6-family-matrix.yaml` (or per-DC + variants if DC1/DC2 ever need to diverge) implementing the confirmed + options. +3. Apply via `juju deploy ./bundle.yaml --overlay ... --overlay + overlays/dc-dc-ipv6-family-matrix.yaml` (stacked overlays, same pattern + as the existing octavia-pki overlay) OR `juju config =` + per-app post-deploy, whichever the charm options actually require. + +Log this as its own DOCFIX/design task once the charm-option research is +done -- it blocks this stage's "Ceph-over-v6 and geneve-over-v6 verified" +gate criterion (Step 12) until it exists. + +--- + +## Steps 7-9 -- phase-03 / phase-04 / phase-05, adapted per DC + +Follow `runbooks/phase-03-core-verify.md`, `runbooks/phase-04-network- +carve.md`, `runbooks/phase-05-octavia-enablement.md` verbatim per DC, with +the same `$DC` selector discipline as Step 1 applied to every command block +in them that reads `lib-net.sh`/`lib-hosts.sh` values. No other adaptation +is currently known to be needed in these three -- if executing them surfaces +a DC0-specific literal baked into a check block that these runbooks' +authors didn't anticipate needing DC-parameterization, LOG it (a DOCFIX +candidate) rather than silently patching around it live. + +--- + +## Step 10 -- `cloud-assert.sh --capture` (read-only + committed BOM) + +```bash +bash scripts/cloud-assert.sh --capture +``` +Per DC, at real deploy-completion. Confirmed repo-agnostic (2026-07-09 +sweep). Commit the resulting `asbuilt/` BOM per this repo's existing +deploy-loop convention, labeled clearly by DC (do not let DC1's and DC2's +BOMs collide/overwrite -- confirm the script's own output path is already +DC-distinguishing, or add a DC-qualified path if not, before running for +the second DC). + +--- + +## Step 11 -- Controller backup [MUTATION, gated] + +```bash +juju create-backup -m +juju download-backup -m +``` +Per D-104: scheduled controller backups, confirmed backup posture (juju 3.6 +`create-backup`/`download-backup`). Store the downloaded backup off-host, +labeled by DC, same secrets-adjacent handling discipline as Vault's init +capture. The RESTORE half of this drill is exercised in Stage 6 (DR wiring +and failover drill) -- this step only takes and stores the backup, it does +not restore it. + +--- + +## Step 12 -- Ceph-over-v6 + geneve-over-v6 verification (READ-ONLY gate) + +```bash +# Ceph binds v6 on the storage/replication ULA planes (ms_bind_ipv6, D-101): +juju run -m ceph-mon/leader 'ceph -s' 2>&1 | grep -i "v6\|v2" +# OVN geneve encapsulation over the data-tenant ULA plane: +juju config -m ovn-central | grep -i encap +``` +The exact expected values here depend on Step 6's overlay actually having +been written and applied -- if Step 6 is still open (charm options +unconfirmed), this gate CANNOT pass yet; do not force a "looks close enough" +pass. Record whichever real state you observe, including "blocked on Step +6" if that's the honest answer. + +**GATE (Stage 5 exit condition, `docs/dc-dc-deployment-workflow.md`):** +`preflight.sh` PASS before add-model (Step 3); post-deploy `cloud-assert.sh +--capture` per DC (Step 10); controller backup taken + stored (Step 11); +Ceph-over-v6 and geneve-over-v6 verified (Step 12, contingent on Step 6). + +-> Repeat this entire runbook for the other DC (DC1 first is recommended, +since only DC1 is unblocked by real address literals today) -> proceed to +Stage 6 (DR wiring and failover drill) once BOTH DCs pass this gate. + +--- + +## Delivery checklist + +- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo + changes made while executing this runbook (the IPv6 overlay file, any + `bundle.yaml` VIP edit for DC2, any DC-qualified `asbuilt/` path fix). +- [ ] Changelog entry per real repo change this runbook produces (the + overlay, any bundle edit, any script fix) -- next DOCFIX number via + `bash scripts/ledger-scan.sh`, same as every other delivery this + session. +- [ ] `docs/session-ledger.md` updated with the outcome, per DC. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 5 row updated once BOTH DCs + have actually passed Step 12's gate (not just DC1). +- [ ] Log the Step 6 charm-config research (v6 SAN support, v6 addressing + option names) as its own follow-up item if not resolved inline during + this stage's real execution -- do not leave it silently unresolved in + the workflow doc's tooling gap register.