diff --git a/docs/changelog-20260709-office1-network-edge.md b/docs/changelog-20260709-office1-network-edge.md new file mode 100644 index 0000000..663c1da --- /dev/null +++ b/docs/changelog-20260709-office1-network-edge.md @@ -0,0 +1,134 @@ +# Changelog 2026-07-09 -- Office1-local network + Office1 OPNsense edge ownership (DOCFIX-163) + +No live infrastructure touched -- OpenTofu module/wiring additions, runbook +updates, and documentation only (prep-only session; no `tofu` binary +available, same UNVALIDATED status as every other OpenTofu module in this +repo). Closes tooling gap register items #12 and #16 in +`docs/dc-dc-deployment-workflow.md`; surfaces a new item, #17, found while +closing #16. + +## Items + +### 1. DOCFIX-163 -- `opentofu/modules/office1-network` (closes gap #12) + +FILES: `opentofu/modules/office1-network/main.tf`, +`opentofu/modules/office1-network/variables.tf`, +`opentofu/modules/office1-network/outputs.tf`, `opentofu/main.tf` (new +`module "office1_network"` block, instantiated for real), `opentofu/README.md`. + +WHAT: a new module, shaped exactly like `modules/dc-planes` (isolated +`libvirt_network`, no `forward`/DHCP block, attribute-style nested-object +syntax per this provider's confirmed schema convention) but sized for ONE +network instead of six planes -- Office1 is a headend, not a per-DC +OpenStack cloud, so the six-plane D-052/D-100 template does not apply to it. +This is the shared L2 segment the three Office1 service VMs (MAAS-region, +NetBox, GitBucket), the vcloud host's own management interface, and Office1's +own OPNsense edge's LAN side all attach to -- previously modeled nowhere in +`opentofu/` (gap #12, found 2026-07-09 while drafting +`runbooks/dc-dc-phase1-office1-standup.md`). + +DESIGN DECISION: a NEW module, not a reused host bridge (the runbook's own +"Open questions" section had left this as a genuine, undecided fork). +Reasoning, in full in the module's own header comment and in +`runbooks/dc-dc-phase1-office1-standup.md`'s Open question #1: +- D-103 already establishes that OpenTofu owns create/destroy of every + virtual network in this topology. A hand-configured host bridge would be + the one network living outside that inventory -- exactly the + not-reproducible, not-versioned debt D-103 exists to eliminate, and the + same class of debt this repo already logs explicitly (as a DOCFIX + candidate) whenever Option B (manual VM creation) is used elsewhere in the + same runbook. +- This repo's own CHECK steps (`virsh net-list --all`, used throughout the + Stage 1/2 runbooks) implicitly assume that command is a COMPLETE inventory + of what exists on vcloud. A pre-existing, OpenTofu-invisible bridge would + silently break that assumption, and Stage 0's teardown path (D-061) would + never reach it. +Instantiated for REAL in root `main.tf` (`module "office1_network"`), unlike +most of the other still-scaffolded modules in this repo -- it needs no +unmeasured value beyond `domain_suffix`/`underlay_mtu`, both already real, +already-required inputs elsewhere in the same file (the identical reasoning +that already lets the three `mesh-link` legs be instantiated today). + +VERIFIED: +- `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy WARN (unchanged). +- **UNVALIDATED against a real provider** -- no `tofu` binary was available + this session (same status as every other module in `opentofu/`; see + `opentofu/README.md`'s top-level status line). Run + `bash scripts/opentofu-validate.sh` on a connected machine before trusting + the schema. + +REVERT: `git checkout HEAD~ -- opentofu/main.tf opentofu/README.md && rm -rf +opentofu/modules/office1-network` (safe -- nothing else references this +module yet). + +### 2. DOCFIX-163 -- Office1 OPNsense edge ownership decision (closes gap #16) + +FILES: `opentofu/main.tf` (new, commented-out `module "office1_opnsense"` +skeleton), `opentofu/README.md`, +`runbooks/dc-dc-phase1-office1-standup.md` (new Step 4b, updated Open +questions/Sequence/GATE/delivery-checklist sections). + +WHAT: the buildout design's topology (Section 1) is explicit -- "full +dark-fiber triangle... plus a **per-site** OPNsense simulated ISP edge," +three sites, three edges. Stage 3's Build list already covered DC1/DC2's +edges (`modules/opnsense-edge` per DC); neither Stage 2's nor Stage 3's +runbook instantiated one for Office1 (gap #16, found 2026-07-09 during final +consolidation of the DC-DC runbook set). + +DECISION: **YES, Office1 gets its own `modules/opnsense-edge` call, a fourth +site alongside DC1/DC2 -- and Stage 2's own runbook +(`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it**, via a new +Step 4b (CHECK-before-MUTATION shape, its own GATE bullet, matching the +runbook's existing style exactly). Rationale: Stage 2's entire scope is +already "Office1 headend standup" -- the OPNsense edge is part of that +headend's own boundary, not DC substrate -- and Stage 1 already creates the +mesh-link networks this edge's neighbors depend on; Stage 3 has no reason to +reach back into Office1's own infrastructure to build a piece of it. + +The new `module "office1_opnsense"` skeleton in `main.tf` mirrors Stage 3's +`dc1_opnsense` template shape exactly (same module, same variable names). +Its `lan_network_name` resolves cleanly to Item 1's new network +(`module.office1_network.network_name`). Its `wan_network_name` does NOT -- +see Item 3 below. The block stays commented out, same as DC1/DC2's own +not-yet-instantiated edges (neither is real either, per +`opentofu/README.md`), pending real `memory_mib`/`vcpu`/`disk_size_bytes`/ +`base_volume_path`/`config_iso_path` values plus gap #17's resolution. + +VERIFIED: `bash scripts/repo-lint.sh`: 0 fail (module block is commented +out; nothing new to `tofu validate` here that Item 1 didn't already cover). + +REVERT: `git checkout HEAD~ -- opentofu/main.tf opentofu/README.md +runbooks/dc-dc-phase1-office1-standup.md` (safe -- Step 4b's module call was +never uncommented/applied). + +### 3. Gap register item #17 (NEW) -- no per-site ISP-uplink/WAN network exists yet, for any site + +Investigated per this task's own instruction to check before inventing a +WAN-side network for Office1's edge: read +`runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 5 directly. Its own +`dc1_opnsense` template carries an IDENTICAL, still-unresolved placeholder: +``` +wan_network_name = module.mesh_dc1_office1... # or a dedicated per-site ISP-uplink segment -- confirm which, do not assume +``` +This confirms the WAN-side network question is NOT Office1-specific -- it is +a genuinely deeper, cross-site gap that also blocks DC1's and DC2's own +edges, not something this delivery could resolve alone without inventing a +value or (worse) defaulting Office1's WAN side to a mesh-link network, which +would directly contradict D-100's own sub-item ruling that the three +dark-fiber mesh legs carry MANAGEMENT TRAFFIC ONLY (Office1<->DC = MAAS/ +Juju/operator), not a simulated ISP uplink. Logged as new tooling gap +register item #17 in `docs/dc-dc-deployment-workflow.md`, cross-referenced +from gaps #4/#12/#16 and from both new module headers -- flagged rather than +invented, per this task's own instruction. + +## Next actionable step + +Tooling gap register items #12 and #16 CLOSED. Gap #17 (NEW) is now the +blocking dependency for instantiating ANY of the three sites' +`opnsense-edge` calls end-to-end -- needs a decision on the WAN-side network +shape (most likely a fourth per-site `libvirt_network`, analogous to +`office1-network`/`mesh-link`/`dc-planes`) before `dc1_opnsense`, +`dc2_opnsense`, or `office1_opnsense` can be uncommented for real. Designing +the three Office1 service VMs' actual `cloudinit-vm` `network_config` +content against the now-real `office1-local` network remains separate, +un-actioned work (unchanged by this delivery). diff --git a/opentofu/README.md b/opentofu/README.md index 007fef3..32e2db8 100644 --- a/opentofu/README.md +++ b/opentofu/README.md @@ -97,6 +97,38 @@ that resource is the wrong one (it composes new VMs; D-103 explicitly rules that out). Not instantiated: needs a real MAAS zone/pool and the vcloud host's real power_address. +- `modules/office1-network` (2026-07-09) -- closes tooling gap register item + #12: Office1's own LOCAL virtual network, the shared L2 segment the three + Office1 service VMs (MAAS-region, NetBox, GitBucket), the vcloud host's own + management interface, and Office1's own OPNsense edge's LAN side (gap #16) + all attach to. Shaped exactly like `modules/dc-planes` (isolated + `libvirt_network`, no `forward`/DHCP, attribute-style syntax) but for ONE + network, not six planes -- Office1 is a headend, not an OpenStack DC, so + the six-plane template does not apply to it. **Design decision (resolves + gap #12's own "new module vs. host-bridge reuse" fork): a new module, NOT a + reused host bridge** -- see the module's own header comment for the full + reasoning (in short: a hand-configured bridge would be the one network in + this topology living outside OpenTofu's D-103-mandated inventory, and would + silently break this repo's own `virsh net-list --all` completeness + assumption used as a CHECK step throughout the runbooks). INSTANTIATED for + real in root `main.tf` (needs no value beyond `domain_suffix`/ + `underlay_mtu`, both already real required inputs elsewhere in this file -- + the same reasoning that already lets the three mesh-link legs be + instantiated today). +- `modules/opnsense-edge` now has a FOURTH intended caller (2026-07-09, + closes tooling gap register item #16): Office1 gets its own OPNsense edge + alongside DC1/DC2 (the buildout design's "per-site" wording is explicit -- + three sites, three edges), and Stage 2's runbook + (`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it, since Stage + 2's whole scope is already "Office1 headend standup" and Stage 1 already + creates the mesh-link networks this edge's neighbors depend on. A `module + "office1_opnsense"` skeleton is written in root `main.tf`, commented out -- + mirroring the identical NOT-YET-REAL status of DC1/DC2's own edges (neither + is instantiated either; see `runbooks/dc-dc-phase2-tofu-dc-substrate.md` + Step 5, which is itself still all placeholders). Its `lan_network_name` + resolves cleanly to `module.office1_network.network_name`; its + `wan_network_name` does NOT -- see the new gap register item #17 below, + a DEEPER, cross-site gap this delivery found rather than invented past. - `modules/netem-link` (2026-07-09) -- applies `tc qdisc ... netem` WAN-simulation parameters (D-100) to a `mesh-link` bridge, via a `terraform_data` resource with `local-exec` provisioners run over SSH (NOT @@ -105,9 +137,9 @@ instantiated: needs the real bridge name, the vcloud host's SSH target, and the actual netem parameters (still an unruled D-100 decision). - Root wiring for DC1 (planes + pool, using DC1's inherited DC0 CIDRs) and - Office1 (pool only), plus all three mesh links, plus a `provider "maas" {}` - block (api_url/api_key as new required variables, `maas_api_key` marked - `sensitive = true`). + Office1 (pool + local network, 2026-07-09), plus all three mesh links, plus + a `provider "maas" {}` block (api_url/api_key as new required variables, + `maas_api_key` marked `sensitive = true`). **Syntax bug fixed 2026-07-09:** `dc-planes`, `mesh-link`, and `dc-storage-pool` originally used classic HCL block syntax (`domain { name = ... }`) for @@ -137,6 +169,35 @@ now built (`modules/netem-link`), but the actual latency/jitter/loss/rate values are still an unruled D-100 open item, so nothing is instantiated. Full breakdown in `docs/dc-dc-deployment-workflow.md` gap register item 4. +- **Office1's own OPNsense edge -- ownership DECIDED 2026-07-09 (gap #16 + CLOSED), instantiation still blocked, same as DC1/DC2's edges.** A `module + "office1_opnsense"` skeleton exists in root `main.tf`, commented out: its + `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` + have the exact same "no invented specs" blockers as DC1/DC2's own + not-yet-instantiated edges. Its LAN side is now resolvable + (`module.office1_network.network_name`), but its WAN side is NOT -- see + the new gap register item #17 immediately below, which is NOT + Office1-specific: it blocks DC1's and DC2's own `wan_network_name` values + identically (confirmed by reading `runbooks/dc-dc-phase2-tofu-dc-substrate.md` + Step 5's own DC1 template, which carries the identical unresolved + placeholder). +- **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in + this repo, for ANY site (gap register item #17, NEW 2026-07-09).** Every + `modules/opnsense-edge` caller needs a `wan_network_name` distinct from + both the LAN-side network (office1-network / a dc-planes plane) and the + three mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC-ONLY + by D-100's own sub-item ruling (Office1<->DC fiber = MAAS/Juju/operator), + not a simulated ISP uplink, so wiring an edge's WAN side to a mesh leg + would misrepresent the topology rather than merely placeholder it. This + was found while resolving gap #16 for Office1 and confirmed to be a + pre-existing, unresolved gap in Stage 3's own DC1/DC2 wiring too (its own + runbook's `wan_network_name` line is still a literal "confirm which, do + not assume" placeholder) -- not invented here for Office1 alone. Needs a + decision: a fourth network shape per site (dc-planes/mesh-link/ + office1-network-style, one dedicated `libvirt_network` per site standing + in for its "internet" uplink), or some other mechanism not yet considered. + Blocks instantiating ANY of the four `opnsense-edge` calls + (`dc1_opnsense`, `dc2_opnsense`, `office1_opnsense`) end-to-end. - **MAAS VM-host registration -- mechanism DONE (`modules/maas-vm-host`), not instantiated.** Needs a real MAAS zone/pool and the vcloud host's power address; neither exists yet. diff --git a/opentofu/main.tf b/opentofu/main.tf index 60e0dfb..07d069c 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -64,6 +64,59 @@ target_path = var.office1_pool_path } +# ---- Office1-local network (closes tooling gap register item #12): the +# shared L2 segment the three Office1 service VMs (MAAS-region, NetBox, +# GitBucket), the vcloud host's own management interface, and Office1's own +# OPNsense edge's LAN side (gap #16) all attach to. See +# modules/office1-network/main.tf's header for the design decision (a new +# module, not a reused host bridge) and opentofu/README.md for the full +# writeup. Instantiated for real here (unlike office1_opnsense below): it +# needs no value beyond domain_suffix/underlay_mtu, both already real, +# required inputs elsewhere in this file -- the same reasoning that already +# lets the three mesh-link legs be instantiated today. ---- + +module "office1_network" { + source = "./modules/office1-network" + domain_suffix = var.domain_suffix + mtu = var.underlay_mtu +} + +# ---- Office1's own OPNsense edge (closes tooling gap register item #16 -- +# decision: YES, Office1 gets a fourth modules/opnsense-edge call alongside +# DC1/DC2, and Stage 2's runbook (runbooks/dc-dc-phase1-office1-standup.md) +# owns creating it -- see that runbook's new Step 4b and its "Open questions" +# section for the full reasoning). NOT instantiated below -- commented out on +# purpose, mirroring the identical NOT-YET-REAL status of module +# "dc1_opnsense"/"dc2_opnsense" (see runbooks/dc-dc-phase2-tofu-dc-substrate.md +# Step 5's own template, which is ALSO still all placeholders, nothing +# committed here either). Every value below needs to be real before this is +# uncommented -- do not fill in invented specs to wire it in sooner: +# +# module "office1_opnsense" { +# source = "./modules/opnsense-edge" +# vm_name = "office1-opnsense" +# memory_mib = +# vcpu = +# pool_name = module.office1_storage.pool_name +# disk_size_bytes = +# base_volume_path = "" +# config_iso_path = "" +# lan_network_name = module.office1_network.network_name +# wan_network_name = DC fiber = MAAS/ +# Juju/operator), not a simulated ISP uplink -- wiring an edge's WAN +# side to a mesh leg would misrepresent the topology, not just +# placeholder it. Resolve gap #17 (a dedicated per-site ISP-edge +# network, analogous in shape to this same office1-network module or +# mesh-link) before uncommenting this block for Office1 OR for DC1/DC2.> +# } + # ---- The D-100 dark-fiber mesh triangle: DC1<->DC2, DC1<->Office1, # DC2<->Office1 -- mesh, not star. All three legs created now even though DC2 # itself is not yet wired above: the LINK is independent of whether DC2's diff --git a/opentofu/modules/office1-network/main.tf b/opentofu/modules/office1-network/main.tf new file mode 100644 index 0000000..c667d75 --- /dev/null +++ b/opentofu/modules/office1-network/main.tf @@ -0,0 +1,61 @@ +# office1-network: Office1's own LOCAL virtual network (closes tooling gap +# register item #12 in docs/dc-dc-deployment-workflow.md) -- the single +# shared L2 segment the three Office1 service VMs (MAAS-region, NetBox, +# GitBucket), the vcloud host's own management interface, and Office1's own +# OPNsense edge's LAN side (gap #16) all attach to. +# +# DESIGN DECISION (2026-07-09, resolves gap #12's own "new module vs. host- +# bridge reuse" fork): a NEW dc-planes-shaped module, NOT a reused host +# bridge. Reasoning: +# 1. `modules/dc-planes`, `modules/mesh-link`, and `modules/dc-storage-pool` +# already establish the precedent that every virtual network/pool this +# repo depends on is a versioned, destroy/recreate-safe OpenTofu +# resource -- per D-103 ("OpenTofu owns create/destroy of ... all +# virtual networks"). A hand-configured host bridge would be the ONE +# network in this whole topology living outside that inventory -- +# exactly the "not reproducible / not versioned" debt D-103 exists to +# eliminate, and the same class of debt this repo already logs +# explicitly when Option B (manual VM creation) is used elsewhere (see +# runbooks/dc-dc-phase1-office1-standup.md's "Provisioning-path +# decision" section). +# 2. This repo's own audit/CHECK steps assume `virsh net-list --all` is a +# COMPLETE inventory of what exists on vcloud (see +# runbooks/dc-dc-phase1-office1-standup.md's Stage-1-gate CHECK). A +# pre-existing host bridge, created outside OpenTofu, would silently +# break that assumption -- a real bridge would exist that no `tofu +# plan`/`tofu destroy` knows about, and Stage 0's teardown path +# (D-061) would not reach it. +# 3. Shaped like `modules/dc-planes` but for ONE network, not six planes: +# Office1 is a headend, not an OpenStack DC, so the D-052/D-100 +# six-plane template does not apply to it -- it needs exactly one +# shared local segment, not a plane family. +# +# Isolated, not NAT/routed -- same reasoning as dc-planes/mesh-link: no +# libvirt-managed DHCP (NetBox is the IPAM apex, skill hard rule 3); +# `forward` is OMITTED entirely, which is documented native-libvirt behavior +# for a private, unforwarded virtual switch (this repo could not confirm the +# provider's `forward.mode` accepted string values from the published docs -- +# see dc-planes/main.tf's identical, still-open caveat). Confirm with `tofu +# providers schema -json` before the first real apply; add `mode = "none"` +# then if it turns out to still be required -- do not guess it in now. +# +# SYNTAX NOTE: this provider's current schema uses attribute-style nested +# objects (`domain = { ... }`, `mtu = { ... }`), not classic HCL nested +# blocks -- the same provider-wide convention confirmed against real example +# .tf files and documented in opentofu/README.md's "Syntax bug fixed +# 2026-07-09" section (dc-planes/mesh-link/dc-storage-pool all originally +# shipped with the wrong, block-style syntax and were corrected). This +# module is written directly in the corrected style from the start. + +resource "libvirt_network" "office1_local" { + name = "office1-local" + autostart = true + + domain = { + name = "office1.${var.domain_suffix}" + } + + mtu = { + size = var.mtu + } +} diff --git a/opentofu/modules/office1-network/outputs.tf b/opentofu/modules/office1-network/outputs.tf new file mode 100644 index 0000000..2da84f8 --- /dev/null +++ b/opentofu/modules/office1-network/outputs.tf @@ -0,0 +1,9 @@ +output "network_id" { + description = "libvirt network id -- for wiring into future modules/cloudinit-vm (the three Office1 service VMs) or modules/opnsense-edge (Office1's own edge LAN side) calls." + value = libvirt_network.office1_local.id +} + +output "network_name" { + description = "libvirt network name (\"office1-local\") -- pass this straight through as a cloudinit-vm network_names entry or opnsense-edge's lan_network_name; do not reconstruct it (same real-attribute-reference discipline as modules/dc-planes' network_names output)." + value = libvirt_network.office1_local.name +} diff --git a/opentofu/modules/office1-network/variables.tf b/opentofu/modules/office1-network/variables.tf new file mode 100644 index 0000000..7f012d7 --- /dev/null +++ b/opentofu/modules/office1-network/variables.tf @@ -0,0 +1,9 @@ +variable "domain_suffix" { + description = "Base domain for this network's libvirt DNS domain (yields \"office1.\"). Anchor value per D-106's naming convention; Office1 is a headend, not a per-DC cloud, so it gets one domain, not a per-plane family like modules/dc-planes. NOT re-derived here, pass it in explicitly -- same convention as dc-planes' own domain_suffix input." + type = string +} + +variable "mtu" { + description = "Underlay MTU for this network. MUST be the Phase-0 MEASURED host L2 MTU (buildout-design Section 3 / D-102) -- do not assume jumbo (9000). No default -- pass explicitly, same discipline as every other network module in this repo (dc-planes, mesh-link)." + type = number +} diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index fc871d5..ea94f14 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -27,10 +27,14 @@ (goal/build/gate) and Section 5 (OpenTofu/MAAS/NetBox/Juju boundary); `docs/dc-dc-deployment-workflow.md` Stage 2 (tracker row -- update its `**State:**` line when this runbook completes a real run) and the Tooling gap -register (items #2, #3, #9 above all bear directly on this stage); +register (items #2, #3, #9 above all bear directly on this stage; items #12 +and #16 were found and CLOSED during this runbook's own authoring/update +pass -- see Open questions #1/#2 below; item #17 was found while closing +#16 and remains OPEN, a cross-site gap, not specific to Office1); `opentofu/README.md` (module scope/status for `modules/cloudinit-vm` / -`modules/base-image`); `netbox/dc-dc-prefixes-import.py` (the NetBox -multi-DC/dual-stack importer, built 2026-07-09 -- MECHANISM only, see Step 7). +`modules/base-image` / `modules/office1-network` / `modules/opnsense-edge`); +`netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer, +built 2026-07-09 -- MECHANISM only, see Step 7). Decisions this runbook owns: **D-103** (OpenTofu/MAAS/Juju lifecycle seam -- Office1 service VMs are OpenTofu-created, eventually; MAAS owns their @@ -65,12 +69,16 @@ virsh pool-list --all ``` Expect: the six DC1 plane networks, the three mesh-link networks -(dc1<->dc2, dc1<->office1, dc2<->office1), and the `dc1`/`office1` storage -pools, all `active` -- this is `dc-dc-phase0-vcloud-prep.md`'s Step 11 output. -If any of this is missing or `inactive`, STOP -- Stage 1 is not actually done; -go re-run/complete that runbook rather than starting Stage 2 on an unfinished -Stage 1. Do not re-measure nested KVM, disk budget, or MTU here -- Stage 1 -already recorded those; this runbook only reads their OUTCOME (the networks +(dc1<->dc2, dc1<->office1, dc2<->office1), the `office1-local` network +(`opentofu/modules/office1-network`, added by this delivery -- 2026-07-09, +gap #12 CLOSED; picked up by whichever `tofu apply` run of +`dc-dc-phase0-vcloud-prep.md`'s Step 10/11 executes against this repo state, +same root `main.tf` Stage 1 already applies), and the `dc1`/`office1` +storage pools, all `active` -- this is `dc-dc-phase0-vcloud-prep.md`'s Step +11 output. If any of this is missing or `inactive`, STOP -- Stage 1 is not +actually done; go re-run/complete that runbook rather than starting Stage 2 +on an unfinished Stage 1. Do not re-measure nested KVM, disk budget, or MTU +here -- Stage 1 already recorded those; this runbook only reads their OUTCOME (the networks existing) as its gate. --- @@ -80,38 +88,76 @@ Per this repo's discipline of naming ambiguity explicitly rather than quietly picking an answer: -1. **What virtual network do the three Office1 service VMs themselves attach - to?** `opentofu/main.tf` (per Stage 1) wires `office1_pool_path` (storage - only) and the three mesh-link networks (Office1<->DC1, Office1<->DC2, - inter-site only, "management traffic only" per D-100's own sub-item - ruling). No Office1-LOCAL network (something the three service VMs and the - vcloud host's own management interface could all share) is modeled - anywhere in `opentofu/`. This is a real gap, not an oversight this runbook - papers over: either a new `modules/dc-planes`-shaped Office1-local network - needs designing, or an existing host bridge needs to be reused deliberately - (and documented as such) -- a decision for whoever designs the - `cloudinit-vm` `network_config` content in the blocked Option A path below. - Flag this to the operator before Step 3; do not invent a network name to - move forward. -2. **NetBox's own address literals are not this stage's job**, per D-101's own +1. **RESOLVED 2026-07-09 (tooling gap register item #12 CLOSED). What + virtual network do the three Office1 service VMs themselves attach to?** + `opentofu/main.tf` (per Stage 1) previously wired only `office1_pool_path` + (storage only) and the three mesh-link networks (Office1<->DC1, + Office1<->DC2, inter-site only, "management traffic only" per D-100's own + sub-item ruling) -- no Office1-LOCAL network (something the three service + VMs and the vcloud host's own management interface could all share) was + modeled anywhere in `opentofu/`. **Decision: a NEW module + (`opentofu/modules/office1-network`), not a reused host bridge.** A + dc-planes-shaped isolated `libvirt_network` sized for ONE network (Office1 + is a headend, not a six-plane OpenStack DC). Reusing an existing host + bridge was explicitly considered and rejected: it would be the one network + in this whole topology living outside OpenTofu's D-103-mandated inventory, + and would silently break this repo's own `virsh net-list --all` + completeness assumption (used as a CHECK step in this runbook's own Entry + condition, above) -- exactly the kind of untracked, unreproducible state + D-103 exists to prevent, and the same class of debt this runbook already + logs explicitly when Option B (manual VM creation) is used. Instantiated + for real in root `main.tf` as `module "office1_network"` (needs no + unmeasured value beyond `domain_suffix`/`underlay_mtu`, both already real + required inputs). See `opentofu/README.md` and + `docs/changelog-20260709-office1-network-edge.md` for the full writeup. + Use `module.office1_network.network_name` (`"office1-local"`) as the + network for each service VM's `cloudinit-vm` `network_config` and for + Step 4b's OPNsense edge LAN side below -- do not invent a different + network name. +2. **RESOLVED 2026-07-09 (tooling gap register item #16 CLOSED). Does + Office1 get its own OPNsense simulated-ISP edge, and if so, which stage + creates it?** The buildout design's topology (Section 1) is explicit: + "full dark-fiber triangle... plus a **per-site** OPNsense simulated ISP + edge" -- three sites, three edges. Stage 3's Build list already covered + DC1/DC2's edges; neither Stage 2 nor Stage 3 previously instantiated one + for Office1. **Decision: YES, Office1 gets its own `modules/opnsense-edge` + call, and Stage 2 (this runbook) owns creating it** -- new Step 4b below. + Rationale: this runbook's entire scope is already "Office1 headend + standup," and the OPNsense edge is part of that headend's own boundary, + not DC substrate; Stage 3 (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`) + has no reason to reach back into Office1's own infrastructure to build a + piece of it. This mirrors Stage 3's own `dc1_opnsense`/`dc2_opnsense` + shape exactly (same module, same variable names) -- see Step 4b. Its LAN + side resolves cleanly to open question #1's new network + (`module.office1_network.network_name`); its WAN side does NOT -- no + dedicated per-site ISP-uplink/WAN network exists for ANY site yet + (DC1/DC2 included -- confirmed by reading Stage 3's own `dc1_opnsense` + template, which carries the identical unresolved `wan_network_name` + placeholder). This is now tracked as tooling gap register item #17 (NEW), + a genuinely deeper, cross-site gap -- not invented here as an Office1-only + workaround. Step 4b's module call therefore stays commented out, same as + DC1/DC2's own not-yet-instantiated edges, until gap #17 closes and real + `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` + values exist. +3. **NetBox's own address literals are not this stage's job**, per D-101's own text ("the org ULA /48, the per-DC GUA carve... and DC2's v4 supernet are NetBox-authoritative... NOT hardcoded in this decision"). This runbook stands NetBox up and can run its import MECHANISM (Step 7), but the underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not exist yet -- see Step 7's own gate. -3. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, +4. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, pre-existing `git.baldurkeep.com` instance from the v1 rehearsal -- this stage stands up a SEPARATE, vcloud-local GitBucket instance (per the buildout design's own "GitBucket (vcloud-local mirror)" wording) -- what it mirrors, and under what repo path, is not decided here. Flag as an open item before Step 8's mutation. -4. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the +5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the front door is "Office1 only" -- the actual tailnet ACL policy, tags, and which services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the service VMs?) are exposed through it versus reachable only from Office1's own local network, is real policy design left to Step 9 and the operator's own tailnet account -- not invented here. -5. **Exact MAAS / NetBox / GitBucket version/channel pins are not decided.** +6. **Exact MAAS / NetBox / GitBucket version/channel pins are not decided.** Each install step below names the well-known official install MECHANISM (snap for MAAS, Docker Compose for NetBox, WAR/Docker for GitBucket) but does not pin an exact version/channel/tag -- consult each project's current @@ -127,9 +173,10 @@ deliberately, and log which path was used in the changelog entry. - **Option A (preferred, long-term, BLOCKED pending design):** create the VM - via `opentofu/modules/cloudinit-vm`, once (a) Open question #1 above is - resolved (what network it attaches to) and (b) real `user_data`/`meta_data`/ - `network_config` content is designed for that specific VM (what packages, + via `opentofu/modules/cloudinit-vm`, now that (a) Open question #1 above is + RESOLVED (attaches to `module.office1_network.network_name`, i.e. + `"office1-local"`) but (b) real `user_data`/`meta_data`/`network_config` + content still needs to be designed for that specific VM (what packages, what static address, what hostname). This is the D-103-correct path (`OpenTofu creates... the Office1 service VMs`) and is NOT available tonight -- doing it right requires a design pass this session did not do and this runbook @@ -152,6 +199,8 @@ 2. Confirm OpenTofu still reaches vcloud libvirt from Office1 (read-only) 3. Read the provisioning-path decision + open questions (above; no cmds) 4. MAAS region controller -- create VM (A or B) + install [MUTATION, gated] +4b. Office1 OPNsense edge -- wire module (gap #16 CLOSED; + instantiation still BLOCKED on gap #17 + real specs) [repo change, gated] 5. NetBox -- create VM (A or B) + install (official Docker) [MUTATION, gated] 6. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; literals pending -- PARTIAL) [MUTATION, gated] @@ -238,7 +287,7 @@ --memory \ --vcpus \ --disk pool=office1,size= \ - --network network= \ + --network network=office1-local \ --os-variant \ --cdrom \ --graphics none --console pty,target_type=serial @@ -281,6 +330,72 @@ --- +## Step 4b -- Office1's own OPNsense edge: wire the module [repo change, gated] (NEW -- gap #16 CLOSED) + +Per Open question #2 above (RESOLVED): Office1 gets its own +`modules/opnsense-edge` call, and this runbook (Stage 2) owns creating it -- +not Stage 3, whose runbook only ever covered DC1/DC2. This step mirrors +`runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 4/5 exactly (same module, +same prep scripts, same variable names), just for the `office1` site instead +of `dc1`/`dc2`. + +**CHECK -- confirm no pre-existing Office1 OPNsense VM under this name** +```bash +virsh -c "" list --all | grep -i office1-opnsense +``` +Expect no match (fresh standup) or a clear, understood match (re-run/repair +case). + +**MUTATION -- prepare Office1's OPNsense base image + config.xml + config ISO +(host-local files, gated)**, same mechanism as Stage 3's Step 4: +```bash +bash scripts/opnsense-prep-image.sh # see the script's own header for its exact args/output path +``` +```bash +export OPNSENSE_HOSTNAME=... DOMAIN=... ROOT_PASSWORD_HASH=... # plus every other token templates/README.md's legend marks REQUIRED for this site +bash scripts/opnsense-render-config.sh office1-opnsense-config.xml +``` +```bash +bash scripts/opnsense-build-config-iso.sh office1-opnsense-config.xml office1-opnsense-config.iso +``` +Every token filled above must be a real, measured/decided value for Office1 +specifically -- no copy-paste of DC1/DC2's config.xml content. + +**Repo change -- wire `module "office1_opnsense"` in `opentofu/main.tf`** + +A commented-out skeleton already exists in root `main.tf` (this delivery, +2026-07-09) with `lan_network_name` already resolved +(`module.office1_network.network_name`). Uncommenting it for a real run +additionally requires: +- `memory_mib` / `vcpu` / `disk_size_bytes` -- real values, OPNsense's own + sizing guidance, not invented (same "no invented specs" rule as every + other VM module in this repo). +- `base_volume_path` / `config_iso_path` -- the real output paths from the + two commands above. +- `wan_network_name` -- **BLOCKED on tooling gap register item #17** (no + dedicated per-site ISP-uplink/WAN network exists anywhere in this repo + yet, for ANY site -- confirmed against Stage 3's own `dc1_opnsense` + template, which carries the identical open placeholder). Do NOT wire this + to any `mesh-link` network: D-100's own sub-item ruling confirms those + three legs carry management traffic only, not a simulated ISP uplink. + **STOP here if gap #17 is still open** -- this step cannot be fully + completed, and that is the honest state to record, not a placeholder to + paper over. + +**GATE:** `opentofu/main.tf` diff (once gap #17 closes and every value above +is real) shows exactly one new `module "office1_opnsense"` block, every +argument a real value, nothing else changed -- same GATE shape as Stage 3's +Step 5. + +**GATE (Stage 2 Gate bullet, NEW):** Office1's own OPNsense edge module +wired with a real LAN side (achievable this stage, gap #12 resolved it) -- +full instantiation (and therefore the running edge VM itself) remains +BLOCKED pending gap #17's resolution, same as DC1/DC2's edges. Do not mark +this bullet fully green until gap #17 closes and `tofu apply` has actually +run for `module.office1_opnsense`. + +--- + ## Step 5 -- NetBox: create VM + install (official Docker method) [MUTATION: gated] **CHECK -- confirm no pre-existing NetBox VM under this name** @@ -454,14 +569,25 @@ Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row: "MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale -confirmed to Office1 only." +confirmed to Office1 only; Office1-local network created; Office1 OPNsense +edge module wired." - **MAAS region reachable:** achievable this stage (Step 4). - **OpenTofu reaches vcloud host libvirt:** already proven in Stage 1; Step 2 here only re-verifies it still holds. +- **Office1-local network created:** ACHIEVED by this delivery (gap #12 + CLOSED) -- `module "office1_network"` is instantiated for real in root + `main.tf`, no longer pending a per-run decision. +- **Office1 OPNsense edge module wired:** PARTIALLY achievable (Step 4b, gap + #16 CLOSED for ownership/LAN-side). Full instantiation and a running edge + VM remain BLOCKED on tooling gap register item #17 (no per-site + ISP-uplink/WAN network exists yet, for any of the three sites) plus real + `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` + values -- do not mark this bullet fully green until gap #17 closes and + `tofu apply` has actually run for `module.office1_opnsense`. - **GitBucket serving:** achievable this stage (Step 7). - **Tailscale confirmed to Office1 only:** achievable this stage (Step 8), - contingent on the operator's own ACL policy design (Open question #4). + contingent on the operator's own ACL policy design (Open question #5). - **NetBox authoritative and populated:** ONLY PARTIALLY achievable right now. The instance can be stood up and reachable (Step 5), and the import MECHANISM exists and is runnable (Step 6) -- but the real address literals @@ -472,11 +598,14 @@ against this real instance. This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** --- MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in full tonight's -sequence; the NetBox bullet closes its mechanism half only, pending the D-101 -literal assignment that is explicitly out of scope for this stage to invent. -Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever -of these is actually true after a real run, not to "DONE" wholesale. +-- MAAS/GitBucket/Tailscale/OpenTofu-reach/Office1-network bullets closeable +in full tonight's sequence; the Office1-OPNsense-edge bullet closes its +ownership/LAN-side half only (gap #16), pending gap #17's cross-site +resolution; the NetBox bullet closes its mechanism half only, pending the +D-101 literal assignment that is explicitly out of scope for this stage to +invent. Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect +whichever of these is actually true after a real run, not to "DONE" +wholesale. -> Proceed to Stage 3 (OpenTofu builds each DC substrate) only once the achievable bullets above are actually green; the NetBox DATA gap does not @@ -508,8 +637,13 @@ resulting debt explicitly (a DOCFIX candidate: migrate that VM to `opentofu/modules/cloudinit-vm` once its cloud-init content is designed) rather than letting it go unrecorded. -- [ ] If Open question #1 (Office1-local network) was resolved this run, - record the resolution and, if it implies a reusable OpenTofu module - (an Office1-local `dc-planes`-shaped network), flag it as a follow-on - delivery rather than silently leaving it as a one-off manual libvirt - network. +- [x] Open question #1 (Office1-local network) RESOLVED 2026-07-09 -- + `opentofu/modules/office1-network` built and instantiated for real in + root `main.tf` (gap #12 CLOSED). See + `docs/changelog-20260709-office1-network-edge.md`. +- [x] Open question #2 (Office1 OPNsense edge ownership) RESOLVED 2026-07-09 + -- Stage 2 owns it (this runbook's new Step 4b); gap #16 CLOSED. Module + skeleton wired (commented) in root `main.tf`; full instantiation + remains blocked on gap #17 (per-site ISP-uplink/WAN network, not yet + resolved for any of the three sites) plus real VM specs -- do not close + Step 4b's own GATE bullet until that instantiation actually runs.