diff --git a/docs/changelog-20260713-d114-voffice1-nested-virt.md b/docs/changelog-20260713-d114-voffice1-nested-virt.md index b93d355..b98afd2 100644 --- a/docs/changelog-20260713-d114-voffice1-nested-virt.md +++ b/docs/changelog-20260713-d114-voffice1-nested-virt.md @@ -76,3 +76,35 @@ through for the same reason. NOT fixed here: DC1 is explicitly GATED behind Office1 completing (D-114), and fixing it now would be scope beyond the current step. It must be fixed before Stage 3 or the DC nodes will come up unable to run a single instance. + +--- + +## ADDENDUM (same day): the DC1 fix is APPLIED, not deferred + +Operator ruling: **"Apply the fixes to DC1 as we find them. Fixes are not blocked, just deployment +at this stage."** So the `modules/node-vm` defect logged above as "NOT actioned" is now FIXED. + +`modules/node-vm` gained an UNCONDITIONAL `cpu = { mode = "host-passthrough" }` -- no +`expose_nested_virt` knob, deliberately. A DC node runs `nova-compute`; it MUST be able to run KVM +guests. There is no correct value other than this one, so no knob that could only ever be set wrong. +(Contrast `cloudinit-vm`, where the flag IS a real per-VM decision, and `opnsense-edge`, which +DISABLES svm as correct hardening for a router.) + +`modules/node-vm` is not yet instantiated, so adding this breaks no live call. Without it the DC +nodes would have come up looking healthy while being unable to launch a single instance -- +the failure would have surfaced during Stage 5 as an inscrutable scheduler error. + +`scripts/opentofu-validate.sh`: PASS (node-vm, cloudinit-vm, all modules). + +## ADDENDUM: voffice1 moved to a RESERVED, out-of-pool address + +`voffice1` first came up on `10.10.0.100` -- the first address of Kea's DYNAMIC pool +(.100-.199). Infrastructure, and a MAAS region controller above all, should not hold a +pool address that can move. + +A Kea host reservation was added through the D-113(a2) REST API (a second live exercise of that +write path): `52:54:00:6a:87:e5 -> 10.10.0.20`, deliberately OUTSIDE the dynamic pool. +`kea/dhcpv4/add_reservation` -> `{"result":"saved"}`, then `kea/service/reconfigure` -> `{"status":"ok"}`. +voffice1 was rebooted and came back on **10.10.0.20**; `.100` is released back to the pool; +`/dev/kvm` still present. Done NOW because nothing yet depends on the address -- it is the cheapest +moment it will ever be. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 9d84773..1d63e0c 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -56,33 +56,57 @@ --- -## Stage 2 -- Phase 1: Office1 headend standup +## Stage 2 -- Phase 1: Office1 SITE standup (RESHAPED 2026-07-13 by D-114) + +> **!!! MODEL CHANGE (D-114, ADOPTED 2026-07-13). The three-sibling-service-VM model is +> SUPERSEDED.** Stage 2 no longer builds three peer VMs (MAAS-region, NetBox, GitBucket) on the +> vcloud host with Tailscale on the headend host. It builds **ONE Office1 site containment VM, +> `voffice1`** (Ubuntu 24.04, OpenTofu-created), which IS the facility: **MAAS-region runs ON it, +> LXD (pinned to the 5.21 LTS track) runs ON it**, that LXD is **registered BACK INTO MAAS as an +> LXD KVM host**, and the non-stack service machines (**NetBox, GitBucket, Tailscale**) are VMs +> **COMPOSED BY MAAS** into it -- MAAS-visible: enlisted, commissioned, deployed, powered. This +> is VR0's own already-proven `lxd` + `tailscale` pattern, applied per site. +> +> **Explicitly NOT affected:** the Juju-deployed OpenStack services on the DC nodes. They stay in +> Juju-created LXD **CONTAINERS** (bundle `lxd:N` placements), invisible to MAAS, exactly as in +> VR0. D-114 grants MAAS a composition right scoped ONLY to the LXD VM host and the non-stack +> machines; `modules/node-vm` still pre-creates the OpenStack node VMs and MAAS merely DISCOVERS +> them (D-103 unchanged on that point). +> +> **Sequencing is an operator ruling:** Office1 FIRST, fully up on this model. **DC1 is GATED +> behind it** and is not started until Office1 is complete; DC2 follows DC1. Stages 3-7 below are +> therefore gated behind THIS stage, and their own content is otherwise unchanged by D-114. | | | |---|---| -| **Goal** | Stand up the operator's deployment headend first, before any DC. | -| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only); Office1's own OPNsense simulated-ISP edge (gap #16, DECIDED 2026-07-09: Stage 2 owns it). | -| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only; Office1-local network created (gap #12 CLOSED); **Office1 OPNsense edge BUILT AND LIVE 2026-07-12/13 -- gate MET** (routing, NAT, egress, DHCP; gap #17 closed for Office1 via `office1-wan`). | -| **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | -| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | -| **Authoring status** | **Runbook WRITTEN 2026-07-09, UPDATED 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. The Office1-local-network gap it originally flagged (gap #12) is now CLOSED (`opentofu/modules/office1-network`, instantiated for real), and a new Step 4b now creates Office1's own OPNsense edge (gap #16, CLOSED -- ownership decided). NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | +| **Goal** | Stand up the Office1 SITE first, before any DC -- and prove the D-114 containment-VM model at its cheapest point. | +| **Build** | `voffice1` site containment VM (OpenTofu, `modules/cloudinit-vm` + `modules/base-image`, on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`); MAAS region controller ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); that LXD registered into MAAS as an LXD KVM host; NetBox / GitBucket / Tailscale as MAAS-COMPOSED LXD VMs inside it; Office1's own OPNsense simulated-ISP edge (gap #16, Stage 2 owns it). | +| **Gate** | Office1 OPNsense edge up (**MET** -- routing, NAT, egress, Kea DHCP; gap #17 closed for Office1 via `office1-wan`); `office1-local` network created (gap #12 CLOSED); OpenTofu reaches vcloud host libvirt (**MET**); **`voffice1` exists and runs (MET 2026-07-13, measured)**; `voffice1` has a reserved address + SSH + nested KVM exposed; MAAS region reachable ON `voffice1`; LXD on the 5.21 track with `lxdbr0` OFF `office1-local`; LXD registered as a MAAS KVM host; **the non-stack machines COMPOSED BY MAAS and deployed -- i.e. an LXD VM actually BOOTS at L3, which is D-114's DC1 entry gate**; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; Tailscale confirmed to Office1 only. | +| **Owns** | **D-114** (site containment VM + MAAS-composed LXD VMs, incl. the LXD 5.21 pin), D-103 as AMENDED by D-114 (OpenTofu owns the containment VM and the site networks; MAAS composes INSIDE the site), D-107 (headend is not a core-service provider). | +| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but LESS new than it looks: the MAAS-composed-LXD-VM half is **exactly what VR0 runs today** (its `lxd` KVM host at LXD 5.21.4, with `tailscale` composed into it at 2 cores / 2 GiB / 25 GB) and carries **NO Roosevelt delta**. The containment VM itself IS a deliberate simulation-only delta -- it is what makes a "hard-down an entire facility" DR drill (`virsh destroy vdc1`) executable at all. NetBox-as-IPAM-apex discipline and the `netbox/*-import.py` pattern transfer unchanged. | +| **Authoring status** | **Runbook REWRITTEN 2026-07-13 to the D-114 model: `runbooks/dc-dc-phase1-office1-standup.md`.** The old Option A / Option B provisioning fork is **CLOSED** -- OpenTofu (Option A) is the path and `module "voffice1"` is a real, applied `cloudinit-vm` instantiation (real image source, real `user_data`/`meta_data`/`network_config`, plus the new `expose_nested_virt` variable). **PENDING VERIFICATION:** the MAAS/LXD install + LXD-KVM-host registration + VM-compose command lines are NOT yet confirmed against Canonical's current docs -- the runbook marks each unconfirmed flag/channel/subcommand rather than fabricating one, and a separate verification pass owns filling them in. | -**State (CORRECTED 2026-07-13): PARTIALLY EXECUTED.** +**State (2026-07-13): PARTIALLY EXECUTED -- substrate DONE, site half remaining.** - **DONE:** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is BUILT and LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on udp/67). - Config is now REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt (11 resources - applied). Tailscale reaches the jumphost via a pre-existing subnet route. -- **NOT DONE -- this is what remains of Stage 2:** the **three headend service VMs** - (MAAS-region, NetBox, GitBucket) do not exist. Measured 2026-07-13: `office1-opnsense` is the - ONLY VM on the vcloud host. The NetBox/GitBucket that answer at `baldurkeep.com` are - PRE-EXISTING services, NOT Office1 headend VMs -- do not mistake them for a met gate. - Per the runbook, these are blocked on a real decision: Option A (`cloudinit-vm`) needs - `user_data`/`meta_data`/`network_config` designed and an image source chosen; Option B is - manual `virt-install`, logged as debt. `modules/cloudinit-vm` + `modules/base-image` now at - least VALIDATE (DOCFIX-194, 2026-07-13) -- they never had before. + Config is now REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1` + EXISTS and is RUNNING** (measured 2026-07-13: `virsh list --all`; `module.voffice1` is in + `opentofu/terraform.tfstate`) -- the D-114 containment VM is applied. +- **NOT DONE -- this is what remains of Stage 2:** MAAS-region ON `voffice1`; LXD ON `voffice1` + (5.21 LTS track); the LXD-KVM-host registration back into MAAS; and **every service machine** + (NetBox, GitBucket, Tailscale), which are now MAAS-COMPOSED LXD VMs, not standalone libvirt + domains. The NetBox/GitBucket that answer at `baldurkeep.com` are PRE-EXISTING services, NOT + Office1 machines -- do not mistake them for a met gate. Likewise, Tailscale reaching the + jumphost via a pre-existing subnet route is a DIFFERENT node and does not meet Stage 2's + Tailscale bullet. +- **The L3 nesting proof is the load-bearing unknown.** LXD *virtual machines* are qemu/KVM + guests, so composing the service machines inside `voffice1` needs nested KVM at L3 -- the same + depth the DC nodes will need. Office1 is deliberately the CHEAP place to prove it (no + OpenStack, no Ceph, small VMs). If an LXD VM boots inside `voffice1`, DC1 inherits the proof; + if it does not, D-114's model is abandoned early having cost one VM. - NetBox's "authoritative and populated" bullet remains PARTIAL (mechanism done via DOCFIX-152; - literals still unassigned). + literals still unassigned) -- unchanged by D-114. --- @@ -261,6 +285,20 @@ every DC node and `netem-link` is the Stage 6 failover-drill mechanism** -- they would have detonated mid-deploy. See `docs/changelog-20260713-docfix194-opentofu-module-validation.md`. + - **`modules/cloudinit-vm` IS NOW INSTANTIATED FOR REAL (2026-07-13, D-114).** + The long-standing "the mechanism exists but no image source is chosen and no + `user_data`/`meta_data`/`network_config` has been designed for it" caveat -- + which is what forced Stage 2's old Option A / Option B fork -- is **CLOSED + for `voffice1`**: `module "voffice1"` in root `main.tf` carries a real image + source (`modules/base-image` -> the official Ubuntu 24.04 noble cloud + image), real cloud-init content, and a `network_config` that DHCPs off the + Office1 edge's Kea (matching the NIC by glob, not by a guessed kernel name). + The module also **gained an `expose_nested_virt` variable** -- load-bearing, + because LXD virtual machines are qemu/KVM guests and D-114's composed + service machines cannot boot without the host CPU feature passed through. + Applied; the domain is running. **The three Office1 service VMs are NOT + cloudinit-vm targets any more at all** -- under D-114 they are MAAS-COMPOSED + LXD VMs, so no further `cloudinit-vm` design work is owed for them. - **Still true:** `tofu plan`/`apply` has NOT been exercised for the Stage 3 DC substrate. Validation is not application. @@ -479,9 +517,16 @@ this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`, already real required inputs elsewhere in the same file. See `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` - (DOCFIX-163) for the full design rationale. Designing the three - service VMs' actual `cloudinit-vm` `network_config` content against this - network remains separate, un-actioned work. + (DOCFIX-163) for the full design rationale. **Follow-on work also DONE + 2026-07-13 (D-114):** the "designing the service VMs' actual `cloudinit-vm` + `network_config` against this network" tail of this gap is closed, but not + the way it was written -- there are no longer three service VMs to attach. + ONE VM attaches to `office1-local`: the `voffice1` containment VM, whose + `network_config` DHCPs from the OPNsense edge's Kea. The service machines + live INSIDE `voffice1`'s LXD (MAAS-composed) and never touch this network + directly. **DHCP hazard to carry forward:** LXD's `lxdbr0` runs its OWN + dnsmasq -- it must stay on LXD's internal NAT bridge and must NEVER be + bridged onto `office1-local`, where Kea is authoritative. 13. **The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real risk still open.** `overlays/dc-dc-ipv6-family-matrix.yaml` now exists, sourced against real charm config/source (`docs/dc-dc-ipv6-charm- @@ -567,6 +612,12 @@ #12's new network; its WAN side does not -- see NEW gap #17 immediately below, a genuinely deeper, cross-site gap surfaced while closing this one, not invented as a substitute for a real answer. + **D-114 follow-on (2026-07-13):** the edge is now BUILT and LIVE, and it is + a load-bearing PREREQUISITE of the rest of Stage 2, not just a gate bullet + -- `voffice1` takes its DHCP lease from the edge's Kea and reaches the + internet through it. Office1's edge is also the only one of the three that + is live; DC1/DC2's edges are Stage 3's problem and are still blocked on gap + #17's DC half. 17. **PARTIALLY CLOSED 2026-07-13 -- the DECISION is made and PROVEN for Office1; DC1/DC2 still have no uplink network.** - **Office1: CLOSED.** The answer was the one this gap proposed -- a @@ -663,8 +714,9 @@ | Design (`dc-dc-buildout-design.md`) | DONE -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) | | Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED | | DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | -| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. NONE executed against real infrastructure (prep-only session). See gaps #12-15 above for real design gaps the authoring pass surfaced. | -| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (OpenTofu v1.12.3 now on the jumphost): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds 11 resources (DC1 planes, pools, 3 mesh links, office1-network, the OPNsense edge). **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. | +| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. Stage 1 EXECUTED 2026-07-10. **Stage 2's runbook REWRITTEN 2026-07-13 to the D-114 model** (one `voffice1` site containment VM + MAAS-composed LXD VMs, replacing the three sibling service VMs and their Option A/B fork); it is PARTIALLY EXECUTED, and its MAAS/LXD command lines are marked PENDING VERIFICATION rather than guessed. Stages 3-7 NOT executed, and D-114 GATES them behind Office1 completing. See gaps #12-15 above for real design gaps the authoring pass surfaced. | +| **D-114 site containment VM (`voffice1`)** | **BUILT + RUNNING 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. | +| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (OpenTofu v1.12.3 now on the jumphost): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds the DC1 planes, pools, 3 mesh links, `office1-network`, the OPNsense edge, and (2026-07-13, D-114) the `ubuntu_noble_base` image + the **`voffice1` containment VM**. **`modules/cloudinit-vm` is now instantiated for real** and gained `expose_nested_virt`. **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. | | OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + RUNNING 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. | | OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **SUPERSEDED 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **OPEN:** reduce the template to a MINIMAL bootstrap (sshd + root key + console + a pre-seeded API key) and retire the ISO path + the module's `config_seed`/cdrom wiring. | | MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` | diff --git a/opentofu/modules/node-vm/main.tf b/opentofu/modules/node-vm/main.tf index 71baaaf..a915675 100644 --- a/opentofu/modules/node-vm/main.tf +++ b/opentofu/modules/node-vm/main.tf @@ -65,6 +65,22 @@ apic = {} } + # CPU: host-passthrough, with the host's `svm` feature PASSED THROUGH. This is + # UNCONDITIONAL and has no opt-out on purpose -- a DC node runs `nova-compute`, so it + # MUST be able to run KVM guests. There is no correct value other than this one. + # + # WHY THIS IS EXPLICIT: with NO cpu block at all, libvirt renders a GENERIC EMULATED CPU + # MODEL that exposes no `svm` -- and the node then comes up looking healthy while being + # unable to launch a single instance. Measured 2026-07-13: the host is an AMD EPYC 9965, + # but a default-CPU guest is handed an "Opteron_G3" with no svm. The identical defect was + # found in modules/cloudinit-vm the same day and broke D-114's model outright. + # + # Contrast modules/opnsense-edge, which DISABLES svm: correct hardening there, because a + # router guest has no business running VMs. A compute node is the opposite case. + cpu = { + mode = "host-passthrough" + } + os = { type = "hvm" type_arch = "x86_64" diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index 8b97fd3..196b983 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -1,85 +1,136 @@ -# DC-DC Phase 1 -- Office1 headend standup (Stage 2) +# DC-DC Phase 1 -- Office1 site standup (Stage 2) -Stand up the operator's deployment headend -- MAAS region controller, confirmed -OpenTofu reach, NetBox, GitBucket, Tailscale -- BEFORE either DC substrate -exists. This is the second DC-DC runbook; it follows -`runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit gate is this -runbook's entry condition. Per this repo's session contract, do NOT re-derive -anything Stage 1 already settled (nested KVM, disk budget, MTU, the DC1/Office1 -libvirt planes and pools, the mesh-link networks) -- confirm Stage 1's gate -passed and move on. +> **REWRITTEN 2026-07-13 to the D-114 model.** The previous version of this +> runbook built THREE SIBLING SERVICE VMs on the vcloud host (MAAS-region, +> NetBox, GitBucket), each behind an "Option A (OpenTofu, blocked) / Option B +> (manual virt-install)" fork, with Tailscale on the headend host. **That model +> is SUPERSEDED.** D-114 (ADOPTED 2026-07-13) replaces it with ONE Office1 site +> containment VM (`voffice1`) that IS the facility, with MAAS + LXD running ON +> it and the non-stack service machines COMPOSED BY MAAS into that LXD. Do not +> work from a cached memory of the old shape -- it will build the wrong thing. -**Authoring context (read before running):** this runbook was authored PREP-ONLY, -with no live vcloud host, MAAS, NetBox, GitBucket, or Tailscale reachable in the -authoring session. Every command below is written to be run for real, later, -by the operator -- it has not been executed against real infrastructure. Per -`opentofu/README.md` and `docs/dc-dc-deployment-workflow.md` Stage 2 (tooling -gap register items #2, #3, #9): **none of the three Office1 service VMs -(MAAS-region, NetBox, GitBucket) have an OpenTofu module instantiation yet** -- -`opentofu/modules/cloudinit-vm` exists as a mechanism, but no image source has -been chosen and no `user_data`/`meta_data`/`network_config` content has been -designed for any of the three. This runbook does not invent that content. It -gives each VM-creation step two explicit paths (Section "Provisioning-path -decision" below) and flags the OpenTofu path as blocked-pending-design rather -than silently choosing the manual path for the operator. +Stand up the Office1 SITE: the containment VM that simulates the facility, the +MAAS region controller that runs inside it, the LXD VM host MAAS composes into, +and the non-stack service machines (NetBox, GitBucket, Tailscale) as +MAAS-visible VMs -- BEFORE either DC substrate exists. This is the second DC-DC +runbook; it follows `runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit +gate is this runbook's entry condition. Per this repo's session contract, do NOT +re-derive anything Stage 1 already settled (nested KVM, disk budget, MTU, the +DC1/Office1 libvirt planes and pools, the mesh-link networks) -- confirm Stage +1's gate passed and move on. + +**Execution status (measured 2026-07-13, not inferred):** + +- Office1's OPNsense edge (`office1-opnsense`) is **BUILT, RUNNING, ROUTING, and + SERVING DHCP** (Kea). Its config is REST-API-managed per D-113(a2). +- `office1-local` and `office1-wan` libvirt networks exist and are active. +- **`voffice1` EXISTS and is RUNNING** (`virsh list --all` -> `voffice1 running`; + `module.voffice1` is in `opentofu/terraform.tfstate`). It was applied by the + main session's `tofu apply`, not by this runbook's authoring pass. +- **NOT yet done:** everything from Step 5 onward -- MAAS-region on `voffice1`, + LXD on `voffice1`, the LXD-KVM-host registration, and every MAAS-composed + service machine. Those are this runbook's remaining work. **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 Phase 1 (goal/build/gate) and Section 5 (OpenTofu/MAAS/NetBox/Juju boundary); `docs/dc-dc-deployment-workflow.md` Stage 2 (tracker row -- update its `**State:**` line when this runbook completes a real run) and the Tooling gap -register (items #2, #3, #9 above all bear directly on this stage; items #12 -and #16 were found and CLOSED during this runbook's own authoring/update -pass -- see Open questions #1/#2 below; item #17 was found while closing -#16 and remains OPEN, a cross-site gap, not specific to Office1); -`opentofu/README.md` (module scope/status for `modules/cloudinit-vm` / -`modules/base-image` / `modules/office1-network` / `modules/opnsense-edge`); -`netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer, -built 2026-07-09 -- MECHANISM only, see Step 7). +register; `opentofu/README.md` (module scope/status for `modules/cloudinit-vm` +/ `modules/base-image` / `modules/office1-network` / `modules/opnsense-edge`); +`netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer -- +MECHANISM only, see Step 10). -Decisions this runbook owns: **D-103** (OpenTofu/MAAS/Juju lifecycle seam -- -Office1 service VMs are OpenTofu-created, eventually; MAAS owns their -provisioning lifecycle once enlisted the same way DC node VMs are), **D-107** -(airgap posture -- Office1 is explicitly OUT of the core-service path: no node -artifacts, no NTP served from Office1; Tailscale is a narrowly-scoped front -door, not a general egress path), **D-106** (naming convention -- Office1 -services use a region-level subdomain outside the per-cloud convention, e.g. -`maas.office1.vr1...`; exact scheme is this stage's job to fix, not yet fixed), -and **D-101** (referenced only -- NetBox is the IPAM apex this stage stands up, -but the address literals D-101 requires are NOT this stage's job to assign). +Decisions this runbook owns: **D-114** (site containment VMs + MAAS-composed +LXD VMs for the non-stack machines -- the model this runbook now implements, +including the LXD 5.21 LTS pin), **D-103** as AMENDED by D-114 (OpenTofu owns +the site containment VM and the site's networks; INSIDE a site, MAAS composes +the service machines -- MAAS still owns the provisioning lifecycle and still +does NOT compose the OpenStack node VMs), **D-107** (airgap posture -- Office1 +is explicitly OUT of the core-service path: no node artifacts, no NTP served +from Office1; Tailscale is a narrowly-scoped front door, not a general egress +path), **D-106** (naming convention -- Office1 services use a region-level +subdomain outside the per-cloud convention, e.g. `maas.office1.vr1...`; exact +scheme is this stage's job to fix, not yet fixed), and **D-101** (referenced +only -- NetBox is the IPAM apex this stage stands up, but the address literals +D-101 requires are NOT this stage's job to assign). -!!! PREP-ONLY / NOT YET EXECUTED. No live infrastructure exists or was - reachable in the authoring session. Every MUTATION below is written for - the operator to run and individually gate later, per this repo's - verify-before-mutate discipline -- none of it has been run. +!!! PENDING VERIFICATION -- the MAAS/LXD install-and-compose sequence (Steps + 5, 6, 7, 8) is the NEW heart of this runbook and its exact command lines + are NOT yet confirmed against Canonical's current documentation. Those + steps name the MECHANISM and the GATE, and mark every unconfirmed flag, + channel, and subcommand as ``. A separate + verification pass owns filling them in. Per this repo's cardinal rule, a + fabricated flag is worse than an honest placeholder -- do NOT "reconstruct" + these from memory, from a different MAAS version's docs, or from this + repo's own `scripts/phase-00-maas-standup.sh` / `scripts/reenroll-hosts.sh` + (those are the PXE-enlisted METAL-NODE path, a DIFFERENT mechanism from + D-114's COMPOSE path -- see Step 8's own note). -!!! STRUCTURAL BLOCKER, flagged not fixed here: the OpenTofu path for all - three Office1 service VMs needs `user_data`/`meta_data`/`network_config` - designed against `modules/cloudinit-vm` before it can be used -- that - design work has NOT been done. Step 3 below presents the resulting - Option A (blocked) / Option B (manual, interim) fork explicitly for each - VM. Do not silently pick one without reading Step 3 first. +!!! LXD IS VERSION-PINNED to the **5.21 LTS track** (D-114). MAAS 3.6/3.7 is + INCOMPATIBLE with LXD >= 6.7: LXD 6.7 consolidated API endpoints and MAAS's + pinned pylxd 2.3.5 cannot speak to them. Canonical's guidance is "use LXD + <= 6.6 or the 5.21 LTS release until further notice" + (https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749). VR0 + runs 5.21.4. Installing LXD from the default `latest/stable` channel would + silently break the MAAS-LXD seam this entire stage depends on. The pin + belongs in `runbooks/appendix-B-asbuilt-version-lock.md` -- record it there + when the real installed version is measured. + +!!! DHCP HAZARD (D-114). OPNsense/Kea is AUTHORITATIVE for DHCP on + `office1-local` (10.10.0.0/24, pool .100-.199). LXD's own `lxdbr0` runs a + SECOND dnsmasq. That is FINE as long as `lxdbr0` stays on LXD's internal + NAT bridge and is NEVER bridged onto `office1-local`. Two DHCP servers on + one L2 is an intermittent, genuinely unpleasant failure to diagnose. Any + LXD networking change in Step 6 must not put `lxdbr0` on the site LAN. --- -## Entry condition (Stage 1 gate -- confirm, do not re-derive) +## Entry condition (Stage 1 gate + the Office1 edge -- confirm, do not re-derive) **CHECK -- read-only, before starting anything in this runbook** ```bash virsh net-list --all virsh pool-list --all +virsh list --all ``` Expect: the six DC1 plane networks, the three mesh-link networks (dc1<->dc2, dc1<->office1, dc2<->office1), the `office1-local` network -(`opentofu/modules/office1-network`, added by this delivery -- 2026-07-09, -gap #12 CLOSED; picked up by whichever `tofu apply` run of -`dc-dc-phase0-vcloud-prep.md`'s Step 10/11 executes against this repo state, -same root `main.tf` Stage 1 already applies), and the `dc1`/`office1` -storage pools, all `active` -- this is `dc-dc-phase0-vcloud-prep.md`'s Step -11 output. If any of this is missing or `inactive`, STOP -- Stage 1 is not -actually done; go re-run/complete that runbook rather than starting Stage 2 -on an unfinished Stage 1. Do not re-measure nested KVM, disk budget, or MTU -here -- Stage 1 already recorded those; this runbook only reads their OUTCOME (the networks -existing) as its gate. +(`opentofu/modules/office1-network`), the `office1-wan` network (the Office1 +ISP uplink -- gap #17 CLOSED for Office1), and the `dc1`/`office1`/`dc2` +storage pools, all `active`. Expect the `office1-opnsense` domain `running`. +If any of this is missing or `inactive`, STOP -- Stage 1 (or the Office1 edge +build) is not actually done; go complete it rather than starting the rest of +Stage 2 on an unfinished substrate. Do not re-measure nested KVM, disk budget, +or MTU here -- Stage 1 already recorded those; this runbook only reads their +OUTCOME (the networks and the edge existing) as its gate. + +--- + +## The D-114 model in one paragraph (read before Step 1) + +Office1 is a SITE, not three peer VMs on a host. OpenTofu creates ONE +containment VM -- `voffice1`, Ubuntu 24.04, on `office1-local`, taking a DHCP +lease from the OPNsense edge's Kea, reaching the internet through that edge like +a real server behind a real site router. `voffice1` IS the facility. The MAAS +REGION CONTROLLER runs ON it. LXD (5.21 LTS) also runs ON it, and that LXD is +REGISTERED BACK INTO MAAS as an LXD KVM host. The non-stack service machines -- +NetBox, GitBucket, Tailscale -- are then VMs **composed by MAAS** into that LXD +host: MAAS-visible, enlisted, commissioned, deployed, powered, releasable. This +is not a new pattern; it is VR0's own already-proven `lxd` + `tailscale` pair, +applied per site. Explicitly NOT in scope: the Juju-deployed OpenStack services +on the DC nodes -- those stay in Juju-created LXD CONTAINERS (bundle `lxd:N` +placements), invisible to MAAS, exactly as in VR0. Sequencing is an operator +ruling: **Office1 FIRST, fully up. DC1 is GATED behind it. DC2 follows DC1.** + +**`expose_nested_virt = true` on `voffice1` is LOAD-BEARING, not a nicety.** LXD +*virtual machines* are qemu/KVM guests, so composing NetBox/GitBucket/Tailscale +as LXD VMs inside `voffice1` requires nested KVM at L3. (Only LXD *containers* +would be free of that requirement -- and MAAS does not enlist containers.) +Office1 is therefore the CHEAP nesting probe for the whole D-114 model: no +OpenStack, no Ceph, small VMs. **If an LXD VM boots inside `voffice1`, the pod +model is proven to L3 and DC1 inherits that proof** (D-114's own DC1 entry +gate). If it does not, the containment-VM model is abandoned early having cost +one VM -- a scoped retreat, by design, not a redesign. --- @@ -88,126 +139,84 @@ Per this repo's discipline of naming ambiguity explicitly rather than quietly picking an answer: -1. **RESOLVED 2026-07-09 (tooling gap register item #12 CLOSED). What - virtual network do the three Office1 service VMs themselves attach to?** - `opentofu/main.tf` (per Stage 1) previously wired only `office1_pool_path` - (storage only) and the three mesh-link networks (Office1<->DC1, - Office1<->DC2, inter-site only, "management traffic only" per D-100's own - sub-item ruling) -- no Office1-LOCAL network (something the three service - VMs and the vcloud host's own management interface could all share) was - modeled anywhere in `opentofu/`. **Decision: a NEW module - (`opentofu/modules/office1-network`), not a reused host bridge.** A - dc-planes-shaped isolated `libvirt_network` sized for ONE network (Office1 - is a headend, not a six-plane OpenStack DC). Reusing an existing host - bridge was explicitly considered and rejected: it would be the one network - in this whole topology living outside OpenTofu's D-103-mandated inventory, - and would silently break this repo's own `virsh net-list --all` - completeness assumption (used as a CHECK step in this runbook's own Entry - condition, above) -- exactly the kind of untracked, unreproducible state - D-103 exists to prevent, and the same class of debt this runbook already - logs explicitly when Option B (manual VM creation) is used. Instantiated - for real in root `main.tf` as `module "office1_network"` (needs no - unmeasured value beyond `domain_suffix`/`underlay_mtu`, both already real - required inputs). See `opentofu/README.md` and - `docs/changelog-20260709-office1-network-edge.md` for the full writeup. - Use `module.office1_network.network_name` (`"office1-local"`) as the - network for each service VM's `cloudinit-vm` `network_config` and for - Step 4b's OPNsense edge LAN side below -- do not invent a different - network name. -2. **RESOLVED 2026-07-09 (tooling gap register item #16 CLOSED). Does - Office1 get its own OPNsense simulated-ISP edge, and if so, which stage - creates it?** The buildout design's topology (Section 1) is explicit: - "full dark-fiber triangle... plus a **per-site** OPNsense simulated ISP - edge" -- three sites, three edges. Stage 3's Build list already covered - DC1/DC2's edges; neither Stage 2 nor Stage 3 previously instantiated one - for Office1. **Decision: YES, Office1 gets its own `modules/opnsense-edge` - call, and Stage 2 (this runbook) owns creating it** -- new Step 4b below. - Rationale: this runbook's entire scope is already "Office1 headend - standup," and the OPNsense edge is part of that headend's own boundary, - not DC substrate; Stage 3 (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`) - has no reason to reach back into Office1's own infrastructure to build a - piece of it. This mirrors Stage 3's own `dc1_opnsense`/`dc2_opnsense` - shape exactly (same module, same variable names) -- see Step 4b. Its LAN - side resolves cleanly to open question #1's new network - (`module.office1_network.network_name`); its WAN side does NOT -- no - dedicated per-site ISP-uplink/WAN network exists for ANY site yet - (DC1/DC2 included -- confirmed by reading Stage 3's own `dc1_opnsense` - template, which carries the identical unresolved `wan_network_name` - placeholder). This is now tracked as tooling gap register item #17 (NEW), - a genuinely deeper, cross-site gap -- not invented here as an Office1-only - workaround. Step 4b's module call therefore stays commented out, same as - DC1/DC2's own not-yet-instantiated edges, until gap #17 closes and real - `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` - values exist. +1. **RESOLVED 2026-07-13 by D-114. The Option A / Option B provisioning fork is + CLOSED.** The old fork ("OpenTofu `cloudinit-vm`, blocked pending user_data + design" vs "manual `virt-install`, logged as debt") existed because no + `cloudinit-vm` instantiation had ever been designed. **OpenTofu (Option A) is + the path, and it is DONE:** `module "voffice1"` in `opentofu/main.tf` is a + real, applied `cloudinit-vm` instantiation -- real image source + (`modules/base-image`, the official Ubuntu 24.04 noble cloud image), real + `user_data` (identity + access + guest agent), real `meta_data`, real + `network_config` (DHCP on `office1-local`, matched by interface GLOB rather + than a guessed kernel NIC name), and the new `expose_nested_virt` variable. + There is nothing left to fork on: **the three service VMs are no longer + OpenTofu VMs at all** -- they are MAAS-composed LXD VMs (Step 8). Do not + re-introduce a manual `virt-install` path for them. +2. **The exact MAAS/LXD install, LXD-KVM-host registration, and VM-compose + command lines are NOT yet verified.** See the PENDING VERIFICATION banner + above. Steps 5-8 carry the placeholders; a separate verification pass against + Canonical's current docs owns filling them in. Do not run those steps from a + remembered flag set. 3. **NetBox's own address literals are not this stage's job**, per D-101's own text ("the org ULA /48, the per-DC GUA carve... and DC2's v4 supernet are - NetBox-authoritative... NOT hardcoded in this decision"). This runbook - stands NetBox up and can run its import MECHANISM (Step 7), but the - underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not - exist yet -- see Step 7's own gate. -4. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, - pre-existing `git.baldurkeep.com` instance from the v1 rehearsal -- this - stage stands up a SEPARATE, vcloud-local GitBucket instance (per the - buildout design's own "GitBucket (vcloud-local mirror)" wording) -- what - it mirrors, and under what repo path, is not decided here. Flag as an - open item before Step 8's mutation. -5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the - front door is "Office1 only" -- the actual tailnet ACL policy, tags, and - which services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the - service VMs?) are exposed through it versus reachable only from Office1's - own local network, is real policy design left to Step 9 and the operator's - own tailnet account -- not invented here. -6. **Exact MAAS / NetBox / GitBucket version/channel pins are not decided.** - Each install step below names the well-known official install MECHANISM - (snap for MAAS, Docker Compose for NetBox, WAR/Docker for GitBucket) but - does not pin an exact version/channel/tag -- consult each project's current - official install docs at execution time and record whatever is actually - installed in the changelog entry for this runbook's real run. - ---- - -## Provisioning-path decision (applies to all three service VMs) - -Per the structural blocker above, EVERY VM-creation step in this runbook (5, -6, 8) has two paths. Do not silently default to one -- read both, pick -deliberately, and log which path was used in the changelog entry. - -- **Option A (preferred, long-term, BLOCKED pending design):** create the VM - via `opentofu/modules/cloudinit-vm`, now that (a) Open question #1 above is - RESOLVED (attaches to `module.office1_network.network_name`, i.e. - `"office1-local"`) but (b) real `user_data`/`meta_data`/`network_config` - content still needs to be designed for that specific VM (what packages, - what static address, what hostname). This is the D-103-correct path (`OpenTofu - creates... the Office1 service VMs`) and is NOT available tonight -- doing - it right requires a design pass this session did not do and this runbook - will not invent. -- **Option B (interim, manual):** create the VM directly against the vcloud - host's libvirt (e.g. `virt-install` with a real official cloud image ISO/ - qcow2, or an interactive install), attach it to whatever network Open - question #1 resolves to, then run the install steps below directly over SSH - once the VM has a reachable address. This gets Office1 stood up now, at the - cost of NOT being reproducible/versioned in OpenTofu -- log it explicitly as - a known debt (a DOCFIX candidate: "Office1 service VM `` was - hand-created via Option B on ``; migrate to `modules/cloudinit-vm` - once its cloud-init content is designed"). + NetBox-authoritative... NOT hardcoded in this decision"). This runbook stands + NetBox up and can run its import MECHANISM (Step 10), but the underlying + `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not exist yet -- + see Step 10's own gate. +4. **GitBucket's VR1 repo path/name is not decided.** + The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a + DIFFERENT, pre-existing + `git.baldurkeep.com` instance from the v1 rehearsal -- this stage stands up a + SEPARATE, Office1-local GitBucket -- what it mirrors, and under what repo + path, is not decided here. Flag as an open item before Step 11's mutation. +5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the front + door is "Office1 only" -- the actual tailnet ACL policy, tags, and which + services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the service + machines?) are exposed through it versus reachable only from Office1's own + local network, is real policy design left to Step 12 and the operator's own + tailnet account -- not invented here. NOTE the placement change under D-114: + Tailscale is now a MAAS-COMPOSED SERVICE MACHINE (a VM in `voffice1`'s LXD, + exactly like VR0's own `tailscale` machine), NOT a package installed on the + headend host. A pre-existing subnet route already reaches the jumphost; that + is a different node and does not satisfy this gate. +6. **Exact MAAS / NetBox / GitBucket / LXD version pins are not decided** (beyond + D-114's hard LXD 5.21-LTS-track constraint, which IS decided). Each install + step below names the well-known official install MECHANISM but does not pin an + exact version/channel/tag -- consult each project's current official install + docs at execution time and record whatever is ACTUALLY installed in the + changelog entry for this runbook's real run, and in + `runbooks/appendix-B-asbuilt-version-lock.md`. +7. **`voffice1`'s stable address is not yet fixed.** It takes a DHCP lease from + the edge's Kea (pool .100-.199). A Kea RESERVATION (added over the OPNsense + REST API, `scripts/opnsense-api.sh`) is what makes it stable -- Step 4. Until + that reservation exists, do NOT hardcode `voffice1`'s address anywhere, + including in MAAS's own `--maas-url`. --- ## Sequence ``` -1. Confirm Stage 1 exit gate (read-only, above) -2. Confirm OpenTofu still reaches vcloud libvirt from Office1 (read-only) -3. Read the provisioning-path decision + open questions (above; no cmds) -4. MAAS region controller -- create VM (A or B) + install [MUTATION, gated] -4b. Office1 OPNsense edge -- wire module (gap #16 CLOSED; - instantiation still BLOCKED on gap #17 + real specs) [repo change, gated] -5. NetBox -- create VM (A or B) + install (official Docker) [MUTATION, gated] -6. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; - literals pending -- PARTIAL) [MUTATION, gated] -7. GitBucket -- create VM (A or B) + install (WAR or Docker) [MUTATION, gated] -8. Tailscale -- install + scope to Office1 only [MUTATION, gated] -9. Post-standup verify against the Stage 2 gate (read-only) - -> EXIT GATE (PARTIAL, see GATE section) -> Stage 3 +1. Confirm Stage 1 exit gate + Office1 edge live (read-only, above) +2. Confirm OpenTofu still reaches vcloud libvirt (read-only) +2b. Office1 OPNsense edge -- BUILT AND LIVE (historical + + DANGER banner; do NOT re-run against the live edge) (read-only) +3. voffice1 -- the site containment VM (tofu apply) [MUTATION, gated] + -- APPLIED 2026-07-13; domain running (measured) +4. voffice1 first contact -- lease, Kea reservation, SSH [MUTATION, gated] +5. MAAS region controller ON voffice1 [MUTATION, gated] PENDING VERIF. +6. LXD ON voffice1, pinned to the 5.21 LTS track + [MUTATION, gated] PENDING VERIF. +7. Register that LXD BACK INTO MAAS as an LXD KVM host + [MUTATION, gated] PENDING VERIF. +8. Compose the non-stack service machines into it (MAAS) + -- THE L3 NESTING PROOF; gates DC1 [MUTATION, gated] PENDING VERIF. +9. NetBox -- install on its composed machine [MUTATION, gated] +10. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; + literals pending -- PARTIAL) [MUTATION, gated] +11. GitBucket -- install on its composed machine [MUTATION, gated] +12. Tailscale -- install on its composed machine, scoped [MUTATION, gated] +13. Post-standup verify against the Stage 2 gate (read-only) + -> EXIT GATE -> Stage 3 (DC1), which D-114 GATES behind this ``` --- @@ -219,141 +228,49 @@ --- -## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt from Office1 (READ-ONLY) +## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt (READ-ONLY) -Stage 1 already proved OpenTofu -> vcloud-host libvirt reachability (its Step -1 CHECK + Step 10 apply). This step is a re-verification, not a re-derivation --- if this session is running some time after Stage 1, confirm nothing -regressed rather than assuming it still holds. +Stage 1 proved OpenTofu -> vcloud-host libvirt reachability and has since +APPLIED for real (the state file holds the DC1 planes, the pools, the three mesh +links, `office1-network`, the OPNsense edge, and `voffice1`). This step is a +re-verification, not a re-derivation. -**CHECK -- from the Office1 operator VM/session** +**CHECK -- from the Office1 operator session** ```bash -cd opentofu -tofu version -virsh -c "$(grep -A1 'variable "libvirt_uri"' -m1 /dev/null 2>/dev/null; true)" list --all 2>/dev/null || true -``` -More directly -- use the exact `libvirt_uri` value recorded in Stage 1's -`opentofu/dc-dc-phase0.auto.tfvars`: -```bash +tofu -chdir="$REPO/opentofu" version virsh -c "" list --all ``` -Expect: the domain list (the DC1 plane/pool objects have no domains yet, so -this may be empty, but the CONNECTION must succeed with no error). If it -fails, STOP -- this is the same blocking condition Stage 1's Step 1 described; -do not proceed into VM creation without confirmed libvirt reach. +(`$REPO` is set once per session -- see `runbooks/README.md` Conventions.) +Expect: the connection succeeds, and the domain list shows `office1-opnsense` +and `voffice1`. If the connection fails, STOP -- do not proceed into any +mutation without confirmed libvirt reach. -**CHECK -- MAAS provider not yet needed but flagged (Stage 1's Known Gap)** +**CHECK -- the tree still validates** ```bash -grep -n 'provider "maas"' opentofu/main.tf +bash scripts/opentofu-validate.sh ``` -Confirm whether Stage 1 worked around the unconditional `maas` provider block -with placeholder values or whether a DOCFIX split it out -- read whatever -Stage 1's changelog entry recorded, and use the same resolved answer here -rather than re-deciding it. +Expect: root + every module PASS (10/10 as of DOCFIX-194 -- the gate now +validates every module STANDALONE, because root-only validation silently +skipped the two modules root does not call). --- -## Step 3 -- Read the provisioning-path decision and open questions +## Step 2b -- Office1's OPNsense edge: BUILT AND LIVE (READ-ONLY / historical) -No commands. Confirm with the operator (or record your own reasoned choice, -logged) which path -- A or B -- is being used for each of the three VMs -below, and how Open question #1 (Office1-local network) is being resolved for -this run. Do not proceed to Step 4 silently. +The Office1 edge is no longer work this runbook does -- it is a satisfied +PREREQUISITE. It was built 2026-07-12/13 and is routing, NAT-ing, providing +egress, and serving Kea DHCP on `office1-local`. `voffice1` depends on all four +of those. This section is retained as the historical record of how the image is +prepped, and for its safety banner, which is still live. ---- - -## Step 4 -- MAAS region controller: create VM + install [MUTATION: gated] - -**CHECK -- confirm no pre-existing MAAS-region VM under this name** -```bash -virsh -c "" list --all | grep -i maas-region -``` -Expect: no match (fresh standup) or a clear, understood match (re-run/repair -case) -- do not proceed if an unexplained existing domain shares the name. - -**MUTATION -- create the VM (Option A or B per Step 3)** - -Option A: not available tonight (blocked, see above) -- instantiate -`module "office1_maas_region"` using `modules/cloudinit-vm` once its -`user_data`/`meta_data`/`network_config` are designed, then `tofu apply`. - -Option B (interim manual path): -```bash -# Illustrative shape only -- fill every with a value you -# measured/assigned this session; none of these are literals to copy verbatim. -virt-install \ - --connect "" \ - --name maas-region-office1 \ - --memory \ - --vcpus \ - --disk pool=office1,size= \ - --network network=office1-local \ - --os-variant \ - --cdrom \ - --graphics none --console pty,target_type=serial -``` -Boot it, complete the OS install per that OS's own docs, and record the -resulting static/DHCP address this session assigns it -- MEASURE/ASSIGN, never -guessed. This is the exact "known debt" Option B commits to logging (see -Provisioning-path decision above). - -**MUTATION -- install MAAS region controller (official snap-based method)** - -Once the VM is up and reachable over SSH: -```bash -ssh 'snap info maas' -``` -Consult MAAS's current official install docs for the exact channel to pin -(this repo does not hardcode a MAAS snap channel anywhere else to copy from -- -pick and record the current LTS/stable channel per those docs, not a guessed -one) and run the equivalent of: -```bash -sudo snap install maas --channel= -``` -Then initialize it as a region controller per MAAS's own current `maas init` -documentation for that installed version -- the exact flags (`--maas-url`, -database backend selection, admin creation) vary by MAAS release; consult the -official docs rather than reusing a flag set from memory or from a different -MAAS version's docs. - -**CHECK -- verify the region controller is reachable** -```bash -curl -sI "http://:5240/MAAS/" | head -1 -``` -Expect an HTTP response, not a connection failure. Naming per D-106: this -service's FQDN, once Designate/DNS exists for Office1, would follow the -region-level subdomain scheme D-106 names as an example -(`maas.office1.vr1...`) -- exact scheme is confirmed here as part of this -stage, not assumed; record whatever real hostname convention you actually use. - -**GATE (Stage 2, first Gate bullet):** MAAS region reachable. - ---- - -## Step 4b -- Office1's own OPNsense edge: wire the module [repo change, gated] (NEW -- gap #16 CLOSED) - -Per Open question #2 above (RESOLVED): Office1 gets its own -`modules/opnsense-edge` call, and this runbook (Stage 2) owns creating it -- -not Stage 3, whose runbook only ever covered DC1/DC2. This step mirrors -`runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 4/5 exactly (same module, -same prep scripts, same variable names), just for the `office1` site instead -of `dc1`/`dc2`. - -**CHECK -- confirm no pre-existing Office1 OPNsense VM under this name** -```bash -virsh -c "" list --all | grep -i office1-opnsense -``` -Expect no match (fresh standup) or a clear, understood match (re-run/repair -case). - -> ## !!! DANGER -- DO NOT RUN THIS STEP AGAINST THE LIVE OFFICE1 EDGE (2026-07-13) !!! +> ## !!! DANGER -- DO NOT RUN A CONFIG PUSH AGAINST THE LIVE OFFICE1 EDGE (2026-07-13) !!! > > **The Office1 edge is BUILT, ROUTING, and SERVING DHCP.** Its DHCP config is now > **API-MANAGED** (D-113(a2), proven live 2026-07-13). Rendering a full `config.xml` and > pushing it to `10.10.0.1` **WOULD CLOBBER THAT LIVE STATE** -- a full-config push replaces > `/conf/config.xml` wholesale and drops ~667 migration-populated elements. > -> **Two things below are now WRONG and must not be followed as written:** +> **Two things are now permanently WRONG and must not be attempted:** > > 1. **The config ISO cannot work AT ALL.** Per **D-112**, the OPNsense Configuration Importer > can NEVER fire on a pre-installed nano image -- `opnsense-importer -b` probes for a @@ -366,166 +283,424 @@ > DOCFIX-192 (dead console) and DOCFIX-193 (no DHCP). > > **What is actually true today:** Office1 was brought up by the D-112(c) console bootstrap, and -> is now managed over SSH + the REST API. This step is retained ONLY as the historical record of -> how the image is prepped. -> -> **If you are re-standing-up Office1 from scratch:** the template has NOT yet been reduced to a -> minimal bootstrap (sshd + root key + console) -- that is the open D-113(a2) work. Until it is, -> STOP and re-derive this step; do not render a full config and push it. +> is now managed over SSH + the REST API. -**MUTATION -- prepare Office1's OPNsense base image (host-local files, gated)**: +**Image prep (host-local files only -- the one part still runnable, and only for +a from-scratch re-standup):** ```bash bash scripts/opnsense-prep-image.sh # see the script's own header for its exact args/output path ``` -The `config.xml` render + config-ISO sub-steps that used to live here are RETIRED -- see the -DANGER banner above. They are preserved verbatim in git history (and in -`docs/changelog-20260712-office1-opnsense-edge-build.md`) rather than kept here as runnable -instructions, because an instruction that clobbers a live router is worse than no instruction. - -**Repo change -- wire `module "office1_opnsense"` in `opentofu/main.tf`** - -A commented-out skeleton already exists in root `main.tf` (this delivery, -2026-07-09) with `lan_network_name` already resolved -(`module.office1_network.network_name`). Uncommenting it for a real run -additionally requires: -- `memory_mib` / `vcpu` / `disk_size_bytes` -- real values, OPNsense's own - sizing guidance, not invented (same "no invented specs" rule as every - other VM module in this repo). -- `base_volume_path` / `config_iso_path` -- the real output paths from the - two commands above. -- `wan_network_name` -- **BLOCKED on tooling gap register item #17** (no - dedicated per-site ISP-uplink/WAN network exists anywhere in this repo - yet, for ANY site -- confirmed against Stage 3's own `dc1_opnsense` - template, which carries the identical open placeholder). Do NOT wire this - to any `mesh-link` network: D-100's own sub-item ruling confirms those - three legs carry management traffic only, not a simulated ISP uplink. - **STOP here if gap #17 is still open** -- this step cannot be fully - completed, and that is the honest state to record, not a placeholder to - paper over. - -**GATE:** `opentofu/main.tf` diff (once gap #17 closes and every value above -is real) shows exactly one new `module "office1_opnsense"` block, every -argument a real value, nothing else changed -- same GATE shape as Stage 3's -Step 5. - -**GATE (Stage 2 Gate bullet, NEW):** Office1's own OPNsense edge module -wired with a real LAN side (achievable this stage, gap #12 resolved it) -- -full instantiation (and therefore the running edge VM itself) remains -BLOCKED pending gap #17's resolution, same as DC1/DC2's edges. Do not mark -this bullet fully green until gap #17 closes and `tofu apply` has actually -run for `module.office1_opnsense`. +**GATE (Stage 2 Gate bullet):** Office1 OPNsense edge up -- **MET 2026-07-12/13** +(routing, NAT, egress, serial console, SSH-key managed, Kea DHCP). Gap #17 is +CLOSED for Office1 via the dedicated `office1-wan` ISP-uplink network. DC1/DC2 +still have no uplink network -- that is Stage 3's problem, not this one's. --- -## Step 5 -- NetBox: create VM + install (official Docker method) [MUTATION: gated] +## Step 3 -- voffice1: the site containment VM [MUTATION: gated] -**CHECK -- confirm no pre-existing NetBox VM under this name** +> **STATUS: APPLIED 2026-07-13.** `module.voffice1` is in +> `opentofu/terraform.tfstate` and the domain is RUNNING (measured this session: +> `virsh list --all` -> `voffice1 running`). The apply was performed by the main +> session, not by this runbook's authoring pass. This step is retained as the +> procedure -- for a re-standup, and so the gate it satisfies is auditable. +> **Re-running `tofu apply` when `voffice1` is already up is NOT a no-op to be +> taken casually:** an apply that reports "updated in-place" can still BOUNCE a +> domain (measured 2026-07-13 -- an apply restarted the live edge). Read the plan +> before approving it. + +This is the D-114 site containment VM. It IS the Office1 facility: MAAS-region +runs on it, LXD runs on it, and the non-stack service machines are composed into +that LXD by MAAS. It sits on `office1-local` ONLY -- it reaches the internet +THROUGH the OPNsense edge, like a real server behind a real site router. + +**CHECK -- confirm the current state before any apply** ```bash -virsh -c "" list --all | grep -i netbox +virsh -c "" list --all | grep -i voffice1 +tofu -chdir="$REPO/opentofu" plan +``` +Expect either no `voffice1` domain (fresh standup -- the apply will create it) +or the domain present AND a plan showing no changes to `module.voffice1`. An +unexplained diff against a running containment VM is a STOP. + +**MUTATION -- create voffice1 (OpenTofu; Option A, and now the ONLY path)** +```bash +tofu -chdir="$REPO/opentofu" apply +``` +Approve only after reading the plan. The module is already fully specified in +`opentofu/main.tf` -- do not re-derive its inputs here. Its load-bearing +properties, for the record: + +- `modules/cloudinit-vm` off `modules/base-image` (the official Ubuntu 24.04 + noble cloud image -- 24.04 is what VR0 deploys for its own `lxd`/`juju`/ + `tailscale` machines, so the image choice carries no new delta). +- `network_names = [office1-local]` ONLY. No mesh leg, no direct WAN. +- `network_config` takes **DHCP** from the edge's Kea, matching the NIC by GLOB + (`en*`) rather than a guessed kernel name -- the NIC's name is not knowable + before first boot, and naming it would be an inferred value. +- **`expose_nested_virt = true`** -- LOAD-BEARING (see "The D-114 model" above). + Without the host CPU feature passed through, no LXD VM can ever boot inside + `voffice1` and D-114's model fails at its first step. +- Cloud-init is DELIBERATELY MINIMAL (identity + access + guest agent). MAAS and + LXD are installed as separate GATED steps below so they are observable and + individually approved, not buried in a first-boot script that either silently + works or silently does not. + +**SECURE THE STATE FILE** after any apply (standing discipline from DOCFIX-175): +confirm `opentofu/**/*.tfstate*` is gitignored, permissions are tight, and an +out-of-band backup exists. The state stores `sensitive = true` values (e.g. +`var.maas_api_key`) in PLAINTEXT. + +**GATE (Stage 2, first Gate bullet):** `voffice1` exists and is `running`. +**MET 2026-07-13 (measured).** + +--- + +## Step 4 -- voffice1 first contact: lease, Kea reservation, SSH [MUTATION: gated] + +`office1-local` is an isolated libvirt network with NO libvirt-managed DHCP -- +`virsh net-dhcp-leases office1-local` is EMPTY BY DESIGN and is NOT evidence that +`voffice1` failed to get an address. Kea, on the OPNsense edge, is the +authoritative DHCP server for that L2. Look for the lease THERE. + +Per D-114, `voffice1` booting on `office1-local` takes the **first real DHCP +lease** the Kea path has ever served -- a path proven so far only at the daemon +level, never end to end. Treat a successful lease as a real result worth +recording, and a failed one as a real finding (not a nuisance to work around by +hand-assigning a static address). + +**CHECK -- read the lease from the edge (read-only)** +```bash +bash scripts/opnsense-api.sh +``` +Expect: one lease, in the .100-.199 pool, whose MAC matches `voffice1`'s +interface (`virsh -c "" domiflist voffice1`). MEASURE both -- do +not assume the lease belongs to `voffice1` because it is the only one. + +**MUTATION -- pin the address with a Kea RESERVATION (gated)** + +Add a Kea host reservation binding `voffice1`'s measured MAC to a stable address, +over the OPNsense REST API (`scripts/opnsense-api.sh`). Do NOT hand-edit +`/conf/config.xml` -- see Step 2b's DANGER banner. Do NOT hardcode an address +anywhere (MAAS `--maas-url` included) until this reservation exists and has been +re-measured after a `voffice1` reboot. + +**CHECK -- SSH reach** +```bash +ssh -i jessea123@ 'hostnamectl; systemd-detect-virt; cloud-init status --long' +``` +Expect: hostname `voffice1`, `kvm`, and cloud-init `status: done`. A cloud-init +that is still `running` or `error` is a STOP -- read its logs before layering +MAAS on top of a half-configured machine. + +**CHECK -- nested virt is actually exposed (the L3 precondition)** +```bash +ssh 'grep -c -w -E "svm|vmx" /proc/cpuinfo; ls -l /dev/kvm' +``` +Expect: a non-zero count and `/dev/kvm` present. **If this fails, STOP** -- +`expose_nested_virt` did not take effect, and every LXD VM in Step 8 will fail +to boot. That is D-114's own early-abandon signal; do not paper over it. + +**GATE (Stage 2 Gate bullet, NEW):** `voffice1` holds a Kea RESERVATION, is +reachable over SSH, and exposes nested KVM. + +--- + +## Step 5 -- MAAS region controller ON voffice1 [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The exact snap channel and the exact `maas init` +> flag set (`--maas-url`, database backend selection, admin creation) VARY BY +> MAAS RELEASE. They are NOT reproduced from memory here. A separate +> verification pass against MAAS's current official install docs owns filling +> these in. Do not substitute this repo's `scripts/phase-00-maas-standup.sh` +> -- that script drives an EXISTING MAAS, it does not install one. + +Under D-114 the MAAS region controller runs INSIDE `voffice1`, not on a sibling +VM of its own. It is the site's provisioning authority: it will own the LXD KVM +host (Step 7) and every service machine composed into it (Step 8). + +**CHECK -- nothing pre-existing** +```bash +ssh 'snap list maas 2>&1 || echo "not installed"' ``` -**MUTATION -- create the VM (Option A or B per Step 3)**, same shape and same -caveats as Step 4's MAAS-region VM creation (Option A blocked pending -`cloudinit-vm` content design for THIS specific VM; Option B manual, logged as -debt). +**MUTATION -- install MAAS (snap, the official method)** +```bash +ssh 'sudo snap install maas --channel=' +``` + +**MUTATION -- initialize it as a region controller** + +Per MAAS's own current `maas init` documentation for the installed version. +`` + +Use `voffice1`'s RESERVED address (Step 4) for the MAAS URL -- never the raw +lease, and never a guessed address. + +**CHECK -- the region controller answers** +```bash +curl -sI "http://:5240/MAAS/" | head -1 +``` +Expect an HTTP response, not a connection failure. Naming per D-106: this +service's FQDN, once DNS exists for Office1, follows the region-level subdomain +scheme D-106 names as an example (`maas.office1.vr1...`) -- record whatever real +hostname convention you actually use; do not assume it. + +**MUTATION -- create the MAAS API key for later steps** + +Steps 7 and 8 (and `opentofu/modules/maas-vm-host`, if it is ever wired for +Office1) need a real MAAS API key. Generate it per MAAS's own docs and handle it +per this repo's secrets discipline: never printed, never committed, passed as an +environment variable or read from a jumphost-local file only. Note DOCFIX-175: +`var.maas_api_key` lands in `terraform.tfstate` in PLAINTEXT if OpenTofu is given +it -- factor that into where the key is allowed to go. + +**GATE (Stage 2 Gate bullet):** MAAS region reachable, running ON `voffice1`. + +--- + +## Step 6 -- LXD ON voffice1, pinned to the 5.21 LTS track [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The `lxd init` answers (storage backend and size, +> bridge configuration, whether the LXD HTTPS API is exposed and on what +> address/port, and how MAAS is expected to authenticate to it -- trust +> password vs. certificate) are NOT reproduced from memory. They are precisely +> what the MAAS-LXD seam depends on. A separate verification pass against +> Canonical's current MAAS + LXD docs owns filling them in. + +**MUTATION -- install LXD, PINNED (D-114)** +```bash +ssh 'sudo snap install lxd --channel=5.21/stable' +``` +The `5.21` track is a HARD D-114 constraint, not a preference: MAAS 3.6/3.7 +cannot talk to LXD >= 6.7 (pylxd 2.3.5 vs LXD's consolidated API endpoints). +`latest/stable` WILL eventually hand you a broken seam. If LXD is already +present from the image at a different track, refresh it onto 5.21 rather than +proceeding -- and record the measured version in appendix-B. + +**CHECK -- confirm the pin actually took** +```bash +ssh 'snap list lxd; lxd --version' +``` +Expect a 5.21.x version. **A 6.x version here is a STOP** -- it is the exact +incompatibility D-114 pins against. + +**MUTATION -- initialize LXD** +```bash +ssh 'sudo lxd init ' +``` + +!!! **`lxdbr0` MUST STAY on LXD's internal NAT bridge.** Do NOT bridge it onto + `office1-local`. Kea (on the OPNsense edge) is the authoritative DHCP server + for that L2; `lxdbr0`'s own dnsmasq on the same L2 would be a second one. + See the DHCP HAZARD banner at the top of this runbook. + +**CHECK -- LXD is up and its bridge is where it belongs** +```bash +ssh 'lxc network list; ip -br addr show lxdbr0' +``` +Expect `lxdbr0` on its own private subnet, NOT on 10.10.0.0/24. + +**GATE (Stage 2 Gate bullet, NEW):** LXD installed on `voffice1`, version on the +5.21 LTS track (MEASURED), `lxdbr0` isolated from `office1-local`. + +--- + +## Step 7 -- Register that LXD BACK INTO MAAS as an LXD KVM host [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The exact MAAS CLI/API call that adds an LXD KVM +> host (its subcommand, its power/auth parameters, the project name, and how +> the LXD certificate trust is established) is NOT reproduced from memory. A +> separate verification pass owns it. VR0's as-built shows the TARGET STATE: +> an LXD KVM host in project `default`, LXD 5.21.4. + +This is the seam D-114 turns on. `voffice1` is a MAAS-managed machine that runs +LXD; that LXD is then registered as a VM HOST inside the same MAAS. It is what +makes the service machines MAAS-visible instead of hand-run containers. + +**Note the D-103/D-114 boundary, precisely:** the composition right granted here +is scoped to THIS LXD VM host and the NON-STACK machines. MAAS still does NOT +compose the OpenStack NODE VMs -- `modules/node-vm` pre-creates those and +`modules/maas-vm-host` registers the VIRSH host so MAAS merely DISCOVERS them. +`maas_vm_host_machine` remains ruled out for nodes. + +**CHECK -- read the current VM-host inventory (read-only)** +```bash +ssh 'maas vm-hosts read' +``` + +**MUTATION -- add the LXD KVM host** + +PENDING VERIFICATION: the exact MAAS "add a VM host of type LXD" invocation -- +its subcommand, power address, project name, and credentials/certificate trust. +Consult MAAS's own current LXD-VM-host documentation. Do NOT adapt this from +`scripts/phase-00-maas-standup.sh` or `scripts/reenroll-hosts.sh`: those are the +PXE-enlisted metal-node path, a different mechanism. + +**CHECK -- MAAS sees the LXD host, with real capacity** +```bash +ssh 'maas vm-hosts read' +``` +Expect the LXD host present, with its cores/memory/storage reported -- MAAS +reading real numbers back out of LXD is the evidence the seam works, not merely +that the record was created. + +**GATE (Stage 2 Gate bullet, NEW):** `voffice1`'s LXD is registered in MAAS as an +LXD KVM host and MAAS reads its capacity. + +--- + +## Step 8 -- Compose the non-stack service machines (NetBox, GitBucket, Tailscale) [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The exact `maas ... vm-host compose` parameters +> (cores, memory, storage, and how the composed machine is attached to the right +> network) are NOT reproduced from memory. A separate verification pass owns +> them. VR0's as-built gives the SIZING PRECEDENT, measured, not invented: +> its `tailscale` machine is **2 cores / 2 GiB / 25 GB, Ubuntu 24.04** -- +> D-114's own figure for what a composed service machine costs. + +**This step is the L3 NESTING PROOF, and it GATES DC1.** The first LXD *virtual +machine* that boots inside `voffice1` proves the whole D-114 pod model to L3. +D-114's DC1 entry gate depends on it. If no LXD VM will boot here, STOP and +report -- do not "work around" it by falling back to LXD containers (MAAS does +not enlist containers) or by hand-running the services on `voffice1` itself +(that reconstructs the flat model D-114 replaced, without saying so). + +**MUTATION -- compose the machines, ONE AT A TIME, individually gated** + +Compose in this order, gating each: **Tailscale first** (it is the smallest, it +is VR0's proven exemplar, and if it boots the model is proven at minimum cost), +then NetBox, then GitBucket. + +```bash +ssh 'maas vm-host compose ' +``` + +Each composed machine then goes through MAAS's OWN provisioning lifecycle -- +this is the entire point of D-114's ruling (they are MAAS-VISIBLE machines, not +hand-run VMs): + +1. **Enlist** -- MAAS registers the composed VM. +2. **Commission** -- MAAS boots it and inventories it. +3. **Deploy** -- MAAS installs the OS (Ubuntu 24.04, matching VR0's own service + machines). +4. **Power** -- MAAS controls it via the LXD VM host. + +`` + +**CHECK -- the machines are real, MAAS-visible, and DEPLOYED** +```bash +ssh 'maas machines read | ' +``` +Expect each service machine `Deployed`, powered via the LXD VM host. This is +VR0's `lxd` + `tailscale` shape, reproduced per site. + +**GATE (Stage 2 Gate bullet, NEW -- and D-114's DC1 entry gate, half (b)):** at +least one LXD **virtual machine** BOOTS inside `voffice1`, and the non-stack +service machines are MAAS-composed, deployed, and powered. **L3 nesting PROVEN.** +Record this explicitly -- DC1 is gated on it. + +--- + +## Step 9 -- NetBox: install on its composed machine [MUTATION: gated] + +The VM is now a MAAS-DEPLOYED machine (Step 8), not a hand-built one. This step +only installs the application onto it. **MUTATION -- install NetBox (official netbox-docker Compose method)** -Once the VM is up and reachable over SSH, follow NetBox's own official +Over SSH to the composed NetBox machine, follow NetBox's own official Docker-based install (the `netbox-community/netbox-docker` repo's documented Compose workflow: clone it, configure its `.env` / `docker-compose.override.yml` per its own current docs, `docker compose up -d`). Do not hand-roll a from-source install -- Docker Compose is NetBox's own recommended path and the -lowest-delta one to keep reproducible. Consult NetBox's current official docs -for the exact compose file / image tag to pin; do not fabricate a version tag. +lowest-delta one to keep reproducible. Consult NetBox's current official docs for +the exact compose file / image tag to pin; do not fabricate a version tag. **CHECK -- verify NetBox is reachable** ```bash -curl -sI "http://:8080/" | head -1 +curl -sI "http://:8080/" | head -1 ``` -Expect an HTTP response. Create/record a NetBox API token per NetBox's own -docs (needed by Step 6) -- handle it per this repo's secrets discipline: never +Expect an HTTP response. Create/record a NetBox API token per NetBox's own docs +(needed by Step 10) -- handle it per this repo's secrets discipline: never printed, never committed, passed as an environment variable only (`NETBOX_TOKEN`, matching `netbox/dc-dc-prefixes-import.py`'s own documented usage). **GATE (Stage 2 Gate, partial):** NetBox reachable and running. NOT yet -"authoritative and populated" -- see Step 6. +"authoritative and populated" -- see Step 10. --- -## Step 6 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] +## Step 10 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] -`netbox/dc-dc-prefixes-import.py` (built 2026-07-09, closes tooling gap #3's -MECHANISM) extends the v1 single-site import to VR1's per-DC, dual-stack -model per D-101. It is idempotent and `--verify-only`-capable. **It does NOT, -and cannot, close the DATA half of gap #3**: the real org ULA /48, the per-DC -GUA carve, and DC2's v4 supernet do not exist yet as assigned values anywhere -in this repo or session. Running it for DC1 only requires the two v6 literals -(`ORG_ULA_48`, `DC_GUA_PREFIX`); running it for DC2 additionally requires -`DC2_V4_SUPERNET` -- none of the three has a default, by the script's own -design, and none is invented here. +`netbox/dc-dc-prefixes-import.py` (DOCFIX-152) extends the v1 single-site import +to VR1's per-DC, dual-stack model per D-101. It is idempotent and +`--verify-only`-capable. **It does NOT, and cannot, close the DATA half of +tooling gap #3**: the real org ULA /48, the per-DC GUA carve, and DC2's v4 +supernet do not exist yet as assigned values anywhere in this repo or session. +Running it for DC1 requires the two v6 literals (`ORG_ULA_48`, `DC_GUA_PREFIX`); +running it for DC2 additionally requires `DC2_V4_SUPERNET` -- none of the three +has a default, by the script's own design, and none is invented here. **CHECK -- dry run, no writes** ```bash -NETBOX_URL="http://:8080" \ -NETBOX_TOKEN="" \ +NETBOX_URL="http://:8080" \ +NETBOX_TOKEN="" \ ORG_ULA_48="" \ -DC_GUA_PREFIX="" \ +DC_GUA_PREFIX="" \ python3 netbox/dc-dc-prefixes-import.py --dc dc1 --verify-only ``` If `ORG_ULA_48` / `DC_GUA_PREFIX` are not yet assigned, this command CANNOT be -run for real yet -- that is the honest state of this stage tonight, not a -missing step in this runbook. Record that as the blocking condition rather -than inventing placeholder-looking values to "get past" it (this script -itself is designed to fail loud on exactly that pattern, per its own header). +run for real yet -- that is the honest state of this stage, not a missing step in +this runbook. Record that as the blocking condition rather than inventing +placeholder-looking values to "get past" it (this script is designed to fail loud +on exactly that pattern, per its own header). See +`docs/dc-dc-netem-and-ula-gua-proposal.md` for the drafted ULA-generation +guidance -- a PROPOSAL, not a ruling. -**MUTATION -- once the literals above are real (a later session/step, not -necessarily tonight)** +**MUTATION -- once the literals above are real (a later session/step)** ```bash NETBOX_URL="..." NETBOX_TOKEN="..." ORG_ULA_48="..." DC_GUA_PREFIX="..." \ python3 netbox/dc-dc-prefixes-import.py --dc dc1 ``` DC1's v4 is hardcoded in the script itself (D-101: "DC1 equals the validated -template," not an inferred value -- it is the explicit text of an ADOPTED -decision). DC2 additionally needs `DC2_V4_SUPERNET`, not assigned yet either. +template," the explicit text of an ADOPTED decision, not an inferred value). DC2 +additionally needs `DC2_V4_SUPERNET`, not assigned yet either. -**GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 5). +**GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 9). "Authoritative and populated (planes, per-DC v4, ULA/GUA carve)" is only -PARTIALLY achievable right now: the IMPORT MECHANISM exists and is runnable -the moment the literals are assigned, but the literals themselves are not -assigned this session, so the DATA half of this gate bullet is NOT met yet. -Do not mark this gate green in the tracker until a real `ORG_ULA_48` / -`DC_GUA_PREFIX` (and, for DC2, `DC2_V4_SUPERNET`) exist and this script has -actually been run against this real NetBox instance. +PARTIALLY achievable: the IMPORT MECHANISM exists and is runnable the moment the +literals are assigned, but the literals themselves are not assigned, so the DATA +half of this gate bullet is NOT met. Do not mark this bullet green in the tracker +until a real `ORG_ULA_48` / `DC_GUA_PREFIX` (and, for DC2, `DC2_V4_SUPERNET`) +exist and this script has actually run against this real NetBox instance. --- -## Step 7 -- GitBucket: create VM + install (official WAR or Docker method) [MUTATION: gated] - -**CHECK -- confirm no pre-existing GitBucket VM under this name** -```bash -virsh -c "" list --all | grep -i gitbucket -``` - -**MUTATION -- create the VM (Option A or B per Step 3)**, same shape/caveats -as Steps 4 and 5. +## Step 11 -- GitBucket: install on its composed machine [MUTATION: gated] **MUTATION -- install GitBucket (official method)** GitBucket ships as a single runnable WAR (`java -jar gitbucket.war`, its own documented standalone-Jetty mode) or as an official Docker image -(`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one -and record which. Consult GitBucket's current official docs/release page for -the exact WAR filename/version or image tag to pin -- do not fabricate one. -Open question #3 above (this instance's purpose/repo-path scheme, distinct -from the existing `git.baldurkeep.com` v1 instance per D-014) must be resolved -before deciding what this instance actually mirrors. +(`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one and +record which. Consult GitBucket's current official docs/release page for the +exact WAR filename/version or image tag to pin -- do not fabricate one. Open +question #4 above (this instance's purpose/repo-path scheme, distinct from the +existing `git.baldurkeep.com` v1 instance per D-014) must be resolved before +deciding what this instance actually mirrors. **CHECK -- verify GitBucket is reachable** ```bash -curl -sI "http://:8080/" | head -1 +curl -sI "http://:/" | head -1 ``` Expect an HTTP response. @@ -533,139 +708,140 @@ --- -## Step 8 -- Tailscale: install + scope to Office1 only [MUTATION: gated] +## Step 12 -- Tailscale: install on its composed machine, scoped to Office1 [MUTATION: gated] -Per D-107, Office1 is explicitly OUT of the core-service path (no node -artifacts, no NTP served from Office1) -- Tailscale here is a narrowly-scoped -OPERATOR front door to Office1's own services, not a general egress/access -path for anything DC-side. Do not wire Tailscale into any DC-facing plane. +Per D-114, Tailscale is a MAAS-COMPOSED SERVICE MACHINE (VR0's own `tailscale` +machine, reproduced per site) -- NOT a package on the headend host, which is what +the superseded model did. Per D-107, Office1 is explicitly OUT of the +core-service path (no node artifacts, no NTP served from Office1) -- Tailscale is +a narrowly-scoped OPERATOR front door to Office1's own services, not a general +egress/access path for anything DC-side. Do not wire it into any DC-facing plane. -**CHECK -- confirm Tailscale is not already installed/conflicting** +A pre-existing subnet route already reaches the jumphost. That is a DIFFERENT +node and does NOT satisfy this gate -- do not mistake it for one. + +**CHECK -- confirm nothing conflicting on the composed machine** ```bash -ssh 'tailscale version 2>&1 || echo "not installed"' +ssh 'tailscale version 2>&1 || echo "not installed"' ``` **MUTATION -- install (official method)** ```bash -ssh 'curl -fsSL https://tailscale.com/install.sh | sh' +ssh 'curl -fsSL https://tailscale.com/install.sh | sh' ``` -Consult Tailscale's own current docs if this host's distro needs a different +Consult Tailscale's own current docs if this machine's distro needs a different install path (the install script itself detects and branches on distro). **MUTATION -- bring up, scoped** ```bash -ssh 'sudo tailscale up --advertise-tags=' +ssh 'sudo tailscale up --advertise-tags=' ``` -The actual ACL policy restricting reachability to Office1's services only -(and NOT to any DC-side network) is configured in the operator's own Tailscale -admin console/ACL file -- this is real policy design (Open question #4), not -invented here. Do not advertise routes into any DC plane from this node. +The actual ACL policy restricting reachability to Office1's services only (and +NOT to any DC-side network) is configured in the operator's own Tailscale admin +console/ACL file -- real policy design (Open question #5), not invented here. Do +not advertise routes into any DC plane from this node. **CHECK -- verify scope** ```bash -ssh 'tailscale status' +ssh 'tailscale status' ``` -Confirm only the expected Office1 headend node(s) appear, and that no route -advertisement reaches into DC1/DC2 plane CIDRs. +Confirm only the expected Office1 node(s) appear, and that no route advertisement +reaches into DC1/DC2 plane CIDRs. -**GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only. +**GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only, from +a MAAS-composed machine. --- -## Step 9 -- Post-standup verify against the Stage 2 gate (READ-ONLY) +## Step 13 -- Post-standup verify against the Stage 2 gate (READ-ONLY) ```bash -curl -sI "http://:5240/MAAS/" | head -1 -curl -sI "http://:8080/" | head -1 -curl -sI "http://:/" | head -1 -ssh 'tailscale status' virsh -c "" list --all +ssh 'snap list maas lxd; lxd --version' +ssh 'maas vm-hosts read' +ssh 'maas machines read' +curl -sI "http://:5240/MAAS/" | head -1 +curl -sI "http://:8080/" | head -1 +curl -sI "http://:/" | head -1 +ssh 'tailscale status' ``` -Cross-check each against the GATE section below before declaring this stage -done or partially done. +Cross-check each against the GATE section below before declaring this stage done +or partially done. --- -## GATE (Stage 2 exit condition -- honest partial state) +## GATE (Stage 2 exit condition) -Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row: "MAAS region -reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA -carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale -confirmed to Office1 only; Office1-local network created; Office1 OPNsense -edge module wired." +Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row, as reshaped by D-114: -- **MAAS region reachable:** achievable this stage (Step 4). -- **OpenTofu reaches vcloud host libvirt:** already proven in Stage 1; Step 2 - here only re-verifies it still holds. -- **Office1-local network created:** ACHIEVED by this delivery (gap #12 - CLOSED) -- `module "office1_network"` is instantiated for real in root - `main.tf`, no longer pending a per-run decision. -- **Office1 OPNsense edge module wired:** PARTIALLY achievable (Step 4b, gap - #16 CLOSED for ownership/LAN-side). Full instantiation and a running edge - VM remain BLOCKED on tooling gap register item #17 (no per-site - ISP-uplink/WAN network exists yet, for any of the three sites) plus real - `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` - values -- do not mark this bullet fully green until gap #17 closes and - `tofu apply` has actually run for `module.office1_opnsense`. -- **GitBucket serving:** achievable this stage (Step 7). -- **Tailscale confirmed to Office1 only:** achievable this stage (Step 8), - contingent on the operator's own ACL policy design (Open question #5). -- **NetBox authoritative and populated:** ONLY PARTIALLY achievable right now. - The instance can be stood up and reachable (Step 5), and the import - MECHANISM exists and is runnable (Step 6) -- but the real address literals - (org ULA /48, per-DC GUA carve, DC2 v4 supernet) are not assigned this - session, so NetBox cannot yet be truly "populated" per D-101. Do not mark - this bullet green in `docs/dc-dc-deployment-workflow.md`'s Stage 2 row until - those literals exist and `netbox/dc-dc-prefixes-import.py` has actually run - against this real instance. +- **Office1 OPNsense edge up (routing, NAT, egress, DHCP):** **MET 2026-07-12/13.** +- **Office1-local + office1-wan networks created:** **MET** (gaps #12 and #17-for-Office1 CLOSED). +- **OpenTofu reaches vcloud host libvirt:** **MET** (Stage 1; re-verified in Step 2). +- **`voffice1` site containment VM exists and runs:** **MET 2026-07-13 (measured).** +- **`voffice1` has a stable (reserved) address and SSH reach, with nested KVM + exposed:** Step 4 -- NOT yet confirmed. +- **MAAS region reachable, running ON `voffice1`:** Step 5 -- NOT yet done. +- **LXD on `voffice1`, on the 5.21 LTS track, `lxdbr0` off `office1-local`:** + Step 6 -- NOT yet done. +- **That LXD registered in MAAS as an LXD KVM host:** Step 7 -- NOT yet done. +- **Non-stack service machines COMPOSED BY MAAS into it, deployed and powered -- + i.e. an LXD VM BOOTS at L3:** Step 8 -- NOT yet done. **This is the D-114 DC1 + entry gate.** +- **GitBucket serving:** Step 11 -- NOT yet done. +- **Tailscale confirmed to Office1 only, from a composed machine:** Step 12 -- + NOT yet done, and contingent on the operator's own ACL policy (Open question #5). +- **NetBox authoritative and populated:** ONLY PARTIALLY achievable. The instance + can be stood up and reachable (Step 9), and the import MECHANISM exists and is + runnable (Step 10) -- but the real address literals (org ULA /48, per-DC GUA + carve, DC2 v4 supernet) are not assigned, so NetBox cannot yet be truly + "populated" per D-101. Do not mark this bullet green until those literals exist + and `netbox/dc-dc-prefixes-import.py` has actually run against the real instance. -This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** --- MAAS/GitBucket/Tailscale/OpenTofu-reach/Office1-network bullets closeable -in full tonight's sequence; the Office1-OPNsense-edge bullet closes its -ownership/LAN-side half only (gap #16), pending gap #17's cross-site -resolution; the NetBox bullet closes its mechanism half only, pending the -D-101 literal assignment that is explicitly out of scope for this stage to -invent. Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect -whichever of these is actually true after a real run, not to "DONE" -wholesale. +This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** -- +the substrate half (networks, edge, `voffice1`) is DONE and measured; the SITE +half (MAAS, LXD, the LXD KVM host, and every composed service machine) is the +remaining work, and its command lines are PENDING VERIFICATION. The NetBox DATA +bullet closes its mechanism half only, pending the D-101 literal assignment that +is explicitly out of scope for this stage to invent. Update +`docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever of these +is actually true after a real run, not to "DONE" wholesale. --> Proceed to Stage 3 (OpenTofu builds each DC substrate) only once the -achievable bullets above are actually green; the NetBox DATA gap does not -block Stage 3's OWN gate (Stage 3 depends on MAAS reachability and OpenTofu -reach, not on NetBox's DC2/GUA literals), but should be tracked as a standing -gap until closed. +-> **Stage 3 (DC1) is GATED behind this stage.** D-114's sequencing is an +operator ruling: Office1 FIRST, fully up on this model; DC1 is not started until +Office1 is complete; DC2 follows DC1. DC1's own entry gate additionally requires +proving nested KVM inside `vdc1` (which also proves L4 for tenant instances -- +the one depth VR0 has never exercised). **If L3 fails at Step 8, the +containment-VM model is abandoned early having cost one VM** -- that is the +designed cheap retreat, and it is the correct outcome to report, not a failure to +engineer around. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo - changes made while executing this runbook (e.g. any real `opentofu/` - instantiation once Option A becomes available; any real tfvars, minus - secrets). -- [ ] If any Option A `cloudinit-vm` instantiation was written this run, - `bash scripts/opentofu-validate.sh` green. -- [ ] Changelog entry for this runbook's first real execution (next DOCFIX - number via `bash scripts/ledger-scan.sh`), noting: which provisioning - path (A or B) was used for each of the three VMs, the real addresses/ - hostnames assigned, and whether NetBox's literal-assignment gap was - closed or remains open. + changes made while executing this runbook. +- [ ] If any `opentofu/` change was written this run, `bash + scripts/opentofu-validate.sh` green (root + EVERY module -- DOCFIX-194). +- [ ] Changelog entry for this runbook's real execution (next number via `bash + scripts/ledger-scan.sh`), noting: the MAAS/LXD versions ACTUALLY installed, + `voffice1`'s reserved address, whether the L3 nesting proof passed, which + service machines were composed, and whether NetBox's literal-assignment gap + was closed or remains open. +- [ ] `runbooks/appendix-B-asbuilt-version-lock.md` updated with the LXD 5.21-LTS + pin (D-114 requires it be recorded there) and the measured MAAS version. - [ ] `docs/session-ledger.md` updated with the outcome. -- [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table - updated to the ACTUAL state reached (likely "PARTIAL," not "DONE," per - the GATE section above) -- do not mark NetBox's gate bullet closed - unless the D-101 literals were genuinely assigned and imported this run. -- [ ] If Option B (manual VM creation) was used for any service VM, log the - resulting debt explicitly (a DOCFIX candidate: migrate that VM to - `opentofu/modules/cloudinit-vm` once its cloud-init content is - designed) rather than letting it go unrecorded. -- [x] Open question #1 (Office1-local network) RESOLVED 2026-07-09 -- - `opentofu/modules/office1-network` built and instantiated for real in - root `main.tf` (gap #12 CLOSED). See - `docs/changelog-20260709-office1-network-edge.md`. -- [x] Open question #2 (Office1 OPNsense edge ownership) RESOLVED 2026-07-09 - -- Stage 2 owns it (this runbook's new Step 4b); gap #16 CLOSED. Module - skeleton wired (commented) in root `main.tf`; full instantiation - remains blocked on gap #17 (per-site ISP-uplink/WAN network, not yet - resolved for any of the three sites) plus real VM specs -- do not close - Step 4b's own GATE bullet until that instantiation actually runs. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table updated to + the ACTUAL state reached -- do not mark NetBox's gate bullet closed unless + the D-101 literals were genuinely assigned and imported this run. +- [ ] Every `` placeholder in Steps 4-8 replaced with a + command confirmed against Canonical's current docs, and the confirmed + sequence folded back into this runbook -- so the next site (DC1, then DC2) + inherits a verified procedure rather than repeating the verification. +- [x] Provisioning-path fork (old Option A / Option B) RESOLVED 2026-07-13 by + D-114 -- OpenTofu is the path, `module "voffice1"` is instantiated and + applied, and the three service VMs are no longer OpenTofu VMs at all. +- [x] Office1-local network RESOLVED 2026-07-09 (`opentofu/modules/office1-network`, + gap #12 CLOSED). +- [x] Office1 OPNsense edge RESOLVED and BUILT 2026-07-12/13 (gap #16 CLOSED for + ownership; gap #17 CLOSED for Office1 via `office1-wan`).