# DC-DC Phase 1 -- Office1 site standup (Stage 2) > **REWRITTEN 2026-07-13 to the D-114 model.** The previous version of this > runbook built THREE SIBLING SERVICE VMs on the vcloud host (MAAS-region, > NetBox, GitBucket), each behind an "Option A (OpenTofu, blocked) / Option B > (manual virt-install)" fork, with Tailscale on the headend host. **That model > is SUPERSEDED.** D-114 (ADOPTED 2026-07-13) replaces it with ONE Office1 site > containment VM (`voffice1`) that IS the facility, with MAAS + LXD running ON > it and the non-stack service machines COMPOSED BY MAAS into that LXD. Do not > work from a cached memory of the old shape -- it will build the wrong thing. Stand up the Office1 SITE: the containment VM that simulates the facility, the MAAS region controller that runs inside it, the LXD VM host MAAS composes into, and the non-stack service machines (NetBox, GitBucket, Tailscale) as MAAS-visible VMs -- BEFORE either DC substrate exists. This is the second DC-DC runbook; it follows `runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit gate is this runbook's entry condition. Per this repo's session contract, do NOT re-derive anything Stage 1 already settled (nested KVM, disk budget, MTU, the DC1/Office1 libvirt planes and pools, the mesh-link networks) -- confirm Stage 1's gate passed and move on. **Execution status (measured 2026-07-13, not inferred):** - Office1's OPNsense edge (`office1-opnsense`) is **BUILT, RUNNING, ROUTING, and SERVING DHCP** (Kea). Its config is REST-API-managed per D-113(a2). - `office1-local` and `office1-wan` libvirt networks exist and are active. - **`voffice1` EXISTS and is RUNNING** (`virsh list --all` -> `voffice1 running`; `module.voffice1` is in `opentofu/terraform.tfstate`). It was applied by the main session's `tofu apply`, not by this runbook's authoring pass. - **NOT yet done:** everything from Step 5 onward -- MAAS-region on `voffice1`, LXD on `voffice1`, the LXD-KVM-host registration, and every MAAS-composed service machine. Those are this runbook's remaining work. **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 Phase 1 (goal/build/gate) and Section 5 (OpenTofu/MAAS/NetBox/Juju boundary); `docs/dc-dc-deployment-workflow.md` Stage 2 (tracker row -- update its `**State:**` line when this runbook completes a real run) and the Tooling gap register; `opentofu/README.md` (module scope/status for `modules/cloudinit-vm` / `modules/base-image` / `modules/office1-network` / `modules/opnsense-edge`); `netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer -- MECHANISM only, see Step 10). Decisions this runbook owns: **D-114** (site containment VMs + MAAS-composed LXD VMs for the non-stack machines -- the model this runbook now implements, including the LXD 5.21 LTS pin), **D-103** as AMENDED by D-114 (OpenTofu owns the site containment VM and the site's networks; INSIDE a site, MAAS composes the service machines -- MAAS still owns the provisioning lifecycle and still does NOT compose the OpenStack node VMs), **D-107** (airgap posture -- Office1 is explicitly OUT of the core-service path: no node artifacts, no NTP served from Office1; Tailscale is a narrowly-scoped front door, not a general egress path), **D-106** (naming convention -- Office1 services use a region-level subdomain outside the per-cloud convention, e.g. `maas.office1.vr1...`; exact scheme is this stage's job to fix, not yet fixed), and **D-101** (referenced only -- NetBox is the IPAM apex this stage stands up, but the address literals D-101 requires are NOT this stage's job to assign). !!! PENDING VERIFICATION -- the MAAS/LXD install-and-compose sequence (Steps 5, 6, 7, 8) is the NEW heart of this runbook and its exact command lines are NOT yet confirmed against Canonical's current documentation. Those steps name the MECHANISM and the GATE, and mark every unconfirmed flag, channel, and subcommand as ``. A separate verification pass owns filling them in. Per this repo's cardinal rule, a fabricated flag is worse than an honest placeholder -- do NOT "reconstruct" these from memory, from a different MAAS version's docs, or from this repo's own `scripts/phase-00-maas-standup.sh` / `scripts/reenroll-hosts.sh` (those are the PXE-enlisted METAL-NODE path, a DIFFERENT mechanism from D-114's COMPOSE path -- see Step 8's own note). !!! LXD IS VERSION-PINNED to the **5.21 LTS track** (D-114). MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7: LXD 6.7 consolidated API endpoints and MAAS's pinned pylxd 2.3.5 cannot speak to them. Canonical's guidance is "use LXD <= 6.6 or the 5.21 LTS release until further notice" (https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749). VR0 runs 5.21.4. Installing LXD from the default `latest/stable` channel would silently break the MAAS-LXD seam this entire stage depends on. The pin belongs in `runbooks/appendix-B-asbuilt-version-lock.md` -- record it there when the real installed version is measured. !!! DHCP HAZARD (D-114). OPNsense/Kea is AUTHORITATIVE for DHCP on `office1-local` (10.10.0.0/24, pool .100-.199). LXD's own `lxdbr0` runs a SECOND dnsmasq. That is FINE as long as `lxdbr0` stays on LXD's internal NAT bridge and is NEVER bridged onto `office1-local`. Two DHCP servers on one L2 is an intermittent, genuinely unpleasant failure to diagnose. Any LXD networking change in Step 6 must not put `lxdbr0` on the site LAN. --- ## Entry condition (Stage 1 gate + the Office1 edge -- confirm, do not re-derive) **CHECK -- read-only, before starting anything in this runbook** ```bash virsh net-list --all virsh pool-list --all virsh list --all ``` Expect: the six DC1 plane networks, the three mesh-link networks (dc1<->dc2, dc1<->office1, dc2<->office1), the `office1-local` network (`opentofu/modules/office1-network`), the `office1-wan` network (the Office1 ISP uplink -- gap #17 CLOSED for Office1), and the `dc1`/`office1`/`dc2` storage pools, all `active`. Expect the `office1-opnsense` domain `running`. If any of this is missing or `inactive`, STOP -- Stage 1 (or the Office1 edge build) is not actually done; go complete it rather than starting the rest of Stage 2 on an unfinished substrate. Do not re-measure nested KVM, disk budget, or MTU here -- Stage 1 already recorded those; this runbook only reads their OUTCOME (the networks and the edge existing) as its gate. --- ## The D-114 model in one paragraph (read before Step 1) Office1 is a SITE, not three peer VMs on a host. OpenTofu creates ONE containment VM -- `voffice1`, Ubuntu 24.04, on `office1-local`, taking a DHCP lease from the OPNsense edge's Kea, reaching the internet through that edge like a real server behind a real site router. `voffice1` IS the facility. The MAAS REGION CONTROLLER runs ON it. LXD (5.21 LTS) also runs ON it, and that LXD is REGISTERED BACK INTO MAAS as an LXD KVM host. The non-stack service machines -- NetBox, GitBucket, Tailscale -- are then VMs **composed by MAAS** into that LXD host: MAAS-visible, enlisted, commissioned, deployed, powered, releasable. This is not a new pattern; it is VR0's own already-proven `lxd` + `tailscale` pair, applied per site. Explicitly NOT in scope: the Juju-deployed OpenStack services on the DC nodes -- those stay in Juju-created LXD CONTAINERS (bundle `lxd:N` placements), invisible to MAAS, exactly as in VR0. Sequencing is an operator ruling: **Office1 FIRST, fully up. DC1 is GATED behind it. DC2 follows DC1.** **`expose_nested_virt = true` on `voffice1` is LOAD-BEARING, not a nicety.** LXD *virtual machines* are qemu/KVM guests, so composing NetBox/GitBucket/Tailscale as LXD VMs inside `voffice1` requires nested KVM at L3. (Only LXD *containers* would be free of that requirement -- and MAAS does not enlist containers.) Office1 is therefore the CHEAP nesting probe for the whole D-114 model: no OpenStack, no Ceph, small VMs. **If an LXD VM boots inside `voffice1`, the pod model is proven to L3 and DC1 inherits that proof** (D-114's own DC1 entry gate). If it does not, the containment-VM model is abandoned early having cost one VM -- a scoped retreat, by design, not a redesign. --- ## Open questions flagged here (not resolved by this runbook) Per this repo's discipline of naming ambiguity explicitly rather than quietly picking an answer: 1. **RESOLVED 2026-07-13 by D-114. The Option A / Option B provisioning fork is CLOSED.** The old fork ("OpenTofu `cloudinit-vm`, blocked pending user_data design" vs "manual `virt-install`, logged as debt") existed because no `cloudinit-vm` instantiation had ever been designed. **OpenTofu (Option A) is the path, and it is DONE:** `module "voffice1"` in `opentofu/main.tf` is a real, applied `cloudinit-vm` instantiation -- real image source (`modules/base-image`, the official Ubuntu 24.04 noble cloud image), real `user_data` (identity + access + guest agent), real `meta_data`, real `network_config` (DHCP on `office1-local`, matched by interface GLOB rather than a guessed kernel NIC name), and the new `expose_nested_virt` variable. There is nothing left to fork on: **the three service VMs are no longer OpenTofu VMs at all** -- they are MAAS-composed LXD VMs (Step 8). Do not re-introduce a manual `virt-install` path for them. 2. **The exact MAAS/LXD install, LXD-KVM-host registration, and VM-compose command lines are NOT yet verified.** See the PENDING VERIFICATION banner above. Steps 5-8 carry the placeholders; a separate verification pass against Canonical's current docs owns filling them in. Do not run those steps from a remembered flag set. 3. **NetBox's own address literals are not this stage's job**, per D-101's own text ("the org ULA /48, the per-DC GUA carve... and DC2's v4 supernet are NetBox-authoritative... NOT hardcoded in this decision"). This runbook stands NetBox up and can run its import MECHANISM (Step 10), but the underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not exist yet -- see Step 10's own gate. 4. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, pre-existing `git.baldurkeep.com` instance from the v1 rehearsal -- this stage stands up a SEPARATE, Office1-local GitBucket -- what it mirrors, and under what repo path, is not decided here. Flag as an open item before Step 11's mutation. 5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the front door is "Office1 only" -- the actual tailnet ACL policy, tags, and which services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the service machines?) are exposed through it versus reachable only from Office1's own local network, is real policy design left to Step 12 and the operator's own tailnet account -- not invented here. NOTE the placement change under D-114: Tailscale is now a MAAS-COMPOSED SERVICE MACHINE (a VM in `voffice1`'s LXD, exactly like VR0's own `tailscale` machine), NOT a package installed on the headend host. A pre-existing subnet route already reaches the jumphost; that is a different node and does not satisfy this gate. 6. **Exact MAAS / NetBox / GitBucket / LXD version pins are not decided** (beyond D-114's hard LXD 5.21-LTS-track constraint, which IS decided). Each install step below names the well-known official install MECHANISM but does not pin an exact version/channel/tag -- consult each project's current official install docs at execution time and record whatever is ACTUALLY installed in the changelog entry for this runbook's real run, and in `runbooks/appendix-B-asbuilt-version-lock.md`. 7. **`voffice1`'s stable address is not yet fixed.** It takes a DHCP lease from the edge's Kea (pool .100-.199). A Kea RESERVATION (added over the OPNsense REST API, `scripts/opnsense-api.sh`) is what makes it stable -- Step 4. Until that reservation exists, do NOT hardcode `voffice1`'s address anywhere, including in MAAS's own `--maas-url`. --- ## Sequence ``` 1. Confirm Stage 1 exit gate + Office1 edge live (read-only, above) 2. Confirm OpenTofu still reaches vcloud libvirt (read-only) 2b. Office1 OPNsense edge -- BUILT AND LIVE (historical + DANGER banner; do NOT re-run against the live edge) (read-only) 3. voffice1 -- the site containment VM (tofu apply) [MUTATION, gated] -- APPLIED 2026-07-13; domain running (measured) 4. voffice1 first contact -- lease, Kea reservation, SSH [MUTATION, gated] 5. MAAS region controller ON voffice1 [MUTATION, gated] PENDING VERIF. 6. LXD ON voffice1, pinned to the 5.21 LTS track [MUTATION, gated] PENDING VERIF. 7. Register that LXD BACK INTO MAAS as an LXD KVM host [MUTATION, gated] PENDING VERIF. 8. Compose the non-stack service machines into it (MAAS) -- THE L3 NESTING PROOF; gates DC1 [MUTATION, gated] PENDING VERIF. 9. NetBox -- install on its composed machine [MUTATION, gated] 10. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; literals pending -- PARTIAL) [MUTATION, gated] 11. GitBucket -- install on its composed machine [MUTATION, gated] 12. Tailscale -- install on its composed machine, scoped [MUTATION, gated] 13. Post-standup verify against the Stage 2 gate (read-only) -> EXIT GATE -> Stage 3 (DC1), which D-114 GATES behind this ``` --- ## Step 1 -- Confirm Stage 1 exit gate (READ-ONLY) Covered above under "Entry condition." Do not proceed past this point until that CHECK is green. --- ## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt (READ-ONLY) Stage 1 proved OpenTofu -> vcloud-host libvirt reachability and has since APPLIED for real (the state file holds the DC1 planes, the pools, the three mesh links, `office1-network`, the OPNsense edge, and `voffice1`). This step is a re-verification, not a re-derivation. **CHECK -- from the Office1 operator session** ```bash tofu -chdir="$REPO/opentofu" version virsh -c "" list --all ``` (`$REPO` is set once per session -- see `runbooks/README.md` Conventions.) Expect: the connection succeeds, and the domain list shows `office1-opnsense` and `voffice1`. If the connection fails, STOP -- do not proceed into any mutation without confirmed libvirt reach. **CHECK -- the tree still validates** ```bash bash scripts/opentofu-validate.sh ``` Expect: root + every module PASS (10/10 as of DOCFIX-194 -- the gate now validates every module STANDALONE, because root-only validation silently skipped the two modules root does not call). --- ## Step 2b -- Office1's OPNsense edge: BUILT AND LIVE (READ-ONLY / historical) The Office1 edge is no longer work this runbook does -- it is a satisfied PREREQUISITE. It was built 2026-07-12/13 and is routing, NAT-ing, providing egress, and serving Kea DHCP on `office1-local`. `voffice1` depends on all four of those. This section is retained as the historical record of how the image is prepped, and for its safety banner, which is still live. > ## !!! DANGER -- DO NOT RUN A CONFIG PUSH AGAINST THE LIVE OFFICE1 EDGE (2026-07-13) !!! > > **The Office1 edge is BUILT, ROUTING, and SERVING DHCP.** Its DHCP config is now > **API-MANAGED** (D-113(a2), proven live 2026-07-13). Rendering a full `config.xml` and > pushing it to `10.10.0.1` **WOULD CLOBBER THAT LIVE STATE** -- a full-config push replaces > `/conf/config.xml` wholesale and drops ~667 migration-populated elements. > > **Two things are now permanently WRONG and must not be attempted:** > > 1. **The config ISO cannot work AT ALL.** Per **D-112**, the OPNsense Configuration Importer > can NEVER fire on a pre-installed nano image -- `opnsense-importer -b` probes for a > read-only root, finds a writable one with a factory `/conf/config.xml` already present, and > `bootstrap_and_exit 0`s without enumerating a single device. The ISO is INERT. Nothing was > ever going to read it. > 2. **Full-`config.xml` delivery is SUPERSEDED** by D-113(a2): edge configuration is done over > the **REST API** (`scripts/opnsense-api.sh`), not by hand-authoring the appliance's > GUI-owned XML. Hand-authoring that XML was the single root cause of DOCFIX-191 (lockout), > DOCFIX-192 (dead console) and DOCFIX-193 (no DHCP). > > **What is actually true today:** Office1 was brought up by the D-112(c) console bootstrap, and > is now managed over SSH + the REST API. **Image prep (host-local files only -- the one part still runnable, and only for a from-scratch re-standup):** ```bash bash scripts/opnsense-prep-image.sh # see the script's own header for its exact args/output path ``` **GATE (Stage 2 Gate bullet):** Office1 OPNsense edge up -- **MET 2026-07-12/13** (routing, NAT, egress, serial console, SSH-key managed, Kea DHCP). Gap #17 is CLOSED for Office1 via the dedicated `office1-wan` ISP-uplink network. DC1/DC2 still have no uplink network -- that is Stage 3's problem, not this one's. --- ## Step 3 -- voffice1: the site containment VM [MUTATION: gated] > **STATUS: APPLIED 2026-07-13.** `module.voffice1` is in > `opentofu/terraform.tfstate` and the domain is RUNNING (measured this session: > `virsh list --all` -> `voffice1 running`). The apply was performed by the main > session, not by this runbook's authoring pass. This step is retained as the > procedure -- for a re-standup, and so the gate it satisfies is auditable. > **Re-running `tofu apply` when `voffice1` is already up is NOT a no-op to be > taken casually:** an apply that reports "updated in-place" can still BOUNCE a > domain (measured 2026-07-13 -- an apply restarted the live edge). Read the plan > before approving it. This is the D-114 site containment VM. It IS the Office1 facility: MAAS-region runs on it, LXD runs on it, and the non-stack service machines are composed into that LXD by MAAS. It sits on `office1-local` ONLY -- it reaches the internet THROUGH the OPNsense edge, like a real server behind a real site router. **CHECK -- confirm the current state before any apply** ```bash virsh -c "" list --all | grep -i voffice1 tofu -chdir="$REPO/opentofu" plan ``` Expect either no `voffice1` domain (fresh standup -- the apply will create it) or the domain present AND a plan showing no changes to `module.voffice1`. An unexplained diff against a running containment VM is a STOP. **MUTATION -- create voffice1 (OpenTofu; Option A, and now the ONLY path)** ```bash tofu -chdir="$REPO/opentofu" apply ``` Approve only after reading the plan. The module is already fully specified in `opentofu/main.tf` -- do not re-derive its inputs here. Its load-bearing properties, for the record: - `modules/cloudinit-vm` off `modules/base-image` (the official Ubuntu 24.04 noble cloud image -- 24.04 is what VR0 deploys for its own `lxd`/`juju`/ `tailscale` machines, so the image choice carries no new delta). - `network_names = [office1-local]` ONLY. No mesh leg, no direct WAN. - `network_config` takes **DHCP** from the edge's Kea, matching the NIC by GLOB (`en*`) rather than a guessed kernel name -- the NIC's name is not knowable before first boot, and naming it would be an inferred value. - **`expose_nested_virt = true`** -- LOAD-BEARING (see "The D-114 model" above). Without the host CPU feature passed through, no LXD VM can ever boot inside `voffice1` and D-114's model fails at its first step. - Cloud-init is DELIBERATELY MINIMAL (identity + access + guest agent). MAAS and LXD are installed as separate GATED steps below so they are observable and individually approved, not buried in a first-boot script that either silently works or silently does not. **SECURE THE STATE FILE** after any apply (standing discipline from DOCFIX-175): confirm `opentofu/**/*.tfstate*` is gitignored, permissions are tight, and an out-of-band backup exists. The state stores `sensitive = true` values (e.g. `var.maas_api_key`) in PLAINTEXT. **GATE (Stage 2, first Gate bullet):** `voffice1` exists and is `running`. **MET 2026-07-13 (measured).** --- ## Step 4 -- voffice1 first contact: lease, Kea reservation, SSH [MUTATION: gated] `office1-local` is an isolated libvirt network with NO libvirt-managed DHCP -- `virsh net-dhcp-leases office1-local` is EMPTY BY DESIGN and is NOT evidence that `voffice1` failed to get an address. Kea, on the OPNsense edge, is the authoritative DHCP server for that L2. Look for the lease THERE. Per D-114, `voffice1` booting on `office1-local` takes the **first real DHCP lease** the Kea path has ever served -- a path proven so far only at the daemon level, never end to end. Treat a successful lease as a real result worth recording, and a failed one as a real finding (not a nuisance to work around by hand-assigning a static address). **CHECK -- read the lease from the edge (read-only)** ```bash bash scripts/opnsense-api.sh ``` Expect: one lease, in the .100-.199 pool, whose MAC matches `voffice1`'s interface (`virsh -c "" domiflist voffice1`). MEASURE both -- do not assume the lease belongs to `voffice1` because it is the only one. **MUTATION -- pin the address with a Kea RESERVATION (gated)** Add a Kea host reservation binding `voffice1`'s measured MAC to a stable address, over the OPNsense REST API (`scripts/opnsense-api.sh`). Do NOT hand-edit `/conf/config.xml` -- see Step 2b's DANGER banner. Do NOT hardcode an address anywhere (MAAS `--maas-url` included) until this reservation exists and has been re-measured after a `voffice1` reboot. **CHECK -- SSH reach** ```bash ssh -i jessea123@ 'hostnamectl; systemd-detect-virt; cloud-init status --long' ``` Expect: hostname `voffice1`, `kvm`, and cloud-init `status: done`. A cloud-init that is still `running` or `error` is a STOP -- read its logs before layering MAAS on top of a half-configured machine. **CHECK -- nested virt is actually exposed (the L3 precondition)** ```bash ssh 'grep -c -w -E "svm|vmx" /proc/cpuinfo; ls -l /dev/kvm' ``` Expect: a non-zero count and `/dev/kvm` present. **If this fails, STOP** -- `expose_nested_virt` did not take effect, and every LXD VM in Step 8 will fail to boot. That is D-114's own early-abandon signal; do not paper over it. **GATE (Stage 2 Gate bullet, NEW):** `voffice1` holds a Kea RESERVATION, is reachable over SSH, and exposes nested KVM. --- ## Step 5 -- MAAS region controller ON voffice1 [MUTATION: gated] (PENDING VERIFICATION) > **PENDING VERIFICATION.** The exact snap channel and the exact `maas init` > flag set (`--maas-url`, database backend selection, admin creation) VARY BY > MAAS RELEASE. They are NOT reproduced from memory here. A separate > verification pass against MAAS's current official install docs owns filling > these in. Do not substitute this repo's `scripts/phase-00-maas-standup.sh` > -- that script drives an EXISTING MAAS, it does not install one. Under D-114 the MAAS region controller runs INSIDE `voffice1`, not on a sibling VM of its own. It is the site's provisioning authority: it will own the LXD KVM host (Step 7) and every service machine composed into it (Step 8). **CHECK -- nothing pre-existing** ```bash ssh 'snap list maas 2>&1 || echo "not installed"' ``` **MUTATION -- install MAAS (snap, the official method)** ```bash ssh 'sudo snap install maas --channel=' ``` **MUTATION -- initialize it as a region controller** Per MAAS's own current `maas init` documentation for the installed version. `` Use `voffice1`'s RESERVED address (Step 4) for the MAAS URL -- never the raw lease, and never a guessed address. **CHECK -- the region controller answers** ```bash curl -sI "http://:5240/MAAS/" | head -1 ``` Expect an HTTP response, not a connection failure. Naming per D-106: this service's FQDN, once DNS exists for Office1, follows the region-level subdomain scheme D-106 names as an example (`maas.office1.vr1...`) -- record whatever real hostname convention you actually use; do not assume it. **MUTATION -- create the MAAS API key for later steps** Steps 7 and 8 (and `opentofu/modules/maas-vm-host`, if it is ever wired for Office1) need a real MAAS API key. Generate it per MAAS's own docs and handle it per this repo's secrets discipline: never printed, never committed, passed as an environment variable or read from a jumphost-local file only. Note DOCFIX-175: `var.maas_api_key` lands in `terraform.tfstate` in PLAINTEXT if OpenTofu is given it -- factor that into where the key is allowed to go. **GATE (Stage 2 Gate bullet):** MAAS region reachable, running ON `voffice1`. --- ## Step 6 -- LXD ON voffice1, pinned to the 5.21 LTS track [MUTATION: gated] (PENDING VERIFICATION) > **PENDING VERIFICATION.** The `lxd init` answers (storage backend and size, > bridge configuration, whether the LXD HTTPS API is exposed and on what > address/port, and how MAAS is expected to authenticate to it -- trust > password vs. certificate) are NOT reproduced from memory. They are precisely > what the MAAS-LXD seam depends on. A separate verification pass against > Canonical's current MAAS + LXD docs owns filling them in. **MUTATION -- install LXD, PINNED (D-114)** ```bash ssh 'sudo snap install lxd --channel=5.21/stable' ``` The `5.21` track is a HARD D-114 constraint, not a preference: MAAS 3.6/3.7 cannot talk to LXD >= 6.7 (pylxd 2.3.5 vs LXD's consolidated API endpoints). `latest/stable` WILL eventually hand you a broken seam. If LXD is already present from the image at a different track, refresh it onto 5.21 rather than proceeding -- and record the measured version in appendix-B. **CHECK -- confirm the pin actually took** ```bash ssh 'snap list lxd; lxd --version' ``` Expect a 5.21.x version. **A 6.x version here is a STOP** -- it is the exact incompatibility D-114 pins against. **MUTATION -- initialize LXD** ```bash ssh 'sudo lxd init ' ``` !!! **`lxdbr0` MUST STAY on LXD's internal NAT bridge.** Do NOT bridge it onto `office1-local`. Kea (on the OPNsense edge) is the authoritative DHCP server for that L2; `lxdbr0`'s own dnsmasq on the same L2 would be a second one. See the DHCP HAZARD banner at the top of this runbook. **CHECK -- LXD is up and its bridge is where it belongs** ```bash ssh 'lxc network list; ip -br addr show lxdbr0' ``` Expect `lxdbr0` on its own private subnet, NOT on 10.10.0.0/24. **GATE (Stage 2 Gate bullet, NEW):** LXD installed on `voffice1`, version on the 5.21 LTS track (MEASURED), `lxdbr0` isolated from `office1-local`. --- ## Step 7 -- Register that LXD BACK INTO MAAS as an LXD KVM host [MUTATION: gated] (PENDING VERIFICATION) > **PENDING VERIFICATION.** The exact MAAS CLI/API call that adds an LXD KVM > host (its subcommand, its power/auth parameters, the project name, and how > the LXD certificate trust is established) is NOT reproduced from memory. A > separate verification pass owns it. VR0's as-built shows the TARGET STATE: > an LXD KVM host in project `default`, LXD 5.21.4. This is the seam D-114 turns on. `voffice1` is a MAAS-managed machine that runs LXD; that LXD is then registered as a VM HOST inside the same MAAS. It is what makes the service machines MAAS-visible instead of hand-run containers. **Note the D-103/D-114 boundary, precisely:** the composition right granted here is scoped to THIS LXD VM host and the NON-STACK machines. MAAS still does NOT compose the OpenStack NODE VMs -- `modules/node-vm` pre-creates those and `modules/maas-vm-host` registers the VIRSH host so MAAS merely DISCOVERS them. `maas_vm_host_machine` remains ruled out for nodes. **CHECK -- read the current VM-host inventory (read-only)** ```bash ssh 'maas vm-hosts read' ``` **MUTATION -- add the LXD KVM host** PENDING VERIFICATION: the exact MAAS "add a VM host of type LXD" invocation -- its subcommand, power address, project name, and credentials/certificate trust. Consult MAAS's own current LXD-VM-host documentation. Do NOT adapt this from `scripts/phase-00-maas-standup.sh` or `scripts/reenroll-hosts.sh`: those are the PXE-enlisted metal-node path, a different mechanism. **CHECK -- MAAS sees the LXD host, with real capacity** ```bash ssh 'maas vm-hosts read' ``` Expect the LXD host present, with its cores/memory/storage reported -- MAAS reading real numbers back out of LXD is the evidence the seam works, not merely that the record was created. **GATE (Stage 2 Gate bullet, NEW):** `voffice1`'s LXD is registered in MAAS as an LXD KVM host and MAAS reads its capacity. --- ## Step 8 -- Compose the non-stack service machines (NetBox, GitBucket, Tailscale) [MUTATION: gated] (PENDING VERIFICATION) > **PENDING VERIFICATION.** The exact `maas ... vm-host compose` parameters > (cores, memory, storage, and how the composed machine is attached to the right > network) are NOT reproduced from memory. A separate verification pass owns > them. VR0's as-built gives the SIZING PRECEDENT, measured, not invented: > its `tailscale` machine is **2 cores / 2 GiB / 25 GB, Ubuntu 24.04** -- > D-114's own figure for what a composed service machine costs. **This step is the L3 NESTING PROOF, and it GATES DC1.** The first LXD *virtual machine* that boots inside `voffice1` proves the whole D-114 pod model to L3. D-114's DC1 entry gate depends on it. If no LXD VM will boot here, STOP and report -- do not "work around" it by falling back to LXD containers (MAAS does not enlist containers) or by hand-running the services on `voffice1` itself (that reconstructs the flat model D-114 replaced, without saying so). **MUTATION -- compose the machines, ONE AT A TIME, individually gated** Compose in this order, gating each: **Tailscale first** (it is the smallest, it is VR0's proven exemplar, and if it boots the model is proven at minimum cost), then NetBox, then GitBucket. ```bash ssh 'maas vm-host compose ' ``` Each composed machine then goes through MAAS's OWN provisioning lifecycle -- this is the entire point of D-114's ruling (they are MAAS-VISIBLE machines, not hand-run VMs): 1. **Enlist** -- MAAS registers the composed VM. 2. **Commission** -- MAAS boots it and inventories it. 3. **Deploy** -- MAAS installs the OS (Ubuntu 24.04, matching VR0's own service machines). 4. **Power** -- MAAS controls it via the LXD VM host. `` **CHECK -- the machines are real, MAAS-visible, and DEPLOYED** ```bash ssh 'maas machines read | ' ``` Expect each service machine `Deployed`, powered via the LXD VM host. This is VR0's `lxd` + `tailscale` shape, reproduced per site. **GATE (Stage 2 Gate bullet, NEW -- and D-114's DC1 entry gate, half (b)):** at least one LXD **virtual machine** BOOTS inside `voffice1`, and the non-stack service machines are MAAS-composed, deployed, and powered. **L3 nesting PROVEN.** Record this explicitly -- DC1 is gated on it. --- ## Step 9 -- NetBox: install on its composed machine [MUTATION: gated] The VM is now a MAAS-DEPLOYED machine (Step 8), not a hand-built one. This step only installs the application onto it. **MUTATION -- install NetBox (official netbox-docker Compose method)** Over SSH to the composed NetBox machine, follow NetBox's own official Docker-based install (the `netbox-community/netbox-docker` repo's documented Compose workflow: clone it, configure its `.env` / `docker-compose.override.yml` per its own current docs, `docker compose up -d`). Do not hand-roll a from-source install -- Docker Compose is NetBox's own recommended path and the lowest-delta one to keep reproducible. Consult NetBox's current official docs for the exact compose file / image tag to pin; do not fabricate a version tag. **CHECK -- verify NetBox is reachable** ```bash curl -sI "http://:8080/" | head -1 ``` Expect an HTTP response. Create/record a NetBox API token per NetBox's own docs (needed by Step 10) -- handle it per this repo's secrets discipline: never printed, never committed, passed as an environment variable only (`NETBOX_TOKEN`, matching `netbox/dc-dc-prefixes-import.py`'s own documented usage). **GATE (Stage 2 Gate, partial):** NetBox reachable and running. NOT yet "authoritative and populated" -- see Step 10. --- ## Step 10 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] `netbox/dc-dc-prefixes-import.py` (DOCFIX-152) extends the v1 single-site import to VR1's per-DC, dual-stack model per D-101. It is idempotent and `--verify-only`-capable. **It does NOT, and cannot, close the DATA half of tooling gap #3**: the real org ULA /48, the per-DC GUA carve, and DC2's v4 supernet do not exist yet as assigned values anywhere in this repo or session. Running it for DC1 requires the two v6 literals (`ORG_ULA_48`, `DC_GUA_PREFIX`); running it for DC2 additionally requires `DC2_V4_SUPERNET` -- none of the three has a default, by the script's own design, and none is invented here. **CHECK -- dry run, no writes** ```bash NETBOX_URL="http://:8080" \ NETBOX_TOKEN="" \ ORG_ULA_48="" \ DC_GUA_PREFIX="" \ python3 netbox/dc-dc-prefixes-import.py --dc dc1 --verify-only ``` If `ORG_ULA_48` / `DC_GUA_PREFIX` are not yet assigned, this command CANNOT be run for real yet -- that is the honest state of this stage, not a missing step in this runbook. Record that as the blocking condition rather than inventing placeholder-looking values to "get past" it (this script is designed to fail loud on exactly that pattern, per its own header). See `docs/dc-dc-netem-and-ula-gua-proposal.md` for the drafted ULA-generation guidance -- a PROPOSAL, not a ruling. **MUTATION -- once the literals above are real (a later session/step)** ```bash NETBOX_URL="..." NETBOX_TOKEN="..." ORG_ULA_48="..." DC_GUA_PREFIX="..." \ python3 netbox/dc-dc-prefixes-import.py --dc dc1 ``` DC1's v4 is hardcoded in the script itself (D-101: "DC1 equals the validated template," the explicit text of an ADOPTED decision, not an inferred value). DC2 additionally needs `DC2_V4_SUPERNET`, not assigned yet either. **GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 9). "Authoritative and populated (planes, per-DC v4, ULA/GUA carve)" is only PARTIALLY achievable: the IMPORT MECHANISM exists and is runnable the moment the literals are assigned, but the literals themselves are not assigned, so the DATA half of this gate bullet is NOT met. Do not mark this bullet green in the tracker until a real `ORG_ULA_48` / `DC_GUA_PREFIX` (and, for DC2, `DC2_V4_SUPERNET`) exist and this script has actually run against this real NetBox instance. --- ## Step 11 -- GitBucket: install on its composed machine [MUTATION: gated] **MUTATION -- install GitBucket (official method)** GitBucket ships as a single runnable WAR (`java -jar gitbucket.war`, its own documented standalone-Jetty mode) or as an official Docker image (`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one and record which. Consult GitBucket's current official docs/release page for the exact WAR filename/version or image tag to pin -- do not fabricate one. Open question #4 above (this instance's purpose/repo-path scheme, distinct from the existing `git.baldurkeep.com` v1 instance per D-014) must be resolved before deciding what this instance actually mirrors. **CHECK -- verify GitBucket is reachable** ```bash curl -sI "http://:/" | head -1 ``` Expect an HTTP response. **GATE (Stage 2 Gate bullet):** GitBucket serving. --- ## Step 12 -- Tailscale: install on its composed machine, scoped to Office1 [MUTATION: gated] Per D-114, Tailscale is a MAAS-COMPOSED SERVICE MACHINE (VR0's own `tailscale` machine, reproduced per site) -- NOT a package on the headend host, which is what the superseded model did. Per D-107, Office1 is explicitly OUT of the core-service path (no node artifacts, no NTP served from Office1) -- Tailscale is a narrowly-scoped OPERATOR front door to Office1's own services, not a general egress/access path for anything DC-side. Do not wire it into any DC-facing plane. A pre-existing subnet route already reaches the jumphost. That is a DIFFERENT node and does NOT satisfy this gate -- do not mistake it for one. **CHECK -- confirm nothing conflicting on the composed machine** ```bash ssh 'tailscale version 2>&1 || echo "not installed"' ``` **MUTATION -- install (official method)** ```bash ssh 'curl -fsSL https://tailscale.com/install.sh | sh' ``` Consult Tailscale's own current docs if this machine's distro needs a different install path (the install script itself detects and branches on distro). **MUTATION -- bring up, scoped** ```bash ssh 'sudo tailscale up --advertise-tags=' ``` The actual ACL policy restricting reachability to Office1's services only (and NOT to any DC-side network) is configured in the operator's own Tailscale admin console/ACL file -- real policy design (Open question #5), not invented here. Do not advertise routes into any DC plane from this node. **CHECK -- verify scope** ```bash ssh 'tailscale status' ``` Confirm only the expected Office1 node(s) appear, and that no route advertisement reaches into DC1/DC2 plane CIDRs. **GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only, from a MAAS-composed machine. --- ## Step 13 -- Post-standup verify against the Stage 2 gate (READ-ONLY) ```bash virsh -c "" list --all ssh 'snap list maas lxd; lxd --version' ssh 'maas vm-hosts read' ssh 'maas machines read' curl -sI "http://:5240/MAAS/" | head -1 curl -sI "http://:8080/" | head -1 curl -sI "http://:/" | head -1 ssh 'tailscale status' ``` Cross-check each against the GATE section below before declaring this stage done or partially done. --- ## GATE (Stage 2 exit condition) Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row, as reshaped by D-114: - **Office1 OPNsense edge up (routing, NAT, egress, DHCP):** **MET 2026-07-12/13.** - **Office1-local + office1-wan networks created:** **MET** (gaps #12 and #17-for-Office1 CLOSED). - **OpenTofu reaches vcloud host libvirt:** **MET** (Stage 1; re-verified in Step 2). - **`voffice1` site containment VM exists and runs:** **MET 2026-07-13 (measured).** - **`voffice1` has a stable (reserved) address and SSH reach, with nested KVM exposed:** Step 4 -- NOT yet confirmed. - **MAAS region reachable, running ON `voffice1`:** Step 5 -- NOT yet done. - **LXD on `voffice1`, on the 5.21 LTS track, `lxdbr0` off `office1-local`:** Step 6 -- NOT yet done. - **That LXD registered in MAAS as an LXD KVM host:** Step 7 -- NOT yet done. - **Non-stack service machines COMPOSED BY MAAS into it, deployed and powered -- i.e. an LXD VM BOOTS at L3:** Step 8 -- NOT yet done. **This is the D-114 DC1 entry gate.** - **GitBucket serving:** Step 11 -- NOT yet done. - **Tailscale confirmed to Office1 only, from a composed machine:** Step 12 -- NOT yet done, and contingent on the operator's own ACL policy (Open question #5). - **NetBox authoritative and populated:** ONLY PARTIALLY achievable. The instance can be stood up and reachable (Step 9), and the import MECHANISM exists and is runnable (Step 10) -- but the real address literals (org ULA /48, per-DC GUA carve, DC2 v4 supernet) are not assigned, so NetBox cannot yet be truly "populated" per D-101. Do not mark this bullet green until those literals exist and `netbox/dc-dc-prefixes-import.py` has actually run against the real instance. This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** -- the substrate half (networks, edge, `voffice1`) is DONE and measured; the SITE half (MAAS, LXD, the LXD KVM host, and every composed service machine) is the remaining work, and its command lines are PENDING VERIFICATION. The NetBox DATA bullet closes its mechanism half only, pending the D-101 literal assignment that is explicitly out of scope for this stage to invent. Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever of these is actually true after a real run, not to "DONE" wholesale. -> **Stage 3 (DC1) is GATED behind this stage.** D-114's sequencing is an operator ruling: Office1 FIRST, fully up on this model; DC1 is not started until Office1 is complete; DC2 follows DC1. DC1's own entry gate additionally requires proving nested KVM inside `vdc1` (which also proves L4 for tenant instances -- the one depth VR0 has never exercised). **If L3 fails at Step 8, the containment-VM model is abandoned early having cost one VM** -- that is the designed cheap retreat, and it is the correct outcome to report, not a failure to engineer around. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook. - [ ] If any `opentofu/` change was written this run, `bash scripts/opentofu-validate.sh` green (root + EVERY module -- DOCFIX-194). - [ ] Changelog entry for this runbook's real execution (next number via `bash scripts/ledger-scan.sh`), noting: the MAAS/LXD versions ACTUALLY installed, `voffice1`'s reserved address, whether the L3 nesting proof passed, which service machines were composed, and whether NetBox's literal-assignment gap was closed or remains open. - [ ] `runbooks/appendix-B-asbuilt-version-lock.md` updated with the LXD 5.21-LTS pin (D-114 requires it be recorded there) and the measured MAAS version. - [ ] `docs/session-ledger.md` updated with the outcome. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table updated to the ACTUAL state reached -- do not mark NetBox's gate bullet closed unless the D-101 literals were genuinely assigned and imported this run. - [ ] Every `` placeholder in Steps 4-8 replaced with a command confirmed against Canonical's current docs, and the confirmed sequence folded back into this runbook -- so the next site (DC1, then DC2) inherits a verified procedure rather than repeating the verification. - [x] Provisioning-path fork (old Option A / Option B) RESOLVED 2026-07-13 by D-114 -- OpenTofu is the path, `module "voffice1"` is instantiated and applied, and the three service VMs are no longer OpenTofu VMs at all. - [x] Office1-local network RESOLVED 2026-07-09 (`opentofu/modules/office1-network`, gap #12 CLOSED). - [x] Office1 OPNsense edge RESOLVED and BUILT 2026-07-12/13 (gap #16 CLOSED for ownership; gap #17 CLOSED for Office1 via `office1-wan`).