# DC-DC Phase 1 -- Office1 headend standup (Stage 2) Stand up the operator's deployment headend -- MAAS region controller, confirmed OpenTofu reach, NetBox, GitBucket, Tailscale -- BEFORE either DC substrate exists. This is the second DC-DC runbook; it follows `runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit gate is this runbook's entry condition. Per this repo's session contract, do NOT re-derive anything Stage 1 already settled (nested KVM, disk budget, MTU, the DC1/Office1 libvirt planes and pools, the mesh-link networks) -- confirm Stage 1's gate passed and move on. **Authoring context (read before running):** this runbook was authored PREP-ONLY, with no live vcloud host, MAAS, NetBox, GitBucket, or Tailscale reachable in the authoring session. Every command below is written to be run for real, later, by the operator -- it has not been executed against real infrastructure. Per `opentofu/README.md` and `docs/dc-dc-deployment-workflow.md` Stage 2 (tooling gap register items #2, #3, #9): **none of the three Office1 service VMs (MAAS-region, NetBox, GitBucket) have an OpenTofu module instantiation yet** -- `opentofu/modules/cloudinit-vm` exists as a mechanism, but no image source has been chosen and no `user_data`/`meta_data`/`network_config` content has been designed for any of the three. This runbook does not invent that content. It gives each VM-creation step two explicit paths (Section "Provisioning-path decision" below) and flags the OpenTofu path as blocked-pending-design rather than silently choosing the manual path for the operator. **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 Phase 1 (goal/build/gate) and Section 5 (OpenTofu/MAAS/NetBox/Juju boundary); `docs/dc-dc-deployment-workflow.md` Stage 2 (tracker row -- update its `**State:**` line when this runbook completes a real run) and the Tooling gap register (items #2, #3, #9 above all bear directly on this stage); `opentofu/README.md` (module scope/status for `modules/cloudinit-vm` / `modules/base-image`); `netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer, built 2026-07-09 -- MECHANISM only, see Step 7). Decisions this runbook owns: **D-103** (OpenTofu/MAAS/Juju lifecycle seam -- Office1 service VMs are OpenTofu-created, eventually; MAAS owns their provisioning lifecycle once enlisted the same way DC node VMs are), **D-107** (airgap posture -- Office1 is explicitly OUT of the core-service path: no node artifacts, no NTP served from Office1; Tailscale is a narrowly-scoped front door, not a general egress path), **D-106** (naming convention -- Office1 services use a region-level subdomain outside the per-cloud convention, e.g. `maas.office1.vr1...`; exact scheme is this stage's job to fix, not yet fixed), and **D-101** (referenced only -- NetBox is the IPAM apex this stage stands up, but the address literals D-101 requires are NOT this stage's job to assign). !!! PREP-ONLY / NOT YET EXECUTED. No live infrastructure exists or was reachable in the authoring session. Every MUTATION below is written for the operator to run and individually gate later, per this repo's verify-before-mutate discipline -- none of it has been run. !!! STRUCTURAL BLOCKER, flagged not fixed here: the OpenTofu path for all three Office1 service VMs needs `user_data`/`meta_data`/`network_config` designed against `modules/cloudinit-vm` before it can be used -- that design work has NOT been done. Step 3 below presents the resulting Option A (blocked) / Option B (manual, interim) fork explicitly for each VM. Do not silently pick one without reading Step 3 first. --- ## Entry condition (Stage 1 gate -- confirm, do not re-derive) **CHECK -- read-only, before starting anything in this runbook** ```bash virsh net-list --all virsh pool-list --all ``` Expect: the six DC1 plane networks, the three mesh-link networks (dc1<->dc2, dc1<->office1, dc2<->office1), and the `dc1`/`office1` storage pools, all `active` -- this is `dc-dc-phase0-vcloud-prep.md`'s Step 11 output. If any of this is missing or `inactive`, STOP -- Stage 1 is not actually done; go re-run/complete that runbook rather than starting Stage 2 on an unfinished Stage 1. Do not re-measure nested KVM, disk budget, or MTU here -- Stage 1 already recorded those; this runbook only reads their OUTCOME (the networks existing) as its gate. --- ## Open questions flagged here (not resolved by this runbook) Per this repo's discipline of naming ambiguity explicitly rather than quietly picking an answer: 1. **What virtual network do the three Office1 service VMs themselves attach to?** `opentofu/main.tf` (per Stage 1) wires `office1_pool_path` (storage only) and the three mesh-link networks (Office1<->DC1, Office1<->DC2, inter-site only, "management traffic only" per D-100's own sub-item ruling). No Office1-LOCAL network (something the three service VMs and the vcloud host's own management interface could all share) is modeled anywhere in `opentofu/`. This is a real gap, not an oversight this runbook papers over: either a new `modules/dc-planes`-shaped Office1-local network needs designing, or an existing host bridge needs to be reused deliberately (and documented as such) -- a decision for whoever designs the `cloudinit-vm` `network_config` content in the blocked Option A path below. Flag this to the operator before Step 3; do not invent a network name to move forward. 2. **NetBox's own address literals are not this stage's job**, per D-101's own text ("the org ULA /48, the per-DC GUA carve... and DC2's v4 supernet are NetBox-authoritative... NOT hardcoded in this decision"). This runbook stands NetBox up and can run its import MECHANISM (Step 7), but the underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not exist yet -- see Step 7's own gate. 3. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, pre-existing `git.baldurkeep.com` instance from the v1 rehearsal -- this stage stands up a SEPARATE, vcloud-local GitBucket instance (per the buildout design's own "GitBucket (vcloud-local mirror)" wording) -- what it mirrors, and under what repo path, is not decided here. Flag as an open item before Step 8's mutation. 4. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the front door is "Office1 only" -- the actual tailnet ACL policy, tags, and which services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the service VMs?) are exposed through it versus reachable only from Office1's own local network, is real policy design left to Step 9 and the operator's own tailnet account -- not invented here. 5. **Exact MAAS / NetBox / GitBucket version/channel pins are not decided.** Each install step below names the well-known official install MECHANISM (snap for MAAS, Docker Compose for NetBox, WAR/Docker for GitBucket) but does not pin an exact version/channel/tag -- consult each project's current official install docs at execution time and record whatever is actually installed in the changelog entry for this runbook's real run. --- ## Provisioning-path decision (applies to all three service VMs) Per the structural blocker above, EVERY VM-creation step in this runbook (5, 6, 8) has two paths. Do not silently default to one -- read both, pick deliberately, and log which path was used in the changelog entry. - **Option A (preferred, long-term, BLOCKED pending design):** create the VM via `opentofu/modules/cloudinit-vm`, once (a) Open question #1 above is resolved (what network it attaches to) and (b) real `user_data`/`meta_data`/ `network_config` content is designed for that specific VM (what packages, what static address, what hostname). This is the D-103-correct path (`OpenTofu creates... the Office1 service VMs`) and is NOT available tonight -- doing it right requires a design pass this session did not do and this runbook will not invent. - **Option B (interim, manual):** create the VM directly against the vcloud host's libvirt (e.g. `virt-install` with a real official cloud image ISO/ qcow2, or an interactive install), attach it to whatever network Open question #1 resolves to, then run the install steps below directly over SSH once the VM has a reachable address. This gets Office1 stood up now, at the cost of NOT being reproducible/versioned in OpenTofu -- log it explicitly as a known debt (a DOCFIX candidate: "Office1 service VM `` was hand-created via Option B on ``; migrate to `modules/cloudinit-vm` once its cloud-init content is designed"). --- ## Sequence ``` 1. Confirm Stage 1 exit gate (read-only, above) 2. Confirm OpenTofu still reaches vcloud libvirt from Office1 (read-only) 3. Read the provisioning-path decision + open questions (above; no cmds) 4. MAAS region controller -- create VM (A or B) + install [MUTATION, gated] 5. NetBox -- create VM (A or B) + install (official Docker) [MUTATION, gated] 6. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; literals pending -- PARTIAL) [MUTATION, gated] 7. GitBucket -- create VM (A or B) + install (WAR or Docker) [MUTATION, gated] 8. Tailscale -- install + scope to Office1 only [MUTATION, gated] 9. Post-standup verify against the Stage 2 gate (read-only) -> EXIT GATE (PARTIAL, see GATE section) -> Stage 3 ``` --- ## Step 1 -- Confirm Stage 1 exit gate (READ-ONLY) Covered above under "Entry condition." Do not proceed past this point until that CHECK is green. --- ## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt from Office1 (READ-ONLY) Stage 1 already proved OpenTofu -> vcloud-host libvirt reachability (its Step 1 CHECK + Step 10 apply). This step is a re-verification, not a re-derivation -- if this session is running some time after Stage 1, confirm nothing regressed rather than assuming it still holds. **CHECK -- from the Office1 operator VM/session** ```bash cd opentofu tofu version virsh -c "$(grep -A1 'variable "libvirt_uri"' -m1 /dev/null 2>/dev/null; true)" list --all 2>/dev/null || true ``` More directly -- use the exact `libvirt_uri` value recorded in Stage 1's `opentofu/dc-dc-phase0.auto.tfvars`: ```bash virsh -c "" list --all ``` Expect: the domain list (the DC1 plane/pool objects have no domains yet, so this may be empty, but the CONNECTION must succeed with no error). If it fails, STOP -- this is the same blocking condition Stage 1's Step 1 described; do not proceed into VM creation without confirmed libvirt reach. **CHECK -- MAAS provider not yet needed but flagged (Stage 1's Known Gap)** ```bash grep -n 'provider "maas"' opentofu/main.tf ``` Confirm whether Stage 1 worked around the unconditional `maas` provider block with placeholder values or whether a DOCFIX split it out -- read whatever Stage 1's changelog entry recorded, and use the same resolved answer here rather than re-deciding it. --- ## Step 3 -- Read the provisioning-path decision and open questions No commands. Confirm with the operator (or record your own reasoned choice, logged) which path -- A or B -- is being used for each of the three VMs below, and how Open question #1 (Office1-local network) is being resolved for this run. Do not proceed to Step 4 silently. --- ## Step 4 -- MAAS region controller: create VM + install [MUTATION: gated] **CHECK -- confirm no pre-existing MAAS-region VM under this name** ```bash virsh -c "" list --all | grep -i maas-region ``` Expect: no match (fresh standup) or a clear, understood match (re-run/repair case) -- do not proceed if an unexplained existing domain shares the name. **MUTATION -- create the VM (Option A or B per Step 3)** Option A: not available tonight (blocked, see above) -- instantiate `module "office1_maas_region"` using `modules/cloudinit-vm` once its `user_data`/`meta_data`/`network_config` are designed, then `tofu apply`. Option B (interim manual path): ```bash # Illustrative shape only -- fill every with a value you # measured/assigned this session; none of these are literals to copy verbatim. virt-install \ --connect "" \ --name maas-region-office1 \ --memory \ --vcpus \ --disk pool=office1,size= \ --network network= \ --os-variant \ --cdrom \ --graphics none --console pty,target_type=serial ``` Boot it, complete the OS install per that OS's own docs, and record the resulting static/DHCP address this session assigns it -- MEASURE/ASSIGN, never guessed. This is the exact "known debt" Option B commits to logging (see Provisioning-path decision above). **MUTATION -- install MAAS region controller (official snap-based method)** Once the VM is up and reachable over SSH: ```bash ssh 'snap info maas' ``` Consult MAAS's current official install docs for the exact channel to pin (this repo does not hardcode a MAAS snap channel anywhere else to copy from -- pick and record the current LTS/stable channel per those docs, not a guessed one) and run the equivalent of: ```bash sudo snap install maas --channel= ``` Then initialize it as a region controller per MAAS's own current `maas init` documentation for that installed version -- the exact flags (`--maas-url`, database backend selection, admin creation) vary by MAAS release; consult the official docs rather than reusing a flag set from memory or from a different MAAS version's docs. **CHECK -- verify the region controller is reachable** ```bash curl -sI "http://:5240/MAAS/" | head -1 ``` Expect an HTTP response, not a connection failure. Naming per D-106: this service's FQDN, once Designate/DNS exists for Office1, would follow the region-level subdomain scheme D-106 names as an example (`maas.office1.vr1...`) -- exact scheme is confirmed here as part of this stage, not assumed; record whatever real hostname convention you actually use. **GATE (Stage 2, first Gate bullet):** MAAS region reachable. --- ## Step 5 -- NetBox: create VM + install (official Docker method) [MUTATION: gated] **CHECK -- confirm no pre-existing NetBox VM under this name** ```bash virsh -c "" list --all | grep -i netbox ``` **MUTATION -- create the VM (Option A or B per Step 3)**, same shape and same caveats as Step 4's MAAS-region VM creation (Option A blocked pending `cloudinit-vm` content design for THIS specific VM; Option B manual, logged as debt). **MUTATION -- install NetBox (official netbox-docker Compose method)** Once the VM is up and reachable over SSH, follow NetBox's own official Docker-based install (the `netbox-community/netbox-docker` repo's documented Compose workflow: clone it, configure its `.env` / `docker-compose.override.yml` per its own current docs, `docker compose up -d`). Do not hand-roll a from-source install -- Docker Compose is NetBox's own recommended path and the lowest-delta one to keep reproducible. Consult NetBox's current official docs for the exact compose file / image tag to pin; do not fabricate a version tag. **CHECK -- verify NetBox is reachable** ```bash curl -sI "http://:8080/" | head -1 ``` Expect an HTTP response. Create/record a NetBox API token per NetBox's own docs (needed by Step 6) -- handle it per this repo's secrets discipline: never printed, never committed, passed as an environment variable only (`NETBOX_TOKEN`, matching `netbox/dc-dc-prefixes-import.py`'s own documented usage). **GATE (Stage 2 Gate, partial):** NetBox reachable and running. NOT yet "authoritative and populated" -- see Step 6. --- ## Step 6 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] `netbox/dc-dc-prefixes-import.py` (built 2026-07-09, closes tooling gap #3's MECHANISM) extends the v1 single-site import to VR1's per-DC, dual-stack model per D-101. It is idempotent and `--verify-only`-capable. **It does NOT, and cannot, close the DATA half of gap #3**: the real org ULA /48, the per-DC GUA carve, and DC2's v4 supernet do not exist yet as assigned values anywhere in this repo or session. Running it for DC1 only requires the two v6 literals (`ORG_ULA_48`, `DC_GUA_PREFIX`); running it for DC2 additionally requires `DC2_V4_SUPERNET` -- none of the three has a default, by the script's own design, and none is invented here. **CHECK -- dry run, no writes** ```bash NETBOX_URL="http://:8080" \ NETBOX_TOKEN="" \ ORG_ULA_48="" \ DC_GUA_PREFIX="" \ python3 netbox/dc-dc-prefixes-import.py --dc dc1 --verify-only ``` If `ORG_ULA_48` / `DC_GUA_PREFIX` are not yet assigned, this command CANNOT be run for real yet -- that is the honest state of this stage tonight, not a missing step in this runbook. Record that as the blocking condition rather than inventing placeholder-looking values to "get past" it (this script itself is designed to fail loud on exactly that pattern, per its own header). **MUTATION -- once the literals above are real (a later session/step, not necessarily tonight)** ```bash NETBOX_URL="..." NETBOX_TOKEN="..." ORG_ULA_48="..." DC_GUA_PREFIX="..." \ python3 netbox/dc-dc-prefixes-import.py --dc dc1 ``` DC1's v4 is hardcoded in the script itself (D-101: "DC1 equals the validated template," not an inferred value -- it is the explicit text of an ADOPTED decision). DC2 additionally needs `DC2_V4_SUPERNET`, not assigned yet either. **GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 5). "Authoritative and populated (planes, per-DC v4, ULA/GUA carve)" is only PARTIALLY achievable right now: the IMPORT MECHANISM exists and is runnable the moment the literals are assigned, but the literals themselves are not assigned this session, so the DATA half of this gate bullet is NOT met yet. Do not mark this gate green in the tracker until a real `ORG_ULA_48` / `DC_GUA_PREFIX` (and, for DC2, `DC2_V4_SUPERNET`) exist and this script has actually been run against this real NetBox instance. --- ## Step 7 -- GitBucket: create VM + install (official WAR or Docker method) [MUTATION: gated] **CHECK -- confirm no pre-existing GitBucket VM under this name** ```bash virsh -c "" list --all | grep -i gitbucket ``` **MUTATION -- create the VM (Option A or B per Step 3)**, same shape/caveats as Steps 4 and 5. **MUTATION -- install GitBucket (official method)** GitBucket ships as a single runnable WAR (`java -jar gitbucket.war`, its own documented standalone-Jetty mode) or as an official Docker image (`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one and record which. Consult GitBucket's current official docs/release page for the exact WAR filename/version or image tag to pin -- do not fabricate one. Open question #3 above (this instance's purpose/repo-path scheme, distinct from the existing `git.baldurkeep.com` v1 instance per D-014) must be resolved before deciding what this instance actually mirrors. **CHECK -- verify GitBucket is reachable** ```bash curl -sI "http://:8080/" | head -1 ``` Expect an HTTP response. **GATE (Stage 2 Gate bullet):** GitBucket serving. --- ## Step 8 -- Tailscale: install + scope to Office1 only [MUTATION: gated] Per D-107, Office1 is explicitly OUT of the core-service path (no node artifacts, no NTP served from Office1) -- Tailscale here is a narrowly-scoped OPERATOR front door to Office1's own services, not a general egress/access path for anything DC-side. Do not wire Tailscale into any DC-facing plane. **CHECK -- confirm Tailscale is not already installed/conflicting** ```bash ssh 'tailscale version 2>&1 || echo "not installed"' ``` **MUTATION -- install (official method)** ```bash ssh 'curl -fsSL https://tailscale.com/install.sh | sh' ``` Consult Tailscale's own current docs if this host's distro needs a different install path (the install script itself detects and branches on distro). **MUTATION -- bring up, scoped** ```bash ssh 'sudo tailscale up --advertise-tags=' ``` The actual ACL policy restricting reachability to Office1's services only (and NOT to any DC-side network) is configured in the operator's own Tailscale admin console/ACL file -- this is real policy design (Open question #4), not invented here. Do not advertise routes into any DC plane from this node. **CHECK -- verify scope** ```bash ssh 'tailscale status' ``` Confirm only the expected Office1 headend node(s) appear, and that no route advertisement reaches into DC1/DC2 plane CIDRs. **GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only. --- ## Step 9 -- Post-standup verify against the Stage 2 gate (READ-ONLY) ```bash curl -sI "http://:5240/MAAS/" | head -1 curl -sI "http://:8080/" | head -1 curl -sI "http://:/" | head -1 ssh 'tailscale status' virsh -c "" list --all ``` Cross-check each against the GATE section below before declaring this stage done or partially done. --- ## GATE (Stage 2 exit condition -- honest partial state) Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row: "MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only." - **MAAS region reachable:** achievable this stage (Step 4). - **OpenTofu reaches vcloud host libvirt:** already proven in Stage 1; Step 2 here only re-verifies it still holds. - **GitBucket serving:** achievable this stage (Step 7). - **Tailscale confirmed to Office1 only:** achievable this stage (Step 8), contingent on the operator's own ACL policy design (Open question #4). - **NetBox authoritative and populated:** ONLY PARTIALLY achievable right now. The instance can be stood up and reachable (Step 5), and the import MECHANISM exists and is runnable (Step 6) -- but the real address literals (org ULA /48, per-DC GUA carve, DC2 v4 supernet) are not assigned this session, so NetBox cannot yet be truly "populated" per D-101. Do not mark this bullet green in `docs/dc-dc-deployment-workflow.md`'s Stage 2 row until those literals exist and `netbox/dc-dc-prefixes-import.py` has actually run against this real instance. This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** -- MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in full tonight's sequence; the NetBox bullet closes its mechanism half only, pending the D-101 literal assignment that is explicitly out of scope for this stage to invent. Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever of these is actually true after a real run, not to "DONE" wholesale. -> Proceed to Stage 3 (OpenTofu builds each DC substrate) only once the achievable bullets above are actually green; the NetBox DATA gap does not block Stage 3's OWN gate (Stage 3 depends on MAAS reachability and OpenTofu reach, not on NetBox's DC2/GUA literals), but should be tracked as a standing gap until closed. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo changes made while executing this runbook (e.g. any real `opentofu/` instantiation once Option A becomes available; any real tfvars, minus secrets). - [ ] If any Option A `cloudinit-vm` instantiation was written this run, `bash scripts/opentofu-validate.sh` green. - [ ] Changelog entry for this runbook's first real execution (next DOCFIX number via `bash scripts/ledger-scan.sh`), noting: which provisioning path (A or B) was used for each of the three VMs, the real addresses/ hostnames assigned, and whether NetBox's literal-assignment gap was closed or remains open. - [ ] `docs/session-ledger.md` updated with the outcome. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table updated to the ACTUAL state reached (likely "PARTIAL," not "DONE," per the GATE section above) -- do not mark NetBox's gate bullet closed unless the D-101 literals were genuinely assigned and imported this run. - [ ] If Option B (manual VM creation) was used for any service VM, log the resulting debt explicitly (a DOCFIX candidate: migrate that VM to `opentofu/modules/cloudinit-vm` once its cloud-init content is designed) rather than letting it go unrecorded. - [ ] If Open question #1 (Office1-local network) was resolved this run, record the resolution and, if it implies a reusable OpenTofu module (an Office1-local `dc-planes`-shaped network), flag it as a follow-on delivery rather than silently leaving it as a one-off manual libvirt network.