diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 1d63e0c..277afd2 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -63,7 +63,7 @@ > vcloud host with Tailscale on the headend host. It builds **ONE Office1 site containment VM, > `voffice1`** (Ubuntu 24.04, OpenTofu-created), which IS the facility: **MAAS-region runs ON it, > LXD (pinned to the 5.21 LTS track) runs ON it**, that LXD is **registered BACK INTO MAAS as an -> LXD KVM host**, and the non-stack service machines (**NetBox, GitBucket, Tailscale**) are VMs +> LXD KVM host**, and the non-stack service machines (**NetBox, Tailscale** -- GitBucket removed per D-116) are VMs > **COMPOSED BY MAAS** into it -- MAAS-visible: enlisted, commissioned, deployed, powered. This > is VR0's own already-proven `lxd` + `tailscale` pattern, applied per site. > @@ -80,8 +80,8 @@ | | | |---|---| | **Goal** | Stand up the Office1 SITE first, before any DC -- and prove the D-114 containment-VM model at its cheapest point. | -| **Build** | `voffice1` site containment VM (OpenTofu, `modules/cloudinit-vm` + `modules/base-image`, on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`); MAAS region controller ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); that LXD registered into MAAS as an LXD KVM host; NetBox / GitBucket / Tailscale as MAAS-COMPOSED LXD VMs inside it; Office1's own OPNsense simulated-ISP edge (gap #16, Stage 2 owns it). | -| **Gate** | Office1 OPNsense edge up (**MET** -- routing, NAT, egress, Kea DHCP; gap #17 closed for Office1 via `office1-wan`); `office1-local` network created (gap #12 CLOSED); OpenTofu reaches vcloud host libvirt (**MET**); **`voffice1` exists and runs (MET 2026-07-13, measured)**; `voffice1` has a reserved address + SSH + nested KVM exposed; MAAS region reachable ON `voffice1`; LXD on the 5.21 track with `lxdbr0` OFF `office1-local`; LXD registered as a MAAS KVM host; **the non-stack machines COMPOSED BY MAAS and deployed -- i.e. an LXD VM actually BOOTS at L3, which is D-114's DC1 entry gate**; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; Tailscale confirmed to Office1 only. | +| **Build** | `voffice1` site containment VM (OpenTofu, `modules/cloudinit-vm` + `modules/base-image`, on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`); MAAS region controller ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); that LXD registered into MAAS as an LXD KVM host; NetBox + Tailscale as MAAS-COMPOSED LXD VMs inside it (**GitBucket REMOVED from scope -- D-116**); Office1's own OPNsense simulated-ISP edge (gap #16, Stage 2 owns it). | +| **Gate** | Office1 OPNsense edge up (**MET** -- routing, NAT, egress, Kea DHCP; gap #17 closed for Office1 via `office1-wan`); `office1-local` network created (gap #12 CLOSED); OpenTofu reaches vcloud host libvirt (**MET**); **`voffice1` exists and runs (MET 2026-07-13, measured)**; `voffice1` has a reserved address + SSH + nested KVM exposed; MAAS region reachable ON `voffice1`; LXD on the 5.21 track with `lxdbr0` OFF `office1-local`; LXD registered as a MAAS KVM host; **the non-stack machines COMPOSED BY MAAS and deployed -- i.e. an LXD VM actually BOOTS at L3, which is D-114's DC1 entry gate**; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); Tailscale confirmed to Office1 only. **GitBucket is NO LONGER A GATE (D-116): no Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record.** | | **Owns** | **D-114** (site containment VM + MAAS-composed LXD VMs, incl. the LXD 5.21 pin), D-103 as AMENDED by D-114 (OpenTofu owns the containment VM and the site networks; MAAS composes INSIDE the site), D-107 (headend is not a core-service provider). | | **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but LESS new than it looks: the MAAS-composed-LXD-VM half is **exactly what VR0 runs today** (its `lxd` KVM host at LXD 5.21.4, with `tailscale` composed into it at 2 cores / 2 GiB / 25 GB) and carries **NO Roosevelt delta**. The containment VM itself IS a deliberate simulation-only delta -- it is what makes a "hard-down an entire facility" DR drill (`virsh destroy vdc1`) executable at all. NetBox-as-IPAM-apex discipline and the `netbox/*-import.py` pattern transfer unchanged. | | **Authoring status** | **Runbook REWRITTEN 2026-07-13 to the D-114 model: `runbooks/dc-dc-phase1-office1-standup.md`.** The old Option A / Option B provisioning fork is **CLOSED** -- OpenTofu (Option A) is the path and `module "voffice1"` is a real, applied `cloudinit-vm` instantiation (real image source, real `user_data`/`meta_data`/`network_config`, plus the new `expose_nested_virt` variable). **PENDING VERIFICATION:** the MAAS/LXD install + LXD-KVM-host registration + VM-compose command lines are NOT yet confirmed against Canonical's current docs -- the runbook marks each unconfirmed flag/channel/subcommand rather than fabricating one, and a separate verification pass owns filling them in. | @@ -95,7 +95,7 @@ `opentofu/terraform.tfstate`) -- the D-114 containment VM is applied. - **NOT DONE -- this is what remains of Stage 2:** MAAS-region ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); the LXD-KVM-host registration back into MAAS; and **every service machine** - (NetBox, GitBucket, Tailscale), which are now MAAS-COMPOSED LXD VMs, not standalone libvirt + (NetBox, Tailscale -- GitBucket dropped, D-116), which are now MAAS-COMPOSED LXD VMs, not standalone libvirt domains. The NetBox/GitBucket that answer at `baldurkeep.com` are PRE-EXISTING services, NOT Office1 machines -- do not mistake them for a met gate. Likewise, Tailscale reaching the jumphost via a pre-existing subnet route is a DIFFERENT node and does not meet Stage 2's diff --git a/docs/design-decisions.md b/docs/design-decisions.md index e0620cf..06efd89 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2035,6 +2035,7 @@ - Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 `create-backup` / `download-backup`). Controller backups scheduled and a restore drill run in Phase 5. - HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation. - Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only). + **[AMENDED by D-116, 2026-07-13: the GitBucket half is SUPERSEDED. No Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record. Office1 = MAAS-region + OpenTofu + NetBox + Tailscale.]** **Rationale / tradeoffs:** per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC. @@ -2743,3 +2744,41 @@ NetBox as the place changes are simulated first. The Office1 NetBox VM is composed and `Ready` but not yet deployed -- so the natural sequence is: deploy it, import this carve there, verify, then feed back to the real NetBox. + +## D-116: no Office1-local GitBucket -- keep using the existing git.baldurkeep.com + +**Status:** **ADOPTED 2026-07-13 (operator ruling).** SUPERSEDES the GitBucket half of D-103 and +removes GitBucket from Stage 2's build list. + +**Operator ruling, verbatim in substance:** "For the gitbucket, we've decided not to build a +separate one on the test cloud. We will continue to use the current setup and create new repos for +deployment tasks as needed." + +**What changes.** D-103 assigned Office1 three service VMs -- MAAS-region, NetBox, GitBucket -- and +D-114 recast them as MAAS-composed LXD VMs. **GitBucket is now OUT OF SCOPE entirely**: no third +service machine is composed, and the existing `git.baldurkeep.com` remains the git service of record +for VR1. New repos are created there as deployment tasks require. + +**What this closes.** The Stage 2 runbook carried an explicit OPEN QUESTION about the Office1-local +GitBucket -- "what it mirrors, and under what repo path, is not decided" -- because a *second*, +Office1-local GitBucket alongside the pre-existing `git.baldurkeep.com` raised a real +mirror/authority question nobody had answered. **That question is now MOOT.** There is one git +service, it is the existing one, and it is authoritative. This is a genuine simplification, not a +deferral. + +**Rationale.** The Office1-local GitBucket only ever existed to rehearse a per-site git mirror. The +operator judges the rehearsal not worth a machine: the real deployment work is served by the +existing instance, and a second one buys a mirroring problem (which repo is authoritative, what +syncs, in which direction) in exchange for no capability the deployment actually needs. + +**Delta to Roosevelt.** Roosevelt may still want a per-site git mirror for node-airgap reasons +(cf. D-107's per-DC artifact mirror, which is a SEPARATE concern and is NOT affected by this +ruling -- D-107 is about OS/package artifacts for node provisioning, not about hosting git repos). +If Roosevelt later needs a site-local git service, it is a fresh decision made against real +requirements, not a rehearsal artifact carried forward. Recording that explicitly so nobody reads +this ruling as "Roosevelt will never have a git mirror". + +**Consequences (all applied in the same change):** GitBucket is removed from Stage 2's Build/Gate in +`docs/dc-dc-deployment-workflow.md`, from the Office1 standup runbook's sequence and its open +questions, and from `docs/vr1-office1-as-built.md`. Office1's composed service machines are +therefore exactly TWO: `office1-netbox` and `office1-tailscale`, both LIVE. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 7c68846..f70d48b 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -80,8 +80,17 @@ **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. - **REMAINING on Office1:** deploy + install NetBox (composed, Ready, not yet deployed); compose + - install GitBucket and Tailscale; then run the NetBox import. + **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.201) runs **NetBox + 4.6.4**; `office1-tailscale` (10.10.1.202, tailnet 100.64.0.53) is a subnet router advertising + `10.10.0.0/22` on the operator's **SELF-HOSTED** control plane. The edge routes `10.10.1.0/24 -> + 10.10.0.20`. Full connection reference: **`docs/vr1-office1-as-built.md`** -- read that first for + any "how do I reach X" question. + **GitBucket: OUT OF SCOPE (D-116, operator ruling).** No Office1-local GitBucket is built; + `git.baldurkeep.com` stays the git service of record. Office1's composed machines are exactly TWO. + **REMAINING on Office1:** (a) OPERATOR: enable the `10.10.0.0/22` route on the Tailscale control + server (`headscale routes enable`) -- it is advertised but not yet enabled, so the subnet is not + reachable from the tailnet yet; (b) import the D-115 carve into the Office1 NetBox; (c) the + pinned Office1 LAN IPv6 (item 1a). 1a. **PINNED -- NEXT AFTER NETBOX: deploy Office1's LAN IPv6** (operator ruling 2026-07-13, D-115 question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). Add `2602:f3e2:f01:100::1/64` to the OPNsense edge LAN + RA, and let `voffice1` take its address. diff --git a/docs/vr1-office1-as-built.md b/docs/vr1-office1-as-built.md index 605bc9b..e708b34 100644 --- a/docs/vr1-office1-as-built.md +++ b/docs/vr1-office1-as-built.md @@ -41,7 +41,7 @@ | **voffice1** | `10.10.0.20` (Kea reservation) | **MAAS 3.7.2** region+rack + PostgreSQL 16.14; **LXD 5.21.5** (registered to MAAS as VM host `office1-lxd`) | L2 | | **office1-netbox** | `10.10.1.201` | **NetBox 4.6.4** (netbox-docker; Django 6.0.6) on `:8000` | L3 (LXD VM) | | **office1-tailscale** | `10.10.1.202` / tailnet **`100.64.0.53`** (`fd7a:115c:a1e0::35`) | **Tailscale 1.98.8**, subnet router advertising `10.10.0.0/22`. Joined the **SELF-HOSTED** control plane `https://tailscale.baldurkeep.com:443` | L3 (LXD VM) | -| **office1-gitbucket** | -- | **NOT BUILT YET** | -- | +| ~~office1-gitbucket~~ | -- | **NOT BUILT -- AND WILL NOT BE (D-116).** `git.baldurkeep.com` remains the git service of record; new repos are created there as deployment tasks require. | -- | `office1-netbox` and `office1-tailscale` are **MAAS-composed LXD VMs** (D-114) -- MAAS enlisted, commissioned, deployed and powers them. The `voffice1` LXD host is `office1-lxd` (vm-host id 1). diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index 196b983..9f1b0ad 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -3,6 +3,7 @@ > **REWRITTEN 2026-07-13 to the D-114 model.** The previous version of this > runbook built THREE SIBLING SERVICE VMs on the vcloud host (MAAS-region, > NetBox, GitBucket), each behind an "Option A (OpenTofu, blocked) / Option B +> [GitBucket is since REMOVED from scope entirely -- D-116.] > (manual virt-install)" fork, with Tailscale on the headend host. **That model > is SUPERSEDED.** D-114 (ADOPTED 2026-07-13) replaces it with ONE Office1 site > containment VM (`voffice1`) that IS the facility, with MAAS + LXD running ON @@ -11,7 +12,7 @@ Stand up the Office1 SITE: the containment VM that simulates the facility, the MAAS region controller that runs inside it, the LXD VM host MAAS composes into, -and the non-stack service machines (NetBox, GitBucket, Tailscale) as +and the non-stack service machines (NetBox, Tailscale -- GitBucket dropped, D-116) as MAAS-visible VMs -- BEFORE either DC substrate exists. This is the second DC-DC runbook; it follows `runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit gate is this runbook's entry condition. Per this repo's session contract, do NOT @@ -114,7 +115,7 @@ a real server behind a real site router. `voffice1` IS the facility. The MAAS REGION CONTROLLER runs ON it. LXD (5.21 LTS) also runs ON it, and that LXD is REGISTERED BACK INTO MAAS as an LXD KVM host. The non-stack service machines -- -NetBox, GitBucket, Tailscale -- are then VMs **composed by MAAS** into that LXD +NetBox and Tailscale -- are then VMs **composed by MAAS** into that LXD host: MAAS-visible, enlisted, commissioned, deployed, powered, releasable. This is not a new pattern; it is VR0's own already-proven `lxd` + `tailscale` pair, applied per site. Explicitly NOT in scope: the Juju-deployed OpenStack services @@ -123,7 +124,7 @@ ruling: **Office1 FIRST, fully up. DC1 is GATED behind it. DC2 follows DC1.** **`expose_nested_virt = true` on `voffice1` is LOAD-BEARING, not a nicety.** LXD -*virtual machines* are qemu/KVM guests, so composing NetBox/GitBucket/Tailscale +*virtual machines* are qemu/KVM guests, so composing NetBox/Tailscale as LXD VMs inside `voffice1` requires nested KVM at L3. (Only LXD *containers* would be free of that requirement -- and MAAS does not enlist containers.) Office1 is therefore the CHEAP nesting probe for the whole D-114 model: no @@ -163,15 +164,17 @@ NetBox up and can run its import MECHANISM (Step 10), but the underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not exist yet -- see Step 10's own gate. -4. **GitBucket's VR1 repo path/name is not decided.** - The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a - DIFFERENT, pre-existing - `git.baldurkeep.com` instance from the v1 rehearsal -- this stage stands up a - SEPARATE, Office1-local GitBucket -- what it mirrors, and under what repo - path, is not decided here. Flag as an open item before Step 11's mutation. +4. **GitBucket: CLOSED, NOT DEFERRED (D-116).** There is NO Office1-local GitBucket. The operator + ruled 2026-07-13 that the existing `git.baldurkeep.com` remains the git service of record, with + new repos created there as deployment tasks require. This also makes the old open question -- + what a second, Office1-local GitBucket would mirror and under what repo path -- MOOT. One git + service, and it is authoritative. (D-107's per-DC ARTIFACT mirror is a separate concern and is + NOT affected.) + + 5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the front door is "Office1 only" -- the actual tailnet ACL policy, tags, and which - services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the service + services (NetBox web UI? MAAS UI? SSH to the service machines?) are exposed through it versus reachable only from Office1's own local network, is real policy design left to Step 12 and the operator's own tailnet account -- not invented here. NOTE the placement change under D-114: @@ -179,7 +182,7 @@ exactly like VR0's own `tailscale` machine), NOT a package installed on the headend host. A pre-existing subnet route already reaches the jumphost; that is a different node and does not satisfy this gate. -6. **Exact MAAS / NetBox / GitBucket / LXD version pins are not decided** (beyond +6. **Exact MAAS / NetBox / LXD version pins are not decided** (beyond D-114's hard LXD 5.21-LTS-track constraint, which IS decided). Each install step below names the well-known official install MECHANISM but does not pin an exact version/channel/tag -- consult each project's current official install @@ -213,7 +216,6 @@ 9. NetBox -- install on its composed machine [MUTATION, gated] 10. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; literals pending -- PARTIAL) [MUTATION, gated] -11. GitBucket -- install on its composed machine [MUTATION, gated] 12. Tailscale -- install on its composed machine, scoped [MUTATION, gated] 13. Post-standup verify against the Stage 2 gate (read-only) -> EXIT GATE -> Stage 3 (DC1), which D-114 GATES behind this @@ -553,7 +555,7 @@ --- -## Step 8 -- Compose the non-stack service machines (NetBox, GitBucket, Tailscale) [MUTATION: gated] (PENDING VERIFICATION) +## Step 8 -- Compose the non-stack service machines (NetBox, Tailscale) [MUTATION: gated] > **PENDING VERIFICATION.** The exact `maas ... vm-host compose` parameters > (cores, memory, storage, and how the composed machine is attached to the right @@ -573,7 +575,7 @@ Compose in this order, gating each: **Tailscale first** (it is the smallest, it is VR0's proven exemplar, and if it boots the model is proven at minimum cost), -then NetBox, then GitBucket. +then NetBox, then Tailscale. ```bash ssh 'maas vm-host compose ' @@ -685,28 +687,23 @@ --- -## Step 11 -- GitBucket: install on its composed machine [MUTATION: gated] +## Step 11 -- GitBucket: REMOVED FROM SCOPE (D-116) -**MUTATION -- install GitBucket (official method)** +**DO NOT BUILD AN OFFICE1-LOCAL GITBUCKET.** Operator ruling 2026-07-13 (D-116): the existing +`git.baldurkeep.com` remains the git service of record for VR1, and new repos are created there as +deployment tasks require. No third service machine is composed. -GitBucket ships as a single runnable WAR (`java -jar gitbucket.war`, its own -documented standalone-Jetty mode) or as an official Docker image -(`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one and -record which. Consult GitBucket's current official docs/release page for the -exact WAR filename/version or image tag to pin -- do not fabricate one. Open -question #4 above (this instance's purpose/repo-path scheme, distinct from the -existing `git.baldurkeep.com` v1 instance per D-014) must be resolved before -deciding what this instance actually mirrors. +This step previously walked an Office1-local GitBucket install. It is retained as a tombstone rather +than deleted because the runbook's step numbering is referenced elsewhere, and because someone WILL +come looking for it after reading D-103 (whose GitBucket half this supersedes). -**CHECK -- verify GitBucket is reachable** -```bash -curl -sI "http://:/" | head -1 -``` -Expect an HTTP response. +The old open question this closes -- what a second, Office1-local GitBucket would mirror, and under +what repo path, given `git.baldurkeep.com` already exists -- is MOOT, not deferred. -**GATE (Stage 2 Gate bullet):** GitBucket serving. +NOTE: D-107's per-DC ARTIFACT mirror (OS/package artifacts for node provisioning) is a SEPARATE +concern and is NOT affected by this ruling. ---- +Proceed directly to Step 12. ## Step 12 -- Tailscale: install on its composed machine, scoped to Office1 [MUTATION: gated]