diff --git a/docs/vr1-office1-as-built.md b/docs/vr1-office1-as-built.md new file mode 100644 index 0000000..800b726 --- /dev/null +++ b/docs/vr1-office1-as-built.md @@ -0,0 +1,113 @@ +# VR1 Office1 -- as-built inventory and connection reference + +**Purpose.** The single operational answer to "what is running, where, on what address, and how do I +reach it." Every value here was MEASURED on the live system, not planned or inferred. Update it in +the SAME change that brings a host or service online -- a stale connection table is worse than none. + +**Scope.** VR1 Office1 only (the site headend). DC1/DC2 get their own sections when they exist. + +**Authority note.** NetBox is the IPAM apex -- this table is an OPERATIONAL reference (how to reach +things, where credentials live), NOT the address authority. Where the two ever disagree, NetBox wins +and the divergence gets fixed here. The D-115 carve is not yet imported into NetBox (open item). + +Last measured: 2026-07-13. + +--- + +## 1. Networks + +| Network | CIDR | Where it lives | Gateway | DHCP authority | Role (D-115) | +|---|---|---|---|---|---| +| `office1-wan` | `172.30.1.0/24` | libvirt NAT net on vcloud (`virbr11`) | `172.30.1.1` (vcloud) | none (static) | **Edge** (`172.30.0.0/16`) -- simulated ISP uplink | +| `office1-local` | `10.10.0.0/24` | libvirt bridge on vcloud (`virbr2`), addressless | `10.10.0.1` (the edge) | **Kea on the OPNsense edge** (pool `.100-.199`) | **Office** -- VR1 Office1 `10.10.0.0/22` | +| `lxdbr0` | `10.10.1.0/24` | INSIDE `voffice1` (LXD-managed, NAT) | `10.10.1.1` (voffice1) | **MAAS** (dynamic `.100-.200`; deployed machines get `.201+`) | **Office** -- same `/22`, compose net | + +**The DHCP split is structural.** Two authorities, two physically separate L2s. `lxdbr0` is NEVER +bridged onto `office1-local`, so Kea and MAAS cannot see each other. LXD's own dnsmasq is disabled +(`ipv4.dhcp=false`, `dns.mode=none`, `raw.dnsmasq=port=0`) and binds **no** `:53` and **no** `:67`. + +**IPv6:** Office1's carve is `2602:f3e2:f01:100::/56` with office subnet `2602:f3e2:f01:100::/64` +(D-115). **NOT YET DEPLOYED** -- pinned as the next step. Note IPv6 does NOT egress the lab +(measured: v6 gateway reachable, v6 internet 100% loss), so v6 here will be LAN-local only. + +--- + +## 2. Hosts and services + +| Host | Address | What runs on it | Level | +|---|---|---|---| +| **vcloud** | `10.17.11.248` (lab), `10.10.0.10` on `virbr2`, `172.30.1.1` on `virbr11` | libvirt/KVM hypervisor; OpenTofu runs from here | L1 (itself a KVM guest) | +| **office1-opnsense** | LAN `10.10.0.1` / WAN `172.30.1.2` | OPNsense 26.1 edge router: routing, NAT, **Kea DHCP**, firewall | L2 | +| **voffice1** | `10.10.0.20` (Kea reservation) | **MAAS 3.7.2** region+rack + PostgreSQL 16.14; **LXD 5.21.5** (registered to MAAS as VM host `office1-lxd`) | L2 | +| **office1-netbox** | `10.10.1.201` | **NetBox 4.6.4** (netbox-docker; Django 6.0.6) on `:8000` | L3 (LXD VM) | +| **office1-tailscale** | `10.10.1.202` | **Tailscale 1.98.8**, subnet router advertising `10.10.0.0/22` | L3 (LXD VM) | +| **office1-gitbucket** | -- | **NOT BUILT YET** | -- | + +`office1-netbox` and `office1-tailscale` are **MAAS-composed LXD VMs** (D-114) -- MAAS enlisted, +commissioned, deployed and powers them. The `voffice1` LXD host is `office1-lxd` (vm-host id 1). + +--- + +## 3. How to reach each thing + +All access is via the Office1 service SSH key: **`~/vr1-office1-creds/office1_svc_ed25519`** on +vcloud (private half NEVER read; SEC-007 tracks its rotation). + +`office1-local` (`10.10.0.0/24`) and `lxdbr0` (`10.10.1.0/24`) are **host-local to vcloud** -- not +routable from a workstation. Everything below therefore tunnels through vcloud. + +| Target | From vcloud | From your workstation | +|---|---|---| +| **voffice1** (shell) | `ssh -i jessea123@10.10.0.20` | `ssh -J jessea123@10.17.11.248 jessea123@10.10.0.20` | +| **OPNsense GUI** | `https://10.10.0.1` | `ssh -L 8443:10.10.0.1:443 jessea123@10.17.11.248` -> `https://localhost:8443` | +| **OPNsense shell** | `ssh -i root@10.10.0.1` | via vcloud. **root's shell is `tcsh`** -- feed commands to `sh -s`. | +| **MAAS UI** | `http://10.10.0.20:5240/MAAS` | `ssh -L 5240:10.10.0.20:5240 jessea123@10.17.11.248` -> `http://localhost:5240/MAAS` | +| **NetBox UI/API** | `http://10.10.1.201:8000` (needs the route below) | `ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.201:8000 jessea123@10.10.0.20` -> `http://localhost:8000` | +| **office1-netbox / office1-tailscale** (shell) | `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201` | chain both jumps | + +**Routing to the compose net.** The OPNsense edge has a static route `10.10.1.0/24 -> 10.10.0.20` +(gateway object `OFFICE1_LXD_GW`, priority 255, NOT the default gateway). So any `office1-local` +client using the edge as its gateway reaches the compose net. **vcloud itself does NOT** -- its +default route is the lab, so it needs an explicit route (operator sudo): +`sudo ip route replace 10.10.1.0/24 via 10.10.0.20 dev virbr2` + +**Tailscale** (once authorized + the route approved in the admin console) makes `10.10.0.0/22` +reachable from the tailnet directly -- i.e. the whole Office1 carve, edge GUI and MAAS UI included, +with no tunnels. That is the intended "front door" (D-107: Office1 ONLY). + +--- + +## 4. Where the credentials live (NEVER commit; never print) + +| What | Location | Notes | +|---|---|---| +| Office1 service SSH key | `~/vr1-office1-creds/office1_svc_ed25519{,.pub}` (vcloud) | SEC-007 -- rotation obligation | +| OPNsense API key/secret | `~/vr1-office1-creds/opnsense-api.txt` (vcloud) | used by `scripts/opnsense-api.sh` | +| OPNsense root password | `~/vr1-office1-creds/opnsense-root-password` (vcloud) | GUI login | +| MAAS admin password + API key; DB password; LXD trust password | `/root/maas-secrets/` on **voffice1** (`0600`) | generated on the box | +| NetBox SECRET_KEY, admin password, API token | `/root/netbox-secrets/` on **office1-netbox** (`0600`) | SECRET_KEY rotated off netbox-docker's PUBLIC default | +| NetBox (upstream, production) token | `~/.vr1-netbox.env` (vcloud) | SEC-006 -- burned, revoke at deployment completion | + +**NetBox API token format (4.6):** the wire form is `nbt_.`. The API's `token` field +alone is NOT usable -- present it raw and you get `403 Invalid v1 token`. See +`docs/changelog-20260713-office1-netbox-deployed.md`. + +--- + +## 5. Standing traps for this site (the short list) + +1. **DHCP on the edge is API-MANAGED.** A rendered `config.xml` push would CLOBBER it. That path is + DELETED (D-113(a2)); configure via `scripts/opnsense-api.sh`. +2. **OPNsense gateways: `defaultgw` in the API output is DERIVED FROM `priority`** (lower = more + preferred), not from what you send. Adding a LAN gateway with a lower priority than `WAN_GW` + SILENTLY STEALS THE DEFAULT ROUTE and kills site egress -- this happened on 2026-07-13. Always + add the gateway, READ THE CONFIG BACK, confirm `WAN_GW` is still default, and only THEN + `reconfigure`. `WAN_GW` is now priority 1; `OFFICE1_LXD_GW` is 255. +3. **root's shell on the edge is `tcsh`** -- `$(...)` and `2>&1` fail (quietly, or with "Ambiguous + output redirect"). Feed remote commands to `sh -s`. +4. **`lxc` reads config from stdin** -- in a piped script every `lxc` call needs `</dev/null` or it + eats the rest of the script. +5. **MAAS's bind9 intermittently SERVFAILs a first-time external lookup** and succeeds on retry + (seen twice: docker registry, tailscale.com). Retry before diagnosing. +6. **A `tofu apply` touching a `libvirt_domain`'s devices BOUNCES THE GUEST**, even when the plan + says "updated in-place". Schedule it as an outage.