Newer
Older
openstack-caracal-dc-dc / docs / vr1-office1-as-built.md

VR1 Office1 -- as-built inventory and connection reference

Purpose. The single operational answer to "what is running, where, on what address, and how do I reach it." Every value here was MEASURED on the live system, not planned or inferred. Update it in the SAME change that brings a host or service online -- a stale connection table is worse than none.

Scope. VR1 Office1 only (the site headend). DC1/DC2 get their own sections when they exist.

Authority note. NetBox is the IPAM apex -- this table is an OPERATIONAL reference (how to reach things, where credentials live), NOT the address authority. Where the two ever disagree, NetBox wins and the divergence gets fixed here. The D-115 carve is not yet imported into NetBox (open item).

Last measured: 2026-07-13.


1. Networks

Network CIDR Where it lives Gateway DHCP authority Role (D-115)
office1-wan 172.30.1.0/24 libvirt NAT net on vcloud (virbr11) 172.30.1.1 (vcloud) none (static) Edge (172.30.0.0/16) -- simulated ISP uplink
office1-local 10.10.0.0/24 libvirt bridge on vcloud (virbr2), addressless 10.10.0.1 (the edge) Kea on the OPNsense edge (pool .100-.199) Office -- VR1 Office1 10.10.0.0/22
lxdbr0 10.10.1.0/24 INSIDE voffice1 (LXD-managed, NAT) 10.10.1.1 (voffice1) MAAS (dynamic .100-.200; deployed machines get .201+) Office -- same /22, compose net

The DHCP split is structural. Two authorities, two physically separate L2s. lxdbr0 is NEVER bridged onto office1-local, so Kea and MAAS cannot see each other. LXD's own dnsmasq is disabled (ipv4.dhcp=false, dns.mode=none, raw.dnsmasq=port=0) and binds no :53 and no :67.

IPv6: Office1's carve is 2602:f3e2:f01:100::/56 with office subnet 2602:f3e2:f01:100::/64 (D-115). NOT YET DEPLOYED -- pinned as the next step. Note IPv6 does NOT egress the lab (measured: v6 gateway reachable, v6 internet 100% loss), so v6 here will be LAN-local only.


2. Hosts and services

Host Address What runs on it Level
vcloud 10.17.11.248 (lab), 10.10.0.10 on virbr2, 172.30.1.1 on virbr11 libvirt/KVM hypervisor; OpenTofu runs from here L1 (itself a KVM guest)
office1-opnsense LAN 10.10.0.1 / WAN 172.30.1.2 OPNsense 26.1 edge router: routing, NAT, Kea DHCP, firewall L2
voffice1 10.10.0.20 (Kea reservation) MAAS 3.7.2 region+rack + PostgreSQL 16.14; LXD 5.21.5 (registered to MAAS as VM host office1-lxd) L2
office1-netbox 10.10.1.201 NetBox 4.6.4 (netbox-docker; Django 6.0.6) on :8000 L3 (LXD VM)
office1-tailscale 10.10.1.202 / tailnet 100.64.0.53 (fd7a:115c:a1e0::35) Tailscale 1.98.8, subnet router advertising 10.10.0.0/22. Joined the SELF-HOSTED control plane https://tailscale.baldurkeep.com:443 L3 (LXD VM)
office1-gitbucket -- NOT BUILT YET --

office1-netbox and office1-tailscale are MAAS-composed LXD VMs (D-114) -- MAAS enlisted, commissioned, deployed and powers them. The voffice1 LXD host is office1-lxd (vm-host id 1).


3. How to reach each thing

All access is via the Office1 service SSH key: ~/vr1-office1-creds/office1_svc_ed25519 on vcloud (private half NEVER read; SEC-007 tracks its rotation).

office1-local (10.10.0.0/24) and lxdbr0 (10.10.1.0/24) are host-local to vcloud -- not routable from a workstation. Everything below therefore tunnels through vcloud.

Target From vcloud From your workstation
voffice1 (shell) ssh -i <key> jessea123@10.10.0.20 ssh -J jessea123@10.17.11.248 jessea123@10.10.0.20
OPNsense GUI https://10.10.0.1 ssh -L 8443:10.10.0.1:443 jessea123@10.17.11.248 -> https://localhost:8443
OPNsense shell ssh -i <key> root@10.10.0.1 via vcloud. root's shell is tcsh -- feed commands to sh -s.
MAAS UI http://10.10.0.20:5240/MAAS ssh -L 5240:10.10.0.20:5240 jessea123@10.17.11.248 -> http://localhost:5240/MAAS
NetBox UI/API http://10.10.1.201:8000 (needs the route below) ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.201:8000 jessea123@10.10.0.20 -> http://localhost:8000
office1-netbox / office1-tailscale (shell) ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201 chain both jumps

Routing to the compose net. The OPNsense edge has a static route 10.10.1.0/24 -> 10.10.0.20 (gateway object OFFICE1_LXD_GW, priority 255, NOT the default gateway). So any office1-local client using the edge as its gateway reaches the compose net. vcloud itself does NOT -- its default route is the lab, so it needs an explicit route (operator sudo): sudo ip route replace 10.10.1.0/24 via 10.10.0.20 dev virbr2

Tailscale front door (D-107: Office1 ONLY). office1-tailscale is a subnet router on the operator's SELF-HOSTED control plane (https://tailscale.baldurkeep.com:443 -- NOT Tailscale's public one). It is UP and online as 100.64.0.53, advertising 10.10.0.0/22 -- exactly the D-115 Office1 carve, so "Office1 only" is enforced by the CARVE, not by policy alone.

  • Brought up with: --login-server=https://tailscale.baldurkeep.com:443 --authkey=<file> --advertise-routes=10.10.0.0/22 --accept-routes --accept-dns=false --hostname=office1-tailscale
  • --accept-dns=false is DELIBERATE (a deviation from the operator's usual command): this is a MAAS-managed machine and MAAS's bind9 serves its internal names. Letting the tailnet take over DNS would break that resolution. Flip it if MagicDNS is wanted here.
  • OPERATOR ACTION STILL REQUIRED: the 10.10.0.0/22 route is ADVERTISED but must be ENABLED on the control server (Headscale: headscale routes enable) before other tailnet nodes route to Office1. Until then the node is reachable but the subnet is not.
  • The auth key was consumed by path from a 0600 file and shredded from the node after use -- tailscaled holds its own node key now. It never entered an operator session or the repo.

Once the route is enabled, the whole Office1 carve -- edge GUI, MAAS UI, NetBox -- is reachable from the tailnet with no SSH tunnels at all, which supersedes most of the tunnel commands above.


4. Where the credentials live (NEVER commit; never print)

What Location Notes
Office1 service SSH key ~/vr1-office1-creds/office1_svc_ed25519{,.pub} (vcloud) SEC-007 -- rotation obligation
OPNsense API key/secret ~/vr1-office1-creds/opnsense-api.txt (vcloud) used by scripts/opnsense-api.sh
OPNsense root password ~/vr1-office1-creds/opnsense-root-password (vcloud) GUI login
MAAS admin password + API key; DB password; LXD trust password /root/maas-secrets/ on voffice1 (0600) generated on the box
NetBox SECRET_KEY, admin password, API token /root/netbox-secrets/ on office1-netbox (0600) SECRET_KEY rotated off netbox-docker's PUBLIC default
NetBox (upstream, production) token ~/.vr1-netbox.env (vcloud) SEC-006 -- burned, revoke at deployment completion
Tailscale auth key (self-hosted tailnet) ~/vr1-office1-creds/tailscale-authkey.txt (vcloud, 0600) Operator-supplied, 48 chars. Consumed by PATH, never printed. The copy on the node was SHREDDED after tailscale up. Rotate/revoke on the control server at v1 close.

NetBox API token format (4.6): the wire form is nbt_<key>.<plaintext>. The API's token field alone is NOT usable -- present it raw and you get 403 Invalid v1 token. See docs/changelog-20260713-office1-netbox-deployed.md.


5. Standing traps for this site (the short list)

  1. DHCP on the edge is API-MANAGED. A rendered config.xml push would CLOBBER it. That path is DELETED (D-113(a2)); configure via scripts/opnsense-api.sh.
  2. OPNsense gateways: defaultgw in the API output is DERIVED FROM priority (lower = more preferred), not from what you send. Adding a LAN gateway with a lower priority than WAN_GW SILENTLY STEALS THE DEFAULT ROUTE and kills site egress -- this happened on 2026-07-13. Always add the gateway, READ THE CONFIG BACK, confirm WAN_GW is still default, and only THEN reconfigure. WAN_GW is now priority 1; OFFICE1_LXD_GW is 255.
  3. root's shell on the edge is tcsh -- $(...) and 2>&1 fail (quietly, or with "Ambiguous output redirect"). Feed remote commands to sh -s.
  4. lxc reads config from stdin -- in a piped script every lxc call needs </dev/null or it eats the rest of the script.
  5. MAAS's bind9 intermittently SERVFAILs a first-time external lookup and succeeds on retry (seen twice: docker registry, tailscale.com). Retry before diagnosing.
  6. A tofu apply touching a libvirt_domain's devices BOUNCES THE GUEST, even when the plan says "updated in-place". Schedule it as an outage.