Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / runbooks / dc-dc-office1-service-reip.md
@JANeumatrix JANeumatrix 19 hours ago 6 KB D-120 ADOPTED + EXECUTED: re-IP Office1 services into the .2-.49 static band
Raw Blame History

Office1 service re-IP -- move NetBox + Tailscale into the D-120 static band

EXECUTED 2026-07-15 (D-120 ADOPTED, option (a)). Both services re-IP'd, no wipe, verified on the wire. The PENDING-VERIFICATION markers below are now RESOLVED with the measured method; this is the reference procedure the DCs reuse for their own compose /24s. It remains a gated mutation on running services -- present each step, justify it as minimal, get individual approval -- if ever re-run.

Target (EXECUTED)

office1-netbox     10.10.1.201 -> 10.10.1.10   (static, band .2-.49)
office1-tailscale  10.10.1.202 -> 10.10.1.11   (static, band .2-.49)

Both VMs are MAAS-COMPOSED LXD VMs on voffice1; their addresses come from MAAS. The edge's static route 10.10.1.0/24 -> 10.10.0.20 is /24-wide and is UNAFFECTED by the change.

Preconditions (READ-ONLY)

  1. D-120 ADOPTED; target addresses confirmed.
  2. Coordinate with sandbox/C2 work -- the NetBox re-IP briefly interrupts the sandbox NetBox.
  3. Record current state: office1-netbox 10.10.1.201, office1-tailscale 10.10.1.202 (as-built).
  4. Access: ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201 (and .202); MAAS on voffice1 (10.10.0.20), secrets in /root/maas-secrets/ on the box (operator-held).

Step 0 -- Confirm the recovery backstop FIRST (READ-ONLY, blocking)

Before severing any network path, prove lxc exec/console reaches each guest independent of guest networking -- it rides the LXD socket, so it survives the re-IP and is the recovery path if the new address does not come up. On voffice1:

sudo -n lxc list                                   # instance names (office1-netbox / -tailscale)
sudo -n lxc exec office1-netbox -- hostname        # must return -- proves the backstop

Step 1 -- MAAS: NOT AVAILABLE on a Deployed machine (finding)

MAAS refuses interface edits on a Deployed machine: interface unlink-subnet ... returns "Cannot unlink subnet interface because the machine is not New, Ready, Allocated, or Broken." The only way to make it Ready is Release, which WIPES the OS -- forbidden (it destroys the seeded sandbox). So the MAAS-model update is skipped and MAAS-model drift is ACCEPTED (MAAS keeps showing .201/.202 mode=auto). Drift reconciles for free at the next teardown/redeploy. .10/.11 are off MAAS's auto-assign path: MAAS allocates nodes from .201+ upward (observed -- the two services themselves landed at .201/.202), so the low .2-.49 band is never auto-handed out. (NB: the whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee -- the allocation DIRECTION is.)

DC replication: a DC building fresh should static-assign .2-.49 in MAAS BEFORE deploy (a Ready machine DOES accept interface link-subnet <sid> <iface> subnet=<id> mode=STATIC ip_address=<addr>), so there is no drift. The Deployed-block only bit Office1 because its services were already deployed in the node band.

Step 2 -- Set the address on the GUEST (both files, via lxc exec) [MUTATION: gated]

Drive this over lxc exec from voffice1, NOT SSH-to-the-guest-IP: the LXD socket is immune to the netplan apply that swaps the address, so your control channel never drops. Edit BOTH files (netplan takes effect now; the curtin cfg makes it survive a reboot / cloud-init re-render), then apply:

for f in /etc/netplan/50-cloud-init.yaml /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg; do
  sudo -n lxc exec office1-netbox -- cp "$f" "$f.bak-reip201"
  sudo -n lxc exec office1-netbox -- sed -i 's#10.10.1.201/24#10.10.1.10/24#' "$f"
done
sudo -n lxc exec office1-netbox -- netplan apply     # gateway4-deprecated warnings are benign
sudo -n lxc exec office1-netbox -- ip -4 addr show enp5s0 | grep inet   # verify .10, do not assume

(Same for office1-tailscale: .202/24 -> .11/24, backups .bak-reip202.)

Step 3 -- Each SERVICE's own address config [MEASURED]

  • NetBox (office1-netbox, netbox-docker): binds 0.0.0.0:8000, follows the host IP. ALLOWED_HOSTS is * (config default in /opt/netbox-docker/configuration/configuration.py; NOT overridden in env/netbox.env), so NO change and NO restart was needed -- it served at .10 immediately (HTTP 200 at http://10.10.1.10:8000/login/).
  • Tailscale (office1-tailscale): re-asserted automatically after the interface IP change -- BackendState=Running, PrimaryRoutes=["10.10.0.0/22"] unchanged, tailnet IP 100.64.0.53 unchanged. NO tailscale up re-assert was needed. (If a future run shows the route dropped, re-assert with tailscale up --advertise-routes=10.10.0.0/22; node identity/key is unchanged.)

Step 4 -- Verify on the wire (READ-ONLY)

ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.10 'ip -4 addr'      # netbox at .10
curl -s -o /dev/null -w "%{http_code}\n" http://10.10.1.10:8000/api/    # 403/200 = up
ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.11 'tailscale status'        # tailscale at .11

Step 5 -- Update the repo references + the importer allowlist [DONE 2026-07-15]

10.10.1.201 -> 10.10.1.10 and 10.10.1.202 -> 10.10.1.11 applied across:

  • CODE (done): SANDBOX_HOSTS .201 -> .10 in netbox/roles-aggregates-import.py, netbox/dc-dc-prefixes-import.py, and netbox/d115-office-carve.py; tests/dc-dc-prefixes-import/test_logic.py (both sandbox-URL hosts). tests/roles-aggregates-import/run-tests.sh only greps for the literal SANDBOX_HOSTS token and uses localhost (still in the allowlist) -- no address change needed there.
  • Docs (done): docs/vr1-office1-as-built.md, docs/session-ledger.md, docs/dc-dc-netbox-buildout-scope.md, docs/dc-dc-deployment-workflow.md, runbooks/dc-dc-phase1-office1-standup.md, and D-120 itself. Dated changelogs LEFT as historical snapshots (they record the deploy-time .201/.202).
  • Gauntlet: bash scripts/run-tests-all.sh + bash scripts/repo-lint.sh re-run green.

Step 6 -- Record the D-120 carve in NetBox (via the sandbox loop)

Register 10.10.1.0/24's child ranges (static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments in the sandbox NetBox, then feed upstream with C2. NetBox is the IPAM authority for this; do not leave the layout recorded only in prose.

Rollback

Restore each guest's saved netplan + curtin backups (.bak-reip201 / .bak-reip202) via lxc exec and netplan apply, then git revert the Step-5 repo change. MAAS needs no rollback (its model was never changed -- see Step 1). Nothing here is destructive -- the VMs and their data are untouched; only the L3 address changes.

DC replication note

When a DC's compose /24 is built, its services get D-120 static-band addresses FROM DAY ONE via this same procedure -- there is no "assign in the node band then re-IP later" for the DCs. This runbook is the reference.