# changelog 2026-08-07 -- dc1-region standup sequence ("sequence and continue") Session scope: sequence and execute the enumerated NEXT threads (dc1 region buildout, NetBox recording, hygiene). Status lives ONLY in `docs/CURRENT-STATE.md`; this file is the per-session body (GA-R2). Each item states what / why (evidence) / how to revert. --- ## Item 1 -- NetBox tool GAP closed + as-built utility hosts LANDED into office1-netbox **What.** New tested importer `netbox/dc-util-hosts-import.py` (+ harness `tests/dc-util-hosts-import/` 19/19) that records a D-134 utility-band HOST's ip-addresses (both planes) into the apex, then landed the four as-built hosts: | host | metal-admin | provider-public | ids | |---|---|---|---| | vr1-dc0-juju-01 (.5) | 10.12.8.5 | 10.12.4.5 | 187, 188 | | vr1-dc0-maas-01 (.6) | 10.12.8.6 | 10.12.4.6 | 189, 190 | | vr1-dc0-tailscale-01 (.7) | 10.12.8.7 | 10.12.4.7 | 191, 192 | | vr1-dc1-maas-01 (.6) | 10.12.68.6 | 10.12.64.6 | 193, 194 | Fidelity: apex `ipam/ip-addresses` 186 -> 194 (+8); idempotent re-run reports `EXISTS / created: 0`. Target `office1-netbox` (10.10.1.10, DOCFIX-195 sandbox apex). **Why.** Landing the operator-flagged NetBox-pending list (changelog-20260807 Item 11) hit a tool gap: `dc-plane-apex-import.py` had loaded the D-134 utility RANGES (.4-.49) and the VIP addresses (.50-.62), but NO importer recorded the individual utility-HOST assignments within the band -- all 16 candidate host IPs measured absent. Per CLAUDE.md hard rule 4 the gap was declared (operator ruled "Build importer, then land") and NOT improvised as raw API POSTs. **Design (advisor-reviewed).** DERIVED, never hardcoded: plane CIDRs from `lib-net.sh`, host octet from `lib-hosts.sh` (an empty/missing octet is a hard die -- the clean-zero failure class). ONE host per invocation (liveness is operator-asserted; no L3 path apex->DC plane, SEC-010). Whole-plan preflight (both planes) before any create; the D-134 utility RANGE is the precondition (a missing range dies). dns_name collision guard; SANDBOX_HOSTS + `--yes-write-upstream` gate; WAF-safe UA. dns_name carries each ruled `vr1-dc--NN` name -> the dc0 RENAMES are recorded by construction (there were no stale old-name records to rename). **Scope boundary / deferred (queued findings).** - The `.4` ARTIFACT host (dc0 apt mirror / dc1 caching proxy, D-135) is OUT of this importer: it is not a `lib-hosts` host, is metal-admin-ONLY (`10.12.8.4` / `10.12.68.4`, no provider-public leg -- measured from `dc-mirror.sh`/`dc-cache-proxy.sh` LISTEN_CIDR), per-DC divergent, and has no host NAME anywhere in the repo. Recording it needs an operator naming ruling -- a separate finding, not this tool. - dc1 `.5` juju-01 + `.7` tailscale-01: record WHEN LIVE (juju not yet bootstrapped; tailscale planned, rebuilt in step E). The `vr1-dc1-region` MAAS region needs no separate NetBox object -- the apex `dcim/regions` are geographic (US-West / VR0 / VR1), not MAAS regions; the region VM is represented by its `.6` ip-addresses. **How to revert.** This is a NetBox state change, not a git revert. Delete the 8 ip-addresses by address from `office1-netbox` (ids 187-194, or by the four host dns_names), e.g. `DELETE /api/ipam/ip-addresses//`. The importer itself and its harness are removable with `git rm netbox/dc-util-hosts-import.py tests/dc-util-hosts-import/` + `bash scripts/run-tests-all.sh --record-manifest`. --- ## Item 2 -- Stage-5/dc1 region STANDUP: topology + IPAM + DHCP cutover (live, gated) All against `--profile vr1-dc1-region --expect-rack vr1-dc1-maas-01`, run from voffice1; each mutation profile-asserted and read back. Operator-approved ("I approve you running these commands"; "Full cutover first"). **B -- topology apply** (`dc-region-topology.sh apply vr1-dc1 --commit`). Built the 5 named plane fabrics, 6 spaces, 4 missing plane subnets (10.12.72/76/80/84.0/22), space bindings, gateways, and the `openstack-vr1-dc1` placement tag. Verified `check` = 37/40 (the 3 remaining = DHCP/DNS, deliberately deferred to the handover). Capture: `docs/audit/dc1-region-topology-20260807.txt`. (The apply's own run exceeded the client's 120s window during output; the writes all landed -- the check is the evidence, no half-write.) **C-IPAM** (`dc-plane-ipam.sh`, `MAAS_PROFILE=vr1-dc1-region`). `carve-v6 --commit` = 6 v6 plane subnets on their v4 twins' VLANs, created + read back (0 errors). `reserve --commit` = 13 v4 reservations (6 utility + 6 VIP bands + FIP pool `10.12.65.0-10.12.67.254`). Read-only `check` = 24/0 PASS. (Note: reserve reported `skipped=6` on v6 bands though carve-v6 had landed; the 24/0 check confirms the ruled state is fully present -- glance owed, non-blocking.) **C-DHCP/DNS cutover** (manual per the dc0 2026-07-30 precedent; `dc-region-topology` deliberately excludes DHCP and no handover TOOL exists -- a Roosevelt-transferable gap, logged not built mid-cutover). Measured exact values (nothing inferred): OLD = Office1 `admin` subnet 11, vlan fabric 142 vid 0, primary nmpcq4; NEW = `vr1-dc1-region` subnet 3, vlan fabric 0 vid 0, rack qtw8pm. As-built config replicated verbatim (dns_servers `10.12.68.3`, allow_dns false, dynamic `10.12.68.201-.254` -- MEASURED, my recollection of `.100-.200` was WRONG). Sequence: OFF admin (dhcp_on=false) -> region is sole server -> set subnet-3 dns/range -> ON region (dhcp_on=true primary=qtw8pm). Verified: topology check 40/0; `.6` runs the sole metal-admin dhcpd (`ss :67` on enp1s0); vvr1-dc1 has NO `:67` on the node segment. FALSE ALARM resolved: a bare `pgrep -c dhcpd` self-matched its own command shell and read 1 -- `ss :67`/`pgrep -af` are authoritative. Capture: `docs/audit/dc1-dhcp-handover-20260807.txt`. **Revert.** Region-side: `maas vr1-dc1-region vlan update 0 0 dhcp_on=false`; re-enable admin: `maas admin vlan update 142 0 dhcp_on=true`. Topology/IPAM are additive (idempotent re-apply / MAAS delete by object). No repo change in this item -- all live MAAS state. --- ## Item 3 -- Stage-5/dc1: power-key installed; image sync BLOCKED on SEC-031 (edge repair) **C-powerkey DONE** (`maas-region-power-key.sh install vr1-dc1 qemu+ssh://jessea123@10.12.68.2/system --commit`, run ON the .6 via office1_svc, key piped on stdin from vcloud -- never argv/intermediate, shredded from the .6's mktemp). Verify 9/9 incl. the artifact test: REAL virsh (snap) connect to the rack succeeds + enumerates 13 domains. SEC-016 per-DC isolation intact (dc1's own key, never dc0's). FRAGILITY (SEC-016): re-assert after any MAAS snap refresh (`/var/snap/maas/current/`). **C-images BLOCKED -- root-caused to SEC-031, not a new problem.** The region has no jammy resources (boot-source `images.maas.io`, only noble selected). The .6 cannot reach images.maas.io: routing is correct (default via edge 10.12.64.1, edge gw pingable, DNS resolves) but `ping 1.1.1.1` FAILS. Measured that the RACK is identical (direct egress also fails; its :8000 proxy only works via an Office1-proxy chain). PRIOR ART (CURRENT-STATE ~3979, SEC-031, capture dc1-edge-assessment-20260802.txt) already diagnosed this: the 2026-08-01 in-place tofu resize bounced the containment VM = hard power cut; fsck on the dc1 OPNsense edge destroyed the FreeBSD base-system user DB (`/etc/master.passwd`, `/etc/group`), so `Configuring firewall.....failed` every boot -> NO pf ruleset -> the edge forwards but does NOT translate (tcpdump: packet leaves WAN with source still 10.12.64.2, untranslated). config.xml is INTACT; verdict REPAIR-not-rebuild. **REFRAME for the egress decision.** dc0 (measured) reaches images.maas.io directly via its WORKING edge NAT (http_proxy=null, jammy present). So the operator's choice "fix .6 egress to match dc0" = **repair the dc1 edge (SEC-031)**: regenerate the user DB from config.xml via the proven D-112(c) `local_user_set()` path (or donate from the pristine `opnsense-26.7-nano.qcow2` in the same pool) -> firewall applies -> NAT restored. This also closes the SEC-031 security exposure (edge currently an open router). CAVEAT: the edge has NO working credential path (sshd down, authenticated API hangs, console login likely fails on the unparseable passwd DB). HELD for operator direction before touching the live edge -- not bundled into image sync. **Revert.** No mutation taken for images (diagnosis only). Power-key revert: remove `/var/snap/maas/current/root/.ssh/id_dc1_power` + its `config` Host stanza on the .6. --- ## Item 4 -- SEC-031 CLOSED: dc1 OPNsense edge REBUILT, DC egress restored (proven dc0 procedure) **What.** Rebuilt the dc1 edge via the proven dc0-2026-08-02 procedure (operator-directed: "look back at the steps that were successful with dc0" -- dc0's edge was REBUILT, not repaired). Result: `dc-egress-check dc1` **pass=8 fail=0**; the `.6` region reaches the internet (`ping 1.1.1.1` OK, `curl images.maas.io` 200), which unblocks jammy image sync. **Steps (now the runbook `runbooks/dc-edge-rebuild.md`):** 1. Rollback prep -- edge `virsh shutdown`, disk copied `...pre-sec031-repair-20260807`; inner tfstate backed up (sha256-identical); TARGETED `tofu plan -replace` (disk+domain) MACHINE-ASSERTED edge-only (2 non-no-op, both `vr1_dc1_opnsense`, 0 others -- the 9 nodes protected); base image confirmed present on voffice1. 2. `tofu apply` -> 2 added / 2 destroyed (edge only), fresh pristine disk, factory boot. 3. Console bootstrap -- generalised `scripts/opnsense-console-rebuild.py --domain vr1-dc1-opnsense --pubkey ` on the rack: `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`, sshd `OK`. 4. Cleared rack known_hosts (rebuilt edge = new host keys), interim reach leg on virbr4, proved key-only SSH (`OPNsense 26.7`, `15.1-RELEASE-p1`). 5. Minted API key (`opnsense-bootstrap-apikey.sh`); set WAN `172.30.3.2/24` gw `172.30.3.1` (pf reload produced automatic outbound NAT here); proved edge egress (ping 1.1.1.1, fetch jammy Release) BEFORE the LAN move; set LAN `10.12.64.1/22` (session drop expected), `configctl interface reconfigure lan`, NAT confirmed `from (vtnet0:network) to any`. 6. `dc-egress-check dc1` 8/8. Cleanup: shredded the transient rack edge key (SEC-015 pattern), removed the reach leg, consolidated the API key to `~/vr1-dc1-creds/` (0600, sha256-verified). **Why a runbook + tool now (operator instruction).** The dc0 rebuild lived ONLY as an audit capture (`docs/audit/dc0-edge-rebuild-20260802.txt`), so dc1 had to reconstruct the whole procedure by reading 683 lines + adapting a one-off driver. Fixed: `runbooks/dc-edge-rebuild.md` (site-parameterised procedure) + `scripts/opnsense-console-rebuild.py` (site-agnostic driver, replaces the per-DC `d112c-console--rebuild.py` copies). SEC-031 CLOSED in the ledger. **Lessons baked in:** rebuilt edge invalidates rack known_hosts; the LAN move drops its own session (config saves+applies first); edge root shell is tcsh (no `2>&1`); assert tofu scope on ACTIONS not the drift note; `opnsense-set-interface-v4.sh` uses the legacy interfaces page by design (D-113 amdt). **How to revert.** Rebuild is forward-only; the pre-rebuild disk copy (`...pre-sec031-repair-20260807`) + tfstate backup on the rack/voffice1 are the rollback point. Repo revert: `git rm scripts/opnsense-console-rebuild.py runbooks/dc-edge-rebuild.md` + reopen SEC-031.