changelog 2026-08-07 -- dc1-region standup sequence ("sequence and continue")

Session scope: sequence and execute the enumerated NEXT threads (dc1 region buildout, NetBox recording, hygiene). Status lives ONLY in docs/CURRENT-STATE.md; this file is the per-session body (GA-R2). Each item states what / why (evidence) / how to revert.

Item 1 -- NetBox tool GAP closed + as-built utility hosts LANDED into office1-netbox

What. New tested importer netbox/dc-util-hosts-import.py (+ harness tests/dc-util-hosts-import/ 19/19) that records a D-134 utility-band HOST's ip-addresses (both planes) into the apex, then landed the four as-built hosts:

Fidelity: apex ipam/ip-addresses 186 -> 194 (+8); idempotent re-run reports EXISTS / created: 0. Target office1-netbox (10.10.1.10, DOCFIX-195 sandbox apex).

Why. Landing the operator-flagged NetBox-pending list (changelog-20260807 Item 11) hit a tool gap: dc-plane-apex-import.py had loaded the D-134 utility RANGES (.4-.49) and the VIP addresses (.50-.62), but NO importer recorded the individual utility-HOST assignments within the band -- all 16 candidate host IPs measured absent. Per CLAUDE.md hard rule 4 the gap was declared (operator ruled "Build importer, then land") and NOT improvised as raw API POSTs.

Design (advisor-reviewed). DERIVED, never hardcoded: plane CIDRs from lib-net.sh, host octet from lib-hosts.sh (an empty/missing octet is a hard die -- the clean-zero failure class). ONE host per invocation (liveness is operator-asserted; no L3 path apex->DC plane, SEC-010). Whole-plan preflight (both planes) before any create; the D-134 utility RANGE is the precondition (a missing range dies). dns_name collision guard; SANDBOX_HOSTS + --yes-write-upstream gate; WAF-safe UA. dns_name carries each ruled vr1-dc<N>-<role>-NN name -> the dc0 RENAMES are recorded by construction (there were no stale old-name records to rename).

Scope boundary / deferred (queued findings).

• The .4 ARTIFACT host (dc0 apt mirror / dc1 caching proxy, D-135) is OUT of this importer: it is not a lib-hosts host, is metal-admin-ONLY (10.12.8.4 / 10.12.68.4, no provider-public leg -- measured from dc-mirror.sh/dc-cache-proxy.sh LISTEN_CIDR), per-DC divergent, and has no host NAME anywhere in the repo. Recording it needs an operator naming ruling -- a separate finding, not this tool.
• dc1 .5 juju-01 + .7 tailscale-01: record WHEN LIVE (juju not yet bootstrapped; tailscale planned, rebuilt in step E). The vr1-dc1-region MAAS region needs no separate NetBox object -- the apex dcim/regions are geographic (US-West / VR0 / VR1), not MAAS regions; the region VM is represented by its .6 ip-addresses.

How to revert. This is a NetBox state change, not a git revert. Delete the 8 ip-addresses by address from office1-netbox (ids 187-194, or by the four host dns_names), e.g. DELETE /api/ipam/ip-addresses/<id>/. The importer itself and its harness are removable with git rm netbox/dc-util-hosts-import.py tests/dc-util-hosts-import/

• bash scripts/run-tests-all.sh --record-manifest.

Item 2 -- Stage-5/dc1 region STANDUP: topology + IPAM + DHCP cutover (live, gated)

All against --profile vr1-dc1-region --expect-rack vr1-dc1-maas-01, run from voffice1; each mutation profile-asserted and read back. Operator-approved ("I approve you running these commands"; "Full cutover first").

B -- topology apply (dc-region-topology.sh apply vr1-dc1 --commit). Built the 5 named plane fabrics, 6 spaces, 4 missing plane subnets (10.12.72/76/80/84.0/22), space bindings, gateways, and the openstack-vr1-dc1 placement tag. Verified check = 37/40 (the 3 remaining = DHCP/DNS, deliberately deferred to the handover). Capture: docs/audit/dc1-region-topology-20260807.txt. (The apply's own run exceeded the client's 120s window during output; the writes all landed -- the check is the evidence, no half-write.)

C-IPAM (dc-plane-ipam.sh, MAAS_PROFILE=vr1-dc1-region). carve-v6 --commit = 6 v6 plane subnets on their v4 twins' VLANs, created + read back (0 errors). reserve --commit = 13 v4 reservations (6 utility + 6 VIP bands + FIP pool 10.12.65.0-10.12.67.254). Read-only check = 24/0 PASS. (Note: reserve reported skipped=6 on v6 bands though carve-v6 had landed; the 24/0 check confirms the ruled state is fully present -- glance owed, non-blocking.)

C-DHCP/DNS cutover (manual per the dc0 2026-07-30 precedent; dc-region-topology deliberately excludes DHCP and no handover TOOL exists -- a Roosevelt-transferable gap, logged not built mid-cutover). Measured exact values (nothing inferred): OLD = Office1 admin subnet 11, vlan fabric 142 vid 0, primary nmpcq4; NEW = vr1-dc1-region subnet 3, vlan fabric 0 vid 0, rack qtw8pm. As-built config replicated verbatim (dns_servers 10.12.68.3, allow_dns false, dynamic 10.12.68.201-.254 -- MEASURED, my recollection of .100-.200 was WRONG). Sequence: OFF admin (dhcp_on=false) -> region is sole server -> set subnet-3 dns/range -> ON region (dhcp_on=true primary=qtw8pm). Verified: topology check 40/0; .6 runs the sole metal-admin dhcpd (ss :67 on enp1s0); vvr1-dc1 has NO :67 on the node segment. FALSE ALARM resolved: a bare pgrep -c dhcpd self-matched its own command shell and read 1 -- ss :67/pgrep -af are authoritative. Capture: docs/audit/dc1-dhcp-handover-20260807.txt.

Revert. Region-side: maas vr1-dc1-region vlan update 0 0 dhcp_on=false; re-enable admin: maas admin vlan update 142 0 dhcp_on=true. Topology/IPAM are additive (idempotent re-apply / MAAS delete by object). No repo change in this item -- all live MAAS state.

Item 3 -- Stage-5/dc1: power-key installed; image sync BLOCKED on SEC-031 (edge repair)

C-powerkey DONE (maas-region-power-key.sh install vr1-dc1 qemu+ssh://jessea123@10.12.68.2/system --commit, run ON the .6 via office1_svc, key piped on stdin from vcloud -- never argv/intermediate, shredded from the .6's mktemp). Verify 9/9 incl. the artifact test: REAL virsh (snap) connect to the rack succeeds + enumerates 13 domains. SEC-016 per-DC isolation intact (dc1's own key, never dc0's). FRAGILITY (SEC-016): re-assert after any MAAS snap refresh (/var/snap/maas/current/).

C-images BLOCKED -- root-caused to SEC-031, not a new problem. The region has no jammy resources (boot-source images.maas.io, only noble selected). The .6 cannot reach images.maas.io: routing is correct (default via edge 10.12.64.1, edge gw pingable, DNS resolves) but ping 1.1.1.1 FAILS. Measured that the RACK is identical (direct egress also fails; its :8000 proxy only works via an Office1-proxy chain). PRIOR ART (CURRENT-STATE ~3979, SEC-031, capture dc1-edge-assessment-20260802.txt) already diagnosed this: the 2026-08-01 in-place tofu resize bounced the containment VM = hard power cut; fsck on the dc1 OPNsense edge destroyed the FreeBSD base-system user DB (/etc/master.passwd, /etc/group), so Configuring firewall.....failed every boot -> NO pf ruleset -> the edge forwards but does NOT translate (tcpdump: packet leaves WAN with source still 10.12.64.2, untranslated). config.xml is INTACT; verdict REPAIR-not-rebuild.

REFRAME for the egress decision. dc0 (measured) reaches images.maas.io directly via its WORKING edge NAT (http_proxy=null, jammy present). So the operator's choice "fix .6 egress to match dc0" = repair the dc1 edge (SEC-031): regenerate the user DB from config.xml via the proven D-112(c) local_user_set() path (or donate from the pristine opnsense-26.7-nano.qcow2 in the same pool) -> firewall applies -> NAT restored. This also closes the SEC-031 security exposure (edge currently an open router). CAVEAT: the edge has NO working credential path (sshd down, authenticated API hangs, console login likely fails on the unparseable passwd DB). HELD for operator direction before touching the live edge -- not bundled into image sync.

Revert. No mutation taken for images (diagnosis only). Power-key revert: remove /var/snap/maas/current/root/.ssh/id_dc1_power + its config Host stanza on the .6.

Item 4 -- SEC-031 CLOSED: dc1 OPNsense edge REBUILT, DC egress restored (proven dc0 procedure)

What. Rebuilt the dc1 edge via the proven dc0-2026-08-02 procedure (operator-directed: "look back at the steps that were successful with dc0" -- dc0's edge was REBUILT, not repaired). Result: dc-egress-check dc1 pass=8 fail=0; the .6 region reaches the internet (ping 1.1.1.1 OK, curl images.maas.io 200), which unblocks jammy image sync.

Steps (now the runbook runbooks/dc-edge-rebuild.md):

1. Rollback prep -- edge virsh shutdown, disk copied ...pre-sec031-repair-20260807; inner tfstate backed up (sha256-identical); TARGETED tofu plan -replace (disk+domain) MACHINE-ASSERTED edge-only (2 non-no-op, both vr1_dc1_opnsense, 0 others -- the 9 nodes protected); base image confirmed present on voffice1.
2. tofu apply -> 2 added / 2 destroyed (edge only), fresh pristine disk, factory boot.
3. Console bootstrap -- generalised scripts/opnsense-console-rebuild.py --domain vr1-dc1-opnsense --pubkey <dc1 edge pub> on the rack: CONFIG-WRITTEN, AK-IN-CONFIG=152, USER-MATERIALIZED, sshd OK.
4. Cleared rack known_hosts (rebuilt edge = new host keys), interim reach leg on virbr4, proved key-only SSH (OPNsense 26.7, 15.1-RELEASE-p1).
5. Minted API key (opnsense-bootstrap-apikey.sh); set WAN 172.30.3.2/24 gw 172.30.3.1 (pf reload produced automatic outbound NAT here); proved edge egress (ping 1.1.1.1, fetch jammy Release) BEFORE the LAN move; set LAN 10.12.64.1/22 (session drop expected), configctl interface reconfigure lan, NAT confirmed from (vtnet0:network) to any.
6. dc-egress-check dc1 8/8. Cleanup: shredded the transient rack edge key (SEC-015 pattern), removed the reach leg, consolidated the API key to ~/vr1-dc1-creds/ (0600, sha256-verified).

Why a runbook + tool now (operator instruction). The dc0 rebuild lived ONLY as an audit capture (docs/audit/dc0-edge-rebuild-20260802.txt), so dc1 had to reconstruct the whole procedure by reading 683 lines + adapting a one-off driver. Fixed: runbooks/dc-edge-rebuild.md (site-parameterised procedure) + scripts/opnsense-console-rebuild.py (site-agnostic driver, replaces the per-DC d112c-console-<dc>-rebuild.py copies). SEC-031 CLOSED in the ledger.

Lessons baked in: rebuilt edge invalidates rack known_hosts; the LAN move drops its own session (config saves+applies first); edge root shell is tcsh (no 2>&1); assert tofu scope on ACTIONS not the drift note; opnsense-set-interface-v4.sh uses the legacy interfaces page by design (D-113 amdt).

How to revert. Rebuild is forward-only; the pre-rebuild disk copy (...pre-sec031-repair-20260807) + tfstate backup on the rack/voffice1 are the rollback point. Repo revert: git rm scripts/opnsense-console-rebuild.py runbooks/dc-edge-rebuild.md + reopen SEC-031.