# changelog 2026-08-07 -- dc1-region standup sequence ("sequence and continue")

Session scope: sequence and execute the enumerated NEXT threads (dc1 region buildout,
NetBox recording, hygiene). Status lives ONLY in `docs/CURRENT-STATE.md`; this file is
the per-session body (GA-R2). Each item states what / why (evidence) / how to revert.

---

## Item 1 -- NetBox tool GAP closed + as-built utility hosts LANDED into office1-netbox

**What.** New tested importer `netbox/dc-util-hosts-import.py` (+ harness
`tests/dc-util-hosts-import/` 19/19) that records a D-134 utility-band HOST's
ip-addresses (both planes) into the apex, then landed the four as-built hosts:

| host | metal-admin | provider-public | ids |
|---|---|---|---|
| vr1-dc0-juju-01 (.5)      | 10.12.8.5  | 10.12.4.5  | 187, 188 |
| vr1-dc0-maas-01 (.6)      | 10.12.8.6  | 10.12.4.6  | 189, 190 |
| vr1-dc0-tailscale-01 (.7) | 10.12.8.7  | 10.12.4.7  | 191, 192 |
| vr1-dc1-maas-01 (.6)      | 10.12.68.6 | 10.12.64.6 | 193, 194 |

Fidelity: apex `ipam/ip-addresses` 186 -> 194 (+8); idempotent re-run reports
`EXISTS / created: 0`. Target `office1-netbox` (10.10.1.10, DOCFIX-195 sandbox apex).

**Why.** Landing the operator-flagged NetBox-pending list (changelog-20260807
Item 11) hit a tool gap: `dc-plane-apex-import.py` had loaded the D-134 utility
RANGES (.4-.49) and the VIP addresses (.50-.62), but NO importer recorded the
individual utility-HOST assignments within the band -- all 16 candidate host IPs
measured absent. Per CLAUDE.md hard rule 4 the gap was declared (operator ruled
"Build importer, then land") and NOT improvised as raw API POSTs.

**Design (advisor-reviewed).** DERIVED, never hardcoded: plane CIDRs from
`lib-net.sh`, host octet from `lib-hosts.sh` (an empty/missing octet is a hard die --
the clean-zero failure class). ONE host per invocation (liveness is operator-asserted;
no L3 path apex->DC plane, SEC-010). Whole-plan preflight (both planes) before any
create; the D-134 utility RANGE is the precondition (a missing range dies). dns_name
collision guard; SANDBOX_HOSTS + `--yes-write-upstream` gate; WAF-safe UA. dns_name
carries each ruled `vr1-dc<N>-<role>-NN` name -> the dc0 RENAMES are recorded by
construction (there were no stale old-name records to rename).

**Scope boundary / deferred (queued findings).**
- The `.4` ARTIFACT host (dc0 apt mirror / dc1 caching proxy, D-135) is OUT of this
  importer: it is not a `lib-hosts` host, is metal-admin-ONLY (`10.12.8.4` /
  `10.12.68.4`, no provider-public leg -- measured from `dc-mirror.sh`/`dc-cache-proxy.sh`
  LISTEN_CIDR), per-DC divergent, and has no host NAME anywhere in the repo. Recording
  it needs an operator naming ruling -- a separate finding, not this tool.
- dc1 `.5` juju-01 + `.7` tailscale-01: record WHEN LIVE (juju not yet bootstrapped;
  tailscale planned, rebuilt in step E). The `vr1-dc1-region` MAAS region needs no
  separate NetBox object -- the apex `dcim/regions` are geographic (US-West / VR0 / VR1),
  not MAAS regions; the region VM is represented by its `.6` ip-addresses.

**How to revert.** This is a NetBox state change, not a git revert. Delete the 8
ip-addresses by address from `office1-netbox` (ids 187-194, or by the four host
dns_names), e.g. `DELETE /api/ipam/ip-addresses/<id>/`. The importer itself and its
harness are removable with `git rm netbox/dc-util-hosts-import.py tests/dc-util-hosts-import/`
+ `bash scripts/run-tests-all.sh --record-manifest`.

---

## Item 2 -- Stage-5/dc1 region STANDUP: topology + IPAM + DHCP cutover (live, gated)

All against `--profile vr1-dc1-region --expect-rack vr1-dc1-maas-01`, run from voffice1;
each mutation profile-asserted and read back. Operator-approved ("I approve you running
these commands"; "Full cutover first").

**B -- topology apply** (`dc-region-topology.sh apply vr1-dc1 --commit`). Built the 5 named
plane fabrics, 6 spaces, 4 missing plane subnets (10.12.72/76/80/84.0/22), space bindings,
gateways, and the `openstack-vr1-dc1` placement tag. Verified `check` = 37/40 (the 3
remaining = DHCP/DNS, deliberately deferred to the handover). Capture:
`docs/audit/dc1-region-topology-20260807.txt`. (The apply's own run exceeded the client's
120s window during output; the writes all landed -- the check is the evidence, no half-write.)

**C-IPAM** (`dc-plane-ipam.sh`, `MAAS_PROFILE=vr1-dc1-region`). `carve-v6 --commit` = 6 v6
plane subnets on their v4 twins' VLANs, created + read back (0 errors). `reserve --commit`
= 13 v4 reservations (6 utility + 6 VIP bands + FIP pool `10.12.65.0-10.12.67.254`).
Read-only `check` = 24/0 PASS. (Note: reserve reported `skipped=6` on v6 bands though
carve-v6 had landed; the 24/0 check confirms the ruled state is fully present -- glance
owed, non-blocking.)

**C-DHCP/DNS cutover** (manual per the dc0 2026-07-30 precedent; `dc-region-topology`
deliberately excludes DHCP and no handover TOOL exists -- a Roosevelt-transferable gap,
logged not built mid-cutover). Measured exact values (nothing inferred): OLD =
Office1 `admin` subnet 11, vlan fabric 142 vid 0, primary nmpcq4; NEW = `vr1-dc1-region`
subnet 3, vlan fabric 0 vid 0, rack qtw8pm. As-built config replicated verbatim
(dns_servers `10.12.68.3`, allow_dns false, dynamic `10.12.68.201-.254` -- MEASURED, my
recollection of `.100-.200` was WRONG). Sequence: OFF admin (dhcp_on=false) -> region is
sole server -> set subnet-3 dns/range -> ON region (dhcp_on=true primary=qtw8pm).
Verified: topology check 40/0; `.6` runs the sole metal-admin dhcpd (`ss :67` on enp1s0);
vvr1-dc1 has NO `:67` on the node segment. FALSE ALARM resolved: a bare `pgrep -c dhcpd`
self-matched its own command shell and read 1 -- `ss :67`/`pgrep -af` are authoritative.
Capture: `docs/audit/dc1-dhcp-handover-20260807.txt`.

**Revert.** Region-side: `maas vr1-dc1-region vlan update 0 0 dhcp_on=false`; re-enable
admin: `maas admin vlan update 142 0 dhcp_on=true`. Topology/IPAM are additive (idempotent
re-apply / MAAS delete by object). No repo change in this item -- all live MAAS state.

---

## Item 3 -- Stage-5/dc1: power-key installed; image sync BLOCKED on SEC-031 (edge repair)

**C-powerkey DONE** (`maas-region-power-key.sh install vr1-dc1 qemu+ssh://jessea123@10.12.68.2/system
--commit`, run ON the .6 via office1_svc, key piped on stdin from vcloud -- never argv/intermediate,
shredded from the .6's mktemp). Verify 9/9 incl. the artifact test: REAL virsh (snap) connect to the
rack succeeds + enumerates 13 domains. SEC-016 per-DC isolation intact (dc1's own key, never dc0's).
FRAGILITY (SEC-016): re-assert after any MAAS snap refresh (`/var/snap/maas/current/`).

**C-images BLOCKED -- root-caused to SEC-031, not a new problem.** The region has no jammy resources
(boot-source `images.maas.io`, only noble selected). The .6 cannot reach images.maas.io: routing is
correct (default via edge 10.12.64.1, edge gw pingable, DNS resolves) but `ping 1.1.1.1` FAILS. Measured
that the RACK is identical (direct egress also fails; its :8000 proxy only works via an Office1-proxy
chain). PRIOR ART (CURRENT-STATE ~3979, SEC-031, capture dc1-edge-assessment-20260802.txt) already
diagnosed this: the 2026-08-01 in-place tofu resize bounced the containment VM = hard power cut; fsck on
the dc1 OPNsense edge destroyed the FreeBSD base-system user DB (`/etc/master.passwd`, `/etc/group`),
so `Configuring firewall.....failed` every boot -> NO pf ruleset -> the edge forwards but does NOT
translate (tcpdump: packet leaves WAN with source still 10.12.64.2, untranslated). config.xml is INTACT;
verdict REPAIR-not-rebuild.

**REFRAME for the egress decision.** dc0 (measured) reaches images.maas.io directly via its WORKING
edge NAT (http_proxy=null, jammy present). So the operator's choice "fix .6 egress to match dc0" =
**repair the dc1 edge (SEC-031)**: regenerate the user DB from config.xml via the proven D-112(c)
`local_user_set()` path (or donate from the pristine `opnsense-26.7-nano.qcow2` in the same pool) ->
firewall applies -> NAT restored. This also closes the SEC-031 security exposure (edge currently an
open router). CAVEAT: the edge has NO working credential path (sshd down, authenticated API hangs,
console login likely fails on the unparseable passwd DB). HELD for operator direction before touching
the live edge -- not bundled into image sync.

**Revert.** No mutation taken for images (diagnosis only). Power-key revert: remove
`/var/snap/maas/current/root/.ssh/id_dc1_power` + its `config` Host stanza on the .6.

---

## Item 4 -- SEC-031 CLOSED: dc1 OPNsense edge REBUILT, DC egress restored (proven dc0 procedure)

**What.** Rebuilt the dc1 edge via the proven dc0-2026-08-02 procedure (operator-directed:
"look back at the steps that were successful with dc0" -- dc0's edge was REBUILT, not
repaired). Result: `dc-egress-check dc1` **pass=8 fail=0**; the `.6` region reaches the
internet (`ping 1.1.1.1` OK, `curl images.maas.io` 200), which unblocks jammy image sync.

**Steps (now the runbook `runbooks/dc-edge-rebuild.md`):**
1. Rollback prep -- edge `virsh shutdown`, disk copied `...pre-sec031-repair-20260807`;
   inner tfstate backed up (sha256-identical); TARGETED `tofu plan -replace` (disk+domain)
   MACHINE-ASSERTED edge-only (2 non-no-op, both `vr1_dc1_opnsense`, 0 others -- the 9 nodes
   protected); base image confirmed present on voffice1.
2. `tofu apply` -> 2 added / 2 destroyed (edge only), fresh pristine disk, factory boot.
3. Console bootstrap -- generalised `scripts/opnsense-console-rebuild.py --domain vr1-dc1-opnsense
   --pubkey <dc1 edge pub>` on the rack: `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`,
   `USER-MATERIALIZED`, sshd `OK`.
4. Cleared rack known_hosts (rebuilt edge = new host keys), interim reach leg on virbr4,
   proved key-only SSH (`OPNsense 26.7`, `15.1-RELEASE-p1`).
5. Minted API key (`opnsense-bootstrap-apikey.sh`); set WAN `172.30.3.2/24` gw `172.30.3.1`
   (pf reload produced automatic outbound NAT here); proved edge egress (ping 1.1.1.1,
   fetch jammy Release) BEFORE the LAN move; set LAN `10.12.64.1/22` (session drop expected),
   `configctl interface reconfigure lan`, NAT confirmed `from (vtnet0:network) to any`.
6. `dc-egress-check dc1` 8/8. Cleanup: shredded the transient rack edge key (SEC-015 pattern),
   removed the reach leg, consolidated the API key to `~/vr1-dc1-creds/` (0600, sha256-verified).

**Why a runbook + tool now (operator instruction).** The dc0 rebuild lived ONLY as an audit
capture (`docs/audit/dc0-edge-rebuild-20260802.txt`), so dc1 had to reconstruct the whole
procedure by reading 683 lines + adapting a one-off driver. Fixed: `runbooks/dc-edge-rebuild.md`
(site-parameterised procedure) + `scripts/opnsense-console-rebuild.py` (site-agnostic driver,
replaces the per-DC `d112c-console-<dc>-rebuild.py` copies). SEC-031 CLOSED in the ledger.

**Lessons baked in:** rebuilt edge invalidates rack known_hosts; the LAN move drops its own
session (config saves+applies first); edge root shell is tcsh (no `2>&1`); assert tofu scope on
ACTIONS not the drift note; `opnsense-set-interface-v4.sh` uses the legacy interfaces page by
design (D-113 amdt).

**How to revert.** Rebuild is forward-only; the pre-rebuild disk copy
(`...pre-sec031-repair-20260807`) + tfstate backup on the rack/voffice1 are the rollback point.
Repo revert: `git rm scripts/opnsense-console-rebuild.py runbooks/dc-edge-rebuild.md` + reopen SEC-031.
