# 2026-07-23 -- G12 [V] leg: dc1 edge bootstrap + v4 addressing (session changelog)

Session scope: resume the G12 [V] leg from the 2026-07-22 close handoff
(edge bootstrap onward). Branch `dc-dc-g12-dc1-substrate`. One changelog per
session (GA-R2/D1); every item carries its revert. Live mutations gated
(operator ran in `manual` permission mode -- see Item 0). Edge-phase mutations
mostly run UNWRAPPED (approval prompt = the gate); the console bootstrap went
through the `~/as-executed/2026-07-22-dc1-deploy.log` wrap.

## Item 0 -- permission-mode friction (recorded; no repo change)

The session opened in `auto` mode, whose classifier BLOCKS remote
`sudo`/mutation shapes (matched by the project `ssh * sudo *` ask rule)
instead of surfacing them for approval -- so it cannot deliver "operator
approves each mutation." Resolved by switching to `manual` mode (alias for
`default`; no classifier): allow-rules flow read-only rack probes, project ask
rules prompt every gated mutation, deny rules + `guard-destructive.py` stay
active. Session allow-rules for the dc1/dc0 rack ssh shapes were added to
`.claude/settings.local.json` (the operator later broadened to `ssh *`).
Revert: none (session-scoped settings; remove the added allow lines at close
if desired).

## Item 1 -- dc1 edge service keypair minted + manifest

`~/vr1-dc1-creds/vr1-dc1-edge_ed25519` (0600/0644) minted on vcloud,
`creds-manifests/vr1-dc1.manifest` extended (edge key + `opnsense-api.txt`),
`creds-audit vr1-dc1` CLEAN (5 entries). Revert: `git checkout` the manifest;
delete the keypair.

## Item 2 -- D-112(c) console bootstrap COMPLETE (edge SSH + service key)

Driver `d112c-console-dc1.py` (dc1 adaptation of the dc0-PROVEN v6, retrieved
verbatim from `vvr1-dc0:~/d112c-console.py`): serial console -> factory
root/opnsense -> shell -> ship+run bootstrap PHP (enable ssh, permitrootlogin,
install dc1 edge pubkey, `write_config`) -> materialize PHP
(`local_user_set()`, dc0 lesson v). Transcript `~/d112c-console-dc1.log` on
the rack: `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`;
`/root/.ssh/authorized_keys` (115 B) created (was absent). Edge banner
confirms **vtnet0 = LAN 192.168.1.1/24, vtnet1 = WAN** (dc0 mapping).
**FINDING (logged, not fixed mid-step):** the dc0 `d112c-bootstrap.php` on the
rack required only `config.inc`, so my first bootstrap PHP threw
`Call to undefined function shell_safe()` at `config.inc:311` (in
`write_config()->make_config_revision_entry()`) and aborted BEFORE writing --
this is **dc0 lesson (iv)** (shell_safe lives in `util.inc`, measured on dc0's
identical 26.7 image). Added `require_once("util.inc")`; two diagnostic lines
made csh-safe (edge root shell is tcsh; `2>&1` = "Ambiguous output redirect").
Re-ran clean. The proven dc0 `.b64` bootstrap artifact is missing this include
-- a real gap if replayed; queue: fold `util.inc` into the canonical console
bootstrap payload. Revert: config-only; re-run factory reset on the edge to
undo (or leave -- it is the intended state).

## Item 3 -- key-only SSH to the edge VERIFIED (D-112(c) proof)

Reached the edge via a ProxyCommand chain from vcloud (rack key for the jump,
edge key for the final hop -- per-hop keys; a plain `-J` applies one key to all
hops and the rack rejected it). `root@192.168.1.1` -> `uname -r` =
`15.1-RELEASE-p1`, `ifconfig -l` = vtnet0/vtnet1/lo0/enc0/pfsync0/pflog0.
Second D-112(c) datapoint after dc0.

## Item 4 -- D-113(a2) API key MINTED + smoke test PASS

Interim reach leg `192.168.1.2/24` added on the rack provider-public bridge
`virbr4` (measured this session) to reach the factory edge LAN. Edge-config
scripts + edge key staged on the rack (dc0 method; the mint script SSHes
edge-direct, no ProxyJump). Key minted via the vendor `opnsense-mint-apikey.php`
(`apikeys->add()` -- the GUI's own path); key/secret 80 chars, 173 B ->
`~/opnsense-api.txt` (0600, secret never printed). Smoke test from the rack:
`GET core/firmware/status` -> exit 0, `product_abi 26.7`. First proof the
D-113(a2) API path works on 26.7 for dc1.
**FINDING (logged):** `opnsense-bootstrap-apikey.sh`'s first `scp` to the edge
failed once with `scp: Connection closed` -- TRANSIENT: the edge sshd was not
ready in the moment right after the console bootstrap's `configctl openssh
restart` (both default `scp` and `scp -O` succeeded minutes later; the edge is
reachable and scp works). Unblocked via an ssh-pipe mint (vendor PHP shipped
by `ssh 'cat >'`, retrieved by `ssh -n 'cat' > file`) -- during which a
missing `-n` on the edge ssh calls let them EAT the rack `bash -s` heredoc
stdin (mint ran, retrieve/cleanup lines were swallowed); re-run with `-n`
retrieved the already-minted key cleanly. Queue: a post-restart sshd
readiness wait / one scp retry in `opnsense-bootstrap-apikey.sh`. Revert: delete
the API key via the edge API + remove `~/vr1-dc1-creds/opnsense-api.txt`.

## Item 5 -- edge v4 addressing (WAN then LAN; reach-handoff)

`opnsense-set-interface-v4.sh` run VERBATIM from the rack (plain scp works
after warm-up; no repo edit). Values confirmed from committed `lib-net.sh`
(provider-public `10.12.64.0/22` gw `10.12.64.1`, D-124 amendment) + the D-124
uplink `172.30.3.0/24`.
- **WAN first** (script ordering rule): dhcp -> `172.30.3.2/24` gw
  `172.30.3.1`, applied + read back on the kernel (vtnet1). Edge egress
  VERIFIED: ping 1.1.1.1 = 0% loss.
- **Permanent rack leg** `10.12.64.2/22` added on `virbr4` (so the rack keeps
  reach after the edge LAN moves).
- **LAN last**: `192.168.1.1/24` -> `10.12.64.1/22` (vtnet0). The apply drops
  the `192.168.1.1` session mid-command (expected; the foreground ssh hung on
  the dead interface and was TaskStop-ped -- config was already saved+applied).
  Verified independently at the NEW LAN via the rack's `10.12.64.2` leg:
  `ifconfig vtnet0` = `inet 10.12.64.1 netmask 0xfffffc00` (/22); API
  `GET core/firmware/status` at `10.12.64.1` -> exit 0, abi 26.7.
- Interim `192.168.1.2/24` removed; `virbr4` now carries only `10.12.64.2/22`
  (dc0 end-state exactly).
Revert: re-address LAN/WAN back via the same script; the leg adds/dels are
`ip addr add/del` (non-persistent).

## Item 6 -- creds consolidated + rack edge-key WIPED (SEC-015 closed transient)

API creds pulled rack -> `~/vr1-dc1-creds/opnsense-api.txt` (0600, 1 key + 1
secret). Rack edge-key copy `shred -u`'d; staged edge-config scripts +
console driver + rack creds copy removed (`ls` confirms none remain). The
non-secret console transcript `~/d112c-console-dc1.log` retained as an audit
artifact. **SEC-015** opened for the transient edge-key-on-rack exposure
(minted->wiped same phase; disclosed). Revert: n/a (cleanup).

## Milestone

dc1 edge fully bootstrapped + addressed -- D-112(c) console bootstrap and
D-113(a2) REST API both PROVEN on 26.7 (second datapoint after dc0). Edge:
WAN `172.30.3.2/24` gw `172.30.3.1` (egress 0% loss), LAN `10.12.64.1/22`
(the ruled provider-public gateway), API answering at the new LAN.

## Item 7 -- dc-rack-net.sh dc1 site-table arm + harness (D-131 sub-1 delivery)

Added the `dc1)` arm to `scripts/dc-rack-net.sh` (the STANDING per-DC forwarder
+ rack-legs pattern), MEASURED/apex-committed values, network-name keyed (no
virbrN literal -- T6 holds):
  vr1-dc1-metal-admin      10.12.68.2/22  rack MAAS/DHCP leg (apex-committed rack IP)
  vr1-dc1-metal-admin      10.12.68.3/22  node-DNS forwarder listen alias (D-120 static)
  vr1-dc1-provider-public  10.12.64.2/22  edge-LAN leg (measured live on the bridge 2026-07-23)
  DNS_LISTEN=10.12.68.3  DNS_UPSTREAM=10.10.0.20 (region BIND over transit).
Harness `tests/dc-rack-net/` extended T15-T18 (dc1 identity + MEASURED-tag);
**18/18**, gauntlet **ALL GREEN (76)**, repo-lint 0-fail. This forwarder is the
D-131 fix that pre-empts dc0's commissioning SERVFAIL on the isolated rack.
Revert: `git checkout` scripts/dc-rack-net.sh tests/dc-rack-net/run-tests.sh.

## Item 8 -- dc-rack-net install on the dc1 rack (persistent legs + forwarder)

`install dc1` run on the rack (operator-gated). Check PASS 10/10
(`docs/audit/dc1-rack-net-install-20260723.txt`): persistent metal-admin legs
`10.12.68.2/22` + `10.12.68.3/22` on virbr6, provider-public `10.12.64.2/22` on
virbr4, `dc1-rack-legs` + `dc1-node-dns` enabled+active. Behavioral proof: the
forwarder answers authoritative `maas-internal SOA` via region BIND -- the
D-131 fix that pre-empts dc0's commissioning SERVFAIL. Revert:
`dc-rack-net.sh` install is idempotent; to undo, disable the two units + remove
the generated files + the interim legs.

## Item 9 -- region-side MAAS config for dc1 metal-admin (DHCP + DNS + range)

MAAS auto-discovered dc1 planes from the rack interfaces: metal-admin
`10.12.68.0/22` = subnet id 11 (VLAN fabric 142/vid 0), provider-public
`10.12.64.0/22` = subnet id 10. On subnet 11 (all operator-gated):
- D-120 dynamic range `10.12.68.100-10.12.68.200` created (iprange id 3) --
  the ruled band applied to dc1's CIDR.
- D-131: `dns_servers=10.12.68.3 allow_dns=false` (nodes resolve via the rack
  forwarder, not MAAS -- the SERVFAIL fix).
- DHCP: VLAN fabric 142/vid 0 `dhcp_on=true primary_rack=nmpcq4`.
Verified BEHAVIORALLY (dc0 lesson -- not the self-report): dhcpd RUNNING on the
rack (`dhcpd -4 ... virbr6`, dhcpd.conf freshly generated); no Temporal
incident (the dc0 region restart fixed it fleet-wide). Revert:
`maas admin vlan update 142 0 dhcp_on=false`; `subnet update 11 dns_servers=
allow_dns=true`; `ipranges delete 3`.

## Item 10 -- dc1 enlistment PROVEN (canary)

`virsh reset vr1-dc1-control-01` -> the node PXE-booted, got DHCP from nmpcq4,
and ENLISTED in MAAS ~2 min later (machine count 11 -> 12). The
DHCP->PXE->enlist chain works end to end for dc1; with the forwarder
pre-installed, dc0's two stacked commissioning faults are pre-mitigated.
Revert: n/a (enlistment; the machine is deleted/re-commissioned as needed).

## Item 11 -- SEC-016 ruling: dedicated dc1 MAAS->libvirt power key (GA-R5)

Commissioning needs the region MAAS snap to SSH the dc1 rack libvirt
(`power_type=virsh`). The dc0 rack authorizes the SEC-012 MAAS key; the dc1
rack does not. Operator RULED (AskUserQuestion, exact utterance): **"Mint a
dedicated dc1 power key"** -- per-DC isolation, NOT cross-DC reuse of SEC-012.
Recorded as **SEC-016** (security-ledger). Wiring (dependent work, next):
mint `vr1-dc1-maas-power_ed25519`; authorize its pubkey on the dc1 rack;
install privkey in the region MAAS snap + snap `ssh config` Host 172.31.0.6;
give the maas-node-power script's virsh reach from voffice1 via the dc1 SERVICE
key (dc0 split). Then `maas-node-power.sh` dc1 (dry -> --commit), commission
9/9. Revert: deauthorize the pubkey on the rack, remove the snap key + config.

## Item 12 -- SEC-016 power key WIRED (dedicated dc1 MAAS->libvirt)

Per the SEC-016 ruling, replicating the dc0 split (script=svc key, MAAS=dedicated
key):
- Minted `~/vr1-dc1-creds/vr1-dc1-maas-power_ed25519` (manifest updated,
  creds-audit CLEAN 7 entries).
- Public half authorized on the dc1 rack jessea123 (2 keys now, matching dc0).
- SCRIPT virsh path: dc1 SERVICE key copied to voffice1 `~/vr1-dc1-creds/` +
  voffice1 `~/.ssh/config` Host 172.31.0.6 -> svc key (voffice1 reaches the rack
  directly on its 172.31.0.5/30 transit leg). Verified: voffice1 virsh lists 9
  dc1 domains.
- MAAS power path: dedicated power PRIVKEY installed in the region MAAS snap
  (`/var/snap/maas/current/root/.ssh/id_dc1_power`, 0600 root) + snap
  `ssh config` Host 172.31.0.6 -> that key (dc0's 172.31.0.2 keeps SEC-012's
  default id_ed25519 -- true per-DC isolation). Revert: deauthorize the pubkey
  on the rack, remove the snap key + Host block, remove the voffice1 svc key +
  Host block.

## Item 13 -- power control set + commissioning 9/9 READY

All 9 dc1 nodes reset -> PXE enlisted (by pinned 52:54:01:d1 MACs; machines
11->20). `maas-node-power.sh --commit` set `power_type=virsh` on all 9, each
verified by a real `query-power-state` -- which PROVES the SEC-016 dedicated
key drives the dc1 rack from the region (2 transient mid-shutdown power=error
cleared on re-query). Region MAAS config: D-120 range + D-131 forwarder DNS +
DHCP primary_rack=nmpcq4. Commissioned all 9 -> **ALL 9 READY in ~3.5 min**
(Commissioning -> Testing -> Ready, NO timeout, NO SERVFAIL), shapes EXACT to
D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), power=virsh
(`docs/audit/dc1-commissioning-verify-20260723.txt`). dc0's two stacked faults
(MAC regen, rack-resolver SERVFAIL) were pre-empted by the pinned MACs (step C)
+ the dc-rack-net forwarder (item 8). Revert: `maas admin machine delete <sid>`
per node; power-off domains.

## Milestone -- G12 [V] leg COMPLETE

dc1 substrate fully built and commissioned: outer vvr1-dc1 + inner 28/28 (prior
sessions) -> edge bootstrapped + addressed (D-112c/D-113a2 on 26.7) -> rack-net
+ region MAAS -> 9/9 nodes READY, shapes exact. G12's remaining leg was the
build; that is done. NEXT: G12 close-out (this changelog consolidated, gauntlet
+ repo-lint, GA-R7 memory review, skill sweep, operator-gated merge to `main`,
branch retirement).
(provider-public `10.12.64.2/22`, metal-admin `10.12.68.2/22`, forwarder
`10.12.68.3` -> region BIND `10.10.0.20`) + harness update, then
`install dc1`; region-side MAAS (metal-admin `dns_servers=10.12.68.3
allow_dns=false`, DHCP `primary_rack=nmpcq4`, D-120 range
`10.12.68.100-.200`); `maas-node-power.sh` dc1 arm; commission 9/9 to Ready
(D-121 Option C); then G12 close-out (gauntlet, repo-lint, consolidation,
memory review, skill sweep, operator-gated merge to `main`).
