# Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap)

Session changelog (GA-R2/D1: ONE per session). Logged window: `dc0-deploy`
(same-day append, per the A/B-boundary handoff). Status lives ONLY in
`docs/CURRENT-STATE.md`; this file is narrative + reverts.

## 1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124)

- **Question as presented (2026-07-19):** "Who should hold 172.31.0.1 (the
  Office1 end of the dc0 transit /30), and by what mechanism? This is the
  prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS
  rack-to-region enrollment path."
- **Operator answer, exact utterance (option selected):** "voffice1 via IaC +
  restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to
  `module "voffice1"` in the outer root; apply in a planned voffice1 restart
  window (graceful shutdown first; region/NetBox/Tailscale return via proven
  autostart).
- Classification: OPS mechanism decision implementing D-124's already-ruled
  addressing ("region/peer .1" -- `docs/design-decisions.md:3727`); no new
  D-number (GA-R3, doubt resolves DOWN).
- **Measured basis (this session, logged):** mesh-vr1-dc0-office1 (virbr7) is
  ISOLATED (no `<forward>`/`<ip>`) with exactly ONE attached interface
  (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing
  holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only
  route `10.10.0.0/22 via 172.31.0.1` (d124-rack.auto.tfvars, consumed by
  step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached
  by `-J voffice1` (D-126 shape) -- NO vcloud host leg / site-baseleg DC row
  needed for this net.

## 2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated)

- **What:** `opentofu/main.tf` `module "voffice1"`: `network_names` gains
  `module.mesh_vr1_dc0_office1.network_name` SECOND (office1-local stays
  first -- PCI order preserves enp1s0); `network_config` rewritten from the
  single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4;
  transit=enp2s0 static `${vr1_dc0_rack_transit_peer_ip}/${prefix}` -- same
  D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init-
  re-run caveats documented in-block: the seed netplan is REBUILD correctness;
  the live transit config is an in-guest netplan drop-in at the gated attach
  step.
- **Why:** section 1's measured gap -- step B (ssh reach + rack enrollment)
  has no L3 path until the region end of the transit exists.
- **Revert:** restore `network_names = [module.office1_network.network_name]`
  and the previous "en*" glob `network_config` (git revert of this commit);
  if already applied, a follow-up plan shows the interface removal and the
  in-guest drop-in `/etc/netplan/60-transit.yaml` is deleted.

## 3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation

- Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply
  (1/1/1 exact, `docs/audit/outer-apply-20260719-voffice1-transit.txt`) ->
  domain up with both NICs, office1-local index 0.
- **INCIDENT (found+fixed in-window):** the provider REGENERATED both NIC
  MACs on the interface-list rebuild (old lan MAC `52:54:00:6a:87:e5` ->
  new `52:54:00:89:e1:19`). The Kea reservation (MAC-keyed) missed and
  voffice1 leased pool address `.108` (measured via edge ARP). FIX: re-keyed
  reservation uuid `b12621e6-63e9-4714-9726-2e130c84e069` to the new MAC via
  the D-113 API (`kea/dhcpv4/set_reservation` -> saved;
  `kea/service/reconfigure` -> ok), then a detached
  `networkctl reconfigure enp1s0` -> voffice1 back on `10.10.0.20`.
- **LOGGED FINDING (not executed):** `modules/cloudinit-vm` does not pin
  MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed
  DHCP reservations. Roosevelt-relevant; candidate module `mac` var or a
  documented ops step. Also: live transit config landed as in-guest netplan
  drop-in `/etc/netplan/60-transit.yaml` (narrows the lan glob to enp1s0 +
  transit enp2s0 static) -- the drop-in must ALSO narrow `lan`, else the
  50-cloud-init `en*` glob claims enp2s0 (measured: first drop-in attempt
  without it left enp2s0 addressless).
- **Revert:** `kea/dhcpv4/set_reservation` back to the old MAC (senseless
  unless the NIC change is also reverted); delete the drop-in.

## 4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix

- Symptom: `172.31.0.2` unreachable; MEASURED: both rack NICs emitted ZERO
  frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive
  (ACPI-responsive; cloud-init completed per offline logs).
- Guest is a SEALED BOX: `modules/cloudinit-vm` domains have NO serial
  console, NO qga channel, NO graphics; qga is NOT in the noble base image
  and its cloud-init install failed (no egress) -- so no console, no agent,
  no network. **LOGGED FINDING: add opt-in serial console (+ the D-129-style
  qga channel var) to `modules/cloudinit-vm` -- observability gap bites every
  future DC VM. NOT executed mid-step.**
- Offline inspection (libguestfs, operator-run, captures
  `~/vvr1-dc0-offline-inspect{,2}.txt`): NIC names enp1s0/enp2s0 CORRECT
  (naming trap closed); root cause = netplan `match.name` + `set-name`:
  boot 1 renames don't apply (.link written after udev add) so the rendered
  config matched nothing; after removing set-name, boot 2 STILL failed
  because the early-boot netplan generator ran against the OLD config and
  renamed the devices to mgmt/uplink before cloud-init wrote the new file
  (one-boot lag, measured in syslog). Boot 3 (current config, no renames)
  is the verification boot.
- FIX (this commit + applied): `network_config` drops set-name, keeps
  MEASURED kernel names; `instance-id` bumped `vvr1-dc0-d123 -> -d123-r2`
  (side effects noted in-block); seed replaced via explicit
  `-replace=module.vvr1_dc0.libvirt_volume.seed` (the sanctioned override
  of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2
  exact (`docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt`).
- CONSEQUENCE for step B: bootstrap runs with
  `--transit-if enp1s0 --uplink-if enp2s0` (script defaults mgmt/uplink no
  longer match reality); runbook prose naming "mgmt" needs a DOCFIX at
  stage close.
- **Revert:** git revert of the netplan/instance-id edit + one more
  seed `-replace` apply + reboot.

## 4b. VERIFICATION (2026-07-20, logged window)

- Boot 3 with the fixed config: `enp1s0 UP 172.31.0.2/30`, region route
  present, `br-vr1-dc0-wan` up.
- voffice1 -> 172.31.0.2 ping 0% loss; ssh
  `-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2`
  works (D-126 -J shape, per-env key); rack -> region `ping 10.10.0.20`
  0% loss (the MAAS-enrollment path for the G10 bootstrap).
- Outer plan CONVERGED back to ZERO DIFF
  (`docs/audit/outer-plan-20260720-postfix-converged.txt`); CURRENT-STATE
  section 5 updated in the same commit.
- Follow-items (logged, not executed): D-126 DC ssh-config alias row for
  vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key);
  cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC
  pinning; DOCFIX for runbook "mgmt/uplink" naming.

## 4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window)

- Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to
  archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped
  answer to the rack's no-egress bootstrap; apt used the same proxy via
  per-invocation env. Enrollment secret piped region->rack
  (`install -m600 /dev/stdin`), verified by byte-count only (32), never
  in context.
- Run 1 (`docs/audit/stepB-bootstrap-20260720.txt`): maas snap 3.7/stable
  installed, **rack ENROLLED to http://10.10.0.20:5240/MAAS**; FAILED at
  qemu apt install on stale image package lists (404s via proxy).
  Fix: `apt-get update` (proxied), re-run.
- Run 2 (`docs/audit/stepB-bootstrap-20260720-run2.txt`): EXIT 0 --
  libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010
  FORWARD-drop loaded on enp1s0 + `sec010-fw.service` enabled
  (boot-persistent), `br-vr1-dc0-wan` verified with enp2s0 enslaved.
- `--check` (`docs/audit/stepB-check-20260720.txt`): all items ok EXCEPT
  opnsense base (the remaining step-B item); rackd running, enrolled.
- Region-side residue for the MAAS phase (per script exit text): enable
  DHCP on metal-admin at the Office1 region naming this rack primary_rack
  (step D territory, NOT done).
- **Revert:** rack enrollment: region-side `maas ... rack-controllers
  delete`; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft;
  packages: apt/snap remove; snap proxy: `snap unset system proxy.http
  proxy.https`.

## 4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20)

- **Question as presented:** reuse verified 26.1 artifact vs fresh download
  (first exchange); then, sharpened after the operator noted "Office1 is
  running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base.
- **Operator answer, exact utterance (option selected):** "Fresh 26.7 nano
  base".
- Consequences implemented: prep runs 26.7 -> `opnsense-26.7-nano.qcow2`;
  `vr1-dc0-substrate` `opnsense_base_path` default updated (no inner state
  exists; pre-apply edit); step-B `--check` passes `--opnsense-base` for
  the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1
  default + hint strings (script change + harness, stage close); the
  D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C
  boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7).
- Mirror: `https://mirror.ams1.nl.leaseweb.net/opnsense` -- MEASURED from
  the vendor's live download page this session (not memory), HEAD-verified
  serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh
  performs no checksum verification of the downloaded image (same property
  as the original 26.1 prep; candidate hardening).

## 4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends

- OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy,
  468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184;
  `docs/audit/stepB-opnsense-prep-20260720.txt`). Inner-root
  `opnsense_base_path` default -> 26.7 (validate PASS).
- Final `--check` EXIT 0, ALL items ok
  (`docs/audit/stepB-check-20260720-final.txt`).
- SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since
  this session wired voffice1's transit leg; voffice1 runs LXD so
  ip_forward=1): identical scoped nft artifact + sec010-fw.service on
  enp2s0, verified loaded + persistent; rack proxy/region paths re-verified
  working post-pin. **SEC-010 row CLOSED (operator ruling, utterance in the
  row).** Open SEC count 8 -> 7.
- LOGGED follow-item: site-headend-install.sh region-role support for the
  transit-if pin (voffice1's was hand-applied to the script's exact
  artifact shape; fold into the script + harness at stage close).
- **Revert:** rm the two files + `systemctl disable sec010-fw` on voffice1;
  `nft delete table inet sec010`; ledger row reopens citing this item.

## 7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28)

- voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt
  source (keyrings byte-copied; version == section-7 pin); repo cloned at
  `29cf7bf` (branch); dc0 private key staged to `~/vr1-dc0-creds/`
  (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config
  entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the
  ED25519 fingerprint cross-checked against the accept-new-trusted entry
  (MATCH).
- Inner-root fixes measured at the FIRST REAL dial/apply (each committed):
  (i) provider needs `keyfile`+`sshauth=privkey` URI params -- its Go ssh
  ignores ~/.ssh/config AND default identities; single-type known_hosts
  fails as "key mismatch" (multi-type scan required);
  (ii) `modules/wan-bridge`: `<mtu>` is ILLEGAL in bridge-mode networks --
  removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never
  applied);
  (iii) base-image location premise WRONG under remote provider -- content
  UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1
  (`~/vr1-dc0-images/`), tfvar override + variables.tf description
  corrected;
  (iv) edge serial-log dir `/var/lib/libvirt/vr1/staging/` is a HARDCODED
  vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack
  (755); LOGGED findings: parameterize the path; fold staging-dir creation
  into site-headend-install --host-nodes; opentofu-validate.sh does not
  cover the inner root.
- Applies (captures `docs/audit/inner-{plan,apply}-20260720-stepC*.txt`):
  run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan
  bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort;
  provider self-cleaned the failed domain -- operator undefine found
  nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff
  (`docs/audit/inner-converge-20260720-stepC.txt`).
- VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge);
  edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112
  serial path (first revalidation datapoint; full D-112(c) bootstrap is the
  next window).
- NEW STATE OF RECORD: the inner tfstate lives on voffice1
  (`~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate`)
  -- back it up with the site; follow-item for the backup set.
- **Revert:** `tofu destroy` in the inner root from voffice1 (site-down
  alternative: `virsh destroy vvr1-dc0`, D-122); module fixes revert by
  commit.

## 9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN

- Per-env edge key minted on vcloud (`~/vr1-dc0-creds/vr1-dc0-edge_ed25519`,
  0600/0644, SEC-009 convention), staged to the rack the same way.
- Rack legs added (operator-ruled "10.12.4.2, manual now + script later"):
  `10.12.4.2/22` on the provider-public inner bridge (virbr5, MEASURED) +
  the D-124-ruled `10.12.8.2/22` on metal-admin (virbr2). INTERIM: `ip addr
  add`, not persistent. **LOGGED FINDING (queued):** site-headend-install.sh
  never implemented the rack legs despite main.tf's comment claiming it
  consumes `vr1_dc0_rack_metal_admin_ip` -- add rack-leg support + harness
  before step D closes. Also interim: `192.168.1.2/24` on virbr5 to reach
  the edge's FACTORY LAN; REMOVE once the edge is re-addressed.
- **D-112(c) console bootstrap: DONE** (`~/d112c-console.log` on the rack).
  Driver evolved v1->v6 against MEASURED failures, each real and worth
  keeping for the runbook:
  (i) serial console wraps long lines -> ship payload as <=160-char base64
  chunks (`openssl base64 -d`), never one long line;
  (ii) `set-name` in the edge netplan is irrelevant here but the same class
  bit the rack (item 4);
  (iii) tcsh on the edge (documented trap) breaks nested-quoted `sed` --
  "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship
  the whole payload instead;
  (iv) `write_config()` needs an include beyond config.inc -- MEASURED
  (`grep -l shell_safe /usr/local/etc/inc/*.inc` -> auth/console/filter/
  interfaces/rrd/system/util.inc), not guessed;
  (v) writing config.xml does NOT materialize `/root/.ssh/authorized_keys`
  -- must call the vendor's `local_user_set()` (auth.inc). This was the
  actual reason key auth failed after a "successful" config write.
- VERIFIED: `ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1` -> key-only auth
  OK, `uname -r` = 15.1-RELEASE-p1, `ifconfig -l` = vtnet0/vtnet1.
  INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it
  answered on virbr5), vtnet1 = WAN -- same order as the office1 edge.
- **API key minted via the vendor model** (`opnsense-bootstrap-apikey.sh`
  + `opnsense-mint-apikey.php`, run ON the rack): key/secret 80 chars ->
  `~/vr1-dc0-creds/opnsense-api.txt` (0600, secret never printed).
  SMOKE TEST PASS: `GET core/firmware/status` -> HTTP 200, `CORE_ABI 26.7`.
  **This is the first proof the D-113(a2) API path works on 26.7** (the
  26.7 revalidation the operator's base-version ruling called for).
  Script note: the bootstrap script requires the minter BESIDE it (its
  own dir), not in `scripts/` -- worth a usage-line DOCFIX.
- **Revert:** delete the API key (`POST auth/user/del_api_key/<id>`, the
  proven negative-tested path) + creds file; `ip addr del` the three
  interim addresses; the console bootstrap reverts by re-imaging the edge
  (inner `tofu destroy`/re-apply of the edge module).

## 10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN,
## upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked.

- Method: operator-ruled "throwaway VM, as written" -- noble cloud image
  fetched ON the rack via the region proxy, one cloud-init guest, SINGLE
  NIC bridged to `br-vr1-dc0-wan`, serial-log probe, then destroyed
  (`leftover-domains=0` asserted). Script: scratchpad
  `d125-egress-test.sh`; capture `docs/audit/d125-egress-gate-20260720.txt`.
- Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and
  came up address-less. MEASURED cause: `vr1-dc0-uplink` carries an `<ip>`
  but NO `<dhcp>` block -- there is no DHCP server on that /24 by design
  (the edge WAN takes a STATIC .2, D-113/D-125). **DOCFIX QUEUED:** the
  gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say
  STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24.
- RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1;
  **gateway ping 3/3, 0% loss** -- a FOREIGN MAC inside vvr1-dc0 reaches
  the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in
  property, and it PASSES. But `ping 1.1.1.1` FAILED (NET-PING-RC=1).
- Follow-up measurement (rack bridge, 172.30.2.51, self-cleaning): ICMP
  AND TCP (`curl http://1.1.1.1` -> 000) both fail past the gateway, so it
  is not ICMP-specific. Controls: vcloud egress 0% loss; the office1 edge
  egresses through the SAME-SHAPE `office1-wan` NAT 0% loss. The two
  network definitions are byte-equivalent in substance (`net-dumpxml`:
  both `forward mode=nat` + `<nat><port 1024-65535>`; only netmask-vs-
  prefix notation differs) -- so the delta is in vcloud's LIVE firewall
  rules for virbr4, NOT in the design or the config.
- **Gate status: OPEN (not FAIL).** The runbook's FAIL remedy (revert to
  double-NAT) is for a DESIGN failure; invoking it against what is very
  likely a host firewall-rule gap would be the wrong remedy and would add
  Roosevelt delta. Next step is a read-only measurement needing operator
  sudo: `sudo nft list ruleset | grep -nE "172\.30\.(1|2)\.0/24"` (or
  `sudo iptables -t nat -S | grep 172.30`) comparing the WORKING virbr11
  rules against virbr4's.
- **Revert:** nothing to revert -- the probe guest and both temporary
  addresses were removed; the only residue is `/tmp/noble.img` on the rack
  (kept deliberately: it is the base for any future throwaway probe).

## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review

- Operator directive (verbatim intent): at the proper time/place, review the
  OPNsense 26.7 changelog + documentation for upgraded features and changes;
  incorporate beneficial improvements (esp. hardening) into the next
  deployment iteration, following project rules.
- Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued
  DOCFIXes). Deliverable: a findings doc citing the official 26.7 release
  notes; actionable items route per GA-R3 -- hardening/profile deltas as
  PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely
  surface), operational items as runbook DOCFIXes. NO self-executing
  incorporation: each finding is ruled per GA-R5 (one per exchange) before
  touching any built surface. Target of incorporation: dc1 standup +
  Roosevelt runbooks; live-edge retrofits individually gated.

## 6. ledger-scan hardening: FAIL-OPEN is terminology, not a status

- SEC-010's close surfaced a scanner false positive: the OPEN-row detector
  (`awk toupper(cell) ~ /OPEN|PENDING/`) matched the substring OPEN inside
  "fail-open" in the closed row's disposition -- the row counted OPEN
  forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from
  the cell before matching (`scripts/ledger-scan.sh`); harness gains a
  CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified:
  7 open, SEC-010 absent.
- **Revert:** git revert of this commit's ledger-scan.sh + test hunks.

## 5. Host tooling: libguestfs-tools installed on vcloud (operator-run)

- `sudo apt-get install -y libguestfs-tools` (1:1.52.0-5ubuntu3) -- enables
  offline guest inspection (virt-cat/virt-ls), used for items 4's captures.
  Standard KVM-host tooling; Roosevelt-transferable.
- **Revert:** `sudo apt-get remove libguestfs-tools`.