Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / archive / changelogs / changelog-20260723-g12-dc1-edge.md
@JANeumatrix JANeumatrix 20 hours ago 13 KB G12 dc1 close-out: consolidation + skill sweep + CURRENT-STATE (G12 CLOSED)
Raw Blame History

2026-07-23 -- G12 [V] leg: dc1 edge bootstrap + v4 addressing (session changelog)

Session scope: resume the G12 [V] leg from the 2026-07-22 close handoff (edge bootstrap onward). Branch dc-dc-g12-dc1-substrate. One changelog per session (GA-R2/D1); every item carries its revert. Live mutations gated (operator ran in manual permission mode -- see Item 0). Edge-phase mutations mostly run UNWRAPPED (approval prompt = the gate); the console bootstrap went through the ~/as-executed/2026-07-22-dc1-deploy.log wrap.

Item 0 -- permission-mode friction (recorded; no repo change)

The session opened in auto mode, whose classifier BLOCKS remote sudo/mutation shapes (matched by the project ssh * sudo * ask rule) instead of surfacing them for approval -- so it cannot deliver "operator approves each mutation." Resolved by switching to manual mode (alias for default; no classifier): allow-rules flow read-only rack probes, project ask rules prompt every gated mutation, deny rules + guard-destructive.py stay active. Session allow-rules for the dc1/dc0 rack ssh shapes were added to .claude/settings.local.json (the operator later broadened to ssh *). Revert: none (session-scoped settings; remove the added allow lines at close if desired).

Item 1 -- dc1 edge service keypair minted + manifest

~/vr1-dc1-creds/vr1-dc1-edge_ed25519 (0600/0644) minted on vcloud, creds-manifests/vr1-dc1.manifest extended (edge key + opnsense-api.txt), creds-audit vr1-dc1 CLEAN (5 entries). Revert: git checkout the manifest; delete the keypair.

Item 2 -- D-112(c) console bootstrap COMPLETE (edge SSH + service key)

Driver d112c-console-dc1.py (dc1 adaptation of the dc0-PROVEN v6, retrieved verbatim from vvr1-dc0:~/d112c-console.py): serial console -> factory root/opnsense -> shell -> ship+run bootstrap PHP (enable ssh, permitrootlogin, install dc1 edge pubkey, write_config) -> materialize PHP (local_user_set(), dc0 lesson v). Transcript ~/d112c-console-dc1.log on the rack: CONFIG-WRITTEN, AK-IN-CONFIG=152, USER-MATERIALIZED; /root/.ssh/authorized_keys (115 B) created (was absent). Edge banner confirms vtnet0 = LAN 192.168.1.1/24, vtnet1 = WAN (dc0 mapping). FINDING (logged, not fixed mid-step): the dc0 d112c-bootstrap.php on the rack required only config.inc, so my first bootstrap PHP threw Call to undefined function shell_safe() at config.inc:311 (in write_config()->make_config_revision_entry()) and aborted BEFORE writing -- this is dc0 lesson (iv) (shell_safe lives in util.inc, measured on dc0's identical 26.7 image). Added require_once("util.inc"); two diagnostic lines made csh-safe (edge root shell is tcsh; 2>&1 = "Ambiguous output redirect"). Re-ran clean. The proven dc0 .b64 bootstrap artifact is missing this include -- a real gap if replayed; queue: fold util.inc into the canonical console bootstrap payload. Revert: config-only; re-run factory reset on the edge to undo (or leave -- it is the intended state).

Item 3 -- key-only SSH to the edge VERIFIED (D-112(c) proof)

Reached the edge via a ProxyCommand chain from vcloud (rack key for the jump, edge key for the final hop -- per-hop keys; a plain -J applies one key to all hops and the rack rejected it). root@192.168.1.1 -> uname -r = 15.1-RELEASE-p1, ifconfig -l = vtnet0/vtnet1/lo0/enc0/pfsync0/pflog0. Second D-112(c) datapoint after dc0.

Item 4 -- D-113(a2) API key MINTED + smoke test PASS

Interim reach leg 192.168.1.2/24 added on the rack provider-public bridge virbr4 (measured this session) to reach the factory edge LAN. Edge-config scripts + edge key staged on the rack (dc0 method; the mint script SSHes edge-direct, no ProxyJump). Key minted via the vendor opnsense-mint-apikey.php (apikeys->add() -- the GUI's own path); key/secret 80 chars, 173 B -> ~/opnsense-api.txt (0600, secret never printed). Smoke test from the rack: GET core/firmware/status -> exit 0, product_abi 26.7. First proof the D-113(a2) API path works on 26.7 for dc1. FINDING (logged): opnsense-bootstrap-apikey.sh's first scp to the edge failed once with scp: Connection closed -- TRANSIENT: the edge sshd was not ready in the moment right after the console bootstrap's configctl openssh restart (both default scp and scp -O succeeded minutes later; the edge is reachable and scp works). Unblocked via an ssh-pipe mint (vendor PHP shipped by ssh 'cat >', retrieved by ssh -n 'cat' > file) -- during which a missing -n on the edge ssh calls let them EAT the rack bash -s heredoc stdin (mint ran, retrieve/cleanup lines were swallowed); re-run with -n retrieved the already-minted key cleanly. Queue: a post-restart sshd readiness wait / one scp retry in opnsense-bootstrap-apikey.sh. Revert: delete the API key via the edge API + remove ~/vr1-dc1-creds/opnsense-api.txt.

Item 5 -- edge v4 addressing (WAN then LAN; reach-handoff)

opnsense-set-interface-v4.sh run VERBATIM from the rack (plain scp works after warm-up; no repo edit). Values confirmed from committed lib-net.sh (provider-public 10.12.64.0/22 gw 10.12.64.1, D-124 amendment) + the D-124 uplink 172.30.3.0/24.

  • WAN first (script ordering rule): dhcp -> 172.30.3.2/24 gw 172.30.3.1, applied + read back on the kernel (vtnet1). Edge egress VERIFIED: ping 1.1.1.1 = 0% loss.
  • Permanent rack leg 10.12.64.2/22 added on virbr4 (so the rack keeps reach after the edge LAN moves).
  • LAN last: 192.168.1.1/24 -> 10.12.64.1/22 (vtnet0). The apply drops the 192.168.1.1 session mid-command (expected; the foreground ssh hung on the dead interface and was TaskStop-ped -- config was already saved+applied). Verified independently at the NEW LAN via the rack's 10.12.64.2 leg: ifconfig vtnet0 = inet 10.12.64.1 netmask 0xfffffc00 (/22); API GET core/firmware/status at 10.12.64.1 -> exit 0, abi 26.7.
  • Interim 192.168.1.2/24 removed; virbr4 now carries only 10.12.64.2/22 (dc0 end-state exactly). Revert: re-address LAN/WAN back via the same script; the leg adds/dels are ip addr add/del (non-persistent).

Item 6 -- creds consolidated + rack edge-key WIPED (SEC-015 closed transient)

API creds pulled rack -> ~/vr1-dc1-creds/opnsense-api.txt (0600, 1 key + 1 secret). Rack edge-key copy shred -u'd; staged edge-config scripts + console driver + rack creds copy removed (ls confirms none remain). The non-secret console transcript ~/d112c-console-dc1.log retained as an audit artifact. SEC-015 opened for the transient edge-key-on-rack exposure (minted->wiped same phase; disclosed). Revert: n/a (cleanup).

Milestone

dc1 edge fully bootstrapped + addressed -- D-112(c) console bootstrap and D-113(a2) REST API both PROVEN on 26.7 (second datapoint after dc0). Edge: WAN 172.30.3.2/24 gw 172.30.3.1 (egress 0% loss), LAN 10.12.64.1/22 (the ruled provider-public gateway), API answering at the new LAN.

Item 7 -- dc-rack-net.sh dc1 site-table arm + harness (D-131 sub-1 delivery)

Added the dc1) arm to scripts/dc-rack-net.sh (the STANDING per-DC forwarder

  • rack-legs pattern), MEASURED/apex-committed values, network-name keyed (no virbrN literal -- T6 holds): vr1-dc1-metal-admin 10.12.68.2/22 rack MAAS/DHCP leg (apex-committed rack IP) vr1-dc1-metal-admin 10.12.68.3/22 node-DNS forwarder listen alias (D-120 static) vr1-dc1-provider-public 10.12.64.2/22 edge-LAN leg (measured live on the bridge 2026-07-23) DNS_LISTEN=10.12.68.3 DNS_UPSTREAM=10.10.0.20 (region BIND over transit). Harness tests/dc-rack-net/ extended T15-T18 (dc1 identity + MEASURED-tag); 18/18, gauntlet ALL GREEN (76), repo-lint 0-fail. This forwarder is the D-131 fix that pre-empts dc0's commissioning SERVFAIL on the isolated rack. Revert: git checkout scripts/dc-rack-net.sh tests/dc-rack-net/run-tests.sh.

Item 8 -- dc-rack-net install on the dc1 rack (persistent legs + forwarder)

install dc1 run on the rack (operator-gated). Check PASS 10/10 (docs/audit/dc1-rack-net-install-20260723.txt): persistent metal-admin legs 10.12.68.2/22 + 10.12.68.3/22 on virbr6, provider-public 10.12.64.2/22 on virbr4, dc1-rack-legs + dc1-node-dns enabled+active. Behavioral proof: the forwarder answers authoritative maas-internal SOA via region BIND -- the D-131 fix that pre-empts dc0's commissioning SERVFAIL. Revert: dc-rack-net.sh install is idempotent; to undo, disable the two units + remove the generated files + the interim legs.

Item 9 -- region-side MAAS config for dc1 metal-admin (DHCP + DNS + range)

MAAS auto-discovered dc1 planes from the rack interfaces: metal-admin 10.12.68.0/22 = subnet id 11 (VLAN fabric 142/vid 0), provider-public 10.12.64.0/22 = subnet id 10. On subnet 11 (all operator-gated):

  • D-120 dynamic range 10.12.68.100-10.12.68.200 created (iprange id 3) -- the ruled band applied to dc1's CIDR.
  • D-131: dns_servers=10.12.68.3 allow_dns=false (nodes resolve via the rack forwarder, not MAAS -- the SERVFAIL fix).
  • DHCP: VLAN fabric 142/vid 0 dhcp_on=true primary_rack=nmpcq4. Verified BEHAVIORALLY (dc0 lesson -- not the self-report): dhcpd RUNNING on the rack (dhcpd -4 ... virbr6, dhcpd.conf freshly generated); no Temporal incident (the dc0 region restart fixed it fleet-wide). Revert: maas admin vlan update 142 0 dhcp_on=false; subnet update 11 dns_servers= allow_dns=true; ipranges delete 3.

Item 10 -- dc1 enlistment PROVEN (canary)

virsh reset vr1-dc1-control-01 -> the node PXE-booted, got DHCP from nmpcq4, and ENLISTED in MAAS ~2 min later (machine count 11 -> 12). The DHCP->PXE->enlist chain works end to end for dc1; with the forwarder pre-installed, dc0's two stacked commissioning faults are pre-mitigated. Revert: n/a (enlistment; the machine is deleted/re-commissioned as needed).

Item 11 -- SEC-016 ruling: dedicated dc1 MAAS->libvirt power key (GA-R5)

Commissioning needs the region MAAS snap to SSH the dc1 rack libvirt (power_type=virsh). The dc0 rack authorizes the SEC-012 MAAS key; the dc1 rack does not. Operator RULED (AskUserQuestion, exact utterance): "Mint a dedicated dc1 power key" -- per-DC isolation, NOT cross-DC reuse of SEC-012. Recorded as SEC-016 (security-ledger). Wiring (dependent work, next): mint vr1-dc1-maas-power_ed25519; authorize its pubkey on the dc1 rack; install privkey in the region MAAS snap + snap ssh config Host 172.31.0.6; give the maas-node-power script's virsh reach from voffice1 via the dc1 SERVICE key (dc0 split). Then maas-node-power.sh dc1 (dry -> --commit), commission 9/9. Revert: deauthorize the pubkey on the rack, remove the snap key + config.

Item 12 -- SEC-016 power key WIRED (dedicated dc1 MAAS->libvirt)

Per the SEC-016 ruling, replicating the dc0 split (script=svc key, MAAS=dedicated key):

  • Minted ~/vr1-dc1-creds/vr1-dc1-maas-power_ed25519 (manifest updated, creds-audit CLEAN 7 entries).
  • Public half authorized on the dc1 rack jessea123 (2 keys now, matching dc0).
  • SCRIPT virsh path: dc1 SERVICE key copied to voffice1 ~/vr1-dc1-creds/ + voffice1 ~/.ssh/config Host 172.31.0.6 -> svc key (voffice1 reaches the rack directly on its 172.31.0.5/30 transit leg). Verified: voffice1 virsh lists 9 dc1 domains.
  • MAAS power path: dedicated power PRIVKEY installed in the region MAAS snap (/var/snap/maas/current/root/.ssh/id_dc1_power, 0600 root) + snap ssh config Host 172.31.0.6 -> that key (dc0's 172.31.0.2 keeps SEC-012's default id_ed25519 -- true per-DC isolation). Revert: deauthorize the pubkey on the rack, remove the snap key + Host block, remove the voffice1 svc key + Host block.

Item 13 -- power control set + commissioning 9/9 READY

All 9 dc1 nodes reset -> PXE enlisted (by pinned 52:54:01:d1 MACs; machines 11->20). maas-node-power.sh --commit set power_type=virsh on all 9, each verified by a real query-power-state -- which PROVES the SEC-016 dedicated key drives the dc1 rack from the region (2 transient mid-shutdown power=error cleared on re-query). Region MAAS config: D-120 range + D-131 forwarder DNS + DHCP primary_rack=nmpcq4. Commissioned all 9 -> ALL 9 READY in ~3.5 min (Commissioning -> Testing -> Ready, NO timeout, NO SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), power=virsh (docs/audit/dc1-commissioning-verify-20260723.txt). dc0's two stacked faults (MAC regen, rack-resolver SERVFAIL) were pre-empted by the pinned MACs (step C)

  • the dc-rack-net forwarder (item 8). Revert: maas admin machine delete <sid> per node; power-off domains.

Milestone -- G12 [V] leg COMPLETE

dc1 substrate fully built and commissioned: outer vvr1-dc1 + inner 28/28 (prior sessions) -> edge bootstrapped + addressed (D-112c/D-113a2 on 26.7) -> rack-net

  • region MAAS -> 9/9 nodes READY, shapes exact. G12's remaining leg was the build; that is done. NEXT: G12 close-out (this changelog consolidated, gauntlet
  • repo-lint, GA-R7 memory review, skill sweep, operator-gated merge to main, branch retirement). (provider-public 10.12.64.2/22, metal-admin 10.12.68.2/22, forwarder 10.12.68.3 -> region BIND 10.10.0.20) + harness update, then install dc1; region-side MAAS (metal-admin dns_servers=10.12.68.3 allow_dns=false, DHCP primary_rack=nmpcq4, D-120 range 10.12.68.100-.200); maas-node-power.sh dc1 arm; commission 9/9 to Ready (D-121 Option C); then G12 close-out (gauntlet, repo-lint, consolidation, memory review, skill sweep, operator-gated merge to main).