diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index cbf7fd6..a56c105 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -475,7 +475,7 @@ | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | | G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | -| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). NEXT (gated): inner apply FROM voffice1 (`opentofu/vr1-dc1-substrate/`, 54 MACs pre-pinned), then D-125 egress gate, edge bootstrap (D-112(c)/D-113(a2) on 26.7), rack standup DoD (dc-rack-net install dc1, forwarder 10.12.68.3, region-side DHCP, maas-node-power) | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). NEXT (gated): D-125 egress gate on br-vr1-dc1-wan, edge bootstrap (D-112(c)/D-113(a2)) + v4 addressing (WAN 172.30.3.2, LAN 10.12.64.1/22), rack standup DoD (dc-rack-net dc1 + forwarder 10.12.68.3, region-side DHCP primary_rack nmpcq4 + D-120 range, maas-node-power) | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | OPEN (execution only; decision content complete) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | diff --git a/docs/audit/inner-converge-20260722-dc1-stepC.txt b/docs/audit/inner-converge-20260722-dc1-stepC.txt new file mode 100644 index 0000000..020c88e --- /dev/null +++ b/docs/audit/inner-converge-20260722-dc1-stepC.txt @@ -0,0 +1,2 @@ +28 +No changes. Your infrastructure matches the configuration. diff --git a/docs/audit/inner-plan-20260722-dc1-stepC.txt b/docs/audit/inner-plan-20260722-dc1-stepC.txt new file mode 100644 index 0000000..328e574 --- /dev/null +++ b/docs/audit/inner-plan-20260722-dc1-stepC.txt @@ -0,0 +1,1678 @@ + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + + create + +OpenTofu will perform the following actions: + + # module.inner_storage.libvirt_pool.dc will be created + + resource "libvirt_pool" "dc" { + + allocation = (known after apply) + + available = (known after apply) + + capacity = (known after apply) + + id = (known after apply) + + name = "vr1-dc1-inner-pool" + + target = { + + path = "/var/lib/libvirt/vr1-dc1-inner" + } + + type = "dir" + + uuid = (known after apply) + } + + # module.vr1_dc1_node["vr1-dc1-compute-01"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-compute-01-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:04:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-compute-01-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 49152 + + memory_unit = "MiB" + + name = "vr1-dc1-compute-01" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 12 + } + + # module.vr1_dc1_node["vr1-dc1-compute-01"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 107374182400 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-compute-01-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-compute-02"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-compute-02-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:05:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-compute-02-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 49152 + + memory_unit = "MiB" + + name = "vr1-dc1-compute-02" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 12 + } + + # module.vr1_dc1_node["vr1-dc1-compute-02"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 107374182400 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-compute-02-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-control-01"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-control-01-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:01:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-control-01-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 65536 + + memory_unit = "MiB" + + name = "vr1-dc1-control-01" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 16 + } + + # module.vr1_dc1_node["vr1-dc1-control-01"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 161061273600 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-control-01-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-control-02"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-control-02-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:02:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-control-02-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 65536 + + memory_unit = "MiB" + + name = "vr1-dc1-control-02" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 16 + } + + # module.vr1_dc1_node["vr1-dc1-control-02"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 161061273600 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-control-02-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-control-03"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-control-03-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:03:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-control-03-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 65536 + + memory_unit = "MiB" + + name = "vr1-dc1-control-03" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 16 + } + + # module.vr1_dc1_node["vr1-dc1-control-03"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 161061273600 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-control-03-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-01"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-01-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:06:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-01-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-01" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-01"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-01-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-02"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-02-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:07:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-02-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-02" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-02"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-02-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-03"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-03-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:08:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-03-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-03" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-03"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-03-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-04"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-04-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:09:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-04-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-04" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-04"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-04-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_opnsense.libvirt_domain.vm will be created + + resource "libvirt_domain" "vm" { + + autostart = true + + cpu = { + + features = [ + + { + + name = "svm" + + policy = "disable" + }, + ] + + mode = "host-passthrough" + } + + devices = { + + channels = [ + + { + + source = { + + unix = { + + mode = "bind" + } + } + + target = { + + virt_io = { + + name = "org.qemu.guest_agent.0" + } + } + }, + ] + + disks = [ + + { + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-opnsense-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-wan" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-opnsense-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 2048 + + memory_unit = "MiB" + + name = "vr1-dc1-opnsense" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "pc" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 2 + } + + # module.vr1_dc1_opnsense.libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = (known after apply) + + create = { + + content = { + + url = "/home/jessea123/vr1-dc1-images/opnsense-26.7-nano.qcow2" + } + } + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-opnsense-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_planes.libvirt_network.plane["data-tenant"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "data-tenant.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-data-tenant" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["metal-admin"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "metal-admin.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-metal-admin" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["metal-internal"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "metal-internal.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-metal-internal" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["provider-public"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "provider-public.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-provider-public" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["replication"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "replication.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-replication" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["storage"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "storage.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-storage" + + uuid = (known after apply) + } + + # module.vr1_dc1_wan.libvirt_network.wan_bridge will be created + + resource "libvirt_network" "wan_bridge" { + + autostart = true + + bridge = { + + name = "br-vr1-dc1-wan" + } + + forward = { + + mode = "bridge" + } + + id = (known after apply) + + name = "vr1-dc1-wan" + + uuid = (known after apply) + } + +Plan: 28 to add, 0 to change, 0 to destroy. + +───────────────────────────────────────────────────────────────────────────── + +Saved the plan to: dc1-inner-20260722.tfplan + +To perform exactly these actions, run the following command to apply: + tofu apply "dc1-inner-20260722.tfplan" diff --git a/docs/changelog-20260722-g12-dc1-build.md b/docs/changelog-20260722-g12-dc1-build.md index 49a1c0e..da50a3f 100644 --- a/docs/changelog-20260722-g12-dc1-build.md +++ b/docs/changelog-20260722-g12-dc1-build.md @@ -147,11 +147,45 @@ (`3981821e3a3c...476627d`). Same bits that passed the D-112 boot path on dc0. Revert: delete `/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2`. +## Item 9 -- inner apply: dc1 substrate step C EXECUTED (from voffice1, D-128 Plane 2) + +Staging on voffice1 (dc0-precedent set): repo clone fetched + switched from the +retired stage-3 branch to `dc-dc-g12-dc1-substrate` @ 61c416e (dc0 inner +tfstate untouched, verified); dc1 private key piped in (0600, 432 bytes by +count); 172.31.0.6 host keys scanned into known_hosts (all 3 types -- dc0 +trap); the 26.7 base copied rack->voffice1 to `~/vr1-dc1-images/` and +sha256-verified (provider streams volume content itself -- the dc0 measured +fix; rack-side copy satisfies only the bootstrap check). Inner tfvars +`d124-inner.auto.tfvars` written on voffice1, every value sourced (measured +transit .6, outer-committed planes map verbatim, Stage-1 mtu 9000, D-106 +suffix, keyfile + local base paths). + +Plan `dc1-inner-20260722.tfplan` = **28/0/0** (dc0's exact step-C count), +verified in-capture: 54 pinned `52:54:01:d1:*` MACs, 9 Option-C nodes + edge, +qga channel present (`docs/audit/inner-plan-20260722-dc1-stepC.txt`). First +apply FAILED at 18/28: the edge domain's serial-log dir +`/var/lib/libvirt/vr1/staging/` did not exist inside vvr1-dc1 (hand-created +on dc0 during the 07-20 serial work; QUEUED finding -- belongs in the +bootstrap/standup DoD so dc2+ does not repeat it). Fix-forward per the +rollback tree: dir created (root:root 0755, dc0 mirror), resume plan 10/0/0, +**apply exit 0**. Final: 28/28 in state, convergence ZERO DIFF +(`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING +inside vvr1-dc1**, edge serial log at the FreeBSD login prompt -- OPNsense +26.7, factory LAN 192.168.1.1/24, WAN unaddressed (second D-112 boot-path +datapoint on 26.7; exactly dc0's post-step-C state). The dc1 INNER tfstate +lives ON voffice1 (`opentofu/vr1-dc1-substrate/terraform.tfstate` -- add to +the site backup set, same as dc0's). + +Revert: `dc-dc-teardown-rollback.md` tree; inner root destroy from voffice1 +tears down the 28 (containment VM unaffected). + ## Next (gated, not run here) -Inner apply FROM voffice1 (D-128 Plane 2; `opentofu/vr1-dc1-substrate/`, -54 MACs pre-pinned) after the step-B `--check` re-run is EXIT 0; then the -D-125 egress isolation gate on `br-vr1-dc1-wan`, edge bootstrap (D-112(c) + -D-113(a2) on 26.7), rack standup DoD (dc-rack-net.sh install dc1, forwarder -10.12.68.3, region-side DHCP on metal-admin, maas-node-power). Runbook + +D-125 egress isolation gate on `br-vr1-dc1-wan` (throwaway guest, two +identical runs); edge bootstrap (D-112(c) console -> key-only SSH -> +D-113(a2) API key on 26.7); edge addressing via `opnsense-set-interface-v4` +(WAN 172.30.3.2/24 gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); +rack standup DoD (dc-rack-net.sh install dc1 + forwarder 10.12.68.3, +region-side DHCP on metal-admin naming nmpcq4 primary_rack, dynamic range +10.12.68.100-.200 per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern.