diff --git a/docs/changelog-20260712-office1-opnsense-edge-build.md b/docs/changelog-20260712-office1-opnsense-edge-build.md new file mode 100644 index 0000000..7256a7e --- /dev/null +++ b/docs/changelog-20260712-office1-opnsense-edge-build.md @@ -0,0 +1,52 @@ +# Changelog 2026-07-12 -- DOCFIX-186: Office1 OPNsense edge build + two first-boot infra findings + +**Context.** Standing up the Office1 OPNsense edge so `office1-local` has a router + DHCP +(operator: bring the edge online before the Office1 NetBox VM). First-ever OPNsense boot in +this repo -- reviewed-but-unexercised module + image chain. This records the repo changes and +the two real infrastructure findings hit on the first apply. + +## Repo change +- **`opentofu/main.tf`**: `module "office1_opnsense"` instantiated (was a commented skeleton): + `modules/opnsense-edge`, 2 vCPU / 2 GiB / 16 GiB, `pool_name = module.office1_storage.pool_name`, + `base_volume_path` = the prepped nano qcow2, `config_iso_path` = the staged config ISO, + `lan_network_name = module.office1_network.network_name`, `wan_network_name = "office1-wan"`. + Validated + planned clean (first real validation of `modules/opnsense-edge`): 3 to add. + +## Infrastructure created (not repo files) +- **`office1-wan`** NAT network (`172.30.1.0/24` -> `enp1s0`), created via `virsh` -- + Office1's per-site simulated-ISP uplink (operator's per-site-uplink ruling). **Logged as + D-103 debt** to formalize into an OpenTofu module later (same class as the retained `wan`). +- OPNsense creds generated + saved 0600 to `~/vr1-office1-creds/` (jumphost-only, NEVER + committed): SSH keypair + root password + its bcrypt hash (for `config.xml`). + +## Two first-boot findings (both real, both DOCFIX-worthy) +1. **`config_iso_path` must live OUTSIDE the pool dir.** `libvirt_volume.config_seed` uses + `create.content.url` to COPY the ISO into the pool as a volume named + `office1-opnsense-config.iso`; if the source ISO sits at that exact pool path, libvirt + errors "volume target path ... already exists". Fixed by staging the ISO at + `/var/lib/libvirt/vr1/staging/` and pointing `config_iso_path` there. (The disk uses + `backing_store` -- a reference, not a copy -- so the in-pool nano image is fine.) +2. **apparmor blocks the custom pool path `/var/lib/libvirt/vr1/`.** The domain was defined + but failed to start: qemu (`libvirt-qemu`) got "Permission denied" on the backing image + despite correct POSIX perms. libvirt's apparmor abstraction only whitelists + `/var/lib/libvirt/images/**` + `.../qemu/**`, not our operator-owned `vr1/` pool tree. + **This is foundational -- every VR1 VM lives under `/var/lib/libvirt/vr1/`.** Fix (operator, + one-time sudo), via the abstraction's `include if exists `: + ``` + echo '/var/lib/libvirt/vr1/** rwk,' | sudo tee /etc/apparmor.d/local/abstractions/libvirt-qemu + sudo systemctl reload apparmor && sudo systemctl restart libvirtd + ``` + **PREREQ:** this belongs in the workstation/host prep (a candidate `scripts/prereqs/` + addition + a phase-0/phase-1 runbook note) since it gates every VR1 VM boot, not just this + edge. Not yet actioned as a prereq installer (needs sudo). + +## State +- Edge build COMPLETE except the boot: image, `office1-wan`, creds, config, ISO, module + instantiation, disk + config_seed volumes all done. Domain boot is blocked ONLY on the + apparmor fix above (operator sudo). After it: re-apply to boot, then verify WAN (`172.30.1.2`) + + LAN DHCP/routing on `office1-local`. + +## Revert +- `git checkout opentofu/main.tf` (re-comment the module); `virsh net-destroy/undefine + office1-wan`; `git rm` this changelog. (Live volumes: `tofu destroy` the office1_opnsense + module resources.) diff --git a/docs/changelog-20260712-opnsense-edge-real-isp-router.md b/docs/changelog-20260712-opnsense-edge-real-isp-router.md new file mode 100644 index 0000000..e4ab1a1 --- /dev/null +++ b/docs/changelog-20260712-opnsense-edge-real-isp-router.md @@ -0,0 +1,39 @@ +# Changelog 2026-07-12 -- DOCFIX-185: OPNsense edge is a real-ISP router, not an egress-airgap + +**Context.** While building the Office1 OPNsense edge, the config template's firewall +turned out to be **DC-airgap-shaped** (WAN `direction=out` default-deny except NTP + a +per-DC artifact-mirror sync). Operator clarified the transport model, correcting a +misunderstanding baked into the template: + +> Each site (both DCs and Office) simulates its **own real ISP connection** -- the OPNsense +> edge is a normal internet-facing router. The **dark fiber (mesh links) is East-West / +> replication only**, NOT an internet path. D-107's node-level artifact posture (per-DC +> mirror; nodes not pulling directly from the public internet) is a **separate** DC-node / +> routing concern, not the edge's WAN egress. + +## What changed +- **`opentofu/templates/opnsense-config.xml.tmpl`**: removed the three WAN-egress-control + rules (`seq 20` NTP-only, `seq 21` mirror-only, `seq 99` default-deny-out) and the + `MIRROR_SYNC_PROTOCOL`/`MIRROR_UPSTREAM_NET`/`MIRROR_SYNC_PORT` tokens they used. The edge + now allows normal outbound (LAN allow-any + default outbound NAT); WAN inbound stays + default-deny + `blockpriv`/`blockbogons`. A comment records the rationale. +- **`scripts/opnsense-render-config.sh`**: `MIRROR_*` dropped from `REQUIRED_VARS`. +- **`tests/opnsense-render-config/run-tests.sh`**: `MIRROR_*` export removed / unset. + (An XML-comment `--` bug in the first template edit was caught by the harness's + well-formed-XML test and fixed -- double-hyphens are illegal inside XML comments.) + +## Verification +- `bash tests/opnsense-render-config/run-tests.sh` -> ALL PASS (8/8). `repo-lint` 0 fail. +- Rendered Office1's real config (2 firewall rules = LAN allow-any inet/inet6; egress-control + gone), well-formed, no leftover tokens. + +## Follow-up (flagged, not actioned) +- The clarified transport model (real-ISP-per-site; dark fiber East-West-only; edges are + normal routers; node-airgap is a separate DC concern) is a **design clarification** that + should carry a short amendment note under **D-100** (fabric) and/or **D-107** (airgap) so + the DC-edge configs (Stage 3) are built to the same understanding, not the egress-airgap + one. Operator to rule whether to formalize as a D-amendment. + +## Revert +- `git checkout opentofu/templates/opnsense-config.xml.tmpl scripts/opnsense-render-config.sh + tests/opnsense-render-config/run-tests.sh`; `git rm` this changelog. diff --git a/docs/changelog-20260712-opnsense-prep-bz2-fallback.md b/docs/changelog-20260712-opnsense-prep-bz2-fallback.md new file mode 100644 index 0000000..60d7687 --- /dev/null +++ b/docs/changelog-20260712-opnsense-prep-bz2-fallback.md @@ -0,0 +1,19 @@ +# Changelog 2026-07-12 -- DOCFIX-184: opnsense-prep-image.sh bz2 python fallback + +**Context.** First real run of `scripts/opnsense-prep-image.sh` (preparing the OPNsense +26.1 nano image for the Office1 edge) failed exit 2 -- `bunzip2` absent (the nano image +is `.img.bz2`). `bzip2` is not installed and would need sudo. + +## Fix +- `scripts/opnsense-prep-image.sh`: the bz2 decompress now prefers `bunzip2` but falls + back to **python3's stdlib `bz2` module** (python3 is a base prereq on this platform), + so a missing `bzip2` package is no longer a blocker. The required-tools check requires + `qemu-img` + (`bunzip2` OR `python3`). + +## Verification +- Re-ran the prep: downloaded/decompressed/converted/grew the 26.1 nano image to + `opnsense-26.1-nano.qcow2` (qcow2, 11 GiB virtual) via the python path. +- `bash tests/opnsense-prep-image/run-tests.sh` -> ALL PASS (3/3). `repo-lint` 0 fail. + +## Revert +- `git checkout scripts/opnsense-prep-image.sh`; `git rm` this changelog. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index f5cd797..6128a30 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1659,6 +1659,39 @@ - **Numbers:** D next-free 112, DOCFIX next-free 184. Branch `dc-dc-stage1-phase0-first-exec` now also carries the merge + D-111 + DOCFIX-183. +## Jumphost stream -- session 2026-07-12: Office1 OPNsense edge (bring the site router+DHCP online) + +- **Operator ruling:** bring the Office1 OPNsense edge online FIRST (router + DHCP for + `office1-local`) before the Office1 NetBox VM. Edges are the real router; do not hang + service VMs on a `wan` shortcut. +- **TRANSPORT MODEL CLARIFIED (operator, corrects an earlier misread):** each site (both + DCs + Office) simulates its OWN real ISP connection -- the OPNsense edge is a normal + internet-facing router. Dark fiber (mesh links) = East-West / replication ONLY, not an + internet path. D-107 node-airgap (per-DC mirror) is a SEPARATE DC concern, not the edge + WAN egress. -> **DOCFIX-185:** stripped the egress-airgap rules (seq 20/21/99 + MIRROR_*) + from `opnsense-config.xml.tmpl` + render script + harness (8/8). FOLLOW-UP: a D-100/D-107 + amendment note so the Stage-3 DC edges are built to this model (operator to rule). +- **Creds (closed test, agent-generated, 0600 at `~/vr1-office1-creds/`, NEVER commit; + operator to REVOKE the NetBox token + these at session end):** SSH ed25519 keypair + + OPNsense root password + bcrypt hash. +- **Build DONE except boot:** `office1-wan` NAT net `172.30.1.0/24`->enp1s0 (virsh; D-103 + debt); OPNsense 26.1 nano image prepped (DOCFIX-184: bz2 -> python3 fallback, bzip2 absent); + config rendered (office1-opnsense, LAN 10.10.0.1/24 + DHCP .100-.199, WAN static 172.30.1.2 + gw .1) + ISO built; `module "office1_opnsense"` instantiated in main.tf + validated/planned + (first real validation of modules/opnsense-edge); disk + config_seed volumes created. +- **TWO first-boot findings (DOCFIX-186):** (1) `config_iso_path` must be OUTSIDE the pool + (create.content.url collides with its own target volume) -> staged at + `/var/lib/libvirt/vr1/staging/`. (2) **apparmor blocks `/var/lib/libvirt/vr1/`** (custom + pool path not in libvirt's abstraction) -> domain defined-but-failed-to-start, qemu + Permission-denied despite good POSIX perms. FOUNDATIONAL (all VR1 VMs). Fix = operator + one-time sudo: `echo '/var/lib/libvirt/vr1/** rwk,' | sudo tee + /etc/apparmor.d/local/abstractions/libvirt-qemu; sudo systemctl reload apparmor && restart + libvirtd`. PREREQ candidate (gates every VR1 VM boot). +- **NEXT:** operator applies the apparmor rule -> clean up the partial apply (domain defined + but not in tofu state) -> re-apply to boot -> verify WAN 172.30.1.2 + LAN DHCP/routing -> + then the Office1 NetBox VM. +- **Numbers:** D next-free 112, DOCFIX next-free 187 (184/185/186 committed this session). + diff --git a/opentofu/main.tf b/opentofu/main.tf index 24606da..39cd111 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -93,30 +93,24 @@ # committed here either). Every value below needs to be real before this is # uncommented -- do not fill in invented specs to wire it in sooner: # -# module "office1_opnsense" { -# source = "./modules/opnsense-edge" -# vm_name = "office1-opnsense" -# memory_mib = -# vcpu = -# pool_name = module.office1_storage.pool_name -# disk_size_bytes = -# base_volume_path = "" -# config_iso_path = "" -# lan_network_name = module.office1_network.network_name -# wan_network_name = DC fiber = MAAS/ -# Juju/operator), not a simulated ISP uplink -- wiring an edge's WAN -# side to a mesh leg would misrepresent the topology, not just -# placeholder it. Resolve gap #17 (a dedicated per-site ISP-edge -# network, analogous in shape to this same office1-network module or -# mesh-link) before uncommenting this block for Office1 OR for DC1/DC2.> -# } +# Office1's own OPNsense simulated-ISP edge (gap #16 CLOSED). gap #17 resolved +# for Office1 via the `office1-wan` NAT network (created via virsh 2026-07-12 -- +# a documented D-103 debt to formalize into an OpenTofu module later, same class +# as the retained `wan`). LAN = the Office1-local segment; WAN = office1-wan (its +# own simulated ISP -> internet). Config is the DOCFIX-185 real-ISP-router config +# (no egress-airgap), rendered to office1-opnsense-config.iso. +module "office1_opnsense" { + source = "./modules/opnsense-edge" + vm_name = "office1-opnsense" + memory_mib = 2048 + vcpu = 2 + pool_name = module.office1_storage.pool_name + disk_size_bytes = 17179869184 # 16 GiB (base OPNsense 26.1 nano is 11 GiB virtual) + base_volume_path = "/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2" + config_iso_path = "/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso" # staging, NOT the pool dir (create.content.url collides with its own target volume there -- first-boot finding 2026-07-12) + lan_network_name = module.office1_network.network_name + wan_network_name = "office1-wan" +} # ---- The D-100 dark-fiber mesh triangle: DC1<->DC2, DC1<->Office1, # DC2<->Office1 -- mesh, not star. All three legs created now even though DC2 diff --git a/opentofu/templates/opnsense-config.xml.tmpl b/opentofu/templates/opnsense-config.xml.tmpl index 1b1e1f0..e6e7f02 100644 --- a/opentofu/templates/opnsense-config.xml.tmpl +++ b/opentofu/templates/opnsense-config.xml.tmpl @@ -122,50 +122,18 @@ any Default allow LAN IPv6 to any rule - - 1 - keep - 20 - pass - 1 - wan - out - inet - UDP - {{WAN_IPADDR}} - any - 123 - D-107 controlled egress: NTP only, WAN uplink - - - 1 - keep - 21 - pass - 1 - wan - out - inet - {{MIRROR_SYNC_PROTOCOL}} - {{WAN_IPADDR}} - {{MIRROR_UPSTREAM_NET}} - {{MIRROR_SYNC_PORT}} - D-107 controlled egress: per-DC mirror upstream sync only - - - 1 - keep - 99 - block - 1 - wan - out - inet - any - any - any - D-107 controlled egress: default-deny everything else outbound on WAN - + diff --git a/scripts/opnsense-prep-image.sh b/scripts/opnsense-prep-image.sh index ee86cc4..a39753a 100644 --- a/scripts/opnsense-prep-image.sh +++ b/scripts/opnsense-prep-image.sh @@ -31,9 +31,17 @@ # noticing until a real deploy fails partway through. MIRROR_BASE="${OPNSENSE_MIRROR_BASE:?set OPNSENSE_MIRROR_BASE to a real OPNsense mirror base URL (see https://opnsense.org/download/) -- not hardcoded here, mirrors change}" -for bin in bunzip2 qemu-img; do - command -v "$bin" >/dev/null 2>&1 || { echo "FAIL: $bin required on PATH"; exit 2; } -done +command -v qemu-img >/dev/null 2>&1 || { echo "FAIL: qemu-img required on PATH"; exit 2; } +# bz2 decompressor: prefer bunzip2, else python3's stdlib bz2 module. python3 is +# a base prereq on this platform, so a missing bzip2 package is NOT a blocker +# (DOCFIX-184 -- first real run hit an absent bzip2). +if command -v bunzip2 >/dev/null 2>&1; then + BZ2_DECOMP=bunzip2 +elif command -v python3 >/dev/null 2>&1; then + BZ2_DECOMP=python3 +else + echo "FAIL: need bunzip2 or python3 to decompress the .bz2 image"; exit 2 +fi if command -v curl >/dev/null 2>&1; then FETCH=(curl -fSL -o) elif command -v wget >/dev/null 2>&1; then @@ -50,9 +58,13 @@ echo "== downloading: $URL ==" "${FETCH[@]}" "$IMG_BZ2" "$URL" -echo "== decompressing ==" -bunzip2 -k "$IMG_BZ2" +echo "== decompressing (via $BZ2_DECOMP) ==" IMG_RAW="${IMG_BZ2%.bz2}" +if [ "$BZ2_DECOMP" = "bunzip2" ]; then + bunzip2 -k "$IMG_BZ2" +else + python3 -c "import bz2,shutil,sys; shutil.copyfileobj(bz2.open(sys.argv[1],'rb'), open(sys.argv[2],'wb'))" "$IMG_BZ2" "$IMG_RAW" +fi echo "== converting raw -> qcow2 ==" qemu-img convert -f raw -O qcow2 "$IMG_RAW" "$OUT" diff --git a/scripts/opnsense-render-config.sh b/scripts/opnsense-render-config.sh index 2dd1ad8..18f2460 100644 --- a/scripts/opnsense-render-config.sh +++ b/scripts/opnsense-render-config.sh @@ -28,8 +28,10 @@ OPNSENSE_HOSTNAME DOMAIN ROOT_PASSWORD_HASH WAN_IF WAN_IPADDR WAN_SUBNET_BITS WAN_GATEWAY LAN_IF LAN_IPADDR LAN_SUBNET_BITS - MIRROR_SYNC_PROTOCOL MIRROR_UPSTREAM_NET MIRROR_SYNC_PORT ) +# MIRROR_SYNC_PROTOCOL/MIRROR_UPSTREAM_NET/MIRROR_SYNC_PORT removed (DOCFIX-185): +# the edge is a normal simulated-ISP router, not an egress-airgap firewall -- the +# WAN mirror/NTP egress-control rules that used them are gone from the template. for v in "${REQUIRED_VARS[@]}"; do req "$v" done diff --git a/tests/opnsense-render-config/run-tests.sh b/tests/opnsense-render-config/run-tests.sh index 1c3ec8b..ab045ab 100644 --- a/tests/opnsense-render-config/run-tests.sh +++ b/tests/opnsense-render-config/run-tests.sh @@ -15,7 +15,8 @@ export ROOT_PASSWORD_HASH='$2y$10$fakehashfakehashfakehashfakehashfakehas' export WAN_IF="vtnet0" WAN_IPADDR="203.0.113.2" WAN_SUBNET_BITS="30" WAN_GATEWAY="203.0.113.1" export LAN_IF="vtnet1" LAN_IPADDR="10.12.8.1" LAN_SUBNET_BITS="22" - export MIRROR_SYNC_PROTOCOL="TCP" MIRROR_UPSTREAM_NET="10.12.5.10/32" MIRROR_SYNC_PORT="443" + # MIRROR_* removed (DOCFIX-185): edge is a normal ISP router, no egress-airgap tokens. + unset MIRROR_SYNC_PROTOCOL MIRROR_UPSTREAM_NET MIRROR_SYNC_PORT unset ROUTE1_NETWORK ROUTE1_GATEWAY ROUTE1_DESCR ROUTE2_NETWORK ROUTE2_GATEWAY ROUTE2_DESCR unset NTP_UPSTREAM_SERVERS NTP_PREFER_SERVER }