diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index f524cce..19c16de 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -95,7 +95,8 @@ | Why is it built this way? / proposing changes | repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number | | Versions, channels, pins | repo `runbooks/appendix-B-asbuilt-version-lock.md` | | Environment facts (hosts, repo, planes) | `references/environment.md` | -| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings) | +| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) | +| Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong | `references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) | ## Standard loops (repeatable session shapes) diff --git a/.claude/skills/openstack-cloud-ops/references/environment.md b/.claude/skills/openstack-cloud-ops/references/environment.md index 8bcbe80..f73e354 100644 --- a/.claude/skills/openstack-cloud-ops/references/environment.md +++ b/.claude/skills/openstack-cloud-ops/references/environment.md @@ -29,6 +29,11 @@ Charmed OpenStack Caracal 2024.1 - Juju 3.6, MAAS 3.7.2, Vault TLS (charm-pki root CA), OVN 24.03, Ceph Squid, Octavia (amphora), Barbican, mysql-innodb-cluster, RabbitMQ, Magnum + magnum-capi-helm driver + azimuth +(SUBSTRATE below the charms -- pins + traps in `references/platform-traps.md`: +libvirt/KVM via the `dmacvicar/libvirt` OpenTofu provider `0.9.8` on OpenTofu +`v1.12.3`; **LXD pinned to the `5.21` LTS track** -- MAAS 3.6/3.7 cannot talk +to LXD >= 6.7, D-114; OPNsense 26.1 edges driven by REST API, D-113; the vcloud +host is ITSELF a KVM guest, so nesting is a per-level requirement) capi-helm-charts (kubeadm engine), in-cloud single-homed CAPI mgmt VM (capi-mgmt-v2, k8s-snap, D-035). NetBox is the IPAM apex: never hand-edit downstream MAAS or overlays for network values. diff --git a/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md b/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md index a9cfc5e..86c1c6a 100644 --- a/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md +++ b/.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md @@ -45,10 +45,25 @@ ## Providers used in `repo opentofu/` ### `dmacvicar/libvirt` (pinned `0.9.8`, confirmed 2026-07-09) + +**Before writing any `libvirt_domain`, read `references/platform-traps.md` +section 1.** Four schema-level traps there have each cost real time and none +of them is a validate-time error: bare `memory` is KiB not MiB; no `features` +block means `acpi=off`; no `cpu` block means no `svm` (nested KVM silently +impossible); and "will be updated in-place" still BOUNCES the guest. + - Registry: https://registry.terraform.io/providers/dmacvicar/libvirt/latest (JS-rendered -- use the GitHub repo for actual content) - GitHub: https://github.com/dmacvicar/terraform-provider-libvirt (default - branch `main`) + branch `main`; there is no `CHANGELOG.md` -- the release NOTES on the + releases page are the changelog, and they are unusually informative: + `.../releases/tag/v0.9.0` is the breaking-rewrite rationale, `v0.9.1` is the + 0.9.0->0.9.1 attribute-rename migration guide) +- **Version boundary that invalidates most examples on the internet:** 0.9.0 + (2025-11-08) is a compatibility-BREAKING rewrite (plugin framework, "HCL maps + almost 1:1 to libvirt XML"); the legacy provider lives on the `v0.8` branch. + 0.9.1 then code-generated the schema and renamed attributes again. Anything + 0.8-era -- including model memory -- is wrong for this pin. - Docs: `docs/resources/*.md`, `docs/data-sources/*.md` (NOT the older `website/docs/r/*.markdown` layout some search results/training data reference -- that's stale) @@ -89,20 +104,46 @@ issue) -- these were the most reliable fetches of the whole session. ### OPNsense (not a Terraform provider -- the guest OS itself) -- Official docs: https://docs.opnsense.org/manual/install.html, - .../manual/backups.html, .../development/backend/autorun.html -- Forum (community consensus, cite specific threads, not the forum - generally): https://forum.opnsense.org/index.php?topic=42517.0 (cloud-init - unreliable on FreeBSD -- the finding that ruled out reusing - `modules/cloudinit-vm` for OPNsense) -- GitHub issues (feature provenance): https://github.com/opnsense/core/issues/5733 - (ISO9660 Configuration-Importer support added specifically for VM/cloud - automation), .../issues/10017 (corroborates it shipped and is live) -- Full sourced writeup: `repo opentofu/README.md`'s "OPNsense deployment - research" section. + +**SUPERSEDED RESEARCH WARNING (2026-07-12/13).** `repo opentofu/README.md`'s +"OPNsense deployment research" section and the 2026-07-09 changelogs describe a +**config-ISO / Configuration-Importer** delivery path. That path is DEAD and its +code is DELETED: +- **D-112:** the Configuration Importer can NEVER fire on a pre-installed nano + image (upstream `opnsense-importer` probes for a read-only root and exits + before scanning media). The ISO was inert; nothing was ever going to read it. +- **D-113 (ADOPTED 2026-07-13, option a2):** OPNsense STAYS, but config moves to + the **REST API**. `opentofu/templates/opnsense-config.xml.tmpl`, + `scripts/opnsense-render-config.sh` and `scripts/opnsense-build-config-iso.sh` + are DELETED, along with the module's `config_seed` volume + cdrom disk. + **A rendered-`config.xml` push would now CLOBBER live API-managed DHCP.** + +Current path: `scripts/opnsense-api.sh` (REST client) + +`scripts/opnsense-bootstrap-apikey.sh` (mints a key via OPNsense's own model +over SSH). Traps, exact endpoints, the mandatory `service/reconfigure` step, the +60-second firewall auto-rollback, and the tcsh trap: `references/platform-traps.md` +section 4. + +- Official docs: https://docs.opnsense.org/manual/install.html; + REST API: https://docs.opnsense.org/development/how-tos/api.html; + 26.1 release notes (ISC-DHCP left core; dnsmasq is the new default): + https://docs.opnsense.org/releases/CE_26.1.html +- Still-valid finding from the 2026-07-09 research: cloud-init is unreliable on + FreeBSD, so `modules/cloudinit-vm` genuinely does not apply to OPNsense + (https://forum.opnsense.org/index.php?topic=42517.0), and the nano image (not + the installer image) is the one to use. +- Historical only (the ISO mechanism these describe cannot fire on nano): + https://github.com/opnsense/core/issues/5733, .../issues/10017. ## Where the full findings live (don't duplicate here, it will drift) +**Caveat first (2026-07-13):** `repo opentofu/README.md` predates D-112/D-113 +and its "OPNsense deployment research" section still describes the DELETED +config-ISO path -- see the superseded-research warning above before acting on +anything it says about OPNsense. Its libvirt schema notes also predate the four +domain traps in `references/platform-traps.md` section 1. Everything else in it +still stands. + `repo opentofu/README.md` is the authoritative, per-module account -- schema notes, what's confirmed vs. assumed per attribute, and a dated "Audit pass" section recording a later re-review (a real documentation diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md new file mode 100644 index 0000000..44419f9 --- /dev/null +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -0,0 +1,427 @@ +# Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense + +The substrate BELOW the cloud (the hypervisor, the provisioner, the edge +routers) has its own version boundaries and its own silent-failure modes. +Every item below is either (a) sourced to a vendor/upstream URL, or (b) +MEASURED on this project and sourced to the repo's own D-NNN / changelog. +Nothing here is generic advice: each entry exists because it cost, or would +have cost, a session. + +Companion files: `references/opentofu-provider-docs.md` (provider schema + +fetch methodology), `references/troubleshooting.md` (method). Start at the +verbatim-error index at the bottom if you have an error string in hand. + +## Version pins that actually matter (and why) + +| Thing | Pin | Why the pin exists | +|---|---|---| +| `dmacvicar/libvirt` provider | `0.9.8` | 0.9.0 was a compatibility-BREAKING rewrite; anything remembered from 0.8-era examples is wrong (see below) | +| OpenTofu | `v1.12.3` (as-run) | the binary that actually executed Stage 1 | +| LXD | **`5.21` LTS track** | MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114; see below) | +| MAAS | 3.7.2 (VR0) | the LXD ceiling above is a property of MAAS 3.6/3.7 | +| OPNsense | 26.1 "Witty Woodpecker" | ISC-DHCP left core in 26.1 -- Kea/dnsmasq only (see below) | +| Vault charm | `1.8/stable` | 1.16 is an incompatible charm, deliberately NOT an upgrade (appendix-B, D-068) | + +Charm channels/revisions are appendix-B's job, not this file's. This file +covers only the substrate the charms sit on. + +--- + +## 1. dmacvicar/libvirt provider (pinned 0.9.8) + +### 1a. The 0.8 -> 0.9 break invalidates most examples you will find + +v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the +plugin framework, with the design principle "HCL maps almost 1:1 to libvirt +XML"; the legacy provider stayed on the `v0.8` branch. v0.9.1 (2025-11-30) +then code-generated the schema from libvirt's XML schema, renaming attributes +again (`unit` -> `memory_unit`, `os.arch` -> `os.type_arch`, `os.machine` -> +`os.type_machine`, `os.kernel_args` -> `os.cmdline`, camelCase -> snake_case +throughout). +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.0 +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1 + +**Consequence:** blog posts, StackOverflow answers, and model memory almost +all describe 0.8. Nested structures are attribute-style objects +(`os = { ... }`, `devices = { ... }`), not HCL blocks. Confirm against +`docs/resources/*.md` + `examples/*.tf` at the pinned tag, or +`tofu providers schema -json`. + +### 1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188) + +The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are +explicit -- whenever libvirt exposes a value with a unit attribute the +provider now has two attributes (`memory` + `memory_unit`, `capacity` + +`capacity_unit`, etc.). **Leaving the unit unset lets libvirt use its +default.**" +(https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1) + +And libvirt's default is KiB: "The units for this value are determined by the +optional attribute `unit`, which defaults to 'KiB'" +(https://libvirt.org/formatdomain.html#memory-allocation). + +So `memory = 2048` with no `memory_unit` is **2 MiB**, not 2 GiB. In 0.8, +`memory` meant MiB -- that is exactly how this shipped here. + + memory = var.memory_mib + memory_unit = "MiB" # REQUIRED. Never omit. + +Signatures (all measured, 2026-07-12): QEMU cmdline `-m size=2048k`; +`virsh dominfo` -> `Max memory: 2048 KiB`; a FreeBSD guest echoes +`/boot.config` (262 bytes of serial output) and then triple-faults handing off +to `/boot/loader`. `tofu validate` CANNOT catch it (the attribute is +optional) -- `scripts/opentofu-validate.sh` check **S1** does. +Evidence: `docs/changelog-20260712-libvirt-memory-unit-rootcause.md`, +`docs/incident-20260712-opnsense-edge-boot-triplefault.md`. + +The provider's own current domain examples now carry `memory_unit = "MiB"` +(`docs/resources/domain.md`) -- a fresh reader of the docs would get this +right; a reader of any 0.8-era example would not. + +### 1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190) + +The whole `features` attribute is optional, so omitting it is schema-valid -- +and libvirt then renders the machine with `acpi=off` (measured on the QEMU +cmdline: `-machine pc-i440fx-noble,...,acpi=off`). + + features = { + acpi = true + apic = {} # acpi is a Boolean; apic is a nested object + } + +(schema: `docs/resources/domain.md`, `features.acpi` (Boolean), +`features.apic` (Attributes).) + +Three distinct consequences, only one of which is loud: +- FreeBSD/OPNsense **panics** (it finds the local APIC via ACPI's MADT and + ships no `atpic` fallback): `panic: running without device atpic requires a + local APIC` -> `db>` prompt -> domain shows `running` while burning 100% of + one core forever. +- Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling. +- **MAAS drives power off/on via ACPI**, so a MAAS-managed node VM without + ACPI can only ever be hard-stopped. That would have been a nasty Stage-3 + debug. +Evidence: `docs/changelog-20260712-libvirt-acpi-kernel-panic.md`. +Guard: `scripts/opentofu-validate.sh` check **S2**. + +### 1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible + +Found 2026-07-13. With no `cpu` attribute, libvirt hands the guest a generic +emulated CPU model with no virtualization flag. Measured: host is an **AMD +EPYC 9965**, the default-CPU guest was handed an **`Opteron_G3`**. Nothing +errors -- the guest just has no `/dev/kvm` and no `svm` in `/proc/cpuinfo`. + +Ubuntu's own nested-virt guide states the guest-side requirement: set the +guest `cpu mode` to `host-model` or `host-passthrough` for the guest to see +`svm`/`vmx` +(https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/). + + cpu = { + mode = "host-passthrough" + # per-caller: pass svm through, or disable it as router hardening + features = var.expose_nested_virt ? [] : [ + { name = "svm", policy = "disable" } + ] + } + +**The host side is a SEPARATE requirement and bites at every level.** The +vcloud host is ITSELF a KVM guest (`systemd-detect-virt` -> `kvm`), so nesting +must be enabled at each level, not just ours: +- check: `cat /sys/module/kvm_amd/parameters/nested` -> `Y` or `1` + (Intel: `/sys/module/kvm_intel/parameters/nested`) +- persist: `options kvm-amd nested=1` in a file under `/etc/modprobe.d/` + (same source as above) + +Why this matters here (D-114): LXD **virtual machines** are qemu/KVM guests, +so MAAS composing service VMs inside a site containment VM needs working +nested KVM at that depth; and `nova-compute` on the DC nodes needs it one +level deeper still. `modules/node-vm` still has the missing-`cpu`-block defect +-- logged, NOT fixed, because DC1 is gated behind Office1 +(`docs/changelog-20260713-d114-voffice1-nested-virt.md`). + +### 1e. "will be updated in-place" DOES NOT mean "the guest stays up" + +A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a +live guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and +no DHCP on the Office1 edge). "In-place" is a statement about the Terraform +RESOURCE, not about the domain. + +Upstream confirms the mechanism: the v0.9.4 release notes describe domain +update as "updating a domain (e.g. changing memory) **would undefine and +re-define it**" (the 0.9.4 fix was only to preserve NVRAM/TPM files across +that undefine), and v0.9.6 added "configurable update shutdown behavior so +domain updates can request guest shutdown and wait with a configurable timeout +before forcing a stop." +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4 +- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.6 + +**Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an +OUTAGE of that guest. Schedule and gate it as one.** Read the plan's resource +list before applying; `0 to change` on the domains is the only assurance that +a running guest is not about to bounce +(`docs/session-ledger.md`, standing lesson 2). + +Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to +libvirt (max boot memory, `machine`, `cpu`). The provider will happily plan an +in-place update that libvirt then ignores. If a changed value does not show up +in `virsh dominfo`, `destroy` + `undefine` the domain and re-apply (volumes +survive an `undefine`). + +### 1f. Two more upstream behaviours worth knowing (from the release notes) + +- `capacity_unit = "GiB"` on a volume caused **"Provider produced inconsistent + result after apply"** on every apply before 0.9.4 (libvirt normalizes to + bytes). Fixed in 0.9.4 -- if you see that error string, check the provider + version first. +- `tofu destroy` of a **`dir` pool** used to DELETE the backing directory + (`StoragePoolDelete`). Since 0.9.4 the directory is preserved by default, + with an explicit `destroy.delete = true` to opt back in. Relevant to + `modules/dc-storage-pool`. + (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4) +- **UNVERIFIED:** the 0.9.4 notes name a `destroy.shutdown.timeout` option + while `docs/resources/domain.md` documents `destroy = { graceful, timeout }`, + and I could not find the 0.9.6 update-shutdown attribute documented at all. + Before relying on either, read the real schema: + `tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'`. + +## 2. AppArmor blocks qemu on any non-default pool path + +libvirt's stock `abstractions/libvirt-qemu` grants qemu the DEFAULT pool path +(`/var/lib/libvirt/images`) only. A pool anywhere else is blocked by AppArmor +**even with perfect POSIX permissions**, and the failure names nothing: + +- libvirt/qemu says only: `Could not open '': Permission denied` +- the actual denial is in the kernel log: + `apparmor="DENIED" operation="open" profile="libvirt-" name=""` +- the domain DEFINES fine and fails at START -- so `tofu apply` can "succeed" + into a dead guest. + +Upstream (still open, provider issue #920, with the same workaround): +https://github.com/dmacvicar/terraform-provider-libvirt/issues/920 + +Fix, in the vendor-sanctioned override include (survives package upgrade -- +do NOT edit the shipped abstraction): + + # /etc/apparmor.d/local/abstractions/libvirt-qemu + /var/lib/libvirt/vr1/** rwk, + +This is now installed by `bash scripts/prereqs/install-apparmor-libvirt.sh` +(pool parent overridable via `VR1_POOL_PARENT`), wired into +`scripts/prereqs/install-all.sh` and reported by `check-prereqs.sh`. It cost a +session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host +state; `docs/changelog-20260713-apparmor-libvirt-prereq.md`. + +## 3. MAAS + LXD + +### 3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114) + +Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that +MAAS's pinned **pylxd 2.3.5** cannot speak to (fixed in **pylxd >= 2.3.9**, +not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6 +or the 5.21 LTS release until further notice." +https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 + +VR0 runs LXD **5.21.4** and is therefore safe by accident of timing; D-114 +makes it a decision. **LXD 5.21 LTS is supported to June 2029** +(https://canonical.com/blog/lxd_5-21-0_lts). + +**The live hazard is the snap auto-refresh, not the install.** LXD's docs: +"By default, installed snaps update automatically when new releases are +published to the channel they're tracking"; the `latest` track "typically +points to the latest feature release", is "a continuously rolling release +track", and is "not recommended for general use"; a bare `snap install lxd` +uses the most recent LTS track, "which is currently 5.21". +https://canonical.com/lxd/docs/latest/reference/releases-snap/ +=> install/refresh explicitly on `5.21/stable` and verify the tracked channel +(`snap list lxd`, `snap info lxd`) before blaming MAAS for an LXD it can no +longer talk to. + +### 3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off + +"Bridges created by LXD are managed, which means that in addition to creating +the bridge interface itself, LXD also sets up a local `dnsmasq` process to +provide DHCP, IPv6 route announcements and DNS services to the network" +(`ipv4.dhcp` defaults to **true**). +https://canonical.com/lxd/docs/latest/reference/network_bridge/ + +MAAS's own LXD VM-host procedure therefore has you disable exactly that, +because MAAS must be the DHCP/PXE authority for the machines it composes: + + lxc network set lxdbr0 dns.mode=none + lxc network set lxdbr0 ipv4.dhcp=false + lxc network set lxdbr0 ipv6.dhcp=false + +https://canonical.com/maas/docs/how-to-manage-machines + +This is the concrete form of D-114's DHCP warning. On `office1-local`, +**OPNsense/Kea is authoritative** (10.10.0.0/24, pool .100-.199). Two DHCP +servers on one L2 is an intermittent, genuinely unpleasant failure. Never +bridge `lxdbr0` onto the site LAN without disabling its DHCP first. + +### 3c. Two different LXDs in this cloud -- do not conflate them (D-114) + +Conflating these caused a real, recorded error: +- **Juju-created LXD CONTAINERS** on the OpenStack nodes (`lxd:N` placements + in `bundle.yaml`, 21 of them). MAAS never sees these and never needs to. +- **A MAAS-registered LXD KVM host** (the VR0 `lxd` machine) into which MAAS + **COMPOSES VMs** (`maas $PROFILE vm-host compose ...`) -- VR0's `tailscale` + machine is one. These ARE MAAS machines: enlisted, commissioned, deployed, + powered, released. + +D-114 extends the second pattern per site; the first is explicitly out of +scope. Also note: MAAS only sees VMs in the LXD **project** the VM host was +registered with -- VMs in other projects are invisible to it and unaffected +(MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251). + +**Composed-VM networking -- MEASURE, do not assume.** A MAAS-bundled copy of +the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses +either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD +profile. The profile is used to determine which bridge to use." +(served by a MAAS instance's own docs: +https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- **I could +NOT locate this sentence on canonical.com/maas.io today; the docs site has been +reorganized and the old URLs 404. Treat the profile->bridge mechanism as +PLAUSIBLE-BUT-UNCONFIRMED against a primary source.**) Since a composed machine +that PXEs on the wrong L2 is the failure this predicts, do the cheap +measurement instead of trusting either source: `lxc profile show maas` / +`lxc profile show default` on the LXD host, and confirm the NIC's parent bridge +before composing. + +Because LXD **VMs** (not containers) are qemu/KVM guests, this pattern needs +nested KVM at whatever depth the LXD host sits -- see 1d. + +## 4. OPNsense 26.1 edge + +### 4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113) + +The config-xml path is **DELETED** (template, renderer, ISO builder, and the +module's `config_seed` volume + cdrom disk -- D-113 amendment, 2026-07-13). +Configure the edge over the **REST API** (`scripts/opnsense-api.sh`); mint an +API key with `scripts/opnsense-bootstrap-apikey.sh`. Three separate defects +(no sshd, no console, an inert `` block) all traced to hand-writing +that file, and a full-config push DROPS ~667 migration-populated elements -- +including the box's only two firewall pass rules -- surviving only on an +UNDOCUMENTED self-heal at boot. If you think the API cannot express something, +that is a finding to raise against D-113, not a reason to write XML. + +Two traps that follow: +- **A config ISO can NEVER be read on a nano image** (D-112, root-caused in + upstream `opnsense/core:src/sbin/opnsense-importer`: the importer probes for + a read-only root and `bootstrap_and_exit 0`s on a pre-installed image before + scanning any media). Any instruction to build one is a trap. +- Factory default: LAN `192.168.1.1/24`, **SSH disabled**, **all inbound + blocked on WAN**. An unanswered ping to the WAN address is NOT a fault. + +### 4b. ISC dhcpd is GONE from core in 26.1 -- an `` config is inert + +26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically +installed during upgrades. **It is not installed on new installations because +it is not being used**, but you can still install and keep using it." Also: +"Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the +box." +https://docs.opnsense.org/releases/CE_26.1.html + +So on a fresh 26.1 image: an ISC `` block configures nothing (that was +DOCFIX-193), and the GUI default backend is **dnsmasq**, not Kea. This edge +deliberately runs **Kea** (10.10.0.0/24, pool .100-.199, first real lease +served to `voffice1` 2026-07-13). + +### 4c. REST API: auth, endpoints, and the reconfigure step everyone forgets + +- Auth is **HTTP basic with an API key/secret pair**, not the root password: + "API access is part of the local user authentication system, but uses + key/secret pairs to separate account information from machine to machine + communication." URL pattern: `https:///api///`. + https://docs.opnsense.org/development/how-tos/api.html +- Kea DHCPv4 (exact paths): + `/api/kea/dhcpv4/get`, `/set`, `/add_subnet`, `/set_subnet/$uuid`, + `/del_subnet/$uuid`, `/add_reservation`, `/set_reservation/$uuid`, + `/get_reservation`, `/add_option`; service control: + `/api/kea/service/reconfigure|restart|start|stop|status`. + **A `set*` call only edits config -- it is NOT live until you POST + `/api/kea/service/reconfigure`.** A silently-unapplied change looks exactly + like a change that did not take. + https://docs.opnsense.org/development/api/core/kea.html +- **Firewall rules have a self-rollback safety net -- use it.** The filter + controller exposes `savepoint`, `apply/$rollback_revision`, + `cancel_rollback/$rollback_revision`, `revert/$revision`: "When calling + `savepoint()` a new config revision will be created and the timestamp will be + returned for later use. **If the `cancelRollback(savepoint)` is not called + within 60 seconds, the firewall will rollback to the previous state**." + For any remote rule change on a router you reach THROUGH that router, a + savepoint means a mistake self-reverts instead of locking you out. + https://docs.opnsense.org/development/api/core/firewall.html +- Do NOT re-implement vendor internals to bootstrap. OPNsense stores API + secrets as `crypt(secret,'$6$')`; we deliberately do not mint keys offline -- + a format drift would fail SILENTLY (keys that never authenticate). Call the + vendor's own model instead (`scripts/opnsense-bootstrap-apikey.sh`). + (D-113 amendment.) + +### 4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH + +Measured on the Office1 edge: root's shell is **tcsh**. tcsh has no `$(...)` +command substitution (it uses backquotes); a `$(...)` inside a quoted remote +command is rejected with: + + Illegal variable name. + +It already cost a defect: a pre-install snapshot silently no-op'd and the +install then ran with no rollback point behind it +(`docs/changelog-20260713-office1-dhcp-apply.md`). Because the step was +non-fatal, nothing stopped. + +**Always feed remote commands to `sh -s`:** + + ssh -i "$OPNSENSE_SSH_KEY" root@ 'sh -s' <<'SH' + ...POSIX shell here... + SH + +(tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh) + +--- + +## Verbatim error -> cause index (grep this first) + +| Exact string you see | Cause | Where | +|---|---|---| +| guest triple-faults right after echoing `/boot.config`; `virsh dominfo` -> `Max memory: 2048 KiB`; QEMU `-m size=2048k` | `libvirt_domain.memory` with no `memory_unit` -- KiB, not MiB | 1b | +| `panic: running without device atpic requires a local APIC` (then `db>`, 100% of one core, domain "running") | no `features` block -> `acpi=off` | 1c | +| guest has no `/dev/kvm`; `/proc/cpuinfo` has no `svm`; CPU model reads `Opteron_G3` on an EPYC host | no `cpu` block -> generic emulated CPU | 1d | +| `Could not open '': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-"` | non-default libvirt pool path not granted in AppArmor | 2 | +| `Provider produced inconsistent result after apply` on a volume with `capacity_unit` | provider < 0.9.4 | 1f | +| plan says `will be updated in-place`, guest uptime resets anyway | provider undefines + redefines the domain | 1e | +| `Illegal variable name.` from an `ssh root@ '...'` | root's shell is tcsh; `$(...)` unsupported | 4d | +| OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more | the Configuration Importer can never fire on a nano image | 4a | +| a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c | +| MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a | +| a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c | + +--- + +## NetBox (upstream, `netbox.baldurkeep.com`) + +### `HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl` + +**Symptom.** `urllib`/`requests`/`pynetbox` get `403 Forbidden` from the upstream NetBox on every +endpoint, including ones a valid token plainly reads. `curl` with the identical token and URL +returns `200`. It reads exactly like a bad or revoked token; it is not. + +**Cause.** Something in front of upstream NetBox (WAF / reverse proxy) filters on **User-Agent**. +The default Python UA (`Python-urllib/3.12`) is blocked; `curl/8.5.0` is allowed. Measured +2026-07-13: same token, same URL, UA the only variable -- `None -> 403`, `curl/8.5.0 -> 200`. + +**Fix.** Send an accepted `User-Agent` header. This applies to **`pynetbox`** too (it sends +`python-requests/...`), so the repo's NetBox tooling must set it: + +```python +nb = pynetbox.api(url, token=tok) +nb.http_session.headers["User-Agent"] = "curl/8.5.0" +``` + +**Do not** diagnose this as a token/permission problem and go re-mint credentials -- that is the +trap. Confirm with `curl` first; if curl works and Python does not, it is the UA. + +**Note:** this is an UPSTREAM-only trap so far. The Office1 sandbox NetBox (4.6.4, no proxy in +front) accepts the default Python UA. Upstream is 4.5.8 and also uses **v1** (40-char, bare) tokens, +while the sandbox is 4.6 and uses **v2** (`nbt_.`). Any tool pointed at BOTH must +handle both auth shapes and the UA -- do not assume they are the same NetBox. diff --git a/.claude/skills/openstack-cloud-ops/references/troubleshooting.md b/.claude/skills/openstack-cloud-ops/references/troubleshooting.md index c5ba069..9f39241 100644 --- a/.claude/skills/openstack-cloud-ops/references/troubleshooting.md +++ b/.claude/skills/openstack-cloud-ops/references/troubleshooting.md @@ -10,6 +10,13 @@ This file gives you the method and the reflexes that must fire BEFORE and WHILE you read those. +**Substrate failures (libvirt/KVM guest won't boot, AppArmor, MAAS+LXD, +OPNsense edge):** these are NOT in appendix-A. Go to +`references/platform-traps.md` -- it ends in a verbatim-error -> cause index +keyed to exact strings (`panic: running without device atpic requires a local +APIC`, `Illegal variable name.`, a bare `Permission denied` that never names +AppArmor, `-m size=2048k`, ...). Match the string BEFORE hypothesizing. + **VR1 DC-DC track:** appendix-A above is VR0-specific -- nothing in the VR1 DC-DC track (`docs/dc-dc-deployment-workflow.md`) has been executed yet, so there is no equivalent incident index for it (an OpenTofu/MAAS-`vm_host`/ diff --git a/creds-manifests/vr1-office1.manifest b/creds-manifests/vr1-office1.manifest new file mode 100644 index 0000000..3b6f8eb --- /dev/null +++ b/creds-manifests/vr1-office1.manifest @@ -0,0 +1,22 @@ +# vr1-office1 creds manifest -- what MUST be consolidated in ~/vr1-office1-creds/ (SEC-009). +# scripts/creds-audit.sh verifies this list against the live folder: presence, mode, +# non-empty. No secret VALUES are read -- only existence, mode, and declared provenance. +# +# fields: <filename> <octal-mode> <source> +# source = "local" -- generated/held on the jumphost (vcloud) +# = "<vm>:<path>" -- MINTED ON A SITE VM; a working copy MUST be pulled into +# the folder at mint time (this is the class SEC-009 missed: +# office1-netbox:/root/netbox-secrets/api.token went +# un-consolidated until it was needed). +# +office1_svc_ed25519 600 local +office1_svc_ed25519.pub 644 local +opnsense-api.txt 600 local +opnsense-root-hash 600 local +opnsense-root-password 600 local +README 600 local +tailscale-authkey.txt 600 local +vr1-netbox.env 600 local +vr1-netbox-sandbox.env 600 office1-netbox:/root/netbox-secrets/api.token +vr1-office1.env 600 local +vr1-stage1.env 600 local diff --git a/docs/changelog-20260710-carve-gua-hang-fix.md b/docs/changelog-20260710-carve-gua-hang-fix.md new file mode 100644 index 0000000..595ddf4 --- /dev/null +++ b/docs/changelog-20260710-carve-gua-hang-fix.md @@ -0,0 +1,48 @@ +# Changelog 2026-07-10 -- DOCFIX-181: carve_gua subnet-enumeration hang + harness watchdog + +**Context.** After the Stage-1 commit, the full `scripts/run-tests-all.sh` +gauntlet was seen to HANG on `tests/dc-dc-prefixes-import/`. Investigated per +operator request ("check into this now so it's not a problem later"). Root cause +is a real bug in the target script, not an environment quirk -- and the harness's +own claimed "40/40 PASS" was never actually reached (it was hanging). + +## Root cause + +`netbox/dc-dc-prefixes-import.py::carve_gua()` built +`list(dc_gua_prefix.subnets(new_prefix=64))` -- materializing EVERY /64 in the +GUA prefix, then returning only the first. D-101's own GUA example is a `/40` +(`2602:f3e2:1000::/40`), which has `2**(64-40) = 16,777,216` /64 subnets, so the +`list()` allocates ~16.7M `IPv6Network` objects -> effective hang / OOM. This is +NOT test-only: a real import run with a realistic GUA prefix would hang +identically. (`carve_ula` is safe -- it carves /48->/56->/64, bounded at 256.) + +Diagnosed with `faulthandler.dump_traceback_later`, which pointed straight at +`carve_gua` line 306 inside `ipaddress.subnets()`. + +## Fix + +- **`netbox/dc-dc-prefixes-import.py::carve_gua`** -- take the first /64 LAZILY + with `next(dc_gua_prefix.subnets(new_prefix=64))` (O(1)) instead of + `list(...)[0]`. Same result, no enumeration. +- **`tests/dc-dc-prefixes-import/test_logic.py`** -- added a + `faulthandler.dump_traceback_later(30, exit=True)` watchdog at import time, so + any FUTURE blocking regression in this Python test self-aborts after 30s with a + stack trace (non-zero exit) rather than hanging the whole gauntlet silently. + +## Verification +- `bash tests/dc-dc-prefixes-import/run-tests.sh` -> `ALL PASS (40 checks)` in + ~0.05s (previously: hung indefinitely, never completing). +- `bash scripts/repo-lint.sh` -> 0 fail. +- Full `scripts/run-tests-all.sh` gauntlet now runs to completion. + +## Revert +- `git checkout netbox/dc-dc-prefixes-import.py tests/dc-dc-prefixes-import/test_logic.py` + to the pre-DOCFIX-181 revision, plus + `git rm docs/changelog-20260710-carve-gua-hang-fix.md`. + (Reverting reinstates the hang -- do not, without replacing the fix.) + +## Note +- The pre-existing ledger claim of "dc-dc-prefixes-import 40/40" was optimistic: + the end-to-end path had been hanging, so those checks never actually ran to a + green count until this fix. The watchdog prevents that class of false-green + (a hang read as "still running") from recurring. diff --git a/docs/changelog-20260710-opentofu-scaffold-first-validation.md b/docs/changelog-20260710-opentofu-scaffold-first-validation.md new file mode 100644 index 0000000..31e8474 --- /dev/null +++ b/docs/changelog-20260710-opentofu-scaffold-first-validation.md @@ -0,0 +1,74 @@ +# Changelog 2026-07-10 -- DOCFIX-179: OpenTofu scaffold, first real validation + apply + +**Context.** First-ever execution of the `opentofu/` tree against a real `tofu` +binary (OpenTofu v1.12.3), on the vcloud host, executing +`runbooks/dc-dc-phase0-vcloud-prep.md` Steps 8-11. The tree had carried a +**SCAFFOLD, UNVALIDATED** banner since authoring (no `tofu` had ever run it). +Two real bugs blocked `tofu init`/`plan`; both fixed here, then the Stage-1 +module slice was validated, planned, and applied clean (13 libvirt objects). + +## What changed + +### FIX 1 -- per-module `required_providers` (the `tofu init` blocker) +- `tofu init` failed: child modules use `libvirt_*`/`maas_*` resources but did + not declare their own `required_providers`, so OpenTofu inferred the + `libvirt_` prefix as **`hashicorp/libvirt`** (nonexistent) instead of + `dmacvicar/libvirt`. Provider SOURCE mapping is NOT inherited by child + modules -- only provider CONFIGURATION is. +- Added `versions.tf` (source-only; version stays pinned at the root) to every + module using a provider resource: 8 libvirt modules (`dc-planes`, `mesh-link`, + `dc-storage-pool`, `office1-network`, `node-vm`, `base-image`, `cloudinit-vm`, + `opnsense-edge`) + 1 maas module (`maas-vm-host`). `netem-link` uses only the + built-in `terraform_data`, so it needs none. Fixed comprehensively (all + modules, not just Stage 1's four) so Stage 2/3 do not re-hit it. + +### FIX 2 -- `tofu fmt` (the whole never-fmt'd scaffold) +- `tofu fmt -recursive opentofu/` normalized 11 files (attribute alignment, + comment reflow) that had drifted because no `tofu` had ever formatted them. + +### CHANGE 3 -- `provider "maas"` deferred to Stage 3 +- The root `provider "maas"` block forced `tofu plan` to demand `maas_api_url` + AND the **sensitive** `maas_api_key` for a Stage-1 plan that creates ZERO + MAAS resources -- which would bake a fake key into `phase0.tfplan` and + `terraform.tfstate` (the DOCFIX-175 plaintext-secret surface). Operator ruled + (2026-07-10) to fix structurally rather than use the runbook's placeholder + workaround. Removed the `provider "maas"` block from `main.tf` and the + `maas_api_url`/`maas_api_key` variables from `variables.tf`, with in-file + notes to re-add them at Stage 3 (when `maas-vm-host` is instantiated). `maas` + stays in the ROOT `versions.tf` `required_providers` (version pinned, + `.terraform.lock.hcl` complete for Stage 3). + +### FIX 4 -- repo-lint must skip OpenTofu's `.terraform/` provider cache +- After the first real `tofu init`, `repo-lint` FAILED L1 (ASCII) on a + downloaded provider plugin's README inside `opentofu/.terraform/`. DOCFIX-175 + gitignored `.terraform/` for git, but `repo_lint.py`'s `all_text()` scans the + filesystem, not git. Added `".terraform" in p.parts` to `all_text()`'s skip + set (alongside `.git`/`__pycache__`) -- third-party, gitignored, not repo + content. `tests/repo-lint/run-tests.sh` -> 34/34 PASS. + +### Result +- `tofu validate` -> Success. `tofu plan` -> 13 to add / 0 / 0, with NO MAAS + vars required. `tofu apply` -> **13 added, 0 changed, 0 destroyed** (6 DC1 + planes + dc1/office1/dc2 pools + office1-local + 3 mesh legs, all MTU 9000). +- `.terraform.lock.hcl` is now generated -- **commit it** (pins provider + hashes: `dmacvicar/libvirt` 0.9.8, `canonical/maas` 2.7.2). +- `opentofu/README.md` status banner updated: Stage-1 module slice VALIDATED + + APPLIED; the not-yet-instantiated Stage 2/3 modules remain UNVALIDATED. + +## Verification +- `bash scripts/opentofu-validate.sh` (== `tofu fmt -check` + `init` + `validate`) + -> clean after the fixes. +- `bash scripts/repo-lint.sh` -> 0 fail. + +## Revert +- `git rm opentofu/modules/*/versions.tf` (the 9 added files) -- restores the + original `tofu init` inference failure. +- Re-add the `provider "maas"` block to `main.tf` and the two `maas_*` variables + to `variables.tf` from this changelog / git history. +- `tofu fmt` reverts are cosmetic; `git checkout` the 11 files to undo. +- (Live infra: `runbooks/dc-dc-teardown-rollback.md` / `tofu destroy` -- not part + of a repo revert.) + +## Follow-up (logged) +- Stage 3's runbook must re-add the `provider "maas"` block + the two variables + when it wires `maas-vm-host` -- noted in `main.tf`/`variables.tf` in-file. diff --git a/docs/changelog-20260710-phase0-runbook-as-executed.md b/docs/changelog-20260710-phase0-runbook-as-executed.md new file mode 100644 index 0000000..1fbf07a --- /dev/null +++ b/docs/changelog-20260710-phase0-runbook-as-executed.md @@ -0,0 +1,49 @@ +# Changelog 2026-07-10 -- DOCFIX-180: dc-dc-phase0 runbook as-executed corrections + +**Context.** End-of-Stage-1 runbook update. Per the operator's model (keep a +DOCFIX log through a whole phase, then update the runbook at phase close), the +corrections accrued during the first real execution of +`runbooks/dc-dc-phase0-vcloud-prep.md` (2026-07-10) are now folded back into the +runbook as draft updates. The live execution itself is DOCFIX-179 (scaffold) + +DOCFIX-178 (prereq installers); this entry is the runbook-text reconciliation. + +## What changed in `runbooks/dc-dc-phase0-vcloud-prep.md` + +1. **Top banner** -- marks the runbook FIRST EXECUTED 2026-07-10 and points at + the DOCFIX-178/179/180 changelogs. +2. **Known-gap section -> RESOLVED.** The `provider "maas"` block DID force + `tofu plan` to demand both `maas_api_url` AND the sensitive `maas_api_key` + for a zero-MAAS plan. Recorded the operator's structural resolution (defer the + provider block + both vars to Stage 3; DOCFIX-179) instead of the placeholder + workaround. Also corrected a stale reference ("Step 4" -> it surfaces at + Step 8/9). +3. **Step 3 (MTU).** Added the two-MTU-domains clarification: the measured host + *uplink* MTU (1500) is the ISP/WAN-edge value (Stage 2/3), while `underlay_mtu` + drives host-internal jumbo-capable virtio bridges -> set it to jumbo (9000) + for the internal fabric. First run: measured 1500, ruled `underlay_mtu=9000`. +4. **Step 5 (pools).** Added the non-pristine-host reality (a prior VR0 topology + + `wan` were present): inventory first, tear down leftovers gated, KEEP `wan` + as a gap-#17 ISP-uplink model. Replaced the bare `sudo mkdir` with the + self-owned-parent pattern (`install -d -o $USER -g libvirt /var/lib/libvirt/vr1` + once, then sudo-free subdirs). Recorded that DC2's storage pool IS wired now + (planes still deferred). +5. **Step 8 (init/validate).** Recorded the first-`init` failure + its two fixes + (per-module `required_providers`; `tofu fmt -recursive`) and the README banner + flip to STAGE-1-VALIDATED. +6. **Step 9 (plan).** Added `-input=false` (non-interactive safety) and corrected + the expected plan from 11 to the real **13 resources** (added the omitted + `office1_network` and the newly-wired `dc2_storage`). + +## Verification +- `bash scripts/repo-lint.sh` -> 0 fail (1 documented legacy WARN). +- No script changed here (doc-only), so no harness; the tooling changes these + notes describe are covered by DOCFIX-178/179's harnesses. + +## Revert +- `git checkout runbooks/dc-dc-phase0-vcloud-prep.md` to the pre-DOCFIX-180 + revision (removes the 6 as-executed edits), and + `git rm docs/changelog-20260710-phase0-runbook-as-executed.md`. + +## Follow-up (logged) +- The other `dc-dc-phaseN` runbooks will get the same end-of-phase as-executed + treatment as each is first-executed (their own DOCFIX numbers). diff --git a/docs/changelog-20260710-prereq-install-scripts.md b/docs/changelog-20260710-prereq-install-scripts.md new file mode 100644 index 0000000..31cd57f --- /dev/null +++ b/docs/changelog-20260710-prereq-install-scripts.md @@ -0,0 +1,60 @@ +# Changelog 2026-07-10 -- DOCFIX-178: workstation prerequisite install scripts + +**Context.** First real execution of `runbooks/dc-dc-phase0-vcloud-prep.md` on the +vcloud host surfaced that `tofu` was not installed, and the runbook's Step 6 only +hand-waved ("install per OpenTofu's official instructions ... not prescribed +here"). Operator ruling (2026-07-10): a runbook must not assume prereq runtimes +are present, and new operators should have individual, repo-provided install +scripts to get a workstation/workspace ready. This delivers that. + +## What changed + +### NEW: `scripts/prereqs/` (idempotent Debian/Ubuntu installers) +- `lib-prereq.sh` -- shared helpers (sourced): `pq_have`, `pq_require_apt`, + `pq_parse_mode` (--check/--dry-run/--help), `pq_run` (dry-run-aware runner). +- `install-opentofu.sh` -- OpenTofu `tofu` >= 1.6.0 via the official **deb** + method (lands `/usr/bin/tofu`, the command name every runbook step + + `scripts/opentofu-validate.sh` expect; the `opentofu` snap may not expose it). + Version-checks and skips if already new enough. +- `install-virtualization.sh` -- libvirt-daemon-system + qemu-kvm + libvirt-clients + + virtinst, and adds the invoking user to the `libvirt` group (re-login noted). +- `install-jq.sh` -- jq (a known cause of harness-gauntlet failures when absent). +- `install-image-tools.sh` -- qemu-utils (`qemu-img`) + `xorriso`, for the Stage + 2/3 OPNsense build path (`opnsense-prep-image.sh`, `opnsense-build-config-iso.sh`). +- `check-prereqs.sh` -- read-only report (all of the above + curl/python3/sudo), + exit 0 all-required-present / 1 missing; each miss names its fixer script. +- `install-all.sh` -- orchestrator (virtualization -> tofu -> jq -> image-tools) + + final check; passes `--dry-run` through. +- `README.md` -- quick start, per-script table, scope/conventions. +- Each installer supports `--check` (read-only) and `--dry-run` (mutate nothing); + `sudo` is used only for the actual package/group mutation, never in check/dry-run. + Non-Debian/Ubuntu OS -> fail loud (exit 3), no package-manager guessing. + +### NEW: `tests/prereqs/run-tests.sh` +- Exercises the parse/guard/`--check`/`--dry-run` paths only (never a real apt + install -- same posture as the opnsense-* harnesses). Includes the safety + assertion that `pq_run` under dry-run does NOT execute its command. **24/24 PASS.** + +### MODIFIED: `runbooks/dc-dc-phase0-vcloud-prep.md` +- New **Prerequisites** section (before "Known gap") with the one-pass + `check-prereqs`/`install-all` flow + a per-prereq table pointing at the installers. +- Step 6's vague "install per official instructions" replaced with a concrete + `bash scripts/prereqs/install-opentofu.sh` pointer + the >=1.6.0 floor. + +## Verification +- `bash tests/prereqs/run-tests.sh` -> `prereqs: 24/24 PASS`. +- `bash scripts/repo-lint.sh` -> 0 fail (1 documented legacy WARN). +- `bash scripts/ledger-scan.sh` -> DOCFIX next-free was 178 before this entry. + +## Revert +- `git rm -r scripts/prereqs tests/prereqs docs/changelog-20260710-prereq-install-scripts.md` +- Revert the two `runbooks/dc-dc-phase0-vcloud-prep.md` edits (Prerequisites + section + Step 6 paragraph) back to their pre-DOCFIX-178 text. + +## Follow-up (logged, not actioned) +- The other `dc-dc-phaseN` runbooks should gain the same short "see + `scripts/prereqs/`" pointer in their intros (their prereq sets overlap: tofu, + libvirt, and Stage 2/3 additionally need qemu-img/xorriso). One-line-per-runbook + sweep, its own DOCFIX. `scripts/prereqs/` itself is already stage-agnostic. +- macOS/RHEL install paths (currently exit-3 fail-loud) if a non-Ubuntu control + point ever appears. diff --git a/docs/changelog-20260711-d111-subcarve-nn-alignment.md b/docs/changelog-20260711-d111-subcarve-nn-alignment.md new file mode 100644 index 0000000..dbdc52a --- /dev/null +++ b/docs/changelog-20260711-d111-subcarve-nn-alignment.md @@ -0,0 +1,48 @@ +# Changelog 2026-07-11 -- DOCFIX-182: D-111 v6 subcarve aligned to deployed NN mnemonic + +**Context.** The Chat/planning stream handed the Code stream the VR1 NetBox buildout +(`docs/dc-dc-netbox-buildout-scope.md`). A read-only export of the live +`netbox.baldurkeep.com` (2026-07-11) confirmed the deployed VR0-DC0/Willamette clouds +use a net-byte (`NN`) mnemonic (provider `:10`+VIP `:11`, metal `:20`, data `:30`, +storage `:40`, repl `:50`) with `/60`-per-plane + `/64`-active. `dc-dc-prefixes-import.py` +did NOT match it (arbitrary contiguous plane indices, `/64`-only) -- scope-doc +sub-decision #1. Operator deferred to the Code stream's recommendation; ratified as +**D-111**. This implements it. + +## What changed + +### `netbox/dc-dc-prefixes-import.py` +- **Replaced `carve_ula` + `carve_gua` with `carve_v6(dc, org_ula_48, dc_gua_prefix)`** + returning `[(cidr, role_slug, kind)]` on the D-111 layout: provider-public GUA + `/60`+`/64`@`:10` + VIP `/64`@`:11`; metal-admin/metal-internal SHARE the metal `/60`@`:20` + (`:20`/`:21`); data `:30`, storage `:40`, replication `:50` each `/60`+`/64` ULA. +- **New `_sub_at()` helper** -- computes the Nth subnet by direct integer arithmetic, + NEVER `list(base.subnets())` enumeration (preserves the DOCFIX-181 no-hang property; + the old `carve_gua` `next()` fix is subsumed). +- **`DC_V6_INDEX`** maps the per-DC ULA `/56` to the GUA site nibble (dc1->`:02xx`, + dc2->`:03xx`) so the 4th hextet reads `DC.NN` in both families. +- `main()` + `verify()` updated to consume the record list; docstring SUBCARVE SCHEME + rewritten. Per-DC prefix count is now 18 (6 v4 + 12 v6), was 12. + +### `tests/dc-dc-prefixes-import/test_logic.py` +- `carve_ula`/`carve_gua` tests replaced with `carve_v6` structure/family/NN-offset/ + non-overlap/no-hang assertions; end-to-end counts 12 -> 18; e2e GUA blocks use the real + `2602:f3e2:f02::/48` / `f03::/48`. **53/53 PASS** (was 40/40). + +### `docs/design-decisions.md` +- **D-111 ADOPTED** (the ruling above), incl. the metal `/60`-sharing resolution and the + explicit out-of-scope note (lbaas-mgmt/vpn/oob mirrored separately; G5 naming is its own D). + +### `docs/dc-dc-netbox-buildout-scope.md` +- Sub-decision #1 marked RATIFIED as D-111. + +## Verification +- `bash tests/dc-dc-prefixes-import/run-tests.sh` -> ALL PASS (53 checks). +- `bash scripts/repo-lint.sh` -> 0 fail. Standalone address math confirmed for dc1/dc2. +- NOT yet run against a real NetBox (the Office1 simulation instance doesn't exist yet; + it will be the first real target, then refinements flow back to `netbox.baldurkeep.com`). + +## Revert +- `git checkout netbox/dc-dc-prefixes-import.py tests/dc-dc-prefixes-import/test_logic.py` + to the pre-DOCFIX-182 revision; remove the D-111 entry from `design-decisions.md` and the + scope-doc ratification note; `git rm` this changelog. diff --git a/docs/changelog-20260711-ula-gen-command-fix.md b/docs/changelog-20260711-ula-gen-command-fix.md new file mode 100644 index 0000000..59cc8b8 --- /dev/null +++ b/docs/changelog-20260711-ula-gen-command-fix.md @@ -0,0 +1,31 @@ +# Changelog 2026-07-11 -- DOCFIX-183: fix invalid ULA-generation command in the netem/ULA proposal + +**Context.** While assigning the org ULA /48 for the VR1 NetBox buildout, the +generation command in `docs/dc-dc-netem-and-ula-gua-proposal.md` section 2.1 +(DOCFIX-168) was actually run and produced an **invalid IPv6 address** +(`fd42ff:19d97a::/48` -- 6-hex-digit hextets). Root cause: its `sed` split the +10 hex digits of `openssl rand -hex 5` as 4+6 (`\1\2:\3\4\5`) instead of the +2+4+4 an IPv6 `/48` needs (`fdXX:XXXX:XXXX`). + +## Fix +- `docs/dc-dc-netem-and-ula-gua-proposal.md` section 2.1: `sed` corrected to + `s/\(..\)\(....\)\(....\)/\1:\2:\3/` -> `fdXX:XXXX:XXXX::/48`. Verified: the + corrected command emits a valid `/48` within `fc00::/7`. +- Added a correction note pointing to the **ratified** VR1 value + `fd50:840e:74e2::/48` (from `docs/dc-dc-netbox-buildout-scope.md` 4c); the + command is now framed as the reference method, not a VR1 re-generation step. + +> **CORRECTION (2026-07-13, D-118).** The word "ratified" above was **FALSE** when +> written. `fd50:840e:74e2::/48` was a *recommendation* in a scope doc (gap G3); +> **no D-number assigned it**, and `grep ORG_ULA docs/design-decisions.md` returned +> nothing. The value was treated as settled for two days on the strength of this +> sentence alone. It is NOW genuinely ratified, as **D-118**. Recorded because the +> failure mode is reusable: a document asserting authority it does not have. + +## Verification +- Ran the corrected command: `fdc0:4468:d6bf::/48` -> parses, /48, in `fc00::/7`. +- `bash scripts/repo-lint.sh` -> 0 fail. + +## Revert +- `git checkout docs/dc-dc-netem-and-ula-gua-proposal.md` to the pre-DOCFIX-183 + revision; `git rm` this changelog. diff --git a/docs/changelog-20260712-docfix189-max-context-session-audit.md b/docs/changelog-20260712-docfix189-max-context-session-audit.md new file mode 100644 index 0000000..88272a1 --- /dev/null +++ b/docs/changelog-20260712-docfix189-max-context-session-audit.md @@ -0,0 +1,92 @@ +# Changelog 2026-07-12 -- DOCFIX-189: re-audit of the max-context session's troubleshooting + docs + +**Why.** The 2026-07-12 OPNsense boot session ran to a context limit *while working from a +wrong diagnosis*. DOCFIX-188 fixed the actual bug (a 2 MiB guest -- missing `memory_unit`). +This pass re-audits everything that session **wrote**, on the operator's instruction, because +work authored under a false theory tends to encode that theory as fact. + +**Headline:** the settings it changed are mostly fine and are RETAINED. The *reasoning* it +recorded was not -- it attributed the fault to whatever knob was being turned at the time, +on no evidence, four separate times. It also left one silently-dead config knob. + +## Class 1 -- false causal claims (the real damage) + +Each change was written up as a contributing cause. **None of them changed the symptom**: the +fault stayed at a deterministic 262 bytes through every one. Corrected in place, at the +source, so the next reader cannot inherit the theory: + +| Claim as written | Truth | +|---|---| +| `opnsense-edge/main.tf` -- COW backing "made boot2 fault ... **the disk shape did** [help]" | FALSE. Fault persisted identically. | +| `opnsense-edge/main.tf` -- q35 "triple-faults ... **confirmed via the serial log**" | FALSE. Serial log showed the SAME fault before and after. q35 was never shown to be a problem. | +| `opnsense-edge/main.tf` -- "with no console the domain triple-faulted" | MISLEADING. It faulted with AND without a console; the console *revealed* the fault, it did not change it. | +| `opnsense-edge/main.tf` -- `svm` disable is "**ROOT CAUSE** of the first-boot triple-fault" | FALSE. (Already corrected in DOCFIX-188.) | + +All four settings are **RETAINED** -- each is defensible on its own merits (serial is a +genuine nano requirement; i440fx is the conventional FreeBSD machine type; direct-copy is the +documented `virt-install --import` flow; `svm`-disable is reasonable hardening). Only the +false *justifications* are gone. The serial console genuinely earned its keep: it is what made +the fault legible at all. + +## Class 2 -- a silently dead knob (a real functional defect) + +`disk_size_bytes = 17179869184` (16 GiB) in `main.tf` **did nothing.** When DOCFIX-187 changed +the disk from a sized COW overlay to a direct copy of the prepped nano, nothing consumed the +variable any more -- but an unused variable is legal HCL, so neither `tofu validate` nor the +plan said a word. + +**Measured:** `virsh vol-info` -> the live disk is **11.00 GiB**, not the 16 GiB `main.tf` +declares. Real sizing comes from `scripts/opnsense-prep-image.sh`'s `GROW` (default `+8G`). + +- Removed `disk_size_bytes` from the `office1_opnsense` instantiation **and** from + `modules/opnsense-edge/variables.tf` (must be removed together or tofu errors). +- Left an explicit note in both places saying sizing is prep-image's job, so nobody re-adds a + sizing input the module does not consume. +- **Proof it was dead:** `tofu plan` is byte-identical before and after removal + (`0 to add, 1 to change, 0 to destroy`). Removing it changes no resource. +- Consequence to know: every VM built from the same prepped base image gets the same disk + size. If the edge needs more than 11 GiB, re-run prep-image with a bigger `GROW`. + +## Class 3 -- stale/unsourced facts + +- **`Opteron_G3` is a RED HERRING** (incident report). The host CPU is really an **AMD EPYC + 9965 (Zen 5, family 26)**; libvirt's CPU-model DB does not know family 26 and falls back to + the oldest matching name. The prior session partly built its nested-virt/old-CPU theory on + this artifact. Corrected + annotated: read `/proc/cpuinfo`, not libvirt's model guess. +- **"OPNsense's 3 GB min" is UNVERIFIED** (incident report, ranked step 3). No source given. + Annotated: do not propagate. That step was right-for-the-wrong-reason -- memory *was* the + problem, but it was 2 MiB, not "2 GB but slightly small", and nobody measured it. +- **DOCFIX-186's "the disk uses `backing_store` -- a reference, not a copy"** is now STALE + (DOCFIX-187 made it a copy). Conclusion still holds -- no target-path collision -- but for + a *different* mechanism (source and target volume names differ). Annotated, because the + original reasoning no longer supports the conclusion. +- **DOCFIX-187's changelog** gains a SUPERSEDED-IN-PART header pointing at DOCFIX-188/189. + +## Kept as-is (audited, no change needed) + +- DOCFIX-186's two infra findings are REAL and measured -- the `config_iso_path`-outside-the-pool + requirement, and the **apparmor** rule for `/var/lib/libvirt/vr1/**` (correctly flagged as + foundational; it gates every VR1 VM). Both stand. +- DOCFIX-185's egress-airgap strip was an operator ruling on transport model, not a + troubleshooting artifact. Untouched. + +## Verification + +- `tofu fmt` clean; `tofu validate` **Success**; `tofu plan` unchanged (proves the knob was dead). +- `scripts/opentofu-validate.sh`: S1 PASS + fmt/init/validate PASS. +- `tests/opentofu-validate/run-tests.sh`: **4 PASS / 0 FAIL**. +- `scripts/repo-lint.sh`: **0 fail** (1 documented legacy warn). + +## Standing lesson (the one worth keeping) + +**When a change does not fix the symptom, do not write it up as though it did.** Four +successive non-fixes were each recorded as a contributing cause, and the accumulated fiction +is what a fresh session would have inherited and continued. A fault that is *deterministic and +immune to every knob you turn* is evidence that you have not yet touched the cause -- it is +not evidence that every knob you turned was load-bearing. + +## Revert + +- `git revert <sha>`. Comment-only + one dead-variable removal; no live-state dependency. +- To restore the dead knob (do not): re-add `disk_size_bytes` to `modules/opnsense-edge/ + variables.tf` and pass it in `main.tf`. It will still do nothing. diff --git a/docs/changelog-20260712-libvirt-acpi-kernel-panic.md b/docs/changelog-20260712-libvirt-acpi-kernel-panic.md new file mode 100644 index 0000000..2a19a6e --- /dev/null +++ b/docs/changelog-20260712-libvirt-acpi-kernel-panic.md @@ -0,0 +1,120 @@ +# Changelog 2026-07-12 -- DOCFIX-190: SECOND boot bug -- libvirt domains had ACPI disabled + +**Context.** DOCFIX-188 fixed the OPNsense boot triple-fault (a 2 MiB guest). With the guest +finally getting its RAM, the kernel got far enough to reach interrupt initialisation -- and +hit a **second, independent defect** that the first bug had been masking. + +## Symptom + +Domain reports `running`, burns **100% of one core indefinitely**, emits nothing further on +the console, and never touches the network. Serial log (measured) ends at: + +``` +---<<BOOT>>--- +panic: running without device atpic requires a local APIC +cpuid = 0 +KDB: stack backtrace: +apic_init() at apic_init+0xfc/frame ... +mi_startup() at mi_startup+0xb5/frame ... +KDB: enter: panic +db> +``` + +The 100% CPU spin is the guest parked at the `db>` kernel debugger prompt. A domain in this +state looks healthy to `virsh` -- `running (booted)`, no error -- which is exactly how it +evades casual checks. + +## Root cause + +The QEMU cmdline carried **`-machine pc-i440fx-noble,...,acpi=off`**. + +None of the three VM modules emitted a `features` block, so libvirt defaulted **ACPI off**. +FreeBSD discovers the local APIC from ACPI's MADT table, and the OPNsense kernel ships no +`atpic` fallback -- so with ACPI disabled it has no usable interrupt controller and panics +in `apic_init()` before userland. + +## Fix -- all three VM modules (same foundational class as DOCFIX-188) + +```hcl +features = { + acpi = true + apic = {} +} +``` + +1. `modules/opnsense-edge` -- the live blocker; FreeBSD **panics outright** without it. +2. `modules/cloudinit-vm` -- MAAS/NetBox/GitBucket. A Linux guest boots without ACPI but + degraded: no clean ACPI shutdown/reboot signalling, unreliable CPU/IRQ enumeration. +3. `modules/node-vm` -- the MAAS-managed node VMs. **MAAS drives power off/on via ACPI + signalling**, so without it a graceful `virsh shutdown` (and therefore MAAS power + control) never works -- nodes could only ever be hard-stopped. This one would have been + a genuinely nasty Stage-3 debug. + +Every ordinary libvirt guest sets both; there was no reason ours did not. + +## Guard -- S2 + +Like `memory_unit`, `tofu validate` **cannot** catch this: the entire `features` block is +optional, so its absence is schema-valid. `scripts/opentofu-validate.sh` gains **S2**, which +fails any `libvirt_domain` that does not enable ACPI. The `--check-memory-unit` flag is +renamed `--static-only` (old name kept as a back-compat alias) since it now runs S1 + S2. + +Harness `tests/opentofu-validate/` 4 -> **6 PASS**. New `fixtures/s2-bad/` is the key case: +it **passes S1 and still fails S2**, proving the two guards catch independent classes rather +than one masking the other. + +## Verification (measured, live) + +| | Before | After | +|---|---|---| +| QEMU machine | `acpi=off` | **`acpi=on`** | +| Kernel | `panic: ... requires a local APIC`, `db>` | boots to userland | +| CPU | **100.0% of one core** (ddb spin) | idle (~19s total) | +| Network | nothing | **DHCP lease `172.30.1.126`, hostname `OPNsense`** | +| Memory (DOCFIX-188, re-confirmed) | -- | `BIOS 639kB/2096108kB available memory` = full 2 GiB | + +**OPNsense now boots and runs.** Both boot bugs are closed. + +- `tests/opentofu-validate/run-tests.sh`: **6 PASS / 0 FAIL**. +- `scripts/repo-lint.sh`: 0 fail (1 documented legacy warn). +- `tofu fmt`/`validate`: clean. + +## STILL OPEN -- the Configuration Importer did not apply (next session's item) + +OPNsense came up on **factory defaults** (WAN via DHCP) instead of our static +`172.30.1.2` / LAN `10.10.0.1`. The config ISO is **verified well-formed** -- it is ISO9660 +labelled `OPNSENSE_CFG` and contains `CONF/CONFIG.XML` with `<opnsense>` and `10.10.0.1` +inside (grepped from the raw image) -- and it is attached as a SATA cdrom. So the payload is +right and the Importer simply did not consume it. + +This is the mechanism `modules/opnsense-edge`'s own header has always flagged UNVERIFIED +("whether the Configuration Importer's ISO9660 support actually behaves as described once +booted for real"). The research is well-sourced; it had just never been exercised. Now it +has, and it did not work as assumed. + +**Do NOT guess the fix.** Read the Importer's own console output first -- it runs early in +boot and reports what it scanned. Untested hypothesis worth checking (NOT a conclusion): +ISO9660 8.3 uppercase naming vs the lowercase `/conf/config.xml` the Importer looks for. + +**Note the unanswered ping to the WAN address is NOT a fault** -- OPNsense blocks inbound on +WAN by default. Do not chase it. + +## Operational finding -- the boot log is unreadable to the agent (real gap) + +libvirt creates the serial log **`root:0600`** and **recreates it on every domain create**, so +reading the single most valuable artifact during a boot incident needs an interactive `sudo` +each time. This materially contributed to the original misdiagnosis. A default ACL on the +staging dir does NOT fix it: libvirt's `0600` creation mode sets the ACL **mask** to `---`, +which nullifies any named-user entry (measured: `user:jessea123:r-- #effective:---`). + +Proposed durable fix (NOT yet implemented/verified): have the runbook/module pre-create the +serial log `0644` before `tofu apply`, so libvirt appends to an already-readable file rather +than minting a `root:0600` one. Verify on a real recreate before trusting it. + +## Revert + +- Remove the `features` blocks from the three `libvirt_domain` resources (restores the panic + -- do not). +- Guard: drop `s2_check` + the `--static-only` rename in `scripts/opentofu-validate.sh`, and + `tests/opentofu-validate/fixtures/s2-bad/` + T5/T6/T7. +- Whole change: `git revert <sha>`. No live-state dependency. diff --git a/docs/changelog-20260712-libvirt-memory-unit-rootcause.md b/docs/changelog-20260712-libvirt-memory-unit-rootcause.md new file mode 100644 index 0000000..ae19e8d --- /dev/null +++ b/docs/changelog-20260712-libvirt-memory-unit-rootcause.md @@ -0,0 +1,119 @@ +# Changelog 2026-07-12 -- DOCFIX-188: ROOT CAUSE of the OPNsense boot triple-fault (libvirt `memory_unit`) + +**The 2026-07-12 boot incident is diagnosed. It was never a CPU, nesting, machine-type, +or console problem.** The Office1 OPNsense edge guest was running with **2 MiB of RAM.** + +## Evidence (measured this session, read-only, every hop) + +| Hop | Measured | +|---|---| +| Module input | `memory_mib = 2048` (`opentofu/main.tf:105`) -- intent: 2 GiB | +| Module wiring | `memory = var.memory_mib` (`modules/opnsense-edge/main.tf:82`), **no `memory_unit`** | +| tofu state | `memory = 2048` | +| Rendered libvirt XML | `<memory unit='KiB'>2048</memory>` | +| `virsh dominfo` | `Max memory: 2048 KiB` | +| **Actual QEMU cmdline** | **`-m size=2048k`** -- i.e. **2 MiB** | + +**Why it presents as a BTX triple-fault:** FreeBSD's `boot2` is tiny and fits in 2 MiB, so +it happily echoes `/boot.config` (the deterministic 262-byte serial capture) -- and then +triple-faults the instant it hands off to `/boot/loader`, which does not fit. This is +exactly why the fault was immune to every CPU/machine/disk/console change tried on +2026-07-12: none of them touched the cause. + +## The defect + +`dmacvicar/libvirt` **>= 0.9** changed `memory` semantics. Per the provider's own schema +(`tofu providers schema -json`): + +- `memory` -- *"interpreted in libvirt memory units (typically **KiB** unless a unit is + specified elsewhere)"* +- `memory_unit` -- *"Sets the unit for the domain's main memory value ... KiB, MiB, or GiB"* + +The old 0.8-era meaning of `memory` was MiB. The modules were authored against that +assumption. With no `memory_unit`, libvirt defaults to KiB, so **every VM this tree creates +gets 1024x too little RAM.** + +## Fix -- all three VM modules (identical one-line defect) + +`memory_unit = "MiB"` added to the `libvirt_domain` resource in each: + +1. `opentofu/modules/opnsense-edge/main.tf` -- the Office1 edge (**the live blocker**). +2. `opentofu/modules/cloudinit-vm/main.tf` -- MAAS / NetBox / GitBucket VMs (Stage 2, the + **next** thing to be built; would have hit this identically). +3. `opentofu/modules/node-vm/main.tf` -- DC node VMs (Stage 3). + +Only `opnsense-edge` was ever instantiated, which is the only reason this surfaced there +first. The `memory_mib = 2048` value is UNCHANGED -- with the unit correct it now means the +2 GiB that was always intended. + +**Also corrected (same file, flagged not smuggled):** the `opnsense-edge` comment asserting +that disabling AMD `svm` was the *"ROOT CAUSE of the first-boot triple-fault"*. That claim +is disproven by this session's evidence (it never resolved the fault). The `svm = disable` +setting itself is RETAINED -- it is legitimate nested-virt hardening for a guest with no +business seeing `svm` -- but the comment now says it was tried and did not fix anything. +Leaving a false root-cause claim in the tree would have misled the next session. + +## Guard -- so this cannot silently recur + +`tofu validate` **cannot** catch this: `memory_unit` is optional, so omitting it is +schema-valid. That is precisely how it shipped. Following the DOCFIX-137 precedent (make +the standing lesson enforceable at the source): + +- `scripts/opentofu-validate.sh` gains **S1**: a static scan failing any `libvirt_domain` + that sets `memory` without `memory_unit`. Runs first, needs no tofu binary and no network. +- New `--check-memory-unit` flag runs S1 alone, so the harness can exercise it without + paying for `tofu init`. +- `tests/opentofu-validate/` gains fixtures + T3/T4/T5. **T3 is the negative test**: the + fixture is the exact defect that shipped, and S1 rejects it. T5 asserts the real tree is + clean. + +## Verification + +- `tests/opentofu-validate/run-tests.sh`: **4 PASS / 0 FAIL** (was 1 PASS). +- `scripts/opentofu-validate.sh`: S1 PASS + fmt + init + validate **Success**. +- `scripts/repo-lint.sh`: **0 fail** (1 documented legacy warn). +- `scripts/run-tests-all.sh`: **ALL GREEN (52 harnesses)**. +- `tofu plan`: `0 to add, 1 to change, 0 to destroy` -- domain only; the 11 GiB direct-copy + disk volume is NOT touched. + +## Live apply -- SEPARATELY GATED, NOT DONE IN THIS CHANGE + +The repo fix above is module source only (no live mutation). Booting the edge is a separate +operator-gated step. **Do not trust the plan's "update in-place":** max-boot-memory is +create-time to libvirt, the same trap DOCFIX-187 recorded for `machine`/`cpu` (provider +plans in-place, libvirt ignores it). The domain must be recreated: + + cd opentofu && source ~/vr1-stage1.env + virsh -c qemu:///system destroy office1-opnsense # currently paused/faulted + virsh -c qemu:///system undefine office1-opnsense # leaves volumes intact + tofu apply + +Re-run `tofu plan` AFTER the `undefine` -- the `1 to change / in-place` plan captured above +is the *pre-undefine* plan and is NOT what will execute. Post-undefine, tofu refreshes, sees +the domain gone, and switches to create. Confirm it then reads `1 to add, 0 to destroy` +(disk + config_seed volumes untouched by the recreate). + +**Pass criteria (do NOT declare success on `tofu apply` exiting 0):** +1. **Authoritative:** `virsh dominfo office1-opnsense` -> `Max memory: 2097152 KiB` (2 GiB). + Anchor on this number; it is unambiguous. Do NOT anchor on the QEMU `-m` string -- with + `unit='MiB'` libvirt may normalize and emit `-m size=2097152k`, which is **success**, not + failure. (Stated as a guess here, not a measurement, until the apply runs.) +2. The serial log grows **past 262 bytes** into the loader/kernel. +3. The router actually works: WAN 172.30.1.2 reachable, LAN DHCP serving `office1-local`. + +**Pre-staged fallbacks (so a surprise does not cost another session):** +- If `dominfo` still shows the wrong size, the provider is not honouring `memory_unit`: fix + is `memory = var.memory_mib * 1024` (leaning on the KiB default), NOT more debugging. +- If it clears 262 bytes and boots further but does not reach a working router, that is a + **sizing** question, not a regression of this fix -- check OPNsense 26.1's documented + minimum RAM (the incident report's "3 GB" is UNVERIFIED) and bump to 4096 if warranted. + Clearing the triple-fault and reaching a working router are two separate bars. + +## Revert (per item) + +- Module fix: remove the `memory_unit = "MiB"` line from the three `libvirt_domain` + resources (restores the 2 MiB bug -- do not). +- Comment correction: `git revert` the doc hunk in `modules/opnsense-edge/main.tf`. +- Guard: delete the S1 block + `--check-memory-unit` in `scripts/opentofu-validate.sh`, drop + T3/T4/T5 and `tests/opentofu-validate/fixtures/`. +- Whole change: `git revert <sha>`. Nothing here is live-state dependent. diff --git a/docs/changelog-20260712-office1-opnsense-edge-build.md b/docs/changelog-20260712-office1-opnsense-edge-build.md new file mode 100644 index 0000000..8ddab6a --- /dev/null +++ b/docs/changelog-20260712-office1-opnsense-edge-build.md @@ -0,0 +1,58 @@ +# Changelog 2026-07-12 -- DOCFIX-186: Office1 OPNsense edge build + two first-boot infra findings + +**Context.** Standing up the Office1 OPNsense edge so `office1-local` has a router + DHCP +(operator: bring the edge online before the Office1 NetBox VM). First-ever OPNsense boot in +this repo -- reviewed-but-unexercised module + image chain. This records the repo changes and +the two real infrastructure findings hit on the first apply. + +## Repo change +- **`opentofu/main.tf`**: `module "office1_opnsense"` instantiated (was a commented skeleton): + `modules/opnsense-edge`, 2 vCPU / 2 GiB / 16 GiB, `pool_name = module.office1_storage.pool_name`, + `base_volume_path` = the prepped nano qcow2, `config_iso_path` = the staged config ISO, + `lan_network_name = module.office1_network.network_name`, `wan_network_name = "office1-wan"`. + Validated + planned clean (first real validation of `modules/opnsense-edge`): 3 to add. + +## Infrastructure created (not repo files) +- **`office1-wan`** NAT network (`172.30.1.0/24` -> `enp1s0`), created via `virsh` -- + Office1's per-site simulated-ISP uplink (operator's per-site-uplink ruling). **Logged as + D-103 debt** to formalize into an OpenTofu module later (same class as the retained `wan`). +- OPNsense creds generated + saved 0600 to `~/vr1-office1-creds/` (jumphost-only, NEVER + committed): SSH keypair + root password + its bcrypt hash (for `config.xml`). + +## Two first-boot findings (both real, both DOCFIX-worthy) +1. **`config_iso_path` must live OUTSIDE the pool dir.** `libvirt_volume.config_seed` uses + `create.content.url` to COPY the ISO into the pool as a volume named + `office1-opnsense-config.iso`; if the source ISO sits at that exact pool path, libvirt + errors "volume target path ... already exists". Fixed by staging the ISO at + `/var/lib/libvirt/vr1/staging/` and pointing `config_iso_path` there. (The disk uses + `backing_store` -- a reference, not a copy -- so the in-pool nano image is fine.) + **CORRECTION 2026-07-12 (DOCFIX-189):** that parenthetical is now STALE -- DOCFIX-187 + changed the disk to a direct copy (`create.content.url`), so it no longer uses + `backing_store`. The in-pool nano image is still fine, but for a DIFFERENT reason: the + copy's target volume name (`<vm>-disk.qcow2`) differs from the source + (`opnsense-26.1-nano.qcow2`), so there is no target-path collision. Same conclusion, + different mechanism -- do not rely on the original reasoning. +2. **apparmor blocks the custom pool path `/var/lib/libvirt/vr1/`.** The domain was defined + but failed to start: qemu (`libvirt-qemu`) got "Permission denied" on the backing image + despite correct POSIX perms. libvirt's apparmor abstraction only whitelists + `/var/lib/libvirt/images/**` + `.../qemu/**`, not our operator-owned `vr1/` pool tree. + **This is foundational -- every VR1 VM lives under `/var/lib/libvirt/vr1/`.** Fix (operator, + one-time sudo), via the abstraction's `include if exists <local/abstractions/libvirt-qemu>`: + ``` + echo '/var/lib/libvirt/vr1/** rwk,' | sudo tee /etc/apparmor.d/local/abstractions/libvirt-qemu + sudo systemctl reload apparmor && sudo systemctl restart libvirtd + ``` + **PREREQ:** this belongs in the workstation/host prep (a candidate `scripts/prereqs/` + addition + a phase-0/phase-1 runbook note) since it gates every VR1 VM boot, not just this + edge. Not yet actioned as a prereq installer (needs sudo). + +## State +- Edge build COMPLETE except the boot: image, `office1-wan`, creds, config, ISO, module + instantiation, disk + config_seed volumes all done. Domain boot is blocked ONLY on the + apparmor fix above (operator sudo). After it: re-apply to boot, then verify WAN (`172.30.1.2`) + + LAN DHCP/routing on `office1-local`. + +## Revert +- `git checkout opentofu/main.tf` (re-comment the module); `virsh net-destroy/undefine + office1-wan`; `git rm` this changelog. (Live volumes: `tofu destroy` the office1_opnsense + module resources.) diff --git a/docs/changelog-20260712-opnsense-edge-boot-fixes.md b/docs/changelog-20260712-opnsense-edge-boot-fixes.md new file mode 100644 index 0000000..983f062 --- /dev/null +++ b/docs/changelog-20260712-opnsense-edge-boot-fixes.md @@ -0,0 +1,55 @@ +# Changelog 2026-07-12 -- DOCFIX-187: opnsense-edge module first-boot fixes (boot still OPEN) + +> **SUPERSEDED IN PART -- read `docs/changelog-20260712-libvirt-memory-unit-rootcause.md` +> (DOCFIX-188) and the DOCFIX-189 audit first.** +> +> This changelog was written under a WRONG diagnosis, at a context limit. The boot fault was +> a **2 MiB guest** (missing `memory_unit`), not a CPU/machine/disk/console problem. +> +> The CHANGES below are retained (each is defensible on its own merits), but the +> **CAUSAL CLAIMS attached to items 2, 3 and 4 are FALSE** and were corrected in the module +> source by DOCFIX-189: +> - item 2 (q35 -> i440fx): did NOT fix the fault. q35 was never shown to be a problem. +> - item 3 (COW overlay -> direct copy): did NOT fix the fault. It also silently orphaned +> `disk_size_bytes`, so the edge got 11 GiB while `main.tf` claimed 16 GiB (DOCFIX-189). +> - item 4 (`svm` disable): did NOT fix the fault. +> - item 1 (serial console) is the one that earned its keep -- but it *revealed* the fault, +> it did not change it. +> +> Standing lesson: when a change does not fix the symptom, do not write it up as though it +> did. Every one of these was recorded as a contributing cause on no evidence. + +**Context.** First real boot of the Office1 OPNsense edge. Several correct-but-necessary +module fixes were made while debugging; the boot is NOT yet resolved (a BTX-loader +triple-fault remains -- see `docs/incident-20260712-opnsense-edge-boot-triplefault.md`). +These changes are kept because a working config needs them regardless. + +## `opentofu/modules/opnsense-edge/main.tf` +1. **Serial console added** (`devices.serials`, file-backed to + `/var/lib/libvirt/vr1/staging/${vm_name}-serial.log`). OPNsense nano is serial-only; the + module had no console, so the guest had no output and faulted with nothing to show. This + got the boot from a blank early fault to the loader stage (and gives boot-log capture). +2. **`machine` q35 -> `pc` (i440fx).** FreeBSD's legacy BTX loader is far more commonly + booted on i440fx; q35 is a known early-fault source. (Did not fix this incident, but is + the safer, documented choice.) +3. **Disk: COW overlay -> direct per-VM copy** (`create.content.url = base_volume_path`, + no `backing_store`). Matches the documented `virt-install --import <nano.qcow2>` flow; + the nano is designed to boot + auto-expand in place, not through a read-only backing. + `var.disk_size_bytes` is now unused (nano is pre-grown by `opnsense-prep-image.sh`). +4. **CPU `host-passthrough` with AMD `svm` disabled** (`cpu.features = [{name=svm, + policy=disable}]`). The documented AMD nested-virt fix (`-cpu host,-svm`). NOTE: the + provider key is `features` (plural) -- `feature` validates but is silently dropped. + +## Important operational note discovered +- `machine` and `cpu` are create-time; the dmacvicar provider plans them as an in-place + "change" that libvirt does NOT apply to an existing domain. To actually change them: + `virsh destroy <dom>; virsh undefine <dom>; tofu apply` (recreate). `tofu apply -replace` + hit "domain already exists" mid-replace this session. + +## Status +- Module `tofu validate` -> Success. NOT booting: OPNsense triple-faults at the BTX loader + regardless of the above (incident report has the full trail + ranked next steps). + +## Revert +- `git checkout opentofu/modules/opnsense-edge/main.tf` to the pre-DOCFIX-187 revision. + (Reverts the serial console + i440fx + direct-disk + svm-disable together.) diff --git a/docs/changelog-20260712-opnsense-edge-real-isp-router.md b/docs/changelog-20260712-opnsense-edge-real-isp-router.md new file mode 100644 index 0000000..e4ab1a1 --- /dev/null +++ b/docs/changelog-20260712-opnsense-edge-real-isp-router.md @@ -0,0 +1,39 @@ +# Changelog 2026-07-12 -- DOCFIX-185: OPNsense edge is a real-ISP router, not an egress-airgap + +**Context.** While building the Office1 OPNsense edge, the config template's firewall +turned out to be **DC-airgap-shaped** (WAN `direction=out` default-deny except NTP + a +per-DC artifact-mirror sync). Operator clarified the transport model, correcting a +misunderstanding baked into the template: + +> Each site (both DCs and Office) simulates its **own real ISP connection** -- the OPNsense +> edge is a normal internet-facing router. The **dark fiber (mesh links) is East-West / +> replication only**, NOT an internet path. D-107's node-level artifact posture (per-DC +> mirror; nodes not pulling directly from the public internet) is a **separate** DC-node / +> routing concern, not the edge's WAN egress. + +## What changed +- **`opentofu/templates/opnsense-config.xml.tmpl`**: removed the three WAN-egress-control + rules (`seq 20` NTP-only, `seq 21` mirror-only, `seq 99` default-deny-out) and the + `MIRROR_SYNC_PROTOCOL`/`MIRROR_UPSTREAM_NET`/`MIRROR_SYNC_PORT` tokens they used. The edge + now allows normal outbound (LAN allow-any + default outbound NAT); WAN inbound stays + default-deny + `blockpriv`/`blockbogons`. A comment records the rationale. +- **`scripts/opnsense-render-config.sh`**: `MIRROR_*` dropped from `REQUIRED_VARS`. +- **`tests/opnsense-render-config/run-tests.sh`**: `MIRROR_*` export removed / unset. + (An XML-comment `--` bug in the first template edit was caught by the harness's + well-formed-XML test and fixed -- double-hyphens are illegal inside XML comments.) + +## Verification +- `bash tests/opnsense-render-config/run-tests.sh` -> ALL PASS (8/8). `repo-lint` 0 fail. +- Rendered Office1's real config (2 firewall rules = LAN allow-any inet/inet6; egress-control + gone), well-formed, no leftover tokens. + +## Follow-up (flagged, not actioned) +- The clarified transport model (real-ISP-per-site; dark fiber East-West-only; edges are + normal routers; node-airgap is a separate DC concern) is a **design clarification** that + should carry a short amendment note under **D-100** (fabric) and/or **D-107** (airgap) so + the DC-edge configs (Stage 3) are built to the same understanding, not the egress-airgap + one. Operator to rule whether to formalize as a D-amendment. + +## Revert +- `git checkout opentofu/templates/opnsense-config.xml.tmpl scripts/opnsense-render-config.sh + tests/opnsense-render-config/run-tests.sh`; `git rm` this changelog. diff --git a/docs/changelog-20260712-opnsense-prep-bz2-fallback.md b/docs/changelog-20260712-opnsense-prep-bz2-fallback.md new file mode 100644 index 0000000..60d7687 --- /dev/null +++ b/docs/changelog-20260712-opnsense-prep-bz2-fallback.md @@ -0,0 +1,19 @@ +# Changelog 2026-07-12 -- DOCFIX-184: opnsense-prep-image.sh bz2 python fallback + +**Context.** First real run of `scripts/opnsense-prep-image.sh` (preparing the OPNsense +26.1 nano image for the Office1 edge) failed exit 2 -- `bunzip2` absent (the nano image +is `.img.bz2`). `bzip2` is not installed and would need sudo. + +## Fix +- `scripts/opnsense-prep-image.sh`: the bz2 decompress now prefers `bunzip2` but falls + back to **python3's stdlib `bz2` module** (python3 is a base prereq on this platform), + so a missing `bzip2` package is no longer a blocker. The required-tools check requires + `qemu-img` + (`bunzip2` OR `python3`). + +## Verification +- Re-ran the prep: downloaded/decompressed/converted/grew the 26.1 nano image to + `opnsense-26.1-nano.qcow2` (qcow2, 11 GiB virtual) via the python path. +- `bash tests/opnsense-prep-image/run-tests.sh` -> ALL PASS (3/3). `repo-lint` 0 fail. + +## Revert +- `git checkout scripts/opnsense-prep-image.sh`; `git rm` this changelog. diff --git a/docs/changelog-20260713-apparmor-libvirt-prereq.md b/docs/changelog-20260713-apparmor-libvirt-prereq.md new file mode 100644 index 0000000..e0409c9 --- /dev/null +++ b/docs/changelog-20260713-apparmor-libvirt-prereq.md @@ -0,0 +1,60 @@ +# changelog 2026-07-13 -- fold the AppArmor/libvirt rule into scripts/prereqs + +## What + +New prereq installer `scripts/prereqs/install-apparmor-libvirt.sh`, wired into +`install-all.sh` (right after `install-virtualization`, since it edits libvirt's own +profile directory) and reported by `check-prereqs.sh`. Harness extended +`tests/prereqs/run-tests.sh` 24 -> 32 PASS. README updated. + +## Why + +The rule `/var/lib/libvirt/vr1/** rwk,` in +`/etc/apparmor.d/local/abstractions/libvirt-qemu` **gates every VR1 VM boot** -- the +Office1 edge, the DC edges, and every node VM. libvirt's stock +`abstractions/libvirt-qemu` grants qemu only the DEFAULT pool path +(`/var/lib/libvirt/images`); VR1 uses a custom pool parent, which is not in that +abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The failure +is nasty and non-obvious: the domain DEFINES fine, then fails to start with a bare +qemu "Permission denied" and **nothing in the libvirt error names AppArmor**. It cost +a session on 2026-07-12 (DOCFIX-186). + +It had been applied BY HAND on the vcloud host and existed ONLY as host state. Nothing +in the repo carried it -- so a rebuild, or a second host, walked straight back into the +identical wall. This closes that gap. It was logged as a PREREQ candidate at the time +and never actioned; the ledger collapse surfaced it again. + +## Design notes + +- **Idempotent, and deliberately quiet on a good host.** If the rule is already present + the script exits 0 having touched nothing -- in particular it does NOT reload apparmor + and does NOT bounce libvirtd. The reload/restart happens ONLY on the run that actually + adds the rule. (Restarting libvirtd does not stop running domains, but it is a real + service action and is gated behind a real change rather than run on every pass.) +- **Writes to `local/`, not the abstraction itself.** `local/abstractions/libvirt-qemu` + is the vendor-sanctioned override include -- it survives package upgrades, which an + edit to the shipped abstraction would not. +- **Pool parent is a variable, not a literal.** Defaults to `/var/lib/libvirt/vr1`, + overridable via `VR1_POOL_PARENT` -- per the repo's prefer-dynamic-over-hardcoded rule. +- **Non-AppArmor hosts report N/A and SUCCEED.** The rule is unnecessary there, not + failed; the script does not manufacture a failure on a system it does not apply to. + +## Verification + +- `tests/prereqs/run-tests.sh`: **32/32 PASS** (was 24/24). The new cases drive the + DETECTION off a pool parent guaranteed absent from the profile, so the negative path + is deterministic regardless of what the host has applied: `--check` must FAIL for an + ungranted parent (a false OK here is the whole bug), `--dry-run` must plan the fix and + mutate nothing, and the planned rule must be exactly `<parent>/** rwk,`. +- Full gauntlet: **ALL GREEN (52 harnesses)**. `repo-lint`: 0 fail (1 documented legacy warn). +- Run live on vcloud: `--check` reports OK (rule already present), exit 0, and the + profile file is unchanged -- confirming the already-satisfied path is a true no-op. + +## Revert + +``` +git revert <this commit> +``` +Removes the installer, its harness cases, the `install-all`/`check-prereqs` wiring, and +the README rows. Reverting touches NO host state: the rule already live on vcloud was +applied by hand long before this commit and is not managed by it. diff --git a/docs/changelog-20260713-config-xml-danger-sweep.md b/docs/changelog-20260713-config-xml-danger-sweep.md new file mode 100644 index 0000000..d26bb2e --- /dev/null +++ b/docs/changelog-20260713-config-xml-danger-sweep.md @@ -0,0 +1,57 @@ +# 2026-07-13 -- safety sweep: the config.xml push path is now a LIVE HAZARD + +After D-113(a2) was proven (edge config is API-managed), every instruction in the repo that +still said "render a config.xml and push it to the edge" became **actively dangerous**. This +sweep finds them and marks them. **No behaviour changes; no live system touched.** Warnings and +headers only. + +## Why this was urgent + +Office1's DHCP is now **API-managed**. A full `config.xml` push REPLACES `/conf/config.xml` +wholesale and drops ~667 migration-populated elements (measured 2026-07-13), including the only +two firewall pass rules on the box. Following the old runbook steps against the live edge would +clobber it. An instruction that destroys a working router is worse than no instruction. + +## What the sweep also caught (a PRE-EXISTING landmine, unrelated to today) + +`runbooks/dc-dc-phase2` Step 4 still instructs building a **config ISO** for DC1's edge -- but +**D-112 established that ISO can never be read**. `opnsense-importer -b` probes for a read-only +root; on a pre-installed nano the root is writable and a factory `/conf/config.xml` already +exists, so it `bootstrap_and_exit 0`s without enumerating a single device. That runbook has been +telling anyone who follows it to build an inert artifact and then wonder why the edge came up on +factory defaults -- which is EXACTLY the day we lost on 2026-07-12. It was never corrected at the +source. + +Its `WAN_IF`/`LAN_IF` "chicken-and-egg" discussion is likewise **moot**: that problem only exists +if you try to seed a full config before first boot. D-112(c) measures the real `vtnetN` mapping +*after* boot, where it is knowable. + +## Marked (5 files) + +| file | what was done | +|---|---| +| `runbooks/dc-dc-phase1-office1-standup.md` | DANGER banner; the config.xml render + ISO sub-steps REMOVED as runnable instructions (they would clobber the live Office1 edge). Image-prep retained. History preserved in git + the build changelog. | +| `runbooks/dc-dc-phase2-tofu-dc-substrate.md` | STOP banner on Step 4: the ISO is inert (D-112), full-config rendering is superseded (D-113(a2)), and the WAN_IF/LAN_IF problem is moot. Points at the proven Office1 path. | +| `scripts/opnsense-render-config.sh` | DANGER header at point-of-use. NOT dead -- under D-113(a2) it is to be REDUCED to a minimal bootstrap render (sshd + key + console). Until then, safe only for a brand-new, not-yet-booted edge. | +| `scripts/opnsense-build-config-iso.sh` | RETIRED header quoting the upstream source that proves the importer can never fire on nano. Kept (not deleted) because D-112 says retire it in the same change that reduces the template -- which has not landed. | +| `docs/dc-dc-deployment-workflow.md` | STALE-CONTENT warning at the top of the tooling gap register (the doc a fresh session reads first). | + +Changelogs were deliberately NOT touched: they are history, not instructions. Marking history is +noise; marking instructions is safety. + +## Verification + +Harnesses still green (headers only): `opnsense-render-config` 24 PASS, `opnsense-build-config-iso` +2 PASS, `opnsense-api` 21 PASS. `repo-lint` 0 fail. + +## Still OPEN (the actual D-113(a2) migration -- NOT done) + +Reduce the template to a minimal bootstrap, retire the config-ISO path and the +`opnsense-edge` module's `config_seed`/cdrom wiring, and rewrite Stage 3's edge steps around the +API. This sweep makes the repo SAFE in the meantime; it does not make it FINISHED. + +## Revert + + git revert <this commit> + +Restores the previous (dangerous) instructions verbatim. Nothing else changes. diff --git a/docs/changelog-20260713-config-xml-path-deleted.md b/docs/changelog-20260713-config-xml-path-deleted.md new file mode 100644 index 0000000..f517f26 --- /dev/null +++ b/docs/changelog-20260713-config-xml-path-deleted.md @@ -0,0 +1,92 @@ +# 2026-07-13 -- D-113(a2) COMPLETE: the config.xml path is DELETED + +Operator ruling ("I'll defer to your lean"): the OPNsense `config.xml` template is **deleted, not +reduced**. + +## Why deleted rather than reduced + +The (a2) plan was to shrink the template to a minimal bootstrap (sshd + root key + console + a +seeded API key). That turned out to be **unnecessary** -- every one of those is covered without a +`config.xml` at all: + +| bootstrap need | how it is met now | +|---|---| +| sshd + root key | the **D-112(c) console bootstrap** (proven on Office1) | +| an API key | **`opnsense-bootstrap-apikey.sh`** -- calls OPNsense's OWN model over SSH; no GUI click, no re-implemented crypto | +| DHCP / firewall / interfaces | **the REST API** (`opnsense-api.sh`), proven read AND write | + +So the provisioning chain contains **no `config.xml` anywhere**: + + boot factory nano -> console bootstrap -> mint API key -> configure over REST + +A `config.xml` renderer that nobody should ever run is **not a safety net -- it is a loaded gun +pointed at a live router.** The 2026-07-13 safety sweep is the evidence: the repo still contained +runbook steps telling an operator to render a config and push it to the edge, which by then would +have **clobbered live API-managed DHCP**. + +## Deleted + +- `opentofu/templates/opnsense-config.xml.tmpl` +- `scripts/opnsense-render-config.sh` + `tests/opnsense-render-config/` +- `scripts/opnsense-build-config-iso.sh` + `tests/opnsense-build-config-iso/` +- the `opnsense-edge` module's `config_seed` volume + cdrom disk, and its `config_iso_path` var +- the `xorriso`/`genisoimage` prereq (it existed ONLY for the ISO builder) + +All remain in git history. `opentofu/templates/README.md` is now a **tombstone** explaining what +happened and what to use instead -- people will go looking for that template. + +## The live change, and why it was safe + +Removing the module's ISO wiring touches an INSTANTIATED resource +(`libvirt_volume.config_seed` is in tfstate). Measured with `tofu plan` BEFORE applying: + + module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE + module.office1_opnsense.libvirt_volume.config_seed will be destroyed + Plan: 0 to add, 1 to change, 1 to destroy. + +**No replacement** of the running domain -- confirmed explicitly (a replacement would have +destroyed the live edge). Only the cdrom entry is removed from the disk list and the inert ISO +volume is dropped. + +### CORRECTION -- "updated in-place" DOES NOT MEAN "no restart" (measured, 2026-07-13) + +**The apply RESTARTED the guest.** This was predicted NOT to happen, and that prediction was +WRONG. Measured: uptime went from `8:36` to `6 secs` across the apply. The dmacvicar/libvirt +provider redefines the domain to apply a disk-list change and **bounces it** -- there was a brief +outage (~30s) during which the edge was not routing and not serving DHCP. + +**`libvirt_domain ... will be updated in-place` means the RESOURCE is not replaced. It says +NOTHING about whether the GUEST keeps running.** Do not read it as "no interruption". + +**This matters for Stage 3.** Any future `tofu apply` that touches a `libvirt_domain`'s devices -- +on a DC edge, or on a node VM -- should be assumed to BOUNCE THE GUEST, and must be scheduled and +gated as an outage, not as a no-op config tidy-up. + +Recovery was clean and needed no intervention: post-boot, `kea-dhcp4` is running and bound to +`10.10.0.1:67` with the subnet/pool intact, WAN/LAN/default-route up, egress 0.0% loss, 8 LAN pass +rules in pf, serial console + getty alive. `tofu plan` reports "No changes" -- repo and state are +back in sync. + +**Applying it now was the SAFE choice, not the risky one.** Leaving the repo/state divergence in +place would mean the next `tofu apply` -- e.g. while building DC1's substrate in Stage 3 -- +sweeps up this pending change as a **surprise side effect on the running Office1 edge, mid-stage**. +Deliberate and gated beats incidental and unnoticed. + +## A lint guard did its job + +`repo-lint`'s **L3** rule (runbooks must not reference missing scripts) went RED on the tombstone +notes, because they named the deleted paths. The rule was RIGHT, and it has no opt-out (only L4 +does). Rather than weaken a guard that exists to stop runbooks pointing at dead scripts, the +tombstones were reworded to name the bare filename instead of the `scripts/` path. The guard stays +fully intact; the information is preserved. + +## Verification + +`repo-lint` 0 fail. `opentofu-validate` PASS -- root + **10/10 modules** standalone. Gauntlet green. + +## Revert + + git revert <this commit> # restores the template, renderer, ISO builder, harnesses + cd opentofu && tofu apply # re-creates the (inert) config_seed volume + cdrom + +Nothing depends on the ISO: it was never read by anything (D-112). diff --git a/docs/changelog-20260713-d114-voffice1-nested-virt.md b/docs/changelog-20260713-d114-voffice1-nested-virt.md new file mode 100644 index 0000000..b98afd2 --- /dev/null +++ b/docs/changelog-20260713-d114-voffice1-nested-virt.md @@ -0,0 +1,110 @@ +# changelog 2026-07-13 -- D-114: voffice1 built; cloudinit-vm gains nested virt + +## What + +1. **`opentofu/modules/cloudinit-vm` gained a `cpu` block and an `expose_nested_virt` variable.** +2. **`opentofu/main.tf` instantiates `modules/base-image` (Ubuntu 24.04 LTS) and `modules/cloudinit-vm` + for `voffice1`** -- the first real instantiation of either module. +3. **APPLIED. `voffice1` is LIVE.** + +## Why -- the module was silently broken for D-114's entire model + +`cloudinit-vm`'s `libvirt_domain` had **no `cpu` block at all**. With none, libvirt renders a +GENERIC EMULATED CPU MODEL that exposes **no `svm` flag** -- so nested KVM is impossible in the +guest. Since LXD **virtual machines** are qemu/KVM guests, that meant **no service VM could ever +have been composed into `voffice1`**, and D-114's model would have failed at its first step, with a +confusing "KVM not available" rather than anything pointing at OpenTofu. + +Measured proof of the masking: the host is an **AMD EPYC 9965**, but a default-CPU guest is handed +an **"Opteron_G3"**. Post-fix, `voffice1` reports `AMD EPYC 9965` and `svm` on all 16 cores. + +`expose_nested_virt` is a per-caller decision with NO default, deliberately: +- **true** -- pass `svm` through. Required for the D-114 site containment VMs. +- **false** -- disable `svm`, matching `modules/opnsense-edge`, which disables it as legitimate + hardening for a router guest that has no business seeing nested virt. (Operator asked whether the + edge's `svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was *tried* as a + triple-fault fix on 2026-07-12 and did NOT resolve it -- the real cause was the memory unit -- and + was retained on hardening grounds. It stays disabled on the edge.) + +## voffice1 as built (measured, not inferred) + +- Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, `host-passthrough`, single NIC on `office1-local`. +- Reaches the internet THROUGH the OPNsense edge (default gw 10.10.0.1); egress to 1.1.1.1 at 4.6 ms. +- Cloud-init is DELIBERATELY MINIMAL (identity + SSH key + qemu-guest-agent). MAAS and LXD are + installed as separate GATED steps so they are observable and individually approved, not buried in + a first-boot script that either silently works or silently does not. +- The SSH public key is read at plan time with `file(var.office1_ssh_pubkey_path)` -- the key + material never enters a command line, the repo, or an agent's context. The private half is never + read (SEC-007 tracks its rotation). +- `network_config` matches the NIC by **glob** (`en*`), not a guessed kernel name: the name + (ens3/enp1s0/...) depends on machine type and PCI topology and is not knowable before first boot. + Naming it would have been an inferred value. (It came up `enp1s0`.) + +## Two gates CLEARED by this apply + +1. **KEA HAS SERVED ITS FIRST REAL DHCP LEASE.** Standing open item -- the daemon was proven + (bound udp/67, subnet + pool intact) but the SERVICE never was: `office1-local` had no client. + Measured via the D-113(a2) REST API: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches + voffice1's NIC exactly), hostname `voffice1`, state active. **DHCP is now proven end to end.** +2. **D-114's NESTING PROBE PASSES at L3.** Inside `voffice1`: `/dev/kvm` PRESENT, `svm` on all 16 + cores, real EPYC CPU model. CAVEAT, stated honestly: this proves nested KVM is AVAILABLE. The + DEFINITIVE L3 proof is an actual guest booting inside `voffice1` -- that lands when the first + LXD VM is composed. Do not record L3 as fully proven until then. + +## Verification + +- `scripts/opentofu-validate.sh`: PASS (all modules). +- `tofu plan`: **5 to add, 0 to change, 0 to destroy** -- confirmed BEFORE applying that the running + OPNsense edge was NOT touched (no in-place update -> no repeat of the 2026-07-13 guest bounce). +- `tofu apply`: **5 added, 0 changed, 0 destroyed.** Edge stayed up throughout. +- repo-lint 0 fail. + +## Revert + +``` +tofu -chdir=opentofu destroy -target=module.voffice1 -target=module.ubuntu_noble_base +git revert <this commit> +``` +Destroys `voffice1`, its disk/seed, and the base image; reverts the module + main.tf changes. The +OPNsense edge and every Stage-1 network/pool are untouched by both halves. NOTE the base image is a +600 MB download -- reverting and re-applying re-fetches it. + +## Follow-on FINDING (logged, NOT actioned -- DC1 is gated) + +`modules/node-vm` has the SAME missing-`cpu`-block defect. The DC OpenStack nodes run +`nova-compute`, which needs working KVM, so they will need `host-passthrough` with `svm` passed +through for the same reason. NOT fixed here: DC1 is explicitly GATED behind Office1 completing +(D-114), and fixing it now would be scope beyond the current step. It must be fixed before Stage 3 +or the DC nodes will come up unable to run a single instance. + +--- + +## ADDENDUM (same day): the DC1 fix is APPLIED, not deferred + +Operator ruling: **"Apply the fixes to DC1 as we find them. Fixes are not blocked, just deployment +at this stage."** So the `modules/node-vm` defect logged above as "NOT actioned" is now FIXED. + +`modules/node-vm` gained an UNCONDITIONAL `cpu = { mode = "host-passthrough" }` -- no +`expose_nested_virt` knob, deliberately. A DC node runs `nova-compute`; it MUST be able to run KVM +guests. There is no correct value other than this one, so no knob that could only ever be set wrong. +(Contrast `cloudinit-vm`, where the flag IS a real per-VM decision, and `opnsense-edge`, which +DISABLES svm as correct hardening for a router.) + +`modules/node-vm` is not yet instantiated, so adding this breaks no live call. Without it the DC +nodes would have come up looking healthy while being unable to launch a single instance -- +the failure would have surfaced during Stage 5 as an inscrutable scheduler error. + +`scripts/opentofu-validate.sh`: PASS (node-vm, cloudinit-vm, all modules). + +## ADDENDUM: voffice1 moved to a RESERVED, out-of-pool address + +`voffice1` first came up on `10.10.0.100` -- the first address of Kea's DYNAMIC pool +(.100-.199). Infrastructure, and a MAAS region controller above all, should not hold a +pool address that can move. + +A Kea host reservation was added through the D-113(a2) REST API (a second live exercise of that +write path): `52:54:00:6a:87:e5 -> 10.10.0.20`, deliberately OUTSIDE the dynamic pool. +`kea/dhcpv4/add_reservation` -> `{"result":"saved"}`, then `kea/service/reconfigure` -> `{"status":"ok"}`. +voffice1 was rebooted and came back on **10.10.0.20**; `.100` is released back to the pool; +`/dev/kvm` still present. Done NOW because nothing yet depends on the address -- it is the cheapest +moment it will ever be. diff --git a/docs/changelog-20260713-docfix194-opentofu-module-validation.md b/docs/changelog-20260713-docfix194-opentofu-module-validation.md new file mode 100644 index 0000000..b6330dd --- /dev/null +++ b/docs/changelog-20260713-docfix194-opentofu-module-validation.md @@ -0,0 +1,103 @@ +# DOCFIX-194 (2026-07-13) -- the OpenTofu gate was green over two broken modules + +`scripts/opentofu-validate.sh` printed **`PASS`** on 2026-07-13 while **two modules were flatly +broken** -- neither could have applied, and one could not even `init`. Both are modules that +Stage 3 and Stage 4 are built on. This fixes both defects and closes the blind spot that hid +them. + +## Root cause of the miss: root-only validation + +`tofu validate` against the ROOT module only parses modules the root actually **instantiates**. +A module that is not called -- or is called only from a **commented-out** block -- is **never +parsed at all**. Root's `main.tf` calls `dc-planes`, `dc-storage-pool`, `mesh-link`, +`office1-network` and `opnsense-edge`. It does NOT call: + + base-image cloudinit-vm maas-vm-host netem-link node-vm + +which is precisely the set Stage 2 (`cloudinit-vm`, `base-image`) and Stage 3 +(`node-vm`, `maas-vm-host`, `netem-link`) depend on. **They had never been parsed by any tool.** +This was only findable because a `tofu` binary now exists on the jumphost (OpenTofu v1.12.3) -- +the tree was authored in sessions with no binary to self-check, exactly as +`opentofu/README.md` warned. + +## Defect 1 -- `modules/node-vm`: `libvirt_volume` had a `format` attribute that does not exist + + format = { # WRONG -- rejected: "An argument named format is not expected here" + type = "qcow2" + } + +Verified against the provider's own schema (`tofu providers schema -json`, dmacvicar/libvirt +v0.9.8): `libvirt_volume` has **no top-level `format`**. It nests under `target`: + + target = { # CORRECT + format = { + type = "qcow2" + } + } + +`modules/base-image` had the correct nesting all along -- which is why it passed and node-vm did +not. **`node-vm` builds EVERY DC node** (PXE-boot blanks). Stage 3/4 would have failed on first +use. The module's own header even said "VERIFY with `tofu providers schema -json` before the +first real apply" -- nobody could, until now. + +## Defect 2 -- `modules/netem-link`: a destroy provisioner referencing `var.*` (could not `init`) + + provisioner "local-exec" { + when = destroy + command = "ssh ${var.vcloud_host_ssh_target} 'sudo tc qdisc del dev ${var.bridge_name} root' || true" + } + +OpenTofu rejects this **at init time**: + + Error: Invalid reference from destroy provisioner + Destroy-time provisioners and their connection configurations may only reference + attributes of the related resource, via 'self', 'count.index', or 'each.key'. + +So the module could not even initialize, let alone apply. Fixed with the canonical pattern -- +stash the values in `input`, read them back as `self.input.*`: + + input = { + ssh_target = var.vcloud_host_ssh_target + bridge_name = var.bridge_name + } + provisioner "local-exec" { + when = destroy + command = "ssh ${self.input.ssh_target} 'sudo tc qdisc del dev ${self.input.bridge_name} root' || true" + } + +`netem-link` is the DR/latency mechanism for Stage 3 and the **Stage 6 failover drill**. + +## The durable fix -- S3: validate EVERY module standalone + +The two bugs are one-line fixes. The real defect was the gate. `opentofu-validate.sh` now runs a +new **S3** stage that validates every directory under `opentofu/modules/` **standalone**, in a +TEMP COPY (so the gate never writes `.terraform/` or a module-level `.terraform.lock.hcl` into +the repo -- only the ROOT lock file is tracked, deliberately). + +**A gate that reports green over unparsed code is worse than no gate: it manufactures +confidence.** That is what happened here, and it is what S3 prevents. + +## Verification + +`tests/opentofu-validate/run-tests.sh`: **9 PASS** (T2 skipped -- can't exercise the +missing-binary guard on a box that has the binary). Three new cases: + +- **T8 (the load-bearing one):** a fixture whose ROOT IS VALID and does NOT call a BROKEN module. + Root-only validation reports PASS on it. S3 must FAIL. **If T8 ever goes green, the blind spot + is back.** +- **T9:** the same fixture shape with a CORRECT module passes -- proving T8 fails on the defect, + not on the fixture's shape. +- **T10:** all 10 real modules validate standalone. + +Fixtures use `terraform_data` (a builtin), so they need no provider download -- the cases stay +fast and need no registry access. + +Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, **all 10 modules +PASS standalone**. + +## Revert + + git revert <this commit> + +Restores the two broken modules and the root-only gate. Nothing was instantiated; no live system +is affected by either the bug or the fix. diff --git a/docs/changelog-20260713-git-credential-store.md b/docs/changelog-20260713-git-credential-store.md new file mode 100644 index 0000000..71dc454 --- /dev/null +++ b/docs/changelog-20260713-git-credential-store.md @@ -0,0 +1,65 @@ +# 2026-07-13 -- unattended git push from the jumphost (SEC-005) + +The jumphost clone could not `git push`: every attempt died with +`could not read Username for 'https://git.baldurkeep.com'`. The branch sat **12 commits +ahead of origin for two days** as a result -- a real data-loss exposure, since all of the +D-112 / OPNsense edge work existed only on this box. Fixed by giving git a stored +credential. Operator ruled the posture; recorded as **SEC-005**. + +## What was configured + + git config --global credential.helper store # -> ~/.git-credentials, mode 600 + +The token itself was written by the OPERATOR via a hidden prompt +(`getpass` + `/dev/tty`), URL-encoded, file created 0600 from `os.open` so it is never +briefly world-readable. **The token was never read into agent context** -- verification is +by *using* the credential (push succeeds / file survives), never by printing it. Per the +CLAUDE.md secrets rule. + +Result: `f5fd4f5..425d434` pushed; `git ls-remote` confirms origin HEAD == local HEAD. + +## Two GitBucket behaviours worth knowing (both cost us a cycle) + +**1. The git username is NOT the login you type into the web UI.** +The operator signs in to GitBucket with `jesse.austin@neumatrix.com`. That address is +rejected for git transport. Measured against the real endpoints: + +| username | `/api/v3/user` | `info/refs?service=git-receive-pack` | verdict | +|---|---|---|---| +| `jesse.austin@neumatrix.com` | 401 | 401 | rejected | +| **`jesse.austin`** | 401 | **200** | **CAN PUSH** | +| `JANeumatrix` (the git `user.name`) | 401 | 401 | rejected | + +Note the whole API column is 401: **GitBucket does not accept PAT basic-auth on +`/api/v3/`, but does accept it for git transport.** So an API probe is the WRONG way to +validate this credential -- it produces a false negative. Validate against +`info/refs?service=git-receive-pack`, which is what push actually uses. + +**2. `credential.helper store` DELETES the credential when the server rejects it.** +git calls `credential reject` on a 401, and the store helper erases the matching line. +The first (bad-username) push therefore emptied `~/.git-credentials` to 0 bytes, and the +follow-up diagnostics were all authenticating with an empty string -- which looked like a +corrupt/unparseable file and sent the investigation the wrong way for a beat. If the file +mysteriously empties, the credential was rejected; it did not fail to write. + +Consequence for tooling: **validate before writing.** The helper script now confirms push +rights against GitBucket and only then persists the token, so a bad entry cannot silently +wipe the store. + +## Security posture (SEC-005) + +A long-lived, account-wide PAT now sits in plaintext on the jumphost, readable by anything +running as `jessea123`. This is the accepted cost of unattended push, ruled by the operator. +Mitigations in place: dedicated token (not a reused personal one), mode 600, and a recorded +revoke path. GitBucket has no per-repo token scoping, so least-privilege is not achievable +here -- the control is rotation, not scope. + +**Revoke/rotate:** GitBucket -> Account Settings -> Applications. Do it at v1 close, or +immediately if the jumphost is shared, rebuilt, or its disk leaves the premises. + +## Revert + + git config --global --unset credential.helper + rm -f ~/.git-credentials + +Then revoke the token in GitBucket. Pushes go back to requiring interactive auth. diff --git a/docs/changelog-20260713-office1-dhcp-apply.md b/docs/changelog-20260713-office1-dhcp-apply.md new file mode 100644 index 0000000..4e7d8f4 --- /dev/null +++ b/docs/changelog-20260713-office1-dhcp-apply.md @@ -0,0 +1,97 @@ +# 2026-07-13 -- DOCFIX-193 APPLIED: the Office1 edge now serves DHCP + +Executed the live apply of the Kea DHCP config built (but deliberately NOT applied) under +DOCFIX-193. The Office1 OPNsense edge is now a router **and** a DHCP server for office1-local. + +## What was executed + +Target: `root@10.10.0.1` (`office1-opnsense`), the live, routing Office1 edge. +Path: the D-112(c) steady-state path -- render -> scp over SSH -> SHA-256 verify on the +guest -> install to `/conf/config.xml` -> reboot. + + set -a; . ~/vr1-office1.env; set +a + bash scripts/opnsense-render-config.sh "$OPNSENSE_CONFIG_OUT" + scp -i "$OPNSENSE_SSH_KEY" "$OPNSENSE_CONFIG_OUT" root@10.10.0.1:/tmp/config.new + # guest: sha256 -q /tmp/config.new == local sha256, else abort + # guest: cp /tmp/config.new /conf/config.xml ; reboot + +Rendered config sha256 `bc100bd49c9d2bfec7dfd799e03da7cc5fa48a2608a6521ed2dfea645366a9be`; +guest hash matched before install. Edge returned in ~30s. + +## Measured end state (post-reboot, over SSH) + +- `kea-dhcp4` running (pid 15090), **bound udp4 10.10.0.1:67** -- the deliverable. +- Kea: `enabled=1`, `interfaces=lan`, subnet `10.10.0.0/24`, pool `10.10.0.100-10.10.0.199`, + routers/DNS `10.10.0.1`. +- Router intact: WAN `172.30.1.2`, LAN `10.10.0.1`, default route `172.30.1.1`, + NAT `automatic`, egress to `1.1.1.1` **0.0% packet loss**. +- Console intact: `kern.console=ttyu0`, getty running (DOCFIX-192 holds). + +NOT yet verified end-to-end: an actual client DHCPDISCOVER->lease. No client exists on +office1-local yet. The udp/67 binding + loaded subnet is the strongest available evidence +until Stage 2 puts a host on that LAN. + +## The finding that nearly bit us (and why the push was still safe) + +A full-config push REPLACES `/conf/config.xml` wholesale. The rendered template is 128 +elements; the live config was 796. The overwrite therefore **drops 667 elements** that +OPNsense's own migrations had populated -- including, critically: + + /OPNsense/Firewall/Filter/rules/rule x2 "Default allow LAN to any" (+ IPv6) + +Those two MVC rules are the ONLY pass rules on the box (`/filter/rule` count is **0** -- +the legacy rules were migrated away in 26.1), and pf's base policy is +`block drop in log inet all`. Dropping them with no replacement = an edge that boots but +does not route. + +This was NOT resolved by inference. It was settled from the box's own `/conf/backup/` +trail, which shows the overwrite is **self-healing** and had already been exercised three +times on this exact host: + +| time (07-12) | elements | MVC rules | legacy /filter | revision | +|---|---|---|---|---| +| 07:50 | 713-733 | 0 | **2** | factory nano; rules live in legacy `/filter` | +| 20:25 | 774 | **2** | 0 | after a template push (0 rules) + boot -> `run_migrations` | +| 23:28 | 776 | **2** | 0 | after a template push (0 rules) + boot -> `run_migrations` | +| 23:30 | 796 | **2** | 0 | after a template push (0 rules) + boot -> `run_migrations` | + +Every push of a rule-less config came back with exactly 2 regenerated default-allow-LAN +rules and working routing. The same trail shows `Created web GUI TLS certificate` after +each boot, so `/ca`, `/cert` and `/system/webgui/ssl-certref` regenerate too. + +Today's apply was cycle #4 of that proven mechanism plus one additive block, and the +post-reboot measurement confirms it: 791 elements, both LAN rules back, TLS cert back, +and our `subnet4` **survived** the migration pass. + +## Defect in the executed command (logged, not fixed here) + +Step 1 of the apply -- the pre-install snapshot -- **silently failed**: + + ssh root@10.10.0.1 'cp /conf/config.xml /conf/config.xml.pre-dhcp && echo "... $(sha256 -q ...)"' + -> Illegal variable name. + +root's shell on OPNsense is **tcsh**, not sh. The `$(...)` inside the quoted remote command +is parsed by tcsh, which rejects it -- and because the failure was non-fatal, the install +proceeded **without the named rollback point**. The config was recoverable anyway (OPNsense +keeps `/conf/config.xml.prev` + `/conf/backup/`), so no harm resulted, but the guard did +not exist when it was relied upon. + +Lesson for the apply path: every remote command to this box must be fed to `sh -s` (as the +install and verify steps correctly were), never passed as a quoted string to root's tcsh. + +## Open gap + +The apply path is still **ad-hoc shell**, not a repo script with a harness -- which is why +the tcsh defect above could ship at all. It should become +`scripts/opnsense-apply-config.sh` (snapshot -> scp -> verify -> install -> reboot -> assert), +with the assertions from this session (kea on udp/67, 2 LAN rules, egress, console) as its +post-conditions. LOGGED, not executed -- it is out of scope for this step. + +## Revert + + ssh -i "$OPNSENSE_SSH_KEY" root@10.10.0.1 'sh -s' <<'SH' + cp /conf/config.xml.prev /conf/config.xml && reboot + SH + +Restores the pre-DHCP routing config (the edge routes; nothing serves DHCP). Repo-side, +DOCFIX-193 (`7a2b396`) is the commit that introduced the Kea template block. diff --git a/docs/changelog-20260713-office1-headend-maas-lxd.md b/docs/changelog-20260713-office1-headend-maas-lxd.md new file mode 100644 index 0000000..f588968 --- /dev/null +++ b/docs/changelog-20260713-office1-headend-maas-lxd.md @@ -0,0 +1,102 @@ +# changelog 2026-07-13 -- Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21, D-114 proven end to end + +## What + +`voffice1` is a working VR1 site headend. **MAAS 3.7.2** (region+rack, PostgreSQL 16.14) and +**LXD 5.21.5** are installed on it, that LXD is **registered back into MAAS as an LXD VM host**, and +MAAS has **composed, PXE-booted and COMMISSIONED** its first service VM. D-114's model is proven. + +New: `scripts/site-headend-install.sh` + `tests/site-headend-install/run-tests.sh` (18/18 PASS). +The script is the install sequence we actually executed, codified -- not written from docs. Its +`--check` mode was run against the live `voffice1` and reports every item OK. + +## The proof (measured, end to end) + +``` +vcloud (L1 -- itself a KVM guest, measured) + |- voffice1 (L2) MAAS 3.7.2 region+rack + LXD 5.21.5, 10.10.0.20 (Kea reservation) + |- office1-netbox (L3) LXD VIRTUAL-MACHINE, composed by MAAS, PXE-booted by MAAS +``` + +- `office1-netbox`: composed via `maas admin vm-host compose`, leased **10.10.1.100** from MAAS's + own DHCP on `lxdbr0`, PXE-booted, ran commissioning, and reached **status `Ready`** with MAAS + correctly detecting 2 cores / 4096 MiB. It then powered off -- correct for a commissioned machine. +- **This is the DEFINITIVE L3 nested-KVM proof** D-114 asked for. Previously we had only shown + nested KVM was AVAILABLE (`/dev/kvm` + `svm`). Now a guest has actually booted, PXE'd, + commissioned and reported its hardware three levels deep. D-114's Office1 gate is CLEARED. +- It reproduces VR0's `lxd` + `tailscale` pattern exactly: MAAS's unit of visibility is the machine; + Juju's LXD containers (the OpenStack API services) remain invisible to MAAS, as in VR0. + +## The DHCP split -- structurally safe, not safe-by-convention + +| network | authority | MAAS `dhcp_on` | +|---|---|---| +| `10.10.0.0/24` (office1-local, the site LAN) | **Kea**, on the OPNsense edge | **False** | +| `10.10.1.0/24` (`lxdbr0`, the compose network) | **MAAS** | **True** | + +Two DHCP servers, two **physically separate L2s** -- `lxdbr0` is never bridged onto the site LAN, so +they cannot see each other. The script asserts this after configuring DHCP and FAILS if MAAS DHCP is +on for any subnet other than the one it was told about. + +**`10.10.1.0/24` is a NEW allocation that NetBox has not blessed.** It was chosen deliberately (DC1 +is `10.12.x`, DC2 `10.13.0.0/19`, the tenant pool is `10.20.0.0/16` per D-016 -- the obvious-looking +`10.20.x` would have collided with tenant space). It must be registered in NetBox once NetBox +exists. This is the known chicken-and-egg: NetBox is one of the VMs this headend composes. + +## FOUR TRAPS, all hit for real, all now encoded in the script and asserted by the harness + +1. **`lxd init --auto` FAILS on a host that also runs MAAS.** It creates `lxdbr0` with LXD's own + DHCP+DNS on, which starts a dnsmasq; dnsmasq binds the **wildcard** `0.0.0.0:53`, and MAAS's + bind9 (`named`) already holds `:53`. Verbatim: + `Failed starting network: The DNS and DHCP service exited prematurely: exit status 2 + ("dnsmasq: failed to create listening socket for 10.10.1.1: Address already in use")` + The message names the BRIDGE address, which sends you hunting the wrong thing -- the real + conflict is the wildcard bind. **`ipv4.dhcp=false` + `dns.mode=none` are NOT sufficient**: LXD + still spawns dnsmasq for a managed bridge and it still tries to bind `:53`. The fix that works is + `raw.dnsmasq="port=0"`. Verified after the fact: dnsmasq runs but holds **zero** sockets on `:53` + and **zero** on `:67`. +2. **`lxc` reads its config from STDIN.** Pipe this script to `bash -s` over ssh and the first `lxc` + call without a redirect **eats the rest of the script as YAML** (`Error: yaml: unmarshal errors: + line 1: cannot unmarshal !!str 'echo "...' into api.StoragePoolPut`) and everything after it + silently vanishes. Every `lxc` call now ends in `</dev/null`. Same class as the OPNsense tcsh trap. +3. **LXD snap track changes are ONE-WAY.** LXD must be installed DIRECTLY onto `5.21/stable` and + never allowed to land on `latest` first, because MAAS 3.6/3.7 is incompatible with LXD >= 6.7 + (pinned pylxd 2.3.5 vs LXD 6.7's consolidated endpoints). Bonus discovered in research: + `core.trust_password` still exists in 5.21 and is GONE in 6.x, so the pin also preserves the + scriptable registration path. https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 +4. **MAAS DHCP on the compose network ONLY.** `--compose-cidr` is REQUIRED and has no default, + precisely so it can never be picked by accident, and the script hard-fails if MAAS DHCP ends up + enabled on any other subnet. + +Also encoded: `systemd-timesyncd` must be disabled (MAAS manages time via chrony); a **dynamic +iprange must exist before `dhcp_on=True` leases anything**; `dir` storage driver chosen over zfs +deliberately (no zfs loop file inside an already-nested guest). + +## Discipline notes + +- Every MAAS/LXD flag used here was read from the tool's OWN `--help` **on the box** before use + (`maas init --help`, `maas createadmin --help`, `maas vm-host compose --help`), rather than + trusted from documentation. `--password` on `createadmin` and `storage=label:size` on `compose` + were confirmed this way. +- All secrets (DB password, MAAS admin password, MAAS API key, LXD trust password) are generated ON + the target, stored `0600` under `/root/maas-secrets/`, and were never printed into the session. +- The harness asserts the four traps are still present in the script -- a "simplification" that drops + one goes RED in the gauntlet rather than in production. + +## Verification + +- `tests/site-headend-install/run-tests.sh`: **18/18 PASS**. Includes a self-check that the harness + actually catches a regression (removing the 5.21 pin makes it fail). +- `scripts/site-headend-install.sh --check` against the LIVE `voffice1`: every item OK. +- `repo-lint`: 0 fail. Full gauntlet: ALL GREEN. + +## Revert + +``` +ssh voffice1 'sudo snap remove maas postgresql lxd' # or destroy voffice1 entirely: +tofu -chdir=opentofu destroy -target=module.voffice1 +git revert <this commit> +``` +Reverting the commit removes the script + harness. The OPNsense edge is untouched. Note the Kea +reservation for voffice1 lives on the edge, not in this commit -- remove it via the REST API +(`kea/dhcpv4/del_reservation`) if voffice1 is destroyed for good. diff --git a/docs/changelog-20260713-office1-netbox-deployed.md b/docs/changelog-20260713-office1-netbox-deployed.md new file mode 100644 index 0000000..ef3b55a --- /dev/null +++ b/docs/changelog-20260713-office1-netbox-deployed.md @@ -0,0 +1,93 @@ +# changelog 2026-07-13 -- office1-netbox DEPLOYED; NetBox 4.6.4 live on the Office1 headend + +## What + +`office1-netbox` -- the MAAS-composed LXD VM inside `voffice1` -- is **Deployed** (Ubuntu 24.04.4, +laid down by MAAS) and runs **NetBox 4.6.4** (netbox-docker, Django 6.0.6). Its API authenticates. + +This completes the D-114 chain in its final form: MAAS composed the machine, PXE-booted it, +commissioned it, deployed an OS onto it, and we installed a real service on it -- all three levels +of nesting deep, inside a containment VM, on a host that is itself a KVM guest. + +## FOUR findings, all hit for real. Two are now guarded in code. + +### 1. MAAS-discovered subnets have NO gateway -- a deployed machine gets NO default route (FIXED) + +MAAS auto-discovers the compose subnet from `lxdbr0`'s interface, and that discovery carries **no +`gateway_ip`**. So the first deploy produced a machine that looked healthy -- it had an address, +and DNS even resolved (MAAS's bind9 is link-local) -- but had **no default route and no egress**. +The failure only surfaces at the first `apt install`, which hangs. + +Fixed in `scripts/site-headend-install.sh`: it now sets `gateway_ip=<bridge-ip>` on the compose +subnet (lxdbr0 IS the gateway -- it does the NAT). `tests/site-headend-install/` asserts it (19/19). +`office1-netbox` was released and redeployed to prove the fix rather than hand-patching a route. + +### 2. NetBox 4.6 v2 API tokens: the wire format is `nbt_<key>.<plaintext>` -- NOT the `token` field + +This cost the most time and is the least guessable thing here. + +NetBox 4.6 **hashes** API tokens. The `Token` model has `key` (a 12-char PREFIX), `plaintext` (40), +`hmac_digest` (64) and a `version`. Authentication (`netbox/api/authentication.py`) infers the +version **from a prefix**: + +```python +version = 2 if auth_value.startswith(TOKEN_PREFIX) else 1 # TOKEN_PREFIX = 'nbt_' +... +key, plaintext = auth_value.removeprefix(TOKEN_PREFIX).split('.', 1) +``` + +So a v2 token must be presented as **`nbt_<key>.<plaintext>`** (57 chars). `POST +/api/users/tokens/provision/` returns `key` and `token` as **separate fields**, and **the `token` +field ALONE IS NOT USABLE** -- present it by itself and NetBox parses it as a legacy v1 token, +fails the lookup, and returns `403 {"detail":"Invalid v1 token"}`. That error names v1 while you +are holding a v2 token, which is thoroughly misleading. + +Consequences for anything scripted against this NetBox (incl. `netbox/dc-dc-prefixes-import.py` +and any `pynetbox` client): **assemble the token, do not use the API's `token` field raw.** + +Related: `SUPERUSER_API_TOKEN` in netbox-docker's env is **NOT honored** on this version (the +entrypoint logs "No API token was created ... SUPERUSER_API_TOKEN and SUPERUSER_API_KEY are not +set" even when it IS set). Use the provisioning endpoint instead. And do NOT try to construct a +Token row by hand -- setting `plaintext` trips the DB's `enforce_version_dependent_fields` check +constraint. Call the interface; do not re-implement it. (This repo already learned that lesson +minting OPNsense API keys -- and I re-learned it here the hard way.) + +### 3. A WRONG HYPOTHESIS, recorded because the mistake is reusable + +When the token kept 403-ing I concluded that rotating `SECRET_KEY` **after** NetBox had initialised +had broken the token HMAC pepper, and I wiped the database (`docker compose down -v`) to re-init +with the key in place. **That hypothesis was WRONG.** The 403 was entirely the v2 wire format above. +The wipe was harmless -- the DB was empty -- but the reasoning was not, and a wipe on a populated +NetBox would have been destructive. The lesson: `Invalid v1 token` is a FORMAT error, not a crypto +or key-rotation error. Read the auth code before reaching for a destructive reset. + +### 4. netbox-docker ships a PUBLIC default `SECRET_KEY` + +`env/netbox.env` in the public netbox-docker repo carries a real, published `SECRET_KEY`. Left +alone it is session-forgery material on any reachable instance. **Rotated** to a generated value. +(`SKIP_SUPERUSER=true` is the shipped default, so at least no default admin/admin exists.) + +## As deployed (measured) + +- `office1-netbox`: Ubuntu 24.04.4, 2 vCPU / 4096 MiB / 40 GB, **10.10.1.201** (MAAS-assigned, + outside the dynamic range), gateway `10.10.1.1`, egress OK. `systemd-detect-virt` -> `kvm` (L3). +- NetBox **4.6.4** on Django 6.0.6, containers `netbox` + `netbox-worker` + `postgres` + `redis` + + `redis-cache`, published on `:8000`. `GET /api/status/` -> **HTTP 200**. +- Secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored `0600` under + `/root/netbox-secrets/`, never printed into the session. + +## Reachability caveat (not yet solved) + +NetBox is on `10.10.1.201`, which sits **behind `voffice1`'s NAT** on `lxdbr0`. It is reachable +from `voffice1` but NOT from `office1-local` or the operator's workstation. To make it reachable, +the OPNsense edge needs a static route for `10.10.1.0/24` via `10.10.0.20`. That is a live edge +change (REST API, D-113(a2)) and is NOT done -- it should be gated with the operator. + +## Still to do + +- The D-115 carve is NOT yet imported into this NetBox (it is the next step). +- `10.10.1.0/24` + the D-115 Office/Edge roles need registering here, then feeding back upstream. + +## Verification + +`tests/site-headend-install/run-tests.sh` 19/19 PASS. `repo-lint` 0 fail. diff --git a/docs/changelog-20260713-opnsense-api-client.md b/docs/changelog-20260713-opnsense-api-client.md new file mode 100644 index 0000000..1a03851 --- /dev/null +++ b/docs/changelog-20260713-opnsense-api-client.md @@ -0,0 +1,63 @@ +# 2026-07-13 -- D-113(a2) step 1: a thin OPNsense REST API client + +First delivery under the D-113 ruling (ADOPTED: option (a2) -- stay on OPNsense, move +configuration OFF hand-authored `config.xml` and ONTO the documented REST API). + +## What shipped + +- `scripts/opnsense-api.sh` -- thin REST client. `[--dry-run] <GET|POST> <api-path> [json-body]`. +- `tests/opnsense-api/run-tests.sh` -- offline harness, **21 PASS**, no network, no real key. +- Gauntlet ALL GREEN (**53** harnesses, was 52). `repo-lint` 0 fail. + +## Why this exists (the D-113 argument, in one line) + +Hand-authoring the appliance's GUI-owned `config.xml` was the SINGLE root cause of DOCFIX-191 +(no sshd/key -> lockout), DOCFIX-192 (no console), DOCFIX-193 (no DHCP), and the 667-element +migration self-heal we ended up depending on. **None of those is expressible through the API**, +where sshd, DHCP and firewall rules are typed resources with defaults. You cannot forget to +enable sshd in a format where sshd is a typed field. + +## Design decisions worth keeping + +**The secret never reaches argv.** Credentials are read from a file and passed to `curl` via +`--config` on STDIN. `curl -u key:secret` would put the secret in the process's argv, where +ANY user on the box can read it out of `ps`. Harness case **T9** stubs `curl` with an argv +recorder and asserts the secret is absent from argv and present on stdin -- so a future +"simplification" to `curl -u` turns the harness red. That test is the point of the file. + +**The API host is never inferred** (hard rule 2). `$OPNSENSE_API_HOST` unset is a loud failure, +not a default of `10.10.0.1`. Harness T3. + +**`--insecure` is deliberate and scoped.** The edge presents a self-signed cert that OPNsense +REGENERATES ON EVERY BOOT (measured: "Created web GUI TLS certificate" appears in the config +revision trail after each reboot), so pinning it is pointless. Acceptable ONLY because the +transport is a private lab LAN leg (virbr2). Not to be copied to anything tenant-facing. + +**Dry-run reads no credentials** (T11), which is what lets the harness prove URL construction +offline. + +## Not done yet -- the blocking step + +**The API is alive but has no key.** Measured on the live edge 2026-07-13: lighttpd IS listening +on 443/80 and the API answers 401 (i.e. it is up and demanding auth), and `root` has **0 API +keys**. + +Minting a key must happen in the **GUI** (System > Access > Users > root > API keys > "+"). +That is deliberate: writing `apikeys` into `config.xml` by hand would be the exact anti-pattern +D-113 just retired. The downloaded key/secret file goes to +`~/vr1-office1-creds/opnsense-api.txt` (operator-only; never read into agent context -- SEC-007 +covers that directory). + +Once the key exists, the proof-of-path is: drive the DHCP subnet we pushed by hand yesterday +through the API instead, and confirm it round-trips. If the API can express it, the template +retires to a MINIMAL bootstrap config (sshd + root key + console + the API key) and everything +else -- DHCP, firewall, interfaces -- moves to the API. If it cannot, that is the signal that +OPNsense is not buying the abstraction we are paying for, and D-113 gets revisited. + +## Revert + + git rm scripts/opnsense-api.sh + git rm -r tests/opnsense-api/ + +Nothing live is touched by this change; the client has never been run against the edge (no key +exists yet). diff --git a/docs/changelog-20260713-opnsense-api-proven.md b/docs/changelog-20260713-opnsense-api-proven.md new file mode 100644 index 0000000..c06ec6c --- /dev/null +++ b/docs/changelog-20260713-opnsense-api-proven.md @@ -0,0 +1,64 @@ +# 2026-07-13 -- D-113(a2) PROVEN: the OPNsense REST API round-trips the edge config + +The API path adopted in D-113 (option a2) is now demonstrated working against the LIVE Office1 +edge. This closes the open question that gated the ruling. + +## Bootstrap (the one manual step, by design) + +An API key was minted for `root` in the GUI. In OPNsense **26.1** this is NOT in the user-edit +dialog and NOT in the ApiKeys tab (that tab only wires `search_api_key` + `del_api_key` -- it +can list and delete, but CANNOT create). It is a **row action on the Users list**: the +`fa-ticket` icon, "Create and download API key for this user" (source: +`mvc/app/views/OPNsense/Auth/user.volt`). Earlier instructions in this repo referencing +"edit user -> API keys -> +" describe the OLD UI and are wrong for 26.1. + +`ApiKeyField::add()` mints `key` and `secret` as `base64(random_bytes(60))` (80 chars each) and +persists `key|crypt(secret,'$6$')` -- **the secret is shown exactly once and is stored hashed**. +Lose it and you delete the key and mint a new one. + +Key stored at `~/vr1-office1-creds/opnsense-api.txt` (0600, operator-only, never read into agent +context). SEC-007 amended to cover it. + +## Measured, against the live edge + + GET core/firmware/status -> HTTP 200, OPNsense 26.1 (auth works) + GET kea/dhcpv4/get -> dhcpv4.general.enabled = 1, interfaces = lan + POST kea/dhcpv4/search_subnet -> 1 row: + + subnet = 10.10.0.0/24 + pools = 10.10.0.100-10.10.0.199 + option_data.routers = 10.10.0.1 + option_data.domain_name_servers = 10.10.0.1 + option_data.domain_name = office1.vr1.cloud.neumatrix.local + description = LAN DHCP + +The API reads back EXACTLY the config we pushed via config.xml yesterday -- and exposes it as +TYPED, NAMED FIELDS (`option_data.routers`, `pools`, ...), not free-form XML. That is the whole +(a2) argument, concretely: DOCFIX-191 (missing sshd), DOCFIX-192 (silenced console) and +DOCFIX-193 (an inert ISC `<dhcpd>` block against a Kea backend) are not expressible here. You +cannot omit a typed field that has a default, and you cannot address the wrong DHCP backend when +the endpoint IS the backend. + +## Endpoint map (measured from Dhcpv4Controller.php, not guessed) + + get | searchSubnet | getSubnet | addSubnet | setSubnet | delSubnet + searchReservation | getReservation | addReservation | setReservation | delReservation + downloadReservations | uploadReservations | searchPeer | ... | delPeer + +Note OPNsense maps camelCase actions to snake_case URLs: `searchSubnetAction` -> +`POST /api/kea/dhcpv4/search_subnet`. Response root key is `dhcpv4` (NOT `dhcp4` -- the config +XML element name differs from the API key name; do not infer one from the other). + +## NOT done yet + +- **No WRITE has been performed through the API.** Read is proven; the write path + (`setSubnet` + a reconfigure) is a live mutation and remains gated. +- The `config.xml` template is still in place. It should now be reduced to a MINIMAL bootstrap + (sshd + root key + console + nothing else), with DHCP/firewall/interfaces moving to the API. + Until that lands, DO NOT push a full rendered config.xml -- it would clobber API-managed state. + +## Revert + +Nothing to revert: this change is read-only against the edge plus documentation. To revoke the +credential: OPNsense GUI -> System > Access > Users > root row > ApiKeys tab -> delete, then +`rm ~/vr1-office1-creds/opnsense-api.txt`. diff --git a/docs/changelog-20260713-opnsense-api-write-proven.md b/docs/changelog-20260713-opnsense-api-write-proven.md new file mode 100644 index 0000000..ca12e82 --- /dev/null +++ b/docs/changelog-20260713-opnsense-api-write-proven.md @@ -0,0 +1,64 @@ +# 2026-07-13 -- D-113(a2) write path PROVEN on the live edge + +The READ half was proven earlier today. This closes the other half: **configuration written +through the REST API reaches the running daemon.** (a2) is now demonstrated end to end, not +argued. + +## What was executed (live, operator-approved) + + POST kea/dhcpv4/set_subnet/93473635-f2aa-5c0a-986c-438e59088c6c -> {"result":"saved"} + POST kea/service/reconfigure -> {"status":"ok"} + GET kea/service/status -> {"status":"running"} + +Deliberately IDEMPOTENT: the payload was the exact values read back from the API moments before +(`subnet 10.10.0.0/24`, `pools 10.10.0.100-10.10.0.199`, `routers`/`domain_name_servers` +`10.10.0.1`, `description "LAN DHCP"`). Nothing about the edge's behaviour should change -- the +point was to exercise the WRITE path, not to alter the edge. + +## GROUND TRUTH after the write (not the API's self-report) + +The API saying `{"result":"saved"}` is the service's own verdict. What was actually measured on +the box: + + kea-dhcp4 pid 30027 (RESTARTED), bound udp4 10.10.0.1:67 + +Kea's OWN generated config (`/usr/local/etc/kea/kea-dhcp4.conf` -- the daemon's JSON, not +OPNsense's XML): + + interfaces : ['vtnet0'] + subnet : 10.10.0.0/24 + pools : ['10.10.0.100-10.10.0.199'] + routers : 10.10.0.1 + domain-name-servers : 10.10.0.1 + domain-name : office1.vr1.cloud.neumatrix.local + +Router unaffected (the endpoint touches only Kea, but verify, never assume): +WAN 172.30.1.2, LAN 10.10.0.1, default route 172.30.1.1, 8 LAN pass rules loaded in pf, +egress to 1.1.1.1 at **0.0% packet loss**. + +**The full chain works: API write -> OPNsense renders -> Kea loads -> daemon serves.** + +## Payload shape (measured -- do not infer this) + +OPNsense GET returns select fields as `{value: {selected: 0|1}}`; SET expects the comma-joined +selected values. A GET response is therefore NOT directly re-POSTable -- it must be flattened: + + {"routers": {"10.10.0.1": {"value": "10.10.0.1", "selected": 1}}} # what GET returns + {"routers": "10.10.0.1"} # what SET expects + +`kea/service/reconfigure` is the apply verb (ServiceController extends +ApiMutableServiceControllerBase; `$internalServiceName = 'kea'`). Without it the config is saved +but the daemon is not reloaded. + +## Consequence: the config.xml push path is now RETIRED IN PRACTICE + +DHCP on this edge is now API-managed. **Pushing a full rendered `config.xml` would clobber it.** +Do not run the old scp/install/reboot path against Office1. The template must be reduced to a +minimal bootstrap (sshd + root key + console) before it is used again -- that is the next step +under D-113(a2), and it is NOT yet done. + +## Revert + +Re-POST the previous values (they are unchanged, so nothing to undo), or from the GUI: +Services > Kea DHCP > DHCPv4 > Subnets. The pre-API `config.xml` backups remain in +`/conf/backup/` on the edge. diff --git a/docs/changelog-20260713-opnsense-apikey-bootstrap.md b/docs/changelog-20260713-opnsense-apikey-bootstrap.md new file mode 100644 index 0000000..02932ef --- /dev/null +++ b/docs/changelog-20260713-opnsense-apikey-bootstrap.md @@ -0,0 +1,83 @@ +# 2026-07-13 -- D-113(a2): API-key bootstrap, with no GUI click and no re-implemented crypto + +The last unknown in D-113(a2) edge provisioning. The API path was proven (read AND write), but +every new edge still needed a **manual GUI click** to mint its API key -- which would have put a +human in the middle of building DC1's and DC2's edges in Stage 3. Closed. + +## What shipped + +- `scripts/opnsense-mint-apikey.php` -- runs ON the edge; calls **OPNsense's own model** + (`Auth\User` -> `apikeys->add()`), the exact code path the GUI's ticket button invokes. +- `scripts/opnsense-bootstrap-apikey.sh` -- ships the minter over SSH, runs it, retrieves the + key, wipes the remote copies. +- `tests/opnsense-bootstrap-apikey/run-tests.sh` -- offline harness, **8 PASS** (ssh/scp stubbed; + no edge contacted, no real key minted). + +## The design decision, and why the alternative was rejected + +OPNsense stores API secrets as `crypt(secret, '$6$')` -- SHA-512 with an EMPTY salt (verified on +the live 26.1 box: the stored hash is `$6$$` + 86 chars). So we *could* mint keys offline and +seed them into a bootstrap `config.xml`. + +**We deliberately do not.** That would couple our provisioning to a vendor hash format we would +have to keep byte-compatible forever -- and a mismatch **fails silently**: keys that simply never +authenticate, with no error at generation time. Calling the vendor's own generator has no format +to keep in sync. + +This is the same principle as D-113(a2) itself: **call the interface, do not re-implement the +internals.** Hand-reimplementing OPNsense's internals is exactly what cost us DOCFIX-191/192/193. + +(An offline-hash attempt was actually blocked by the permission classifier mid-session. The block +was correct, and it pushed us to the better design.) + +## VERIFIED END TO END on the live edge (2026-07-12/13) + +Minted a SECOND key on Office1 via the vendor model, proved it works, then deleted it: + + mint via OPNsense's own model -> key=80 chars, secret=80 chars (base64 of 60 random bytes) + GET core/firmware/status -> HTTP 200 [the vendor-minted key AUTHENTICATES] + POST auth/user/del_api_key/<id> -> {"result":"deleted"} + search_api_key -> 1 row remaining (the operator's original key) + retired key re-tested -> REJECTED [negative test: deletion really took] + +The operator's existing key was untouched throughout, and the temporary creds file was shredded +from the jumphost. The edge is back to its prior state. + +## Guard clause worth keeping (harness T5) + +The script **refuses to overwrite an existing creds file.** An existing file means a LIVE key on +the edge; overwriting the local copy would **strand it there permanently**, because the secret is +hashed on save and can never be read back. That is unrecoverable credential loss, and a guard +clause is the only thing standing in front of it. T5 asserts both the refusal AND that the +existing file is left byte-intact. + +The secret is never printed: creation is the only moment it exists in cleartext. It goes straight +to a 0600 file in the `key=`/`secret=` form `scripts/opnsense-api.sh` parses. + +## What this unblocks + +Edge provisioning is now automatable end to end, with no human in the loop: + + boot factory nano + -> D-112(c) console bootstrap (enable SSH + install key) + -> scripts/opnsense-bootstrap-apikey.sh <-- THIS (no GUI click) + -> scripts/opnsense-api.sh <-- everything else: DHCP, firewall, ifaces + +No `config.xml` anywhere in that chain. + +## Still OPEN + +- **The template reduction itself is NOT done.** `opentofu/templates/opnsense-config.xml.tmpl` + still renders a FULL config. Under D-113(a2) it should shrink to a minimal bootstrap or vanish + entirely -- the chain above suggests it may not be needed at all, since the console bootstrap + plus this script covers first contact. That is the next call. +- The `opnsense-edge` module still wires `config_seed` + a cdrom disk for the INERT config-ISO + path (D-112). Removing them touches an INSTANTIATED resource (`libvirt_volume.config_seed` is + in state), so it is a live change, not a docs change -- deliberately not done here. + +## Revert + + git rm scripts/opnsense-bootstrap-apikey.sh scripts/opnsense-mint-apikey.php + git rm -r tests/opnsense-bootstrap-apikey/ + +Nothing live depends on these: the live edge's key was minted via the GUI and is unaffected. diff --git a/docs/changelog-20260713-workflow-doc-status-correction.md b/docs/changelog-20260713-workflow-doc-status-correction.md new file mode 100644 index 0000000..c8be9e2 --- /dev/null +++ b/docs/changelog-20260713-workflow-doc-status-correction.md @@ -0,0 +1,43 @@ +# 2026-07-13 -- the workflow doc's status was materially FALSE + +`docs/dc-dc-deployment-workflow.md` is the first thing a fresh session reads. Its status claims +were written 2026-07-09 and never updated, so they had drifted into being **wrong in ways that +would misdirect real work**. Corrected against MEASURED state, not impressions. Docs only; no +code, no live system touched. + +## What was false, and what is true + +| claim in the doc | reality (measured 2026-07-13) | +|---|---| +| gap #2: "UNVALIDATED -- no `tofu` binary available" | **OpenTofu v1.12.3 IS installed.** The tree validates: root + **10/10 modules**. It has also been **APPLIED** -- 11 resources in state. | +| `opentofu/` "SCAFFOLDED, UNVALIDATED" | VALIDATED and partly APPLIED. But **2 modules were BROKEN and had never been parsed** (DOCFIX-194) -- see below. | +| gap #17: "no per-site ISP-uplink network exists anywhere, for ANY site" | **CLOSED FOR OFFICE1.** `office1-wan` is live (virbr11, NAT, `172.30.1.0/24`); the edge's WAN is on it at `172.30.1.2`, egress verified. The gap's own proposed answer (a dedicated per-site uplink network, NOT a mesh leg) is what was built. **DC1/DC2 still have none** -- `dc1-provider-public` has no IP and no forward. | +| `modules/opnsense-edge` "BUILT, UNVALIDATED (no real boot)" | **BUILT, INSTANTIATED, RUNNING.** Routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on udp/67. | +| `templates/opnsense-config.xml.tmpl` "BUILT, renderer fully tested" | **SUPERSEDED by D-113(a2)** -- config is REST-API-managed. A full config.xml push would now **CLOBBER** live DHCP. | +| `modules/netem-link` "BUILT, UNVALIDATED" | **WAS BROKEN** (destroy provisioner referencing `var.*` -- could not even `init`). Fixed 2026-07-13 (DOCFIX-194). | +| Stage 2 "RUNBOOK WRITTEN, NOT YET EXECUTED" | **PARTIALLY EXECUTED.** The edge + both Office1 networks are live. The **three headend VMs are NOT built.** | + +## The trap this correction removes + +Stage 2's gate row said the edge was "blocked on gap #17's WAN-side network." Both halves are +now false, and someone reading it would go build a network that already exists. + +Worse, the Stage 2 gate lists "NetBox authoritative" and "GitBucket serving" -- and NetBox and +GitBucket **do** answer, at `baldurkeep.com`. But those are **PRE-EXISTING services, not Office1 +headend VMs**. Measured: `office1-opnsense` is the ONLY VM on the vcloud host. It would be very +easy to tick that gate as met and move to Stage 3 with no headend at all. The corrected State +section says so explicitly. + +## What remains of Stage 2 (the real critical path) + +The three headend service VMs -- MAAS-region, NetBox, GitBucket -- do not exist. Per the runbook +they are blocked on a genuine decision, not on typing: Option A (`modules/cloudinit-vm`) needs +`user_data`/`meta_data`/`network_config` designed and an image source chosen; Option B is manual +`virt-install`, logged as debt. Both `cloudinit-vm` and `base-image` now at least VALIDATE +(DOCFIX-194) -- they never had before. + +## Revert + + git revert <this commit> + +Restores the stale (false) status claims. diff --git a/docs/changelog-20260714-c4-sandbox-loop-and-upstream-guard.md b/docs/changelog-20260714-c4-sandbox-loop-and-upstream-guard.md new file mode 100644 index 0000000..5814dd3 --- /dev/null +++ b/docs/changelog-20260714-c4-sandbox-loop-and-upstream-guard.md @@ -0,0 +1,54 @@ +# 2026-07-14 -- C4 (sandbox loop documented) + the upstream-write guard it exposed + +## C4 -- the NetBox sandbox loop is now documented (Stage 2 close-out) + +Two placements, no duplication of rationale: +- `docs/dc-dc-netbox-buildout-scope.md` **section 8** -- the design: why a sandbox, why a fidelity + check (the 17-of-90 silent scope drop), the 2026-07-14 hardening (do not trust a pre-hardening + "faithful" verdict), the five-tool write-guard table, the WAF trap, and the C2 preconditions. +- `runbooks/dc-dc-phase1-office1-standup.md` **Step 10b** -- the operational sequence (dump -> + seed -> apply delta -> fidelity-check -> operator-gated upstream write), with the two-host / + two-token separation called out as the safety property. Step 10's stale "the literals do not + exist yet" note was corrected (D-115/117/118 assigned them). + +Drafted by a subagent, then corrected against this session's own changes before placement (the +draft predated the two fixes below and would otherwise have documented a now-false state). + +## The guard C4 exposed: the two pynetbox importers had NO upstream gate + +Documenting the write-guard architecture surfaced that `roles-aggregates-import.py` and +`dc-dc-prefixes-import.py` had **no hostname gate at all** -- unlike `sandbox-seed.py` (structural +refusal) and `d115-office-carve.py` (`SANDBOX_HOSTS` + `--yes-write-upstream`). Until earlier today +the WAF's 403 on the default User-Agent was the ONLY thing stopping an accidental `--commit` against +the production apex -- and the WAF fix (`changelog-20260714-netbox-importers-waf-useragent.md`) +removed that accidental safety. So an explicit gate was required, and is now added: + +- both importers gain `--yes-write-upstream`; a `--commit` whose `NETBOX_URL` host is not in + `SANDBOX_HOSTS = {localhost, 127.0.0.1, 10.10.1.201}` REFUSES without it. Same pattern as + `d115-office-carve.py`, so all three write-capable importers now behave identically. + +This is defence-in-depth with the whole-plan preflights (which prevent a half-write); the guard +prevents the write from *starting* against the wrong target. + +## Tests + +- `tests/dc-dc-prefixes-import/` 86 -> **91**: `--commit` to a non-sandbox host is REFUSED without + `--yes-write-upstream` (and writes nothing); WITH the flag it proceeds; a sandbox host needs no + flag. The write-path fake tests now target the known sandbox host so the guard passes transparently. +- `tests/roles-aggregates-import/` 20 -> **24**: asserts the flag, the `SANDBOX_HOSTS` gate, the + refusal message, and that the guard actually checks the flag. + +GAUNTLET ALL GREEN (58); repo-lint 0 fail. + +## Stage 2 close-out after this change + +C1 DONE, C3 DONE, **C4 DONE**. C2 (feed the carve upstream) is the ONLY remaining item -- a +production write, still gated, with its preconditions now documented (section 8.6): re-verify the +sandbox under the hardened check, run the pynetbox importers from office1-netbox, pass +`--yes-write-upstream`, approve each mutation. + +## Revert + + git revert <this commit> + +Docs + one argparse flag and guard block per importer + their tests. No NetBox was written. diff --git a/docs/changelog-20260714-d119-region-qualified-dc-namespace.md b/docs/changelog-20260714-d119-region-qualified-dc-namespace.md new file mode 100644 index 0000000..8d06fe3 --- /dev/null +++ b/docs/changelog-20260714-d119-region-qualified-dc-namespace.md @@ -0,0 +1,108 @@ +# 2026-07-14 -- D-119: region-qualify the VR1 DC namespace (Stage 2 close-out C3 / gap #19) + +Closes the collision that BLOCKED Stage 3. **D-119 COMPLETES D-117; it does not reverse it.** + +## The bug, in one line + +The token `dc0` meant **two different clouds** depending on which file you were reading: VR0's LIVE +testcloud in `scripts/lib-net.sh`, and VR1's FIRST DC in the NetBox importer. One string, two +clouds, one of them in production. + +## The ruling (operator, 2026-07-14) + +The repo adopts the apex's names verbatim, region-qualified: + + vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region) + vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48 + vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48 + +Bare `dc0`/`dc1`/`dc2` are now **REJECTED LOUDLY** everywhere. Accepting them "for compatibility" +would preserve the exact ambiguity being deleted. + +**NO PRODUCTION IPAM WRITE.** The apex was already correct and self-consistent; the REPO was the +only surface out of step. MEASURED against `netbox.baldurkeep.com` before any edit: + + 2602:f3e2:f02::/48 -> site vr1-dc0 "Virtual Region 1 (VR1) Datacenter 0 (DC0)" + 2602:f3e2:f03::/48 -> site vr1-dc1 "Virtual Region 1 (VR1) Datacenter 1 (DC1)" + +The alternative (rename the apex to match the repo's `dc1`/`dc2`) was considered and REJECTED: it +needed a production write on the IPAM authority, made VR1 the only 1-indexed region, and would have +created a worse ambiguity (`vr1-dc1` = VR1's FIRST DC vs `vr0-dc1` = VR0's SECOND). See D-119. + +## The prize: the off-by-one becomes structurally impossible + +The importer's DC->site map is now an **IDENTITY** (`vr1-dc0` -> slug `vr1-dc0`). There is no offset +table left to get wrong -- and the original defect was precisely a WRONG LOOKUP TABLE. An assert +enforces it: + + assert all(k == v["slug"] for k, v in SITES.items()) # D-119 + +## TWO REAL BUGS found by the review sweep, which the naming fix alone would NOT have closed + +**BREAK-1 -- corrupted descriptions.** The importer built labels with `f"VR1 {dc.upper()} ..."`. +Fine when `dc` was `"dc0"`; under D-119 `dc` is `"vr1-dc0"`, rendering **`"VR1 VR1-DC0 +provider-public"`** on all 36 prefixes. Same defect class as the original bug -- DERIVING a label by +munging a token instead of looking it up -- hiding in the description field, where slug-focused +review missed it. Now `f"{site_cfg['name']} ..."`, looked up. + +**BREAK-2 -- `DC_GUA_PREFIX` was never cross-checked against `--dc`.** THIS IS THE IMPORTANT ONE. +`--dc vr1-dc0 DC_GUA_PREFIX=<f03::/48>` was **accepted** -- it writes the SECOND DC's GUA prefixes, +carved with the FIRST DC's ULA nibble (`DC_V6_INDEX` is keyed off `--dc`, independently), all scoped +to the FIRST DC's site. **A silently mis-bound datacenter, assembled from two disagreeing sources.** +Identity-mapping the site slug makes the *slug* unfalsifiable but leaves the *addressing* +free-floating -- so the naming fix alone would NOT have finished the job. Now guarded by +`EXPECTED_GUA`, which pins the apex's measured binding and refuses a mismatched pair. + +## What changed + +- `scripts/lib-net.sh`, `lib-hosts.sh` -- region-qualified selectors; bare `dcN` REJECTED. + `vr0-dc0` and `vr1-dc0` are SEPARATE case arms on purpose: they are behaviourally identical today + but semantically independent (`vr1-dc0` no-ops because D-101 says it INHERITS VR0 DC0's v4 + layout). The day D-101 stops holding, one diverges and the other must not. +- `netbox/dc-dc-prefixes-import.py` -- identity `SITES`, the `EXPECTED_GUA` guard, the description + fix, `DC1_V4_SUPERNET`/`DC2_V4_SUPERNET` -> **`VR1_DC1_V4_SUPERNET`** (BOTH old names rejected BY + NAME: an unqualified "DC1" could mean VR0's dc0 OR VR1's second DC). +- `scripts/dc-dc-{dr-drill,radosgw-multisite,rbd-mirror,ceph-disk-budget}.sh`, + `{phase-00-maas-standup,carve-host-interfaces,reenroll-hosts}.sh` -- selectors + `DC=` hints. + `rbd-mirror`'s `--site-name` aligned too: it is a DIFFERENT namespace that LOOKS identical, and + `--dc vr1-dc0 --site-name dc1` on one command line is a re-created collision. +- `opentofu/` -- `dc1_*` -> `vr1_dc0_*`, `dc2_*` -> `vr1_dc1_*`, with **`moved {}` blocks**. +- All 6 VR1 runbooks + `netbox/README.md` + `docs/vr1-office1-as-built.md` call sites. +- **D-114 amendment:** the DC containment VMs `vdc1`/`vdc2` -> **`vvr1-dc0`/`vvr1-dc1`**. Neither is + built, so this is free -- and D-114's own DR primitive is **`virsh destroy vdc1`**, which under the + old naming reads as "destroy VR1 DC1" while MEANING "destroy VR1 DC0". A mislabelled destroy + command in a DR drill is not cosmetic. `voffice1` KEEPS its name. + +## Tests + +`tests/dc-selector` 21 -> **30** (new: every bare `dcN` must be REJECTED). +`tests/dc-dc-prefixes-import` 40 -> **82** (new: the identity invariant, `EXPECTED_GUA` in BOTH +mismatch directions, that a mismatched run writes NOTHING, that the MATCHING pair still succeeds, +and that no description contains the munged `VR1 VR1-DC0`). +**GAUNTLET ALL GREEN (57 harnesses); repo-lint 0 fail.** + +The old harnesses were not merely stale -- they **pinned the wrong mapping**, and would have gone +green while enforcing the bug. That is the failure mode this change exists to kill. + +## The `tofu apply` is NOT done -- it is GATED + +`opentofu/` changes rename the libvirt objects (`dc_name`/`link_name` interpolate straight into each +object's `name`, which is ForceNew), so the plan REPLACES them. `moved {}` cannot suppress a +ForceNew -- it is there so the plan reads as 11 clean "must be replaced" lines instead of 11 +unrelated destroy+create pairs, i.e. so it is REVIEWABLE. + + Plan: 11 to add, 0 to change, 11 to destroy. + +**Safe, MEASURED before the rename:** all 11 objects are EMPTY -- no guests attached (Stage 3 has +not run, so no DC node VMs exist) and no volumes in either DC pool (the only volumes in state belong +to `office1_opnsense`, `ubuntu_noble_base`, `voffice1`, none of which sit in a DC pool). Office1's +live networks do not appear in the plan at all. + +**Not applied.** A destroying apply is operator-gated. Re-run `tofu plan` at execution and STOP if +anything shows as ATTACHED (standing lesson 2: "updated in-place" does NOT mean "no restart"). + +## Revert + + git revert <this commit> + # opentofu: the moved{} blocks make the reverse direction symmetrical; re-plan before applying. + # No NetBox state to revert -- this change made ZERO writes to any NetBox. diff --git a/docs/changelog-20260714-d119-tofu-apply.md b/docs/changelog-20260714-d119-tofu-apply.md new file mode 100644 index 0000000..60d29f1 --- /dev/null +++ b/docs/changelog-20260714-d119-tofu-apply.md @@ -0,0 +1,43 @@ +# 2026-07-14 -- D-119 OpenTofu rename APPLIED (operator-gated destroying apply) + +The `tofu apply` deferred by the D-119 commit. Executed after explicit operator authorization. + +## Pre-apply discipline (verify before mutate) + +A LIVE re-check immediately before the apply (not the earlier measurement): + + dc1-* / mesh-dc* networks -> 0 DHCP leases each + dc1-pool / dc2-pool -> 0 volumes each + every libvirt domain -> none references a dc1-*/mesh-dc* network + +Then a fresh `tofu plan`: **11 to add, 0 to change, 11 to destroy** -- all `must be replaced` +(the `moved{}` blocks). Confirmed empty, so the replace is safe (standing lesson 2: an apply +touching a live libvirt object's devices is an outage -- none of these carried anything). + +## Result + + Apply complete! Resources: 11 added, 0 changed, 11 destroyed. + +Verified ON THE HOST, not just in state: +- state holds all 11 under `vr1_dc0_*` / `vr1_dc1_*` / `mesh_vr1_*`; +- `tofu plan` -> **"No changes"** (repo/state back in sync); +- libvirt shows `vr1-dc0-{provider-public,metal-admin,metal-internal,data-tenant,storage,replication}`, + `mesh-vr1-dc0-office1`, `mesh-vr1-dc0-vr1-dc1`, `mesh-vr1-dc1-office1`, `vr1-dc0-pool`, `vr1-dc1-pool`; +- ZERO old `dc1-*`/`dc2-*`/`mesh-dc*` network or pool names remain. + +## No collateral + +Office1 was never in the plan and is untouched: `office1-local`, `office1-wan`, `wan`, the +`office1-opnsense` edge and `voffice1` all present; edge (10.10.0.1) and voffice1 (10.10.0.20) +both reachable after the apply. + +## Why this went before Stage 3 + +Carrying divergent repo/state into Stage 3 would let a Stage-3 `tofu apply` silently bundle this +11-resource rename with real DC provisioning. Applying it now, while the objects are empty, kept +the rename a clean standalone change. + +## Revert + +The `moved{}` blocks are symmetric; reverting the D-119 code commit and re-applying renames back. +There is nothing to preserve in these objects (they are empty), so a revert is a pure rename. diff --git a/docs/changelog-20260714-netbox-importers-waf-useragent.md b/docs/changelog-20260714-netbox-importers-waf-useragent.md new file mode 100644 index 0000000..e7297c7 --- /dev/null +++ b/docs/changelog-20260714-netbox-importers-waf-useragent.md @@ -0,0 +1,54 @@ +# 2026-07-14 -- the two pynetbox importers could NEVER have written upstream (WAF User-Agent) + +A C2 (feed-the-carve-upstream) blocker, found while establishing C2's feasibility. Measured, not +inferred. + +## The gap + +The upstream apex `netbox.baldurkeep.com` sits behind a filter that **403s the default Python +User-Agent** (`references/platform-traps.md`). The same token works from curl -- it looks EXACTLY +like an auth failure and is not. **MEASURED 2026-07-14 from vcloud:** `curl` -> HTTP 200; +`urllib`/pynetbox -> HTTP 403. + +The stdlib tools (`prod-draft-dump.py`, `sandbox-seed.py`, `d115-office-carve.py`) already send an +accepted UA (`curl/8.5.0`). But the two **pynetbox-based** importers -- +`roles-aggregates-import.py` and `dc-dc-prefixes-import.py` -- built `pynetbox.api(url, token)` with +the **default** UA and never overrode it. So they would 403 against upstream: **they could only ever +have written to the sandbox (which has no WAF), never to the apex C2 must feed.** The gap was +invisible because every real run so far was against the sandbox. + +## The fix + +`get_nb()` in both importers now sets the accepted UA on the underlying requests session: + + nb.http_session.headers["User-Agent"] = "curl/8.5.0" + +Same value and rationale as the stdlib tools -- pynetbox exposes its `requests.Session` as +`.http_session`, so one line covers every call it makes. + +## Tests + +- `tests/dc-dc-prefixes-import/` -> **86**: the fake now has an `.http_session` (mirroring + pynetbox's real one), and the test asserts AT RUNTIME that `get_nb()` sets the UA to `curl/8.5.0` + -- not just that the string appears in source. +- `tests/roles-aggregates-import/` -> **20**: grep assertion that `get_nb` sets the WAF-safe UA. + +GAUNTLET ALL GREEN (58); repo-lint 0 fail. + +## What this does and does NOT unblock + +**Does:** removes the WAF 403 from the two pynetbox importers' upstream write path. + +**Does NOT** (still open for C2): +- `pynetbox` is NOT installed on vcloud (measured). The two importers must run on a host that HAS it + and can reach upstream -- `office1-netbox` has pynetbox 7.0.0. `d115-office-carve.py` is stdlib and + runs anywhere. +- The sandbox re-verify (hardened `sandbox-fidelity-check.py`) needs the operator-held sandbox token. +- The upstream write itself is a PRODUCTION IPAM mutation and stays operator-gated. + +## Revert + + git revert <this commit> + +The single line in each `get_nb()` is the only functional change; the rest is tests. No NetBox was +written. diff --git a/docs/changelog-20260714-netbox-write-path-hardening.md b/docs/changelog-20260714-netbox-write-path-hardening.md new file mode 100644 index 0000000..7814771 --- /dev/null +++ b/docs/changelog-20260714-netbox-write-path-hardening.md @@ -0,0 +1,85 @@ +# 2026-07-14 -- NetBox write-path hardening (the bugs that would have corrupted the apex at C2) + +Found by an adversarial review sweep during D-119. **Independent of the naming work** -- these are +the defects that would have bitten during **C2 (feed the validated carve UPSTREAM)**, which is the +production IPAM write Stage 2 still needs. + +## 1. `roles-aggregates-import.py` still died HALF-WAY through a production write + +The preflight added on 2026-07-13 -- after this exact script created 4 roles, 400'd on the 5th, and +left the apex half-populated -- **only covered role-NAME collisions.** Two `die()` calls remained +BELOW the first write: + +- the **ARIN RIR** lookup (an apex without ARIN dies *after* 5 roles are committed), and +- **`validate_ula_48()`** (a typo'd `ORG_ULA_48` dies *after* roles AND RIRs are committed). + +So the tool could still leave the production IPAM half-written with no rollback -- **the exact +failure the preflight was added to prevent, still armed one section lower.** The RIR half of the +preflight loop was also a literal no-op (`if ...: continue` followed by a comment). + +**Fixed:** both checks moved INTO the preflight; the RIR name-collision check is now real. +**A preflight that does not cover every `die()` downstream of the first write is not a preflight.** + +## 2. `sandbox-fidelity-check.py` could return a FALSE GREEN + +This is the one that stings: it is the script we have been citing as **proof the sandbox is a +faithful replica.** + +It field-compares only the **SHARED** objects. The entire planned delta was checked merely as a +*subset* of `EXPECTED_NEW`: + + unexpected = extra - expected # proves the delta is a SUBSET of the plan + # NEVER proves the plan was CARRIED OUT + +So with **zero** of the 36 DC prefixes created, `extra = {}`, `unexpected = {}`, and it printed +*"Nothing lost, nothing stray"* and exited **0**. `EXPECTED_NEW` was an upper bound masquerading as +an assertion. A delta prefix created with the **wrong scope, role or status** also passed clean -- +it was never field-compared against anything. + +**Fixed:** `EXPECTED_NEW` is now BOTH bounds (`EXPECTED-BUT-ABSENT` is a hard failure), and delta +prefixes are checked for scope and role. That second check is the 17-prefix scope-drop bug relocated +into the delta, where the old check was structurally blind to it. + +## 3. `dc-dc-prefixes-import.py` died mid-loop on a missing role + +`find_role()` was called lazily INSIDE the write loop. Against the apex **as it stands today** (no +six-plane roles exist upstream), a `--commit` would create the site and the first 4 prefixes, then +die on the 5th: **a half-written datacenter in the production IPAM.** Now a whole-plan preflight +resolves every role BEFORE the first write. + +## 4. The test fake was KINDER THAN REALITY at exactly the failure point + +`fake_pynetbox.get()` returned `matches[0]` on a multi-match. **Real `pynetbox` RAISES.** NetBox +permits duplicate prefixes (this importer's own docstring anticipates them against `vr0-dc0`), so a +global `.get(prefix=...)` would blow up mid-loop in production while the harness sailed through. +A fake that is gentler than the real thing hides the bug it exists to catch. It now raises. + +## Tests + +- **`tests/sandbox-fidelity-check/` -- NEW, 10/10.** The checker **shipped with no harness at all**, + which is precisely why the false green survived. T3 reproduces it: an unapplied plan used to exit 0. + T1 derives its expectation FROM THE CHECKER'S OWN `EXPECTED_NEW` rather than re-typing it -- a + harness holding its own copy of the expectation cannot detect the two drifting apart. +- `tests/roles-aggregates-import/` 16 -> **19**: asserts by POSITION that the ARIN check and the ULA + validation both precede the first `create`. The old harness only asserted the *word* "PREFLIGHT" + appeared before the first write -- so it went **green while the landmine was armed**. +- `tests/dc-dc-prefixes-import/` **82** (unchanged count; the fake now raises). + +**GAUNTLET ALL GREEN (58 harnesses); repo-lint 0 fail.** + +## Still OPEN (logged, not fixed) + +- **Same-dumper blind spot:** the fidelity check compares two dumps produced by the SAME field list, + so any field the dumper omits is invisible on BOTH sides by construction (prefix `vrf`/`tenant`/ + `vlan`/`tags`/`custom_fields`; site `tenant`/`group`/`facility`). Structurally the same shape as the + region-scope bug. Benign only while upstream uses none of them -- nothing enforces that. +- **Duplicate-CIDR collapse:** prefix-keyed dicts in the seeder and the checker collapse duplicate + prefixes, so a whole prefix object could vanish with a green check. Latent (no duplicates upstream + today) -- but the importer's `.get(prefix=...)` is a GLOBAL lookup, so `--update` could rebind + another site's prefix. Scoping it by `scope_id` is the real fix. + +## Revert + + git revert <this commit> + +No NetBox was written during this change -- all work was offline against synthetic fixtures. diff --git a/docs/changelog-20260714-office1-lan-ipv6-executed.md b/docs/changelog-20260714-office1-lan-ipv6-executed.md new file mode 100644 index 0000000..02fab54 --- /dev/null +++ b/docs/changelog-20260714-office1-lan-ipv6-executed.md @@ -0,0 +1,116 @@ +# 2026-07-14 -- Office1 LAN IPv6 is LIVE (Stage 2 close-out C1: OPEN -> DONE) + +The last build item on Office1. Executed against the live edge and proven on the wire, not by +self-report. Closes the item pinned since 2026-07-13 (D-115 question 3). + +## Operator ruling that shaped it (2026-07-14) + +The session opened with a proposal to build an **IPv4-to-IPv6 NAT** (a second OPNsense, or nginx) +to *simulate* the v6 forwarding the lab does not provide. **Not built, and the proposal was +withdrawn** in favour of: *"configure IPv6 internally as much as possible and we will configure the +IPv6 edge after deployment has been completed."* + +Why that was the right call, recorded so it is not re-proposed: + +- **D-101 already rules it out:** *"native GUA routing, no NAT66 -- NAT66 is discouraged and we own + the edge."* Translation is a REJECTED approach here, not an open one. +- **D-115's amendment already ruled on this exact symptom:** v6 not routing past the lab boundary + is *"a network issue OUTSIDE this cloud and outside this repo -- it needs the lab/upstream + operator, not a change here."* +- **MINIMIZE DELTA TO ROOSEVELT:** Roosevelt has a real ISP carrying real v6. A translation box is + precisely the component Roosevelt will NOT have, so a NAT shim is the *least* faithful possible + simulation -- it would make the runbooks teach a topology that must later be deleted. +- **nginx was not a peer option:** it is an L7 HTTP(S) reverse proxy, not NAT. It proves nothing + about RA, ICMPv6, Ceph-over-v6 or geneve-over-v6 -- i.e. nothing about D-101's family matrix. + +**Re-measured 2026-07-14, confirming D-115:** vcloud holds a GUA and a v6 default route via RA, the +v6 gateway is reachable, but v6 internet is **100% loss** while v4 from the same host is fine. The +lab's upstream advertises RA and a prefix and does not route v6 out. Unchanged, and not ours to fix. + +## What was done + + scripts/opnsense-set-interface-v6.sh --commit 10.10.0.1 lan 2602:f3e2:f01:100::1 64 + POST /api/radvd/settings/addEntry {"entries":{"enabled":"1","interface":"lan","mode":"unmanaged"}} + POST /api/radvd/service/reconfigure + +Address `2602:f3e2:f01:100::1/64` is the D-115 carve value, not an invented one. + +**The address half is NOT API-doable** -- see the D-113 amendment. The RA half IS API-native. Both +halves are now exercised for real, which is what makes them reusable verbatim for the DC0/DC1 edges +at Stage 3. + +## `mode=unmanaged`, deliberately NOT the model's default + +The radvd model defaults to `mode=stateless`, which sets the RA **Other-config flag** -- telling +clients to fetch DNS and other options from a **DHCPv6 server**. MEASURED on the edge: Kea DHCPv6 is +`enabled=0` and no DHCPv6 daemon is running. Accepting the default would have advertised a DHCPv6 +server **that does not exist** -- a quiet misconfiguration that would look fine and produce clients +that hang waiting on it. `unmanaged` (M=0, O=0) is the honest configuration for SLAAC-only. + +Field names, the request key (`entries`, from `addBase('entries','entries')`), the legal `mode` +values, and the apply endpoint were all READ OFF THE EDGE (the model XML and the controller), not +recalled. Nothing here was guessed. + +## PROVEN ON THE WIRE (not by `{"result":"saved"}`) + + edge ifconfig vtnet0 -> inet6 2602:f3e2:f01:100::1 prefixlen 64 [the KERNEL, not the config] + edge radvd running -> prefix 2602:f3e2:f01:100::/64, AdvAutonomous on, AdvOnLink on + RDNSS 2602:f3e2:f01:100::1, no M/O flags [= unmanaged took effect] + voffice1 -> inet6 2602:f3e2:f01:100:5054:ff:fe6a:87e5/64 [SLAAC] + + 2602:f3e2:f01:100::/64 proto ra [RA-learned route] + edge -> voffice1 GUA -> 3 packets, 0.0% loss, 0.838ms avg [TRAFFIC PASSES] + edge ndp -an -> neighbour 52:54:00:6a:87:e5 [matches voffice1's Kea + lease hwaddr -- it IS voffice1] + voffice1 dig @2602:f3e2:f01:100::1 example.com -> ANSWERS [a v6-BOUND SERVICE really + serves: "services binding + v6" is now EVIDENCE] + +## The RDNSS we did not set, and why it is nonetheless correct + +The generated `radvd.conf` advertises `RDNSS 2602:f3e2:f01:100::1` -- **we never set that field.** +OPNsense auto-defaulted it to the edge's own LAN v6 address. That is the SAME failure class we +deliberately avoided with `mode=unmanaged` (advertising a service that may not exist), one field +over, and it slipped in precisely BECAUSE the field was left unset. + +**Checked, not assumed:** the edge's Unbound DOES answer DNS on that v6 address (`dig` from +`voffice1` resolves). It works despite there being no v6 egress because Unbound queries upstream +over **v4**. So the default is correct here -- but it was luck, not design. **Set RDNSS explicitly +when this RA config is reused on the DC0/DC1 edges at Stage 3,** and re-check that whatever address +it names actually answers. + +**No collateral damage** (`configctl interface reconfigure lan` re-applies the LAN that all of +Office1 sits behind): v4 `10.10.0.1` intact, Kea DHCPv4 still running and listening on udp/67, +edge + `voffice1` + the routed compose net all still reachable. The interface-count assertion held +(3 interfaces, unchanged). + +## A trap worth recording: `accept_ra=0` does NOT mean "RAs ignored" on Ubuntu + +`voffice1` has kernel `net.ipv6.conf.all.forwarding=1` (it runs LXD) and `accept_ra=0` on `enp1s0`. +Classic Linux behaviour says a forwarding host ignores RAs unless `accept_ra=2` -- so this looked +like it could not possibly have autoconfigured. **It did.** netplan/systemd-networkd processes RAs +in **USERSPACE**, independent of the kernel sysctl. Do not diagnose "no SLAAC" from the sysctl alone +on an Ubuntu box; read `ip -6 addr` and look for a `proto ra` route. + +## What this does and does NOT prove + +**Proves** (most of D-101's family matrix): v6 addressing, RA/SLAAC, on-link routing, and the edge +speaking v6 on the LAN. This is the mechanism DC0/DC1 will reuse. + +**Does NOT prove, and cannot in VR1:** internet GUA reachability. The WAN v6 leg is DEFERRED per +D-115 -- it would be a path to nowhere and untestable. Per the 2026-07-14 operator ruling, the +**IPv6 edge is configured AFTER deployment completes.** Do not read this green result as "v6 works +end to end." + +## Revert + + # RA + bash scripts/opnsense-api.sh POST /api/radvd/settings/delEntry/627e3f12-8150-4909-9910-a275f2a0a6fc + bash scripts/opnsense-api.sh POST /api/radvd/service/reconfigure + # address: clear ipaddrv6/subnetv6 on Interfaces > [LAN], then Save+Apply. + # NOTE: `opnsense-set-interface-v6.sh` CANNOT express a clear -- it requires a 4-arg + # address+prefixlen and validates the address. Passing `:: 64` would SET the unspecified + # address, not clear the field. Do not use it as a revert. (An explicit --clear flag would + # be the honest fix; not built, since nothing needs it yet.) + +(`OPNSENSE_API_HOST=10.10.0.1` for the API calls.) Reverting is not expected to be needed: nothing +depends on the v6 leg yet, and the v4 LAN is untouched by it. diff --git a/docs/changelog-20260714-opnsense-set-interface-v6.md b/docs/changelog-20260714-opnsense-set-interface-v6.md new file mode 100644 index 0000000..b632bdb --- /dev/null +++ b/docs/changelog-20260714-opnsense-set-interface-v6.md @@ -0,0 +1,85 @@ +# 2026-07-14 -- D-113 AMENDMENT: the LAN IPv6 the REST API cannot set + +Retroactive changelog for commit `522ba7d` (the script) and `3b01dd4` (the D-113 amendment). +The code shipped with a green harness and a clean lint, but **without this entry** -- the +delivery rule requires all three. Closing that gap; no code changes here. + +## The measurement that forced it + +D-113(a2) asserts the OPNsense edge is configured "via the REST API", and lists **interfaces** +among the API-covered surfaces. **Measured on the live Office1 edge (OPNsense 26.1): that is +false.** `Interfaces > [LAN]` is served by the LEGACY page `/interfaces.php?if=lan`. Only the +Devices and Neighbors sub-trees were ever migrated to MVC. **There is no REST endpoint for a base +interface's `ipaddrv6`.** + +The claim was **inferred** from DHCP and firewall being API-native, and never measured. It sat in +an ADOPTED decision as if it were a fact. D-113 now carries the amendment. + +## What shipped (already committed) + +- `scripts/opnsense-set-iface-v6.php` -- runs ON the edge; loads the LIVE config through + OPNsense's own `Config` singleton, mutates exactly two leaf fields (`ipaddrv6`, `subnetv6`), + asserts the interface count is unchanged, and lets the vendor's own code serialize it. +- `scripts/opnsense-set-interface-v6.sh` -- ships the setter over SSH, runs it, applies with + `configctl interface reconfigure`, then proves the address is **on the wire**. +- `tests/opnsense-set-interface-v6/run-tests.sh` -- offline harness, **40/40 PASS** (ssh/scp + stubbed; no edge contacted). + +**DRY BY DEFAULT.** Without `--commit` nothing is written and nothing is applied. + +## Why this is NOT the forbidden config.xml push + +The distinction is the whole point, and it is the difference between this and the path D-113(a2) +deleted. **It never authors a config.** It mutates two leaves of the edge's own live config +object through the vendor's own model -- the identical code path `interfaces.php` runs on Save. +Nothing is replaced wholesale, so the **API-managed Kea DHCP and the firewall rules cannot be +clobbered** (standing lesson 1: a full rendered config.xml push WOULD clobber them). + +Same principle as `opnsense-bootstrap-apikey.sh`: **call the vendor's interface, do not +re-implement the internals.** + +## Two traps it is built against + +- **root's shell on the edge is `tcsh`.** A `$(...)` inside a quoted remote command dies with + "Illegal variable name", and it dies QUIETLY. Every remote command here is fed to `sh -s` over + stdin (standing lesson 5). +- **Saving the config does NOT reconfigure the kernel.** Hence the explicit `configctl interface + reconfigure`, and hence the verify reads `ifconfig` -- the KERNEL -- and not the config file. A + `{"result":"saved"}`-style self-report is not evidence. + +## Status: WRITTEN AND TESTED, NEVER RUN AGAINST THE LIVE EDGE + +This is Stage 2 close-out item **C1**, and C1 is still **OPEN**. The script exists; the address is +not on the edge. Executing it is a live mutation and remains operator-gated. + +Two things to carry into that execution: + +- **The RA half is API-native** and is a separate step (`/api/radvd/settings/addEntry`). The + address alone is not the deliverable; the LAN needs RA to be useful. +- **IPv6 does not egress the lab** (measured 2026-07-13: v6 gateway reachable, v6 internet 100% + loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding -- most of D-101's family + matrix -- but NOT internet GUA reachability, which is unprovable in VR1. The WAN v6 leg is + DEFERRED per D-115's amendment; it would be a path to nowhere. + +## Reusable by the DC edges + +Stage 3 builds DC0's and DC1's edges from the same module. This script takes them verbatim -- only +the host and address change. That was the operator's stated reason for scripting it rather than +clicking it once in the GUI: **the same treatment gets paid three times.** + +## Known limitation (logged, not actioned) + +The interface-name-to-device mapping is a hardcoded `lan->vtnet0 / wan->vtnet1` sed. It is correct +for the Office1 edge as built, but it is an ASSUMPTION about device ordering that a DC edge with a +different NIC count could break. It would fail loudly (the verify reads the wrong device and the +script exits 2), not silently. Worth deriving from the edge's own config before Stage 3 reuses it. + +## Revert + + git rm scripts/opnsense-set-interface-v6.sh scripts/opnsense-set-iface-v6.php + git rm -r tests/opnsense-set-interface-v6/ + git rm docs/changelog-20260714-opnsense-set-interface-v6.md + +Nothing live depends on these: **the script has never been run with `--commit`,** so the live edge +is unaffected by its existence. The D-113 amendment prose (`3b01dd4`) is a separate revert and +should be KEPT regardless -- it records a measurement, not a tool. diff --git a/docs/changelog-20260715-creds-audit.md b/docs/changelog-20260715-creds-audit.md new file mode 100644 index 0000000..69a6a7f --- /dev/null +++ b/docs/changelog-20260715-creds-audit.md @@ -0,0 +1,60 @@ +# 2026-07-15 -- creds-audit: enforce per-site credential consolidation (SEC-009), so the DC0/DC1 spread cannot recur + +## Why + +SEC-009 established the convention "ALL of a site's secrets live in `~/<site>-creds/` +(0700, files 0600), no loose env files in `~`." But nothing ENFORCED it, and it had +two blind spots that a checklist alone would not catch: + +1. **VM-minted secrets.** A secret generated ON a site VM (office1-netbox's sandbox + NetBox token at `/root/netbox-secrets/api.token`) has no forcing function to get + pulled into the jumphost folder. It went un-consolidated until the C2 fidelity + re-check needed it (SEC-009 addendum, 2026-07-15). +2. **Silent sprawl.** A loose `*.env`/`*.token` in `~` outside a `*-creds/` folder -- + the ORIGINAL SEC-009 exposure -- had no standing detector. + +Standing up DC0 and DC1 multiplies both (each mints MAAS API keys, per-DC NetBox +tokens, node/edge SSH keys, OPNsense creds, tailscale authkeys on multiple VMs). The +operator asked directly how we stop the spread recurring for the DCs. Answer: a +manifest + an audit + a mint-time runbook step. + +## What was built + +- **`creds-manifests/<site>.manifest`** -- declares EXACTLY the secrets a site must + hold: `<filename> <octal-mode> <source>`, where source is `local` or `<vm>:<path>` + for VM-minted secrets (making the easy-to-miss class explicit and provenance-tracked). + `creds-manifests/vr1-office1.manifest` seeded with 11 entries from the live folder; + it audits CLEAN. +- **`scripts/creds-audit.sh <site>`** -- verifies the live `~/<site>-creds/` against + the manifest: folder is 0700, every entry present with its declared mode and + non-empty; flags any UNDECLARED file in the folder; and scans the home root for + SPRAWL (`*.env`/`*.token`/`*authkey*`/`*appcred*`/`*-cred*.txt` loose outside any + `*-creds/`). **Reads NO secret values -- metadata only** (stat/size), verified by a + harness assertion (T7: no cat/head/tail/od/xxd in the script). Env overrides + `CREDS_ROOT`/`MANIFEST_DIR` make it fully offline-testable. +- **`tests/creds-audit/run-tests.sh`** (7/7) -- synthetic folders under a tmp root: + complete+correct -> CLEAN; missing (VM-sourced) entry -> fail; wrong mode -> fail; + undeclared file -> fail; home-root sprawl -> fail; non-0700 folder -> fail; + metadata-only. Auto-discovered by the gauntlet. +- **Runbook discipline** -- `runbooks/dc-dc-phase3-maas-enlist-deploy.md` GATE now + carries a non-negotiable close-out: consolidate + manifest each secret AT MINT TIME, + and `bash scripts/creds-audit.sh $DC` MUST be CLEAN before a DC is declared done. +- **`docs/security-ledger.md`** -- SEC-009 ENFORCEMENT note recording the above. + +## Verification + +- Harness: `tests/creds-audit/run-tests.sh` -> 7/7 PASS. +- Live: `bash scripts/creds-audit.sh vr1-office1` -> CLEAN (folder 0700, 11/11 entries + present + correct + non-empty, no undeclared files, no sprawl). This also CONFIRMS + the SEC-009 sandbox-token gap is now closed and tool-verified. +- Gauntlet ALL GREEN; repo-lint 0 fail. + +## Revert + +- `git rm scripts/creds-audit.sh tests/creds-audit/run-tests.sh creds-manifests/vr1-office1.manifest docs/changelog-20260715-creds-audit.md` + (remove `creds-manifests/` if now empty). +- `git checkout HEAD -- docs/security-ledger.md runbooks/dc-dc-phase3-maas-enlist-deploy.md` + reverts the SEC-009 enforcement note and the runbook close-out. + +Nothing here reads, moves, or mutates any secret; the audit is read-only metadata +inspection and the manifest holds only filenames/modes/provenance. diff --git a/docs/changelog-20260715-creds-consolidation.md b/docs/changelog-20260715-creds-consolidation.md new file mode 100644 index 0000000..8d1932b --- /dev/null +++ b/docs/changelog-20260715-creds-consolidation.md @@ -0,0 +1,52 @@ +# 2026-07-15 -- credential/env consolidation into ~/vr1-office1-creds/ (SEC-009) + +Operator-approved. Closes a credential SPRAWL + world-readable exposure on vcloud, and establishes +the standing per-site consolidation convention. + +## The finding + +Three env files sat loose in `~`, outside the consolidated `~/vr1-office1-creds/` folder: + + ~/.vr1-netbox.env 0600 upstream/v1-draft NetBox token (NETBOX_URL + NETBOX_TOKEN) + ~/vr1-office1.env 0600 OPNsense edge config (WAN/LAN/DHCP/root-hash/SSH-key vars) + ~/vr1-stage1.env 0664 OpenTofu TF_VAR_* -- INCLUDING TF_VAR_maas_api_key (a MAAS secret) + +`vr1-stage1.env` at **0664 (group/world-readable)** held a MAAS API key -- a real exposure, not just +untidiness. `~/vr1-office1-creds/tailscale-authkey.txt` was also 0664. None were created this +session (mtimes 2026-07-10..12); a prior session in this continuity scattered them. + +## What was done (files OUTSIDE the repo; not committed, recorded here) + +- Moved all three into `~/vr1-office1-creds/` (`vr1-netbox.env`, `vr1-office1.env`, `vr1-stage1.env`). +- `chmod 600` on every sensitive file in the folder -- fixed the two 0664 files. +- No file CONTENTS were read; key NAMES only were listed to identify each file's purpose. +- Fixed a D-119 follow-up the sweep could not reach (it is outside the repo): `vr1-stage1.env` had + `TF_VAR_dc1_pool_path` / `TF_VAR_dc2_pool_path` -> renamed to `TF_VAR_vr1_dc0_pool_path` / + `TF_VAR_vr1_dc1_pool_path` to match the renamed OpenTofu variables. + +## Repo changes (committed) + +- Active path references updated old `~/.vr1-netbox.env` -> `~/vr1-office1-creds/vr1-netbox.env`: + `netbox/prod-draft-dump.py` (docstring + usage + error message), `runbooks/dc-dc-phase1-office1- + standup.md` (Step 10b table), `docs/vr1-office1-as-built.md`, `docs/session-ledger.md`. + Dated changelog/incident records that reference the OLD paths are left as historical snapshots. +- `docs/vr1-office1-as-built.md` secrets table: added the three env files + the consolidation note. + Corrected a wrong assumption recorded mid-session -- the SANDBOX NetBox token is NOT on vcloud; it + is on office1-netbox (`/root/netbox-secrets/`). `vr1-netbox.env` is the UPSTREAM token only. +- `docs/security-ledger.md`: SEC-009 (this finding, REMEDIATED) + the STANDING CONVENTION. + +## Standing convention (SEC-009) + +Per-site: ALL sensitive + env/config files live in one `~/<site>-creds/` folder (0700; files 0600; +public keys 0644). No loose env files in `~`. `~/vr1-dc0-creds/` and `~/vr1-dc1-creds/` follow the +same pattern FROM DAY ONE when DC deployment starts. This is a DATA-LOSS / continuity control first: +scattered creds get lost between sessions and after compaction. Real credential hardening (rotation, +revocation) stays deferred to post-teardown redeploy cycles -- this is a closed in-house test. + +Also seeded agent memory (was empty): the creds convention and the closed-test posture, so neither +is re-lost at the next compaction. + +## Revert + +Repo edits: `git revert <this commit>`. The file moves are outside the repo -- to reverse, move the +three env files back to `~` (not recommended: it re-opens the sprawl and the 0664 exposure). diff --git a/docs/changelog-20260715-d120-load-and-iprange-tooling.md b/docs/changelog-20260715-d120-load-and-iprange-tooling.md new file mode 100644 index 0000000..82aa282 --- /dev/null +++ b/docs/changelog-20260715-d120-load-and-iprange-tooling.md @@ -0,0 +1,53 @@ +# 2026-07-15 -- C2 sub-task (ii): D-120 bands loaded + verified; sandbox loop extended to ip-ranges/ip-addresses + +## Context + +C2 sub-task (ii), per DOCFIX-195: load the D-120 compose-/24 band ranges + the two +static service IPs into office1-netbox (the VR1 IPAM apex) and VERIFY them. The logged +tooling gap: the sandbox loop (`prod-draft-dump.py` + `sandbox-fidelity-check.py`) did +not cover `ip-ranges`/`ip-addresses` at all, so those objects were invisible to the +fidelity check. This closes that gap and then does the load, verified. + +## What changed + +**Tooling (the "A" extension):** +- `netbox/prod-draft-dump.py` -- ENDPOINTS += `ipam/ip-ranges` and `ipam/ip-addresses`. + Ranges get a synthetic `range` key (`start-end`) since the checker keys on one field. + Verified live: the endpoints dump (0/0 before the load). +- `netbox/sandbox-fidelity-check.py` -- `KEY` += the two endpoints; `EXPECTED_NEW` += the + 3 D-120 ranges (keyed start-end) + the 2 service IPs (keyed address). They are pure + delta (the frozen v1 draft carries none), so both-bounds already gives endpoint/address + correctness -- a wrong endpoint is a different key (UNEXPECTED-EXTRA + EXPECTED-BUT-ABSENT). + Success banner now reads D-115/D-117/D-118/**D-120**. +- `tests/sandbox-fidelity-check/run-tests.sh` -- +T11 (a dropped D-120 range -> + EXPECTED-BUT-ABSENT) +T12 (a stray ip-address -> UNEXPECTED-EXTRA). 12 -> 14. + +**Importer:** +- `netbox/d120-compose-bands.py` -- stdlib, UA-aware, `SANDBOX_HOSTS`-guarded, + DRY-BY-DEFAULT (`--commit` to write, `--yes-write-upstream` for a non-sandbox target), + with a WHOLE-PLAN PREFLIGHT (parent `10.10.1.0/24` must exist; every endpoint in-parent; + start<=end) so a bad plan cannot half-write. Creates the 3 ranges + 2 IPs, idempotent. + Harness `tests/d120-compose-bands/` (16/16) pins the band values + the guards. + +**The load (executed 2026-07-15, operator-approved):** +Dry-run previewed 5 creates / 0 present; `--commit` against office1-netbox created: +- ip-ranges: `10.10.1.2-.49` (static), `.100-.200` (dynamic), `.201-.254` (node) +- ip-addresses: `10.10.1.10` (office1-netbox), `10.10.1.11` (office1-tailscale) +Re-dump -> ip-ranges 3, ip-addresses 2. Fidelity check -> **exit 0**: "upstream draft + +EXACTLY the planned delta (D-115/D-117/D-118/D-120)." + +## Result + +- **C2 sub-task (ii) CLOSED; C2 fully closed.** office1-netbox now holds the complete + validated VR1 dataset, verified by a fidelity check that covers every object class in it. +- **D-120 Step 6 CLOSED** -- the band layout is now recorded in NetBox, no apex write. +- Gauntlet ALL GREEN (60 harnesses); repo-lint 0 fail. +- Stage 2 close-out (C1..C5) is COMPLETE -> the branch->main merge point is reached + (the merge itself remains an operator-gated action). + +## Revert + +- `git rm netbox/d120-compose-bands.py tests/d120-compose-bands/run-tests.sh docs/changelog-20260715-d120-load-and-iprange-tooling.md` +- `git checkout HEAD -- netbox/prod-draft-dump.py netbox/sandbox-fidelity-check.py tests/sandbox-fidelity-check/run-tests.sh docs/session-ledger.md docs/dc-dc-deployment-workflow.md` +- The 5 NetBox objects (ip-ranges 1-3, ip-addresses 1-2 on office1-netbox) are additive; + delete them in the NetBox UI if the load must be undone. No existing object was edited. diff --git a/docs/changelog-20260715-d120-office1-service-reip.md b/docs/changelog-20260715-d120-office1-service-reip.md new file mode 100644 index 0000000..34273c3 --- /dev/null +++ b/docs/changelog-20260715-d120-office1-service-reip.md @@ -0,0 +1,66 @@ +# Changelog 2026-07-15 -- D-120 ADOPTED: Office1 service re-IP into the static band + +## What changed + +Adopted **D-120** (static-services addressing convention for MAAS-managed compose /24s) as +**option (a), executed**, and re-IP'd Office1's two running services out of the MAAS node band into +the `.2-.49` static band: + + office1-netbox 10.10.1.201 -> 10.10.1.10 + office1-tailscale 10.10.1.202 -> 10.10.1.11 + +No wipe. Both services verified on the wire: +- NetBox: HTTP 200 at `http://10.10.1.10:8000/login/`, reachable vcloud-side; old `.201` dead. +- Tailscale: `BackendState=Running`, `PrimaryRoutes=["10.10.0.0/22"]` unchanged, tailnet + `100.64.0.53` unchanged; old `.202` dead. + +## How (method, for the DC replication that reuses this) + +- **MAAS could NOT be updated** -- it refuses interface edits on a **Deployed** machine ("Cannot + unlink subnet interface because the machine is not New, Ready, Allocated, or Broken"). The only + lift is Release->redeploy = WIPE (forbidden; destroys the seeded sandbox). So MAAS-model drift is + ACCEPTED: MAAS still shows `.201`/`.202` `mode=auto`. It reconciles for free at the next + teardown/redeploy. No collision: MAAS auto-assigns nodes from `.201+` upward (observed -- the two + services themselves landed at `.201`/`.202`), so the low `.2-.49` band is off its allocation path. + (Being outside the dynamic range is NOT the guarantee; the whole non-dynamic space is the static pool.) +- The live address was set on each **guest** via BOTH `/etc/netplan/50-cloud-init.yaml` and + `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (the latter so it survives a reboot / + cloud-init re-render), then `netplan apply`. Backups saved on each guest: `.bak-reip201` / + `.bak-reip202`. +- Driven over **`lxc exec` from voffice1**, NOT SSH-to-the-guest-IP -- the LXD socket is immune to + the `netplan apply` that swaps the address, so the control channel never dropped. `lxc exec`/ + `console` was also confirmed as the recovery backstop BEFORE the first apply. +- NetBox `ALLOWED_HOSTS` is `*` (config default, not overridden in `env/netbox.env`), so it served + at `.10` with no restart. Tailscale re-asserted automatically (no `tailscale up` needed). + +## Repo references swept (`.201->.10`, `.202->.11`) + +- **Code:** `SANDBOX_HOSTS` allowlist in `netbox/dc-dc-prefixes-import.py`, + `netbox/roles-aggregates-import.py`, `netbox/d115-office-carve.py`; + `tests/dc-dc-prefixes-import/test_logic.py` (both sandbox-URL hosts). The on-box working copy + `/home/ubuntu/nbimport/netbox/d115-office-carve.py` was updated too (else a `--commit` against the + now-`.10` sandbox would be REFUSED as non-sandbox). +- **Docs:** `docs/vr1-office1-as-built.md` (+ a MAAS-model-drift caveat), `docs/session-ledger.md`, + `docs/dc-dc-netbox-buildout-scope.md`, `docs/dc-dc-deployment-workflow.md`, + `runbooks/dc-dc-phase1-office1-standup.md`, `docs/design-decisions.md` (D-120 flip + Execution), + `runbooks/dc-dc-office1-service-reip.md` (PENDING markers resolved to the measured method). +- Dated changelogs left as historical snapshots (they record the deploy-time `.201`/`.202`). + +## Gauntlet + +`bash scripts/run-tests-all.sh` -> ALL GREEN (58 harnesses). `bash scripts/repo-lint.sh` -> 0 fail, +1 documented legacy WARN (design-decisions non-ASCII carve-out; my additions are ASCII-clean). + +## Still owed (NOT done here) + +- **D-120 Step 6 / C2:** register the `10.10.1.0/24` child ranges (static `.2-.49` / dynamic + `.100-.200` / node `.201-.254`) and the two service IPs IN the sandbox NetBox, then feed upstream. + Rides with C2. + +## Revert + +- **Repo:** `git revert <this commit>` restores the `.201`/`.202` references. +- **Live services:** on each guest via `lxc exec`, restore `/etc/netplan/50-cloud-init.yaml` and + `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` from the `.bak-reip201` / `.bak-reip202` backups, + then `netplan apply`. MAAS needs no rollback (its model was never changed). The on-box importer + copy reverts with `sed -i s#10.10.1.10#10.10.1.201#g /home/ubuntu/nbimport/netbox/d115-office-carve.py`. diff --git a/docs/changelog-20260715-docfix195-c2-netbox-of-record.md b/docs/changelog-20260715-docfix195-c2-netbox-of-record.md new file mode 100644 index 0000000..657ec1a --- /dev/null +++ b/docs/changelog-20260715-docfix195-c2-netbox-of-record.md @@ -0,0 +1,75 @@ +# 2026-07-15 -- DOCFIX-195: office1-netbox holds the VR1 IPAM APEX role (working draft); apex write-back deferred; C2 redefined + +## Why + +Operator ruling (2026-07-15): **`office1-netbox` (10.10.1.10) HOLDS THE IPAM APEX ROLE during VR1** -- +it is the authoritative NetBox every consuming system (OpenTofu, MAAS, overlays, the Juju bundle) +derives its values from. It is a *working draft* (edited as the buildout proceeds), but it is the apex. +**`netbox.baldurkeep.com` does not get a write at all during VR1**; it is the eventual production apex, +frozen as a READ-ONLY v1 reference draft and not consulted by VR1 systems. Merging `office1-netbox` back +into `netbox.baldurkeep.com` is a deferred maybe at the very end -- NOT a Stage 2 gate. + +**Confirmed while scoping:** no live system config points at any NetBox URL (`opentofu/`, `scripts/`, +`lib-*.sh` carry none) -- "consuming systems use office1-netbox as apex" is a value-DERIVATION workflow +(we read NetBox by hand to populate `lib-net.sh`/tfvars/bundle), not a live API wiring, so no config +repoint is needed; this is a documentation change only. + +The repo encoded the OPPOSITE model: it called `netbox.baldurkeep.com` the "apex/source of truth" and +`office1-netbox` a throwaway "sandbox", and defined close-out **C2** (the last open Stage 2 gate, and +the branch->`main` merge trigger) as an operator-gated write that feeds the carve UPSTREAM. Under the +ruling that write does not happen in Stage 2, so C2-as-written was wrong. + +**Finding surfaced while drafting (new, previously unlogged):** D-120 Step 6 registers `10.10.1.0/24`'s +child ranges (`ipam/ip-ranges`) and the two service IPs (`ipam/ip-addresses`). Those object classes are +NOT covered by the sandbox loop -- `prod-draft-dump.py`'s `ENDPOINTS` and `sandbox-fidelity-check.py`'s +`KEY`/`EXPECTED_NEW` both stop at rirs/roles/regions/sites/aggregates/prefixes. So the hardened fidelity +check cannot verify the D-120 load; that verification needs a tool extension (+ harness) or a recorded +manual read-back. C2 now records this as a tooling gap and an open C2-execution sub-question. + +This is a DOCFIX, not a D-NNN: it defers the apex WRITE and does not revoke D-010 (netbox.baldurkeep.com +remains the eventual source of truth). Doc-only; no script or live-cloud change. + +## What changed (all documentation) + +1. **`docs/dc-dc-deployment-workflow.md`** -- Stage 2 State-paragraph NetBox bullet + the **C2** row. + C2 redefined: office1-netbox holds the COMPLETE validated dataset (carve + roles + both DCs + the + D-120 child ranges + service IPs), verified by (i) a hardened-fidelity-check re-run on the shared + objects and (ii) a load+verify of the D-120 objects; the tooling gap is logged; apex write-back + deferred. C2 stays **OPEN**. +2. **`docs/dc-dc-netbox-buildout-scope.md`** -- section 8.1 (inverted the upstream/office1 roles: office1 + is the record, the apex is a read-only v1 draft, merge-back deferred) and section 8.6 (retitled and + rewritten from "Feeding the delta upstream" to "office1-netbox holds the validated dataset; apex + write-back DEFERRED", with the D-120 tooling gap and the deferred-merge-back procedure). +3. **`docs/session-ledger.md`** -- five spots corrected (D-120 Step 6 item; OPEN-WORK item 2; the + NetBox WRITE-PATH BUGS preamble; the fidelity-recheck sequence note; the PINNED merge-point bullet) + plus a NEW PINNED bullet recording the deferred apex write-back. All edits are in narrative sections; + the machine-derived block was not touched. +4. **`docs/design-decisions.md`** -- a dated Note under **D-010** recording the VR1-phase apex role + (office1-netbox is the working-draft apex; apex write-back deferred; defers the write, does not + revoke D-010), AND a pointer on **D-103**'s "NetBox remains the IPAM apex; MAAS and overlays consume + NetBox values" bullet naming office1-netbox as that apex during VR1. +5. **`netbox/README.md`** -- a banner so the `NETBOX_URL=https://netbox.baldurkeep.com` examples are not + mistaken for the VR1 write target (that is office1-netbox; the apex URL is for the deferred merge-back + only, gated by `--yes-write-upstream`). + +## Also logged (NOT actioned here) + +- **Tooling gap:** the sandbox dumper + fidelity check do not cover `ip-ranges`/`ip-addresses`; the + D-120 load cannot be machine-verified until they are extended (each with its harness) or a manual + read-back is recorded. This is the real remaining C2-execution work. +- **Durability finding:** `office1-netbox` is now load-bearing VR1 IPAM state on a lab VM; its + backup/durability matters once it is the record (log-only; not part of this DOCFIX). + +## Revert + +Each item is an isolated text edit; revert individually: + +1. `git checkout HEAD -- docs/dc-dc-deployment-workflow.md` +2. `git checkout HEAD -- docs/dc-dc-netbox-buildout-scope.md` +3. `git checkout HEAD -- docs/session-ledger.md` +4. `git checkout HEAD -- docs/design-decisions.md` +5. `git checkout HEAD -- netbox/README.md` +6. `git rm docs/changelog-20260715-docfix195-c2-netbox-of-record.md` + +Reverting restores the prior "feed the carve upstream = C2" framing across all five docs. No code paths, +scripts, harnesses, or live NetBox state are affected by this change or its revert. diff --git a/docs/changelog-20260715-fidelity-check-scope-correctness.md b/docs/changelog-20260715-fidelity-check-scope-correctness.md new file mode 100644 index 0000000..7a4983d --- /dev/null +++ b/docs/changelog-20260715-fidelity-check-scope-correctness.md @@ -0,0 +1,83 @@ +# 2026-07-15 -- C2 sub-task (i): sandbox re-verified faithful; fidelity checker scope false-positive fixed + +## Context + +C2 (Stage 2 close-out) sub-task (i), per DOCFIX-195: re-run the HARDENED fidelity +check against `office1-netbox` (the VR1 IPAM apex, a working draft), because the +"faithful" verdict predated the 2026-07-14 hardening. Executed this session from the +jumphost, off the newly-consolidated sandbox token (see SEC-009 note below) through +an SSH tunnel to `10.10.1.10:8000`. + +## What happened + +1. **Read-only dump** of `office1-netbox` via `netbox/prod-draft-dump.py` (token + sourced from `~/vr1-office1-creds/vr1-netbox-sandbox.env`, never printed). Sandbox: + 3 RIRs / 29 roles / 5 regions / 11 sites / 5 aggregates / 134 prefixes. + +2. **The hardened checker flagged 2 problems** -- `10.10.0.0/16` (office) and + `172.30.0.0/16` (edge) "have NO SCOPE" -- exit 1. + +3. **Investigated rather than accepted the failure.** Both are role-level container + /16s that `netbox/d115-office-carve.py` creates with `scope=None` **by design** + (lines 74/84). And the upstream apex draft ALREADY carries **27 unscoped** role + containers (`10.0.0.0/8`, `172.16.0.0/12`, `23.157.124.0/24`, ...). So unscoped + role containers are the established convention; these two conform. A full + cross-check of all 44 delta prefixes against their INTENDED scope (from the two + importers) was **44/44 exact** -- every DC prefix bound to the CORRECT DC site + (f02/nibble-2 -> vr1-dc0, f03/nibble-3 -> vr1-dc1), no wrong-target binding, no + hidden scope drop. **The sandbox is faithful.** The checker over-flagged. + +4. **The checker had a false-positive defect.** Its 2026-07-14 hardening added a + "DELTA prefix has NO SCOPE -> fail" rule to catch silent scope drops, but it + over-corrected: it failed ANY unscoped delta prefix, including the legitimately- + unscoped role containers. A checker that reds on a faithful sandbox is as useless + as one that greens on an unfaithful one, so it had to be fixed before sub-task (i) + could close on a real exit 0. + +## The fix (`netbox/sandbox-fidelity-check.py` + harness) + +- New `EXPECTED_PREFIX_SCOPE`: the INTENDED scope per delta prefix -- `("site",slug)` + / `("region",slug)` / `None` for an intentionally-unscoped org container. The DC + half is generated by `_dc_prefix_scopes()` (the D-119 DC->site identity), so the + CIDR set and the scope map cannot drift apart; `EXPECTED_NEW["ipam/prefixes"]` is + now `set(EXPECTED_PREFIX_SCOPE)`. +- The delta check now asserts **scope == INTENDED** (reading the dumper's + slug-rewritten `scope_site`/`scope_region`), not merely "has a scope." This kills + the false-positive AND is strictly STRONGER: it still catches a dropped scope, and + now also catches a **WRONG-TARGET** binding -- a DC0 prefix bound to `vr1-dc1`, the + exact D-117 near-miss the old presence-only test was blind to. +- **NOT fixed by exempting the two CIDRs** (that would re-open a false-green hole on + precisely those prefixes). Intended-None is asserted like any other intended scope. +- Harness (`tests/sandbox-fidelity-check/run-tests.sh`): fixtures moved to the dump + format (`scope_site`), T1 now sets each planned prefix's scope from the checker's + own `EXPECTED_PREFIX_SCOPE`, and two cases added -- **T9** (wrong-target scope -> + fail) and **T10** (legitimately-unscoped role container -> pass). 10 -> 12 tests. + +## Result + +- Re-run against the real sandbox dump: **exit 0** -- "SANDBOX = upstream draft + + EXACTLY the planned delta (D-115/D-117/D-118). Nothing lost, nothing stray." +- Harness **12/12**; gauntlet **ALL GREEN (58)**; repo-lint 0 fail (1 legacy WARN). +- **C2 sub-task (i) is CLOSED** -- the sandbox is faithful, established by a checker + that no longer false-fails. Sub-task (ii) (load + verify the D-120 ip-ranges + + service IPs) remains, with its logged tooling gap (dumper/checker don't yet cover + ip-ranges/ip-addresses). + +## SEC-009 note (sandbox token consolidation) + +The SEC-009 consolidation had MISSED the sandbox NetBox token: it lived only at +`/root/netbox-secrets/api.token` on office1-netbox, never in `~/vr1-office1-creds/`, +contrary to the standing convention ("ALL sensitive files for a site live in +`~/<site>-creds/`"). Consolidated this session to +`~/vr1-office1-creds/vr1-netbox-sandbox.env` (0600), token copied off the VM without +being printed. Recorded in the SEC-009 row. + +## Revert + +- `git checkout HEAD -- netbox/sandbox-fidelity-check.py tests/sandbox-fidelity-check/run-tests.sh` + restores the pre-fix checker (which false-fails on the two role-container /16s). +- `git checkout HEAD -- docs/security-ledger.md docs/session-ledger.md` reverts the notes. +- `git rm docs/changelog-20260715-fidelity-check-scope-correctness.md`. + +No NetBox state was changed by any of this -- the dump and the check are read-only, +and the token consolidation is a jumphost-local file copy. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 40ca0e8..5934adf 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -43,22 +43,105 @@ | **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase0-vcloud-prep.md`** -- command-level steps (host identify -> measure CPU/RAM/disk/nested-KVM/MTU -> enable nested KVM -> prepare pool paths -> install/confirm OpenTofu -> tfvars -> init/validate/plan/apply -> post-apply gate verify), each MUTATION individually gated per this repo's discipline. Flags a real open structural question up front (the `provider "maas"` block in `main.tf` is unconditional even though this stage only touches libvirt resources -- may force `maas_api_url`/`maas_api_key` to be set at `plan` time with nothing yet using them; noted as a possible DOCFIX candidate, not silently worked around). NOT YET EXECUTED against real infrastructure -- this is the first runbook that will be, tomorrow morning, per the operator's own stated plan. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet -- this runbook's Step 8 is the first real run. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. +**State:** **DONE -- executed 2026-07-10** (first DC-DC runbook run against real +infrastructure; vcloud host, OpenTofu v1.12.3). As-built: nested KVM already-on; +`underlay_mtu=9000` (jumbo, operator ruling -- the isolated planes/mesh are +host-internal virtio bridges, jumbo-capable regardless of the 1500 ISP uplink); +Ceph size=3 disk-budget plausibility PASS (4.60 TiB margin); 13 libvirt objects +applied (6 dc1 planes + dc1/office1/dc2 pools + office1-local + 3 mesh legs, all +MTU 9000). Prior VR0 topology torn down (`wan` kept as a gap-#17 model). DC2 +PLANES deferred (Option B -- awaits NetBox supernet); DC2 storage pool wired. +Delivered DOCFIX-178 (prereq installers), DOCFIX-179 (scaffold: per-module +required_providers + fmt + maas-provider deferral). See those changelogs. --- -## Stage 2 -- Phase 1: Office1 headend standup +## Stage 2 -- Phase 1: Office1 SITE standup (RESHAPED 2026-07-13 by D-114) + +> **!!! MODEL CHANGE (D-114, ADOPTED 2026-07-13). The three-sibling-service-VM model is +> SUPERSEDED.** Stage 2 no longer builds three peer VMs (MAAS-region, NetBox, GitBucket) on the +> vcloud host with Tailscale on the headend host. It builds **ONE Office1 site containment VM, +> `voffice1`** (Ubuntu 24.04, OpenTofu-created), which IS the facility: **MAAS-region runs ON it, +> LXD (pinned to the 5.21 LTS track) runs ON it**, that LXD is **registered BACK INTO MAAS as an +> LXD KVM host**, and the non-stack service machines (**NetBox, Tailscale** -- GitBucket removed per D-116) are VMs +> **COMPOSED BY MAAS** into it -- MAAS-visible: enlisted, commissioned, deployed, powered. This +> is VR0's own already-proven `lxd` + `tailscale` pattern, applied per site. +> +> **Explicitly NOT affected:** the Juju-deployed OpenStack services on the DC nodes. They stay in +> Juju-created LXD **CONTAINERS** (bundle `lxd:N` placements), invisible to MAAS, exactly as in +> VR0. D-114 grants MAAS a composition right scoped ONLY to the LXD VM host and the non-stack +> machines; `modules/node-vm` still pre-creates the OpenStack node VMs and MAAS merely DISCOVERS +> them (D-103 unchanged on that point). +> +> **Sequencing is an operator ruling:** Office1 FIRST, fully up on this model. **DC1 is GATED +> behind it** and is not started until Office1 is complete; DC2 follows DC1. Stages 3-7 below are +> therefore gated behind THIS stage, and their own content is otherwise unchanged by D-114. | | | |---|---| -| **Goal** | Stand up the operator's deployment headend first, before any DC. | -| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only); Office1's own OPNsense simulated-ISP edge (gap #16, DECIDED 2026-07-09: Stage 2 owns it). | -| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only; Office1-local network created (gap #12 CLOSED); Office1 OPNsense edge module wired (instantiation still blocked on real specs + gap #17's WAN-side network). | -| **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | -| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | -| **Authoring status** | **Runbook WRITTEN 2026-07-09, UPDATED 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. The Office1-local-network gap it originally flagged (gap #12) is now CLOSED (`opentofu/modules/office1-network`, instantiated for real), and a new Step 4b now creates Office1's own OPNsense edge (gap #16, CLOSED -- ownership decided). NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | +| **Goal** | Stand up the Office1 SITE first, before any DC -- and prove the D-114 containment-VM model at its cheapest point. | +| **Build** | `voffice1` site containment VM (OpenTofu, `modules/cloudinit-vm` + `modules/base-image`, on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`); MAAS region controller ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); that LXD registered into MAAS as an LXD KVM host; NetBox + Tailscale as MAAS-COMPOSED LXD VMs inside it (**GitBucket REMOVED from scope -- D-116**); Office1's own OPNsense simulated-ISP edge (gap #16, Stage 2 owns it). | +| **Gate** | Office1 OPNsense edge up (**MET** -- routing, NAT, egress, Kea DHCP; gap #17 closed for Office1 via `office1-wan`); `office1-local` network created (gap #12 CLOSED); OpenTofu reaches vcloud host libvirt (**MET**); **`voffice1` exists and runs (MET 2026-07-13, measured)**; `voffice1` has a reserved address + SSH + nested KVM exposed; MAAS region reachable ON `voffice1`; LXD on the 5.21 track with `lxdbr0` OFF `office1-local`; LXD registered as a MAAS KVM host; **the non-stack machines COMPOSED BY MAAS and deployed -- i.e. an LXD VM actually BOOTS at L3, which is D-114's DC1 entry gate**; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); Tailscale confirmed to Office1 only. **GitBucket is NO LONGER A GATE (D-116): no Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record.** | +| **Owns** | **D-114** (site containment VM + MAAS-composed LXD VMs, incl. the LXD 5.21 pin), D-103 as AMENDED by D-114 (OpenTofu owns the containment VM and the site networks; MAAS composes INSIDE the site), D-107 (headend is not a core-service provider). | +| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but LESS new than it looks: the MAAS-composed-LXD-VM half is **exactly what VR0 runs today** (its `lxd` KVM host at LXD 5.21.4, with `tailscale` composed into it at 2 cores / 2 GiB / 25 GB) and carries **NO Roosevelt delta**. The containment VM itself IS a deliberate simulation-only delta -- it is what makes a "hard-down an entire facility" DR drill (`virsh destroy vvr1-dc0`) executable at all. NetBox-as-IPAM-apex discipline and the `netbox/*-import.py` pattern transfer unchanged. | +| **Authoring status** | **Runbook REWRITTEN 2026-07-13 to the D-114 model: `runbooks/dc-dc-phase1-office1-standup.md`.** The old Option A / Option B provisioning fork is **CLOSED** -- OpenTofu (Option A) is the path and `module "voffice1"` is a real, applied `cloudinit-vm` instantiation (real image source, real `user_data`/`meta_data`/`network_config`, plus the new `expose_nested_virt` variable). **PENDING VERIFICATION:** the MAAS/LXD install + LXD-KVM-host registration + VM-compose command lines are NOT yet confirmed against Canonical's current docs -- the runbook marks each unconfirmed flag/channel/subcommand rather than fabricating one, and a separate verification pass owns filling them in. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned). +### !!! STAGE 2 IS THE BRANCH MERGE POINT (operator ruling, 2026-07-14) !!! + +Branch `dc-dc-stage1-phase0-first-exec` merges to `main` at **Stage 2 CLOSE** -- when every +gate bullet above is MET, not before. This is NEW policy: `main` has never taken a feature +merge (its only merge commits are inherited from the pre-rename `openstack-caracal-ipv4` +remote, D-110), so the entire VR1 buildout -- D-112 through D-118, the Office1 headend, the +NetBox tooling -- currently exists ONLY on the branch. Until the merge lands, `main` is not +tracking the mission and must not be treated as trunk-of-record for VR1. + +The merge is an operator-gated action, not a session-end habit. See the Stage 2 close-out +checklist below for what must be true first. + +--- + +**State (2026-07-14): SUBSTRATE + SITE BOTH DONE. Stage 2 is in CLOSE-OUT, not in build.** + +> The 2026-07-13 "NOT DONE" list that stood here is **SUPERSEDED and was already stale when +> written** -- MAAS-region, LXD, the LXD-KVM-host registration and both composed service +> machines all landed the same day. This doc's status table has now drifted three times; the +> correction below is measured, not assumed. + +- **DONE (substrate):** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is + BUILT and LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on + udp/67). Config is REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1` + EXISTS and is RUNNING** (`module.voffice1` is in `opentofu/terraform.tfstate`). +- **DONE (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5 + ON `voffice1`; that LXD **registered back into MAAS as an LXD VM host**; and both service + machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.10, NetBox 4.6.4) and + `office1-tailscale` (10.10.1.11, subnet router for `10.10.0.0/22`, route now ENABLED on the + self-hosted control server). GitBucket is out of scope (D-116), so Office1's composed machines + are exactly TWO. Codified in `scripts/site-headend-install.sh` (harness 18/18) -- reusable for + the DCs. +- **The L3 nesting proof is NO LONGER an unknown -- it is DEFINITIVE.** A guest booted, PXE'd and + COMMISSIONED three levels deep. DC1 inherits the proof; D-114's model stands. +- **NetBox is populated in `office1-netbox` -- the VR1 IPAM APEX, a working draft (DOCFIX-195)** (D-115 + office carve + the six-plane roles + both DCs, 90 -> 134 prefixes). The apex literals that blocked it + are now assigned: **D-117** (site naming) and **D-118** (`ORG_ULA_48 = fd50:840e:74e2::/48`). + **Operator ruling 2026-07-15 (DOCFIX-195):** during VR1, `office1-netbox` (10.10.1.10) takes ALL + NetBox writes and HOLDS THE IPAM APEX ROLE -- every consuming system (OpenTofu, MAAS, overlays, the + Juju bundle) derives its NetBox values from it. It is a *working draft* (edited as the buildout + proceeds), but it is the authoritative NetBox. `netbox.baldurkeep.com` is the production apex, held + as a READ-ONLY v1 reference draft and NOT consulted by VR1 systems; the write-back to it is DEFERRED + to end-of-deployment (may not happen) -- it is NOT a Stage 2 gate. So the "NetBox authoritative + populated" gate is met when `office1-netbox` holds + the COMPLETE validated dataset, NOT by any upstream write. See C2 for what "complete" and "verified" + require (the D-120 additions, plus a fidelity-check re-run -- the earlier "fidelity check clean" + verdict predates the 2026-07-14 hardening and must be re-established). + +**Stage 2 close-out checklist (what must be true before the merge):** + +| # | Item | State | +|---|---|---| +| C1 | Office1 LAN IPv6 deployed (`2602:f3e2:f01:100::1/64` + RA on the edge LAN). WAN v6 leg DEFERRED: v6 does not egress the lab. | **DONE 2026-07-14** -- live and PROVEN ON THE WIRE: address on the kernel, radvd advertising the prefix (`mode=unmanaged`), `voffice1` autoconfigured `...:5054:ff:fe6a:87e5` by SLAAC, edge->voffice1 GUA ping 0.0% loss. NOTE the D-113 AMENDMENT: the ADDRESS half is **not** REST-API-doable (`scripts/opnsense-set-interface-v6.sh`); only the RA half is. `docs/changelog-20260714-office1-lan-ipv6-executed.md` | +| C2 | `office1-netbox` (10.10.1.10) is the VR1 IPAM apex -- a working draft (DOCFIX-195) -- and holds the COMPLETE validated dataset: D-115 carve + six-plane roles + both DCs' prefixes/aggregates/RIRs, PLUS the D-120 additions -- `10.10.1.0/24`'s child ranges (`ip-ranges`: static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11). `netbox.baldurkeep.com` stays a READ-ONLY v1 reference draft; the write-back to it is DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15) -- NOT a Stage 2 gate. | **DONE 2026-07-15.** (i) The HARDENED fidelity check was re-run against office1-netbox -> exit 0 (faithful); it also surfaced + fixed a checker scope false-positive (`docs/changelog-20260715-fidelity-check-scope-correctness.md`). (ii) The tooling gap was CLOSED -- `prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover `ip-ranges`/`ip-addresses` -- and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox via `netbox/d120-compose-bands.py --commit` and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned delta (D-115/D-117/D-118/D-120)." `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. Gauntlet ALL GREEN (60). | +| C3 | The `$DC` shell-selector collision resolved. | **DONE 2026-07-14 (D-119)** -- selectors are REGION-QUALIFIED (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`) across `lib-net.sh`, `lib-hosts.sh`, `opentofu/`, the importer and every runbook call site. Bare `dcN` is REJECTED. The importer's DC->site map is now an IDENTITY, so the off-by-one is structurally impossible. Gap #19 CLOSED. | +| C4 | The NetBox sandbox loop documented. | **DONE 2026-07-14** -- `docs/dc-dc-netbox-buildout-scope.md` section 8 (rationale + write-guard table + WAF trap) and `runbooks/dc-dc-phase1-office1-standup.md` Step 10b (operational sequence). | +| C5 | This doc's Stage 2 row reconciled against reality (it has drifted three times). | **DONE 2026-07-14** | --- @@ -169,6 +252,32 @@ ## Tooling gap register (audited 2026-07-09; update as items close) +> ### !!! STALE-CONTENT WARNING (2026-07-13) -- READ BEFORE ACTING ON ANY OPNSENSE ITEM BELOW !!! +> +> Everything in this document describing OPNsense edge config delivery via a rendered +> `config.xml` and/or a **config ISO** is **OBSOLETE and, against the live Office1 edge, +> DANGEROUS**: +> +> - **The config ISO is INERT (D-112).** The OPNsense Configuration Importer can NEVER fire on a +> pre-installed nano image -- it probes for a read-only root, finds a writable one with a +> factory `/conf/config.xml`, and exits without enumerating any device. Nothing ever reads it. +> - **Full-`config.xml` delivery is SUPERSEDED (D-113(a2), ruled + PROVEN 2026-07-13).** Edge +> config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read +> AND write -- against the live Office1 edge. +> - **Office1's DHCP is now API-MANAGED.** A full rendered `config.xml` push to `10.10.0.1` would +> CLOBBER live state (it replaces `/conf/config.xml` wholesale, dropping ~667 +> migration-populated elements including the only 2 firewall pass rules). +> +> The as-built path that actually worked: prep image -> boot on factory defaults -> **D-112(c)** +> console bootstrap -> enable SSH + install key -> configure over SSH and the REST API. See +> `docs/changelog-20260713-opnsense-api-proven.md` and +> `docs/changelog-20260713-opnsense-api-write-proven.md`. +> +> **OPEN (D-113(a2), not yet done):** reduce `opentofu/templates/opnsense-config.xml.tmpl` to a +> MINIMAL bootstrap (sshd + root key + console), retire the config-ISO path and the module's +> `config_seed`/cdrom wiring, and rewrite Stage 3's edge steps around the API. Until that lands, +> treat every OPNsense step in this doc and in `runbooks/dc-dc-phase1/phase2` as reference-only. + Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not inferred from the design doc alone. Ordered roughly by "you'll hit this first," not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc; @@ -188,7 +297,50 @@ `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` when it's authored -- that call-site wiring is part of writing that runbook, not this delivery. -2. **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded +2. **OpenTofu** -- **STATUS CORRECTED 2026-07-13. The "UNVALIDATED / no tofu + binary" framing below is now FALSE and must not be relied on:** + - **A `tofu` binary EXISTS on the jumphost: OpenTofu v1.12.3.** The tree can + self-check. The whole "authored in a session with no binary" caveat that + rides along with every OpenTofu instruction in this repo is now closed as + a *capability* statement. + - **The tree VALIDATES: 10/10 modules**, root + every module standalone + (`bash scripts/opentofu-validate.sh`). + - **It has also been APPLIED for real.** `opentofu/terraform.tfstate` holds + 11 resources: the DC1 planes, the pools, the three mesh links, the + `office1-network`, and the Office1 OPNsense edge domain + volumes. This is + no longer a paper tree. + - **BUT two modules were BROKEN and had NEVER been parsed by any tool** + (DOCFIX-194, 2026-07-13): `node-vm` (`libvirt_volume` with a bogus + top-level `format` -- it nests under `target`) and `netem-link` (a + destroy-time provisioner referencing `var.*`, which OpenTofu rejects at + INIT -- the module could not even initialize). Root's `main.tf` does not + call either one, and `tofu validate` on the root module never parses an + uncalled module -- so the gate printed **PASS** over both. Both are fixed; + the gate now validates EVERY module standalone (S3). **`node-vm` builds + every DC node and `netem-link` is the Stage 6 failover-drill mechanism** -- + they would have detonated mid-deploy. See + `docs/changelog-20260713-docfix194-opentofu-module-validation.md`. + - **`modules/cloudinit-vm` IS NOW INSTANTIATED FOR REAL (2026-07-13, D-114).** + The long-standing "the mechanism exists but no image source is chosen and no + `user_data`/`meta_data`/`network_config` has been designed for it" caveat -- + which is what forced Stage 2's old Option A / Option B fork -- is **CLOSED + for `voffice1`**: `module "voffice1"` in root `main.tf` carries a real image + source (`modules/base-image` -> the official Ubuntu 24.04 noble cloud + image), real cloud-init content, and a `network_config` that DHCPs off the + Office1 edge's Kea (matching the NIC by glob, not by a guessed kernel name). + The module also **gained an `expose_nested_virt` variable** -- load-bearing, + because LXD virtual machines are qemu/KVM guests and D-114's composed + service machines cannot boot without the host CPU feature passed through. + Applied; the domain is running. **The three Office1 service VMs are NOT + cloudinit-vm targets any more at all** -- under D-114 they are MAAS-COMPOSED + LXD VMs, so no further `cloudinit-vm` design work is owed for them. + - **Still true:** `tofu plan`/`apply` has NOT been exercised for the Stage 3 + DC substrate. Validation is not application. + + *Original 2026-07-09 text follows (kept for the module inventory; its status + claims are superseded by the block above).* + + **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded 2026-07-09 (was: zero files).** `opentofu/` now has: `modules/dc-planes` (six per-DC planes), `modules/mesh-link` (the D-100 dark-fiber triangle legs), `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot @@ -400,9 +552,16 @@ this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`, already real required inputs elsewhere in the same file. See `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` - (DOCFIX-163) for the full design rationale. Designing the three - service VMs' actual `cloudinit-vm` `network_config` content against this - network remains separate, un-actioned work. + (DOCFIX-163) for the full design rationale. **Follow-on work also DONE + 2026-07-13 (D-114):** the "designing the service VMs' actual `cloudinit-vm` + `network_config` against this network" tail of this gap is closed, but not + the way it was written -- there are no longer three service VMs to attach. + ONE VM attaches to `office1-local`: the `voffice1` containment VM, whose + `network_config` DHCPs from the OPNsense edge's Kea. The service machines + live INSIDE `voffice1`'s LXD (MAAS-composed) and never touch this network + directly. **DHCP hazard to carry forward:** LXD's `lxdbr0` runs its OWN + dnsmasq -- it must stay on LXD's internal NAT bridge and must NEVER be + bridged onto `office1-local`, where Kea is authoritative. 13. **The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real risk still open.** `overlays/dc-dc-ipv6-family-matrix.yaml` now exists, sourced against real charm config/source (`docs/dc-dc-ipv6-charm- @@ -488,7 +647,34 @@ #12's new network; its WAN side does not -- see NEW gap #17 immediately below, a genuinely deeper, cross-site gap surfaced while closing this one, not invented as a substitute for a real answer. -17. **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in + **D-114 follow-on (2026-07-13):** the edge is now BUILT and LIVE, and it is + a load-bearing PREREQUISITE of the rest of Stage 2, not just a gate bullet + -- `voffice1` takes its DHCP lease from the edge's Kea and reaches the + internet through it. Office1's edge is also the only one of the three that + is live; DC1/DC2's edges are Stage 3's problem and are still blocked on gap + #17's DC half. +17. **PARTIALLY CLOSED 2026-07-13 -- the DECISION is made and PROVEN for + Office1; DC1/DC2 still have no uplink network.** + - **Office1: CLOSED.** The answer was the one this gap proposed -- a + dedicated per-site ISP-uplink `libvirt_network`, **not** a mesh leg. + `office1-wan` exists and is live (measured 2026-07-13: `virbr11`, NAT + forward, `172.30.1.0/24`). The Office1 edge's WAN sits on it at + `172.30.1.2` with default route `172.30.1.1`, and egress to the internet + is verified (0.0% loss). So the topology question is ANSWERED and the + pattern is PROVEN in production, not merely designed. + - **DC1/DC2: STILL OPEN.** Neither has an ISP-uplink network. Measured: + `dc1-provider-public` has **no IP and no forward** (isolated), and there + is no `dc1-wan`/`dc2-wan` at all. Stage 3's `dc1_opnsense` still carries + the unresolved `wan_network_name` placeholder described below. + - **The fix is now mechanical, not a decision:** replicate the + `office1-wan` shape per DC. The D-100 reasoning below (mesh legs are + MANAGEMENT-TRAFFIC-ONLY and must NOT carry an edge's WAN side) stands + unchanged and is what the Office1 build honoured. + + *Original 2026-07-09 text follows (the reasoning is still correct; the "for + ANY site / nowhere" status claim is superseded by the block above).* + + **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in this repo, for ANY site** (found 2026-07-09 while closing gap #16 for Office1). Every `modules/opnsense-edge` caller needs a `wan_network_name` distinct from BOTH its LAN-side network (a @@ -554,6 +740,39 @@ has no real machines to test against yet -- the runbook states the principle and points at MAAS's own docs rather than guessing flags. +19. **NEW 2026-07-14 -- THE `dc1`/`dc0` NAME NOW MEANS TWO DIFFERENT DATACENTERS. + This BLOCKS Stage 3 and is the same off-by-one D-117 just fixed, one layer down.** + + D-117 (ADOPTED, Option B) moved the repo onto the NetBox apex's 0-indexed + convention: VR1's **first** DC is `vr1-dc0`, its **second** is `vr1-dc1`. + That rename was executed on the IMPORT PATH ONLY (`netbox/dc-dc-prefixes- + import.py` + the scope doc). It was **NOT** executed on the two other + surfaces that speak `$DC`, and they now disagree with the apex: + + | Surface | says `dc1` means | says `dc0` means | + |---|---|---| + | NetBox apex (authority, D-117) | VR1's **SECOND** DC | VR1's **FIRST** DC | + | `opentofu/main.tf` (`dc1_planes`, `dc2_storage`, `mesh_dc1_dc2`) | VR1's **FIRST** DC | -- (no `dc0`) | + | `scripts/lib-net.sh` / `lib-hosts.sh` (DOCFIX-151 selectors) | VR1's first DC | **VR0's** DC0 -- a different REGION | + + So "the next DC we deploy" is, simultaneously: apex `vr1-dc0`, OpenTofu + `dc1_*`, and a `lib-net.sh` `$DC` value that ALSO names VR0's live testcloud. + Three names, one machine, and one of them collides with a **running cloud**. + + **Why a mechanical rename is a trap (D-117 TRAP 1/TRAP 2, now live here):** + `sed s/dc1/dc0/` on `opentofu/main.tf` would hand VR1's first DC the string + `dc0`, which `lib-net.sh` already resolves to **VR0's deployed literals**. The + fix is a SEMANTIC remap through placeholders, and the shell selectors must be + region-qualified (`vr0-dc0` / `vr1-dc0` / `vr1-dc1`), not merely renumbered. + + **Consequence if ignored:** `module "dc1_nodes"` gets authored against the + wrong site's prefixes, and the first VR1 DC deploys with the second DC's + addressing -- discovered, at best, at `juju deploy`. + + **Status: OPEN. It is Stage 2 close-out item C3 and the FIRST step of any + Stage 3 OpenTofu work.** Nothing in `opentofu/` should gain a new DC-keyed + module until it is closed. + --- ## Where things stand today (2026-07-09) -- update this as stages close @@ -563,15 +782,16 @@ | Design (`dc-dc-buildout-design.md`) | DONE -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) | | Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED | | DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | -| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. NONE executed against real infrastructure (prep-only session). See gaps #12-15 above for real design gaps the authoring pass surfaced. | -| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` | -| OPNsense image+config mechanism (`modules/opnsense-edge`) | BUILT 2026-07-09, UNVALIDATED (no `tofu`/ISO-tool/real boot to confirm), see `opentofu/README.md` | -| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | BUILT 2026-07-09, renderer fully tested (8/8) -- several tokens still pending Stage 0 ratification or a real boot, see `opentofu/templates/README.md` | +| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. Stage 1 EXECUTED 2026-07-10. **Stage 2's runbook REWRITTEN 2026-07-13 to the D-114 model** (one `voffice1` site containment VM + MAAS-composed LXD VMs, replacing the three sibling service VMs and their Option A/B fork); it is PARTIALLY EXECUTED, and its MAAS/LXD command lines are marked PENDING VERIFICATION rather than guessed. Stages 3-7 NOT executed, and D-114 GATES them behind Office1 completing. See gaps #12-15 above for real design gaps the authoring pass surfaced. | +| **D-114 site containment VM (`voffice1`)** | **BUILT + RUNNING 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. | +| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (OpenTofu v1.12.3 now on the jumphost): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds the DC1 planes, pools, 3 mesh links, `office1-network`, the OPNsense edge, and (2026-07-13, D-114) the `ubuntu_noble_base` image + the **`voffice1` containment VM**. **`modules/cloudinit-vm` is now instantiated for real** and gained `expose_nested_virt`. **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. | +| OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + RUNNING 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. | +| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **SUPERSEDED 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **OPEN:** reduce the template to a MINIMAL bootstrap (sshd + root key + console + a pre-seeded API key) and retire the ISO path + the module's `config_seed`/cdrom wiring. | | MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` | -| `tc netem` mechanism (`modules/netem-link`) | BUILT 2026-07-09, UNVALIDATED -- real latency/jitter/loss/rate parameters still an unruled D-100 item, see `opentofu/README.md` | +| `tc netem` mechanism (`modules/netem-link`) | **WAS BROKEN, FIXED 2026-07-13 (DOCFIX-194)** -- its destroy provisioner referenced `var.*`, which OpenTofu rejects at INIT, so the module could not even initialize. Now validates. Real latency/jitter/loss/rate parameters are STILL an unruled D-100 item. Never applied. | | `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` | | Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` | -| Office1 OPNsense edge ownership | DECIDED 2026-07-09 (DOCFIX-163, gap #16 CLOSED) -- Stage 2 owns it; `module "office1_opnsense"` skeleton in `main.tf` (commented, same as DC1/DC2's own not-yet-instantiated edges); WAN-side network still blocked on NEW gap #17 (no per-site ISP-uplink network exists anywhere yet, DC1/DC2 included) | +| Office1 OPNsense edge ownership | **BUILT AND LIVE 2026-07-12/13.** `module "office1_opnsense"` is instantiated (not commented). **Gap #17 is CLOSED FOR OFFICE1**: `office1-wan` (virbr11, NAT, 172.30.1.0/24) is the dedicated per-site ISP uplink, and the edge's WAN sits on it -- the D-100-honouring answer (NOT a mesh leg). **DC1/DC2 still have NO uplink network** -- replicating the `office1-wan` shape per DC is now mechanical, not a decision. | | Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` | | `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 4 and Stage 5's runbooks (both written 2026-07-09) mandate calling it, not yet executed for real | | NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned | diff --git a/docs/dc-dc-netbox-buildout-scope.md b/docs/dc-dc-netbox-buildout-scope.md index 8331c9a..0b4e323 100644 --- a/docs/dc-dc-netbox-buildout-scope.md +++ b/docs/dc-dc-netbox-buildout-scope.md @@ -55,7 +55,7 @@ | G2 | **Aggregates** | **0 aggregates** despite ARIN RIR present | Create aggregates (section 4b) -- IPAM apex should declare its allocations | | G3 | **Org ULA `/48`** | Not present (only `fc00::/8`, `fd00::/8` containers) | Assign + create (section 4c); the D-101 literal | | G4 | **VR1 VLAN groups** | VR1 DC0/DC1 have **no VLAN groups** (VR0 DC0 + prod sites do) | Create per-DC VLAN groups + VLANs | -| G5 | **DC naming** | NetBox = `vr1-dc0`/`vr1-dc1`; decisions + import tool = **DC1/DC2** (`--dc dc1` targets `vr1-dc1`, DC2 -> `vr1-dc2` which does not exist) | Operator IPAM-hygiene call: rename to `vr1-dc1`/`vr1-dc2`, or re-point the tool. **Blocks the import** until resolved | +| G5 | **DC naming** | ~~NetBox = `vr1-dc0`/`vr1-dc1`; tool = DC1/DC2~~ | **CLOSED by D-117 (ADOPTED 2026-07-13).** The REPO moved to the apex's 0-indexed names: `--dc dc0` -> `vr1-dc0`, `--dc dc1` -> `vr1-dc1`. The recommendation BELOW (rename the apex to `vr1-dc1`/`vr1-dc2`) was **REJECTED** -- the apex is not bent to fit the repo. The office KEEPS its number (`vr1-off1`). | | G6 | **VR0 DC0 / Willamette prefixes** | Fully built on the **OLD** model: single roles (`provider`/`metal`/`data`/`storage`/`repl`/`lbaas-mgmt`), **all GUA**, no ULA | This is the v1/DC0-era model, **not** the VR1 template. Leave as-is or retire (hygiene call, D-101 site-collision note) | --- @@ -113,9 +113,9 @@ literals. Recommended literals (pending G5 naming + section 5 ratification): ``` # DC1 (v4 inherited; only v6 top-level blocks required) -ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc1 +ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # D-117: VR1 DC0 # DC2 (distinct v4 supernet ALSO required -- >= /19; tool re-packs 6 sequential /22s, see 5.3) -ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f03::/48 DC2_V4_SUPERNET=10.13.0.0/19 --dc dc2 +ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f03::/48 DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # D-117: VR1 DC1 ``` Run `--verify-only` first. The tool creates the six-plane v4 + the D-101 family-matrix v6 legs. @@ -143,15 +143,19 @@ `/56`-per-DC index is independent of the GUA geography -- confirm that's acceptable (it matches the "internal `/48` reused across the domain" intent). **Recommend:** align the tool's per-plane `/64` offsets to the `NN` mnemonic so GUA and ULA read consistently, then ratify. + **RATIFIED as D-111 (2026-07-11) + implemented in `carve_v6` (DOCFIX-182):** NN + offsets, `/60`+`/64` per plane, metal `/60` shared by admin `:20` / internal `:21`, + ULA `/56` indexed to the GUA nibble so the 4th hextet reads `DC.NN` in both families. -2. **DC naming (G5).** `vr1-dc0`/`vr1-dc1` (NetBox) vs DC1/DC2 (decisions/tool). **Recommend:** - rename sites to `vr1-dc1`/`vr1-dc2` to match the adopted DC1/DC2 language and the tool's - `--dc dc2 -> vr1-dc2` expectation; retire or repurpose the now-orphaned `vr1-dc0`. +2. ~~**DC naming (G5).**~~ **SUPERSEDED -- do NOT follow this.** This recommended renaming the APEX + to `vr1-dc1`/`vr1-dc2`. **D-117 (ADOPTED 2026-07-13) ruled the opposite:** the repo moves to the + apex's names (`dc0`/`dc1`), because NetBox is the IPAM apex and is not bent to fit the repo. The + import tool now REJECTS `--dc dc2` and the `DC2_V4_SUPERNET` env var by name. 3. **DC2 v4 supernet value + size.** D-101 leaves it NetBox-assigned. The import tool (`DC2_MIN_SUPERNET_PREFIXLEN = 19`) carves **six SEQUENTIAL `/22`s** from the supernet base and FAILS LOUD below `/19` (a `/19` = 8 `/22`s; six used, two spare). So a **`/19` is correct** - (e.g. `10.13.0.0/19`); DC2's planes land at sequential offsets (`.0/.4/.8/.12/.16/.20`), NOT + (`10.12.64.0/19` per D-115; was `10.13.0.0/19`); DC2's planes land at sequential offsets (`.0/.4/.8/.12/.16/.20`), NOT mirroring DC1's `.4/.8/.12/.16/.32/.36` -- fine per D-101 (same six-plane *structure*, not same offsets). Just pick a non-colliding `/19` (per D-074). @@ -180,6 +184,127 @@ | Literal | Status | Recommendation | |---|---|---| | `ORG_ULA_48` | unassigned | `fd50:840e:74e2::/48` (ratify) | -| `DC_GUA_PREFIX` dc1 | ambiguous | `2602:f3e2:f02::/48` (existing VR1 DC0 site block, renamed dc1) | -| `DC_GUA_PREFIX` dc2 | ambiguous | `2602:f3e2:f03::/48` (existing VR1 DC1 site block, renamed dc2) | -| `DC2_V4_SUPERNET` | unassigned | `10.13.0.0/19` (distinct; tool needs >= `/19`, re-packs 6 sequential `/22`s -- see 5.3) | +| `ORG_ULA_48` | **ASSIGNED (D-118)** | **`fd50:840e:74e2::/48`** -- RFC 4193 valid, no collision with Tailscale`s `fd7a:115c:a1e0::/48`. Unblocks the DC import. | +| `DC_GUA_PREFIX` **dc0** | **MEASURED** (D-117) | `2602:f3e2:f02::/48` -- the apex's `vr1-dc0` block. Not renamed; the repo adopted the apex's name. | +| `DC_GUA_PREFIX` **dc1** | **MEASURED** (D-117) | `2602:f3e2:f03::/48` -- the apex's `vr1-dc1` block. | +| `DC2_V4_SUPERNET` | **ASSIGNED (D-115)** | **`10.12.64.0/19`** -- INSIDE the Cloud `/16`, free (DC1 holds `10.12.{4,8,12,16,32,36}`), DC1<->DC2 stay routable. The old `10.13.0.0/19` recommendation was OUTSIDE every allocated block (it would have squatted); D-115 supersedes it. Tool needs >= `/19` and re-packs 6 sequential `/22`s -- see 5.3. | + +## 8. The NetBox sandbox loop (C4) + +### 8.1 Why a second NetBox exists + +**DOCFIX-195 (operator ruling 2026-07-15) -- READ THIS FIRST; it INVERTS the roles the rest of this +section was originally written around.** During VR1 the VR1 IPAM APEX is `office1-netbox` +(`10.10.1.10:8000`, NetBox 4.6.4 on the Office1 VM, `docs/changelog-20260713-office1-netbox-deployed.md`): +it takes ALL VR1 writes (the section-4 buildout, the D-120 additions, any further refinement) and every +consuming system (OpenTofu, MAAS, overlays, the Juju bundle) derives its NetBox values from it, not from +`netbox.baldurkeep.com`. It is a *working draft* -- edited as the buildout proceeds -- but it is the +authoritative NetBox. `netbox.baldurkeep.com` is held as a READ-ONLY v1 REFERENCE DRAFT for +the duration -- it gets NO write during VR1. Merging the validated `office1-netbox` dataset back into +`netbox.baldurkeep.com` is a DEFERRED, end-of-deployment MAYBE (8.6), not a Stage 2 gate. This DEFERS +the apex write; it does NOT revoke D-010 -- `netbox.baldurkeep.com` remains the eventual IPAM source of +truth. + +The loop that builds and validates `office1-netbox` from the draft is unchanged; only the FINAL step +(feeding the apex) moved OUT of Stage 2: + + upstream draft (READ-ONLY, v1 reference) --[prod-draft-dump.py]--> netbox/draft/vr1-draft.json + vr1-draft.json --[sandbox-seed.py]------> office1-netbox (the RECORD) + ...apply the planned delta in office1-netbox (the section-4 importers + the D-120 additions)... + ...prove office1-netbox == draft + EXACTLY the planned delta (fidelity check)... + validated dataset --[DEFERRED, end-of-deployment, operator-gated]-> apex + +The importers NEVER `--commit` the apex as a side effect (8.4). Writing `office1-netbox` is normal and +unguarded (it is in `SANDBOX_HOSTS`); writing `netbox.baldurkeep.com` stays behind +`--yes-write-upstream` and is out of scope until the deferred merge-back. + +### 8.2 Why a FIDELITY CHECK, and not just counts + +Counts and idempotency CANNOT prove fidelity. On 2026-07-13 the seeder silently dropped the scope on +17 of 90 prefixes -- it mapped only `dcim.site`, and most of the VR1 draft is `dcim.region`-scoped -- +and every count still matched and re-running stayed idempotent. It was caught by luck on a spot-check +printing `site None`. `netbox/sandbox-fidelity-check.py` is the check that catches it on purpose: it +diffs FIELD BY FIELD on every shared object, so a dropped or mangled field is a hard failure. + +### 8.3 The 2026-07-14 hardening -- do NOT trust a pre-hardening "faithful" verdict + +Until 2026-07-14 the fidelity check could return a FALSE GREEN. It only proved the sandbox delta was +a SUBSET of the plan (`unexpected = extra - expected`); it NEVER proved the plan was CARRIED OUT. +With ZERO of the 36 DC prefixes created, `extra` and `unexpected` were both empty, and it printed +"Nothing lost, nothing stray" and exited 0. `EXPECTED_NEW` was an upper bound masquerading as an +assertion. Fixed 2026-07-14 (`docs/changelog-20260714-netbox-write-path-hardening.md`): `EXPECTED_NEW` +is now BOTH bounds (`EXPECTED-BUT-ABSENT` is a hard failure), and delta prefixes are checked for scope +and role. A NEW harness `tests/sandbox-fidelity-check/run-tests.sh` (10/10; T3 IS the false-green +scenario) ships with it -- the checker had shipped with NO harness, which is why the false green +survived. + +CONSEQUENCE: any "sandbox is faithful" claim dated before 2026-07-14 was made by a checker that could +not see an unapplied or under-applied plan. **Re-run the current checker before trusting fidelity** +(this is why the ledger flags a sandbox re-verify owed before C2). + +Two blind spots remain OPEN (logged, not fixed): SAME-DUMPER (both sides use the same field list, so +a field the dumper omits is invisible on both), and DUPLICATE-CIDR collapse (prefix-keyed dicts +collapse duplicates; latent -- none upstream today). "Faithful" means faithful within the dumped +field list, not absolutely. + +### 8.4 Write-guard architecture -- FIVE tools, and how each is kept off the apex + +"The sim never writes upstream" is enforced differently per tool. As of 2026-07-14 the three +write-capable importers ALL gate an upstream `--commit`, but by two different mechanisms: + +| Tool | Default | Upstream write-guard | +|---|---|---| +| `prod-draft-dump.py` | read-only; no `--commit` | N/A -- performs no writes | +| `sandbox-seed.py` | DRY; `--commit` writes | STRUCTURAL: holds no upstream URL/token and dies if pointed at the draft's own source host -- it CANNOT reach upstream | +| `d115-office-carve.py` | DRY; `--commit` writes | `SANDBOX_HOSTS` allowlist `{localhost,127.0.0.1,10.10.1.10}`; a non-sandbox `--commit` REFUSES without `--yes-write-upstream` | +| `roles-aggregates-import.py` | PREVIEW; `--commit` writes | **`SANDBOX_HOSTS` + `--yes-write-upstream` (added 2026-07-14)** -- plus a whole-plan PREFLIGHT so it cannot half-write | +| `dc-dc-prefixes-import.py` | DRY; `--commit` writes | **`SANDBOX_HOSTS` + `--yes-write-upstream` (added 2026-07-14)** -- plus a whole-plan role PREFLIGHT so it cannot half-write | + +Before 2026-07-14 the two pynetbox importers had NO target guard at all; the only thing stopping an +accidental upstream `--commit` was the WAF 403 (8.5) -- and fixing the WAF (below) removed that +accidental safety, which is exactly why the explicit `--yes-write-upstream` gate was added in the +same change (`docs/changelog-20260714-netbox-importers-waf-useragent.md`). The whole-plan preflights +are the OTHER layer: both importers had previously died mid-loop and left the production IPAM +half-populated (`docs/changelog-20260714-netbox-write-path-hardening.md`). + +### 8.5 The upstream WAF trap (FIXED 2026-07-14) + +Upstream NetBox sits behind a User-Agent-filtering WAF: `urllib`/`requests`/`pynetbox` get a 403 +that looks EXACTLY like an auth failure, while the SAME token works from `curl` (measured 2026-07-13, +re-measured 2026-07-14: `curl` -> 200, `urllib`/pynetbox -> 403; `references/platform-traps.md`). Do +NOT diagnose it as a token problem and re-mint credentials -- confirm with `curl` first. + +The three stdlib tools always sent an accepted UA (`curl/8.5.0`). The two pynetbox importers did NOT +until 2026-07-14 -- meaning they could NEVER have written to the apex, only to the WAF-less sandbox. +`get_nb()` in both now sets `nb.http_session.headers["User-Agent"] = "curl/8.5.0"` +(`docs/changelog-20260714-netbox-importers-waf-useragent.md`). NOTE: `pynetbox` is NOT installed on +`vcloud` -- the two pynetbox importers must run on `office1-netbox` (pynetbox 7.0.0), which reaches +upstream through the edge. + +### 8.6 C2 -- office1-netbox holds the validated dataset (apex write-back DEFERRED) + +**DOCFIX-195 (operator ruling 2026-07-15) REDEFINED C2.** C2 is NO LONGER "feed the carve upstream" -- +there is NO `netbox.baldurkeep.com` write in Stage 2 at all. C2 is met when `office1-netbox` (the VR1 +IPAM apex) holds the COMPLETE validated dataset AND it is VERIFIED. Two sub-tasks remain: + + (i) Re-run the HARDENED fidelity check against `office1-netbox` for the shared carve/roles/DCs -- + the "faithful" verdict predates the 2026-07-14 hardening (8.3), so it must be re-established. + (ii) Load the D-120 additions into `office1-netbox` -- `10.10.1.0/24`'s child ranges (`ip-ranges`: + static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments + (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11) -- and verify them. + +**TOOLING GAP (logged DOCFIX-195, NOT fixed here).** The D-120 objects are `ip-ranges`/`ip-addresses` +-- object classes the sandbox loop does NOT cover: `prod-draft-dump.py`'s ENDPOINTS and +`sandbox-fidelity-check.py`'s KEY/EXPECTED_NEW both stop at rirs/roles/regions/sites/aggregates/ +prefixes. So the hardened fidelity check CANNOT see the D-120 load. Verifying sub-task (ii) requires +EITHER extending both tools (add the two endpoints + KEY entries + EXPECTED_NEW bounds + delta field +checks, each with its harness) OR a recorded manual read-back against `office1-netbox`. That choice is +an open C2-execution sub-question, to be ruled when C2 runs. + +**The DEFERRED merge-back (end-of-deployment, MAYBE).** If/when the operator elects to seed +`netbox.baldurkeep.com` from the validated `office1-netbox` dataset, it is a production IPAM write and +NOT part of the sim: (a) run the two pynetbox importers from `office1-netbox` (pynetbox + apex reach); +(b) each needs `--yes-write-upstream` on top of `--commit` for the non-sandbox target; (c) the NetBox +write-path bugs (session-ledger "NetBox WRITE-PATH BUGS") gate THAT write, not Stage 2 close; (d) +present and individually approve each mutation -- never batched. As of 2026-07-15 the apex is untouched +and still holds only the v1 draft (measured read-only). diff --git a/docs/dc-dc-netem-and-ula-gua-proposal.md b/docs/dc-dc-netem-and-ula-gua-proposal.md index c5b4e0c..74b2820 100644 --- a/docs/dc-dc-netem-and-ula-gua-proposal.md +++ b/docs/dc-dc-netem-and-ula-gua-proposal.md @@ -69,7 +69,7 @@ actually does under the hood, not a shortcut invented here): ```bash -printf 'fd%s::/48\n' "$(openssl rand -hex 5 | sed 's/\(..\)\(..\)\(..\)\(..\)\(..\)/\1\2:\3\4\5/')" +printf 'fd%s::/48\n' "$(openssl rand -hex 5 | sed 's/\(..\)\(....\)\(....\)/\1:\2:\3/')" ``` Run this ONCE, by the operator, on a real machine -- the output is the @@ -78,6 +78,14 @@ not just as an environment variable -- this is a permanent organizational identifier, not a per-session value. +> **Correction (DOCFIX-183):** the `sed` above was fixed to split the 10 hex +> digits as 2+4+4 hextets (`fdXX:XXXX:XXXX::/48`); the prior 4+6 split produced +> INVALID IPv6 (a 6-hex-digit hextet -- caught when this command was actually +> run 2026-07-11). **The ratified VR1 value is `fd50:840e:74e2::/48`** (CSPRNG, +> collision-checked vs the Tailscale `/48` in NetBox -- see +> `docs/dc-dc-netbox-buildout-scope.md` section 4c); this command is the +> reference method, not a re-generation step for VR1. + ### 2.2 -- Per-DC GUA carve (from ARIN 2602:f3e2::/32 region-0 /36) This is NOT something to generate randomly -- D-101's own text names this diff --git a/docs/design-decisions.md b/docs/design-decisions.md index b7b868e..a08a471 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -193,6 +193,8 @@ **Decision:** NetBox is the single source of truth for IPAM at the **role and cloud-level pool** layer. Per-project tenant subnets are exempt under the hybrid model (D-016). +**Note (DOCFIX-195, operator ruling 2026-07-15 -- VR1 phase):** During VR1 `office1-netbox` (10.10.1.10) HOLDS THE IPAM APEX ROLE: it is the authoritative NetBox every consuming system (OpenTofu, MAAS, overlays, the Juju bundle) derives its values from. It is a *working draft* (edited as the buildout proceeds), but it is the apex. `netbox.baldurkeep.com` is the eventual PRODUCTION apex, held during VR1 as a READ-ONLY v1 reference draft that takes NO write and is NOT consulted by VR1 systems. Merging the validated `office1-netbox` dataset back into `netbox.baldurkeep.com` is DEFERRED to end-of-deployment (a maybe). This DEFERS the apex write; it does NOT revoke D-010 -- `netbox.baldurkeep.com` remains the single eventual source of truth. See `docs/dc-dc-netbox-buildout-scope.md` section 8 and close-out C2 in `docs/dc-dc-deployment-workflow.md`. + **Workflow:** Update NetBox → update bundle/overlay → commit both with cross-reference. **Standing imports for v1 (gating the bundle):** @@ -2016,7 +2018,7 @@ - OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt. - OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs. - MAAS owns commission / deploy / power / release of those node VMs; it does NOT compose new ones. Juju then deploys the OpenStack bundle onto the MAAS-deployed machines. -- NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse. +- NetBox remains the IPAM apex; MAAS and overlays consume NetBox values, never the reverse. **(DOCFIX-195, 2026-07-15: during VR1 that apex is `office1-netbox` (10.10.1.10) -- the working draft every consuming system reads; `netbox.baldurkeep.com` is the eventual production apex, frozen as the v1 reference draft with merge-back deferred. Defers the write, does not change this consume-not-reverse rule.)** **Rationale / tradeoffs:** this keeps the entire physical substrate in versioned OpenTofu (auditable, reproducible, and exactly the "operator scripts the deployment from Office1" model), while MAAS owns the provisioning lifecycle -- mirroring Roosevelt, where MAAS enlists hardware that already exists rather than creating it. "MAAS owns lifecycle" here means the provisioning lifecycle, not domain creation; that reconciles the two operator inputs. @@ -2035,6 +2037,7 @@ - Single-unit (not HA) for VR1, with the confirmed backup posture (juju 3.6 `create-backup` / `download-backup`). Controller backups scheduled and a restore drill run in Phase 5. - HA (3-unit) DEFERRED to Roosevelt, when bare-metal resources allow. Recorded as a known VR1 test-only posture, not a Roosevelt recommendation. - Office1 hosts NO OpenStack Juju controller (Office1 = MAAS-region + OpenTofu + NetBox + GitBucket only). + **[AMENDED by D-116, 2026-07-13: the GitBucket half is SUPERSEDED. No Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record. Office1 = MAAS-region + OpenTofu + NetBox + Tailscale.]** **Rationale / tradeoffs:** per-DC single controller is DR-honest (independence) and resource-frugal on the 256 vCPU / 1 TiB / 10 TiB host. A shared or Office1-hosted controller would couple both DCs' manageability -- a controller outage would take out both -- the opposite of what a DR drill should prove. Single-vs-HA is a deliberate test-only economy; backups are the mitigation and are drilled. Roosevelt scales to 3-unit HA per DC. @@ -2144,3 +2147,1112 @@ **Log-for-sweep (not actioned):** `ledger-scan` DOCFIX / BUNDLEFIX patterns use `[0-9]{3}` -- safe through 999, latent at the 1000 boundary (the same class as the 100-boundary bug already fixed for those series). --- + +## D-111: VR1 per-DC v6 subcarve aligned to the deployed NN mnemonic + +**Status:** ADOPTED 2026-07-11 (operator deferred to Code stream's recommendation; ratifies `docs/dc-dc-netbox-buildout-scope.md` sub-decision #1). + +**Decision:** +- `netbox/dc-dc-prefixes-import.py`'s per-plane v6 subcarve is aligned to the DEPLOYED VR0-DC0 / Willamette net-byte (NN) mnemonic, replacing the original contiguous-plane-index proposal (which was explicitly un-ratified). Each plane gets a `/60` container + `/64` active, mirroring the live template: + - provider-public (GUA, out of `DC_GUA_PREFIX`): `/60` + `/64` active at `:10`, API-VIP `/64` at `:11`. + - metal-admin + metal-internal (ULA) SHARE the metal `/60` at `:20` -- admin `/64` at `:20`, internal `/64` at `:21` -- mirroring how provider `:10` / VIP `:11` share the provider `/60` (the deployed model had a single `metal` plane; the D-052/D-053 six-plane split nests both legs as `/64`s in it). + - data-tenant `:30`, storage `:40`, replication `:50` (ULA): each its own `/60` + `/64`. +- ULA legs are carved from a per-DC `/56` of the org ULA `/48`, indexed to the GUA site nibble (dc1 -> `...:02xx`, dc2 -> `...:03xx`) so the 4th hextet reads `DC.NN` in BOTH families. +- Addresses are computed by direct index arithmetic, NEVER `list(...subnets())` enumeration (preserves the DOCFIX-181 no-hang property). + +**Rationale:** the operator's standing directive is to keep VR1 as close to the real deployment as possible. The live NetBox export (2026-07-11) shows VR0-DC0 / Willamette use exactly this NN mnemonic + `/60`-per-plane structure; aligning the tool means VR1's GUA and ULA both read `:20`=metal, `:30`=data, etc., consistent with the deployed clouds, instead of arbitrary contiguous indices. + +**Out of scope (mirrored separately, not via this six-plane tool):** the `lbaas-mgmt` (`:80`), `vpn` (`:e0`), `oob` (`:f0`) networks from the deployed template. The G5 DC-site rename (`vr1-dc0`/`vr1-dc1` -> `vr1-dc1`/`vr1-dc2`) is a distinct ruling, recorded under its own D-number when executed. + +**References:** `netbox/dc-dc-prefixes-import.py` (`carve_v6`), `docs/dc-dc-netbox-buildout-scope.md` section 5.1, D-101 (family matrix), D-052/D-053 (six-plane), the 2026-07-11 live NetBox export. Harness `tests/dc-dc-prefixes-import/` 53/53. Applies to the Office1 simulation NetBox first; flows back to `netbox.baldurkeep.com` as refined draft. + +--- + +## D-112: OPNsense edge config delivery -- the Configuration Importer cannot work on a nano image + +**Status:** ADOPTED 2026-07-12 -- **option (c), post-boot network provisioning.** Operator ruling: +"C since it is the way people automate opnsense" -- prefer the mainstream, industry-standard +automation path (the OPNsense REST API, as used by `opn-cli` and the Ansible OPNsense collection) +over a bespoke image-bake or media-import trick. Consistent with the standing "transferable answers +beat quick fixes" constraint: a real Roosevelt edge is provisioned over the network too. + +IMPLEMENTATION NOT STARTED. The BOOTSTRAP sub-problem below is unsolved and is itself unruled -- +"push the config over the API" is not reachable from a factory-default OPNsense without first +solving how we authenticate and how we reach it. Do not start building until B1/B2 are ruled. + +**Bootstrap problem (measured/sourced, must be solved BEFORE any config push):** +1. **No route to it.** Factory-default OPNsense serves LAN on `192.168.1.1/24` on `office1-local`. + `office1-local` is an ISOLATED libvirt network -- the vcloud host has NO leg on `virbr2` + (measured: no IPv4 on the bridge). So nothing can currently talk to the edge's LAN at all. + The WAN side is reachable at the DHCP lease `172.30.1.126`, but OPNsense **blocks all inbound on + WAN by default** -- so the WAN leg is NOT a way in. +2. **No credentialed API.** The OPNsense REST API authenticates with an **API key + secret**, not the + root password. Keys are minted in the GUI -- a chicken-and-egg for a fresh box. SSH is + **DISABLED** in the factory default, so `ssh` is not an entry point either. +3. **Our serial console is write-only.** `modules/opnsense-edge` attaches `<serial type='file'>` -- + a one-way capture. It CANNOT be typed into, so the console is not currently an entry point + either. Making it interactive means `type='pty'` + `virsh console` (a module change + recreate). + +**Unruled sub-choices (operator, next session):** +- **B1 -- how do we REACH it?** (i) temporarily add a host leg on `virbr2` + (`ip addr add 192.168.1.10/24`) -- reversible, non-persistent, cheapest; or (ii) give + `office1-local` a permanent host address in the OpenTofu network definition (touches D-100's + network design); or (iii) drive it entirely over an interactive serial console and never need IP + reachability for bootstrap. +- **B2 -- how do we AUTHENTICATE the first time?** (i) interactive serial console (`type='pty'`) -> + root login -> enable SSH + install the ed25519 pubkey we already generated in + `~/vr1-office1-creds/`, after which everything is plain SSH/SCP and fully scriptable; or + (ii) authenticated GUI session with the default root credentials (curl + CSRF token) to mint an + API key, then use the REST API proper. + NOTE: (i) is console-driven but is NOT the rejected option (d) -- (d) was rejected for racing the + importer's 7-second timeout. A console *login shell* has no timing race and can be driven at + leisure. These are materially different risks; do not conflate them. + +**Option (a) RE-EXAMINED 2026-07-12 at operator request and MEASURED INFEASIBLE on this host.** +Recorded so it is not re-litigated. Baking `/conf/config.xml` into the nano requires writing +FreeBSD UFS2 from Linux, and every route is closed here: + +| Requirement | Measured result | +|---|---| +| Kernel UFS write | `# CONFIG_UFS_FS_WRITE is not set` -- driver is READ-ONLY (`CONFIG_UFS_FS=m`) | +| A FAT partition to write instead | NONE -- MBR (`dos`), a single FreeBSD `a5` partition; UFS only | +| FUSE UFS driver (`fuse-ufs`, `ufs2-tools`) | not present, not packaged | +| libguestfs (`guestfish` / `virt-copy-in`) | not installed | + +Two ways (a) could still be FORCED, both REJECTED: +- **FreeBSD helper VM** (boot a FreeBSD live system with the nano as a second disk; FreeBSD mounts + UFS rw natively; drop the file; power off). It genuinely works and `xorriso` IS present, so a live + ISO could be remastered with an auto-run script -- but it means fetching + remastering a FreeBSD + ISO and maintaining a builder VM forever to deliver one 4.5 KB file. Cost exceeds (c) entirely. +- **Raw byte-patch of the image** (pad the new XML to the default `config.xml`'s exact byte length + and overwrite its data blocks in place, leaving the inode untouched). Probably works -- and + REJECTED anyway: it silently corrupts if the file is not block-contiguous, and it is + NON-TRANSFERABLE (you cannot byte-patch a real appliance's disk on Roosevelt hardware), which is + precisely what this repo's governing constraint forbids. + +**Net: examining (a) properly STRENGTHENED (c).** (a) is not merely less mainstream here -- it is +unavailable without a FreeBSD-shaped detour costing more than (c)'s whole bootstrap. + +**BOOTSTRAP RULED 2026-07-12 (operator: "proceed with the C bootstrap"):** +- **B1 (reach) = (i)** temporary host leg on `virbr2` (`ip addr add 192.168.1.10/24`) -- reversible, + non-persistent, no change to the D-100 network definition. Removed/re-addressed once the edge's + LAN moves to `10.10.0.1`. +- **B2 (authenticate) = (i)** one interactive console login -> enable SSH + install the ed25519 + pubkey ALREADY generated in `~/vr1-office1-creds/`. Thereafter everything is plain SSH/SCP and + fully scriptable. Chosen over GUI/CSRF-scraping to mint an API key (brittle, no gain). + +**Once bootstrapped, the steady-state mechanism is:** push `config.xml` (or the equivalent API +calls) to the edge, reload, and verify. The config we already render (`opnsense-config.xml.tmpl`, +LAN `10.10.0.1/24` + DHCP `.100-.199`, WAN static `172.30.1.2`) is REUSED unchanged -- only the +DELIVERY changes. + +**Consequently RETIRED (do not delete until (c) actually works):** the config-ISO delivery path -- +`scripts/opnsense-build-config-iso.sh` + its harness, and the module's `config_seed` volume + cdrom +disk. They are inert (the Importer never reads them), not harmful. Remove them in the same change +that lands (c), not before. + +**Finding (MEASURED, not inferred).** The Office1 edge boots (DOCFIX-188 memory + DOCFIX-190 ACPI) +but comes up on FACTORY DEFAULTS. The config ISO is well-formed and correctly attached -- verified: +ISO9660 labelled `OPNSENSE_CFG`, containing `CONF/CONFIG.XML` with our `<opnsense>` payload and +`10.10.0.1`; the guest sees it as `cd0` (`QEMU QEMU DVD-ROM`, 186x2048 = exactly our 380928-byte +image). The boot log shows the importer being invoked and then doing NOTHING: + + >>> Invoking import script 'importer' + (no further output) + +**Root cause -- architectural, not a bug.** `rc.syshook.d/import/20-importer` runs +`/usr/local/sbin/opnsense-importer -b`. Upstream source (opnsense/core, `src/sbin/opnsense-importer`): + + INSTALL="/.probe.for.readonly" + ... + if [ -n "${DO_BOOT}" ]; then + touch ${INSTALL} 2> /dev/null + if [ -f ${INSTALL} -a -f /conf/config.xml ]; then + bootstrap_and_exit 0 + fi + +The `touch` is a PROBE FOR A READ-ONLY ROOT. On INSTALLER media the root is read-only, the touch +fails, the marker is absent -- so the importer proceeds to scan attached media (and yes, its scan +path does support cd9660). On a PRE-INSTALLED image (our nano) the root is WRITABLE and a factory +`/conf/config.xml` ALREADY EXISTS, so BOTH conditions hold and it `bootstrap_and_exit 0`s +immediately -- it never enumerates a single device. + +**Therefore the Configuration Importer can NEVER fire on a pre-installed nano image, by design.** +Nothing was ever going to read our ISO. + +**What the prior research got right and wrong** (`opentofu/README.md` "OPNsense deployment +research", items 2-3): it correctly established that the Importer exists and that its scan path +supports ISO9660 for VM automation. It never established that the Importer would RUN AT ALL on a +nano image -- and it does not. The module header has always flagged this exact mechanism UNVERIFIED; +that flag was correct and is now discharged NEGATIVE. + +**Options (operator to rule; none implemented):** + +- **(a) Bake `config.xml` into the image before first boot.** `opnsense-prep-image.sh` writes + `/conf/config.xml` directly into the nano's UFS root. Cleanest and most immutable-infra; no + importer, no ISO, no boot-time interactivity. RISK/UNVERIFIED: writing FreeBSD UFS2 from Linux -- + most distro kernels ship `CONFIG_UFS_FS_WRITE` off, so this may be infeasible without libguestfs + or a FreeBSD helper. FEASIBILITY MUST BE PROVEN BEFORE ADOPTING. +- **(b) Switch from the nano image to the installer image.** On installer media the root IS + read-only, so the Importer runs and the existing ISO mechanism works AS DESIGNED and unchanged. + Cost: an actual install step per edge; needs checking whether it is non-interactive. +- **(c) Boot factory-default, then push config over the network.** Provision post-boot via + OPNsense's API/SSH from Office1 (factory default LAN is `192.168.1.1/24` on `office1-local`, which + we own). Most Roosevelt-transferable -- real edges get provisioned over the network too -- but it + is two-phase and needs a bootstrap path onto the default LAN. +- **(d) Serial-console expect-scripting.** Drive the interactive importer prompt over the serial + console. NOT RECOMMENDED: fragile, timing-dependent (the prompt has a 7-second timeout). + +**Impacts if not (b):** `scripts/opnsense-build-config-iso.sh`, its harness, the module's +`config_seed` volume + cdrom disk, and the README research section all assume the ISO mechanism. + +**References:** `docs/incident-20260712-opnsense-edge-boot-triplefault.md`, +`docs/changelog-20260712-libvirt-acpi-kernel-panic.md` (DOCFIX-190), upstream +`opnsense/core:src/sbin/opnsense-importer` + `src/etc/rc.syshook.d/import/20-importer`. +Also gates Stage 3 (the per-DC OPNsense edges use this same module). + +--- + +## D-113: is OPNsense the right edge platform, or should the simulated edges be VyOS/Linux? + +**Status:** **ADOPTED 2026-07-13 -- option (a2): STAY ON OPNSENSE, MOVE CONFIG TO THE REST API.** +Operator ruling ("go ahead and continue with your rest API recommendation"), after two +constraints were surfaced during the discussion that narrowed the field decisively: + +1. **GUI access is a REQUIREMENT** (operator, 2026-07-13). This ELIMINATES option (b) VyOS and + option (c) plain-Linux outright -- neither has a GUI. Both had been the leading candidates + until this was stated; recording it so the reasoning is not re-run from the wrong premise. +2. **VyOS LTS binaries are subscription-gated** (verified 2026-07-13, vyos.net/get + the + Software Access Subscription page). Free VyOS means the ROLLING release -- a moving target + that fights this repo's appendix-B version-pinning discipline -- or building from source. + Independent of the GUI point, this was real friction. + +Remaining open-source field was OPNsense / pfSense CE / OpenWrt / IPFire. pfSense REJECTED +(same GUI-owned XML lineage -- a migration that buys nothing). IPFire: weakest automation +story. **OpenWrt was the credible alternative** -- its UCI config is text-first BY DESIGN +(LuCI and the CLI read/write the same `/etc/config/*` files, so editing config as text is a +supported path, structurally unlike OPNsense's GUI-owned `config.xml`) -- but it costs a new +image pipeline, a fresh bootstrap problem (it also defaults to LAN `192.168.1.1`), and dnsmasq +instead of Kea, all to escape a problem fixable in place. + +**The ruling rests on this:** the bug class was never "OPNsense". It was hand-authoring +`config.xml`, a GUI-owned format, when OPNsense ships a documented REST API with mature Ansible +collections modelling exactly what we got wrong (`dhcp_subnet`, firewall rules, system settings +as typed first-class resources). **None of DOCFIX-191/192/193 is expressible through the API** -- +you cannot forget to enable sshd in a format where sshd is a typed field with a default. (a2) +keeps the GUI, keeps a WORKING routing+DHCP edge, keeps the two libvirt fixes, and removes the +defect source -- and it is the smallest change of anything considered. + +**AMENDMENT 2026-07-13 -- the template is DELETED, not reduced (operator ruling: "I'll defer +to your lean").** The original (a2) plan was to REDUCE `opnsense-config.xml.tmpl` to a minimal +bootstrap (sshd + root key + console + a seeded API key). That turned out to be unnecessary: +every one of those is covered without any `config.xml` at all. + +- sshd + root key: the **D-112(c) console bootstrap** installs them (proven on Office1). +- the API key: **`scripts/opnsense-bootstrap-apikey.sh`** mints one by calling OPNsense's OWN + model (`Auth\User -> apikeys->add()`) over SSH -- no GUI click, and no re-implementation of the + vendor's `$6$` crypt format (which would fail SILENTLY on any format drift). +- everything else -- DHCP, firewall, interfaces: **the REST API**. + +So the provisioning chain has NO `config.xml` in it, and a renderer that nobody should ever run is +not a safety net -- it is a loaded gun pointed at a live router. The 2026-07-13 safety sweep is the +evidence: the repo still contained runbook steps instructing an operator to render a config and push +it to the edge, which by then would have CLOBBERED live API-managed DHCP. + +**DELETED:** `opentofu/templates/opnsense-config.xml.tmpl`, `scripts/opnsense-render-config.sh`, +`scripts/opnsense-build-config-iso.sh`, and both their harnesses. The `opnsense-edge` module's +`config_seed` volume + cdrom disk are removed (the ISO was INERT -- D-112). The +`xorriso`/`genisoimage` prereq is dropped. All remain in git history. + +**If you think the API cannot express something:** that is a finding to raise against D-113 -- not +a reason to hand-write XML. + +**Superseded by this ruling:** the "stay on (a1)" lean in the original recommendation below. +(a1) is now explicitly REJECTED: its cost compounds per feature and it depends on an +undocumented self-heal of an internal format (see the 667-element finding). + +**Scope.** The VR1 edges are SIMULATED customer-site / ISP routers (Office1 today; Stage 3 adds +one per DC). They are lab infrastructure, not a product surface. Nothing a tenant touches +depends on this choice; what depends on it is our own automation cost, for three edges. + +### First, an honest cost accounting of 2026-07-12 (five bugs) + +| # | bug | OPNsense's fault? | +|---|---|---| +| DOCFIX-188 | guest got 2 MiB RAM (`memory_unit`) -- triple fault | **NO** -- libvirt module bug; would hit ANY guest | +| DOCFIX-190 | `acpi=off` -> FreeBSD kernel panic | **NO** -- libvirt module bug; generic | +| DOCFIX-191 | rendered config had no sshd + no key -> would have locked management out | yes (hand-authored `config.xml`) | +| DOCFIX-192 | rendered config silenced the only console | yes (hand-authored `config.xml`) | +| DOCFIX-193 | LAN DHCP was never implemented (and ISC `<dhcpd>` would have been inert -- 26.1 is Kea) | yes (hand-authored `config.xml`) | + +Two of five were platform-agnostic libvirt bugs whose fixes now protect every VR1 VM +(MAAS/NetBox/GitBucket, the DC nodes). Finding them via OPNsense was lucky, not costly. + +**The other three share ONE root cause: we hand-author the appliance's internal, GUI-owned +`config.xml`.** That is a known-bad idea in any product of this class -- and pfSense would be +identical, since it is the same format lineage. + +**2026-07-13 adds a fourth data point of the same kind.** A full-config push REPLACES +`/conf/config.xml`; the rendered template is 128 elements against a live 796, so it DROPS ~667 +migration-populated elements -- including the only two firewall pass rules on the box +("Default allow LAN to any"), against a pf base policy of block-drop-all. It works only because +OPNsense regenerates them on boot (proven across 4 push+boot cycles, `/conf/backup/` trail). +**We are depending on an undocumented self-heal of an internal format.** That is the ongoing +cost, and it is the thing to weigh -- not the boot bugs. + +### THE CONFOUND (must be resolved before ruling) + +**D-112 ADOPTED option (c) with the explicit rationale "C since it is the way people automate +opnsense" -- i.e. the OPNsense REST API (`opn-cli`, the Ansible OPNsense collection).** + +**What we actually built is NOT that.** We render a full `config.xml` from a Jinja-ish template +and scp it over the wire. That is a config-file bake delivered post-boot -- it satisfies the +letter of (c) (provisioning happens over the network, after boot) but NOT the rationale the +ruling was made on. + +**Every OPNsense-specific bug we are now charging against the platform (DOCFIX-191/192/193, and +the 667-element drop) is an artifact of hand-authoring XML -- exactly what the REST API exists +to avoid.** The API models DHCP, sshd, console, and firewall rules as first-class resources; +none of those three bugs is expressible through it. + +So the fair comparison is not "OPNsense vs VyOS". It is: + +- **(a1) OPNsense + `config.xml` templating** -- the status quo. Cheapest right now; every new + feature (VPN, HA, VLANs) means reverse-engineering another slice of an internal format, and + the self-heal dependency above is permanent. +- **(a2) OPNsense + the REST API** -- what D-112's rationale actually called for. Config becomes + declarative API calls; the internal format stops being our problem. Cost: retire the template, + build/adopt an API client (`opn-cli` is packaged and mainstream), and keep only a MINIMAL + bootstrap config (sshd + key + console) as a file. The boot bugs stay fixed either way. +- **(b) VyOS** -- purpose-built for this. Plain declarative text config, first-class cloud-init, + a real config API; it exists to be a lab/edge router. Friction: LTS images are + subscription-gated (rolling is free), it is less "appliance-like" than a real site router, and + switching discards a WORKING, routing, DHCP-serving edge to re-run discovery on a new platform. +- **(c) Plain Linux router** (Debian/Ubuntu + nftables + Kea + FRR, provisioned by cloud-init). + Maximum fidelity to the rest of our tooling -- same image pipeline, same cloud-init, same + Ansible/OpenTofu idioms as every other VR1 VM, and zero appliance-internal formats. Friction: + we hand-build router behaviour that OPNsense/VyOS ship; least faithful to "a real site runs an + appliance". +- **(d) pfSense** -- **REJECT.** Same config-format lineage as OPNsense; buys nothing, costs a + migration. + +### Recommendation (mine; the operator rules) + +**Stay on OPNsense -- but the real question is (a1) vs (a2), and I lean (a2).** + +Rationale for staying: the hard part is DONE and MEASURED WORKING (boots, routes, NATs, reaches +the internet, serves DHCP, managed over SSH with a verified push pipeline). Edges 2 and 3 reuse +the module and template -- different tokens, same everything -- so the cost paid was one-time, +not per-edge. Switching platforms discards that and re-runs the same class of discovery +elsewhere. VyOS is the defensible blank-sheet pick, but we are not on a blank sheet. + +Rationale for (a2) over (a1): (a1)'s cost COMPOUNDS. Each new feature is another slice of +reverse-engineered XML plus a self-heal we do not control. (a2) pays a bounded, one-time cost to +get onto the path D-112's ruling actually named, and it is the honest reading of that ruling. + +**What would change my mind entirely (i.e. rule (b)/(c)):** if, having moved to (a2), the API +still forces us to reverse-engineer the config format for each new feature -- then OPNsense is +not buying us the appliance abstraction we are paying for, and VyOS's plain-text config wins. + +### Cost of deferring + +Low but not zero: Stage 3 builds two more edges from this same module. Ruling BEFORE Stage 3 +means the choice is made once; ruling after means migrating three edges instead of one. + +**Evidence:** `docs/changelog-20260712-office1-opnsense-edge-build.md`, +`docs/changelog-20260712-opnsense-edge-boot-fixes.md`, +`docs/changelog-20260713-office1-dhcp-apply.md` (the 667-element self-heal, measured), +D-112 (the (c) ruling and its stated rationale). + +### D-113 -- AMENDMENT (2026-07-14): "interfaces: the REST API" IS FALSE. Base-interface addressing has NO API. + +A factual correction to this decision's own text, not a re-ruling. The (a2) rationale above ends +with: *"everything else -- DHCP, firewall, **interfaces**: the REST API."* **The `interfaces` half +of that claim is wrong**, and it was never measured before being written -- it was inferred from +DHCP and firewall being API-native. + +**MEASURED on the live Office1 edge (2026-07-14), OPNsense 26.1:** + +| Surface | Served by | API? | +|---|---|---| +| Interfaces > [LAN] / [WAN] -- **address, incl. `ipaddrv6`** | `/interfaces.php?if=lan` | **NO -- legacy PHP page, config.xml-backed** | +| Interfaces > Devices (bridge/gif/gre/lagg/loopback/vlan/vxlan) | `/ui/interfaces/*` | yes | +| Interfaces > Virtual IPs | `/api/interfaces/vip_settings/*` | yes (`ipalias`/`carp`/`proxyarp`; iface `wan`/`lan`/`lo0`) | +| Services > Router Advertisements (radvd) | `/api/radvd/settings/*` | yes (`getEntry`/`addEntry`/`searchEntry`) | +| Services > Kea DHCPv4 / DHCPv6 | `/api/kea/*` | yes | + +Only the **Devices** and **Neighbors** sub-trees were migrated to MVC. A base interface's IP +addressing was not. So there is no API call that sets LAN's `ipaddrv6`. + +**What this does and does NOT change:** + +- **It does NOT reopen (a2).** The prohibition D-113 actually adopted is on **hand-authored + `config.xml` and full rendered pushes** -- the clobber path that would wipe the API-managed DHCP. + D-113 also records **"GUI access is a REQUIREMENT"** as the constraint that eliminated VyOS and + plain-Linux. A one-time operator GUI edit is therefore the vendor interface this decision + deliberately PRESERVED. It is not a violation and must not be described as one. +- **It DOES mean the provisioning chain is not 100% API-scriptable** for any surface that needs a + base-interface address -- which is every edge that gets IPv6, i.e. Office1 now and the DC0/DC1 + edges at Stage 3. That is a per-edge manual step unless the VIP path below works. + +**The open question (an EXPERIMENT, not a plan):** the `radvd` entry model has **no prefix field** +-- it derives the advertised prefix from the interface's configured IPv6. Whether an **`ipalias` +VIP** counts as "the interface's configured IPv6" for OPNsense's radvd config generator, or whether +the generator only reads the primary `ipaddrv6`, is **UNKNOWN and must not be assumed**. If the VIP +path works, the whole IPv6 edge build is API-native and reproducible per DC. If it does not, the +LAN GUA is an operator GUI step at every edge. This is cheap and reversible to test on the lab edge +and is gated on an operator ruling. + +--- + +## D-114: VR1 site containment VMs + MAAS-composed LXD VMs for the non-stack machines + +**Status:** **ADOPTED 2026-07-13 (operator ruling).** AMENDS D-103 (which assigned Office1 three +sibling service VMs and put every VR1 domain directly on the vcloud host). Supersedes nothing else. + +**Context.** Stage 2 was about to build Office1 per D-103: three peer VMs (MAAS-region, NetBox, +GitBucket) created directly on vcloud libvirt, with Tailscale installed on the headend. The +operator described a different shape -- each site (Office1, DC1, DC2) living inside its OWN VM +environment, with MAAS running LXD to hold "the rest of the servers/services." An initial reading +of `bundle.yaml` found only Juju's `lxd:8`-style CONTAINER placements and concluded there was no +precedent for MAAS composing LXD VMs. **That conclusion was WRONG**, and is recorded here because +the error is instructive: the bundle's Juju LXD containers and MAAS's LXD VM host are TWO DIFFERENT +LXD usages in the same cloud, and reading only the bundle collapses them into one. + +**Measured as-built evidence (VR0 / `openstack-caracal-ipv4` MAAS 3.7.2, operator screenshots +2026-07-13).** Seven machines, all Deployed, ALL powered via Virsh: + +- `openstack0-3` -- 8 cores / 32 GiB / 1.1 TB, 6 spaces, tags `openstack, virtual`. Juju machines + 8-11; the OpenStack API services run in JUJU-created LXD CONTAINERS on these (21 `lxd:N` + placements in `bundle.yaml`). MAAS never sees those containers, and never needed to. +- `juju` -- 2 cores / 4 GiB, tags `juju, virtual`. The controller. +- `lxd` -- 8 cores / 16 GiB / 687 GB, tags `lxd, virtual`. A MAAS-DEPLOYED, virsh-powered libvirt + VM that is then REGISTERED BACK INTO MAAS as an LXD KVM host (project `default`, LXD **5.21.4**; + host NICs enp1s0 / enp7s0 / lxdbr0). +- `tailscale` -- 2 cores / 2 GiB / 25 GB, Ubuntu 24.04, tags `pod-console-logging, virtual`. + **It is a VM COMPOSED BY MAAS INSIDE the `lxd` KVM host.** This is the proof of the pattern. + +So the pattern the operator asked for is not new -- it is VR0's own, already proven in production +on this cloud, and it costs 2 cores / 2 GiB per service machine. + +### Ruling + +1. **Non-stack machines become MAAS-composed LXD VMs.** Per site, one MAAS-deployed machine runs + LXD and is registered back into MAAS as an LXD KVM host; the machines that live OUTSIDE the + OpenStack stack (NetBox, GitBucket, Tailscale, and any future service box) are COMPOSED into it + by MAAS. They are therefore MAAS-visible: enlisted, commissioned, deployed, powered, released. + This is what the operator meant by visibility, and it is `lxd` + `tailscale` reproduced per site. + **Explicitly NOT in scope:** the Juju-deployed OpenStack services on the DC nodes. Those stay in + Juju-created LXD CONTAINERS (bundle `lxd:N` placements), invisible to MAAS, exactly as in VR0. + Operator: "I don't care about visibility into the juju deployments into the openstack0-3 + machines as much as the other machines running outside of the stack." + +2. **Each site gets a containment VM** -- `voffice1`, `vdc1`, `vdc2` -- simulating the facility that + the site's servers and services operate within. Rationale (operator): **full-facility snapshots** + and **the ability to simulate total geographical failure of a facility.** The second is the + decisive one: D-108's DR failover drill currently has NO honest "hard-down an entire DC" + primitive, and `virsh destroy vdc1` IS that primitive. This is a capability the flat model + cannot offer at any price. + +3. **Sequencing (operator ruling):** Office1 FIRST, fully up and running, on this model. DC1 is + **GATED** behind it and is not started until Office1 is complete. DC2 follows DC1. + +4. **LXD is version-pinned to the 5.21 LTS track.** MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7: + LXD 6.7 consolidated API endpoints and MAAS's pinned pylxd 2.3.5 cannot speak to them (fixed in + pylxd >= 2.3.9, not yet in a MAAS release). Canonical's own guidance is "use LXD <= 6.6 or the + 5.21 LTS release until further notice." VR0 is already on 5.21.4, i.e. on the safe side by + accident of timing rather than by decision -- this makes it a decision. Goes in appendix-B. + Source: https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 + +### What this changes in D-103 + +D-103 said OpenTofu owns creation of "the Office1 service VMs (MAAS-region, NetBox, GitBucket)" as +domains on the vcloud host. Under D-114, OpenTofu owns creation of the SITE CONTAINMENT VMs and the +site's networks; INSIDE a site, the service machines are composed by MAAS into LXD. D-103's core +principle is UNCHANGED and is in fact extended: MAAS still owns the provisioning lifecycle, and +still does not compose the OpenStack NODE VMs (`modules/node-vm` pre-creates those; `maas_vm_host` +registers the virsh host so MAAS DISCOVERS them -- `maas_vm_host_machine` remains ruled out for +nodes). The composition right granted here is scoped to the LXD VM host and the non-stack machines. + +### Known risk, and its entry gate + +**Nesting depth.** `vcloud` is ITSELF a KVM guest (measured: `systemd-detect-virt` -> `kvm`). A site +containment VM therefore puts the site's servers one level deeper than VR0 runs them: DC node VMs +land at L3 and OpenStack TENANT INSTANCES at L4. VR0 proves nesting works to the depth IT uses +(`tailscale` inside `lxd`), but VR0's hosts sit beside vcloud on the same 10.17.11.0/22 lab network +and are thus one level SHALLOWER than a VR1 pod. L4 tenant instances are UNPROVEN, and nova-compute +needs working KVM at the bottom of that stack. The 2026-07-12 OPNsense BTX triple-fault is prior +evidence that this environment is already fragile at depth. + +**Office1 IS the nesting probe -- and that is a feature of the sequencing, not a coincidence.** +(An earlier draft of this entry claimed "Office1 needs no nested KVM." That was WRONG and is +corrected here: LXD **virtual machines** are qemu/KVM guests, so composing NetBox/GitBucket/ +Tailscale as LXD VMs inside `voffice1` requires nested KVM at exactly the same depth the DC nodes +would need it -- L3. Only LXD *containers* would be free, and MAAS does not enlist containers.) +Office1 is therefore the CHEAP place to prove the model: no OpenStack, no Ceph, small VMs. If an +LXD VM boots inside `voffice1`, the pod model is proven to L3 and DC1 inherits that proof. + +**Entry gate for DC1:** (a) Office1 fully up on this model (operator ruling: DC1 is gated behind +it); (b) inside `vdc1`, confirm nested KVM is exposed and a guest actually BOOTS -- which also +proves L4 is reachable for tenant instances, the one depth VR0 has never exercised. If L3 fails at +Office1, the containment-VM model is abandoned early having cost one VM. If L3 holds but L4 fails +at DC1, DC compute stays flat and Office1 keeps its containment VM. Both outcomes are scoped +retreats, not redesigns -- the ruling is deliberately written to make them cheap. + +**DHCP.** OPNsense/Kea is authoritative on `office1-local` (10.10.0.0/24, pool .100-.199). LXD's +`lxdbr0` runs its OWN dnsmasq -- that is FINE as long as it stays on LXD's internal NAT bridge and +is never bridged onto `office1-local`. Two DHCP servers on one L2 is an intermittent failure that +is genuinely unpleasant to diagnose. Any LXD networking change must not put `lxdbr0` on the site LAN. +Corollary bonus: `voffice1` booting on `office1-local` will take the FIRST REAL DHCP LEASE from Kea +-- a path proven only at the daemon level so far, never end to end. + +### Delta to Roosevelt + +The containment VM is simulation-only machinery: in Roosevelt a "facility" is a building full of +real servers, not a VM, and it is torn down by pulling power, not `virsh destroy`. That is an +ACCEPTED delta, bought deliberately: it is what makes a geographic-failure drill executable at all +in a virtual rehearsal. The LXD half carries NO delta -- it is exactly what VR0 runs today and what +Roosevelt would run on real hardware (a service host running LXD, registered to MAAS). + +> **D-119 NAMING AMENDMENT (2026-07-14).** The DC containment VMs named `vdc1`/`vdc2` above are +> **RENAMED `vvr1-dc0` / `vvr1-dc1`** to match the region-qualified DC namespace. Neither is built +> yet (only `voffice1` exists), so this costs nothing -- and it removes a genuinely dangerous +> ambiguity: this decision's own DR primitive is **`virsh destroy vdc1`**, which under the old +> naming would read as "destroy VR1 DC1" while actually meaning "destroy VR1 DC0". A mislabelled +> destroy command in a DR drill is not cosmetic. `voffice1` KEEPS its name (the office keeps its +> number, per D-119). The `vdc1`/`vdc2` spellings in the body above are HISTORICAL. + +## D-115: VR1 Office1 addressing -- the office carve, and the v4 role gaps it exposed + +**Status:** **ADOPTED 2026-07-13 (operator ruling -- see the amendment at the end of this entry).** Originally PROPOSED/OPEN the same day. Found during Office1 deployment +testing: the headend was built on address space that NetBox never assigned. Per the operator, the +IPAM carve matters MORE than the deployment already standing on it -- "we can always change the +current IP addressing on the deployment; getting the IPAM carve proper and correct is more +important than going back and correcting deployment work." Nothing here is blocked; Office1 runs. +This is about ratifying or replacing what it runs on, BEFORE NetBox/GitBucket land on it. + +### What the IPAM apex actually encodes (MEASURED 2026-07-13 against netbox.baldurkeep.com) + +**IPv6 is a complete, consistent, region-hierarchical model.** `2602:f3e2::/36` (ARIN, Abysius Inc). +Every REGION gets a `/40`; inside it, SITES are `/48`s on a FIXED slot pattern: + +| slot | meaning | VR0 (`e00::/40`) | VR1 (`f00::/40`) | Eugene (`100::/40`) | +|---|---|---|---|---| +| `X00::/48` | Infrastructure / site-to-site links | `e00::` | `f00::` | `100::` | +| `X01::/48` | **Offices** | `e01::` | **`f01::`** | `101::` | +| `X02::/48` | Datacenter (first) | `e02::` (VR0 DC0) | `f02::` | `102::` Willamette | +| `X03::/48` | Datacenter (second) | `e03::` (VR0 DC1) | `f03::` | `103::` Roosevelt | + +Inside an office `/48`: a reserved `/56`, then ONE `/56` PER OFFICE, then a `/64` office subnet. +VR0 Off0 = `e01:100::/56` -> `e01:100::/64`. Eugene's Charnelton = `101:100::/56`. Inside a DC `/48`: +the D-111 NN mnemonic (`:10` provider, `:11` provider API VIP, `:20` metal, `:30` data, `:40` +storage, `:50` replication, `:80` lbaas-mgmt, `:e0` VPN, `:f0` OOB), each `/60` + `/64`. +Also global: `2602:f3e2::/40` inter-regional infrastructure, `2602:f3e2:fe::/48` **Edge Networks**, +`2602:f3e2:ff::/48` OpenStack tenant reservation. + +**IPv6 for Office1 therefore has NO design freedom** -- the pattern determines it: +`2602:f3e2:f01:100::/56`, office subnet `2602:f3e2:f01:100::/64`. VR1's office `/48` already exists. + +**IPv4 is a much thinner, ROLE-based model** -- and this is where the gaps are: +`10.0.0.0/8` "Corp Private" has exactly three children: `10.12.0.0/16` **Cloud**, `10.16.0.0/16` +Datacenter (Willamette dev), `10.17.0.0/16` Datacenter (Roosevelt dev -- the physical lab net that +vcloud and the VR0 jumphost actually sit on). Plus `172.16.0.0/12` "Corp OOB Private". + +### The gaps (all four are real; none was known before today) + +1. **There is NO Office role in v4.** Office1's LAN `10.10.0.0/24` (chosen during the OPNsense edge + build, 2026-07-12) and its LXD compose network `10.10.1.0/24` (chosen 2026-07-13) are SQUAT -- + unallocated, unregistered, colliding with nothing but blessed by nothing. +2. **DC2's planned `10.13.0.0/19` is OUTSIDE the Cloud `/16`.** `docs/dc-dc-netbox-buildout-scope.md` + recommends it, but NetBox's Cloud role is `10.12.0.0/16`; `10.13.x` is unallocated. DC2 was + already set to squat, and nobody had noticed. +3. **`office1-wan` (`172.30.1.0/24`) squats INSIDE `172.16.0.0/12` "Corp OOB Private"** -- and it is + not OOB, it is a SIMULATED ISP UPLINK. That is a wrong-role problem, not merely an unregistered one. +4. **"Edge Networks" has a v6 `/48` (`2602:f3e2:fe::/48`) but NO v4 counterpart**, so the simulated + ISP/WAN segments have nowhere legitimate to live in v4 at all. + +(Also, still open and pre-existing: NetBox holds ZERO aggregates -- gap G2 -- so nothing declares +these allocations at the RIR level. And `2602:f3e2:f01:100::/56` needs a `VR1 Off1` SITE, which +does not exist in NetBox either.) + +### The design tension the operator must rule on + +**v6 here is REGION-hierarchical; v4 is ROLE-based.** That asymmetry is defensible and, I think, +correct: v6 is abundant, so it can afford geography; v4 is scarce, so it is pooled by function. +The alternative -- forcing v4 to mirror the v6 region tree -- burns scarce v4 AND requires +RENUMBERING DC1 off `10.12.x`, which is exactly the inheritance D-101 bought in order to reuse +VR0 DC0's bundle unchanged (minimize delta to Roosevelt). That is a large cost for conceptual tidiness. + +### Options + +**(a) RECOMMENDED -- keep v4 role-based; fill in the MISSING roles. Ratify the deployed addresses.** +- New v4 role **Office**: `10.10.0.0/16`, carved **`/22` per office**. + VR1 Office1 = **`10.10.0.0/22`** -> LAN `10.10.0.0/24` (as deployed) + LXD compose `10.10.1.0/24` + (as deployed) + `.2`/`.3` spare. **This BLESSES exactly what is already running -- zero renumber.** +- New v4 role **Edge** (mirroring v6's `2602:f3e2:fe::/48`): `172.30.0.0/16` for simulated ISP/WAN + segments, carved out of the `172.16.0.0/12` container but with its OWN role, not OOB. + Office1's WAN `172.30.1.0/24` (as deployed) becomes legitimate; DC1/DC2 edges take `.2`/`.3`. +- **DC2 moves INSIDE the Cloud `/16`**: `10.12.64.0/19` (free -- DC1's six `/22`s occupy + `10.12.4/8/12/16/32/36`, so `10.12.64.0/19` collides with nothing and DC1<->DC2 stay routable + over the dark fiber). SUPERSEDES the scope doc's `10.13.0.0/19` recommendation. +- IPv6 per the pattern: Office1 = `2602:f3e2:f01:100::/56` + `/64` subnet; create the `VR1 Off1` site. +- COST: two NetBox roles + a handful of prefixes. **No deployment renumber at all.** + +**(b) Mirror the v6 region model in v4.** Give each virtual region a v4 `/16` (e.g. VR1 = `10.15.0.0/16`) +and carve sites inside it (infra / office / DC1 / DC2). Conceptually cleanest; the two families would +read identically. COST: RENUMBERS DC1's six planes off `10.12.x`, which BREAKS D-101's +inherit-DC0-unchanged economy and forces edits to `lib-net.sh`, MAAS subnets, and the bundle's VIPs. +It also spends a whole `/16` per virtual region on a test environment. Recommended ONLY if the +operator wants v4 and v6 structurally identical and accepts the DC renumber. + +**(c) Ratify the squat as-is and register it verbatim.** Record `10.10.0.0/24`, `10.10.1.0/24`, +`172.30.1.0/24`, `10.13.0.0/19` in NetBox exactly as deployed, with no new roles and no structure. +Cheapest today, and WRONG: it enshrines an ISP-sim network inside the corp OOB block, leaves offices +with no allocatable block for the next office, and leaves DC2 outside the Cloud role. Rejected +unless the operator wants pure speed. + +### What is NOT at stake + +Nothing is blocked and nothing is broken. Office1 is live and routing on the squat addresses. The +question is only what to write into the IPAM apex -- and the renumber cost is LOWEST RIGHT NOW, +before NetBox and GitBucket are deployed onto that network. Under option (a) the renumber cost is +ZERO, because (a) is shaped to bless what is already there. + +**References:** measured 2026-07-13 against `netbox.baldurkeep.com` (read-only; SEC-006 token). +D-101 (family matrix + DC1-inherits-DC0), D-111 (NN mnemonic subcarve), D-016 (tenant pool +`10.20.0.0/16` -- the reason the obvious `10.20.x` was NOT used for the compose net), +`docs/dc-dc-netbox-buildout-scope.md` (whose `DC2_V4_SUPERNET = 10.13.0.0/19` recommendation this +decision would supersede under option (a)), D-114 (which built the headend that exposed all of this). + +### D-115 -- AMENDMENT (2026-07-13): ADOPTED by operator ruling, with a measured constraint attached + +**Operator ruled all three questions:** +1. **v4 stays ROLE-BASED; fill in the missing roles.** Option (a). v6 keeps its region hierarchy; + v4 is pooled by function. No DC renumber; D-101's inherit-DC0-unchanged economy is preserved. +2. **`/22` per office.** VR1 Office1 = **`10.10.0.0/22`** -- which BLESSES the deployed LAN + (`10.10.0.0/24`) and LXD compose net (`10.10.1.0/24`) exactly. Zero deployment renumber. +3. **Register AND deploy dual-stack on Office1 NOW** -- Office1 becomes the first real dual-stack + proof and de-risks D-101's family matrix early. + +**THE ADOPTED CARVE** + +v4 (new roles marked): +``` +10.0.0.0/8 Corp Private + |- 10.10.0.0/16 Office <-- NEW ROLE + | |- 10.10.0.0/22 VR1 Office1 + | |- 10.10.0.0/24 office1-local (site LAN; Kea on the edge) [as deployed] + | |- 10.10.1.0/24 lxdbr0 compose net (MAAS DHCP/PXE) [as deployed] + | |- 10.10.2.0/24, 10.10.3.0/24 spare + |- 10.12.0.0/16 Cloud + | |- 10.12.{4,8,12,16,32,36}.0/22 DC1 planes [unchanged -- D-101 inheritance] + | |- 10.12.64.0/19 DC2 <-- MOVED INSIDE the Cloud /16 + |- 10.16.0.0/16 Datacenter (Willamette dev) + |- 10.17.0.0/16 Datacenter (Roosevelt dev -- the physical lab net) +172.16.0.0/12 Corp + |- 172.30.0.0/16 Edge (simulated ISP uplinks) <-- NEW ROLE, mirrors v6's fe::/48 + |- 172.30.1.0/24 office1-wan [as deployed] +``` +**This SUPERSEDES `docs/dc-dc-netbox-buildout-scope.md`'s `DC2_V4_SUPERNET = 10.13.0.0/19`**, which +sat outside every allocated block. DC2 becomes `10.12.64.0/19` -- free (DC1 occupies +`10.12.4/8/12/16/32/36`), inside the Cloud role, and still routable to DC1 over the dark fiber. + +v6 (determined by the existing pattern -- no design freedom): +``` +2602:f3e2:f01::/48 VR1 Offices [exists] + |- 2602:f3e2:f01:100::/56 VR1 Office1 <-- CREATE (+ the "VR1 Off1" site) + |- 2602:f3e2:f01:100::/64 Office1 office subnet <-- CREATE +``` + +### MEASURED CONSTRAINT (2026-07-13) -- IPv6 DOES NOT EGRESS THE LAB + +Discovered while planning ruling 3. On vcloud: +- Global v6 addresses present (`2602:f3e2:e00:100::/64` SLAAC + a `/128`), and a v6 default route + via RA. +- The v6 default gateway (link-local `fe80::e68d:8cff:fe14:7cd5`) **IS reachable** -- RA and L2 work. +- **But v6 INTERNET is UNREACHABLE**: both `2606:4700:4700::1111` and `2001:4860:4860::8888` fail, + 100% loss. v4 egress from the same host is fine. + +**Consequence, stated plainly so nobody builds on a false premise:** deploying dual-stack on Office1 +proves v6 ADDRESSING, RA, and services binding v6 -- which is most of D-101's family matrix and is +worth doing. It does **NOT** prove v6 INTERNET reachability, and it cannot: the GUA is not routed +past the lab boundary. **Public-facing GUA reachability (public API endpoints on v6, tenant GUA +egress) is UNPROVABLE in VR1 until the lab's upstream actually carries v6.** That is a network +issue OUTSIDE this cloud and outside this repo -- it needs the lab/upstream operator, not a change +here. Do not schedule a VR1 test that depends on it, and do not let a green internal-v6 result be +read as "v6 works end to end". + +`office1-wan` currently has **no v6 at all** (v4 NAT only), and `office1-local` is an addressless +isolated bridge (correct -- the OPNsense edge owns addressing on it). + +### NetBox write path -- NOT done unilaterally + +The carve above is recorded HERE (the repo) as the decision of record. Writing it into +`netbox.baldurkeep.com` is a MUTATION ON A PRODUCTION SYSTEM and is gated on explicit operator +approval; the standing architecture is read=source / write=feedback-only, with the Office1 sandbox +NetBox as the place changes are simulated first. The Office1 NetBox VM is composed and `Ready` but +not yet deployed -- so the natural sequence is: deploy it, import this carve there, verify, then +feed back to the real NetBox. + +## D-116: no Office1-local GitBucket -- keep using the existing git.baldurkeep.com + +**Status:** **ADOPTED 2026-07-13 (operator ruling).** SUPERSEDES the GitBucket half of D-103 and +removes GitBucket from Stage 2's build list. + +**Operator ruling, verbatim in substance:** "For the gitbucket, we've decided not to build a +separate one on the test cloud. We will continue to use the current setup and create new repos for +deployment tasks as needed." + +**What changes.** D-103 assigned Office1 three service VMs -- MAAS-region, NetBox, GitBucket -- and +D-114 recast them as MAAS-composed LXD VMs. **GitBucket is now OUT OF SCOPE entirely**: no third +service machine is composed, and the existing `git.baldurkeep.com` remains the git service of record +for VR1. New repos are created there as deployment tasks require. + +**What this closes.** The Stage 2 runbook carried an explicit OPEN QUESTION about the Office1-local +GitBucket -- "what it mirrors, and under what repo path, is not decided" -- because a *second*, +Office1-local GitBucket alongside the pre-existing `git.baldurkeep.com` raised a real +mirror/authority question nobody had answered. **That question is now MOOT.** There is one git +service, it is the existing one, and it is authoritative. This is a genuine simplification, not a +deferral. + +**Rationale.** The Office1-local GitBucket only ever existed to rehearse a per-site git mirror. The +operator judges the rehearsal not worth a machine: the real deployment work is served by the +existing instance, and a second one buys a mirroring problem (which repo is authoritative, what +syncs, in which direction) in exchange for no capability the deployment actually needs. + +**Delta to Roosevelt.** Roosevelt may still want a per-site git mirror for node-airgap reasons +(cf. D-107's per-DC artifact mirror, which is a SEPARATE concern and is NOT affected by this +ruling -- D-107 is about OS/package artifacts for node provisioning, not about hosting git repos). +If Roosevelt later needs a site-local git service, it is a fresh decision made against real +requirements, not a rehearsal artifact carried forward. Recording that explicitly so nobody reads +this ruling as "Roosevelt will never have a git mirror". + +**Consequences (all applied in the same change):** GitBucket is removed from Stage 2's Build/Gate in +`docs/dc-dc-deployment-workflow.md`, from the Office1 standup runbook's sequence and its open +questions, and from `docs/vr1-office1-as-built.md`. Office1's composed service machines are +therefore exactly TWO: `office1-netbox` and `office1-tailscale`, both LIVE. + +--- + +## D-117: VR1 site naming -- the apex says DC0/DC1, the repo says DC1/DC2 (G5) + +**Status:** **ADOPTED 2026-07-13 (operator ruling: Option B, DC-only). FULLY EXECUTED BY D-119 +(2026-07-14) -- the "PARTIALLY EXECUTED" caveat below is DISCHARGED; read D-119 for the end state.** +The REPO moves to the apex's 0-indexed convention **for the DATACENTERS**. NetBox is the IPAM apex +and is NOT bent to fit the repo. Supersedes the `docs/dc-dc-netbox-buildout-scope.md` G5 +recommendation (which proposed the opposite, Option A) and the D-106 `dc1`/`dc2` zone labels. +Closes G5. + +> **D-119 completed this decision, it did not reverse it.** D-117 renamed only the NetBox IMPORT +> PATH, which left `dc0` meaning VR1's first DC in the importer and **VR0's live cloud** in +> `lib-net.sh` -- one string, two clouds. D-119 executes this decision's own amendment +> (region-qualify the selectors: `vr0-dc0` / `vr1-dc0` / `vr1-dc1`) across `lib-net.sh`, +> `lib-hosts.sh` and `opentofu/`, making the importer's DC->site mapping an IDENTITY. **The +> alternative -- renaming the apex to match the repo's `dc1`/`dc2` -- was considered and REJECTED; +> see D-119.** The apex is NOT renamed and no production IPAM write is involved. + +**Operator refinement (verbatim):** *"go ahead and rename the DC numbers. The office number is +fine."* -- so the **OFFICE KEEPS ITS NUMBER**: it stays **Office1**, and its NetBox site is +**`vr1-off1` / "VR1 Off1"**. That site does NOT exist upstream yet, so it is CREATED, not renamed -- +no collision (VR0's `vr0-off0` is untouched), and **nothing deployed changes**. + +**This is the G5 gap** (`docs/dc-dc-netbox-buildout-scope.md`), promoted to a decision because it +blocks the Office1 NetBox import and because the tool that trips over it WRITES BY DEFAULT. + +**The collision, measured live against `netbox.baldurkeep.com` (4.5.8) on 2026-07-13:** + +| v6 site `/48` | Prod NetBox site (the apex) | `dc-dc-prefixes-import.py` | +|---|---|---| +| `2602:f3e2:f02::/48` | **VR1 DC0** (`vr1-dc0`, id 43) | `--dc dc1` (its docstring: "`f02::/48` for dc1") | +| `2602:f3e2:f03::/48` | **VR1 DC1** (`vr1-dc1`, id 44) | `--dc dc2` | + +The **addressing agrees**; only the **labels are off by one**. The repo's "DC1" and the apex's "DC0" +are the SAME datacenter. Same off-by-one on the office: the repo's "Office1" carve +(`2602:f3e2:f01:100::/56`, D-115) exactly mirrors VR0's **Off0** (`e01:100::/56`). + +**Why this is dangerous, not merely cosmetic.** `dc-dc-prefixes-import.py` maps `dc1 -> slug +vr1-dc1` and **has no `--commit`** -- `--verify-only` is its read-only mode, so it WRITES BY +DEFAULT. Run it as documented against a NetBox seeded from prod and it binds DC1's `f02::/48` +prefixes to the site `vr1-dc1`, which upstream holds `f03::/48` -- i.e. **the other DC**, silently. +(`roles-aggregates-import.py` is the opposite: dry by default. Do not assume the tools are uniform.) + +**How it happened, recorded because the mistake is reusable.** The script's own comment: *"Slugs are +new -- no prior convention existed in this repo (grepped `opentofu/*.tf` and the buildout design; +none found)."* It grepped the **repo**. The convention lived in the **apex**. NetBox is declared the +IPAM apex (D-010/D-103) precisely so that naming is NOT a repo-local invention -- but the check for +prior art was run against the repo. Same class of error as concluding "no VR0 precedent for +MAAS-composed LXD VMs" after grepping only `bundle.yaml` (2026-07-13, same session). + +**What the site byte does and does NOT mean.** Within a region `/40`: `00`=Infrastructure, +`01`=Offices, `02`/`03`=the DCs. That is **sequential allocation, not a DC index** -- there is no +`02 -> DC0` mnemonic to preserve. An earlier draft of this entry claimed there was; that was wrong +and is corrected here, because it materially weakens the case against Option A. + +**Blast radius, measured.** `vr1-dc0` and `vr1-dc1` are **skeletal**: 1 prefix each (their own `/48` +container), and **0** devices / racks / VLANs / clusters. (Contrast `vr0-dc0`: 18 prefixes.) Renaming +them upstream touches 2 site records + their 2 prefix descriptions. Renaming the REPO instead touches +the live surfaces (`lib-net.sh`, the `--dc` flag, `opentofu/`, runbooks, SKILL.md, and the text of +D-101/D-106/D-111/D-115) and contradicts **already-ratified D-106** (Designate `dc1`/`dc2` zones). + +### Options + +**A. Rename the two skeletal apex sites to `vr1-dc1`/`vr1-dc2`; add `vr1-off1`. (RECOMMENDED)** +The scope doc's standing recommendation. Cheapest by a wide margin, and it preserves the `dc1`/`dc2` +language already ADOPTED in D-106 and wired through every live repo surface. Cost: VR1 reads DC1/DC2 +while VR0 reads DC0/DC1 -- a cosmetic cross-region inconsistency, and VR1's office becomes `Off1` +against VR0's `Off0`. +**TRAP -- rename ORDER matters:** `vr1-dc0 -> vr1-dc1` while `vr1-dc1` still exists is a **slug +collision**. Rename **`vr1-dc1` -> `vr1-dc2` FIRST**, then `vr1-dc0` -> `vr1-dc1`. Update the two +`/48` descriptions ("Datacenter 0 (DC0)") in the same change or they contradict their own site. + +**B. Rename the repo to the apex's 0-indexed `DC0`/`DC1` + `Off0`.** +Uniform 0-indexing across VR0 and VR1, and the apex is never bent to fit the repo. Cost: reopens +ratified D-106, and rewrites every live repo surface. Historical changelogs are NOT rewritten. + +**C. Re-point the tool only (`SITES = {dc1: vr1-dc0, dc2: vr1-dc1}`).** +Two lines, nothing else moves. **Argued against:** it makes `--dc dc1` write to a site *displayed* as +"VR1 DC0" forever -- a permanent hidden indirection, which is how this class of bug got here. + +**Whichever is chosen, one fix is unconditional:** give `dc-dc-prefixes-import.py` a `--commit` flag +so it is DRY BY DEFAULT, matching `roles-aggregates-import.py`. A write-by-default IPAM tool is a +standing hazard regardless of how the naming lands. + +### D-117 -- AMENDMENT (2026-07-13): ADOPTED (Option B), scope narrowed by operator ruling + +**Ruled:** VR1's datacenters are **DC0** (`vr1-dc0`, `2602:f3e2:f02::/48`) and **DC1** (`vr1-dc1`, +`2602:f3e2:f03::/48`) -- matching the apex, which already holds both sites. The repo's `dc1`/`dc2` +labels are retired in favour of **`dc0`/`dc1`**. + +VR1's office **remains Office1** -> NetBox site **`vr1-off1` ("VR1 Off1")**, carrying the D-115 carve +(`2602:f3e2:f01:100::/56` + `/64`, `10.10.0.0/22`). Note VR1 therefore reads **DC0/DC1/Off1** while +VR0 reads DC0/DC1/Off0 -- an accepted cosmetic asymmetry. The office site is NEW upstream, so it is +created cleanly and **no deployed object is renamed**. + +**SCOPE -- what B does and does NOT touch.** B is a ruling about **IPAM naming authority**: the +NetBox site record and the selector that resolves to it. It does NOT extend to deployment substrate. + +- **IN scope (renamed, DC ONLY):** the DC half of the NetBox site identity; the importer's `SITES` + + GUA map; the `--dc` selector (`dc1|dc2` -> `dc0|dc1`); `lib-net.sh`/`lib-hosts.sh` DC keys; + OpenTofu DC module labels; forward-looking DC prose. +- **OUT of scope -- NOTHING office-side is renamed.** Per the operator refinement above, `Office1` + keeps its number, so `voffice1`, `office1-opnsense`, the `office1-local`/`office1-wan` networks, + `office1-pool`, the MAAS VM host `office1-lxd`, the machines `office1-netbox` / + `office1-tailscale`, the OPNsense `OFFICE1_LXD_GW` object and the Tailscale node all stand as-is. + **No teardown, no rebuild.** +- **Also out of scope, independently:** deployment substrate is not a carve. Even where a name DID + change (the DC libvirt objects), the rename is a label on empty, unused infrastructure -- see the + table below. + +**TRAP 1 -- this is a SEMANTIC REMAP, not a string replace.** `dc1 -> dc0` AND `dc2 -> dc1`, so +`dc1` denotes different things before and after. A sequential `sed` double-applies and corrupts the +tree. Any mechanical pass must go through a placeholder (`dc1 -> __DCA__`, `dc2 -> __DCB__`, then +`__DCA__ -> dc0`, `__DCB__ -> dc1`). + +**TRAP 2 -- and this one NO mechanical pass can catch.** Existing prose uses "DC0" to mean **VR0's +rehearsal DC** while using "DC1"/"DC2" to mean **VR1's** DCs. The importer's docstring: *"DC1 +INHERITS the DC0 six-plane CIDRs unchanged (D-101 -- 'DC1 equals the DC0 template') ... not the +existing `vr0-dc0` site."* Remap VR1's `dc1 -> dc0` and that reads *"DC0 equals the DC0 template"* -- +two DIFFERENT DC0s (`vr0-dc0` and `vr1-dc0`) that were previously disambiguated by NUMBER ALONE. +**Every "DCn" in prose must be region-qualified by hand** (`VR0 DC0` vs `VR1 DC0`). This is why the +decision text below is ANNOTATED, not rewritten. + +**Treatment by surface kind (adopted with the ruling):** + +| Surface | Treatment | +|---|---| +| Operational/structured (importer, `lib-net.sh`, `--dc`, tofu labels) | Mechanical rename. **This tier ALONE unblocks the NetBox import.** | +| **ADOPTED decision text** (D-101, D-106, D-111, D-115) | **ANNOTATE in place, do NOT rewrite** -- the D-103->D-116 supersession precedent. Ratified prose is a historical record. | +| Forward-looking prose (runbooks, design docs) | Hand-edit, region-qualifying every "DCn" (Trap 2). | +| DC libvirt objects (`dc1-*`/`dc2-*` nets, `dc1-pool`/`dc2-pool`) | **Empty and unused** (0 volumes, no DC VMs). Rename the module labels; let the next **GATED** DC apply reconcile them. **No surprise destroy/recreate.** | +| Historical changelogs | **Never rewritten.** | + +**Unconditional companion fix (also ruled):** `netbox/dc-dc-prefixes-import.py` becomes **DRY BY +DEFAULT** with an explicit `--commit`, matching `roles-aggregates-import.py`. A write-by-default tool +pointed at the IPAM apex is a standing hazard, and it is the mechanism by which this naming bug would +have landed silently. + +### D-117 -- AMENDMENT (2026-07-13): the `$DC` shell selector is a SECOND, COLLIDING namespace + +Found while executing the rename. **`scripts/lib-net.sh` / `lib-hosts.sh` use a THREE-value `$DC` +selector in which `dc0` ALREADY MEANS VR0's DC0** (the deployed rehearsal), with `dc1`/`dc2` meaning +VR1's two DCs: + +``` +dc0|dc1) # lib-net.sh: VR0 DC0 and VR1's first DC share v4 literals (D-101 inheritance) +dc2) # VR1's second DC -- FAILS LOUD, no literals assigned yet +``` + +So the repo has been carrying **two different `dcN` namespaces at once**: the shell libs' (where +`dc0` = VR0) and the NetBox importer's (where `dc0` did not exist and `dc1` = VR1's first DC). That +is Trap 2 in its purest form, and it is why a mechanical `sed` across this rename would have been +**actively dangerous**: renaming VR1's `dc1 -> dc0` in `lib-net.sh` collides HEAD-ON with VR0's +existing `dc0` and would silently hand VR1's selector VR0's deployed literals. + +**Proposed fix (NOT yet executed -- does not block the NetBox import):** region-qualify the shell +selector to **the apex's own slugs** -- `vr0-dc0`, `vr1-dc0`, `vr1-dc1` -- so there is ONE namespace +across NetBox, the shell libs, and the importer, and every identifier is unambiguous by construction. +This is the same principle D-117 already adopts: the apex names things; the repo follows. + +**Blast radius:** `lib-net.sh`, `lib-hosts.sh`, and the scripts that take `$DC` +(`carve-host-interfaces.sh`, `dc-dc-ceph-disk-budget.sh`, `dc-dc-dr-drill.sh`, +`dc-dc-radosgw-multisite.sh`, `dc-dc-rbd-mirror.sh`, `phase-00-maas-standup.sh`, +`reenroll-hosts.sh`) plus their harnesses and `tests/dc-selector/`. + +**Until it is executed, the shell `$DC` selector retains its OLD meaning** (`dc0`=VR0 DC0, +`dc1`/`dc2`=VR1) while `netbox/dc-dc-prefixes-import.py --dc` uses the NEW, apex-aligned meaning +(`dc0`/`dc1` = VR1's two DCs). **These two are NOT interchangeable.** Do not copy a `$DC` value from +a shell script into `--dc`, or vice versa, without re-reading this entry. + +--- + +## D-118: the org ULA /48 -- ADOPT `fd50:840e:74e2::/48` + +**Status:** **ADOPTED 2026-07-13 (operator ruling).** Closes gap **G3** +(`docs/dc-dc-netbox-buildout-scope.md`) and the D-101 open literal `ORG_ULA_48`. **UNBLOCKS the VR1 +DC six-plane NetBox import**, which could not run without it. + +**Ruled:** `ORG_ULA_48 = **fd50:840e:74e2::/48**`. + +D-101 rules the org ULA `/48` a NetBox-assigned literal, "NOT hardcoded in this decision", and every +tool takes it as a REQUIRED env var with no default. Until now no decision assigned it, so +`netbox/dc-dc-prefixes-import.py` was unrunnable -- supplying a value would have been an inferred +value (hard rule 2). + +**Validation (measured 2026-07-13, not asserted):** + +| Property | Result | +|---|---| +| Inside `fc00::/7` | yes | +| First byte `0xfd` -> **L=1**, locally assigned (RFC 4193) | yes | +| 40-bit Global ID | `0x50840e74e2` -- no discernible structure | +| Prefix length | `/48` -> **256 x `/56`**, and D-101 allocates one `/56` per DC | +| Collides with Tailscale's ULA `fd7a:115c:a1e0::/48` | **NO** | +| Collides with anything else in the apex | no -- the apex holds only the `fc00::/8` / `fd00::/8` containers and Tailscale's `/48` | + +**The Tailscale non-collision is the load-bearing check, not a formality.** `office1-tailscale` runs +with `--accept-routes` on the self-hosted control plane, so an org ULA overlapping +`fd7a:115c:a1e0::/48` would be a LIVE routing conflict on the tailnet, not a paper one. `fd50` vs +`fd7a` -- distinct. + +**Provenance, stated honestly.** `docs/dc-dc-netbox-buildout-scope.md` 4c describes this value as +CSPRNG-generated. That claim originates in an earlier session and **cannot be re-verified now**. It +does not change the ruling: RFC 4193's "SHA1 of time + MAC" construction is a **SHOULD**, not a MUST +-- the actual requirement is a pseudo-random 40-bit Global ID, which this satisfies. Regenerating +would not make the value MORE correct; it would only substitute a different random number, at the +cost of re-editing the 7 places that already carry this one and risking a stale literal surviving in +one of them. Not worth it. + +**Corrects a FALSE claim in the repo.** `docs/changelog-20260711-ula-gen-command-fix.md` called this +"the **ratified** VR1 value" -- it was NOT ratified; no D-number ever assigned it. That phantom +ratification is very likely why the value has been treated as settled for two days while +`grep ORG_ULA docs/design-decisions.md` returned nothing. This entry makes the claim true, and the +changelog is annotated. **Same failure mode as D-117:** a value treated as authoritative because a +document asserted it, with no decision behind it. + +**Consequences.** The DC import can now run: +``` +ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # VR1 DC0 +ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f03::/48 \ + DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # VR1 DC1 +``` +(`DC_GUA_PREFIX` values are MEASURED off the apex per D-117; `DC1_V4_SUPERNET` is assigned by D-115.) +Per D-101/D-111 each DC takes a `/56` of this `/48`, indexed to its GUA site nibble, so the 4th +hextet reads `DC.NN` in BOTH families. + +## D-119: the VR1 DC namespace -- region-qualify the selectors; the repo adopts the apex + +**Status:** **ADOPTED 2026-07-14 (operator ruling).** COMPLETES D-117 rather than reversing it: +it executes D-117's own amendment (region-qualify the shell selectors) across the three surfaces +D-117 never touched. D-117 stays ADOPTED; its "partially executed" caveat is discharged here. +Closes tooling-gap #19 and Stage 2 close-out item C3. + +### The problem, stated exactly + +After D-117, the token `dc0` meant TWO DIFFERENT CLOUDS depending on which file you were reading: + +| Surface | `dc0` meant | `dc1` meant | +|---|---|---| +| NetBox apex (the IPAM authority) | VR1's FIRST DC | VR1's SECOND DC | +| `scripts/lib-net.sh` / `lib-hosts.sh` | **VR0's DC0 -- a DIFFERENT, LIVE cloud** | VR1's first DC | +| `opentofu/main.tf` | (no `dc0`) | VR1's first DC | + +One string, two clouds, one of them in production. A mechanical renumber makes it WORSE, not +better (D-117 TRAP 1): `sed s/dc1/dc0/` over `opentofu/` would hand VR1's first DC the string +`dc0`, which `lib-net.sh` already resolves to VR0's deployed literals. + +### The ruling + +**Region-qualify the DC selector everywhere. The repo adopts the apex's names verbatim.** + + vr0-dc0 VR0's DC0 the LIVE testcloud (a different region) + vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48 + vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48 + +Bare `dc0` / `dc1` / `dc2` are REJECTED loudly by the selectors. They are the ambiguity being +deleted; accepting them "for compatibility" would preserve the bug. + +**The apex is NOT renamed. There is NO production IPAM write in this decision.** The apex was +already self-consistent and 0-indexed; MEASURED 2026-07-14 against `netbox.baldurkeep.com`: + + 2602:f3e2:f02::/48 -> site vr1-dc0 "Virtual Region 1 (VR1) Datacenter 0 (DC0)" + 2602:f3e2:f03::/48 -> site vr1-dc1 "Virtual Region 1 (VR1) Datacenter 1 (DC1)" + (VR0, for contrast: vr0-dc0, vr0-dc1, vr0-off0 -- the same 0-indexed convention) + +The repo was the only surface out of step, because -- operator, verbatim in substance -- *"we +didn't catch the dc0 missing and went straight to one."* + +### Why this beats the alternative (renaming NetBox to match the repo) + +Considered and REJECTED: rename the apex to `vr1-dc1`/`vr1-dc2` so the ordinals match the repo's +`dc1`/`dc2`. It was rejected because it would: +- require a PRODUCTION IPAM write on the authority, ordered highest-slug-first or it violates + NetBox's unique-slug constraint; +- make VR1 the ONLY 1-indexed region, diverging from VR0 -- the very region VR1 rehearses; and +- create a worse ambiguity: `vr1-dc1` would mean VR1's FIRST DC while `vr0-dc1` still means VR0's + SECOND DC. Same suffix, different ordinal, one NetBox. + +(For the record, the apex is NOT uniformly `dcN`: the Eugene/Roosevelt sites are place-named -- +Charnelton, Cleveland, Roosevelt, Stevenson, Willamette. The `dcN` convention belongs to the +REHEARSAL regions VR0/VR1, and within those it is 0-indexed. The consistency claim is scoped to +them, not to the whole apex.) + +### The prize: the off-by-one becomes structurally impossible + +The importer's DC->site mapping becomes an IDENTITY (`vr1-dc0` -> slug `vr1-dc0`). **There is no +offset table left to get wrong.** This matters because the original defect was precisely a WRONG +LOOKUP TABLE -- `--dc dc1` bound to slug `vr1-dc1` while treating `f02::/48` as dc1's, though the +apex binds `f02` to VR1 DC0. It would have written the FIRST DC's prefixes onto the SECOND DC's +site, silently. Deleting the mapping deletes the bug class. + +### Scope -- what this decision does NOT change + +- **The office KEEPS its number** (operator ruling): NetBox site `vr1-off1`, and `Office1` / + `voffice1` / `office1-local` / `office1-netbox` / `~/vr1-office1-creds` throughout. The site is + BUILT AND RUNNING; renaming it churns hostnames, creds paths and libvirt networks for zero + functional gain. This leaves a known, ACCEPTED asymmetry with VR0's `vr0-off0`. Documented, not + fixed. +- **VR0 is not renamed.** Its DC stays DC0; only the SELECTOR STRING is region-qualified + (`dc0` -> `vr0-dc0`). No VR0 literal, host, or deployed object changes. +- **The ULA hextet still mirrors the GUA site nibble, NOT the DC number** (D-111): `vr1-dc0` -> + `fd50:840e:74e2:2NN::`, `vr1-dc1` -> `:3NN::`, tracking `f02`/`f03`. This is DELIBERATE so the + 4th hextet reads the same in both families. Do not "fix" it to match the DC index -- that would + break the already-imported sandbox prefixes. +- **NetBox data is untouched.** No re-import, no re-dump of `netbox/draft/vr1-draft.json`. + +### Blast radius (MEASURED 2026-07-14, before any edit) + +`opentofu/` renames change the libvirt objects' `name` attribute, so `tofu apply` will +DESTROY AND RECREATE them. Measured: the 6 `dc1_planes` networks, both DC storage pools, and the +3 mesh links are live in state but **EMPTY** -- no guests attached (Stage 3 has not run) and no +volumes in the DC pools (the only volumes belong to `office1_opnsense`, `ubuntu_noble_base`, and +`voffice1`, none of which are in a DC pool). The recreate is therefore safe, but it IS a +destroying apply and is gated: the plan is presented before it runs. + +### Standing rule this leaves behind + +**Never write a bare `dcN` for a VR1 datacenter again, in code or in prose.** Region-qualify it. +The one durable lesson of D-117 -- which survives its completion -- is that an unqualified `dcN` +in a repo that operates more than one cloud is a latent off-by-one waiting for a reader who does +not know which cloud the sentence is about. + +## D-120: static-services addressing convention for the MAAS-managed compose /24s (ADOPTED) + +**Status:** **ADOPTED 2026-07-15 -- option (a), EXECUTED.** Operator endorsed the direction (static +services belong in a low band, not mixed with deployed nodes) and ruled "it doesn't matter where the +IPs land in the sub-50 scope; go ahead with changing the IPs." The band layout below and the Office1 +service addresses (`.10`/`.11`) are ratified, and the Office1 re-IP was executed the same day (see +"Execution" and `runbooks/dc-dc-office1-service-reip.md`). Refines D-115 (which ratified that +`10.10.1.0/24` exists as the Office1 compose net but left its INTERNAL layout as just "MAAS +DHCP/PXE"). Does NOT reopen D-115's /22 carve. + +### The problem + +On `10.10.1.0/24` (Office1 `lxdbr0`, MAAS-managed), the persistent site services sit at +`office1-netbox = 10.10.1.201` and `office1-tailscale = 10.10.1.202` -- addressed IDENTICALLY to how +a deployed OpenStack node would be. MAAS auto-assigns deployed machines from `.201+` (just past its +`.100-.200` dynamic range), so the services landed there by default, not by design. They are NOT in +the DHCP dynamic range (no collision), but they are not distinguishable from ephemeral compute nodes +either. There is no ratified sub-layout for the /24. + +### Decision (PROPOSED) -- a functional band layout for EVERY MAAS-managed compose /24 + +Applies to Office1's `10.10.1.0/24` AND to each DC's compose /24 when the DCs are built (they inherit +the template -- minimize-delta): + + .1 gateway (the site containment VM: voffice1, and each DC's equivalent) + .2 - .49 STATIC site infrastructure/services (MAAS static-assign; OUTSIDE the dynamic range) + .50 - .99 reserved / spare (headroom for more static services) + .100 - .200 MAAS DYNAMIC (enlistment / commissioning / PXE / DHCP) [as already deployed] + .201 - .254 deployed compute / OpenStack nodes (MAAS auto-assign) [as already deployed] + +The dynamic and node bands are UNCHANGED from what MAAS already does -- only the static-services band +`.2-.49` is newly reserved, and it already sits outside the dynamic range, so no MAAS range edit is +needed, only per-service static IP assignment. + +### Office1 service addresses (PROPOSED; operator to ratify) + + office1-netbox 10.10.1.201 -> 10.10.1.10 + office1-tailscale 10.10.1.202 -> 10.10.1.11 + +`.10`/`.11` are arbitrary within `.2-.49`; chosen low and adjacent. Operator may pick otherwise. + +### Recording it in NetBox (the IPAM authority deciding where NetBox itself lives) + +The band layout is a carve that belongs IN NetBox: register `10.10.1.0/24`'s child ranges (static / +dynamic / node) and the two service IP assignments. Do this through the sandbox loop (D-115 / the +C4-documented loop), not by hand -- NetBox is the authority for exactly this. + +### Consequence / cost (why this is not a free change) + +Re-IPing Office1's two RUNNING services cascaded to the repo references, INCLUDING the hardcoded +`SANDBOX_HOSTS` allowlist (`{localhost, 127.0.0.1, 10.10.1.201}` -> `.10`) in +`roles-aggregates-import.py`, `dc-dc-prefixes-import.py`, and `d115-office-carve.py` (added +2026-07-14) plus its test. The sweep + gauntlet ran with the re-IP (see "Execution" above and +`runbooks/dc-dc-office1-service-reip.md`). Dated changelogs keep `.201`/`.202` as historical +snapshots. + +### Options + +- **(a) ADOPT the convention AND re-IP Office1 now** (recommended) -- Office1 is the proven template; + making it match means the DCs inherit a clean, already-exercised layout. Cost: a gated re-IP of two + running services + the 20-reference sweep. +- **(b) ADOPT the convention for the DCs ONLY; leave Office1 at `.201/.202`** -- zero disruption now, + but Office1 (the template everyone copies) permanently diverges from the convention, and the + `SANDBOX_HOSTS` allowlist keeps a "deployed-node-band" address for a static service. +- **(c) Reject** -- keep services in the node band. Not recommended: it is the ambiguity this entry + exists to remove. + +**Recommendation: (a).** Get the template right once. + +### Execution (2026-07-15) -- option (a), done + +Re-IP executed on the two running services, no wipe, verified on the wire: + + office1-netbox 10.10.1.201 -> 10.10.1.10 NetBox HTTP 200 at .10, reachable vcloud-side + office1-tailscale 10.10.1.202 -> 10.10.1.11 tailscale Running, still advertises 10.10.0.0/22 + +Two findings that shaped the method (and that the DC replication inherits): + +- **MAAS will NOT edit an interface on a Deployed machine** ("Cannot unlink subnet interface because + the machine is not New, Ready, Allocated, or Broken"). The only lift is Release->redeploy, which + WIPES the OS -- forbidden here (it would destroy the seeded sandbox). So the durable MAAS-model + update is unavailable without a wipe. Resolution: set the live address on the GUEST only, via BOTH + `/etc/netplan/50-cloud-init.yaml` and `/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (so it + survives reboot), and ACCEPT MAAS-model drift (MAAS still shows `.201`/`.202` `mode=auto`). The + drift reconciles for free at the next teardown/redeploy from a corrected model. `.10`/`.11` are + off MAAS's auto-assign path -- MAAS allocates nodes from `.201+` upward (observed: the two services + themselves landed at `.201`/`.202`), so the low `.2-.49` band is never auto-handed out. NB: the + whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee; + the allocation DIRECTION is. A DC building fresh sidesteps this by MAAS-static-assigning `.2-.49` + before deploy (Step 1 below), which pins the addresses explicitly rather than relying on direction. +- **NetBox `ALLOWED_HOSTS` = `*`** (config default; not overridden in `env/netbox.env`), so NetBox + served at `.10` with no `DisallowedHost` change needed. +- **Method note for the DCs:** drive the guest change over `lxc exec` from the containment VM, NOT + SSH-to-the-guest-IP -- the LXD socket is immune to the `netplan apply` that swaps the address, so + the control channel never drops. `lxc console`/`exec` is also the recovery backstop if the new + address does not come up. For a DC building fresh, prefer static-assigning `.2-.49` in MAAS BEFORE + deploy (a Ready machine accepts the link edit) so there is no drift to accept -- the Deployed-block + above only bit Office1 because its services were already deployed in the node band. + +The NetBox-side carve record (Step 6 -- register the `.2-.49`/dynamic/node child ranges + the two +service IPs in the sandbox, then feed upstream) is still OWED and rides with C2; it is NOT done here. diff --git a/docs/incident-20260712-opnsense-edge-boot-triplefault.md b/docs/incident-20260712-opnsense-edge-boot-triplefault.md new file mode 100644 index 0000000..48060e6 --- /dev/null +++ b/docs/incident-20260712-opnsense-edge-boot-triplefault.md @@ -0,0 +1,113 @@ +# Incident 2026-07-12: Office1 OPNsense edge triple-faults at the BTX loader + +> ## ROOT CAUSE FOUND 2026-07-12 (DOCFIX-188) -- READ THIS FIRST, THEN STOP +> +> **The guest had 2 MiB of RAM.** Not a CPU, nesting, machine-type, or console problem. +> Everything below this box is the ORIGINAL (wrong) investigation, retained for the record. +> **Do not work the "ranked next steps" -- they all chase the wrong layer.** +> +> `dmacvicar/libvirt` >= 0.9 changed `memory` from MiB (the 0.8-era meaning the modules were +> written against) to **raw libvirt units, defaulting to KiB**. With no `memory_unit`, +> `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` -> QEMU **`-m size=2048k`** = +> **2 MiB**. Measured end-to-end: module input -> tofu state -> domain XML -> `virsh dominfo` +> (`Max memory: 2048 KiB`) -> the live QEMU cmdline. +> +> `boot2` is tiny and fits in 2 MiB, so it echoes `/boot.config` and *then* triple-faults +> handing off to `/boot/loader`, which does not fit. **That is why the fault was +> deterministic at exactly 262 bytes and immune to every CPU/machine/disk/console change +> tried -- none of them touched the cause.** +> +> **Fix:** `memory_unit = "MiB"` on the `libvirt_domain` in all three VM modules +> (`opnsense-edge`, `cloudinit-vm`, `node-vm` -- the same defect was latent in all of them +> and would have broken every future VR1 VM). Guarded against recurrence by +> `scripts/opentofu-validate.sh` S1. See +> `docs/changelog-20260712-libvirt-memory-unit-rootcause.md`. +> +> **Lesson for the next incident:** a bootloader that dies at a fixed byte offset, immune to +> every knob you turn, is a *resource* problem, not a CPU-feature problem. The domain XML and +> the QEMU cmdline are ground truth -- read them before theorising about nested virt. +> +> **Status:** root cause fixed in repo; live boot verification is the remaining gated step. + +**Original status (SUPERSEDED):** OPEN. The Office1 OPNsense edge VM is built and *starts*, but OPNsense +triple-faults ~262 bytes into boot. Blocks the Office1 headend (router+DHCP for +`office1-local`) and therefore the Office1 NetBox VM. Written at a context limit as a +resume artifact -- a fresh session should read this + `docs/session-ledger.md` first. + +## Symptom (confirmed, reproducible) +- Domain `office1-opnsense` (libvirt, `qemu:///system`) enters `running (booted)` then + goes to `paused (unknown)`. `virsh resume` fails with *"cont: Resetting the Virtual + Machine is required"* -> the guest **triple-faulted**. +- Serial capture (`/var/lib/libvirt/vr1/staging/office1-opnsense-serial.log`, `root:600`, + read with `sudo cat`) contains exactly, every boot: + ``` + /boot.config: -S115200 -h -D + ``` + i.e. FreeBSD `boot2` echoes `/boot.config` (serial 115200, `-h` serial console, `-D` + dual console) and then triple-faults **handing off to the BTX `/boot/loader`**. The log + mtime updates on each boot (confirmed fresh, not stale); it is deterministically 262 bytes. + +## Environment +- **Double-nested virt:** OPNsense guest -> `vcloud` host (itself a VM: virtio NIC/disk) + -> outer hypervisor. Host CPU model reported by libvirt: **AMD `Opteron_G3`**. Nested + KVM on (`kvm_amd/parameters/nested = 1`). + - **CORRECTION 2026-07-12 (DOCFIX-189): the `Opteron_G3` label is a RED HERRING.** The + real CPU is an **AMD EPYC 9965 (Zen 5, family 26)** (`/proc/cpuinfo`). libvirt's CPU + model database does not know family 26, so it falls back to the oldest matching model + name. The prior session built a nested-virt/ancient-CPU theory partly on this artifact. + Read `/proc/cpuinfo`, not libvirt's model guess. +- Image: OPNsense **26.1 nano** amd64 (`scripts/opnsense-prep-image.sh 26.1`), prepped to + `/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2` (11 GiB virtual). +- Module: `opentofu/modules/opnsense-edge`, instantiated as `module "office1_opnsense"` in + `opentofu/main.tf`. LAN=`office1-local` (`vtnet0`), WAN=`office1-wan` (`vtnet1`, NAT + `172.30.1.0/24`). Config ISO (real-ISP-router config, DOCFIX-185) at + `/var/lib/libvirt/vr1/staging/office1-opnsense-config.iso`. + +## What was tried -- ALL applied and verified in the domain XML, NONE resolved it +1. **Serial console added** (module gap -- nano is serial-only). Got the boot *to* the + loader stage (from a no-console early fault to the 262-byte `/boot.config` point). +2. **`machine` q35 -> i440fx** (`pc-i440fx-noble`, verified). No change. (Note: machine + + cpu are create-time; the provider does an in-place "change" that does NOT apply -- + must recreate the domain: `virsh destroy && virsh undefine`, then `tofu apply`.) +3. **Disk: COW overlay -> direct per-VM copy** of the nano (verified 11 GiB / 2.14 GiB + allocated, no backing). No change. +4. **CPU `host-passthrough`** (verified). No change. +5. **Disable AMD `svm`** (`<feature policy='disable' name='svm'/>` in XML -- the documented + AMD nested-virt fix, forum: `-cpu host,-svm`). **Still 262 bytes.** NOTE: the provider's + CPU-feature key is `features` (plural); `feature` validates but is silently dropped. + +## Ranked next steps for a fresh session +1. **Full forum CPU flag set**, not just `-svm`: add `+kvm_pv_eoi,+kvm_pv_unhalt` (and try + without `svm` masking too). Likely via `libvirt_domain.qemu_commandline` since the + provider's CPU `features` may not fully translate under `host-passthrough`. Confirm the + masking actually reaches the guest CPUID. +2. **Video device (`-D` dual console).** `/boot.config` requests dual console but the domain + has **no video/graphics device** -- `boot2`'s VGA init may fault. Add a `graphics` + + `video` (VGA/std) device and retry. (Cheap, plausible, not yet tried.) +3. **Memory 2 GB -> 4 GB** (below OPNsense's 3 GB min; guides use 4096). Recreate to apply. + - **CORRECTION (DOCFIX-189): this step was RIGHT FOR THE WRONG REASON, and its premise is + unsourced.** The memory *was* the problem -- but it was 2 MiB, not "2 GB but a bit + small", and no measurement was taken to notice that. The "3 GB min" figure is + UNVERIFIED (no source given); do not propagate it. If 2 GiB proves insufficient after + the real fix, check OPNsense 26.1's documented minimum before picking a number. +4. **UEFI boot (OVMF)** instead of legacy BIOS/BTX -- sidesteps BTX entirely, but the nano + is a BIOS/MBR image, so this needs care (or a different image build). +5. **Outer-hypervisor CPU:** under double-nesting, vcloud's exposed CPU (`Opteron_G3`) may + not provide what FreeBSD's BTX needs. Consider testing a minimal FreeBSD/OPNsense boot + directly on vcloud to isolate whether it's nesting-depth-specific. + +## Reproduce / operate +- Recreate + boot: `cd opentofu && source ~/vr1-stage1.env && virsh -c qemu:///system + destroy office1-opnsense; virsh -c qemu:///system undefine office1-opnsense; tofu apply`. +- Watch: `virsh domstate --reason office1-opnsense`; serial via `sudo cat` the log above. +- Provider CPU/serial schema is attribute-style + nested under `devices`/`cpu`; introspect + with `tofu providers schema -json` (see how `serials`/`features` were found this session). +- As-executed log: `~/as-executed/2026-07-12-dc-dc-phase1-office1.log`. +- Creds (jumphost-only, 0600, `~/vr1-office1-creds/`): SSH key + OPNsense root pw/hash. + **Operator: revoke the pasted NetBox token + these at close.** + +## Cross-refs +- `docs/changelog-20260712-opnsense-edge-boot-fixes.md` (DOCFIX-187, the module changes). +- `docs/changelog-20260712-office1-opnsense-edge-build.md` (DOCFIX-186, the build + the + apparmor/config-iso-staging findings). +- `docs/changelog-20260712-opnsense-edge-real-isp-router.md` (DOCFIX-185, config posture). diff --git a/docs/security-ledger.md b/docs/security-ledger.md index efa92a3..8f7329f 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -13,3 +13,43 @@ | SEC-002 | 2026-06-17 | juju action params persist in the operation log -- charm authorization must use short-lived child tokens | DOCFIX-011 | operator | STANDING RULE (verify each vault authorize) | | SEC-003 | 2026-07-03 | Vault unseal-key custody is single-operator (bus factor) | D-069 | operator | OPEN -- assign custodians + rehearse second-person unseal (DEFERRED 2026-07-06 by operator; note: keeps d011-06 MANUAL, gates D-011 full close) | | SEC-004 | 2026-05-27 | repo temporarily PUBLIC for v1 web_fetch workflow | project completion list | operator | OPEN -- flip to private at v1 close (deferral reaffirmed 2026-07-06) | +| SEC-005 | 2026-07-13 | long-lived GitBucket PAT stored in PLAINTEXT on the jumphost (`credential.helper store`, mode 600) so the agent can push unattended; GitBucket PATs are account-wide -- no per-repo scoping exists | docs/changelog-20260713-git-credential-store.md | operator | OPEN -- dedicated token (not a reused personal one); revoke + rotate at v1 close, or immediately if the jumphost is shared/rebuilt. Revoke: GitBucket -> Account Settings -> Applications | +| SEC-006 | 2026-07-12 | NetBox API token PASTED INTO AGENT CHAT CONTEXT during D-111 IPv6 reconciliation -- the token is BURNED (it exists in a transcript that is not under our control) and must be revoked + reissued, not merely rotated on schedule | docs/changelog-20260712-office1-opnsense-edge-build.md; flagged again at 2026-07-12 session close | operator | OPEN -- **DEFERRED by operator ruling 2026-07-13: revoke at completion of this deployment.** Until then the token is live and exposed. Revoke: NetBox -> Admin -> API Tokens | +| SEC-007 | 2026-07-12 | `~/vr1-office1-creds/` on the jumphost holds the Office1 edge root password + its bcrypt hash + the `office1_svc` SSH PRIVATE key + (added 2026-07-13) the OPNsense **API key/secret** (`opnsense-api.txt`, 0600), all in plaintext (dir 0700, files 0600) | 2026-07-12 edge build; custody detail deliberately not recorded here per D-069 | operator | OPEN -- required for edge management (D-112(c) makes SSH the only management path), so this is a ROTATION obligation, not a delete-me. Rotate at v1 close / if the jumphost is rebuilt or shared | +| SEC-008 | 2026-07-13 | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, 0600) -- the operator's 48-char Tailscale auth key for the SELF-HOSTED control plane `tailscale.baldurkeep.com`. Used to enrol `office1-tailscale`. Consumed BY PATH and never printed into a session; the copy shipped to the node was SHREDDED after `tailscale up` (tailscaled holds its own node key now). | 2026-07-13 Office1 Tailscale enrolment; docs/vr1-office1-as-built.md | operator | OPEN -- ROTATION obligation, not a delete-me: the key is still on vcloud for re-enrolment. Revoke/reissue on the control server at v1 close, or immediately if vcloud is rebuilt or shared. | +| SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | + +**STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive +files AND environment/config files for a site live in a single `~/<site>-creds/` folder on vcloud, +mode 0700, files 0600 (public keys 0644). No loose env files in `~`. Sites: `~/vr1-office1-creds/` +(exists); when DC deployment starts, `~/vr1-dc0-creds/` and `~/vr1-dc1-creds/` follow the SAME +pattern from day one -- do not accumulate loose `vr1-dc0*.env` in `~` and consolidate later. This is +a DATA-LOSS / continuity control as much as a security one: scattered creds get lost between +sessions and after compaction. Rationale is looser-security-but-strict-hygiene: this is a closed +in-house test, so real credential hardening (rotation, revocation, non-shared tokens) is deferred to +post-teardown redeploy cycles (test2/3/4+), but consolidation happens NOW so nothing is lost. + +**ADDENDUM (2026-07-15): the sandbox NetBox token was MISSED by the original consolidation.** It lived +only at `/root/netbox-secrets/api.token` on `office1-netbox`, never in `~/vr1-office1-creds/` -- the +prior consolidation moved the three loose `~` env files but did not pull this VM-resident secret into +the folder, contrary to the standing convention ("ALL sensitive files for a site live in +`~/<site>-creds/`"). Found while starting C2 sub-task (i) (the fidelity re-check needed it). Now +consolidated to `~/vr1-office1-creds/vr1-netbox-sandbox.env` (mode 0600; `NETBOX_URL=http://10.10.1.10:8000` ++ the assembled `nbt_<key>.<plaintext>` token), copied off the VM without being printed into the +session. The `/root/netbox-secrets/` copy remains the source-of-record on the VM; this is a working +copy on the jumphost, mirroring how `vr1-netbox.env` holds the upstream token. See +`docs/changelog-20260715-fidelity-check-scope-correctness.md`. + +**ENFORCEMENT (2026-07-15) -- so the miss cannot recur, esp. for DC0/DC1.** The convention was never +ENFORCED; a VM-minted secret had no forcing function to get consolidated and no check to catch a miss. +Now built (`docs/changelog-20260715-creds-audit.md`): +- **`creds-manifests/<site>.manifest`** -- declares EXACTLY the secrets a site must hold, each with its + mode and PROVENANCE (`local`, or `<vm>:<path>` for VM-minted secrets -- the class that was missed). + `creds-manifests/vr1-office1.manifest` seeded (11 entries) and the live folder audits CLEAN. +- **`scripts/creds-audit.sh <site>`** -- verifies the live `~/<site>-creds/` against the manifest + (presence, 0700 folder / declared file modes, non-empty), flags UNDECLARED files in the folder, and + scans the home root for SPRAWL (loose `*.env`/`*.token`/`*authkey*` outside any `*-creds/`). Reads NO + secret values -- metadata only. Harness `tests/creds-audit/` (7/7). +- **Standing rule:** run `bash scripts/creds-audit.sh vr1-dcN` at the CLOSE of each DC standup, and + consolidate+manifest every secret AT MINT TIME (not "later" -- later is when it is lost). Wired into + the `dc-dc-phase3` runbook. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 92acb7e..6f1bf6e 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -2,9 +2,8 @@ **Purpose.** Long ops sessions on this cloud routinely exceed a single context window and get COMPACTED (sometimes several times). Anything living only in the chat scrollback -is lost at compaction. This ledger is the durable, committed record of what is in flight, -so any session -- after a compaction, or a fresh one, or the parallel Claude Code stream -- -can resume without losing pending work. +is lost at compaction. This ledger is the durable, committed record of what is IN FLIGHT, +so any session -- after a compaction, or a fresh one -- can resume without losing pending work. **How to use it (standing practice).** 1. At session start: read this ledger AND run `bash scripts/ledger-scan.sh`. Reconcile. @@ -13,1551 +12,537 @@ This narrative must not claim CLOSED anything the scan shows OPEN, nor omit what it surfaces. 3. Update this ledger at every deliverable/commit -- it is a standing deliverable like the changelog. The changelog says what CHANGED; this ledger says what is still OPEN. -4. Numbers and the machine-derived section below are seeded from a scan; re-run the scan and - re-seed rather than editing those by hand. +4. The machine-derived block below is seeded from a scan; re-run the scan and re-seed rather + than editing it by hand. + +**SINGLE STREAM (collapsed 2026-07-13).** This ledger previously carried three parallel, +separately-owned stream sections (`main-chat`, `jumphost`, `shared`) plus ~30 append-only +session narratives. Those streams are CLOSED and reconciled into the one list below. There is +now ONE stream. Do not re-introduce per-stream sections. + +**Where the history went.** The 2,189-line session-by-session narrative is NOT lost -- it is in +git history (the parent of the collapse commit) and, in durable form, in the 65 `docs/changelog-*.md` +files, `docs/design-decisions.md`, and the incident reports. This ledger deliberately carries only +what is still OPEN, plus the facts that would otherwise be lost because they live nowhere else. --- <!-- SECTION: machine-derived | OWNER: ledger-scan (regenerate; never hand-edit) --> ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) -_As of 2026-07-09 scan (post-DOCFIX-161, final consolidation pass); -re-run the scan to refresh:_ +_Re-seeded from the 2026-07-15 scan. Re-run `bash scripts/ledger-scan.sh` to refresh._ -- **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 - ruled out per amendment; off-EOL path OPEN), D-071 (routine update cadence + - controller patch policy -- operator to rule; gated on the pre-DC-DC HA/backup - session). **D-100..D-110 are ALL ADOPTED as of 2026-07-09** (D-102 MERGED into - D-101; D-109 confirmed standalone) -- Stage 0 of - `docs/dc-dc-deployment-workflow.md` is CLEARED; see - `docs/dc-dc-buildout-design.md` Section 10 for the ruling record. _(D-050 - RESOLVED/CLOSED 2026-07-06 via D-051; D-073 APPLIED live 2026-07-06 -- not - scan-tracked as open. D-076..D-099 are a RESERVED source-project band, never - assigned in this repo, per D-110.)_ -- **OPEN security rows:** SEC-001 (rotate at v1 close, re-ruled 2026-07-06), - SEC-003 (custodians + second-person unseal -- DEFERRED; keeps d011-06 MANUAL), - SEC-004 (flip repo to private at v1 close -- deferral reaffirmed). -- **Next-free numbers:** D = 111, DOCFIX = 162, BUNDLEFIX = 012. - (Seeded from the 2026-07-09 scan, post DOCFIX-161.) +- **PROPOSED / OPEN decisions:** D-068 (Vault substrate hardening, Roosevelt -- 1.16 ruled OUT + per amendment; the off-EOL-1.8.8 path remains OPEN), D-071 (routine update cadence + Juju + controller patch policy -- AMENDED, still PROPOSED, awaiting operator ratification; gated on + the pre-DC-DC controller HA/backup session). +- **OPEN security rows:** SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007, **SEC-008**. + (SEC-008 -- the vcloud re-enrolment key -- was MISSING from the prior block.) +- **Next-free numbers:** D = **121**, DOCFIX = 196, BUNDLEFIX = 013. + (Since the prior block: D-119 + D-120 assigned and ADOPTED, and DOCFIX-195 assigned by this change + -- so next-free advanced D 119->121 and DOCFIX 195->196.) +- **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) + ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- `ledger-scan` reads + prose and a decoy token inflates the next-free counter. This has bitten twice. +- **KNOWN `ledger-scan` DEFECT (found 2026-07-14, logged not fixed):** the scan reports **D-115 as + PROPOSED/OPEN. It is not -- D-115 is ADOPTED.** The scan matches the words "PROPOSED/OPEN" + anywhere in an entry's prose, and D-115's Status line reads "ADOPTED ... Originally PROPOSED/OPEN + the same day". It should read the `**Status:**` line, not the body. This is the *status* twin of + the decoy-token rule above. **Until it is fixed, do not trust the scan's PROPOSED/OPEN list + without checking the Status line.** D-115 is the only current false positive. + +<!-- END: machine-derived --> --- -<!-- END: machine-derived --> -<!-- SECTION: main-chat | OWNER: main claude.ai chat stream only --> -## Active build (this session) +## Live state (measured 2026-07-13; nothing is mid-flight) -- **validate.sh D-011 runner -- modular.** Foundation committed: `scripts/lib-validate.sh` - (exit contract 0/1/2/3/4, emit, vr_json stderr-safe, run, env hygiene, disruptive gate) + - `scripts/validate.sh` orchestrator (profiles, verdict, --stop-on-fail, --include-disruptive). - - Batch 1 committed: `checks/d011-01-charms`, `checks/d011-06-vault-unseal` (MANUAL until - the SEC-003 rehearsal closes). - - Batch 2 committed: `checks/d011-02-vip-jumphost` (catalog-derive public origins, TLS+HTTP), - `checks/d011-03-vip-tenant` (ephemeral agnhost pod egress -> keystone VIP, D-035 proof). - `post-restart` profile now fully populated. - - Batch 3 DRAFTED (mock-tested only, NEEDS LIVE VALIDATION): `d011-04-octavia-lb` - (RR+member additive; amphora failover --disruptive, N+1 headroom-guarded) + - `d011-05-magnum-e2e` (wraps tenant-acceptance + timing). full-d011 profile now COMPLETE. - - **Next:** live-validate batch 3 (see Verify-live queue), then item-3/#5-#8 backlog. - - D-011 AMENDED bar: 1 charms; 2 VIP jumphost; 3 VIP tenant; 4 octavia RR/failover/recovery; - 5 magnum e2e + OCCM; 6 second-person manual unseal (attestation, D-069); 7 DROPPED - (D-070 supersedes D-012 snapshots); 8 DROPPED (D-019, no Designate). +- **VR1 / Office1 edge `office1-opnsense` is UP and healthy.** LAN 10.10.0.1/24, WAN 172.30.1.2 + (gw 172.30.1.1). Routing + automatic NAT, egress 0.0% loss, 8 LAN pass rules in pf, serial + console + getty alive, and `kea-dhcp4` serving DHCP on udp/67 (subnet 10.10.0.0/24, pool + .100-.199). **DHCP is API-MANAGED (D-113(a2)).** +- **OpenTofu repo/state back IN SYNC -- D-119 rename APPLIED (2026-07-14, operator-gated).** + `tofu apply`: **11 added, 0 changed, 11 destroyed**; `tofu plan` now reports "No changes". Verified + on the host: six `vr1-dc0-*` networks, three `mesh-vr1-*` links, `vr1-dc0-pool`/`vr1-dc1-pool`; ZERO + old `dc1-*`/`dc2-*`/`mesh-dc*` names remain. Pre-apply LIVE re-check confirmed all 11 objects empty + (0 leases, 0 volumes, no domain attached). Office1 (`office1-local`/`office1-wan`/`wan`/edge/ + `voffice1`) was NOT in the plan and is untouched -- edge + voffice1 still reachable post-apply. + tofu v1.12.3; tree validates (10/10 modules); lock committed. +- **VR0 / DC0 testcloud:** fleet fully current (juju 3.6.25, 91 agents); D-011 CLOSED except + item 6 (manual unseal, blocked on SEC-003). devteam is a live tenant running their own + cluster. No beta cluster, no foil tenant, no orphan trustees (see State facts). +- Branch `dc-dc-stage1-phase0-first-exec`, **pushed and level with origin (2026-07-14)**. VR1 Stage 1 + COMPLETE; Stage 2 in CLOSE-OUT (C1..C4 open, C5 done). +- **SESSION CONSOLIDATION (2026-07-14).** Work had fragmented across FOUR concurrent Claude sessions + on this one clone -- including two long-lived `--permission-mode auto` background agents (idle 14-15h + but still holding auto-approve rights against the live cloud), and a second interactive session that + received the SAME "resume last night's work" prompt six minutes before this one and began bootstrapping + in parallel. All three were terminated and the supervising daemon stopped (it had respawned the agents + once). **Standing lesson: the daemon ADOPTS AND RESPAWNS background workers -- killing the child is not + enough; stop the daemon (`claude daemon stop --any`).** The working tree was clean and 3-ahead + throughout; nothing was lost or clobbered. ONE session is now the rule. +- **Last night's three commits (D-113 amendment + the v6 script) were made by the auto-permission + background agent and left UNPUSHED and UNDOCUMENTED.** Both closed 2026-07-14: pushed, and + `docs/changelog-20260714-opnsense-set-interface-v6.md` written. -## Script backlog (changelog-tracked) +--- -- DONE: tenant-offboard.sh, vault-kv-health.sh, cloud-assert.sh (item 3 largely), validate - foundation + batch 1. -- **item 3 remainder:** fold `vault-kv-health.sh` into cloud-assert as a section; use the - combined gate to regenerate the stale restart-procedure runbook. -- **item 5:** `scripts/tenant-cluster-create.sh` -- generalize the t2-02 stage6 + watch + - D-066-evidence wrapper. Not started. -- **item 6:** `scripts/keystone-policy-drift.sh` -- D-051 base_* alignment (LIVE-READ PENDING). - Not started. -- **item 7:** `scripts/cloud-snapshot.sh` -- juju-export baseline capture (D-070 retired KVM - snapshots but KEPT the export/inventory baseline). Not started. -- **item 8:** `tests/tenant-acceptance/` harness -- pending (tenant-onboard harness landed). +## OPEN WORK -- VR1 DC-DC (the current mission) -## Register / operator-gated (see docs/handoff-20260703-open-items.md) +**D-120 ADOPTED + EXECUTED (2026-07-15) -- static-services addressing for the compose /24s.** +Office1's two services were re-IP'd OUT of the MAAS node band into the `.2-.49` static band: +`office1-netbox .201 -> 10.10.1.10`, `office1-tailscale .202 -> 10.10.1.11`. On the wire, no wipe +(NetBox HTTP 200 at .10; tailscale still advertises `10.10.0.0/22`). Band layout ratified: `.2-.49` +static / `.100-.200` dynamic / `.201+` nodes, for every MAAS compose /24; DCs inherit it from day +one. Method finding: MAAS won't edit a Deployed machine's interface (lift = redeploy = WIPE), so the +address is set on the guest (netplan + curtin, over `lxc exec`) and MAAS-model drift is accepted +(reconciles at next teardown/redeploy). Sweep + gauntlet done; `runbooks/dc-dc-office1-service-reip.md` +now records the measured method. **D-120 Step 6 DONE 2026-07-15** -- the /24 child ranges + the two +service IPs are registered in `office1-netbox` (the VR1 IPAM apex) via `netbox/d120-compose-bands.py +--commit` and VERIFIED by the (now-extended) fidelity check -> exit 0. The `ip-ranges`/`ip-addresses` +tooling gap is CLOSED (dumper + checker now cover both classes). NO apex write -- the upstream +write-back stays DEFERRED to end-of-deployment. This closed the last piece of close-out C2. See +`docs/changelog-20260715-d120-load-and-iprange-tooling.md`. -- **R-1 / SEC-003:** Vault unseal-key custodian assignment (OPEN). Gates V-2 and closes - `d011-06-vault-unseal` (currently MANUAL). Operator input, recorded off-repo. -- **V-2:** second-person unseal rehearsal (blocked on R-1). -- **R-2 (D-043 caveat):** capi-mgmt-v2 auto-resume exclusion -- rule or accept. -- **R-3:** was D-063 -- now CLOSED. The handoff doc still lists R-3 OPEN; update it. -- **D-1:** Pattern-A full redeploy (VR0 DC0). -## Verify-live queue (batch 3 drafts -- confirm on real cloud before trusting) +The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling +gap register). Read it FIRST. This list is only what that doc does not already carry. -- d011-04 amphora_headroom() field parsing (`server show` .flavor; `hypervisor list --long` - vcpu/ram) -- safe default HOLDs on any parse miss, but confirm the OK path live. -- d011-04 OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior. -- d011-05 needs a FOIL tenant (2nd tenant) for P3 isolation -- onboard one or set VR_FOIL_APPCRED - (acme, the former foil, was offboarded). -- Sequence: run d011-04 non-disruptively first (RR+member), then --include-disruptive once - headroom confirmed. +1. **Stage 2 -- the Office1 HEADEND IS LIVE and D-114 is PROVEN END TO END (2026-07-13).** + `voffice1`: Ubuntu 24.04.4, 16 vCPU / 32 GiB / 600 GiB, **10.10.0.20** (Kea reservation, outside + the pool), egress through the OPNsense edge. On it: **MAAS 3.7.2** region+rack (PostgreSQL 16.14) + and **LXD 5.21.5**, with that LXD **registered into MAAS as an LXD VM host**. MAAS then + **composed, PXE-booted and COMMISSIONED** `office1-netbox` (LXD VIRTUAL-MACHINE, leased + 10.10.1.100 from MAAS, reached status **Ready**, 2 cores / 4096 MiB detected). + **The L3 nesting proof is now DEFINITIVE** -- a guest booted, PXE'd and commissioned three levels + deep, not merely "`/dev/kvm` exists". VR0's `lxd` + `tailscale` pattern, reproduced. + Codified in `scripts/site-headend-install.sh` (+ harness, 18/18) -- reusable for DC1/DC2. + **THE DHCP SPLIT (structural, not conventional):** `10.10.0.0/24` -> **Kea** on the edge (MAAS + dhcp_on=False); `10.10.1.0/24` (`lxdbr0`) -> **MAAS** (dhcp_on=True). Separate L2s; `lxdbr0` is + NEVER bridged onto the site LAN. The install script hard-fails if MAAS DHCP is on any other subnet. + **OFFICE1 IS ESSENTIALLY COMPLETE (2026-07-13).** `office1-netbox` (10.10.1.10) runs **NetBox + 4.6.4**; `office1-tailscale` (10.10.1.11, tailnet 100.64.0.53) is a subnet router advertising + `10.10.0.0/22` on the operator's **SELF-HOSTED** control plane. The edge routes `10.10.1.0/24 -> + 10.10.0.20`. Full connection reference: **`docs/vr1-office1-as-built.md`** -- read that first for + any "how do I reach X" question. + **GitBucket: OUT OF SCOPE (D-116, operator ruling).** No Office1-local GitBucket is built; + `git.baldurkeep.com` stays the git service of record. Office1's composed machines are exactly TWO. + **REMAINING on Office1 (reconciled 2026-07-14):** (a) route enable -- **DONE**, operator enabled + `10.10.0.0/22` on the control server; the edge GUI answers at 10.10.0.1 from the workstation with + no tunnel; (b) import the D-115 carve into the Office1 NetBox -- **DONE** (sandbox seeded from the + upstream draft and PROVEN faithful field-by-field; carve + six-plane roles + BOTH DCs imported, + 90 -> 134 prefixes); (c) the pinned Office1 LAN IPv6 (item 1a) -- **STILL OPEN**, and it is now + the last build item on Office1. + **STAGE 2 IS IN CLOSE-OUT, NOT BUILD. It is also the BRANCH MERGE POINT** (see PINNED). The + close-out checklist C1..C5 is in `docs/dc-dc-deployment-workflow.md`; do not re-derive it here. +1a. **DONE 2026-07-14 -- Office1's LAN IPv6 is LIVE and PROVEN ON THE WIRE. C1 is CLOSED.** + Address `2602:f3e2:f01:100::1/64` on the edge LAN (kernel-verified) + radvd advertising the prefix + with `mode=unmanaged`; `voffice1` autoconfigured `2602:f3e2:f01:100:5054:ff:fe6a:87e5` by SLAAC and + edge->voffice1 GUA ping is 0.0% loss. No collateral damage (v4, Kea DHCPv4, compose-net route all + intact). See `docs/changelog-20260714-office1-lan-ipv6-executed.md`. + **OPERATOR RULING 2026-07-14 -- DO NOT RE-PROPOSE A v6 NAT.** A v4-to-v6 NAT (second OPNsense, or + nginx) was proposed to *simulate* the v6 the lab will not forward, and was WITHDRAWN in favour of + *"configure IPv6 internally as much as possible; configure the IPv6 edge AFTER deployment + completes."* It contradicts **D-101** (*"native GUA routing, no NAT66"*) and **D-115**'s amendment + (upstream's problem, not ours), and a translation box is the one component Roosevelt will NOT have + -- maximum delta, zero proof value. nginx is an L7 proxy, not NAT, and proves nothing about + RA/ICMPv6/Ceph-over-v6. + **TRAP recorded:** `voffice1` has `forwarding=1` + `accept_ra=0` yet STILL autoconfigured -- + netplan/systemd-networkd handles RA in USERSPACE, independent of the kernel sysctl. Never diagnose + "no SLAAC" from the sysctl alone on Ubuntu; read `ip -6 addr` + look for a `proto ra` route. + **The WAN v6 leg remains DEFERRED** -- v6 does not egress the lab (re-measured 2026-07-14: v6 + internet 100% loss, v4 fine). Internet GUA reachability is UNPROVABLE in VR1. -## Logged-not-actioned (small; would vanish at compaction) +1a-historical. **(superseded -- the original pinned item)** deploy Office1's LAN IPv6 (operator ruling 2026-07-13, D-115 + question 3: "register AND deploy dual-stack on Office1 now", sequenced AFTER the NetBox deploy). + Add `2602:f3e2:f01:100::1/64` to the OPNsense edge LAN + RA, and let `voffice1` take its address. + **CORRECTED 2026-07-14 (D-113 AMENDMENT -- the old instruction here was WRONG):** the ADDRESS half + CANNOT be done via the REST API. Measured on the live edge (OPNsense 26.1): `Interfaces > [LAN]` is + the LEGACY page `/interfaces.php?if=lan`; only Devices and Neighbors were migrated to MVC, and there + is **NO REST endpoint for a base interface's `ipaddrv6`**. D-113(a2)'s "interfaces are API-covered" + was INFERRED, never measured, and is false. + - **Address:** `scripts/opnsense-set-interface-v6.sh` (dry by default; harness 40/40). It is NOT a + config.xml push -- it mutates two leaves of the edge's LIVE config through OPNsense's own model, + so API-managed Kea and the firewall rules cannot be clobbered. **WRITTEN AND TESTED, NEVER RUN + WITH `--commit`.** + - **RA:** that half IS API-native -- `/api/radvd/settings/addEntry` via `scripts/opnsense-api.sh`. + Still forbidden either way: a rendered config.xml push. + **KNOWN LIMIT, do not forget it:** IPv6 DOES NOT EGRESS THE LAB (measured 2026-07-13: v6 gateway + reachable, v6 internet 100% loss, v4 fine). So this proves v6 ADDRESSING / RA / services-binding, + which is most of D-101's family matrix -- but NOT internet GUA reachability, which is unprovable + in VR1 until the lab's upstream carries v6. Deploy the LAN leg; DEFER the WAN v6 leg (it would be + a path to nowhere and untestable). See D-115's amendment. +1b. **`10.10.1.0/24` IS NOW A BLESSED ALLOCATION (D-115 ADOPTED).** VR1 Office1 = `10.10.0.0/22` + (Office role, `10.10.0.0/16`): LAN `10.10.0.0/24` + compose `10.10.1.0/24` + 2 spare. Zero + renumber. Also adopted: Edge role `172.30.0.0/16` (legitimises `office1-wan`'s 172.30.1.0/24, + which had been squatting in the Corp OOB block), and **DC2 moved to `10.12.64.0/19`** inside the + Cloud /16 (its old `10.13.0.0/19` sat outside every allocated block). STILL TO DO: write the + carve into NetBox -- production write is operator-gated; the plan is to import into the Office1 + sandbox NetBox first, verify, then feed back. +1c. **(superseded note kept for context)** the original unblessed-allocation finding: Chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, + tenant pool=10.20.0.0/16 per D-016 -- the obvious `10.20.x` would have COLLIDED with tenant space), + but NetBox has not assigned it. Register it once NetBox exists. Known chicken-and-egg: NetBox is + one of the VMs this headend composes. +1c. **`modules/node-vm` nested-virt defect: FIXED 2026-07-13** (operator: fixes land as we find them; + only DC1 *deployment* is gated). Unconditional `cpu = { mode = "host-passthrough" }` -- DC nodes + run `nova-compute` and MUST run KVM guests. Without it they would have come up looking healthy and + been unable to launch a single instance. +2. **NetBox sandbox loop -- DOCUMENTED 2026-07-14 (close-out C4 DONE).** scope-doc section 8 + phase1 runbook Step 10b. (Was: BUILT 2026-07-13/14, undocumented.) + Architecture: draft-source `netbox.baldurkeep.com` (READ ONLY) -> a NEW NetBox on the Office1 VM + (sandbox) -> import draft + planned changes -> simulate VR1 there -> feed validated refinements + BACK. The sim NEVER `--commit`s upstream. **The tooling now exists and is proven:** + `netbox/prod-draft-dump.py` (read-only, no write path at all), `netbox/sandbox-seed.py` + (dry-by-default, structurally REFUSES an upstream URL), `netbox/sandbox-fidelity-check.py` + (field-by-field -- counts alone HID a bug where 17 of 90 prefixes silently lost their region + scope). **STILL PENDING:** capture the loop in `docs/dc-dc-netbox-buildout-scope.md` + the Stage 2 + runbook, and **ensure `office1-netbox` holds the COMPLETE validated dataset** (close-out C2, + REDEFINED by DOCFIX-195: `office1-netbox` is the VR1 IPAM apex (the working draft every consuming + system reads); `netbox.baldurkeep.com` is the production apex, held as a READ-ONLY v1 reference draft, + and the write-back to it is DEFERRED to end-of-deployment -- NOT a Stage 2 gate). + **C2 sub-task (i) DONE 2026-07-15** -- the hardened fidelity check was re-run against `office1-netbox` + (read-only dump via the consolidated sandbox token + an SSH tunnel), and the sandbox is PROVEN + faithful: exit 0, "upstream draft + EXACTLY the planned delta." Along the way the hardened checker's + own scope rule was found to FALSE-FAIL on the two intentionally-unscoped role-container /16s + (`10.10.0.0/16`, `172.30.0.0/16`); fixed to assert scope == INTENDED (now also catches a WRONG-TARGET + binding -- the D-117 near-miss), harness 10->12, gauntlet ALL GREEN (58). SEC-009 gap closed: the + sandbox token was never consolidated; it now lives at `~/vr1-office1-creds/vr1-netbox-sandbox.env`. + See `docs/changelog-20260715-fidelity-check-scope-correctness.md`. + **C2 sub-task (ii) DONE 2026-07-15 -> C2 FULLY CLOSED -> STAGE 2 CLOSE-OUT COMPLETE (C1..C5 all done).** + The tooling gap was closed (`prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover + `ip-ranges`/`ip-addresses`; harnesses +T11/T12 and a new `netbox/d120-compose-bands.py` +16/16 + harness), and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox + (`--commit`, operator-approved) and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned + delta (D-115/D-117/D-118/D-120)." Gauntlet ALL GREEN (60). D-120 Step 6 CLOSED (no apex write). See + `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. **NEXT: the branch->`main` merge** (Stage 2 + close is the merge point, PINNED) -- an operator-gated action, present it, do not run it unasked. +3. **`pynetbox` -- CLOSED 2026-07-13.** Installed (7.0.0) on `office1-netbox`, which is where the + imports actually run. The stdlib tools (`prod-draft-dump`/`sandbox-seed`/`sandbox-fidelity-check`) + deliberately need no dependency at all. + **TRAP, recorded in `platform-traps.md`:** the upstream NetBox sits behind a **User-Agent-filtering + WAF**. `Python-urllib/3.12` gets a **403 that looks exactly like an auth failure**; the same token + works from curl. This affects pynetbox too. +4. **The apparmor rule is now IN THE REPO -- CLOSED 2026-07-13.** + `scripts/prereqs/install-apparmor-libvirt.sh` (+ `install-all` / `check-prereqs` wiring; harness + 24 -> 32 PASS; gauntlet ALL GREEN 52). It grants `<pool-parent>/** rwk,` in + `/etc/apparmor.d/local/abstractions/libvirt-qemu`, which GATES EVERY VR1 VM BOOT: without it a + domain DEFINES but qemu fails "Permission denied" and nothing in the error names AppArmor. + Idempotent, and a true no-op on an already-good host (it does NOT reload apparmor or bounce + libvirtd unless it actually adds the rule). Pool parent defaults to `/var/lib/libvirt/vr1`, + override with `VR1_POOL_PARENT`. `docs/changelog-20260713-apparmor-libvirt-prereq.md`. +5. **The DEFERRED literals are now ALL ASSIGNED -- item CLOSED 2026-07-14.** The operator's Option-B + ruling ("do not invent CIDRs, wait for NetBox") held, and NetBox has now assigned them: + **ORG_ULA_48 = `fd50:840e:74e2::/48`** (**D-118 ADOPTED** -- RFC 4193 valid, and verified NOT to + collide with Tailscale's `fd7a:115c:a1e0::/48`, which matters because `office1-tailscale` runs + `--accept-routes`); GUA per D-117: first DC `2602:f3e2:f02::/48`, second `f03::/48`; the second + DC's v4 = six sequential `/22`s from `10.12.64.0/19` (D-115 moved it INSIDE the Cloud /16). + All are IMPORTED and fidelity-checked in the sandbox. + **CAUTION -- `DC2_V4_SUPERNET` IS A RETIRED NAME (D-117).** The importer now hard-rejects it and + rejects `--dc dc2`. Any doc still saying `10.13.0.0/19` is pre-D-115 and wrong. +6. **D-119 (ADOPTED 2026-07-14) -- THE DC NAMESPACE IS REGION-QUALIFIED. C3/gap #19 CLOSED.** + `vr0-dc0` (VR0's LIVE testcloud) / `vr1-dc0` (VR1's FIRST DC, GUA f02::/48) / `vr1-dc1` (VR1's + SECOND, f03::/48). Bare `dcN` is REJECTED everywhere -- it meant VR0's live cloud in `lib-net.sh` + but VR1's first DC in the importer: one string, two clouds. D-119 **COMPLETES D-117** (executes + its own amendment); it does not reverse it. **ZERO NetBox writes** -- the apex was already right; + the REPO was the odd one out. The importer's DC->site map is now an IDENTITY, so the off-by-one is + STRUCTURALLY IMPOSSIBLE, not merely defended. Also renamed: `vdc1`/`vdc2` -> `vvr1-dc0`/`vvr1-dc1` + (D-114's own DR primitive is `virsh destroy vdc1` -- it read as "destroy VR1 DC1" while MEANING + "destroy VR1 DC0"). Office KEEPS its number (`vr1-off1`, `Office1`, `voffice1`). + Env var: `DC1_/DC2_V4_SUPERNET` -> **`VR1_DC1_V4_SUPERNET`** (both old names rejected by name). + Gauntlet ALL GREEN (57); dc-selector 21->30, prefixes-import 40->82. + **THE `tofu apply` IS NOT DONE -- IT IS GATED.** Plan: 11 to add, 0 to change, 11 to destroy (the + libvirt object NAMES change, which is ForceNew; `moved{}` makes the plan reviewable but cannot + suppress a replace). MEASURED SAFE: all 11 are EMPTY (no guests, no volumes; Stage 3 hasn't run). + Re-plan at execution and STOP if anything shows ATTACHED. + See `docs/changelog-20260714-d119-region-qualified-dc-namespace.md`. -- offboard v2 `--sweep-magnum-orphans` mode (orphan per-cluster trustee in the magnum domain). -- offboard stage-3 app-cred idempotency re-run guard. -- `host_href=None` barbican observation (exercised fine in stage 6; probably closeable). -- DOCFIX candidate: sweep repo for other `-f json ... 2>&1` stderr-merge hazards (DOCFIX-085 class). -- D-050 (keystone policyd override) and the D-063-adjacent `identity:list_trusts=""` hardening - remain PROPOSED/OPEN, not actioned. +6a. **(historical) G5 NetBox site rename -- CLOSED as D-117 (ADOPTED, Option B).** The repo moved to the apex's + 0-indexed convention (first VR1 DC = `vr1-dc0`), rather than bending the apex to the repo. This + fixed a REAL off-by-one: `dc-dc-prefixes-import.py` bound `--dc dc1` to slug `vr1-dc1` while + treating `f02::/48` as dc1 -- but the apex binds `f02` to `VR1 DC0`. It would have written the + first DC's prefixes onto the second DC's site. The importer is now also **dry-by-default**. + **D-117 IS ONLY PARTIALLY EXECUTED** -- see gap #19 / close-out C3: `opentofu/main.tf` and + `lib-net.sh`/`lib-hosts.sh` still speak the OLD naming, and `lib-net.sh`'s `dc0` means **VR0's** + DC0. That collision BLOCKS Stage 3. + G2 (aggregates) and G3 (ULA /48) are both CLOSED by the same work. +7. **DOCFIX-185 follow-up (operator to rule):** a D-100/D-107 amendment note so the Stage-3 DC + edges are built to the per-site-ISP transport model (each site simulates its OWN ISP; dark + fiber is East-West/replication ONLY, never an internet path). +8. `docs/dc-dc-deployment-workflow.md`'s status table was corrected 2026-07-13, but re-read it + against reality before trusting any row -- it has drifted twice. +9. **DHCP PROVEN END TO END -- CLOSED 2026-07-13.** `voffice1` (the first client ever to sit on + `office1-local`) took a real lease from Kea: `10.10.0.100`, hwaddr `52:54:00:6a:87:e5` (matches + its NIC), hostname `voffice1`, state active -- read back through the D-113(a2) REST API. The + service, not just the daemon, now works. +10. **Doc-accuracy finding (measured 2026-07-13, logged not actioned):** `docs/security-ledger.md` + SEC-007 says "D-112(c) makes SSH the only management path". MEASURED on the live edge: the web + GUI IS listening on the LAN side (10.10.0.1:443 HTTPS 200; :80 301-redirects to it), and is + correctly firewalled OFF on the WAN side (172.30.1.2:443/:80 closed). This is OPNsense-default + and not an exposure -- the LAN is the office network, and D-113 kept OPNsense precisely BECAUSE + the GUI is an operator requirement. But "the only management path" is imprecise: SSH is the only + path used by AUTOMATION. Reword SEC-007/D-112(c) so nobody concludes the GUI is disabled. + Reaching the GUI from a workstation requires an SSH tunnel via vcloud (the LAN is a host-local + libvirt bridge, `virbr2`, not routable off-host): `ssh -L 8443:10.10.0.1:443 <user>@10.17.11.248` + -> `https://localhost:8443`. -<!-- END: main-chat --> -<!-- SECTION: jumphost | OWNER: Claude Code jumphost stream only --> -## Jumphost stream -- ops-update-20260705: WINDOW CLOSED 2026-07-05 (addenda 13-16) +## OPEN WORK -- VR0 / DC0 (the running testcloud) -First execution of runbooks/ops-update-procedure.md (DOCFIX-086; addendum 13). -Logged session ops-update-20260705; checkpointed here at each milestone for -cross-stream sync. State as of the Section 2 entry checkpoint: +1. **d011 batch-3 live-validation queue** -- `scripts/checks/d011-04-octavia-lb.sh` and + `d011-05-magnum-e2e.sh` are DRAFTED and mock-tested; the live PASS was recorded in the + 2026-07-06 batch-3 window, but these remain the checks to re-confirm on any rebuild: + d011-04 `amphora_headroom()` field parsing (safe default HOLDs on a parse miss -- confirm the + OK path live); the OCCM LB-name-contains-service-name assumption; d011-05 needs a FOIL tenant + (a 2nd tenant) for its P3 isolation case -- foil1 was offboarded, so onboard one or set + `VR_FOIL_APPCRED`. Sequence: non-disruptive half first, then `--include-disruptive`. +2. **Script backlog item 3 (remainder):** fold `scripts/vault-kv-health.sh` into `cloud-assert.sh` + as a section (cloud-assert has its own A1 vault check but does NOT absorb the KV health script + -- VERIFIED still open), then use the combined gate to regenerate the stale restart-procedure + runbook. +3. **Script backlog item 5:** `scripts/tenant-cluster-create.sh` -- generalize the stage-6 + + watch + D-066-evidence wrapper. NOT STARTED (verified absent). + (Items 6/7/8 -- `keystone-policy-drift.sh`, `cloud-snapshot.sh`, `tests/tenant-acceptance/` -- + are DELIVERED. The old main-chat section called them "not started"; that was FALSE. Reconciled.) +4. **S-class script quality (`docs/script-quality-findings-20260707.md`):** S4/S7/S8a/S8b DONE; + S6 no-action; S8c ruled WON'T-DO. **S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/ + offboard) are HELD** by the operator until the DOCFIX-126/127 live-verify lands. +5. **DOCFIX-126/127 live-verify is still OPEN** -- the offboard-E0 and onboard-cred-file fixes have + never been exercised end-to-end (foil1's offboard predated them). The next REAL onboard/offboard + confirms. Related known gap: onboard cred-write stages 1/2 have no unit coverage. +6. **`test1` orphan user -- approved for deletion, staged, NOT run.** Role-less shell in the + devteam domain, 0 roles, no resources. Exact gated command (RE-VERIFY role-less at execution): + `openstack user delete 9a62f88e331f41adb5c9e41225ec8698`. +7. **The handoff doc is stale:** `docs/handoff-20260703-open-items.md` still lists R-3 (D-063) as + PROPOSED/OPEN. D-063 is `RESOLVED / CLOSED` in `design-decisions.md` (verified). Correct the + handoff doc. +8. **Upstream reports queued (operator):** magnum `can-upgrade-to` anomaly -- Charmhub magnum + 2024.1/stable latest (r101) is s390x-ONLY, no valid amd64 target, so magnum was EXCLUDED from + the update window; report to OpenStack Charmers. (The dashboard-TLS upstream bug was WITHDRAWN + -- operator judged it site misconfig; the draft is retained.) +9. **Barbican `host_href`:** the charm renders a literal `https://None:9312`, suppressing the + upstream request-derived fallback. Benign (catalog unaffected, exercised fine live). A + charm-level fix would need its own D-NNN. Logged, not actioned. +10. **v1-close checklist is QUEUED** (handoff section 6) -- gated on D-011 item 6, which is gated + on the SEC-003 unseal rehearsal. -- **DONE Section 1 (pre-flight):** client 3.6.25 / controller 3.6.24 / all 91 agents - uniformly 3.6.24 -> target 3.6.25. Worklist = 17 apps. Quiesce clean. Pre-change BOM - asbuilt/20260705-102951 committed on a TRUE cloud-assert PASS (required the DOCFIX-087 - A7 false-negative fix, addendum 14 -- window was HELD on the operator's Option B - ruling until it landed). -- **DONE Section 2.1:** controller state backup via `juju create-backup -m - admin/controller` (902MB, checksummed, ~/openstack-baseline/, jumphost-only). - NOTE: controller model is admin/controller, NOT <user>/controller. -- **DONE Section 2.2/2.3:** controller upgraded 3.6.24 -> 3.6.25 (blind window ~1 min, - as expected); post-controller cloud-assert PASS. -- **DONE Section 3:** all 91 openstack-model agents + controller machine at 3.6.25, - states clean (63 idle / 28 started), nothing stuck. -- **DONE Section 4 G0:** keystone 778->817; token probe OK; cloud-assert PASS. -- **DONE Section 4 G1 (11 apps, standing group approval, serial+gated):** - placement 125->154, nova-cloud-controller 795->823, neutron-api 650->710, - neutron-api-plugin-ovn 178->215, glance 642->681, glance-simplestreams-sync - 124->152, octavia-diskimage-retrofit 196->232, cinder 733->820, cinder-ceph - 533->568, barbican 209->265, barbican-vault 75->99. All probes OK; - boundary cloud-assert PASS. -- **DONE Section 4 G2:** octavia 441->542; LBs 2/2 ACTIVE/ONLINE; cloud-assert PASS. -- **DONE Section 4 G3:** dashboards 728->750 / 59->122 / 120->168; D-044 override - INTACT; HTTP+login healthy. RCA: VIP HTTPS dead SINCE DEPLOY (haproxy 443 backend - targets vhost-less internal addr; L4 check masks; phase-03 3.3 check fails open) -- - NOT a G3 regression; G3 stands (operator ruling; addendum 15). Upstream bug + - 2 DOCFIX candidates queued post-window. -- **DONE (coordinated):** vault bundle revert 1.16->1.8/stable (BUNDLEFIX-010) + - D-068 AMENDMENT recorded; 1.16 ruled out (certs V0/V1, Raft-only, Ceph, BUSL); - "off EOL 1.8" remains OPEN. Evidence: docs/D-068-vault-1.8-vs-1.16-analysis.md. -- **DONE Section 4 G4:** nova-compute 827->894; hypervisors up; guests - byte-identical pre/post. -- **DONE Sections 5-6:** post cloud-assert PASS; post BOM asbuilt/20260705-194617 - committed; version coherence exact (91 agents @ 3.6.25); BOM diff = 17 expected - rev pairs + 1 explained metadata delta (cinder secrets-storage endpoint); - appendix-B B.1 fully re-baselined; runbook as-executed corrections DOCFIX-088; - D-071 amended (backup exists). WINDOW COMPLETE. -- **DONE follow-on (addendum 17):** magnum 70->96 + vault 372->714 refreshed after - download-and-diff analysis (mass-rebuild campaign upstream; vault stayed reactive/V0, - handlers byte-identical, never sealed). Fleet FULLY CURRENT (zero can-upgrade-to); - appendix-B re-baselined from asbuilt/20260705-201118. -- **DONE (addendum 18):** D-072 ADOPTED+EXECUTED -- dashboard VIP TLS repaired - (cluster binding -> metal-admin; BUNDLEFIX-011 durable; https 200 first time on - this build). DOCFIX-089 phase-03 3.3 fail-closed + appendix-A entry. Upstream bug - DRAFTED: docs/upstream-bug-draft-dashboard-tls.md. -- **POST-SESSION QUEUE (operator):** submit the upstream bug + record LP number in - D-072; rule on D-071 (still PROPOSED); Vault off-EOL path (Roosevelt-scale, 1.16 - ruled out per D-068 amendment; in-channel 714 applied). -- **Window findings logged for close-out (do not action mid-window):** - 1. magnum can-upgrade-to anomaly REPRODUCED (points at magnum-dashboard-122; - evidence ~/openstack-baseline/magnum-can-upgrade-anomaly-20260705.json) AND - Charmhub magnum 2024.1/stable latest release (r101, 2026-07-01) is s390x-ONLY -- - no valid amd64 target; magnum EXCLUDED; report upstream to OpenStack Charmers. - 2. vault NEWLY offers in-channel 372->714 on 1.8/stable -- NOT refreshed (D-068); - logged only. - 3. `juju create-backup`/`download-backup` EXIST on juju 3.6 -- authoring assumption - ("removed in 3.0, expected absent") is WRONG; correct ops-update-procedure 2.1 - and the D-071 risk section at window close (DOCFIX candidate). - 4. juju status --format=line omits workload messages on 3.6 (root cause of the A7 - false negative; fixed by DOCFIX-087). +## OPEN -- NetBox WRITE-PATH BUGS (found 2026-07-14 by adversarial review; LOGGED, NOT FIXED) -## Jumphost stream -- session 2026-07-05 (post-window resume) +These are independent of D-119's naming work. **DOCFIX-195 (2026-07-15) reclassified WHEN they matter:** +C2 no longer writes the PRODUCTION apex `netbox.baldurkeep.com` (office1-netbox is the VR1 IPAM apex; +the `netbox.baldurkeep.com` write-back is DEFERRED to end-of-deployment). So these bugs gate the DEFERRED merge-back, NOT Stage 2 +close -- they are what would corrupt the production apex during THAT write. They no longer block C2 or +the Stage 2 merge. -- Orientation + reconcile DONE: machine-derived block re-seeded from today's scan - (DOCFIX next-free 090 -> 091 was stale); shared-section stale-facts correction - appended per the handoff reconcile item. Fences validate 4/0/0. -- FINDING (logged, not actioned): `repo_lint.py` L5 "next-free identifiers" info - line counts ANY textual mention of D-/DOCFIX-/BUNDLEFIX-NNN across all files, so - it is inflated by the deliberate high-numbered decoys in the tests/ledger-scan - fixture ("must NOT inflate") and by next-free pointer lines themselves. ledger-scan - (heading-based, pointer-excluding) is the numbering authority and is correct. - DOCFIX candidate: make L5 reuse ledger-scan's exclusion rules or drop the line. - No number consumed. NOTE (2026-07-06 correction): the first version of THIS bullet - quoted the decoy tokens literally and thereby inflated ledger-scan's own - DOCFIX/BUNDLEFIX next-free (docs/ prose is scan-scanned; only "next-free" lines - are excluded). Tokens de-fanged; standing rule: never write identifier-shaped - tokens above the real high-water mark in docs/ or runbooks/ prose. -- Standing by for batch-3 live validation support (verify-live queue). +1. **`roles-aggregates-import.py` still dies HALF-WAY through a production write.** The preflight it + advertises only checks role-NAME collisions. The `ARIN` RIR lookup and `validate_ula_48()` both run + **after** the role-creation loop has already committed 5 roles. A typo'd `ORG_ULA_48`, or an apex + without ARIN, writes 7 objects then dies with **no rollback**. The RIR preflight loop is a literal + no-op (`if ...: continue` + a comment). The harness cannot catch it: it only asserts the STRING + "PREFLIGHT" appears before the first create. **This is the same bug class that already bit us once.** +2. **`sandbox-fidelity-check.py` can return a FALSE GREEN.** It field-compares only the SHARED objects; + the planned delta is checked merely as a SUBSET of `EXPECTED_NEW`. So a prefix created with the + WRONG scope/role/status passes clean -- and if the importer created ZERO of the 36 DC prefixes, + `extra subset-of expected` still holds and it exits 0 saying "nothing lost, nothing stray." + **We have been citing this script as proof the sandbox is faithful. It cannot prove delta + correctness.** It WOULD have caught the historical 17-prefix scope drop (those were shared objects). +3. **`find_role()` dies mid-loop** in `dc-dc-prefixes-import.py` -- roles are resolved lazily INSIDE + the write loop. Against upstream as it stands today (no six-plane roles), a `--commit` creates the + site + 4 prefixes then dies on the 5th. No whole-plan preflight. +4. **Same-dumper blind spot:** `sandbox-fidelity-check` compares two dumps produced by the SAME field + list, so any field the dumper omits is invisible on BOTH sides by construction (prefix `vrf`, + `tenant`, `vlan`, `tags`, `custom_fields`; site `tenant`/`group`/`facility`). Structurally the same + shape as the region-scope bug that dropped 17 prefixes. +5. **Duplicate-CIDR collapse:** prefix-keyed dicts in the seeder and the fidelity check collapse + duplicate prefixes (NetBox permits them, and the importer's own docstring anticipates them vs + `vr0-dc0`). A whole prefix object can vanish with a green check. Latent -- no duplicates upstream today. +6. **The test fake diverges from reality at the failure point:** `fake_pynetbox.get()` returns + `matches[0]` on a multi-match where REAL pynetbox RAISES. So the harness cannot catch #5. -## Jumphost stream -- session 2026-07-06 (operator ruling sweep; addendum 20) +**Recommended sequence: fix 1-3 BEFORE C2.** 1-3 are DONE (2026-07-14, +`docs/changelog-20260714-netbox-write-path-hardening.md`): the roles-importer preflight now covers +ARIN + the ULA, the prefixes-importer preflights every role, the fidelity check is BOTH-bounds + +delta-scope-aware, and the test fake raises like real pynetbox. Gauntlet 58 ALL GREEN. +**CONSEQUENCE TO ACT ON:** the sandbox was declared "proven faithful" (2026-07-14) using the +fidelity check BEFORE it was hardened -- i.e. under the version that could FALSE-GREEN. RE-RUN +`sandbox-fidelity-check.py` against the sandbox under the hardened check before trusting that +"faithful" verdict -- this re-run IS C2 sub-task (i) (DOCFIX-195), run against `office1-netbox` (the +VR1 IPAM apex), NOT against the production apex `netbox.baldurkeep.com`. (Not done this session: it +needs the office1-netbox token, which is operator-held; folds naturally into the C2 step.) +Items 4-6 remain LOGGED not fixed. -- RECORDED: D-050 RESOLVED/CLOSED via D-051 (operator ruling); D-073 ADOPTED - (identity:list_trusts tightened to modern keystone default -- IMPLEMENTATION + - TESTING PENDING, nothing live yet; proposed sequencing: before batch-3 d011-05); - D-072 amendment (upstream dashboard-TLS bug WITHDRAWN -- operator judges site - misconfig, draft retained); SEC-001 -> rotate at v1 close; SEC-003/004 deferred. -- PINNED (operator): D-071 ratification and D-068 off-EOL path choice held until - A1/A2 (batch-3 entry) complete. Magnum can-upgrade-to upstream report pinned. -- A1 foil-tenant onboarding APPROVED in principle; tenant-structure model under - discussion first (SCS Domain Manager persona confirmation + operator/tenant - responsibility split). A2 (d011-04 disruptive gating) held behind A1. -- IN DISCUSSION (B2): OpenBao evaluation; D-068 item 2 (vault_url cleartext http); - D-068 item 3 (AppRole TTL audit + proactive probe) scoping. -- DELIVERED (addendum 21): DOCFIX-091 docs/tenant-onboarding-contract.md per A1 - rulings 1-4 (boundary, roles, intake, dangers, hardening register H1-H5 -- - proposals logged, none implemented). A1.4 interpretation pending operator - confirm: "SSH access into their domain" read as canary access-proof (H3). - Verify-live additions: Horizon manager-role GUI identity probe (contract 5.3). -- RULED (2026-07-06 cont.): A1.3 H1-H5 ADOPTED as pre-v1-close tasks; A1.4 canary - interpretation CONFIRMED + adopted as pre-deployment task. Full onboarding - completion package (scripts + internal docs + client-facing drafts + live - end-to-end validation) added to handoff section 6 v1-close checklist. -- RULED (2026-07-06 cont.): clientdocs/ TOP-LEVEL for client-facing drafts; - build INTERLEAVES with batch-3 (foil onboarding = live workflow validation); - A2 disruptive half PRE-APPROVED conditional on live N+1 headroom PASS. -- DELIVERED (addendum 22): H1 scripts/tenant-assert.sh + 7-case harness (green); - clientdocs/ package DOCFIX-092 (README/intake/welcome/self-service drafts). - Remaining in the interleave before foil onboarding: H2 idempotency guard, - H3 canary stage, D-073 list_trusts zip implementation (then gated live apply). -- DELIVERED (operator-away batch, addenda 23-25): H2 re-run guards + H3 canary - stage7 (harness 12/12); D-073 implementation STAGED (39-rule zip, drift check - PASS -- live apply GATED, block in the D-073 amendment); DOCFIX-093 - runbooks/d011-batch3-window-DRAFT.md (the one-window do-doc: D-073 apply + - foil stages 0-4 + tenant-assert + canary + d011-04/05 + close-out); H5 embedded; - docs/D-068-openbao-assessment-DRAFT.md (B2 research input -- no OpenBao charm - exists, MySQL backend removed upstream, V1 goes to Sunbeam not legacy charms; - research rec: deadline-bound risk acceptance). Stderr-merge sweep DONE: - 10-site inventory logged in addendum 25 (fixes = DOCFIX candidate, not actioned). -- REPO-SIDE QUEUE now EMPTY except: H4 policy-drift script (main-chat backlog - item 6 -- coordinate before building) and the client-docs sweep-on-change rule. - EVERYTHING ELSE waits on the operator-present live window (the do-doc). -- DELIVERED (operator-away batch part 2, addendum 26): DOCFIX-094 stderr-separation - fixes (cloud-assert A5 FAIL-OPEN closed + harness proof; onboard resolvers; - offboard blind-sweep guard exit 20; 3 runbook paste blocks; acceptance:43 rides - its future harness). Vault measurements: charm `channel` + ssl-* options - verified; snap has NO 1.13/1.14 track -- max MPL payload in-charm = 1.12.11; - OpenBao memo updated. Orphan-trustee sweep draft NOT started (queued). -- BLOCKER: juju controller macaroon EXPIRED ~17:10 2026-07-06 -- all juju cmds - prompt for password. OPERATOR: `juju login -u jessea123` at next presence. - Blocked reads queued: keystone policy.json PENDING-LIVE-READ trio; barbican - host_href look. (Vault reads above completed BEFORE expiry.) - CLEARED later 2026-07-06: auth verified working; both queued reads DONE - (see addendum 27 closures). +(Historical rationale: 1 and 3 are partial-write-no-rollback on the production IPAM; 2 is what we would use to prove the write was correct.) -## Jumphost stream -- session 2026-07-06 (cont.): d011-batch3 WINDOW EXECUTED (addendum 27) +## Operator-gated / PINNED (do NOT re-ask; these are rulings, not open questions) -- **WINDOW COMPLETE, all gates green** (logged d011-batch3, per-command wrap -- - operator-ruled Claude Code lane, now in the convention doc as DOCFIX-097): - D-073 APPLIED live (verify pair + H4-live PASS proof); foil1 onboarded - 10.20.29.0/24 (stages 0-4 first-try, tenant-assert PASS 15/15 after fix, - H2 double-SKIP, H3 canary PASS FIP 10.12.7.177/ubuntu); d011-04 PASS both - halves (A2 conditional honored; headroom OK on all 3 hypervisors); d011-05 - PASS (P1/P2/P3 with foil1); baseline + close-out cloud-assert PASS. -- **In-window checker fixes (each harness-proven then re-run live):** - DOCFIX-095 tenant-assert owner-context app-cred probe (admin-side list is - always-403 by base policy admin_or_owner -- measured, NOT a D-073 effect); - DOCFIX-096 d011-04 amphora_headroom --long + display-name hypervisor keys. -- **D-011: batch-3 CLOSED; D-011 CLOSED-except-item-6** (SEC-003 manual unseal - rehearsal). v1-close checklist (handoff section 6) now QUEUED. -- **Queued-read closures:** keystone trio CONFIRMED (D-064 values explicit in - live base policy; overlay comment tags dischargeable -- DOCFIX candidate); - barbican host_href CLOSED benign-with-corrected-mechanism (charm renders - literal https://None:9312, suppressing the upstream request-derived - fallback; catalog unaffected; charm-level fix = future item needing D-NNN). -- **Operator rulings this session:** devteam pinned as next tenant, onboarding - HELD until a cloud snapshot exists (item-7 cloud-snapshot.sh draft launched); - orphan-sweep audit stays rc 0 report-only; clientdocs L7 receipt keeps all - 14 files; H4 design defaults accepted as delivered. -- **Worktree drafts DELIVERED (uncommitted, integration queued):** orphan-trustee - sweep (14/14), clientdocs L7 (17/17), tenant-acceptance harness (16 checks; - closes the last stderr-inventory site), H4 keystone-policy-drift (10/10 + - first live PASS), clientdocs handover pack (lint clean), cloud-snapshot.sh - (in flight). Integration order: after this window commit; tenant-acceptance - + L7 receipt last (receipt must hash post-integration file states). -- **Open (small):** Horizon manual checks on foil1 (operator, browser, - non-blocking); capi-test-1 CREATE_FAILED leftover (orphan-sweep candidate); - beta-cluster lbtest leftover; repo-lint worktree-exclusion DOCFIX candidate. -- **SCRIPT-QUALITY BATCH LANDED (addendum 33, 2026-07-07):** repo-wide sweep - (110 scripts) -> operator-ruled fix scope shipped as DOCFIX-112..117: - stage2 anti-escalation guard fail-closed (SECURITY -- was fail-open both - ways); tenant-offboard hardening cluster (stderr-as-argv, Phase B counter - CLOSED after 2 days logged, LB-wait fails loud, cred-file auth_url); - validate.sh nonstd-exit HOLD; juju-spaces-check macaroon-vs-absent; pipefail - sweep + vault-kv-health rc-test; guard-hook blocklist extension. Gauntlet - 38/38. Deferred S-class + minor R items in - docs/script-quality-findings-20260707.md (operator has NOT ruled S1-S5/S7/S8). +- **BRANCH MERGE POINT = STAGE 2 CLOSE (operator ruling 2026-07-14).** Branch + `dc-dc-stage1-phase0-first-exec` merges to `main` when every Stage 2 gate bullet is MET -- + not at session end, not "when it feels done." The close-out checklist (C1..C5) lives in + `docs/dc-dc-deployment-workflow.md`'s Stage 2 section; that doc is authoritative for what + "closed" means. **DOCFIX-195 (2026-07-15) REDEFINED C2** (the last open C-item): it is no longer an + apex write but "`office1-netbox` holds AND verifies the complete validated dataset"; the + `netbox.baldurkeep.com` write-back is DEFERRED to end-of-deployment. The merge still waits on every + gate bullet MET -- only C2's bar changed. NEW policy: `main` has never taken a feature merge (its only merge commits + are inherited from the pre-rename `openstack-caracal-ipv4` remote, D-110), so ALL VR1 work + -- D-112..D-118, the Office1 headend, the NetBox tooling, 52 commits -- lives only on the + branch. `main`'s tip is two ad-hoc operator uploads; it is NOT trunk-of-record for VR1 yet. + The merge itself is a gated action: present it, do not run it unasked. +- **UPSTREAM APEX WRITE-BACK -- DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15).** + Seeding `netbox.baldurkeep.com` from the validated `office1-netbox` dataset is a MAYBE at the very + end, NOT a Stage 2 gate. Until then the apex stays a READ-ONLY v1 reference draft (no write during + VR1); `office1-netbox` (10.10.1.10) is the VR1 IPAM apex (the working draft every consuming system + reads). The "NetBox WRITE-PATH BUGS" section gates THAT deferred write, not Stage 2 close. Defers the + write; does NOT revoke D-010. +- **D-071** ratification -- held for the dedicated pre-DC-DC Juju controller HA/backup planning + session. Backups now exist (a 902MB controller backup was taken; `juju create-backup` / + `download-backup` DO exist on juju 3.6 -- the original "removed in 3.0" premise was wrong). + Controller HA (single-controller, no-HA) is the unresolved half. +- **D-068** off-EOL Vault path (Roosevelt-scale). 1.16 is RULED OUT; vault stays 1.8/stable rev 714. +- **SEC-001 / SEC-004 / SEC-005 / SEC-007:** rotate / flip-to-private / revoke at v1 close. +- **SEC-003:** custodian assignment + second-person unseal rehearsal -- DEFERRED by the operator. + This keeps `d011-06-vault-unseal` MANUAL and gates the FULL close of D-011. +- **SEC-006** (burned NetBox API token) -- **DEFERRED by operator ruling 2026-07-13 to deployment + completion.** Until then the token is LIVE and EXPOSED. +- **D-076 is NOT this repo's to work.** D-076..D-099 are a RESERVED source-project band, never + assigned here (D-110). The dashboard-policy-override workstream and its uncommitted draft live + in the SOURCE project, not this repo. Operator also pinned it: no D-076 work while devteam are + running their own tests. +- **quota-vs-capacity: CLOSED** -- operator ACCEPTS as-documented (no quota trim). -## Jumphost stream -- session 2026-07-08: devteam cluster-template incident (addendum 39) +--- -- **DIAGNOSED + DELIVERED:** devteam "Unable to create cluster template" = - Public/Hidden flag -> `clustertemplate:publish` (admin_api) refusal, BY - DESIGN; read-only diagnosis, no cloud change. DOCFIX-121 (clientdocs - template-creation guidance + Public/Hidden warning) + DOCFIX-122 - (appendix-A symptom entry); L7 receipt re-recorded; ~/devteam-package - re-rendered (kubernetes.md, self-service-guide.md) for tonight's push. -- **OPEN (operator option, unruled): stage5 for devteam** -- `devteam-k8s` - was never delivered (window ran stages 0-4); one gated command - (`bash scripts/tenant-onboard.sh devteam 5`) closes the docs' delivered- - template promise; else devteam self-creates per new docs. -- **OPEN (operator, small):** role-less GUI test user in the devteam - domain (grant member or delete); Role-dropdown question on the - dashboard-override test (why load-balancer_admin was submitted) -- - evidence folded into the D-076 draft (worktree, uncommitted). -- **TestCluster NoValidHost + decommission audit (addendum 40):** devteam - 1+3 cluster stalled on RAM capacity (NOT quota); operator deleted it, - devteam re-created 1+1 (TestCluster2). QUEUED RULINGS from the audit: - (a) beta-cluster teardown (~17GB reclaim) -- cross-stream, main-chat - owns the resize-acceptance role; (b) lbtest LB leftover reclaim via - kubectl (beta kubeconfig, main-chat asset); (c) foil1 offboard - (RECOMMENDED -- dormant shell); (d) quota-vs-capacity expectation - (trim testcloud quotas or document in clientdocs) -- unruled. +## Standing lessons and traps (load-bearing; these live nowhere else) -## Jumphost stream -- session 2026-07-08 (cont.): Jenkins+K8s guide overnight (addendum 41) +1. **THE EDGE TRAP -- DHCP on Office1 is API-MANAGED.** A full rendered `config.xml` push WOULD + CLOBBER IT. The old scp/install/reboot path is DELETED (template, renderer, ISO builder all + gone -- `opentofu/templates/README.md` is a tombstone). Configure the edge via the REST API + (`scripts/opnsense-api.sh`); mint keys with `scripts/opnsense-bootstrap-apikey.sh`. +2. **"updated in-place" DOES NOT mean "no restart".** A `tofu apply` whose plan says + `libvirt_domain.vm will be updated in-place` **RESTARTED the guest** (measured: uptime 8:36 -> + 6 secs; ~30s with no routing and no DHCP). "In-place" means the RESOURCE is not replaced -- it + says NOTHING about the guest staying up. The dmacvicar/libvirt provider redefines the domain to + apply a disk-list change and BOUNCES it. **Assume ANY apply touching a `libvirt_domain`'s + devices (DC edges, node VMs) is an OUTAGE. Schedule and gate it as one.** +3. **Do not re-implement vendor internals; call the interface.** OPNsense stores API secrets as + `crypt(secret,'$6$')` (SHA-512, EMPTY salt). We COULD mint keys offline and seed them -- we + deliberately DO NOT: it would couple provisioning to a vendor hash format we must keep + byte-compatible forever, and a mismatch FAILS SILENTLY (keys that never authenticate). +4. **A config ISO can NEVER be read on an OPNsense nano image** (D-112). Any instruction to build + one is a trap; the dc-dc-phase2 runbook directed people into it for days. +5. **root's shell on the OPNsense edge is `tcsh`, not `sh`.** A `$(...)` inside a quoted remote + command dies with "Illegal variable name" -- and it dies QUIETLY. This already cost one defect: + a pre-install snapshot silently no-op'd and the install then ran with no snapshot behind it. + **Always feed remote commands to `sh -s`.** (The `scripts/opnsense-apply-config.sh` that was + once wanted here is SUPERSEDED -- that snapshot/scp/install/reboot path is the very config.xml + path D-113(a2) deleted. Do not build it. The tcsh trap still applies to any SSH to the edge.) +6. **Guide changes must sweep the tenant AI skill in the SAME change.** Skill-lag was caught twice + (DOCFIX-135/136) and is now ENFORCED by repo-lint L8 (guide<->skill coupling; the guide's sha256 + is pinned, and any guide edit FAILS lint until re-recorded). +7. **The controller model is `admin/controller`,** not `<user>/controller`. +8. **`juju status --format=line` omits workload messages on 3.6** -- this caused a cloud-assert + false negative (fixed by DOCFIX-087). Don't rely on it. +9. **repo-lint's guards are load-bearing; reword rather than weaken.** L3 (docs must not reference + missing scripts) correctly went RED on the config.xml tombstones -- the tombstones were reworded, + the guard kept. -- **DELIVERED (DOCFIX-123):** new clientdocs/jenkins-kubernetes-guide.md -- - the single end-to-end Jenkins-to-Kubernetes workflow HUB (Pattern A deploy- - into-cluster + Pattern B build-agents; self-create cluster path; kubeconfig- - as-Jenkins-credential; NoValidHost sizing; flannel/Public-Hidden/keypair - traps; troubleshooting; glossary; cross-links). Flannel hole also closed in - tenant-skill kubernetes.md. Guide added to LEAK_FILES (harness change), - README updated, L7 receipt 30 files, gauntlet 38/38. Instantiated to - ~/devteam-package (devteam values). Operator pushes tonight. -- **OPERATOR RULINGS (on return, 2026-07-08 PM) + status:** - 1. devteam cluster: DELETE + DELIVER TEMPLATE ruled. DONE (addendum 42): - TestCluster2 confirmed gone + devteam-k8s calico template delivered. - 2b. Handover package RE-consolidation (operator re-ruling, addenda 46-47): - glossary folded into handover-pack + glossary.md removed + account-table - hard de-dup (DOCFIX-128); tenant AI skill aligned to the consolidated - package -- 3 missing failure signatures added, capacity caveat, Jenkins- - guide integration (DOCFIX-129). Top-level docs 9->8. Package re-instantiated - (glossary appendix in handover-pack, skill refreshed). DONE. - 2. Consolidation review M1-M7: ALL APPROVED. DONE (addendum 43, DOCFIX-124): - M5 error-holes in self-service-guide; M3 entry point + goal-map in welcome; - M1/M2 single-source-of-truth + pointers (softer than hard-delete -- flagged, - protects the AI skill + P2 test); M6 Jenkinsfile path/stage fixes; M4 new - glossary.md. Package re-instantiation for the changed set = next step. - FLAGGED: package handover-pack.md has unfilled {{CUSTODIAN_1/2}} (operator - value; pre-existing, devteam placeholder-custodian test). - 3. Gated script fixes: BOTH DONE (addendum 45). DOCFIX-126 offboard E0 - always-403 no longer FATALs before Phase F (harness case added, 21/21); - DOCFIX-127 onboard cred files now carry dashboard_url (env-overridable, - as-built https://10.12.4.58/horizon). LIVE VERIFY STILL GATED: next real - onboard/offboard confirms end-to-end (onboard cred-write stages 1/2 have - no unit coverage -- pre-existing; logged future harness item). - 4. DOCFIX list CLOSED (operator); handover-pack.md fine as-is (already carries - {{DASHBOARD_URL}}/{{AUTH_URL}}). - 5. HANDOVER READY (operator confirmed 2026-07-08). Two flags ACCEPTED for the - devteam FIRST INTERNAL TEST: (a) package handover-pack.md keeps literal - {{CUSTODIAN_1/2}} -- fine for the test, fill with real names for a real - client handover; (b) existing ~/tenant-devteam cred files predate the - DOCFIX-127 dashboard_url field (written 2026-07-07) -- fine for today, the - dashboard URL is in handover-pack section 3 which the team receives. Repo - fully pushed (HEAD db284fc). REMAINING = operator-manual delivery only: - copy ~/devteam-package + ~/tenant-devteam to the two custodians in person - (guard blocks agent access to cred material). -- **Offboard E0 defect (for the fix):** admin-side - identity:list_application_credentials is always-403 (base admin_or_owner; - measured), so offboard v2 --apply counts 3 false failures and FATALs at exit - 23 BEFORE Phase F (cred-dir retirement). foil1 offboard hit this live; cred - dir retired by hand. Fix: owner-context-or-skip E0 inventory, do not count - known-403 as failure; + harness case. - -## Jumphost stream -- session 2026-07-08 (cont.): devteam self-service cluster live (post-handover) - -- **devteam self-created their first cluster (read-only audit, no cloud change).** - `TestCluster` (uuid 92b92eed-afa3-4aaf-ab33-13d3ddeea869) CREATE_COMPLETE + - HEALTHY (cluster / infrastructure / controlplane / nodegroup all Ready), created - 2026-07-08 17:18 UTC in devteam-prod. 1 master + 1 worker, keypair devteam-key, - api https://10.12.7.130:6443. calico + small-first + correct keypair + no - Public/Hidden flag -- the DOCFIX-121 dashboard symptom resolved IN PRACTICE on - their first real self-service create. (Note: this fresh TestCluster reuses the - name of the earlier deleted TestCluster/TestCluster2, addenda 40/42 -- cosmetic.) -- **Built via HORIZON, not the CLI.** They created their own template - `TestTemplate` (calico, public/hidden False, keypair baked in = devteam-key) in - the dashboard and launched TestCluster from it, rather than the delivered CLI - template devteam-k8s (keypair_id None; measured, correct-by-design, still unused). - Both are valid Magnum objects. -- **What the Horizon path changes (LOGGED, not actioned; DOCFIX candidate, no - number consumed):** - 1. Nothing about the running cluster -- same Magnum API object either way. - 2. Explains the keypair-on-template confusion this session: Horizon's Create - Cluster Template dialog exposes an OPTIONAL Keypair field, so their - GUI-built TestTemplate shows a key while the delivered devteam-k8s (CLI - stage5 omits it by design) shows none. Comparing the two in-GUI drove the - "no keypair -> must be required" read. It is not; the key is supplied at - cluster-create time and their live cluster already carries it. - 3. Doc-gap: jenkins-kubernetes-guide.md is CLI-centric. Their next guide step - (Section 3, kubeconfig fetch via `openstack coe cluster config`) has NO - Horizon path -- the supported retrieval is the CLI. (Earlier draft of this - bullet claimed magnum-ui "offers a cluster-config download" -- UNVERIFIED, - retracted; do not assert it.) The Jenkins integration requires the - CLI/app-cred path regardless. -- **ADDRESSED: DOCFIX-130 (addendum 48, COMMITTED f0f11ae; ~/devteam-package - guide RE-INSTANTIATED 2026-07-08, 34937 bytes, tokens filled from live-measured - AUTH_URL/REGION).** Grew from 2 gaps to a full guide pass on operator - instruction ("fold and add 1-7... make GUI/CLI paths better defined"). - clientdocs/jenkins-kubernetes-guide.md, additive only, no token touched: - - STRUCTURE: new Section 0 dashboard-vs-CLI selector + **On the command line** / - **In the dashboard** markers; dashboard equivalents for create (2.3), resize - (2.5 new), teardown (8). Rule: lifecycle = either path; everything Kubernetes - (kubeconfig onward) = CLI only. - - GAPS 1-7 closed: registry/imagePullSecret (5.0; VERIFIED no registry service - in catalog), ingress-nginx how-to + no-DNS (5.4 new), cluster resize - (2.5 new; VERIFIED syntax), persistent storage self-verify (5.5 new), cert - expiry read one-liner (3), tooling install + kubectl-skew (1); + ImagePullBackOff - and PVC-Pending troubleshooting rows (9). - - Leak harness CAUGHT two self-inflicted client-doc leaks ("Horizon", "magnum") - on first pass -- reworded, now PASS 8/0; gauntlet 38/38. - - ~/devteam-package NOT re-instantiated (refresh at next package sweep). - See the changelog DOCFIX-130 entry. Numbers: DOCFIX next-free now 131. -- **STORAGE OPERATOR FOLLOW-UP (logged, not actioned):** delivered cluster - template has volume_driver unset + no docker volume (MEASURED). Whether the - built cluster gets a default StorageClass from the cinder-CSI default is - UNVERIFIED (verifying = kubectl into devteam's live cluster; deferred for tenant - isolation). If none, persistent volumes are unavailable out of the box -- - candidate template improvement (would need a D-NNN). 5.5 is written self-verifying - so the doc is correct either way. -- **CLEANUP QUEUE RECONCILED against live (2026-07-08 read-only audit) -- LIVE IS - AHEAD OF LEDGER, several "open" items already resolved:** - - foil1 offboard (was RECOMMENDED, addendum-40 c): DONE -- no foil project/user, - no orphan trustee. (Closes the dormant-shell item; also means the gated - DOCFIX-126/127 offboard live-verify was NOT exercised by us -- still unverified - end-to-end, next real offboard confirms.) - - capi-test-1 CREATE_FAILED delete (REMAINING-OPEN item 3, was operator-executes): - DONE -- no cluster record, trustee reaped. - - beta-cluster teardown + lbtest LB (addendum-40 a/b, cross-stream/main-chat): - DONE -- no beta cluster, no stray LB. (Main-chat to log its side.) - - Audit state: only devteam TestCluster (CREATE_COMPLETE) + its kubeapi LB + - its protected trustee remain; orphan sweep ORPHAN=0 PROTECTED=2 UNPROVEN=0; - zero unattached floating IPs. - - STILL OPEN from the addendum-40 audit: only (d) quota-vs-capacity expectation - -- trim testcloud quotas or document the capacity ceiling in clientdocs - (unruled, operator). Everything else in that audit is closed. - -## Jumphost stream -- session 2026-07-08 (cont.): S-class script-quality batch (operator-greenlit) - -Operator greenlit the recommended sequence: (1) Phase A stderr fix, (2) S7+S4, -(3) S8; HOLD S1/S2/S3/S5 (lib-validate adoption on tenant-onboard/offboard) until -the DOCFIX-126/127 live-verify lands. Source: docs/script-quality-findings-20260707.md. - -- **DONE -- DOCFIX-131 (addendum 49): tenant-offboard.sh Phase A stdout-only.** - New `tinv()` (tenant-scoped `inv()` twin) for the 3 Phase A `coe cluster list` - captures; wait loop retries on list-error, final check fails closed. Latent bug: - merged stderr could false-trip exit 21 on an already-clean tenant. Harness un- - excludes coe from warnstderr + new coe-stderr-not-argv assertion (22/22). - Gauntlet 38/38, lint 0 fail, L7 re-recorded. Committed + pushed. -- **DONE -- S7 (DOCFIX-132) + S4 (DOCFIX-133), addendum 50.** S7: removed dead - `CF` (d011-04) + subsumed `vr_is_ipv4` guard (d011-03). S4: three inline trust - filters -> one fixture-tested scripts/trust_filter.py (+tests/trust_filter/ 9/9); - SDIR added to offboard + acceptance. Gauntlet 39 ALL GREEN. Committed + pushed. -- **S8 RE-SCOPED (bigger than its "S" estimate; operator decision needed before I - wire it):** the KEYSTONE_VIP literal (10.12.4.50) lives in 6 files, and 3 of - them (tenant-onboard/offboard/acceptance) do NOT source lib-net.sh -- so - centralizing means adding `. lib-net.sh` sourcing into client-critical scripts I - just modified (DOCFIX-131 + S4), which stacks risk. Breakdown: - - S8a (FIP pool FIP_START/END -> lib-net.sh): DONE (DOCFIX-134). FIP_POOL_START/END - added to lib-net.sh; phase-04 create/verify reference them. Gauntlet 39/39. - - S8b (KEYSTONE_VIP -> lib-net.sh): DEFERRED -- needs sourcing plumbing into 6 - scripts incl. the 3 tenant scripts (onboard/offboard/acceptance) that do NOT - source lib-net. Operator decision needed before wiring client-critical scripts. - - S8c (provider-bundle-check.py restates plane CIDRs): DEFERRED -- design change - (python can't source bash -- needs a generated/parsed form). -- **DONE -- DOCFIX-135 (addendum 51): tenant AI skill registry gap.** devteam asked - (via the assistant) if there's a registry setup step; the DOCFIX-130 registry - content was not mirrored into the skill (grep found zero). Added "Images: bring - your own registry" to references/kubernetes.md + ImagePullBackOff signature to - references/troubleshooting.md; skill harness 8/8, gauntlet 39/39; package skill - files re-instantiated. Committed + pushed. (Skill-lag class; watch for it on - future guide changes.) -- **DONE -- DOCFIX-136 (addendum 52): tenant AI skill FULL SWEEP vs DOCFIX-130.** - Operator asked for a full sweep after the registry gap. Grep-audit found 6 more - guide topics absent; all added concise (pointer to guide for detail): resize - command, kubeconfig cert-expiry (+openssl read), dashboard-vs-CLI split, no-DNS, - NEW Persistent storage section, ServiceAccount pointer; + PVC-Pending and - cert-expired troubleshooting signatures. Skill now covers the full DOCFIX-130 set - (ingress stays recommendation+pointer by design). Harness 8/8, gauntlet 39/39, - package re-instantiated. Committed + pushed. STANDING LESSON: guide changes must - sweep the skill in the same change -- skill-lag caught twice (DOCFIX-135/136). -- **DONE -- DOCFIX-137 (addendum 53): the standing lesson is now ENFORCEABLE.** - repo-lint L8 guide<->skill coupling guard: clientdocs/guide-skill-coupling.txt - pins the guide's sha256; any guide change FAILS lint (naming the skill files to - review) until re-recorded with `bash scripts/repo-lint.sh - --record-guide-skill-coupling`. Harness +5 (28/28), gauntlet 39/39. So the next - guide edit CANNOT land without a conscious skill review. Skill-lag closed at the - source. -- **S-CLASS BATCH STATUS (operator ruled 2026-07-08):** DONE = S7, S4, S8a - (DOCFIX-132/133/134). **S8b APPROVED -- DO IT (next actionable):** centralize - KEYSTONE_VIP into lib-net.sh. **DONE (DOCFIX-138):** KEYSTONE_VIP_DEFAULT in - lib-net.sh; all 6 sites reference it (3 forms: KEYSTONE_VIP, KEYSTONE_HOSTPORT, - acceptance inline); lib-net source added where absent. Gauntlet 39/39, lint 0 fail. - **S8 FULLY DONE (S8a+S8b).** **S8c RULED WON'T-DO** (operator: marginal value for as-built - constants vs bash<->python emit/parse machinery). S1/S2/S3/S5 (lib-validate) - STILL HELD -- operator keeps DOCFIX-126/127 live-verify open for an upcoming - opportunity (devteam actively using resources; not now). S6 no-action. -- **quota-vs-capacity (addendum-40 d): CLOSED -- operator ACCEPTS as-documented** - (DOCFIX-130 sec 2.3/5.5 + skill cover NoValidHost/capacity thoroughly; no quota trim). -- **StorageClass follow-up (DOCFIX-130 sec 5.5): RESOLVED 2026-07-08.** Operator - authorized full read of devteam's cluster (fully internal domain); scoped via - ~/devteam-cluster-openrc (snap needs $HOME for --dir; kubeconfig fetched under - $HOME, used, DELETED). devteam TestCluster HAS a default StorageClass: - `default (default)` provisioner cinder.csi.openstack.org, Retain, - WaitForFirstConsumer, expansion=true. So the delivered template (volume_driver - unset) DOES yield PV support via the cinder-CSI magnum default -- persistent - storage works out of the box; no template change needed. sec 5.5 doc is correct - and self-verifying; OPTIONAL enhancement: state the expected `default` cinder - class (would trigger L8 skill sweep). Cleanup: kubeconfig removed, no residue. -- **D-076 assessment (operator #6: "tested working w/ devteam domain-admin? check - for additional work"):** NOT fully working -- optimistic recollection. Revision 1 - FAILED (full-page 403s + forced logout), reverted. Revision 2 PARTIALLY passed: - identity panels render for the manager persona + session intact, BUT the - create-user GUI flow submits a NON-managed role (load-balancer_admin) whose grant - 403s, stranding a role-less user (flow not atomic). ADDITIONAL WORK before it can - advance: (1) root-cause why the Role dropdown submitted load-balancer_admin - (form default-role preselection vs mis-select); (2) add a per-view SEMANTIC harness - test asserting create-user grants exactly `member` (invisible to the existing 50 - policy pass/fail assertions); (3) clean up the orphaned test user - 9a62f88e331f41adb5c9e41225ec8698 (name `test1`, CONFIRMED still present in devteam, - ZERO roles -- delete or grant member); (4) operator RULING: adopt rev2 (a) vs - close GUI/CLI-only (b) + accept/reject the charmhelpers blacklist-hole reliance; - (5) another operator-present gated window to re-test after rework. NOTE: live - override state UNVERIFIED -- juju macaroon expired (needs `juju login`); confirm - use-policyd-override is reverted to baseline. Draft in worktree agent-a73ec798 - (uncommitted). Clients unaffected: CLI identity path is the documented behavior. -- **D-076 PINNED (operator 2026-07-08):** do NOT work D-076 while devteam are - running their own tests in the env. Deferred until devteam testing completes; - no further D-076 investigation/window until then. Draft stays in worktree. -- **test1 orphan SCHEDULED FOR REMOVAL (operator 2026-07-08):** approved to delete; - staged, NOT run now. Exact gated command (re-verify role-less at execution): - `openstack user delete 9a62f88e331f41adb5c9e41225ec8698` (name test1, devteam - domain, 0 roles measured 2026-07-08). Run at the next convenient window; it is a - role-less shell with no resources, so removal does not affect devteam's tests. - -## REMAINING OPEN WORK (single list for session resume; 2026-07-07, updated post-devteam-window) - -1. **Devteam delivery -- last two steps are OPERATOR-MANUAL (everything - else DONE, addenda 34-35):** intake collected 2026-07-07 (internal - test; placeholder custodians operator-accepted; filed jumphost-local - ~/devteam-intake-record.md, NEVER commit); quota applied+verified - (10/20/51200, runbook standard); pack instantiated to - ~/devteam-package (17 files, tokens verified zero, banners stripped; - templates in clientdocs/ untouched -- L7 receipt intact). REMAINING - (operator): (a) copy the CA into the package: `cp - ~/vault-init/vault-ca-root.pem ~/devteam-package/omega-cloud-ca.pem` - (guard correctly blocks agent access to ~/vault-init); (b) deliver - ~/devteam-package + the ~/tenant-devteam credential files in-person - to the two named custodians. -2. **Horizon contract checks: CLOSED for devteam (2026-07-07 full - three-account walkthrough, addendum 38).** 5.1 VERIFIED (login + - Identity-only view + expected popups); 5.3 verdict: GUI identity probe - FAILS as the contract anticipated -- CLOSED via the CLI-path clause - (DOCFIX-120 contract amendment); D-075 TLS-leg redirect CONFIRMED - (operator browser). Isolation, keypair asymmetry, per-user app-cred - visibility, and OBJECT STORAGE (container create/delete -- first live - proof, closes the acceptance-checklist caveat) all verified in-GUI. - foil1's own 5.1/5.3 rows: surface now proven on devteam; operator may - run or waive foil1's pass (non-blocking either way). -2c. **Dashboard policy override (GUI identity for manager persona): - first live apply FAILED acceptance + REVERTED 2026-07-07; rework IN - FLIGHT (background agent, worktree).** Zip mechanics + PO: activation - worked; panels rendered but page-level checks refused (Domains/ - Projects/Users/Groups "not authorized"); Roles panel popped two errors - then FORCED LOGOUT. Reverted to baseline in one config cycle - (operator-confirmed). Decision draft stays PROPOSED (uncommitted, - agent worktree; next-free D claimed at commit). Next: agent rework - (per-view rule/target mapping, logout root cause, scope-down to - Users+Projects, offline per-view semantic tests), then a second gated - apply in a later window. Clients are unaffected: CLI identity path is - the documented behavior. -2b. **D-075 RULED (option c) + EXECUTED 2026-07-07** (was: root-autoindex - finding): redirect / -> /horizon + Options -Indexes via - conf-enabled/99-omega-root.conf on the dashboard unit. First attempt - (RedirectMatch) caught leaking the backend port in verify and replaced - with per-dir mod_rewrite. Delivered as D-075 entry + DOCFIX-118 - (phase-03 PER-REBUILD block + appendix-A symptom entry). PER-REBUILD - like the D-044 override. -3. **capi-test-1 CREATE_FAILED record delete: RULED YES (2026-07-07), - execution handed to operator** -- the Claude Code permission classifier - (correctly) refuses agent-side deletes of resources it did not create; - exact command staged in-session (uuid fae7ec70-712c-4e58-88dd-da0d7b8c89b2, - measured this window). After it runs: next `tenant-offboard.sh - --sweep-magnum-orphans --audit` reaps its trustee. -4. **S-class consolidation + minor R items:** proposals in - docs/script-quality-findings-20260707.md -- present options to operator - before building (S1-S5/S7/S8 unruled; S6 ruled no-action). -5. **Pinned decisions (do NOT re-ask):** D-071 ratification (pre-DC-DC HA/ - backup session); D-068 off-EOL path (Roosevelt-scale); SEC-001/003/004 - at v1 close; d011-06 stays MANUAL until SEC-003 rehearsal (gates D-011 - FULL close; v1-close checklist QUEUED per handoff section 6). -6. **Cross-stream:** main-chat to reconcile its ledger section against - addenda 27-33 (its backlog items 6/7/8 delivered here; its Verify-live - queue fully cleared; D-050/D-073/D-074 all resolved). - -- **Horizon three-account walkthrough + docs alignment (2026-07-07 cont., - addendum 38):** operator live-tested all three devteam accounts in the - GUI; every observation contract-correct (details in item 2 above). - DOCFIX-119: clientdocs aligned to observed GUI reality (5 files by - background docs agent -- first-login expectations, CLI user-creation - commands, popup signatures, object-storage row de-hedged + acceptance - script aligned; harnesses 8/8 + 47/47; L7 receipt re-recorded) + README - instantiation guidance (DASHBOARD_URL includes webroot). DOCFIX-120: - contract section 5 amended with live verdicts (5.1 verified; 5.3 - closed-by-CLI-clause; stage-7 canary marked IMPLEMENTED). Package - RE-INSTANTIATED to ~/devteam-package from updated templates (incl. - /horizon URL); OPERATOR: re-copy to local + deliver. Dashboard - override attempt/revert logged in item 2c. -- **devteam intake + quota + pack instantiation (2026-07-07 cont., - addendum 35):** intake walkthrough with operator (7 sections; CIDR + - short name pre-settled by rulings); quota set devteam-prod 10/20/51200 - (values equal nova class defaults -- set explicitly per runbook "documents - the record"); first-ever clientdocs pack instantiation: 17 files to - ~/devteam-package via scratchpad script (9 tokens: endpoints MEASURED - in-session -- AUTH_URL https://10.12.4.50:5000/v3, RegionOne; dashboard - https://10.12.4.58 carried from D-072 as-executed verification). - Guard/classifier blocks honored: no ~/vault-init access (CA copy handed - to operator), no curl -k probe. LOGGED-NOT-ACTIONED (DOCFIX candidate): - tenant-skill/references/day2-operations.md line ~7 explains the {{THIS}} - notation in DELIVERED prose, so the literal token survives instantiation; - reword at next clientdocs sweep (L7 receipt re-record required). -- **devteam ONBOARDED (2026-07-07 window ops-devteam-onboard, addendum 34):** - stages 0-4 + tenant-assert (15/15 PASS) + stage7 canary (SSH via FIP - 10.12.7.43, teardown clean) ALL FIRST-TRY on TENANT_CIDR=10.100.0.0/24 - (D-074 ruled fallback -- first live use of the fallback path and first - client-facing workflow execution; ZERO as-executed corrections needed). - Stage-2 anti-escalation guard (DOCFIX-112 batch) first live client run: - admin grant denied as designed. Baseline reused per ruling - (asbuilt/20260706-224851, not stale). Open cloud-assert PASS (A5-A7 held - on first run -- one-shot shell lacked admin scope, known jumphost trait, - re-run sourced) and close cloud-assert PASS. Handover creds staged in - ~/tenant-devteam (0600, jumphost-only). Pack instantiation HELD on intake - answers (item 1 above). capi-test-1 delete handed to operator (item 3). -- **CLIENT PACKAGE RELEASED (addendum 31, 2026-07-07):** operator-requested - polish sweep integrated (DOCFIX-110) + banner ruling applied (DRAFT - removed; the three per-client templates keep TEMPLATE; README says - RELEASED-for-handover). Package is handover-ready pending only per-client - instantiation. In flight: repo-wide script quality sweep (report-only - agent; findings to operator triage next). -- **CLIENT PACKAGE COMPLETE (addendum 30):** starter kit (DOCFIX-108, 6 - deliverables incl. tenancy-audit from the skill delivery; pipeline-smoke + - leftover-check deduped away) + tenant AI skill omega-cloud-tenant - (DOCFIX-109, leakage-gated) integrated; L7 coverage recursive; receipt 29 - files; gauntlet 37/37. REMAINING before handover: operator-requested - polish pass over the final clientdocs set (in flight), then devteam - onboarding next session instantiates the pack. -- **D-074 EXECUTED (addendum 29):** operator ADOPTED the main-chat CIDR - correction PLAN same-day (overlap-allowed tenant CIDRs; client-choice + - 10.100.0.0/24 fallback; capi-mgmt stays reserved; L5 print removed). - Phase 0 gates honored: allow_overlapping_ips=True measured; overlap+Magnum - proof via foil1 (EXACT 10.20.28.0/24 overlap with beta; cluster ~14.5 min; - 2 Ready nodes via API-LB while beta served the same CIDR); teardown clean; - cloud-assert PASS. Guard rewritten (DOCFIX-106, net_overlap.py, harness - 12->17), docs aligned (contract/runbook/intake), D-016 amended, repo_lint - L5 next-free removed (DOCFIX-107). FINDING-doc BUNDLEFIX token de-fanged - (counter was inflated to 052; restored 012). Phase 3 (devteam onboarding, - fallback CIDR) remains NEXT SESSION. Corrections fed back to main-chat via - addendum 29: northwind example fictional; runbook filename wrong in PLAN. -- **INTEGRATED (addendum 28):** all six worktree drafts -> main as - DOCFIX-098 (H4 drift detector), 099/100 (offboard orphan sweep + Phase E0 - guard), 101 (clientdocs handover pack), 102 (cloud-snapshot.sh -- unblocks - the devteam snapshot gate), 103 (tenant-acceptance harness + the last - stderr-inventory fix), 104 (lint L7 sweep-on-change ENFORCED; receipt live, - 17 files). Gauntlet 35/35 on the integrated tree. Main-chat backlog items - 6/7/8 are now DELIVERED (by this stream, operator-assigned) -- main chat to - reconcile its section. -- **devteam gate SATISFIED (2026-07-06):** cloud-snapshot first live runs both - green -- asbuilt/20260706-224851 committed (12 redactions verified; scrubber - caught a real embedded CA private key) + 1.1GB controller backup - jumphost-local (~/openstack-baseline/, NEVER commit). Operator ruled: devteam - onboarding NOT this session -- next session runs stages 0-4 + tenant-assert - + canary on 10.20.30.0/24 (recommended, unruled), then instantiates the - handover pack with dev-team intake answers (7 questions in the DOCFIX-101 - entry). Also still open: foil1 Horizon manual checks (contract 5.1/5.3). - -## Jumphost stream -- session 2026-07-09: new repo `openstack-caracal-dc-dc` first load (repo-rename sweep) - -- **Environment note:** this session's shell is Windows (win32), not the Linux - jumphost (`vopenstack-jesse`) CLAUDE.md describes -- flagged to operator. - `run-tests-all.sh` gauntlet on this box shows 23/39 pre-existing failures - (missing `jq`, tty/exit-code quirks) unrelated to any change made this - session; `repo-lint` (the only suite touched) is ALL PASS standalone and in - the gauntlet. Confirm actual jumphost behavior at next Linux session. -- **DONE -- DOCFIX-139 + DOCFIX-140 (changelog - `docs/changelog-20260709-dc-dc-repo-rename-sweep.md`):** repo-rename sweep - (stale `openstack-caracal-ipv4` path/URL references repaired to - `openstack-caracal-dc-dc` across the skill, 4 runbook RUN blocks, and the - repo_lint.py docstring; clientdocs L7 receipt re-recorded after confirming no - client-facing drift) + a real repo_lint.py bug fix (L1 Windows - path-separator: the `docs/design-decisions.md` legacy-carve-out comparison - used `str(p.relative_to(R))`, which never matches on Windows, so the - documented WARN silently became a FAIL -- fixed to `.as_posix()` at all 3 - affected sites; harness gained T3b covering the carve-out path directly, - 29/29 green). NOT actioned (flagged, larger scope): README.md and - environment.md's body content still describe VR0 DC0/v1 single-DC scope - wholesale -- awaiting operator direction on whether/when to rewrite for VR1 - DC-DC. Uncommitted -- operator has not yet asked to commit. -- **DONE -- DOCFIX-141 (changelog `docs/changelog-20260709-repo-agnostic-sweep.md`):** - operator asked to scope the "make scripts repo-agnostic" idea as its own - sweep + a lint rule, following directly from the DOCFIX-139/140 experience. - New repo-lint check L9 (self-reference guard) FAILS any script/RUN-block/ - skill file that hardcodes the current repo's own directory name; added with - a 5-case harness (34/34 green). Along the way, narrowed `all_text()`'s - `.claude` exclusion from "skip all of `.claude/`" to "skip only - `.claude/worktrees/`" -- the blanket exclusion meant checked-in - `.claude/skills/` and `.claude/hooks/` content was invisible to lint - entirely (why the SKILL.md hardcode from DOCFIX-139 wasn't caught). - Established a `$REPO` session-variable convention (documented in - `runbooks/README.md` Conventions) and converted the 8 remaining hardcoded - clone-path references (SKILL.md + 4 runbooks) to require it explicitly - (fail loud if unset, no inferred fallback, per hard rule 2). environment.md's - repo URL is KEPT (documented anchor fact, same as its other facts) but - tagged with the new `repo-lint: allow-selfref` opt-out. LOGGED, NOT ACTIONED: - L3/L4/L6/L9 only recognize triple-backtick fences for "in code" detection; - `d011-batch3-window-DRAFT.md` (indented pseudo-code) and - `tenant-onboarding-v2-DRAFT.md` (BEGIN/END block comments) use different - conventions and are invisible to these checks -- a pre-existing gap, not - introduced here, DOCFIX candidate (teach the checks the other conventions, - or standardize runbooks onto fences). Uncommitted -- operator has not yet - asked to commit. -- **DONE -- workflow doc + gap register (2026-07-09):** `docs/dc-dc-deployment-workflow.md` - added (Stage 0-7 followable checklist + tooling gap register audited - directly against the repo) + a companion visual Artifact - (https://claude.ai/code/artifact/e752f6dd-23ff-4c59-9ce5-1dedb917e5ea). - COMMITTED + PUSHED (374aef5). -- **DONE -- DOCFIX-142 (changelog `docs/changelog-20260709-opentofu-scaffold.md`):** - first OpenTofu content in the repo (gap register item 2). `opentofu/` - network + storage-pool scaffold: `dc-planes` (six per-DC planes), - `mesh-link` (D-100 dark-fiber triangle), `dc-storage-pool` modules; DC1 + - Office1 wired, DC2 held back pending its CIDR assignment (D-101). Provider - schema (`dmacvicar/libvirt` v0.9.8) verified against its actual current docs - this session, not memory -- differs materially from older common examples - (network isolation is `forward.mode`, nested, not top-level `mode`; domain - resource restructured to ~40 args mirroring raw libvirt XML). Node-VM/volume - resources, OPNsense, DC2 planes, and `tc netem` application DELIBERATELY - DEFERRED -- schema too unfamiliar to author safely without a real - `tofu providers schema` pass; see `opentofu/README.md`. Added - `scripts/opentofu-validate.sh` + `tests/opentofu-validate/` (2/2 green -- - only the guard-clause paths testable without a `tofu` binary, which is not - available anywhere this repo has been worked in this session; the actual - fmt/init/validate path is UNTESTED, logged not hidden). Extended - `repo_lint.py`'s L1 file-extension set to include `.tf` (previously - uncovered since no Terraform/OpenTofu files existed). repo-lint 34/34 - harness green, 0 fail/1 documented warn. Uncommitted -- operator has not yet - asked to commit this piece. -- **NEXT ACTIONABLE STEP (logged):** on a machine with the `tofu` binary + - registry network access, run `scripts/opentofu-validate.sh`, then - `tofu providers schema -json` against the initialized provider, and read the - real `libvirt_domain`/`libvirt_volume` schema directly before authoring the - node-VM module. -- **DONE -- OPNsense gap detail (2026-07-09):** gap register item 4 split into - its 4 separable sub-gaps (network substrate done; VM/image blocked on the - domain module; OPNsense's own routing config not started, no mechanism - chosen; `tc netem` a separate OS-level thing blocked on an unruled D-100 - parameter). Mirrored into `opentofu/README.md` + the visual tracker. - COMMITTED + PUSHED (0bb6cff). -- **DONE -- DOCFIX-143 (changelog `docs/changelog-20260709-repo-lint-crlf-fix.md`):** - fixed the CRLF `write_text` bug flagged during the DOCFIX-141 commit -- - `--record-clientdocs-sweep`/`--record-guide-skill-coupling` now - `write_bytes()` with an explicit ascii-encode instead of relying on - `.gitattributes` eol=lf to clean up CRLF after the fact. Verified 0 CR bytes - on a live re-record; harness 34/34 unchanged; repo-lint 0 fail/1 warn. - COMMITTED + PUSHED (f93dd0d). -- **DONE -- DOCFIX-144 (changelog `docs/changelog-20260709-opentofu-node-vm.md`):** - moved into the domain module as requested. Built `opentofu/modules/node-vm` - (blank-disk PXE-boot MAAS-managed node VM, D-103's pattern for Stage 3) -- - this time from the provider's own real example `.tf` files, not - doc-summarization alone, after last session deferred `libvirt_domain` for - exactly that confidence gap. **Found and fixed a real bug in the - already-committed DOCFIX-142 delivery in the process:** `dc-planes`/ - `mesh-link`/`dc-storage-pool` used classic HCL block syntax - (`domain { ... }`) where the provider's current schema actually needs - attribute-style objects (`domain = { ... }`) -- confirmed once real - provider examples surfaced the convention; doc-summarization alone had NOT - caught this. All three corrected; `opentofu/README.md`'s schema notes now - state the attribute-object convention as a provider-wide assumption. - UNVERIFIED, flagged: the per-device `boot`-order attribute's exact shape - (an inference by pattern). Still deferred: the cloud-init/base-image VM - pattern (Office1 services + OPNsense), DC2 planes, netem. - COMMITTED + PUSHED (3324387). -- **DONE -- DOCFIX-145 (changelog `docs/changelog-20260709-opentofu-cloudinit-vm.md`):** - built `opentofu/modules/base-image` + `modules/cloudinit-vm` (the OTHER VM - pattern, for Office1's own service VMs -- MAAS-region, NetBox, GitBucket). - Same high-confidence source as DOCFIX-144 (`examples/alpine_cloudinit.tf`, - fetched as real code). Threaded the base image's `.path` output across the - module boundary as a real attribute reference rather than reconstructing a - volume path from pool+name strings -- a cleaner design than the earlier - draft, avoiding an unconfirmed-convention guess entirely. - `user_data`/`meta_data`/`network_config` are required inputs with no - default (Office1 VM configs aren't designed yet; a plausible DHCP default - would also silently fail since these planes carry no libvirt DHCP). - **OPNsense applicability explicitly NOT confirmed** -- FreeBSD-based, - cloud-init/NoCloud support not the same guarantee as Linux images, flagged - prominently rather than assumed. Neither module instantiated (no image - source chosen for any service VM). Workflow doc + visual tracker updated. - Uncommitted -- operator has not yet asked to commit this piece. -- **DONE -- OPNsense deployment research (changelog - `docs/changelog-20260709-opnsense-research.md`, no DOCFIX -- research - delivery, matches the design-doc precedent):** operator asked to check - OPNsense's actual documentation. Confirmed cloud-init is genuinely - unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing - the DOCFIX-145 open question. Found the real mechanism instead: the - Configuration Importer (official docs) scans an attached volume for - `/conf/config.xml` in a 2-3s boot window; ISO9660 support was added to it - specifically for VM/cloud automation (a closed, milestone-targeted, - core-dev-assigned GitHub issue) -- mechanically identical to - `cloudinit-vm`'s cdrom-attach shape, different payload. Use the nano image - for KVM (pre-installed, serial-ready), not the installer images. Fully - sourced in `opentofu/README.md`. NOT built yet: a decompress/convert - preprocessing script for the nano image, an ISO9660 config.xml-seed - builder, and the real config.xml content -- itemized as the next - actionable step. COMMITTED + PUSHED (75ca823). -- **DONE -- DOCFIX-146 (changelog `docs/changelog-20260709-opnsense-edge-module.md`):** - operator asked to continue with OPNsense while the research was fresh. - Built all three itemized next-steps: `scripts/opnsense-prep-image.sh` - (nano image decompress+convert, no hardcoded mirror -- `OPNSENSE_MIRROR_BASE` - required), `scripts/opnsense-build-config-iso.sh` (ISO9660 config.xml - seed), and `opentofu/modules/opnsense-edge` wiring both together -- - mechanically identical to `cloudinit-vm`'s cdrom-attach shape, no - `libvirt_cloudinit_disk` involved. Explicit `lan_network_name`/ - `wan_network_name` variables (not an ordered list) to prevent a silent - LAN/WAN swap. Both harnesses green (4/4, 3/3) exercising REAL missing-tool - guards (`qemu-img`/`genisoimage`/`xorriso` are all genuinely absent this - session). Two things flagged, not presented as fact: whether the - Configuration Importer's ISO9660 behavior holds on a real boot, and - whether `devices.interfaces` list order reliably maps to LAN/WAN NIC - enumeration for this guest. `config_iso_path` has no default -- real - config.xml content per site is still undesigned. Not instantiated in root - `main.tf`. `opentofu/README.md` + workflow doc + visual tracker updated to - reflect BUILT (not just researched). COMMITTED + PUSHED (a3ff07f). -- **DONE -- DOCFIX-147 (changelog `docs/changelog-20260709-opentofu-maas-netem.md`):** - operator asked to pull docs and validate before drafting the next modules - -- the two remaining unbuilt mechanisms from the OpenTofu scope - (MAAS VM-host registration, tc netem application). Researched a real - official MAAS provider (`canonical/maas` v2.7.2) and OpenTofu's own - `local-exec`/`terraform_data` docs BEFORE writing any code, per - instruction. Caught a real design mistake before it happened: read both - `maas_vm_host` (register) and `maas_vm_host_machine` (compose -- takes - cores/memory/disks as INPUTS, MAAS's "Compose machine" pod feature) - schemas, cross-checked against D-103's explicit "MAAS does NOT compose new - ones" ruling, and built `modules/maas-vm-host` using ONLY `maas_vm_host` -- - the other resource would have had MAAS and `node-vm` fighting over VM - creation. Also confirmed `local-exec` runs on the machine invoking `tofu - apply` (Office1 per D-103), not the vcloud host where bridges live -- - `modules/netem-link` wraps its `tc qdisc` command in an explicit SSH hop - because of this, using `terraform_data` (OpenTofu's own recommended - replacement for `null_resource`) with a destroy-time provisioner to clean - up. Neither module instantiated (MAAS zone/pool, vcloud SSH target, and - real netem parameters all still pending). `opentofu/README.md` gained a - new sourced research section; workflow doc + visual tracker updated. - COMMITTED + PUSHED (ea276c5). -- **DONE -- DOCFIX-148 (changelog `docs/changelog-20260709-opentofu-audit.md`):** - operator asked to check back over all current modules, pull docs for - anything drafted without it, and review for errors/assumptions. Grepped - every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules and - researched each further rather than re-stating hedges unchanged. **Found - and fixed a genuine documentation error, not just a hedge:** - `opnsense-edge`'s comments claimed interface list-order directly sets - OPNsense's LAN/WAN role -- researching OPNsense's own interface-assignment - model showed role assignment is a SEPARATE, explicit config.xml mapping, - not automatic from device order ("vtnet0=WAN" is a convention some guides - use, not an enforced rule). Comments + both LAN/WAN variable descriptions - corrected; no config.xml has been written yet so this fixes documentation - accuracy for whoever writes it next, not current runtime behavior. - Confidence UPGRADED (not fully resolved) on three items: `node-vm`'s - `boot`-order shape (confirmed the provider 1:1-mirrors libvirt's own XML, - and libvirt's own docs confirm `<boot order='N'/>`'s single-attribute - shape), `create.content.url` accepting local paths (second independent - source), and the `genisoimage` flags (confirmed standard, cleanly - separated from the still-open "does OPNsense's Importer actually read it" - question). Confirmed safe: `maas_vm_host`'s `zone`/`pool` computed-if-unset - behavior. `opentofu/README.md` gained a new "Audit pass" section. Deliberately - did NOT re-research already-well-sourced findings (Importer mechanism, - vm_host-vs-vm_host_machine, the syntax-bug fix) -- targeted only real gaps. - Also added (same delivery): `.claude/skills/openstack-cloud-ops/references/ - opentofu-provider-docs.md` -- indexes every provider/doc source used - building `opentofu/` with exact URLs, confirmed versions, default - branches, AND the fetch methodology that actually worked this session - (Registry doc pages are JS-rendered, use GitHub instead; branch names vary, - check `default_branch` via the API; real example `.tf` files beat - doc-summarized prose for syntax questions). New SKILL.md routing entry; - cross-referenced from `opentofu/README.md` in both directions, no content - duplicated (README.md stays the single source of truth for per-module - findings). COMMITTED + PUSHED (96ecbad). -- **DONE -- DOCFIX-149 (changelog `docs/changelog-20260709-opnsense-config-design.md`):** - operator asked to move into designing the real config.xml content. - Fetched OPNsense's own real, currently-shipped `config.xml.sample` - (`opnsense/core` repo) and its `Route.xml` static-routes model before - drafting. Built `opentofu/templates/opnsense-config.xml.tmpl` - (`{{TOKEN}}`-parameterized, reusing this repo's existing clientdocs - convention) + `scripts/opnsense-render-config.sh`. The real sample fetch - DIRECTLY CONFIRMED the DOCFIX-148 audit finding with an actual example: - ships literal placeholder device names inside each interface's own `<if>` - element, proving LAN/WAN role assignment really is the explicit per-block - mapping DOCFIX-148 concluded. **The renderer is tested END-TO-END, not - just guard clauses** (8/8 -- it needs no external tool, a first among the - opnsense-* scripts) -- and the harness caught a REAL bug before it shipped: - the token `HOSTNAME` collides with bash's own built-in `$HOSTNAME` - variable (`unset` doesn't actually clear it), silently passing a test that - should have failed. Renamed to `OPNSENSE_HOSTNAME` throughout. Full token - legend in `opentofu/templates/README.md`, marking exactly which values - are real (NTP pool, D-106 naming) vs. still pending Stage 0 ratification - (D-100/D-101/D-107) vs. only measurable on a real boot (`vtnetN` - assignment) vs. a security requirement (root password hash must be - freshly generated, never the stock sample's own shipped default). - `opentofu/README.md` + workflow doc + visual tracker updated. Uncommitted - -- operator has not yet asked to commit this piece. -- **DONE -- Stage 0 (decision ratification) RATIFIED 2026-07-09 (changelog - `docs/changelog-20260709-stage0-ratification.md`, DOCFIX-150):** operator - walked through the 6 buildout-design redline items live (AskUserQuestion, - 2 rounds): Ceph size=3-by-default (size=2 only as an explicit Phase-0 - fallback); netem same-metro lean; D-102 folds into D-101, D-109 stays - standalone; metal-admin GAINS a ULA leg (reverses the stated lean -- a - real deviation, not a rubber-stamp); COS per-DC-only confirmed; mirror - sync topology independent-per-DC confirmed. All 11 of D-100..D-110 - individually flipped PROPOSED -> ADOPTED in `docs/design-decisions.md` - (D-102's content preserved under a MERGED-INTO-D-101 status line, per the - append-only discipline -- not deleted). `docs/dc-dc-buildout-design.md` - Section 10 rewritten from "open items" to a ruling record; Section 3's - Ceph gate and the stray "Section D-102" cross-reference corrected to - D-101. **Verified via `ledger-scan.sh`: D-100..D-110 no longer appear - under PROPOSED/OPEN** -- only D-068/D-071 remain (separate, pre-existing, - unrelated to this gate). repo-lint 0 fail/1 documented warn, harness - 34/34. Stage 0 of `docs/dc-dc-deployment-workflow.md` is CLEARED. -- **Operator context, 2026-07-09 evening:** operator stepped away from the - workstation for the night, granted blanket git permission for the - session, and authorized autonomous subagent use to make maximum progress - toward an end-to-end-ready project by morning. Operator clarified this - session runs from the WORKSTATION, not the vopenstack-jesse jumphost -- - tonight's scope is PREP ONLY (docs, scripts, tests, hardened runbooks); - no live infra (juju/MAAS/OpenStack APIs) is reachable or in scope tonight. - Operator will switch to the vcloud host in the morning to run actual - installs/deployments. This explains (and validates) why - `run-tests-all.sh` shows 23 pre-existing failures in this environment -- - all either "jq required" (not installed on the workstation) or - live-cloud-dependent suites (`cloud-assert`, `preflight`, `tenant-*`, - etc.) -- confirmed via `git stash` A/B to be identical with and without - this session's own changes, i.e. an environment gap, not a regression. -- **DONE -- Tooling gap register #1 CLOSED 2026-07-09 ($DC selector - convention, changelog `docs/changelog-20260709-dc-selector-convention.md`, - DOCFIX-151):** added `lib_net_select_dc()` to `scripts/lib-net.sh` - (dc0/dc1 no-op per D-101's inherited-layout ruling; dc2 fails loud -- - gap #3 still open) and `lib_hosts_select_dc()` to `scripts/lib-hosts.sh` - (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment - exists for either yet, an intentional asymmetry vs. the net selector, - documented in-line in both files). Backward compatible by construction -- - no existing caller of either library is affected. New harness - `tests/dc-selector/run-tests.sh`: 21/21 PASS (caught and fixed a real - `pipefail`-vs-nonzero-exit-code bug in the harness itself while writing - it). repo-lint 0 fail/1 documented warn (unchanged); full gauntlet - confirmed via `git stash` A/B to introduce zero new failures. Stage 5's - runbook (not yet authored) still needs to actually CALL - `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` -- the mechanism - exists, the call sites don't yet. -- **DONE -- Tooling gap register #3 mechanism CLOSED 2026-07-09 (NetBox - DC-DC pipeline, changelog `docs/changelog-20260709-netbox-dc-dc-pipeline.md`, - DOCFIX-152):** `netbox/dc-dc-prefixes-import.py` extends the v1 single-site - IPv4-only NetBox import to VR1's two-DC dual-stack model (D-101). DC1's - v4 planes hardcoded (D-101: inherited unchanged, this is decision text, - not an inference); DC2's v4 supernet, the org ULA /48, and the per-DC GUA - carve are all REQUIRED env vars with no defaults -- fails loud rather than - inventing any of them. The per-plane subdivision scheme for the ULA/GUA - blocks is flagged in the script's own docstring as a proposed, unratified - convention pending operator review. New harness - `tests/dc-dc-prefixes-import/` (a fake in-memory NetBox client): 37/37 - PASS at the time (later corrected to 40/40 by DOCFIX-160 -- a dead-code - bug in the harness itself had left 3 happy-path assertions silently - uncounted; see that entry below). Caught and fixed two real authoring - bugs at delivery time: (1) example GUA addresses - in the first draft had host bits set beyond the /40 boundary, caught by - Python's own `ipaddress.ip_network(strict=True)`; (2) the test's own - "count CREATED lines" assertion was inflated by the script's unrelated - site-creation message also containing the word CREATED -- tightened to - match prefix-specific lines. Flagged, not fixed (out of scope): the - existing v1 `ipv4-prefixes-import.py`'s role slugs predate the current - six-plane model in `scripts/lib-net.sh`; this new script uses the CURRENT - slugs instead. repo-lint 0 fail/1 documented warn (also caught and fixed - a CRLF regression introduced by a Python find/replace during test-data - cleanup). Full gauntlet 23/45 FAILED -- confirmed identical pre-existing - environment-gap failures (jq missing / no live juju-cloud access on this - workstation), zero regressions. UNVALIDATED against a real NetBox (none - reachable this session). This closes the TOOLING half of gap #3 only -- - the literals themselves (ULA /48, GUA carve, DC2 supernet) still need - real assignment before this script can be run for real. -- **DONE -- Stage 1 runbook WRITTEN 2026-07-09 (changelog - `docs/changelog-20260709-dc-dc-phase0-runbook.md`, DOCFIX-153):** - `runbooks/dc-dc-phase0-vcloud-prep.md` -- command-level Phase 0 (vcloud - host prep) steps, matching the phase-00 runbook's style (sequence table, - read-only CHECK before every mutation, individually gated mutations, an - explicit gate section, delivery checklist). Surfaces a genuine open - structural question rather than silently resolving it: `opentofu/main.tf`'s - unconditional `provider "maas"` block may force MAAS credentials to be set - at `tofu plan` time even though this stage's plan touches only libvirt - resources -- documented both possible outcomes and what to do for each. - Also flags the VR0/DC0-pool-path-collision risk if the vcloud host is the - same physical machine being repurposed (buildout design's own framing). - NOT YET EXECUTED -- this is the runbook the operator will run tomorrow - morning after switching to the vcloud host; nothing in this delivery - claims a real run happened. repo-lint 0 fail/1 documented warn (unchanged, - prose-only). -- **DONE -- Stages 2, 3, and 5 runbooks WRITTEN 2026-07-09**, three of five - remaining runbooks delegated to parallel subagents (Stages 2, 3, 4, 6, 7) - while Stage 5 was authored directly (the ground-truth/copy-point stage): - - **Stage 2** (changelog `docs/changelog-20260709-dc-dc-phase1-runbook.md`, - DOCFIX-154): `runbooks/dc-dc-phase1-office1-standup.md`. Every VM- - creation step forks explicit Option A (OpenTofu, blocked pending - cloud-init content design)/Option B (manual, logged debt). Found a real, - previously-unnoticed gap: no Office1-local virtual network is modeled - anywhere in `opentofu/`. Caught and fixed a repo-lint L2 false-flag - during review (a correctly-sourced historical D-014 path reference, - reworded onto one line so the rule's own EXEMPT pattern applies, not - deleted). - - **Stage 3** (changelog `docs/changelog-20260709-dc-dc-phase2-runbook.md`, - DOCFIX-155): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`. DC1-first; - DC2 hard-gated (not "later"). Five `!!!` callouts naming real - constraints (Stage 2 dependency, DC2 gate, no-Roosevelt-analog shim, - undecided node sizing, unruled netem params) plus OPNsense tool- - presence check. GATE section concludes "CONDITIONALLY MET AT BEST," - refusing to claim more than is actually true. - - **Stage 5** (changelog `docs/changelog-20260709-dc-dc-phase4-runbook.md`, - DOCFIX-156): `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`. Describes - adapting (not duplicating) `phase-01..08` to run twice, once per DC. - DC1 needs zero VIP/CIDR edits (D-101 inheritance); DC2 blocked on the - same NetBox literals as Stages 3/4. Mandates `lib_net_select_dc "$DC"`/ - `lib_hosts_select_dc "$DC"` before every adapted command block touching - those libraries (DOCFIX-151's first real consumer). Surfaces two - genuine open design gaps rather than inventing around them: the D-101 - IPv6 family-matrix overlay (exact charm config option names - unconfirmed) and D-109's IPv6-SAN cert-issuance charm support - (unverified assumption). - All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED (prep-only - session, no live infrastructure reachable). -- **DONE -- Stages 4, 6, 7 runbooks WRITTEN 2026-07-09 -- ALL SEVEN Stage - 1-7 runbooks now complete, tooling gap register item #9 CLOSED:** - - **Stage 4** (changelog `docs/changelog-20260709-dc-dc-phase3-runbook.md`, - DOCFIX-157): `runbooks/dc-dc-phase3-maas-enlist-deploy.md`. Mandates - `lib_net_select_dc "$DC"` then HONESTLY documents `lib_hosts_select_dc - "$DC"` as expected to fail for both DCs until this stage's own - enrollment populates real host data -- a deliberate chicken-and-egg. - Names 7 real gaps in its own section (no OpenTofu module stands up a - per-DC MAAS rack controller VM; three scripts not `$DC`-parameterized; - node sizing undecided; DC2 blocked at the selector layer; no owner for - the per-DC mirror; VM-host-discovery MAAS status unconfirmed; DC0- - specific tag/fabric literals). - - **Stage 6** (changelog `docs/changelog-20260709-dc-dc-phase5-runbook.md`, - DOCFIX-158): `runbooks/dc-dc-phase5-dr-failover-drill.md`. Found three - real `bundle.yaml` gaps by grepping it directly (no `cinder-backup` - charm, no `ceph-rbd-mirror` charm, `ceph-mon`'s `rbd-mirror` binding on - the wrong plane) and makes fixing them the runbook's own first gated - mutation. Includes a negative-control drill (partition-only, expected - to correctly refuse promotion) before the real hard-down drill. Every - Ceph object name is an explicit placeholder. - - **Stage 7** (changelog `docs/changelog-20260709-dc-dc-phase6-runbook.md`, - DOCFIX-159): `runbooks/dc-dc-phase6-designate-cos-magnum.md`. Quotes - the bundle's actual current "NO designate" text verbatim before - describing the reversal; correctly identifies D-106's reactivation - also reverses the bundle's separate B5 os-public-hostname-dropped - posture (two coupled changes, not one); proposes per-DC hostname - overlays to keep `bundle.yaml` DC-agnostic; quotes D-046's domain-setup - trap verbatim twice; leaves COS's deployment mechanism as an explicit - operator decision point rather than inventing one. - All three: repo-lint 0 fail/1 documented warn; NOT YET EXECUTED. Four new - tooling-gap-register items (#12-15) added for the cross-cutting findings - this whole runbook-authoring pass surfaced (Office1-local network - missing, IPv6 family-matrix overlay missing, D-108 bundle gaps, Stage-4 - MAAS/multi-rack gaps) -- see `docs/dc-dc-deployment-workflow.md`. -- **DONE -- Adversarial review pass across the whole evening's work - (changelog `docs/changelog-20260709-adversarial-review-fixes.md`, - DOCFIX-160):** ran a dedicated fresh-eyes review subagent (no authoring - context) across `lib-net.sh`/`lib-hosts.sh`'s new selectors, - `netbox/dc-dc-prefixes-import.py`, and all seven Stage 1-7 runbooks, - instructed to find real defects and cross-check every claim against - ground truth directly (not trust the runbooks' own self-report) -- - including running both new test harnesses live. Found and fixed 3 real - issues: (1) a genuine Python control-flow bug in `tests/dc-dc-prefixes- - import/test_logic.py`'s `expect_ok()` -- a `return` inside `try` made - the `else` clause unreachable dead code, silently leaving 3 happy-path - assertions uncounted (37/37 was true for what ran, but overstated what - was verified; now 40/40 with the fix, independently re-verified via a - live `python3 -c` reproduction before trusting the agent's claim); (2) a - self-contradictory pair of adjacent comments in `dc-dc-prefixes- - import.py` about /19-vs-/22 arithmetic (no functional bug, just wrong - prose); (3) stale "as of this writing" self-references in the Stage 4 - runbook, written before Stage 3 existed and never reconciled once it - did. Propagated the corrected 40/40 count into `docs/dc-dc-deployment- - workflow.md` and this ledger's own earlier entry (left the original - delivery changelog un-edited as historical record of what was believed - at the time). The review explicitly found NOTHING wrong with: the $DC - selector asymmetry description (consistent everywhere), the hand- - verified ULA/GUA carve arithmetic, Stage 6's `bundle.yaml` claims - (confirmed byte-for-byte), 3+ direct D-NNN quotes (verbatim-accurate), - and DOCFIX/D-NNN numbering consistency across every file -- and found no - inferred/invented literal value anywhere in the reviewed material. - repo-lint 0 fail/1 documented warn; both affected test suites re-run - clean (40/40 and 21/21) after the fixes. -- **DONE -- Final consolidation pass (changelog - `docs/changelog-20260709-final-consolidation.md`, DOCFIX-161), closing - out tonight's autonomous session:** a last full read-through of - `docs/dc-dc-deployment-workflow.md` for internal cross-stage - consistency found ONE MORE real gap -- no stage explicitly owns creating - Office1's own OPNsense edge VM, despite the topology being explicit that - the design is per-site (three sites, three edges: DC1, DC2, AND - Office1) -- logged as gap #16, not silently decided. Also fixed two - stale cross-references left over from earlier in the same evening (the - design-doc summary row's confusing "DONE, PROPOSED" phrasing; the - lib-net/lib-hosts row's "Stage 5 runbook still needs to call it," now - true of both Stage 4 and Stage 5). Companion Artifact redeployed twice - more: all seven stage pills moved to a new "Runbook written" state (a - 4th click-cycle state added, distinct from not-started/in-progress/ - done), gap register items 1/3/9 marked closed and 12-16 added, and the - 37/37-to-40/40 test-count correction propagated in. Final full gauntlet - re-run: same 23 pre-existing environment-gap failures as every earlier - check tonight (jq missing / no live juju-cloud access on this - workstation) -- zero regressions across the entire evening's cumulative - changes. repo-lint 0 fail/1 documented warn. - **Session summary for the operator's morning read:** 11 commits - (DOCFIX-151 through DOCFIX-161) delivered gap #1 (closed), gap #3's - tooling half (closed, data half open), and all seven Stage 1-7 runbooks - (gap #9 closed), plus an adversarial review pass and this consolidation. - Nothing was executed against real infrastructure -- prep-only session, - as the operator specified. The morning's first real action is Stage 1's - runbook (`runbooks/dc-dc-phase0-vcloud-prep.md`) on the actual vcloud - host. Genuinely open blockers, roughly in the order they'll be hit: gap - #7 (MTU/Ceph disk-budget arithmetic), gap #16 (Office1 OPNsense edge - ownership), gap #3's data half (NetBox literals), gap #2's UNVALIDATED - status (no `tofu` binary has run yet), gaps #12-15 (Office1-local - network, IPv6 family-matrix overlay, bundle.yaml DR gaps, MAAS/ - multi-rack gaps). -- **DONE -- Tooling gap register item #7 CLOSED 2026-07-09 (MTU/geneve + - Ceph disk-budget calculators, changelog `docs/changelog-20260709-mtu- - ceph-budget-calculators.md`, DOCFIX-162 -- numbering deferred to the - orchestrating session to avoid a collision with other parallel - in-flight agents tonight):** `scripts/dc-dc-mtu-geneve-budget.sh` - (required `--underlay-mtu`, no default) reproduces D-101's own worked - example exactly (1500 - 56 = 1444), leaves tenant MTU at 1500 for a - measured jumbo (>=9000) underlay, and adds a clearly-separated - IPv6-minimum-link-MTU (1280, RFC 8200) sanity floor beyond D-101's own - text. `scripts/dc-dc-ceph-disk-budget.sh` (required `--total-disk`, - per-DC `--dcN-nodes`/`--dcN-per-node-osd`, and required - `--backup-overhead-fraction` with deliberately NO default -- not a hard - number anywhere in this repo, must come from real DC1 Ceph/radosgw-admin - measurement) computes whether size=3 across DC1+DC2 plus overhead fits - the measured total disk, and if not, names size=2 as the ONLY documented - fallback (D-101/Section 3) without silently applying it -- an explicit - operator-logged decision. New harnesses: `tests/dc-dc-mtu-geneve-budget/ - run-tests.sh` 19/19 PASS, `tests/dc-dc-ceph-disk-budget/run-tests.sh` - 16/16 PASS -- both cover the exact worked/constructed numeric examples, - the jumbo/fits vs fallback branches, and every required-arg/bad-format - fail-loud path. `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy - warn (unchanged). Both scripts are read-only calculators (mutate - nothing) and have not yet been run against the real vcloud host -- that - first real run is part of executing `runbooks/dc-dc-phase0-vcloud- - prep.md` Step 3. Uncommitted -- operator has not yet asked to commit - this piece; the orchestrating session will assign the real DOCFIX number - and find-replace `DOCFIX-162` at integration time. -- **DONE -- Tooling gap register items #12 and #16 CLOSED 2026-07-09 - (changelog `docs/changelog-20260709-office1-network-edge.md`, - DOCFIX-163 -- numbering deferred to the orchestrating session, same - reason as the #7 item above):** built `opentofu/modules/office1-network` - (a new, dc-planes-shaped isolated `libvirt_network` sized for ONE network, - not six planes -- closes gap #12) and INSTANTIATED it for real in root - `main.tf` (`module "office1_network"`, needs no unmeasured value beyond - already-real `domain_suffix`/`underlay_mtu`). Design decision: a new - module, deliberately NOT a reused host bridge -- a bridge would be the one - network in this topology living outside OpenTofu's D-103-mandated - inventory and would silently break this repo's own `virsh net-list --all` - completeness assumption; full reasoning in the module's own header and in - `runbooks/dc-dc-phase1-office1-standup.md`'s Open question #1 (now marked - RESOLVED). Also decided gap #16 (Office1 OPNsense edge ownership): YES, - Office1 gets its own `modules/opnsense-edge` call, a fourth site alongside - DC1/DC2, and Stage 2's own runbook owns creating it (new Step 4b, - CHECK-before-MUTATION shape, its own GATE bullet) -- Stage 2's whole scope - is already "Office1 headend standup" and Stage 1 already builds this - edge's network neighbors, so Stage 3 has no reason to reach back into - Office1's own infrastructure. A commented-out `module "office1_opnsense"` - skeleton now exists in `main.tf`, mirroring Stage 3's `dc1_opnsense` - template exactly; its `lan_network_name` resolves cleanly to the new - network, but its `wan_network_name` does not -- investigated rather than - invented, and found to be a genuinely DEEPER, cross-site gap: Stage 3's - own `dc1_opnsense` template carries the identical unresolved - `wan_network_name` placeholder, confirming no dedicated per-site - ISP-uplink/WAN network exists yet for ANY site (DC1/DC2 included). Logged - as NEW tooling gap register item #17 rather than defaulted to a mesh-link - network, which would have directly contradicted D-100's own sub-item - ruling that the three mesh legs carry management traffic only. `bash - scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn (unchanged); - `opentofu/` remains UNVALIDATED overall (no `tofu` binary available this - session, same standing status as every other module here -- not - newly introduced by this delivery). Uncommitted -- operator has not yet - asked to commit; the orchestrating session will assign the real DOCFIX - number and find-replace `DOCFIX-163` at integration time. -- **DONE (uncommitted) -- Gap #13 (D-101 IPv6 family-matrix overlay) - DRAFTED, one real risk left open, researched directly by the - orchestrating session (not delegated) given the accuracy stakes:** - `overlays/dc-dc-ipv6-family-matrix.yaml` + `docs/dc-dc-ipv6-charm- - research.md` (full sourcing). Fetched REAL charm config/source, not - memory: `charm-keystone/config.yaml` confirms `prefer-ipv6` (bool) is a - real, shared charms.openstack option; `charm-nova-cloud-controller`'s - actual "Dual Stack VIPs" commit confirms it's genuinely ADDITIVE (HAProxy - binds both v4 and v6 wildcard addresses simultaneously), not an - either/or switch -- resolving the biggest open question about whether - D-101's dual-stack ambition is even mechanically possible. `charm-ceph- - mon/config.yaml` confirms its OWN separate `prefer-ipv6` + - `ceph-public-network`/`ceph-cluster-network`, which per the charm's own - docs IS a straight switch (not additive) -- the correct shape for the - ULA-only storage/replication planes. Fetched the FULL `charm-layer- - ovn/config.yaml` (25 options) and confirmed NO IPv6/encapsulation option - exists at all -- OVN needs no overlay entry, geneve family follows the - bound interface once the plane is ULA-only. Fetched `charm-vault`'s - actual `vault_pki.py` `sort_sans()` code and confirmed no IPv4/IPv6 - distinction in cert-issuance -- D-109's IPv6-SAN requirement is - code-confirmed, not assumed. **Found a real, open upstream risk while - researching, not invented or glossed over:** Octavia's `lb-mgmt-net` - IPv6 support has two real, still-referenced Launchpad bugs (#1911788, - #1913409) describing failures -- #1911788's root cause is an OVN/LXD/ - MAAS hostname-resolution mismatch, the SAME CLASS of problem this repo's - own D-008 bootstrap order already hardens against, so not necessarily - fatal, but genuinely unresolved. The overlay deliberately excludes an - Octavia entry rather than silently forcing the risk through. Updated - Stage 5's runbook (`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`) Step 6 - and the tooling gap register item #13 to reflect all of this. `bash - scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. Changelog: - `docs/changelog-20260709-ipv6-family-matrix-overlay.md`, **DOCFIX-164** - (number confirmed and closed out here -- this line previously ended - with unresolved "will assign at integration time" boilerplate; caught by - the full-project-sweep's DC-DC-runbook-consistency agent as an - integration miss, fixed 2026-07-10). -- **DONE -- `$DC` parameterization for the 3 MAAS scripts (changelog - `docs/changelog-20260709-maas-scripts-dc-param.md`, DOCFIX-166), - closing gap #15 sub-item 2's CLI-invocation half:** `scripts/reenroll- - hosts.sh`, `scripts/carve-host-interfaces.sh`, and `scripts/phase-00-maas- - standup.sh` now accept an opt-in `$DC` env var and call `lib_net_select_dc` - / `lib_hosts_select_dc` immediately after sourcing, matching the same - convention (and the same dc1-network-no-op/dc1-hosts-fail asymmetry) - DOCFIX-151 gave the two library files the night before. Unset `$DC` is - provably byte-for-byte unchanged (verified by direct, jq-independent - invocation of all three scripts across the unset/dc0/dc1/dc2/bogus - matrix, since the selector calls sit before any MAAS/jq touch-point in - every script). Added `$DC` test cases to both existing harnesses - (`carve-host-interfaces`, `phase-00-maas-standup`) and created a new one - (`reenroll-hosts` had none before). Did NOT invent any per-DC host/CIDR - literal, and did NOT make `phase-00-maas-standup.sh`'s hardcoded `PLANES` - table dc2-aware -- both remain honestly open (dc1 only "works" there - because D-101 makes its real target identical to dc0's). Did not run a - `git stash` A/B this time since the working tree carries other parallel - agents' in-flight uncommitted work tonight; verified directly instead. - repo-lint 0 fail/1 documented warn. `docs/dc-dc-deployment-workflow.md` - item #15 and `runbooks/dc-dc-phase3-maas-enlist-deploy.md` (gap #2, Steps - 5 and 8, delivery checklist) updated to reflect the partial closure -- - six of the seven gap-#15 sub-findings remain open, said plainly rather - than overclaiming. No live infrastructure touched. (DOCFIX-166, already - cited above -- this closing line's stale "will assign at integration - time" boilerplate corrected 2026-07-10, flagged by the same sweep pass.) -- **DONE (uncommitted) -- Gaps #6 + #14 (bundle.yaml: Designate reactivation - + D-108 DR mechanism), done directly by the orchestrating session given - the stakes (this is the SAME `bundle.yaml` the jumphost operates the - live VR0/DC0 rehearsal cloud with, not a DC-DC-only file):** - `bundle.yaml` gained `designate`/`designate-bind`/`designate-mysql- - router`/`designate-hacluster` (D-106) and `cinder-backup`/`ceph-rbd- - mirror` (D-108) + `ceph-mon`'s `rbd-mirror` binding corrected from - `storage` to `replication`. **Caught and fixed three real mistakes - during self-review, by fetching each new charm's REAL metadata.yaml - rather than trusting the standard-pattern draft or the Stage 7 runbook's - own earlier "template" section:** (1) designate's real provides-side - endpoint is `dnsaas`, not `public` like the other API charms; (2) - `cinder-backup` is a SUBORDINATE charm (confirmed `subordinate: true`) - -- the first draft wrongly gave it `num_units`/`to`/an `amqp` relation - that doesn't exist on this charm at all; (3) `ceph-rbd-mirror` is a - principal with `ceph-local`/`ceph-remote` requires endpoints (confirmed - `subordinate: false`) -- the first draft invented a single wrong binding - key. **A genuine, unresolved cross-DC wiring limit documented rather - than worked around:** `ceph-remote` (the peer DC's ceph-mon) cannot be - a same-bundle relation given D-104's per-DC-independent-Juju-controller - design -- only `ceph-local` (this DC's own leg) is wired; the cross-DC - peer relationship needs `juju offer`/`consume` or the manual `rbd - mirror pool peer bootstrap` CLI flow Stage 6's runbook already - documents. Deliberately did NOT reactivate `os-public-hostname` on the - 11 API charms in this shared bundle (D-106's OTHER coupled change) -- - pushed to a proposed per-DC overlay instead, to keep `bundle.yaml` - itself DC-agnostic. Verified: `python3 -c "import yaml; yaml.safe_load( - open('bundle.yaml'))"` parses clean (56 apps, 108 relations); `python3 - scripts/provider-bundle-check.py` PASS, all 6 existing invariants - unaffected (`tests/provider-bundle-check/`: 8/8 unaffected); `bash - scripts/repo-lint.sh` 0 fail/1 documented warn. Reconciled Stage 6's and - Stage 7's runbooks to point at the now-real bundle content instead of - their earlier "template"/"known gap" framing. NOT applied to any live - model this session -- no live cloud reachable; this is a repo-source - change only, to be reviewed via `juju deploy --dry-run` before any real - apply. Changelog `docs/changelog-20260709-designate-cinderbackup- - rbdmirror.md`, **DOCFIX-167** (number confirmed and closed out here -- - this line previously ended with unresolved "will assign at integration - time" boilerplate; caught as an integration miss by the full-project - sweep's DC-DC-runbook-consistency agent, fixed 2026-07-10). -- **DONE (uncommitted) -- Tooling gap register item #5 MECHANISM CLOSED - 2026-07-09 (changelog `docs/changelog-20260709-ceph-replication- - tooling.md`, DOCFIX-165 -- numbering deferred to the orchestrating - session, same reason as the other parallel deliveries tonight):** the - radosgw multisite / rbd-mirror command sequences already written out in - `runbooks/dc-dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and - 11.2-11.4 (verbatim upstream Ceph administration, not invented) are now - real, callable scripts: `scripts/dc-dc-radosgw-multisite.sh` - (`master-init`/`join-readonly`/`enable-two-way`), `scripts/dc-dc-rbd- - mirror.sh` (`bootstrap-primary`/`bootstrap-secondary --direction - rx-only|rx-tx`), and `scripts/dc-dc-dr-drill.sh` (`failover`/`failback`). - Every realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument - -- no invented defaults, matching the runbook's own `<UPPER_SNAKE_TBD>` - placeholder discipline. Default mode is `--dry-run` (prints the exact - `juju ssh`/`juju run` command sequence, secrets redacted); `--apply` - executes one command at a time, aborting on first failure. **The - single most safety-critical property in the whole drill -- demote the - CURRENT PRIMARY before promoting the RECOVERING side during failback -- - is HARD-CODED as fixed call order in `dc-dc-dr-drill.sh`'s `failback` - subcommand**, not left to documentation or an operator-orderable flag; - in apply mode the promote step is only ever reached if the preceding - demote succeeded. Step 10.2's per-image Glance re-registration is always - printed as a flagged MANUAL reminder and NEVER auto-executed (the - runbook is explicit the real invocation depends on unconfirmed Glance - driver behavior); Step 11.3's Cinder reconciliation is likewise not - scripted (tenant/volume-specific). Each script sources `scripts/lib- - net.sh` and calls `lib_net_select_dc` per DC token touched (DOCFIX-151 - convention) -- informational in dry-run (a plan preview is not a - mutation), hard-blocking before any `--apply` (dc2 --apply refused - today, exit 3, since D-101/gap #3 leaves DC2 with no assigned network - literals). New harnesses `tests/dc-dc-radosgw-multisite/`, - `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23 - PASS) test ONLY argument parsing, guard clauses, and dry-run output -- - including a line-order grep proving the failback demote-before-promote - property -- NOT real radosgw/rbd-mirror behavior; no live Ceph cluster - exists this session to test against (same UNVALIDATED posture as every - other tooling delivery tonight). Stage 6 runbook's own "Follow-up, NOT - built now" delivery-checklist item and `docs/dc-dc-deployment- - workflow.md` item #5 both updated to point at these scripts instead of - describing them as future work. `bash scripts/repo-lint.sh`: 0 fail, 1 - documented legacy warn (unchanged). No live infrastructure touched. - Uncommitted -- orchestrating session will assign the real DOCFIX number - and find-replace `DOCFIX-165` at integration time. -- **DONE (uncommitted) -- Gaps #4(d)/#11: proposed netem parameters + ULA/ - GUA/DC2-supernet generation guidance (changelog `docs/changelog- - 20260709-netem-ula-gua-proposal.md`, DOCFIX-168), `docs/dc-dc-netem-and-ula-gua- - proposal.md`:** does NOT rule the netem decision or generate any of the - three real NetBox literals itself -- presents a concrete, reasoned - proposal (1ms delay / 0.2ms jitter / 0.01% loss / uncapped rate, each - with rationale grounded in the buildout design's own "same-metro dark - fiber, low single-digit ms" lean) for OPERATOR ratification, and gives - the exact safe command (`openssl rand -hex 5`, RFC 4193-compliant) for - the operator to generate the real org ULA /48 themselves. Explicitly - does NOT attempt to generate the per-DC GUA carve or DC2's supernet -- - both require real coordination (the ARIN-block administrator; NetBox's - full address inventory) this session has no authority or visibility to - perform; the section instead explains precisely what each requires and - how to close it once ready. `docs/dc-dc-deployment-workflow.md` gap - items #4(d)/#11 updated to reference it; item #10 (harness debt) also - corrected -- LARGELY CLOSED given how many new test suites landed - tonight (dc-selector, dc-dc-prefixes-import, dc-dc-mtu-geneve-budget, - dc-dc-ceph-disk-budget, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, - dc-dc-dr-drill, plus MAAS script additions). `bash scripts/repo-lint.sh`: - 0 fail, 1 documented legacy warn. No live infrastructure touched. - Uncommitted -- orchestrating session will assign the real DOCFIX number - at integration time. -- **DONE -- Full-project sweep for errors/architectural problems, requested - after the seven gap-closure commits: FIRST finding, fixed (changelog - `docs/changelog-20260710-phase01-bundle-staleness-fix.md`, DOCFIX-169):** - `runbooks/phase-01-bundle-deploy.md` -- an EXISTING, LIVE-CLOUD-relevant - VR0 runbook, not a DC-DC artifact -- had gone stale the moment DOCFIX-167 - edited the SAME shared `bundle.yaml` it deploys. Its "expected plan: 50 - apps, 97 relations" (Constants section + Step 1.2's dry-run GATE) was - wrong (real count now 56/108); its GATE's "NO designate (D-019)" text - flatly contradicted the new reality. Fixed both LIVE-gate references; - deliberately left the historical "As-built reference (2026-06-03)" - section's own `50 apps/97 relations` line UNCHANGED (append-only audit- - trail discipline -- it documents a real past deploy, not something to - retroactively edit). Flagged a real, unresolved architectural tension - rather than silently picking a side: this bundle now serves BOTH a plain - VR0-only redeploy AND standing up DC1, and reactivating Designate/the DR - mechanism in the shared file means any future `phase-01` run gets them - regardless of intent -- a future plain-VR0 redeploy needing to exclude - them would need an explicit override, not an assumption either way. - Extended the EXIT GATE's "waiting on vault certs" list to include - designate/designate-bind (inferred from their real `certificates` - relation to vault, clearly labeled NOT YET OBSERVED against a live - deploy). `bash scripts/repo-lint.sh`: 0 fail, 1 documented legacy warn. - Sweep continues across the rest of the repo (scripts, other runbooks, - OpenTofu modules, test quality, security/secrets posture). -- **DONE -- Full-project sweep COMPLETE: 8 parallel agents (bundle.yaml, - VR0 runbooks, scripts, OpenTofu modules, DC-DC runbook consistency, test - harness quality, security/secrets, NetBox/IPAM), all findings triaged - and real ones fixed, changelogs DOCFIX-170/171/172/173:** - - **DOCFIX-170** (`docs/changelog-20260710-sweep-vr0-runbook-opentofu- - fixes.md`): `phase-08-workload-cluster-acceptance.md`'s Designate - framing (same cascading-staleness class as DOCFIX-169), nuanced not a - blanket flip -- charms now active/idle-checkable, but D-011.8's real - resolution criterion still needs Stage 7's os-public-hostname overlay, - not automatic from deployment alone. Plus a stale `opentofu/modules/ - base-image/outputs.tf` comment (cosmetic, no functional effect). - - **DOCFIX-171** (`docs/changelog-20260710-sweep-dcdc-runbook-fixes.md`): - Stage 1's calculator scripts (DOCFIX-162) never wired into its own - Step 3 -- fixed to actually invoke them; Stage 5's Sequence-list still - said `[NOT YET WRITTEN]` for the IPv6 overlay after the Step 6 body - was updated -- fixed; Stage 6's checklist claimed its steps call the - new Ceph scripts when the step bodies still only showed raw commands - -- added real "PREFER `scripts/dc-dc-...`" pointers to Steps 4, 5, 8, - 10.2, 11.2 themselves, closing the gap between claim and text. Also - closed out two orphaned ledger entries (DOCFIX-164, 167) whose closing - lines never got their real number written in during the earlier - integration pass -- an integration miss the sweep itself caught. - - **DOCFIX-172** (`docs/changelog-20260710-sweep-bundle-fixes.md`): - `magnum-dashboard` had NO `bindings:` block AT ALL -- a real, - pre-existing bug (confirmed via `git log`, present since the D-052 - explicit-bindings commit, not tonight's regression) violating this - bundle's own stated "explicit per-application blocks" discipline. - Fixed with a minimal, metadata-confirmed block (subordinate, only a - real `dashboard` endpoint -- NOT a copy of `octavia-dashboard`'s - fuller shape, which has a `certificates` endpoint this charm doesn't). - Also relocated two section-header comments that sat before the WRONG - app's bindings block (functionally harmless, confirmed via - `yaml.safe_load` -- purely a future-edit-risk fix) and corrected the - header's stale "11 API charms" VIP-chain count to 12 (with the - VIP-triple count of 11 correctly left alone, since designate doesn't - carry a `vip:` option). `provider-bundle-check.py` PASS throughout, - 56 apps/108 relations unchanged. - - **DOCFIX-173** (`docs/changelog-20260710-sweep-script-fixes.md`): 4 - confirmed logic bugs + 1 security hardening, ALL independently - reproduced in isolation before being trusted (the same discipline - that caught the dc-dc-prefixes-import bug earlier tonight) -- - including catching that a FIRST proposed fix for one of them - (`scripts/phase-06-capi-stack.sh`) was itself wrong: `|| true` after a - failing pipeline under `pipefail` silently discards `PIPESTATUS`, - which would have made the "GATE FAIL" check never fire again, WORSE - than the original bug. The correct `if pipeline; then :; fi` idiom was - verified via reproduction before committing. Full list: `carve-host- - interfaces.sh`'s `emit()` returning 1 unguarded under `set -e` (killed - apply-mode runs on the first MAAS error, before the FATAL-accumulator - summary could print); `phase-06-capi-stack.sh`'s `PIPESTATUS` read - (as above); `juju-spaces-check.sh` and `osd-blank-check.sh`'s one - unguarded display-only pipe each (both safely `|| true`-able since - nothing downstream needed their PIPESTATUS); `scripts/ledger-scan.sh`'s - DOCFIX/BUNDLEFIX regex using a fixed `{3}` instead of `{3,}` -- the - SAME bug class this repo already got bitten by once for D-numbers - (fixed there via `{3,}`, never mirrored here) -- a dormant landmine, - not yet reachable (repo is at DOCFIX-170ish) but real; and - `dc-dc-rbd-mirror.sh`'s bootstrap token using a predictable `/tmp` - path with no permission hardening -- hardened to a random-suffixed - path + immediate `chmod 600`. - Full test gauntlet re-run after all fixes: 23/51 FAILED, identical to - every prior check tonight (same pre-existing jq-missing environment - gaps) -- zero regressions across the entire sweep-and-fix pass. - **Also flagged, not a repo issue:** the scripts-audit agent reported - that a skill's tool output contained injected text resembling a system - reminder, instructing it to conceal a date change from the operator -- - it correctly disregarded the concealment instruction and reported the - anomaly rather than act on it. The main session observed an - identically-worded, identically-formatted message earlier this same - evening (a genuine date-rollover notice, consistent with the session - crossing midnight) and treated it as benign/expected rather than a live - attack -- noted here for the operator's own awareness per this - environment's prompt-injection disclosure norm, not because either - session took any unsafe action in response. -- **DOCFIX-174** (`docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md`): - caught before assigning the next real DOCFIX number for follow-up work -- - `bash scripts/ledger-scan.sh` reported `DOCFIX next-free=1005`, wildly - implausible against a repo at DOCFIX-173. Root cause: DOCFIX-173's OWN - changelog narrated its `{3}`->`{3,}` regex fix using a literal - illustrative example (the real prefix plus one-zero-zero-four), which the - just-widened regex then matched as a real assignment -- a false-positive - one commit after fixing - a false-negative of the opposite kind. Fixed: reworded the illustrative - example in prose (both the DOCFIX-173 changelog and this file's own - mirrored comment), added a `reproduc`-line exclusion to - `nextfree_mentions()` as defense-in-depth (this repo's delivery style - narrates bug reproductions constantly, so this will recur), and added a - regression fixture to `tests/ledger-scan/run-tests.sh` (37/37 PASS, up - from 33). True next-free after this delivery: D-111, DOCFIX-175, - BUNDLEFIX-012. Caught the same way as everything else tonight: by - running the tool and checking whether its own output was plausible - before trusting it, not by code review. -- **DOCFIX-177** (`docs/changelog-20260710-skill-vr1-awareness.md`): - operator was heading to the vcloud host to begin real execution; asked - "is the skill ready?" -- it wasn't fully. `.claude/skills/openstack-cloud- - ops/SKILL.md` and `runbooks/README.md` (what its routing table points to) - had ZERO mentions of the entire VR1 DC-DC track -- a fresh session reading - only the skill would be routed to VR0's phase-01..08 runbooks with no - indication the DC-DC work (today's actual mission) exists. Fixed: - `runbooks/README.md` gained a full "VR1 DC-DC track" index table (7 Stage - runbooks + `dc-dc-teardown-rollback.md`, explicit that its numbering is a - separate sequence from VR0's), `SKILL.md`'s routing table + standard loops - gained VR1-specific rows, `references/environment.md` gained VR1 pointers - (a third "deployment" entry + a repo-map addition), `references/ - troubleshooting.md` gained a note that appendix-A is VR0-only and VR1 has - no incident index yet. Also found and fixed first: `main` was 2 commits - ahead of `origin/main` (DOCFIX-174, 175/176) -- pushed before this - delivery, since the vcloud host's session bootstrap starts with `git - pull` and would otherwise have missed them this morning. -- **DOCFIX-176** (`docs/changelog-20260710-vr1-teardown-rollback-runbook.md`): - closes gap register item #19. New `runbooks/dc-dc-teardown-rollback.md` -- - the OpenTofu/libvirt/MAAS-vm_host layer had no teardown runbook (D-061's - scripts cover a DIFFERENT layer, the juju/MAAS-machine one). Carries - D-061's PRINCIPLE (MAAS-side cleanup before destroying the libvirt - resource underneath) but not its specific mechanism (this repo doesn't use - MAAS pod-composition for OpenTofu-created VMs, so `--keep-instance` - doesn't directly transfer) -- flagged as untested/residual since no VR1 DC - has reached Stage 4 yet. Two paths (scoped `-target` teardown with - HashiCorp's own targeting-risk warning quoted, or full VR1 destroy), a - rollback decision tree favoring fix-forward over destroy for partial - apply failures, explicit mesh-link shared-infrastructure handling, and - confirmed (by reading the actual `.tf` files) that `netem-link`'s - destroy-time provisioner already works correctly. **Caught and fixed - mid-delivery:** re-running `ledger-scan.sh` before assigning this number - found DOCFIX-174's OWN changelog/ledger narration had reintroduced the - exact false-positive class it had just fixed (quoting the illustrative - example literally again) -- fixed by rewording, folded into DOCFIX-174's - existing changelog rather than a new number, since it corrects that - delivery before commit. Also confirmed DOCFIX-175's Stage-1-not-Stage-3 - correction (see its own entry above) was fully propagated before this - runbook's cross-references were written. -- **DOCFIX-175** (`docs/changelog-20260710-opentofu-state-file-security.md`): - answering the operator's own "what have we missed" question surfaced a - real, previously-unlogged gap -- `terraform.tfstate` stores - `var.maas_api_key` in plaintext regardless of `sensitive = true` (that flag - only suppresses CLI output), and nothing gitignored the state file, - documented backup/permission handling, or noted local state's lack of - locking. Fixed: `.gitignore` excludes `opentofu/**/*.tfstate*` + related - local artifacts, `opentofu/README.md` gained a full "State file handling" - section, `variables.tf`'s `maas_api_key` description carries the caveat, - and `dc-dc-phase2-tofu-dc-substrate.md` Step 8 (the actual first real - `tofu apply` in this repo) gained a "SECURE THE STATE FILE" callout right - after its GATE line -- closing the loop rather than leaving a "someone - should add this" note dangling. Gap register gained item #18 (this, CLOSED) - and #19 (VR1 teardown/rollback runbook -- OPEN, tracked as the next - delivery). Remote backend + real locking explicitly flagged as - Roosevelt/production-scope, not invented here. - -<!-- END: jumphost --> - -<!-- SECTION: shared | OWNER: shared -- append-only; never rewrite another stream's line --> ## State facts to remember -- beta cluster left at **node_count=2** (deliberate; bonus resize acceptance coverage). -- repo is temporarily **PUBLIC** for Claude web_fetch (SEC-004) -- flip private at v1 close. -- Jumphost stream: see "Active window" section above (ops-update-20260705 in flight). - Vault stays 1.8/stable (D-068 unruled). -- CORRECTION (2026-07-05, jumphost stream, per handoff reconcile item): the two bullets - above are stale -- ops-update-20260705 is CLOSED (fleet fully current, juju 3.6.25); - D-068 is RULED on the 1.16 question (1.16 ruled out; vault stays 1.8/stable rev 714; - the off-EOL-1.8.8 path remains OPEN under D-068). +- **The beta cluster and the foil1 tenant are GONE** (both confirmed absent by the 2026-07-08 + read-only audit). The long-standing "beta cluster left at node_count=2" note was STALE and is + retired here. `capi-test-1` and the `lbtest` LB leftover are also gone; orphan sweep reads + ORPHAN=0, zero unattached floating IPs. +- **devteam is a LIVE tenant** running a self-created cluster (built via Horizon, calico, + keypair `devteam-key`). Their template has `volume_driver` unset but the cluster DOES get a + default StorageClass from the cinder-CSI magnum default -- persistent storage works out of the + box; no template change needed. +- The vcloud host was NOT pristine: 9 orphaned `vr0-*` libvirt nets were torn down; **`wan` was + KEPT** (dual-stack NAT-out-enp1s0, GUA `2602:f3e2:fe:10::/64`) as the working model for the + per-site ISP-uplink network. +- MTU has TWO domains: the ISP uplink `enp1s0` is 1500; the host-internal plane/dark-fiber fabric + is **jumbo, underlay_mtu=9000** (operator ruling; verified on all 10 nets). +- The repo is temporarily **PUBLIC** (SEC-004) -- flip private at v1 close. +- Secrets live jumphost-local and are NEVER committed: `~/vr1-office1-creds/` (edge SSH key, + root password, API key), `~/vr1-office1-creds/vr1-netbox.env` (NetBox token), `~/tenant-devteam/`, `~/vault-init/`. ## Project-completion (execute after D-011 passes) -- Consolidate 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. -- Set repo visibility PRIVATE (SEC-004). +- Consolidate the 10 per-phase do-documents into `docs/v1-deploy-runbook.md`. +- Set repo visibility PRIVATE (SEC-004); revoke/rotate the SEC-005/006/007 credentials. - v2-deferred: SSH on GitBucket (port 29418), IPv6 dual-stack, NetBox import bundle. -### Cross-session pins (append-only) +## NetBox import -- DONE for Office1; DC import BLOCKED on one operator ruling (2026-07-13) -- D-071 completion sweep (2026-07-05, DONE): the ops-update window closed; D-071 was AMENDED - (create-backup premise corrected -- `juju create-backup`/`download-backup` DO exist on juju 3.6; - a 902MB controller backup was taken). D-071 REMAINS PROPOSED -- not ratified to ADOPTED. Operator - ratification is pending; the single-controller (no-HA) half of the risk is unresolved by backups - alone. See D-071 + its 2026-07-05 amendment. -- Pre-DC-DC controller HA/backup planning (PINNED): before the Roosevelt (DC-DC) phase, hold a - dedicated planning session on Juju controller High-Availability / backup architecture. Backups - now exist (per D-071 amendment); controller HA is the open design item. This resolves the - single-controller risk recorded under D-071 for production. +**Done.** The Office1 sandbox NetBox was EMPTY and had no seeder. Built the two-phase loop +(`netbox/prod-draft-dump.py` READ-ONLY -> `netbox/draft/vr1-draft.json` -> `netbox/sandbox-seed.py`), +seeded 129 objects from the upstream draft, then applied the **D-115 office carve** +(`netbox/d115-office-carve.py`): site `vr1-off1`, roles `office` + **`edge`** (new), and 8 prefixes. +Idempotent, and **verified FIELD-BY-FIELD** against the upstream draft +(`netbox/sandbox-fidelity-check.py`): every shared object is IDENTICAL, and the delta is EXACTLY +`+edge` role, `+vr1-off1` site, `+8` D-115 prefixes. The sandbox is a proven-faithful replica, which +is what makes it a valid baseline for the later feed-back diff. See `docs/vr1-office1-as-built.md` s6. -<!-- END: shared --> +**Why that check exists:** counts and idempotency CANNOT prove fidelity. The seeder silently dropped +the scope on 17 of 90 prefixes (it mapped only `dcim.site`; most of the VR1 draft is +`dcim.region`-scoped) -- and every count still matched and re-runs were still idempotent. It was +caught by LUCK on a spot-check. The fidelity check is that catch, made deliberate. + +**D-117 Option B is only PARTIALLY EXECUTED.** Done: the import path (`dc-dc-prefixes-import.py`) and +`docs/dc-dc-netbox-buildout-scope.md`. NOT done: the `$DC` shell selector (below) and the +forward-looking DC prose in the runbooks. Do not read "D-117 ADOPTED" as "fully renamed". + +**D-117 ADOPTED (operator ruling, Option B, DC-only):** VR1's DCs are **dc0/dc1**, matching the apex. +The **office KEEPS its number** (Off1). Fixed the importer's off-by-one site binding and made it +**dry by default**. + +### ~~BLOCKED~~ **UNBLOCKED and DONE** -- D-118 ratified the org ULA; both DCs imported + +**D-118 ADOPTED:** `ORG_ULA_48 = fd50:840e:74e2::/48` (RFC 4193 valid; **no collision with +Tailscale's `fd7a:115c:a1e0::/48`**, which matters because `office1-tailscale` runs `--accept-routes` +-- an overlap would have been a LIVE tailnet routing conflict). Also corrected a **FALSE claim** in +`changelog-20260711-ula-gen-command-fix.md` that called the value "ratified" when no D-number ever +assigned it -- the same document-asserts-authority failure mode as D-117. + +**Imported into the sandbox:** six-plane roles + 2 RIRs + 5 aggregates, then **VR1 DC0 (18 prefixes) +and VR1 DC1 (18)**. The D-117 fix is now **LIVE-PROVEN**, not just offline: `--dc dc0` bound to the +apex site `VR1 DC0` (id=8) and `--dc dc1` to `VR1 DC1` (id=9). ULA reads `DC.NN` in the 4th hextet +exactly as D-111 intends (`fd50:840e:74e2:220::` .. `:350::`). + +**Final state PROVEN by `netbox/sandbox-fidelity-check.py`:** the sandbox is the upstream draft plus +EXACTLY the planned delta (D-115 + D-117 + D-118) -- every shared object field-identical, nothing +lost, nothing stray. 90 -> 134 prefixes. + +### A REAL BUG in `roles-aggregates-import.py`, found by running it for the first time for real + +It checked existence by **slug only**. NetBox also enforces **unique NAMES**. The draft's legacy role +`Replication` (slug `repl`) blocks the six-plane role (slug `replication`, name `Replication`), so it +created 4 roles, **400'd on the 5th, and DIED -- leaving the apex half-populated** with no RIRs and no +aggregates and no rollback. It had **never been run against a NetBox holding the real draft** +(only empty ones) and **shipped with NO harness** -- which is why this survived. + +Fixed: the six-plane role is now named **"Replication Plane"** (slug unchanged -- `lib-net.sh` +SPACES6 and the prefix importer look it up), and the script now **PREFLIGHTS every name/slug and +writes nothing unless the whole plan is viable**. `tests/roles-aggregates-import/` created: 16/16. + +### (historical) the original blocker + +`netbox/dc-dc-prefixes-import.py --dc dc0` cannot run. It requires `ORG_ULA_48`, and that literal is +**still unassigned** -- it is gap **G3** in `docs/dc-dc-netbox-buildout-scope.md`, which *recommends* +`fd50:840e:74e2::/48` (CSPRNG-generated, Global ID `0x50840e74e2`) but **no ADOPTED decision assigns +it** (`grep ORG_ULA docs/design-decisions.md` -> nothing). D-101 explicitly leaves it NetBox-assigned. + +Using it anyway would violate hard rule 2 (never use an inferred value), so the import STOPPED here. +This is a decision gate, not a tooling gap -- the tooling is fixed, tested, and ready. + +**To unblock:** ratify an org ULA `/48` under its own D-number. Then: +``` +ORG_ULA_48=<ratified /48> DC_GUA_PREFIX=2602:f3e2:f02::/48 --dc dc0 # VR1 DC0 +ORG_ULA_48=<ratified /48> DC_GUA_PREFIX=2602:f3e2:f03::/48 \ + DC1_V4_SUPERNET=10.12.64.0/19 --dc dc1 # VR1 DC1 +``` +(`DC_GUA_PREFIX` values are MEASURED off the apex; `DC1_V4_SUPERNET` is assigned by D-115. +`DC2_V4_SUPERNET` is RETIRED -- the tool rejects it by name.) + +### Also open (recorded, not blocking) +- **`$DC` shell-selector namespace collision** (D-117 amendment): `lib-net.sh`'s `dc0` means **VR0's** + DC0, while the importer's `dc0` now means **VR1's**. NOT interchangeable. Proposed fix: region- + qualify the shell selector to the apex slugs (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`). +- Feed the D-115 carve upstream to `netbox.baldurkeep.com` (operator-gated; the tool refuses a + non-sandbox target without `--yes-write-upstream`). +- Office1 LAN IPv6 (`2602:f3e2:f01:100::/64` + RA via the OPNsense REST API) -- still pinned. diff --git a/docs/vr1-office1-as-built.md b/docs/vr1-office1-as-built.md new file mode 100644 index 0000000..c210ffc --- /dev/null +++ b/docs/vr1-office1-as-built.md @@ -0,0 +1,194 @@ +# VR1 Office1 -- as-built inventory and connection reference + +**Purpose.** The single operational answer to "what is running, where, on what address, and how do I +reach it." Every value here was MEASURED on the live system, not planned or inferred. Update it in +the SAME change that brings a host or service online -- a stale connection table is worse than none. + +**Scope.** VR1 Office1 only (the site headend). DC1/DC2 get their own sections when they exist. + +**Authority note.** NetBox is the IPAM apex -- this table is an OPERATIONAL reference (how to reach +things, where credentials live), NOT the address authority. Where the two ever disagree, NetBox wins +and the divergence gets fixed here. **The D-115 carve IS now imported** into the Office1 sandbox +NetBox (2026-07-13) -- site `vr1-off1`, roles `office` + `edge`, and every prefix below. It has NOT +yet been fed back upstream to `netbox.baldurkeep.com`; that is an operator-gated step. + +Last measured: 2026-07-13. + +--- + +## 1. Networks + +| Network | CIDR | Where it lives | Gateway | DHCP authority | Role (D-115) | +|---|---|---|---|---|---| +| `office1-wan` | `172.30.1.0/24` | libvirt NAT net on vcloud (`virbr11`) | `172.30.1.1` (vcloud) | none (static) | **Edge** (`172.30.0.0/16`) -- simulated ISP uplink | +| `office1-local` | `10.10.0.0/24` | libvirt bridge on vcloud (`virbr2`), addressless | `10.10.0.1` (the edge) | **Kea on the OPNsense edge** (pool `.100-.199`) | **Office** -- VR1 Office1 `10.10.0.0/22` | +| `lxdbr0` | `10.10.1.0/24` | INSIDE `voffice1` (LXD-managed, NAT) | `10.10.1.1` (voffice1) | **MAAS** (dynamic `.100-.200`; deployed machines get `.201+`) | **Office** -- same `/22`, compose net | + +**The DHCP split is structural.** Two authorities, two physically separate L2s. `lxdbr0` is NEVER +bridged onto `office1-local`, so Kea and MAAS cannot see each other. LXD's own dnsmasq is disabled +(`ipv4.dhcp=false`, `dns.mode=none`, `raw.dnsmasq=port=0`) and binds **no** `:53` and **no** `:67`. + +**IPv6:** Office1's carve is `2602:f3e2:f01:100::/56` with office subnet `2602:f3e2:f01:100::/64` +(D-115). **NOT YET DEPLOYED** -- pinned as the next step. Note IPv6 does NOT egress the lab +(measured: v6 gateway reachable, v6 internet 100% loss), so v6 here will be LAN-local only. + +--- + +## 2. Hosts and services + +| Host | Address | What runs on it | Level | +|---|---|---|---| +| **vcloud** | `10.17.11.248` (lab), `10.10.0.10` on `virbr2`, `172.30.1.1` on `virbr11` | libvirt/KVM hypervisor; OpenTofu runs from here | L1 (itself a KVM guest) | +| **office1-opnsense** | LAN `10.10.0.1` / WAN `172.30.1.2` | OPNsense 26.1 edge router: routing, NAT, **Kea DHCP**, firewall | L2 | +| **voffice1** | `10.10.0.20` (Kea reservation) | **MAAS 3.7.2** region+rack + PostgreSQL 16.14; **LXD 5.21.5** (registered to MAAS as VM host `office1-lxd`) | L2 | +| **office1-netbox** | `10.10.1.10` | **NetBox 4.6.4** (netbox-docker; Django 6.0.6) on `:8000` | L3 (LXD VM) | +| **office1-tailscale** | `10.10.1.11` / tailnet **`100.64.0.53`** (`fd7a:115c:a1e0::35`) | **Tailscale 1.98.8**, subnet router advertising `10.10.0.0/22`. Joined the **SELF-HOSTED** control plane `https://tailscale.baldurkeep.com:443` | L3 (LXD VM) | +| ~~office1-gitbucket~~ | -- | **NOT BUILT -- AND WILL NOT BE (D-116).** `git.baldurkeep.com` remains the git service of record; new repos are created there as deployment tasks require. | -- | + +`office1-netbox` and `office1-tailscale` are **MAAS-composed LXD VMs** (D-114) -- MAAS enlisted, +commissioned, deployed and powers them. The `voffice1` LXD host is `office1-lxd` (vm-host id 1). + +**D-120 static-band re-IP (executed 2026-07-15).** Both services were moved from the MAAS +node band (`.201`/`.202`) into the static-services band: `office1-netbox .201 -> .10`, +`office1-tailscale .202 -> .11`. **MAAS-model drift, by design:** MAAS refuses to edit an +interface on a **Deployed** machine (the only lift is Release->redeploy, which would WIPE the +seeded sandbox), so MAAS's interface model still records `.201`/`.202` as `mode=auto`. The LIVE +address is set on each guest via `/etc/netplan/50-cloud-init.yaml` AND +`/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg` (both edited, so it survives reboot). The drift +reconciles for free at the next teardown/redeploy, when MAAS re-renders from a corrected model. +MAAS auto-assigns nodes from `.201+` upward (observed: netbox/tailscale themselves landed at +`.201`/`.202`), so the low `.2-.49` band is off its allocation path -- no collision. (Being outside +the dynamic range is NOT itself the guarantee; the whole non-dynamic space is MAAS's static pool.) +Guest +control channel during the change was `lxc exec` (LXD socket), immune to the IP swap. + +--- + +## 3. How to reach each thing + +All access is via the Office1 service SSH key: **`~/vr1-office1-creds/office1_svc_ed25519`** on +vcloud (private half NEVER read; SEC-007 tracks its rotation). + +`office1-local` (`10.10.0.0/24`) and `lxdbr0` (`10.10.1.0/24`) are **host-local to vcloud** -- not +routable from a workstation. Everything below therefore tunnels through vcloud. + +| Target | From vcloud | From your workstation | +|---|---|---| +| **voffice1** (shell) | `ssh -i <key> jessea123@10.10.0.20` | `ssh -J jessea123@10.17.11.248 jessea123@10.10.0.20` | +| **OPNsense GUI** | `https://10.10.0.1` | `ssh -L 8443:10.10.0.1:443 jessea123@10.17.11.248` -> `https://localhost:8443` | +| **OPNsense shell** | `ssh -i <key> root@10.10.0.1` | via vcloud. **root's shell is `tcsh`** -- feed commands to `sh -s`. | +| **MAAS UI** | `http://10.10.0.20:5240/MAAS` | `ssh -L 5240:10.10.0.20:5240 jessea123@10.17.11.248` -> `http://localhost:5240/MAAS` | +| **NetBox UI/API** | `http://10.10.1.10:8000` (needs the route below) | on tailnet: `http://10.10.1.10:8000` direct. Else `ssh -J jessea123@10.17.11.248 -L 8000:10.10.1.10:8000 jessea123@10.10.0.20` -> `http://localhost:8000` | +| **office1-netbox / office1-tailscale** (shell) | `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.10` | chain both jumps | + +**Routing to the compose net.** The OPNsense edge has a static route `10.10.1.0/24 -> 10.10.0.20` +(gateway object `OFFICE1_LXD_GW`, priority 255, NOT the default gateway). So any `office1-local` +client using the edge as its gateway reaches the compose net. **vcloud itself does NOT** -- its +default route is the lab, so it needs an explicit route (operator sudo): +`sudo ip route replace 10.10.1.0/24 via 10.10.0.20 dev virbr2` + +**Tailscale front door (D-107: Office1 ONLY).** `office1-tailscale` is a subnet router on the +operator's **SELF-HOSTED** control plane (`https://tailscale.baldurkeep.com:443` -- NOT Tailscale's +public one). It is UP and online as `100.64.0.53`, advertising **`10.10.0.0/22`** -- exactly the +D-115 Office1 carve, so "Office1 only" is enforced by the CARVE, not by policy alone. + +- **Brought up with:** `--login-server=https://tailscale.baldurkeep.com:443 --authkey=<file> + --advertise-routes=10.10.0.0/22 --accept-routes --accept-dns=false --hostname=office1-tailscale` +- **`--accept-dns=false` is DELIBERATE** (a deviation from the operator's usual command): this is a + MAAS-managed machine and MAAS's bind9 serves its internal names. Letting the tailnet take over DNS + would break that resolution. Flip it if MagicDNS is wanted here. +- **ROUTE IS ENABLED** (operator, 2026-07-13). The `10.10.0.0/22` subnet route is live on the + self-hosted control server and **verified from the workstation** -- the OPNsense GUI answers + directly at `https://10.10.0.1` with **no SSH tunnel**. This SUPERSEDES the tunnel commands in the + table above: from a tailnet-joined workstation, every Office1 address in the `/22` is reachable + directly. The tunnels remain valid as a fallback when off-tailnet. +- The auth key was consumed by path from a `0600` file and **shredded from the node** after use -- + tailscaled holds its own node key now. It never entered an operator session or the repo. + +Once the route is enabled, the whole Office1 carve -- edge GUI, MAAS UI, NetBox -- is reachable from +the tailnet with **no SSH tunnels at all**, which supersedes most of the tunnel commands above. + +--- + +## 4. Where the credentials live (NEVER commit; never print) + +| What | Location | Notes | +|---|---|---| +| Office1 service SSH key | `~/vr1-office1-creds/office1_svc_ed25519{,.pub}` (vcloud) | SEC-007 -- rotation obligation | +| OPNsense API key/secret | `~/vr1-office1-creds/opnsense-api.txt` (vcloud) | used by `scripts/opnsense-api.sh` | +| OPNsense root password | `~/vr1-office1-creds/opnsense-root-password` (vcloud) | GUI login | +| MAAS admin password + API key; DB password; LXD trust password | `/root/maas-secrets/` on **voffice1** (`0600`) | generated on the box | +| NetBox SECRET_KEY, admin password, API token | `/root/netbox-secrets/` on **office1-netbox** (`0600`) | SECRET_KEY rotated off netbox-docker's PUBLIC default | +| NetBox (upstream / v1-draft) token env | `~/vr1-office1-creds/vr1-netbox.env` (vcloud) | `NETBOX_URL`+`NETBOX_TOKEN` for `netbox.baldurkeep.com`. SEC-006 -- burned, revoke at completion. The SANDBOX token is NOT here -- it is on office1-netbox (`/root/netbox-secrets/`). | +| Office1 edge config env | `~/vr1-office1-creds/vr1-office1.env` (vcloud) | OPNsense WAN/LAN/DHCP/root-hash/SSH-key vars. Consolidated 2026-07-15 (SEC-009). | +| VR1 OpenTofu env | `~/vr1-office1-creds/vr1-stage1.env` (vcloud) | `TF_VAR_*` incl. `TF_VAR_maas_api_key`. Consolidated + chmod 600 2026-07-15 (was loose + 0664, SEC-009). TF_VAR pool-path names updated to D-119 (`vr1_dc0`/`vr1_dc1`). | +| Tailscale auth key (self-hosted tailnet) | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, `0600`) | Operator-supplied, 48 chars. Consumed by PATH, never printed. The copy on the node was SHREDDED after `tailscale up`. Rotate/revoke on the control server at v1 close. | + +**Credential consolidation (SEC-009, standing convention):** ALL sensitive + env/config files for a +site live in one `~/<site>-creds/` folder (0700; files 0600; public keys 0644) -- no loose env files +in `~`. When DC deployment starts, `~/vr1-dc0-creds/` and `~/vr1-dc1-creds/` follow the same pattern +from day one. See `docs/security-ledger.md` SEC-009. + +**NetBox API token format (4.6):** the wire form is `nbt_<key>.<plaintext>`. The API's `token` field +alone is NOT usable -- present it raw and you get `403 Invalid v1 token`. See +`docs/changelog-20260713-office1-netbox-deployed.md`. + +--- + +## 5. Standing traps for this site (the short list) + +1. **DHCP on the edge is API-MANAGED.** A rendered `config.xml` push would CLOBBER it. That path is + DELETED (D-113(a2)); configure via `scripts/opnsense-api.sh`. +2. **OPNsense gateways: `defaultgw` in the API output is DERIVED FROM `priority`** (lower = more + preferred), not from what you send. Adding a LAN gateway with a lower priority than `WAN_GW` + SILENTLY STEALS THE DEFAULT ROUTE and kills site egress -- this happened on 2026-07-13. Always + add the gateway, READ THE CONFIG BACK, confirm `WAN_GW` is still default, and only THEN + `reconfigure`. `WAN_GW` is now priority 1; `OFFICE1_LXD_GW` is 255. +3. **root's shell on the edge is `tcsh`** -- `$(...)` and `2>&1` fail (quietly, or with "Ambiguous + output redirect"). Feed remote commands to `sh -s`. +4. **`lxc` reads config from stdin** -- in a piped script every `lxc` call needs `</dev/null` or it + eats the rest of the script. +5. **MAAS's bind9 intermittently SERVFAILs a first-time external lookup** and succeeds on retry + (seen twice: docker registry, tailscale.com). Retry before diagnosing. +6. **A `tofu apply` touching a `libvirt_domain`'s devices BOUNCES THE GUEST**, even when the plan + says "updated in-place". Schedule it as an outage. + +--- + +## 6. IPAM -- what NetBox now says about this site (D-115 carve, imported 2026-07-13) + +The Office1 **sandbox** NetBox (`10.10.1.10:8000`) was seeded from the upstream draft (129 objects: +1 RIR, 23 roles, 5 regions, 10 sites, 90 prefixes) and then had the D-115 carve applied on top. +Site **`vr1-off1` ("VR1 Off1")**, region VR1. + +| Prefix | Role | Status | What it is | +|---|---|---|---| +| `10.10.0.0/16` | office | container | The Office role's v4 block -- `/22` per office (D-115) | +| `10.10.0.0/22` | office | container | **VR1 Off1's office `/22`** (`.2`/`.3` spare) | +| `10.10.0.0/24` | office | active | `office1-local` LAN -- Kea DHCP on the edge | +| `10.10.1.0/24` | office | active | `lxdbr0` LXD compose net -- MAAS DHCP | +| `172.30.0.0/16` | **edge** | container | The NEW Edge role -- simulated ISP/WAN (mirrors v6 `2602:f3e2:fe::/48`) | +| `172.30.1.0/24` | edge | active | `office1-wan` -- the simulated ISP uplink | +| `2602:f3e2:f01:100::/56` | office | container | VR1 Off1's v6 `/56` (mirrors VR0 Off0's `e01:100::/56`) | +| `2602:f3e2:f01:100::/64` | office | active | Office subnet -- **NOT YET DEPLOYED** (v6 does not egress the lab) | + +**Every one of these addresses was already running.** D-115 blesses what is deployed rather than +renumbering it -- the carve cost ZERO renumber. What it fixed is that `10.10.0.0/24`, `10.10.1.0/24` +and `172.30.1.0/24` were **SQUAT**: on the wire with no allocation behind them. + +**Naming (D-117 + D-119):** VR1's datacenters are **vr1-dc0** (first) and **vr1-dc1** (second) -- the selector IS the NetBox apex slug. Bare `dcN` is RETIRED (it meant VR0's live cloud in `lib-net.sh`). The office KEEPS its number: **Office1** / `vr1-off1` (matching the apex, which already held both +sites) but **the office keeps its number** -- it is **Off1**. VR1 therefore reads DC0/DC1/**Off1** +while VR0 reads DC0/DC1/Off0. Accepted cosmetic asymmetry; the site is NEW upstream so nothing was +renamed and **no deployed object changed**. + +**Not yet upstream.** The sandbox is a simulation. Feeding this carve back to +`netbox.baldurkeep.com` is a separate, operator-gated write (`netbox/d115-office-carve.py` refuses a +non-sandbox target unless given `--yes-write-upstream`). + +**Reproduce the sandbox from scratch:** +``` +python3 netbox/prod-draft-dump.py --out netbox/draft/vr1-draft.json # on vcloud, READ-ONLY +# then on office1-netbox, with the sandbox token: +python3 sandbox-seed.py --draft draft/vr1-draft.json --commit +python3 d115-office-carve.py --commit +``` diff --git a/netbox/README.md b/netbox/README.md index 0e8d52e..303a150 100644 --- a/netbox/README.md +++ b/netbox/README.md @@ -4,12 +4,20 @@ (IPv4-only, single-site) deployment and the VR1 DC-DC (two-DC, dual-stack) buildout. Each script is idempotent; re-running is safe. +> **DOCFIX-195 (operator ruling 2026-07-15) -- TARGET NetBox during VR1.** The VR1 IPAM apex is +> `office1-netbox` (`http://10.10.1.10:8000`) -- a working draft, edited as the buildout proceeds, and +> the NetBox every consuming system (OpenTofu, MAAS, overlays, bundle) derives its values from. Point +> the importers there (it is in `SANDBOX_HOSTS`, so the writes are unguarded). `netbox.baldurkeep.com` is held as a READ-ONLY v1 reference draft and takes +> NO write during VR1 -- the `NETBOX_URL=https://netbox.baldurkeep.com` shown in the examples below is +> for the DEFERRED end-of-deployment merge-back ONLY, and each apex write additionally requires +> `--yes-write-upstream`. See `docs/dc-dc-netbox-buildout-scope.md` section 8. + ## VR1 DC-DC scripts (2026-07-09, closes tooling gap register #3) ### `dc-dc-prefixes-import.py` Extends the v1 single-site IPv4-only pattern to VR1's two-DC, dual-stack -model (D-101, `docs/design-decisions.md`). Per DC (`--dc dc1` or `--dc dc2`): +model (D-101, `docs/design-decisions.md`). Per DC (`--dc vr1-dc0` or `--dc vr1-dc1`): creates the six-plane IPv4 structure plus the D-101 family-matrix IPv6 legs (ULA for metal-admin/metal-internal/data-tenant/storage/replication, GUA for provider-public). @@ -18,7 +26,7 @@ explicitly "NOT hardcoded in this decision": the org ULA /48, the per-DC GUA carve out of the real ARIN 2602:f3e2::/32 block, and DC2's v4 supernet. All three are REQUIRED environment variables with no defaults -- -`ORG_ULA_48`, `DC_GUA_PREFIX`, and (for `--dc dc2` only) `DC2_V4_SUPERNET`. +`ORG_ULA_48`, `DC_GUA_PREFIX`, and (for `--dc vr1-dc1` only) `VR1_DC1_V4_SUPERNET`. The script validates format/containment and FAILS LOUD if any is missing or malformed; it never silently substitutes DC1's values for DC2 or invents a /48. DC1's v4 planes ARE hardcoded here, deliberately -- D-101 rules DC1 @@ -34,12 +42,12 @@ ``` NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=<token> \ ORG_ULA_48=fd00:1234:5678::/48 DC_GUA_PREFIX=2602:f3e2:1000::/40 \ - python3 dc-dc-prefixes-import.py --dc dc1 + python3 dc-dc-prefixes-import.py --dc vr1-dc0 NETBOX_URL=... NETBOX_TOKEN=... \ ORG_ULA_48=fd00:1234:5678::/48 DC_GUA_PREFIX=2602:f3e2:2000::/40 \ - DC2_V4_SUPERNET=10.13.0.0/19 \ - python3 dc-dc-prefixes-import.py --dc dc2 + VR1_DC1_V4_SUPERNET=10.13.0.0/19 \ + python3 dc-dc-prefixes-import.py --dc vr1-dc1 ``` **Role slugs used here are NEW** (`provider-public`/`metal-admin`/ diff --git a/netbox/d115-office-carve.py b/netbox/d115-office-carve.py new file mode 100755 index 0000000..a25e13a --- /dev/null +++ b/netbox/d115-office-carve.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +""" +Apply the D-115 Office carve (and the roles it needs) to a NetBox. + +D-115 option (a), ADOPTED 2026-07-13. This BLESSES what is already running -- +every address here was MEASURED off the live Office1 build, not invented, so the +carve costs ZERO renumbering. What it fixes is that those addresses were SQUAT: +they existed on the wire with no allocation behind them. + +What D-115 found, and what this creates: + + 1. There is NO Office role in IPv4. Office1's LAN (10.10.0.0/24) and its LXD + compose net (10.10.1.0/24) were squatting. + -> new role `office` at 10.10.0.0/16, carved /22 per office. + VR1 Off1 = 10.10.0.0/22 -> LAN .0/24 + compose .1/24 (+ .2/.3 spare). + + 2. "Edge Networks" has a v6 /48 (2602:f3e2:fe::/48) but NO v4 counterpart, so + the simulated ISP uplink 172.30.1.0/24 was squatting inside the OOB /12. + -> new role `edge` at 172.30.0.0/16, its OWN role (NOT oob), carved out of + the existing 172.16.0.0/12 container. Office1's WAN becomes legitimate. + + 3. IPv6 has NO design freedom here -- the region pattern determines it: + VR1's office /48 (2602:f3e2:f01::/48) already exists upstream, and VR0's + Off0 shows the shape (e01:100::/56 container -> e01:100::/64 subnet). + -> VR1 Off1 = 2602:f3e2:f01:100::/56 container + :100::/64 office subnet. + + 4. That /56 needs a SITE. -> `vr1-off1` ("VR1 Off1"), region VR1. + +NAMING (D-117): VR1's DCs are dc0/dc1 (matching the apex) but the OFFICE KEEPS +ITS NUMBER -- it is Off1, site `vr1-off1`. That site does not exist upstream, so +it is CREATED, not renamed, and nothing deployed changes. VR1 therefore reads +DC0/DC1/Off1 while VR0 reads DC0/DC1/Off0: an accepted cosmetic asymmetry. + +NOT DONE HERE: VR1 DC1's v4 supernet (10.12.64.0/19, also D-115) belongs to +netbox/dc-dc-prefixes-import.py --dc dc1, which owns the six-plane carve. Doing +it in two places is how they drift. + +DRY BY DEFAULT. Nothing is written without --commit. + +WRITING UPSTREAM IS GATED IN CODE. Per the standing architecture the sim never +writes upstream casually: feeding refinements back to the production apex is an +operator decision. Pointing this at anything that is not an obvious sandbox +requires --yes-write-upstream ON TOP of --commit. + +Usage (sandbox, on office1-netbox): + sudo NETBOX_URL=http://localhost:8000 \ + NETBOX_TOKEN="$(cat /root/netbox-secrets/api.token)" \ + python3 d115-office-carve.py # preview + ... --commit # apply +""" + +import argparse +import ipaddress +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request + +UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md + +SITE = {"slug": "vr1-off1", "name": "VR1 Off1", "region": "vr1"} + +ROLES = [ + {"slug": "office", "name": "Office"}, # exists upstream; created if absent + {"slug": "edge", "name": "Edge"}, # NEW -- v6 had Edge Networks, v4 had nothing +] + +# Every CIDR below was MEASURED on the live Office1 build (docs/vr1-office1-as-built.md). +# status: container = an allocation block; active = a real subnet on the wire. +PREFIXES = [ + # --- IPv4: the Office role (D-115 gap 1) --- + ("10.10.0.0/16", "office", "container", None, + "Office (v4) -- /22 per office (D-115)"), + ("10.10.0.0/22", "office", "container", "vr1-off1", + "VR1 Off1 -- office /22 (D-115)"), + ("10.10.0.0/24", "office", "active", "vr1-off1", + "VR1 Off1 office1-local LAN -- Kea DHCP on the OPNsense edge (D-115)"), + ("10.10.1.0/24", "office", "active", "vr1-off1", + "VR1 Off1 LXD compose net (lxdbr0) -- MAAS DHCP (D-114/D-115)"), + + # --- IPv4: the Edge role (D-115 gap 4) --- + ("172.30.0.0/16", "edge", "container", None, + "Edge (v4) -- simulated ISP/WAN segments; mirrors v6 2602:f3e2:fe::/48 (D-115)"), + ("172.30.1.0/24", "edge", "active", "vr1-off1", + "VR1 Off1 office1-wan -- simulated ISP uplink (D-115)"), + + # --- IPv6: forced by the region pattern (D-115 gap 3) --- + ("2602:f3e2:f01:100::/56", "office", "container", "vr1-off1", + "VR1 Off1 (D-115) -- mirrors VR0 Off0 e01:100::/56"), + ("2602:f3e2:f01:100::/64", "office", "active", "vr1-off1", + "VR1 Off1 office subnet -- NOT YET DEPLOYED (v6 does not egress the lab)"), +] + +# A sandbox is local, or the known Office1 sandbox address. Anything else is +# treated as production and requires the explicit upstream flag. +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +class NB: + def __init__(self, base, token): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, + headers={"Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + "User-Agent": UA}) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " + "nbt_<key>.<plaintext>, not the API's bare `token` field.") + if exc.code == 403: + die(f"403 on {path}. If curl works with this token, it is the upstream " + f"User-Agent filter, NOT the token (references/platform-traps.md).") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def one(self, path, **flt): + res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") + return res["results"][0] if res["results"] else None + + def create(self, path, payload): + return self._req("POST", f"{path}/", payload) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--commit", action="store_true", + help="WRITE. Default is a DRY RUN that writes nothing.") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox. " + "Feeding the production apex is an operator decision.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set.") + + host = (urllib.parse.urlparse(url).hostname or url).lower() + is_sandbox = host in SANDBOX_HOSTS + + print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") + if args.commit and not is_sandbox and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{host}': it is not a known sandbox, so this is " + f"treated as the PRODUCTION apex. Feeding validated refinements upstream is " + f"an operator decision, not a side effect of running an import. Re-run with " + f"--yes-write-upstream if that is genuinely what you intend.") + + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" + if not args.commit else "\n*** COMMITTING. ***") + + nb = NB(url, token) + created = existing = 0 + + print("\nIPAM roles:") + for r in ROLES: + if nb.one("ipam/roles", slug=r["slug"]): + print(f" EXISTS role {r['slug']}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE role {r['slug']} ({r['name']})") + created += 1 + continue + o = nb.create("ipam/roles", {"name": r["name"], "slug": r["slug"]}) + print(f" CREATED role {r['slug']} (id={o['id']})") + created += 1 + + print("\nSite:") + site = nb.one("dcim/sites", slug=SITE["slug"]) + site_id = site["id"] if site else None + if site: + print(f" EXISTS site {SITE['slug']} (id={site_id})") + existing += 1 + else: + region = nb.one("dcim/regions", slug=SITE["region"]) + if region is None: + die(f"region '{SITE['region']}' is absent -- seed the draft first " + f"(netbox/sandbox-seed.py).") + if not args.commit: + print(f" [dry-run] would CREATE site {SITE['slug']} ({SITE['name']}) in region {SITE['region']}") + created += 1 + else: + o = nb.create("dcim/sites", {"name": SITE["name"], "slug": SITE["slug"], + "region": region["id"], "status": "active"}) + site_id = o["id"] + print(f" CREATED site {SITE['slug']} (id={site_id})") + created += 1 + + print("\nPrefixes:") + for cidr, role_slug, status, scope_slug, desc in PREFIXES: + ipaddress.ip_network(cidr) # fail loud on a typo, before any write + if nb.one("ipam/prefixes", prefix=cidr): + print(f" EXISTS {cidr}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE {cidr:<24} role={role_slug:<7} {status:<9} " + f"scope={scope_slug or '-'}") + created += 1 + continue + role = nb.one("ipam/roles", slug=role_slug) + if role is None: + die(f"role '{role_slug}' absent -- cannot place {cidr}") + payload = {"prefix": cidr, "role": role["id"], "status": status, "description": desc} + if scope_slug: + if site_id is None: + die(f"site '{scope_slug}' absent -- cannot scope {cidr}") + payload["scope_type"] = "dcim.site" + payload["scope_id"] = site_id + o = nb.create("ipam/prefixes", payload) + print(f" CREATED {cidr} (id={o['id']}) role={role_slug}") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") + if not args.commit: + print("DRY RUN -- nothing was written. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/netbox/d120-compose-bands.py b/netbox/d120-compose-bands.py new file mode 100755 index 0000000..ece353b --- /dev/null +++ b/netbox/d120-compose-bands.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +""" +Register the D-120 compose-/24 band ranges + the static service IP assignments. + +D-120 (ADOPTED 2026-07-15) ratified a functional band layout for every MAAS-managed +compose /24, and D-120 Step 6 owes NetBox the record of it. On Office1's compose net +10.10.1.0/24 (lxdbr0) this creates: + + ip-ranges (the child bands D-120 names for NetBox: static / dynamic / node): + 10.10.1.2/24 - 10.10.1.49/24 static site-services (.2-.49) + 10.10.1.100/24 - 10.10.1.200/24 MAAS dynamic (.100-.200) + 10.10.1.201/24 - 10.10.1.254/24 deployed nodes (.201-.254) + ip-addresses (the two re-IP'd static services): + 10.10.1.10/24 office1-netbox (re-IP from .201, D-120) + 10.10.1.11/24 office1-tailscale (re-IP from .202, D-120) + +(D-120's layout also names .1 gateway and .50-.99 reserved, but its NetBox instruction +is explicitly "child ranges (static/dynamic/node)" -- so those three, not the others.) + +DRY BY DEFAULT -- nothing is written without --commit. + +WRITING UPSTREAM IS GATED IN CODE. The sim never writes the production apex casually: +SANDBOX_HOSTS below is the allowlist; a --commit at any other host REFUSES without +--yes-write-upstream. WHOLE-PLAN PREFLIGHT: the parent compose net must exist and every +range/IP must fall inside it, checked BEFORE any create, so a bad plan cannot half-write. + +Usage (on office1-netbox / through a tunnel, with the sandbox token): + NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=<tok> python3 netbox/d120-compose-bands.py + ... same, add --commit, to write. +""" +import argparse +import ipaddress +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request + +UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md + +# A sandbox is local, or the known Office1 sandbox address. Anything else is treated as +# the production apex and requires the explicit upstream flag. +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + +PARENT = "10.10.1.0/24" # the compose net these bands/IPs live within (must pre-exist) + +# start/end carry the /24 mask, per NetBox convention (netbox/ipv4-prefixes-import.py). +IP_RANGES = [ + ("10.10.1.2/24", "10.10.1.49/24", "active", + "D-120 static site-services band (.2-.49) -- MAAS static-assign, outside the dynamic range"), + ("10.10.1.100/24", "10.10.1.200/24", "active", + "D-120 MAAS dynamic band (.100-.200) -- enlistment/commissioning/PXE/DHCP"), + ("10.10.1.201/24", "10.10.1.254/24", "active", + "D-120 deployed-node band (.201-.254) -- MAAS auto-assign for compute/OpenStack nodes"), +] +IP_ADDRESSES = [ + ("10.10.1.10/24", "active", "office1-netbox", + "office1-netbox service (D-120 static band; re-IP from .201)"), + ("10.10.1.11/24", "active", "office1-tailscale", + "office1-tailscale subnet router (D-120 static band; re-IP from .202)"), +] + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +class NB: + """Stdlib NetBox client -- same shape as the other sandbox-loop tools; UA-aware so + it is not 403'd by the upstream User-Agent filter.""" + def __init__(self, base, token): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, + headers={"Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + "User-Agent": UA}) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " + "nbt_<key>.<plaintext>, not the API's bare `token` field.") + if exc.code == 403: + die(f"403 on {path}. If curl works with this token, it is the upstream " + f"User-Agent filter, NOT the token (references/platform-traps.md).") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def one(self, path, **flt): + res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") + return res["results"][0] if res["results"] else None + + def create(self, path, payload): + return self._req("POST", f"{path}/", payload) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--commit", action="store_true", + help="WRITE. Default is a DRY RUN that writes nothing.") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set.") + + host = (urllib.parse.urlparse(url).hostname or url).lower() + is_sandbox = host in SANDBOX_HOSTS + print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") + if args.commit and not is_sandbox and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{host}': not a known sandbox, so treated as the " + f"PRODUCTION apex. Re-run with --yes-write-upstream if that is intended.") + + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" + if not args.commit else "\n*** COMMITTING. ***") + + nb = NB(url, token) + + # WHOLE-PLAN PREFLIGHT -- fail loud BEFORE any create so a bad plan cannot half-write. + net = ipaddress.ip_network(PARENT) + for s, e, _st, _d in IP_RANGES: + for a in (s, e): + if ipaddress.ip_interface(a).ip not in net: + die(f"range endpoint {a} is outside the parent {PARENT}") + if ipaddress.ip_interface(s).ip > ipaddress.ip_interface(e).ip: + die(f"range {s} - {e} has start > end") + for a, _st, _dns, _d in IP_ADDRESSES: + if ipaddress.ip_interface(a).ip not in net: + die(f"address {a} is outside the parent {PARENT}") + if nb.one("ipam/prefixes", prefix=PARENT) is None: + die(f"parent prefix {PARENT} absent in NetBox -- seed the D-115 carve first " + f"(netbox/d115-office-carve.py). Refusing to place bands in an unallocated /24.") + + created = existing = 0 + + print("\nIP ranges:") + for start, end, status, desc in IP_RANGES: + if nb.one("ipam/ip-ranges", start_address=start, end_address=end): + print(f" EXISTS {start} - {end}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE range {start} - {end} ({status})") + created += 1 + continue + o = nb.create("ipam/ip-ranges", {"start_address": start, "end_address": end, + "status": status, "description": desc}) + print(f" CREATED range {start} - {end} (id={o['id']})") + created += 1 + + print("\nIP addresses:") + for addr, status, dns, desc in IP_ADDRESSES: + if nb.one("ipam/ip-addresses", address=addr): + print(f" EXISTS {addr}") + existing += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE address {addr} dns={dns}") + created += 1 + continue + o = nb.create("ipam/ip-addresses", {"address": addr, "status": status, + "dns_name": dns, "description": desc}) + print(f" CREATED address {addr} (id={o['id']})") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") + if not args.commit: + print("DRY RUN -- nothing was written. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/netbox/dc-dc-prefixes-import.py b/netbox/dc-dc-prefixes-import.py index 9bf69db..f6471fd 100644 --- a/netbox/dc-dc-prefixes-import.py +++ b/netbox/dc-dc-prefixes-import.py @@ -6,8 +6,8 @@ extends the v1 (netbox/ipv4-prefixes-import.py) single-site IPv4-only import to the VR1 two-DC, dual-stack model ratified in D-101 (docs/design-decisions.md). -Per DC (dc1 or dc2, selected via --dc, mirroring the scripts/lib-net.sh -$DC selector convention, DOCFIX-151), creates the six-plane structure with +Per DC (vr1-dc0 or vr1-dc1, selected via --dc, mirroring the scripts/lib-net.sh +$DC selector convention, DOCFIX-151/D-119), creates the six-plane structure with the D-101 family matrix: Plane v4 v6 family @@ -18,16 +18,32 @@ storage (see below) ULA-only replication (see below) ULA-only -DC1 v4: INHERITS the DC0 six-plane CIDRs unchanged (D-101 -- "DC1 equals the -validated template"), i.e. the SAME literals as netbox/ipv4-prefixes-import.py, -hardcoded here for that reason (this is not an inferred value; it is the -explicit text of an ADOPTED decision). Scoped to a DISTINCT NetBox site -(vr1-dc1), not the existing vr0-dc0 site. +NAMING (D-117 + D-119): VR1's DCs are **VR1 DC0** (slug vr1-dc0) and **VR1 DC1** +(slug vr1-dc1), matching the NetBox apex, which already holds both site records. +This script once called them dc1/dc2 -- an off-by-one against the apex that would +have bound VR1 DC0's prefixes to the site vr1-dc1 (i.e. the OTHER DC), silently, +because it also used to write by default. Both defects are fixed. -DC2 v4: D-101 explicitly states DC2's supernet is "NetBox-assigned... not -hardcoded." No default is provided here -- DC2_V4_SUPERNET is a REQUIRED -environment variable with no fallback. Running with --dc dc2 and no -DC2_V4_SUPERNET set FAILS LOUD rather than guessing or reusing DC1's range. +D-119 goes further: **the --dc selector IS the apex slug** (identity map). There is +no offset table left to get wrong. Bare dc0/dc1/dc2 are RETIRED -- 'dc0' used to +mean VR0's LIVE testcloud in scripts/lib-net.sh but VR1's FIRST DC here: one +string, two clouds. EVERY "DCn" below is REGION-QUALIFIED, because an unqualified +"DC0" is ambiguous across VR0 and VR1 and that ambiguity is what produced the bug. + +VR1 DC0 v4: INHERITS the **VR0 DC0** six-plane CIDRs unchanged (D-101 -- "the +first DC equals the validated template"), i.e. the SAME literals as +netbox/ipv4-prefixes-import.py, hardcoded here for that reason (this is not an +inferred value; it is the explicit text of an ADOPTED decision). Scoped to the +DISTINCT NetBox site **vr1-dc0** -- NOT the existing **vr0-dc0** site, which is +the rehearsal environment those literals came from. + +VR1 DC1 v4: D-101 explicitly states the second DC's supernet is "NetBox-assigned +... not hardcoded." No default is provided here -- VR1_DC1_V4_SUPERNET is a REQUIRED +environment variable with no fallback. Running with --dc vr1-dc1 and no +VR1_DC1_V4_SUPERNET set FAILS LOUD rather than guessing or reusing VR1 DC0's range. +(It was DC2_V4_SUPERNET, then briefly DC1_V4_SUPERNET. BOTH old names are now +REJECTED BY NAME with a pointer: an unqualified "DC1" could mean VR0's dc0 OR VR1's +second DC, which is precisely the ambiguity D-119 deletes.) IPv6 (both DCs): D-101 names two literals as NetBox-assigned and "NOT hardcoded in this decision": the org ULA /48, and the per-DC GUA carve out @@ -47,39 +63,47 @@ NetBox. (Precedent: D-102's MTU sub-policy was folded into D-101 only after explicit operator ratification; this carve scheme has not had that pass.) -SUBCARVE SCHEME (proposed, unratified -- review before real use): - - ULA: each DC gets a /56 out of the org /48 (dc1 = ::0/56, dc2 = ::1:00/56, - i.e. the 8 bits immediately after the /48 index the DC -- room for up to - 256 DCs, far beyond VR1's two). Within a DC's /56, each of the three - ULA-family planes (data-tenant, storage, replication) plus the two - dual-stack planes' ULA leg (metal-admin, metal-internal) gets a /64, - indexed in the fixed plane order below (provider-public has no ULA leg, - so its index is skipped -- the /64 offsets are NOT contiguous with the - plane list index, they are assigned only to the five ULA-bearing planes). - - GUA: each DC's already-carved GUA prefix (DC_GUA_PREFIX, e.g. a /40 or - /44 -- length is read from what's actually passed in, not assumed) is - subdivided the same way: provider-public gets the whole thing as a - single /64 (VR1 has no other GUA-bearing plane in the matrix above; - tenant-facing GUA delegation is a separate, later, per-tenant carve out - of this same block -- not this script's job, see netbox/README.md). +SUBCARVE SCHEME (D-111 -- aligned to the DEPLOYED VR0-DC0/Willamette NN +mnemonic; see the D-111 decision record + docs/dc-dc-netbox-buildout-scope.md +section 5.1. Supersedes the original contiguous-plane-index proposal): + - Net-byte (NN) offsets match the live cloud: provider :10 (+ API VIP :11), + metal :20/:21, data :30, storage :40, replication :50. Each plane gets a + /60 container + a /64 active, mirroring the deployed template. + - GUA: provider-public only (D-101 family matrix). Out of DC_GUA_PREFIX (the + per-DC site /48, e.g. 2602:f3e2:f02::/48 for VR1 DC0): /60 + /64 active at :10, + API-VIP /64 at :11. + - ULA: the five internal planes, out of a per-DC /56 of the org ULA /48. The + /56 index is the GUA site nibble (VR1 DC0 -> ...:02xx, VR1 DC1 -> ...:03xx) so the + 4th hextet reads DC.NN in BOTH families. metal-admin (:20) and metal-internal + (:21) SHARE the metal /60 (at :20), exactly as provider (:10) / API-VIP (:11) + share the provider /60. data-tenant :30, storage :40, replication :50 each + get their own /60 + /64. + - Addresses are computed by direct index arithmetic (_sub_at), NEVER + list(...subnets()) enumeration -- the DOCFIX-181 no-hang property is preserved. Idempotent: re-running is safe (existing prefixes are detected and skipped unless --update). NetBox version: 4.x (scope_type/scope_id, not legacy site=). Usage: - # DC1 (v4 inherited, only v6 top-level blocks required): + # DRY-RUN IS THE DEFAULT (D-117). Nothing is written without --commit. + + # VR1 DC0 (v4 inherited from VR0 DC0; only v6 top-level blocks required): NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=<token> \\ - ORG_ULA_48=fd00:1234:5678::/48 DC_GUA_PREFIX=2602:f3e2:1000::/40 \\ - python3 dc-dc-prefixes-import.py --dc dc1 + ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f02::/48 \\ + python3 dc-dc-prefixes-import.py --dc vr1-dc0 # preview + ... python3 dc-dc-prefixes-import.py --dc vr1-dc0 --commit # write - # DC2 (v4 supernet ALSO required, nothing defaulted): + # VR1 DC1 (v4 supernet ALSO required, nothing defaulted): NETBOX_URL=... NETBOX_TOKEN=... \\ - ORG_ULA_48=fd00:1234:5678::/48 DC_GUA_PREFIX=2602:f3e2:2000::/40 \\ - DC2_V4_SUPERNET=10.13.0.0/19 \\ - python3 dc-dc-prefixes-import.py --dc dc2 + ORG_ULA_48=fd50:840e:74e2::/48 DC_GUA_PREFIX=2602:f3e2:f03::/48 \\ + VR1_DC1_V4_SUPERNET=10.12.64.0/19 \\ + python3 dc-dc-prefixes-import.py --dc vr1-dc1 --commit - # Preview only, no writes: - ... python3 dc-dc-prefixes-import.py --dc dc1 --verify-only + # DC_GUA_PREFIX is CROSS-CHECKED against --dc (D-119): a mismatched pair is + # rejected, not silently written. + + # Verification block only (no plan, no writes): + ... python3 dc-dc-prefixes-import.py --dc vr1-dc0 --verify-only KNOWN PRE-EXISTING STALENESS (flagged, not fixed here): netbox/ipv4-prefixes- import.py's role slugs (provider/metal/lbaas-management/openstack-tenant) @@ -103,6 +127,7 @@ import ipaddress import os import sys +import urllib.parse try: import pynetbox @@ -111,14 +136,48 @@ sys.exit(1) # ----------------------------------------------------------------------------- -# DC1/DC2 site identity. Slugs are new -- no prior convention existed in this -# repo (grepped opentofu/*.tf and the buildout design; none found) -- chosen -# to match the D-106 Designate naming already ratified (dc1/dc2, vr1 label). +# VR1 DC site identity. +# +# HISTORY (kept deliberately -- this is the bug's own confession): the original +# comment here read "Slugs are new -- no prior convention existed in this repo +# (grepped opentofu/*.tf and the buildout design; none found)". It grepped the +# REPO. The convention lived in the APEX (NetBox), which is precisely where +# D-010/D-103 say naming authority lives. That is how the off-by-one got in. # ----------------------------------------------------------------------------- +# D-119: THE SELECTOR *IS* THE APEX SLUG. This is an IDENTITY map, and that is the +# whole point -- there is no offset table left to get wrong. The original defect was +# a WRONG LOOKUP TABLE (--dc dc1 bound to slug vr1-dc1 while treating f02::/48 as +# dc1's, though the apex binds f02 to VR1 DC0), which would have written the FIRST +# DC's prefixes onto the SECOND DC's site, silently. Deleting the mapping deletes +# the bug class. +# +# Slugs verified live against netbox.baldurkeep.com (2026-07-14): +# 2602:f3e2:f02::/48 -> site vr1-dc0 "Virtual Region 1 (VR1) Datacenter 0 (DC0)" +# 2602:f3e2:f03::/48 -> site vr1-dc1 "Virtual Region 1 (VR1) Datacenter 1 (DC1)" +# They are NOT a repo-local invention. Do not "fix" them to dc1/dc2 -- that is the +# exact off-by-one D-117 closed and D-119 made unrepresentable. +# A sandbox is local or the known Office1 sandbox address; anything else is treated +# as the production apex and needs --yes-write-upstream (see the guard in main()). +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + SITES = { - "dc1": {"slug": "vr1-dc1", "name": "VR1 DC1"}, - "dc2": {"slug": "vr1-dc2", "name": "VR1 DC2"}, + "vr1-dc0": {"slug": "vr1-dc0", "name": "VR1 DC0"}, + "vr1-dc1": {"slug": "vr1-dc1", "name": "VR1 DC1"}, +} +# The structural invariant D-119 buys us. If a future edit reintroduces an offset, +# this fails at import time rather than silently mis-binding a datacenter. +assert all(k == v["slug"] for k, v in SITES.items()), \ + "SITES must be an IDENTITY map (D-119): the --dc selector IS the apex site slug" + +# The apex's MEASURED DC->GUA binding. The selector cannot be allowed to disagree +# with the addressing: without this, `--dc vr1-dc0 DC_GUA_PREFIX=<f03>` is accepted +# and writes the SECOND DC's GUA, with the FIRST DC's ULA nibble, onto the FIRST +# DC's site -- a silently mis-bound datacenter assembled from two disagreeing +# sources. Identity-mapping the slug alone does NOT close that; this does. +EXPECTED_GUA = { + "vr1-dc0": "2602:f3e2:f02::/48", + "vr1-dc1": "2602:f3e2:f03::/48", } # Fixed plane order -- must match scripts/lib-net.sh's PLANE_NAME exactly @@ -151,7 +210,7 @@ # historical single-plane set is NOT the comparison here -- these are the # CURRENT six-plane values from scripts/lib-net.sh's PLANE_CIDRS, which is # this repo's single source of truth for that layout. -DC1_V4_PREFIXES = { +VR0_DC0_TEMPLATE_V4 = { "provider-public": "10.12.4.0/22", "metal-admin": "10.12.8.0/22", "metal-internal": "10.12.12.0/22", @@ -160,12 +219,12 @@ "replication": "10.12.36.0/22", } -# DC2 v4: NO default. Must come from DC2_V4_SUPERNET (env var), carved into +# VR1 DC1 (the SECOND DC) v4: NO default. From VR1_DC1_V4_SUPERNET (env), carved into # six sequential /22s in PLANE_ORDER, exactly mirroring DC1's /22-per-plane # structure. The supernet must be large enough to hold six non-overlapping # /22s (a /19 holds exactly eight /22s -- six used, two spare; anything # smaller than /19 FAILS LOUD). -DC2_MIN_SUPERNET_PREFIXLEN = 19 # a /19 = exactly 8x /22, room for the six planes +VR1_DC1_MIN_SUPERNET_PREFIXLEN = 19 # a /19 = exactly 8x /22, room for the six planes def die(msg: str, code: int = 1) -> None: @@ -181,6 +240,14 @@ if not token: die("NETBOX_TOKEN environment variable not set") nb = pynetbox.api(url, token=token) + # WAF TRAP (references/platform-traps.md): upstream NetBox 403s the DEFAULT + # python-requests User-Agent -- it looks EXACTLY like an auth failure but is + # not (the same token works from curl). pynetbox is bitten too. The stdlib + # tools (prod-draft-dump/sandbox-seed/d115-office-carve) already send an + # accepted UA; these two pynetbox importers did NOT, so they could never have + # written to the upstream apex (only the sandbox, which has no WAF). Set it + # on the underlying requests session so EVERY call carries it. + nb.http_session.headers["User-Agent"] = "curl/8.5.0" try: _ = nb.status() except Exception as exc: # noqa: BLE001 @@ -188,10 +255,28 @@ return nb -def find_or_create_site(nb, slug: str, name: str): +class _PlannedSite: + """A site that does not exist yet, in a run that is not allowed to create it. + + D-117: a dry run against a FRESH NetBox must still be able to show the whole + plan -- that is the entire point of previewing against an empty apex. Bailing + out with "no site id" would make --dry-run useless exactly when it matters + most. Nothing is ever written from one of these: id is None, and every write + path returns before touching the API.""" + def __init__(self, slug: str, name: str): + self.id = None + self.slug = slug + self.name = name + + +def find_or_create_site(nb, slug: str, name: str, commit: bool): + """D-117: a site is a WRITE. Without --commit we return a _PlannedSite stub + instead of creating one, so the rest of the plan can still be printed.""" site = nb.dcim.sites.get(slug=slug) if site is not None: return site, False + if not commit: + return _PlannedSite(slug, name), False # Minimal site record -- status defaults to NetBox's own default (active). created = nb.dcim.sites.create(name=name, slug=slug) return created, True @@ -259,52 +344,76 @@ return net -def validate_dc2_supernet(cidr: str) -> ipaddress.IPv4Network: +def validate_vr1_dc1_supernet(cidr: str) -> ipaddress.IPv4Network: try: net = ipaddress.ip_network(cidr, strict=True) except ValueError as exc: - die(f"DC2_V4_SUPERNET='{cidr}' is not a valid CIDR: {exc}") + die(f"VR1_DC1_V4_SUPERNET='{cidr}' is not a valid CIDR: {exc}") if net.version != 4: - die(f"DC2_V4_SUPERNET='{cidr}' is not an IPv4 prefix") - if net.prefixlen > DC2_MIN_SUPERNET_PREFIXLEN: + die(f"VR1_DC1_V4_SUPERNET='{cidr}' is not an IPv4 prefix") + if net.prefixlen > VR1_DC1_MIN_SUPERNET_PREFIXLEN: die( - f"DC2_V4_SUPERNET='{cidr}' (/{net.prefixlen}) is too small to hold " - f"six /22 planes -- need at least a /{DC2_MIN_SUPERNET_PREFIXLEN}" + f"VR1_DC1_V4_SUPERNET='{cidr}' (/{net.prefixlen}) is too small to hold " + f"six /22 planes -- need at least a /{VR1_DC1_MIN_SUPERNET_PREFIXLEN}" ) - dc1_nets = [ipaddress.ip_network(c) for c in DC1_V4_PREFIXES.values()] - for d1 in dc1_nets: + tmpl_nets = [ipaddress.ip_network(c) for c in VR0_DC0_TEMPLATE_V4.values()] + for d1 in tmpl_nets: if net.overlaps(d1): die( - f"DC2_V4_SUPERNET='{cidr}' overlaps DC1's inherited range {d1} -- " - f"D-101 requires DC2 to be non-overlapping. Choose a different supernet." + f"VR1_DC1_V4_SUPERNET='{cidr}' overlaps VR1 DC0's inherited range {d1} -- " + f"D-101 requires VR1 DC1 to be non-overlapping. Choose a different supernet." ) return net -def carve_dc2_v4(supernet: ipaddress.IPv4Network) -> dict: +def carve_vr1_dc1_v4(supernet: ipaddress.IPv4Network) -> dict: subnets = list(supernet.subnets(new_prefix=22)) if len(subnets) < len(PLANE_ORDER): die( - f"DC2_V4_SUPERNET yields only {len(subnets)} /22 subnet(s), " + f"VR1_DC1_V4_SUPERNET yields only {len(subnets)} /22 subnet(s), " f"need {len(PLANE_ORDER)}" ) return {plane: str(subnets[i]) for i, plane in enumerate(PLANE_ORDER)} -def carve_ula(org_ula_48: ipaddress.IPv6Network, dc: str) -> dict: - dc_index = 0 if dc == "dc1" else 1 - # /48 -> per-DC /56 (8-bit DC index). See SUBCARVE SCHEME in module docstring. - dc_56_candidates = list(org_ula_48.subnets(new_prefix=56)) - dc_56 = dc_56_candidates[dc_index] - ula_planes = [p for p in PLANE_ORDER if PLANE_V6_FAMILY[p] in ("ula", "ula-only")] - dc_64_candidates = list(dc_56.subnets(new_prefix=64)) - return {plane: str(dc_64_candidates[i]) for i, plane in enumerate(ula_planes)} +# D-111 alignment: lay the v6 legs out on the DEPLOYED VR0-DC0/Willamette NN +# net-byte mnemonic (provider :10 [+VIP :11], metal :20/:21, data :30, storage +# :40, replication :50), /60-per-plane container + /64 active, mirroring the +# live template. Replaces the earlier contiguous-plane-index subcarve. See the +# D-111 decision record + docs/dc-dc-netbox-buildout-scope.md section 5.1. +# +# Per-DC ULA /56 index aligned to the GUA site nibble (VR1 DC0 -> ...:02xx, VR1 DC1 -> +# ...:03xx) so the 4th hextet reads DC.NN in BOTH families. +DC_V6_INDEX = {"vr1-dc0": 0x02, "vr1-dc1": 0x03} -def carve_gua(dc_gua_prefix: ipaddress.IPv6Network) -> dict: - # Only provider-public carries GUA in the current family matrix. - gua_64_candidates = list(dc_gua_prefix.subnets(new_prefix=64)) - return {"provider-public": str(gua_64_candidates[0])} +def _sub_at(base: ipaddress.IPv6Network, new_prefixlen: int, index: int) -> ipaddress.IPv6Network: + """The index-th subnet of `base` at new_prefixlen, computed by direct + arithmetic -- NOT list(base.subnets(...)) enumeration, which would + materialize up to millions of objects and hang (DOCFIX-181).""" + if new_prefixlen < base.prefixlen: + die(f"internal: new_prefixlen /{new_prefixlen} shorter than base {base}") + net_int = int(base.network_address) + index * (1 << (128 - new_prefixlen)) + return ipaddress.ip_network((net_int, new_prefixlen)) + + +def carve_v6(dc: str, org_ula_48: ipaddress.IPv6Network, dc_gua_prefix: ipaddress.IPv6Network) -> list: + """Return [(cidr, role_slug, kind)] for this DC's v6 legs on the deployed NN + layout. provider-public = GUA out of DC_GUA_PREFIX; the five internal planes + = ULA out of a per-DC /56 of the org ULA /48 (D-101 family matrix).""" + ula = _sub_at(org_ula_48, 56, DC_V6_INDEX[dc]) + recs = [ + (str(_sub_at(dc_gua_prefix, 60, 0x10 >> 4)), "provider-public", "GUA /60"), + (str(_sub_at(dc_gua_prefix, 64, 0x10)), "provider-public", "GUA /64 active"), + (str(_sub_at(dc_gua_prefix, 64, 0x11)), "provider-public", "GUA VIP /64"), + (str(_sub_at(ula, 60, 0x20 >> 4)), "metal-admin", "ULA metal /60 (admin+internal)"), + (str(_sub_at(ula, 64, 0x20)), "metal-admin", "ULA /64 active"), + (str(_sub_at(ula, 64, 0x21)), "metal-internal", "ULA /64 active (in metal /60)"), + ] + for plane, nn in (("data-tenant", 0x30), ("storage", 0x40), ("replication", 0x50)): + recs.append((str(_sub_at(ula, 60, nn >> 4)), plane, "ULA /60")) + recs.append((str(_sub_at(ula, 64, nn)), plane, "ULA /64 active")) + return recs # ----------------------------------------------------------------------------- @@ -312,7 +421,10 @@ # ----------------------------------------------------------------------------- -def create_or_report_prefix(nb, cidr: str, role_slug: str, site, description: str, update: bool) -> None: +def create_or_report_prefix(nb, cidr: str, role_slug: str, site, description: str, update: bool, commit: bool) -> None: + """D-117: DRY BY DEFAULT. Without commit=True this writes NOTHING -- it only + prints the plan. This tool used to write by default, which is how an + off-by-one site binding could have landed in the IPAM apex unannounced.""" role = find_role(nb, role_slug) existing = nb.ipam.prefixes.get(prefix=cidr) payload = { @@ -323,16 +435,22 @@ "scope_id": site.id, } if existing is None: + if not commit: + print(f" [dry-run] would CREATE prefix {cidr} role={role_slug} -> site {site.name}") + return created = nb.ipam.prefixes.create(**payload) print(f" CREATED prefix {cidr} (id={created.id}) role={role_slug}") elif update: + if not commit: + print(f" [dry-run] would UPDATE prefix {cidr} (id={existing.id}) role={role_slug} -> site {site.name}") + return existing.update(payload) print(f" UPDATED prefix {cidr} (id={existing.id}) role={role_slug}") else: print(f" EXISTS prefix {cidr} (id={existing.id}) -- skipped (use --update to overwrite)") -def verify(nb, site, v4: dict, v6_ula: dict, v6_gua: dict) -> None: +def verify(nb, site, v4: dict, v6_recs: list) -> None: print() print("=" * 72) print(f"Verification -- final state for site {site.name} (id={site.id})") @@ -343,78 +461,130 @@ p = nb.ipam.prefixes.get(prefix=cidr) status = "MISSING" if p is None else f"OK (id={p.id})" print(f" {plane.ljust(16)} {cidr.ljust(18)} {status}") - print("\nIPv6 ULA (per plane, where applicable):") - for plane, cidr in v6_ula.items(): + print("\nIPv6 (D-101 family matrix / D-111 NN layout):") + for cidr, role_slug, kind in v6_recs: p = nb.ipam.prefixes.get(prefix=cidr) status = "MISSING" if p is None else f"OK (id={p.id})" - print(f" {plane.ljust(16)} {cidr.ljust(24)} {status}") - print("\nIPv6 GUA (per plane, where applicable):") - for plane, cidr in v6_gua.items(): - p = nb.ipam.prefixes.get(prefix=cidr) - status = "MISSING" if p is None else f"OK (id={p.id})" - print(f" {plane.ljust(16)} {cidr.ljust(24)} {status}") + print(f" {role_slug.ljust(16)} {cidr.ljust(26)} {kind.ljust(24)} {status}") def main() -> int: parser = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) - parser.add_argument("--dc", required=True, choices=["dc1", "dc2"], help="Which DC to import (no default -- explicit, measured selection per the $DC convention)") - parser.add_argument("--update", action="store_true", help="Update existing prefixes in place") - parser.add_argument("--verify-only", action="store_true", help="Skip writes; only print verification block") + parser.add_argument("--dc", required=True, choices=["vr1-dc0", "vr1-dc1"], help="Which VR1 DC to import. D-119: the selector IS the apex site slug (identity map, no offset table). Bare dc0/dc1/dc2 are RETIRED -- they were ambiguous across regions.") + parser.add_argument("--update", action="store_true", help="Update existing prefixes in place (requires --commit)") + parser.add_argument("--commit", action="store_true", help="WRITE to NetBox. Default is a DRY RUN that writes nothing (D-117 -- this tool used to write by default)") + parser.add_argument("--verify-only", action="store_true", help="Skip the plan entirely; only print the verification block") + parser.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox. Feeding " + "the production apex is an operator decision, not an import side effect.") args = parser.parse_args() + # UPSTREAM-WRITE GUARD (2026-07-14). This tool had NO hostname gate, and until + # today the WAF's 403 on the default User-Agent was the only thing stopping an + # accidental --commit against the production apex. Now that get_nb() sends an + # accepted UA, that accidental-safety is GONE -- so an explicit gate is required, + # matching d115-office-carve.py. A sandbox is local or the known Office1 address; + # anything else is treated as production. + if args.commit and not args.verify_only: + _host = (urllib.parse.urlparse(os.environ.get("NETBOX_URL", "")).hostname or "").lower() + if _host not in SANDBOX_HOSTS and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{_host or os.environ.get('NETBOX_URL','?')}': it is not a " + "known sandbox, so this is treated as the PRODUCTION apex. Feeding validated prefixes " + "upstream is an operator decision (C2), not a side effect of running an import. " + "Re-run with --yes-write-upstream if that is genuinely what you intend.") + + # D-119: reject BOTH historical names by NAME rather than ignoring them. An + # operator with muscle memory could plausibly export either -- and both are now + # ambiguous ("DC1" could mean VR0's dc0 OR VR1's second DC). Silence would give + # them "required env missing" with no hint the variable moved under them. + for retired in ("DC1_V4_SUPERNET", "DC2_V4_SUPERNET"): + if os.environ.get(retired) and not os.environ.get("VR1_DC1_V4_SUPERNET"): + die(f"{retired} is set, but that name is RETIRED (D-119). An unqualified " + "'DC1'/'DC2' is exactly the cross-region ambiguity D-119 deletes. The " + "variable is now VR1_DC1_V4_SUPERNET (VR1's SECOND DC, slug vr1-dc1). " + "Re-export it under the new name -- refusing to guess.") + dc = args.dc + commit = args.commit site_cfg = SITES[dc] org_ula_48 = validate_ula_48(require_env("ORG_ULA_48", "D-101's org ULA /48 -- not defaulted, must be a real RFC 4193 /48 you generated")) dc_gua_prefix = validate_gua_prefix(require_env("DC_GUA_PREFIX", f"the {dc} carve out of ARIN 2602:f3e2::/32 region-0 /36 cited in D-101 -- not defaulted")) - if dc == "dc1": - v4 = dict(DC1_V4_PREFIXES) - else: - dc2_supernet = validate_dc2_supernet(require_env("DC2_V4_SUPERNET", "D-101 explicitly requires DC2's v4 supernet be NetBox-assigned, not hardcoded")) - v4 = carve_dc2_v4(dc2_supernet) + # D-119 BREAK-2 GUARD. The selector must not be allowed to disagree with the + # addressing. Without this, `--dc vr1-dc0 DC_GUA_PREFIX=<f03::/48>` is accepted: + # it writes the SECOND DC's GUA prefixes, with the FIRST DC's ULA nibble (the + # ULA index is keyed off --dc, independently), all scoped to the FIRST DC's + # site -- a silently mis-bound datacenter assembled from two disagreeing + # sources. Identity-mapping the site slug does NOT close this hole; this does. + if str(dc_gua_prefix) != EXPECTED_GUA[dc]: + die(f"DC_GUA_PREFIX={dc_gua_prefix} does not match {dc}'s MEASURED apex block " + f"({EXPECTED_GUA[dc]}). Refusing to bind one DC's addressing to another " + f"DC's site -- that is the exact off-by-one D-117/D-119 exist to kill.") - v6_ula = carve_ula(org_ula_48, dc) - v6_gua = carve_gua(dc_gua_prefix) + if dc == "vr1-dc0": + v4 = dict(VR0_DC0_TEMPLATE_V4) + else: + second_supernet = validate_vr1_dc1_supernet(require_env("VR1_DC1_V4_SUPERNET", "D-101 explicitly requires the SECOND DC's v4 supernet be NetBox-assigned, not hardcoded (D-115 assigns 10.12.64.0/19)")) + v4 = carve_vr1_dc1_v4(second_supernet) + + v6_recs = carve_v6(dc, org_ula_48, dc_gua_prefix) nb = get_nb() - site, site_created = find_or_create_site(nb, site_cfg["slug"], site_cfg["name"]) - print(f"Connected. Site '{site_cfg['name']}' (id={site.id}){' -- CREATED' if site_created else ''}.") - if dc == "dc1": + # --- WHOLE-PLAN PREFLIGHT (2026-07-14 hardening) -------------------------- + # Resolve EVERY IPAM role this run needs BEFORE the first write. find_role() + # used to be called lazily INSIDE the write loop, so against an apex missing a + # six-plane role (true of upstream TODAY) a --commit would create the site and + # the first 4 prefixes, then die on the 5th -- a half-written datacenter in the + # production IPAM, with no rollback. Same failure class as the roles importer's + # 2026-07-13 partial write. An importer that can die partway through corrupts + # the thing it exists to populate. + needed_roles = list(PLANE_ORDER) + [r for _c, r, _k in v6_recs] + missing = sorted({r for r in needed_roles if nb.ipam.roles.get(slug=r) is None}) + if missing: + print("PREFLIGHT FAILED -- nothing was written:\n") + for r in missing: + print(f" * IPAM role {r!r} does not exist in NetBox") + die("create the role(s) above (netbox/roles-aggregates-import.py), then re-run. " + "NO changes were made -- refusing to half-write a datacenter.") + # ------------------------------------------------------------------------- + + site, site_created = find_or_create_site(nb, site_cfg["slug"], site_cfg["name"], commit) + if isinstance(site, _PlannedSite): + print(f"[dry-run] would CREATE site '{site.name}' ({site.slug}) -- it does not exist yet.") + else: + print(f"Connected. Site '{site_cfg['name']}' (id={site.id}){' -- CREATED' if site_created else ''}.") + + if dc == "vr1-dc0": print( "\nNOTE: this may create prefixes identical to the existing 'vr0-dc0' " - "site's entries (D-101: DC1 inherits DC0's v4 layout unchanged). " + "site's entries (D-101: VR1 DC0 inherits VR0 DC0's v4 layout unchanged). " "NetBox permits duplicate global prefixes (soft-warn, not a hard " "block) but this is a real IPAM-hygiene decision for the operator: " - "either retire/delete the vr0-dc0 entries once DC1 supersedes that " + "either retire/delete the vr0-dc0 entries once VR1 DC0 supersedes that " "rehearsal environment, or knowingly keep both for historical " "record. This script does not decide that for you." ) if not args.verify_only: + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit to apply. ***" + if not commit else "\n*** COMMITTING to NetBox. ***") print("\nIPv4 prefixes:") for plane in PLANE_ORDER: create_or_report_prefix( nb, v4[plane], plane, site, - f"VR1 {dc.upper()} {plane} (v4; D-101)", args.update, + f"{site_cfg['name']} {plane} (v4; D-101)", args.update, commit, ) - print("\nIPv6 ULA prefixes:") - for plane, cidr in v6_ula.items(): + print("\nIPv6 prefixes (D-101 family matrix on the D-111 NN layout):") + for cidr, role_slug, kind in v6_recs: create_or_report_prefix( - nb, cidr, plane, site, - f"VR1 {dc.upper()} {plane} (ULA leg; D-101, proposed subcarve -- see script docstring)", - args.update, - ) - print("\nIPv6 GUA prefixes:") - for plane, cidr in v6_gua.items(): - create_or_report_prefix( - nb, cidr, plane, site, - f"VR1 {dc.upper()} {plane} (GUA leg; D-101, proposed subcarve -- see script docstring)", - args.update, + nb, cidr, role_slug, site, + f"{site_cfg['name']} {role_slug} ({kind}; D-101/D-111)", + args.update, commit, ) - verify(nb, site, v4, v6_ula, v6_gua) + verify(nb, site, v4, v6_recs) print("\nDone.") return 0 diff --git a/netbox/draft/vr1-draft.json b/netbox/draft/vr1-draft.json new file mode 100644 index 0000000..e358f73 --- /dev/null +++ b/netbox/draft/vr1-draft.json @@ -0,0 +1,1132 @@ +{ + "_note": "READ-ONLY dump. IDs discarded; refs are by slug.", + "_source": "https://netbox.baldurkeep.com", + "dcim/regions": [ + { + "description": "Top-level region for US West metros. Holds Eugene (and future West Coast metros).", + "name": "US West", + "parent": null, + "slug": "us-west" + }, + { + "description": "Region0 \u2014 Eugene, OR metro. Nested under US West. Assigned 10.16.0.0/16 IPv4.", + "name": "Eugene", + "parent": "us-west", + "slug": "eugene" + }, + { + "description": "Virtual Regions for testing", + "name": "Virtual Regions", + "parent": null, + "slug": "virtual-region" + }, + { + "description": "Virtual Region for testing", + "name": "VR0", + "parent": "virtual-region", + "slug": "vr0" + }, + { + "description": "Virtual Region for testing", + "name": "VR1", + "parent": "virtual-region", + "slug": "vr1" + } + ], + "dcim/sites": [ + { + "description": "Primary office \u2014 Office0 at 10.16.4.0/24", + "name": "Charnelton", + "region": "eugene", + "slug": "charnelton", + "status": "active" + }, + { + "description": "Future manufacturing/OT \u2014 Cloud3 at 10.16.64.0/20 (reserved)", + "name": "Cleveland", + "region": "eugene", + "slug": "cleveland", + "status": "planned" + }, + { + "description": "Primary DC and current Internet edge \u2014 Cloud0 at 10.16.16.0/20", + "name": "Roosevelt", + "region": "eugene", + "slug": "roosevelt", + "status": "active" + }, + { + "description": "Future tertiary DC \u2014 Cloud2 at 10.16.48.0/20 (reserved)", + "name": "Stevenson", + "region": "eugene", + "slug": "stevenson", + "status": "planned" + }, + { + "description": "Virtual Site for testing", + "name": "VR0 DC0", + "region": "vr0", + "slug": "vr0-dc0", + "status": "active" + }, + { + "description": "Virtual Site for testing", + "name": "VR0 DC1", + "region": "vr0", + "slug": "vr0-dc1", + "status": "active" + }, + { + "description": "Virtual Site for testing", + "name": "VR0 Off0", + "region": "vr0", + "slug": "vr0-off0", + "status": "active" + }, + { + "description": "Virtual Site for testing", + "name": "VR1 DC0", + "region": "vr1", + "slug": "vr1-dc0", + "status": "active" + }, + { + "description": "Virtual Site for testing", + "name": "VR1 DC1", + "region": "vr1", + "slug": "vr1-dc1", + "status": "active" + }, + { + "description": "Secondary DC \u2014 Cloud1 at 10.16.32.0/20. Future Internet edge once ARIN allocation and transit contracts move.", + "name": "Willamette", + "region": "eugene", + "slug": "willamette", + "status": "active" + } + ], + "ipam/aggregates": [], + "ipam/prefixes": [ + { + "description": "Local Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "0.0.0.0/8", + "role": "localnet", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Corp Private", + "is_pool": false, + "mark_utilized": false, + "prefix": "10.0.0.0/8", + "role": "private", + "scope_type": null, + "status": "container" + }, + { + "description": "Virtual Cloud IPs", + "is_pool": false, + "mark_utilized": false, + "prefix": "10.12.0.0/16", + "role": "cloud", + "scope_type": null, + "status": "active" + }, + { + "description": "Current assignment to Willamette dev cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "10.16.0.0/16", + "role": "dc", + "scope_type": null, + "status": "deprecated" + }, + { + "description": "Current assignment to Roosevelt dev cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "10.17.0.0/16", + "role": "dc", + "scope_type": null, + "status": "deprecated" + }, + { + "description": "Public ARIN assignment to Abysius Inc", + "is_pool": false, + "mark_utilized": false, + "prefix": "23.157.124.0/24", + "role": "public", + "scope_type": null, + "status": "container" + }, + { + "description": "CGNAT Reserved", + "is_pool": false, + "mark_utilized": false, + "prefix": "100.64.0.0/10", + "role": "cgnat", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Localhost", + "is_pool": false, + "mark_utilized": false, + "prefix": "127.0.0.0/8", + "role": "localhost", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Link-Local", + "is_pool": false, + "mark_utilized": false, + "prefix": "169.254.0.0/16", + "role": "link-local", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Corp OOB Private", + "is_pool": false, + "mark_utilized": false, + "prefix": "172.16.0.0/12", + "role": "private", + "scope_type": null, + "status": "container" + }, + { + "description": "Residential Private", + "is_pool": false, + "mark_utilized": false, + "prefix": "192.168.0.0/16", + "role": "private", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Public Multicast", + "is_pool": false, + "mark_utilized": false, + "prefix": "224.0.0.0/4", + "role": "multicast", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Public Reserved - Enterprise Private", + "is_pool": false, + "mark_utilized": false, + "prefix": "240.0.0.0/4", + "role": "private", + "scope_type": null, + "status": "container" + }, + { + "description": "Localhost", + "is_pool": false, + "mark_utilized": false, + "prefix": "::1/128", + "role": "localhost", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Discard Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "100::/64", + "role": "discard", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Public ARIN assignment to Abysius Inc", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2::/36", + "role": "public", + "scope_type": null, + "status": "container" + }, + { + "description": "Inter-Regional Infrastructure", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2::/40", + "role": "infra", + "scope_type": null, + "status": "container" + }, + { + "description": "Inter-Regional Site to Site Links", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2::/48", + "role": "infra", + "scope_type": null, + "status": "container" + }, + { + "description": "Edge Networks", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:fe::/48", + "role": null, + "scope_type": null, + "status": "container" + }, + { + "description": "Private cloud tenant reservation", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:ff::/48", + "role": "openstack-tenant", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Eugene Region; Sites receive /24 allocations starting at fc01:0100::/24", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:100::/40", + "role": "region", + "scope_region": "eugene", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Eugene Regional Infrastructure", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:100::/48", + "role": "infra", + "scope_region": "eugene", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Eugene Site to Site Links", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:100::/56", + "role": "infra", + "scope_region": "eugene", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Eugene Regional Offices; Offices are allocated /32 subnets", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:101::/48", + "role": "office", + "scope_region": "eugene", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Reserved for fun and profit", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:101::/56", + "role": "office", + "scope_region": "eugene", + "scope_type": "dcim.region", + "status": "reserved" + }, + { + "description": "Charnelton", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:101:100::/56", + "role": "office", + "scope_site": "charnelton", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Willamette Site", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102::/48", + "role": "dc", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Omega Cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102::/56", + "role": "cloud", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:10::/60", + "role": "provider", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:10::/64", + "role": "provider", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "Willamette Omega Cloud \u2014 Provider API VIPs (charm-managed via hacluster; excluded from Neutron pools)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:11::/64", + "role": "provider", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:20::/60", + "role": "metal", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:20::/64", + "role": "metal", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:30::/60", + "role": "data", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:30::/64", + "role": "data", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:40::/60", + "role": "storage", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:40::/64", + "role": "storage", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:50::/60", + "role": "repl", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:50::/64", + "role": "repl", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:80::/60", + "role": "lbaas-mgmt", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:80::/64", + "role": "lbaas-mgmt", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:e0::/60", + "role": "vpn", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:f0::/60", + "role": "oob", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:f0::/64", + "role": "oob", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "Psi Cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:100::/56", + "role": "cloud", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Beta Cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:1600::/56", + "role": "cloud", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Alpha Cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:102:1700::/56", + "role": "cloud", + "scope_site": "willamette", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Roosevelt Site", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:103::/48", + "role": "dc", + "scope_site": "roosevelt", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Second Region", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:200::/40", + "role": "region", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Third Region", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:300::/40", + "role": "region", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Virtual Region 0 (VR0)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e00::/40", + "role": "region", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Virtual Region 0 (VR0) Infrastructure", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e00::/48", + "role": "infra", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "VR0 Site to Site Links", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e00::/56", + "role": "infra", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "active" + }, + { + "description": "VR0 DC1 to DC0 Site to Site Link", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e00:1::/127", + "role": "infra", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "active" + }, + { + "description": "VR0 Off0 to DC0 Site to Site Link", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e00:2::/127", + "role": "infra", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "active" + }, + { + "description": "Virtual Region 0 (VR0) Offices", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e01::/48", + "role": "office", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Virtual Region 0 (VR0) Offices; Reserved for fun and profit", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e01::/56", + "role": "office", + "scope_region": "vr0", + "scope_type": "dcim.region", + "status": "reserved" + }, + { + "description": "Virtual Region 0 (VR0) Office 0 (Off0)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e01:100::/56", + "role": "office", + "scope_site": "vr0-off0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 Off0 Office Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e01:100::/64", + "role": "office", + "scope_site": "vr0-off0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "Virtual Region 0 (VR0) Datacenter 0 (DC0)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02::/48", + "role": "dc", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02::/56", + "role": "cloud", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud Provider Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:10::/60", + "role": "provider", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud Provider Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:10::/64", + "role": "provider", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud \u2014 Provider API VIPs (charm-managed via hacluster; excluded from Neutron pools)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:11::/64", + "role": "provider", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud Metal Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:20::/60", + "role": "metal", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud Metal Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:20::/64", + "role": "metal", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud Data Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:30::/60", + "role": "data", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud Data Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:30::/64", + "role": "data", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud Storage Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:40::/60", + "role": "storage", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud Storage Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:40::/64", + "role": "storage", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud Replication Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:50::/60", + "role": "repl", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud Replication Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:50::/64", + "role": "repl", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud LBaaS Management Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:80::/60", + "role": "lbaas-mgmt", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud LBaaS Management Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:80::/64", + "role": "lbaas-mgmt", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "VR0 DC0 Omega Cloud VPN Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:e0::/60", + "role": "vpn", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud OOB Prefix", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:f0::/60", + "role": "oob", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "VR0 DC0 Omega Cloud OOB Subnet", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e02:f0::/64", + "role": "oob", + "scope_site": "vr0-dc0", + "scope_type": "dcim.site", + "status": "active" + }, + { + "description": "Virtual Region 0 (VR0) Datacenter 1 (DC1)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:e03::/48", + "role": "dc", + "scope_site": "vr0-dc1", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Virtual Region 1 (VR1)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f00::/40", + "role": "region", + "scope_region": "vr1", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Virtual Region 1 (VR1) Infrastructure", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f00::/48", + "role": "infra", + "scope_region": "vr1", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "VR1 Site to Site Links", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f00::/56", + "role": "infra", + "scope_region": "vr1", + "scope_type": "dcim.region", + "status": "active" + }, + { + "description": "Virtual Region 1 (VR1) Offices", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f01::/48", + "role": "office", + "scope_region": "vr1", + "scope_type": "dcim.region", + "status": "container" + }, + { + "description": "Virtual Region 1 (VR1) Offices; Reserved for fun and profit", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f01::/56", + "role": "office", + "scope_region": "vr1", + "scope_type": "dcim.region", + "status": "reserved" + }, + { + "description": "Virtual Region 1 (VR1) Datacenter 0 (DC0)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f02::/48", + "role": "dc", + "scope_site": "vr1-dc0", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Virtual Region 1 (VR1) Datacenter 1 (DC1)", + "is_pool": false, + "mark_utilized": false, + "prefix": "2602:f3e2:f03::/48", + "role": "dc", + "scope_site": "vr1-dc1", + "scope_type": "dcim.site", + "status": "container" + }, + { + "description": "Centrally assigned ULA (Not Active)", + "is_pool": false, + "mark_utilized": false, + "prefix": "fc00::/8", + "role": "private", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Unique Local Addresses (ULA)", + "is_pool": false, + "mark_utilized": false, + "prefix": "fd00::/8", + "role": "private", + "scope_type": null, + "status": "container" + }, + { + "description": "Tailscale Addresses", + "is_pool": false, + "mark_utilized": true, + "prefix": "fd7a:115c:a1e0::/48", + "role": "vpn", + "scope_type": null, + "status": "active" + }, + { + "description": "Link-Local", + "is_pool": false, + "mark_utilized": false, + "prefix": "fe80::/10", + "role": "link-local", + "scope_type": null, + "status": "reserved" + }, + { + "description": "Public Multicast", + "is_pool": false, + "mark_utilized": false, + "prefix": "ff00::/8", + "role": "multicast", + "scope_type": null, + "status": "reserved" + } + ], + "ipam/rirs": [ + { + "description": "", + "is_private": false, + "name": "ARIN", + "slug": "arin" + } + ], + "ipam/roles": [ + { + "description": "", + "name": "CGNAT", + "slug": "cgnat", + "weight": 1000 + }, + { + "description": "Cloud container", + "name": "Cloud", + "slug": "cloud", + "weight": 1000 + }, + { + "description": "Openstack data plane - shared across datacenters", + "name": "Data", + "slug": "data", + "weight": 1000 + }, + { + "description": "Datacenter container", + "name": "Datacenter", + "slug": "dc", + "weight": 1000 + }, + { + "description": "", + "name": "Discard", + "slug": "discard", + "weight": 1000 + }, + { + "description": "Shared infrastructure \u2014 ptp transit, device loopbacks, RR loopbacks, anycast.", + "name": "Infrastructure", + "slug": "infra", + "weight": 1000 + }, + { + "description": "Load-balancer management (Octavia control plane, not dataplane).", + "name": "LBaaS Management", + "slug": "lbaas-mgmt", + "weight": 1000 + }, + { + "description": "", + "name": "Link-Local", + "slug": "link-local", + "weight": 1000 + }, + { + "description": "", + "name": "Localhost", + "slug": "localhost", + "weight": 1000 + }, + { + "description": "", + "name": "Localnet", + "slug": "localnet", + "weight": 1000 + }, + { + "description": "Bare-metal server NICs + bundled infra: OpenStack control plane, in-band switch mgmt (VLAN 50), Edge-eBGP/DDoS sub-ranges (Roosevelt), RR loopback references. Sized as 2x physical server count.", + "name": "Metal", + "slug": "metal", + "weight": 1000 + }, + { + "description": "", + "name": "Multicast", + "slug": "multicast", + "weight": 1000 + }, + { + "description": "Office site allocation (/24 within Region0 /20 reserved block).", + "name": "Office", + "slug": "office", + "weight": 1000 + }, + { + "description": "Out of band management network", + "name": "OOB", + "slug": "oob", + "weight": 1000 + }, + { + "description": "OpenStack tenant pool \u2014 shared across tenants", + "name": "Openstack Tenant", + "slug": "openstack-tenant", + "weight": 1000 + }, + { + "description": "Reserved addresses for private networks", + "name": "Private", + "slug": "private", + "weight": 1000 + }, + { + "description": "Tenant-reachable. Carries ext_net (FIPs, SNAT egress) and OpenStack public API VIPs. Within each /60: first /64 Neutron FIP pool; next /64 charm API VIPs. Neutron allocation_pools must exclude VIPs.", + "name": "Provider", + "slug": "provider", + "weight": 1000 + }, + { + "description": "Public addresses assigned by a RIR", + "name": "Public", + "slug": "public", + "weight": 1000 + }, + { + "description": "", + "name": "Region", + "slug": "region", + "weight": 1000 + }, + { + "description": "RBD mirror for cross-site Ceph replication. Routed via EVPN type-5 between DCs.", + "name": "Replication", + "slug": "repl", + "weight": 1000 + }, + { + "description": "Ceph-Public (L3-routed) + Ceph-Cluster (non-routed VLAN inside /23 L2).", + "name": "Storage", + "slug": "storage", + "weight": 1000 + }, + { + "description": "OpenStack tenant networks (Geneve underlay touchdown). Also absorbs DMZ roles under regional scheme.", + "name": "Tenant", + "slug": "tenant", + "weight": 1000 + }, + { + "description": "VPN client pools \u2014 site-independent (10.2.0.0/16 Corp Private).", + "name": "VPN", + "slug": "vpn", + "weight": 1000 + } + ] +} diff --git a/netbox/prod-draft-dump.py b/netbox/prod-draft-dump.py new file mode 100755 index 0000000..af7793a --- /dev/null +++ b/netbox/prod-draft-dump.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +""" +Dump the UPSTREAM NetBox draft to a reviewable JSON artifact. READ-ONLY. + +This is phase 1 of the two-phase sandbox seeding loop documented in +docs/session-ledger.md: + + upstream (draft source) --[this script, READ-ONLY]--> vr1-draft.json + vr1-draft.json --[netbox/sandbox-seed.py]--> Office1 sandbox + ...simulate VR1 in the sandbox, apply planned changes (D-115/D-117)... + validated refinements --[operator-gated]----------> upstream + +WHY TWO PHASES instead of one source->dest script: + + 1. The upstream token NEVER travels. This script runs on vcloud (where + ~/vr1-office1-creds/vr1-netbox.env lives); sandbox-seed.py runs on office1-netbox (where + the sandbox token lives). Neither host holds the other's credential. + 2. "The sim NEVER writes upstream" becomes STRUCTURAL, not a discipline: the + script that writes has no upstream credentials and no upstream URL. + 3. The JSON dump is a reviewable, diffable artifact. You can read exactly what + is about to be seeded before anything is written. + +This script performs NO writes of any kind. It has no --commit flag because it +has nothing to commit. + +TRAP -- User-Agent (measured 2026-07-13, see references/platform-traps.md): +upstream NetBox sits behind a filter that 403s the default Python User-Agent. +The SAME token works from curl. It looks exactly like an auth failure and it is +not. We send an accepted UA. pynetbox (python-requests/...) is bitten too. + +Usage: + . ~/vr1-office1-creds/vr1-netbox.env # NETBOX_URL + NETBOX_TOKEN + python3 netbox/prod-draft-dump.py --out netbox/draft/vr1-draft.json +""" + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request + +# Upstream is 4.5.8 and uses v1 (bare, 40-char) tokens. The Office1 sandbox is +# 4.6.4 and uses v2 (nbt_<key>.<plaintext>). Do NOT assume one shape fits both. +UA = "curl/8.5.0" + +# What the VR1 carve actually needs to exist. Order is DEPENDENCY order and the +# seeder replays it verbatim -- parents before children. +ENDPOINTS = [ + ("ipam/rirs", ["name", "slug", "is_private", "description"]), + ("ipam/roles", ["name", "slug", "weight", "description"]), + ("dcim/regions", ["name", "slug", "parent", "description"]), + ("dcim/sites", ["name", "slug", "status", "region", "description"]), + ("ipam/aggregates", ["prefix", "rir", "description"]), + ("ipam/prefixes", ["prefix", "status", "role", "scope_type", "scope_id", + "description", "is_pool", "mark_utilized"]), + # D-120: the compose-/24 band ranges + the static service IP assignments. These + # object classes were previously NOT dumped, so the sandbox loop could not verify + # the D-120 load at all (the "tooling gap" logged in the session ledger). Ranges + # get a synthetic "range" key (start-end) below, since KEY is one field. + ("ipam/ip-ranges", ["start_address", "end_address", "status", "role", "description"]), + ("ipam/ip-addresses", ["address", "status", "role", "dns_name", "description"]), +] + + +def die(msg: str) -> "None": + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +def require_env(name: str) -> str: + v = os.environ.get(name) + if not v: + die(f"{name} is not set. Source the upstream env first: . ~/vr1-office1-creds/vr1-netbox.env") + return v + + +def get_all(base: str, token: str, path: str) -> list: + """Paginate an endpoint fully. Read-only.""" + out = [] + url = f"{base}/api/{path}/?limit=200" + while url: + req = urllib.request.Request(url, headers={ + "Authorization": f"Token {token}", + "Accept": "application/json", + "User-Agent": UA, # <-- the WAF trap. Do not remove. + }) + try: + with urllib.request.urlopen(req, timeout=45) as r: + data = json.load(r) + except urllib.error.HTTPError as exc: + if exc.code == 403: + die(f"403 on {path}. If curl works with this same token, it is the " + f"User-Agent filter, NOT the token -- see references/platform-traps.md.") + die(f"HTTP {exc.code} on {path}: {exc.reason}") + out += data["results"] + nxt = data.get("next") + url = nxt if nxt else None + return out + + +def slim(obj: dict, fields: list) -> dict: + """Keep only what the seeder needs, resolving nested refs to SLUGS. + + IDs are deliberately DISCARDED. A sandbox assigns its own ids; carrying + upstream ids across would either collide or silently mis-link. Everything is + re-linked by slug/prefix on the way in. + """ + rec = {} + for f in fields: + v = obj.get(f) + if isinstance(v, dict): + # nested object (role, rir, region, parent, status) -> slug or value + rec[f] = v.get("slug") or v.get("value") + else: + rec[f] = v + return rec + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--out", required=True, help="Path to write the JSON draft artifact") + args = ap.parse_args() + + base = require_env("NETBOX_URL").rstrip("/") + token = require_env("NETBOX_TOKEN") + + # Site id -> slug, so prefix scope_id (an upstream id) can be rewritten to a + # slug the seeder can re-resolve against ITS OWN site ids. + print(f"Reading upstream draft from {base} (READ-ONLY) ...") + raw_sites = get_all(base, token, "dcim/sites") + site_id_to_slug = {s["id"]: s["slug"] for s in raw_sites} + # Prefixes can be scoped to a REGION as well as a site -- and most of the VR1 + # draft is region-scoped (the f00/f01 containers hang off region VR1, not off + # a site). Mapping only dcim.site silently DROPPED those scopes on the way in. + raw_regions = get_all(base, token, "dcim/regions") + region_id_to_slug = {r["id"]: r["slug"] for r in raw_regions} + + draft = {"_source": base, "_note": "READ-ONLY dump. IDs discarded; refs are by slug."} + for path, fields in ENDPOINTS: + rows = raw_sites if path == "dcim/sites" else get_all(base, token, path) + slimmed = [] + for o in rows: + rec = slim(o, fields) + if path == "ipam/prefixes": + # scope_id is an upstream id -- rewrite to a slug, or drop the scope. + sid = o.get("scope_id") + st = o.get("scope_type") + if sid is not None and st == "dcim.site": + rec["scope_site"] = site_id_to_slug.get(sid) + elif sid is not None and st == "dcim.region": + rec["scope_region"] = region_id_to_slug.get(sid) + rec.pop("scope_id", None) + if path == "ipam/ip-ranges": + # a range is identified by BOTH endpoints; give the fidelity check a + # single stable key (start-end) the way prefixes key on "prefix". + rec["range"] = f'{rec.get("start_address")}-{rec.get("end_address")}' + slimmed.append(rec) + draft[path] = slimmed + print(f" {path:<18} {len(slimmed)}") + + os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True) + with open(args.out, "w") as fh: + json.dump(draft, fh, indent=2, sort_keys=True) + fh.write("\n") + print(f"\nWrote {args.out}. NOTHING was written to NetBox.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/netbox/roles-aggregates-import.py b/netbox/roles-aggregates-import.py index 9480246..7243bc4 100644 --- a/netbox/roles-aggregates-import.py +++ b/netbox/roles-aggregates-import.py @@ -61,6 +61,7 @@ import ipaddress import os import sys +import urllib.parse try: import pynetbox @@ -82,6 +83,14 @@ if not token: die("NETBOX_TOKEN environment variable not set") nb = pynetbox.api(url, token=token) + # WAF TRAP (references/platform-traps.md): upstream NetBox 403s the DEFAULT + # python-requests User-Agent -- it looks EXACTLY like an auth failure but is + # not (the same token works from curl). pynetbox is bitten too. The stdlib + # tools (prod-draft-dump/sandbox-seed/d115-office-carve) already send an + # accepted UA; these two pynetbox importers did NOT, so they could never have + # written to the upstream apex (only the sandbox, which has no WAF). Set it + # on the underlying requests session so EVERY call carries it. + nb.http_session.headers["User-Agent"] = "curl/8.5.0" try: _ = nb.status() except Exception as exc: # noqa: BLE001 @@ -91,6 +100,10 @@ # --- Data (facts / adopted model; the only operator literal is ORG_ULA_48 via env) --- +# A sandbox is local or the known Office1 sandbox address; anything else is treated +# as the production apex and needs --yes-write-upstream (guard in main()). +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + ROLES = [ ("provider-public", "Provider Public", 100, "Six-plane: provider public API VIPs + FIP/ext_net (GUA). D-052/D-101."), @@ -101,7 +114,15 @@ ("data-tenant", "Data Tenant", 130, "Six-plane: geneve overlay, ULA-only; tenant GUA delegated separately. D-052/D-101."), # storage (weight 140) already exists -- intentionally omitted. - ("replication", "Replication", 150, + # NAME is "Replication Plane", NOT "Replication" -- the legacy pre-six-plane role + # `repl` ALREADY HOLDS the name "Replication" in the real draft, and NetBox enforces + # UNIQUE NAMES as well as unique slugs. Creating this as "Replication" returns + # 400 {'name': ['role with this name already exists.']}. This script had never met + # a NetBox carrying the draft (only empty ones), so the collision went unseen until + # 2026-07-13 -- when it 400'd MID-RUN and left the roles half-created. + # The slug stays `replication` because that is what lib-net.sh SPACES6 and + # dc-dc-prefixes-import.py look up. + ("replication", "Replication Plane", 150, "Six-plane: Ceph cluster incl. cross-DC leg, ULA-only. D-052/D-101/D-108."), ] @@ -140,8 +161,23 @@ ap = argparse.ArgumentParser(description="Create VR1 six-plane roles + aggregates (idempotent).") ap.add_argument("--commit", action="store_true", help="write to NetBox (default: preview only, no writes)") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox. Feeding " + "the production apex is an operator decision, not an import side effect.") args = ap.parse_args() commit = args.commit + + # UPSTREAM-WRITE GUARD (2026-07-14). This tool had NO hostname gate; until today + # the WAF's 403 on the default User-Agent was the only thing stopping an accidental + # --commit against the production apex. get_nb() now sends an accepted UA, so that + # accidental-safety is gone -- an explicit gate is required (matches d115-office-carve.py). + if commit: + _host = (urllib.parse.urlparse(os.environ.get("NETBOX_URL", "")).hostname or "").lower() + if _host not in SANDBOX_HOSTS and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{_host or os.environ.get('NETBOX_URL','?')}': it is not a " + "known sandbox, so this is treated as the PRODUCTION apex. Feeding roles/aggregates " + "upstream is an operator decision (C2), not a side effect of running an import. " + "Re-run with --yes-write-upstream if that is genuinely what you intend.") mode = "COMMIT" if commit else "PREVIEW (no writes -- add --commit to apply)" nb = get_nb() @@ -159,6 +195,60 @@ print(f" WOULD {kind:10s} {label}") created += 1 + # --- PREFLIGHT ----------------------------------------------------------- + # Validate EVERY name/slug against NetBox BEFORE writing anything. + # + # On 2026-07-13 this script created 4 roles, then hit a 400 on the 5th (a NAME + # collision with the legacy `repl` role) and DIED -- leaving the apex half + # populated, with no RIRs and no aggregates, and no rollback. An IPAM importer + # that can fail partway through is a tool that corrupts the thing it is meant + # to populate. Check first; write only if the whole plan is viable. + conflicts = [] + for slug, name, _w, _d in ROLES: + if nb.ipam.roles.get(slug=slug): + continue # same slug -> we skip it anyway + clash = nb.ipam.roles.get(name=name) + if clash: + conflicts.append( + f"role name {name!r} is already held by slug {clash.slug!r} (id={clash.id}) " + f"-- cannot create slug {slug!r}. NetBox enforces UNIQUE NAMES, not just slugs.") + for name, slug, _p in RIRS: + if nb.ipam.rirs.get(slug=slug) or nb.ipam.rirs.get(name=name): + continue + clash = nb.ipam.rirs.get(name=name) # unique NAME, same as roles + if clash: + conflicts.append( + f"RIR name {name!r} is already held by slug {clash.slug!r} (id={clash.id}) " + f"-- cannot create slug {slug!r}.") + + # 2026-07-14 HARDENING. These two checks USED TO RUN AFTER THE ROLE-CREATION + # LOOP. That is not a preflight -- it is a landmine: an apex without ARIN, or a + # typo'd ORG_ULA_48, would commit 5 roles (and the RIRs) and THEN die, leaving + # the production IPAM half-populated with no rollback. That is the EXACT failure + # this preflight block was added to prevent, still armed one section lower. + # A preflight that does not cover every die() downstream of the first write is + # not a preflight. + if not (nb.ipam.rirs.get(slug="arin") or nb.ipam.rirs.get(name="ARIN")): + conflicts.append( + "RIR 'ARIN' not found (expected pre-existing) -- the aggregates below are " + "scoped to it. Create it first, then re-run.") + + ula_raw = os.environ.get("ORG_ULA_48") + if ula_raw: + try: + validate_ula_48(ula_raw) # format-check BEFORE any write + except SystemExit: + conflicts.append( + f"ORG_ULA_48={ula_raw!r} is not a valid RFC 4193 ULA /48. It is validated " + "HERE, before the first write -- a typo must not cost a half-populated apex.") + + if conflicts: + print("PREFLIGHT FAILED -- nothing was written:\n") + for c in conflicts: + print(f" * {c}") + die("resolve the problem(s) above, then re-run. NO changes were made.") + # ------------------------------------------------------------------------- + # --- Roles --- print("== IPAM roles (six-plane; slugs match lib-net.sh SPACES6) ==") for slug, name, weight, desc in ROLES: diff --git a/netbox/sandbox-fidelity-check.py b/netbox/sandbox-fidelity-check.py new file mode 100755 index 0000000..4bd4261 --- /dev/null +++ b/netbox/sandbox-fidelity-check.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +""" +Prove a seeded sandbox is a FAITHFUL replica of the upstream draft, plus an +expected delta. Read-only; compares two JSON dumps, touches no NetBox. + +WHY THIS EXISTS. Counts and idempotency CANNOT prove fidelity. On 2026-07-13 the +seeder silently dropped the scope on 17 of 90 prefixes (it mapped only +`dcim.site`, and most of the VR1 draft is `dcim.region`-scoped) -- and every +count still matched, and re-running was still perfectly idempotent. It was caught +by luck on a spot-check that happened to print `site None`. This script is the +check that would have caught it on purpose. + +It diffs FIELD BY FIELD on every shared object, so a dropped or mangled field is +a hard failure rather than an invisible one. + +Usage: + # dump upstream (on vcloud), and the sandbox (on office1-netbox) -- SAME + # read-only dumper, pointed at each in turn: + python3 netbox/prod-draft-dump.py --out netbox/draft/vr1-draft.json + python3 netbox/prod-draft-dump.py --out /tmp/sandbox-dump.json # NETBOX_URL=sandbox + + python3 netbox/sandbox-fidelity-check.py \ + --upstream netbox/draft/vr1-draft.json --sandbox /tmp/sandbox-dump.json + +Exit 0 = faithful replica + exactly the expected delta. Exit 1 = the seeder is +dropping or mangling data. +""" + +import argparse +import json +import sys + +ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) +ap.add_argument("--upstream", required=True, help="JSON dump of the upstream draft") +ap.add_argument("--sandbox", required=True, help="JSON dump of the seeded sandbox") +args = ap.parse_args() + +up = json.load(open(args.upstream)) +sb = json.load(open(args.sandbox)) + +KEY={"ipam/rirs":"slug","ipam/roles":"slug","dcim/regions":"slug","dcim/sites":"slug", + "ipam/aggregates":"prefix","ipam/prefixes":"prefix", + # D-120: the compose-/24 band ranges + the static service IPs. Ranges key on the + # dumper's synthetic "range" (start-end); IPs on "address". + "ipam/ip-ranges":"range","ipam/ip-addresses":"address"} +# The FULL planned delta the sandbox is allowed to have over the upstream draft. +# Anything outside this is a tooling bug (a silent drop, or a stray write). +# D-115 the office carve (site vr1-off1, edge role, 8 prefixes) +# D-117 VR1 DC0/DC1 six-plane (36 prefixes -- 18 per DC) +# D-118 the org ULA /48 (RIRs + aggregates) +# G1/G2 six-plane roles + aggregates +# DC delta prefixes AND their INTENDED scope (the DC's own site). The DC->site +# binding is the D-119 identity: GUA f02 / ULA nibble 2 / the first six /22s -> +# vr1-dc0; f03 / nibble 3 / the last six -> vr1-dc1. Built in ONE place so the CIDR +# set and its scope map cannot drift apart. +def _dc_prefix_scopes(): + scope = {} + v4 = {"vr1-dc0": ["10.12.4.0/22","10.12.8.0/22","10.12.12.0/22", + "10.12.16.0/22","10.12.32.0/22","10.12.36.0/22"], # VR1 DC0 (inherited) + "vr1-dc1": ["10.12.64.0/22","10.12.68.0/22","10.12.72.0/22", + "10.12.76.0/22","10.12.80.0/22","10.12.84.0/22"]} # VR1 DC1 (D-115 /19) + for site, cidrs in v4.items(): + for c in cidrs: + scope[c] = ("site", site) + for dc, site in (("f02", "vr1-dc0"), ("f03", "vr1-dc1")): + for c in (f"2602:f3e2:{dc}:10::/60", f"2602:f3e2:{dc}:10::/64", f"2602:f3e2:{dc}:11::/64"): + scope[c] = ("site", site) + for n, site in (("2", "vr1-dc0"), ("3", "vr1-dc1")): # DC nibble + for nn in ("20", "30", "40", "50"): + scope[f"fd50:840e:74e2:{n}{nn}::/60"] = ("site", site) + scope[f"fd50:840e:74e2:{n}{nn}::/64"] = ("site", site) + scope[f"fd50:840e:74e2:{n}21::/64"] = ("site", site) # metal-internal shares metal /60 + return scope + +def _dc_prefixes(): + return set(_dc_prefix_scopes()) + +# INTENDED scope per DELTA prefix: ("site", slug) / ("region", slug) / None for an +# intentionally UNSCOPED org-level role container. The two role /16s (office, edge) +# are unscoped BY DESIGN in d115-office-carve.py -- mirroring the 27 unscoped role +# containers ALREADY in the upstream apex (10.0.0.0/8, 172.16.0.0/12, ...). Checking +# scope == INTENDED (not merely "has a scope") kills the false "NO SCOPE" failure on +# those two AND still catches a DROPPED scope (the 17-prefix region loss) or a +# WRONG-TARGET binding (a DC0 prefix bound to vr1-dc1 -- the D-117 near-miss). +EXPECTED_PREFIX_SCOPE = { + "10.10.0.0/16": None, # office role container -- unscoped by design + "172.30.0.0/16": None, # edge role container -- unscoped by design + "10.10.0.0/22": ("site", "vr1-off1"), + "10.10.0.0/24": ("site", "vr1-off1"), + "10.10.1.0/24": ("site", "vr1-off1"), + "172.30.1.0/24": ("site", "vr1-off1"), + "2602:f3e2:f01:100::/56": ("site", "vr1-off1"), + "2602:f3e2:f01:100::/64": ("site", "vr1-off1"), + **_dc_prefix_scopes(), +} + +EXPECTED_NEW = { + "ipam/roles": {"edge", "provider-public", "metal-admin", "metal-internal", + "data-tenant", "replication"}, + "dcim/sites": {"vr1-off1"}, + "ipam/rirs": {"rfc-4193-ula", "rfc-1918"}, + "ipam/aggregates": {"2602:f3e2::/36", "23.157.124.0/24", "10.0.0.0/8", + "172.16.0.0/12", "fd50:840e:74e2::/48"}, + "ipam/prefixes": set(EXPECTED_PREFIX_SCOPE), + # D-120 (must match netbox/d120-compose-bands.py). Ranges keyed start-end (the + # dumper's synthetic "range"); IPs by address. Both carry the /24 mask. These are + # pure delta -- the frozen v1 draft carries no ip-ranges/ip-addresses -- so + # both-bounds already gives endpoint/address correctness (a wrong endpoint is a + # different key: it shows as UNEXPECTED-EXTRA + EXPECTED-BUT-ABSENT). + "ipam/ip-ranges": {"10.10.1.2/24-10.10.1.49/24", + "10.10.1.100/24-10.10.1.200/24", + "10.10.1.201/24-10.10.1.254/24"}, + "ipam/ip-addresses": {"10.10.1.10/24", "10.10.1.11/24"}, +} +bad=0 +for ep,k in KEY.items(): + U={r[k]:r for r in up.get(ep,[])} + S={r[k]:r for r in sb.get(ep,[])} + missing=set(U)-set(S) + extra=set(S)-set(U) + exp=EXPECTED_NEW.get(ep,set()) + print(f"\n{ep}: upstream={len(U)} sandbox={len(S)}") + if missing: + bad+=1; print(f" !! MISSING FROM SANDBOX ({len(missing)}): {sorted(missing)[:6]}") + unexpected=extra-exp + if unexpected: + bad+=1; print(f" !! UNEXPECTED EXTRA ({len(unexpected)}): {sorted(unexpected)[:6]}") + elif extra: + print(f" + expected D-115 additions ({len(extra)}): {sorted(extra)}") + + # 2026-07-14 HARDENING -- THE FALSE-GREEN THIS SCRIPT USED TO RETURN. + # `unexpected = extra - exp` only proves the delta is a SUBSET of what was + # planned. It NEVER asserted the plan was actually CARRIED OUT. If the importer + # created ZERO of the 36 DC prefixes, extra={} -> unexpected={} -> this script + # printed "Nothing lost, nothing stray" and exited 0. EXPECTED_NEW was an upper + # bound masquerading as an assertion. It must be BOTH bounds. + missing_expected = exp - extra + if missing_expected: + bad+=1 + print(f" !! EXPECTED-BUT-ABSENT ({len(missing_expected)}): {sorted(missing_expected)[:8]}") + print( " The planned delta was NOT fully applied. A subset-only check would") + print( " have called this run clean -- that is the false green this catches.") + + # DELTA SCOPE-CORRECTNESS. The shared-object drift loop below cannot see these + # (they exist only in the sandbox). Assert each delta prefix's scope == its + # INTENDED scope (EXPECTED_PREFIX_SCOPE), target and all -- reading the dumper's + # slug-rewritten scope_site/scope_region (scope_id is popped on the way in). This + # catches a DROPPED scope (the 17-prefix region loss), a WRONG-TARGET binding (a + # DC0 prefix bound to vr1-dc1 -- the D-117 near-miss), and a spurious scope on an + # intentionally-unscoped container -- WITHOUT false-flagging the role /16s that + # are unscoped by design (which the old "has a scope?" heuristic wrongly failed). + if ep == "ipam/prefixes": + for key in sorted(extra & exp): + rec = S[key] + if rec.get("scope_site"): + actual = ("site", rec["scope_site"]) + elif rec.get("scope_region"): + actual = ("region", rec["scope_region"]) + else: + actual = None + if key not in EXPECTED_PREFIX_SCOPE: + bad+=1; print(f" !! DELTA prefix {key}: no intended scope encoded (extend EXPECTED_PREFIX_SCOPE)") + elif actual != EXPECTED_PREFIX_SCOPE[key]: + bad+=1; print(f" !! DELTA prefix {key}: scope {actual} != intended {EXPECTED_PREFIX_SCOPE[key]}") + if not rec.get("role"): + bad+=1; print(f" !! DELTA prefix {key} has NO ROLE") + # FIELD-LEVEL fidelity on the shared keys -- this is the check that counts cannot make + drift=[] + for key in sorted(set(U)&set(S)): + u,s=U[key],S[key] + for f in set(u)|set(s): + if f.startswith("_"): continue + if u.get(f)!=s.get(f): + drift.append((key,f,u.get(f),s.get(f))) + if drift: + bad+=1 + print(f" !! FIELD DRIFT on {len(set(d[0] for d in drift))} object(s), {len(drift)} field(s):") + for key,f,uv,sv in drift[:12]: + print(f" {key:<26} {f:<14} upstream={uv!r} sandbox={sv!r}") + else: + print(" field-level: IDENTICAL on every shared object") +print("\n" + "=" * 70) +if bad == 0: + print("SANDBOX = upstream draft + EXACTLY the planned delta (D-115/D-117/D-118/D-120). Nothing lost, nothing stray.") + sys.exit(0) +print(f"!! {bad} fidelity problem(s) -- the seeder is dropping or mangling data") +sys.exit(1) + diff --git a/netbox/sandbox-seed.py b/netbox/sandbox-seed.py new file mode 100755 index 0000000..83e8a0b --- /dev/null +++ b/netbox/sandbox-seed.py @@ -0,0 +1,280 @@ +#!/usr/bin/env python3 +""" +Seed a SANDBOX NetBox from the JSON draft produced by netbox/prod-draft-dump.py. + +Phase 2 of the two-phase seeding loop (see prod-draft-dump.py for the why). + +THIS SCRIPT CANNOT WRITE UPSTREAM. It has no upstream URL and no upstream token +-- it only knows the JSON file and the sandbox it is pointed at. That is +deliberate: "the sim never writes upstream" is enforced by STRUCTURE, not by +remembering to pass the right flag. + +DRY BY DEFAULT. Nothing is written without --commit. (The repo learned this the +hard way: dc-dc-prefixes-import.py used to write with no flag at all, and would +have silently bound VR1 DC0's prefixes to the wrong site -- D-117.) + +Idempotent: existing objects are detected by natural key (slug, or the prefix +itself) and skipped. Re-running is safe. + +IDs are never carried across. The draft refs everything by slug; this script +re-resolves those against the SANDBOX's own ids. + +Auth: the sandbox (NetBox 4.6) uses v2 tokens -- the wire form is +`nbt_<key>.<plaintext>`. The API's `token` field alone is NOT usable (it parses +as a legacy v1 token and 403s with "Invalid v1 token", which is thoroughly +misleading). Pass the ASSEMBLED value. + +Usage (on office1-netbox, where NetBox is on localhost and the token lives): + sudo NETBOX_URL=http://localhost:8000 \ + NETBOX_TOKEN="$(cat /root/netbox-secrets/api.token)" \ + python3 sandbox-seed.py --draft vr1-draft.json # preview + ... --draft vr1-draft.json --commit # apply +""" + +import argparse +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request + +# Seeding order is DEPENDENCY order: a site needs its region; a prefix needs its +# role and its site. Prefixes do NOT need parent-prefix ordering -- NetBox +# derives the hierarchy from the CIDR itself. +ORDER = ["ipam/rirs", "ipam/roles", "dcim/regions", "dcim/sites", + "ipam/aggregates", "ipam/prefixes"] + +# Natural key per endpoint -- how we decide "does this already exist?" +NATURAL_KEY = { + "ipam/rirs": "slug", + "ipam/roles": "slug", + "dcim/regions": "slug", + "dcim/sites": "slug", + "ipam/aggregates": "prefix", + "ipam/prefixes": "prefix", +} + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +class NB: + def __init__(self, base: str, token: str): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method: str, path: str, body=None): + url = f"{self.base}/api/{path}" + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(url, data=data, method=method, headers={ + "Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + }) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token'. This NetBox is 4.6: the token must be " + "the ASSEMBLED v2 form nbt_<key>.<plaintext>, not the API's bare " + "`token` field. See docs/vr1-office1-as-built.md.") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def get_one(self, path: str, **flt): + q = urllib.parse.urlencode(flt) + res = self._req("GET", f"{path}/?{q}&limit=1") + return res["results"][0] if res["results"] else None + + def create(self, path: str, payload: dict): + return self._req("POST", f"{path}/", payload) + + def patch(self, path: str, payload: dict): + return self._req("PATCH", f"{path}/", payload) + + +def resolve(nb: NB, cache: dict, path: str, slug: str): + """slug -> sandbox id, cached. None if the slug is absent.""" + if slug is None: + return None + key = (path, slug) + if key not in cache: + obj = nb.get_one(path, slug=slug) + cache[key] = obj["id"] if obj else None + return cache[key] + + +def dep(nb: NB, cache: dict, planned: set, path: str, slug: str, commit: bool): + """Resolve a dependency to (ok, id). + + In a DRY RUN the dependency may not exist yet because we are not creating + anything -- but it IS in the plan, and will exist by the time we commit. + Treating that as "missing" made every prefix cascade-skip behind its role, + which reported a plan of 27/129 and looked like the draft was broken. A dry + run must model the END STATE, not the current empty one. + """ + if slug is None: + return True, None + rid = resolve(nb, cache, path, slug) + if rid is not None: + return True, rid + if not commit and (path, slug) in planned: + return True, None # will exist; no id needed, we never POST + return False, None + + +def build_payload(nb: NB, cache: dict, ep: str, rec: dict, planned: set, commit: bool): + """Draft record -> sandbox POST body, re-linking every ref by slug.""" + p = {k: v for k, v in rec.items() + if k not in ("role", "rir", "region", "parent", "scope_site", + "scope_region", "scope_type", "status") and v is not None} + + if rec.get("status"): + p["status"] = rec["status"] + + if ep == "dcim/regions" and rec.get("parent"): + ok, rid = dep(nb, cache, planned, "dcim/regions", rec["parent"], commit) + if not ok: + return None, f"parent region '{rec['parent']}' not present" + p["parent"] = rid + + if ep == "dcim/sites" and rec.get("region"): + ok, rid = dep(nb, cache, planned, "dcim/regions", rec["region"], commit) + if not ok: + return None, f"region '{rec['region']}' not present" + p["region"] = rid + + if ep == "ipam/aggregates": + ok, rid = dep(nb, cache, planned, "ipam/rirs", rec.get("rir"), commit) + if not ok: + return None, f"rir '{rec.get('rir')}' not present" + p["rir"] = rid + + if ep == "ipam/prefixes": + if rec.get("role"): + ok, rid = dep(nb, cache, planned, "ipam/roles", rec["role"], commit) + if not ok: + return None, f"role '{rec['role']}' not present" + p["role"] = rid + if rec.get("scope_site"): + ok, sid = dep(nb, cache, planned, "dcim/sites", rec["scope_site"], commit) + if not ok: + return None, f"site '{rec['scope_site']}' not present" + p["scope_type"] = "dcim.site" + if sid is not None: + p["scope_id"] = sid + elif rec.get("scope_region"): + ok, rid = dep(nb, cache, planned, "dcim/regions", rec["scope_region"], commit) + if not ok: + return None, f"region '{rec['scope_region']}' not present" + p["scope_type"] = "dcim.region" + if rid is not None: + p["scope_id"] = rid + return p, None + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--draft", required=True, help="JSON produced by prod-draft-dump.py") + ap.add_argument("--commit", action="store_true", + help="WRITE to the sandbox. Default is a DRY RUN that writes nothing.") + ap.add_argument("--update", action="store_true", + help="Also PATCH objects that already exist so they match the draft " + "(otherwise existing objects are left alone). Requires --commit.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set (the SANDBOX's, not upstream's).") + + with open(args.draft) as fh: + draft = json.load(fh) + + src = draft.get("_source", "?") + + # ----------------------------------------------------------------------- + # SAFETY GUARD -- runs BEFORE anything else touches the network. + # + # Never write to the instance the draft was READ from. This is + # self-configuring: the draft records its own _source, so the guard keeps + # working if upstream ever moves hosts. A hardcoded hostname denylist would + # silently stop protecting the day the URL changed -- which is exactly the + # class of stale-literal bug this repo keeps finding. + # + # An earlier version of this guard sat AFTER json.load() and after the + # banner. It was dead code for any malformed draft, and it read as "safe" + # to a reviewer. Guards go FIRST. + # ----------------------------------------------------------------------- + def host(u: str) -> str: + return (urllib.parse.urlparse(u).hostname or u).lower() + + if src != "?" and host(url) == host(src): + die(f"REFUSING: NETBOX_URL points at the UPSTREAM NetBox ({host(src)}) -- the " + f"instance this draft was READ FROM. This script seeds a SANDBOX only; " + f"the sim never writes upstream.") + + nb = NB(url, token) + print(f"Draft source : {src}") + print(f"Sandbox : {url}") + if not args.commit: + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit to apply. ***") + else: + print("\n*** COMMITTING to the sandbox. ***") + + cache, planned = {}, set() + created = existing = skipped = updated = 0 + + for ep in ORDER: + rows = draft.get(ep, []) + if not rows: + continue + print(f"\n{ep} ({len(rows)} in draft)") + nkey = NATURAL_KEY[ep] + for rec in rows: + nval = rec.get(nkey) + found = nb.get_one(ep, **{nkey: nval}) + if found: + existing += 1 + cache[(ep, nval)] = found["id"] + if args.update: + payload, why = build_payload(nb, cache, ep, rec, planned, args.commit) + if payload is None: + print(f" SKIP {nval} -- {why}") + continue + if not args.commit: + print(f" [dry-run] would UPDATE {nval}") + continue + nb.patch(f"{ep}/{found['id']}", payload) + print(f" UPDATED {nval} (id={found['id']})") + updated += 1 + continue + payload, why = build_payload(nb, cache, ep, rec, planned, args.commit) + if payload is None: + print(f" SKIP {nval} -- {why}") + skipped += 1 + continue + if not args.commit: + print(f" [dry-run] would CREATE {nval}") + planned.add((ep, nval)) + created += 1 + continue + obj = nb.create(ep, payload) + cache[(ep, nval)] = obj["id"] + print(f" CREATED {nval} (id={obj['id']})") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'='*66}") + print(f"{verb}: {created} already present: {existing} updated: {updated} skipped: {skipped}") + if not args.commit: + print("DRY RUN -- the sandbox was NOT modified. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/opentofu/.terraform.lock.hcl b/opentofu/.terraform.lock.hcl new file mode 100644 index 0000000..51c8b6c --- /dev/null +++ b/opentofu/.terraform.lock.hcl @@ -0,0 +1,71 @@ +# This file is maintained automatically by "tofu init". +# Manual edits may be lost in future updates. + +provider "registry.opentofu.org/canonical/maas" { + version = "2.7.2" + constraints = "2.7.2" + hashes = [ + "h1:0/+f28bV5/BqmlBZdmQiTWmzVnjn8/3VNwU+7RLHi1E=", + "h1:0NQfNgN9iJEcuT3WK2+vC9Ss/C1cnyepVP6uahpvSSo=", + "h1:3LC/Yd6KHwKKUkeAOi0g6lRO3Aj6+vMwc5xGf6ARRcY=", + "h1:Be77+i/FQe/egck+r/Bi/qtUVTtB1+BXnwVS7rhQwGI=", + "h1:Cd7Ed7paJzuEhCpkcauEl18JkfRnvX7zK2KnbZfOeQ0=", + "h1:GcMMY3WdhRt6xJ6uXSoKRxs0tpmt43GL4a/a/gBlJV8=", + "h1:IdwSAz83vIz0gQvRt9VT+rs9F0+1MgLIi2IjU3U/8gA=", + "h1:JWEzos28aI09FfC9nW8Odb9h/J2KWnBwoOAsTz0OZ1g=", + "h1:JY1cOBhZyAYVKEWkM2TU/z+CgXI0GJ53HIdzIgmORRM=", + "h1:M4bq+7JZB9hf2hPzr1x85dAhbWcmMBDCoqGvYcSc/hk=", + "h1:PTbVBZc9uhi9zztYpAiWIRHc7zNr4Uwr5wR/atlbuI0=", + "h1:WNzlm/e2Rn6mVhLEgG5yBvMJEpcUCvSQxExvY77pAEo=", + "h1:jc8UQ6JbaFzi+zxpf8AzfZXs94sMIWYwu34adBDWvBk=", + "h1:u+mlHkUKpqQFmDB1zRtBFC7Nl+gMDvFw0edd71q5FH0=", + "zh:07de224747a55736794b771cfe42bcd7a84a0656120df73f3bba8fa5a7869fed", + "zh:0aa77c14a1e4826cc1b345b6dc61086a2d9c20a77b151b7677718544e275636e", + "zh:2a4a659601e97d10ff8d07871fa74dd895946f5e683fb7fb4a6050b78beb3012", + "zh:4627c138c5b36d36612bea1d6d5ed895b5edc082f15bdc2af3e3f6c1153d0568", + "zh:4a948b51b701b1e60794e7cee3f7f5183838fd2e1e53898ddd824a3d31e7b267", + "zh:54d987940d7e6c591a1d4b837adf01b6cace17698227649fa72da8113d8c3531", + "zh:5b111497a82b898478fbea60b244d7e932039bf95460df9d984d0f05e2567854", + "zh:68ed5d31ac46fb0e79b69e981021f83166600d222d7b2adecbea9393f8ff93b9", + "zh:7fd8f29946937cd2c2716603081fb5c71f524465cc4b0bfeeeacab13eb927515", + "zh:824e97b9b68192bf04cc56e94e7d279468d1c4d48c2a98d9f71f86b804053500", + "zh:ce5669c6279fa6cd72ab8e39ccffeb90be3f0d30d7b4077f154b8cb1fad710c0", + "zh:d508aa6858a9212e6b71924ec9e0ddeaada4bab74e4e040ce036ef4189e5076a", + "zh:db819ed07eaafbab90f8292b0a6bc4551c3bb7dcbd1006af67e68719f36720b0", + "zh:f42d9b7c7b0a8adcd1135f295b40425d5a47191d14468a5725f665d027aa9647", + "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", + ] +} + +provider "registry.opentofu.org/dmacvicar/libvirt" { + version = "0.9.8" + constraints = "0.9.8" + hashes = [ + "h1:/9TSq9ibeojseTyT6Jx3z/dbvT5uw8ka3/18NKonrQc=", + "h1:BN5hy1RaHlpcV0BDJYrSOEklK4YNdZCpiXOOGn0ce9k=", + "h1:MNJ6dcFru4DwQM7maIUAADJIHn3aF+L8XsnRJlleNaM=", + "h1:P1Cx0VZGTe6cpWpQ4JFOPJJb+Z+N50t4e1VA76T1tAA=", + "h1:Pm9du7Rzsg1bhLq+XnW9VaLSjTu0Yytr9AOdP9FLQvM=", + "h1:Q/JvNhRskJqDKPJoO9DW3+Ons3mk8kc++zlT4TucyFw=", + "h1:Tzk+wRzhboY3F3wB1u0y3ySMMd4ZPtJbOYlTbbkxZc0=", + "h1:Uuk8gkDEzmXfOTW/nm8GMG2aen7n/+X5GsTAf7eVRJc=", + "h1:cRBlnB/TG4+l0y0i42QLU0JoLDBxxJEwCguLXS9Rv8A=", + "h1:jn+hA9lIJzH5T0550IaKNsHxEAbyTGtvYWm02JCxexM=", + "h1:pKAwAcCGEwsKvVy5eIxdmb1gLhTTxB0xGVi8/jqj0uI=", + "h1:wS5VlGinhpnMCva0tKwLRPRQxmMUl80L05AueWshzNg=", + "h1:yqZeKoJ+EZc3687/+ZBqBmtwzvBPLNwaEHW74+bSc6Y=", + "zh:061e5187853729e1d8ba20938402ad6e778b4097436925d0bef7741c8aa26ee1", + "zh:69a2ac8ee0cb0c1581bdc9a1296d89abcd1808229ff0a2fa20678ba4f7472944", + "zh:8b02bf349191b4e7d9529d620da9e9265de5bd01d6983bb079589acdf80dda7a", + "zh:9620872ed1da1ff421e85dd4ba2a110c284f3f0d06369c36a9aa75d501155fad", + "zh:9ae7c932b7fb6f62ff20b863a15a7cd713415faa4c789472dc403062808366cb", + "zh:a369c3e271b49e085940e6181fe6c1086c2126eebf5c7ad2b1db8fc8792dd30c", + "zh:a45ad97d7192bea104b32521839dd705a330f13364cf1007dc912b6a0b6be3c8", + "zh:b69f2a13ff688ad4b79e14deb6ef6341187864f50960bc0fd9c682f6077b4318", + "zh:c20b467ff0ac76431a7bdb99662b9414b922ea603327509518c0179416cdfea3", + "zh:c22d45913bd1f00a33d1a4b80547e4a2854409a298a0868c1730124f03a210ad", + "zh:cf82bb14884fcbdcd57620cee248994b1633ea246fa49fd668d2f1739abb982d", + "zh:ddc1cb4417206dd46eb22c3b81204b3a5edda03c4436e6a5914d541fcfd4c27f", + "zh:e6b1983b8f1b9658448d0d09635619cc75dde314e6b4d7fd16d827f9e4bdb767", + ] +} diff --git a/opentofu/README.md b/opentofu/README.md index 3c71088..08bc8ce 100644 --- a/opentofu/README.md +++ b/opentofu/README.md @@ -10,17 +10,25 @@ for where every provider fact below was sourced from and the fetch methodology that worked -- read that first if extending any module here. -**Status: SCAFFOLD, UNVALIDATED.** No `tofu`/`terraform` binary was available -in the session that authored this -- nothing here has been run through -`tofu init`/`validate`/`plan`, let alone `apply`. Before trusting any of it: +**Status: STAGE-1 VALIDATED + APPLIED (2026-07-10); Stage 2/3 modules still +UNVALIDATED.** First real `tofu` run: 2026-07-10 on the vcloud host (OpenTofu +v1.12.3), executing `runbooks/dc-dc-phase0-vcloud-prep.md`. The root module + +the four Stage-1 module types (`dc-planes`, `dc-storage-pool`, `mesh-link`, +`office1-network`) passed `tofu init`/`validate`/`plan` and were `apply`-ed +clean (13 libvirt objects, all MTU 9000). Two fixes were required (DOCFIX-179): +per-module `required_providers` (child modules do NOT inherit provider SOURCE +mapping -- without their own `versions.tf`, OpenTofu infers `hashicorp/libvirt`, +which does not exist), and `tofu fmt -recursive`. The root `provider "maas"` +block was also deferred to Stage 3 (it forced Stage 1 to supply a sensitive key +for zero MAAS resources). STILL UNVALIDATED -- the modules not yet instantiated +from root: `node-vm`, `base-image`, `cloudinit-vm`, `opnsense-edge`, +`maas-vm-host`, `netem-link` (Stage 2/3). Re-run before trusting those, on a +machine with the binary + provider-registry access: ``` bash scripts/opentofu-validate.sh ``` -on a machine with the binary and network access to the provider registry, and -read the result before proceeding. - ## Scope of this delivery Built: diff --git a/opentofu/main.tf b/opentofu/main.tf index 07d069c..f75981d 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -6,53 +6,54 @@ uri = var.libvirt_uri } -provider "maas" { - api_url = var.maas_api_url - api_key = var.maas_api_key -} - -# ---- modules/maas-vm-host and modules/netem-link are NOT instantiated below -# yet: both need real, measured inputs (a MAAS zone/pool that exists, the -# vcloud host's SSH target, real netem parameters) that don't exist yet -- -# see docs/dc-dc-deployment-workflow.md gap register items 2/4/11 and -# opentofu/README.md. The provider block above is wired so the modules are -# ready to call once those values are real. ---- +# ---- The provider "maas" CONFIG block is deliberately DEFERRED to Stage 3 +# (DOCFIX-179, decided on the first real `tofu plan`). No module instantiated in +# Stage 1-2 uses the maas provider (modules/maas-vm-host is a Stage-3 module, +# commented out below), yet a configured provider "maas" block here forced +# `tofu plan` to demand maas_api_url AND the SENSITIVE maas_api_key for a plan +# that creates ZERO MAAS resources -- which would bake a fake key into +# plan/state (the DOCFIX-175 plaintext-secret surface). maas stays in +# versions.tf's required_providers (version pinned, lock-file complete); Stage 3 +# re-adds this provider block + its two variables when it actually instantiates +# modules/maas-vm-host. modules/netem-link is likewise not instantiated yet +# (needs a real bridge/SSH target + netem params). See gap register items +# 2/4/11 and opentofu/README.md. ---- # ---- DC1: inherits the DC0 six-plane v4 layout unchanged (D-101) ---- -module "dc1_planes" { +module "vr1_dc0_planes" { source = "./modules/dc-planes" - dc_name = "dc1" + dc_name = "vr1-dc0" domain_suffix = var.domain_suffix mtu = var.underlay_mtu - planes = var.dc1_planes + planes = var.vr1_dc0_planes } -module "dc1_storage" { +module "vr1_dc0_storage" { source = "./modules/dc-storage-pool" - dc_name = "dc1" - target_path = var.dc1_pool_path + dc_name = "vr1-dc0" + target_path = var.vr1_dc0_pool_path } -# ---- DC2: NOT YET WIRED. ---- -# D-101 has not assigned DC2's supernet yet (open sub-item). Once NetBox -# assigns it, add a `dc2_planes` variable (same shape as `dc1_planes` in -# variables.tf) and uncomment/adapt this block -- do not fill in guessed -# CIDRs to make this "work" sooner. +# ---- DC2: PLANES deferred (operator ruling 2026-07-10, Option B) -- wait for +# NetBox to assign D-101's supernet/ULA/GUA before wiring vr1_dc1_planes. Do NOT +# fill in guessed CIDRs to make it "work" sooner. When NetBox assigns it, add a +# `vr1_dc1_planes` variable (same shape as `vr1_dc0_planes`) and uncomment this block. +# The storage pool + mesh legs are NOT address-dependent, so they ARE wired now. # -# module "dc2_planes" { +# module "vr1_dc1_planes" { # source = "./modules/dc-planes" -# dc_name = "dc2" +# dc_name = "vr1-dc1" # domain_suffix = var.domain_suffix # mtu = var.underlay_mtu -# planes = var.dc2_planes +# planes = var.vr1_dc1_planes # } -# -# module "dc2_storage" { -# source = "./modules/dc-storage-pool" -# dc_name = "dc2" -# target_path = var.dc2_pool_path -# } + +module "vr1_dc1_storage" { + source = "./modules/dc-storage-pool" + dc_name = "vr1-dc1" + target_path = var.vr1_dc1_pool_path +} # ---- Office1 headend: its own storage pool for MAAS-region/NetBox/GitBucket # service-VM disks (D-103). Office1 does not get a dc-planes module -- the @@ -87,35 +88,31 @@ # owns creating it -- see that runbook's new Step 4b and its "Open questions" # section for the full reasoning). NOT instantiated below -- commented out on # purpose, mirroring the identical NOT-YET-REAL status of module -# "dc1_opnsense"/"dc2_opnsense" (see runbooks/dc-dc-phase2-tofu-dc-substrate.md +# "vr1_dc0_opnsense"/"vr1_dc1_opnsense" (see runbooks/dc-dc-phase2-tofu-dc-substrate.md # Step 5's own template, which is ALSO still all placeholders, nothing # committed here either). Every value below needs to be real before this is # uncommented -- do not fill in invented specs to wire it in sooner: # -# module "office1_opnsense" { -# source = "./modules/opnsense-edge" -# vm_name = "office1-opnsense" -# memory_mib = <REAL VALUE -- OPNsense's own sizing guidance, not invented> -# vcpu = <REAL VALUE> -# pool_name = module.office1_storage.pool_name -# disk_size_bytes = <REAL VALUE> -# base_volume_path = "<real prepared base image path -- scripts/opnsense-prep-image.sh output>" -# config_iso_path = "<real config ISO path -- scripts/opnsense-build-config-iso.sh output, built from a real Office1 config.xml>" -# lan_network_name = module.office1_network.network_name -# wan_network_name = <UNRESOLVED -- see docs/dc-dc-deployment-workflow.md -# tooling gap register item #17 (NEW, added alongside this delivery): -# no dedicated ISP-uplink/WAN-side network exists for ANY site's OPNsense -# edge yet, DC1/DC2 included -- runbooks/dc-dc-phase2-tofu-dc-substrate.md -# Step 5's own dc1_opnsense template has the IDENTICAL unresolved -# wan_network_name placeholder. Do NOT default this to a mesh-link -# network: D-100's own sub-item ruling is explicit that the three -# mesh legs carry MANAGEMENT TRAFFIC ONLY (Office1<->DC fiber = MAAS/ -# Juju/operator), not a simulated ISP uplink -- wiring an edge's WAN -# side to a mesh leg would misrepresent the topology, not just -# placeholder it. Resolve gap #17 (a dedicated per-site ISP-edge -# network, analogous in shape to this same office1-network module or -# mesh-link) before uncommenting this block for Office1 OR for DC1/DC2.> -# } +# Office1's own OPNsense simulated-ISP edge (gap #16 CLOSED). gap #17 resolved +# for Office1 via the `office1-wan` NAT network (created via virsh 2026-07-12 -- +# a documented D-103 debt to formalize into an OpenTofu module later, same class +# as the retained `wan`). LAN = the Office1-local segment; WAN = office1-wan (its +# own simulated ISP -> internet). Config is the DOCFIX-185 real-ISP-router config +# (no egress-airgap), rendered to office1-opnsense-config.iso. +module "office1_opnsense" { + source = "./modules/opnsense-edge" + vm_name = "office1-opnsense" + memory_mib = 2048 + vcpu = 2 + pool_name = module.office1_storage.pool_name + base_volume_path = "/var/lib/libvirt/vr1/office1/opnsense-26.1-nano.qcow2" + # No disk_size_bytes: the disk is a direct copy of the prepped nano, so its size comes + # from scripts/opnsense-prep-image.sh's GROW (default +8G -> 11 GiB). The old + # `disk_size_bytes = 16 GiB` here was SILENTLY IGNORED -- measured: the live volume is + # 11 GiB. Removed in DOCFIX-189. + lan_network_name = module.office1_network.network_name + wan_network_name = "office1-wan" +} # ---- The D-100 dark-fiber mesh triangle: DC1<->DC2, DC1<->Office1, # DC2<->Office1 -- mesh, not star. All three legs created now even though DC2 @@ -123,20 +120,149 @@ # plane networks exist yet, and Phase 0's gate wants "virtual networks present # and isolated as designed" before Phase 2 needs them. ---- -module "mesh_dc1_dc2" { +module "mesh_vr1_dc0_vr1_dc1" { source = "./modules/mesh-link" - link_name = "dc1-dc2" + link_name = "vr1-dc0-vr1-dc1" mtu = var.underlay_mtu } -module "mesh_dc1_office1" { +module "mesh_vr1_dc0_office1" { source = "./modules/mesh-link" - link_name = "dc1-office1" + link_name = "vr1-dc0-office1" mtu = var.underlay_mtu } -module "mesh_dc2_office1" { +module "mesh_vr1_dc1_office1" { source = "./modules/mesh-link" - link_name = "dc2-office1" + link_name = "vr1-dc1-office1" mtu = var.underlay_mtu } + +# ===================================================================== +# D-114 -- Office1 site containment VM + its base image. +# +# voffice1 IS the Office1 "facility": MAAS-region runs on it, LXD runs on it, and the +# non-stack service machines (NetBox, GitBucket, Tailscale) are VMs MAAS COMPOSES into +# that LXD host -- VR0's proven `lxd` + `tailscale` pattern, applied per site. +# +# expose_nested_virt = true is LOAD-BEARING, not a nicety: LXD virtual machines are +# qemu/KVM guests, so without the host `svm` feature passed through NO service VM can +# ever boot inside voffice1 and D-114's model fails at its first step. (cloudinit-vm had +# NO cpu block at all before 2026-07-13, which silently gave every guest an emulated CPU +# with no svm -- measured: host is an EPYC 9965, a default-CPU guest sees "Opteron_G3".) +# +# Only network is office1-local: voffice1 reaches the internet THROUGH the OPNsense edge, +# like a real server behind a real site router. It takes its address by DHCP from the +# edge's Kea -- which also makes it the FIRST REAL LEASE that path has ever served. +# +# Cloud-init here is DELIBERATELY MINIMAL (identity + access + guest agent). MAAS and LXD +# are installed as separate GATED steps so they are observable and individually approved, +# not buried in a first-boot script that either silently works or silently does not. +# ===================================================================== + +module "ubuntu_noble_base" { + source = "./modules/base-image" + image_name = "ubuntu-24.04-base.qcow2" + pool_name = module.office1_storage.pool_name + # Official Ubuntu 24.04 LTS (noble) cloud image. VERIFIED reachable 2026-07-13 (HTTP 200, + # Last-Modified 2026-07-05). 24.04 matches what VR0 deploys for its lxd/juju/tailscale + # machines, so the image choice carries no new delta. + source_url = "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" +} + +module "voffice1" { + source = "./modules/cloudinit-vm" + vm_name = "voffice1" + vcpu = var.voffice1_vcpu + memory_mib = var.voffice1_memory_mib + disk_size_bytes = var.voffice1_disk_bytes + pool_name = module.office1_storage.pool_name + base_volume_path = module.ubuntu_noble_base.path + network_names = [module.office1_network.network_name] + expose_nested_virt = true + + user_data = <<-EOT + #cloud-config + hostname: voffice1 + fqdn: voffice1.${var.domain_suffix} + manage_etc_hosts: true + users: + - name: jessea123 + groups: [adm, sudo] + shell: /bin/bash + sudo: "ALL=(ALL) NOPASSWD:ALL" + ssh_authorized_keys: + - ${trimspace(file(var.office1_ssh_pubkey_path))} + package_update: true + packages: + - qemu-guest-agent + runcmd: + - [systemctl, enable, --now, qemu-guest-agent] + EOT + + meta_data = <<-EOT + instance-id: voffice1-d114 + local-hostname: voffice1 + EOT + + # Match by GLOB, not a guessed interface name: the NIC's kernel name (ens3 / enp1s0 / + # ...) depends on the machine type and PCI topology and is NOT known until first boot. + # Naming it would be an inferred value. dhcp4 is a DELIBERATE choice here, not the + # module's refused default: office1-local genuinely has an authoritative DHCP server + # (Kea on the OPNsense edge, pool .100-.199). A Kea RESERVATION is added afterwards, + # via the OPNsense REST API, to make the address stable. + network_config = <<-EOT + version: 2 + ethernets: + lan: + match: + name: "en*" + dhcp4: true + EOT +} + +# ----------------------------------------------------------------------------- +# D-119 state migration. `moved` rewrites the STATE ADDRESS only -- it cannot +# suppress a ForceNew driven by an attribute, and `dc_name`/`link_name` are +# interpolated straight into each libvirt object's `name` (which IS ForceNew). +# So the libvirt networks/pools ARE replaced; these blocks exist so `tofu plan` +# reads as N clean "must be replaced" lines instead of N unrelated destroy+create +# pairs -- i.e. so the plan is REVIEWABLE. +# +# SAFE TO REPLACE, MEASURED 2026-07-14 BEFORE THE RENAME: all of these objects are +# EMPTY -- no guests are attached (Stage 3 has not run, so no DC node VMs exist) +# and no volumes live in either DC pool (the only volumes in state belong to +# office1_opnsense, ubuntu_noble_base and voffice1, none of which sit in a DC pool). +# Re-verify with `tofu plan` before applying; if a plan shows anything ATTACHED, +# STOP -- standing lesson 2: an apply touching a libvirt_domain's devices is an +# OUTAGE, and "updated in-place" does NOT mean "no restart". +# ----------------------------------------------------------------------------- +moved { + from = module.dc1_planes + to = module.vr1_dc0_planes +} + +moved { + from = module.dc1_storage + to = module.vr1_dc0_storage +} + +moved { + from = module.dc2_storage + to = module.vr1_dc1_storage +} + +moved { + from = module.mesh_dc1_dc2 + to = module.mesh_vr1_dc0_vr1_dc1 +} + +moved { + from = module.mesh_dc1_office1 + to = module.mesh_vr1_dc0_office1 +} + +moved { + from = module.mesh_dc2_office1 + to = module.mesh_vr1_dc1_office1 +} diff --git a/opentofu/modules/base-image/variables.tf b/opentofu/modules/base-image/variables.tf index 33c194e..008ba2f 100644 --- a/opentofu/modules/base-image/variables.tf +++ b/opentofu/modules/base-image/variables.tf @@ -20,5 +20,5 @@ is not guaranteed the same way). Do not assume this module covers OPNsense without checking that first. EOT - type = string + type = string } diff --git a/opentofu/modules/base-image/versions.tf b/opentofu/modules/base-image/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/base-image/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf index 9ebd8e5..8f9cc8d 100644 --- a/opentofu/modules/cloudinit-vm/main.tf +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -34,12 +34,12 @@ backing_store = { path = var.base_volume_path # the caller passes module.<base_image>.path - # straight through -- a real attribute - # reference, matching - # examples/alpine_cloudinit.tf's - # `libvirt_volume.alpine_base.path` exactly, - # just threaded across a module boundary - # instead of within one file. + # straight through -- a real attribute + # reference, matching + # examples/alpine_cloudinit.tf's + # `libvirt_volume.alpine_base.path` exactly, + # just threaded across a module boundary + # instead of within one file. format = { type = "qcow2" } @@ -66,11 +66,43 @@ } resource "libvirt_domain" "vm" { - name = var.vm_name - memory = var.memory_mib - vcpu = var.vcpu - type = "kvm" - running = true + name = var.vm_name + # memory_unit is REQUIRED -- see the opnsense-edge module for the full write-up. On + # dmacvicar/libvirt >=0.9, bare `memory` is interpreted in libvirt's default unit (KiB), + # NOT MiB, so omitting this gives the guest 1024x too little RAM and it triple-faults + # in its bootloader. (2026-07-12 incident.) + memory = var.memory_mib + memory_unit = "MiB" + vcpu = var.vcpu + type = "kvm" + running = true + + # ACPI/APIC: with no `features` block libvirt renders `acpi=off` -- see the + # opnsense-edge module for the full write-up (a FreeBSD guest PANICS outright). + # A Linux guest will usually still boot without ACPI, but degraded: no clean + # ACPI shutdown/reboot signalling, and unreliable CPU/IRQ enumeration. Every + # normal libvirt guest sets both; there is no reason for ours not to. + features = { + acpi = true + apic = {} + } + + # host-passthrough ALWAYS: with no cpu block at all libvirt renders a generic emulated + # CPU model that exposes NO `svm` flag, so nested KVM is impossible in the guest no + # matter what is asked for. (Measured 2026-07-13: the host is an EPYC 9965 while a + # default-CPU guest is handed an "Opteron_G3".) `svm` itself is then passed through or + # disabled per var.expose_nested_virt -- see modules/opnsense-edge, which disables it as + # hardening for a router guest. D-114's site containment VMs REQUIRE it passed through: + # they run LXD VMs composed by MAAS. + cpu = { + mode = "host-passthrough" + features = var.expose_nested_virt ? [] : [ + { + name = "svm" + policy = "disable" + } + ] + } os = { type = "hvm" diff --git a/opentofu/modules/cloudinit-vm/variables.tf b/opentofu/modules/cloudinit-vm/variables.tf index 0815632..9fcd1c1 100644 --- a/opentofu/modules/cloudinit-vm/variables.tf +++ b/opentofu/modules/cloudinit-vm/variables.tf @@ -31,7 +31,7 @@ + volume name; libvirt's actual pool-relative path convention was not independently confirmed this session (see main.tf's note on this). EOT - type = string + type = string } variable "network_names" { @@ -48,7 +48,7 @@ hasn't been done yet, and inventing placeholder cloud-config content would misrepresent it as a real decision. Pass the actual config once written. EOT - type = string + type = string } variable "meta_data" { @@ -64,5 +64,25 @@ 3), so a real static address from NetBox is required here, not a plausible-looking default that would silently fail to reach anything. EOT - type = string + type = string +} + +variable "expose_nested_virt" { + description = <<-EOT + Whether this guest may itself run KVM guests (nested virtualisation). + + TRUE -- pass the host's `svm` feature through. REQUIRED for any VM that is + registered to MAAS as an LXD VM host and has VMs composed into it + (D-114: the site containment VMs voffice1/vvr1-dc0/vvr1-dc1). Without it the + guest gets no svm flag, LXD virtual machines cannot start, and the + D-114 model fails at its first step. + FALSE -- disable `svm`, matching modules/opnsense-edge. Correct hardening for + a guest that has no business seeing nested virt. + + No default: this is a real decision per VM, not a fallback. NOTE the domain + always uses cpu mode "host-passthrough" -- with NO cpu block at all (this + module's state before 2026-07-13) libvirt renders a generic emulated CPU model + that exposes no svm regardless of what this flag says. + EOT + type = bool } diff --git a/opentofu/modules/cloudinit-vm/versions.tf b/opentofu/modules/cloudinit-vm/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/cloudinit-vm/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/dc-planes/variables.tf b/opentofu/modules/dc-planes/variables.tf index ae2fd94..d21eb3a 100644 --- a/opentofu/modules/dc-planes/variables.tf +++ b/opentofu/modules/dc-planes/variables.tf @@ -18,10 +18,10 @@ EOT type = map(object({ cidr = string # anchor fact only -- NOT applied to the libvirt_network - # resource (isolated planes carry no forward/ips block, see - # main.tf); recorded here purely so `tofu plan` output and - # `terraform output` reflect the intended assignment for - # cross-checking against NetBox. + # resource (isolated planes carry no forward/ips block, see + # main.tf); recorded here purely so `tofu plan` output and + # `terraform output` reflect the intended assignment for + # cross-checking against NetBox. })) } diff --git a/opentofu/modules/dc-planes/versions.tf b/opentofu/modules/dc-planes/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/dc-planes/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/dc-storage-pool/variables.tf b/opentofu/modules/dc-storage-pool/variables.tf index 82c3f8a..e052d08 100644 --- a/opentofu/modules/dc-storage-pool/variables.tf +++ b/opentofu/modules/dc-storage-pool/variables.tf @@ -12,5 +12,5 @@ Section 3). MUST be a real, measured path on the vcloud host -- never invented; no default. EOT - type = string + type = string } diff --git a/opentofu/modules/dc-storage-pool/versions.tf b/opentofu/modules/dc-storage-pool/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/dc-storage-pool/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/maas-vm-host/variables.tf b/opentofu/modules/maas-vm-host/variables.tf index 8d0bd34..d688787 100644 --- a/opentofu/modules/maas-vm-host/variables.tf +++ b/opentofu/modules/maas-vm-host/variables.tf @@ -14,7 +14,7 @@ the root module in practice, but is kept as its own input here rather than silently assumed identical. EOT - type = string + type = string } variable "zone" { diff --git a/opentofu/modules/maas-vm-host/versions.tf b/opentofu/modules/maas-vm-host/versions.tf new file mode 100644 index 0000000..332837a --- /dev/null +++ b/opentofu/modules/maas-vm-host/versions.tf @@ -0,0 +1,11 @@ +# Provider source mapping: maps local name `maas` to canonical/maas so +# `tofu init` does not infer the `maas_` resource prefix as a nonexistent +# hashicorp/maas. Version pinned at the root (opentofu/versions.tf). +# (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + maas = { + source = "canonical/maas" + } + } +} diff --git a/opentofu/modules/mesh-link/versions.tf b/opentofu/modules/mesh-link/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/mesh-link/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/netem-link/main.tf b/opentofu/modules/netem-link/main.tf index b0a64db..3ce5608 100644 --- a/opentofu/modules/netem-link/main.tf +++ b/opentofu/modules/netem-link/main.tf @@ -23,12 +23,30 @@ resource "terraform_data" "netem" { triggers_replace = [var.bridge_name, var.netem_args] + # `input` exists ONLY to make the values reachable from the destroy-time + # provisioner below. A destroy provisioner may reference ONLY `self`, + # `count.index`, or `each.key` -- never `var.*`. The original code + # interpolated var.vcloud_host_ssh_target and var.bridge_name directly into + # the destroy command, which OpenTofu rejects at INIT time: + # + # Error: Invalid reference from destroy provisioner + # Destroy-time provisioners and their connection configurations may only + # reference attributes of the related resource, via 'self', ... + # + # So this module could not even initialize, let alone apply. Stashing the + # values in `input` and reading them back as `self.input.*` is the canonical + # fix (OpenTofu's own guidance for exactly this case). (DOCFIX-194, 2026-07-13.) + input = { + ssh_target = var.vcloud_host_ssh_target + bridge_name = var.bridge_name + } + provisioner "local-exec" { command = "ssh ${var.vcloud_host_ssh_target} 'sudo tc qdisc replace dev ${var.bridge_name} root netem ${var.netem_args}'" } provisioner "local-exec" { when = destroy - command = "ssh ${var.vcloud_host_ssh_target} 'sudo tc qdisc del dev ${var.bridge_name} root' || true" + command = "ssh ${self.input.ssh_target} 'sudo tc qdisc del dev ${self.input.bridge_name} root' || true" } } diff --git a/opentofu/modules/netem-link/variables.tf b/opentofu/modules/netem-link/variables.tf index 49e85db..59616cf 100644 --- a/opentofu/modules/netem-link/variables.tf +++ b/opentofu/modules/netem-link/variables.tf @@ -14,7 +14,7 @@ + passwordless sudo on the vcloud host for the invoking user are assumed prerequisites, not configured by this module. No default -- measured. EOT - type = string + type = string } variable "bridge_name" { @@ -27,7 +27,7 @@ <network-name>` (or `tofu show` post-apply) and pass it explicitly here -- no default. EOT - type = string + type = string } variable "netem_args" { @@ -42,5 +42,5 @@ research target, not tc itself); confirm your exact netem_args string with `man tc-netem` or a live `tc qdisc` test before relying on it. EOT - type = string + type = string } diff --git a/opentofu/modules/node-vm/main.tf b/opentofu/modules/node-vm/main.tf index e939cbb..a915675 100644 --- a/opentofu/modules/node-vm/main.tf +++ b/opentofu/modules/node-vm/main.tf @@ -27,17 +27,59 @@ name = "${var.vm_name}-disk.qcow2" pool = var.pool_name capacity = var.disk_size_bytes - format = { - type = "qcow2" + + # `format` is NOT a top-level attribute of libvirt_volume -- it nests under + # `target`. Verified against the provider's own schema (dmacvicar/libvirt + # v0.9.8, `tofu providers schema -json`): libvirt_volume's top-level + # attributes are name/pool/capacity/allocation/physical/type/target/create/ + # backing_store/... and there is NO `format` among them. The old top-level + # form was rejected outright -- "An argument named format is not expected + # here" -- so this module could never have applied. modules/base-image had + # the correct nesting all along; this one did not. (DOCFIX-194, 2026-07-13.) + target = { + format = { + type = "qcow2" + } } } resource "libvirt_domain" "node" { - name = var.vm_name - memory = var.memory_mib - vcpu = var.vcpu - type = "kvm" - running = true + name = var.vm_name + # memory_unit is REQUIRED -- see the opnsense-edge module for the full write-up. On + # dmacvicar/libvirt >=0.9, bare `memory` is interpreted in libvirt's default unit (KiB), + # NOT MiB, so omitting this gives the guest 1024x too little RAM and it triple-faults + # in its bootloader. (2026-07-12 incident.) + memory = var.memory_mib + memory_unit = "MiB" + vcpu = var.vcpu + type = "kvm" + running = true + + # ACPI/APIC: with no `features` block libvirt renders `acpi=off` -- see the + # opnsense-edge module for the full write-up. Especially important here: these are + # the MAAS-managed node VMs, and MAAS drives power/shutdown via ACPI signalling. + # Without ACPI a graceful `virsh shutdown` (and therefore MAAS power-off) does not + # work -- the node would only ever hard-stop. + features = { + acpi = true + apic = {} + } + + # CPU: host-passthrough, with the host's `svm` feature PASSED THROUGH. This is + # UNCONDITIONAL and has no opt-out on purpose -- a DC node runs `nova-compute`, so it + # MUST be able to run KVM guests. There is no correct value other than this one. + # + # WHY THIS IS EXPLICIT: with NO cpu block at all, libvirt renders a GENERIC EMULATED CPU + # MODEL that exposes no `svm` -- and the node then comes up looking healthy while being + # unable to launch a single instance. Measured 2026-07-13: the host is an AMD EPYC 9965, + # but a default-CPU guest is handed an "Opteron_G3" with no svm. The identical defect was + # found in modules/cloudinit-vm the same day and broke D-114's model outright. + # + # Contrast modules/opnsense-edge, which DISABLES svm: correct hardening there, because a + # router guest has no business running VMs. A compute node is the opposite case. + cpu = { + mode = "host-passthrough" + } os = { type = "hvm" diff --git a/opentofu/modules/node-vm/variables.tf b/opentofu/modules/node-vm/variables.tf index 102fcdb..1e71a17 100644 --- a/opentofu/modules/node-vm/variables.tf +++ b/opentofu/modules/node-vm/variables.tf @@ -30,5 +30,5 @@ given PXE boot priority (see main.tf's boot-order note; MAAS-managed node VMs PXE-boot per D-103, they are not given a pre-built OS image). EOT - type = list(string) + type = list(string) } diff --git a/opentofu/modules/node-vm/versions.tf b/opentofu/modules/node-vm/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/node-vm/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/office1-network/versions.tf b/opentofu/modules/office1-network/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/office1-network/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/modules/opnsense-edge/main.tf b/opentofu/modules/opnsense-edge/main.tf index db213da..6504b98 100644 --- a/opentofu/modules/opnsense-edge/main.tf +++ b/opentofu/modules/opnsense-edge/main.tf @@ -1,18 +1,29 @@ # opnsense-edge: one OPNsense edge VM (D-100's per-site independent OPNsense # edge with a simulated ISP uplink; D-103: OpenTofu owns creating it). # -# NOT the cloud-init pattern (modules/cloudinit-vm) -- confirmed unreliable -# on OPNsense/FreeBSD this session (see opentofu/README.md's "OPNsense -# deployment research" section, fully sourced). This module instead uses -# OPNsense's own Configuration Importer: a plain ISO9660 volume containing -# /conf/config.xml, attached as a secondary cdrom disk -- mechanically -# IDENTICAL to cloudinit-vm's seed-volume shape (a libvirt_volume with -# create.content.url, attached as device="cdrom"), just a different payload -# and no libvirt_cloudinit_disk resource (that resource is NoCloud-specific; -# wrong format here). Both the base image and the config ISO are prepared -# OUTSIDE OpenTofu by scripts/opnsense-prep-image.sh and -# scripts/opnsense-build-config-iso.sh respectively -- see each variable's -# description for why. +# THIS MODULE NO LONGER SEEDS ANY CONFIG (2026-07-13). It creates a domain that +# boots the FACTORY nano image, and nothing more. Configuration happens AFTER +# boot, over the network. +# +# The config-seed ISO (a libvirt_volume + a cdrom disk) was REMOVED. It was +# INERT: per D-112, OPNsense's Configuration Importer can NEVER fire on a +# pre-installed nano image -- `opnsense-importer -b` probes for a read-only +# root, finds a writable one with a factory /conf/config.xml already present, +# and exits without enumerating a single device. Nothing ever read that ISO. +# Discovering that cost a full session (2026-07-12). +# +# The config.xml template and its renderer are DELETED under D-113(a2) -- +# hand-authoring the appliance's GUI-owned XML caused DOCFIX-191 (management +# lockout), DOCFIX-192 (dead console) and DOCFIX-193 (no DHCP). Edge config is +# now done over the REST API. The provisioning chain, proven on Office1: +# +# boot factory nano +# -> D-112(c) console bootstrap (enable SSH + install the key) +# -> scripts/opnsense-bootstrap-apikey.sh (mint an API key, no GUI click) +# -> scripts/opnsense-api.sh (DHCP, firewall, interfaces) +# +# The base image is still prepared OUTSIDE OpenTofu, by +# scripts/opnsense-prep-image.sh -- see that variable's description for why. # # UNVERIFIED, flagged plainly (see opentofu/README.md for the full account, # including a 2026-07-09 audit pass that corrected the LAN/WAN claim below): @@ -42,46 +53,93 @@ # touching for this module at all. resource "libvirt_volume" "disk" { - name = "${var.vm_name}-disk.qcow2" - pool = var.pool_name - capacity = var.disk_size_bytes + name = "${var.vm_name}-disk.qcow2" + pool = var.pool_name + + # Direct per-VM COPY of the prepared nano image, NOT a backing/overlay. Matches the + # documented `virt-install --import <nano.qcow2>` workflow; OPNsense nano is designed + # to be booted + auto-expanded in place. + # + # CORRECTION 2026-07-12 (DOCFIX-189): the original version of this comment claimed the + # COW backing "made boot2 fault ... the disk shape did [help]". That is FALSE -- the + # fault persisted identically after this change. The real cause was the memory unit + # (see the domain below). This shape is RETAINED because it is the documented import + # workflow, not because it fixed anything. + # + # DISK SIZING IS NOT OWNED HERE. The copy inherits the prepped image's capacity, so the + # size lever is scripts/opnsense-prep-image.sh's GROW arg (default +8G -> an 11 GiB + # nano). There is deliberately no `capacity`/`disk_size_bytes` input: setting one here + # would be silently ignored, which is exactly the trap DOCFIX-189 removed. Consequence + # to know: every VM built from the same prepped base gets the same disk size. + create = { + content = { + url = var.base_volume_path + } + } target = { format = { type = "qcow2" } } - - backing_store = { - path = var.base_volume_path - format = { - type = "qcow2" - } - } -} - -resource "libvirt_volume" "config_seed" { - name = "${var.vm_name}-config.iso" - pool = var.pool_name - - create = { - content = { - url = var.config_iso_path - } - } } resource "libvirt_domain" "vm" { - name = var.vm_name - memory = var.memory_mib - vcpu = var.vcpu - type = "kvm" - running = true + name = var.vm_name + # memory_unit is REQUIRED. On dmacvicar/libvirt >=0.9, `memory` is a raw libvirt + # value, NOT MiB (that was the 0.8-era meaning): with no unit libvirt defaults to + # KiB, so `memory = 2048` renders `-m size=2048k` = 2 MiB. That was the ACTUAL root + # cause of the 2026-07-12 boot triple-fault (measured: `virsh dominfo` Max memory + # 2048 KiB) -- boot2 fits in 2 MiB and echoes /boot.config, then dies handing off to + # /boot/loader, which does not. See docs/incident-20260712-opnsense-edge-boot-triplefault.md. + memory = var.memory_mib + memory_unit = "MiB" + vcpu = var.vcpu + type = "kvm" + running = true + + # ACPI IS REQUIRED. With no `features` block, libvirt renders the machine with + # `acpi=off`, and FreeBSD/OPNsense then panics on the FIRST interrupt setup: + # + # panic: running without device atpic requires a local APIC + # apic_init() at ... mi_startup() + # + # FreeBSD discovers the local APIC from ACPI's MADT table, and the OPNsense kernel + # carries no `atpic` fallback -- so no ACPI means no usable interrupt controller. The + # guest lands in the ddb debugger (`db>`), which presents as a domain that is "running" + # while burning 100% of a core and emitting nothing further on the console. + # + # This was the SECOND boot bug (2026-07-12), found only after the memory-unit fix let + # the kernel get far enough to reach interrupt init. See the incident report. + features = { + acpi = true + apic = {} + } + + # host-passthrough with the AMD `svm` (nested-virt) feature disabled. NOTE: this was + # tried as a triple-fault fix on 2026-07-12 and did NOT resolve it (the real cause was + # the memory unit above). Retained because it is a legitimate nested-virt hardening for + # a guest that has no business seeing svm, not because it fixed anything. + cpu = { + mode = "host-passthrough" + features = [ + { + name = "svm" + policy = "disable" + } + ] + } os = { - type = "hvm" - type_arch = "x86_64" - type_machine = "q35" + type = "hvm" + type_arch = "x86_64" + # i440fx (pc) rather than q35. CORRECTION 2026-07-12 (DOCFIX-189): the original + # comment here claimed q35 "triple-faults ... confirmed via the serial log". That is + # FALSE -- the serial log showed the SAME 262-byte fault before AND after this change; + # the real cause was the memory unit (see above). i440fx is RETAINED only because it + # is the conventional, widely-exercised machine type for FreeBSD/OPNsense guests -- + # NOT because it fixed this incident. q35 was never actually shown to be a problem. + type_machine = "pc" } devices = { @@ -101,19 +159,6 @@ type = "qcow2" } }, - { - device = "cdrom" - source = { - volume = { - pool = libvirt_volume.config_seed.pool - volume = libvirt_volume.config_seed.name - } - } - target = { - dev = "sda" - bus = "sata" - } - } ] # index 0 = the network intended for LAN, index 1 = the network intended @@ -144,5 +189,39 @@ } } ] + + # OPNsense nano images are SERIAL-console only -- the image ships /boot.config with + # `-S115200 -h`, so its loader expects a serial device. + # + # `pty` source + `log` file. Both roles are needed and they are DIFFERENT: + # source.pty -> an INTERACTIVE, BIDIRECTIONAL console, reached with `virsh console`. + # D-112(c) needs this: the one-time bootstrap is a console login. A `file` source is + # WRITE-ONLY capture and cannot be typed into, so it can never bootstrap anything. + # log -> preserves the full boot capture. `virsh console` only shows output from + # the moment you attach, so without this, everything emitted before attach is lost. + # That capture is what made both 2026-07-12 boot bugs legible; do not drop it. + # + # PTY, not a unix socket -- MEASURED, not preferred. A unix socket looks more scriptable + # (socat, no TTY), but `virsh console` REFUSES it: + # error: internal error: character device serial0 is not using a PTY + # and going direct to the socket is not an option either: libvirt creates it + # libvirt-qemu:kvm, and the staging dir's default ACL stamps a named-user entry that + # outranks group perms, so we get r-- and cannot write. Going through libvirtd (which + # runs as root) via `virsh console` needs no extra privilege -- membership of the + # `libvirt` group is enough. PTY satisfies both constraints; unix satisfies neither. + # pexpect drives `virsh console` fine, which is what the bootstrap uses. + # + # (DOCFIX-189 correction, for the record: adding a console in DOCFIX-187 did not CHANGE + # the original triple-fault, it only revealed it. The domain faulted with and without.) + # NOTE: `source` is deliberately OMITTED. The provider's `source.pty` requires a `path`, + # but a pty's path is ALLOCATED BY LIBVIRT at start time (/dev/pts/N) and cannot be known + # in advance -- so it is unusable here. Omitting source yields libvirt's default + # `<serial type='pty'>`, which is exactly what we want. + serials = [ + { + log = { file = "/var/lib/libvirt/vr1/staging/${var.vm_name}-serial.log", append = "on" } + target = { port = "0" } + } + ] } } diff --git a/opentofu/modules/opnsense-edge/variables.tf b/opentofu/modules/opnsense-edge/variables.tf index d86c1d4..220fb75 100644 --- a/opentofu/modules/opnsense-edge/variables.tf +++ b/opentofu/modules/opnsense-edge/variables.tf @@ -18,10 +18,13 @@ type = string } -variable "disk_size_bytes" { - description = "Boot disk size in bytes (the backing_store overlay's capacity, not the base nano image's own size). No default -- pass explicitly." - type = number -} +# NOTE: there is deliberately NO `disk_size_bytes` here. It was REMOVED 2026-07-12 +# (DOCFIX-189) because it was DEAD: once the disk became a direct copy of the prepped +# nano (DOCFIX-187), nothing consumed it, so `disk_size_bytes = 16 GiB` was silently +# ignored and the edge got the base image's 11 GiB instead. An unused variable is legal +# HCL, so neither `tofu validate` nor the plan flagged the lie. +# Disk size is owned by scripts/opnsense-prep-image.sh's GROW arg. Do not re-add a sizing +# input here unless the module actually consumes it. variable "base_volume_path" { description = <<-EOT @@ -34,21 +37,7 @@ create.content.url almost certainly does a plain fetch with no decompression/conversion; that work must already be done. EOT - type = string -} - -variable "config_iso_path" { - description = <<-EOT - Host filesystem path of a plain ISO9660 image containing - /conf/config.xml -- the output of - `scripts/opnsense-build-config-iso.sh`. This is OPNsense's own - Configuration Importer mechanism (NOT cloud-init -- confirmed unreliable - on OPNsense/FreeBSD; see opentofu/README.md's OPNsense research section). - No default: the real config.xml content for this site's LAN/WAN/routing - role is design work that has to happen first -- do not point this at a - placeholder/empty config.xml to make the module "work" sooner. - EOT - type = string + type = string } variable "lan_network_name" { @@ -67,7 +56,7 @@ stays clear in this file even though the real role still comes from config.xml. EOT - type = string + type = string } variable "wan_network_name" { @@ -78,5 +67,5 @@ mesh-link legs (modules/mesh-link) or a per-site ISP-simulated segment are the expected callers here. EOT - type = string + type = string } diff --git a/opentofu/modules/opnsense-edge/versions.tf b/opentofu/modules/opnsense-edge/versions.tf new file mode 100644 index 0000000..a6a130f --- /dev/null +++ b/opentofu/modules/opnsense-edge/versions.tf @@ -0,0 +1,13 @@ +# Provider source mapping for this module. Child modules MUST declare their own +# required_providers so OpenTofu maps the local name `libvirt` to +# dmacvicar/libvirt. WITHOUT this, OpenTofu infers the `libvirt_` resource +# prefix as hashicorp/libvirt (which does not exist) and `tofu init` fails. +# Version is pinned once at the ROOT (opentofu/versions.tf); child modules +# declare source only. (Found on the first real `tofu init`; DOCFIX-179.) +terraform { + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + } + } +} diff --git a/opentofu/templates/README.md b/opentofu/templates/README.md index 2ae5226..fca3177 100644 --- a/opentofu/templates/README.md +++ b/opentofu/templates/README.md @@ -1,66 +1,56 @@ -# opentofu/templates/ -- OPNsense config.xml template +# opentofu/templates/ -- the OPNsense config.xml template is DELETED (2026-07-13) -`opnsense-config.xml.tmpl` is the `{{TOKEN}}`-parameterized OPNsense -`config.xml` for a DC-DC edge (Office1, DC1, or DC2 -- one instantiation per -site). Built directly from OPNsense's own real, currently-shipped -`config.xml.sample` (`opnsense/core`, fetched 2026-07-09 -- see -`.claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md` -for the fetch method) plus the confirmed `staticroutes` schema -(`opnsense/core`'s `Route.xml` model). This is real design work, not a -placeholder -- but several tokens genuinely cannot be filled with a real -value yet because the decisions they depend on (D-100/D-101/D-107) are -still PROPOSED, or because the value can only be measured on a real boot. -Never invent a value for those; leave the token required and unfilled until -the real one exists. +`opnsense-config.xml.tmpl` **no longer exists.** Neither do `scripts/opnsense-render-config.sh` +(the renderer) or `scripts/opnsense-build-config-iso.sh` (the config-seed ISO builder). If you +came here looking for them, this is why -- and what to use instead. -## Pipeline +## Why they are gone -``` -opnsense-config.xml.tmpl - --(scripts/opnsense-render-config.sh, env-var tokens)--> -real config.xml - --(scripts/opnsense-build-config-iso.sh)--> -config.iso (plain ISO9660, /conf/config.xml) - --(opentofu/modules/opnsense-edge, config_iso_path variable)--> -attached as a secondary cdrom disk to the domain -``` +**Hand-authoring the appliance's GUI-owned `config.xml` was the single root cause of every +OPNsense-specific bug this project has had:** -## Token legend +| bug | what hand-authored XML did | +|---|---| +| DOCFIX-191 | the rendered config had no sshd and no key -- it would have **locked management out** | +| DOCFIX-192 | the rendered config silenced the edge's **only console** (no serial, no video) | +| DOCFIX-193 | LAN DHCP was never implemented -- and an old-style ISC `<dhcpd>` block would have been **inert**, because OPNsense 26.1 uses **Kea** | +| 2026-07-13 | a full-config push **drops ~667 migration-populated elements**, including the only 2 firewall pass rules on the box. It survives only because OPNsense silently regenerates them -- an undocumented self-heal we were depending on | -| Token | Status | Notes | -|---|---|---| -| `{{OPNSENSE_HOSTNAME}}` | Design choice, not yet made | e.g. `opnsense`. NOT named `HOSTNAME` -- that collides with bash's own built-in `$HOSTNAME` variable (caught by this delivery's own test harness: `unset HOSTNAME` doesn't actually clear it, since bash repopulates it). | -| `{{DOMAIN}}` | Real, applies D-106's naming convention | e.g. `omega.dc1.vr1.cloud.neumatrix.local`, following the (still-PROPOSED, but already-settled-as-design-intent) `<service>.omega.<dc>.vr1.cloud.neumatrix.local` pattern -- same status as DC1's plane CIDRs elsewhere in this repo: carried-forward design intent, not invented. | -| `{{ROOT_PASSWORD_HASH}}` | MUST be freshly generated, never reused | The real `config.xml.sample` ships a bcrypt hash (`$2y$10$...`) for a well-known default OPNsense password -- reusing it verbatim would mean every edge shares a public default root password, a real exposure. Generate a fresh one per deployment, e.g. `htpasswd -bnBC 10 "" '<password>' \| tr -d ':\n'` (produces a `$2y$10$...`-compatible bcrypt hash) -- NOT independently verified against a real OPNsense login this session; confirm login works before relying on it operationally. | -| `{{NTP_UPSTREAM_SERVERS}}` / `{{NTP_PREFER_SERVER}}` | Real default supplied (optional to override) | Defaults to the exact same public pool OPNsense's own sample ships (`N.opnsense.pool.ntp.org`) -- a real, confirmed default, not invented. D-107 says the edge "syncs upstream to internet NTP pools," consistent with keeping this default. Override only if a different upstream is decided. | -| `{{WAN_IF}}` / `{{LAN_IF}}` | MUST be measured on a real boot | The `<if>` value is the actual `vtnetN` (or similar) device name FreeBRSD assigns -- confirmed from the real sample that this, not interface declaration order, is what assigns the LAN/WAN role (see `opentofu/README.md`'s "Audit pass" section for the full account of this correction). Boot the domain, run `ifconfig`, and set these to match `opentofu/modules/opnsense-edge`'s actual `lan_network_name`/`wan_network_name` wiring. | -| `{{WAN_IPADDR}}` / `{{WAN_SUBNET_BITS}}` / `{{WAN_GATEWAY}}` | Pending D-100/D-101 | The "simulated ISP uplink" network's real addressing isn't assigned yet (NetBox-pending). | -| `{{LAN_IPADDR}}` / `{{LAN_SUBNET_BITS}}` | Pending a design decision | WHICH plane(s) OPNsense's LAN interface actually serves (metal-admin? provider-public? something else?) is not fully specified in the buildout design as read this session -- confirm before filling this in, don't assume. | -| `{{MIRROR_SYNC_PROTOCOL}}` / `{{MIRROR_UPSTREAM_NET}}` / `{{MIRROR_SYNC_PORT}}` | Pending -- mirror software not chosen | D-107 requires the per-DC artifact mirror's own upstream sync as one of exactly two allowed WAN egress paths (the other is NTP), but doesn't specify which mirror software (apt-mirror? aptly? a generic reverse proxy?) -- the port/protocol depend on that choice. Tooling gap register item 3 territory. | -| `{{ROUTE1_NETWORK}}`/`{{ROUTE1_GATEWAY}}`/`{{ROUTE1_DESCR}}`, `{{ROUTE2_*}}` | Optional, pending DC2 CIDR assignment | At most 2 static routes needed per the known topology (a peer-DC replication-plane route at DC1/DC2; an Office1-management route). Leave a slot's `*_NETWORK` var unset to omit it entirely -- `scripts/opnsense-render-config.sh` drops an unset slot rather than emitting an empty/invalid `<route>`. DC2's own CIDRs aren't assigned yet (D-101 open item), so DC1's route to DC2 can't be filled in until then. | +**None of those is expressible through the REST API,** where sshd, DHCP and firewall rules are +typed resources with defaults. You cannot forget to enable sshd in a format where sshd is a typed +field. -## Design choices baked into the template (not tokens -- read before assuming these are wrong) +**And the config-seed ISO never worked at all.** Per **D-112**, the OPNsense Configuration +Importer can NEVER fire on a pre-installed nano image: `opnsense-importer -b` probes for a +read-only root, finds a writable one with a factory `/conf/config.xml` already present, and exits +without enumerating a single device. Nothing was ever going to read that ISO. It cost a full +session on 2026-07-12 to discover. -- **Firewall: default-deny WAN egress, explicit NTP + mirror-sync allow rules** - (D-107's "controlled egress ... narrowly scoped: NTP-only, plus the - mirror's upstream sync"). The two default LAN-allow rules are kept exactly - as OPNsense's own sample ships them (basic operation depends on them). -- **`gateways.gateway_item`** is new versus the stock sample (which only has - an implicit DHCP-derived gateway) -- required because the WAN side here is - a static "simulated ISP uplink," not DHCP, and `staticroutes.route.gateway` - must reference a named gateway object (confirmed from the real - `Route.xml` model: `gateway` is a `JsonKeyValueStoreField` needing "a valid - gateway from the list"). Named `WAN_GW` -- change consistently in both - places if you rename it. -- **`unbound`/`nat`/`rrd`/legacy `<filter>`** sections are left exactly as - the real stock sample ships them -- minimal, working defaults, not - independently redesigned. +## What to use instead (D-113(a2), ADOPTED + PROVEN 2026-07-13) -## What this delivery does NOT do +Edge configuration is done over the **OPNsense REST API**. The provisioning chain has **no +`config.xml` in it anywhere**: -Write the actual, final, per-site `config.xml` files (Office1/DC1/DC2) -- -that needs the pending values above filled in with real ones, which in turn -needs Stage 0 ratification (D-100/D-101/D-107) and a real boot to measure -`vtnetN` assignment. This delivery is the mechanism (template + a tested -renderer), consistent with everything else built this session: parameterized -and reusable, not three files with invented IP addresses. + boot the factory nano image + -> D-112(c) console bootstrap (enable SSH + install the service key) + -> scripts/opnsense-bootstrap-apikey.sh (mints an API key via OPNsense's OWN model -- + no GUI click, no re-implemented crypto) + -> scripts/opnsense-api.sh (DHCP, firewall, interfaces -- everything else) + +Proven end to end against the live Office1 edge on 2026-07-13, read AND write: +`docs/changelog-20260713-opnsense-api-proven.md`, +`docs/changelog-20260713-opnsense-api-write-proven.md`, +`docs/changelog-20260713-opnsense-apikey-bootstrap.md`. + +## If you are tempted to bring the template back + +Don't. A `config.xml` renderer is not a safety net -- it is a loaded gun pointed at a live +router. On 2026-07-13 the repo still contained runbook steps telling an operator to render a +config and push it to the edge, which by then would have **clobbered live API-managed DHCP** +(see `docs/changelog-20260713-config-xml-danger-sweep.md`). + +The governing decision is **D-113** (`docs/design-decisions.md`). If you believe the API cannot +express something the edge needs, that is a finding worth raising against D-113 -- not a reason +to hand-write XML. + +Deleted in: `docs/changelog-20260713-config-xml-path-deleted.md`. The files remain in git history. diff --git a/opentofu/templates/opnsense-config.xml.tmpl b/opentofu/templates/opnsense-config.xml.tmpl deleted file mode 100644 index 1b1e1f0..0000000 --- a/opentofu/templates/opnsense-config.xml.tmpl +++ /dev/null @@ -1,176 +0,0 @@ -<?xml version="1.0"?> -<opnsense> - <trigger_initial_wizard/> - <theme>opnsense</theme> - <system> - <optimization>normal</optimization> - <hostname>{{OPNSENSE_HOSTNAME}}</hostname> - <domain>{{DOMAIN}}</domain> - <dnsallowoverride>1</dnsallowoverride> - <dnsallowoverride_exclude/> - <group> - <name>admins</name> - <description>System Administrators</description> - <scope>system</scope> - <gid>1999</gid> - <member>0</member> - <priv>page-all</priv> - </group> - <user> - <name>root</name> - <descr>System Administrator</descr> - <scope>system</scope> - <groupname>admins</groupname> - <password>{{ROOT_PASSWORD_HASH}}</password> - <uid>0</uid> - </user> - <timezone>Etc/UTC</timezone> - <timeservers>{{NTP_UPSTREAM_SERVERS}}</timeservers> - <webgui> - <protocol>https</protocol> - </webgui> - <disablenatreflection>yes</disablenatreflection> - <usevirtualterminal>1</usevirtualterminal> - <disableconsolemenu/> - <ipv6allow>1</ipv6allow> - <bogons> - <interval>monthly</interval> - </bogons> - <pf_share_forward>1</pf_share_forward> - <ssh> - <group>admins</group> - </ssh> - </system> - <interfaces> - <wan> - <enable>1</enable> - <if>{{WAN_IF}}</if> - <mtu/> - <ipaddr>{{WAN_IPADDR}}</ipaddr> - <subnet>{{WAN_SUBNET_BITS}}</subnet> - <gateway>WAN_GW</gateway> - <blockpriv>1</blockpriv> - <blockbogons>1</blockbogons> - </wan> - <lan> - <enable>1</enable> - <if>{{LAN_IF}}</if> - <ipaddr>{{LAN_IPADDR}}</ipaddr> - <subnet>{{LAN_SUBNET_BITS}}</subnet> - </lan> - </interfaces> - <gateways> - <gateway_item> - <interface>wan</interface> - <gateway>{{WAN_GATEWAY}}</gateway> - <name>WAN_GW</name> - <weight>1</weight> - <ipprotocol>inet</ipprotocol> - <descr>simulated ISP uplink gateway (D-100)</descr> - </gateway_item> - </gateways> - <staticroutes> -{{ROUTE1_BLOCK}} -{{ROUTE2_BLOCK}} - </staticroutes> - <unbound> - <enable>1</enable> - </unbound> - <nat> - <outbound> - <mode>automatic</mode> - </outbound> - </nat> - <filter> - </filter> - <rrd> - <enable/> - </rrd> - <ntpd> - <prefer>{{NTP_PREFER_SERVER}}</prefer> - <ispool>{{NTP_UPSTREAM_SERVERS}}</ispool> - </ntpd> - <OPNsense> - <Firewall> - <Filter> - <rules> - <rule> - <enabled>1</enabled> - <statetype>keep</statetype> - <sequence>1</sequence> - <action>pass</action> - <quick>1</quick> - <interface>lan</interface> - <direction>in</direction> - <ipprotocol>inet</ipprotocol> - <protocol>any</protocol> - <source_net>lan</source_net> - <destination_net>any</destination_net> - <description>Default allow LAN to any rule</description> - </rule> - <rule> - <enabled>1</enabled> - <statetype>keep</statetype> - <sequence>11</sequence> - <action>pass</action> - <quick>1</quick> - <interface>lan</interface> - <direction>in</direction> - <ipprotocol>inet6</ipprotocol> - <protocol>any</protocol> - <source_net>lan</source_net> - <destination_net>any</destination_net> - <description>Default allow LAN IPv6 to any rule</description> - </rule> - <rule> - <enabled>1</enabled> - <statetype>keep</statetype> - <sequence>20</sequence> - <action>pass</action> - <quick>1</quick> - <interface>wan</interface> - <direction>out</direction> - <ipprotocol>inet</ipprotocol> - <protocol>UDP</protocol> - <source_net>{{WAN_IPADDR}}</source_net> - <destination_net>any</destination_net> - <destination_port>123</destination_port> - <description>D-107 controlled egress: NTP only, WAN uplink</description> - </rule> - <rule> - <enabled>1</enabled> - <statetype>keep</statetype> - <sequence>21</sequence> - <action>pass</action> - <quick>1</quick> - <interface>wan</interface> - <direction>out</direction> - <ipprotocol>inet</ipprotocol> - <protocol>{{MIRROR_SYNC_PROTOCOL}}</protocol> - <source_net>{{WAN_IPADDR}}</source_net> - <destination_net>{{MIRROR_UPSTREAM_NET}}</destination_net> - <destination_port>{{MIRROR_SYNC_PORT}}</destination_port> - <description>D-107 controlled egress: per-DC mirror upstream sync only</description> - </rule> - <rule> - <enabled>1</enabled> - <statetype>keep</statetype> - <sequence>99</sequence> - <action>block</action> - <quick>1</quick> - <interface>wan</interface> - <direction>out</direction> - <ipprotocol>inet</ipprotocol> - <protocol>any</protocol> - <source_net>any</source_net> - <destination_net>any</destination_net> - <description>D-107 controlled egress: default-deny everything else outbound on WAN</description> - </rule> - </rules> - <snatrules/> - <npt/> - <onetoone/> - </Filter> - </Firewall> - </OPNsense> -</opnsense> diff --git a/opentofu/variables.tf b/opentofu/variables.tf index c5a4674..25e0aa3 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -3,16 +3,12 @@ type = string } -variable "maas_api_url" { - description = "MAAS API endpoint, e.g. http://<maas-region-host>:5240/MAAS (confirmed format from the canonical/maas provider's own docs). Measured, no default." - type = string -} - -variable "maas_api_key" { - description = "MAAS API key for the account OpenTofu authenticates as. Retrieved via the operator's own secret-handling process -- never hardcoded, never printed (this repo's own secrets discipline applies here same as everywhere else). No default. Pass via TF_VAR_maas_api_key (never -var, which lands in shell history). CAUTION: sensitive=true suppresses CLI/plan output only -- the real value is still stored in PLAINTEXT in terraform.tfstate regardless of this flag. See opentofu/README.md 'State file handling' (DOCFIX-175) for the required file-permission/backup posture before running tofu apply." - type = string - sensitive = true -} +# maas_api_url / maas_api_key variables are DEFERRED to Stage 3 (DOCFIX-179): +# removed from Stage 1's root because the provider "maas" block that consumed +# them is deferred (see main.tf's note). Re-add both here when Stage 3 +# instantiates modules/maas-vm-host. The SENSITIVE maas_api_key MUST return via +# TF_VAR_maas_api_key only -- never a committed file (opentofu/README.md +# "State file handling", DOCFIX-175). variable "domain_suffix" { description = "Base domain for per-plane libvirt DNS domains. Anchor value per D-106's naming convention (<service>.omega.<dc>.vr1.cloud.neumatrix.local)." @@ -25,7 +21,7 @@ type = number } -variable "dc1_pool_path" { +variable "vr1_dc0_pool_path" { description = "Host filesystem path for DC1's libvirt storage pool. Measured, real path on the vcloud host -- no default." type = string } @@ -35,7 +31,12 @@ type = string } -variable "dc1_planes" { +variable "vr1_dc1_pool_path" { + description = "Host filesystem path for DC2's libvirt storage pool. Measured, real path on the vcloud host -- no default. NOTE: only DC2's STORAGE POOL is wired (operator ruling 2026-07-10, dc-dc-phase0 Step 5/7). DC2's PLANES stay deferred (Option B) pending NetBox assigning the D-101 supernet -- a storage pool has no address dependency, plane networks do." + type = string +} + +variable "vr1_dc0_planes" { description = <<-EOT DC1's six planes. DC1 INHERITS the DC0 v4 layout UNCHANGED (D-101) -- these CIDRs are copied from scripts/lib-net.sh's PLANE_CIDRS array as it @@ -57,9 +58,41 @@ } } -# dc2_planes has NO variable/default here yet, deliberately: DC2's supernet is -# a D-101 OPEN SUB-ITEM ("exact prefixes assigned in NetBox, non-colliding with -# DC1..."), not yet assigned anywhere. Do not add a guessed default -- once -# NetBox assigns it, add a dc2_planes variable shaped exactly like dc1_planes -# above and a matching module "dc2_planes" block in main.tf (see the commented -# skeleton there). +# vr1_dc1_planes has NO variable/default here yet, deliberately: VR1 DC1's v4 +# supernet is a D-101 OPEN SUB-ITEM ("exact prefixes assigned in NetBox, +# non-colliding with..."), not yet wired here. Do not add a guessed default -- +# once NetBox assigns it, add a vr1_dc1_planes variable shaped exactly like +# vr1_dc0_planes above and a matching module "vr1_dc1_planes" block in main.tf +# (see the commented skeleton there). +# +# D-119 NAMING: vr1-dc0 is VR1's FIRST DC, vr1-dc1 its SECOND -- matching the +# NetBox apex slugs exactly. Never write a bare "dcN" for a VR1 DC here: "dc0" +# means VR0's LIVE testcloud in scripts/lib-net.sh, and that collision is what +# D-119 exists to delete. + +# ---- D-114: Office1 site containment VM (voffice1) ---------------------------- +variable "office1_ssh_pubkey_path" { + description = <<-EOT + Path to the PUBLIC half of the Office1 service SSH key, injected into voffice1's + cloud-init. Read with file() at plan time so the key material never has to be pasted + into the repo or a command line. The PRIVATE half stays jumphost-local and is never + read (SEC-007 tracks its rotation). Public keys are not secret; the path is a var so + the location is not hardcoded. + EOT + type = string +} + +variable "voffice1_vcpu" { + description = "voffice1 vCPU count. Hosts MAAS-region + the LXD VM host and the service VMs MAAS composes into it (D-114)." + type = number +} + +variable "voffice1_memory_mib" { + description = "voffice1 memory in MiB." + type = number +} + +variable "voffice1_disk_bytes" { + description = "voffice1 boot disk size in bytes." + type = number +} diff --git a/runbooks/dc-dc-office1-service-reip.md b/runbooks/dc-dc-office1-service-reip.md new file mode 100644 index 0000000..3992e27 --- /dev/null +++ b/runbooks/dc-dc-office1-service-reip.md @@ -0,0 +1,114 @@ +# Office1 service re-IP -- move NetBox + Tailscale into the D-120 static band + +**EXECUTED 2026-07-15** (D-120 ADOPTED, option (a)). Both services re-IP'd, no wipe, verified on the +wire. The PENDING-VERIFICATION markers below are now RESOLVED with the measured method; this is the +reference procedure the DCs reuse for their own compose /24s. It remains a gated mutation on running +services -- present each step, justify it as minimal, get individual approval -- if ever re-run. + +## Target (EXECUTED) + + office1-netbox 10.10.1.201 -> 10.10.1.10 (static, band .2-.49) + office1-tailscale 10.10.1.202 -> 10.10.1.11 (static, band .2-.49) + +Both VMs are MAAS-COMPOSED LXD VMs on `voffice1`; their addresses come from MAAS. The edge's static +route `10.10.1.0/24 -> 10.10.0.20` is /24-wide and is UNAFFECTED by the change. + +## Preconditions (READ-ONLY) + +1. D-120 ADOPTED; target addresses confirmed. +2. Coordinate with sandbox/C2 work -- the NetBox re-IP briefly interrupts the sandbox NetBox. +3. Record current state: `office1-netbox 10.10.1.201`, `office1-tailscale 10.10.1.202` (as-built). +4. Access: `ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.201` (and `.202`); MAAS on `voffice1` + (`10.10.0.20`), secrets in `/root/maas-secrets/` on the box (operator-held). + +## Step 0 -- Confirm the recovery backstop FIRST (READ-ONLY, blocking) + +Before severing any network path, prove `lxc exec`/`console` reaches each guest independent of guest +networking -- it rides the LXD socket, so it survives the re-IP and is the recovery path if the new +address does not come up. On `voffice1`: + + sudo -n lxc list # instance names (office1-netbox / -tailscale) + sudo -n lxc exec office1-netbox -- hostname # must return -- proves the backstop + +## Step 1 -- MAAS: NOT AVAILABLE on a Deployed machine (finding) + +**MAAS refuses interface edits on a Deployed machine:** `interface unlink-subnet ...` returns +"Cannot unlink subnet interface because the machine is not New, Ready, Allocated, or Broken." The +only way to make it Ready is Release, which **WIPES the OS** -- forbidden (it destroys the seeded +sandbox). So the MAAS-model update is skipped and MAAS-model drift is ACCEPTED (MAAS keeps showing +`.201`/`.202` `mode=auto`). Drift reconciles for free at the next teardown/redeploy. `.10`/`.11` are +off MAAS's auto-assign path: MAAS allocates nodes from `.201+` upward (observed -- the two services +themselves landed at `.201`/`.202`), so the low `.2-.49` band is never auto-handed out. (NB: the +whole non-dynamic space is MAAS's static pool, so "outside the dynamic range" is NOT the guarantee -- +the allocation DIRECTION is.) + +> **DC replication:** a DC building fresh should static-assign `.2-.49` in MAAS BEFORE deploy (a +> Ready machine DOES accept `interface link-subnet <sid> <iface> subnet=<id> mode=STATIC +> ip_address=<addr>`), so there is no drift. The Deployed-block only bit Office1 because its services +> were already deployed in the node band. + +## Step 2 -- Set the address on the GUEST (both files, via lxc exec) [MUTATION: gated] + +Drive this over `lxc exec` from `voffice1`, NOT SSH-to-the-guest-IP: the LXD socket is immune to the +`netplan apply` that swaps the address, so your control channel never drops. Edit BOTH files (netplan +takes effect now; the curtin cfg makes it survive a reboot / cloud-init re-render), then apply: + + for f in /etc/netplan/50-cloud-init.yaml /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg; do + sudo -n lxc exec office1-netbox -- cp "$f" "$f.bak-reip201" + sudo -n lxc exec office1-netbox -- sed -i 's#10.10.1.201/24#10.10.1.10/24#' "$f" + done + sudo -n lxc exec office1-netbox -- netplan apply # gateway4-deprecated warnings are benign + sudo -n lxc exec office1-netbox -- ip -4 addr show enp5s0 | grep inet # verify .10, do not assume + +(Same for `office1-tailscale`: `.202/24 -> .11/24`, backups `.bak-reip202`.) + +## Step 3 -- Each SERVICE's own address config [MEASURED] + +- **NetBox (`office1-netbox`, netbox-docker):** binds `0.0.0.0:8000`, follows the host IP. + `ALLOWED_HOSTS` is **`*`** (config default in `/opt/netbox-docker/configuration/configuration.py`; + NOT overridden in `env/netbox.env`), so NO change and NO restart was needed -- it served at `.10` + immediately (HTTP 200 at `http://10.10.1.10:8000/login/`). +- **Tailscale (`office1-tailscale`):** re-asserted automatically after the interface IP change -- + `BackendState=Running`, `PrimaryRoutes=["10.10.0.0/22"]` unchanged, tailnet IP `100.64.0.53` + unchanged. NO `tailscale up` re-assert was needed. (If a future run shows the route dropped, + re-assert with `tailscale up --advertise-routes=10.10.0.0/22`; node identity/key is unchanged.) + +## Step 4 -- Verify on the wire (READ-ONLY) + + ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.10 'ip -4 addr' # netbox at .10 + curl -s -o /dev/null -w "%{http_code}\n" http://10.10.1.10:8000/api/ # 403/200 = up + ssh -J jessea123@10.10.0.20 ubuntu@10.10.1.11 'tailscale status' # tailscale at .11 + +## Step 5 -- Update the repo references + the importer allowlist [DONE 2026-07-15] + +`10.10.1.201 -> 10.10.1.10` and `10.10.1.202 -> 10.10.1.11` applied across: + +- **CODE (done):** `SANDBOX_HOSTS` `.201 -> .10` in `netbox/roles-aggregates-import.py`, + `netbox/dc-dc-prefixes-import.py`, and `netbox/d115-office-carve.py`; + `tests/dc-dc-prefixes-import/test_logic.py` (both sandbox-URL hosts). + `tests/roles-aggregates-import/run-tests.sh` only greps for the literal `SANDBOX_HOSTS` token and + uses `localhost` (still in the allowlist) -- no address change needed there. +- **Docs (done):** `docs/vr1-office1-as-built.md`, `docs/session-ledger.md`, + `docs/dc-dc-netbox-buildout-scope.md`, `docs/dc-dc-deployment-workflow.md`, + `runbooks/dc-dc-phase1-office1-standup.md`, and D-120 itself. Dated changelogs LEFT as historical + snapshots (they record the deploy-time `.201`/`.202`). +- Gauntlet: `bash scripts/run-tests-all.sh` + `bash scripts/repo-lint.sh` re-run green. + +## Step 6 -- Record the D-120 carve in NetBox (via the sandbox loop) + +Register `10.10.1.0/24`'s child ranges (static `.2-.49` / dynamic `.100-.200` / node `.201-.254`) and +the two service IP assignments in the sandbox NetBox, then feed upstream with C2. NetBox is the IPAM +authority for this; do not leave the layout recorded only in prose. + +## Rollback + +Restore each guest's saved netplan + curtin backups (`.bak-reip201` / `.bak-reip202`) via `lxc exec` +and `netplan apply`, then `git revert` the Step-5 repo change. MAAS needs no rollback (its model was +never changed -- see Step 1). Nothing +here is destructive -- the VMs and their data are untouched; only the L3 address changes. + +## DC replication note + +When a DC's compose /24 is built, its services get D-120 static-band addresses FROM DAY ONE via this +same procedure -- there is no "assign in the node band then re-IP later" for the DCs. This runbook is +the reference. diff --git a/runbooks/dc-dc-phase0-vcloud-prep.md b/runbooks/dc-dc-phase0-vcloud-prep.md index 915e286..785aab2 100644 --- a/runbooks/dc-dc-phase0-vcloud-prep.md +++ b/runbooks/dc-dc-phase0-vcloud-prep.md @@ -1,5 +1,11 @@ # DC-DC Phase 0 -- vcloud host preparation (Stage 1) +> **FIRST EXECUTED 2026-07-10** (vcloud host, OpenTofu v1.12.3) -- run +> end-to-end against real infrastructure. As-executed corrections are folded in +> inline, tagged **(DOCFIX-180)**; the OpenTofu scaffold fixes that run required +> are DOCFIX-179; the workstation prereq installers are DOCFIX-178. See the +> matching `docs/changelog-20260710-*.md`. + Turn the bare vcloud host into a substrate Office1 can deploy VR1 (DC1, DC2, Office1) from. This is the FIRST DC-DC runbook executed against real infrastructure -- everything before this session was repo-only prep (docs, @@ -29,26 +35,49 @@ --- +## Prerequisites -- runtimes on the vcloud host / Office1 control point (DOCFIX-178) + +Do NOT assume these are already installed. This runbook drives a real libvirt +host and a real `tofu` binary; confirm each BEFORE Step 1. Idempotent installer +scripts live in `scripts/prereqs/` (Debian/Ubuntu; each supports `--check` and +`--dry-run`) -- see that directory's `README.md`. One-pass setup on a fresh +workstation/workspace: + +```bash +bash scripts/prereqs/check-prereqs.sh # read-only: what's missing? +bash scripts/prereqs/install-all.sh # install the missing prereqs (uses sudo) +# re-login (or: newgrp libvirt) if the libvirt group was just added, then: +bash scripts/prereqs/check-prereqs.sh # confirm all required present +``` + +| Prerequisite | Needed by | Installer | +|---|---|---| +| libvirt + qemu-kvm + `virsh`; `qemu:///system` reachable; user in `libvirt` group | Steps 1, 5, 11 | `scripts/prereqs/install-virtualization.sh` | +| OpenTofu `tofu` >= 1.6.0 (installs as `/usr/bin/tofu`, not the `opentofu` snap) | Steps 8-10 | `scripts/prereqs/install-opentofu.sh` | +| Provider-registry reach (registry.opentofu.org) for `tofu init` | Step 8 | network (local mirror if airgapped) | +| `sudo`, or a pre-owned pool-parent dir under `/var/lib/libvirt` | Step 5 mkdir | -- | +| jq; qemu-img + xorriso (later dc-dc stages) | ops scripts; Stage 2/3 OPNsense build | `install-jq.sh`, `install-image-tools.sh` | + +If the host is airgapped by policy, the Step 8 `tofu init` registry fetch needs a +local provider mirror instead -- out of scope here; resolve before Step 8. + +--- + ## Known gap, flagged before you start (not fixed in this runbook) -`opentofu/main.tf` declares `provider "maas" { api_url = var.maas_api_url; -api_key = var.maas_api_key }` UNCONDITIONALLY at the root, even though this -Phase-0 stage only touches `libvirt`-backed resources (planes, storage pools, -mesh links) -- no MAAS resource is instantiated yet (Stage 3's job). Depending -on how strictly OpenTofu validates a declared-but-unused provider block, `tofu -init`/`plan` in Step 4 MAY demand `maas_api_url`/`maas_api_key` be set even -though nothing in this stage's plan uses MAAS. If you hit that: -- Cheapest workaround: pass placeholder-shaped-but-clearly-fake values (e.g. - `maas_api_url=http://not-yet-provisioned.invalid:5240/MAAS`) ONLY if OpenTofu - merely wants the variable non-null and does not attempt to actually reach - it during `plan` for a provider with zero resources. Verify this is true - (a `plan`, not `apply`, should never open a real connection) before relying - on it. -- If OpenTofu actually attempts to validate MAAS reachability at `plan` time - even with zero MAAS resources, that is a real structural finding -- LOG it - (a DOCFIX candidate: split the MAAS provider block into a Stage-3-only root - module) rather than working around it by fabricating real-looking - credentials or skipping `tofu validate`. +**RESOLVED on first execution (2026-07-10, DOCFIX-179/180).** The root +`provider "maas"` block DID force `tofu plan` to demand BOTH `maas_api_url` AND +the SENSITIVE `maas_api_key` for a Stage-1 plan that creates zero MAAS resources +(it fails at variable validation, *before* any provider is configured -- so no +real MAAS connection is ever attempted). Rather than the placeholder workaround +(which bakes a fake sensitive key into `phase0.tfplan`/`terraform.tfstate` -- +the DOCFIX-175 surface), the operator ruled the STRUCTURAL fix: the +`provider "maas"` block and the `maas_api_url`/`maas_api_key` variables are now +REMOVED from the root module until Stage 3 instantiates `maas-vm-host` +(DOCFIX-179). `maas` stays in the root `versions.tf` `required_providers` +(version pinned, lock file complete). **Stage 1 therefore needs no MAAS input +at all.** Stage 3's runbook MUST re-add the provider block + both variables when +it wires `maas-vm-host` (noted in `main.tf`/`variables.tf` in-file). --- @@ -147,11 +176,23 @@ ```bash ip -o link show | awk '{print $2, $0}' | grep -i mtu ``` -Identify which interface(s) carry the vcloud host's real uplink (the one -OpenTofu's virtual networks will ultimately ride over, or bridge to, for the -simulated ISP edges) and record its MTU. Per D-101 (folded in from D-102): -"Prefer jumbo (9000) end-to-end... if pinned at 1500, set the reduced tenant -MTU consistently." **Do not assume jumbo** -- use the measured value. +Identify which interface(s) carry the vcloud host's real uplink and record its +MTU. Per D-101 (folded in from D-102): "Prefer jumbo (9000) end-to-end... if +pinned at 1500, set the reduced tenant MTU consistently." + +**As-executed clarification (2026-07-10, DOCFIX-180) -- there are TWO distinct +MTU domains; do not conflate them:** the measured host *uplink* MTU (e.g. +`enp1s0`) is the ISP/WAN-edge MTU and governs the future per-site ISP-uplink +networks (Stage 2/3), which are naturally ~1500 like real internet. But +`underlay_mtu` here drives the six planes + mesh legs + `office1-local`, which +are *host-internal isolated virtio bridges* -- they never traverse the physical +uplink and ARE jumbo-capable regardless of it. Per D-101's explicit "prefer +jumbo end-to-end" intent (and because real dark fiber is jumbo-capable), set +`underlay_mtu` to the JUMBO value (9000) for the internal fabric when the host +supports it -- keeping tenant MTU at 1500 with full geneve headroom and avoiding +the nested-virt+geneve MTU-stacking failure the design itself flags. First run: +measured uplink 1500, ruled `underlay_mtu=9000`. Run the calculator below for +BOTH the measured uplink and the chosen jumbo value to see each verdict. **Decision to record (D-101 MTU sub-policy, buildout-design Section 3) -- use the tested calculator (DOCFIX-162, tooling gap #7, 19/19 tests) instead @@ -175,8 +216,8 @@ instead of hand arithmetic:** ```bash bash scripts/dc-dc-ceph-disk-budget.sh --total-disk <MEASURED_TOTAL_DISK> \ - --dc1-nodes <N> --dc1-per-node-osd <SIZE> \ - --dc2-nodes <N> --dc2-per-node-osd <SIZE> \ + --vr1-dc0-nodes <N> --dc1-per-node-osd <SIZE> \ + --vr1-dc1-nodes <N> --dc2-per-node-osd <SIZE> \ --backup-overhead-fraction <FRACTION> ``` Target is size=3/min_size=2 BY DEFAULT (ADOPTED, not still open). The exact @@ -242,22 +283,37 @@ ``` Choose paths under whichever filesystem has the room computed in Step 3's disk-budget check. Do not reuse VR0/DC0's existing pool path if this vcloud -host is the SAME physical host that ran the single-DC testcloud (check -`virsh pool-list --all` for any existing pool already pointed at a -candidate path) -- a path collision between VR0's retiring pool and VR1's -new one is exactly the kind of silent-overlap risk this repo's discipline -flags rather than assumes away. +host is the SAME physical host that ran the single-DC testcloud. -**MUTATION** +**As-executed (2026-07-10, DOCFIX-180) -- the vcloud host was NOT pristine.** It +carried a prior hand-built VR0 topology (9 `vr0-*` isolated networks + a `wan` +NAT network) plus the stock `default` pool at `/var/lib/libvirt/images`. +Inventory FIRST: ```bash -sudo mkdir -p /path/you/chose/dc1 -sudo mkdir -p /path/you/chose/office1 +virsh -c qemu:///system net-list --all +virsh -c qemu:///system pool-list --all ``` -(DC2's path is not created yet -- no CIDRs, no plan to instantiate it in -`main.tf` yet either; see `opentofu/main.tf`'s commented DC2 block.) +Tear down leftover prior-generation networks (gated, per +`runbooks/dc-dc-teardown-rollback.md`) for a clean substrate + clean Step-11 +verify -- BUT KEEP a pre-existing `wan` NAT-to-uplink network: it is a +ready-made model for the per-site ISP-uplink network (gap #17) Stage 2/3 needs. +New pool paths must NOT be `/var/lib/libvirt/images` itself (that is `default`'s +path). -Record the two real paths -- they go into Step 7's tfvars as -`dc1_pool_path`/`office1_pool_path`. +**MUTATION** -- `/var/lib/libvirt` is root-owned. Either run the mkdir with +`sudo`, OR (recommended, sudo-free thereafter) have the operator create a +self-owned VR1 parent ONCE, then create the pool subdirs as your own user: +```bash +# one-time, operator (real terminal for the sudo password): +sudo install -d -o "$USER" -g libvirt -m 2775 /var/lib/libvirt/vr1 +# then, no sudo, and DC2's pool too (see below): +mkdir -p /var/lib/libvirt/vr1/dc1 /var/lib/libvirt/vr1/office1 /var/lib/libvirt/vr1/dc2 +``` +DC2's pool dir + the `dc2_storage` module ARE wired now (operator ruling: a +storage pool has no address dependency); only DC2's PLANES stay deferred pending +NetBox's supernet. Record the real paths -- they become Step 7's +`dc1_pool_path`/`office1_pool_path`/`dc2_pool_path`. First run used +`/var/lib/libvirt/vr1/{dc1,office1,dc2}`. --- @@ -267,10 +323,17 @@ ```bash tofu version ``` -If absent, install per OpenTofu's own official install instructions for -this host's OS (not prescribed here -- an OS-specific package/binary install -is a one-time host-prep action the operator runs directly, matching this -repo's practice of not hardcoding install mechanics that vary by distro). +If absent (or older than 1.6.0), install it with the repo's own idempotent +installer -- this is now a documented prerequisite (see the **Prerequisites** +section at the top of this runbook), not an out-of-band step: +```bash +bash scripts/prereqs/install-opentofu.sh # deb method -> /usr/bin/tofu; needs sudo +``` +Ideally the whole prereq set is already satisfied before Step 1 +(`bash scripts/prereqs/check-prereqs.sh`); this Step is the in-sequence +confirmation that `tofu` specifically is ready. For a non-Debian/non-Ubuntu +control point, `install-opentofu.sh` fails loud with a pointer to OpenTofu's +own portable install options rather than guessing a package manager. **CHECK -- registry network access** (needed for `tofu init` in Step 8 to fetch `dmacvicar/libvirt` 0.9.8 and `canonical/maas` 2.7.2) @@ -331,11 +394,16 @@ tofu validate ``` Or equivalently, from the repo root: `bash scripts/opentofu-validate.sh`. -This is the FIRST real run of this script against a real `tofu` binary -- -`opentofu/README.md` has carried a **SCAFFOLD, UNVALIDATED** banner since -authoring; a clean run here is the first evidence closing that banner (do -not remove the banner from the README until this has actually run clean -- -update it as part of this stage's completion, not preemptively). + +**As-executed (2026-07-10, DOCFIX-179): the first real `tofu init` FAILED** +until two fixes landed, now both in the repo (a fresh clone inits clean): +1. Each child module using a provider resource needs its OWN `required_providers` + (`modules/*/versions.tf`). Child modules do NOT inherit provider SOURCE + mapping, so OpenTofu inferred the `libvirt_` prefix as a nonexistent + `hashicorp/libvirt` and init failed. Fixed for all libvirt + maas modules. +2. `tofu fmt -recursive` on the never-`fmt`'d tree (11 files). +The `opentofu/README.md` banner is now updated from SCAFFOLD/UNVALIDATED to +STAGE-1-VALIDATED (the Stage 2/3 modules remain unexercised). If `validate` surfaces a schema mismatch against the flagged UNVERIFIED notes in `opentofu/README.md` (the `node-vm` boot-order attribute shape is @@ -352,16 +420,23 @@ ```bash cd opentofu -tofu plan -out=phase0.tfplan +tofu plan -input=false -out=phase0.tfplan ``` -Review the plan output line by line against what Step 7's tfvars specify: -expect creates for `module.dc1_planes` (six `libvirt_network` resources), -`module.dc1_storage` + `module.office1_storage` (two `libvirt_pool` -resources), and `module.mesh_dc1_dc2` / `module.mesh_dc1_office1` / -`module.mesh_dc2_office1` (three more `libvirt_network` resources for the -D-100 dark-fiber triangle legs). Confirm nothing else is planned (no DC2 -plane resources -- that module block is commented out in `main.tf` and -should stay that way until gap #3's DATA half closes). +(`-input=false` so a missing required var ERRORS instead of hanging on an +interactive prompt in a non-interactive/agent shell -- DOCFIX-180.) + +Review the plan line by line. **As-executed (2026-07-10, DOCFIX-180): the plan +is 13 resources, not the 11 an earlier draft of this step implied:** +- `module.dc1_planes` -- six `libvirt_network` (the six planes) +- `module.dc1_storage` + `module.office1_storage` + **`module.dc2_storage`** -- + THREE `libvirt_pool` (DC2's storage pool is wired; only its planes are deferred) +- **`module.office1_network`** -- one `libvirt_network` (`office1-local`; + DOCFIX-163 added this, and an earlier draft's expected-list omitted it) +- `module.mesh_dc1_dc2` / `mesh_dc1_office1` / `mesh_dc2_office1` -- three + `libvirt_network` (the D-100 dark-fiber legs) + +Confirm NO DC2 *plane* resources (that block stays commented in `main.tf` until +NetBox assigns the supernet). Expect `Plan: 13 to add, 0 to change, 0 to destroy`. **GATE:** the plan matches this expectation exactly. If it doesn't (extra resources, missing resources, or an unexpected diff), STOP and reconcile @@ -414,7 +489,7 @@ Expect: the six DC1 plane networks, the three mesh-link networks, and the `dc1`/`office1` storage pools, all `active`. Cross-check each plane network's CIDR against `opentofu/variables.tf`'s `dc1_planes` default (which mirrors -`scripts/lib-net.sh`'s `PLANE_CIDRS` -- DOCFIX-151's `lib_net_select_dc dc1` +`scripts/lib-net.sh`'s `PLANE_CIDRS` -- DOCFIX-151's `lib_net_select_dc vr1-dc0` no-op is the same six values, confirming both sources still agree). ```bash diff --git a/runbooks/dc-dc-phase1-office1-standup.md b/runbooks/dc-dc-phase1-office1-standup.md index ea94f14..2cd2f9a 100644 --- a/runbooks/dc-dc-phase1-office1-standup.md +++ b/runbooks/dc-dc-phase1-office1-standup.md @@ -1,85 +1,137 @@ -# DC-DC Phase 1 -- Office1 headend standup (Stage 2) +# DC-DC Phase 1 -- Office1 site standup (Stage 2) -Stand up the operator's deployment headend -- MAAS region controller, confirmed -OpenTofu reach, NetBox, GitBucket, Tailscale -- BEFORE either DC substrate -exists. This is the second DC-DC runbook; it follows -`runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit gate is this -runbook's entry condition. Per this repo's session contract, do NOT re-derive -anything Stage 1 already settled (nested KVM, disk budget, MTU, the DC1/Office1 -libvirt planes and pools, the mesh-link networks) -- confirm Stage 1's gate -passed and move on. +> **REWRITTEN 2026-07-13 to the D-114 model.** The previous version of this +> runbook built THREE SIBLING SERVICE VMs on the vcloud host (MAAS-region, +> NetBox, GitBucket), each behind an "Option A (OpenTofu, blocked) / Option B +> [GitBucket is since REMOVED from scope entirely -- D-116.] +> (manual virt-install)" fork, with Tailscale on the headend host. **That model +> is SUPERSEDED.** D-114 (ADOPTED 2026-07-13) replaces it with ONE Office1 site +> containment VM (`voffice1`) that IS the facility, with MAAS + LXD running ON +> it and the non-stack service machines COMPOSED BY MAAS into that LXD. Do not +> work from a cached memory of the old shape -- it will build the wrong thing. -**Authoring context (read before running):** this runbook was authored PREP-ONLY, -with no live vcloud host, MAAS, NetBox, GitBucket, or Tailscale reachable in the -authoring session. Every command below is written to be run for real, later, -by the operator -- it has not been executed against real infrastructure. Per -`opentofu/README.md` and `docs/dc-dc-deployment-workflow.md` Stage 2 (tooling -gap register items #2, #3, #9): **none of the three Office1 service VMs -(MAAS-region, NetBox, GitBucket) have an OpenTofu module instantiation yet** -- -`opentofu/modules/cloudinit-vm` exists as a mechanism, but no image source has -been chosen and no `user_data`/`meta_data`/`network_config` content has been -designed for any of the three. This runbook does not invent that content. It -gives each VM-creation step two explicit paths (Section "Provisioning-path -decision" below) and flags the OpenTofu path as blocked-pending-design rather -than silently choosing the manual path for the operator. +Stand up the Office1 SITE: the containment VM that simulates the facility, the +MAAS region controller that runs inside it, the LXD VM host MAAS composes into, +and the non-stack service machines (NetBox, Tailscale -- GitBucket dropped, D-116) as +MAAS-visible VMs -- BEFORE either DC substrate exists. This is the second DC-DC +runbook; it follows `runbooks/dc-dc-phase0-vcloud-prep.md` (Stage 1), whose exit +gate is this runbook's entry condition. Per this repo's session contract, do NOT +re-derive anything Stage 1 already settled (nested KVM, disk budget, MTU, the +DC1/Office1 libvirt planes and pools, the mesh-link networks) -- confirm Stage +1's gate passed and move on. + +**Execution status (measured 2026-07-13, not inferred):** + +- Office1's OPNsense edge (`office1-opnsense`) is **BUILT, RUNNING, ROUTING, and + SERVING DHCP** (Kea). Its config is REST-API-managed per D-113(a2). +- `office1-local` and `office1-wan` libvirt networks exist and are active. +- **`voffice1` EXISTS and is RUNNING** (`virsh list --all` -> `voffice1 running`; + `module.voffice1` is in `opentofu/terraform.tfstate`). It was applied by the + main session's `tofu apply`, not by this runbook's authoring pass. +- **NOT yet done:** everything from Step 5 onward -- MAAS-region on `voffice1`, + LXD on `voffice1`, the LXD-KVM-host registration, and every MAAS-composed + service machine. Those are this runbook's remaining work. **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 Phase 1 (goal/build/gate) and Section 5 (OpenTofu/MAAS/NetBox/Juju boundary); `docs/dc-dc-deployment-workflow.md` Stage 2 (tracker row -- update its `**State:**` line when this runbook completes a real run) and the Tooling gap -register (items #2, #3, #9 above all bear directly on this stage; items #12 -and #16 were found and CLOSED during this runbook's own authoring/update -pass -- see Open questions #1/#2 below; item #17 was found while closing -#16 and remains OPEN, a cross-site gap, not specific to Office1); -`opentofu/README.md` (module scope/status for `modules/cloudinit-vm` / -`modules/base-image` / `modules/office1-network` / `modules/opnsense-edge`); -`netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer, -built 2026-07-09 -- MECHANISM only, see Step 7). +register; `opentofu/README.md` (module scope/status for `modules/cloudinit-vm` +/ `modules/base-image` / `modules/office1-network` / `modules/opnsense-edge`); +`netbox/dc-dc-prefixes-import.py` (the NetBox multi-DC/dual-stack importer -- +MECHANISM only, see Step 10). -Decisions this runbook owns: **D-103** (OpenTofu/MAAS/Juju lifecycle seam -- -Office1 service VMs are OpenTofu-created, eventually; MAAS owns their -provisioning lifecycle once enlisted the same way DC node VMs are), **D-107** -(airgap posture -- Office1 is explicitly OUT of the core-service path: no node -artifacts, no NTP served from Office1; Tailscale is a narrowly-scoped front -door, not a general egress path), **D-106** (naming convention -- Office1 -services use a region-level subdomain outside the per-cloud convention, e.g. -`maas.office1.vr1...`; exact scheme is this stage's job to fix, not yet fixed), -and **D-101** (referenced only -- NetBox is the IPAM apex this stage stands up, -but the address literals D-101 requires are NOT this stage's job to assign). +Decisions this runbook owns: **D-114** (site containment VMs + MAAS-composed +LXD VMs for the non-stack machines -- the model this runbook now implements, +including the LXD 5.21 LTS pin), **D-103** as AMENDED by D-114 (OpenTofu owns +the site containment VM and the site's networks; INSIDE a site, MAAS composes +the service machines -- MAAS still owns the provisioning lifecycle and still +does NOT compose the OpenStack node VMs), **D-107** (airgap posture -- Office1 +is explicitly OUT of the core-service path: no node artifacts, no NTP served +from Office1; Tailscale is a narrowly-scoped front door, not a general egress +path), **D-106** (naming convention -- Office1 services use a region-level +subdomain outside the per-cloud convention, e.g. `maas.office1.vr1...`; exact +scheme is this stage's job to fix, not yet fixed), and **D-101** (referenced +only -- NetBox is the IPAM apex this stage stands up, but the address literals +D-101 requires are NOT this stage's job to assign). -!!! PREP-ONLY / NOT YET EXECUTED. No live infrastructure exists or was - reachable in the authoring session. Every MUTATION below is written for - the operator to run and individually gate later, per this repo's - verify-before-mutate discipline -- none of it has been run. +!!! PENDING VERIFICATION -- the MAAS/LXD install-and-compose sequence (Steps + 5, 6, 7, 8) is the NEW heart of this runbook and its exact command lines + are NOT yet confirmed against Canonical's current documentation. Those + steps name the MECHANISM and the GATE, and mark every unconfirmed flag, + channel, and subcommand as `<PENDING VERIFICATION: ...>`. A separate + verification pass owns filling them in. Per this repo's cardinal rule, a + fabricated flag is worse than an honest placeholder -- do NOT "reconstruct" + these from memory, from a different MAAS version's docs, or from this + repo's own `scripts/phase-00-maas-standup.sh` / `scripts/reenroll-hosts.sh` + (those are the PXE-enlisted METAL-NODE path, a DIFFERENT mechanism from + D-114's COMPOSE path -- see Step 8's own note). -!!! STRUCTURAL BLOCKER, flagged not fixed here: the OpenTofu path for all - three Office1 service VMs needs `user_data`/`meta_data`/`network_config` - designed against `modules/cloudinit-vm` before it can be used -- that - design work has NOT been done. Step 3 below presents the resulting - Option A (blocked) / Option B (manual, interim) fork explicitly for each - VM. Do not silently pick one without reading Step 3 first. +!!! LXD IS VERSION-PINNED to the **5.21 LTS track** (D-114). MAAS 3.6/3.7 is + INCOMPATIBLE with LXD >= 6.7: LXD 6.7 consolidated API endpoints and MAAS's + pinned pylxd 2.3.5 cannot speak to them. Canonical's guidance is "use LXD + <= 6.6 or the 5.21 LTS release until further notice" + (https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749). VR0 + runs 5.21.4. Installing LXD from the default `latest/stable` channel would + silently break the MAAS-LXD seam this entire stage depends on. The pin + belongs in `runbooks/appendix-B-asbuilt-version-lock.md` -- record it there + when the real installed version is measured. + +!!! DHCP HAZARD (D-114). OPNsense/Kea is AUTHORITATIVE for DHCP on + `office1-local` (10.10.0.0/24, pool .100-.199). LXD's own `lxdbr0` runs a + SECOND dnsmasq. That is FINE as long as `lxdbr0` stays on LXD's internal + NAT bridge and is NEVER bridged onto `office1-local`. Two DHCP servers on + one L2 is an intermittent, genuinely unpleasant failure to diagnose. Any + LXD networking change in Step 6 must not put `lxdbr0` on the site LAN. --- -## Entry condition (Stage 1 gate -- confirm, do not re-derive) +## Entry condition (Stage 1 gate + the Office1 edge -- confirm, do not re-derive) **CHECK -- read-only, before starting anything in this runbook** ```bash virsh net-list --all virsh pool-list --all +virsh list --all ``` Expect: the six DC1 plane networks, the three mesh-link networks (dc1<->dc2, dc1<->office1, dc2<->office1), the `office1-local` network -(`opentofu/modules/office1-network`, added by this delivery -- 2026-07-09, -gap #12 CLOSED; picked up by whichever `tofu apply` run of -`dc-dc-phase0-vcloud-prep.md`'s Step 10/11 executes against this repo state, -same root `main.tf` Stage 1 already applies), and the `dc1`/`office1` -storage pools, all `active` -- this is `dc-dc-phase0-vcloud-prep.md`'s Step -11 output. If any of this is missing or `inactive`, STOP -- Stage 1 is not -actually done; go re-run/complete that runbook rather than starting Stage 2 -on an unfinished Stage 1. Do not re-measure nested KVM, disk budget, or MTU -here -- Stage 1 already recorded those; this runbook only reads their OUTCOME (the networks -existing) as its gate. +(`opentofu/modules/office1-network`), the `office1-wan` network (the Office1 +ISP uplink -- gap #17 CLOSED for Office1), and the `dc1`/`office1`/`dc2` +storage pools, all `active`. Expect the `office1-opnsense` domain `running`. +If any of this is missing or `inactive`, STOP -- Stage 1 (or the Office1 edge +build) is not actually done; go complete it rather than starting the rest of +Stage 2 on an unfinished substrate. Do not re-measure nested KVM, disk budget, +or MTU here -- Stage 1 already recorded those; this runbook only reads their +OUTCOME (the networks and the edge existing) as its gate. + +--- + +## The D-114 model in one paragraph (read before Step 1) + +Office1 is a SITE, not three peer VMs on a host. OpenTofu creates ONE +containment VM -- `voffice1`, Ubuntu 24.04, on `office1-local`, taking a DHCP +lease from the OPNsense edge's Kea, reaching the internet through that edge like +a real server behind a real site router. `voffice1` IS the facility. The MAAS +REGION CONTROLLER runs ON it. LXD (5.21 LTS) also runs ON it, and that LXD is +REGISTERED BACK INTO MAAS as an LXD KVM host. The non-stack service machines -- +NetBox and Tailscale -- are then VMs **composed by MAAS** into that LXD +host: MAAS-visible, enlisted, commissioned, deployed, powered, releasable. This +is not a new pattern; it is VR0's own already-proven `lxd` + `tailscale` pair, +applied per site. Explicitly NOT in scope: the Juju-deployed OpenStack services +on the DC nodes -- those stay in Juju-created LXD CONTAINERS (bundle `lxd:N` +placements), invisible to MAAS, exactly as in VR0. Sequencing is an operator +ruling: **Office1 FIRST, fully up. DC1 is GATED behind it. DC2 follows DC1.** + +**`expose_nested_virt = true` on `voffice1` is LOAD-BEARING, not a nicety.** LXD +*virtual machines* are qemu/KVM guests, so composing NetBox/Tailscale +as LXD VMs inside `voffice1` requires nested KVM at L3. (Only LXD *containers* +would be free of that requirement -- and MAAS does not enlist containers.) +Office1 is therefore the CHEAP nesting probe for the whole D-114 model: no +OpenStack, no Ceph, small VMs. **If an LXD VM boots inside `voffice1`, the pod +model is proven to L3 and DC1 inherits that proof** (D-114's own DC1 entry +gate). If it does not, the containment-VM model is abandoned early having cost +one VM -- a scoped retreat, by design, not a redesign. --- @@ -88,126 +140,85 @@ Per this repo's discipline of naming ambiguity explicitly rather than quietly picking an answer: -1. **RESOLVED 2026-07-09 (tooling gap register item #12 CLOSED). What - virtual network do the three Office1 service VMs themselves attach to?** - `opentofu/main.tf` (per Stage 1) previously wired only `office1_pool_path` - (storage only) and the three mesh-link networks (Office1<->DC1, - Office1<->DC2, inter-site only, "management traffic only" per D-100's own - sub-item ruling) -- no Office1-LOCAL network (something the three service - VMs and the vcloud host's own management interface could all share) was - modeled anywhere in `opentofu/`. **Decision: a NEW module - (`opentofu/modules/office1-network`), not a reused host bridge.** A - dc-planes-shaped isolated `libvirt_network` sized for ONE network (Office1 - is a headend, not a six-plane OpenStack DC). Reusing an existing host - bridge was explicitly considered and rejected: it would be the one network - in this whole topology living outside OpenTofu's D-103-mandated inventory, - and would silently break this repo's own `virsh net-list --all` - completeness assumption (used as a CHECK step in this runbook's own Entry - condition, above) -- exactly the kind of untracked, unreproducible state - D-103 exists to prevent, and the same class of debt this runbook already - logs explicitly when Option B (manual VM creation) is used. Instantiated - for real in root `main.tf` as `module "office1_network"` (needs no - unmeasured value beyond `domain_suffix`/`underlay_mtu`, both already real - required inputs). See `opentofu/README.md` and - `docs/changelog-20260709-office1-network-edge.md` for the full writeup. - Use `module.office1_network.network_name` (`"office1-local"`) as the - network for each service VM's `cloudinit-vm` `network_config` and for - Step 4b's OPNsense edge LAN side below -- do not invent a different - network name. -2. **RESOLVED 2026-07-09 (tooling gap register item #16 CLOSED). Does - Office1 get its own OPNsense simulated-ISP edge, and if so, which stage - creates it?** The buildout design's topology (Section 1) is explicit: - "full dark-fiber triangle... plus a **per-site** OPNsense simulated ISP - edge" -- three sites, three edges. Stage 3's Build list already covered - DC1/DC2's edges; neither Stage 2 nor Stage 3 previously instantiated one - for Office1. **Decision: YES, Office1 gets its own `modules/opnsense-edge` - call, and Stage 2 (this runbook) owns creating it** -- new Step 4b below. - Rationale: this runbook's entire scope is already "Office1 headend - standup," and the OPNsense edge is part of that headend's own boundary, - not DC substrate; Stage 3 (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`) - has no reason to reach back into Office1's own infrastructure to build a - piece of it. This mirrors Stage 3's own `dc1_opnsense`/`dc2_opnsense` - shape exactly (same module, same variable names) -- see Step 4b. Its LAN - side resolves cleanly to open question #1's new network - (`module.office1_network.network_name`); its WAN side does NOT -- no - dedicated per-site ISP-uplink/WAN network exists for ANY site yet - (DC1/DC2 included -- confirmed by reading Stage 3's own `dc1_opnsense` - template, which carries the identical unresolved `wan_network_name` - placeholder). This is now tracked as tooling gap register item #17 (NEW), - a genuinely deeper, cross-site gap -- not invented here as an Office1-only - workaround. Step 4b's module call therefore stays commented out, same as - DC1/DC2's own not-yet-instantiated edges, until gap #17 closes and real - `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` - values exist. +1. **RESOLVED 2026-07-13 by D-114. The Option A / Option B provisioning fork is + CLOSED.** The old fork ("OpenTofu `cloudinit-vm`, blocked pending user_data + design" vs "manual `virt-install`, logged as debt") existed because no + `cloudinit-vm` instantiation had ever been designed. **OpenTofu (Option A) is + the path, and it is DONE:** `module "voffice1"` in `opentofu/main.tf` is a + real, applied `cloudinit-vm` instantiation -- real image source + (`modules/base-image`, the official Ubuntu 24.04 noble cloud image), real + `user_data` (identity + access + guest agent), real `meta_data`, real + `network_config` (DHCP on `office1-local`, matched by interface GLOB rather + than a guessed kernel NIC name), and the new `expose_nested_virt` variable. + There is nothing left to fork on: **the three service VMs are no longer + OpenTofu VMs at all** -- they are MAAS-composed LXD VMs (Step 8). Do not + re-introduce a manual `virt-install` path for them. +2. **The exact MAAS/LXD install, LXD-KVM-host registration, and VM-compose + command lines are NOT yet verified.** See the PENDING VERIFICATION banner + above. Steps 5-8 carry the placeholders; a separate verification pass against + Canonical's current docs owns filling them in. Do not run those steps from a + remembered flag set. 3. **NetBox's own address literals are not this stage's job**, per D-101's own text ("the org ULA /48, the per-DC GUA carve... and DC2's v4 supernet are - NetBox-authoritative... NOT hardcoded in this decision"). This runbook - stands NetBox up and can run its import MECHANISM (Step 7), but the - underlying `ORG_ULA_48` / `DC_GUA_PREFIX` / `DC2_V4_SUPERNET` values do not - exist yet -- see Step 7's own gate. -4. **GitBucket's VR1 repo path/name is not decided.** The historical D-014 path `jesse.austin/openstack-caracal-ipv4` names a DIFFERENT, - pre-existing `git.baldurkeep.com` instance from the v1 rehearsal -- this - stage stands up a SEPARATE, vcloud-local GitBucket instance (per the - buildout design's own "GitBucket (vcloud-local mirror)" wording) -- what - it mirrors, and under what repo path, is not decided here. Flag as an - open item before Step 8's mutation. -5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the - front door is "Office1 only" -- the actual tailnet ACL policy, tags, and - which services (NetBox web UI? GitBucket web UI? MAAS UI? SSH to the - service VMs?) are exposed through it versus reachable only from Office1's - own local network, is real policy design left to Step 9 and the operator's - own tailnet account -- not invented here. -6. **Exact MAAS / NetBox / GitBucket version/channel pins are not decided.** - Each install step below names the well-known official install MECHANISM - (snap for MAAS, Docker Compose for NetBox, WAR/Docker for GitBucket) but - does not pin an exact version/channel/tag -- consult each project's current - official install docs at execution time and record whatever is actually - installed in the changelog entry for this runbook's real run. + NetBox-authoritative... NOT hardcoded in this decision"). This runbook stands + NetBox up and can run its import MECHANISM (Step 10), but the underlying + `ORG_ULA_48` / `DC_GUA_PREFIX` / `VR1_DC1_V4_SUPERNET` values do not exist yet -- + see Step 10's own gate. +4. **GitBucket: CLOSED, NOT DEFERRED (D-116).** There is NO Office1-local GitBucket. The operator + ruled 2026-07-13 that the existing `git.baldurkeep.com` remains the git service of record, with + new repos created there as deployment tasks require. This also makes the old open question -- + what a second, Office1-local GitBucket would mirror and under what repo path -- MOOT. One git + service, and it is authoritative. (D-107's per-DC ARTIFACT mirror is a separate concern and is + NOT affected.) ---- -## Provisioning-path decision (applies to all three service VMs) - -Per the structural blocker above, EVERY VM-creation step in this runbook (5, -6, 8) has two paths. Do not silently default to one -- read both, pick -deliberately, and log which path was used in the changelog entry. - -- **Option A (preferred, long-term, BLOCKED pending design):** create the VM - via `opentofu/modules/cloudinit-vm`, now that (a) Open question #1 above is - RESOLVED (attaches to `module.office1_network.network_name`, i.e. - `"office1-local"`) but (b) real `user_data`/`meta_data`/`network_config` - content still needs to be designed for that specific VM (what packages, - what static address, what hostname). This is the D-103-correct path (`OpenTofu - creates... the Office1 service VMs`) and is NOT available tonight -- doing - it right requires a design pass this session did not do and this runbook - will not invent. -- **Option B (interim, manual):** create the VM directly against the vcloud - host's libvirt (e.g. `virt-install` with a real official cloud image ISO/ - qcow2, or an interactive install), attach it to whatever network Open - question #1 resolves to, then run the install steps below directly over SSH - once the VM has a reachable address. This gets Office1 stood up now, at the - cost of NOT being reproducible/versioned in OpenTofu -- log it explicitly as - a known debt (a DOCFIX candidate: "Office1 service VM `<name>` was - hand-created via Option B on `<date>`; migrate to `modules/cloudinit-vm` - once its cloud-init content is designed"). +5. **Tailscale ACL/tag scoping specifics are not decided.** D-107 says the front + door is "Office1 only" -- the actual tailnet ACL policy, tags, and which + services (NetBox web UI? MAAS UI? SSH to the service + machines?) are exposed through it versus reachable only from Office1's own + local network, is real policy design left to Step 12 and the operator's own + tailnet account -- not invented here. NOTE the placement change under D-114: + Tailscale is now a MAAS-COMPOSED SERVICE MACHINE (a VM in `voffice1`'s LXD, + exactly like VR0's own `tailscale` machine), NOT a package installed on the + headend host. A pre-existing subnet route already reaches the jumphost; that + is a different node and does not satisfy this gate. +6. **Exact MAAS / NetBox / LXD version pins are not decided** (beyond + D-114's hard LXD 5.21-LTS-track constraint, which IS decided). Each install + step below names the well-known official install MECHANISM but does not pin an + exact version/channel/tag -- consult each project's current official install + docs at execution time and record whatever is ACTUALLY installed in the + changelog entry for this runbook's real run, and in + `runbooks/appendix-B-asbuilt-version-lock.md`. +7. **`voffice1`'s stable address is not yet fixed.** It takes a DHCP lease from + the edge's Kea (pool .100-.199). A Kea RESERVATION (added over the OPNsense + REST API, `scripts/opnsense-api.sh`) is what makes it stable -- Step 4. Until + that reservation exists, do NOT hardcode `voffice1`'s address anywhere, + including in MAAS's own `--maas-url`. --- ## Sequence ``` -1. Confirm Stage 1 exit gate (read-only, above) -2. Confirm OpenTofu still reaches vcloud libvirt from Office1 (read-only) -3. Read the provisioning-path decision + open questions (above; no cmds) -4. MAAS region controller -- create VM (A or B) + install [MUTATION, gated] -4b. Office1 OPNsense edge -- wire module (gap #16 CLOSED; - instantiation still BLOCKED on gap #17 + real specs) [repo change, gated] -5. NetBox -- create VM (A or B) + install (official Docker) [MUTATION, gated] -6. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; - literals pending -- PARTIAL) [MUTATION, gated] -7. GitBucket -- create VM (A or B) + install (WAR or Docker) [MUTATION, gated] -8. Tailscale -- install + scope to Office1 only [MUTATION, gated] -9. Post-standup verify against the Stage 2 gate (read-only) - -> EXIT GATE (PARTIAL, see GATE section) -> Stage 3 +1. Confirm Stage 1 exit gate + Office1 edge live (read-only, above) +2. Confirm OpenTofu still reaches vcloud libvirt (read-only) +2b. Office1 OPNsense edge -- BUILT AND LIVE (historical + + DANGER banner; do NOT re-run against the live edge) (read-only) +3. voffice1 -- the site containment VM (tofu apply) [MUTATION, gated] + -- APPLIED 2026-07-13; domain running (measured) +4. voffice1 first contact -- lease, Kea reservation, SSH [MUTATION, gated] +5. MAAS region controller ON voffice1 [MUTATION, gated] PENDING VERIF. +6. LXD ON voffice1, pinned to the 5.21 LTS track + [MUTATION, gated] PENDING VERIF. +7. Register that LXD BACK INTO MAAS as an LXD KVM host + [MUTATION, gated] PENDING VERIF. +8. Compose the non-stack service machines into it (MAAS) + -- THE L3 NESTING PROOF; gates DC1 [MUTATION, gated] PENDING VERIF. +9. NetBox -- install on its composed machine [MUTATION, gated] +10. NetBox -- run dc-dc-prefixes-import.py (MECHANISM only; + literals pending -- PARTIAL) [MUTATION, gated] +12. Tailscale -- install on its composed machine, scoped [MUTATION, gated] +13. Post-standup verify against the Stage 2 gate (read-only) + -> EXIT GATE -> Stage 3 (DC1), which D-114 GATES behind this ``` --- @@ -219,431 +230,664 @@ --- -## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt from Office1 (READ-ONLY) +## Step 2 -- Confirm OpenTofu still reaches vcloud libvirt (READ-ONLY) -Stage 1 already proved OpenTofu -> vcloud-host libvirt reachability (its Step -1 CHECK + Step 10 apply). This step is a re-verification, not a re-derivation --- if this session is running some time after Stage 1, confirm nothing -regressed rather than assuming it still holds. +Stage 1 proved OpenTofu -> vcloud-host libvirt reachability and has since +APPLIED for real (the state file holds the DC1 planes, the pools, the three mesh +links, `office1-network`, the OPNsense edge, and `voffice1`). This step is a +re-verification, not a re-derivation. -**CHECK -- from the Office1 operator VM/session** +**CHECK -- from the Office1 operator session** ```bash -cd opentofu -tofu version -virsh -c "$(grep -A1 'variable "libvirt_uri"' -m1 /dev/null 2>/dev/null; true)" list --all 2>/dev/null || true -``` -More directly -- use the exact `libvirt_uri` value recorded in Stage 1's -`opentofu/dc-dc-phase0.auto.tfvars`: -```bash +tofu -chdir="$REPO/opentofu" version virsh -c "<the libvirt_uri value recorded in Stage 1's tfvars>" list --all ``` -Expect: the domain list (the DC1 plane/pool objects have no domains yet, so -this may be empty, but the CONNECTION must succeed with no error). If it -fails, STOP -- this is the same blocking condition Stage 1's Step 1 described; -do not proceed into VM creation without confirmed libvirt reach. +(`$REPO` is set once per session -- see `runbooks/README.md` Conventions.) +Expect: the connection succeeds, and the domain list shows `office1-opnsense` +and `voffice1`. If the connection fails, STOP -- do not proceed into any +mutation without confirmed libvirt reach. -**CHECK -- MAAS provider not yet needed but flagged (Stage 1's Known Gap)** +**CHECK -- the tree still validates** ```bash -grep -n 'provider "maas"' opentofu/main.tf +bash scripts/opentofu-validate.sh ``` -Confirm whether Stage 1 worked around the unconditional `maas` provider block -with placeholder values or whether a DOCFIX split it out -- read whatever -Stage 1's changelog entry recorded, and use the same resolved answer here -rather than re-deciding it. +Expect: root + every module PASS (10/10 as of DOCFIX-194 -- the gate now +validates every module STANDALONE, because root-only validation silently +skipped the two modules root does not call). --- -## Step 3 -- Read the provisioning-path decision and open questions +## Step 2b -- Office1's OPNsense edge: BUILT AND LIVE (READ-ONLY / historical) -No commands. Confirm with the operator (or record your own reasoned choice, -logged) which path -- A or B -- is being used for each of the three VMs -below, and how Open question #1 (Office1-local network) is being resolved for -this run. Do not proceed to Step 4 silently. +The Office1 edge is no longer work this runbook does -- it is a satisfied +PREREQUISITE. It was built 2026-07-12/13 and is routing, NAT-ing, providing +egress, and serving Kea DHCP on `office1-local`. `voffice1` depends on all four +of those. This section is retained as the historical record of how the image is +prepped, and for its safety banner, which is still live. ---- +> ## !!! DANGER -- DO NOT RUN A CONFIG PUSH AGAINST THE LIVE OFFICE1 EDGE (2026-07-13) !!! +> +> **The Office1 edge is BUILT, ROUTING, and SERVING DHCP.** Its DHCP config is now +> **API-MANAGED** (D-113(a2), proven live 2026-07-13). Rendering a full `config.xml` and +> pushing it to `10.10.0.1` **WOULD CLOBBER THAT LIVE STATE** -- a full-config push replaces +> `/conf/config.xml` wholesale and drops ~667 migration-populated elements. +> +> **Two things are now permanently WRONG and must not be attempted:** +> +> 1. **The config ISO cannot work AT ALL.** Per **D-112**, the OPNsense Configuration Importer +> can NEVER fire on a pre-installed nano image -- `opnsense-importer -b` probes for a +> read-only root, finds a writable one with a factory `/conf/config.xml` already present, and +> `bootstrap_and_exit 0`s without enumerating a single device. The ISO is INERT. Nothing was +> ever going to read it. +> 2. **Full-`config.xml` delivery is SUPERSEDED** by D-113(a2): edge configuration is done over +> the **REST API** (`scripts/opnsense-api.sh`), not by hand-authoring the appliance's +> GUI-owned XML. Hand-authoring that XML was the single root cause of DOCFIX-191 (lockout), +> DOCFIX-192 (dead console) and DOCFIX-193 (no DHCP). +> +> **What is actually true today:** Office1 was brought up by the D-112(c) console bootstrap, and +> is now managed over SSH + the REST API. -## Step 4 -- MAAS region controller: create VM + install [MUTATION: gated] - -**CHECK -- confirm no pre-existing MAAS-region VM under this name** -```bash -virsh -c "<libvirt_uri>" list --all | grep -i maas-region -``` -Expect: no match (fresh standup) or a clear, understood match (re-run/repair -case) -- do not proceed if an unexplained existing domain shares the name. - -**MUTATION -- create the VM (Option A or B per Step 3)** - -Option A: not available tonight (blocked, see above) -- instantiate -`module "office1_maas_region"` using `modules/cloudinit-vm` once its -`user_data`/`meta_data`/`network_config` are designed, then `tofu apply`. - -Option B (interim manual path): -```bash -# Illustrative shape only -- fill every <ANGLE-BRACKET> with a value you -# measured/assigned this session; none of these are literals to copy verbatim. -virt-install \ - --connect "<libvirt_uri>" \ - --name maas-region-office1 \ - --memory <MEASURE/ASSIGN -- RAM budget from Stage 1 host measurement> \ - --vcpus <MEASURE/ASSIGN> \ - --disk pool=office1,size=<MEASURE/ASSIGN> \ - --network network=office1-local \ - --os-variant <the guest OS you are installing, per its own docs> \ - --cdrom <path to the official install ISO you fetched, per that OS's docs> \ - --graphics none --console pty,target_type=serial -``` -Boot it, complete the OS install per that OS's own docs, and record the -resulting static/DHCP address this session assigns it -- MEASURE/ASSIGN, never -guessed. This is the exact "known debt" Option B commits to logging (see -Provisioning-path decision above). - -**MUTATION -- install MAAS region controller (official snap-based method)** - -Once the VM is up and reachable over SSH: -```bash -ssh <the address you just assigned> 'snap info maas' -``` -Consult MAAS's current official install docs for the exact channel to pin -(this repo does not hardcode a MAAS snap channel anywhere else to copy from -- -pick and record the current LTS/stable channel per those docs, not a guessed -one) and run the equivalent of: -```bash -sudo snap install maas --channel=<consult official docs for current channel> -``` -Then initialize it as a region controller per MAAS's own current `maas init` -documentation for that installed version -- the exact flags (`--maas-url`, -database backend selection, admin creation) vary by MAAS release; consult the -official docs rather than reusing a flag set from memory or from a different -MAAS version's docs. - -**CHECK -- verify the region controller is reachable** -```bash -curl -sI "http://<the address you assigned>:5240/MAAS/" | head -1 -``` -Expect an HTTP response, not a connection failure. Naming per D-106: this -service's FQDN, once Designate/DNS exists for Office1, would follow the -region-level subdomain scheme D-106 names as an example -(`maas.office1.vr1...`) -- exact scheme is confirmed here as part of this -stage, not assumed; record whatever real hostname convention you actually use. - -**GATE (Stage 2, first Gate bullet):** MAAS region reachable. - ---- - -## Step 4b -- Office1's own OPNsense edge: wire the module [repo change, gated] (NEW -- gap #16 CLOSED) - -Per Open question #2 above (RESOLVED): Office1 gets its own -`modules/opnsense-edge` call, and this runbook (Stage 2) owns creating it -- -not Stage 3, whose runbook only ever covered DC1/DC2. This step mirrors -`runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 4/5 exactly (same module, -same prep scripts, same variable names), just for the `office1` site instead -of `dc1`/`dc2`. - -**CHECK -- confirm no pre-existing Office1 OPNsense VM under this name** -```bash -virsh -c "<libvirt_uri>" list --all | grep -i office1-opnsense -``` -Expect no match (fresh standup) or a clear, understood match (re-run/repair -case). - -**MUTATION -- prepare Office1's OPNsense base image + config.xml + config ISO -(host-local files, gated)**, same mechanism as Stage 3's Step 4: +**Image prep (host-local files only -- the one part still runnable, and only for +a from-scratch re-standup):** ```bash bash scripts/opnsense-prep-image.sh # see the script's own header for its exact args/output path ``` -```bash -export OPNSENSE_HOSTNAME=... DOMAIN=... ROOT_PASSWORD_HASH=... # plus every other token templates/README.md's legend marks REQUIRED for this site -bash scripts/opnsense-render-config.sh office1-opnsense-config.xml -``` -```bash -bash scripts/opnsense-build-config-iso.sh office1-opnsense-config.xml office1-opnsense-config.iso -``` -Every token filled above must be a real, measured/decided value for Office1 -specifically -- no copy-paste of DC1/DC2's config.xml content. -**Repo change -- wire `module "office1_opnsense"` in `opentofu/main.tf`** - -A commented-out skeleton already exists in root `main.tf` (this delivery, -2026-07-09) with `lan_network_name` already resolved -(`module.office1_network.network_name`). Uncommenting it for a real run -additionally requires: -- `memory_mib` / `vcpu` / `disk_size_bytes` -- real values, OPNsense's own - sizing guidance, not invented (same "no invented specs" rule as every - other VM module in this repo). -- `base_volume_path` / `config_iso_path` -- the real output paths from the - two commands above. -- `wan_network_name` -- **BLOCKED on tooling gap register item #17** (no - dedicated per-site ISP-uplink/WAN network exists anywhere in this repo - yet, for ANY site -- confirmed against Stage 3's own `dc1_opnsense` - template, which carries the identical open placeholder). Do NOT wire this - to any `mesh-link` network: D-100's own sub-item ruling confirms those - three legs carry management traffic only, not a simulated ISP uplink. - **STOP here if gap #17 is still open** -- this step cannot be fully - completed, and that is the honest state to record, not a placeholder to - paper over. - -**GATE:** `opentofu/main.tf` diff (once gap #17 closes and every value above -is real) shows exactly one new `module "office1_opnsense"` block, every -argument a real value, nothing else changed -- same GATE shape as Stage 3's -Step 5. - -**GATE (Stage 2 Gate bullet, NEW):** Office1's own OPNsense edge module -wired with a real LAN side (achievable this stage, gap #12 resolved it) -- -full instantiation (and therefore the running edge VM itself) remains -BLOCKED pending gap #17's resolution, same as DC1/DC2's edges. Do not mark -this bullet fully green until gap #17 closes and `tofu apply` has actually -run for `module.office1_opnsense`. +**GATE (Stage 2 Gate bullet):** Office1 OPNsense edge up -- **MET 2026-07-12/13** +(routing, NAT, egress, serial console, SSH-key managed, Kea DHCP). Gap #17 is +CLOSED for Office1 via the dedicated `office1-wan` ISP-uplink network. DC1/DC2 +still have no uplink network -- that is Stage 3's problem, not this one's. --- -## Step 5 -- NetBox: create VM + install (official Docker method) [MUTATION: gated] +## Step 3 -- voffice1: the site containment VM [MUTATION: gated] -**CHECK -- confirm no pre-existing NetBox VM under this name** +> **STATUS: APPLIED 2026-07-13.** `module.voffice1` is in +> `opentofu/terraform.tfstate` and the domain is RUNNING (measured this session: +> `virsh list --all` -> `voffice1 running`). The apply was performed by the main +> session, not by this runbook's authoring pass. This step is retained as the +> procedure -- for a re-standup, and so the gate it satisfies is auditable. +> **Re-running `tofu apply` when `voffice1` is already up is NOT a no-op to be +> taken casually:** an apply that reports "updated in-place" can still BOUNCE a +> domain (measured 2026-07-13 -- an apply restarted the live edge). Read the plan +> before approving it. + +This is the D-114 site containment VM. It IS the Office1 facility: MAAS-region +runs on it, LXD runs on it, and the non-stack service machines are composed into +that LXD by MAAS. It sits on `office1-local` ONLY -- it reaches the internet +THROUGH the OPNsense edge, like a real server behind a real site router. + +**CHECK -- confirm the current state before any apply** ```bash -virsh -c "<libvirt_uri>" list --all | grep -i netbox +virsh -c "<libvirt_uri>" list --all | grep -i voffice1 +tofu -chdir="$REPO/opentofu" plan +``` +Expect either no `voffice1` domain (fresh standup -- the apply will create it) +or the domain present AND a plan showing no changes to `module.voffice1`. An +unexplained diff against a running containment VM is a STOP. + +**MUTATION -- create voffice1 (OpenTofu; Option A, and now the ONLY path)** +```bash +tofu -chdir="$REPO/opentofu" apply +``` +Approve only after reading the plan. The module is already fully specified in +`opentofu/main.tf` -- do not re-derive its inputs here. Its load-bearing +properties, for the record: + +- `modules/cloudinit-vm` off `modules/base-image` (the official Ubuntu 24.04 + noble cloud image -- 24.04 is what VR0 deploys for its own `lxd`/`juju`/ + `tailscale` machines, so the image choice carries no new delta). +- `network_names = [office1-local]` ONLY. No mesh leg, no direct WAN. +- `network_config` takes **DHCP** from the edge's Kea, matching the NIC by GLOB + (`en*`) rather than a guessed kernel name -- the NIC's name is not knowable + before first boot, and naming it would be an inferred value. +- **`expose_nested_virt = true`** -- LOAD-BEARING (see "The D-114 model" above). + Without the host CPU feature passed through, no LXD VM can ever boot inside + `voffice1` and D-114's model fails at its first step. +- Cloud-init is DELIBERATELY MINIMAL (identity + access + guest agent). MAAS and + LXD are installed as separate GATED steps below so they are observable and + individually approved, not buried in a first-boot script that either silently + works or silently does not. + +**SECURE THE STATE FILE** after any apply (standing discipline from DOCFIX-175): +confirm `opentofu/**/*.tfstate*` is gitignored, permissions are tight, and an +out-of-band backup exists. The state stores `sensitive = true` values (e.g. +`var.maas_api_key`) in PLAINTEXT. + +**GATE (Stage 2, first Gate bullet):** `voffice1` exists and is `running`. +**MET 2026-07-13 (measured).** + +--- + +## Step 4 -- voffice1 first contact: lease, Kea reservation, SSH [MUTATION: gated] + +`office1-local` is an isolated libvirt network with NO libvirt-managed DHCP -- +`virsh net-dhcp-leases office1-local` is EMPTY BY DESIGN and is NOT evidence that +`voffice1` failed to get an address. Kea, on the OPNsense edge, is the +authoritative DHCP server for that L2. Look for the lease THERE. + +Per D-114, `voffice1` booting on `office1-local` takes the **first real DHCP +lease** the Kea path has ever served -- a path proven so far only at the daemon +level, never end to end. Treat a successful lease as a real result worth +recording, and a failed one as a real finding (not a nuisance to work around by +hand-assigning a static address). + +**CHECK -- read the lease from the edge (read-only)** +```bash +bash scripts/opnsense-api.sh <PENDING VERIFICATION: the read endpoint for Kea leases -- see the script's own --help/usage; do not guess an endpoint path> +``` +Expect: one lease, in the .100-.199 pool, whose MAC matches `voffice1`'s +interface (`virsh -c "<libvirt_uri>" domiflist voffice1`). MEASURE both -- do +not assume the lease belongs to `voffice1` because it is the only one. + +**MUTATION -- pin the address with a Kea RESERVATION (gated)** + +Add a Kea host reservation binding `voffice1`'s measured MAC to a stable address, +over the OPNsense REST API (`scripts/opnsense-api.sh`). Do NOT hand-edit +`/conf/config.xml` -- see Step 2b's DANGER banner. Do NOT hardcode an address +anywhere (MAAS `--maas-url` included) until this reservation exists and has been +re-measured after a `voffice1` reboot. + +**CHECK -- SSH reach** +```bash +ssh -i <the PRIVATE half of the Office1 service key -- jumphost-local, never read into a session> jessea123@<voffice1's measured address> 'hostnamectl; systemd-detect-virt; cloud-init status --long' +``` +Expect: hostname `voffice1`, `kvm`, and cloud-init `status: done`. A cloud-init +that is still `running` or `error` is a STOP -- read its logs before layering +MAAS on top of a half-configured machine. + +**CHECK -- nested virt is actually exposed (the L3 precondition)** +```bash +ssh <voffice1> 'grep -c -w -E "svm|vmx" /proc/cpuinfo; ls -l /dev/kvm' +``` +Expect: a non-zero count and `/dev/kvm` present. **If this fails, STOP** -- +`expose_nested_virt` did not take effect, and every LXD VM in Step 8 will fail +to boot. That is D-114's own early-abandon signal; do not paper over it. + +**GATE (Stage 2 Gate bullet, NEW):** `voffice1` holds a Kea RESERVATION, is +reachable over SSH, and exposes nested KVM. + +--- + +## Step 5 -- MAAS region controller ON voffice1 [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The exact snap channel and the exact `maas init` +> flag set (`--maas-url`, database backend selection, admin creation) VARY BY +> MAAS RELEASE. They are NOT reproduced from memory here. A separate +> verification pass against MAAS's current official install docs owns filling +> these in. Do not substitute this repo's `scripts/phase-00-maas-standup.sh` +> -- that script drives an EXISTING MAAS, it does not install one. + +Under D-114 the MAAS region controller runs INSIDE `voffice1`, not on a sibling +VM of its own. It is the site's provisioning authority: it will own the LXD KVM +host (Step 7) and every service machine composed into it (Step 8). + +**CHECK -- nothing pre-existing** +```bash +ssh <voffice1> 'snap list maas 2>&1 || echo "not installed"' ``` -**MUTATION -- create the VM (Option A or B per Step 3)**, same shape and same -caveats as Step 4's MAAS-region VM creation (Option A blocked pending -`cloudinit-vm` content design for THIS specific VM; Option B manual, logged as -debt). +**MUTATION -- install MAAS (snap, the official method)** +```bash +ssh <voffice1> 'sudo snap install maas --channel=<PENDING VERIFICATION: the current MAAS LTS/stable channel per MAAS official docs -- 3.6 and 3.7 are both in play in D-114; pick and RECORD, do not guess>' +``` + +**MUTATION -- initialize it as a region controller** + +Per MAAS's own current `maas init` documentation for the installed version. +`<PENDING VERIFICATION: the exact init mode and flags, including the database +backend -- these differ materially between MAAS releases; consult the official +docs for the version actually installed, not a remembered flag set.>` + +Use `voffice1`'s RESERVED address (Step 4) for the MAAS URL -- never the raw +lease, and never a guessed address. + +**CHECK -- the region controller answers** +```bash +curl -sI "http://<voffice1's reserved address>:5240/MAAS/" | head -1 +``` +Expect an HTTP response, not a connection failure. Naming per D-106: this +service's FQDN, once DNS exists for Office1, follows the region-level subdomain +scheme D-106 names as an example (`maas.office1.vr1...`) -- record whatever real +hostname convention you actually use; do not assume it. + +**MUTATION -- create the MAAS API key for later steps** + +Steps 7 and 8 (and `opentofu/modules/maas-vm-host`, if it is ever wired for +Office1) need a real MAAS API key. Generate it per MAAS's own docs and handle it +per this repo's secrets discipline: never printed, never committed, passed as an +environment variable or read from a jumphost-local file only. Note DOCFIX-175: +`var.maas_api_key` lands in `terraform.tfstate` in PLAINTEXT if OpenTofu is given +it -- factor that into where the key is allowed to go. + +**GATE (Stage 2 Gate bullet):** MAAS region reachable, running ON `voffice1`. + +--- + +## Step 6 -- LXD ON voffice1, pinned to the 5.21 LTS track [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The `lxd init` answers (storage backend and size, +> bridge configuration, whether the LXD HTTPS API is exposed and on what +> address/port, and how MAAS is expected to authenticate to it -- trust +> password vs. certificate) are NOT reproduced from memory. They are precisely +> what the MAAS-LXD seam depends on. A separate verification pass against +> Canonical's current MAAS + LXD docs owns filling them in. + +**MUTATION -- install LXD, PINNED (D-114)** +```bash +ssh <voffice1> 'sudo snap install lxd --channel=5.21/stable' +``` +The `5.21` track is a HARD D-114 constraint, not a preference: MAAS 3.6/3.7 +cannot talk to LXD >= 6.7 (pylxd 2.3.5 vs LXD's consolidated API endpoints). +`latest/stable` WILL eventually hand you a broken seam. If LXD is already +present from the image at a different track, refresh it onto 5.21 rather than +proceeding -- and record the measured version in appendix-B. + +**CHECK -- confirm the pin actually took** +```bash +ssh <voffice1> 'snap list lxd; lxd --version' +``` +Expect a 5.21.x version. **A 6.x version here is a STOP** -- it is the exact +incompatibility D-114 pins against. + +**MUTATION -- initialize LXD** +```bash +ssh <voffice1> 'sudo lxd init <PENDING VERIFICATION: interactive vs --preseed, and the actual answers -- storage backend/size, bridge, HTTPS listen address, and the auth mechanism MAAS requires. Consult LXD + MAAS official docs; do not invent a preseed.>' +``` + +!!! **`lxdbr0` MUST STAY on LXD's internal NAT bridge.** Do NOT bridge it onto + `office1-local`. Kea (on the OPNsense edge) is the authoritative DHCP server + for that L2; `lxdbr0`'s own dnsmasq on the same L2 would be a second one. + See the DHCP HAZARD banner at the top of this runbook. + +**CHECK -- LXD is up and its bridge is where it belongs** +```bash +ssh <voffice1> 'lxc network list; ip -br addr show lxdbr0' +``` +Expect `lxdbr0` on its own private subnet, NOT on 10.10.0.0/24. + +**GATE (Stage 2 Gate bullet, NEW):** LXD installed on `voffice1`, version on the +5.21 LTS track (MEASURED), `lxdbr0` isolated from `office1-local`. + +--- + +## Step 7 -- Register that LXD BACK INTO MAAS as an LXD KVM host [MUTATION: gated] (PENDING VERIFICATION) + +> **PENDING VERIFICATION.** The exact MAAS CLI/API call that adds an LXD KVM +> host (its subcommand, its power/auth parameters, the project name, and how +> the LXD certificate trust is established) is NOT reproduced from memory. A +> separate verification pass owns it. VR0's as-built shows the TARGET STATE: +> an LXD KVM host in project `default`, LXD 5.21.4. + +This is the seam D-114 turns on. `voffice1` is a MAAS-managed machine that runs +LXD; that LXD is then registered as a VM HOST inside the same MAAS. It is what +makes the service machines MAAS-visible instead of hand-run containers. + +**Note the D-103/D-114 boundary, precisely:** the composition right granted here +is scoped to THIS LXD VM host and the NON-STACK machines. MAAS still does NOT +compose the OpenStack NODE VMs -- `modules/node-vm` pre-creates those and +`modules/maas-vm-host` registers the VIRSH host so MAAS merely DISCOVERS them. +`maas_vm_host_machine` remains ruled out for nodes. + +**CHECK -- read the current VM-host inventory (read-only)** +```bash +ssh <voffice1> 'maas <PENDING VERIFICATION: profile> vm-hosts read' +``` + +**MUTATION -- add the LXD KVM host** + +PENDING VERIFICATION: the exact MAAS "add a VM host of type LXD" invocation -- +its subcommand, power address, project name, and credentials/certificate trust. +Consult MAAS's own current LXD-VM-host documentation. Do NOT adapt this from +`scripts/phase-00-maas-standup.sh` or `scripts/reenroll-hosts.sh`: those are the +PXE-enlisted metal-node path, a different mechanism. + +**CHECK -- MAAS sees the LXD host, with real capacity** +```bash +ssh <voffice1> 'maas <profile> vm-hosts read' +``` +Expect the LXD host present, with its cores/memory/storage reported -- MAAS +reading real numbers back out of LXD is the evidence the seam works, not merely +that the record was created. + +**GATE (Stage 2 Gate bullet, NEW):** `voffice1`'s LXD is registered in MAAS as an +LXD KVM host and MAAS reads its capacity. + +--- + +## Step 8 -- Compose the non-stack service machines (NetBox, Tailscale) [MUTATION: gated] + +> **PENDING VERIFICATION.** The exact `maas ... vm-host compose` parameters +> (cores, memory, storage, and how the composed machine is attached to the right +> network) are NOT reproduced from memory. A separate verification pass owns +> them. VR0's as-built gives the SIZING PRECEDENT, measured, not invented: +> its `tailscale` machine is **2 cores / 2 GiB / 25 GB, Ubuntu 24.04** -- +> D-114's own figure for what a composed service machine costs. + +**This step is the L3 NESTING PROOF, and it GATES DC1.** The first LXD *virtual +machine* that boots inside `voffice1` proves the whole D-114 pod model to L3. +D-114's DC1 entry gate depends on it. If no LXD VM will boot here, STOP and +report -- do not "work around" it by falling back to LXD containers (MAAS does +not enlist containers) or by hand-running the services on `voffice1` itself +(that reconstructs the flat model D-114 replaced, without saying so). + +**MUTATION -- compose the machines, ONE AT A TIME, individually gated** + +Compose in this order, gating each: **Tailscale first** (it is the smallest, it +is VR0's proven exemplar, and if it boots the model is proven at minimum cost), +then NetBox, then Tailscale. + +```bash +ssh <voffice1> 'maas <profile> vm-host compose <PENDING VERIFICATION: the exact compose parameters -- cores/memory/storage/interfaces. Consult MAAS official docs.>' +``` + +Each composed machine then goes through MAAS's OWN provisioning lifecycle -- +this is the entire point of D-114's ruling (they are MAAS-VISIBLE machines, not +hand-run VMs): + +1. **Enlist** -- MAAS registers the composed VM. +2. **Commission** -- MAAS boots it and inventories it. +3. **Deploy** -- MAAS installs the OS (Ubuntu 24.04, matching VR0's own service + machines). +4. **Power** -- MAAS controls it via the LXD VM host. + +`<PENDING VERIFICATION: the commission/deploy invocations for a COMPOSED machine. +Note that a composed machine may land in a different MAAS status than a +PXE-enlisted one -- that is an explicitly UNCONFIRMED item in the Stage 4 +runbook's own known-gaps list. Confirm it here, on real output, rather than +assuming the PXE flow's statuses.>` + +**CHECK -- the machines are real, MAAS-visible, and DEPLOYED** +```bash +ssh <voffice1> 'maas <profile> machines read | <a jq/grep that prints hostname + status + power type>' +``` +Expect each service machine `Deployed`, powered via the LXD VM host. This is +VR0's `lxd` + `tailscale` shape, reproduced per site. + +**GATE (Stage 2 Gate bullet, NEW -- and D-114's DC1 entry gate, half (b)):** at +least one LXD **virtual machine** BOOTS inside `voffice1`, and the non-stack +service machines are MAAS-composed, deployed, and powered. **L3 nesting PROVEN.** +Record this explicitly -- DC1 is gated on it. + +--- + +## Step 9 -- NetBox: install on its composed machine [MUTATION: gated] + +The VM is now a MAAS-DEPLOYED machine (Step 8), not a hand-built one. This step +only installs the application onto it. **MUTATION -- install NetBox (official netbox-docker Compose method)** -Once the VM is up and reachable over SSH, follow NetBox's own official +Over SSH to the composed NetBox machine, follow NetBox's own official Docker-based install (the `netbox-community/netbox-docker` repo's documented Compose workflow: clone it, configure its `.env` / `docker-compose.override.yml` per its own current docs, `docker compose up -d`). Do not hand-roll a from-source install -- Docker Compose is NetBox's own recommended path and the -lowest-delta one to keep reproducible. Consult NetBox's current official docs -for the exact compose file / image tag to pin; do not fabricate a version tag. +lowest-delta one to keep reproducible. Consult NetBox's current official docs for +the exact compose file / image tag to pin; do not fabricate a version tag. **CHECK -- verify NetBox is reachable** ```bash -curl -sI "http://<the address you assigned>:8080/" | head -1 +curl -sI "http://<the composed NetBox machine's address>:8080/" | head -1 ``` -Expect an HTTP response. Create/record a NetBox API token per NetBox's own -docs (needed by Step 6) -- handle it per this repo's secrets discipline: never +Expect an HTTP response. Create/record a NetBox API token per NetBox's own docs +(needed by Step 10) -- handle it per this repo's secrets discipline: never printed, never committed, passed as an environment variable only (`NETBOX_TOKEN`, matching `netbox/dc-dc-prefixes-import.py`'s own documented usage). **GATE (Stage 2 Gate, partial):** NetBox reachable and running. NOT yet -"authoritative and populated" -- see Step 6. +"authoritative and populated" -- see Step 10. --- -## Step 6 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] +## Step 10 -- NetBox: run the multi-DC/dual-stack import (MECHANISM only -- PARTIAL) [MUTATION: gated] -`netbox/dc-dc-prefixes-import.py` (built 2026-07-09, closes tooling gap #3's -MECHANISM) extends the v1 single-site import to VR1's per-DC, dual-stack -model per D-101. It is idempotent and `--verify-only`-capable. **It does NOT, -and cannot, close the DATA half of gap #3**: the real org ULA /48, the per-DC -GUA carve, and DC2's v4 supernet do not exist yet as assigned values anywhere -in this repo or session. Running it for DC1 only requires the two v6 literals -(`ORG_ULA_48`, `DC_GUA_PREFIX`); running it for DC2 additionally requires -`DC2_V4_SUPERNET` -- none of the three has a default, by the script's own -design, and none is invented here. +`netbox/dc-dc-prefixes-import.py` (DOCFIX-152) extends the v1 single-site import +to VR1's per-DC, dual-stack model per D-101. It is idempotent and +`--verify-only`-capable. **UPDATE 2026-07-14: the literals ARE now assigned** -- `ORG_ULA_48 = +fd50:840e:74e2::/48` (D-118), `DC_GUA_PREFIX` `2602:f3e2:f02::/48` / `f03::/48` +(D-117, measured off the apex), `VR1_DC1_V4_SUPERNET = 10.12.64.0/19` (D-115). The +"do not exist yet" framing below is HISTORICAL. What remains is running the loop +against a real NetBox and feeding the result upstream (C2). See **Step 10b** for the +full loop and **`docs/dc-dc-netbox-buildout-scope.md` section 8** for the rationale. + +Running it for VR1 DC0 requires the two v6 literals (`ORG_ULA_48`, `DC_GUA_PREFIX`); +running it for VR1 DC1 additionally requires `VR1_DC1_V4_SUPERNET` -- none has a +default, by the script's own design, and none is invented here. **CHECK -- dry run, no writes** ```bash -NETBOX_URL="http://<the address you assigned>:8080" \ -NETBOX_TOKEN="<the token from Step 5, passed as an env var, never printed>" \ +NETBOX_URL="http://<the composed NetBox machine's address>:8080" \ +NETBOX_TOKEN="<the token from Step 9, passed as an env var, never printed>" \ ORG_ULA_48="<MEASURE/ASSIGN -- a real RFC 4193 random ULA /48; not generated by this runbook>" \ -DC_GUA_PREFIX="<MEASURE/ASSIGN -- the real per-DC GUA carve from the org's ARIN 2602:f3e2::/32, per whoever administers that block>" \ -python3 netbox/dc-dc-prefixes-import.py --dc dc1 --verify-only +DC_GUA_PREFIX="<MEASURE/ASSIGN -- the real per-DC GUA carve, per whoever administers that block>" \ +python3 netbox/dc-dc-prefixes-import.py --dc vr1-dc0 --verify-only ``` If `ORG_ULA_48` / `DC_GUA_PREFIX` are not yet assigned, this command CANNOT be -run for real yet -- that is the honest state of this stage tonight, not a -missing step in this runbook. Record that as the blocking condition rather -than inventing placeholder-looking values to "get past" it (this script -itself is designed to fail loud on exactly that pattern, per its own header). +run for real yet -- that is the honest state of this stage, not a missing step in +this runbook. Record that as the blocking condition rather than inventing +placeholder-looking values to "get past" it (this script is designed to fail loud +on exactly that pattern, per its own header). See +`docs/dc-dc-netem-and-ula-gua-proposal.md` for the drafted ULA-generation +guidance -- a PROPOSAL, not a ruling. -**MUTATION -- once the literals above are real (a later session/step, not -necessarily tonight)** +**MUTATION -- once the literals above are real (a later session/step)** ```bash NETBOX_URL="..." NETBOX_TOKEN="..." ORG_ULA_48="..." DC_GUA_PREFIX="..." \ - python3 netbox/dc-dc-prefixes-import.py --dc dc1 + python3 netbox/dc-dc-prefixes-import.py --dc vr1-dc0 ``` DC1's v4 is hardcoded in the script itself (D-101: "DC1 equals the validated -template," not an inferred value -- it is the explicit text of an ADOPTED -decision). DC2 additionally needs `DC2_V4_SUPERNET`, not assigned yet either. +template," the explicit text of an ADOPTED decision, not an inferred value). DC2 +additionally needs `VR1_DC1_V4_SUPERNET`, not assigned yet either. -**GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 5). +**GATE (Stage 2 Gate, HONEST partial):** NetBox is reachable/running (Step 9). "Authoritative and populated (planes, per-DC v4, ULA/GUA carve)" is only -PARTIALLY achievable right now: the IMPORT MECHANISM exists and is runnable -the moment the literals are assigned, but the literals themselves are not -assigned this session, so the DATA half of this gate bullet is NOT met yet. -Do not mark this gate green in the tracker until a real `ORG_ULA_48` / -`DC_GUA_PREFIX` (and, for DC2, `DC2_V4_SUPERNET`) exist and this script has -actually been run against this real NetBox instance. +PARTIALLY achievable: the IMPORT MECHANISM exists and is runnable the moment the +literals are assigned, but the literals themselves are not assigned, so the DATA +half of this gate bullet is NOT met. Do not mark this bullet green in the tracker +until a real `ORG_ULA_48` / `DC_GUA_PREFIX` (and, for DC2, `VR1_DC1_V4_SUPERNET`) +exist and this script has actually run against this real NetBox instance. --- -## Step 7 -- GitBucket: create VM + install (official WAR or Docker method) [MUTATION: gated] +## Step 10b -- the NetBox sandbox loop (C4) [dry-run first; upstream write OPERATOR-GATED] -**CHECK -- confirm no pre-existing GitBucket VM under this name** -```bash -virsh -c "<libvirt_uri>" list --all | grep -i gitbucket -``` +Step 10 is one importer inside a larger loop. The loop rehearses EVERY NetBox +mutation on a sandbox before the apex is touched. Full rationale + the write-guard +table + the WAF trap live in **`docs/dc-dc-netbox-buildout-scope.md` section 8**; +this is the operational sequence only. -**MUTATION -- create the VM (Option A or B per Step 3)**, same shape/caveats -as Steps 4 and 5. +**Two hosts, two tokens -- do not cross them (the separation IS the safety property):** -**MUTATION -- install GitBucket (official method)** +| Phase | Host | NetBox | Token | +|---|---|---|---| +| dump (read upstream) | `vcloud` | `netbox.baldurkeep.com` (apex, READ-ONLY) | upstream, `~/vr1-office1-creds/vr1-netbox.env` | +| seed / carve / import / verify | `office1-netbox` (`10.10.1.10`) | sandbox `localhost:8000` | sandbox (operator-held on the box) | -GitBucket ships as a single runnable WAR (`java -jar gitbucket.war`, its own -documented standalone-Jetty mode) or as an official Docker image -(`gitbucket/gitbucket`). Either is GitBucket's own supported path; pick one -and record which. Consult GitBucket's current official docs/release page for -the exact WAR filename/version or image tag to pin -- do not fabricate one. -Open question #3 above (this instance's purpose/repo-path scheme, distinct -from the existing `git.baldurkeep.com` v1 instance per D-014) must be resolved -before deciding what this instance actually mirrors. +`pynetbox` is on `office1-netbox`, NOT on vcloud -- the two pynetbox importers +(`roles-aggregates-import.py`, `dc-dc-prefixes-import.py`) MUST run there. -**CHECK -- verify GitBucket is reachable** -```bash -curl -sI "http://<the address you assigned>:8080/" | head -1 -``` -Expect an HTTP response. +1. **Dump upstream (READ-ONLY, vcloud):** `python3 netbox/prod-draft-dump.py --out + netbox/draft/vr1-draft.json`. A 403 here is the WAF User-Agent trap, NOT the + token (section 8.5) -- confirm with `curl` before touching credentials. +2. **Seed the sandbox (office1-netbox, DRY then `--commit`):** `sandbox-seed.py + --draft draft/vr1-draft.json`. It STRUCTURALLY cannot reach upstream. NetBox 4.6 + needs the assembled v2 token `nbt_<key>.<plaintext>`. +3. **Apply the planned delta in the SANDBOX (DRY then `--commit`), in order:** + `roles-aggregates-import.py` (roles + aggregates + org ULA) -> `d115-office-carve.py` + (site `vr1-off1`, edge role, 8 prefixes) -> `dc-dc-prefixes-import.py --dc vr1-dc0` + then `--dc vr1-dc1` (D-119: the selector IS the apex slug). Literals as in the + Step 10 UPDATE note. The prefix importer PREFLIGHT-fails if the six-plane roles + are absent, so roles-aggregates MUST precede it. +4. **Prove fidelity (READ-ONLY):** dump the sandbox with the same dumper, then + `sandbox-fidelity-check.py --upstream netbox/draft/vr1-draft.json --sandbox + <sandbox-dump>`. Exit 0 = upstream draft + EXACTLY the planned delta. Use the + HARDENED checker (post-2026-07-14) -- a pre-hardening green does not count (8.3). +5. **Feed upstream (C2 -- PRODUCTION WRITE, OPERATOR-GATED, STILL PENDING):** run the + same importers against `netbox.baldurkeep.com` with `--commit` AND + `--yes-write-upstream` (the non-sandbox target requires the explicit override). + Present and individually approve each mutation; never batch. Precondition: re-run + the fidelity check first -- the sandbox's "faithful" verdict predates the hardening. -**GATE (Stage 2 Gate bullet):** GitBucket serving. +**GATE:** the loop's tooling is complete and green; the sandbox-side steps 1-4 are +runnable now. Step 5 (C2) has NOT run -- the apex still lacks the six-plane roles, +RIRs/aggregates, the `vr1-off1` carve, and the 36 DC prefixes (measured 2026-07-14). --- -## Step 8 -- Tailscale: install + scope to Office1 only [MUTATION: gated] +## Step 11 -- GitBucket: REMOVED FROM SCOPE (D-116) -Per D-107, Office1 is explicitly OUT of the core-service path (no node -artifacts, no NTP served from Office1) -- Tailscale here is a narrowly-scoped -OPERATOR front door to Office1's own services, not a general egress/access -path for anything DC-side. Do not wire Tailscale into any DC-facing plane. +**DO NOT BUILD AN OFFICE1-LOCAL GITBUCKET.** Operator ruling 2026-07-13 (D-116): the existing +`git.baldurkeep.com` remains the git service of record for VR1, and new repos are created there as +deployment tasks require. No third service machine is composed. -**CHECK -- confirm Tailscale is not already installed/conflicting** +This step previously walked an Office1-local GitBucket install. It is retained as a tombstone rather +than deleted because the runbook's step numbering is referenced elsewhere, and because someone WILL +come looking for it after reading D-103 (whose GitBucket half this supersedes). + +The old open question this closes -- what a second, Office1-local GitBucket would mirror, and under +what repo path, given `git.baldurkeep.com` already exists -- is MOOT, not deferred. + +NOTE: D-107's per-DC ARTIFACT mirror (OS/package artifacts for node provisioning) is a SEPARATE +concern and is NOT affected by this ruling. + +Proceed directly to Step 12. + +## Step 12 -- Tailscale: install on its composed machine, scoped to Office1 [MUTATION: gated] + +Per D-114, Tailscale is a MAAS-COMPOSED SERVICE MACHINE (VR0's own `tailscale` +machine, reproduced per site) -- NOT a package on the headend host, which is what +the superseded model did. Per D-107, Office1 is explicitly OUT of the +core-service path (no node artifacts, no NTP served from Office1) -- Tailscale is +a narrowly-scoped OPERATOR front door to Office1's own services, not a general +egress/access path for anything DC-side. Do not wire it into any DC-facing plane. + +A pre-existing subnet route already reaches the jumphost. That is a DIFFERENT +node and does NOT satisfy this gate -- do not mistake it for one. + +**CHECK -- confirm nothing conflicting on the composed machine** ```bash -ssh <office1-headend-host-or-VM> 'tailscale version 2>&1 || echo "not installed"' +ssh <the composed Tailscale machine> 'tailscale version 2>&1 || echo "not installed"' ``` **MUTATION -- install (official method)** ```bash -ssh <office1-headend-host-or-VM> 'curl -fsSL https://tailscale.com/install.sh | sh' +ssh <the composed Tailscale machine> 'curl -fsSL https://tailscale.com/install.sh | sh' ``` -Consult Tailscale's own current docs if this host's distro needs a different +Consult Tailscale's own current docs if this machine's distro needs a different install path (the install script itself detects and branches on distro). **MUTATION -- bring up, scoped** ```bash -ssh <office1-headend-host-or-VM> 'sudo tailscale up --advertise-tags=<MEASURE/ASSIGN -- your own tailnet ACL tag for "Office1 headend", per Open question #4 above>' +ssh <the composed Tailscale machine> 'sudo tailscale up --advertise-tags=<MEASURE/ASSIGN -- your own tailnet ACL tag for "Office1", per Open question #5 above>' ``` -The actual ACL policy restricting reachability to Office1's services only -(and NOT to any DC-side network) is configured in the operator's own Tailscale -admin console/ACL file -- this is real policy design (Open question #4), not -invented here. Do not advertise routes into any DC plane from this node. +The actual ACL policy restricting reachability to Office1's services only (and +NOT to any DC-side network) is configured in the operator's own Tailscale admin +console/ACL file -- real policy design (Open question #5), not invented here. Do +not advertise routes into any DC plane from this node. **CHECK -- verify scope** ```bash -ssh <office1-headend-host-or-VM> 'tailscale status' +ssh <the composed Tailscale machine> 'tailscale status' ``` -Confirm only the expected Office1 headend node(s) appear, and that no route -advertisement reaches into DC1/DC2 plane CIDRs. +Confirm only the expected Office1 node(s) appear, and that no route advertisement +reaches into DC1/DC2 plane CIDRs. -**GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only. +**GATE (Stage 2 Gate bullet):** Tailscale access confirmed to Office1 only, from +a MAAS-composed machine. --- -## Step 9 -- Post-standup verify against the Stage 2 gate (READ-ONLY) +## Step 13 -- Post-standup verify against the Stage 2 gate (READ-ONLY) ```bash -curl -sI "http://<maas-region-address>:5240/MAAS/" | head -1 -curl -sI "http://<netbox-address>:8080/" | head -1 -curl -sI "http://<gitbucket-address>:<port>/" | head -1 -ssh <office1-headend-host-or-VM> 'tailscale status' virsh -c "<libvirt_uri>" list --all +ssh <voffice1> 'snap list maas lxd; lxd --version' +ssh <voffice1> 'maas <profile> vm-hosts read' +ssh <voffice1> 'maas <profile> machines read' +curl -sI "http://<voffice1's reserved address>:5240/MAAS/" | head -1 +curl -sI "http://<the composed NetBox machine's address>:8080/" | head -1 +curl -sI "http://<the composed GitBucket machine's address>:<port>/" | head -1 +ssh <the composed Tailscale machine> 'tailscale status' ``` -Cross-check each against the GATE section below before declaring this stage -done or partially done. +Cross-check each against the GATE section below before declaring this stage done +or partially done. --- -## GATE (Stage 2 exit condition -- honest partial state) +## GATE (Stage 2 exit condition) -Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row: "MAAS region -reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA -carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale -confirmed to Office1 only; Office1-local network created; Office1 OPNsense -edge module wired." +Per `docs/dc-dc-deployment-workflow.md` Stage 2's Gate row, as reshaped by D-114: -- **MAAS region reachable:** achievable this stage (Step 4). -- **OpenTofu reaches vcloud host libvirt:** already proven in Stage 1; Step 2 - here only re-verifies it still holds. -- **Office1-local network created:** ACHIEVED by this delivery (gap #12 - CLOSED) -- `module "office1_network"` is instantiated for real in root - `main.tf`, no longer pending a per-run decision. -- **Office1 OPNsense edge module wired:** PARTIALLY achievable (Step 4b, gap - #16 CLOSED for ownership/LAN-side). Full instantiation and a running edge - VM remain BLOCKED on tooling gap register item #17 (no per-site - ISP-uplink/WAN network exists yet, for any of the three sites) plus real - `memory_mib`/`vcpu`/`disk_size_bytes`/`base_volume_path`/`config_iso_path` - values -- do not mark this bullet fully green until gap #17 closes and - `tofu apply` has actually run for `module.office1_opnsense`. -- **GitBucket serving:** achievable this stage (Step 7). -- **Tailscale confirmed to Office1 only:** achievable this stage (Step 8), - contingent on the operator's own ACL policy design (Open question #5). -- **NetBox authoritative and populated:** ONLY PARTIALLY achievable right now. - The instance can be stood up and reachable (Step 5), and the import - MECHANISM exists and is runnable (Step 6) -- but the real address literals - (org ULA /48, per-DC GUA carve, DC2 v4 supernet) are not assigned this - session, so NetBox cannot yet be truly "populated" per D-101. Do not mark - this bullet green in `docs/dc-dc-deployment-workflow.md`'s Stage 2 row until - those literals exist and `netbox/dc-dc-prefixes-import.py` has actually run - against this real instance. +- **Office1 OPNsense edge up (routing, NAT, egress, DHCP):** **MET 2026-07-12/13.** +- **Office1-local + office1-wan networks created:** **MET** (gaps #12 and #17-for-Office1 CLOSED). +- **OpenTofu reaches vcloud host libvirt:** **MET** (Stage 1; re-verified in Step 2). +- **`voffice1` site containment VM exists and runs:** **MET 2026-07-13 (measured).** +- **`voffice1` has a stable (reserved) address and SSH reach, with nested KVM + exposed:** Step 4 -- NOT yet confirmed. +- **MAAS region reachable, running ON `voffice1`:** Step 5 -- NOT yet done. +- **LXD on `voffice1`, on the 5.21 LTS track, `lxdbr0` off `office1-local`:** + Step 6 -- NOT yet done. +- **That LXD registered in MAAS as an LXD KVM host:** Step 7 -- NOT yet done. +- **Non-stack service machines COMPOSED BY MAAS into it, deployed and powered -- + i.e. an LXD VM BOOTS at L3:** Step 8 -- NOT yet done. **This is the D-114 DC1 + entry gate.** +- **GitBucket serving:** Step 11 -- NOT yet done. +- **Tailscale confirmed to Office1 only, from a composed machine:** Step 12 -- + NOT yet done, and contingent on the operator's own ACL policy (Open question #5). +- **NetBox authoritative and populated:** ONLY PARTIALLY achievable. The instance + can be stood up and reachable (Step 9), and the import MECHANISM exists and is + runnable (Step 10) -- but the real address literals (org ULA /48, per-DC GUA + carve, DC2 v4 supernet) are not assigned, so NetBox cannot yet be truly + "populated" per D-101. Do not mark this bullet green until those literals exist + and `netbox/dc-dc-prefixes-import.py` has actually run against the real instance. -This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** --- MAAS/GitBucket/Tailscale/OpenTofu-reach/Office1-network bullets closeable -in full tonight's sequence; the Office1-OPNsense-edge bullet closes its -ownership/LAN-side half only (gap #16), pending gap #17's cross-site -resolution; the NetBox bullet closes its mechanism half only, pending the -D-101 literal assignment that is explicitly out of scope for this stage to -invent. Update `docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect -whichever of these is actually true after a real run, not to "DONE" -wholesale. +This runbook's honest exit state is therefore: **Stage 2 PARTIALLY complete** -- +the substrate half (networks, edge, `voffice1`) is DONE and measured; the SITE +half (MAAS, LXD, the LXD KVM host, and every composed service machine) is the +remaining work, and its command lines are PENDING VERIFICATION. The NetBox DATA +bullet closes its mechanism half only, pending the D-101 literal assignment that +is explicitly out of scope for this stage to invent. Update +`docs/dc-dc-deployment-workflow.md`'s Stage 2 row to reflect whichever of these +is actually true after a real run, not to "DONE" wholesale. --> Proceed to Stage 3 (OpenTofu builds each DC substrate) only once the -achievable bullets above are actually green; the NetBox DATA gap does not -block Stage 3's OWN gate (Stage 3 depends on MAAS reachability and OpenTofu -reach, not on NetBox's DC2/GUA literals), but should be tracked as a standing -gap until closed. +-> **Stage 3 (DC1) is GATED behind this stage.** D-114's sequencing is an +operator ruling: Office1 FIRST, fully up on this model; DC1 is not started until +Office1 is complete; DC2 follows DC1. DC1's own entry gate additionally requires +proving nested KVM inside `vvr1-dc0` (which also proves L4 for tenant instances -- +the one depth VR0 has never exercised). **If L3 fails at Step 8, the +containment-VM model is abandoned early having cost one VM** -- that is the +designed cheap retreat, and it is the correct outcome to report, not a failure to +engineer around. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing any repo - changes made while executing this runbook (e.g. any real `opentofu/` - instantiation once Option A becomes available; any real tfvars, minus - secrets). -- [ ] If any Option A `cloudinit-vm` instantiation was written this run, - `bash scripts/opentofu-validate.sh` green. -- [ ] Changelog entry for this runbook's first real execution (next DOCFIX - number via `bash scripts/ledger-scan.sh`), noting: which provisioning - path (A or B) was used for each of the three VMs, the real addresses/ - hostnames assigned, and whether NetBox's literal-assignment gap was - closed or remains open. + changes made while executing this runbook. +- [ ] If any `opentofu/` change was written this run, `bash + scripts/opentofu-validate.sh` green (root + EVERY module -- DOCFIX-194). +- [ ] Changelog entry for this runbook's real execution (next number via `bash + scripts/ledger-scan.sh`), noting: the MAAS/LXD versions ACTUALLY installed, + `voffice1`'s reserved address, whether the L3 nesting proof passed, which + service machines were composed, and whether NetBox's literal-assignment gap + was closed or remains open. +- [ ] `runbooks/appendix-B-asbuilt-version-lock.md` updated with the LXD 5.21-LTS + pin (D-114 requires it be recorded there) and the measured MAAS version. - [ ] `docs/session-ledger.md` updated with the outcome. -- [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table - updated to the ACTUAL state reached (likely "PARTIAL," not "DONE," per - the GATE section above) -- do not mark NetBox's gate bullet closed - unless the D-101 literals were genuinely assigned and imported this run. -- [ ] If Option B (manual VM creation) was used for any service VM, log the - resulting debt explicitly (a DOCFIX candidate: migrate that VM to - `opentofu/modules/cloudinit-vm` once its cloud-init content is - designed) rather than letting it go unrecorded. -- [x] Open question #1 (Office1-local network) RESOLVED 2026-07-09 -- - `opentofu/modules/office1-network` built and instantiated for real in - root `main.tf` (gap #12 CLOSED). See - `docs/changelog-20260709-office1-network-edge.md`. -- [x] Open question #2 (Office1 OPNsense edge ownership) RESOLVED 2026-07-09 - -- Stage 2 owns it (this runbook's new Step 4b); gap #16 CLOSED. Module - skeleton wired (commented) in root `main.tf`; full instantiation - remains blocked on gap #17 (per-site ISP-uplink/WAN network, not yet - resolved for any of the three sites) plus real VM specs -- do not close - Step 4b's own GATE bullet until that instantiation actually runs. +- [ ] `docs/dc-dc-deployment-workflow.md` Stage 2 row and tracker table updated to + the ACTUAL state reached -- do not mark NetBox's gate bullet closed unless + the D-101 literals were genuinely assigned and imported this run. +- [ ] Every `<PENDING VERIFICATION: ...>` placeholder in Steps 4-8 replaced with a + command confirmed against Canonical's current docs, and the confirmed + sequence folded back into this runbook -- so the next site (DC1, then DC2) + inherits a verified procedure rather than repeating the verification. +- [x] Provisioning-path fork (old Option A / Option B) RESOLVED 2026-07-13 by + D-114 -- OpenTofu is the path, `module "voffice1"` is instantiated and + applied, and the three service VMs are no longer OpenTofu VMs at all. +- [x] Office1-local network RESOLVED 2026-07-09 (`opentofu/modules/office1-network`, + gap #12 CLOSED). +- [x] Office1 OPNsense edge RESOLVED and BUILT 2026-07-12/13 (gap #16 CLOSED for + ownership; gap #17 CLOSED for Office1 via `office1-wan`). diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index 61e43a4..a57aa64 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -73,13 +73,14 @@ log it as exactly what it is: a placeholder pending D-100's still-open sub-item, not a final value. -!!! **OPNsense prep tools not confirmed present.** `scripts/opnsense-prep- - image.sh` requires `bunzip2` + `qemu-img` on PATH; `scripts/opnsense- - build-config-iso.sh` requires `genisoimage` OR `xorriso`. Neither has - been confirmed present on the vcloud host as of this writing - (`opentofu/README.md`). Step 2 below checks for all four before anything - else in this runbook proceeds -- treat a missing tool as a blocking - finding (install it, or escalate), not something to route around. +!!! **OPNsense prep tools.** `scripts/opnsense-prep-image.sh` requires + `bunzip2` + `qemu-img` on PATH. **The ISO9660 builder requirement is GONE + (2026-07-13):** the config-ISO builder (`opnsense-build-config-iso.sh`, + formerly under scripts/) is DELETED (D-112 -- + the Importer can never fire on a nano image, so the ISO was never read; + D-113(a2) -- config is done over the REST API). `genisoimage`/`xorriso` are + no longer needed and are no longer installed by + `scripts/prereqs/install-image-tools.sh`. Step 2 checks the remaining tools. --- @@ -89,10 +90,10 @@ 1. Pre-flight: confirm Stage 1 + Stage 2 gates closed (read-only) 2. Confirm OPNsense prep-tool prereqs on the vcloud host (read-only) 3. Confirm/record still-open decisions this stage depends - on (node sizing, config.xml tokens, MAAS zone/pool, - netem parameters) -- STOP on any that are unresolved (read-only) -4. Prepare DC1's OPNsense base image + render config.xml - + build the config ISO [MUTATION: host-local files, gated] + on (node sizing, MAAS zone/pool, netem parameters) + -- STOP on any that are unresolved (read-only) +4. Prepare DC1's OPNsense base image (render/config-ISO steps DELETED + 2026-07-13 -- config is done over the REST API) [MUTATION: host-local files, gated] 5. Wire modules/opnsense-edge for DC1 into main.tf [repo change, gated] 6. Wire modules/node-vm calls for DC1's node VMs into main.tf (ONLY once sizing is real) [repo change, gated] @@ -152,7 +153,7 @@ done ``` Required: `bunzip2` + `qemu-img` (for `scripts/opnsense-prep-image.sh`), and -EITHER `genisoimage` OR `xorriso` (for `scripts/opnsense-build-config-iso.sh`), +(The `genisoimage`/`xorriso` requirement is GONE -- the config-ISO path is deleted.) plus `curl` or `wget` for the image fetch. If any required tool is absent, install it on the vcloud host (an OS-specific package install, out of scope to prescribe here) before Step 4 -- do not skip the tool and hand-build the @@ -209,6 +210,40 @@ ## Step 4 -- Prepare DC1's OPNsense base image + config.xml + config ISO [MUTATION: host-local files, gated] +> ## !!! STOP -- THIS STEP IS OBSOLETE AS WRITTEN (2026-07-13) !!! +> +> Nothing here clobbers a live system (DC1's edge does not exist yet), but **two of the three +> sub-steps below cannot achieve what they claim**, and following them will waste a session and +> produce a dead edge: +> +> 1. **The config ISO is INERT -- it can never be read.** Per **D-112** (root-caused from +> upstream source): `opnsense-importer -b` probes for a read-only root. On INSTALLER media the +> probe fails and the importer scans attached media. On our PRE-INSTALLED NANO image the root +> is writable AND a factory `/conf/config.xml` already exists, so BOTH conditions hold and it +> `bootstrap_and_exit 0`s **without enumerating a single device**. The Configuration Importer +> can NEVER fire on a nano image, by design. Building the ISO is pure waste. +> 2. **Full-`config.xml` rendering is SUPERSEDED** by **D-113(a2)** (ruled 2026-07-13): edge +> config is done over the **REST API** (`scripts/opnsense-api.sh`), which was proven end to +> end -- read AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the +> appliance's GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) +> and DOCFIX-193 (no DHCP, and an ISC `<dhcpd>` block that would have been inert against a Kea +> backend). None of those bugs is expressible through the API. +> +> **The `WAN_IF`/`LAN_IF` chicken-and-egg problem discussed below is therefore MOOT** -- it was an +> artifact of trying to seed a full config before first boot. The D-112(c) bootstrap (boot on +> factory defaults -> reach the console -> enable SSH + install the key -> configure over the +> network) measures the real `vtnetN` mapping *after* boot, where it is knowable. +> +> **What DC1's edge should actually do (the proven Office1 path):** prep the image -> boot it -> +> D-112(c) console bootstrap -> configure over SSH + the REST API. See +> `docs/changelog-20260712-office1-opnsense-edge-build.md`, +> `docs/changelog-20260713-opnsense-api-proven.md`, and +> `docs/changelog-20260713-opnsense-api-write-proven.md`. +> +> **This step is NOT yet rewritten.** Reducing the template to a minimal bootstrap and rewriting +> Stage 3's edge steps around the API is the open D-113(a2) work. Until that lands, STOP here and +> re-derive rather than following the sub-steps below. + This step produces plain files on the vcloud host filesystem -- no libvirt or MAAS object is created yet. Run from the vcloud host (or wherever these scripts execute with reach to write there). @@ -220,16 +255,9 @@ **Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a real output path -- record that path, it feeds Step 5's `base_volume_path`. -**MUTATION -- render config.xml (only if every non-boot-measured token from -Step 3 is real)** -```bash -export OPNSENSE_HOSTNAME=... DOMAIN=... ROOT_PASSWORD_HASH=... -export WAN_IPADDR=... WAN_SUBNET_BITS=... WAN_GATEWAY=... -export LAN_IPADDR=... LAN_SUBNET_BITS=... -export MIRROR_SYNC_PROTOCOL=... MIRROR_UPSTREAM_NET=... MIRROR_SYNC_PORT=... -# WAN_IF / LAN_IF: cannot be set for real yet -- see the note below. -bash scripts/opnsense-render-config.sh dc1-opnsense-config.xml -``` +**(DELETED 2026-07-13.)** The `config.xml` render step is GONE -- the renderer +(`opnsense-render-config.sh`, formerly under scripts/) no longer exists. Edge config is done over the REST API +(D-113(a2)). See the replacement chain at the end of this step. `WAN_IF`/`LAN_IF` are the one pair the script's own `REQUIRED_VARS` list demands that genuinely cannot be measured before the domain exists (Step 8 creates it). If Step 3 flagged these as not yet measurable, this render @@ -241,12 +269,35 @@ re-seed -- that re-seed mechanic is not designed in this delivery; treat it as a real follow-up need if it comes to that, not an invented shortcut here.) -**MUTATION -- build the config ISO (only once config.xml render succeeded)** -```bash -bash scripts/opnsense-build-config-iso.sh dc1-opnsense-config.xml dc1-opnsense-config.iso -``` -**Expect:** a plain ISO9660 image containing `/conf/config.xml` at a real -output path -- record it, it feeds Step 5's `config_iso_path`. +**(DELETED 2026-07-13.)** The config-ISO step is GONE -- the ISO builder +(`opnsense-build-config-iso.sh`, formerly under scripts/) no longer exists, and the module no longer takes a +`config_iso_path`. The ISO was never read by anything (D-112: the Importer cannot fire on a +pre-installed nano image). + +--- + +### The REPLACEMENT chain (proven on Office1, 2026-07-12/13) + +No `config.xml` anywhere. After Step 5 creates the domain and it boots on FACTORY DEFAULTS: + +1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service + public key. This is the only step that needs the console; everything after it is scripted. +2. **Mint the API key** -- no GUI click: + ```bash + export OPNSENSE_SSH_KEY=... # the service private key + bash scripts/opnsense-bootstrap-apikey.sh <edge-lan-ip> <creds-out-file> + ``` +3. **Configure over REST** -- interfaces, DHCP, firewall: + ```bash + export OPNSENSE_API_HOST=<edge-lan-ip> OPNSENSE_API_CREDS=<creds-out-file> + bash scripts/opnsense-api.sh GET core/firmware/status # auth smoke test + bash scripts/opnsense-api.sh POST kea/dhcpv4/set_subnet/<uuid> '<json>' + bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}' + ``` + +**`WAN_IF`/`LAN_IF` are measured AFTER boot** (`ifconfig` over SSH), where they are knowable -- +which is why the chicken-and-egg problem this step used to agonize over simply does not exist +anymore. **GATE:** base image path and (if unblocked) config ISO path recorded as real host-local file paths. If the config.xml render step is blocked on diff --git a/runbooks/dc-dc-phase3-maas-enlist-deploy.md b/runbooks/dc-dc-phase3-maas-enlist-deploy.md index 854aeae..00cbc70 100644 --- a/runbooks/dc-dc-phase3-maas-enlist-deploy.md +++ b/runbooks/dc-dc-phase3-maas-enlist-deploy.md @@ -3,7 +3,7 @@ Bring the per-DC node VMs that Stage 3 (`docs/dc-dc-deployment-workflow.md` -- "Stage 3 -- Phase 2: OpenTofu builds each DC substrate") caused MAAS to DISCOVER through to a deployed OS, ready for Juju (Stage 5 / Phase 4). Run -this runbook ONCE PER DC: set `DC=dc1` or `DC=dc2` for the whole session and +this runbook ONCE PER DC: set `DC=vr1-dc0` or `DC=vr1-dc1` for the whole session and do not mix the two DCs' commands in one shell. This is the FIRST DC-DC runbook whose subject is MAAS itself in the multi-rack VR1 shape -- everything before it (Stage 0-3) either ratified decisions or built the substrate MAAS @@ -74,15 +74,15 @@ `scripts/phase-00-maas-standup.sh` now accept an opt-in `$DC` env var and call `lib_net_select_dc`/`lib_hosts_select_dc` immediately after sourcing (unset `$DC` is byte-for-byte unchanged VR0/DC0 behavior). See - `docs/changelog-20260709-maas-scripts-dc-param.md`. This means `DC=dc1 + `docs/changelog-20260709-maas-scripts-dc-param.md`. This means `DC=vr1-dc0 bash scripts/reenroll-hosts.sh` (etc.) is now a REAL invocation, not hypothetical -- but it does not by itself unblock anything: calling any - of the three with `DC=dc1` or `DC=dc2` today fails loud immediately at + of the three with `DC=vr1-dc0` or `DC=vr1-dc1` today fails loud immediately at `lib_hosts_select_dc` (reenroll-hosts.sh, carve-host-interfaces.sh -- no per-DC host inventory exists yet) or, for dc2, at `lib_net_select_dc` first (gap #4 below). The one case genuinely simplified is `phase-00-maas-standup.sh` against `dc1`: since D-101 makes dc1 inherit - dc0's plane literals unchanged, `DC=dc1 bash + dc0's plane literals unchanged, `DC=vr1-dc0 bash scripts/phase-00-maas-standup.sh` is now a real, correct, gated call -- see the updated Step 8. Parameterizing these scripts to accept REAL per-DC data (once it exists) required no further script change; the @@ -98,11 +98,11 @@ (`openstack0-3`) carry over to `$DC` -- discover however many nodes Stage 3 actually created, live, in Step 2 below. 4. **DC2 is additionally blocked at the network-selector layer.** - `lib_net_select_dc dc2` FAILS LOUD today (D-101's NetBox-literals open + `lib_net_select_dc vr1-dc1` FAILS LOUD today (D-101's NetBox-literals open item, tooling gap register #3 -- no CIDRs assigned yet for DC2). This - runbook cannot proceed past Step 1 for `DC=dc2` until that closes. `dc1` + runbook cannot proceed past Step 1 for `DC=vr1-dc1` until that closes. `dc1` is unblocked at this layer (D-101: DC1 inherits DC0's v4 layout - unchanged, so `lib_net_select_dc dc1` is a documented no-op). + unchanged, so `lib_net_select_dc vr1-dc0` is a documented no-op). 5. **The per-DC artifact mirror (D-107) has no identified owning stage/module in this repo as surveyed for this runbook.** Stage 2's build list (Office1 headend) mentions GitBucket (the REPO mirror) and NetBox -- @@ -327,7 +327,7 @@ The VLAN-103 metal-internal stack is a REAL, already-established repo constant, safe to cite verbatim (`scripts/lib-net.sh`): `METAL_INTERNAL_VID="103"`, `METAL_INTERNAL_IFACE="br-internal"`, `METAL_INTERNAL_CIDR="10.12.12.0/22"` (the -last one is DC0/DC1's value only under `lib_net_select_dc dc1`'s no-op -- +last one is DC0/DC1's value only under `lib_net_select_dc vr1-dc0`'s no-op -- confirm the same call resolves the right CIDR for whichever `$DC` you are running against; it fails loud for `dc2` per gap #4). @@ -440,7 +440,7 @@ hardcoded `PLANES` table is ALSO dc1's real target, so ```bash -DC=dc1 bash scripts/phase-00-maas-standup.sh +DC=vr1-dc0 bash scripts/phase-00-maas-standup.sh ``` is now a real, correct, gated call for `dc1` -- run it directly instead of the manual space-read-and-compare below. For `dc2`, run it the same way and @@ -491,6 +491,16 @@ second DC; the two DCs' completions are independent (D-100: no shared control plane), so one can be done before the other. +> **CREDS-CONSOLIDATION CLOSE-OUT (SEC-009 enforcement -- non-negotiable).** Standing up a DC MINTS +> secrets on its VMs and on the jumphost -- MAAS API key, per-DC NetBox token (if any), node/edge SSH +> keys, OPNsense root creds, tailscale authkey, Juju/Vault material. **The moment each is minted, +> consolidate it into `~/$DC-creds/` (0700; files 0600) and add it to `creds-manifests/$DC.manifest` +> with its provenance (`local` or `<vm>:<path>`).** Do NOT defer this -- "later" is when secrets get +> lost (and after a compaction they are gone). This is the exact class that bit us: office1's sandbox +> NetBox token, minted on the VM at `/root/netbox-secrets/`, went un-consolidated until it was needed +> (SEC-009 addendum). Before declaring this DC done, `bash scripts/creds-audit.sh $DC` MUST report +> **CLEAN** (folder perms, every manifest entry present, no undeclared files, no home-root sprawl). + --- ## Next diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 5bd2aa2..81428a2 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -95,7 +95,7 @@ **CHECK -- jumphost / wherever this session runs juju against this DC** ```bash -export DC=dc1 # or dc2 -- explicit, per this repo's $DC convention (never omitted) +export DC=vr1-dc0 # or dc2 -- explicit, per this repo's $DC convention (never omitted) source scripts/lib-net.sh; lib_net_select_dc "$DC" source scripts/lib-hosts.sh; lib_hosts_select_dc "$DC" echo "PLANE_CIDRS: ${PLANE_CIDRS[*]}" @@ -103,7 +103,7 @@ ``` **GATE:** both selector calls exit 0 (for DC2, this REQUIRES Stage 4 having already populated real host data into `scripts/lib-hosts.sh` for dc2 -- if -`lib_hosts_select_dc dc2` still fails loud here, Stage 4 for DC2 is not +`lib_hosts_select_dc vr1-dc1` still fails loud here, Stage 4 for DC2 is not actually done yet, stop and go back). --- @@ -161,7 +161,7 @@ note either way), not silently default to reuse. - **DC2:** BLOCKED until its VIP band exists (see above). Once NetBox assigns DC2's real supernet and it's imported via `netbox/dc-dc-prefixes- - import.py --dc dc2` (DOCFIX-152), `bundle.yaml`'s VIP lines need editing + import.py --dc vr1-dc1` (DOCFIX-152), `bundle.yaml`'s VIP lines need editing to DC2's real `.50-.60`-equivalent band within DC2's own provider-public plane -- a genuine, reviewed bundle edit, not a copy-paste of DC1's values. - Both DCs: `bundle.yaml` explicitly ships "NO designate" per phase-01's own diff --git a/runbooks/dc-dc-phase5-dr-failover-drill.md b/runbooks/dc-dc-phase5-dr-failover-drill.md index 7d60f76..b85ce2f 100644 --- a/runbooks/dc-dc-phase5-dr-failover-drill.md +++ b/runbooks/dc-dc-phase5-dr-failover-drill.md @@ -113,7 +113,7 @@ **CHECK -- both DCs independently, from wherever this session runs** ```bash source scripts/lib-net.sh -lib_net_select_dc "dc1" # explicit, once -- gap #1 / DOCFIX-151 call-site wiring +lib_net_select_dc "vr1-dc0" # explicit, once -- gap #1 / DOCFIX-151 call-site wiring # repeat the pair below for dc2 in its own shell/session context; do not assume dc2's # values from dc1 -- lib_net_select_dc fails loud for dc2 until its literals are real bash scripts/cloud-assert.sh diff --git a/runbooks/dc-dc-phase6-designate-cos-magnum.md b/runbooks/dc-dc-phase6-designate-cos-magnum.md index 53541fa..b63e6cb 100644 --- a/runbooks/dc-dc-phase6-designate-cos-magnum.md +++ b/runbooks/dc-dc-phase6-designate-cos-magnum.md @@ -121,7 +121,7 @@ ```bash # Set once per DC-pass through this runbook -- MEASURE from Stage 4's as-built # record, never guess a naming scheme: -# DC=dc1|dc2 +# DC=vr1-dc0|dc2 # DC_CONTROLLER=<this DC's own Juju controller name, from Stage 4> # DC_MODEL=<this DC's OpenStack model name, from Stage 4 -- may or may not be # literally "openstack"; VR0/DC0 used "openstack" but VR1 per-DC diff --git a/scripts/carve-host-interfaces.sh b/scripts/carve-host-interfaces.sh index 321fde5..b3ef7b0 100644 --- a/scripts/carve-host-interfaces.sh +++ b/scripts/carve-host-interfaces.sh @@ -27,7 +27,7 @@ # VLAN object id by CIDR. Idempotent: skips a bridge/vlan/link that already exists. # Requires the host to be Ready (link-subnet/update are rejected on Deployed). # -# DC selector (opt-in, DOCFIX-166): set DC=dc0|dc1|dc2 to call +# DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc/lib_hosts_select_dc explicitly. Unset (default) == DC0's # real plane scheme + enrolled hosts, unchanged. dc1 no-ops at the network # layer but FAILS at the host layer (no per-DC HOST_OCTET/HOST_BOOT_MAC yet); @@ -49,7 +49,7 @@ # dc-dc-phase3 runbook's Step 1: an explicit $DC env var, never an inferred # default. Unset/empty $DC changes NOTHING -- this script runs exactly as it # always has, implicitly against DC0/VR0's real plane scheme and enrolled -# hosts (backward compatible by construction). Set DC=dc0|dc1|dc2 to select +# hosts (backward compatible by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select (D-119) # explicitly; the selectors' own fail-loud behavior becomes this script's own # exit code via set -e. No second validation layer is added. DC="${DC:-}" diff --git a/scripts/creds-audit.sh b/scripts/creds-audit.sh new file mode 100755 index 0000000..009f030 --- /dev/null +++ b/scripts/creds-audit.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# scripts/creds-audit.sh -- per-site credential-consolidation audit (SEC-009 enforcement). +# +# Verifies a site's consolidated creds folder (~/<site>-creds/) holds EXACTLY the +# secrets its manifest requires -- each present, non-empty, and with the right mode -- +# and that no sensitive file is sprawled loose in the home dir OUTSIDE a *-creds/ +# folder. It reads NO secret values: only existence, mode, size, and declared +# provenance (from creds-manifests/<site>.manifest). +# +# WHY THIS EXISTS. SEC-009 established the "ALL site secrets live in ~/<site>-creds/" +# convention, but nothing ENFORCED it, and a secret MINTED ON A SITE VM +# (office1-netbox:/root/netbox-secrets/api.token) was never pulled into the folder -- +# it went unnoticed until the fidelity re-check needed it. This audit is the forcing +# check: run it at the END of every site standup and periodically. For the DCs, run +# `creds-audit.sh vr1-dc0` / `vr1-dc1` at standup close so the same gap cannot recur. +# +# Usage: +# bash scripts/creds-audit.sh <site> # e.g. vr1-office1 +# Env overrides (used by the harness; not for normal operation): +# CREDS_ROOT parent dir of <site>-creds (default: $HOME) +# MANIFEST_DIR dir holding <site>.manifest (default: <repo>/creds-manifests) +set -uo pipefail + +SITE="${1:-}" +[ -n "$SITE" ] || { echo "usage: creds-audit.sh <site> (e.g. vr1-office1)"; exit 2; } + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/.." && pwd)" +CREDS_ROOT="${CREDS_ROOT:-$HOME}" +MANIFEST_DIR="${MANIFEST_DIR:-$REPO/creds-manifests}" +CREDS="$CREDS_ROOT/${SITE}-creds" +MANIFEST="$MANIFEST_DIR/${SITE}.manifest" + +F=0 +fail() { echo " [FAIL] $1"; F=$((F+1)); } +okln() { echo " [ok] $1"; } + +[ -f "$MANIFEST" ] || { echo "FAIL: no manifest at $MANIFEST"; exit 1; } +[ -d "$CREDS" ] || { echo "FAIL: no creds folder at $CREDS"; exit 1; } + +echo "=== creds-audit: $SITE ($CREDS) ===" + +# 1. folder must be 0700 (owner-only) +fmode=$(stat -c '%a' "$CREDS") +[ "$fmode" = 700 ] && okln "folder mode 0700" || fail "folder mode $fmode (want 0700)" + +# 2. every manifest entry present, right mode, non-empty +declare -A expected=() +while read -r fname fmode2 fsrc; do + case "$fname" in ''|\#*) continue;; esac + expected["$fname"]=1 + path="$CREDS/$fname" + if [ ! -e "$path" ]; then + fail "MISSING: $fname (provenance: ${fsrc:-?})" + continue + fi + amode=$(stat -c '%a' "$path") + [ "$amode" = "$fmode2" ] || fail "$fname mode $amode (want $fmode2)" + [ -s "$path" ] || fail "$fname is EMPTY" +done < "$MANIFEST" +[ "$F" = 0 ] && okln "all ${#expected[@]} manifest entries present, correct mode, non-empty" + +# 3. no UNDECLARED file in the folder (an undeclared secret sitting here is spread) +for path in "$CREDS"/* "$CREDS"/.[!.]*; do + [ -e "$path" ] || continue + b=$(basename "$path") + [ -n "${expected[$b]:-}" ] || fail "UNDECLARED file in folder: $b (add it to the manifest, or remove it)" +done + +# 4. sprawl: sensitive-looking files loose in CREDS_ROOT, OUTSIDE any *-creds/ folder +shopt -s nullglob +for path in "$CREDS_ROOT"/*.env "$CREDS_ROOT"/.*.env "$CREDS_ROOT"/*appcred* \ + "$CREDS_ROOT"/*-cred*.txt "$CREDS_ROOT"/*authkey* "$CREDS_ROOT"/*.token; do + [ -f "$path" ] || continue + fail "SPRAWL: sensitive file loose in $CREDS_ROOT: $(basename "$path") (belongs in a *-creds/ folder)" +done +shopt -u nullglob + +echo +if [ "$F" = 0 ]; then echo "creds-audit $SITE: CLEAN"; exit 0; fi +echo "creds-audit $SITE: $F problem(s) -- consolidate/fix before proceeding"; exit 1 diff --git a/scripts/dc-dc-ceph-disk-budget.sh b/scripts/dc-dc-ceph-disk-budget.sh index bd0809d..deda4a2 100644 --- a/scripts/dc-dc-ceph-disk-budget.sh +++ b/scripts/dc-dc-ceph-disk-budget.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# scripts/dc-dc-ceph-disk-budget.sh --total-disk <SIZE> --dc1-nodes <N> -# --dc1-per-node-osd <SIZE> --dc2-nodes <N> --dc2-per-node-osd <SIZE> +# scripts/dc-dc-ceph-disk-budget.sh --total-disk <SIZE> --vr1-dc0-nodes <N> +# --dc1-per-node-osd <SIZE> --vr1-dc1-nodes <N> --dc2-per-node-osd <SIZE> # --backup-overhead-fraction <FRACTION> # # Phase-0 gate calculator (tooling gap register item #7): turns @@ -52,21 +52,21 @@ # # Usage: # dc-dc-ceph-disk-budget.sh --total-disk <SIZE> \ -# --dc1-nodes <N> --dc1-per-node-osd <SIZE> \ -# --dc2-nodes <N> --dc2-per-node-osd <SIZE> \ +# --vr1-dc0-nodes <N> --dc1-per-node-osd <SIZE> \ +# --vr1-dc1-nodes <N> --dc2-per-node-osd <SIZE> \ # --backup-overhead-fraction <FRACTION> # # --total-disk SIZE REQUIRED. Measured total disk on the # vcloud host (Step 2's `df`/`lsblk` # output). No default. -# --dc1-nodes N REQUIRED. Measured/planned Ceph OSD node +# --vr1-dc0-nodes N REQUIRED. Measured/planned Ceph OSD node # count, DC1. No default. # --dc1-per-node-osd SIZE REQUIRED. Measured/planned per-node raw # OSD disk footprint, DC1 (already # reflecting DC1's own size=3 replication # as it would actually consume disk). No # default. -# --dc2-nodes N REQUIRED. Same, DC2. No default. +# --vr1-dc1-nodes N REQUIRED. Same, DC2. No default. # --dc2-per-node-osd SIZE REQUIRED. Same, DC2. No default. # --backup-overhead-fraction F REQUIRED. Decimal fraction (e.g. 0.4 for # 40%) of the combined DC1+DC2 Ceph total @@ -90,18 +90,18 @@ usage() { sed -n '2,85p' "$HERE/$(basename "${BASH_SOURCE[0]}")" | sed 's/^# \{0,1\}//'; } -TOTAL_DISK=""; DC1_NODES=""; DC1_PER_NODE=""; DC2_NODES=""; DC2_PER_NODE=""; OVERHEAD_FRAC="" +TOTAL_DISK=""; VR1_DC0_NODES=""; DC1_PER_NODE=""; VR1_DC1_NODES=""; DC2_PER_NODE=""; OVERHEAD_FRAC="" while [ $# -gt 0 ]; do case "$1" in --total-disk) TOTAL_DISK="${2:-}"; shift 2 ;; --total-disk=*) TOTAL_DISK="${1#*=}"; shift ;; - --dc1-nodes) DC1_NODES="${2:-}"; shift 2 ;; - --dc1-nodes=*) DC1_NODES="${1#*=}"; shift ;; + --vr1-dc0-nodes) VR1_DC0_NODES="${2:-}"; shift 2 ;; + --vr1-dc0-nodes=*) VR1_DC0_NODES="${1#*=}"; shift ;; --dc1-per-node-osd) DC1_PER_NODE="${2:-}"; shift 2 ;; --dc1-per-node-osd=*) DC1_PER_NODE="${1#*=}"; shift ;; - --dc2-nodes) DC2_NODES="${2:-}"; shift 2 ;; - --dc2-nodes=*) DC2_NODES="${1#*=}"; shift ;; + --vr1-dc1-nodes) VR1_DC1_NODES="${2:-}"; shift 2 ;; + --vr1-dc1-nodes=*) VR1_DC1_NODES="${1#*=}"; shift ;; --dc2-per-node-osd) DC2_PER_NODE="${2:-}"; shift 2 ;; --dc2-per-node-osd=*) DC2_PER_NODE="${1#*=}"; shift ;; --backup-overhead-fraction) OVERHEAD_FRAC="${2:-}"; shift 2 ;; @@ -113,8 +113,8 @@ # --- validation: every input REQUIRED, no defaults (hard rule 2) ------------ missing=0 -for pair in "--total-disk:$TOTAL_DISK" "--dc1-nodes:$DC1_NODES" \ - "--dc1-per-node-osd:$DC1_PER_NODE" "--dc2-nodes:$DC2_NODES" \ +for pair in "--total-disk:$TOTAL_DISK" "--vr1-dc0-nodes:$VR1_DC0_NODES" \ + "--dc1-per-node-osd:$DC1_PER_NODE" "--vr1-dc1-nodes:$VR1_DC1_NODES" \ "--dc2-per-node-osd:$DC2_PER_NODE" \ "--backup-overhead-fraction:$OVERHEAD_FRAC"; do name="${pair%%:*}"; val="${pair#*:}" @@ -126,8 +126,8 @@ [ "$missing" -eq 1 ] && exit 2 is_pos_int() { [[ "$1" =~ ^[0-9]+$ ]] && [ "$1" -gt 0 ]; } -if ! is_pos_int "$DC1_NODES"; then echo "FAIL: --dc1-nodes must be a positive integer; got '$DC1_NODES'" >&2; exit 2; fi -if ! is_pos_int "$DC2_NODES"; then echo "FAIL: --dc2-nodes must be a positive integer; got '$DC2_NODES'" >&2; exit 2; fi +if ! is_pos_int "$VR1_DC0_NODES"; then echo "FAIL: --vr1-dc0-nodes must be a positive integer; got '$VR1_DC0_NODES'" >&2; exit 2; fi +if ! is_pos_int "$VR1_DC1_NODES"; then echo "FAIL: --vr1-dc1-nodes must be a positive integer; got '$VR1_DC1_NODES'" >&2; exit 2; fi # fraction: decimal >= 0 (e.g. 0, 0.4, 1, 1.25) if ! [[ "$OVERHEAD_FRAC" =~ ^[0-9]+(\.[0-9]+)?$ ]]; then @@ -166,8 +166,8 @@ printf '%d.%02d' "$((tib_scaled / 100))" "$((tib_scaled % 100))" } -DC1_TOTAL_B=$((DC1_NODES * DC1_PER_NODE_B)) -DC2_TOTAL_B=$((DC2_NODES * DC2_PER_NODE_B)) +DC1_TOTAL_B=$((VR1_DC0_NODES * DC1_PER_NODE_B)) +DC2_TOTAL_B=$((VR1_DC1_NODES * DC2_PER_NODE_B)) CEPH_TOTAL_B=$((DC1_TOTAL_B + DC2_TOTAL_B)) # overhead = ceph_total * fraction, computed in fixed-point (avoid bash's @@ -179,8 +179,8 @@ REQUIRED_B=$((CEPH_TOTAL_B + OVERHEAD_B)) echo "=== dc-dc-ceph-disk-budget: Section 3 / D-101 Ceph disk-budget gate ===" -echo " DC1: $DC1_NODES node(s) x $DC1_PER_NODE per-node OSD footprint = $(to_tib_str "$DC1_TOTAL_B") TiB" -echo " DC2: $DC2_NODES node(s) x $DC2_PER_NODE per-node OSD footprint = $(to_tib_str "$DC2_TOTAL_B") TiB" +echo " DC1: $VR1_DC0_NODES node(s) x $DC1_PER_NODE per-node OSD footprint = $(to_tib_str "$DC1_TOTAL_B") TiB" +echo " DC2: $VR1_DC1_NODES node(s) x $DC2_PER_NODE per-node OSD footprint = $(to_tib_str "$DC2_TOTAL_B") TiB" echo " Combined DC1+DC2 Ceph (size=3, as measured/planned): $(to_tib_str "$CEPH_TOTAL_B") TiB" echo " Backup/mirror/image overhead (--backup-overhead-fraction $OVERHEAD_FRAC of combined Ceph): $(to_tib_str "$OVERHEAD_B") TiB" echo " Required total (size=3 + overhead): $(to_tib_str "$REQUIRED_B") TiB" diff --git a/scripts/dc-dc-dr-drill.sh b/scripts/dc-dc-dr-drill.sh index e3dc4fd..3a8b111 100644 --- a/scripts/dc-dc-dr-drill.sh +++ b/scripts/dc-dc-dr-drill.sh @@ -54,11 +54,11 @@ # assigned network literals, so no real DC2 Ceph cluster exists to target). # # Usage: -# dc-dc-dr-drill.sh failover --dc <dc1|dc2> --model M --unit U --pool NAME \ +# dc-dc-dr-drill.sh failover --dc <vr1-dc0|vr1-dc1> --model M --unit U --pool NAME \ # --confirmed-down [--reregister-image ID ...] [--apply] # dc-dc-dr-drill.sh failback --pool NAME \ -# --recovering-dc <dc1|dc2> --recovering-model M --recovering-unit U \ -# --primary-dc <dc1|dc2> --primary-model M --primary-unit U \ +# --recovering-dc <vr1-dc0|vr1-dc1> --recovering-model M --recovering-unit U \ +# --primary-dc <vr1-dc0|vr1-dc1> --primary-model M --primary-unit U \ # [--skip-11-4] [--apply] [--no-prompt] # dc-dc-dr-drill.sh --help # @@ -71,7 +71,7 @@ usage() { cat <<'EOF' usage: - dc-dc-dr-drill.sh failover --dc dc1|dc2 --model M --unit U --pool NAME \ + dc-dc-dr-drill.sh failover --dc vr1-dc0|vr1-dc1 --model M --unit U --pool NAME \ --confirmed-down [--reregister-image ID ...] [--apply] (--dc/--model/--unit identify the DC being PROMOTED -- the surviving side. --confirmed-down is a required acknowledgement that runbook Step @@ -79,8 +79,8 @@ already been done; this script does not perform that check itself.) dc-dc-dr-drill.sh failback --pool NAME \ - --recovering-dc dc1|dc2 --recovering-model M --recovering-unit U \ - --primary-dc dc1|dc2 --primary-model M --primary-unit U \ + --recovering-dc vr1-dc0|vr1-dc1 --recovering-model M --recovering-unit U \ + --primary-dc vr1-dc0|vr1-dc1 --primary-model M --primary-unit U \ [--skip-11-4] [--apply] [--no-prompt] (recovering = the DC coming back up; primary = the DC currently serving as primary since the failover. Order is hard-coded: demote @@ -147,8 +147,8 @@ esac done - [ -n "$DC" ] || { echo "FAIL: --dc is required (dc1|dc2 -- the DC being promoted)" >&2; exit 1; } - case "$DC" in dc1|dc2) ;; *) echo "FAIL: --dc must be dc1 or dc2 (got '$DC')" >&2; exit 1 ;; esac + [ -n "$DC" ] || { echo "FAIL: --dc is required (vr1-dc0|vr1-dc1 -- the DC being promoted)" >&2; exit 1; } + case "$DC" in vr1-dc0|vr1-dc1) ;; *) echo "FAIL: --dc must be vr1-dc0 or vr1-dc1 (got '$DC'). D-119: bare dcN is RETIRED -- it was ambiguous across regions." >&2; exit 1 ;; esac [ -n "$UNIT" ] || { echo "FAIL: --unit is required" >&2; exit 1; } [ -n "$POOL" ] || { echo "FAIL: --pool is required (Glance pool name -- confirm, never assume 'glance')" >&2; exit 1; } @@ -221,10 +221,10 @@ done [ -n "$POOL" ] || { echo "FAIL: --pool is required" >&2; exit 1; } - [ -n "$REC_DC" ] || { echo "FAIL: --recovering-dc is required (dc1|dc2)" >&2; exit 1; } - [ -n "$PRI_DC" ] || { echo "FAIL: --primary-dc is required (dc1|dc2)" >&2; exit 1; } - case "$REC_DC" in dc1|dc2) ;; *) echo "FAIL: --recovering-dc must be dc1 or dc2 (got '$REC_DC')" >&2; exit 1 ;; esac - case "$PRI_DC" in dc1|dc2) ;; *) echo "FAIL: --primary-dc must be dc1 or dc2 (got '$PRI_DC')" >&2; exit 1 ;; esac + [ -n "$REC_DC" ] || { echo "FAIL: --recovering-dc is required (vr1-dc0|vr1-dc1)" >&2; exit 1; } + [ -n "$PRI_DC" ] || { echo "FAIL: --primary-dc is required (vr1-dc0|vr1-dc1)" >&2; exit 1; } + case "$REC_DC" in vr1-dc0|vr1-dc1) ;; *) echo "FAIL: --recovering-dc must be vr1-dc0 or vr1-dc1 (got '$REC_DC'). D-119: bare dcN is RETIRED." >&2; exit 1 ;; esac + case "$PRI_DC" in vr1-dc0|vr1-dc1) ;; *) echo "FAIL: --primary-dc must be vr1-dc0 or vr1-dc1 (got '$PRI_DC'). D-119: bare dcN is RETIRED." >&2; exit 1 ;; esac [ "$REC_DC" != "$PRI_DC" ] || { echo "FAIL: --recovering-dc and --primary-dc must differ (got '$REC_DC' twice)" >&2; exit 1; } [ -n "$REC_UNIT" ] || { echo "FAIL: --recovering-unit is required" >&2; exit 1; } [ -n "$PRI_UNIT" ] || { echo "FAIL: --primary-unit is required" >&2; exit 1; } diff --git a/scripts/dc-dc-radosgw-multisite.sh b/scripts/dc-dc-radosgw-multisite.sh index 815e36a..a66d18b 100644 --- a/scripts/dc-dc-radosgw-multisite.sh +++ b/scripts/dc-dc-radosgw-multisite.sh @@ -64,13 +64,13 @@ usage() { cat <<'EOF' usage: - dc-dc-radosgw-multisite.sh master-init --dc dc1|dc2 --model M --unit U \ + dc-dc-radosgw-multisite.sh master-init --dc vr1-dc0|vr1-dc1 --model M --unit U \ --realm NAME --zonegroup NAME --zone NAME --endpoint URL \ [--restart-action ACTION] [--apply] - dc-dc-radosgw-multisite.sh join-readonly --dc dc1|dc2 --model M --unit U \ + dc-dc-radosgw-multisite.sh join-readonly --dc vr1-dc0|vr1-dc1 --model M --unit U \ --zonegroup NAME --zone NAME --endpoint URL \ --access-key KEY --secret SECRET [--restart-action ACTION] [--apply] - dc-dc-radosgw-multisite.sh enable-two-way --dc dc1|dc2 --model M --unit U \ + dc-dc-radosgw-multisite.sh enable-two-way --dc vr1-dc0|vr1-dc1 --model M --unit U \ --zone NAME [--apply] dc-dc-radosgw-multisite.sh --help @@ -115,8 +115,8 @@ # subcommands (Step 4) -- enable-two-way (Step 8) only MODIFIES an existing # zone's read-only flag, so it does not need them; forcing them here would # make an operator supply irrelevant values just to satisfy the parser. -[ -n "$DC" ] || { echo "FAIL: --dc is required (dc1|dc2)" >&2; exit 1; } -case "$DC" in dc1|dc2) ;; *) echo "FAIL: --dc must be dc1 or dc2 (got '$DC')" >&2; exit 1 ;; esac +[ -n "$DC" ] || { echo "FAIL: --dc is required (vr1-dc0|vr1-dc1)" >&2; exit 1; } +case "$DC" in vr1-dc0|vr1-dc1) ;; *) echo "FAIL: --dc must be vr1-dc0 or vr1-dc1 (got '$DC'). D-119: bare dcN is RETIRED -- it was ambiguous across regions." >&2; exit 1 ;; esac [ -n "$UNIT" ] || { echo "FAIL: --unit is required (e.g. ceph-radosgw/0 -- measured this session, never assumed)" >&2; exit 1; } [ -n "$ZONE" ] || { echo "FAIL: --zone is required" >&2; exit 1; } diff --git a/scripts/dc-dc-rbd-mirror.sh b/scripts/dc-dc-rbd-mirror.sh index 2876abf..13433dd 100644 --- a/scripts/dc-dc-rbd-mirror.sh +++ b/scripts/dc-dc-rbd-mirror.sh @@ -61,9 +61,9 @@ usage() { cat <<'EOF' usage: - dc-dc-rbd-mirror.sh bootstrap-primary --dc dc1|dc2 --model M --unit U \ + dc-dc-rbd-mirror.sh bootstrap-primary --dc vr1-dc0|vr1-dc1 --model M --unit U \ --pool NAME --site-name NAME [--token-out /remote/path] [--apply] - dc-dc-rbd-mirror.sh bootstrap-secondary --dc dc1|dc2 --model M --unit U \ + dc-dc-rbd-mirror.sh bootstrap-secondary --dc vr1-dc0|vr1-dc1 --model M --unit U \ --pool NAME --site-name NAME --direction rx-only|rx-tx \ --token-in /remote/path [--apply] dc-dc-rbd-mirror.sh --help @@ -105,8 +105,8 @@ done # --- guard clauses: required args, no invented defaults --- -[ -n "$DC" ] || { echo "FAIL: --dc is required (dc1|dc2)" >&2; exit 1; } -case "$DC" in dc1|dc2) ;; *) echo "FAIL: --dc must be dc1 or dc2 (got '$DC')" >&2; exit 1 ;; esac +[ -n "$DC" ] || { echo "FAIL: --dc is required (vr1-dc0|vr1-dc1)" >&2; exit 1; } +case "$DC" in vr1-dc0|vr1-dc1) ;; *) echo "FAIL: --dc must be vr1-dc0 or vr1-dc1 (got '$DC'). D-119: bare dcN is RETIRED -- it was ambiguous across regions." >&2; exit 1 ;; esac [ -n "$UNIT" ] || { echo "FAIL: --unit is required (e.g. ceph-rbd-mirror/0 or ceph-mon/0 -- measured, never assumed)" >&2; exit 1; } [ -n "$POOL" ] || { echo "FAIL: --pool is required (confirm against 'ceph osd pool ls' + 'juju config glance rbd-pool' -- do not assume 'glance', per the runbook)" >&2; exit 1; } [ -n "$SITE_NAME" ] || { echo "FAIL: --site-name is required" >&2; exit 1; } diff --git a/scripts/lib-hosts.sh b/scripts/lib-hosts.sh index 07ac646..46a25b6 100644 --- a/scripts/lib-hosts.sh +++ b/scripts/lib-hosts.sh @@ -59,28 +59,37 @@ # --- $DC selector convention (2026-07-09, tooling gap register #1) --- # -# Mirrors lib-net.sh's lib_net_select_dc -- but the answer here is honestly -# different, not a parallel no-op: HOSTS / HOST_OCTET / HOST_BOOT_MAC above -# are VR0/DC0's real, measured, enrolled hosts. DC1's and DC2's node VMs are -# created by opentofu/modules/node-vm (Stage 3 / buildout-design Phase 2), -# which has not run -- their hostnames, boot MACs, and octets do not exist -# yet, and VIRSH_POWER_ADDRESS above is DC0's own vcloud host connection -# (the buildout design's VR1 vcloud host is a distinct, not-yet-measured -# endpoint -- see opentofu's root `libvirt_uri` variable, also unset). -# Selecting dc1 or dc2 FAILS LOUDLY rather than inventing placeholder host -# data or silently reusing DC0's real hosts for a different DC. +# Mirrors lib-net.sh's lib_net_select_dc -- SAME region-qualified selectors +# (D-119), but the answer here is honestly different, not a parallel no-op: +# HOSTS / HOST_OCTET / HOST_BOOT_MAC above are VR0 DC0's real, measured, +# enrolled hosts. VR1 DC0's and VR1 DC1's node VMs are created by +# opentofu/modules/node-vm (Stage 3 / buildout-design Phase 2), which has not +# run -- their hostnames, boot MACs, and octets do not exist yet, and +# VIRSH_POWER_ADDRESS above is VR0 DC0's own vcloud host connection (the +# buildout design's VR1 vcloud host is a distinct, not-yet-measured endpoint -- +# see opentofu's root `libvirt_uri` variable, also unset). Selecting either VR1 +# DC FAILS LOUDLY rather than inventing placeholder host data or silently +# reusing VR0 DC0's real hosts for a different DC. +# +# DELIBERATE ASYMMETRY WITH lib-net.sh: `vr1-dc0` is a NO-OP there (D-101 says +# it inherits VR0 DC0's v4 layout) but FAILS here (it has no enrolled hosts). +# That is intentional and is asserted by tests/dc-selector. lib_hosts_select_dc() { - local dc="${1:?usage: lib_hosts_select_dc <dc0|dc1|dc2>}" + local dc="${1:?usage: lib_hosts_select_dc <vr0-dc0|vr1-dc0|vr1-dc1>}" case "$dc" in - dc0) + vr0-dc0) : # already the sourced defaults above -- explicit no-op, not a guess. ;; - dc1|dc2) + vr1-dc0|vr1-dc1) echo "FAIL: $dc has no enrolled hosts yet -- Phase 2/3 node-VM creation (opentofu/modules/node-vm) and MAAS enrollment haven't run for it. Do not invent HOSTS/HOST_OCTET/HOST_BOOT_MAC/VIRSH_POWER_ADDRESS values; populate this file with a real \$dc block once those hosts are actually enrolled and measured." >&2 return 1 ;; + dc0|dc1|dc2) + echo "FAIL: bare '$dc' is RETIRED (D-119). It was AMBIGUOUS ACROSS REGIONS: 'dc0' meant VR0's live DC0 here, but VR1's FIRST DC in the NetBox importer. Use the region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" >&2 + return 1 + ;; *) - echo "FAIL: unknown DC '$dc' (expected dc0, dc1, or dc2)" >&2 + echo "FAIL: unknown DC '$dc' (expected vr0-dc0, vr1-dc0, or vr1-dc1)" >&2 return 1 ;; esac diff --git a/scripts/lib-net.sh b/scripts/lib-net.sh index 5db72bd..ec326a6 100644 --- a/scripts/lib-net.sh +++ b/scripts/lib-net.sh @@ -83,26 +83,40 @@ # the $REPO convention (DOCFIX-139/141): one explicit, measured selector, # never an inferred default. # -# dc0 and dc1 are the SAME values: D-101 (ADOPTED 2026-07-09) rules that -# DC1 inherits DC0's v4 layout UNCHANGED, so there is deliberately no -# second copy of these literals to drift out of sync with the ones above -- -# selecting either is a no-op against what sourcing this file already set. -# dc2 has NO assigned values yet (D-101's NetBox-literals open item, gap #3 -# in the tooling gap register) -- selecting it FAILS LOUDLY. It does not -# silently fall back to dc0/dc1's values or invent one; that silent-reuse -# failure mode is exactly what this convention exists to prevent. +# SELECTORS ARE REGION-QUALIFIED (D-119, ADOPTED 2026-07-14). The selector +# string IS the NetBox apex site slug -- vr0-dc0 / vr1-dc0 / vr1-dc1 -- so the +# shell, the OpenTofu modules and the IPAM authority all speak ONE namespace. +# +# WHY, because a bare `dcN` looks harmless: before D-119, `dc0` meant VR0's +# LIVE testcloud HERE, but VR1's FIRST DC in the NetBox importer. One string, +# two clouds, one of them in production. A bare dcN is therefore REJECTED -- +# accepting it "for compatibility" would preserve exactly the ambiguity D-119 +# exists to delete. lib_net_select_dc() { - local dc="${1:?usage: lib_net_select_dc <dc0|dc1|dc2>}" + local dc="${1:?usage: lib_net_select_dc <vr0-dc0|vr1-dc0|vr1-dc1>}" case "$dc" in - dc0|dc1) - : # already the sourced defaults above -- explicit no-op, not a guess. + vr0-dc0) + : # VR0's DC0 -- the LIVE testcloud. The literals sourced above ARE its + # real, measured values. Explicit no-op, not a guess. ;; - dc2) - echo "FAIL: dc2 has no assigned network literals yet (D-101 NetBox-literals open item, tooling gap register #3) -- do not select dc2 until NetBox assigns real CIDRs for it" >&2 + vr1-dc0) + : # VR1's FIRST DC. Same values as vr0-dc0, but for a DIFFERENT REASON: + # D-101 rules that VR1's first DC INHERITS VR0 DC0's v4 layout UNCHANGED. + # This is a SEPARATE case arm ON PURPOSE. The day D-101 stops holding, + # this arm diverges and vr0-dc0's must not. Folding the two arms together + # would encode the coincidence and lose the reason -- and the reason is + # the thing that changes (VR1 DC0 gains its own v6 per the family matrix). + ;; + vr1-dc1) + echo "FAIL: vr1-dc1 (VR1's SECOND DC) has no assigned network literals yet (D-101 NetBox-literals open item, tooling gap register #3) -- do not select it until NetBox assigns real CIDRs for it" >&2 + return 1 + ;; + dc0|dc1|dc2) + echo "FAIL: bare '$dc' is RETIRED (D-119). It was AMBIGUOUS ACROSS REGIONS: 'dc0' meant VR0's live DC0 here, but VR1's FIRST DC in the NetBox importer. Use the region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" >&2 return 1 ;; *) - echo "FAIL: unknown DC '$dc' (expected dc0, dc1, or dc2)" >&2 + echo "FAIL: unknown DC '$dc' (expected vr0-dc0, vr1-dc0, or vr1-dc1)" >&2 return 1 ;; esac diff --git a/scripts/opentofu-validate.sh b/scripts/opentofu-validate.sh index 63165f7..ecbee79 100644 --- a/scripts/opentofu-validate.sh +++ b/scripts/opentofu-validate.sh @@ -1,16 +1,33 @@ #!/usr/bin/env bash -# scripts/opentofu-validate.sh [tofu-dir] +# scripts/opentofu-validate.sh [--static-only] [tofu-dir] # -# Thin syntax/schema gate for opentofu/ (VR1 IaC, D-103). Read-only: runs -# `tofu fmt -check` + `tofu init -backend=false` + `tofu validate` against the -# root module. Does NOT plan or apply -- no provider connection, no libvirt -# reach required. This is the harness opentofu/README.md tells you to run -# before trusting anything under opentofu/, since it was authored in a -# session with no tofu binary available to self-check. +# Syntax/schema/semantic gate for opentofu/ (VR1 IaC, D-103). Read-only: runs the +# static semantic guards (S1, S2) + `tofu fmt -check` + `tofu init -backend=false` +# + `tofu validate` against the root module + S3, which validates EVERY module +# STANDALONE (root-only validation never parses uncalled modules -- that blind spot +# hid two broken modules until 2026-07-13; see S3's comment). Does NOT plan or apply -- no provider +# connection, no libvirt reach required. This is the harness opentofu/README.md +# tells you to run before trusting anything under opentofu/, since it was +# authored in a session with no tofu binary available to self-check. # -# Exit: 0 clean | 1 fmt/init/validate failed | 2 tofu binary not found. +# The S* guards catch libvirt-domain defects that `tofu validate` structurally +# CANNOT see (optional attributes whose ABSENCE is the bug). Both were paid for in +# blood by the 2026-07-12 Office1 OPNsense boot incident: +# S1 memory without memory_unit -> 1024x too little RAM (guest triple-faults) +# S2 domain without ACPI enabled -> FreeBSD panics; Linux loses clean shutdown +# +# --static-only runs ONLY the S* guards and exits (no tofu binary, no network +# needed). That is how the harness exercises them cheaply. +# +# Exit: 0 clean | 1 S*/fmt/init/validate failed | 2 tofu binary not found. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +MODE="full" +if [ "${1:-}" = "--static-only" ] || [ "${1:-}" = "--check-memory-unit" ]; then + MODE="static" # --check-memory-unit kept as a back-compat alias + shift +fi TOFU_DIR="${1:-$HERE/../opentofu}" if [ ! -d "$TOFU_DIR" ]; then @@ -18,12 +35,96 @@ exit 1 fi +# S1: every libvirt_domain that sets `memory` MUST also set `memory_unit`. +# +# `tofu validate` CANNOT catch this: memory_unit is an optional attribute, so +# omitting it is schema-valid. But on dmacvicar/libvirt >=0.9 a bare `memory` is +# interpreted in libvirt's DEFAULT unit (KiB), not MiB (the 0.8-era meaning), so +# `memory = 2048` silently yields a 2 MiB guest that triple-faults in its +# bootloader. That cost a full session on 2026-07-12 (Office1 OPNsense edge; see +# docs/incident-20260712-opnsense-edge-boot-triplefault.md). This is the guard +# that stops the next VM module from reintroducing it. +# +# Block-scan is safe because `tofu fmt -check` (below) enforces canonical layout: +# top-level resource braces at column 0, attributes at two-space indent. Run +# order puts S1 first so it also works with no tofu binary present. +s1_check() { + local out + out="$(find "$TOFU_DIR" -name '*.tf' -not -path '*/.terraform/*' -print0 2>/dev/null \ + | xargs -0 -r awk ' + /^resource "libvirt_domain"/ { inblk=1; mem=0; unit=0; rline=FNR; next } + inblk && /^ memory[[:space:]]*=/ { mem=1 } + inblk && /^ memory_unit[[:space:]]*=/ { unit=1 } + inblk && /^}/ { + if (mem && !unit) + printf " [FAIL] %s:%d sets memory with no memory_unit -- guest gets KiB, not MiB (1024x too little RAM)\n", FILENAME, rline + inblk=0 + } + ')" + if [ -n "$out" ]; then + echo "$out" + echo ' FIX: add memory_unit = "MiB" to each domain listed above.' + return 1 + fi + echo " [PASS] every libvirt_domain that sets memory also sets memory_unit" + return 0 +} + +# S2: every libvirt_domain MUST enable ACPI (`features = { acpi = true }`). +# +# Also invisible to `tofu validate` (the whole `features` block is optional). With no +# features block libvirt renders the machine `acpi=off`. A FreeBSD/OPNsense guest then +# PANICS at interrupt init ("running without device atpic requires a local APIC") and +# spins in ddb at 100% CPU while looking "running". A Linux guest boots but loses clean +# ACPI shutdown -- which breaks MAAS power control on the node VMs. Cost: the second half +# of the 2026-07-12 boot incident, found only once the S1 fix let the kernel get far +# enough to reach interrupt init. +s2_check() { + local out + out="$(find "$TOFU_DIR" -name '*.tf' -not -path '*/.terraform/*' -print0 2>/dev/null \ + | xargs -0 -r awk ' + /^resource "libvirt_domain"/ { inblk=1; acpi=0; rline=FNR; next } + inblk && /^[[:space:]]*acpi[[:space:]]*=[[:space:]]*true/ { acpi=1 } + inblk && /^}/ { + if (!acpi) + printf " [FAIL] %s:%d libvirt_domain does not enable ACPI -- libvirt renders acpi=off; FreeBSD guests panic, Linux guests lose clean shutdown\n", FILENAME, rline + inblk=0 + } + ')" + if [ -n "$out" ]; then + echo "$out" + echo ' FIX: add features = { acpi = true, apic = {} } to each domain listed above.' + return 1 + fi + echo " [PASS] every libvirt_domain enables ACPI" + return 0 +} + +echo "== S1 libvirt_domain memory_unit guard ==" +if s1_check; then + s1_rc=0 +else + s1_rc=1 +fi + +echo "== S2 libvirt_domain ACPI guard ==" +if s2_check; then + s2_rc=0 +else + s2_rc=1 +fi + +if [ "$MODE" = "static" ]; then + [ "$s1_rc" -eq 0 ] && [ "$s2_rc" -eq 0 ] && exit 0 + exit 1 +fi + if ! command -v tofu >/dev/null 2>&1; then echo "FAIL: tofu binary not found on PATH -- install OpenTofu, then re-run this script" exit 2 fi -fail=0 +fail=$(( s1_rc | s2_rc )) echo "== tofu fmt -check -recursive ==" if ! tofu fmt -check -recursive -diff "$TOFU_DIR"; then @@ -37,12 +138,53 @@ exit 1 fi -echo "== tofu validate ==" +echo "== tofu validate (root module) ==" if ! ( cd "$TOFU_DIR" && tofu validate ); then echo " [FAIL] validate failed" fail=1 fi +# S3: validate EVERY module standalone, not just the ones root happens to call. +# +# THE BLIND SPOT THIS CLOSES (DOCFIX-194, 2026-07-13): `tofu validate` on the root +# module only reaches modules the root actually INSTANTIATES. Modules that are not +# called -- or are called only from a commented-out block -- are NEVER PARSED. This +# gate printed a green "PASS" on 2026-07-13 while TWO modules were flat-out broken: +# +# node-vm libvirt_volume had a top-level `format = {...}` -- not a real +# attribute (it nests under `target`). Could never have applied. +# node-vm builds EVERY DC node. +# netem-link a destroy-time provisioner referenced var.* -- OpenTofu rejects +# that at INIT. The module could not even initialize. netem-link is +# the DR/latency mechanism for Stage 3 and the Stage 6 failover drill. +# +# Both would have detonated at first use, mid-deploy, in a stage where the runbook +# says "fix-forward is usually correct". A gate that reports green over unparsed code +# is worse than no gate: it manufactures false confidence. Hence this loop. +# +# Each module is validated in a TEMP COPY so the gate never writes .terraform/ or +# .terraform.lock.hcl into the repo tree (module-level lock files are noise; only the +# ROOT lock file is tracked, deliberately -- see .gitignore). +echo "== tofu validate (every module standalone -- the root-only blind spot) ==" +if [ -d "$TOFU_DIR/modules" ]; then + for moddir in "$TOFU_DIR"/modules/*/; do + mod="$(basename "$moddir")" + tmp="$(mktemp -d)" + cp -R "$moddir." "$tmp/" 2>/dev/null + rm -rf "$tmp/.terraform" "$tmp/.terraform.lock.hcl" + if out="$( cd "$tmp" && tofu init -backend=false -input=false -no-color 2>&1 && tofu validate -no-color 2>&1 )"; then + echo " [PASS] modules/$mod" + else + echo " [FAIL] modules/$mod" + printf '%s\n' "$out" | grep -E "^(Error| on | [0-9]+:)" | head -8 | sed 's/^/ /' + fail=1 + fi + rm -rf "$tmp" + done +else + echo " [WARN] no modules/ dir under $TOFU_DIR -- nothing to validate standalone" +fi + if [ "$fail" -eq 0 ]; then echo "PASS: opentofu-validate ($TOFU_DIR)" exit 0 diff --git a/scripts/opnsense-api.sh b/scripts/opnsense-api.sh new file mode 100755 index 0000000..4f9d9af --- /dev/null +++ b/scripts/opnsense-api.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# scripts/opnsense-api.sh [--dry-run] <METHOD> <api-path> [json-body] +# +# Thin REST client for the OPNsense API -- the D-113 (a2) config path. +# +# D-113 ADOPTED (a2): configuration of the edge moves OFF hand-authored +# config.xml and ONTO the documented REST API. Hand-editing the appliance's +# GUI-owned config.xml was the single root cause of DOCFIX-191 (no sshd/key), +# DOCFIX-192 (no console) and DOCFIX-193 (no DHCP), plus the 667-element +# migration self-heal we ended up depending on. None of those is expressible +# through the API, where sshd/DHCP/rules are typed resources with defaults. +# +# Credentials: read from a file (NEVER from argv, NEVER echoed). Mint the key +# in the GUI -- System > Access > Users > root > API keys > "+" -- which +# downloads a file containing: +# key=... +# secret=... +# Save it to $OPNSENSE_API_CREDS (default ~/vr1-office1-creds/opnsense-api.txt). +# The secret is passed to curl via --config on STDIN, so it never appears in +# argv (i.e. never in `ps`). Do not "improve" this into `curl -u`. +# +# TLS: the edge presents a SELF-SIGNED cert that OPNsense REGENERATES ON EVERY +# BOOT (measured 2026-07-13 -- "Created web GUI TLS certificate" appears in the +# config revision trail after each boot). Pinning it is therefore pointless, so +# we use --insecure. This is acceptable ONLY because the transport is a private +# lab LAN leg (virbr2). Do NOT copy this flag to anything tenant-facing. +# +# Exit: 0 ok | 1 usage/credential/config error | 2 HTTP error from the API. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +: "${OPNSENSE_API_CREDS:=$HOME/vr1-office1-creds/opnsense-api.txt}" + +DRY_RUN=0 +if [ "${1:-}" = "--dry-run" ]; then DRY_RUN=1; shift; fi + +METHOD="${1:-}" +APIPATH="${2:-}" +BODY="${3:-}" + +usage() { + echo "usage: opnsense-api.sh [--dry-run] <GET|POST> <api-path> [json-body]" >&2 + echo " e.g. opnsense-api.sh GET core/firmware/status" >&2 + echo " opnsense-api.sh POST kea/dhcpv4/set '{\"dhcpv4\":{...}}'" >&2 + exit 1 +} + +[ -n "$METHOD" ] && [ -n "$APIPATH" ] || usage +case "$METHOD" in + GET|POST) ;; + *) echo "FAIL: method must be GET or POST (got '$METHOD')" >&2; usage ;; +esac + +# The API host is NEVER inferred (hard rule 2): it is measured and passed in. +if [ -z "${OPNSENSE_API_HOST:-}" ]; then + echo "FAIL: \$OPNSENSE_API_HOST not set (e.g. 10.10.0.1) -- refusing to guess the edge address" >&2 + exit 1 +fi + +# Leading slashes on the path are a common typo; normalize rather than 404. +APIPATH="${APIPATH#/}" +APIPATH="${APIPATH#api/}" +URL="https://${OPNSENSE_API_HOST}/api/${APIPATH}" + +if [ "$DRY_RUN" = "1" ]; then + # Dry-run exists so the harness can prove URL construction WITHOUT credentials + # and without touching the network. It must never read the creds file. + echo "DRY-RUN ${METHOD} ${URL}" + [ -n "$BODY" ] && echo "DRY-RUN body: ${BODY}" + exit 0 +fi + +[ -f "$OPNSENSE_API_CREDS" ] || { + echo "FAIL: credentials file not found: $OPNSENSE_API_CREDS" >&2 + echo " Mint one in the GUI: System > Access > Users > root > API keys > '+'" >&2 + exit 1 +} + +# Parse key/secret WITHOUT echoing them. Tolerate OPNsense's downloaded format +# (key=...\nsecret=...), CRLF, and surrounding quotes. +API_KEY="$(sed -n 's/\r$//; s/^[[:space:]]*key[[:space:]]*=[[:space:]]*"\{0,1\}\([^"]*\)"\{0,1\}[[:space:]]*$/\1/p' "$OPNSENSE_API_CREDS" | head -1)" +API_SECRET="$(sed -n 's/\r$//; s/^[[:space:]]*secret[[:space:]]*=[[:space:]]*"\{0,1\}\([^"]*\)"\{0,1\}[[:space:]]*$/\1/p' "$OPNSENSE_API_CREDS" | head -1)" + +[ -n "$API_KEY" ] || { echo "FAIL: no 'key=' line in $OPNSENSE_API_CREDS" >&2; exit 1; } +[ -n "$API_SECRET" ] || { echo "FAIL: no 'secret=' line in $OPNSENSE_API_CREDS" >&2; exit 1; } + +# Secret goes to curl over STDIN via --config, so it is absent from argv/ps. +CURL_ARGS=(--silent --show-error --insecure --config - --request "$METHOD" --url "$URL" + --write-out '\n__HTTP_STATUS__:%{http_code}\n') +if [ -n "$BODY" ]; then + CURL_ARGS+=(--header 'Content-Type: application/json' --data "$BODY") +fi + +RESP="$(printf 'user = "%s:%s"\n' "$API_KEY" "$API_SECRET" | curl "${CURL_ARGS[@]}")" || { + echo "FAIL: curl could not reach $URL" >&2 + exit 2 +} + +STATUS="$(printf '%s' "$RESP" | sed -n 's/^__HTTP_STATUS__:\([0-9]*\)$/\1/p' | tail -1)" +printf '%s\n' "$RESP" | grep -v '^__HTTP_STATUS__:' + +case "$STATUS" in + 2*) exit 0 ;; + 401|403) + echo "FAIL: HTTP $STATUS -- the API rejected the credential." >&2 + echo " Note OPNsense API keys authenticate as their OWNING USER; a GUI password will NOT work here." >&2 + exit 2 ;; + "") + echo "FAIL: no HTTP status returned from $URL" >&2 + exit 2 ;; + *) + echo "FAIL: HTTP $STATUS from $URL" >&2 + exit 2 ;; +esac diff --git a/scripts/opnsense-bootstrap-apikey.sh b/scripts/opnsense-bootstrap-apikey.sh new file mode 100755 index 0000000..97e11e0 --- /dev/null +++ b/scripts/opnsense-bootstrap-apikey.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# scripts/opnsense-bootstrap-apikey.sh [--dry-run] <edge-host> <output-creds-file> +# +# Bootstraps an OPNsense REST API key on an edge that we can already reach over SSH, +# WITHOUT a GUI click and WITHOUT re-implementing any vendor crypto. This is the +# missing link that makes D-113(a2) edge provisioning fully automatable -- Stage 3's +# DC1/DC2 edges included. +# +# HOW: ships scripts/opnsense-mint-apikey.php to the edge and runs it there. That +# script calls OPNSENSE'S OWN MODEL (Auth\User -> apikeys->add()) -- the exact code +# path the GUI's "Create and download API key for this user" ticket button invokes. +# +# WHY NOT MINT THE KEY OFFLINE: OPNsense stores API secrets as crypt(secret,'$6$') +# (SHA-512, empty salt -- verified on the live 26.1 box). We COULD reproduce that and +# seed a key into a bootstrap config.xml. We deliberately DO NOT: it would couple our +# provisioning to a vendor hash format we would have to keep byte-compatible forever, +# and a mismatch fails SILENTLY -- keys that simply never authenticate. Calling the +# vendor's generator has no format to keep in sync. Same principle as D-113(a2) +# itself: call the interface, do not re-implement the internals. +# +# PREREQUISITE: SSH to the edge as root with a key (i.e. the D-112(c) console +# bootstrap has already run). This script does NOT solve first-contact; it solves +# "we have SSH, now get an API key so everything else can be REST". +# +# THE SECRET IS NEVER PRINTED. OPNsense hashes it on save, so creation is the ONLY +# moment it exists in cleartext. It goes straight to <output-creds-file> (0600) in the +# `key=`/`secret=` form scripts/opnsense-api.sh expects. Stdout gets lengths only. +# +# Exit: 0 minted | 1 usage/precondition error | 2 mint or retrieval failed. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MINTER="$HERE/opnsense-mint-apikey.php" + +DRY_RUN=0 +if [ "${1:-}" = "--dry-run" ]; then DRY_RUN=1; shift; fi + +EDGE="${1:-}" +OUT="${2:-}" +: "${OPNSENSE_API_USER:=root}" + +usage() { + echo "usage: opnsense-bootstrap-apikey.sh [--dry-run] <edge-host> <output-creds-file>" >&2 + echo " e.g. opnsense-bootstrap-apikey.sh 10.10.0.1 ~/vr1-office1-creds/opnsense-api.txt" >&2 + echo " env: OPNSENSE_SSH_KEY (required) OPNSENSE_API_USER (default: root)" >&2 + exit 1 +} + +[ -n "$EDGE" ] && [ -n "$OUT" ] || usage +[ -f "$MINTER" ] || { echo "FAIL: minter not found: $MINTER" >&2; exit 1; } + +# Refuse to clobber an existing credential file. Overwriting it would strand a live +# API key on the edge with no local copy -- unrecoverable, since the secret is hashed +# and can never be read back. Delete the old key first, deliberately. +if [ -e "$OUT" ]; then + echo "FAIL: $OUT already exists -- refusing to overwrite." >&2 + echo " An existing creds file means a live key. Overwriting would STRAND it on the" >&2 + echo " edge with no local copy (the secret is hashed there and can never be read" >&2 + echo " back). Delete the old key via the API first, then re-run." >&2 + exit 1 +fi + +if [ "$DRY_RUN" = "1" ]; then + echo "DRY-RUN would ship $MINTER -> ${OPNSENSE_API_USER}@${EDGE}:/tmp/" + echo "DRY-RUN would mint for user '${OPNSENSE_API_USER}' and write ${OUT}" + exit 0 +fi + +[ -n "${OPNSENSE_SSH_KEY:-}" ] || { echo "FAIL: \$OPNSENSE_SSH_KEY not set" >&2; exit 1; } + +SSH=(ssh -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15) +SCP=(scp -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15 -q) + +REMOTE_PHP="/tmp/opnsense-mint-apikey.$$.php" +REMOTE_OUT="/tmp/opnsense-apikey.$$.txt" +# Always wipe the remote copies, even on failure -- the remote file holds the +# cleartext secret until we have it. +cleanup() { "${SSH[@]}" "root@${EDGE}" "rm -f '$REMOTE_PHP' '$REMOTE_OUT'" >/dev/null 2>&1 || true; } +trap cleanup EXIT + +"${SCP[@]}" "$MINTER" "root@${EDGE}:${REMOTE_PHP}" || { echo "FAIL: could not ship the minter to $EDGE" >&2; exit 2; } + +# load_phalcon.php is resolved RELATIVE to /usr/local/opnsense/mvc -- must cd there. +"${SSH[@]}" "root@${EDGE}" \ + "cd /usr/local/opnsense/mvc && php '$REMOTE_PHP' '$OPNSENSE_API_USER' '$REMOTE_OUT'" \ + || { echo "FAIL: minting failed on $EDGE" >&2; exit 2; } + +umask 077 +"${SCP[@]}" "root@${EDGE}:${REMOTE_OUT}" "$OUT" || { echo "FAIL: could not retrieve the key" >&2; exit 2; } +chmod 600 "$OUT" + +# Verify shape without printing anything sensitive. +k=$(grep -c '^key=' "$OUT" || true) +s=$(grep -c '^secret=' "$OUT" || true) +if [ "$k" != "1" ] || [ "$s" != "1" ]; then + echo "FAIL: $OUT does not contain exactly one key= and one secret= line" >&2 + exit 2 +fi + +echo "OK: API key minted on ${EDGE} for '${OPNSENSE_API_USER}' -> ${OUT} (0600, secret not printed)" +echo " verify with: OPNSENSE_API_HOST=${EDGE} OPNSENSE_API_CREDS=${OUT} bash scripts/opnsense-api.sh GET core/firmware/status" diff --git a/scripts/opnsense-build-config-iso.sh b/scripts/opnsense-build-config-iso.sh deleted file mode 100644 index 27877bd..0000000 --- a/scripts/opnsense-build-config-iso.sh +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env bash -# scripts/opnsense-build-config-iso.sh <config.xml path> <output.iso path> -# -# Builds a plain ISO9660 image containing /conf/config.xml, for OPNsense's -# Configuration Importer (see opentofu/README.md's "OPNsense deployment -# research" section) -- ISO9660 support was added to the Importer -# specifically so a config ISO could be attached as a secondary CD-ROM, -# mechanically identical to how modules/cloudinit-vm attaches its NoCloud -# seed via a libvirt_volume + create.content.url pointing at a local path. -# -# NOTE (updated after a 2026-07-09 audit pass): the genisoimage/xorriso flags -# below (-V volume label, -J Joliet, -R Rock Ridge) are CONFIRMED standard, -# correct usage for building a plain cross-platform ISO9660 image from a -# directory -- this part is no longer a guess. What remains genuinely -# unverified is a DIFFERENT claim: whether OPNsense's Configuration Importer -# actually reads /conf/config.xml back off an ISO9660 volume built this way -# once booted for real (neither genisoimage/xorriso nor a real OPNsense VM -# was available this session to test that end-to-end). Confirm that -# specifically before relying on it operationally. -# -# Requires: genisoimage or xorriso. Read-only w.r.t. the repo; writes only -# to <output.iso path> and a self-cleaning temp dir. -# Exit: 0 built | 1 bad input | 2 required tool missing. -set -euo pipefail - -CONFIG_XML="${1:?usage: opnsense-build-config-iso.sh <config.xml> <output.iso>}" -OUT="${2:?usage: opnsense-build-config-iso.sh <config.xml> <output.iso>}" - -[ -r "$CONFIG_XML" ] || { echo "FAIL: cannot read $CONFIG_XML"; exit 1; } - -TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT -mkdir -p "$TMP/conf" -cp "$CONFIG_XML" "$TMP/conf/config.xml" - -if command -v genisoimage >/dev/null 2>&1; then - genisoimage -o "$OUT" -V "OPNSENSE_CFG" -J -R "$TMP" -elif command -v xorriso >/dev/null 2>&1; then - xorriso -as genisoimage -o "$OUT" -V "OPNSENSE_CFG" -J -R "$TMP" -else - echo "FAIL: genisoimage or xorriso required on PATH" - exit 2 -fi - -echo "PASS: built $OUT (contains /conf/config.xml)" diff --git a/scripts/opnsense-mint-apikey.php b/scripts/opnsense-mint-apikey.php new file mode 100644 index 0000000..c262b6d --- /dev/null +++ b/scripts/opnsense-mint-apikey.php @@ -0,0 +1,69 @@ +<?php +/* + * mint-apikey.php <username> <output-file> + * + * Mints an OPNsense API key using OPNSENSE'S OWN MODEL CODE -- the exact path the + * GUI's "Create and download API key for this user" ticket button invokes + * (Auth\Api\UserController::addApiKeyAction -> $user->apikeys->add()). + * + * WHY THIS AND NOT AN OFFLINE GENERATOR: minting the key ourselves would mean + * re-implementing OPNsense's $6$ SHA-512 crypt storage format and keeping it + * byte-compatible forever. A mismatch fails SILENTLY (keys that never + * authenticate). Calling the vendor's own generator has no format to keep in + * sync -- same principle as D-113(a2) itself: call the interface, don't + * re-implement the internals. + * + * The secret is written to <output-file> (0600) and NEVER printed: OPNsense + * stores it as crypt(secret,'$6$'), so creation is the only moment it exists in + * cleartext. Stdout gets lengths only. + * + * Run from /usr/local/opnsense/mvc (load_phalcon.php is relative to it). + */ +require_once('script/load_phalcon.php'); + +use OPNsense\Core\Config; +use OPNsense\Auth\User; + +if ($argc < 3) { + fwrite(STDERR, "usage: mint-apikey.php <username> <output-file>\n"); + exit(1); +} +$username = $argv[1]; +$outfile = $argv[2]; + +Config::getInstance()->lock(); + +$mdl = new User(); +$user = $mdl->getUserByName($username); +if ($user == null) { + fwrite(STDERR, "FAIL: no such user: {$username}\n"); + exit(1); +} + +$new = $user->apikeys->add(); +if (empty($new) || empty($new['key']) || empty($new['secret'])) { + fwrite(STDERR, "FAIL: apikeys->add() returned nothing\n"); + exit(1); +} + +$mdl->serializeToConfig(); +Config::getInstance()->save(); + +$payload = "key=" . $new['key'] . "\nsecret=" . $new['secret'] . "\n"; +$fh = fopen($outfile, 'w'); +if ($fh === false) { + fwrite(STDERR, "FAIL: cannot write {$outfile}\n"); + exit(1); +} +fwrite($fh, $payload); +fclose($fh); +chmod($outfile, 0600); + +printf( + "OK: minted for %s -- key=%d chars, secret=%d chars, wrote %d bytes to %s (0600, secret NOT printed)\n", + $username, + strlen($new['key']), + strlen($new['secret']), + strlen($payload), + $outfile +); diff --git a/scripts/opnsense-prep-image.sh b/scripts/opnsense-prep-image.sh index ee86cc4..a39753a 100644 --- a/scripts/opnsense-prep-image.sh +++ b/scripts/opnsense-prep-image.sh @@ -31,9 +31,17 @@ # noticing until a real deploy fails partway through. MIRROR_BASE="${OPNSENSE_MIRROR_BASE:?set OPNSENSE_MIRROR_BASE to a real OPNsense mirror base URL (see https://opnsense.org/download/) -- not hardcoded here, mirrors change}" -for bin in bunzip2 qemu-img; do - command -v "$bin" >/dev/null 2>&1 || { echo "FAIL: $bin required on PATH"; exit 2; } -done +command -v qemu-img >/dev/null 2>&1 || { echo "FAIL: qemu-img required on PATH"; exit 2; } +# bz2 decompressor: prefer bunzip2, else python3's stdlib bz2 module. python3 is +# a base prereq on this platform, so a missing bzip2 package is NOT a blocker +# (DOCFIX-184 -- first real run hit an absent bzip2). +if command -v bunzip2 >/dev/null 2>&1; then + BZ2_DECOMP=bunzip2 +elif command -v python3 >/dev/null 2>&1; then + BZ2_DECOMP=python3 +else + echo "FAIL: need bunzip2 or python3 to decompress the .bz2 image"; exit 2 +fi if command -v curl >/dev/null 2>&1; then FETCH=(curl -fSL -o) elif command -v wget >/dev/null 2>&1; then @@ -50,9 +58,13 @@ echo "== downloading: $URL ==" "${FETCH[@]}" "$IMG_BZ2" "$URL" -echo "== decompressing ==" -bunzip2 -k "$IMG_BZ2" +echo "== decompressing (via $BZ2_DECOMP) ==" IMG_RAW="${IMG_BZ2%.bz2}" +if [ "$BZ2_DECOMP" = "bunzip2" ]; then + bunzip2 -k "$IMG_BZ2" +else + python3 -c "import bz2,shutil,sys; shutil.copyfileobj(bz2.open(sys.argv[1],'rb'), open(sys.argv[2],'wb'))" "$IMG_BZ2" "$IMG_RAW" +fi echo "== converting raw -> qcow2 ==" qemu-img convert -f raw -O qcow2 "$IMG_RAW" "$OUT" diff --git a/scripts/opnsense-render-config.sh b/scripts/opnsense-render-config.sh deleted file mode 100644 index 2dd1ad8..0000000 --- a/scripts/opnsense-render-config.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -# scripts/opnsense-render-config.sh <output-config.xml> -# -# Renders opentofu/templates/opnsense-config.xml.tmpl into a real config.xml -# by substituting {{TOKEN}} markers (this repo's existing clientdocs -# convention, reused here) from required environment variables -- no -# invented defaults except NTP_UPSTREAM_SERVERS/NTP_PREFER_SERVER, which -# default to the SAME public pool OPNsense's own shipped config.xml.sample -# uses (a real, confirmed default, not an invention -- see -# opentofu/templates/README.md). Everything else fails loud if unset. -# -# Feed the result to scripts/opnsense-build-config-iso.sh next. -# Exit: 0 rendered | 1 required token/template missing. -set -euo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -TEMPLATE="$HERE/../opentofu/templates/opnsense-config.xml.tmpl" -OUT="${1:?usage: opnsense-render-config.sh <output-config.xml>}" - -: "${NTP_UPSTREAM_SERVERS:=0.opnsense.pool.ntp.org 1.opnsense.pool.ntp.org 2.opnsense.pool.ntp.org 3.opnsense.pool.ntp.org}" -: "${NTP_PREFER_SERVER:=0.opnsense.pool.ntp.org}" - -req() { # req VARNAME -- fail loud if unset, no inferred fallback - local name="$1" - [ -n "${!name:-}" ] || { echo "FAIL: \$$name not set -- see opentofu/templates/README.md"; exit 1; } -} - -REQUIRED_VARS=( - OPNSENSE_HOSTNAME DOMAIN ROOT_PASSWORD_HASH - WAN_IF WAN_IPADDR WAN_SUBNET_BITS WAN_GATEWAY - LAN_IF LAN_IPADDR LAN_SUBNET_BITS - MIRROR_SYNC_PROTOCOL MIRROR_UPSTREAM_NET MIRROR_SYNC_PORT -) -for v in "${REQUIRED_VARS[@]}"; do - req "$v" -done - -[ -r "$TEMPLATE" ] || { echo "FAIL: template not found: $TEMPLATE"; exit 1; } - -route_block() { # route_block <network-var-name> <gateway-var-name> <descr-var-name> - local net="${!1:-}" gw="${!2:-}" descr="${!3:-}" - if [ -z "$net" ]; then - return 0 # no route needed at this site -- emit nothing, not a placeholder - fi - [ -n "$gw" ] || { echo "FAIL: \$$1 is set but \$$2 is not"; exit 1; } - printf ' <route>\n <network>%s</network>\n <gateway>%s</gateway>\n <descr>%s</descr>\n <enabled>1</enabled>\n </route>\n' \ - "$net" "$gw" "${descr:-static route}" -} - -ROUTE1_BLOCK="$(route_block ROUTE1_NETWORK ROUTE1_GATEWAY ROUTE1_DESCR)" -ROUTE2_BLOCK="$(route_block ROUTE2_NETWORK ROUTE2_GATEWAY ROUTE2_DESCR)" - -content="$(cat "$TEMPLATE")" -for v in "${REQUIRED_VARS[@]}" NTP_UPSTREAM_SERVERS NTP_PREFER_SERVER; do - content="${content//"{{$v}}"/${!v}}" -done -content="${content//"{{ROUTE1_BLOCK}}"/$ROUTE1_BLOCK}" -content="${content//"{{ROUTE2_BLOCK}}"/$ROUTE2_BLOCK}" - -if grep -qE '\{\{[A-Z0-9_]+\}\}' <<<"$content"; then - echo "FAIL: unresolved {{TOKEN}} remains in rendered output -- template/script token lists are out of sync:" - grep -oE '\{\{[A-Z0-9_]+\}\}' <<<"$content" | sort -u - exit 1 -fi - -printf '%s' "$content" > "$OUT" -echo "PASS: rendered $OUT" diff --git a/scripts/opnsense-set-iface-v6.php b/scripts/opnsense-set-iface-v6.php new file mode 100644 index 0000000..dd92e8b --- /dev/null +++ b/scripts/opnsense-set-iface-v6.php @@ -0,0 +1,146 @@ +<?php +/* + * opnsense-set-iface-v6.php [--commit] <interface> <address> <prefixlen> + * + * e.g. opnsense-set-iface-v6.php --commit lan 2602:f3e2:f01:100::1 64 + * + * Sets a STATIC IPv6 address on an OPNsense base interface, using OPNSENSE'S OWN + * CONFIG OBJECT -- the same object the GUI's Interfaces > [LAN] page mutates when + * you press Save. + * + * WHY THIS EXISTS (D-113 AMENDMENT, 2026-07-14): D-113(a2)'s text claims + * "DHCP, firewall, interfaces: the REST API". The `interfaces` half is FALSE and + * was never measured. On OPNsense 26.1, Interfaces > [LAN] is served by the LEGACY + * page /interfaces.php?if=lan -- only the Devices and Neighbors sub-trees were ever + * migrated to MVC. There is NO REST endpoint that sets a base interface's ipaddrv6. + * Measured on the live Office1 edge, not inferred. + * + * WHY THIS IS NOT THE FORBIDDEN config.xml PUSH. The thing D-113 banned is a + * HAND-AUTHORED / RENDERED config.xml pushed wholesale over the live one -- the + * clobber path that would have wiped the API-managed Kea DHCP (667 migration- + * populated elements, 8 firewall rules). This does the OPPOSITE: + * + * - it never AUTHORS a config. It LOADS the edge's live config through the + * vendor's own Config singleton, mutates exactly TWO leaf fields, and lets + * the vendor's own code serialize it back. + * - it is byte-for-byte the code path interfaces.php runs on Save. + * - it asserts, before and after, that the interface COUNT is unchanged -- so a + * silent structural drop cannot pass unnoticed. + * + * Same principle as scripts/opnsense-mint-apikey.php, which mints an API key by + * calling Auth\User->apikeys->add() rather than re-implementing OPNsense's $6$ + * crypt format: CALL THE VENDOR'S INTERFACE, DO NOT RE-IMPLEMENT THE INTERNALS. + * + * DRY BY DEFAULT. Without --commit it prints the current -> desired delta and + * writes NOTHING. (D-117's lesson: a write-by-default IPAM/config tool is how an + * off-by-one lands in production unannounced.) + * + * This script does NOT apply the change to the running kernel -- saving config and + * reconfiguring the interface are separate steps, deliberately. The caller + * (scripts/opnsense-set-interface-v6.sh) runs `configctl interface reconfigure`. + * + * Run from /usr/local/opnsense/mvc (load_phalcon.php is resolved relative to it). + * + * Exit: 0 ok (or dry run) | 1 usage/validation error | 2 save failed. + */ +require_once('script/load_phalcon.php'); + +use OPNsense\Core\Config; + +$argvv = $argv; +array_shift($argvv); +$commit = false; +if (isset($argvv[0]) && $argvv[0] === '--commit') { + $commit = true; + array_shift($argvv); +} + +if (count($argvv) < 3) { + fwrite(STDERR, "usage: opnsense-set-iface-v6.php [--commit] <interface> <address> <prefixlen>\n"); + fwrite(STDERR, " e.g. opnsense-set-iface-v6.php --commit lan 2602:f3e2:f01:100::1 64\n"); + exit(1); +} +list($ifname, $address, $prefixlen) = $argvv; + +/* ---- validate BEFORE touching the config. Never write an unvalidated value. ---- */ +if (filter_var($address, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6) === false) { + fwrite(STDERR, "FAIL: '{$address}' is not a valid IPv6 address\n"); + exit(1); +} +if (!ctype_digit((string)$prefixlen) || (int)$prefixlen < 1 || (int)$prefixlen > 128) { + fwrite(STDERR, "FAIL: '{$prefixlen}' is not a valid IPv6 prefix length (1-128)\n"); + exit(1); +} +$prefixlen = (string)(int)$prefixlen; + +Config::getInstance()->lock(); +$cfg = Config::getInstance()->object(); + +if (!isset($cfg->interfaces)) { + fwrite(STDERR, "FAIL: config has no <interfaces> node -- refusing to guess\n"); + exit(1); +} +if (!isset($cfg->interfaces->$ifname)) { + $known = array(); + foreach ($cfg->interfaces->children() as $k => $v) { + $known[] = $k; + } + fwrite(STDERR, "FAIL: no such interface '{$ifname}'. Known: " . implode(', ', $known) . "\n"); + exit(1); +} + +/* Structural guard: the interface COUNT must not change. If serializing the + * vendor's own object ever drops a node, that is a silent corruption of a live + * router and it must fail loud, not be discovered later by a ping that stopped. */ +$iface_count_before = count($cfg->interfaces->children()); + +$node = $cfg->interfaces->$ifname; +$cur_ip = isset($node->ipaddrv6) ? (string)$node->ipaddrv6 : ''; +$cur_len = isset($node->subnetv6) ? (string)$node->subnetv6 : ''; + +printf("interface : %s (%s)\n", $ifname, isset($node->descr) ? (string)$node->descr : ''); +printf(" ipaddrv6 : %s -> %s\n", $cur_ip === '' ? '(none)' : $cur_ip, $address); +printf(" subnetv6 : %s -> %s\n", $cur_len === '' ? '(none)' : $cur_len, $prefixlen); +printf(" ipaddr (v4) : %s [UNTOUCHED]\n", isset($node->ipaddr) ? (string)$node->ipaddr : '(none)'); +printf(" interfaces in config: %d\n", $iface_count_before); + +if ($cur_ip === $address && $cur_len === $prefixlen) { + print("\nNO CHANGE -- already set to the desired value. Nothing to do.\n"); + exit(0); +} + +if (!$commit) { + print("\nDRY RUN -- nothing was written. Re-run with --commit to apply.\n"); + exit(0); +} + +$node->ipaddrv6 = $address; +$node->subnetv6 = $prefixlen; + +$iface_count_after = count($cfg->interfaces->children()); +if ($iface_count_after !== $iface_count_before) { + fwrite(STDERR, "FAIL: interface count changed {$iface_count_before} -> {$iface_count_after}. NOT SAVING.\n"); + exit(2); +} + +Config::getInstance()->save(); + +/* Read back from a FRESH load -- the service's own verdict, not our in-memory copy. */ +Config::getInstance()->forceReload(); +$verify = Config::getInstance()->object(); +$got_ip = (string)$verify->interfaces->$ifname->ipaddrv6; +$got_len = (string)$verify->interfaces->$ifname->subnetv6; +$got_cnt = count($verify->interfaces->children()); + +if ($got_ip !== $address || $got_len !== $prefixlen) { + fwrite(STDERR, "FAIL: read-back mismatch -- got {$got_ip}/{$got_len}, wanted {$address}/{$prefixlen}\n"); + exit(2); +} +if ($got_cnt !== $iface_count_before) { + fwrite(STDERR, "FAIL: read-back interface count {$got_cnt} != {$iface_count_before} -- CONFIG DAMAGED\n"); + exit(2); +} + +printf("\nOK: saved and read back -- %s = %s/%s (%d interfaces intact)\n", $ifname, $got_ip, $got_len, $got_cnt); +print("NOTE: config is saved but NOT yet applied to the running kernel.\n"); +print(" The caller must run: configctl interface reconfigure {$ifname}\n"); diff --git a/scripts/opnsense-set-interface-v6.sh b/scripts/opnsense-set-interface-v6.sh new file mode 100755 index 0000000..6e8383b --- /dev/null +++ b/scripts/opnsense-set-interface-v6.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# scripts/opnsense-set-interface-v6.sh [--commit] <edge-host> <interface> <address> <prefixlen> +# +# e.g. OPNSENSE_SSH_KEY=~/.ssh/office1-edge \ +# bash scripts/opnsense-set-interface-v6.sh --commit 10.10.0.1 lan 2602:f3e2:f01:100::1 64 +# +# Sets a STATIC IPv6 address on an OPNsense base interface and applies it, then +# proves the address is actually on the wire. +# +# WHY IT IS NOT THE REST API (D-113 AMENDMENT, 2026-07-14). Measured on the live +# Office1 edge (OPNsense 26.1): Interfaces > [LAN] is served by the LEGACY page +# /interfaces.php?if=lan. Only the Devices and Neighbors sub-trees were migrated to +# MVC. There is NO REST endpoint for a base interface's ipaddrv6. D-113(a2)'s claim +# that "interfaces" are API-covered was inferred, never measured, and is false. +# +# WHY IT IS NOT THE FORBIDDEN config.xml PUSH. It never authors a config. It ships +# scripts/opnsense-set-iface-v6.php, which loads the edge's LIVE config through +# OPNsense's own Config singleton, mutates two leaf fields, asserts the interface +# count is unchanged, and lets the vendor's own code serialize it -- the identical +# code path interfaces.php runs on Save. Nothing is replaced wholesale, so the +# API-managed Kea DHCP and the firewall rules cannot be clobbered. Same principle as +# opnsense-bootstrap-apikey.sh: call the vendor's interface, do not re-implement it. +# +# REUSABLE BY THE DC EDGES. Stage 3 builds DC0's and DC1's edges from the same +# module; this script takes them verbatim, only the host/address change. +# +# DRY BY DEFAULT -- without --commit, nothing is written and nothing is applied. +# +# root's shell on the edge is TCSH, not sh. A $(...) inside a quoted remote command +# dies with "Illegal variable name", and it dies QUIETLY. Every remote command here +# is therefore fed to `sh -s` over stdin, never interpolated into a tcsh -c string. +# +# Exit: 0 ok (or dry run) | 1 usage/precondition | 2 remote set/apply/verify failed. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SETTER="$HERE/opnsense-set-iface-v6.php" + +COMMIT=0 +if [ "${1:-}" = "--commit" ]; then COMMIT=1; shift; fi + +EDGE="${1:-}" +IFNAME="${2:-}" +ADDR="${3:-}" +PLEN="${4:-}" + +usage() { + echo "usage: opnsense-set-interface-v6.sh [--commit] <edge-host> <interface> <address> <prefixlen>" >&2 + echo " e.g. opnsense-set-interface-v6.sh --commit 10.10.0.1 lan 2602:f3e2:f01:100::1 64" >&2 + echo " env: OPNSENSE_SSH_KEY (required)" >&2 + echo " default: DRY RUN -- prints the delta, writes nothing." >&2 + exit 1 +} + +[ -n "$EDGE" ] && [ -n "$IFNAME" ] && [ -n "$ADDR" ] && [ -n "$PLEN" ] || usage +[ -f "$SETTER" ] || { echo "FAIL: setter not found: $SETTER" >&2; exit 1; } + +# Validate locally too. The PHP validates as well -- but a bad value should never +# even reach the live router. +case "$ADDR" in *:*) : ;; *) echo "FAIL: '$ADDR' is not an IPv6 address" >&2; exit 1 ;; esac +case "$PLEN" in ''|*[!0-9]*) echo "FAIL: '$PLEN' is not a prefix length" >&2; exit 1 ;; esac +[ "$PLEN" -ge 1 ] && [ "$PLEN" -le 128 ] || { echo "FAIL: prefix length '$PLEN' out of range 1-128" >&2; exit 1; } + +[ -n "${OPNSENSE_SSH_KEY:-}" ] || { echo "FAIL: \$OPNSENSE_SSH_KEY not set" >&2; exit 1; } + +SSH=(ssh -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15) +SCP=(scp -i "$OPNSENSE_SSH_KEY" -o BatchMode=yes -o ConnectTimeout=15 -q) + +REMOTE_PHP="/tmp/opnsense-set-iface-v6.$$.php" +cleanup() { "${SSH[@]}" "root@${EDGE}" "rm -f '$REMOTE_PHP'" >/dev/null 2>&1 || true; } +trap cleanup EXIT + +"${SCP[@]}" "$SETTER" "root@${EDGE}:${REMOTE_PHP}" \ + || { echo "FAIL: could not ship the setter to $EDGE" >&2; exit 2; } + +echo "=== BEFORE (the edge's own view) ===" +"${SSH[@]}" "root@${EDGE}" sh -s <<EOF || { echo "FAIL: could not read the edge" >&2; exit 2; } +ifconfig $(printf '%s' "$IFNAME" | sed 's/lan/vtnet0/; s/wan/vtnet1/') inet6 2>/dev/null | grep inet6 || echo " (no inet6 on the interface)" +EOF + +# load_phalcon.php is resolved RELATIVE to /usr/local/opnsense/mvc -- must cd there. +PHPARGS="" +[ "$COMMIT" = "1" ] && PHPARGS="--commit" + +echo +echo "=== SET (via OPNsense's own config model) ===" +"${SSH[@]}" "root@${EDGE}" sh -s <<EOF || { echo "FAIL: the setter failed on $EDGE" >&2; exit 2; } +cd /usr/local/opnsense/mvc && php '$REMOTE_PHP' $PHPARGS '$IFNAME' '$ADDR' '$PLEN' +EOF + +if [ "$COMMIT" != "1" ]; then + echo + echo "DRY RUN -- nothing written, nothing applied. Re-run with --commit." + exit 0 +fi + +echo +echo "=== APPLY (configctl -- saving config does NOT reconfigure the kernel) ===" +"${SSH[@]}" "root@${EDGE}" sh -s <<EOF || { echo "FAIL: configctl reconfigure failed" >&2; exit 2; } +configctl interface reconfigure $IFNAME +EOF + +echo +echo "=== AFTER: GROUND TRUTH (the kernel, not the config file) ===" +DEV="$(printf '%s' "$IFNAME" | sed 's/lan/vtnet0/; s/wan/vtnet1/')" +OUT="$("${SSH[@]}" "root@${EDGE}" sh -s <<EOF +ifconfig $DEV inet6 2>/dev/null | grep inet6 || true +EOF +)" +printf '%s\n' "$OUT" + +# The address must actually be ON THE INTERFACE. `{"result":"saved"}`-style +# self-reports are not evidence; the kernel is. +if printf '%s' "$OUT" | grep -qi "$(printf '%s' "$ADDR" | tr 'A-Z' 'a-z')"; then + echo + echo "OK: $ADDR/$PLEN is live on $IFNAME ($DEV)." + echo " Next: Router Advertisements -- that half IS API-native:" + echo " bash scripts/opnsense-api.sh POST /api/radvd/settings/addEntry ..." + exit 0 +fi + +echo +echo "FAIL: $ADDR is NOT on $DEV after reconfigure. Config saved but not in effect." >&2 +exit 2 diff --git a/scripts/phase-00-maas-standup.sh b/scripts/phase-00-maas-standup.sh index 46d8c65..8044462 100644 --- a/scripts/phase-00-maas-standup.sh +++ b/scripts/phase-00-maas-standup.sh @@ -34,7 +34,7 @@ # CLI forms verified against Canonical MAAS how-to-manage-networks. # ASCII + LF. # -# DC selector (opt-in, DOCFIX-166): set DC=dc0|dc1|dc2 to call +# DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc explicitly. Unset (default) == today's D-052/D-053 # behavior, unchanged. dc1 is a genuine no-op (D-101); dc2 FAILS LOUD (gap # #3) -- but note the PLANES table itself is still DC0-hardcoded, so this @@ -52,7 +52,7 @@ # runbook's Step 1: an explicit $DC env var, never an inferred default. # Unset/empty $DC changes NOTHING -- this script runs exactly as it always # has, implicitly against the D-052/D-053 plane scheme (backward compatible -# by construction). Set DC=dc0|dc1|dc2 to select explicitly; dc2 FAILS LOUD +# by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly (D-119); vr1-dc1 FAILS LOUD # today (no NetBox-assigned literals yet -- gap #3) via set -e, no second # validation layer added. This script does not source lib-hosts.sh (it never # touches host identity), so only the network selector is called here. diff --git a/scripts/prereqs/README.md b/scripts/prereqs/README.md new file mode 100644 index 0000000..0e8e3ee --- /dev/null +++ b/scripts/prereqs/README.md @@ -0,0 +1,53 @@ +# scripts/prereqs/ -- workstation/workspace prerequisite installers + +New operators start here. These scripts get a Debian/Ubuntu workstation (or the +vcloud host / Office1 control point) ready to run the VR1 DC-DC workflow, so a +runbook never assumes a runtime is already present (DOCFIX-178). Each installer +is **idempotent** (a satisfied prereq is a no-op) and supports `--check` +(read-only report) and `--dry-run` (print actions, mutate nothing). + +## Quick start + +```bash +bash scripts/prereqs/check-prereqs.sh # read-only: what's missing? +bash scripts/prereqs/install-all.sh --dry-run # preview +bash scripts/prereqs/install-all.sh # install everything missing (uses sudo) +``` + +Then re-login (or `newgrp libvirt`) if the virtualization installer just added +you to the `libvirt` group, and re-run `check-prereqs.sh` to confirm all green. + +## What each script covers + +| Script | Prereq | Needed by | +|---|---|---| +| `install-opentofu.sh` | OpenTofu `tofu` >= 1.6.0 (deb repo -> `/usr/bin/tofu`) | dc-dc Stage 1-6 `tofu` steps + `scripts/opentofu-validate.sh` | +| `install-virtualization.sh` | libvirt + qemu-kvm + virsh + `libvirt` group | dc-dc Stage 1+ (pools, networks, node VMs) | +| `install-apparmor-libvirt.sh` | AppArmor grant `<pool-parent>/** rwk,` so qemu can open disks under the **custom** VR1 pool parent (default `/var/lib/libvirt/vr1`; override `VR1_POOL_PARENT`) | **Every VR1 VM boot** -- Office1 edge, DC edges, node VMs | +| `install-jq.sh` | jq | cloud-assert + many ops scripts (and the test gauntlet) | +| `install-image-tools.sh` | qemu-utils (`qemu-img`) | Stage 2/3 OPNsense image prep (`opnsense-prep-image.sh`). The `xorriso`/ISO9660 requirement was REMOVED 2026-07-13: the config-seed ISO path is deleted (D-112 -- the Importer can never fire on a nano image; D-113(a2) -- config is done over the REST API). | +| `check-prereqs.sh` | (read-only report of all of the above + curl/python3/sudo) | run first, on any new workspace | +| `install-all.sh` | orchestrator over the five installers + a final check | one-pass setup | +| `lib-prereq.sh` | shared helpers (sourced, not run) | the installers | + +## Scope + conventions + +- **Debian/Ubuntu (apt) only.** On another OS each installer fails loud (exit 3) + and tells you to install manually -- it does not guess a package manager. + `tofu` and the ISO builder have portable upstream options; see each script. +- **`sudo`** is used only for the actual package install / group change / profile + edit. The `--check` and `--dry-run` modes never touch the system, never call + sudo, and never reload a service. +- **The AppArmor one is the trap, not a formality.** libvirt's stock + `abstractions/libvirt-qemu` only grants qemu the DEFAULT pool path + (`/var/lib/libvirt/images`). VR1 uses a custom pool parent, so AppArmor blocks + qemu even with perfect POSIX permissions -- the domain DEFINES, then fails to + start with a bare "Permission denied" that never mentions AppArmor. It cost a + session on 2026-07-12. `install-apparmor-libvirt.sh` reloads apparmor and + restarts libvirtd ONLY on the run that actually adds the rule, never on an + already-good host. +- **`tofu`, not `opentofu`.** The OpenTofu installer uses the deb method on + purpose: every runbook step and `scripts/opentofu-validate.sh` call the `tofu` + command, which the `opentofu` snap may not expose. +- Run via `bash scripts/prereqs/<name>.sh` (these are not marked executable, per + this repo's convention). Harness: `tests/prereqs/run-tests.sh`. diff --git a/scripts/prereqs/check-prereqs.sh b/scripts/prereqs/check-prereqs.sh new file mode 100644 index 0000000..b1b97f7 --- /dev/null +++ b/scripts/prereqs/check-prereqs.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# scripts/prereqs/check-prereqs.sh +# +# Read-only prereq report for the VR1 dc-dc workflow -- run this first on any new +# workstation/workspace to see what is missing before starting a stage. Mutates +# NOTHING. Each REQUIRED miss has a matching scripts/prereqs/install-*.sh. +# +# Exit: 0 all REQUIRED present | 1 one or more REQUIRED missing. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" + +miss=0 +line() { # <status> <label> <detail> <fixer> + printf ' [%-7s] %-22s %s\n' "$1" "$2" "$3" + if [ -n "${4:-}" ]; then printf ' fix: %s\n' "$4"; fi + return 0 # MUST return 0: callers use `pq_have X && line OK.. || line WARN..` +} + +echo "== VR1 dc-dc prereq report ($(hostname)) ==" + +# tofu >= 1.6.0 (REQUIRED, Stage 1-6) +if pq_have tofu; then + v="$(tofu version 2>/dev/null | head -n1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -n1)" + if [ "$(printf '1.6.0\n%s\n' "$v" | sort -V | head -n1)" = "1.6.0" ]; then + line OK tofu "$v" + else + line OLD tofu "${v:-unknown} < 1.6.0" "bash scripts/prereqs/install-opentofu.sh"; miss=1 + fi +else + line MISSING tofu "not on PATH" "bash scripts/prereqs/install-opentofu.sh"; miss=1 +fi + +# virsh / libvirt (REQUIRED, Stage 1+) +if pq_have virsh; then line OK virsh "$(virsh --version 2>/dev/null)" +else line MISSING virsh "libvirt not installed" "bash scripts/prereqs/install-virtualization.sh"; miss=1; fi +if id -nG 2>/dev/null | tr ' ' '\n' | grep -qx libvirt; then line OK libvirt-group "user in libvirt group" +else line MISSING libvirt-group "user not in libvirt group" "bash scripts/prereqs/install-virtualization.sh (then re-login)"; miss=1; fi + +# AppArmor grant for the VR1 pool parent (REQUIRED on AppArmor hosts, Stage 1+). +# Without it every VR1 VM DEFINES but fails to START (qemu: Permission denied) -- +# see scripts/prereqs/install-apparmor-libvirt.sh for the full why. +VR1_PP="${VR1_POOL_PARENT:-/var/lib/libvirt/vr1}" +AA_F="/etc/apparmor.d/local/abstractions/libvirt-qemu" +if [ ! -d /etc/apparmor.d ]; then + line OK apparmor-libvirt "AppArmor not in use; override not needed" +elif [ -r "$AA_F" ] && grep -qxF "$VR1_PP/** rwk," "$AA_F"; then + line OK apparmor-libvirt "qemu granted $VR1_PP/**" +else + line MISSING apparmor-libvirt "qemu cannot open disks under $VR1_PP" "bash scripts/prereqs/install-apparmor-libvirt.sh"; miss=1 +fi + +# jq (REQUIRED, many ops scripts) +if pq_have jq; then line OK jq "$(jq --version 2>/dev/null)" +else line MISSING jq "JSON processor absent" "bash scripts/prereqs/install-jq.sh"; miss=1; fi + +# image tools (REQUIRED for Stage 2/3 OPNsense build) +if pq_have qemu-img; then line OK qemu-img "present" +else line MISSING qemu-img "qemu-utils absent" "bash scripts/prereqs/install-image-tools.sh"; miss=1; fi +if pq_have xorriso || pq_have genisoimage; then line OK iso-builder "$(pq_have xorriso && echo xorriso || echo genisoimage)" +else line MISSING iso-builder "no ISO9660 builder" "bash scripts/prereqs/install-image-tools.sh"; miss=1; fi + +# near-universal, reported but not auto-installed here +pq_have curl && line OK curl "present" || line WARN curl "absent (needed for tofu init + downloads)" +pq_have python3 && line OK python3 "$(python3 --version 2>/dev/null)" || line WARN python3 "absent (netbox + provider-bundle-check)" +pq_have sudo && line OK sudo "present" || line WARN sudo "absent (installers need root)" + +echo +if [ "$miss" -eq 0 ]; then echo "PREREQS: all required present"; exit 0; fi +echo "PREREQS: missing required tooling (see 'fix:' lines, or run scripts/prereqs/install-all.sh)"; exit 1 diff --git a/scripts/prereqs/install-all.sh b/scripts/prereqs/install-all.sh new file mode 100644 index 0000000..09d794f --- /dev/null +++ b/scripts/prereqs/install-all.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-all.sh [--dry-run] +# +# Orchestrator: runs every prereq installer in a sensible order, then a final +# read-only check. Passes --dry-run through to each. Each installer is itself +# idempotent, so re-running is safe. Debian/Ubuntu (apt). +# +# --dry-run print what each installer would do; mutate nothing +# (no flag) install all missing prereqs (uses sudo) +# +# Exit: 0 all ok | 1 an installer failed | 2 bad args. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-all.sh [--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" +FLAG=""; [ "$PQ_DRYRUN" = "1" ] && FLAG="--dry-run" + +# virtualization first (substrate), then its AppArmor override (must follow the +# libvirt install -- it edits libvirt's own profile dir), then tofu, then the +# smaller tools. +for s in install-virtualization install-apparmor-libvirt install-opentofu install-jq install-image-tools; do + echo "== $s ==" + bash "$HERE/$s.sh" $FLAG || { echo "FAIL: $s.sh exited $?" >&2; exit 1; } + echo +done + +echo "== final read-only check ==" +bash "$HERE/check-prereqs.sh" || true # report only; install-all's own success is per-installer above +echo +echo "install-all: done${FLAG:+ (dry-run)}" diff --git a/scripts/prereqs/install-apparmor-libvirt.sh b/scripts/prereqs/install-apparmor-libvirt.sh new file mode 100644 index 0000000..f5db608 --- /dev/null +++ b/scripts/prereqs/install-apparmor-libvirt.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-apparmor-libvirt.sh [--check|--dry-run] +# +# Prereq: let qemu read/write the VR1 libvirt pool parent under AppArmor. +# +# WHY THIS EXISTS (learned the hard way, 2026-07-12): libvirt's stock +# `abstractions/libvirt-qemu` profile only grants qemu access to the DEFAULT pool +# path (/var/lib/libvirt/images). VR1 uses a CUSTOM pool parent, which is NOT in +# that abstraction -- so AppArmor blocks qemu even when the POSIX permissions are +# perfect. The symptom is nasty: the domain DEFINES fine, then fails to start with +# a bare qemu "Permission denied", and nothing in the libvirt error names AppArmor. +# This GATES EVERY VR1 VM BOOT (Office1 edge, DC edges, node VMs). +# +# It had been applied by hand on the vcloud host and existed ONLY as host state -- +# so a rebuild, or a second host, walked straight back into the same wall. That is +# what this script closes. +# +# Grants: `<pool-parent>/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu +# (the `local/` include is the vendor-sanctioned override point -- it survives +# package upgrades, which editing the abstraction itself would not). +# +# --check report whether the rule is present; exit 0 ok / 1 missing +# --dry-run print what it would do; mutate nothing +# (no flag) append the rule (if absent), reload apparmor, restart libvirtd (sudo) +# +# Idempotent: if the rule is already present it exits 0 having touched NOTHING -- +# in particular it does NOT reload apparmor or bounce libvirtd on an already-good +# host. The reload/restart happens ONLY on the run that actually adds the rule. +# Restarting libvirtd does not stop running domains, but it is a real service +# action -- hence it is gated behind an actual change. +# +# Pool parent defaults to /var/lib/libvirt/vr1; override with VR1_POOL_PARENT. +# +# Exit: 0 ok/installed | 1 check-failed | 2 bad args | 3 unsupported OS | +# 4 install failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" + +POOL_PARENT="${VR1_POOL_PARENT:-/var/lib/libvirt/vr1}" +AA_DIR="/etc/apparmor.d/local/abstractions" +AA_FILE="$AA_DIR/libvirt-qemu" +RULE="$POOL_PARENT/** rwk," + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-apparmor-libvirt.sh [--check|--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" + +# Is AppArmor actually the LSM in play? On a host without it (or with it disabled) +# the rule is unnecessary, not failed -- report N/A and succeed. Do not manufacture +# a failure on a system this does not apply to. +aa_in_use() { + [ -d /etc/apparmor.d ] || return 1 + if pq_have aa-enabled; then aa-enabled --quiet 2>/dev/null && return 0 || return 1; fi + [ -e /sys/module/apparmor/parameters/enabled ] || return 1 + grep -qi '^y' /sys/module/apparmor/parameters/enabled 2>/dev/null +} + +rule_present() { [ -r "$AA_FILE" ] && grep -qxF "$RULE" "$AA_FILE"; } + +report() { + if ! aa_in_use; then + echo "N/A: AppArmor is not in use on this host -- no libvirt-qemu override needed" + return 0 + fi + if rule_present; then + echo "OK: qemu is granted '$POOL_PARENT/**' in $AA_FILE" + return 0 + fi + echo "ABSENT: no AppArmor grant for '$POOL_PARENT/**' in $AA_FILE" + echo " Every VR1 VM will DEFINE but FAIL TO START (qemu: Permission denied)." + return 1 +} + +report; st=$? +[ "$PQ_CHECK" = "1" ] && exit "$st" +[ "$st" = "0" ] && { echo "-- apparmor/libvirt prereq already satisfied --"; exit 0; } + +pq_require_apt "the apparmor/libvirt override" || exit 3 + +pq_run "create $AA_DIR" -- sudo mkdir -p "$AA_DIR" \ + || { echo "FAIL: could not create $AA_DIR" >&2; exit 4; } +pq_run "grant qemu access to $POOL_PARENT/**" -- \ + sudo sh -c "printf '%s\n' '$RULE' >> '$AA_FILE'" \ + || { echo "FAIL: could not append the rule to $AA_FILE" >&2; exit 4; } +pq_run "reload apparmor" -- sudo systemctl reload apparmor \ + || { echo "FAIL: apparmor reload" >&2; exit 4; } +pq_run "restart libvirtd (picks up the new profile for NEW domain starts)" -- \ + sudo systemctl restart libvirtd \ + || { echo "FAIL: libvirtd restart" >&2; exit 4; } + +[ "$PQ_DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } + +rule_present || { echo "FAIL: rule still absent after append -- inspect $AA_FILE" >&2; exit 4; } +echo "INSTALLED: qemu may now open disks under $POOL_PARENT/. Re-run --check to confirm." +exit 0 diff --git a/scripts/prereqs/install-image-tools.sh b/scripts/prereqs/install-image-tools.sh new file mode 100644 index 0000000..49baf6f --- /dev/null +++ b/scripts/prereqs/install-image-tools.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-image-tools.sh [--check|--dry-run] +# +# Prereq installer: image tooling for the OPNsense-edge build path (dc-dc +# Stage 2/3). qemu-img (from qemu-utils) is used by scripts/opnsense-prep-image.sh. +# Debian/Ubuntu (apt). Idempotent. +# +# THE ISO9660 BUILDER IS NO LONGER NEEDED (2026-07-13). It existed for +# scripts/opnsense-build-config-iso.sh, which built a config-seed ISO for the +# OPNsense Configuration Importer -- an importer that CANNOT FIRE on a +# pre-installed nano image (D-112: it probes for a read-only root, finds a +# writable one with a factory /conf/config.xml, and exits without enumerating a +# single device). The ISO was never read by anything. That script, and the whole +# config.xml delivery path, are DELETED under D-113(a2): edge config is done over +# the REST API (scripts/opnsense-api.sh). xorriso/genisoimage are therefore no +# longer installed or checked here. +# +# --check report qemu-img presence; exit 0/1 +# --dry-run print what it would do; mutate nothing +# (no flag) install qemu-utils (sudo) +# +# Exit: 0 ok/installed | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" +PKGS="qemu-utils" + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-image-tools.sh [--check|--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" + +report() { + local ok=0 + pq_have qemu-img && echo "OK: qemu-img present" || { echo "ABSENT: qemu-img (qemu-utils)"; ok=1; } + # No ISO9660 builder check: the config-seed ISO path is DELETED (D-112/D-113(a2)). + return "$ok" +} + +report; st=$? +[ "$PQ_CHECK" = "1" ] && exit "$st" +[ "$st" = "0" ] && { echo "-- image/ISO tooling already satisfied --"; exit 0; } + +pq_require_apt "image tools" || exit 3 +pq_run "apt-get update" -- sudo apt-get update || { echo "FAIL: apt-get update" >&2; exit 4; } +# shellcheck disable=SC2086 +pq_run "install: $PKGS" -- sudo apt-get install -y $PKGS || { echo "FAIL: package install" >&2; exit 4; } +[ "$PQ_DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } +report >/dev/null && { echo "INSTALLED: qemu-img + ISO builder"; exit 0; } +echo "FAIL: post-install check incomplete" >&2; exit 4 diff --git a/scripts/prereqs/install-jq.sh b/scripts/prereqs/install-jq.sh new file mode 100644 index 0000000..8469364 --- /dev/null +++ b/scripts/prereqs/install-jq.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-jq.sh [--check|--dry-run] +# +# Prereq installer: jq -- the JSON processor many of this repo's operational +# scripts rely on (cloud-assert, tenant tooling, BOM/diff work). Its absence is +# a known cause of harness-gauntlet failures on under-provisioned workstations. +# Debian/Ubuntu (apt). Idempotent. +# +# --check report present/absent; exit 0/1 +# --dry-run print what it would do; mutate nothing +# (no flag) apt-get install jq (sudo) +# +# Exit: 0 ok/installed | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-jq.sh [--check|--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" + +if pq_have jq; then + echo "OK: jq present ($(jq --version 2>/dev/null))"; exit 0 +fi +echo "ABSENT: jq not on PATH" +[ "$PQ_CHECK" = "1" ] && exit 1 + +pq_require_apt jq || exit 3 +pq_run "apt-get update" -- sudo apt-get update || { echo "FAIL: apt-get update" >&2; exit 4; } +pq_run "install: jq" -- sudo apt-get install -y jq || { echo "FAIL: package install" >&2; exit 4; } +[ "$PQ_DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } +pq_have jq && { echo "INSTALLED: jq $(jq --version 2>/dev/null)"; exit 0; } +echo "FAIL: post-install check did not find jq" >&2; exit 4 diff --git a/scripts/prereqs/install-opentofu.sh b/scripts/prereqs/install-opentofu.sh new file mode 100644 index 0000000..934f9c8 --- /dev/null +++ b/scripts/prereqs/install-opentofu.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-opentofu.sh [--check|--dry-run] +# +# Prereq installer: OpenTofu `tofu` (>= 1.6.0) -- the IaC runtime every dc-dc +# Stage 1-6 runbook and scripts/opentofu-validate.sh depend on. Idempotent: if a +# new-enough tofu is already on PATH it does nothing. Debian/Ubuntu via the +# official installer's deb method, which lands /usr/bin/tofu (the `tofu` command +# name our tooling expects; the `opentofu` snap may not expose a plain `tofu`). +# +# --check report present/absent + version only; exit 0 if OK, 1 if missing/old +# --dry-run print what it would do; mutate nothing +# (no flag) install if absent/too-old (uses sudo for the deb install) +# +# Registry reach (registry.opentofu.org) is needed later by `tofu init`, not here. +# Exit: 0 ok/installed | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" +MIN_VER="1.6.0" + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-opentofu.sh [--check|--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" + +ver_ge() { [ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -n1)" = "$2" ]; } # $1 >= $2 ? +cur_ver() { tofu version 2>/dev/null | head -n1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -n1; } + +if pq_have tofu; then + v="$(cur_ver)" + if [ -n "$v" ] && ver_ge "$v" "$MIN_VER"; then + echo "OK: tofu $v present (>= $MIN_VER)"; exit 0 + fi + echo "PRESENT-BUT-OLD: tofu ${v:-unknown} < $MIN_VER" +else + echo "ABSENT: tofu not on PATH" +fi +[ "$PQ_CHECK" = "1" ] && exit 1 + +pq_require_apt tofu || exit 3 +tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT +inst="$tmp/install-opentofu.sh" +pq_run "download the official OpenTofu installer" -- \ + curl -fsSL https://get.opentofu.org/install-opentofu.sh -o "$inst" \ + || { echo "FAIL: could not download installer (registry/internet reach?)" >&2; exit 4; } +pq_run "run installer (deb method; needs sudo)" -- \ + sudo bash "$inst" --install-method deb \ + || { echo "FAIL: installer exited non-zero" >&2; exit 4; } + +[ "$PQ_DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } +v="$(cur_ver)" +if pq_have tofu && [ -n "$v" ] && ver_ge "$v" "$MIN_VER"; then + echo "INSTALLED: tofu $v"; exit 0 +fi +echo "FAIL: post-install check did not find tofu >= $MIN_VER on PATH" >&2; exit 4 diff --git a/scripts/prereqs/install-virtualization.sh b/scripts/prereqs/install-virtualization.sh new file mode 100644 index 0000000..1de525a --- /dev/null +++ b/scripts/prereqs/install-virtualization.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# scripts/prereqs/install-virtualization.sh [--check|--dry-run] +# +# Prereq installer: libvirt + KVM/QEMU + virsh -- the virtualization substrate +# the vcloud host runs on (dc-dc Stage 1+: pools, networks, node VMs). Installs +# libvirt-daemon-system, qemu-kvm, libvirt-clients, virtinst, and adds the +# invoking user to the `libvirt` group. Debian/Ubuntu (apt). Idempotent. +# +# --check report virsh/qemu presence + qemu:///system reach + group; exit 0/1 +# --dry-run print what it would do; mutate nothing +# (no flag) install missing packages + add user to libvirt group (sudo) +# +# NOTE: a freshly-added group membership needs a re-login (or `newgrp libvirt`) +# to take effect. Exit: 0 ok/installed | 1 check-failed | 2 bad args | +# 3 unsupported OS | 4 install failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +. "$HERE/lib-prereq.sh" +PKGS="libvirt-daemon-system qemu-kvm libvirt-clients virtinst" + +pq_parse_mode "$@"; rc=$? +[ "$rc" = "10" ] && { echo "usage: bash scripts/prereqs/install-virtualization.sh [--check|--dry-run]"; exit 0; } +[ "$rc" = "0" ] || exit "$rc" + +report() { + local ok=0 + pq_have virsh && echo "OK: virsh present ($(virsh --version 2>/dev/null))" || { echo "ABSENT: virsh"; ok=1; } + pq_have qemu-system-x86_64 && echo "OK: qemu-system-x86_64 present" || { echo "ABSENT: qemu-system-x86_64"; ok=1; } + if pq_have virsh && virsh -c qemu:///system list --all >/dev/null 2>&1; then + echo "OK: qemu:///system reachable" + else + echo "WARN: qemu:///system not reachable yet (daemon down, or group not active until re-login)" + fi + if id -nG 2>/dev/null | tr ' ' '\n' | grep -qx libvirt; then + echo "OK: user in 'libvirt' group" + else + echo "ABSENT: user not in 'libvirt' group (re-login needed after install)"; ok=1 + fi + return "$ok" +} + +report; st=$? +[ "$PQ_CHECK" = "1" ] && exit "$st" +[ "$st" = "0" ] && { echo "-- all virtualization prereqs already satisfied --"; exit 0; } + +pq_require_apt "libvirt/qemu" || exit 3 +pq_run "apt-get update" -- sudo apt-get update || { echo "FAIL: apt-get update" >&2; exit 4; } +# shellcheck disable=SC2086 +pq_run "install: $PKGS" -- sudo apt-get install -y $PKGS || { echo "FAIL: package install" >&2; exit 4; } +pq_run "add user '${USER:-$(id -un)}' to the libvirt group" -- sudo usermod -aG libvirt "${USER:-$(id -un)}" \ + || { echo "FAIL: usermod" >&2; exit 4; } + +[ "$PQ_DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } +echo "INSTALLED. NOTE: log out/in (or run 'newgrp libvirt') for the group to take effect, then re-run --check." +exit 0 diff --git a/scripts/prereqs/lib-prereq.sh b/scripts/prereqs/lib-prereq.sh new file mode 100644 index 0000000..a15e4a9 --- /dev/null +++ b/scripts/prereqs/lib-prereq.sh @@ -0,0 +1,41 @@ +# scripts/prereqs/lib-prereq.sh +# Shared helpers for the prereq installers in this directory. SOURCED, never run +# directly. Provides mode parsing (--check/--dry-run), a presence check, an apt +# guard, and a dry-run-aware command runner. ASCII + LF. + +pq_have() { command -v "$1" >/dev/null 2>&1; } + +# pq_require_apt <toolname> : succeed on Debian/Ubuntu, else fail loud (return 3). +pq_require_apt() { + pq_have apt-get && return 0 + echo "FAIL: ${1:-this prereq}'s installer supports Debian/Ubuntu (apt-get) only." >&2 + echo " Install it manually for this OS, then re-run with --check to confirm." >&2 + return 3 +} + +# pq_parse_mode "$@" : sets PQ_CHECK / PQ_DRYRUN. Returns 10 for --help, 2 on bad arg. +pq_parse_mode() { + PQ_CHECK=0; PQ_DRYRUN=0 + local a + for a in "$@"; do + case "$a" in + --check) PQ_CHECK=1 ;; + --dry-run) PQ_DRYRUN=1 ;; + -h|--help) return 10 ;; + *) echo "FAIL: unknown arg '$a' (use: --check | --dry-run)" >&2; return 2 ;; + esac + done + return 0 +} + +# pq_run <desc> -- <cmd...> : print-and-skip under dry-run, else announce+execute. +pq_run() { + local desc="$1"; shift + [ "${1:-}" = "--" ] && shift + if [ "${PQ_DRYRUN:-0}" = "1" ]; then + printf ' [dry-run] %s\n $ %s\n' "$desc" "$*" + return 0 + fi + echo " -> $desc" + "$@" +} diff --git a/scripts/reenroll-hosts.sh b/scripts/reenroll-hosts.sh index daa8d92..8a835dc 100644 --- a/scripts/reenroll-hosts.sh +++ b/scripts/reenroll-hosts.sh @@ -11,7 +11,7 @@ # (default) create any of the four that are MISSING, then poll all four to Ready # --check read-only: report current status of openstack0-3 (no mutation) # -# DC selector (opt-in, DOCFIX-166): set DC=dc0|dc1|dc2 to call +# DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc/lib_hosts_select_dc explicitly. Unset (default) == DC0's # real, enrolled hosts, unchanged. dc1/dc2 currently FAIL LOUD (no per-DC host # inventory exists yet) -- see scripts/lib-hosts.sh. @@ -42,7 +42,7 @@ # dc-dc-phase3 runbook's Step 1: an explicit $DC env var, never an inferred # default. Unset/empty $DC changes NOTHING -- this script runs exactly as it # always has, implicitly against DC0/VR0's real, enrolled hosts (backward -# compatible by construction). Set DC=dc0|dc1|dc2 to select explicitly; the +# compatible by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly (D-119); the # selectors' own fail-loud behavior becomes this script's own exit code via # set -e (lib_hosts_select_dc fails for BOTH dc1 and dc2 today -- no per-DC # host inventory exists yet for either). No second validation layer is added. diff --git a/scripts/repo_lint.py b/scripts/repo_lint.py index ee80e75..c88877b 100644 --- a/scripts/repo_lint.py +++ b/scripts/repo_lint.py @@ -183,7 +183,12 @@ # which keys on the exact repo-relative path) -- session-local, not # repo content (DOCFIX-111). The rest of .claude/ (hooks, skills) IS # checked-in repo content and stays in scope for L1/L9. + # .terraform/ is OpenTofu's downloaded provider-plugin cache (created + # by `tofu init`, gitignored per DOCFIX-175). Its third-party provider + # docs carry non-ASCII bytes and are NOT repo content -- skip like .git + # (DOCFIX-179: surfaced on the first real `tofu init` in opentofu/). if not p.is_file() or ".git" in p.parts or "__pycache__" in p.parts \ + or ".terraform" in p.parts \ or (".claude" in p.parts and "worktrees" in p.parts): continue if p.suffix.lower() == ".zip" or p.name in skip_names: diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh new file mode 100644 index 0000000..46e7dd7 --- /dev/null +++ b/scripts/site-headend-install.sh @@ -0,0 +1,277 @@ +#!/usr/bin/env bash +# scripts/site-headend-install.sh [--check|--dry-run] --compose-cidr <CIDR> [--maas-url-ip <IP>] +# +# Installs a VR1 SITE HEADEND on the host it runs on: MAAS (region+rack) + LXD, with that +# LXD registered back into MAAS as an LXD VM host so MAAS can COMPOSE the site's non-stack +# service machines into it. This is D-114, and it is VR0's proven `lxd` + `tailscale` +# pattern applied per site. +# +# RUN IT ON THE SITE HOST (voffice1 today; vvr1-dc0/vvr1-dc1 later), as root: +# ssh <site-host> 'sudo bash -s' -- --compose-cidr 10.10.1.0/24 < scripts/site-headend-install.sh +# +# --check report what is installed/configured; mutate nothing; exit 0 ok / 1 incomplete +# --dry-run print what it would do; mutate nothing +# (no flag) install + configure (idempotent; safe to re-run) +# +# EXIT: 0 ok | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. ASCII + LF. +# +# ===================================================================================== +# THE FOUR TRAPS THIS SCRIPT EXISTS TO ENCODE (all hit for real on 2026-07-13; do not +# "simplify" any of them away): +# +# 1. `lxd init --auto` FAILS OUTRIGHT on a host that also runs MAAS. +# It creates lxdbr0 with LXD's own DHCP+DNS ON, which starts a dnsmasq; dnsmasq binds +# the WILDCARD 0.0.0.0:53, and MAAS's bind9 (`named`) already holds :53. Verbatim: +# Failed starting network: The DNS and DHCP service exited prematurely: exit status 2 +# ("dnsmasq: failed to create listening socket for 10.10.1.1: Address already in use") +# Note the message names the BRIDGE address, which sends you hunting the wrong thing -- +# the real conflict is the wildcard bind. So: NEVER `lxd init --auto` here. Create the +# bridge explicitly with DHCP/DNS off AND `raw.dnsmasq=port=0`, which stops dnsmasq +# binding :53 at all. (`ipv4.dhcp=false` + `dns.mode=none` alone are NOT enough -- LXD +# still spawns dnsmasq for a managed bridge, and it still tries to bind :53.) +# +# 2. `lxc` READS ITS CONFIG FROM STDIN. If you pipe this script to `bash -s` over ssh, the +# FIRST `lxc` call with no stdin redirect EATS THE REST OF THE SCRIPT as YAML and dies +# with `yaml: unmarshal errors`, silently truncating everything after it. Every `lxc` +# call below therefore ends in `</dev/null`. Same class as the OPNsense tcsh trap. +# +# 3. LXD SNAP TRACK CHANGES ARE ONE-WAY -- you cannot downgrade a track. LXD MUST be +# installed DIRECTLY onto 5.21/stable and never allowed to land on `latest` first, +# because MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (LXD 6.7 consolidated API +# endpoints; MAAS's pinned pylxd 2.3.5 cannot speak to them; fixed in pylxd >= 2.3.9, +# not yet in a MAAS release). Canonical's guidance: LXD <= 6.6, or the 5.21 LTS track. +# https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749 +# Bonus: `core.trust_password` still EXISTS in 5.21 and is GONE in 6.x -- the pin also +# preserves the scriptable registration path. +# +# 4. MAAS DHCP IS ENABLED ON THE COMPOSE NETWORK ONLY -- NEVER on the site LAN. The site +# LAN already has an authoritative DHCP server (Kea, on the OPNsense edge). Two DHCP +# servers on one L2 is an intermittent failure that is genuinely unpleasant to +# diagnose. --compose-cidr is REQUIRED and has no default precisely so this can never +# be picked by accident, and the script REFUSES to touch DHCP on any other subnet. +# ===================================================================================== +set -uo pipefail + +CHECK=0; DRYRUN=0; COMPOSE_CIDR=""; MAAS_IP="" +while [ $# -gt 0 ]; do + case "$1" in + --check) CHECK=1 ;; + --dry-run) DRYRUN=1 ;; + --compose-cidr) shift; COMPOSE_CIDR="${1:-}" ;; + --maas-url-ip) shift; MAAS_IP="${1:-}" ;; + -h|--help) + echo "usage: site-headend-install.sh [--check|--dry-run] --compose-cidr <CIDR> [--maas-url-ip <IP>]" + exit 0 ;; + *) echo "FAIL: unknown arg '$1'" >&2; exit 2 ;; + esac + shift +done + +[ -n "$COMPOSE_CIDR" ] || { echo "FAIL: --compose-cidr is REQUIRED (e.g. 10.10.1.0/24). It has no default on purpose: MAAS must serve DHCP on the compose network ONLY, never on the site LAN, which the edge's Kea already owns." >&2; exit 2; } +echo "$COMPOSE_CIDR" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.0/24$' || { echo "FAIL: --compose-cidr must be a /24 like 10.10.1.0/24 (got '$COMPOSE_CIDR')" >&2; exit 2; } + +BASE="${COMPOSE_CIDR%.0/24}" # 10.10.1 +BRIDGE_IP="${BASE}.1" +RANGE_LO="${BASE}.100" +RANGE_HI="${BASE}.200" +SECRETS=/root/maas-secrets +MAAS_CHANNEL="3.7/stable" +PG_CHANNEL="16/stable" +LXD_CHANNEL="5.21/stable" # TRAP 3 -- do not change without reading it + +run() { # run <desc> -- <cmd...> + local d="$1"; shift; [ "${1:-}" = "--" ] && shift + if [ "$DRYRUN" = "1" ]; then printf ' [dry-run] %s\n $ %s\n' "$d" "$*"; return 0; fi + echo " -> $d"; "$@" +} +have() { command -v "$1" >/dev/null 2>&1; } + +# ---------------- check mode (read-only) ---------------- +report() { + local bad=0 + snap list maas >/dev/null 2>&1 && echo "OK: maas snap $(snap list maas 2>/dev/null | awk 'NR>1{print $2" ("$4")"}')" || { echo "ABSENT: maas snap"; bad=1; } + snap list postgresql >/dev/null 2>&1 && echo "OK: postgresql $(snap list postgresql 2>/dev/null | awk 'NR>1{print $2}')" || { echo "ABSENT: postgresql snap"; bad=1; } + if snap list lxd >/dev/null 2>&1; then + local tr; tr="$(snap list lxd 2>/dev/null | awk 'NR>1{print $4}')" + if [ "$tr" = "$LXD_CHANNEL" ]; then echo "OK: lxd on $tr" + else echo "WRONG TRACK: lxd is on '$tr', must be '$LXD_CHANNEL' (MAAS breaks on LXD >= 6.7; track changes are ONE-WAY -- a reinstall is needed)"; bad=1; fi + else echo "ABSENT: lxd snap"; bad=1; fi + + if have lxc && lxc network show lxdbr0 </dev/null >/dev/null 2>&1; then + echo "OK: lxdbr0 exists (ipv4.dhcp=$(lxc network get lxdbr0 ipv4.dhcp </dev/null 2>/dev/null), dns.mode=$(lxc network get lxdbr0 dns.mode </dev/null 2>/dev/null))" + local n53 n67 + n53=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':53 ') + n67=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':67 ') + if [ "${n53:-0}" = "0" ] && [ "${n67:-0}" = "0" ]; then + echo "OK: dnsmasq binds NO :53 and NO :67 -- MAAS and the edge's Kea remain sole DHCP/DNS authorities" + else + echo "FAIL: dnsmasq holds :53=$n53 :67=$n67 -- it is FIGHTING MAAS/Kea (see trap 1)"; bad=1 + fi + else echo "ABSENT: lxdbr0"; bad=1; fi + + if have maas && maas status 2>/dev/null | grep -q regiond; then echo "OK: MAAS initialised"; else echo "ABSENT: MAAS not initialised"; bad=1; fi + return "$bad" +} + +report_st=0 +if [ "$CHECK" = "1" ]; then report; exit $?; fi +report >/dev/null 2>&1; report_st=$? +[ "$report_st" = "0" ] && [ "$DRYRUN" = "0" ] && { echo "-- site headend already installed and configured --"; exit 0; } + +have apt-get || { echo "FAIL: Debian/Ubuntu (apt) only." >&2; exit 3; } +# root is required to MUTATE, not to plan: --dry-run stays runnable by anyone (and by the harness). +[ "$DRYRUN" = "1" ] || [ "$(id -u)" = "0" ] || { echo "FAIL: must run as root (use: ssh <host> 'sudo bash -s' -- ... < $0)" >&2; exit 4; } +[ -n "$MAAS_IP" ] || MAAS_IP="$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{print $7; exit}')" +[ -n "$MAAS_IP" ] || { echo "FAIL: could not measure this host's outbound IP; pass --maas-url-ip" >&2; exit 4; } + +install -d -m 0700 "$SECRETS" +newpass() { # newpass <file> -- generate once, 0600, NEVER echoed + [ -s "$1" ] && return 0 + openssl rand -base64 30 | tr -dc 'A-Za-z0-9' | head -c 28 > "$1"; chmod 0600 "$1" +} + +echo "== 1. time: MAAS manages time via chrony; systemd-timesyncd conflicts ==" +run "disable systemd-timesyncd" -- systemctl disable --now systemd-timesyncd >/dev/null 2>&1 + +echo "== 2. MAAS $MAAS_CHANNEL + PostgreSQL $PG_CHANNEL ==" +snap list maas >/dev/null 2>&1 || run "install maas" -- snap install maas --channel="$MAAS_CHANNEL" || exit 4 +snap list postgresql >/dev/null 2>&1 || run "install postgresql" -- snap install postgresql --channel="$PG_CHANNEL" || exit 4 + +echo "== 3. MAAS database (password generated here, 0600, never printed) ==" +if [ "$DRYRUN" = "0" ]; then + newpass "$SECRETS/db.pass"; DBPASS="$(cat "$SECRETS/db.pass")" + postgresql.psql -U postgres -h /tmp -tAc "SELECT 1 FROM pg_roles WHERE rolname='maas'" 2>/dev/null | grep -q 1 \ + || postgresql.psql -U postgres -h /tmp -c "CREATE ROLE maas LOGIN PASSWORD '$DBPASS'" >/dev/null 2>&1 + postgresql.psql -U postgres -h /tmp -tAc "SELECT 1 FROM pg_database WHERE datname='maas'" 2>/dev/null | grep -q 1 \ + || postgresql.createdb -U postgres -h /tmp -O maas maas >/dev/null 2>&1 + echo " -> role + database 'maas' present" +else + echo " [dry-run] create role+db 'maas' with a generated password" +fi + +echo "== 4. maas init region+rack ==" +if [ "$DRYRUN" = "0" ]; then + if maas status 2>/dev/null | grep -q regiond; then + echo " -> already initialised" + else + maas init region+rack --maas-url "http://${MAAS_IP}:5240/MAAS" \ + --database-uri "postgres://maas:$(cat "$SECRETS/db.pass")@localhost/maas" --force >/dev/null 2>&1 \ + || { echo "FAIL: maas init" >&2; exit 4; } + echo " -> initialised" + fi + newpass "$SECRETS/admin.pass" + maas apikey --username admin >/dev/null 2>&1 || \ + maas createadmin --username admin --password "$(cat "$SECRETS/admin.pass")" --email "admin@$(hostname)" >/dev/null 2>&1 + maas apikey --username admin > "$SECRETS/admin.apikey" 2>/dev/null; chmod 0600 "$SECRETS/admin.apikey" + for i in $(seq 1 60); do curl -sf -o /dev/null --max-time 3 "http://${MAAS_IP}:5240/MAAS/" && break; sleep 5; done + maas login admin "http://${MAAS_IP}:5240/MAAS" "$(cat "$SECRETS/admin.apikey")" >/dev/null 2>&1 + echo " -> admin ready, CLI logged in" +else + echo " [dry-run] maas init region+rack + createadmin + login" +fi + +echo "== 5. LXD, DIRECTLY onto $LXD_CHANNEL (trap 3: track changes are ONE-WAY) ==" +if snap list lxd >/dev/null 2>&1; then + tr="$(snap list lxd 2>/dev/null | awk 'NR>1{print $4}')" + [ "$tr" = "$LXD_CHANNEL" ] || { echo "FAIL: lxd already on track '$tr'; you CANNOT downgrade a snap track. Remove it (snap remove lxd) and re-run." >&2; exit 4; } +else + run "install lxd" -- snap install lxd --channel="$LXD_CHANNEL" || exit 4 + run "hold lxd auto-refresh" -- snap refresh --hold lxd >/dev/null 2>&1 +fi +[ "$DRYRUN" = "0" ] && lxd waitready --timeout=90 + +echo "== 6. LXD storage + compose bridge (built explicitly; see trap 1 -- the auto-init path FAILS here) ==" +if [ "$DRYRUN" = "0" ]; then + # dir driver: conservative inside an already-nested guest (no zfs loop file). + lxc storage show default </dev/null >/dev/null 2>&1 || lxc storage create default dir </dev/null >/dev/null 2>&1 + if ! lxc network show lxdbr0 </dev/null >/dev/null 2>&1; then + lxc network create lxdbr0 \ + ipv4.address="${BRIDGE_IP}/24" ipv4.nat=true ipv4.dhcp=false \ + ipv6.address=none dns.mode=none raw.dnsmasq="port=0" </dev/null >/dev/null 2>&1 \ + || { echo "FAIL: could not create lxdbr0 (see trap 1)" >&2; exit 4; } + fi + lxc profile device remove default eth0 </dev/null >/dev/null 2>&1 + lxc profile device add default root disk path=/ pool=default </dev/null >/dev/null 2>&1 + lxc profile device add default eth0 nic network=lxdbr0 name=eth0 </dev/null >/dev/null 2>&1 + lxc config set core.https_address '[::]:8443' </dev/null + newpass "$SECRETS/lxd-trust.pass" + lxc config set core.trust_password "$(cat "$SECRETS/lxd-trust.pass")" </dev/null + echo " -> lxdbr0 ${BRIDGE_IP}/24, NAT on, LXD DHCP/DNS OFF; API on :8443" + + n53=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':53 ') + n67=$(ss -lunp 2>/dev/null | grep dnsmasq | grep -c ':67 ') + [ "${n53:-0}" = "0" ] && [ "${n67:-0}" = "0" ] \ + || { echo "FAIL: dnsmasq is holding :53=$n53 :67=$n67 -- it will fight MAAS/Kea (trap 1)" >&2; exit 4; } + echo " -> VERIFIED: dnsmasq binds no :53 and no :67" +else + echo " [dry-run] create dir pool + lxdbr0 ${BRIDGE_IP}/24 (dhcp off, dns off, raw.dnsmasq=port=0), expose API" +fi + +echo "== 7. register LXD into MAAS as a VM host ==" +if [ "$DRYRUN" = "0" ]; then + if maas admin vm-hosts read 2>/dev/null | grep -q '"type": "lxd"'; then + echo " -> LXD vm-host already registered" + else + maas admin vm-hosts create type=lxd name="$(hostname)-lxd" \ + power_address="https://${MAAS_IP}:8443" project=default \ + password="$(cat "$SECRETS/lxd-trust.pass")" >/dev/null 2>&1 \ + && echo " -> registered as $(hostname)-lxd" || { echo "FAIL: vm-hosts create" >&2; exit 4; } + fi +else + echo " [dry-run] maas admin vm-hosts create type=lxd power_address=https://${MAAS_IP}:8443" +fi + +echo "== 8. MAAS DHCP on the COMPOSE network ONLY (trap 4) ==" +if [ "$DRYRUN" = "0" ]; then + SUB_JSON="$(maas admin subnets read 2>/dev/null)" + SUB_ID="$(printf '%s' "$SUB_JSON" | python3 -c " +import sys,json +t=sys.argv[1] +for s in json.load(sys.stdin): + if s.get('cidr')==t: print(s['id']); break +" "$COMPOSE_CIDR")" + [ -n "$SUB_ID" ] || { echo "FAIL: MAAS has not discovered $COMPOSE_CIDR (is lxdbr0 up?)" >&2; exit 4; } + FAB_ID="$(printf '%s' "$SUB_JSON" | python3 -c " +import sys,json +t=sys.argv[1] +for s in json.load(sys.stdin): + if s.get('cidr')==t: print(s['vlan']['fabric_id']); break +" "$COMPOSE_CIDR")" + RACK="$(maas admin rack-controllers read 2>/dev/null | python3 -c "import sys,json;print(json.load(sys.stdin)[0]['system_id'])")" + + # GATEWAY: MAAS auto-discovers the compose subnet from lxdbr0's interface, and that discovery + # carries NO gateway. Without this, a DEPLOYED machine gets a netplan with NO DEFAULT ROUTE -- + # it comes up looking fine (DNS even resolves, because MAAS's bind9 is link-local) but has NO + # EGRESS, so the first `apt install` on it hangs. lxdbr0 IS the gateway (it does the NAT). + # Measured 2026-07-13: office1-netbox deployed with gateway_ip=None and had to be redeployed. + maas admin subnet update "$SUB_ID" gateway_ip="$BRIDGE_IP" >/dev/null 2>&1 \ + && echo " -> gateway_ip=$BRIDGE_IP set on $COMPOSE_CIDR" + + maas admin ipranges create type=dynamic subnet="$SUB_ID" start_ip="$RANGE_LO" end_ip="$RANGE_HI" \ + comment="MAAS DHCP/PXE for LXD-composed VMs (D-114)" >/dev/null 2>&1 + maas admin vlan update "$FAB_ID" 0 dhcp_on=True primary_rack="$RACK" >/dev/null 2>&1 \ + && echo " -> DHCP ON for $COMPOSE_CIDR ($RANGE_LO-$RANGE_HI), rack=$RACK" + + # HARD GUARD: assert MAAS did not turn DHCP on for any OTHER subnet. + BADSUB="$(maas admin subnets read 2>/dev/null | python3 -c " +import sys,json +t=sys.argv[1] +bad=[s['cidr'] for s in json.load(sys.stdin) if s['vlan'].get('dhcp_on') and s.get('cidr')!=t] +print(','.join(bad)) +" "$COMPOSE_CIDR")" + if [ -n "$BADSUB" ]; then + echo "FAIL: MAAS DHCP is ON for a subnet that is NOT the compose network: $BADSUB" >&2 + echo " The site LAN's DHCP belongs to the edge's Kea. Turn this OFF before proceeding." >&2 + exit 4 + fi + echo " -> VERIFIED: MAAS DHCP is on for $COMPOSE_CIDR and NO other subnet" +else + echo " [dry-run] iprange $RANGE_LO-$RANGE_HI + dhcp_on for $COMPOSE_CIDR only" +fi + +[ "$DRYRUN" = "1" ] && { echo " (dry-run: no changes made)"; exit 0; } +echo +echo "INSTALLED. MAAS UI: http://${MAAS_IP}:5240/MAAS (admin password: $SECRETS/admin.pass, 0600)" +echo "Compose a service VM with:" +echo " maas admin vm-host compose <id> cores=2 memory=4096 storage=root:40 hostname=<name>" +exit 0 diff --git a/tests/carve-host-interfaces/fakebin/maas b/tests/carve-host-interfaces/fakebin/maas old mode 100644 new mode 100755 diff --git a/tests/carve-host-interfaces/run-tests.sh b/tests/carve-host-interfaces/run-tests.sh index 2fc25f5..9574a53 100644 --- a/tests/carve-host-interfaces/run-tests.sh +++ b/tests/carve-host-interfaces/run-tests.sh @@ -79,15 +79,15 @@ echo echo "=== \$DC selector wiring (DOCFIX-166; tooling gap register #15 sub-item 2) ===" -run 0 "DC=dc0 explicit: identical no-op full carve plan" "$FIX/sub_ok.json" "$FIX/if_fresh.json" dc0 +run 0 "DC=vr0-dc0 explicit: identical no-op full carve plan" "$FIX/sub_ok.json" "$FIX/if_fresh.json" vr0-dc0 has 'WOULD: create br-ex \(OVS\) parent=enp1s0\(id=100\)' has 'Summary: 0 fatal' -run_dc 1 "DC=dc1: hosts selector fails loud (no per-DC host data)" dc1 +run_dc 1 "DC=vr1-dc0: hosts selector fails loud (no per-DC host data)" vr1-dc0 has 'no enrolled hosts yet' absent 'DO:|WOULD:' # must exit before any MAAS interaction -run_dc 1 "DC=dc2: net selector fails loud first (NetBox gap)" dc2 +run_dc 1 "DC=vr1-dc1: net selector fails loud first (NetBox gap)" vr1-dc1 has 'no assigned network literals yet' run_dc 1 "DC=bogus: unknown token fails loud" bogus diff --git a/tests/creds-audit/run-tests.sh b/tests/creds-audit/run-tests.sh new file mode 100755 index 0000000..7318a57 --- /dev/null +++ b/tests/creds-audit/run-tests.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# tests/creds-audit/run-tests.sh +# +# Harness for scripts/creds-audit.sh. Offline: builds synthetic creds folders + +# manifests under a tmp CREDS_ROOT and asserts the audit's verdict. Touches no real +# secret and no real home dir (CREDS_ROOT / MANIFEST_DIR are redirected to tmp). +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +AUDIT="$REPO/scripts/creds-audit.sh" +P=0; F=0 +ok() { echo " [PASS] $1"; P=$((P+1)); } +no() { echo " [FAIL] $1"; F=$((F+1)); } + +[ -f "$AUDIT" ] || { echo "FAIL: $AUDIT not found"; exit 1; } + +# fresh synthetic environment: CREDS_ROOT/<site>-creds + MANIFEST_DIR/<site>.manifest +SITE=testsite +mkenv() { # mkenv -> echoes the tmp root; caller sets CREDS_ROOT/MANIFEST_DIR to it + local root; root="$(mktemp -d)" + mkdir -p "$root/${SITE}-creds" "$root/manifests" + chmod 700 "$root/${SITE}-creds" + printf 'a_secret 600 local\nb_key.pub 644 local\nc.env 600 somevm:/root/x.token\n' \ + > "$root/manifests/${SITE}.manifest" + printf 'secret\n' > "$root/${SITE}-creds/a_secret"; chmod 600 "$root/${SITE}-creds/a_secret" + printf 'pub\n' > "$root/${SITE}-creds/b_key.pub"; chmod 644 "$root/${SITE}-creds/b_key.pub" + printf 'tok\n' > "$root/${SITE}-creds/c.env"; chmod 600 "$root/${SITE}-creds/c.env" + echo "$root" +} +run() { CREDS_ROOT="$1" MANIFEST_DIR="$1/manifests" bash "$AUDIT" "$SITE" >"$1/out" 2>&1; echo $?; } + +echo "=== creds-audit ===" + +# T1: a complete, correct, non-sprawled folder -> CLEAN, exit 0 +R="$(mkenv)"; rc=$(run "$R") +[ "$rc" = 0 ] && grep -q "CLEAN" "$R/out" && ok "T1 complete + correct folder -> CLEAN, exit 0" \ + || { no "T1 a correct folder should pass (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T2: a manifest-required file MISSING (the SEC-009 gap: a VM secret never pulled in) +R="$(mkenv)"; rm -f "$R/${SITE}-creds/c.env"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "MISSING: c.env" "$R/out" && ok "T2 a manifested (VM-sourced) secret absent -> MISSING, exit 1" \ + || { no "T2 a missing manifest entry should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T3: wrong permissions on a secret -> FAIL +R="$(mkenv)"; chmod 644 "$R/${SITE}-creds/a_secret"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "a_secret mode 644" "$R/out" && ok "T3 a secret with loose mode (644) -> caught, exit 1" \ + || { no "T3 wrong mode should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T4: an UNDECLARED file in the folder -> FAIL (spread: a secret nobody declared) +R="$(mkenv)"; printf 'x\n' > "$R/${SITE}-creds/stray.env"; chmod 600 "$R/${SITE}-creds/stray.env"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "UNDECLARED file in folder: stray.env" "$R/out" && ok "T4 an undeclared file in the folder -> caught, exit 1" \ + || { no "T4 an undeclared file should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T5: SPRAWL -- a sensitive file loose in CREDS_ROOT, outside any *-creds/ folder +R="$(mkenv)"; printf 'x\n' > "$R/loose.env"; chmod 600 "$R/loose.env"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "SPRAWL: sensitive file loose" "$R/out" && ok "T5 a loose .env in the home root -> SPRAWL, exit 1" \ + || { no "T5 sprawl in the home root should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T6: the folder itself not 0700 -> FAIL +R="$(mkenv)"; chmod 755 "$R/${SITE}-creds"; rc=$(run "$R") +[ "$rc" = 1 ] && grep -q "folder mode 755" "$R/out" && ok "T6 a world-listable creds folder (0755) -> caught, exit 1" \ + || { no "T6 a non-0700 folder should fail (rc=$rc)"; sed 's/^/ /' "$R/out"; } +rm -rf "$R" + +# T7: the audit reads NO secret content -- no content-reading verbs in the script +grep -qE '\b(cat|head|tail|less|more|od|xxd|hexdump)\b' "$AUDIT" \ + && no "T7 the audit contains a content-reading verb -- it must inspect metadata only" \ + || ok "T7 metadata-only: no cat/head/tail/od/xxd in the audit" + +echo +if [ "$F" = 0 ]; then echo "creds-audit: $P/$P PASS"; exit 0; fi +echo "creds-audit: FAILURES: $F (passed $P)"; exit 1 diff --git a/tests/d115-office-carve/run-tests.sh b/tests/d115-office-carve/run-tests.sh new file mode 100755 index 0000000..d3a5242 --- /dev/null +++ b/tests/d115-office-carve/run-tests.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# tests/d115-office-carve/run-tests.sh +# +# Harness for netbox/d115-office-carve.py. OFFLINE -- touches no NetBox. +# +# The carve's VALUES are the point. Every CIDR in that script was MEASURED off +# the live Office1 build; D-115 blesses exactly what is running, so a "tidy-up" +# that changes one of these silently turns a zero-renumber carve into a +# renumber. This harness pins them. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/d115-office-carve.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" + +# no env -> must fail loud, not guess a target +NETBOX_URL= NETBOX_TOKEN= python3 "$S" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" + +# DRY BY DEFAULT (the D-117 lesson). +grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" +grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" + +# WRITING UPSTREAM IS GATED IN CODE, not by discipline. Feeding the production +# apex is an operator decision; it must never be a side effect of an import. +grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" --commit 2>&1)" +printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ + || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" + +# THE D-115 VALUES. Each was measured on the live build; changing one costs a renumber. +for cidr in \ + "10.10.0.0/16" "10.10.0.0/22" "10.10.0.0/24" "10.10.1.0/24" \ + "172.30.0.0/16" "172.30.1.0/24" \ + "2602:f3e2:f01:100::/56" "2602:f3e2:f01:100::/64"; do + grep -qF "\"$cidr\"" "$S" && ok || bad "D-115 prefix $cidr is GONE from the carve" +done + +# The two roles D-115 exists to create. `edge` is the NEW one -- v6 had "Edge +# Networks", v4 had nothing, so the WAN was squatting inside the OOB /12. +grep -q '"slug": "office"' "$S" && ok || bad "the office role is gone" +grep -q '"slug": "edge"' "$S" && ok || bad "the edge role is gone -- the WAN goes back to squatting in OOB" + +# The office KEEPS its number (D-117 operator ruling): site is vr1-off1, NOT off0. +grep -q '"slug": "vr1-off1"' "$S" && ok || bad "site slug is not vr1-off1 (D-117: the office keeps its number)" +grep -q 'vr1-off0' "$S" && bad "vr1-off0 appears -- the office was NOT renamed (D-117)" || ok + +# The DC v4 supernet must NOT be duplicated here -- it belongs to the six-plane +# importer. Two owners for one prefix is how they drift apart. +# (match the PREFIXES table, not the docstring that explains why it is absent) +grep -qE '^\s*\("10\.12\.64\.0/19"' "$S" \ + && bad "DC1's v4 supernet is duplicated here -- it belongs to dc-dc-prefixes-import.py" || ok + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "d115-office-carve: $pass/$total PASS"; exit 0; fi +echo "d115-office-carve: $fail/$total FAIL"; exit 1 diff --git a/tests/d120-compose-bands/run-tests.sh b/tests/d120-compose-bands/run-tests.sh new file mode 100755 index 0000000..b63bbf6 --- /dev/null +++ b/tests/d120-compose-bands/run-tests.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# tests/d120-compose-bands/run-tests.sh +# +# Harness for netbox/d120-compose-bands.py. OFFLINE -- touches no NetBox. +# +# Pins the D-120 band VALUES (a changed endpoint silently re-bands the /24), and the +# guard behaviour every sandbox-loop writer must have: dry by default, upstream-write +# gated IN CODE, and a whole-plan preflight so a bad plan cannot half-write. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/d120-compose-bands.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" + +# no env -> must fail loud, not guess a target +NETBOX_URL= NETBOX_TOKEN= python3 "$S" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" + +# DRY BY DEFAULT +grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" +grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" + +# UPSTREAM WRITE GATED IN CODE (not by discipline) +grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" --commit 2>&1)" +printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ + || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" + +# WHOLE-PLAN PREFLIGHT -- must validate the plan BEFORE any create (no half-writes) +grep -q "outside the parent" "$S" && ok || bad "lost the in-parent preflight" +grep -q "parent prefix .* absent" "$S" && ok || bad "lost the parent-exists preflight" + +# THE D-120 VALUES -- the 3 child ranges D-120 names for NetBox (static/dynamic/node) +for pair in \ + '"10.10.1.2/24", "10.10.1.49/24"' \ + '"10.10.1.100/24", "10.10.1.200/24"' \ + '"10.10.1.201/24", "10.10.1.254/24"'; do + grep -qF "$pair" "$S" && ok || bad "D-120 range [$pair] is GONE / changed" +done + +# The two re-IP'd service addresses (D-120: .201->.10, .202->.11) and their dns_names +grep -qF '"10.10.1.10/24"' "$S" && grep -q 'office1-netbox' "$S" && ok || bad "office1-netbox 10.10.1.10 assignment is gone" +grep -qF '"10.10.1.11/24"' "$S" && grep -q 'office1-tailscale' "$S" && ok || bad "office1-tailscale 10.10.1.11 assignment is gone" + +# The parent compose net must be the D-115 /24, not something invented +grep -qF 'PARENT = "10.10.1.0/24"' "$S" && ok || bad "parent compose net is not 10.10.1.0/24" + +# The node band must NOT still say .201/.202 for the services (the pre-re-IP snapshot) +grep -qE '"10\.10\.1\.20[12]/24", "active", "office1' "$S" && bad "a service is still addressed .201/.202 (pre-D-120 re-IP)" || ok + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "d120-compose-bands: $pass/$total PASS"; exit 0; fi +echo "d120-compose-bands: $fail/$total FAIL"; exit 1 diff --git a/tests/dc-dc-ceph-disk-budget/run-tests.sh b/tests/dc-dc-ceph-disk-budget/run-tests.sh index d43afcc..28284ab 100644 --- a/tests/dc-dc-ceph-disk-budget/run-tests.sh +++ b/tests/dc-dc-ceph-disk-budget/run-tests.sh @@ -19,10 +19,10 @@ fi } -FITS_ARGS=(--total-disk 10Ti --dc1-nodes 3 --dc1-per-node-osd 500G \ - --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3) -SHORT_ARGS=(--total-disk 2Ti --dc1-nodes 3 --dc1-per-node-osd 500G \ - --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3) +FITS_ARGS=(--total-disk 10Ti --vr1-dc0-nodes 3 --dc1-per-node-osd 500G \ + --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3) +SHORT_ARGS=(--total-disk 2Ti --vr1-dc0-nodes 3 --dc1-per-node-osd 500G \ + --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3) # --- constructed numeric example: size=3 + overhead FITS -------------------- # DC1 = 3*500G = 1.46TiB, DC2 same = 1.46TiB, combined ~2.92TiB, +30% overhead @@ -45,21 +45,21 @@ run 2 'REQUIRED' "T7 all args missing FAILS loud (rc 2)" run 2 '\-\-backup-overhead-fraction is REQUIRED' \ "T8 backup-overhead-fraction has NO default -- missing it alone FAILS loud" \ - --total-disk 10Ti --dc1-nodes 3 --dc1-per-node-osd 500G --dc2-nodes 3 --dc2-per-node-osd 500G + --total-disk 10Ti --vr1-dc0-nodes 3 --dc1-per-node-osd 500G --vr1-dc1-nodes 3 --dc2-per-node-osd 500G run 2 '\-\-total-disk is REQUIRED' "T9 missing --total-disk alone FAILS loud" \ - --dc1-nodes 3 --dc1-per-node-osd 500G --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 + --vr1-dc0-nodes 3 --dc1-per-node-osd 500G --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 # --- bad value formats fail loud -------------------------------------------- -run 2 'positive integer' "T10 non-numeric --dc1-nodes FAILS loud (rc 2)" \ - --total-disk 10Ti --dc1-nodes abc --dc1-per-node-osd 500G --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 +run 2 'positive integer' "T10 non-numeric --vr1-dc0-nodes FAILS loud (rc 2)" \ + --total-disk 10Ti --vr1-dc0-nodes abc --dc1-per-node-osd 500G --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 run 2 'non-negative decimal' "T11 non-numeric --backup-overhead-fraction FAILS loud (rc 2)" \ - --total-disk 10Ti --dc1-nodes 3 --dc1-per-node-osd 500G --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction abc + --total-disk 10Ti --vr1-dc0-nodes 3 --dc1-per-node-osd 500G --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction abc run 2 'integer byte count or Ki/Mi/Gi/Ti' "T12 bad size suffix on --total-disk FAILS loud (rc 2)" \ - --total-disk 10XB --dc1-nodes 3 --dc1-per-node-osd 500G --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 + --total-disk 10XB --vr1-dc0-nodes 3 --dc1-per-node-osd 500G --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 # --- size parsing: plain bytes and 1024-based suffixes agree ---------------- run 0 'verdict=PASS' "T13 plain-byte total-disk equivalent to 10Ti still PASSes" \ - --total-disk 10995116277760 --dc1-nodes 3 --dc1-per-node-osd 500G --dc2-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 + --total-disk 10995116277760 --vr1-dc0-nodes 3 --dc1-per-node-osd 500G --vr1-dc1-nodes 3 --dc2-per-node-osd 500G --backup-overhead-fraction 0.3 # --- --help works, exits 0, cites Section 3 / D-101 and the no-invented-fraction rule --- run 0 'Section 3' "T14 --help exits 0 and cites buildout-design Section 3" --help diff --git a/tests/dc-dc-dr-drill/run-tests.sh b/tests/dc-dc-dr-drill/run-tests.sh index fb42b11..1bd4664 100644 --- a/tests/dc-dc-dr-drill/run-tests.sh +++ b/tests/dc-dc-dr-drill/run-tests.sh @@ -32,31 +32,31 @@ # --- failover arg parsing --- run 1 '\-\-dc is required' "T4 failover missing --dc FAILS" failover -run 1 '\-\-dc must be dc1 or dc2' "T5 failover bad --dc token FAILS" failover --dc bogus --unit ceph-mon/0 --pool glance -run 1 '\-\-unit is required' "T6 failover missing --unit FAILS" failover --dc dc1 -run 1 'never assume .glance.' "T7 failover missing --pool FAILS, cites no-assume-glance" failover --dc dc1 --unit ceph-mon/0 +run 1 '\-\-dc must be vr1-dc0 or vr1-dc1' "T5 failover bad --dc token FAILS" failover --dc bogus --unit ceph-mon/0 --pool glance +run 1 '\-\-unit is required' "T6 failover missing --unit FAILS" failover --dc vr1-dc0 +run 1 'never assume .glance.' "T7 failover missing --pool FAILS, cites no-assume-glance" failover --dc vr1-dc0 --unit ceph-mon/0 # --- failback arg parsing --- run 1 '\-\-pool is required' "T8 failback missing --pool FAILS" failback run 1 '\-\-recovering-dc is required' "T9 failback missing --recovering-dc FAILS" failback --pool glance -run 1 '\-\-primary-dc is required' "T10 failback missing --primary-dc FAILS" failback --pool glance --recovering-dc dc1 -run 1 'must differ' "T11 failback recovering-dc == primary-dc FAILS" failback --pool glance --recovering-dc dc1 --primary-dc dc1 --recovering-unit u/0 --primary-unit u/1 -run 1 '\-\-recovering-unit is required' "T12 failback missing --recovering-unit FAILS" failback --pool glance --recovering-dc dc1 --primary-dc dc2 -run 1 '\-\-primary-unit is required' "T13 failback missing --primary-unit FAILS" failback --pool glance --recovering-dc dc1 --primary-dc dc2 --recovering-unit u/0 +run 1 '\-\-primary-dc is required' "T10 failback missing --primary-dc FAILS" failback --pool glance --recovering-dc vr1-dc0 +run 1 'must differ' "T11 failback recovering-dc == primary-dc FAILS" failback --pool glance --recovering-dc vr1-dc0 --primary-dc vr1-dc0 --recovering-unit u/0 --primary-unit u/1 +run 1 '\-\-recovering-unit is required' "T12 failback missing --recovering-unit FAILS" failback --pool glance --recovering-dc vr1-dc0 --primary-dc vr1-dc1 +run 1 '\-\-primary-unit is required' "T13 failback missing --primary-unit FAILS" failback --pool glance --recovering-dc vr1-dc0 --primary-dc vr1-dc1 --recovering-unit u/0 # --- failover: --confirmed-down gates --apply, but not the dry-run plan print --- -FO_DRY="$(bash "$SCRIPT" failover --dc dc1 --unit ceph-mon/0 --pool glance 2>&1)"; FO_DRY_RC=$? +FO_DRY="$(bash "$SCRIPT" failover --dc vr1-dc0 --unit ceph-mon/0 --pool glance 2>&1)"; FO_DRY_RC=$? [[ "$FO_DRY_RC" == 0 ]] && grep -q 'promote --force glance' <<<"$FO_DRY" && grep -q 'confirmed-down is required' <<<"$FO_DRY" \ && { echo " PASS T14 failover dry-run (no --confirmed-down) still prints the plan, rc 0"; PASS=$((PASS+1)); } || { echo " FAIL T14"; echo "$FO_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 1 'confirmed-down is required' "T15 failover --apply WITHOUT --confirmed-down is BLOCKED (rc 1)" failover --dc dc1 --unit ceph-mon/0 --pool glance --apply +run 1 'confirmed-down is required' "T15 failover --apply WITHOUT --confirmed-down is BLOCKED (rc 1)" failover --dc vr1-dc0 --unit ceph-mon/0 --pool glance --apply -FO_REREG="$(bash "$SCRIPT" failover --dc dc1 --unit ceph-mon/0 --pool glance --confirmed-down --reregister-image img1 2>&1)" +FO_REREG="$(bash "$SCRIPT" failover --dc vr1-dc0 --unit ceph-mon/0 --pool glance --confirmed-down --reregister-image img1 2>&1)" grep -q 'NEVER EXECUTED BY THIS SCRIPT' <<<"$FO_REREG" && grep -q 'img1' <<<"$FO_REREG" \ && { echo " PASS T16 failover re-registration step is printed but flagged manual (never auto-executed)"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$FO_REREG" | sed 's/^/ /'; FAIL=$((FAIL+1)); } # --- failback: the safety-critical step ORDER, verified by LINE NUMBER --- -FB_OUT="$(bash "$SCRIPT" failback --pool glance --recovering-dc dc1 --recovering-unit rec/0 --primary-dc dc2 --primary-unit pri/0 2>&1)"; FB_RC=$? +FB_OUT="$(bash "$SCRIPT" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 2>&1)"; FB_RC=$? [[ "$FB_RC" == 0 ]] && { echo " PASS T17 failback dry-run (dc1 recovering, dc2 primary) exits 0"; PASS=$((PASS+1)); } || { echo " FAIL T17"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } L_112=$(grep -n '11.2 (demote recovering' <<<"$FB_OUT" | head -1 | cut -d: -f1) @@ -72,21 +72,21 @@ && grep -A0 '11.4b' <<<"$FB_OUT" | grep -q 'promote glance' \ && { echo " PASS T19 11.4a is a demote, 11.4b is a promote (not swapped)"; PASS=$((PASS+1)); } || { echo " FAIL T19"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -FB_SKIP="$(bash "$SCRIPT" failback --pool glance --recovering-dc dc1 --recovering-unit rec/0 --primary-dc dc2 --primary-unit pri/0 --skip-11-4 2>&1)" +FB_SKIP="$(bash "$SCRIPT" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 --skip-11-4 2>&1)" grep -q '11.4 SKIPPED' <<<"$FB_SKIP" && ! grep -q '11.4a' <<<"$FB_SKIP" \ && { echo " PASS T20 --skip-11-4 stops the plan after 11.2/11.3"; PASS=$((PASS+1)); } || { echo " FAIL T20"; echo "$FB_SKIP" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run (dc2 involved), blocking before --apply --- -grep -q 'gate: FAILED' <<<"$FB_OUT" && grep -q 'gate: OK (dc1)' <<<"$FB_OUT" \ - && { echo " PASS T21 failback dry-run reports dc1 OK / dc2 FAILED, still prints full plan"; PASS=$((PASS+1)); } || { echo " FAIL T21"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: informational in dry-run (vr1-dc1 involved), blocking before --apply --- +grep -q 'gate: FAILED' <<<"$FB_OUT" && grep -q 'gate: OK (vr1-dc0)' <<<"$FB_OUT" \ + && { echo " PASS T21 failback dry-run reports vr1-dc0 OK / vr1-dc1 FAILED, still prints full plan"; PASS=$((PASS+1)); } || { echo " FAIL T21"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T22 failback --apply is BLOCKED by the dc2 gate (rc 3)" failback --pool glance --recovering-dc dc1 --recovering-unit rec/0 --primary-dc dc2 --primary-unit pri/0 --apply --no-prompt +run 3 '\$DC gate refused' "T22 failback --apply is BLOCKED by the dc2 gate (rc 3)" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 --apply --no-prompt # --- --apply never runs without --apply: no juju invocation attempted in dry-run --- if command -v juju >/dev/null 2>&1; then echo " SKIP T23 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" else - run 2 'juju required on PATH' "T23 failover --apply (dc1 gate OK) but juju absent FAILS (rc 2)" failover --dc dc1 --unit ceph-mon/0 --pool glance --confirmed-down --apply + run 2 'juju required on PATH' "T23 failover --apply (dc1 gate OK) but juju absent FAILS (rc 2)" failover --dc vr1-dc0 --unit ceph-mon/0 --pool glance --confirmed-down --apply fi echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" diff --git a/tests/dc-dc-prefixes-import/fake_pynetbox.py b/tests/dc-dc-prefixes-import/fake_pynetbox.py index e9e8013..7fa7bc0 100644 --- a/tests/dc-dc-prefixes-import/fake_pynetbox.py +++ b/tests/dc-dc-prefixes-import/fake_pynetbox.py @@ -44,6 +44,16 @@ matches = [i for i in self._items if self._matches(i, filters)] if not matches: return None + # REAL pynetbox RAISES on a multi-match. This fake used to return matches[0], + # which meant the harness DIVERGED FROM REALITY AT EXACTLY THE FAILURE POINT: + # NetBox permits duplicate prefixes (this importer's own docstring anticipates + # them against vr0-dc0), and a global .get(prefix=...) would blow up mid-loop + # in production while the tests sailed through. A fake that is kinder than the + # real thing hides the bug it exists to catch. + if len(matches) > 1: + raise ValueError( + f"get() returned more than one result ({len(matches)}). " + f"Filters: {filters}. Use .filter() and scope the lookup.") return matches[0] def filter(self, **filters): @@ -72,6 +82,13 @@ self.vlan_groups = Collection() +class _HttpSession: + """Stands in for pynetbox.api().http_session (a requests.Session).""" + + def __init__(self): + self.headers = {} + + class FakeApi: """Stands in for pynetbox.api(url, token=...).""" @@ -80,6 +97,9 @@ self.token = token self.dcim = _Dcim() self.ipam = _Ipam() + # Mirrors pynetbox.api's real .http_session (a requests.Session): get_nb() + # sets the WAF-safe User-Agent on it. A .headers dict is enough to record it. + self.http_session = _HttpSession() def status(self): return {"netbox-version": "fake"} diff --git a/tests/dc-dc-prefixes-import/test_logic.py b/tests/dc-dc-prefixes-import/test_logic.py index 13821a5..329ba5b 100644 --- a/tests/dc-dc-prefixes-import/test_logic.py +++ b/tests/dc-dc-prefixes-import/test_logic.py @@ -20,6 +20,15 @@ import os import sys +import faulthandler + +# Watchdog: self-abort WITH a traceback after 30s instead of hanging the whole +# run-tests-all.sh gauntlet if any test ever blocks. This is exactly how the +# carve_gua /40 subnet-enumeration hang (DOCFIX-181) was diagnosed; now a future +# blocking regression fails loudly (non-zero exit + stack dump) rather than +# stalling the gauntlet indefinitely. +faulthandler.dump_traceback_later(30, exit=True) + HERE = os.path.dirname(os.path.abspath(__file__)) REPO_ROOT = os.path.dirname(os.path.dirname(HERE)) STUB_SYSPATH = os.path.join(HERE, "stub_syspath") @@ -104,52 +113,73 @@ expect_dies("gua unrelated GUA block rejected", T.validate_gua_prefix, "2001:db8::/40") # ----------------------------------------------------------------------------- -# validate_dc2_supernet +# validate_dc1_supernet # ----------------------------------------------------------------------------- -dc2_super = expect_ok("dc2 supernet valid /19 no overlap", T.validate_dc2_supernet, "10.13.0.0/19") -expect_dies("dc2 supernet too small (/20) rejected", T.validate_dc2_supernet, "10.13.0.0/20") -expect_dies("dc2 supernet overlapping DC1 rejected", T.validate_dc2_supernet, "10.12.0.0/19") -expect_dies("dc2 supernet IPv6 input rejected", T.validate_dc2_supernet, "fd00::/19") +dc2_super = expect_ok("second-DC supernet valid /19 no overlap", T.validate_vr1_dc1_supernet, "10.12.64.0/19") +expect_dies("second-DC supernet too small (/20) rejected", T.validate_vr1_dc1_supernet, "10.12.64.0/20") +expect_dies("second-DC supernet overlapping VR1 DC0 rejected", T.validate_vr1_dc1_supernet, "10.12.0.0/19") +expect_dies("second-DC supernet IPv6 input rejected", T.validate_vr1_dc1_supernet, "fd00::/19") # ----------------------------------------------------------------------------- -# carve_dc2_v4 +# carve_dc1_v4 # ----------------------------------------------------------------------------- if dc2_super is not None: - v4 = T.carve_dc2_v4(dc2_super) - check(set(v4.keys()) == set(T.PLANE_ORDER), "carve_dc2_v4 covers all six planes") + v4 = T.carve_vr1_dc1_v4(dc2_super) + check(set(v4.keys()) == set(T.PLANE_ORDER), "carve_vr1_dc1_v4 covers all six planes") all_22 = all(ipaddress.ip_network(c).prefixlen == 22 for c in v4.values()) - check(all_22, "carve_dc2_v4 all /22") + check(all_22, "carve_vr1_dc1_v4 all /22") within = all(ipaddress.ip_network(c).subnet_of(dc2_super) for c in v4.values()) - check(within, "carve_dc2_v4 all within supernet") + check(within, "carve_vr1_dc1_v4 all within supernet") distinct = len(set(v4.values())) == len(v4) - check(distinct, "carve_dc2_v4 all distinct") + check(distinct, "carve_vr1_dc1_v4 all distinct") # ----------------------------------------------------------------------------- -# carve_ula +# carve_v6 (D-111 NN layout: /60+/64 per plane, deployed net-byte mnemonic) # ----------------------------------------------------------------------------- org_ula = ipaddress.ip_network("fd12:3456:789a::/48") -ula_dc1 = T.carve_ula(org_ula, "dc1") -ula_dc2 = T.carve_ula(org_ula, "dc2") -expected_ula_planes = {p for p in T.PLANE_ORDER if T.PLANE_V6_FAMILY[p] in ("ula", "ula-only")} -check(set(ula_dc1.keys()) == expected_ula_planes, "carve_ula dc1 covers exactly the ULA-bearing planes") -check(set(ula_dc2.keys()) == expected_ula_planes, "carve_ula dc2 covers exactly the ULA-bearing planes") -all_64_dc1 = all(ipaddress.ip_network(c).prefixlen == 64 for c in ula_dc1.values()) -check(all_64_dc1, "carve_ula dc1 all /64") +gua_dc0 = ipaddress.ip_network("2602:f3e2:f02::/48") +recs1 = T.carve_v6("vr1-dc0", org_ula, gua_dc0) +recs2 = T.carve_v6("vr1-dc1", org_ula, ipaddress.ip_network("2602:f3e2:f03::/48")) + +# structure: 12 records (provider 3, metal-admin 2, metal-internal 1, data/storage/repl 2 each) +check(len(recs1) == 12, "carve_v6 VR1-DC0 yields 12 records", str(len(recs1))) +roles1 = [r for _, r, _ in recs1] +check(roles1.count("provider-public") == 3, "carve_v6 provider-public has 3 (/60,/64,VIP)") +check(roles1.count("metal-admin") == 2, "carve_v6 metal-admin has 2 (/60,/64)") +check(roles1.count("metal-internal") == 1, "carve_v6 metal-internal has 1 (/64 in metal /60)") +for role in ("data-tenant", "storage", "replication"): + check(roles1.count(role) == 2, f"carve_v6 {role} has 2 (/60,/64)") + +# family matrix: provider-public is GUA (within its /48); the rest are ULA (fd..) +addrs1 = {c for c, _, _ in recs1} +prov = [ipaddress.ip_network(c) for c, r, _ in recs1 if r == "provider-public"] +check(all(n.subnet_of(gua_dc0) for n in prov), "carve_v6 provider-public within the GUA /48") +nonprov = [ipaddress.ip_network(c) for c, r, _ in recs1 if r != "provider-public"] +check(all(str(n.network_address).startswith("fd") for n in nonprov), "carve_v6 non-provider planes are ULA") + +# NN offsets land on the deployed mnemonic; DC.NN reads in the 4th hextet +check("2602:f3e2:f02:10::/60" in addrs1, "carve_v6 VR1-DC0 provider GUA /60 at :10") +check("2602:f3e2:f02:11::/64" in addrs1, "carve_v6 VR1-DC0 provider API-VIP /64 at :11") +check("fd12:3456:789a:220::/64" in addrs1, "carve_v6 VR1-DC0 metal-admin ULA /64 at DC.NN :220") +check("fd12:3456:789a:221::/64" in addrs1, "carve_v6 VR1-DC0 metal-internal ULA /64 at :221") +check("fd12:3456:789a:230::/64" in addrs1, "carve_v6 VR1-DC0 data-tenant ULA /64 at :230") +check("fd12:3456:789a:250::/64" in addrs1, "carve_v6 VR1-DC0 replication ULA /64 at :250") +m60 = ipaddress.ip_network("fd12:3456:789a:220::/60") +check(ipaddress.ip_network("fd12:3456:789a:221::/64").subnet_of(m60), "carve_v6 metal-internal /64 sits inside metal /60") + +# VR1 DC1 uses the :3NN nibble; the two DCs never overlap +addrs2 = {c for c, _, _ in recs2} +check("fd12:3456:789a:320::/64" in addrs2, "carve_v6 VR1-DC1 metal-admin at :320 (DC nibble 3)") +check("2602:f3e2:f03:10::/64" in addrs2, "carve_v6 VR1-DC1 provider GUA at f03:10") no_overlap = not any( ipaddress.ip_network(a).overlaps(ipaddress.ip_network(b)) - for a in ula_dc1.values() for b in ula_dc2.values() + for a, _, _ in recs1 for b, _, _ in recs2 ) -check(no_overlap, "carve_ula dc1/dc2 non-overlapping") +check(no_overlap, "carve_v6 vr1-dc0/vr1-dc1 non-overlapping") -# ----------------------------------------------------------------------------- -# carve_gua -# ----------------------------------------------------------------------------- -if gua is not None: - gua_carved = T.carve_gua(gua) - check(set(gua_carved.keys()) == {"provider-public"}, "carve_gua only provider-public") - g64 = ipaddress.ip_network(gua_carved["provider-public"]) - check(g64.prefixlen == 64, "carve_gua result is /64") - check(g64.subnet_of(gua), "carve_gua result within input prefix") +# DOCFIX-181 no-hang property preserved: a /40 GUA computes by math, not enumeration +recs_big = T.carve_v6("vr1-dc1", org_ula, ipaddress.ip_network("2602:f3e2:1000::/40")) +check(len(recs_big) == 12, "carve_v6 with a /40 GUA completes fast (no subnet enumeration)") # ----------------------------------------------------------------------------- # require_env @@ -181,77 +211,286 @@ sys.argv = old_argv -os.environ["NETBOX_URL"] = "https://fake.example/netbox" +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # the known sandbox (guard passes) os.environ["NETBOX_TOKEN"] = "fake-token" os.environ["ORG_ULA_48"] = "fd12:3456:789a::/48" fake1 = fake_pynetbox.FakeApi() fake1.preseed_roles(T.PLANE_ORDER) -os.environ["DC_GUA_PREFIX"] = "2602:f3e2:1000::/40" +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f02::/48" # apex: vr1-dc0 +for _v in ("DC1_V4_SUPERNET", "DC2_V4_SUPERNET", "VR1_DC1_V4_SUPERNET"): + os.environ.pop(_v, None) # both old names RETIRED by D-119 +# --- D-117: DRY RUN IS THE DEFAULT. This is the guard that matters most: the +# --- tool used to write with no flag at all, straight into the IPAM apex. +with captured_stdout() as out0: + rc0 = run_main(["--dc", "vr1-dc0"], fake1) +check(rc0 == 0, "main vr1-dc0 DRY RUN rc==0", str(rc0)) +check(out0.getvalue().count("CREATED prefix") == 0, + "D-117: dry run (no --commit) CREATES NOTHING") +check(out0.getvalue().count("[dry-run] would CREATE prefix") == 18, + "D-117: dry run PLANS all 18 prefixes", + str(out0.getvalue().count("[dry-run] would CREATE prefix"))) +check("[dry-run] would CREATE site" in out0.getvalue(), + "D-117: dry run PLANS the site too (fresh-NetBox preview must not bail)") +check("DRY RUN" in out0.getvalue(), "D-117: dry run says so, loudly") +check(len(fake1.ipam.prefixes._items) == 0, + "D-117: dry run left the NetBox EMPTY (no side effects)", + str(len(fake1.ipam.prefixes._items))) + +# --- now commit for real with captured_stdout() as out1: - rc1 = run_main(["--dc", "dc1"], fake1) -check(rc1 == 0, "main dc1 first run rc==0", str(rc1)) + rc1 = run_main(["--dc", "vr1-dc0", "--commit"], fake1) +check(rc1 == 0, "main vr1-dc0 --commit rc==0", str(rc1)) created_count = out1.getvalue().count("CREATED prefix") -check(created_count == 12, "main dc1 first run creates 12 prefixes (6 v4 + 5 ula + 1 gua)", str(created_count)) -check("vr0-dc0" in out1.getvalue(), "main dc1 prints the vr0-dc0 collision note") +check(created_count == 18, "main vr1-dc0 --commit creates 18 prefixes (6 v4 + 12 v6)", str(created_count)) +check("vr0-dc0" in out1.getvalue(), "main vr1-dc0 prints the vr0-dc0 (VR0 rehearsal) collision note") +dc0_site = fake1.dcim.sites.get(slug="vr1-dc0") +check(dc0_site is not None, "D-119: --dc vr1-dc0 created the APEX site vr1-dc0 (identity)") +check(fake1.dcim.sites.get(slug="vr1-dc1") is None, + "D-119: --dc vr1-dc0 did NOT touch vr1-dc1 -- the off-by-one is dead") with captured_stdout() as out2: - rc2 = run_main(["--dc", "dc1"], fake1) -check(rc2 == 0, "main dc1 second run (idempotent) rc==0") -check(out2.getvalue().count("CREATED prefix") == 0, "main dc1 second run creates nothing new") -check(out2.getvalue().count("EXISTS prefix") == 12, "main dc1 second run reports all 12 as EXISTS") + rc2 = run_main(["--dc", "vr1-dc0", "--commit"], fake1) +check(rc2 == 0, "main vr1-dc0 second run (idempotent) rc==0") +check(out2.getvalue().count("CREATED prefix") == 0, "main vr1-dc0 second run creates nothing new") +check(out2.getvalue().count("EXISTS prefix") == 18, "main vr1-dc0 second run reports all 18 as EXISTS") # ----------------------------------------------------------------------------- -# main() end-to-end -- dc2 happy path (fresh site, distinct v4/GUA) +# main() end-to-end -- VR1 DC1 (the SECOND DC): fresh site, distinct v4/GUA # ----------------------------------------------------------------------------- fake2 = fake_pynetbox.FakeApi() fake2.preseed_roles(T.PLANE_ORDER) -os.environ["DC_GUA_PREFIX"] = "2602:f3e2:2000::/40" -os.environ["DC2_V4_SUPERNET"] = "10.13.0.0/19" +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f03::/48" # apex: vr1-dc1 +os.environ["VR1_DC1_V4_SUPERNET"] = "10.12.64.0/19" # D-115 with captured_stdout() as out3: - rc3 = run_main(["--dc", "dc2"], fake2) -check(rc3 == 0, "main dc2 run rc==0", str(rc3)) -check(out3.getvalue().count("CREATED prefix") == 12, "main dc2 run creates 12 prefixes", str(out3.getvalue().count("CREATED prefix"))) -dc2_site = fake2.dcim.sites.get(slug="vr1-dc2") -check(dc2_site is not None, "main dc2 creates the vr1-dc2 site") + rc3 = run_main(["--dc", "vr1-dc1", "--commit"], fake2) +check(rc3 == 0, "main vr1-dc1 (second DC) run rc==0", str(rc3)) +check(out3.getvalue().count("CREATED prefix") == 18, "main vr1-dc1 creates 18 prefixes", + str(out3.getvalue().count("CREATED prefix"))) +check(fake2.dcim.sites.get(slug="vr1-dc1") is not None, "main vr1-dc1 binds to the APEX site vr1-dc1") +check(fake2.dcim.sites.get(slug="vr1-dc2") is None, "D-119: no phantom vr1-dc2 site is ever created") # ----------------------------------------------------------------------------- -# main() -- dc2 with DC2_V4_SUPERNET missing FAILS LOUD +# main() -- second DC with DC1_V4_SUPERNET missing FAILS LOUD # ----------------------------------------------------------------------------- -del os.environ["DC2_V4_SUPERNET"] +del os.environ["VR1_DC1_V4_SUPERNET"] fake3 = fake_pynetbox.FakeApi() fake3.preseed_roles(T.PLANE_ORDER) try: - run_main(["--dc", "dc2"], fake3) - no("main dc2 without DC2_V4_SUPERNET fails loud", "did not raise SystemExit") + with captured_stdout(): + run_main(["--dc", "vr1-dc1", "--commit"], fake3) + no("main vr1-dc1 without VR1_DC1_V4_SUPERNET fails loud", "did not raise SystemExit") except SystemExit: - ok("main dc2 without DC2_V4_SUPERNET fails loud") + ok("main vr1-dc1 without VR1_DC1_V4_SUPERNET fails loud") # ----------------------------------------------------------------------------- -# argparse: missing/invalid --dc +# D-119: BOTH retired env var names are rejected, not silently ignored +# ----------------------------------------------------------------------------- +os.environ["DC2_V4_SUPERNET"] = "10.12.64.0/19" # muscle memory +os.environ.pop("DC1_V4_SUPERNET", None) +try: + with captured_stdout(): + run_main(["--dc", "vr1-dc1", "--commit"], fake_pynetbox.FakeApi()) + no("D-119: retired DC2_V4_SUPERNET is rejected by name") +except SystemExit: + ok("D-119: retired DC2_V4_SUPERNET is rejected by name") +os.environ.pop("DC2_V4_SUPERNET", None) + +# ----------------------------------------------------------------------------- +# argparse: missing/invalid --dc (dc2 is now an INVALID choice -- D-117) # ----------------------------------------------------------------------------- fake4 = fake_pynetbox.FakeApi() fake4.preseed_roles(T.PLANE_ORDER) -os.environ["DC2_V4_SUPERNET"] = "10.13.0.0/19" +os.environ["VR1_DC1_V4_SUPERNET"] = "10.12.64.0/19" try: - run_main([], fake4) + with captured_stdout(): + run_main([], fake4) no("main missing --dc fails loud") except SystemExit: ok("main missing --dc fails loud") +# D-119: EVERY bare dcN is now an invalid choice. They were ambiguous across +# regions ('dc0' = VR0's live cloud in lib-net.sh, VR1's first DC here). +for _bare in ("dc0", "dc1", "dc2"): + try: + with captured_stdout(): + run_main(["--dc", _bare], fake4) + no(f"D-119: bare --dc {_bare} is REJECTED (retired, cross-region ambiguous)") + except SystemExit: + ok(f"D-119: bare --dc {_bare} is REJECTED (retired, cross-region ambiguous)") + try: - run_main(["--dc", "dc3"], fake4) + with captured_stdout(): + run_main(["--dc", "dc3"], fake4) no("main invalid --dc choice fails loud") except SystemExit: ok("main invalid --dc choice fails loud") # ----------------------------------------------------------------------------- +# D-119 BREAK-2: DC_GUA_PREFIX must AGREE with --dc. +# +# Without this guard, `--dc vr1-dc0 DC_GUA_PREFIX=<f03::/48>` was ACCEPTED: it +# writes the SECOND DC's GUA prefixes, carved with the FIRST DC's ULA nibble +# (DC_V6_INDEX is keyed off --dc, independently), all scoped to the FIRST DC's +# site. A silently mis-bound datacenter assembled from two disagreeing sources. +# Identity-mapping the site slug does NOT close this -- it is one layer down. +# ----------------------------------------------------------------------------- +fake5 = fake_pynetbox.FakeApi() +fake5.preseed_roles(T.PLANE_ORDER) +os.environ["ORG_ULA_48"] = "fd50:840e:74e2::/48" +os.environ["VR1_DC1_V4_SUPERNET"] = "10.12.64.0/19" + +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f03::/48" # the SECOND DC's block... +try: + with captured_stdout(): + run_main(["--dc", "vr1-dc0", "--commit"], fake5) # ...but the FIRST DC's site + no("D-119 BREAK-2: mismatched DC_GUA_PREFIX/--dc is REJECTED") +except SystemExit: + ok("D-119 BREAK-2: mismatched DC_GUA_PREFIX/--dc is REJECTED") +check(fake5.dcim.sites.get(slug="vr1-dc0") is None, + "D-119 BREAK-2: the mismatched run wrote NOTHING (no site created)") + +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f02::/48" # the FIRST DC's block... +try: + with captured_stdout(): + run_main(["--dc", "vr1-dc1", "--commit"], fake_pynetbox.FakeApi()) # ...second DC's site + no("D-119 BREAK-2: the reverse mismatch is REJECTED too") +except SystemExit: + ok("D-119 BREAK-2: the reverse mismatch is REJECTED too") + +# The MATCHING pair still works -- the guard must not be a blanket refusal. +fake6 = fake_pynetbox.FakeApi() +fake6.preseed_roles(T.PLANE_ORDER) +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f02::/48" +with captured_stdout(): + rc_ok = run_main(["--dc", "vr1-dc0", "--commit"], fake6) +check(rc_ok == 0, "D-119 BREAK-2: the MATCHING --dc/DC_GUA_PREFIX pair still succeeds") + +# ----------------------------------------------------------------------------- +# D-119 BREAK-1: descriptions must not munge the selector into the label. +# +# The old code did f"VR1 {dc.upper()} ..." -- fine when dc was "dc0", but under +# D-119 dc is "vr1-dc0", which renders "VR1 VR1-DC0 provider-public". Same defect +# class as the original bug: DERIVING a label from a token instead of looking it +# up. It hid in the description field, where slug-focused review missed it. +# ----------------------------------------------------------------------------- +descs = [p.description for p in fake6.ipam.prefixes.all()] +check(any(d.startswith("VR1 DC0 ") for d in descs), + "D-119 BREAK-1: descriptions read 'VR1 DC0 ...' (from SITES[name], looked up)") +check(not any("VR1-DC0" in d.upper().replace("VR1 DC0", "") for d in descs), + "D-119 BREAK-1: no description contains the munged 'VR1 VR1-DC0'") + +# ----------------------------------------------------------------------------- +# WHOLE-PLAN ROLE PREFLIGHT (2026-07-14 hardening). +# +# find_role() used to be called lazily INSIDE the write loop, so against an apex +# missing a six-plane role a --commit would create the site + the first prefixes, +# then die mid-loop: a HALF-WRITTEN datacenter in the production IPAM. Every other +# test in this file preseeds ALL roles, so they exercise the preflight only when it +# PASSES -- silently. This exercises it FAILING, and asserts it wrote NOTHING. +# Without this, the preflight could be deleted and the harness would stay green. +# ----------------------------------------------------------------------------- +fake_missing = fake_pynetbox.FakeApi() +fake_missing.preseed_roles([r for r in T.PLANE_ORDER if r != "storage"]) # one role short +os.environ["ORG_ULA_48"] = "fd50:840e:74e2::/48" +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f02::/48" +try: + with captured_stdout() as _out: + run_main(["--dc", "vr1-dc0", "--commit"], fake_missing) + no("preflight: a missing IPAM role is REJECTED before any write") +except SystemExit: + ok("preflight: a missing IPAM role is REJECTED before any write") +check(fake_missing.dcim.sites.get(slug="vr1-dc0") is None, + "preflight: the rejected run created NO site (no half-written datacenter)") +check(sum(1 for _ in fake_missing.ipam.prefixes.all()) == 0, + "preflight: the rejected run created NO prefixes") + +# ----------------------------------------------------------------------------- +# UPSTREAM-WRITE GUARD (2026-07-14). This tool had NO hostname gate; the WAF 403 on +# the default User-Agent was the only thing stopping an accidental --commit to the +# production apex, and that accidental-safety was removed by the UA fix. So a +# --commit to a NON-sandbox host must now REFUSE without --yes-write-upstream. +# ----------------------------------------------------------------------------- +fake_up = fake_pynetbox.FakeApi() +fake_up.preseed_roles(T.PLANE_ORDER) +os.environ["NETBOX_URL"] = "https://netbox.baldurkeep.com" # the PRODUCTION apex +os.environ["ORG_ULA_48"] = "fd50:840e:74e2::/48" +os.environ["DC_GUA_PREFIX"] = "2602:f3e2:f02::/48" +try: + with captured_stdout(): + run_main(["--dc", "vr1-dc0", "--commit"], fake_up) + no("guard: --commit to the production apex WITHOUT --yes-write-upstream is REFUSED") +except SystemExit: + ok("guard: --commit to the production apex WITHOUT --yes-write-upstream is REFUSED") +check(fake_up.dcim.sites.get(slug="vr1-dc0") is None, + "guard: the refused upstream --commit wrote NOTHING") + +# WITH the flag, the same upstream --commit proceeds (guard is an opt-in gate, not a wall). +fake_up2 = fake_pynetbox.FakeApi() +fake_up2.preseed_roles(T.PLANE_ORDER) +with captured_stdout(): + rc_up = run_main(["--dc", "vr1-dc0", "--commit", "--yes-write-upstream"], fake_up2) +check(rc_up == 0, "guard: --commit --yes-write-upstream to the apex is ALLOWED") +check(fake_up2.dcim.sites.get(slug="vr1-dc0") is not None, + "guard: the authorized upstream --commit DID write") + +# A sandbox host needs NO flag -- the default C2-safe path for the sim. +fake_sb = fake_pynetbox.FakeApi() +fake_sb.preseed_roles(T.PLANE_ORDER) +os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # known sandbox +with captured_stdout(): + rc_sb = run_main(["--dc", "vr1-dc0", "--commit"], fake_sb) +check(rc_sb == 0, "guard: --commit to a SANDBOX host needs no flag") + +# ----------------------------------------------------------------------------- # Summary # ----------------------------------------------------------------------------- print() + +# --------------------------------------------------------------------------- +# D-117 REGRESSION GUARDS. These are the whole reason this rename happened: +# the tool bound VR1 DC0's prefixes to the site vr1-dc1 (the OTHER DC), and it +# WROTE BY DEFAULT, so it would have landed silently in the IPAM apex. +# The slugs below are the APEX's, measured live 2026-07-13. Do not "fix" them. +# --------------------------------------------------------------------------- +# THE D-119 STRUCTURAL INVARIANT: the selector IS the slug. If a future edit +# reintroduces an offset table, this check fails -- the bug becomes untypeable. +check(all(k == v["slug"] for k, v in T.SITES.items()), + "D-119: SITES is an IDENTITY map -- the --dc selector IS the apex site slug") +check(set(T.SITES) == {"vr1-dc0", "vr1-dc1"}, "D-119: only the two apex slugs are selectable") +check(not any(k in T.SITES for k in ("dc0", "dc1", "dc2")), + "D-119: every bare dcN selector is GONE from SITES") +check(T.DC_V6_INDEX["vr1-dc0"] == 0x02, "D-119: vr1-dc0 -> GUA site nibble 02 (== apex f02::/48)") +check(T.DC_V6_INDEX["vr1-dc1"] == 0x03, "D-119: vr1-dc1 -> GUA site nibble 03 (== apex f03::/48)") +check(T.EXPECTED_GUA == {"vr1-dc0": "2602:f3e2:f02::/48", "vr1-dc1": "2602:f3e2:f03::/48"}, + "D-119: EXPECTED_GUA pins the apex's MEASURED DC->GUA binding") + +# dry-by-default: --commit must exist, and the writers must take a commit flag. +import inspect +_src = inspect.getsource(T) +check("--commit" in _src, "dry-by-default: --commit flag exists") +check("commit" in inspect.signature(T.create_or_report_prefix).parameters, + "dry-by-default: create_or_report_prefix is commit-gated") +check("commit" in inspect.signature(T.find_or_create_site).parameters, + "dry-by-default: find_or_create_site is commit-gated (a site is a WRITE)") +check("DC2_V4_SUPERNET" in _src and "DC1_V4_SUPERNET" in _src and "RETIRED (D-119)" in _src, + "D-119: BOTH retired supernet env vars are rejected by name, not ignored") + +# WAF trap (references/platform-traps.md): get_nb() MUST set the accepted +# User-Agent on the pynetbox requests session, or every upstream call 403s and +# this importer could never write to the apex (only the WAF-less sandbox). +# Asserted at RUNTIME: get_nb() runs against the fake, which records the header. +_fake_ua = fake_pynetbox.FakeApi() +pynetbox.api = lambda url=None, token=None: _fake_ua +os.environ["NETBOX_URL"] = "https://netbox.baldurkeep.com" +os.environ["NETBOX_TOKEN"] = "x" +T.get_nb() +check(_fake_ua.http_session.headers.get("User-Agent") == "curl/8.5.0", + "WAF: get_nb sets the accepted User-Agent on the pynetbox session (runtime)") + if F == 0: print(f"ALL PASS ({P} checks)") sys.exit(0) diff --git a/tests/dc-dc-radosgw-multisite/run-tests.sh b/tests/dc-dc-radosgw-multisite/run-tests.sh index 1f36502..e570c14 100644 --- a/tests/dc-dc-radosgw-multisite/run-tests.sh +++ b/tests/dc-dc-radosgw-multisite/run-tests.sh @@ -39,33 +39,33 @@ run 1 'unknown subcommand' "T2 bogus subcommand FAILS (rc 1)" bogus run 0 'usage:' "T3 --help exits 0 with usage" --help run 1 '\-\-dc is required' "T4 master-init missing --dc FAILS (rc 1)" master-init -run 1 '\-\-dc must be dc1 or dc2' "T5 master-init bad --dc token FAILS" master-init --dc bogus --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z --endpoint http://x:80 -run 1 '\-\-unit is required' "T6 master-init missing --unit FAILS" master-init --dc dc1 -run 1 '\-\-realm is required' "T7 master-init missing --realm FAILS" master-init --dc dc1 --unit ceph-radosgw/0 --zonegroup ZG --zone Z --endpoint http://x:80 -run 1 '\-\-access-key is required' "T8 join-readonly missing --access-key FAILS" join-readonly --dc dc1 --unit ceph-radosgw/0 --zonegroup ZG --zone Z --endpoint http://x:80 --secret S -run 1 '\-\-secret is required' "T9 join-readonly missing --secret FAILS" join-readonly --dc dc1 --unit ceph-radosgw/0 --zonegroup ZG --zone Z --endpoint http://x:80 --access-key K -run 1 '\-\-zone is required' "T10 enable-two-way missing --zone FAILS" enable-two-way --dc dc1 --unit ceph-radosgw/0 +run 1 '\-\-dc must be vr1-dc0 or vr1-dc1' "T5 master-init bad --dc token FAILS" master-init --dc bogus --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z --endpoint http://x:80 +run 1 '\-\-unit is required' "T6 master-init missing --unit FAILS" master-init --dc vr1-dc0 +run 1 '\-\-realm is required' "T7 master-init missing --realm FAILS" master-init --dc vr1-dc0 --unit ceph-radosgw/0 --zonegroup ZG --zone Z --endpoint http://x:80 +run 1 '\-\-access-key is required' "T8 join-readonly missing --access-key FAILS" join-readonly --dc vr1-dc0 --unit ceph-radosgw/0 --zonegroup ZG --zone Z --endpoint http://x:80 --secret S +run 1 '\-\-secret is required' "T9 join-readonly missing --secret FAILS" join-readonly --dc vr1-dc0 --unit ceph-radosgw/0 --zonegroup ZG --zone Z --endpoint http://x:80 --access-key K +run 1 '\-\-zone is required' "T10 enable-two-way missing --zone FAILS" enable-two-way --dc vr1-dc0 --unit ceph-radosgw/0 # --- dry-run plan content (dc1, gate OK) --- -MI_OUT="$(bash "$SCRIPT" master-init --dc dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://10.12.36.10:80 2>&1)"; MI_RC=$? +MI_OUT="$(bash "$SCRIPT" master-init --dc vr1-dc0 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://10.12.36.10:80 2>&1)"; MI_RC=$? [[ "$MI_RC" == 0 ]] && grep -q 'OK (dry-run)' <<<"$MI_OUT" && { echo " PASS T11 master-init dry-run rc 0 + OK marker"; PASS=$((PASS+1)); } || { echo " FAIL T11"; FAIL=$((FAIL+1)); } -grep -q 'gate: OK (dc1)' <<<"$MI_OUT" && { echo " PASS T12 dc1 gate reports OK"; PASS=$((PASS+1)); } || { echo " FAIL T12"; FAIL=$((FAIL+1)); } +grep -q 'gate: OK (vr1-dc0)' <<<"$MI_OUT" && { echo " PASS T12 vr1-dc0 gate reports OK"; PASS=$((PASS+1)); } || { echo " FAIL T12"; FAIL=$((FAIL+1)); } grep -qE 'realm create --rgw-realm=R' <<<"$MI_OUT" && grep -qE 'zonegroup create.*--master' <<<"$MI_OUT" && grep -qE 'zone create.*--master' <<<"$MI_OUT" && grep -q 'period update --commit' <<<"$MI_OUT" \ && { echo " PASS T13 master-init plan contains realm/zonegroup/zone/period steps"; PASS=$((PASS+1)); } || { echo " FAIL T13"; echo "$MI_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } grep -q 'no --restart-action given' <<<"$MI_OUT" && { echo " PASS T14 no-restart-action reminder present"; PASS=$((PASS+1)); } || { echo " FAIL T14"; FAIL=$((FAIL+1)); } -RESTART_OUT="$(bash "$SCRIPT" master-init --dc dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --restart-action restart 2>&1)" +RESTART_OUT="$(bash "$SCRIPT" master-init --dc vr1-dc0 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --restart-action restart 2>&1)" grep -qE 'juju run ceph-radosgw/0 restart -m openstack' <<<"$RESTART_OUT" && { echo " PASS T15 --restart-action adds a juju run step"; PASS=$((PASS+1)); } || { echo " FAIL T15"; echo "$RESTART_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } # --- $DC gate: informational in dry-run, blocking before --apply --- -DC2_DRY="$(bash "$SCRIPT" master-init --dc dc2 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 2>&1)"; DC2_DRY_RC=$? +DC2_DRY="$(bash "$SCRIPT" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 2>&1)"; DC2_DRY_RC=$? [[ "$DC2_DRY_RC" == 0 ]] && grep -q 'gate: FAILED' <<<"$DC2_DRY" && grep -q 'OK (dry-run)' <<<"$DC2_DRY" \ && { echo " PASS T16 dc2 dry-run still prints the plan (gate is informational here)"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$DC2_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T17 dc2 --apply is BLOCKED by the gate (rc 3)" master-init --dc dc2 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply +run 3 '\$DC gate refused' "T17 dc2 --apply is BLOCKED by the gate (rc 3)" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply # --- secret redaction --- -JR_OUT="$(bash "$SCRIPT" join-readonly --dc dc1 --unit ceph-radosgw/0 --zonegroup ZG --zone Z2 --endpoint http://x:80 --access-key AK123 --secret TOPSECRETVALUE 2>&1)" +JR_OUT="$(bash "$SCRIPT" join-readonly --dc vr1-dc0 --unit ceph-radosgw/0 --zonegroup ZG --zone Z2 --endpoint http://x:80 --access-key AK123 --secret TOPSECRETVALUE 2>&1)" grep -q '<REDACTED>' <<<"$JR_OUT" && ! grep -q 'TOPSECRETVALUE' <<<"$JR_OUT" \ && { echo " PASS T18 --secret value is redacted in printed plan"; PASS=$((PASS+1)); } || { echo " FAIL T18 (secret leaked or not redacted)"; echo "$JR_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } @@ -73,7 +73,7 @@ if command -v juju >/dev/null 2>&1; then echo " SKIP T19 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" else - run 2 'juju required on PATH' "T19 --apply with dc1 (gate OK) but juju absent FAILS (rc 2)" master-init --dc dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply + run 2 'juju required on PATH' "T19 --apply with dc1 (gate OK) but juju absent FAILS (rc 2)" master-init --dc vr1-dc0 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply fi echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" diff --git a/tests/dc-dc-rbd-mirror/run-tests.sh b/tests/dc-dc-rbd-mirror/run-tests.sh index b4345ee..302c20f 100644 --- a/tests/dc-dc-rbd-mirror/run-tests.sh +++ b/tests/dc-dc-rbd-mirror/run-tests.sh @@ -28,40 +28,40 @@ run 1 'unknown subcommand' "T2 bogus subcommand FAILS (rc 1)" bogus run 0 'usage:' "T3 --help exits 0 with usage" --help run 1 '\-\-dc is required' "T4 bootstrap-primary missing --dc FAILS" bootstrap-primary -run 1 '\-\-dc must be dc1 or dc2' "T5 bootstrap-primary bad --dc token FAILS" bootstrap-primary --dc bogus --unit ceph-mon/0 --pool glance --site-name dc1 -run 1 '\-\-unit is required' "T6 bootstrap-primary missing --unit FAILS" bootstrap-primary --dc dc1 -run 1 'do not assume .glance.' "T7 bootstrap-primary missing --pool FAILS, cites no-assume-glance" bootstrap-primary --dc dc1 --unit ceph-mon/0 --site-name dc1 -run 1 '\-\-site-name is required' "T8 bootstrap-primary missing --site-name FAILS" bootstrap-primary --dc dc1 --unit ceph-mon/0 --pool glance -run 1 '\-\-direction is required' "T9 bootstrap-secondary missing --direction FAILS" bootstrap-secondary --dc dc1 --unit ceph-mon/0 --pool glance --site-name dc2 --token-in /tmp/tok -run 1 '\-\-direction must be rx-only or rx-tx' "T10 bootstrap-secondary bad --direction FAILS" bootstrap-secondary --dc dc1 --unit ceph-mon/0 --pool glance --site-name dc2 --direction bogus --token-in /tmp/tok -run 1 '\-\-token-in is required' "T11 bootstrap-secondary missing --token-in FAILS" bootstrap-secondary --dc dc1 --unit ceph-mon/0 --pool glance --site-name dc2 --direction rx-only +run 1 '\-\-dc must be vr1-dc0 or vr1-dc1' "T5 bootstrap-primary bad --dc token FAILS" bootstrap-primary --dc bogus --unit ceph-mon/0 --pool glance --site-name vr1-dc0 +run 1 '\-\-unit is required' "T6 bootstrap-primary missing --unit FAILS" bootstrap-primary --dc vr1-dc0 +run 1 'do not assume .glance.' "T7 bootstrap-primary missing --pool FAILS, cites no-assume-glance" bootstrap-primary --dc vr1-dc0 --unit ceph-mon/0 --site-name vr1-dc0 +run 1 '\-\-site-name is required' "T8 bootstrap-primary missing --site-name FAILS" bootstrap-primary --dc vr1-dc0 --unit ceph-mon/0 --pool glance +run 1 '\-\-direction is required' "T9 bootstrap-secondary missing --direction FAILS" bootstrap-secondary --dc vr1-dc0 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --token-in /tmp/tok +run 1 '\-\-direction must be rx-only or rx-tx' "T10 bootstrap-secondary bad --direction FAILS" bootstrap-secondary --dc vr1-dc0 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --direction bogus --token-in /tmp/tok +run 1 '\-\-token-in is required' "T11 bootstrap-secondary missing --token-in FAILS" bootstrap-secondary --dc vr1-dc0 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --direction rx-only # --- dry-run plan content --- -BP_OUT="$(bash "$SCRIPT" bootstrap-primary --dc dc1 --unit ceph-mon/0 --pool glance --site-name dc1 2>&1)"; BP_RC=$? +BP_OUT="$(bash "$SCRIPT" bootstrap-primary --dc vr1-dc0 --unit ceph-mon/0 --pool glance --site-name vr1-dc0 2>&1)"; BP_RC=$? [[ "$BP_RC" == 0 ]] && grep -q 'OK (dry-run)' <<<"$BP_OUT" && { echo " PASS T12 bootstrap-primary dry-run rc 0 + OK marker"; PASS=$((PASS+1)); } || { echo " FAIL T12"; echo "$BP_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -grep -q 'rbd mirror pool enable glance image' <<<"$BP_OUT" && grep -q 'peer bootstrap create --site-name dc1 glance' <<<"$BP_OUT" \ +grep -q 'rbd mirror pool enable glance image' <<<"$BP_OUT" && grep -q 'peer bootstrap create --site-name vr1-dc0 glance' <<<"$BP_OUT" \ && { echo " PASS T13 bootstrap-primary plan contains enable + peer-bootstrap-create"; PASS=$((PASS+1)); } || { echo " FAIL T13"; echo "$BP_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } grep -q 'transfer it to the DC2 unit OUT OF BAND' <<<"$BP_OUT" && { echo " PASS T14 bootstrap-primary reminds out-of-band token transfer"; PASS=$((PASS+1)); } || { echo " FAIL T14"; FAIL=$((FAIL+1)); } -BS_RXONLY="$(bash "$SCRIPT" bootstrap-secondary --dc dc1 --unit ceph-mon/1 --pool glance --site-name dc2 --direction rx-only --token-in /tmp/tok 2>&1)" -grep -q 'peer bootstrap import --site-name dc2 --direction rx-only glance /tmp/tok' <<<"$BS_RXONLY" \ +BS_RXONLY="$(bash "$SCRIPT" bootstrap-secondary --dc vr1-dc0 --unit ceph-mon/1 --pool glance --site-name vr1-dc1 --direction rx-only --token-in /tmp/tok 2>&1)" +grep -q 'peer bootstrap import --site-name vr1-dc1 --direction rx-only glance /tmp/tok' <<<"$BS_RXONLY" \ && { echo " PASS T15 bootstrap-secondary rx-only plan is correct"; PASS=$((PASS+1)); } || { echo " FAIL T15"; echo "$BS_RXONLY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -BS_RXTX="$(bash "$SCRIPT" bootstrap-secondary --dc dc1 --unit ceph-mon/1 --pool glance --site-name dc2 --direction rx-tx --token-in /tmp/tok2 2>&1)" -grep -q 'peer bootstrap import --site-name dc2 --direction rx-tx glance /tmp/tok2' <<<"$BS_RXTX" \ +BS_RXTX="$(bash "$SCRIPT" bootstrap-secondary --dc vr1-dc0 --unit ceph-mon/1 --pool glance --site-name vr1-dc1 --direction rx-tx --token-in /tmp/tok2 2>&1)" +grep -q 'peer bootstrap import --site-name vr1-dc1 --direction rx-tx glance /tmp/tok2' <<<"$BS_RXTX" \ && { echo " PASS T16 bootstrap-secondary rx-tx (two-way) plan is correct"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$BS_RXTX" | sed 's/^/ /'; FAIL=$((FAIL+1)); } # --- $DC gate: informational in dry-run, blocking before --apply --- -DC2_DRY="$(bash "$SCRIPT" bootstrap-primary --dc dc2 --unit ceph-mon/0 --pool glance --site-name dc2 2>&1)"; DC2_DRY_RC=$? +DC2_DRY="$(bash "$SCRIPT" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 2>&1)"; DC2_DRY_RC=$? [[ "$DC2_DRY_RC" == 0 ]] && grep -q 'gate: FAILED' <<<"$DC2_DRY" && grep -q 'OK (dry-run)' <<<"$DC2_DRY" \ && { echo " PASS T17 dc2 dry-run still prints the plan (gate is informational here)"; PASS=$((PASS+1)); } || { echo " FAIL T17"; echo "$DC2_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T18 dc2 --apply is BLOCKED by the gate (rc 3)" bootstrap-primary --dc dc2 --unit ceph-mon/0 --pool glance --site-name dc2 --apply +run 3 '\$DC gate refused' "T18 dc2 --apply is BLOCKED by the gate (rc 3)" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --apply if command -v juju >/dev/null 2>&1; then echo " SKIP T19 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" else - run 2 'juju required on PATH' "T19 --apply with dc1 (gate OK) but juju absent FAILS (rc 2)" bootstrap-primary --dc dc1 --unit ceph-mon/0 --pool glance --site-name dc1 --apply + run 2 'juju required on PATH' "T19 --apply with dc1 (gate OK) but juju absent FAILS (rc 2)" bootstrap-primary --dc vr1-dc0 --unit ceph-mon/0 --pool glance --site-name vr1-dc0 --apply fi echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" diff --git a/tests/dc-selector/run-tests.sh b/tests/dc-selector/run-tests.sh index 4c2caed..b745bd2 100644 --- a/tests/dc-selector/run-tests.sh +++ b/tests/dc-selector/run-tests.sh @@ -36,34 +36,52 @@ chk "HOSTS array unaffected" "${HOSTS[0]}" "openstack0" chk "HOST_OCTET unaffected" "${HOST_OCTET[openstack0]}" "40" -# --- lib_net_select_dc: dc0/dc1 no-op, dc2 fails loud, unknown fails loud --- -( lib_net_select_dc dc0 ); chk "net dc0 no-op rc" "$?" 0 -( lib_net_select_dc dc1 ); chk "net dc1 no-op rc" "$?" 0 -( lib_net_select_dc dc2 2>/dev/null ); chk "net dc2 fails-loud rc" "$?" 1 -NET_ERR="$(lib_net_select_dc dc2 2>&1 1>/dev/null || true)" -grep -q "NetBox" <<<"$NET_ERR" && ok "net dc2 error cites NetBox gap" || no "net dc2 error cites NetBox gap" +# --- lib_net_select_dc (D-119 region-qualified): vr0-dc0/vr1-dc0 no-op, +# vr1-dc1 fails loud, RETIRED bare dcN fails loud, unknown fails loud --- +( lib_net_select_dc vr0-dc0 ); chk "net vr0-dc0 no-op rc" "$?" 0 +( lib_net_select_dc vr1-dc0 ); chk "net vr1-dc0 no-op rc" "$?" 0 +( lib_net_select_dc vr1-dc1 2>/dev/null ); chk "net vr1-dc1 fails-loud rc" "$?" 1 +NET_ERR="$(lib_net_select_dc vr1-dc1 2>&1 1>/dev/null || true)" +grep -q "NetBox" <<<"$NET_ERR" && ok "net vr1-dc1 error cites NetBox gap" || no "net vr1-dc1 error cites NetBox gap" ( lib_net_select_dc bogus 2>/dev/null ); chk "net unknown-token fails-loud rc" "$?" 1 ( lib_net_select_dc 2>/dev/null ); chk "net missing-arg fails-loud rc" "$?" 1 -# no-op really means no-op: values identical after selecting either dc0 or dc1 -DC0_VAL="$(lib_net_select_dc dc0 >/dev/null 2>&1; echo "${PLANE_CIDRS[0]}")" -DC1_VAL="$(lib_net_select_dc dc1 >/dev/null 2>&1; echo "${PLANE_CIDRS[0]}")" -chk "net dc0/dc1 identical values (D-101)" "$DC0_VAL" "$DC1_VAL" +# D-119 REGRESSION GUARD: the bare dcN tokens are RETIRED and must be REJECTED. +# Accepting them "for compatibility" would preserve the exact cross-region +# ambiguity D-119 deletes -- 'dc0' meant VR0's LIVE cloud in lib-net.sh but VR1's +# FIRST DC in the NetBox importer. Silence here is how the off-by-one comes back. +for retired in dc0 dc1 dc2; do + ( lib_net_select_dc "$retired" 2>/dev/null ); chk "net RETIRED '$retired' rejected (D-119)" "$?" 1 + R_ERR="$(lib_net_select_dc "$retired" 2>&1 1>/dev/null || true)" + grep -q "RETIRED" <<<"$R_ERR" && ok "net '$retired' error says RETIRED" || no "net '$retired' error says RETIRED" +done -# --- lib_hosts_select_dc: dc0 no-op, dc1 AND dc2 both fail loud --- -( lib_hosts_select_dc dc0 ); chk "hosts dc0 no-op rc" "$?" 0 -( lib_hosts_select_dc dc1 2>/dev/null ); chk "hosts dc1 fails-loud rc" "$?" 1 -( lib_hosts_select_dc dc2 2>/dev/null ); chk "hosts dc2 fails-loud rc" "$?" 1 -HOSTS_ERR1="$(lib_hosts_select_dc dc1 2>&1 1>/dev/null || true)" -grep -q "no enrolled hosts" <<<"$HOSTS_ERR1" && ok "hosts dc1 error cites no-enrollment" || no "hosts dc1 error cites no-enrollment" -HOSTS_ERR2="$(lib_hosts_select_dc dc2 2>&1 1>/dev/null || true)" -grep -q "no enrolled hosts" <<<"$HOSTS_ERR2" && ok "hosts dc2 error cites no-enrollment" || no "hosts dc2 error cites no-enrollment" +# no-op really means no-op: values identical after selecting vr0-dc0 or vr1-dc0. +# They match by INHERITANCE (D-101: VR1's first DC inherits VR0 DC0's v4 layout), +# not by coincidence -- and they are separate case arms so they can diverge later. +VR0_VAL="$(lib_net_select_dc vr0-dc0 >/dev/null 2>&1; echo "${PLANE_CIDRS[0]}")" +VR1_VAL="$(lib_net_select_dc vr1-dc0 >/dev/null 2>&1; echo "${PLANE_CIDRS[0]}")" +chk "net vr0-dc0/vr1-dc0 identical values (D-101 inheritance)" "$VR0_VAL" "$VR1_VAL" + +# --- lib_hosts_select_dc: vr0-dc0 no-op, BOTH VR1 DCs fail loud --- +( lib_hosts_select_dc vr0-dc0 ); chk "hosts vr0-dc0 no-op rc" "$?" 0 +( lib_hosts_select_dc vr1-dc0 2>/dev/null ); chk "hosts vr1-dc0 fails-loud rc" "$?" 1 +( lib_hosts_select_dc vr1-dc1 2>/dev/null ); chk "hosts vr1-dc1 fails-loud rc" "$?" 1 +HOSTS_ERR1="$(lib_hosts_select_dc vr1-dc0 2>&1 1>/dev/null || true)" +grep -q "no enrolled hosts" <<<"$HOSTS_ERR1" && ok "hosts vr1-dc0 error cites no-enrollment" || no "hosts vr1-dc0 error cites no-enrollment" +HOSTS_ERR2="$(lib_hosts_select_dc vr1-dc1 2>&1 1>/dev/null || true)" +grep -q "no enrolled hosts" <<<"$HOSTS_ERR2" && ok "hosts vr1-dc1 error cites no-enrollment" || no "hosts vr1-dc1 error cites no-enrollment" ( lib_hosts_select_dc bogus 2>/dev/null ); chk "hosts unknown-token fails-loud rc" "$?" 1 ( lib_hosts_select_dc 2>/dev/null ); chk "hosts missing-arg fails-loud rc" "$?" 1 +for retired in dc0 dc1 dc2; do + ( lib_hosts_select_dc "$retired" 2>/dev/null ); chk "hosts RETIRED '$retired' rejected (D-119)" "$?" 1 +done -# --- the documented asymmetry itself: dc1 differs between the two libs --- -( lib_net_select_dc dc1 ); NET_DC1_RC=$? -( lib_hosts_select_dc dc1 2>/dev/null ); HOSTS_DC1_RC=$? -[ "$NET_DC1_RC" = 0 ] && [ "$HOSTS_DC1_RC" = 1 ] && ok "dc1 asymmetry: net no-ops, hosts fails (documented, not a bug)" || no "dc1 asymmetry broken (net=$NET_DC1_RC hosts=$HOSTS_DC1_RC)" +# --- the documented asymmetry: vr1-dc0 differs between the two libs --- +# net no-ops (D-101 says it inherits VR0 DC0's v4), hosts FAILS (it has no +# enrolled hosts yet). Intentional; asserted so nobody "fixes" one to match. +( lib_net_select_dc vr1-dc0 ); NET_RC=$? +( lib_hosts_select_dc vr1-dc0 2>/dev/null ); HOSTS_RC=$? +[ "$NET_RC" = 0 ] && [ "$HOSTS_RC" = 1 ] && ok "vr1-dc0 asymmetry: net no-ops, hosts fails (documented, not a bug)" || no "vr1-dc0 asymmetry broken (net=$NET_RC hosts=$HOSTS_RC)" echo; [ "$F" = 0 ] && { echo "ALL PASS ($P checks)"; exit 0; } || { echo "FAILURES: $F"; exit 1; } diff --git a/tests/opentofu-validate/fixtures/s1-bad/main.tf b/tests/opentofu-validate/fixtures/s1-bad/main.tf new file mode 100644 index 0000000..067c049 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s1-bad/main.tf @@ -0,0 +1,10 @@ +# S1 fixture (BAD): libvirt_domain sets `memory` with NO `memory_unit`. +# This is the exact defect that shipped the 2026-07-12 Office1 OPNsense +# 2 MiB guest. scripts/opentofu-validate.sh S1 MUST reject this. +resource "libvirt_domain" "bad" { + name = "s1-bad" + memory = 2048 + vcpu = 2 + type = "kvm" + running = true +} diff --git a/tests/opentofu-validate/fixtures/s1-good/main.tf b/tests/opentofu-validate/fixtures/s1-good/main.tf new file mode 100644 index 0000000..fdf5c6e --- /dev/null +++ b/tests/opentofu-validate/fixtures/s1-good/main.tf @@ -0,0 +1,27 @@ +# S1+S2 fixture (GOOD): libvirt_domain sets `memory` AND `memory_unit`, AND enables ACPI. +# Also pins the near-miss: `memory_backing` / `memory_tune` are DIFFERENT +# attributes and must not be mistaken for `memory` by the guard's scan. +resource "libvirt_domain" "good" { + name = "s1-good" + memory = 2048 + memory_unit = "MiB" + vcpu = 2 + type = "kvm" + running = true + + features = { + acpi = true + apic = {} + } +} + +resource "libvirt_domain" "no_memory_at_all" { + name = "s1-no-memory" + vcpu = 1 + type = "kvm" + running = true + + features = { + acpi = true + } +} diff --git a/tests/opentofu-validate/fixtures/s2-bad/main.tf b/tests/opentofu-validate/fixtures/s2-bad/main.tf new file mode 100644 index 0000000..69876c6 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s2-bad/main.tf @@ -0,0 +1,15 @@ +# S2 fixture (BAD): memory/memory_unit are correct, but the domain does NOT enable ACPI. +# libvirt then renders the machine `acpi=off`, and a FreeBSD/OPNsense guest PANICS at +# interrupt init ("running without device atpic requires a local APIC") and spins in ddb +# at 100% CPU while still reporting as "running". That was the SECOND half of the +# 2026-07-12 boot incident. scripts/opentofu-validate.sh S2 MUST reject this. +# +# Note this fixture PASSES S1 -- it exists to prove S2 catches its own class independently. +resource "libvirt_domain" "no_acpi" { + name = "s2-bad" + memory = 2048 + memory_unit = "MiB" + vcpu = 2 + type = "kvm" + running = true +} diff --git a/tests/opentofu-validate/fixtures/s3-bad/main.tf b/tests/opentofu-validate/fixtures/s3-bad/main.tf new file mode 100644 index 0000000..6dac935 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s3-bad/main.tf @@ -0,0 +1,7 @@ +# Root is deliberately VALID and deliberately does NOT call modules/broken. +# That is the whole point of this fixture: root-only validation reports PASS, +# because an uncalled module is never parsed. S3 must still catch it. +variable "unused" { + type = string + default = "root is fine" +} diff --git a/tests/opentofu-validate/fixtures/s3-bad/modules/broken/main.tf b/tests/opentofu-validate/fixtures/s3-bad/modules/broken/main.tf new file mode 100644 index 0000000..7bd4be8 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s3-bad/modules/broken/main.tf @@ -0,0 +1,15 @@ +# Reproduces the real netem-link defect (DOCFIX-194): a destroy-time provisioner +# may reference ONLY self/count.index/each.key -- never var.*. OpenTofu rejects +# this at INIT. Uses terraform_data (a builtin) so the fixture needs NO provider +# download: the harness stays fast and offline. +variable "target" { + type = string + default = "host" +} + +resource "terraform_data" "broken" { + provisioner "local-exec" { + when = destroy + command = "echo ${var.target}" + } +} diff --git a/tests/opentofu-validate/fixtures/s3-good/main.tf b/tests/opentofu-validate/fixtures/s3-good/main.tf new file mode 100644 index 0000000..4c4e88f --- /dev/null +++ b/tests/opentofu-validate/fixtures/s3-good/main.tf @@ -0,0 +1,4 @@ +variable "unused" { + type = string + default = "root is fine" +} diff --git a/tests/opentofu-validate/fixtures/s3-good/modules/fine/main.tf b/tests/opentofu-validate/fixtures/s3-good/modules/fine/main.tf new file mode 100644 index 0000000..da8ffe5 --- /dev/null +++ b/tests/opentofu-validate/fixtures/s3-good/modules/fine/main.tf @@ -0,0 +1,17 @@ +# The correct form: stash the value in `input`, read it back via self.input.* +# in the destroy provisioner. +variable "target" { + type = string + default = "host" +} + +resource "terraform_data" "fine" { + input = { + target = var.target + } + + provisioner "local-exec" { + when = destroy + command = "echo ${self.input.target}" + } +} diff --git a/tests/opentofu-validate/run-tests.sh b/tests/opentofu-validate/run-tests.sh index 7c2586b..222be17 100644 --- a/tests/opentofu-validate/run-tests.sh +++ b/tests/opentofu-validate/run-tests.sh @@ -1,11 +1,12 @@ #!/usr/bin/env bash # tests/opentofu-validate/run-tests.sh -- offline harness for -# scripts/opentofu-validate.sh. Only tests the guard clauses this environment -# can actually exercise (no tofu binary is available anywhere this repo has -# been worked in so far -- see opentofu/README.md). The fmt/init/validate path -# itself is UNTESTED here; that is real, logged residual risk, not an -# oversight -- run the script for real on a machine with the tofu binary -# before trusting opentofu/. +# scripts/opentofu-validate.sh. +# +# UPDATED 2026-07-13: a tofu binary IS now present on the jumphost (OpenTofu +# v1.12.3), so the fmt/init/validate path is no longer untestable. T8-T10 exercise +# S3 (module-standalone validation) for real; they SKIP if no tofu is on PATH. +# The old header claimed "no tofu binary is available anywhere this repo has been +# worked in" -- that was true when written and is now false. # Exit: 0 all pass | 1 any case failed. ASCII + LF. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -32,5 +33,59 @@ run 2 'tofu binary not found' "T2 tofu binary missing FAILS (rc 2)" "$TMP" fi +# --- S1/S2: libvirt_domain static guards (both halves of the 2026-07-12 boot incident) --- +# --static-only runs the static scans only: no tofu binary, no network. + +run 1 'sets memory with no memory_unit' \ + "T3 S1 REJECTS a libvirt_domain with memory and no memory_unit (rc 1)" \ + --static-only "$HERE/fixtures/s1-bad" + +run 0 'every libvirt_domain that sets memory also sets memory_unit' \ + "T4 S1+S2 ACCEPT memory+memory_unit+acpi, and a domain with no memory at all (rc 0)" \ + --static-only "$HERE/fixtures/s1-good" + +run 1 'does not enable ACPI' \ + "T5 S2 REJECTS a libvirt_domain with no ACPI (rc 1) -- and it PASSES S1, proving S2 is independent" \ + --static-only "$HERE/fixtures/s2-bad" + +run 0 'every libvirt_domain enables ACPI' \ + "T6 S1+S2 PASS against the real opentofu/ tree (all VM modules carry memory_unit + acpi)" \ + --static-only "$HERE/../../opentofu" + +run 0 'every libvirt_domain that sets memory also sets memory_unit' \ + "T7 back-compat: the old --check-memory-unit flag still works" \ + --check-memory-unit "$HERE/../../opentofu" + +# --- S3: the root-only blind spot (DOCFIX-194) ------------------------------- +# +# `tofu validate` on the root module NEVER PARSES a module the root does not +# instantiate. On 2026-07-13 this gate printed PASS while node-vm (a bogus +# top-level `format` on libvirt_volume) and netem-link (a destroy provisioner +# referencing var.*) were both broken -- neither is called from root's main.tf. +# Both would have detonated at first use, mid-deploy. +# +# T8's fixture is the load-bearing one: its ROOT IS VALID and does NOT call the +# broken module. A gate that only validates root reports PASS on it. If T8 ever +# goes green-when-it-should-be-red, the blind spot is back. +# +# The fixtures use `terraform_data` (a builtin), so no provider download is +# needed -- these cases stay fast and need no registry access. + +if command -v tofu >/dev/null 2>&1; then + run 1 '\[FAIL\] modules/broken' \ + "T8 S3 REJECTS a broken module that root does NOT call (rc 1) -- THE blind-spot regression test" \ + "$HERE/fixtures/s3-bad" + + run 0 '\[PASS\] modules/fine' \ + "T9 S3 ACCEPTS a correct standalone module (rc 0) -- proves T8 fails on the defect, not on the fixture shape" \ + "$HERE/fixtures/s3-good" + + run 0 '\[PASS\] modules/node-vm' \ + "T10 S3 PASSES against the real opentofu/ tree (all 10 modules validate standalone)" \ + "$HERE/../../opentofu" +else + echo " SKIP T8-T10 S3 module-standalone cases (no tofu binary on PATH)" +fi + echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" [[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1 diff --git a/tests/opnsense-api/run-tests.sh b/tests/opnsense-api/run-tests.sh new file mode 100755 index 0000000..8453d46 --- /dev/null +++ b/tests/opnsense-api/run-tests.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# tests/opnsense-api/run-tests.sh -- offline harness for scripts/opnsense-api.sh. +# +# Tests the guard clauses, URL construction, and credential parsing WITHOUT +# touching the network and WITHOUT a real credential. The fixture key/secret +# below are throwaway fakes, never real key material. +# +# The load-bearing case is T9: the secret must never reach argv. We assert that +# by stubbing `curl` with a recorder that dumps its own argv -- if a future edit +# "simplifies" the script to `curl -u key:secret`, T9 goes red. That is the +# point: the secret is visible in `ps` for anyone on the box otherwise. +# +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$HERE/../../scripts/opnsense-api.sh" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +PASS=0; FAIL=0 + +FAKE_KEY="FAKEKEYaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +FAKE_SECRET="FAKESECRETbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + +ok() { PASS=$((PASS+1)); echo " PASS $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL $1"; } +check(){ if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (want '$3', got '$2')"; fi; } + +mkcreds() { printf 'key=%s\nsecret=%s\n' "$FAKE_KEY" "$FAKE_SECRET" > "$1"; } + +echo "T1: no args -> usage, exit 1" +out="$(bash "$SCRIPT" 2>&1)"; rc=$? +check "T1 exit" "$rc" "1" +case "$out" in *usage*) ok "T1 prints usage";; *) bad "T1 usage text";; esac + +echo "T2: bad method rejected" +OPNSENSE_API_HOST=10.10.0.1 out="$(bash "$SCRIPT" DELETE core/firmware/status 2>&1)"; rc=$? +check "T2 exit" "$rc" "1" +case "$out" in *"must be GET or POST"*) ok "T2 method guard";; *) bad "T2 method guard";; esac + +echo "T3: host NEVER inferred -- unset OPNSENSE_API_HOST must fail loud" +out="$(env -u OPNSENSE_API_HOST bash "$SCRIPT" GET core/firmware/status 2>&1)"; rc=$? +check "T3 exit" "$rc" "1" +case "$out" in *OPNSENSE_API_HOST*) ok "T3 refuses to guess the edge address";; *) bad "T3 host guard";; esac + +echo "T4: dry-run builds the right URL" +out="$(OPNSENSE_API_HOST=10.10.0.1 bash "$SCRIPT" --dry-run GET core/firmware/status 2>&1)"; rc=$? +check "T4 exit" "$rc" "0" +check "T4 url" "$out" "DRY-RUN GET https://10.10.0.1/api/core/firmware/status" + +echo "T5: leading '/' and 'api/' in the path are normalized, not 404'd" +out="$(OPNSENSE_API_HOST=10.10.0.1 bash "$SCRIPT" --dry-run GET /api/kea/dhcpv4/get 2>&1)" +check "T5 url" "$out" "DRY-RUN GET https://10.10.0.1/api/kea/dhcpv4/get" + +echo "T6: missing creds file -> fail loud, exit 1" +out="$(OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/nope.txt" bash "$SCRIPT" GET core/firmware/status 2>&1)"; rc=$? +check "T6 exit" "$rc" "1" +case "$out" in *"not found"*) ok "T6 says which file is missing";; *) bad "T6 message";; esac + +echo "T7: creds file with no key= -> fail loud" +printf 'secret=%s\n' "$FAKE_SECRET" > "$TMP/nokey.txt" +out="$(OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/nokey.txt" bash "$SCRIPT" GET core/firmware/status 2>&1)"; rc=$? +check "T7 exit" "$rc" "1" +case "$out" in *"no 'key='"*) ok "T7 detects missing key";; *) bad "T7 message";; esac + +echo "T8: creds file with no secret= -> fail loud" +printf 'key=%s\n' "$FAKE_KEY" > "$TMP/nosecret.txt" +out="$(OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/nosecret.txt" bash "$SCRIPT" GET core/firmware/status 2>&1)"; rc=$? +check "T8 exit" "$rc" "1" +case "$out" in *"no 'secret='"*) ok "T8 detects missing secret";; *) bad "T8 message";; esac + +echo "T9: THE SECRET MUST NEVER REACH ARGV (it would show up in \`ps\`)" +mkcreds "$TMP/creds.txt" +# stub curl: record argv, then emit a plausible response + status line +cat > "$TMP/curl" <<'STUB' +#!/usr/bin/env bash +printf '%s\n' "$@" > "$TMP_ARGV" +cat > "$TMP_STDIN" +echo '{"status":"ok"}' +echo '__HTTP_STATUS__:200' +STUB +chmod +x "$TMP/curl" +TMP_ARGV="$TMP/argv.txt" TMP_STDIN="$TMP/stdin.txt" \ + PATH="$TMP:$PATH" OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/creds.txt" \ + bash "$SCRIPT" GET core/firmware/status >/dev/null 2>&1 +if grep -q "$FAKE_SECRET" "$TMP/argv.txt" 2>/dev/null; then + bad "T9 SECRET LEAKED INTO ARGV -- visible in ps" +else + ok "T9 secret absent from argv" +fi +if grep -q "$FAKE_SECRET" "$TMP/stdin.txt" 2>/dev/null; then + ok "T9 secret passed via stdin --config (correct)" +else + bad "T9 secret did not reach curl via stdin -- auth would fail" +fi + +echo "T10: the secret is never printed to stdout/stderr" +out="$(TMP_ARGV="$TMP/argv2.txt" TMP_STDIN="$TMP/stdin2.txt" \ + PATH="$TMP:$PATH" OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/creds.txt" \ + bash "$SCRIPT" GET core/firmware/status 2>&1)" +case "$out" in + *"$FAKE_SECRET"*) bad "T10 SECRET PRINTED to output";; + *"$FAKE_KEY"*) bad "T10 KEY PRINTED to output";; + *) ok "T10 neither key nor secret appears in output";; +esac + +echo "T11: dry-run must NOT read the creds file at all" +out="$(OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/nope.txt" \ + bash "$SCRIPT" --dry-run GET core/firmware/status 2>&1)"; rc=$? +check "T11 exit (no creds needed)" "$rc" "0" + +echo "T12: a 401 from the API is surfaced as exit 2, not silently swallowed" +cat > "$TMP/curl" <<'STUB' +#!/usr/bin/env bash +cat > /dev/null +echo '{"status":"denied"}' +echo '__HTTP_STATUS__:401' +STUB +chmod +x "$TMP/curl" +out="$(PATH="$TMP:$PATH" OPNSENSE_API_HOST=10.10.0.1 OPNSENSE_API_CREDS="$TMP/creds.txt" \ + bash "$SCRIPT" GET core/firmware/status 2>&1)"; rc=$? +check "T12 exit" "$rc" "2" +case "$out" in *401*) ok "T12 reports the 401";; *) bad "T12 message";; esac + +echo +echo "RESULT: PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "FAILURES"; exit 1; } diff --git a/tests/opnsense-bootstrap-apikey/run-tests.sh b/tests/opnsense-bootstrap-apikey/run-tests.sh new file mode 100755 index 0000000..9748a5a --- /dev/null +++ b/tests/opnsense-bootstrap-apikey/run-tests.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# tests/opnsense-bootstrap-apikey/run-tests.sh -- offline harness for +# scripts/opnsense-bootstrap-apikey.sh. Stubs ssh/scp on PATH, so no edge is ever +# contacted and no real key is ever minted. +# +# The load-bearing case is T5: the script MUST refuse to overwrite an existing creds +# file. An existing file means a LIVE key on the edge; overwriting the local copy +# would strand it there permanently, because OPNsense stores the secret as +# crypt(secret,'$6$') and it can never be read back. That is an unrecoverable +# credential loss, and a guard clause is the only thing standing in front of it. +# +# Exit: 0 all pass | 1 any case failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$HERE/../../scripts/opnsense-bootstrap-apikey.sh" +MINTER="$HERE/../../scripts/opnsense-mint-apikey.php" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +PASS=0; FAIL=0 + +FAKE_KEY="FAKEKEYaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +FAKE_SECRET="FAKESECRETbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + +ok() { PASS=$((PASS+1)); echo " PASS $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL $1"; } + +# --- stub ssh/scp: scp "retrieving" the remote file writes a fake creds payload --- +mk_stubs() { + cat > "$TMP/bin/ssh" <<'STUB' +#!/usr/bin/env bash +exit 0 +STUB + cat > "$TMP/bin/scp" <<STUB +#!/usr/bin/env bash +# last arg is the destination; if it is a LOCAL path (no colon), we are retrieving +dest="\${!#}" +case "\$dest" in + *:*) exit 0 ;; # shipping TO the edge -- nothing to do + *) printf 'key=%s\nsecret=%s\n' "$FAKE_KEY" "$FAKE_SECRET" > "\$dest"; exit 0 ;; +esac +STUB + chmod +x "$TMP/bin/ssh" "$TMP/bin/scp" +} +mkdir -p "$TMP/bin"; mk_stubs + +echo "T1: the minter PHP exists and is what we ship" +if [ -f "$MINTER" ] && grep -q "apikeys->add()" "$MINTER"; then + ok "T1 minter present and calls the VENDOR model (apikeys->add)" +else + bad "T1 minter missing or does not call apikeys->add()" +fi + +echo "T2: the minter never prints the secret" +if grep -qE 'printf|echo' "$MINTER" && ! grep -E "printf|echo" "$MINTER" | grep -q "\['secret'\]"; then + ok "T2 minter prints lengths, never the secret itself" +else + bad "T2 minter may print the secret" +fi + +echo "T3: no args -> usage, exit 1" +out="$(bash "$SCRIPT" 2>&1)"; rc=$? +{ [ "$rc" = "1" ] && grep -q usage <<<"$out"; } && ok "T3 usage guard" || bad "T3 usage guard (rc=$rc)" + +echo "T4: dry-run needs no ssh key and contacts nothing" +out="$(env -u OPNSENSE_SSH_KEY bash "$SCRIPT" --dry-run 10.10.0.1 "$TMP/nope.txt" 2>&1)"; rc=$? +{ [ "$rc" = "0" ] && grep -q "DRY-RUN" <<<"$out"; } && ok "T4 dry-run" || bad "T4 dry-run (rc=$rc)" + +echo "T5: REFUSES to overwrite an existing creds file (the unrecoverable-loss guard)" +printf 'key=existing\nsecret=existing\n' > "$TMP/existing.txt" +out="$(OPNSENSE_SSH_KEY=/dev/null PATH="$TMP/bin:$PATH" bash "$SCRIPT" 10.10.0.1 "$TMP/existing.txt" 2>&1)"; rc=$? +if [ "$rc" = "1" ] && grep -q "refusing to overwrite" <<<"$out"; then + # and it must NOT have touched the file + if grep -q "^key=existing$" "$TMP/existing.txt"; then + ok "T5 refuses to clobber a live credential, and leaves it intact" + else + bad "T5 refused but MODIFIED the existing file" + fi +else + bad "T5 did NOT refuse to overwrite an existing creds file (rc=$rc) -- would strand a live key" +fi + +echo "T6: missing SSH key -> fail loud (not a silent agent-auth attempt)" +out="$(env -u OPNSENSE_SSH_KEY PATH="$TMP/bin:$PATH" bash "$SCRIPT" 10.10.0.1 "$TMP/out6.txt" 2>&1)"; rc=$? +{ [ "$rc" = "1" ] && grep -q "OPNSENSE_SSH_KEY" <<<"$out"; } && ok "T6 ssh-key guard" || bad "T6 ssh-key guard (rc=$rc)" + +echo "T7: happy path writes a 0600 creds file in the shape opnsense-api.sh parses" +out="$(OPNSENSE_SSH_KEY=/dev/null PATH="$TMP/bin:$PATH" bash "$SCRIPT" 10.10.0.1 "$TMP/out7.txt" 2>&1)"; rc=$? +if [ "$rc" = "0" ] && [ -f "$TMP/out7.txt" ]; then + mode=$(stat -c %a "$TMP/out7.txt") + k=$(grep -c '^key=' "$TMP/out7.txt"); s=$(grep -c '^secret=' "$TMP/out7.txt") + if [ "$mode" = "600" ] && [ "$k" = "1" ] && [ "$s" = "1" ]; then + ok "T7 minted file: mode 600, one key= line, one secret= line" + else + bad "T7 bad output file (mode=$mode key=$k secret=$s)" + fi +else + bad "T7 happy path failed (rc=$rc): $out" +fi + +echo "T8: the secret never appears on stdout/stderr" +case "$out" in + *"$FAKE_SECRET"*) bad "T8 SECRET PRINTED to output";; + *) ok "T8 secret absent from output";; +esac + +echo +echo "RESULT: PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] && { echo "ALL PASS"; exit 0; } || { echo "FAILURES"; exit 1; } diff --git a/tests/opnsense-build-config-iso/run-tests.sh b/tests/opnsense-build-config-iso/run-tests.sh deleted file mode 100644 index 1ad46b3..0000000 --- a/tests/opnsense-build-config-iso/run-tests.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# tests/opnsense-build-config-iso/run-tests.sh -- offline harness for -# scripts/opnsense-build-config-iso.sh. Only tests the guard clauses this -# environment can actually exercise. Whether the built ISO is actually -# picked up by OPNsense's Configuration Importer is UNTESTED here (no -# genisoimage/xorriso and no real OPNsense VM were available this session) -# -- logged as residual risk, not hidden; see opentofu/README.md. -# Exit: 0 all pass | 1 any case failed. ASCII + LF. -set -uo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -SCRIPT="$HERE/../../scripts/opnsense-build-config-iso.sh" -PASS=0; FAIL=0 - -run() { # run <want_rc> <regex> <label> -- args... - local want="$1" rx="$2" label="$3"; shift 3 - local out rc - out="$(bash "$SCRIPT" "$@" 2>&1)"; rc=$? - if [[ "$rc" == "$want" ]] && grep -qE "$rx" <<<"$out"; then - echo " PASS $label"; PASS=$((PASS+1)) - else - echo " FAIL $label (rc=$rc want=$want)"; echo "$out" | sed 's/^/ /'; FAIL=$((FAIL+1)) - fi -} - -TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT - -run 1 'usage: opnsense-build-config-iso.sh' "T1 missing all args FAILS (rc 1)" -run 1 'cannot read' "T2 nonexistent config.xml FAILS (rc 1)" "$TMP/does-not-exist.xml" "$TMP/out.iso" - -echo '<opnsense></opnsense>' > "$TMP/config.xml" -if command -v genisoimage >/dev/null 2>&1 || command -v xorriso >/dev/null 2>&1; then - echo " SKIP T3 genisoimage/xorriso-missing case (one IS present in this environment -- can't exercise the missing-tool guard here)" -else - run 2 'genisoimage or xorriso required' "T3 no ISO tool available FAILS (rc 2)" "$TMP/config.xml" "$TMP/out.iso" -fi - -echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" -[[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1 diff --git a/tests/opnsense-render-config/run-tests.sh b/tests/opnsense-render-config/run-tests.sh deleted file mode 100644 index 1c3ec8b..0000000 --- a/tests/opnsense-render-config/run-tests.sh +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env bash -# tests/opnsense-render-config/run-tests.sh -- offline harness for -# scripts/opnsense-render-config.sh. Unlike the other opnsense-* scripts, -# this one needs no external tool (pure bash + template substitution), so -# this harness tests the REAL behavior end-to-end, not just guard clauses. -# Exit: 0 all pass | 1 any case failed. ASCII + LF. -set -uo pipefail -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -SCRIPT="$HERE/../../scripts/opnsense-render-config.sh" -TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT -PASS=0; FAIL=0 - -base_env() { - export OPNSENSE_HOSTNAME="opnsense" DOMAIN="omega.dc1.vr1.cloud.neumatrix.local" - export ROOT_PASSWORD_HASH='$2y$10$fakehashfakehashfakehashfakehashfakehas' - export WAN_IF="vtnet0" WAN_IPADDR="203.0.113.2" WAN_SUBNET_BITS="30" WAN_GATEWAY="203.0.113.1" - export LAN_IF="vtnet1" LAN_IPADDR="10.12.8.1" LAN_SUBNET_BITS="22" - export MIRROR_SYNC_PROTOCOL="TCP" MIRROR_UPSTREAM_NET="10.12.5.10/32" MIRROR_SYNC_PORT="443" - unset ROUTE1_NETWORK ROUTE1_GATEWAY ROUTE1_DESCR ROUTE2_NETWORK ROUTE2_GATEWAY ROUTE2_DESCR - unset NTP_UPSTREAM_SERVERS NTP_PREFER_SERVER -} - -run_ok() { # run_ok <label> <out-file> - local label="$1" out="$2" rc - bash "$SCRIPT" "$out" >/dev/null 2>&1; rc=$? - if [[ "$rc" == 0 ]] && [ -s "$out" ]; then - echo " PASS $label"; PASS=$((PASS+1)) - else - echo " FAIL $label (rc=$rc)"; FAIL=$((FAIL+1)) - fi -} - -run_fail() { # run_fail <want_rc> <label> <out-file> - local want="$1" label="$2" out="$3" rc - bash "$SCRIPT" "$out" >/dev/null 2>&1; rc=$? - if [[ "$rc" == "$want" ]]; then - echo " PASS $label"; PASS=$((PASS+1)) - else - echo " FAIL $label (rc=$rc want=$want)"; FAIL=$((FAIL+1)) - fi -} - -# T1: happy path, no optional routes -- renders, no leftover {{TOKEN}}, well-formed XML -base_env -run_ok "T1 happy path renders" "$TMP/t1.xml" -if ! grep -qE '\{\{[A-Z0-9_]+\}\}' "$TMP/t1.xml" 2>/dev/null; then - echo " PASS T1b no leftover {{TOKEN}} markers"; PASS=$((PASS+1)) -else - echo " FAIL T1b leftover tokens found: $(grep -oE '\{\{[A-Z0-9_]+\}\}' "$TMP/t1.xml" | tr '\n' ' ')"; FAIL=$((FAIL+1)) -fi -if python3 -c "import xml.dom.minidom,sys; xml.dom.minidom.parse(sys.argv[1])" "$TMP/t1.xml" >/dev/null 2>&1; then - echo " PASS T1c well-formed XML"; PASS=$((PASS+1)) -else - echo " FAIL T1c not well-formed XML"; FAIL=$((FAIL+1)) -fi - -# T2: missing required var FAILS -base_env -unset OPNSENSE_HOSTNAME -run_fail 1 "T2 missing required var FAILS (rc 1)" "$TMP/t2.xml" - -# T3: NTP defaults apply when unset (real OPNsense default, not invented) -base_env -bash "$SCRIPT" "$TMP/t3.xml" >/dev/null 2>&1 -if grep -q "0.opnsense.pool.ntp.org" "$TMP/t3.xml" 2>/dev/null; then - echo " PASS T3 NTP default pool applied when unset"; PASS=$((PASS+1)) -else - echo " FAIL T3 NTP default pool missing"; FAIL=$((FAIL+1)) -fi - -# T4: one static route renders correctly, second slot stays empty -base_env -export ROUTE1_NETWORK="fd00:dc2::/64" ROUTE1_GATEWAY="fd00:mesh::2" ROUTE1_DESCR="to DC2" -bash "$SCRIPT" "$TMP/t4.xml" >/dev/null 2>&1 -if grep -q "fd00:dc2::/64" "$TMP/t4.xml" 2>/dev/null && grep -q "<staticroutes>" "$TMP/t4.xml" 2>/dev/null; then - echo " PASS T4 static route 1 renders"; PASS=$((PASS+1)) -else - echo " FAIL T4 static route 1 missing from output"; FAIL=$((FAIL+1)) -fi - -# T5: route network set but gateway missing FAILS -base_env -export ROUTE1_NETWORK="10.0.0.0/24" -unset ROUTE1_GATEWAY -run_fail 1 "T5 route network without gateway FAILS (rc 1)" "$TMP/t5.xml" - -# T6: missing output arg FAILS -base_env -out="$(bash "$SCRIPT" 2>&1)"; rc=$? -if [[ "$rc" == 1 ]] && grep -q "usage: opnsense-render-config.sh" <<<"$out"; then - echo " PASS T6 missing output arg FAILS (rc 1)"; PASS=$((PASS+1)) -else - echo " FAIL T6 missing output arg (rc=$rc)"; FAIL=$((FAIL+1)) -fi - -echo; echo "RESULT: PASS=$PASS FAIL=$FAIL" -[[ "$FAIL" -eq 0 ]] && { echo "ALL PASS"; exit 0; } || exit 1 diff --git a/tests/opnsense-set-interface-v6/run-tests.sh b/tests/opnsense-set-interface-v6/run-tests.sh new file mode 100755 index 0000000..cb44de6 --- /dev/null +++ b/tests/opnsense-set-interface-v6/run-tests.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# tests/opnsense-set-interface-v6/run-tests.sh +# +# Harness for scripts/opnsense-set-interface-v6.sh + scripts/opnsense-set-iface-v6.php. +# +# OFFLINE. Touches no edge. It exercises the arg contract and the SAFETY PROPERTIES, +# which are the entire reason this script is allowed to exist at all: it is the ONLY +# tool in this repo that writes an OPNsense base-interface config, on a LIVE router +# whose DHCP and firewall rules are API-managed and would be destroyed by a wholesale +# config.xml replace. +# +# Pinned here, each one load-bearing: +# 1. DRY BY DEFAULT. --commit is required to write. (D-117's lesson.) +# 2. It NEVER renders or pushes a config.xml. It mutates the vendor's own live +# Config object in place -- the thing D-113 banned is the wholesale push. +# 3. The interface COUNT is asserted unchanged before saving, and re-asserted on a +# fresh read-back. A silent structural drop on a live router must fail loud. +# 4. Values are validated BEFORE they reach the router (both layers). +# 5. Remote commands go through `sh -s`. root's shell on the edge is TCSH and a +# $(...) in a quoted remote command dies QUIETLY ("Illegal variable name"). +# 6. Ground truth is the KERNEL (ifconfig), never the config file or a self-report. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SDIR="$(cd "$HERE/../../scripts" && pwd)" +SH="$SDIR/opnsense-set-interface-v6.sh" +PHP="$SDIR/opnsense-set-iface-v6.php" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +# 0. both exist and the shell half parses +[ -f "$SH" ] && ok || bad "opnsense-set-interface-v6.sh missing" +[ -f "$PHP" ] && ok || bad "opnsense-set-iface-v6.php missing" +bash -n "$SH" 2>/dev/null && ok || bad "opnsense-set-interface-v6.sh does not parse" +# php is not installed on the jumphost (it lives on the edge). Lint only if present -- +# never silently skip-and-pass a real syntax check when the tool IS available. +if command -v php >/dev/null 2>&1; then + php -l "$PHP" >/dev/null 2>&1 && ok || bad "opnsense-set-iface-v6.php does not parse (php -l)" +else + ok # documented: no php on this host; the edge parses it at run time +fi + +# 1. arg contract -- every missing arg must fail, not default to something. +"$SH" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "no args must fail" +"$SH" 10.10.0.1 >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "host only must fail" +"$SH" 10.10.0.1 lan >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "no address must fail" +"$SH" 10.10.0.1 lan 2602:f3e2:f01:100::1 >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "no prefixlen must fail" + +# 2. VALIDATION BEFORE THE ROUTER. A bad value must die locally -- it must never be +# shipped to a live edge to be rejected there. +out="$("$SH" 10.10.0.1 lan 10.10.0.1 64 2>&1)" +printf '%s' "$out" | grep -q "not an IPv6 address" && ok || bad "a v4 address was not rejected locally" +out="$("$SH" 10.10.0.1 lan 2602:f3e2:f01:100::1 129 2>&1)" +printf '%s' "$out" | grep -q "out of range" && ok || bad "prefixlen 129 was not rejected locally" +out="$("$SH" 10.10.0.1 lan 2602:f3e2:f01:100::1 abc 2>&1)" +printf '%s' "$out" | grep -q "not a prefix length" && ok || bad "a non-numeric prefixlen was not rejected" + +# 3. no key -> refuse. (Validation runs first, so use an otherwise-valid call.) +out="$(env -u OPNSENSE_SSH_KEY "$SH" 10.10.0.1 lan 2602:f3e2:f01:100::1 64 2>&1)" +printf '%s' "$out" | grep -q 'OPNSENSE_SSH_KEY not set' && ok || bad "missing OPNSENSE_SSH_KEY was not refused" + +# 4. DRY BY DEFAULT -- in BOTH halves. +grep -q '\-\-commit' "$SH" && ok || bad "the wrapper has no --commit flag" +grep -q '\-\-commit' "$PHP" && ok || bad "the php has no --commit flag" +grep -q 'DRY RUN -- nothing written, nothing applied' "$SH" && ok || bad "the wrapper does not announce its dry run" +grep -q 'DRY RUN -- nothing was written' "$PHP" && ok || bad "the php does not announce its dry run" +# the php must not save unless committed +grep -q 'if (!\$commit)' "$PHP" && ok || bad "the php has no dry-run guard before the write" + +# 5. IT MUST NEVER RENDER OR PUSH A config.xml. This is the D-113 red line: a +# wholesale replace would wipe the API-managed Kea DHCP and the firewall rules. +grep -qiE 'config\.xml\.tmpl|render-config|scp .*config\.xml|>[[:space:]]*/conf/config\.xml' "$SH" \ + && bad "the wrapper renders or pushes a config.xml -- THE FORBIDDEN CLOBBER PATH" || ok +grep -qE 'file_put_contents|fopen\(.*/conf/config\.xml' "$PHP" \ + && bad "the php writes config.xml directly instead of via the vendor's Config object" || ok +# it must go through the vendor's own singleton +grep -q 'Config::getInstance()' "$PHP" && ok || bad "the php does not use OPNsense's own Config singleton" +grep -q 'use OPNsense\\Core\\Config;' "$PHP" && ok || bad "the php no longer imports OPNsense's Config" + +# 6. THE STRUCTURAL GUARD. Serializing the vendor's object must not drop nodes, and +# if it ever does, a LIVE ROUTER is corrupt. Fail loud, do not discover it later. +grep -q 'iface_count_before' "$PHP" && ok || bad "the interface-count guard is gone" +grep -q 'NOT SAVING' "$PHP" && ok || bad "the php no longer refuses to save on a count mismatch" +grep -q 'CONFIG DAMAGED' "$PHP" && ok || bad "the post-save count re-check is gone" +# and it must READ BACK from a fresh load, not trust its in-memory copy +grep -q 'forceReload' "$PHP" && ok || bad "the php does not re-read from disk -- it would trust its own memory" +grep -q 'read-back mismatch' "$PHP" && ok || bad "the php does not verify what it actually wrote" + +# 7. only ipaddrv6/subnetv6 may be touched. Anything else on a live edge is scope creep +# with a router at the end of it. +touched="$(grep -oE '\$node->[a-zA-Z0-9_]+ =' "$PHP" | sort -u | sed 's/\$node->//; s/ =//')" +[ "$(printf '%s\n' "$touched" | grep -c .)" -eq 2 ] && ok \ + || bad "the php assigns to more than 2 interface fields: $(printf '%s' "$touched" | tr '\n' ' ')" +printf '%s\n' "$touched" | grep -qx 'ipaddrv6' && ok || bad "the php no longer sets ipaddrv6" +printf '%s\n' "$touched" | grep -qx 'subnetv6' && ok || bad "the php no longer sets subnetv6" +grep -q 'UNTOUCHED' "$PHP" && ok || bad "the php no longer shows that the v4 address is left alone" + +# 8. THE TCSH TRAP. root's shell on the edge is tcsh; a $(...) inside a quoted remote +# command dies QUIETLY. Every remote command must be fed to `sh -s`. +grep -q 'sh -s' "$SH" && ok || bad "remote commands are not fed to 'sh -s' -- the TCSH trap (root's shell is tcsh)" +grep -qi 'tcsh' "$SH" && ok || bad "the tcsh trap is no longer documented -- the next person will re-learn it the hard way" + +# 9. saving config != applying it. The reconfigure step must exist, and the php must +# say plainly that it does NOT apply. +grep -q 'configctl interface reconfigure' "$SH" && ok || bad "the wrapper never applies the change (configctl reconfigure)" +grep -q 'NOT yet applied to the running kernel' "$PHP" && ok || bad "the php no longer warns that a save is not an apply" + +# 10. GROUND TRUTH. The kernel decides, not the config file and not a self-report. +grep -q 'ifconfig' "$SH" && ok || bad "the wrapper never checks the KERNEL for the address" +grep -q 'GROUND TRUTH' "$SH" && ok || bad "the ground-truth read-back is gone" +grep -q 'is NOT on' "$SH" && ok || bad "the wrapper does not FAIL when the address is absent after reconfigure" + +# 11. the D-113 amendment must stay cited -- this script only exists because the API +# cannot do this, and that finding is the whole justification for the SSH path. +grep -q 'D-113' "$SH" && ok || bad "the wrapper no longer cites D-113 (why this is not the REST API)" +grep -q 'D-113' "$PHP" && ok || bad "the php no longer cites D-113" +grep -qi 'interfaces.php' "$SH" && ok || bad "the measured evidence (legacy /interfaces.php page) is no longer cited" + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "opnsense-set-interface-v6: $pass/$total PASS"; exit 0; fi +echo "opnsense-set-interface-v6: $fail/$total FAIL"; exit 1 diff --git a/tests/phase-00-maas-standup/fakebin/maas b/tests/phase-00-maas-standup/fakebin/maas old mode 100644 new mode 100755 diff --git a/tests/phase-00-maas-standup/run-tests.sh b/tests/phase-00-maas-standup/run-tests.sh index 15a26df..91554a8 100644 --- a/tests/phase-00-maas-standup/run-tests.sh +++ b/tests/phase-00-maas-standup/run-tests.sh @@ -86,15 +86,15 @@ echo echo "=== \$DC selector wiring (DOCFIX-166; tooling gap register #15 sub-item 2) ===" -run 0 "DC=dc0 explicit: identical no-drift six-plane done" done dc0 +run 0 "DC=vr0-dc0 explicit: identical no-drift six-plane done" done vr0-dc0 has 'no drift' has 'OK \(dryrun\) -- topology consistent with D-052/D-053' -run 0 "DC=dc1: net selector no-ops (D-101), same hardcoded PLANES table" done dc1 +run 0 "DC=vr1-dc0: net selector no-ops (D-101 inheritance), same hardcoded PLANES table" done vr1-dc0 has 'no drift' has 'OK \(dryrun\) -- topology consistent with D-052/D-053' -run_dc 1 "DC=dc2: net selector fails loud (NetBox gap)" dc2 +run_dc 1 "DC=vr1-dc1: net selector fails loud (NetBox gap)" vr1-dc1 has 'no assigned network literals yet' absent 'DO:|WOULD:' # must exit before any MAAS interaction diff --git a/tests/phase-02/fakebin/jq b/tests/phase-02/fakebin/jq old mode 100644 new mode 100755 diff --git a/tests/phase-02/fakebin/juju b/tests/phase-02/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-03-adminrc/fakebin/juju b/tests/phase-03-adminrc/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-03-adminrc/fakebin/openssl b/tests/phase-03-adminrc/fakebin/openssl old mode 100644 new mode 100755 diff --git a/tests/phase-03-adminrc/fakebin/openstack b/tests/phase-03-adminrc/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-03/fakebin/jq b/tests/phase-03/fakebin/jq old mode 100644 new mode 100755 diff --git a/tests/phase-03/fakebin/juju b/tests/phase-03/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-04-create/fakebin/maas b/tests/phase-04-create/fakebin/maas old mode 100644 new mode 100755 diff --git a/tests/phase-04-create/fakebin/openstack b/tests/phase-04-create/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-04-internal-cert-san/fakebin/juju b/tests/phase-04-internal-cert-san/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-04-internal-cert-san/fakebin/openstack b/tests/phase-04-internal-cert-san/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-04/fakebin/maas b/tests/phase-04/fakebin/maas old mode 100644 new mode 100755 diff --git a/tests/phase-04/fakebin/openstack b/tests/phase-04/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-05-amphora/fakebin/curl b/tests/phase-05-amphora/fakebin/curl old mode 100644 new mode 100755 diff --git a/tests/phase-05-amphora/fakebin/juju b/tests/phase-05-amphora/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-05-amphora/fakebin/openstack b/tests/phase-05-amphora/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-05-amphora/fakebin/sha256sum b/tests/phase-05-amphora/fakebin/sha256sum old mode 100644 new mode 100755 diff --git a/tests/phase-05-amphora/fakebin/wget b/tests/phase-05-amphora/fakebin/wget old mode 100644 new mode 100755 diff --git a/tests/phase-05/fakebin/juju b/tests/phase-05/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-05/fakebin/openstack b/tests/phase-05/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-06-bootstrap/fakebin/curl b/tests/phase-06-bootstrap/fakebin/curl old mode 100644 new mode 100755 diff --git a/tests/phase-06-bootstrap/fakebin/openstack b/tests/phase-06-bootstrap/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-06-bootstrap/fakebin/sha256sum b/tests/phase-06-bootstrap/fakebin/sha256sum old mode 100644 new mode 100755 diff --git a/tests/phase-06-bootstrap/fakebin/wget b/tests/phase-06-bootstrap/fakebin/wget old mode 100644 new mode 100755 diff --git a/tests/phase-06-capi-stack/fakebin/ssh b/tests/phase-06-capi-stack/fakebin/ssh old mode 100644 new mode 100755 diff --git a/tests/phase-06-k8s-bootstrap/fakebin/ssh b/tests/phase-06-k8s-bootstrap/fakebin/ssh old mode 100644 new mode 100755 diff --git a/tests/phase-06-kubeconfig-gate/fakebin/kubectl b/tests/phase-06-kubeconfig-gate/fakebin/kubectl old mode 100644 new mode 100755 diff --git a/tests/phase-06-kubeconfig-gate/fakebin/ssh b/tests/phase-06-kubeconfig-gate/fakebin/ssh old mode 100644 new mode 100755 diff --git a/tests/phase-06-mgmt-vm/fakebin/openstack b/tests/phase-06-mgmt-vm/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-06-net-setup/fakebin/openstack b/tests/phase-06-net-setup/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/phase-07-conductor-graft/fakebin/juju b/tests/phase-07-conductor-graft/fakebin/juju old mode 100644 new mode 100755 diff --git a/tests/phase-07-conductor-graft/fakebin/kubectl b/tests/phase-07-conductor-graft/fakebin/kubectl old mode 100644 new mode 100755 diff --git a/tests/phase-07-conductor-graft/fakebin/openstack b/tests/phase-07-conductor-graft/fakebin/openstack old mode 100644 new mode 100755 diff --git a/tests/prereqs/run-tests.sh b/tests/prereqs/run-tests.sh new file mode 100644 index 0000000..a61afb1 --- /dev/null +++ b/tests/prereqs/run-tests.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# tests/prereqs/run-tests.sh +# Harness for scripts/prereqs/* -- exercises the mode-parse / guard / --check / +# --dry-run paths only. It NEVER runs a real apt install (that needs root and +# would mutate the machine; same posture as the opnsense-* harnesses). ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PREQ="$(cd "$HERE/../../scripts/prereqs" && pwd)" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } +# t <desc> <expected-rc> <cmd...> +t() { local d="$1" exp="$2"; shift 2; "$@" >/dev/null 2>&1; local rc=$?; if [ "$rc" = "$exp" ]; then ok; else bad "$d (rc=$rc want=$exp)"; fi; } + +INSTALLERS="install-opentofu install-virtualization install-jq install-image-tools install-apparmor-libvirt" + +# 1. lib-prereq sourcing + mode parsing +. "$PREQ/lib-prereq.sh" +type pq_have >/dev/null 2>&1 && ok || bad "lib-prereq did not define pq_have" +pq_parse_mode --check >/dev/null 2>&1; [ "${PQ_CHECK:-x}" = "1" ] && ok || bad "parse --check set PQ_CHECK" +pq_parse_mode --dry-run >/dev/null 2>&1; [ "${PQ_DRYRUN:-x}" = "1" ] && ok || bad "parse --dry-run set PQ_DRYRUN" +pq_parse_mode --bogus >/dev/null 2>&1; [ "$?" = "2" ] && ok || bad "parse bad arg -> 2" +pq_parse_mode --help >/dev/null 2>&1; [ "$?" = "10" ] && ok || bad "parse --help -> 10" + +# 2. pq_run: dry-run must NOT execute; real mode must +tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT +PQ_DRYRUN=1 pq_run "sentinel" -- touch "$tmp/nope" >/dev/null 2>&1 +[ ! -e "$tmp/nope" ] && ok || bad "pq_run dry-run executed the command" +PQ_DRYRUN=0 pq_run "sentinel" -- touch "$tmp/yes" >/dev/null 2>&1 +[ -e "$tmp/yes" ] && ok || bad "pq_run real-mode did not execute" + +# 3. each installer: --help=0, bad-arg=2, --dry-run=0 (never a real install) +for s in $INSTALLERS; do + t "$s --help" 0 bash "$PREQ/$s.sh" --help + t "$s bad-arg" 2 bash "$PREQ/$s.sh" --bogus + t "$s --dry-run" 0 bash "$PREQ/$s.sh" --dry-run +done + +# 4. orchestrator +t "install-all --help" 0 bash "$PREQ/install-all.sh" --help +t "install-all bad-arg" 2 bash "$PREQ/install-all.sh" --bogus +t "install-all --dry-run" 0 bash "$PREQ/install-all.sh" --dry-run + +# 5. check-prereqs is read-only and exits 0 (all present) or 1 (something missing) +bash "$PREQ/check-prereqs.sh" >/dev/null 2>&1; rc=$? +{ [ "$rc" = "0" ] || [ "$rc" = "1" ]; } && ok || bad "check-prereqs rc=$rc (want 0|1)" + +# 5b. each tool row appears at most once -- guards the line() return-value bug +# (a helper whose last cmd is a failing test makes `A && line OK || line WARN` +# print BOTH). Found on first dogfood run; fixed via line() return 0. +crep="$(bash "$PREQ/check-prereqs.sh" 2>&1)" +dupbad=0 +for tool in tofu virsh jq qemu-img curl python3 sudo apparmor-libvirt; do + n="$(printf '%s\n' "$crep" | grep -cE "\][ ]+${tool}[ ]+")" + [ "$n" -le 1 ] || dupbad=1 +done +[ "$dupbad" = "0" ] && ok || bad "check-prereqs printed a tool row more than once" + +# 6. an absent tool's --dry-run must reach + print the dry-run plan (marker) +if pq_have jq; then + ok # jq present -> install-jq short-circuits at OK before the plan; marker not expected +else + bash "$PREQ/install-jq.sh" --dry-run 2>&1 | grep -q '\[dry-run\]' && ok || bad "install-jq --dry-run lacked [dry-run] marker" +fi + +# 7. install-apparmor-libvirt: the DETECTION is what has to be right -- a false OK +# here means every VR1 VM defines-then-fails-to-start with a bare qemu +# "Permission denied" and nothing naming AppArmor. Drive it off a pool parent +# that is guaranteed absent from the profile, so the negative path is +# deterministic regardless of what this host happens to have applied. +AAF="/etc/apparmor.d/local/abstractions/libvirt-qemu" +BOGUS="/var/lib/libvirt/zz-not-granted-$$" +if [ -d /etc/apparmor.d ]; then + # 7a. --check must FAIL (rc 1) for a parent that was never granted + VR1_POOL_PARENT="$BOGUS" bash "$PREQ/install-apparmor-libvirt.sh" --check >/dev/null 2>&1 + [ "$?" = "1" ] && ok || bad "apparmor --check returned OK for an ungranted pool parent (false OK)" + + # 7b. --dry-run must plan the fix but mutate NOTHING (the profile must not gain the rule) + out="$(VR1_POOL_PARENT="$BOGUS" bash "$PREQ/install-apparmor-libvirt.sh" --dry-run 2>&1)" + printf '%s\n' "$out" | grep -q '\[dry-run\]' && ok || bad "apparmor --dry-run printed no [dry-run] plan" + if [ -r "$AAF" ] && grep -qF "$BOGUS" "$AAF"; then + bad "apparmor --dry-run MUTATED $AAF (wrote the rule)" + else + ok + fi + + # 7c. the rule it writes must be exactly the grant qemu needs + printf '%s\n' "$out" | grep -qF "$BOGUS/** rwk," && ok || bad "apparmor plan lacks the '<parent>/** rwk,' grant" +else + # no AppArmor on this host: the installer must report N/A and SUCCEED, not fail + VR1_POOL_PARENT="$BOGUS" bash "$PREQ/install-apparmor-libvirt.sh" --check >/dev/null 2>&1 + [ "$?" = "0" ] && ok || bad "apparmor --check must exit 0 (N/A) on a non-AppArmor host" + ok; ok; ok # the three AppArmor-host assertions do not apply here +fi + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "prereqs: $pass/$total PASS"; exit 0; fi +echo "prereqs: $fail/$total FAIL"; exit 1 diff --git a/tests/reenroll-hosts/fakebin/maas b/tests/reenroll-hosts/fakebin/maas old mode 100644 new mode 100755 diff --git a/tests/reenroll-hosts/run-tests.sh b/tests/reenroll-hosts/run-tests.sh index 0ed29f7..f37f9ee 100644 --- a/tests/reenroll-hosts/run-tests.sh +++ b/tests/reenroll-hosts/run-tests.sh @@ -50,18 +50,26 @@ has 'NOT-ENROLLED' has 'read-only check mode; no changes made' -run 0 "DC=dc0 explicit: identical --check output" dc0 +run 0 "DC=vr0-dc0 explicit: identical --check output" vr0-dc0 has 'NOT-ENROLLED' has 'read-only check mode; no changes made' -run_dc 1 "DC=dc1: hosts selector fails loud (no per-DC host data)" dc1 +run_dc 1 "DC=vr1-dc0: hosts selector fails loud (no per-DC host data)" vr1-dc0 has 'no enrolled hosts yet' absent 'Current host status' # must exit before report() ever runs -run_dc 1 "DC=dc2: net selector fails loud first (NetBox gap)" dc2 +run_dc 1 "DC=vr1-dc1: net selector fails loud first (NetBox gap)" vr1-dc1 has 'no assigned network literals yet' absent 'Current host status' +# D-119 REGRESSION GUARD: the bare dcN tokens are RETIRED. 'dc0' used to mean +# VR0's LIVE cloud here but VR1's FIRST DC in the NetBox importer -- one string, +# two clouds. Accepting it silently is how the off-by-one comes back. +for _bare in dc0 dc1 dc2; do + run_dc 1 "DC=$_bare: RETIRED bare selector is REJECTED (D-119)" "$_bare" + has 'RETIRED' +done + run_dc 1 "DC=bogus: unknown token fails loud" bogus has "unknown DC 'bogus'" diff --git a/tests/roles-aggregates-import/run-tests.sh b/tests/roles-aggregates-import/run-tests.sh new file mode 100755 index 0000000..4d0736c --- /dev/null +++ b/tests/roles-aggregates-import/run-tests.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# tests/roles-aggregates-import/run-tests.sh +# +# Harness for netbox/roles-aggregates-import.py. OFFLINE -- touches no NetBox. +# +# THIS SCRIPT SHIPPED WITH NO HARNESS, and that is not incidental to the bug it +# carried. It was only ever exercised against an EMPTY NetBox. The moment it met +# one holding the real upstream draft (2026-07-13) it created 4 roles, hit a 400 +# on the 5th, and DIED -- leaving the IPAM apex half-populated with no RIRs and +# no aggregates, and no rollback. +# +# Two properties are pinned here, both learned from that failure: +# 1. NetBox enforces UNIQUE ROLE NAMES, not just unique slugs. The legacy `repl` +# role already holds the name "Replication", so the six-plane role (slug +# `replication`) MUST NOT be named "Replication". +# 2. The importer must PREFLIGHT every name/slug before writing ANYTHING. An +# IPAM importer that can die partway corrupts the thing it exists to populate. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/roles-aggregates-import.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +# This script needs pynetbox (unlike the stdlib tools). On a host without it, it +# must say so CLEANLY rather than traceback -- so accept either. +out="$(python3 "$S" --help 2>&1)" +if printf '%s' "$out" | grep -q "pynetbox not installed"; then ok # clean, actionable +elif printf '%s' "$out" | grep -q -- "--commit"; then ok # real help text +else bad "--help neither printed help nor a clean 'pynetbox not installed' message"; fi + +# DRY BY DEFAULT -- this one already was, and must stay so. +grep -q -- '--commit' "$S" && ok || bad "no --commit flag" +grep -q 'default: preview only, no writes' "$S" && ok || bad "no longer documents preview-by-default" + +# 1. THE NAME COLLISION. Naming the six-plane role "Replication" 400s against any +# NetBox carrying the draft, because legacy slug `repl` holds that name. +grep -q '"Replication Plane"' "$S" && ok \ + || bad "the six-plane replication role is not named 'Replication Plane' -- it will 400 against the real draft (legacy slug 'repl' holds the name 'Replication')" +grep -qE '\("replication", "Replication",' "$S" \ + && bad "the six-plane role is named exactly 'Replication' -- COLLIDES with legacy slug 'repl'" || ok +# the slug must NOT drift: lib-net.sh SPACES6 and dc-dc-prefixes-import.py look it up. +grep -q '"replication", "Replication Plane"' "$S" && ok \ + || bad "the replication slug drifted -- lib-net.sh SPACES6 and the prefix importer look up slug 'replication'" + +# 2. PREFLIGHT. Nothing may be written until the whole plan is known viable. +grep -q 'PREFLIGHT FAILED -- nothing was written' "$S" && ok \ + || bad "the preflight is gone -- a name collision can again leave the apex HALF-WRITTEN" +grep -q 'NetBox enforces UNIQUE NAMES, not just slugs' "$S" && ok \ + || bad "the preflight no longer checks NAME collisions (only slugs) -- that IS the bug" +# the preflight must run BEFORE the first create +pf=$(grep -n 'PREFLIGHT' "$S" | head -1 | cut -d: -f1) +cr=$(grep -n 'nb.ipam.roles.create' "$S" | head -1 | cut -d: -f1) +[ -n "$pf" ] && [ -n "$cr" ] && [ "$pf" -lt "$cr" ] && ok \ + || bad "preflight does not run BEFORE the first create -- half-written apex is back" + +# 2026-07-14: EVERY die() downstream of the first write must be COVERED by the +# preflight. The old harness asserted only that the WORD "PREFLIGHT" appeared before +# the first create -- so it went green while the ARIN lookup and the ORG_ULA_48 +# validation still ran AFTER 5 roles had been committed. A typo'd ULA, or an apex +# with no ARIN, left the PRODUCTION IPAM half-populated with no rollback. That is +# the very failure this preflight exists to prevent, still armed one section lower. +# Assert by POSITION, not by vocabulary. +arin=$(grep -n "rirs.get(slug=\"arin\")" "$S" | head -1 | cut -d: -f1) +[ -n "$arin" ] && [ "$arin" -lt "$cr" ] && ok \ + || bad "the ARIN RIR check runs AFTER the first role create -- a missing ARIN half-writes the apex" +ula=$(grep -n 'validate_ula_48(' "$S" | sed -n '2p' | cut -d: -f1) # [1] is the def +[ -n "$ula" ] && [ "$ula" -lt "$cr" ] && ok \ + || bad "ORG_ULA_48 is validated AFTER the first role create -- a typo half-writes the apex" +grep -q 'a typo must not cost a half-populated apex' "$S" && ok \ + || bad "the ULA preflight rationale is gone -- someone will move it back below the writes" + +# WAF trap: get_nb must set the accepted User-Agent on the pynetbox session, or +# every upstream call 403s (looks like auth failure; references/platform-traps.md). +grep -q 'http_session.headers\["User-Agent"\] = "curl/8.5.0"' "$S" && ok \ + || bad "get_nb does not set the WAF-safe User-Agent -- upstream writes will 403" + +# UPSTREAM-WRITE GUARD: --commit to a non-sandbox host must require --yes-write-upstream. +# The WAF 403 used to be the only accidental-safety; the UA fix removed it. +grep -q -- '--yes-write-upstream' "$S" && ok \ + || bad "no --yes-write-upstream flag -- an accidental --commit can hit the production apex" +grep -q 'SANDBOX_HOSTS' "$S" && ok \ + || bad "no SANDBOX_HOSTS gate -- the upstream-write guard is gone" +grep -q 'REFUSING to --commit' "$S" && ok \ + || bad "the upstream-write refusal message is gone" +# the guard must gate on args.yes_write_upstream before writing +grep -q 'not args.yes_write_upstream' "$S" && ok \ + || bad "the guard does not actually check the --yes-write-upstream flag" + +# 3. the six-plane slugs must match lib-net.sh SPACES6 exactly. +for slug in provider-public metal-admin metal-internal data-tenant replication; do + grep -q "\"$slug\"" "$S" && ok || bad "six-plane role slug '$slug' is gone" +done +# storage is deliberately NOT created -- it pre-exists upstream and is reused. +grep -q 'already exists -- intentionally omitted' "$S" && ok \ + || bad "the note that 'storage' is reused (not created) is gone" + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "roles-aggregates-import: $pass/$total PASS"; exit 0; fi +echo "roles-aggregates-import: $fail/$total FAIL"; exit 1 diff --git a/tests/sandbox-fidelity-check/run-tests.sh b/tests/sandbox-fidelity-check/run-tests.sh new file mode 100755 index 0000000..92bb14f --- /dev/null +++ b/tests/sandbox-fidelity-check/run-tests.sh @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# tests/sandbox-fidelity-check/run-tests.sh +# +# Harness for netbox/sandbox-fidelity-check.py. Offline: builds synthetic upstream +# / sandbox JSON dumps and asserts the checker's verdict. Touches no NetBox. +# +# WHY THIS HARNESS EXISTS AT ALL (2026-07-14): the checker SHIPPED WITH NO HARNESS, +# and it was the script we were citing as PROOF the sandbox was a faithful replica. +# It could return a FALSE GREEN: `unexpected = extra - expected` proves the delta is +# a SUBSET of the plan but NEVER proved the plan was CARRIED OUT. With ZERO of the +# 36 DC prefixes created, extra={} -> unexpected={} -> it printed "Nothing lost, +# nothing stray" and exited 0. EXPECTED_NEW was an upper bound masquerading as an +# assertion. T3 is that exact scenario, now caught. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +CHECK="$REPO/netbox/sandbox-fidelity-check.py" +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +P=0; F=0 +ok() { echo " [PASS] $1"; P=$((P+1)); } +no() { echo " [FAIL] $1"; F=$((F+1)); } + +[ -f "$CHECK" ] || { echo "FAIL: $CHECK not found"; exit 1; } + +# --- fixtures ----------------------------------------------------------------- +# A minimal upstream draft: one shared prefix. The sandbox must reproduce it +# EXACTLY, and may add only what EXPECTED_NEW allows. +mk() { # mk <file> <json> + printf '%s' "$2" > "$TMP/$1" +} + +SHARED='{"prefix":"10.99.0.0/24","status":"active","role":"storage","scope_type":"dcim.site","scope_id":1,"description":"shared","is_pool":false,"mark_utilized":false}' +# an EXPECTED delta prefix in DUMP FORMAT (scope_site slug, as prod-draft-dump.py +# rewrites it; scope_id is popped on the way in). 10.12.4.0/22 is intended for site +# vr1-dc0 per EXPECTED_PREFIX_SCOPE. +DELTA_OK='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":"dcim.site","scope_site":"vr1-dc0","description":"VR1 DC0 provider-public","is_pool":false,"mark_utilized":false}' +# the SAME expected prefix but UNSCOPED -- the 17-prefix-scope-drop bug's shape +DELTA_NOSCOPE='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":null,"description":"VR1 DC0 provider-public","is_pool":false,"mark_utilized":false}' +# the SAME expected prefix bound to the WRONG DC site (vr1-dc1) -- the D-117 near-miss +DELTA_WRONGSITE='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":"dcim.site","scope_site":"vr1-dc1","description":"wrong site","is_pool":false,"mark_utilized":false}' +# a stray object nobody planned +STRAY='{"prefix":"192.0.2.0/24","status":"active","role":"storage","scope_type":"dcim.site","scope_id":1,"description":"stray","is_pool":false,"mark_utilized":false}' + +up_json() { printf '{"ipam/prefixes":[%s]}' "$SHARED"; } +run() { # run <upstream-json> <sandbox-json> ; echoes rc + printf '%s' "$1" > "$TMP/u.json"; printf '%s' "$2" > "$TMP/s.json" + python3 "$CHECK" --upstream "$TMP/u.json" --sandbox "$TMP/s.json" >"$TMP/out" 2>&1 + echo $? +} + +echo "=== sandbox-fidelity-check ===" + +# T1: THE POSITIVE PATH. A faithful replica carrying the COMPLETE planned delta +# must pass. The expected set is DERIVED FROM THE CHECKER ITSELF (its own +# EXPECTED_NEW), never re-typed here -- a harness that hardcodes its own copy of +# the expectation cannot detect the two drifting apart. +python3 - "$CHECK" "$TMP" <<'PY' +import json, re, sys +src = open(sys.argv[1]).read() +tmp = sys.argv[2] +# exec only the declarative prologue (up to the comparison loop), with the two +# argparse-required inputs stubbed, to harvest EXPECTED_NEW without running the diff. +head = src.split("bad=0")[0] +head = re.sub(r"^ap\s*=.*$|^ap\.add_argument.*$|^args\s*=.*$|^up\s*=.*$|^sb\s*=.*$", + "", head, flags=re.M) +ns = {} +exec(compile(head, "checker-prologue", "exec"), ns) +SHARED = {"prefix":"10.99.0.0/24","status":"active","role":"storage", + "scope_type":"dcim.site","scope_id":1,"description":"shared", + "is_pool":False,"mark_utilized":False} +sandbox = {} +upstream = {} +for ep, key in ns["KEY"].items(): + upstream[ep] = [SHARED] if ep == "ipam/prefixes" else [] + rows = [SHARED] if ep == "ipam/prefixes" else [] + for v in sorted(ns["EXPECTED_NEW"].get(ep, set())): + rec = {key: v} + if ep == "ipam/prefixes": + rec.update({"status":"active","role":"provider-public", + "description":"planned","is_pool":False,"mark_utilized":False}) + sc = ns["EXPECTED_PREFIX_SCOPE"].get(v) # INTENDED scope, harvested from the checker itself + if sc is not None: + kind, slug = sc + rec["scope_type"] = "dcim.site" if kind == "site" else "dcim.region" + rec["scope_site" if kind == "site" else "scope_region"] = slug + else: + rec["scope_type"] = None # intentionally-unscoped org container + rows.append(rec) + sandbox[ep] = rows +json.dump(upstream, open(f"{tmp}/u1.json","w")) +json.dump(sandbox, open(f"{tmp}/s1.json","w")) +PY +python3 "$CHECK" --upstream "$TMP/u1.json" --sandbox "$TMP/s1.json" >"$TMP/out" 2>&1; rc=$? +[ "$rc" = 0 ] && ok "T1 faithful replica WITH the complete planned delta -> exit 0" \ + || { no "T1 the complete, correct delta should PASS (rc=$rc)"; sed 's/^/ /' "$TMP/out" | tail -6; } + +# T2: a shared object with a MANGLED field -> FAIL (the original 17-prefix bug, +# as it appeared on SHARED objects; the check already caught this) +MANGLED='{"prefix":"10.99.0.0/24","status":"active","role":"storage","scope_type":null,"scope_id":null,"description":"shared","is_pool":false,"mark_utilized":false}' +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s]}' "$MANGLED")") +[ "$rc" = 1 ] && grep -q "FIELD DRIFT" "$TMP/out" && ok "T2 dropped scope on a SHARED object -> FIELD DRIFT, exit 1" \ + || { no "T2 should fail on shared-object field drift (rc=$rc)"; sed 's/^/ /' "$TMP/out"; } + +# T3: THE FALSE GREEN. The planned delta was NOT applied (zero DC prefixes created). +# A subset-only check calls this clean. It must FAIL. +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s]}' "$SHARED")") +# ^ T1 is the no-delta-expected case; to exercise EXPECTED-BUT-ABSENT we need the +# checker's own EXPECTED_NEW to be non-empty, which it always is. T1 above already +# proves it: with an empty sandbox delta, the 36+ EXPECTED_NEW prefixes are absent. +grep -q "EXPECTED-BUT-ABSENT" "$TMP/out" \ + && ok "T3 FALSE GREEN CAUGHT: the planned delta was never applied -> EXPECTED-BUT-ABSENT" \ + || no "T3 the false green is BACK -- a subset-only check passes an unapplied plan" +[ "$rc" = 1 ] && ok "T3 an unapplied plan exits 1 (was exit 0 -- 'Nothing lost, nothing stray')" \ + || no "T3 unapplied plan still exits $rc" + +# T4: an UNEXPECTED stray write -> FAIL +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$STRAY")") +[ "$rc" = 1 ] && grep -q "UNEXPECTED EXTRA" "$TMP/out" && ok "T4 a stray write -> UNEXPECTED EXTRA, exit 1" \ + || { no "T4 should fail on a stray write (rc=$rc)"; sed 's/^/ /' "$TMP/out"; } + +# T5: an object MISSING from the sandbox -> FAIL +rc=$(run "$(up_json)" '{"ipam/prefixes":[]}') +[ "$rc" = 1 ] && grep -q "MISSING FROM SANDBOX" "$TMP/out" && ok "T5 a lost upstream object -> MISSING, exit 1" \ + || { no "T5 should fail on a lost object (rc=$rc)"; sed 's/^/ /' "$TMP/out"; } + +# T6: DELTA SCOPE-CORRECTNESS. An EXPECTED new prefix whose INTENDED scope was +# DROPPED (10.12.4.0/22 should be site vr1-dc0, arrives unscoped) must fail. The +# shared-object drift loop CANNOT see this (the object exists only in the +# sandbox) -- it is precisely the 17-prefix scope-drop bug, relocated into the +# delta where the old check was blind to it. +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_NOSCOPE")") +grep -q "!= intended" "$TMP/out" && ok "T6 an expected DELTA prefix with its scope DROPPED -> caught" \ + || { no "T6 delta scope-correctness is blind again"; sed 's/^/ /' "$TMP/out"; } +[ "$rc" = 1 ] && ok "T6 a scope-dropped delta prefix exits 1" || no "T6 scope-dropped delta exits $rc" + +# T7: a correctly-SCOPED expected delta prefix (right target) is NOT flagged +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_OK")") +grep -q "!= intended" "$TMP/out" && no "T7 a correctly-scoped delta prefix was wrongly flagged" \ + || ok "T7 a correctly-scoped delta prefix is NOT flagged (guard is not a blanket refusal)" + +# T9: WRONG-TARGET scope. 10.12.4.0/22 is intended for vr1-dc0; bound to vr1-dc1 it +# must fail. This is the D-117 near-miss (a DC's prefixes silently bound to the +# OTHER DC's site) -- a mere "has a scope?" test was blind to it; asserting scope +# == INTENDED catches it. +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_WRONGSITE")") +[ "$rc" = 1 ] && grep -q "!= intended" "$TMP/out" && ok "T9 a WRONG-TARGET scope (DC0 prefix on vr1-dc1) -> caught, exit 1" \ + || { no "T9 wrong-target scope not caught (rc=$rc)"; sed 's/^/ /' "$TMP/out"; } + +# T10: the legitimately-UNSCOPED role container (10.10.0.0/16, intended None) present +# as unscoped must NOT be flagged. This is the exact false-positive the fix +# removes: the old "has a scope?" heuristic FAILED a faithful sandbox on this +# prefix (mirroring the 27 unscoped role containers already in the upstream apex). +DELTA_UNSCOPED_OK='{"prefix":"10.10.0.0/16","status":"container","role":"office","scope_type":null,"description":"office role container","is_pool":false,"mark_utilized":false}' +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_UNSCOPED_OK")") +grep -q "!= intended" "$TMP/out" && no "T10 an intentionally-unscoped role container was wrongly flagged (false positive is back)" \ + || ok "T10 an intentionally-unscoped role container (10.10.0.0/16) is NOT flagged" + +# T11: a D-120 ip-range DROPPED from an otherwise-complete sandbox -> EXPECTED-BUT-ABSENT. +# Proves the new ip-ranges endpoint is wired into the both-bounds check (the D-120 +# "tooling gap" this closes: those objects used to be invisible to the checker). +python3 - "$TMP" <<'PY' +import json, sys +tmp = sys.argv[1] +sb = json.load(open(f"{tmp}/s1.json")) +sb["ipam/ip-ranges"] = [r for r in sb["ipam/ip-ranges"] if r["range"] != "10.10.1.2/24-10.10.1.49/24"] +json.dump(sb, open(f"{tmp}/s11.json", "w")) +PY +python3 "$CHECK" --upstream "$TMP/u1.json" --sandbox "$TMP/s11.json" >"$TMP/out" 2>&1; rc=$? +[ "$rc" = 1 ] && grep -q "EXPECTED-BUT-ABSENT" "$TMP/out" && grep -q "10.10.1.2/24-10.10.1.49/24" "$TMP/out" \ + && ok "T11 a dropped D-120 ip-range -> EXPECTED-BUT-ABSENT, exit 1" \ + || { no "T11 a missing D-120 range should fail (rc=$rc)"; grep -i "range\|absent" "$TMP/out" | sed 's/^/ /'; } + +# T12: a STRAY ip-address not in EXPECTED_NEW -> UNEXPECTED EXTRA. Proves the +# ip-addresses endpoint is likewise covered (a stray write cannot hide). +python3 - "$TMP" <<'PY' +import json, sys +tmp = sys.argv[1] +sb = json.load(open(f"{tmp}/s1.json")) +sb["ipam/ip-addresses"].append({"address": "10.10.1.99/24"}) +json.dump(sb, open(f"{tmp}/s12.json", "w")) +PY +python3 "$CHECK" --upstream "$TMP/u1.json" --sandbox "$TMP/s12.json" >"$TMP/out" 2>&1; rc=$? +[ "$rc" = 1 ] && grep -q "UNEXPECTED EXTRA" "$TMP/out" && grep -q "10.10.1.99/24" "$TMP/out" \ + && ok "T12 a stray ip-address -> UNEXPECTED EXTRA, exit 1" \ + || { no "T12 a stray ip-address should fail (rc=$rc)"; grep -i "address\|extra" "$TMP/out" | sed 's/^/ /'; } + +# T8: read-only -- no write verbs anywhere in the checker +grep -qE '\.(create|update|delete|save)\(' "$CHECK" \ + && no "T8 the checker contains a WRITE verb -- it must be read-only" \ + || ok "T8 read-only: no create/update/delete/save in the checker" + +echo +if [ "$F" = 0 ]; then echo "sandbox-fidelity-check: $P/$P PASS"; exit 0; fi +echo "sandbox-fidelity-check: FAILURES: $F (passed $P)"; exit 1 diff --git a/tests/sandbox-seed/run-tests.sh b/tests/sandbox-seed/run-tests.sh new file mode 100755 index 0000000..2c1817b --- /dev/null +++ b/tests/sandbox-seed/run-tests.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# tests/sandbox-seed/run-tests.sh +# +# Harness for netbox/sandbox-seed.py and netbox/prod-draft-dump.py -- the +# two-phase upstream->sandbox seeding loop. +# +# OFFLINE. Touches no NetBox, real or sandbox. It exercises the arg contract and +# the SAFETY PROPERTIES, which are the whole reason these two scripts are split: +# +# 1. prod-draft-dump.py must have NO write path at all (no --commit, no POST). +# 2. sandbox-seed.py must be DRY BY DEFAULT and must REFUSE an upstream URL. +# 3. The upstream User-Agent trap must stay encoded (a 403 that looks like auth). +# +# Each of these cost real time on 2026-07-13. A regression that quietly drops one +# would not surface until something was written to the production IPAM apex. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +NBDIR="$(cd "$HERE/../../netbox" && pwd)" +SEED="$NBDIR/sandbox-seed.py" +DUMP="$NBDIR/prod-draft-dump.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +# 0. both parse +python3 -c "import ast;ast.parse(open('$SEED').read())" 2>/dev/null && ok || bad "sandbox-seed.py does not parse" +python3 -c "import ast;ast.parse(open('$DUMP').read())" 2>/dev/null && ok || bad "prod-draft-dump.py does not parse" + +# 1. arg contract +python3 "$SEED" --help >/dev/null 2>&1 && ok || bad "sandbox-seed --help -> 0" +python3 "$DUMP" --help >/dev/null 2>&1 && ok || bad "prod-draft-dump --help -> 0" +python3 "$SEED" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "sandbox-seed with no --draft must fail" +python3 "$DUMP" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "prod-draft-dump with no --out must fail" + +# 2. THE DUMP HAS NO WRITE PATH. It reads production. If it ever grows a POST, +# that is a write to the IPAM apex from a script whose whole contract is +# "read-only", and no reviewer would be looking for it. +grep -qE 'add_argument\("--commit"' "$DUMP" && bad "prod-draft-dump.py has a --commit flag -- it MUST be read-only" || ok +grep -qE '"(POST|PATCH|PUT|DELETE)"' "$DUMP" && bad "prod-draft-dump.py contains a WRITE method -- it MUST be read-only" || ok + +# 3. sandbox-seed is DRY BY DEFAULT (the D-117 lesson: a write-by-default IPAM +# tool is how an off-by-one lands in production unannounced). +grep -q '"--commit", action="store_true"' "$SEED" && ok || bad "sandbox-seed has no --commit flag" +grep -q 'DRY RUN -- nothing will be written' "$SEED" && ok || bad "sandbox-seed does not announce its dry run" + +# 4. sandbox-seed must REFUSE to point at upstream. "The sim never writes +# upstream" is enforced structurally, not by remembering a flag. +grep -q 'REFUSING: NETBOX_URL points at the UPSTREAM' "$SEED" && ok \ + || bad "sandbox-seed lost its upstream-URL refusal guard" +grep -q 'Guards go FIRST' "$SEED" && ok \ + || bad "the upstream guard is no longer documented as running FIRST (it was dead code once)" +tmp="$(mktemp)"; printf '{"_source":"https://netbox.baldurkeep.com"}' > "$tmp" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$SEED" --draft "$tmp" --commit 2>&1)" +rm -f "$tmp" +printf '%s' "$out" | grep -q "REFUSING" && ok \ + || bad "sandbox-seed did NOT refuse an upstream NETBOX_URL (it would have written to production)" + +# 5. the upstream User-Agent trap: a 403 that is NOT an auth failure. +grep -q 'User-Agent' "$DUMP" && ok || bad "prod-draft-dump lost the User-Agent header -- upstream 403s the default python UA" +grep -q 'User-Agent filter, NOT the token' "$DUMP" && ok \ + || bad "prod-draft-dump lost the 403 explanation -- the next person will blame the token" + +# 6. ids must never be carried across instances. +grep -q 'IDs are deliberately DISCARDED' "$DUMP" && ok || bad "prod-draft-dump lost the id-discard contract" + +# 7. region scopes. Mapping only dcim.site silently dropped the scope on 17 of the +# 90 upstream prefixes -- including the ENTIRE VR1 region-scoped draft. Caught +# only because a spot-check printed 'site None'. +grep -q 'dcim.region' "$DUMP" && ok || bad "prod-draft-dump no longer carries REGION-scoped prefixes (17 of 90 upstream)" +grep -q 'dcim.region' "$SEED" && ok || bad "sandbox-seed no longer resolves REGION scopes" + +# 8. the fidelity checker must exist and must be READ-ONLY. Counts and +# idempotency cannot prove a faithful seed -- the region-scope drop proved that +# (17 of 90 prefixes lost their scope; every count still matched). +FID="$NBDIR/sandbox-fidelity-check.py" +[ -f "$FID" ] && ok || bad "netbox/sandbox-fidelity-check.py is gone -- nothing proves the seed is faithful" +python3 -c "import ast;ast.parse(open('$FID').read())" 2>/dev/null && ok || bad "sandbox-fidelity-check.py does not parse" +grep -qE '"(POST|PATCH|PUT|DELETE)"' "$FID" && bad "sandbox-fidelity-check.py can WRITE -- it must be read-only" || ok +grep -q 'FIELD BY FIELD' "$FID" && ok || bad "the fidelity check no longer compares field-by-field -- counts alone hid the last bug" + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "sandbox-seed: $pass/$total PASS"; exit 0; fi +echo "sandbox-seed: $fail/$total FAIL"; exit 1 diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh new file mode 100644 index 0000000..4aa5c18 --- /dev/null +++ b/tests/site-headend-install/run-tests.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# tests/site-headend-install/run-tests.sh +# +# Harness for scripts/site-headend-install.sh. Exercises arg parsing, the SAFETY GUARDS, +# and --dry-run only. It NEVER installs a snap, never touches MAAS/LXD, and never mutates +# the machine (same posture as tests/prereqs and the opnsense-* harnesses). +# +# The guards are the point of this harness. The script's whole reason to exist is that +# four specific traps cost a live session on 2026-07-13; a regression that quietly drops +# one of them would not show up in any deploy until something broke in production. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../scripts" && pwd)/site-headend-install.sh" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } +t() { local d="$1" exp="$2"; shift 2; "$@" >/dev/null 2>&1; local rc=$?; if [ "$rc" = "$exp" ]; then ok; else bad "$d (rc=$rc want=$exp)"; fi; } + +# 1. arg contract +t "--help -> 0" 0 bash "$S" --help +t "unknown arg -> 2" 2 bash "$S" --bogus +t "missing --compose-cidr -> 2" 2 bash "$S" --dry-run +t "non-/24 cidr -> 2" 2 bash "$S" --dry-run --compose-cidr 10.10.1.0/16 +t "garbage cidr -> 2" 2 bash "$S" --dry-run --compose-cidr not-a-cidr +t "valid --dry-run -> 0" 0 bash "$S" --dry-run --compose-cidr 10.10.1.0/24 + +# 2. --compose-cidr must have NO default. A default here is how you end up serving MAAS +# DHCP on the site LAN, where the edge's Kea is authoritative (trap 4). +out="$(bash "$S" --dry-run 2>&1)" +printf '%s' "$out" | grep -q -- "--compose-cidr is REQUIRED" && ok || bad "missing --compose-cidr must fail LOUD and say why" + +# 3. --dry-run must MUTATE NOTHING: no snap install, no lxc, no maas invoked. +out="$(bash "$S" --dry-run --compose-cidr 10.10.1.0/24 2>&1)" +printf '%s' "$out" | grep -q '\[dry-run\]' && ok || bad "--dry-run printed no [dry-run] plan" +if snap list lxd >/dev/null 2>&1 || snap list maas >/dev/null 2>&1; then + bad "--dry-run appears to have INSTALLED a snap on the test host" +else ok; fi + +# 4. the derived addresses must track --compose-cidr, not be hardcoded. +out="$(bash "$S" --dry-run --compose-cidr 10.77.5.0/24 2>&1)" +printf '%s' "$out" | grep -q "10.77.5.1/24" && ok || bad "bridge IP not derived from --compose-cidr" +printf '%s' "$out" | grep -q "10.77.5.100" && ok || bad "range start not derived from --compose-cidr" +printf '%s' "$out" | grep -q "10.77.5.200" && ok || bad "range end not derived from --compose-cidr" + +# 5. THE FOUR TRAPS must remain encoded in the script. Each of these cost a live session; +# if someone "simplifies" one away, this harness goes red rather than the cloud. +grep -q 'raw.dnsmasq' "$S" && ok || bad "trap 1 LOST: raw.dnsmasq=port=0 is what stops dnsmasq binding :53 against MAAS's bind9" +grep -q 'ipv4.dhcp=false' "$S" && ok || bad "trap 1 LOST: lxdbr0 must have LXD's own DHCP off" + +# Fold backslash-continuations first, and strip comments + echo'd strings, so we test what +# the script EXECUTES rather than what it says. (Both of these bit this harness on its +# first run: a multi-line `lxc network create` reads as missing its redirect, and an echo +# that WARNS about `lxd init --auto` reads as calling it.) +CODE="$(sed -e 's/#.*$//' "$S" | sed -e ':a' -e '/\\$/{N; s/\\\n//; ta}' | grep -vE '^[[:space:]]*echo ')" + +printf '%s' "$CODE" | grep -qE '\blxd init --auto\b' \ + && bad "trap 1 LOST: 'lxd init --auto' is EXECUTED -- it FAILS when MAAS's bind9 holds :53" || ok + +n="$(printf '%s' "$CODE" | grep -cE '\blxc\b.*</dev/null')" +m="$(printf '%s' "$CODE" | grep -cE '\blxc\b')" +[ "$m" -ge 1 ] && [ "$n" = "$m" ] && ok || bad "trap 2 LOST: every lxc call must redirect </dev/null ($n of $m do) -- lxc reads stdin and EATS the piped script" +grep -q '5.21/stable' "$S" && ok || bad "trap 3 LOST: LXD must be pinned to the 5.21 LTS track (MAAS breaks on LXD >= 6.7)" +grep -q 'is ON for a subnet that is NOT the compose network' "$S" \ + && ok || bad "trap 4 LOST: the guard asserting MAAS DHCP is on NO other subnet is gone" +# trap 5: MAAS discovers the compose subnet from lxdbr0 with NO gateway. A deployed machine then +# gets a netplan with no default route -- it LOOKS healthy (DNS resolves; bind9 is link-local) but +# has no egress, and the first apt install on it hangs. Cost a redeploy on 2026-07-13. +grep -q 'subnet update .*gateway_ip' "$S" && ok || bad "trap 5 LOST: gateway_ip is never set on the compose subnet -- deployed machines will have NO default route" + +echo +total=$((pass+fail)) +if [ "$fail" -eq 0 ]; then echo "site-headend-install: $pass/$total PASS"; exit 0; fi +echo "site-headend-install: $fail/$total FAIL"; exit 1