diff --git a/docs/changelog-20260713-workflow-doc-status-correction.md b/docs/changelog-20260713-workflow-doc-status-correction.md new file mode 100644 index 0000000..c8be9e2 --- /dev/null +++ b/docs/changelog-20260713-workflow-doc-status-correction.md @@ -0,0 +1,43 @@ +# 2026-07-13 -- the workflow doc's status was materially FALSE + +`docs/dc-dc-deployment-workflow.md` is the first thing a fresh session reads. Its status claims +were written 2026-07-09 and never updated, so they had drifted into being **wrong in ways that +would misdirect real work**. Corrected against MEASURED state, not impressions. Docs only; no +code, no live system touched. + +## What was false, and what is true + +| claim in the doc | reality (measured 2026-07-13) | +|---|---| +| gap #2: "UNVALIDATED -- no `tofu` binary available" | **OpenTofu v1.12.3 IS installed.** The tree validates: root + **10/10 modules**. It has also been **APPLIED** -- 11 resources in state. | +| `opentofu/` "SCAFFOLDED, UNVALIDATED" | VALIDATED and partly APPLIED. But **2 modules were BROKEN and had never been parsed** (DOCFIX-194) -- see below. | +| gap #17: "no per-site ISP-uplink network exists anywhere, for ANY site" | **CLOSED FOR OFFICE1.** `office1-wan` is live (virbr11, NAT, `172.30.1.0/24`); the edge's WAN is on it at `172.30.1.2`, egress verified. The gap's own proposed answer (a dedicated per-site uplink network, NOT a mesh leg) is what was built. **DC1/DC2 still have none** -- `dc1-provider-public` has no IP and no forward. | +| `modules/opnsense-edge` "BUILT, UNVALIDATED (no real boot)" | **BUILT, INSTANTIATED, RUNNING.** Routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on udp/67. | +| `templates/opnsense-config.xml.tmpl` "BUILT, renderer fully tested" | **SUPERSEDED by D-113(a2)** -- config is REST-API-managed. A full config.xml push would now **CLOBBER** live DHCP. | +| `modules/netem-link` "BUILT, UNVALIDATED" | **WAS BROKEN** (destroy provisioner referencing `var.*` -- could not even `init`). Fixed 2026-07-13 (DOCFIX-194). | +| Stage 2 "RUNBOOK WRITTEN, NOT YET EXECUTED" | **PARTIALLY EXECUTED.** The edge + both Office1 networks are live. The **three headend VMs are NOT built.** | + +## The trap this correction removes + +Stage 2's gate row said the edge was "blocked on gap #17's WAN-side network." Both halves are +now false, and someone reading it would go build a network that already exists. + +Worse, the Stage 2 gate lists "NetBox authoritative" and "GitBucket serving" -- and NetBox and +GitBucket **do** answer, at `baldurkeep.com`. But those are **PRE-EXISTING services, not Office1 +headend VMs**. Measured: `office1-opnsense` is the ONLY VM on the vcloud host. It would be very +easy to tick that gate as met and move to Stage 3 with no headend at all. The corrected State +section says so explicitly. + +## What remains of Stage 2 (the real critical path) + +The three headend service VMs -- MAAS-region, NetBox, GitBucket -- do not exist. Per the runbook +they are blocked on a genuine decision, not on typing: Option A (`modules/cloudinit-vm`) needs +`user_data`/`meta_data`/`network_config` designed and an image source chosen; Option B is manual +`virt-install`, logged as debt. Both `cloudinit-vm` and `base-image` now at least VALIDATE +(DOCFIX-194) -- they never had before. + +## Revert + + git revert + +Restores the stale (false) status claims. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 8797930..9d84773 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -62,12 +62,27 @@ |---|---| | **Goal** | Stand up the operator's deployment headend first, before any DC. | | **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only); Office1's own OPNsense simulated-ISP edge (gap #16, DECIDED 2026-07-09: Stage 2 owns it). | -| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only; Office1-local network created (gap #12 CLOSED); Office1 OPNsense edge module wired (instantiation still blocked on real specs + gap #17's WAN-side network). | +| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only; Office1-local network created (gap #12 CLOSED); **Office1 OPNsense edge BUILT AND LIVE 2026-07-12/13 -- gate MET** (routing, NAT, egress, DHCP; gap #17 closed for Office1 via `office1-wan`). | | **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). | | **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. | | **Authoring status** | **Runbook WRITTEN 2026-07-09, UPDATED 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. The Office1-local-network gap it originally flagged (gap #12) is now CLOSED (`opentofu/modules/office1-network`, instantiated for real), and a new Step 4b now creates Office1's own OPNsense edge (gap #16, CLOSED -- ownership decided). NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. | -**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned). +**State (CORRECTED 2026-07-13): PARTIALLY EXECUTED.** + +- **DONE:** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is BUILT and + LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on udp/67). + Config is now REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt (11 resources + applied). Tailscale reaches the jumphost via a pre-existing subnet route. +- **NOT DONE -- this is what remains of Stage 2:** the **three headend service VMs** + (MAAS-region, NetBox, GitBucket) do not exist. Measured 2026-07-13: `office1-opnsense` is the + ONLY VM on the vcloud host. The NetBox/GitBucket that answer at `baldurkeep.com` are + PRE-EXISTING services, NOT Office1 headend VMs -- do not mistake them for a met gate. + Per the runbook, these are blocked on a real decision: Option A (`cloudinit-vm`) needs + `user_data`/`meta_data`/`network_config` designed and an image source chosen; Option B is + manual `virt-install`, logged as debt. `modules/cloudinit-vm` + `modules/base-image` now at + least VALIDATE (DOCFIX-194, 2026-07-13) -- they never had before. +- NetBox's "authoritative and populated" bullet remains PARTIAL (mechanism done via DOCFIX-152; + literals still unassigned). --- @@ -223,7 +238,36 @@ `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` when it's authored -- that call-site wiring is part of writing that runbook, not this delivery. -2. **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded +2. **OpenTofu** -- **STATUS CORRECTED 2026-07-13. The "UNVALIDATED / no tofu + binary" framing below is now FALSE and must not be relied on:** + - **A `tofu` binary EXISTS on the jumphost: OpenTofu v1.12.3.** The tree can + self-check. The whole "authored in a session with no binary" caveat that + rides along with every OpenTofu instruction in this repo is now closed as + a *capability* statement. + - **The tree VALIDATES: 10/10 modules**, root + every module standalone + (`bash scripts/opentofu-validate.sh`). + - **It has also been APPLIED for real.** `opentofu/terraform.tfstate` holds + 11 resources: the DC1 planes, the pools, the three mesh links, the + `office1-network`, and the Office1 OPNsense edge domain + volumes. This is + no longer a paper tree. + - **BUT two modules were BROKEN and had NEVER been parsed by any tool** + (DOCFIX-194, 2026-07-13): `node-vm` (`libvirt_volume` with a bogus + top-level `format` -- it nests under `target`) and `netem-link` (a + destroy-time provisioner referencing `var.*`, which OpenTofu rejects at + INIT -- the module could not even initialize). Root's `main.tf` does not + call either one, and `tofu validate` on the root module never parses an + uncalled module -- so the gate printed **PASS** over both. Both are fixed; + the gate now validates EVERY module standalone (S3). **`node-vm` builds + every DC node and `netem-link` is the Stage 6 failover-drill mechanism** -- + they would have detonated mid-deploy. See + `docs/changelog-20260713-docfix194-opentofu-module-validation.md`. + - **Still true:** `tofu plan`/`apply` has NOT been exercised for the Stage 3 + DC substrate. Validation is not application. + + *Original 2026-07-09 text follows (kept for the module inventory; its status + claims are superseded by the block above).* + + **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded 2026-07-09 (was: zero files).** `opentofu/` now has: `modules/dc-planes` (six per-DC planes), `modules/mesh-link` (the D-100 dark-fiber triangle legs), `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot @@ -523,7 +567,28 @@ #12's new network; its WAN side does not -- see NEW gap #17 immediately below, a genuinely deeper, cross-site gap surfaced while closing this one, not invented as a substitute for a real answer. -17. **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in +17. **PARTIALLY CLOSED 2026-07-13 -- the DECISION is made and PROVEN for + Office1; DC1/DC2 still have no uplink network.** + - **Office1: CLOSED.** The answer was the one this gap proposed -- a + dedicated per-site ISP-uplink `libvirt_network`, **not** a mesh leg. + `office1-wan` exists and is live (measured 2026-07-13: `virbr11`, NAT + forward, `172.30.1.0/24`). The Office1 edge's WAN sits on it at + `172.30.1.2` with default route `172.30.1.1`, and egress to the internet + is verified (0.0% loss). So the topology question is ANSWERED and the + pattern is PROVEN in production, not merely designed. + - **DC1/DC2: STILL OPEN.** Neither has an ISP-uplink network. Measured: + `dc1-provider-public` has **no IP and no forward** (isolated), and there + is no `dc1-wan`/`dc2-wan` at all. Stage 3's `dc1_opnsense` still carries + the unresolved `wan_network_name` placeholder described below. + - **The fix is now mechanical, not a decision:** replicate the + `office1-wan` shape per DC. The D-100 reasoning below (mesh legs are + MANAGEMENT-TRAFFIC-ONLY and must NOT carry an edge's WAN side) stands + unchanged and is what the Office1 build honoured. + + *Original 2026-07-09 text follows (the reasoning is still correct; the "for + ANY site / nowhere" status claim is superseded by the block above).* + + **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in this repo, for ANY site** (found 2026-07-09 while closing gap #16 for Office1). Every `modules/opnsense-edge` caller needs a `wan_network_name` distinct from BOTH its LAN-side network (a @@ -599,14 +664,14 @@ | Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED | | DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 | | Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. NONE executed against real infrastructure (prep-only session). See gaps #12-15 above for real design gaps the authoring pass surfaced. | -| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` | -| OPNsense image+config mechanism (`modules/opnsense-edge`) | BUILT 2026-07-09, UNVALIDATED (no `tofu`/ISO-tool/real boot to confirm), see `opentofu/README.md` | -| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | BUILT 2026-07-09, renderer fully tested (8/8) -- several tokens still pending Stage 0 ratification or a real boot, see `opentofu/templates/README.md` | +| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (OpenTofu v1.12.3 now on the jumphost): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds 11 resources (DC1 planes, pools, 3 mesh links, office1-network, the OPNsense edge). **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. | +| OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + RUNNING 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. | +| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **SUPERSEDED 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **OPEN:** reduce the template to a MINIMAL bootstrap (sshd + root key + console + a pre-seeded API key) and retire the ISO path + the module's `config_seed`/cdrom wiring. | | MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` | -| `tc netem` mechanism (`modules/netem-link`) | BUILT 2026-07-09, UNVALIDATED -- real latency/jitter/loss/rate parameters still an unruled D-100 item, see `opentofu/README.md` | +| `tc netem` mechanism (`modules/netem-link`) | **WAS BROKEN, FIXED 2026-07-13 (DOCFIX-194)** -- its destroy provisioner referenced `var.*`, which OpenTofu rejects at INIT, so the module could not even initialize. Now validates. Real latency/jitter/loss/rate parameters are STILL an unruled D-100 item. Never applied. | | `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` | | Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` | -| Office1 OPNsense edge ownership | DECIDED 2026-07-09 (DOCFIX-163, gap #16 CLOSED) -- Stage 2 owns it; `module "office1_opnsense"` skeleton in `main.tf` (commented, same as DC1/DC2's own not-yet-instantiated edges); WAN-side network still blocked on NEW gap #17 (no per-site ISP-uplink network exists anywhere yet, DC1/DC2 included) | +| Office1 OPNsense edge ownership | **BUILT AND LIVE 2026-07-12/13.** `module "office1_opnsense"` is instantiated (not commented). **Gap #17 is CLOSED FOR OFFICE1**: `office1-wan` (virbr11, NAT, 172.30.1.0/24) is the dedicated per-site ISP uplink, and the edge's WAN sits on it -- the D-100-honouring answer (NOT a mesh leg). **DC1/DC2 still have NO uplink network** -- replicating the `office1-wan` shape per DC is now mechanical, not a decision. | | Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` | | `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 4 and Stage 5's runbooks (both written 2026-07-09) mandate calling it, not yet executed for real | | NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned | diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 92cdd18..a168b5f 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -2011,6 +2011,22 @@ scp/install/reboot path against this edge. Template reduction to a minimal bootstrap (sshd + key + console) is the next step and is NOT yet done. +## WORKFLOW-DOC STATUS CORRECTED 2026-07-13 (docs only) + +- `docs/dc-dc-deployment-workflow.md` (the doc a fresh session reads FIRST) had 2026-07-09 status + claims that were now materially FALSE. Corrected against measured state: gap #2 (tofu binary + EXISTS, v1.12.3; tree validates 10/10 and is partly APPLIED), gap #17 (**CLOSED for Office1** -- + `office1-wan` virbr11 NAT 172.30.1.0/24 carries the live edge WAN; **DC1/DC2 still have none**), + the opnsense-edge / config.xml / netem-link status rows, and Stage 2's gate + state. +- **THE TRAP REMOVED:** Stage 2's gate lists "NetBox authoritative" + "GitBucket serving" -- and + both DO answer at baldurkeep.com. **But those are PRE-EXISTING services, NOT Office1 headend + VMs.** Measured: `office1-opnsense` is the ONLY VM on the vcloud host. It would be easy to tick + that gate as met and walk into Stage 3 with no headend. The corrected State section says so. +- **What actually remains of Stage 2:** the three headend VMs (MAAS-region, NetBox, GitBucket) + DO NOT EXIST. Blocked on a real decision (cloudinit-vm user_data/meta_data/network_config + + image source, vs manual virt-install as debt) -- not on typing. +- `docs/changelog-20260713-workflow-doc-status-correction.md`. + ## DOCFIX-194 2026-07-13: the OpenTofu gate was GREEN over two BROKEN modules - **`scripts/opentofu-validate.sh` printed PASS while `node-vm` and `netem-link` were flatly