diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index a7e2546..b955575 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -131,6 +131,20 @@ to D-121 Option C. `opentofu/vr1-dc0-maas/` is retained but UNUSED (the pod route is refuted); retire-or-keep is a stage-close question, as is the D-103/D-123 amendment text. Session changelog items 15-17. + **INCIDENT 2026-07-21 (pod-delete cascade): the 9 DC0 MAAS machine + records are GONE.** During the operator-ruled retire of + `opentofu/vr1-dc0-maas`, deleting the stale pod object (id=4 + `vr1-dc0-inner`) cascaded to the nine machine records the failed + 2026-07-20 pod refresh had silently linked to it -- the association + check was run AFTER the delete (agent process error, owned; capture + `docs/audit/incident-20260721-pod-delete-cascade.txt`). Substrate is + MEASURED INTACT (all 10 domains inside vvr1-dc0; edge running, 9 + nodes shut off; MAC pins config-carried; rack DNS/legs/dhcpd + untouched). LOST: enlistment records, Ready state, per-machine power + wiring. Recovery = the proven 07-21 flow (power-on -> PXE re-enlist + -> maas-node-power.sh -> re-commission), operator-gated, PENDING. + Every "9 nodes Ready" claim below this point describes the + PRE-INCIDENT state and is quoted history until recovery re-verifies. **COMMISSIONING RESOLVED 2026-07-21: all 9 nodes Ready** (logged window ops-commissioning-diag; adjudication `docs/audit/commissioning-diag-20260721.txt`; session changelog diff --git a/docs/audit/incident-20260721-pod-delete-cascade.txt b/docs/audit/incident-20260721-pod-delete-cascade.txt new file mode 100644 index 0000000..5260e66 --- /dev/null +++ b/docs/audit/incident-20260721-pod-delete-cascade.txt @@ -0,0 +1,44 @@ +# INCIDENT 2026-07-21: stale-pod deletion cascaded to the 9 DC0 machine records +# +# Context: operator-ruled "Retire fully" for opentofu/vr1-dc0-maas (stage +# close). Cleanup step 2 was deleting the stale pod object the failed +# 2026-07-20 pod apply left on the region. +# +# WHAT WAS RUN (gated, in sequence): +$ ssh voffice1 'maas admin vm-host delete 4' # stale pod vr1-dc0-inner (virsh) +DELETE-EXIT=0 +$ maas admin vm-hosts read # read-back + id=1 name=office1-lxd type=lxd # pod gone, as intended +$ maas admin machines read # cascade check -- RUN TOO LATE (see ERROR) +2 machines +Counter({'Deployed': 2}) # office1-netbox, office1-tailscale ONLY +# => the 9 DC0 machine records (previously ALL READY, shapes exact to D-121 +# Option C, per-machine virsh power) were DELETED with the pod. +# +# ERROR (agent process failure, owned): the association check was performed +# AFTER the delete. The correct order -- read the pod's machine list BEFORE +# deleting -- was skipped on the reasoning "the pod registration failed, so +# no machines can be associated". That reasoning was WRONG: the 2026-07-20 +# pod apply created the pod object and its refresh got far enough (it died +# on domblkinfo DURING enumeration) to link the nine existing machines to +# pod id=4. MAAS deletes pod-linked machine RECORDS on vm-host delete even +# without decompose. +# +# BLAST RADIUS (measured): +# - Substrate INTACT: inside vvr1-dc0, virsh -c qemu:///system list --all +# shows all 10 domains (vr1-dc0-opnsense RUNNING id 16; the 9 node VMs +# shut off) -- no decompose reached libvirt (the pod's virsh path was +# broken; no decompose was requested). +# - MAC pins INTACT (config-pinned in the inner root since 2026-07-21). +# - Rack services INTACT: dc0-node-dns / rack legs / dhcpd untouched. +# - LOST: the 9 MAAS machine records -- enlistment identity, commissioning +# results (Ready state), per-machine power_type=virsh wiring. +# +# RECOVERY PATH (the 2026-07-21 proven flow, all steps operator-gated): +# 1. Power on the 9 node VMs on the rack (virsh start) -> they PXE-boot +# to the region across the transit and ENLIST as New (pinned MACs). +# 2. scripts/maas-node-power.sh (MAC-matched, dry-run first) re-wires +# per-machine virsh power; verify by query-power-state. +# 3. Commission the fleet -> Ready expected in minutes (07-21 proof: +# canary ~3 min, fleet of 8 concurrent ~4 min, forwarder path live). +# 4. Verify shapes against D-121 Option C, capture, correct CURRENT-STATE. diff --git a/docs/changelog-20260721-netem-install-verify.md b/docs/changelog-20260721-netem-install-verify.md index 897f828..e5320c3 100644 --- a/docs/changelog-20260721-netem-install-verify.md +++ b/docs/changelog-20260721-netem-install-verify.md @@ -180,3 +180,44 @@ id=4 `vr1-dc0-inner` type=virsh (beside the legitimate office1-lxd id=1). Its deletion is a gated mutation awaiting the retire-or-keep ruling cluster (with SEC-013's on-disk key + tfstate disposition). +- Manifest gap noted (finding, not fixed): neither creds manifest + tracks `maas-api-key.txt` or `maas-virsh_ed25519` in + `~/vr1-dc0-creds/` on voffice1 (SEC-012/013 rows are the only + tracking). Candidate manifest addition at SEC-012's disposition. + +## 13. RULING: vr1-dc0-maas RETIRED (stage-close ruling 1) + repo removal + +- RULING (GA-R5). Question as presented: "Stage-close ruling 1 of 3: + what happens to `opentofu/vr1-dc0-maas/` (the refuted pod root)?" + with measured context (dead mechanism, stale pod id=4, SEC-013 + surfaces). Operator selection, exact: "Retire fully (Recommended)". +- Executed (repo part): `git rm -r opentofu/vr1-dc0-maas/`; + opentofu-validate PASS with the root gone (validator iterates roots + dynamically -- no reference edits needed). +- **Revert:** git revert restores the root (history preserves it + regardless; D-103/D-123 amendments remain the mechanism authority). + +## 14. INCIDENT: stale-pod delete cascaded to the 9 DC0 machine records + +- Cleanup step 2 (`maas admin vm-host delete 4`, gated) ALSO deleted + the nine DC0 machine records: the failed 2026-07-20 pod refresh had + linked the existing machines to the pod (it died on domblkinfo + DURING enumeration -- late enough to associate), and MAAS deletes + pod-linked machine records on vm-host delete even without decompose. +- AGENT ERROR, owned: the pod's machine-association list should have + been read BEFORE the delete; it was checked after, on the wrong + reasoning that a failed registration implies no associations. Lesson + for appendix-A (queued): before ANY `vm-host delete`, read the pod's + machines list and expect ZERO; a non-empty list is a STOP. +- Blast radius measured (capture + docs/audit/incident-20260721-pod-delete-cascade.txt): substrate + fully intact (10/10 domains inside vvr1-dc0, edge running, nodes + shut off; MAC pins config-carried; rack DNS/legs/dhcpd untouched). + Lost: MAAS enlistment records, Ready state, per-machine power + wiring. CURRENT-STATE corrected in the same commit (GA-R1 C2). +- Recovery (operator-gated, PENDING): power-on -> PXE re-enlist -> + maas-node-power.sh re-wire -> re-commission (the proven 07-21 fleet + flow, ~minutes). SEC-013's remaining cleanup steps (tfstate + key + file on voffice1) HELD until recovery is decided. +- **Revert:** n/a (records); the MAAS-side loss is repaired by the + recovery flow, not a git operation. diff --git a/opentofu/vr1-dc0-maas/.terraform.lock.hcl b/opentofu/vr1-dc0-maas/.terraform.lock.hcl deleted file mode 100644 index 9e129c7..0000000 --- a/opentofu/vr1-dc0-maas/.terraform.lock.hcl +++ /dev/null @@ -1,38 +0,0 @@ -# This file is maintained automatically by "tofu init". -# Manual edits may be lost in future updates. - -provider "registry.opentofu.org/canonical/maas" { - version = "2.7.2" - constraints = "2.7.2" - hashes = [ - "h1:0/+f28bV5/BqmlBZdmQiTWmzVnjn8/3VNwU+7RLHi1E=", - "h1:0NQfNgN9iJEcuT3WK2+vC9Ss/C1cnyepVP6uahpvSSo=", - "h1:3LC/Yd6KHwKKUkeAOi0g6lRO3Aj6+vMwc5xGf6ARRcY=", - "h1:Be77+i/FQe/egck+r/Bi/qtUVTtB1+BXnwVS7rhQwGI=", - "h1:Cd7Ed7paJzuEhCpkcauEl18JkfRnvX7zK2KnbZfOeQ0=", - "h1:GcMMY3WdhRt6xJ6uXSoKRxs0tpmt43GL4a/a/gBlJV8=", - "h1:IdwSAz83vIz0gQvRt9VT+rs9F0+1MgLIi2IjU3U/8gA=", - "h1:JWEzos28aI09FfC9nW8Odb9h/J2KWnBwoOAsTz0OZ1g=", - "h1:JY1cOBhZyAYVKEWkM2TU/z+CgXI0GJ53HIdzIgmORRM=", - "h1:M4bq+7JZB9hf2hPzr1x85dAhbWcmMBDCoqGvYcSc/hk=", - "h1:PTbVBZc9uhi9zztYpAiWIRHc7zNr4Uwr5wR/atlbuI0=", - "h1:WNzlm/e2Rn6mVhLEgG5yBvMJEpcUCvSQxExvY77pAEo=", - "h1:jc8UQ6JbaFzi+zxpf8AzfZXs94sMIWYwu34adBDWvBk=", - "h1:u+mlHkUKpqQFmDB1zRtBFC7Nl+gMDvFw0edd71q5FH0=", - "zh:07de224747a55736794b771cfe42bcd7a84a0656120df73f3bba8fa5a7869fed", - "zh:0aa77c14a1e4826cc1b345b6dc61086a2d9c20a77b151b7677718544e275636e", - "zh:2a4a659601e97d10ff8d07871fa74dd895946f5e683fb7fb4a6050b78beb3012", - "zh:4627c138c5b36d36612bea1d6d5ed895b5edc082f15bdc2af3e3f6c1153d0568", - "zh:4a948b51b701b1e60794e7cee3f7f5183838fd2e1e53898ddd824a3d31e7b267", - "zh:54d987940d7e6c591a1d4b837adf01b6cace17698227649fa72da8113d8c3531", - "zh:5b111497a82b898478fbea60b244d7e932039bf95460df9d984d0f05e2567854", - "zh:68ed5d31ac46fb0e79b69e981021f83166600d222d7b2adecbea9393f8ff93b9", - "zh:7fd8f29946937cd2c2716603081fb5c71f524465cc4b0bfeeeacab13eb927515", - "zh:824e97b9b68192bf04cc56e94e7d279468d1c4d48c2a98d9f71f86b804053500", - "zh:ce5669c6279fa6cd72ab8e39ccffeb90be3f0d30d7b4077f154b8cb1fad710c0", - "zh:d508aa6858a9212e6b71924ec9e0ddeaada4bab74e4e040ce036ef4189e5076a", - "zh:db819ed07eaafbab90f8292b0a6bc4551c3bb7dcbd1006af67e68719f36720b0", - "zh:f42d9b7c7b0a8adcd1135f295b40425d5a47191d14468a5725f665d027aa9647", - "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", - ] -} diff --git a/opentofu/vr1-dc0-maas/main.tf b/opentofu/vr1-dc0-maas/main.tf deleted file mode 100644 index 70b00f9..0000000 --- a/opentofu/vr1-dc0-maas/main.tf +++ /dev/null @@ -1,56 +0,0 @@ -# ===================================================================== -# vr1-dc0 MAAS registration root (Model B step D / runbook Step 9). -# -# WHY A THIRD ROOT, rather than adding `provider "maas"` to an existing one: -# DOCFIX-179 removed the maas provider block from the OUTER root after measuring -# that its mere PRESENCE forces every `tofu plan` to demand maas_api_url AND the -# SENSITIVE maas_api_key -- even for a plan that creates zero MAAS resources, -# which would bake a fake key into plan/state (the DOCFIX-175 plaintext-secret -# surface). That lesson applies verbatim to the INNER substrate root: coupling -# routine substrate plans to MAAS credential availability would be repeating the -# defect the repo already paid to learn. So MAAS registration gets its own small -# root with its own state, and the substrate roots stay credential-free. -# -# WHAT IT DOES: registers the DC's inner libvirt as a MAAS VM host (pod) so MAAS -# DISCOVERS the already-existing, OpenTofu-created node VMs. It deliberately does -# NOT compose machines -- see modules/maas-vm-host's header for the -# maas_vm_host vs maas_vm_host_machine reasoning (D-103). -# -# WHERE IT RUNS: from the Office1 headend (voffice1), the D-128 Plane-2 host -- -# same place the inner substrate root runs. -# -# SECRET HANDLING: maas_api_key comes ONLY from TF_VAR_maas_api_key in the -# environment, read from the operator-placed 0600 file. It is never written to a -# tfvars file. NOTE: it DOES land in this root's terraform.tfstate (unavoidable -# with this provider), so that state file inherits the same handling as any -# credential-bearing artifact -- 0600, on voffice1, never committed. -# ===================================================================== - -provider "maas" { - api_url = var.maas_api_url - api_key = var.maas_api_key -} - -module "vr1_dc0_vm_host" { - source = "../modules/maas-vm-host" - vm_host_name = var.vm_host_name - power_address = var.power_address - zone = var.zone - pool = var.pool -} - -output "vm_host_id" { - description = "MAAS id of the registered VM host -- the handle for a later teardown." - value = module.vr1_dc0_vm_host.id -} - -# MAAS's OWN view of the host's capacity. Not inputs -- a post-registration -# cross-check against the D-121/D-123 sizing math (vvr1-dc0 = 108 vCPU / 416 GiB): -# if MAAS reports something else, the nesting or the sizing is not what we think. -output "maas_reported_cores" { - value = module.vr1_dc0_vm_host.resources_cores_total -} - -output "maas_reported_memory" { - value = module.vr1_dc0_vm_host.resources_memory_total -} diff --git a/opentofu/vr1-dc0-maas/variables.tf b/opentofu/vr1-dc0-maas/variables.tf deleted file mode 100644 index d863bc3..0000000 --- a/opentofu/vr1-dc0-maas/variables.tf +++ /dev/null @@ -1,43 +0,0 @@ -variable "maas_api_url" { - description = "Office1 MAAS region API endpoint, e.g. http://10.10.0.20:5240/MAAS. MEASURED (the rack enrolled against exactly this URL), never inferred." - type = string -} - -variable "maas_api_key" { - description = <<-EOT - MAAS API key (consumer:token:secret). SECRET -- supply ONLY via the - environment as TF_VAR_maas_api_key, sourced from the operator-placed 0600 - file (SEC-009 per-site creds convention). NEVER put it in a tfvars file, - never echo it, and never let it reach a plan file that gets committed. - EOT - type = string - sensitive = true -} - -variable "vm_host_name" { - description = "MAAS VM host (pod) name for this DC's inner libvirt." - type = string -} - -variable "power_address" { - description = <<-EOT - libvirt URI MAAS uses to reach this VM host. Under D-123 Model B this is - LOCAL to the rack (`qemu:///system`): vvr1-dc0 is BOTH the MAAS rack - controller AND the libvirt host holding the node VMs, so MAAS dials no - cross-fiber virsh. MEASURED prerequisite: the MAAS rack process must be - able to open that socket on the rack. - EOT - type = string -} - -variable "zone" { - description = "MAAS zone for the VM host. null = let MAAS compute it (the module's documented, provider-supported no-op)." - type = string - default = null -} - -variable "pool" { - description = "MAAS resource pool for the VM host. null = let MAAS compute it." - type = string - default = null -} diff --git a/opentofu/vr1-dc0-maas/versions.tf b/opentofu/vr1-dc0-maas/versions.tf deleted file mode 100644 index 0818fd1..0000000 --- a/opentofu/vr1-dc0-maas/versions.tf +++ /dev/null @@ -1,18 +0,0 @@ -# Pins mirror the outer root's versions.tf EXACTLY (canonical/maas 2.7.2) -- -# this root is a third entry point, not a different provider generation. If the -# outer root's pin is bumped, bump it here in the SAME change: two copies of one -# fact drift silently otherwise (the same caveat opentofu/variables.tf records -# about vr1_dc0_planes vs scripts/lib-net.sh). -# -# NO libvirt provider here ON PURPOSE. This root touches MAAS and nothing else. - -terraform { - required_version = ">= 1.6.0" - - required_providers { - maas = { - source = "canonical/maas" - version = "2.7.2" - } - } -}