diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b955575..3ac5862 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -131,20 +131,26 @@ to D-121 Option C. `opentofu/vr1-dc0-maas/` is retained but UNUSED (the pod route is refuted); retire-or-keep is a stage-close question, as is the D-103/D-123 amendment text. Session changelog items 15-17. - **INCIDENT 2026-07-21 (pod-delete cascade): the 9 DC0 MAAS machine - records are GONE.** During the operator-ruled retire of + **INCIDENT 2026-07-21 (pod-delete cascade): RESOLVED SAME-DAY -- all + 9 nodes READY again.** During the operator-ruled retire of `opentofu/vr1-dc0-maas`, deleting the stale pod object (id=4 `vr1-dc0-inner`) cascaded to the nine machine records the failed 2026-07-20 pod refresh had silently linked to it -- the association check was run AFTER the delete (agent process error, owned; capture - `docs/audit/incident-20260721-pod-delete-cascade.txt`). Substrate is - MEASURED INTACT (all 10 domains inside vvr1-dc0; edge running, 9 - nodes shut off; MAC pins config-carried; rack DNS/legs/dhcpd - untouched). LOST: enlistment records, Ready state, per-machine power - wiring. Recovery = the proven 07-21 flow (power-on -> PXE re-enlist - -> maas-node-power.sh -> re-commission), operator-gated, PENDING. - Every "9 nodes Ready" claim below this point describes the - PRE-INCIDENT state and is quoted history until recovery re-verifies. + `docs/audit/incident-20260721-pod-delete-cascade.txt`). Substrate + was measured intact throughout (10/10 domains, MAC pins + config-carried, rack services untouched); only MAAS records were + lost. Operator-ruled recovery executed immediately: virsh power-on + -> PXE re-enlist (9/9 in ~2 min, pinned MACs) -> maas-node-power.sh + dry+commit (9/9, power verified) -> re-commission -> **ALL 9 READY + in ~3 min, shapes exact to D-121 Option C, power=virsh** + (`docs/audit/incident-20260721-recovery-verify.txt`; note the MAAS + hostnames are NEW random names -- any doc quoting the old ones is + history). The retire-fully ruling is now FULLY EXECUTED: repo root + removed, stale pod gone, voffice1 tfstate remnants + the SEC-013 + on-disk key file deleted (absence verified; SEC-013 row narrowed to + CLI-profile-only). Lesson shipped to appendix-A: read a pod's + machine list BEFORE `vm-host delete`; non-empty = STOP. **COMMISSIONING RESOLVED 2026-07-21: all 9 nodes Ready** (logged window ops-commissioning-diag; adjudication `docs/audit/commissioning-diag-20260721.txt`; session changelog diff --git a/docs/audit/incident-20260721-recovery-verify.txt b/docs/audit/incident-20260721-recovery-verify.txt new file mode 100644 index 0000000..1770991 --- /dev/null +++ b/docs/audit/incident-20260721-recovery-verify.txt @@ -0,0 +1,13 @@ +# Pod-delete-cascade RECOVERY verification -- 2026-07-21 +# Flow: virsh start 9 -> PXE re-enlist (~2 min) -> maas-node-power.sh dry+commit (9/9 ok, power verified off) -> commission -> ALL 9 READY (~3 min). +# Expected shapes (D-121 Option C): 3x 16cpu/64GiB control + 2x 12cpu/48GiB compute + 4x 8cpu/24GiB storage. + +ace-robin Ready cpu=16 mem_MiB=65536 power=virsh +alert-cub Ready cpu=8 mem_MiB=24576 power=virsh +first-oryx Ready cpu=8 mem_MiB=24576 power=virsh +keen-dove Ready cpu=12 mem_MiB=49152 power=virsh +moral-salmon Ready cpu=8 mem_MiB=24576 power=virsh +real-filly Ready cpu=16 mem_MiB=65536 power=virsh +superb-piglet Ready cpu=12 mem_MiB=49152 power=virsh +wired-thrush Ready cpu=16 mem_MiB=65536 power=virsh +wise-stud Ready cpu=8 mem_MiB=24576 power=virsh diff --git a/docs/changelog-20260721-netem-install-verify.md b/docs/changelog-20260721-netem-install-verify.md index e5320c3..0b9f15f 100644 --- a/docs/changelog-20260721-netem-install-verify.md +++ b/docs/changelog-20260721-netem-install-verify.md @@ -221,3 +221,28 @@ file on voffice1) HELD until recovery is decided. - **Revert:** n/a (records); the MAAS-side loss is repaired by the recovery flow, not a git operation. + +## 15. RECOVERY EXECUTED (operator-ruled) -- all 9 nodes Ready again; retire-fully COMPLETE + +- RULING (GA-R5). Question: recover now via the proven 07-21 flow? + Operator selection, exact: "Recover now (Recommended)". +- Executed, all gated: virsh start x9 on the rack -> PXE re-enlist + (9/9 in ~2 min; pinned MACs = stable identity) -> voffice1 clone + fast-forwarded -> maas-node-power.sh DRY (9/9 MAC-matched) then + --commit (9/9 ok, each verified by query-power-state) -> commission + x9 -> ALL 9 READY in ~3 min. Shapes verified exact to D-121 Option C + (3x16/64GiB, 2x12/48GiB, 4x8/24GiB), power=virsh + (docs/audit/incident-20260721-recovery-verify.txt). NOTE: new random + MAAS hostnames (ace-robin, alert-cub, ... replace the pre-incident + set); nothing durable quoted the old names as identity (the power + script is MAC-matched by design). +- Held cleanup then completed (the rest of ruling 1): voffice1's + vr1-dc0-maas dir incl. terraform.tfstate REMOVED; SEC-013 key file + `~/vr1-dc0-creds/maas-api-key.txt` REMOVED; absence verified by + listing (creds dir now: maas-virsh_ed25519 [SEC-012, in use] + the + dc0 service keypair). SEC-013 row narrowed to CLI-profile-only; + CURRENT-STATE incident paragraph resolved; NEW appendix-A entry + (pod-delete cascade: check the pod's machine list BEFORE delete). +- **Revert:** n/a for the live repairs; docs via git revert. The + deleted key file is not restorable from the repo (by design) -- the + operator re-mints from MAAS if a maas-provider root ever returns. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index cd96bfd..8e93cdd 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -20,7 +20,7 @@ | SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/archive/changelogs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | | SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. **MODEL B UPDATE (2026-07-16):** after the D-123 Model B reshape, the OUTER `vvr1-dc0` is single-leg (transit only); metal-admin + the other 5 planes are now INNER bridges on `vvr1-dc0`'s own libvirtd. The forwarding hardening is MORE critical (vvr1-dc0 now bridges ALL 6 inner planes + the transit) and belongs on the inner libvirt host, not just a 2-leg rack -- wire it in the C3 bootstrap (`site-headend-install.sh` node-host mode). | 2026-07-16 plane-segregation review + Model B reshape cross-check; `opentofu/main.tf` `module "vvr1_dc0"` | operator | **CLOSED 2026-07-20** (was: artifact COMMITTED Phase D 2026-07-16, gated on apply+verify -- both now done, see close note at end of this cell). Enforced via a FORWARD-drop across the transit leg, NOT a global `ip_forward=0`. `scripts/site-headend-install.sh --host-nodes` writes `/etc/nftables-sec010.nft` (drop FORWARD in+out the transit interface, default `mgmt`) + a boot-persistent `sec010-fw.service`; `--host-nodes --check` is the MECHANICAL pre-apply gate (fails if the rule is absent, AND -- hardened 2026-07-16 -- if the keyed transit interface does not EXIST, since an nftables oifname on an absent iface loads clean but matches nothing = fail-open). **D-125 UPDATE (2026-07-16):** the earlier rationale (a global `ip_forward=0` being unusable because the inner `vr1-dc0-wan` NAT forced `ip_forward=1`) NO LONGER applies -- under bridge-in the inner `vr1-dc0-wan` is a BRIDGE (no inner NAT). The scoped FORWARD-drop is RETAINED as-is (correct either way). **br_netfilter CONSTRAINT:** it MUST stay interface-scoped and NEVER be globalized -- with `br_netfilter` loaded, bridged WAN frames traverse the L3 FORWARD chain, so a global forward-drop would silently kill the `br-vr1-dc0-wan` WAN path. `--check` now also verifies the WAN bridge exists with its uplink enslaved (same fail-open class). Nothing routes across the fiber THROUGH vvr1-dc0; the rack proxies MAAS at the app layer (originated/terminated, not forwarded). Same pin follows onto `voffice1` when its transit leg is wired. Region route must target only the rack transit /30, never 10.12.8.0/22. **CLOSED 2026-07-20 (operator ruling, exact utterance: "Close SEC-010 (Recommended)"): applied + verified BOTH ends in the logged dc0-deploy step-B window.** vvr1-dc0: written by `site-headend-install.sh --role rack --host-nodes`, verified by the named `--check` EXIT 0 (`docs/audit/stepB-check-20260720-final.txt` -- rule present + `enp1s0` exists + `br-vr1-dc0-wan`/`enp2s0` enslaved; note the transit interface is `enp1s0`, not the script-default `mgmt` -- netplan set-name dropped, session changelog). voffice1 (the "same pin follows" clause -- its transit leg `enp2s0` was wired 2026-07-20): identical scoped artifact + `sec010-fw.service` enabled, `nft list table inet sec010` verified, proxy/region paths re-verified working post-pin. | | SEC-012 | 2026-07-20 | **New SERVICE credential: a dedicated MAAS -> libvirt SSH keypair.** Minted during the DC0 deploy so MAAS can power-control the inner node VMs (`power_type=virsh`). It is a SERVICE credential held by a daemon, not an operator key: the private half is installed inside the MAAS snap on BOTH controllers (region + DC rack) because MAAS dials the power address from whichever controller it chooses -- MEASURED to be the REGION, not the rack that owns the hardware. The public half authorizes a user that is in the rack's `libvirt` group, so anyone holding this key can drive libvirt on `vvr1-dc0` -- i.e. power/define/destroy every inner node VM and the DC edge. Blast radius is deliberately narrowed by being per-purpose (operator-ruled 2026-07-20: "Dedicated MAAS->libvirt key" over reusing the dc0 service key). | 2026-07-20 step-D power wiring; `scripts/maas-node-power.sh` header; session changelog items 15-17 | operator | **OPEN -- rotation obligation + a scope question.** (1) ROTATE at v1 close, or immediately if either controller is rebuilt or shared. (2) SCOPE: the key authorizes a `libvirt`-group user, which is broader than the power verbs MAAS actually needs; a tighter grant (dedicated unix user + polkit rule limiting to domain start/stop/status) is the hardening candidate and is Roosevelt-relevant, since bare metal replaces this with IPMI credentials that have the same "power-only vs full-control" question. Custody detail off-repo per D-069. | -| SEC-013 | 2026-07-20 | **MAAS API key materialized to disk on the region.** An admin-scoped MAAS API key (`consumer:token:secret`) was placed by the operator in a 0600 file on `voffice1` so the `opentofu/vr1-dc0-maas` root could consume it via `TF_VAR_maas_api_key`. It grants FULL MAAS admin API access (machines, power, deploy, users). Two exposure surfaces beyond the file itself: (a) the OpenTofu **state file** of any root that uses the maas provider records it -- unavoidable with this provider, so that state inherits credential handling (0600, never committed); (b) a MAAS CLI profile was also created for the operator user from the same key. It was verified by FORMAT ONLY (71 bytes, 3 colon-separated parts) and never printed, echoed, or passed in argv. | 2026-07-20 step-D part 2; `opentofu/vr1-dc0-maas/main.tf` header; session changelog item 15 | operator | **OPEN -- rotation obligation.** Rotate at v1 close, or immediately if `voffice1` is rebuilt/shared or the tfstate leaves the host. NOTE: the pod route this key was placed for was subsequently REFUTED (see changelog item 16) and `opentofu/vr1-dc0-maas` is currently UNUSED -- if that root is retired at stage close, this key's on-disk copy and the tfstate should be removed with it, leaving only the CLI profile. Custody detail off-repo per D-069. | +| SEC-013 | 2026-07-20 | **MAAS API key materialized to disk on the region.** An admin-scoped MAAS API key (`consumer:token:secret`) was placed by the operator in a 0600 file on `voffice1` so the `opentofu/vr1-dc0-maas` root could consume it via `TF_VAR_maas_api_key`. It grants FULL MAAS admin API access (machines, power, deploy, users). Two exposure surfaces beyond the file itself: (a) the OpenTofu **state file** of any root that uses the maas provider records it -- unavoidable with this provider, so that state inherits credential handling (0600, never committed); (b) a MAAS CLI profile was also created for the operator user from the same key. It was verified by FORMAT ONLY (71 bytes, 3 colon-separated parts) and never printed, echoed, or passed in argv. | 2026-07-20 step-D part 2; `opentofu/vr1-dc0-maas/main.tf` header; session changelog item 15 | operator | **OPEN -- rotation obligation (surface NARROWED 2026-07-21).** The retire-fully ruling was executed 2026-07-21: `opentofu/vr1-dc0-maas` removed from the repo, the on-disk key file (`~/vr1-dc0-creds/maas-api-key.txt` on voffice1) and the root's `terraform.tfstate` DELETED, absence verified by listing (session changelog 2026-07-21 items 13-15; NOTE the pod-object deletion cascaded to the 9 machine records -- incident capture `docs/audit/incident-20260721-pod-delete-cascade.txt`, recovered same-day). REMAINING surface: the operator's MAAS CLI profile only. Rotate the underlying API key at v1 close, or immediately if `voffice1` is rebuilt/shared. Custody detail off-repo per D-069. | | SEC-014 | 2026-07-21 | **Rack cluster secret exposed to session context.** During the commissioning diagnosis, a read of `/var/snap/maas//agent.yaml` on the DC0 rack (hunting the agent resolver's config surface) returned the rack's MAAS cluster `secret` into the Claude session context, the operator terminal scrollback, and the as-executed log (`~/as-executed/2026-07-21-ops-commissioning-diag.log`, 0600, jumphost-only). The secret authenticates rack<->region enrollment. The read was not anticipated to contain a credential (config file, not a key file); disclosed same-session. | 2026-07-21 session changelog item 6; docs/audit/commissioning-diag-20260721.txt | operator | **OPEN -- rotation obligation.** Rotate the MAAS shared secret (= rack re-enrollment for `7chphy`) at a convenient maintenance point, or immediately if session artifacts leave the jumphost. Process fix queued: add `agent.yaml` to the guard hook's never-read list alongside key/cred globs. | | SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk). | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` | operator | **CLOSED 2026-07-16 (operator ruling -- keep uniform 6-NIC).** Review R3-F10: A2's cross-examination refuted the attack-surface concern -- in the isolated-L2 sim the unbound vNICs have no reachability out, and pruning would INCREASE Roosevelt-delta (baremetal trunks all VLANs to every node on bonded NICs, so all planes are present regardless of L3 binding). Uniform 6-NIC is the more Roosevelt-faithful model. Accepted non-issue; no code change. | diff --git a/runbooks/appendix-A-troubleshooting.md b/runbooks/appendix-A-troubleshooting.md index e19add3..bb5f6b9 100644 --- a/runbooks/appendix-A-troubleshooting.md +++ b/runbooks/appendix-A-troubleshooting.md @@ -777,3 +777,33 @@ region-side runbook step: metal-admin subnet dns_servers=, allow_dns=false. Verify: `dig @ maas-internal SOA` answers; a canary commission goes Ready in minutes instead of timing out. + +## Symptom: `maas ... vm-host delete ` (or pod delete) silently REMOVES enlisted machines -- `machines read` count drops [2026-07-21] + +CAUSE: MAAS deletes every machine record LINKED to a pod when the pod is + deleted -- no decompose flag needed, no warning, records only (it + does not touch the hypervisor; a broken pod connection changes + nothing). Machines get linked at pod REGISTRATION/refresh: even a + pod whose refresh FAILED partway (e.g. the 2026-07-20 virsh pod + that died on `domblkinfo ... missing storage backend for 'volume'`) + can have enumerated far enough to claim existing machines. So "the + pod apply failed" does NOT imply "no machines are associated". + +CHECK: BEFORE any vm-host/pod delete, read the pod's machine list and + expect ZERO: + + maas admin vm-host read # inspect + maas admin machines read | + + A non-empty list is a STOP: unlink or accept the record loss + deliberately (operator-gated), never as a cleanup side effect. + +FIX: If the records are already gone and the domains still exist + (2026-07-21 incident, capture docs/audit/ + incident-20260721-pod-delete-cascade.txt): power the domains on + (virsh) -> they PXE re-enlist (MACs pinned = same identity basis) + -> scripts/maas-node-power.sh dry then --commit -> re-commission + -> verify shapes + Ready (recovery measured ~5 min fleet-wide, + docs/audit/incident-20260721-recovery-verify.txt). NOTE: MAAS + assigns NEW random hostnames at re-enlistment -- update anything + quoting the old ones.