diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 707064f..94e9c30 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -184,7 +184,9 @@ were wrong; its instrument found the cause). D-131 PARTIALLY RULED (sub-1 RULED 2026-07-21: the forwarder is the STANDING per-DC pattern, repo-carried + part of DC standup definition-of-done; - sub-2 RULED 2026-07-21: metal-admin-only scope; sub-3..4 OPEN -- + sub-2 RULED 2026-07-21: metal-admin-only scope; sub-3 RESOLVED + 2026-07-21 by measurement: no dhcpd option-6 defect, stale read, + no second LP; sub-4 OPEN + pinned DNS architectural review -- status line in design-decisions.md is the authority). SEC-014 OPENED (rack cluster secret exposure during diagnosis). Queued delivery: incident docs @@ -529,6 +531,15 @@ 6. Audit Phase 4 rulings: GA-R1..GA-R6 individually, plus the legal stage-status vocabulary A/B (GA-F10 operator note) -- each gated, one ruling per exchange. +7. D-132 -- Roosevelt per-DC MAAS topology (PROPOSED 2026-07-21, + operator-pinned to the next deployment; three question groups: HA + regions per DC, rack-top rack controllers, cross-site backup + custody). Present at Roosevelt MAAS design time, not before. +8. D-131 sub-4 + pinned DNS architectural review (operator-directed + 2026-07-21: stack best practices, forwarder security implications, + vendor guidance on the "utility nodes" DC-to-DC configuration) -- + executes at next-deployment design time; feeds the sub-4 ruling + with the LP outcome. ## 9. Additional defect found while authoring (FIXED in sweep Batch 0.3, 2026-07-19 -- wrap-aware exclusion, GA-F15; history below) diff --git a/docs/audit/d131-sub3-dhcpd-option6-20260721.txt b/docs/audit/d131-sub3-dhcpd-option6-20260721.txt new file mode 100644 index 0000000..be35248 --- /dev/null +++ b/docs/audit/d131-sub3-dhcpd-option6-20260721.txt @@ -0,0 +1,37 @@ +D-131 sub-3 decisive check -- rendered dhcpd.conf option 6 on the dc0 rack +Date: 2026-07-21 (evening session, queue pass) +Runner: OPERATOR (command classifier-walled for the agent; operator ran via +the session's `!` prefix and pasted output back -- gated copy-paste model). +Command + output VERBATIM: + +jessea123@vcloud:~$ ! ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2 'sudo grep -B2 -A12 "subnet 10.12.8.0" /var/snap/maas/common/maas/dhcpd.conf' +# +shared-network vlan-5005 { + subnet 10.12.8.0 netmask 255.255.252.0 { + ignore-client-uids true; + next-server 10.12.8.2; + option subnet-mask 255.255.252.0; + option broadcast-address 10.12.11.255; + option domain-name-servers 10.12.8.3; + option domain-name "maas"; + option domain-search "maas"; + option ntp-servers 10.12.8.2; + # + # Bootloaders + # + if option user-class = "iPXE" or (exists ipxe.http and ( exists ipxe.bzimage or exists ipxe.efi )) { + +ADJUDICATION (pre-committed interpretation, stated before the run): +option domain-name-servers = 10.12.8.3 ONLY. The 2026-07-21 morning +observation (option 6 = "10.12.8.2, 10.12.8.3") was a STALE READ taken +before the Temporal-driven DHCP re-render propagated the allow_dns=false +flip -- consistent with the shipped source (snap rev 41649, MAAS 3.7.2, +maasserver/dhcp.py: get_default_dns_servers() returns [] when +not subnet.allow_dns, BEFORE any rack-IP/use_rack_proxy logic runs; +make_subnet_config() then appends only subnet.dns_servers). The +maastemporalworker DHCP workflow carries no DNS composition of its own. +CONCLUSION: no defect, no LP report of its own; nodes do NOT rely on +SERVFAIL failover -- they receive the forwarder address alone. dhcpd.conf +mtime at check time: Jul 21 19:16 (re-rendered since the observation). +Config file itself remains root-only (0640); this capture is the citable +evidence for the sub-3 closure. diff --git a/docs/changelog-20260721-tenant-review-pin.md b/docs/changelog-20260721-tenant-review-pin.md index a7c80d1..04a6156 100644 --- a/docs/changelog-20260721-tenant-review-pin.md +++ b/docs/changelog-20260721-tenant-review-pin.md @@ -48,3 +48,34 @@ surface touched -- dc0-node-dns.service already runs metal-admin-only, so the ruling codifies the running shape; zero config delta. - **Revert:** git revert this commit (records only). + +## 4. D-131 sub-3 RESOLVED by measurement (dispatch ruled "Investigate now (Recommended)") + +- Read-only investigation, split execution: agent read the shipped source + over the measured rack reach (ssh -J voffice1, dc0 key, no sudo); the + decisive dhcpd.conf read is root-only AND the agent's ssh+sudo was + classifier-walled, so the OPERATOR ran it via the `!` prefix and pasted + output (capture: docs/audit/d131-sub3-dhcpd-option6-20260721.txt). +- Result: option 6 = 10.12.8.3 ALONE. Source (maasserver/dhcp.py, snap rev + 41649): allow_dns=false short-circuits get_default_dns_servers() to [] + before any rack-IP prepend logic; maastemporalworker has no DNS + composition of its own. The morning prepend observation was a stale + pre-re-render read. NO defect; NO second LP; no live surface carried the + stale claim (archived changelog stays as history). +- **Revert:** git revert this commit (records only). + +## 5. Two operator pins landed: DNS architectural review + D-132 (MAAS topology) + +- PINNED REVIEW (operator-directed, recorded in the D-131 Status block, + feeds sub-4): next-deployment architectural review of the DNS setup -- + (a) stack best-practice conformance, (b) forwarder security + implications, (c) vendor-documented guidance for the "utility nodes" + DC-to-DC traffic configuration. +- D-132 FILED, PROPOSED / OPEN (next-free verified 132 pre-assignment): + Roosevelt per-DC MAAS topology -- HA regional VMs per DC, rack-top rack + controllers deferring to the site region (multi-rack DCs), cross-site + regional backup custody for rebuild. Operator directive quoted verbatim + in the entry; each question rules individually per GA-R5 at Roosevelt + MAAS design time. Ledger next-free re-seeded D=133; CURRENT-STATE + section 8 items 7-8 added (same-commit coupling). +- **Revert:** git revert this commit (records only). diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 22a0282..5413104 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4220,7 +4220,22 @@ exact utterance: "Metal-admin only (Recommended)". Scope is therefore FIXED at metal-admin only (dnsmasq bound to the single metal-admin alias; edge keeps its own WAN-side DNS; SEC-010 untouched); revisit only if a D-129 (iii)/(iv) ruling later consolidates site -services onto the edge. Sub-decisions 3-4 remain OPEN. +services onto the edge. Sub-decision 3 RESOLVED 2026-07-21 by measurement (dispatch ruling +GA-R5: question = "how should we dispatch the dhcpd option-6 prepend (rack IP still offered +first despite allow_dns=false)?"; operator selection, exact utterance: "Investigate now +(Recommended)"). Investigation result: the shipped 3.7.2 source (snap rev 41649, +maasserver/dhcp.py) makes the rack-IP prepend UNREACHABLE when allow_dns=false -- +get_default_dns_servers() returns [] before any rack-IP logic -- and the rendered +dhcpd.conf now carries option 6 = 10.12.8.3 ALONE (operator-run capture, +docs/audit/d131-sub3-dhcpd-option6-20260721.txt). The morning prepend observation was a +STALE READ pre-Temporal-re-render. NO defect, NO second LP; nodes do not rely on SERVFAIL +failover. Sub-decision 4 remains OPEN, and now carries a PINNED ARCHITECTURAL REVIEW +(operator-directed 2026-07-21) to be executed at next-deployment design time: (a) does the +node-DNS setup conform to stack best practices; (b) security implications of the applied +metal-admin forwarder (exposure scope, cache behavior, logging, hardening); (c) whether a +vendor-documented configuration recommendation exists for the "utility nodes" DC-to-DC +traffic configuration we are utilizing. Review feeds the sub-4 Roosevelt ruling alongside +the LP outcome. **Context (measured, 2026-07-21).** MAAS 3.7's rack-only controllers serve node DNS from the `maas-agent` resolver, which on an internet-isolated rack SERVFAILs ALL queries -- including its @@ -4244,8 +4259,43 @@ 1. Adopt the forwarder as the STANDING per-DC pattern (repo-carried, site-keyed unit + module or script delivery, harness) vs treat as temporary until the MAAS agent defect is fixed upstream. 2. RULED 2026-07-21 (Status block above): metal-admin only. -3. dhcpd still prepends the rack IP to option 6 despite `allow_dns=false` (relies on client - SERVFAIL failover -- works, proven, but is an extra moving part): investigate template - behavior; possibly LP material of its own. +3. RESOLVED 2026-07-21 (Status block above): stale read, no defect -- option 6 carries the + forwarder alone post-re-render; prepend logic is source-unreachable under allow_dns=false. 4. Roosevelt: same forwarder shape, or region-DNS-over-WAN with a firewall exception, or wait for the upstream fix and require a minimum MAAS version. Evidence needed: LP outcome. + +## D-132: PROPOSED -- Roosevelt per-DC MAAS topology: HA regions, rack-top racks, cross-site backup custody [ARCH] + +**Status:** PROPOSED / OPEN 2026-07-21. Operator-pinned to the NEXT DEPLOYMENT -- no ruling +yet; present options at Roosevelt MAAS design time, BEFORE any DC's first commissioning. +Filed from the queue-pass session; the pin directive is quoted verbatim below (the pin is +an instruction to carry the questions, not a ruling on any of them -- GA-R5 applies to each +question individually when it is presented). + +**Operator directive (2026-07-21, exact utterance):** "Pin the following MAAS questions and +changes to the next deployment: HA MAAS regional VMs per DC, rack top MAAS that defers back +to the site specific regional MAAS controller since there will be multiple racks at some +DCs. Regional MAAS controllers at other sites hold the backup data for other DCs in case of +a rebuild requirement." + +**Questions to rule (each its own GA-R5 exchange when presented):** +1. **HA MAAS region per DC.** Move from the current single-region-per-site shape (VR1: + one region on voffice1) to HA regional controller VMs per DC. Sub-questions: HA member + count, postgres HA mechanism, VIP/load-balancing for the region API, and whether the + region VMs live on utility/infra nodes per DC. +2. **Rack-top rack controllers deferring to the site region.** One rack controller per rack + (some DCs will have multiple racks), each enrolled to the DC's OWN regional MAAS. + Interactions to carry into the options: D-131 (the node-DNS forwarder is rack-scoped -- + composes naturally with per-rack controllers, but each rack needs its own forwarder + + dc-rack-net delivery), the MAAS 3.7 rack-only agent-resolver defect (D-131 context; LP + outcome may set a minimum MAAS version), and SEC-010-class transit posture per rack. +3. **Cross-site backup custody.** Each site's regional MAAS backup data held by OTHER + sites' regional controllers for rebuild-from-loss. Sub-questions: what constitutes the + backup set (postgres dump, images, secrets -- secrets handling is a SEC row when built), + transport + encryption, custody rotation, and a restore REHEARSAL as definition-of-done. + +**Context (measured, VR1).** dc0 today: single Office1 region + single dc0 rack over the +D-126 transit; per-machine virsh power (D-103/D-123 amendments); rack-side twin invariant +(D-131 sub-1) delivered via scripts/dc-rack-net.sh. The A1 test passes: any Roosevelt build +session must grep this before laying out MAAS -- topology decides commissioning order, +DNS/DHCP shape per rack, and backup custody across DCs. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 64851ee..3d6344f 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -36,14 +36,16 @@ update cadence + Juju controller patch policy), D-129 (OPNsense edge plugin/add-on base profile -- OPEN / partially ruled, four sub-decisions), D-131 (node-facing DNS strategy for rack-only controllers -- PARTIALLY RULED: sub-1 ruled + delivered 2026-07-21, sub-2 ruled - 2026-07-21 (metal-admin only), sub-3..4 open; - now surfaced by the scan itself after the PARTIAL fix). Status lines in + 2026-07-21 (metal-admin only), sub-3 resolved 2026-07-21 (no defect), sub-4 open + pinned + DNS architectural review; + now surfaced by the scan itself after the PARTIAL fix), D-132 (Roosevelt per-DC MAAS + topology -- PROPOSED 2026-07-21, operator-pinned to the next deployment). Status lines in `docs/design-decisions.md` are the only ruling authority. - **OPEN security rows:** 10 open per `bash scripts/ledger-scan.sh` (re-verified at stage close; SEC-013 surface NARROWED 2026-07-21, row still open for rotation). The SEC register of record is `docs/security-ledger.md`; row-level dispositions live THERE only (GA-R4 amendment F3) -- this block carries pointer + count, never rows. -- **Next-free numbers:** D = **132**, DOCFIX = **198**, BUNDLEFIX = 052. +- **Next-free numbers:** D = **133**, DOCFIX = **198**, BUNDLEFIX = 052. - **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- historically a decoy token in prose inflated the next-free counter (hardened in DOCFIX-174); the authoring discipline