diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 99a72a5..7a2ed4e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1308,6 +1308,44 @@ **G17's dc0 half is NOW GENUINELY ARMABLE and this is its one-shot window** -- the existing capture was taken on the CONTROLLER VM and says so; the nine ROLE nodes are booting for the first time as of this entry. + **>>> THE DEPLOY IS BLOCKED ON ONE PRECISE ARTIFACT DEFECT: THE dc0 MIRROR DOES NOT CARRY + `jammy-backports`. <<<** MEASURED 2026-07-31, end to end: 22 of 33 units in `error`, every + one `hook failed: "install"`, and the hook output is + `E: The repository 'http://10.12.8.4/ubuntu jammy-backports Release' does not have a Release + file.` The mirror serves `jammy` / `jammy-security` / `jammy-updates` **200** and + `jammy-backports` **404** (`curl` per suite against `dists//Release`), matching D-135's + "jammy triple" scope exactly. The node's `/etc/apt/sources.list` carries + `deb http://10.12.8.4/ubuntu jammy-backports main restricted universe multiverse` -- the + `apt-mirror` model-config set at Step 3.5 rewrites the stock Ubuntu sources to point EVERY + suite at the DC mirror, including the one it was never built to hold. All 9 machines are + `started` / `running`, so this is purely the artifact layer; units retry, so they clear on + their own once the suite answers 200. **THE FIX IS A D-135 SCOPE QUESTION AND IS NOT TAKEN + HERE (hard rule 1 + ruled surface):** add `jammy-backports` to the debmirror config and + re-sync, or remove backports from the nodes' sources -- an operator ruling either way. + **AND `dc-mirror.sh check dc0` PASSES while this is true** -- it verifies sync STATUS, not + that the mirror carries every suite the deployed image asks for. That is the + checker-that-cannot-see-the-real-failure class again, and the gap is SUITE COVERAGE. + **>>> QUEUED BY OPERATOR DIRECTION 2026-07-31, NOT BUILT: a per-DC Tailscale subnet router + is now a STANDING DC-STANDUP REQUIREMENT. <<<** Operator, verbatim: **"A closer real work + analog would be to install tailscale in DC0, DC1, and any future DCs as they would be running + their own tailscale on, or beside, the utility node for that particular DC. Queue up the + tailscale installation in DC0 and add it as a requirement for DC1 and any future + installations."** Home of record: `docs/dc-dc-deployment-workflow.md` tooling gap register + **item 21**, which carries the requirement, its definition-of-done per DC, the + vendor-documented build constraints, and the four sub-decisions that need GA-R5 rulings + BEFORE any build (the utility-band OCTET, since D-134's map is a standing cross-DC standard; + the admin-reachability model -- star vs mesh, which is the real architecture question at + region-region scale; HA count per site; SNAT on/off). **This is EXECUTION of the already-ruled + D-129(iii) shape** (dedicated node per site on metal-admin, edge excluded), not a new + decision. **MEASURED while scoping it: SEC-010 does NOT need relaxing** -- Tailscale reaches + each router as ordinary WireGuard UDP to the VM's own address, and the forwarding to permit + is on that VM between `tailscale0` and metal-admin, which is a different decision from the + transit-leg DROP rule. **ALSO MEASURED, and both are defects in the EXISTING Office1 + installation:** the node is UNTAGGED (`AdvertiseTags: None`), so it is owned by a user + identity and carries a 180-day key-expiry clock on the operator's only tailnet path; and + **D-129(iii)'s own text cites D-107 as governing the Office1 installation while D-107 + ("Airgap posture, per-DC artifact mirror, and NTP") rules nothing about Tailscale** -- read in + full, the SECOND instance of the F7 miscitation class, this time inside a ruling. **RUNBOOK DEFECT FOUND IN PASSING, LOGGED NOT FIXED (DOCFIX material):** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557` gives the dc0 deploy WITHOUT `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below includes its diff --git a/docs/changelog-20260731-prefer-ipv6-research.md b/docs/changelog-20260731-prefer-ipv6-research.md index 194b766..7b1554e 100644 --- a/docs/changelog-20260731-prefer-ipv6-research.md +++ b/docs/changelog-20260731-prefer-ipv6-research.md @@ -446,3 +446,61 @@ **Revert.** Not revertible as a repo change; the deploy is live cloud state. Reversing it means `juju remove-application` per app or destroying the model, which is an operator decision. + +## Item 14 -- DEPLOY BLOCKER FOUND, LOGGED NOT FIXED: the dc0 mirror lacks `jammy-backports` + +**Measured end to end.** 22 of 33 units in `error`, every one `hook failed: "install"`. +Hook output: `E: The repository 'http://10.12.8.4/ubuntu jammy-backports Release' does not +have a Release file.` Per-suite probe against the mirror: `jammy` / `jammy-security` / +`jammy-updates` **200**, `jammy-backports` **404** -- exactly D-135's "jammy triple" +scope. The node's `/etc/apt/sources.list` carries `deb http://10.12.8.4/ubuntu +jammy-backports ...`, because the Step-3.5 `apt-mirror` model-config rewrites the stock +Ubuntu sources to point EVERY suite at the DC mirror, including one it was never built to +hold. All 9 machines are `started`/`running` -- purely the artifact layer, and units retry. + +**Not fixed: D-135 scope is a ruled surface** (hard rule 1). Add the suite and re-sync, or +drop backports from the nodes' sources -- operator's call. + +**`dc-mirror.sh check dc0` PASSES while this is true.** It verifies sync STATUS, not suite +COVERAGE against what the deployed image asks for. Logged as the gap, not fixed. + +**Revert.** Nothing to revert. + +## Item 15 -- QUEUED BY OPERATOR DIRECTION: per-DC Tailscale as a standing standup requirement + +**Operator direction, verbatim (2026-07-31):** *"A closer real work analog would be to +install tailscale in DC0, DC1, and any future DCs as they would be running their own +tailscale on, or beside, the utility node for that particular DC. Queue up the tailscale +installation in DC0 and add it as a requirement for DC1 and any future installations."* + +**QUEUED, NOT EXECUTED** -- the current step is the Stage-5 deploy. Home of record: +`docs/dc-dc-deployment-workflow.md` tooling gap register **item 21** (new), carrying the +requirement, a per-DC definition-of-done, the vendor-documented build constraints, and the +four sub-decisions needing GA-R5 rulings before any build: the utility-band OCTET (D-134's +map is a standing cross-DC standard, so it cannot be chosen per-DC), the +admin-reachability model (star vs mesh -- the real architecture question at region-region +scale), HA count per site, and SNAT on/off. + +**This is EXECUTION of the already-ruled D-129(iii) shape**, not a new decision. + +**Measured while scoping it, all read-only:** +- **SEC-010 does not need relaxing.** Tailscale reaches each router as ordinary WireGuard + UDP to the VM's own address; the forwarding to permit is on that VM between `tailscale0` + and metal-admin -- a different decision from the transit-leg DROP rule. +- **voffice1 holds no route to `10.12.x` at all** (`ip route get 10.12.8.1` falls to the + default; only the two transit /30s), so the tailnet stops at Office1 by construction. + A subnet router can only forward to nets its own host can reach. +- **The existing Office1 node is UNTAGGED** (`AdvertiseTags: None`) -- user-owned, with a + 180-day key-expiry clock on the operator's only tailnet path. A defect to fix alongside. +- **D-129(iii) cites D-107 as governing the Office1 installation; D-107 rules nothing + about Tailscale** (read in full). Second instance of the F7 miscitation class, this time + inside a ruling. + +**Research basis.** Tailscale KB (subnet routers, HA, site-to-site, ACL tags, autoApprovers, +key expiry, 4via6, exit nodes) and Headscale's own docs, fetched and quoted rather than +recalled. Two findings the build must honour: `autoApprovers` does not apply retroactively, +so the policy must exist BEFORE a router first advertises; and Headscale does not document +4via6 support, which makes globally-unique per-DC management CIDRs a hard constraint at +region scale rather than a nicety. + +**Revert.** `git revert` -- register item 21 and the CURRENT-STATE entry move together. diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 42cb042..02bac93 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -1114,6 +1114,118 @@ verdict NO LEG REQUIRED on vcloud, with the voffice1-side transit addressing UNMEASURED as above.** +21. **NEW 2026-07-31 -- OPERATOR-DIRECTED, QUEUED NOT BUILT: a per-DC Tailscale + subnet router is a STANDING DC-STANDUP REQUIREMENT, dc0 first.** + + Operator direction, verbatim (2026-07-31): *"A closer real work analog would be to + install tailscale in DC0, DC1, and any future DCs as they would be running their own + tailscale on, or beside, the utility node for that particular DC. Queue up the + tailscale installation in DC0 and add it as a requirement for DC1 and any future + installations."* + + **QUEUED, NOT EXECUTED** (hard rule 1 -- the current step is the Stage-5 deploy). This + row is the requirement's home until it is built. + + **Why it is a requirement and not an improvement.** Today operator reach into a DC is + SSH ProxyJump only: workstation -> vcloud -> voffice1 -> the DC rack's transit /30 -> + (on the rack) metal-admin. There is no tailnet presence in any DC. The Office1 + subnet router advertises `10.10.0.0/22` and NOTHING toward `10.12.x`, and it could not + usefully advertise it either -- MEASURED 2026-07-31: voffice1 holds no route to + `10.12.x` at all (`ip route get 10.12.8.1` falls through to the default via + `10.10.0.1`; its only DC-facing addresses are the two transit /30s). A subnet router + can only forward to nets its own host can reach, so the tailnet stops at Office1 by + construction, not by configuration. + + **THE RULED SHAPE ALREADY EXISTS -- this row is its per-DC execution, not a new + decision.** D-129 sub-decision (iii), RULED 2026-07-21 (GA-R5, operator: *"Dedicated + node per site (Recommended)"*), makes a dedicated subnet-router node on each site's + METAL-ADMIN plane the standing per-site pattern, with the edge excluded (no + management leg, no `os-tailscale`). The operator's own clarification quoted inside + that ruling is the standing context: *"Roosevelt deisgn, and this deployment, will + have a site tailscale installation to access the metal-admin plane at each location."* + [sic] D-129(iii) also states that per-site ACL scoping, key custody and a SEC row land + AT IMPLEMENTATION -- i.e. here. + + **CITATION DEFECT, recorded so it is not propagated: D-129(iii) says "D-107 remains + the governing decision for the Office1 installation". D-107 is titled "Airgap posture, + per-DC artifact mirror, and NTP (VR1)" and rules NOTHING about Tailscale** -- read in + full 2026-07-31. This is the SAME miscitation finding F7 raised against the skill's + routing table on 2026-07-29, now found a second time inside a ruling. The Office1 + installation currently has no governing decision at all. + + **SEC-010 DOES NOT NEED RELAXING, and this is the load-bearing point.** Tailscale + reaches each site's router as ordinary WireGuard UDP addressed to the router VM + itself; the forwarding that must be permitted is ON THAT VM, between `tailscale0` and + its metal-admin NIC. That is a different forwarding decision from the transit-leg + DROP rule (`table inet sec010`), which is untouched. The isolation D-138 was ruled to + work around stays intact -- the same application-layer-vs-kernel-forwarding + distinction SEC-010's own justification draws, and the same one that makes the + voffice1->rack MAAS tunnel legitimate. + + **WHAT MUST BE RULED BEFORE ANYTHING IS BUILT** (each its own GA-R5 exchange; none of + these is engineering judgement): + - **(a) The utility-band OCTET.** D-134's octet map is a STANDING CROSS-DC STANDARD + and divergence between DCs at the same octet is a DEFECT, so this cannot be chosen + per-DC at build time. Currently `.4` artifact service / `.5` Juju controller / + `.6` MAAS region; `.7` is next free. Also to settle: "ON, or BESIDE, the utility + node" -- a dedicated VM in the utility band (the D-129(iii) shape) versus + co-locating the daemon on an existing utility VM. + - **(b) THE ADMIN-REACHABILITY MODEL, and it is the real architecture question.** Does + any DC need to reach ANOTHER DC's metal-admin plane, or is admin traffic strictly + operator->DC? A star keeps a compromised DC's blast radius inside that DC and is + what the existing SEC-010 / D-125 posture already encodes; a mesh creates lateral + management paths and grows ACL surface as O(N^2), which does not survive the + region-region-with-N-DCs shape the operator described on 2026-07-31. Cross-DC + REPLICATION (Ceph rbd-mirror, RADOSGW multisite) runs on the replication plane, not + metal-admin, so it is not an argument for a management mesh. + - **(c) HA count per site** -- one router or two. Two is Tailscale's documented HA + shape; one is a single point of failure for operator access. + - **(d) SNAT on or off** (`--snat-subnet-routes=false`). Off preserves the operator's + source IP in DC-side audit logs; it requires a return route for `100.64.0.0/10` on + the served hosts. + + **VENDOR-DOCUMENTED REQUIREMENTS the build must honour** (researched 2026-07-31 + against tailscale.com/kb and headscale.net; full citations in the session record): + - **Tagged identity, not user identity.** Tailscale KB 1068: "it's more common for + subnet routers to use a tag-based identity", and applying a tag "removes any + user-based authentication" and disables key expiry by default. The EXISTING Office1 + node is UNTAGGED (`AdvertiseTags: None`, MEASURED 2026-07-31) and therefore carries + a user's 180-day key-expiry clock -- a standing outage waiting to happen and a + defect to fix alongside this work. + - **`autoApprovers` must be written BEFORE the router first advertises.** KB 1337: + "Auto-approver policies only apply when Tailscale first receives a subnet route + advertisement. Updating the tailnet policy file ... does not retroactively approve + existing unapproved routes." Get the policy in first or approve by hand forever. + - **Non-overlapping management CIDRs per DC, permanently.** Tailscale's remedy for + overlapping subnets is 4via6 (KB 1201); the SELF-HOSTED Headscale control plane does + not document 4via6 support and carries a closed issue reporting it fails. At + region-region scale this makes "every DC's metal-admin gets a globally unique CIDR" + a HARD ADDRESSING CONSTRAINT, cheap to hold now at three sites and expensive later. + Currently satisfied: `10.10.0.0/22`, `10.12.8.0/22`, `10.12.68.0/22`. + - **An unpoliced Headscale is allow-all.** Its policy reference states that by default + "no policy is loaded which means that Headscale allows all traffic between nodes" -- + so decision (b) is not merely a design preference, it is the thing that stops DC0's + router reaching DC1's management plane. + - **Version gates, and the version is NOT YET MEASURED:** autoApprovers >= 0.17.0, + subnet-router failover >= 0.18.0, Tailscale-model tags >= 0.28.0, HA health probing + + `node.expiry` >= 0.29.0. **Establish the running Headscale version before relying + on any of these.** Headscale documents NO failover time (10s/5s are sample-config + probe defaults); do not carry Tailscale SaaS's documented 15 s across. + - **Encryption terminates at the subnet router.** True by construction -- the served + host runs no Tailscale and holds no key -- but Tailscale's own encryption KB pages + do NOT state it (checked; only a blog post does). Treat last-hop cleartext on + metal-admin as a design fact to reason about, not one to cite. Management protocols + must carry their own encryption (SSH, TLS), which they do. + + **DEFINITION OF DONE, per DC** (so this does not reopen per-DC, the same shape as + item 20's DC generalization): the router VM exists in the ruled utility octet on + metal-admin; IP forwarding enabled ON THE VM only; joined with a TAGGED pre-auth key; + route approved (ideally via `autoApprovers`); a NAMED executable check proving the + route is live from outside the site; a SEC row for the node's key custody + (D-129(iii)); and the addressing constraint above re-verified against the new DC's + CIDR. **Roosevelt analog: this is the operator's out-of-band access path to every + site, so it is a day-1 standup item, not a convenience added later.** + --- ## Where things stand today (2026-07-09) -- update this as stages close