# changelog 2026-08-07 -- dc0 .7 Tailscale install: office1-mirrored (untagged), site-tailscale.sh prep+forwarding

Session goal: finish the dc0 `.7` Tailscale subnet-router install so the operator can log into
Headscale and approve the route -> unblocks dc0 Step 3.3 (Horizon-over-tailnet). Operator
directive: mirror the working Office1 router; defer tags/autoApprovers/ACL to the bare-metal build.

Branch `dc-dc-stage5-preconditions`. Status authority is CURRENT-STATE.md (this file is history).

## Measured ground truth (read-only, this session)

- **dc0 `.7` (`10.12.8.7`, MAAS name `known-marten`, user `ubuntu`, sudo ok):** tailscale package
  **ABSENT**; snap present; default route `via 10.12.4.1 dev enp2s0`; apt proxy
  `http://10.12.8.6:8000/`; **IP forwarding OFF** (`ip_forward=0`, `all.forwarding=0`), no
  `/etc/sysctl.d/99-tailscale.conf`. iptables 1.8.7 + iproute2 5.15 present (tailscale deps).
- **office1-tailscale (working reference):** deb `tailscale 1.98.9`, tailscaled active, forwarding
  ON via `/etc/sysctl.d/99-tailscale.conf`; `tailscale debug prefs`: `Tags: None` (UNTAGGED),
  `AdvertiseRoutes: ['10.10.0.0/22']`, `RouteAll: True` (accept-routes), `NoSNAT: False` (SNAT on),
  control `https://tailscale.baldurkeep.com:443`.
- **Egress for the package:** `.7` cannot resolve/reach `pkgs.tailscale.com` (external DNS blocked;
  apt-proxy will not forward it). The **rack and vcloud both have egress**; vcloud pulls office1's
  exact `pool/tailscale_1.98.9_amd64.deb` (SHA256 `c739c29ec2342cc7df1a24cd102a3dfb51b359f79338d433ce8f052aeebc62ff`,
  Depends: `iptables`). Method = staged `.deb` via a host with egress -> scp to `.7` -> `dpkg -i`.
- **Custody:** operator placed the plain Headscale pre-auth key on vcloud at
  `~/vr1-tailnet-creds/headscale-preauth-vr1-dc0.key` (0600; sha256 `d2e74720...`); never read into
  session context. Used only via `$TS_AUTHKEY_FILE`.

## Changes (repo delivery; ALL GREEN before any live step)

1. **`scripts/site-tailscale.sh` -- three verbs (`prep|install|check`) + untagged VR1 + forwarding.**
   - New **`prep <site>`**: installs the tailscale package from a rack-staged `.deb` (`$TS_DEB`,
     since `.7` has no egress) and enables IP forwarding (writes `/etc/sysctl.d/99-tailscale.conf`,
     `sysctl --system`, then ASSERTS v4+v6 forwarding took).
   - **Tag now OPTIONAL**: `TS_TAG` defaults EMPTY = untagged (office1-mirrored, VR1); `install`
     omits `--advertise-tags` and adds `--accept-routes` (office1 `RouteAll`); `check` skips the tag
     assertion when untagged. `TS_TAG=tag:subnet-router` restores the D-129(iii) tagged design.
   - **`check` now ASSERTS IP forwarding** -- the load-bearing subnet-router property. Root cause it
     closes (advisor 2026-08-07): `tailscale up --advertise-routes` WARNS-and-succeeds with
     forwarding off, so the join completes and Headscale approves the route while NOTHING forwards
     to Horizon -- a green gate blind to the one property that makes the router work.
   - Why (revert): `git checkout <prev> -- scripts/site-tailscale.sh` restores the tagged-only,
     no-prep, no-forwarding form. Evidence: harness below.
2. **`tests/site-tailscale/run-tests.sh` -- 23 cases, up from ~15.** New failing-direction fixtures
   for every new path: `prep-noforward`->4, `prep-nopkg-nodeb`->4, `prep-fromdeb`->0, `check-noforward`
   ->1, `check-tag-happy`->0, `check-tag-notag`->1, `install-tag-happy`->0. Fakes `sysctl`/`dpkg` added.
   Result: **23 pass / 0 fail**. Revert: restore the file with the script.
3. **`docs/design-decisions.md` -- D-129(iii) AMENDMENT (2026-08-07 b), [OPS], no new D-number.**
   Records the operator's untagged/office1-mirror deferral (verbatim utterance quoted) and the
   accepted VR1 exposure (key-expiry clock + no ACL star boundary yet); the Roosevelt tagged/ACL
   design in ruling (b)/notes 1-2 is UNCHANGED, only deferred. Revert: delete the `### D-129(iii)
   -- AMENDMENT (2026-08-07 b)` subsection.

Gates: `repo-lint` 0 fail / 1 pre-existing legacy L1 warn; harness 23/23.

## NOT YET DONE (live, gated -- pending this delivery's commit)

Stage the `.deb` (vcloud dl + sha256 verify -> scp to `.7`) -> `dpkg -i` -> `prep vr1-dc0` ->
`install vr1-dc0` (untagged, key via `$TS_AUTHKEY_FILE`, removed after) -> **operator approves the
`10.12.8.0/22` route in Headscale** -> `check vr1-dc0` green -> confirm Horizon over the tailnet
(closes Step 3.3). dc1 `.7` is downstream of the dc1 node rebuild -- not prepped this session.

## Findings queued for the close sweep

- F1: `.7` MAAS machine name is still the auto name (`known-marten`), not `vr1-dc0-tailscale-01`
  (naming convention owed; cosmetic, does not affect the join -- tailnet name set by `--hostname`).
- F2: raw `grep DOCFIX-` returns a stray `DOCFIX-1004` string somewhere (ledger-scan's next-free is
  213); verify it is not a real assignment collision.

## UPDATE (same session) -- install incident + fix + live re-join

**INCIDENT:** the first live join succeeded (node registered, `machineAuthorized=true`) but bringing
`.7` up as a subnet router with `--accept-routes` **blackholed its own L3 on both legs** and locked
us out (L2/ARP alive, no IP). Root cause: a subnet router that also advertises the `/22` its own
SSH path lives in must NOT `--accept-routes` -- it imports a route that sends its own return traffic
into `tailscale0`. office1 tolerates accept-routes only because it is steady-state / reached
differently. Reboot-race recovery lost (tailscaled reconnects in seconds with a cached control key).
**Recovered via `qemu-nbd` offline mask of `tailscaled`** on the rack (VM disk
`/var/lib/libvirt/vr1-dc0-inner/vr1-dc0-tailscale-01-disk.qcow2`, root = `nbd0p2`), then purged
tailscale + wiped state for a clean base (operator: "remove and rebuild tailscale").

**SECONDARY:** the pre-auth key leaked -- it was on the `tailscale up` **argv** (visible in `ps`) and
reached a session transcript. Operator reused the existing key for the re-join (accepted); it should
still be rotated. **The tool now passes `--authkey=file:` so the value never touches argv again.**

**TOOL FIXES (this commit, harness 27/0, gauntlet ALL GREEN 102, repo-lint 0-fail):**
- `install`: **dropped `--accept-routes`** (advertise-only subnet router) -- the incident fix; and
  `--authkey=file:<path>` (no argv leak; `$TS_AUTHKEY` env is staged to a 0600 temp file).
- `check`: two new assertions with failing-direction fixtures -- **own-subnet-not-via-tailscale0**
  (the blackhole guard) and **control-plane reachability** (HTTP response from `$LOGIN_SERVER`, not
  just DNS -- the green-except-route failure class).
- Revert: `git revert` this commit; the prior form is the accept-routes/argv-key version.

**LIVE RE-JOIN (advertise-only, safety-net protected):** disabled v6 at runtime on `.7` (fast join;
no global v6 to lose), armed a 180s `systemd-run` auto-`tailscale down` net, ran `install` -> **up
Running/Online, TSIP `100.64.0.57`, own-subnet NOT hijacked, control reachable, .7 stayed
reachable** (no lockout). `check` FAILs only on the route pending Headscale approval (expected).
Safety net cancelled, key removed from `.7`.

**OWED / follow-ups:**
- **Operator: approve `10.12.8.0/22` on `vr1-dc0-tailscale` (100.64.0.57)** in Headscale; delete the
  **stale node `100.64.0.56`** from the failed first attempt. Then `check` -> green + Horizon confirm.
- `disable_ipv6` on `.7` is **runtime-only** (not persisted) -- on reboot v6 returns and the re-join
  takes ~7 min (v6 control backoff before v4 fallback). Decide whether to persist it (fast reconnect)
  vs accept the delay. NOT baked into `prep` (unproven-persistence; advisor guidance).
- **F3 (new): the maas snap refresh (rev 41649) wiped the `vr1-dc0-region` CLI profile** on voffice1
  -- a true MAAS rebuild is blocked until the profile is re-logged in (region API key + tunnel to
  `10.12.8.6:5240`). Likely affects dc1 too. Snap-per-revision profile store, the known trap.
  **>>> RETRACTED 2026-08-08 (measurement, GA-R1 C2): F3 is WRONG. <<<** voffice1's
  `~/snap/maas/41649/.maascli.db` AND `~/snap/maas/current/.maascli.db` BOTH hold
  `[admin, vr1-dc0-region, vr1-dc1-region]`; the `:5243` tunnel is up; `maas vr1-dc1-region version
  read` returns capabilities. The "empty db" reading queried the wrong paths. The REAL, narrower
  finding: the dc0 RACK (172.31.0.2) has the maas CLI but an EMPTY profile db, so phase-04/05 scripts
  that need maas+openstack co-located cannot run on the rack as-is (a D-138 co-location gap) -- see
  `docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt` F2/F3.
- Rotate the leaked pre-auth key once dc0 is confirmed green.