Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260807-dc0-tailscale-install.md
@JANeumatrix JANeumatrix 6 hours ago 7 KB site-tailscale: advertise-only subnet router + authkey=file: + check guards
Raw Blame History

changelog 2026-08-07 -- dc0 .7 Tailscale install: office1-mirrored (untagged), site-tailscale.sh prep+forwarding

Session goal: finish the dc0 .7 Tailscale subnet-router install so the operator can log into Headscale and approve the route -> unblocks dc0 Step 3.3 (Horizon-over-tailnet). Operator directive: mirror the working Office1 router; defer tags/autoApprovers/ACL to the bare-metal build.

Branch dc-dc-stage5-preconditions. Status authority is CURRENT-STATE.md (this file is history).

Measured ground truth (read-only, this session)

  • dc0 .7 (10.12.8.7, MAAS name known-marten, user ubuntu, sudo ok): tailscale package ABSENT; snap present; default route via 10.12.4.1 dev enp2s0; apt proxy http://10.12.8.6:8000/; IP forwarding OFF (ip_forward=0, all.forwarding=0), no /etc/sysctl.d/99-tailscale.conf. iptables 1.8.7 + iproute2 5.15 present (tailscale deps).
  • office1-tailscale (working reference): deb tailscale 1.98.9, tailscaled active, forwarding ON via /etc/sysctl.d/99-tailscale.conf; tailscale debug prefs: Tags: None (UNTAGGED), AdvertiseRoutes: ['10.10.0.0/22'], RouteAll: True (accept-routes), NoSNAT: False (SNAT on), control https://tailscale.baldurkeep.com:443.
  • Egress for the package: .7 cannot resolve/reach pkgs.tailscale.com (external DNS blocked; apt-proxy will not forward it). The rack and vcloud both have egress; vcloud pulls office1's exact pool/tailscale_1.98.9_amd64.deb (SHA256 c739c29ec2342cc7df1a24cd102a3dfb51b359f79338d433ce8f052aeebc62ff, Depends: iptables). Method = staged .deb via a host with egress -> scp to .7 -> dpkg -i.
  • Custody: operator placed the plain Headscale pre-auth key on vcloud at ~/vr1-tailnet-creds/headscale-preauth-vr1-dc0.key (0600; sha256 d2e74720...); never read into session context. Used only via $TS_AUTHKEY_FILE.

Changes (repo delivery; ALL GREEN before any live step)

  1. scripts/site-tailscale.sh -- three verbs (prep|install|check) + untagged VR1 + forwarding.
    • New prep <site>: installs the tailscale package from a rack-staged .deb ($TS_DEB, since .7 has no egress) and enables IP forwarding (writes /etc/sysctl.d/99-tailscale.conf, sysctl --system, then ASSERTS v4+v6 forwarding took).
    • Tag now OPTIONAL: TS_TAG defaults EMPTY = untagged (office1-mirrored, VR1); install omits --advertise-tags and adds --accept-routes (office1 RouteAll); check skips the tag assertion when untagged. TS_TAG=tag:subnet-router restores the D-129(iii) tagged design.
    • check now ASSERTS IP forwarding -- the load-bearing subnet-router property. Root cause it closes (advisor 2026-08-07): tailscale up --advertise-routes WARNS-and-succeeds with forwarding off, so the join completes and Headscale approves the route while NOTHING forwards to Horizon -- a green gate blind to the one property that makes the router work.
    • Why (revert): git checkout <prev> -- scripts/site-tailscale.sh restores the tagged-only, no-prep, no-forwarding form. Evidence: harness below.
  2. tests/site-tailscale/run-tests.sh -- 23 cases, up from ~15. New failing-direction fixtures for every new path: prep-noforward->4, prep-nopkg-nodeb->4, prep-fromdeb->0, check-noforward ->1, check-tag-happy->0, check-tag-notag->1, install-tag-happy->0. Fakes sysctl/dpkg added. Result: 23 pass / 0 fail. Revert: restore the file with the script.
  3. docs/design-decisions.md -- D-129(iii) AMENDMENT (2026-08-07 b), [OPS], no new D-number. Records the operator's untagged/office1-mirror deferral (verbatim utterance quoted) and the accepted VR1 exposure (key-expiry clock + no ACL star boundary yet); the Roosevelt tagged/ACL design in ruling (b)/notes 1-2 is UNCHANGED, only deferred. Revert: delete the ### D-129(iii) -- AMENDMENT (2026-08-07 b) subsection.

Gates: repo-lint 0 fail / 1 pre-existing legacy L1 warn; harness 23/23.

NOT YET DONE (live, gated -- pending this delivery's commit)

Stage the .deb (vcloud dl + sha256 verify -> scp to .7) -> dpkg -i -> prep vr1-dc0 -> install vr1-dc0 (untagged, key via $TS_AUTHKEY_FILE, removed after) -> operator approves the 10.12.8.0/22 route in Headscale -> check vr1-dc0 green -> confirm Horizon over the tailnet (closes Step 3.3). dc1 .7 is downstream of the dc1 node rebuild -- not prepped this session.

Findings queued for the close sweep

  • F1: .7 MAAS machine name is still the auto name (known-marten), not vr1-dc0-tailscale-01 (naming convention owed; cosmetic, does not affect the join -- tailnet name set by --hostname).
  • F2: raw grep DOCFIX- returns a stray DOCFIX-1004 string somewhere (ledger-scan's next-free is 213); verify it is not a real assignment collision.

UPDATE (same session) -- install incident + fix + live re-join

INCIDENT: the first live join succeeded (node registered, machineAuthorized=true) but bringing .7 up as a subnet router with --accept-routes blackholed its own L3 on both legs and locked us out (L2/ARP alive, no IP). Root cause: a subnet router that also advertises the /22 its own SSH path lives in must NOT --accept-routes -- it imports a route that sends its own return traffic into tailscale0. office1 tolerates accept-routes only because it is steady-state / reached differently. Reboot-race recovery lost (tailscaled reconnects in seconds with a cached control key). Recovered via qemu-nbd offline mask of tailscaled on the rack (VM disk /var/lib/libvirt/vr1-dc0-inner/vr1-dc0-tailscale-01-disk.qcow2, root = nbd0p2), then purged tailscale + wiped state for a clean base (operator: "remove and rebuild tailscale").

SECONDARY: the pre-auth key leaked -- it was on the tailscale up argv (visible in ps) and reached a session transcript. Operator reused the existing key for the re-join (accepted); it should still be rotated. The tool now passes --authkey=file: so the value never touches argv again.

TOOL FIXES (this commit, harness 27/0, gauntlet ALL GREEN 102, repo-lint 0-fail):

  • install: dropped --accept-routes (advertise-only subnet router) -- the incident fix; and --authkey=file:<path> (no argv leak; $TS_AUTHKEY env is staged to a 0600 temp file).
  • check: two new assertions with failing-direction fixtures -- own-subnet-not-via-tailscale0 (the blackhole guard) and control-plane reachability (HTTP response from $LOGIN_SERVER, not just DNS -- the green-except-route failure class).
  • Revert: git revert this commit; the prior form is the accept-routes/argv-key version.

LIVE RE-JOIN (advertise-only, safety-net protected): disabled v6 at runtime on .7 (fast join; no global v6 to lose), armed a 180s systemd-run auto-tailscale down net, ran install -> up Running/Online, TSIP 100.64.0.57, own-subnet NOT hijacked, control reachable, .7 stayed reachable (no lockout). check FAILs only on the route pending Headscale approval (expected). Safety net cancelled, key removed from .7.

OWED / follow-ups:

  • Operator: approve 10.12.8.0/22 on vr1-dc0-tailscale (100.64.0.57) in Headscale; delete the stale node 100.64.0.56 from the failed first attempt. Then check -> green + Horizon confirm.
  • disable_ipv6 on .7 is runtime-only (not persisted) -- on reboot v6 returns and the re-join takes ~7 min (v6 control backoff before v4 fallback). Decide whether to persist it (fast reconnect) vs accept the delay. NOT baked into prep (unproven-persistence; advisor guidance).
  • F3 (new): the maas snap refresh (rev 41649) wiped the vr1-dc0-region CLI profile on voffice1 -- a true MAAS rebuild is blocked until the profile is re-logged in (region API key + tunnel to 10.12.8.6:5240). Likely affects dc1 too. Snap-per-revision profile store, the known trap.
  • Rotate the leaked pre-auth key once dc0 is confirmed green.