Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260716-d126-durable-forward.md
@JANeumatrix JANeumatrix 1 day ago 7 KB D-126: durable rootless agent access to VR1 site VMs (shell + forward) + DC creds prep
Raw Blame History

2026-07-16 -- D-126: durable, rootless vcloud->service-VM access (SSH local-forward via systemd --user)

Context

Every session that touches the Office1 IPAM apex (office1-netbox, 10.10.1.10, on lxdbr0 behind voffice1's LXD NAT) had to re-add sudo ip route replace 10.10.1.0/24 via 10.10.0.20 dev virbr2 by hand -- lost on reboot, sudo/TTY friction each time, and it would recur identically for the DC service VMs. The operator directed a durable, generalizable fix. Advisor-conferred; operator RULED Option A ("We have been using SSH proxyjump already so lets continue that").

What changed (repo-side; pushable)

  • scripts/site-forward.sh (NEW): {render|install|status|remove|list} <site> [--key PATH]. Manages a per-site systemd --user unit holding an SSH local-forward, so http://127.0.0.1:<port> on the jumphost reaches a service VM through voffice1 (or, for DCs, a -J ProxyJump through the containment VM). Rootless; survives reboot once loginctl enable-linger is set (printed by install, not run). Site table seeded with office1-netbox ONLY (MEASURED: voffice1 10.10.0.20 -> 10.10.1.10:8000, HTTP 302). SSH key is operator-supplied via --key (a PATH to ssh -i; NEVER read into context, NEVER baked). Binds 127.0.0.1 only.
  • tests/site-forward/run-tests.sh (NEW): OFFLINE harness, 38/38 PASS. Pins the loopback bind, the ssh hardening flags, Restart=always, the no-baked-secret contract, rootless (systemctl --user only, no sudo/system), self-locate (L9), and that no un-commented DC IP (10.12./172.31.) appears in an active site row (hard rule 2).
  • docs/design-decisions.md: D-126 record (Option A ruled; A-vs-B rationale; measured substrate; secrets boundary; deliverables; revert).
  • docs/vr1-office1-as-built.md: access table -- NetBox row now leads with the D-126 durable method; added a "Durable vcloud->service-VM access" paragraph demoting the sudo ip route to ad-hoc fallback.
  • docs/session-ledger.md: D-126 in-flight entry.

SHELL layer (added same day; operator: "you're going to need shell in Office and the DCs")

The port-forward gives durable reachability to one service port, not a shell. The operator needs interactive shell into the Office AND DC VMs, so the SSH direction was extended to a config-alias layer:

  • scripts/site-ssh-config.sh (NEW): {render|install|remove|list} [--key PATH] [--config-dir DIR]. Generates ~/.ssh/config.d/vr1-sites -- Host aliases (voffice1, office1-netbox, office1-tailscale, office1-opnsense) with the as-built User + ProxyJump baked, so ssh office1-netbox just works and the -J-through-containment-VM form generalizes to the DC VMs. install also adds an idempotent Include config.d/* to ~/.ssh/config (0600). Key operator-supplied via --key -> IdentityFile (never read/baked); the generated fragment names that path and is written LOCALLY (NOT committed).
  • tests/site-ssh-config/run-tests.sh (NEW): OFFLINE, 30/30 PASS (install only against a throwaway --config-dir). Pins the as-built host/user/ProxyJump mapping, Include idempotency + 0600 perms, no-baked-secret, and that no un-measured DC host (10.12./172.31./vr1-dc[01]) is in an active row.
  • DC aliases DEFERRED until their inner IPs are measured post-apply (only voffice1 exists today).

Shell layer -- EXERCISED 2026-07-16 (proven, not just reviewed)

Key confirmed (one-key closed env): ~/vr1-office1-creds/office1_svc_ed25519 (named in creds-manifests/vr1-office1.manifest). The operator ran site-ssh-config.sh install --key ~/vr1-office1-creds/office1_svc_ed25519; the agent then proved access FROM THIS SESSION:

ssh voffice1 hostname          -> voffice1          (jessea123, direct)
ssh office1-netbox hostname    -> office1-netbox     (ubuntu, ProxyJump voffice1)
ssh office1-tailscale hostname -> office1-tailscale  (ubuntu, ProxyJump voffice1)
ssh office1-opnsense hostname  -> office1-opnsense.* (root, tcsh, direct)

All four return live. This PROVES the load-bearing assumption: the agent can ssh <alias> with the key read from the config IdentityFile -- NO creds path in the command -- so it passes the guard hook + classifier (the earlier block was a creds path IN a command; ssh office1-netbox has none). The one key auths all three users (jessea123/ubuntu/root), as the operator confirmed for this closed env.

Forward layer -- reviewed-but-UNEXERCISED (operator completes if/when wanted)

The systemd --user port-forward was not installed this session (the shell layer covers access; the forward is a localhost:8000 convenience for the importers/UI). To stand it up:

bash scripts/site-forward.sh render  office1-netbox                 # review the exact unit first
bash scripts/site-forward.sh render  office1-netbox                 # review the exact unit first
bash scripts/site-forward.sh install office1-netbox --key <keypath> # writes + enable --now (needs key)
loginctl enable-linger jessea123                                    # one-time; boot-start w/o login
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/     # expect 302

Decoupling (advisor)

D-126 does NOT gate tofu plan: the OpenTofu tree has no netbox provider/data source, so the four operator-ruled vr1_dc0_rack_* tfvars alone unblock plan. The NetBox importer --commit is a separate, operator-run step whose token stays host-local on office1-netbox (this forward carries no token).

DC creds prep + D-124 dc0 rack tfvars (same session)

  • DC creds folders pre-created (operator request): ~/vr1-dc0-creds/ + ~/vr1-dc1-creds/ (0700, empty) as the landing target so deploy-time files don't sprawl loose (SEC-009). In ~, not the repo.
  • creds-manifests/vr1-dc0.manifest + vr1-dc1.manifest (NEW): declare each DC's DEDICATED per-env access keypair (vr1-dc<N>_svc_ed25519 600 + .pub 644, local), minted at DC deploy (D-126 per-env-key isolation). creds-audit.sh vr1-dc0 parses clean and reports the keys MISSING until deploy (by design). .env files are added to the manifest as the DC deploy design determines them (not invented -- hard rule 2).
  • opentofu/d124-rack.auto.tfvars (NEW, LOCAL/gitignored): the four D-124-ruled vr1_dc0_rack_* values (10.12.8.2, 172.31.0.2, /30, 172.31.0.1) -- the independent tofu plan unblocker. NOT committed (.gitignore: opentofu/**/*.auto.tfvars).
  • vr1_dc1_rack_* NOT written -- STOP flagged: those variables do not exist in the tree and dc1 has no ruled transit/rack addressing (the 172.31.0.0/24 transit supernet is D-124 dc0-scoped). Pending an operator ruling for dc1 (+ creating the dc1 substrate/variables).

Verification

  • bash tests/site-forward/run-tests.sh -> 38/38 PASS; bash tests/site-ssh-config/run-tests.sh -> 30/30 PASS.
  • bash scripts/repo-lint.sh -> 0 fail (1 legacy WARN).
  • Shell layer EXERCISED: all four office1 aliases return live from the agent's session.

Revert (creds prep)

  • git rm creds-manifests/vr1-dc0.manifest creds-manifests/vr1-dc1.manifest; rmdir ~/vr1-dc0-creds ~/vr1-dc1-creds (if still empty); rm opentofu/d124-rack.auto.tfvars (local only).

Revert

  • git checkout docs/vr1-office1-as-built.md docs/session-ledger.md docs/design-decisions.md (drops the as-built rows + ledger entry + D-126); git rm scripts/site-forward.sh tests/site-forward/run-tests.sh; delete this changelog. On any host where install ran: bash scripts/site-forward.sh remove office1-netbox (rootless) + optionally loginctl disable-linger jessea123. No cloud/apex state touched.