| 2026-07-14 |
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-115 office carve IMPORTED into the Office1 NetBox; DC import blocked on ORG_ULA_48
...
netbox/d115-office-carve.py (+ tests/d115-office-carve/ 20/20 PASS)
Applied to the Office1 sandbox: site vr1-off1, roles office + edge (edge is NEW),
and 8 prefixes. Idempotent (re-run: 0 created / 11 present).
10.10.0.0/16 office container the Office v4 block, /22 per office
10.10.0.0/22 office container VR1 Off1's /22
10.10.0.0/24 office active office1-local LAN (Kea on the edge)
10.10.1.0/24 office active lxdbr0 compose net (MAAS DHCP)
172.30.0.0/16 edge container NEW Edge role; mirrors v6 2602:f3e2:fe::/48
172.30.1.0/24 edge active office1-wan, the simulated ISP uplink
2602:f3e2:f01 :/56 office container mirrors VR0 Off0's e01:/56
2602:f3e2:f01:/64 office active office subnet (NOT deployed; v6 has no egress)
Every address was already RUNNING. D-115 blesses what is deployed -- zero
renumber. What it fixes is that three of them were SQUAT: on the wire with no
allocation behind them.
Writing upstream is GATED IN CODE: the tool refuses a non-sandbox target unless
given --yes-write-upstream. Feeding the production apex is an operator decision,
never a side effect of running an import.
BLOCKED, deliberately: dc-dc-prefixes-import.py --dc dc0 CANNOT run. It needs
ORG_ULA_48, which is still UNASSIGNED (gap G3; the scope doc recommends
fd50:840e:74e2::/48 but no ADOPTED decision assigns it -- grep ORG_ULA
design-decisions.md returns nothing). Using it anyway would be an inferred value
(hard rule 2). This is a DECISION gate, not a tooling gap: the tooling is fixed,
tested and ready. Recorded in the ledger with the exact commands to unblock.
Also: docs/vr1-office1-as-built.md gains the IPAM table, and records that the
Tailscale 10.10.0.0/22 route is now ENABLED (edge GUI reachable from the
workstation with no tunnel), superseding the tunnel instructions.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-13 |
D-116 ADOPTED: no Office1-local GitBucket -- keep using git.baldurkeep.com
...
Operator ruling: "we've decided not to build a separate one on the test cloud. We will continue to
use the current setup and create new repos for deployment tasks as needed."
SUPERSEDES the GitBucket half of D-103 (which gave Office1 three service VMs) and removes GitBucket
from Stage 2's build list entirely. Office1's MAAS-composed service machines are now exactly TWO --
office1-netbox and office1-tailscale -- and both are LIVE.
WHAT THIS ACTUALLY CLOSES, and why it is a simplification rather than a deferral: the Stage 2 runbook
carried an OPEN QUESTION about the Office1-local GitBucket -- "what it mirrors, and under what repo
path, is not decided" -- because a SECOND GitBucket alongside the pre-existing git.baldurkeep.com
raised a real authority/mirroring question nobody had answered. That question is now MOOT. There is
one git service, it is the existing one, and it is authoritative.
Recorded explicitly in D-116 so nobody misreads the scope: D-107's per-DC ARTIFACT mirror (OS/package
artifacts for node provisioning) is a SEPARATE concern and is NOT affected. And if Roosevelt later
wants a site-local git service, that is a fresh decision against real requirements -- not something
this ruling forecloses.
Applied everywhere it was an instruction, not just recorded:
- runbooks/dc-dc-phase1-office1-standup.md: Step 11 replaced with a TOMBSTONE (kept rather than
deleted -- step numbering is referenced elsewhere, and someone WILL come looking after reading
D-103); removed from the sequence, the compose steps, and the open-questions list.
- docs/dc-dc-deployment-workflow.md: dropped from Stage 2's Build and, importantly, from its GATE --
"GitBucket serving" is no longer a gate condition.
- docs/design-decisions.md: D-103 annotated in place with the supersession, so grepping for GitBucket
lands on the amendment rather than the stale instruction.
- docs/vr1-office1-as-built.md + session ledger updated.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Tailscale: office1-tailscale joined the SELF-HOSTED tailnet as the Office1 subnet router
...
Operator's control plane is self-hosted (https://tailscale.baldurkeep.com:443), NOT Tailscale's
public one -- so the public login-URL flow from the previous commit was void. Node reset off
controlplane.tailscale.com and re-enrolled against the operator's server with their auth key.
REACHABILITY PROVEN FIRST, not assumed: DNS + TCP 443 + HTTPS to tailscale.baldurkeep.com confirmed
from the node BEFORE running `tailscale up`, so a failure would have been a network fact rather than
a confusing registration error.
AS BROUGHT UP: --login-server=https://tailscale.baldurkeep.com:443 --authkey=<from 0600 file>
--advertise-routes=10.10.0.0/22 --accept-routes --accept-dns=false --hostname=office1-tailscale
Online as 100.64.0.53 (fd7a:115c:a1e0::35), no health warnings, advertising 10.10.0.0/22 -- exactly
the D-115 Office1 carve, so D-107's "Office1 ONLY" front door is enforced by the CARVE itself rather
than by policy alone.
ONE DELIBERATE DEVIATION from the operator's usual command: --accept-dns=false. This is a
MAAS-managed machine and MAAS's bind9 serves its internal names; letting the tailnet take over DNS
would break that resolution. Flagged rather than silently applied -- trivial to flip if MagicDNS is
wanted here.
OPERATOR ACTION STILL REQUIRED: the 10.10.0.0/22 route is ADVERTISED but must be ENABLED on the
control server (Headscale: `headscale routes enable`) before other tailnet nodes route to Office1.
Once enabled, the whole Office1 carve -- edge GUI, MAAS UI, NetBox -- is reachable from the tailnet
with NO SSH tunnels, superseding most of the tunnel commands in the as-built doc.
SECRET HANDLING: the auth key was written by the operator to a 0600 file OUTSIDE this session,
consumed BY PATH, never printed into context, and the copy shipped to the node was SHREDDED after
enrolment (tailscaled holds its own node key now). Tracked as SEC-008 -- a rotation obligation, not
a delete-me, since the key remains on vcloud for re-enrolment.
docs/vr1-office1-as-built.md updated with the tailnet addresses, the exact bring-up command, the
DNS deviation, and the pending route-enable step.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1: edge route to the compose net + Tailscale subnet router + an as-built connection reference
...
THE ROUTE -- and an OUTAGE I CAUSED AND REVERTED. Adding a LAN gateway on OPNsense for voffice1
SILENTLY STOLE THE DEFAULT ROUTE: the edge's default flipped from 172.30.1.1 (ISP) to 10.10.0.20,
and the site lost egress. Root cause: OPNsense's `defaultgw` field in the API output is DERIVED
FROM `priority` (lower = more preferred), NOT from the value you send -- I sent defaultgw="0", it
came back True, and I had ALREADY run `reconfigure` before reading anything back. Detected within
~2 minutes by checking egress, reverted by deleting the gateway; default route and egress restored.
Redone SAFELY: add the gateway, READ THE CONFIG BACK, verify WAN_GW is still the default, and only
THEN apply. WAN_GW is now priority 1 (explicitly preferred); OFFICE1_LXD_GW is 255. Verified after
apply: default still 172.30.1.1, egress OK, and the edge now routes 10.10.1.0/24 -> 10.10.0.20 and
reaches NetBox. The lesson is in the as-built doc's trap list.
Also re-learned the hard way: root's shell on the edge is tcsh, so `2>&1` in my verification command
died with "Ambiguous output redirect" -- the exact trap already written down in this repo's standing
lessons. Feed remote commands to `sh -s`.
TAILSCALE: office1-tailscale composed by MAAS (2 cores / 2 GiB / 25 GB -- VR0's sizing), deployed
Ubuntu 24.04, Tailscale 1.98.8 installed, IP forwarding on, brought up as a SUBNET ROUTER advertising
10.10.0.0/22 -- exactly the D-115 Office1 carve, so "Office1 only" (D-107) is enforced by the carve
itself. Authenticated via the login-URL flow, NOT an auth key: no tailnet secret passes through the
session. PENDING OPERATOR: authorize the node, and approve the 10.10.0.0/22 route in the admin
console. The tailnet ACL/tag policy remains operator design (runbook open question 5).
NEW: docs/vr1-office1-as-built.md -- the operator-requested connection reference. Every value
MEASURED, not planned: networks + who owns DHCP on each, hosts/services/levels, how to reach each
thing from vcloud AND from a workstation, where every credential lives, and the site's standing
traps. Explicitly NOT an address authority -- NetBox is the IPAM apex; this is operational.
Also noted: MAAS's bind9 intermittently SERVFAILs a first-time external lookup and succeeds on
retry (seen twice -- docker registry, tailscale.com). Retry before diagnosing.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
