| 2026-07-14 |
D-117 ADOPTED (B, DC-only): fix the off-by-one site binding + make the importer dry-by-default
...
Operator ruling: rename the DC numbers to match the NetBox apex; the office
keeps its number (Office1 -> new site vr1-off1, so nothing deployed changes).
netbox/dc-dc-prefixes-import.py:
--dc dc1|dc2 -> --dc dc0|dc1, bound to the APEX slugs measured live:
dc0 -> vr1-dc0 (2602:f3e2:f02::/48) dc1 -> vr1-dc1 (f03::/48)
It previously mapped dc1 -> vr1-dc1 while treating f02::/48 as dc1, i.e. it
would have written VR1 DC0's prefixes to the OTHER DC's site.
Now DRY BY DEFAULT with --commit (it used to write with no flag at all).
A dry run against a FRESH NetBox now plans the site too (_PlannedSite) rather
than bailing with 'no site id' -- previewing an empty apex is the whole point.
DC2_V4_SUPERNET is REJECTED BY NAME (retired) rather than silently ignored.
Every 'DCn' in prose is region-qualified: 'DC0' meant VR0's rehearsal DC in
some sentences and VR1's first DC in others. No sed could have caught that.
tests/dc-dc-prefixes-import/: 73 checks ALL PASS, incl. regression guards that
the dry run creates NOTHING, that --dc dc0 never touches vr1-dc1, and that the
invented vr1-dc2 site is never created.
FOUND, RECORDED, NOT YET FIXED (D-117 amendment): scripts/lib-net.sh carries a
SECOND, colliding $DC namespace where dc0 already means VR0's DC0. Renaming
VR1's dc1->dc0 there would collide with VR0. Proposed fix: region-qualify the
shell selector to the apex slugs (vr0-dc0/vr1-dc0/vr1-dc1). Does not block the
import; the two selectors are NOT interchangeable meanwhile.
repo-lint 0 fail. run-tests-all: ALL GREEN (53 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-117 (PROPOSED): VR1 site naming collision blocks the NetBox import
...
Measured live against the apex: prod NetBox binds 2602:f3e2:f02::/48 -> site
VR1 DC0, but dc-dc-prefixes-import.py maps --dc dc1 -> slug vr1-dc1 and treats
f02::/48 as dc1. The addressing agrees; the labels are off by one. Because that
importer has NO --commit (it WRITES BY DEFAULT), running it as documented against
a prod-seeded NetBox would silently bind DC1's prefixes to the other DC's site.
This is the known G5 gap, promoted to a decision because it blocks the import.
Options A/B/C recorded; A (rename the two skeletal apex sites) recommended, with
the slug-collision rename ORDER called out as a trap.
Also records the upstream-NetBox User-Agent WAF trap in platform-traps.md: the
same token that works from curl gets 403 from urllib/requests/pynetbox. It will
bite pynetbox and looks exactly like an auth failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-13 |
D-116 ADOPTED: no Office1-local GitBucket -- keep using git.baldurkeep.com
...
Operator ruling: "we've decided not to build a separate one on the test cloud. We will continue to
use the current setup and create new repos for deployment tasks as needed."
SUPERSEDES the GitBucket half of D-103 (which gave Office1 three service VMs) and removes GitBucket
from Stage 2's build list entirely. Office1's MAAS-composed service machines are now exactly TWO --
office1-netbox and office1-tailscale -- and both are LIVE.
WHAT THIS ACTUALLY CLOSES, and why it is a simplification rather than a deferral: the Stage 2 runbook
carried an OPEN QUESTION about the Office1-local GitBucket -- "what it mirrors, and under what repo
path, is not decided" -- because a SECOND GitBucket alongside the pre-existing git.baldurkeep.com
raised a real authority/mirroring question nobody had answered. That question is now MOOT. There is
one git service, it is the existing one, and it is authoritative.
Recorded explicitly in D-116 so nobody misreads the scope: D-107's per-DC ARTIFACT mirror (OS/package
artifacts for node provisioning) is a SEPARATE concern and is NOT affected. And if Roosevelt later
wants a site-local git service, that is a fresh decision against real requirements -- not something
this ruling forecloses.
Applied everywhere it was an instruction, not just recorded:
- runbooks/dc-dc-phase1-office1-standup.md: Step 11 replaced with a TOMBSTONE (kept rather than
deleted -- step numbering is referenced elsewhere, and someone WILL come looking after reading
D-103); removed from the sequence, the compose steps, and the open-questions list.
- docs/dc-dc-deployment-workflow.md: dropped from Stage 2's Build and, importantly, from its GATE --
"GitBucket serving" is no longer a gate condition.
- docs/design-decisions.md: D-103 annotated in place with the supersession, so grepping for GitBucket
lands on the amendment rather than the stale instruction.
- docs/vr1-office1-as-built.md + session ledger updated.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Tailscale: office1-tailscale joined the SELF-HOSTED tailnet as the Office1 subnet router
...
Operator's control plane is self-hosted (https://tailscale.baldurkeep.com:443), NOT Tailscale's
public one -- so the public login-URL flow from the previous commit was void. Node reset off
controlplane.tailscale.com and re-enrolled against the operator's server with their auth key.
REACHABILITY PROVEN FIRST, not assumed: DNS + TCP 443 + HTTPS to tailscale.baldurkeep.com confirmed
from the node BEFORE running `tailscale up`, so a failure would have been a network fact rather than
a confusing registration error.
AS BROUGHT UP: --login-server=https://tailscale.baldurkeep.com:443 --authkey=<from 0600 file>
--advertise-routes=10.10.0.0/22 --accept-routes --accept-dns=false --hostname=office1-tailscale
Online as 100.64.0.53 (fd7a:115c:a1e0::35), no health warnings, advertising 10.10.0.0/22 -- exactly
the D-115 Office1 carve, so D-107's "Office1 ONLY" front door is enforced by the CARVE itself rather
than by policy alone.
ONE DELIBERATE DEVIATION from the operator's usual command: --accept-dns=false. This is a
MAAS-managed machine and MAAS's bind9 serves its internal names; letting the tailnet take over DNS
would break that resolution. Flagged rather than silently applied -- trivial to flip if MagicDNS is
wanted here.
OPERATOR ACTION STILL REQUIRED: the 10.10.0.0/22 route is ADVERTISED but must be ENABLED on the
control server (Headscale: `headscale routes enable`) before other tailnet nodes route to Office1.
Once enabled, the whole Office1 carve -- edge GUI, MAAS UI, NetBox -- is reachable from the tailnet
with NO SSH tunnels, superseding most of the tunnel commands in the as-built doc.
SECRET HANDLING: the auth key was written by the operator to a 0600 file OUTSIDE this session,
consumed BY PATH, never printed into context, and the copy shipped to the node was SHREDDED after
enrolment (tailscaled holds its own node key now). Tracked as SEC-008 -- a rotation obligation, not
a delete-me, since the key remains on vcloud for re-enrolment.
docs/vr1-office1-as-built.md updated with the tailnet addresses, the exact bring-up command, the
DNS deviation, and the pending route-enable step.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1: edge route to the compose net + Tailscale subnet router + an as-built connection reference
...
THE ROUTE -- and an OUTAGE I CAUSED AND REVERTED. Adding a LAN gateway on OPNsense for voffice1
SILENTLY STOLE THE DEFAULT ROUTE: the edge's default flipped from 172.30.1.1 (ISP) to 10.10.0.20,
and the site lost egress. Root cause: OPNsense's `defaultgw` field in the API output is DERIVED
FROM `priority` (lower = more preferred), NOT from the value you send -- I sent defaultgw="0", it
came back True, and I had ALREADY run `reconfigure` before reading anything back. Detected within
~2 minutes by checking egress, reverted by deleting the gateway; default route and egress restored.
Redone SAFELY: add the gateway, READ THE CONFIG BACK, verify WAN_GW is still the default, and only
THEN apply. WAN_GW is now priority 1 (explicitly preferred); OFFICE1_LXD_GW is 255. Verified after
apply: default still 172.30.1.1, egress OK, and the edge now routes 10.10.1.0/24 -> 10.10.0.20 and
reaches NetBox. The lesson is in the as-built doc's trap list.
Also re-learned the hard way: root's shell on the edge is tcsh, so `2>&1` in my verification command
died with "Ambiguous output redirect" -- the exact trap already written down in this repo's standing
lessons. Feed remote commands to `sh -s`.
TAILSCALE: office1-tailscale composed by MAAS (2 cores / 2 GiB / 25 GB -- VR0's sizing), deployed
Ubuntu 24.04, Tailscale 1.98.8 installed, IP forwarding on, brought up as a SUBNET ROUTER advertising
10.10.0.0/22 -- exactly the D-115 Office1 carve, so "Office1 only" (D-107) is enforced by the carve
itself. Authenticated via the login-URL flow, NOT an auth key: no tailnet secret passes through the
session. PENDING OPERATOR: authorize the node, and approve the 10.10.0.0/22 route in the admin
console. The tailnet ACL/tag policy remains operator design (runbook open question 5).
NEW: docs/vr1-office1-as-built.md -- the operator-requested connection reference. Every value
MEASURED, not planned: networks + who owns DHCP on each, hosts/services/levels, how to reach each
thing from vcloud AND from a workstation, where every credential lives, and the site's standing
traps. Explicitly NOT an address authority -- NetBox is the IPAM apex; this is operational.
Also noted: MAAS's bind9 intermittently SERVFAILs a first-time external lookup and succeeds on
retry (seen twice -- docker registry, tailscale.com). Retry before diagnosing.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
office1-netbox DEPLOYED: NetBox 4.6.4 live on the Office1 headend, three levels deep
...
MAAS composed the machine, PXE-booted it, commissioned it, and DEPLOYED Ubuntu 24.04.4 onto it --
an LXD VM inside voffice1, inside vcloud, which is itself a KVM guest. NetBox 4.6.4 (Django 6.0.6)
runs on it and its API returns HTTP 200. The D-114 chain is complete in its final form.
FOUR FINDINGS. TWO ARE NOW GUARDED IN CODE.
1. MAAS-DISCOVERED SUBNETS HAVE NO GATEWAY (FIXED). MAAS discovers the compose subnet from lxdbr0's
interface, and that discovery carries no gateway_ip. The first deploy produced a machine that
LOOKED healthy -- it had an address and DNS even resolved (MAAS's bind9 is link-local) -- but had
NO default route and NO egress. It only surfaces when the first apt install hangs.
scripts/site-headend-install.sh now sets gateway_ip on the compose subnet; the harness asserts it
(19/19). office1-netbox was released and REDEPLOYED to prove the fix rather than hand-patching a
route onto the running box.
2. NETBOX 4.6 v2 TOKENS: THE WIRE FORMAT IS `nbt_<key>.<plaintext>`, NOT THE API's `token` FIELD.
NetBox 4.6 hashes tokens: `key` is a 12-char PREFIX, plus `plaintext` and an hmac_digest.
Authentication infers the version FROM A PREFIX (TOKEN_PREFIX = 'nbt_') and splits on '.'. So the
57-char assembled form is what authenticates. POST /api/users/tokens/provision/ returns `key` and
`token` as SEPARATE fields, and the `token` field alone is NOT usable -- present it raw and NetBox
parses it as a legacy v1 token and returns 403 "Invalid v1 token", an error that names v1 while
you are holding a v2 token. Anything scripted against this NetBox (including
netbox/dc-dc-prefixes-import.py and any pynetbox client) must ASSEMBLE the token.
Also: netbox-docker's SUPERUSER_API_TOKEN env is NOT honored on this version, and hand-building a
Token row trips the DB's enforce_version_dependent_fields constraint. Call the interface; do not
re-implement it -- the same lesson this repo already learned minting OPNsense API keys, which I
proceeded to re-learn the hard way.
3. A WRONG HYPOTHESIS, RECORDED BECAUSE THE MISTAKE IS REUSABLE. When the token kept 403-ing I
concluded that rotating SECRET_KEY after NetBox had initialised must have broken the token HMAC
pepper, and I WIPED THE DATABASE (docker compose down -v) to re-init. That was WRONG -- the 403
was purely the wire format above. The wipe was harmless because the DB was empty, but the
reasoning was not: on a populated NetBox it would have been destructive. "Invalid v1 token" is a
FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a
destructive reset.
4. netbox-docker ships a PUBLIC default SECRET_KEY (it is in the public repo) -- session-forgery
material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the
shipped default, so no default admin/admin exists.)
All secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under
/root/netbox-secrets/, never printed into the session.
REACHABILITY CAVEAT, not solved: NetBox is on 10.10.1.201, behind voffice1's NAT. Reachable from
voffice1, NOT from office1-local or a workstation. Making it reachable needs a static route for
10.10.1.0/24 via 10.10.0.20 on the OPNsense edge -- a live edge change, deliberately NOT done
unilaterally.
NEXT: import the D-115 carve into this NetBox.
repo-lint 0 fail. tests/site-headend-install 19/19 PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger: pin Office1 LAN IPv6 as the next step after the NetBox deploy (D-115 q3)
...
Operator sequenced it: NetBox first, then dual-stack. Pinned with its known limit attached so the
next session cannot mistake internal v6 for working v6 -- IPv6 does not egress the lab (measured),
so the LAN leg proves addressing/RA/services-binding and nothing about internet GUA reachability.
The WAN v6 leg is DEFERRED: it would be a path to nowhere and untestable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-115 ADOPTED: v4 stays role-based; Office = /22 per office; DC2 moves INSIDE the Cloud /16
...
Operator ruled all three questions:
1. v4 stays ROLE-BASED, fill the missing roles (v6 keeps its region hierarchy; v4 is pooled by
function -- v6 is abundant, v4 is scarce). No DC renumber; D-101's inherit-DC0-unchanged
economy is preserved.
2. /22 per office. VR1 Office1 = 10.10.0.0/22 -- which BLESSES the deployed LAN (10.10.0.0/24) and
LXD compose net (10.10.1.0/24) exactly. ZERO deployment renumber.
3. Register AND deploy dual-stack on Office1 now.
THE CARVE. New v4 roles: Office (10.10.0.0/16, /22 per office) and Edge (172.30.0.0/16, mirroring
v6's Edge Networks fe::/48) so the simulated ISP uplinks stop living inside the Corp OOB block --
office1-wan's 172.30.1.0/24 becomes legitimate. DC2 moves from 10.13.0.0/19 to 10.12.64.0/19,
INSIDE the Cloud /16: free (DC1 holds 10.12.{4,8,12,16,32,36}), still routable to DC1 over the
dark fiber. docs/dc-dc-netbox-buildout-scope.md is corrected -- its 10.13.0.0/19 recommendation sat
outside every allocated block and would have had DC2 squatting too.
v6 is determined by the existing pattern, no design freedom: Office1 = 2602:f3e2:f01 :/56 with
a /64 office subnet, plus a "VR1 Off1" site. (VR0 Off0 = e01:/56 -> /64; Eugene's Charnelton =
101:/56. VR1's office /48 already existed and we were simply not using it.)
MEASURED CONSTRAINT, attached before anyone builds on a false premise: IPv6 DOES NOT EGRESS THE LAB.
vcloud has global v6 and a v6 default route, and the v6 gateway IS reachable (RA and L2 work) -- but
v6 INTERNET is unreachable (2606:4700:4700::1111 and 2001:4860:4860::8888 both 100% loss; v4 from the
same host is fine). So deploying dual-stack on Office1 proves v6 ADDRESSING/RA/services-binding-v6 --
most of D-101's family matrix, worth doing -- but it CANNOT prove v6 internet reachability. Public
GUA reachability (v6 public API endpoints, tenant GUA egress) is UNPROVABLE IN VR1 until the lab's
upstream carries v6. That is outside this cloud and this repo. Do not schedule a VR1 test that
depends on it, and do not read a green internal-v6 result as "v6 works end to end".
The NetBox WRITE is deliberately NOT done unilaterally: netbox.baldurkeep.com is production, the
standing architecture is read=source / write=feedback-only, and the Office1 sandbox NetBox (composed,
Ready, not yet deployed) is where this carve gets simulated first.
repo-lint 0 fail -- and its L5 collision guard did its job, catching that my amendment header read
as a second DEFINITION of D-115. Renamed to the AMENDMENT form the guard exempts rather than
weakening the rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-115 PROPOSED: Office1 addressing -- the office carve, and the four v4 role gaps it exposed
...
Operator challenge during deployment testing ("I thought there were corp office subnet carves in
the planning") -- and they were right. I had checked for COLLISIONS but never asked the IPAM apex
whether Office1 had an ASSIGNED carve. It does, in v6, and we were not using it.
MEASURED against the live NetBox (read-only), not the repo -- the repo's planning docs cover only
the DC planes, so grepping them would never have found this:
IPv6 is a complete region-hierarchical model. Every region gets a /40; sites are /48s on a FIXED
slot pattern: X00=infrastructure, X01=OFFICES, X02/X03=datacenters. VR1's office /48 ALREADY EXISTS
(2602:f3e2:f01::/48). Offices take one /56 each, then a /64 subnet -- VR0 Off0 = e01:/56 ->
e01:/64; Eugene's Charnelton = 101:/56. So Office1's v6 has NO design freedom: it is
2602:f3e2:f01:/56 + a /64. Office1 currently has NO IPv6 AT ALL -- D-101's dual-stack family
matrix silently skipped the office.
FOUR v4 GAPS, none previously known:
1. There is NO Office role in v4. Office1's LAN (10.10.0.0/24) and LXD compose net (10.10.1.0/24)
are squat -- colliding with nothing, blessed by nothing.
2. DC2's planned 10.13.0.0/19 is OUTSIDE the Cloud /16 (10.12.0.0/16). The scope doc recommends it;
NetBox never allocated it. DC2 was already set to squat and nobody had noticed.
3. office1-wan (172.30.1.0/24) squats INSIDE 172.16.0.0/12 "Corp OOB Private" -- and it is not OOB,
it is a simulated ISP uplink. A wrong-ROLE problem, not just an unregistered one.
4. "Edge Networks" has a v6 /48 (2602:f3e2:fe::/48) but NO v4 counterpart, so the ISP-sim WAN
segments have nowhere legitimate to live in v4.
THE TENSION TO RULE ON: v6 here is region-hierarchical, v4 is role-based. That asymmetry is
defensible -- v6 is abundant so it gets geography, v4 is scarce so it gets pooled by function.
Mirroring v6's region tree in v4 would force RENUMBERING DC1 off 10.12.x, breaking exactly the
inherit-DC0-unchanged economy D-101 bought to reuse VR0's bundle. Large cost for tidiness.
Recommended option (a): keep v4 role-based and fill the MISSING roles -- an Office /16 carved /22
per office (VR1 Office1 = 10.10.0.0/22, which BLESSES the deployed LAN + compose net exactly, zero
renumber), an Edge role (172.30.0.0/16, mirroring v6's fe::/48) so the ISP-sim WANs stop living in
the OOB block, and DC2 pulled INSIDE the Cloud /16 at 10.12.64.0/19 (free; DC1<->DC2 stay routable).
Option (b) mirrors v6 and renumbers the DCs. Option (c) ratifies the squat verbatim -- rejected.
PROPOSED, not adopted: presenting options, not picking. Nothing is blocked -- Office1 is live and
routing. But the renumber cost is LOWEST RIGHT NOW, before NetBox and GitBucket deploy onto it.
repo-lint 0 fail. No live system touched (NetBox reads were read-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END.
...
MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.
vcloud (L1 -- itself a KVM guest, measured)
+- voffice1 (L2) MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
+- office1-netbox (L3) LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected
THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.
THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
10.10.0.0/24 (site LAN) -> Kea on the OPNsense edge MAAS dhcp_on=False
10.10.1.0/24 (lxdbr0) -> MAAS MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.
NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.
FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
be picked by accident.
DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.
10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.
Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.
Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114 follow-through: fix node-vm's nested-virt defect NOW; voffice1 pinned to a reserved address
...
Operator ruling: fixes land as we find them -- only DC1 DEPLOYMENT is gated, not DC1 fixes.
1. modules/node-vm: UNCONDITIONAL `cpu = { mode = "host-passthrough" }`. Same defect just found in
cloudinit-vm: with no cpu block libvirt hands the guest a generic emulated CPU with NO svm. A DC
node runs nova-compute, so it MUST run KVM guests -- there is no correct value other than this,
hence no knob that could only ever be set wrong (contrast cloudinit-vm, where it IS a real per-VM
decision, and opnsense-edge, which disables svm as correct hardening for a router). Not yet
instantiated, so no live call breaks. Without it the DC nodes would have come up looking healthy
and been unable to launch a single instance -- surfacing in Stage 5 as an inscrutable scheduler error.
2. voffice1 moved off the dynamic pool onto a RESERVED address. It first took 10.10.0.100 -- the
first address of Kea's pool (.100-.199). A MAAS region controller must not hold an address that
can move. Added a Kea host reservation via the D-113(a2) REST API (a second live exercise of that
write path): 52:54:00:6a:87:e5 -> 10.10.0.20, deliberately OUTSIDE the pool. add_reservation ->
"saved", service/reconfigure -> "ok". Rebooted; voffice1 came back on 10.10.0.20, .100 released,
/dev/kvm still present. Done now because nothing depends on the address yet.
3. runbooks/dc-dc-phase1-office1-standup.md rewritten to the D-114 model (was: three sibling service
VMs + an unruled Option A/B fork -- it would have actively misdirected whoever ran it). New
sequence: voffice1 via tofu -> first contact (lease/SSH/nested-KVM) -> MAAS on voffice1 -> LXD
(5.21 LTS) -> register LXD as a MAAS VM host -> MAAS-COMPOSE NetBox/GitBucket/Tailscale. The
MAAS/LXD command lines carry explicit PENDING VERIFICATION markers rather than fabricated flags.
docs/dc-dc-deployment-workflow.md Stage 2 + gap register updated to match.
repo-lint 0 fail. opentofu-validate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114 ADOPTED: VR1 site containment VMs + MAAS-composed LXD VMs for the non-stack machines
...
Operator ruling. AMENDS D-103 (which gave Office1 three sibling service VMs on vcloud).
WHAT THE OPERATOR'S SCREENSHOTS PROVED -- and where I was wrong. I had grepped bundle.yaml,
found only Juju's `lxd:N` CONTAINER placements, and concluded "there is no VR0 precedent for
MAAS composing LXD VMs -- zero." FLATLY WRONG. VR0's MAAS (3.7.2) has an `lxd` machine
registered back into MAAS as an LXD KVM host (LXD 5.21.4), and `tailscale` is a VM COMPOSED
BY MAAS INSIDE IT. The bundle's Juju LXD containers and MAAS's LXD VM host are TWO DIFFERENT
LXD usages in the same cloud; reading only the bundle collapses them into one. Recorded in the
decision because the error is instructive.
The ruling:
1. Non-stack machines (NetBox, GitBucket, Tailscale) become MAAS-COMPOSED LXD VMs -- MAAS-visible,
enlisted/commissioned/deployed/powered. This is VR0's proven `lxd` + `tailscale` pattern per
site, at ~2 cores / 2 GiB each. The Juju-deployed OpenStack services stay in Juju-created LXD
CONTAINERS, invisible to MAAS, exactly as in VR0 -- per the operator, that visibility is not wanted.
2. Each site gets a containment VM (voffice1/vdc1/vdc2). The decisive rationale is the operator's:
full-facility snapshots, and simulating total geographical failure of a facility. D-108's DR
drill currently has NO honest "hard-down a DC" primitive -- `virsh destroy vdc1` IS that primitive.
The flat model cannot offer this at any price.
3. Office1 FIRST, fully up; DC1 GATED behind it.
4. LXD pinned to the 5.21 LTS track: MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (endpoint
consolidation vs pinned pylxd 2.3.5). VR0 is on 5.21.4 -- safe by timing, now safe by decision.
KNOWN RISK: vcloud is itself a KVM guest (measured), so a containment VM puts servers one level
deeper than VR0 runs them. Office1 IS the nesting probe -- LXD VMs are qemu/KVM guests, so it
needs nested KVM at the same L3 depth the DCs would (an earlier draft of the entry said Office1
needed none; corrected in-entry). Office1 is simply the cheap place to prove it: no OpenStack,
no Ceph. DC1's gate additionally proves L4, the depth VR0 has never exercised.
Bonus: voffice1 booting on office1-local takes the FIRST REAL DHCP LEASE from Kea -- a path
proven only at the daemon level, never end to end.
repo-lint 0 fail. No live system touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fold the AppArmor/libvirt rule into scripts/prereqs -- it gated every VR1 VM boot
...
`/var/lib/libvirt/vr1/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu had
been applied BY HAND on vcloud and existed ONLY as host state. Nothing in the repo
carried it, so a rebuild or a second host walked straight back into the same wall.
Why it matters: libvirt's stock abstractions/libvirt-qemu grants qemu only the DEFAULT
pool path (/var/lib/libvirt/images). VR1 uses a custom pool parent, which is not in that
abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The domain
DEFINES fine, then fails to start with a bare qemu "Permission denied" -- and NOTHING in
the libvirt error names AppArmor. It cost a session on 2026-07-12 (DOCFIX-186), was
logged as a PREREQ candidate, and was never actioned until the ledger collapse resurfaced it.
New: scripts/prereqs/install-apparmor-libvirt.sh, wired into install-all.sh (after
install-virtualization -- it edits libvirt's own profile dir) and reported by
check-prereqs.sh.
- Idempotent and deliberately QUIET on a good host: if the rule is present it exits 0
having touched nothing -- it does NOT reload apparmor or bounce libvirtd. Those happen
only on the run that actually adds the rule.
- Writes to local/, the vendor-sanctioned override include, so it survives package upgrades.
- Pool parent is a variable (VR1_POOL_PARENT, default /var/lib/libvirt/vr1), not a literal.
- Non-AppArmor hosts report N/A and SUCCEED -- no manufactured failure on a host it
does not apply to.
Verification: tests/prereqs 32/32 PASS (was 24/24) -- the new cases drive DETECTION off a
pool parent guaranteed absent from the profile, so the negative path is deterministic on
any host: a false OK from --check IS the bug, --dry-run must plan and mutate nothing, and
the planned rule must be exactly `<parent>/** rwk,`. Full gauntlet ALL GREEN (52 harnesses),
repo-lint 0 fail. Run live on vcloud: --check reports OK and leaves the profile byte-identical,
confirming the already-satisfied path is a true no-op.
No host state was mutated by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger collapse: close the coverage gap in the carry-forward
...
The collapse commit's guards (lint, ASCII, ledger-scan, size) validated FORM, not
COVERAGE. Two ledger regions had only been keyword-grepped, not read. Read them.
Three items surfaced; each was VERIFIED against the repo rather than trusted:
- run-tests-all.sh dirtying the tree with 37 spurious mode-change diffs: NOW FIXED
(the fakebin stubs are committed 100755 -- 36 of them, measured). Not carried.
- `scripts/opnsense-apply-config.sh` (once "wanted"): SUPERSEDED, and dangerous to
build -- its snapshot/scp/install/reboot flow IS the config.xml path D-113(a2)
deleted. Recorded as "do not build this", so nobody resurrects it from the history.
- The tcsh trap it taught is REAL and carried forward: root's shell on the OPNsense
edge is tcsh, so `$(...)` in a quoted remote command dies with "Illegal variable
name" -- QUIETLY. It already cost a silently no-op'd pre-install snapshot. Always
feed remote commands to `sh -s`.
Also carried (this one matters and was nowhere in the open lists): DHCP HAS NEVER
SERVED A REAL LEASE. kea-dhcp4 is bound to udp/67 with the subnet+pool intact, but
office1-local has no client host, so the DAEMON is proven and the SERVICE is not.
The first VM on office1-local (the Stage 2 headend VMs) is the real test -- logged
as that stage's gate rather than an assumption.
Plus a measured doc-accuracy finding: SEC-007 says "D-112(c) makes SSH the only
management path", but the edge's web GUI IS listening on the LAN (10.10.0.1:443,
HTTPS 200) and is correctly firewalled off on the WAN. Not an exposure -- OPNsense
default, and D-113 kept OPNsense *because* the GUI is an operator requirement -- but
the wording is imprecise. Logged, not actioned (hard rule 1).
Guards: repo-lint 0 fail (1 documented legacy warn), 0 non-ASCII. No live system
touched; the edge probes were read-only TCP/HTTP reachability checks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Collapse the three ledger streams into one; reconcile against repo ground truth
...
The ledger carried three separately-owned parallel stream sections (main-chat,
jumphost, shared) plus ~30 append-only session narratives -- 2189 lines, several
of them contradicting each other and the repo. Collapsed to ONE stream, 222 lines,
carrying only what is still OPEN plus the facts that live nowhere else.
The narrative is NOT lost: it is in this commit's parent (f81177a) and, durably,
in the 65 docs/changelog-*.md files, design-decisions.md, and the incident reports.
RECONCILED (repo ground truth wins over any stream's self-claim; all verified):
- machine-derived block was STALE: said D=111/DOCFIX=162/BUNDLEFIX=012 and listed
only SEC-001/003/004. Re-seeded from today's scan: D=114/DOCFIX=195/BUNDLEFIX=013,
six open SEC rows (001/003/004/005/006/007).
- main-chat called backlog items 6/7/8 "not started". FALSE -- keystone-policy-drift.sh,
cloud-snapshot.sh and tests/tenant-acceptance/ all exist. Delivered by the jumphost
stream; this was the cross-stream reconcile the ledger itself had pending.
- item 3 remainder (fold vault-kv-health into cloud-assert): VERIFIED still open.
- item 5 (tenant-cluster-create.sh): VERIFIED still absent.
- R-3/D-063: ledger claimed CLOSED, handoff doc still says PROPOSED/OPEN. D-063 is
RESOLVED/CLOSED in design-decisions.md -- so the HANDOFF DOC is the stale one.
Carried forward as an open item rather than fixed here (hard rule 1).
- D-076: not this repo's to work. D-076..D-099 is a RESERVED source-project band
(D-110), so its "uncommitted worktree draft" was never here. No work lost.
- "beta cluster left at node_count=2" was STALE -- the 2026-07-08 audit found no beta
cluster and no foil1 tenant. Retired.
CARRIED FORWARD (would have been operationally lost -- the skill loads this file):
the API-managed-DHCP edge trap; "updated in-place" still BOUNCES the guest; do not
re-implement OPNsense's key crypto, call the interface; config ISO can never be read
on a nano image; guide<->skill coupling (repo-lint L8); controller model is
admin/controller; State facts; Project-completion.
NEW, measured this session: the apparmor rule /var/lib/libvirt/vr1/** GATES EVERY VR1
VM BOOT but exists only as host state -- it is NOT in scripts/prereqs/, so a rebuild
hits the identical foundational failure. Logged as an open item.
Guards: repo-lint 0 fail (1 documented legacy warn), 0 non-ASCII, ledger-scan
reconciles against the re-seeded machine block. No live system touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
APPLIED + CORRECTION: "updated in-place" does NOT mean "no restart" -- the apply bounced the edge
...
The config_seed/cdrom removal is APPLIED to the live Office1 edge. Apply complete: 0 added,
1 changed, 1 destroyed. `tofu plan` now reports "No changes" -- repo and state are back in sync,
so Stage 3 cannot trip over the divergence.
CORRECTION, recorded because the prediction was WRONG and the mistake is reusable:
I predicted the apply would NOT interrupt the guest, on the grounds that the plan said
"libvirt_domain.vm will be updated IN-PLACE". THE APPLY RESTARTED THE GUEST. Measured: uptime went
from 8:36 to 6 secs across the apply. The dmacvicar/libvirt provider redefines the domain to apply
a disk-list change and BOUNCES it -- a ~30s outage during which the edge was neither routing nor
serving DHCP.
"updated in-place" means the RESOURCE is not replaced. It says NOTHING about whether the GUEST
keeps running. Do not read it as "no interruption" -- that is exactly what I did.
STAGE 3 CONSEQUENCE (the reason this is worth a commit of its own): any future tofu apply that
touches a libvirt_domain's devices -- on a DC edge, or on a node VM -- must be assumed to BOUNCE
THE GUEST, and scheduled/gated as an outage rather than a no-op config tidy-up.
Recovery was clean and needed no intervention. Post-boot, measured: kea-dhcp4 running and bound
udp4 10.10.0.1:67 with subnet 10.10.0.0/24 + pool .100-.199 intact; WAN 172.30.1.2, LAN 10.10.0.1,
default route 172.30.1.1; egress to 1.1.1.1 at 0.0% packet loss; 8 LAN pass rules in pf; serial
console + getty alive (DOCFIX-192 holds). The cdrom is gone from the persistent domain XML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2): API-key bootstrap -- no GUI click, no re-implemented crypto
...
The last unknown in D-113(a2) edge provisioning. The API path was proven (read AND write), but
every new edge still needed a MANUAL GUI CLICK to mint its key -- which would have put a human in
the middle of building DC1's and DC2's edges in Stage 3. Closed.
scripts/opnsense-mint-apikey.php runs ON the edge; calls OPNsense's OWN model
(Auth\User -> apikeys->add()) -- the exact code path
the GUI's ticket button invokes
scripts/opnsense-bootstrap-apikey.sh ships it over SSH, runs it, retrieves the key, wipes
the remote copies
tests/opnsense-bootstrap-apikey/ offline harness, 8 PASS (ssh/scp stubbed)
Gauntlet ALL GREEN (54 harnesses, was 53); repo-lint 0 fail.
VERIFIED END TO END on the live Office1 edge: minted a SECOND key via the vendor model ->
GET core/firmware/status returned HTTP 200 (the vendor-minted key AUTHENTICATES) ->
POST auth/user/del_api_key/<id> -> {"result":"deleted"} -> search_api_key shows 1 row remaining
(the operator's original key, untouched throughout) -> the retired key is now REJECTED (a real
negative test, not just an absence of errors). Temp creds shredded from the jumphost. The edge is
back to its prior state.
THE DESIGN DECISION, and why the alternative was rejected: OPNsense stores API secrets as
crypt(secret,'$6$') -- SHA-512 with an EMPTY salt (verified on the live 26.1 box: the stored hash
is "$6$$" + 86 chars). We COULD mint keys offline and seed them into a bootstrap config.xml. We
deliberately DO NOT. That would couple our provisioning to a vendor hash format we would have to
keep byte-compatible forever, and a mismatch FAILS SILENTLY -- keys that simply never
authenticate, with no error at generation time. Calling the vendor's own generator has no format
to keep in sync. Same principle as D-113(a2) itself: call the interface, do not re-implement the
internals -- which is exactly what DOCFIX-191/192/193 cost us. (An offline-hash attempt was
blocked by the permission classifier mid-session; the block was correct and produced this better
design.)
GUARD CLAUSE WORTH KEEPING (harness T5): the script REFUSES to overwrite an existing creds file.
An existing file means a LIVE key on the edge; overwriting the local copy would strand it there
PERMANENTLY, because the secret is hashed on save and can never be read back. That is
unrecoverable credential loss. T5 asserts both the refusal and that the existing file is left
byte-intact. The secret is never printed -- creation is the only moment it exists in cleartext.
WHAT THIS UNBLOCKS: edge provisioning is now automatable with no human in the loop --
boot factory nano -> D-112(c) console bootstrap (SSH + key) -> opnsense-bootstrap-apikey.sh
-> opnsense-api.sh for DHCP/firewall/interfaces.
There is NO config.xml anywhere in that chain.
STILL OPEN: the template reduction itself. And the chain above raises the real question -- the
template may not need REDUCING so much as DELETING, since the console bootstrap plus the key
minter cover first contact. NOT ruled; not done unilaterally. (Note the opnsense-edge module's
config_seed volume is INSTANTIATED -- it is in tfstate -- so removing the ISO wiring is a LIVE
change, not a docs change.)
Revert: git rm the two scripts + tests/opnsense-bootstrap-apikey/ (nothing live depends on them --
the live edge's key was minted via the GUI and is unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
Correct the workflow doc's status: it was materially FALSE and would misdirect work
...
docs/dc-dc-deployment-workflow.md is the first thing a fresh session reads. Its status claims
dated from 2026-07-09 and had drifted into being wrong. Corrected against MEASURED state.
Docs only; no code, no live system touched.
gap #2 "UNVALIDATED -- no tofu binary" -> OpenTofu v1.12.3 IS installed; tree validates
(root + 10/10 modules) and is partly APPLIED
(11 resources in state). But 2 modules were
BROKEN and had never been parsed (DOCFIX-194).
gap #17 "no per-site ISP-uplink network -> CLOSED FOR OFFICE1: office1-wan is live
exists anywhere, for ANY site" (virbr11, NAT, 172.30.1.0/24); the edge's WAN
sits on it at 172.30.1.2, egress verified. The
gap's own proposed answer (a dedicated per-site
uplink, NOT a mesh leg) is what got built.
DC1/DC2 STILL have none.
opnsense-edge "BUILT, UNVALIDATED" -> BUILT, INSTANTIATED, RUNNING (routing, NAT,
egress, console, SSH, Kea DHCP on udp/67).
config.xml tmpl "BUILT, fully tested" -> SUPERSEDED by D-113(a2); a full push would now
CLOBBER live API-managed DHCP.
netem-link "BUILT, UNVALIDATED" -> WAS BROKEN; fixed 2026-07-13 (DOCFIX-194).
Stage 2 "NOT YET EXECUTED" -> PARTIALLY EXECUTED.
THE TRAP THIS REMOVES: Stage 2's gate lists "NetBox authoritative" and "GitBucket serving" -- and
NetBox and GitBucket DO answer, at baldurkeep.com. But those are PRE-EXISTING services, NOT
Office1 headend VMs. Measured: office1-opnsense is the ONLY VM on the vcloud host. It would be
very easy to tick that gate as met and walk into Stage 3 with no headend at all. The corrected
State section now says this explicitly.
What actually remains of Stage 2 (the real critical path): the three headend service VMs
(MAAS-region, NetBox, GitBucket) do not exist. Blocked on a genuine decision, not on typing:
Option A (modules/cloudinit-vm) needs user_data/meta_data/network_config designed and an image
source chosen; Option B is manual virt-install, logged as debt. Both cloudinit-vm and base-image
now at least VALIDATE (DOCFIX-194) -- they never had before.
Revert: git revert this commit (restores the stale, false status claims).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SAFETY SWEEP: mark the config.xml push path as a LIVE HAZARD (no behaviour change)
...
After D-113(a2) was proven (Office1's config is now API-managed), every instruction in the repo
that still said "render a config.xml and push it to the edge" became ACTIVELY DANGEROUS: a full
config.xml push replaces /conf/config.xml wholesale and drops ~667 migration-populated elements
(measured), including the only 2 firewall pass rules on the box. Following the old runbook steps
against the live edge would clobber it.
Warnings and headers only. No behaviour changes, no live system touched.
Marked (5 files):
runbooks/dc-dc-phase1-office1-standup.md DANGER banner; the config.xml render + config-ISO
sub-steps removed as RUNNABLE instructions (they
would clobber the live edge). Image-prep retained;
history preserved in git + the build changelog.
runbooks/dc-dc-phase2-tofu-dc-substrate.md STOP banner on Step 4.
scripts/opnsense-render-config.sh DANGER header at point-of-use. NOT dead: under
D-113(a2) it is to be reduced to a MINIMAL bootstrap
render. Until then, safe only for a brand-new,
not-yet-booted edge.
scripts/opnsense-build-config-iso.sh RETIRED header quoting the upstream source proving
the importer can never fire on a nano image.
docs/dc-dc-deployment-workflow.md STALE-CONTENT warning atop the tooling gap register.
THE SWEEP ALSO CAUGHT A PRE-EXISTING LANDMINE, unrelated to today's work: dc-dc-phase2 Step 4
still instructs building a CONFIG ISO for DC1's edge -- but D-112 established that ISO can never
be read (opnsense-importer probes for a read-only root; on a pre-installed nano the root is
writable and a factory /conf/config.xml already exists, so it bootstrap_and_exit 0's without
enumerating a single device). That runbook has been telling anyone who follows it to build an
inert artifact and then wonder why the edge came up on factory defaults -- which is EXACTLY the
day lost on 2026-07-12. It was never corrected at the source. Now it is. Its WAN_IF/LAN_IF
"chicken-and-egg" discussion is moot for the same reason: that problem only exists if you try to
seed a full config before first boot, and D-112(c) measures vtnetN AFTER boot.
Changelogs deliberately NOT touched -- they are history, not instructions. Marking history is
noise; marking instructions is safety.
Verification: opnsense-render-config 24 PASS, opnsense-build-config-iso 2 PASS, opnsense-api
21 PASS; repo-lint 0 fail.
STILL OPEN (the actual D-113(a2) migration): reduce the template to a minimal bootstrap, retire
the config-ISO path and the opnsense-edge module's config_seed/cdrom wiring, and rewrite Stage 3's
edge steps around the API. This sweep makes the repo SAFE, not FINISHED.
Revert: git revert this commit (restores the previous, dangerous instructions verbatim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
CHECKPOINT 2026-07-13: session close (edge routing + serving DHCP; D-113(a2) proven)
...
Session-ledger checkpoint so a fresh session can resume cold.
LIVE STATE (measured): office1-opnsense is UP -- routing, NAT, egress 0.0% loss, serial console,
SSH key management, and kea-dhcp4 serving DHCP on udp/67 (pool .100-.199). 8 LAN pass rules in
pf. Nothing broken, nothing mid-flight.
LANDED TODAY:
1. DOCFIX-193 APPLIED -- DHCP live (was built-but-unapplied at last close)
2. SEC-005 -- GitBucket credential stored; unattended git push now WORKS (the branch had sat
12 commits ahead of origin for two days -- a real data-loss exposure, now closed)
3. SEC-006/007 -- the two known credential exposures put ON the ledger (prose only before)
4. D-113 RULED (a2): stay on OPNsense, move config to the REST API
5. D-113(a2) PROVEN END TO END, read AND write: API -> OPNsense -> Kea -> daemon.
scripts/opnsense-api.sh + harness (21 PASS); gauntlet 53 ALL GREEN.
THE ONE TRAP: DHCP on Office1 is now API-MANAGED. A full rendered config.xml push WOULD CLOBBER
IT. Do NOT run the old scp/install/reboot path against this edge until the template is reduced.
NEXT: reduce opnsense-config.xml.tmpl to a minimal bootstrap (sshd + root key + console) and move
DHCP/firewall/interfaces to the API. Repo work; no live mutation needed to land it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) write path PROVEN on the live edge: API -> OPNsense -> Kea -> daemon
...
The read half was proven earlier today; this closes the other half. Configuration written
through the REST API reaches the running daemon. (a2) is now demonstrated end to end.
Executed (live, operator-approved, deliberately IDEMPOTENT -- the payload was the exact values
read back from the API moments before, so the edge's behaviour should not change; the point was
to exercise the WRITE path):
POST kea/dhcpv4/set_subnet/93473635-... -> {"result":"saved"}
POST kea/service/reconfigure -> {"status":"ok"}
GROUND TRUTH after the write (the API's {"result":"saved"} is only the service's own verdict):
kea-dhcp4 pid 30027 (RESTARTED), bound udp4 10.10.0.1:67
Kea's OWN generated /usr/local/etc/kea/kea-dhcp4.conf:
interfaces ['vtnet0'], subnet 10.10.0.0/24, pools ['10.10.0.100-10.10.0.199'],
routers 10.10.0.1, domain-name-servers 10.10.0.1
Router unaffected: WAN 172.30.1.2, LAN 10.10.0.1, default route 172.30.1.1, 8 LAN pass rules in
pf, egress to 1.1.1.1 at 0.0% packet loss.
Payload gotcha, measured -- do NOT infer this: a GET response is not directly re-POSTable.
OPNsense GET returns select fields as {value: {selected: 0|1}}; SET expects the comma-joined
selected values, so the GET must be flattened first. And kea/service/reconfigure is the apply
verb -- without it the config saves but the daemon never reloads.
CONSEQUENCE: the config.xml push path is now RETIRED IN PRACTICE for Office1. DHCP is
API-managed, so pushing a full rendered config.xml WOULD CLOBBER IT. Do not run the old
scp/install/reboot path against this edge. Reducing the template to a minimal bootstrap
(sshd + root key + console) is the next step under D-113(a2) and is NOT yet done.
Revert: re-POST the previous values (unchanged, so nothing to undo), or edit in the GUI at
Services > Kea DHCP > DHCPv4 > Subnets. Pre-API config.xml backups remain in /conf/backup/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) PROVEN: the OPNsense REST API round-trips the edge config
...
The API path adopted in D-113 is now demonstrated working against the LIVE Office1 edge,
closing the question that gated the ruling.
Measured:
GET core/firmware/status -> HTTP 200, OPNsense 26.1 (auth works)
GET kea/dhcpv4/get -> dhcpv4.general.enabled=1, interfaces=lan
POST kea/dhcpv4/search_subnet -> 1 row: subnet 10.10.0.0/24, pools 10.10.0.100-10.10.0.199,
option_data.routers 10.10.0.1, dns 10.10.0.1,
domain office1.vr1.cloud.neumatrix.local, descr "LAN DHCP"
The API reads back EXACTLY what we pushed via config.xml yesterday, and exposes it as TYPED,
NAMED FIELDS -- not free-form XML. That is the (a2) argument made concrete: DOCFIX-191 (missing
sshd), DOCFIX-192 (silenced console) and DOCFIX-193 (an inert ISC <dhcpd> block against a Kea
backend) are NOT EXPRESSIBLE here. You cannot omit a typed field that has a default, and you
cannot address the wrong DHCP backend when the endpoint IS the backend.
GUI gotcha worth recording (26.1): API keys are NOT created in the user-edit dialog, and NOT in
the ApiKeys tab -- that tab wires only search_api_key + del_api_key, so it can list and delete
but CANNOT create. Creation is a ROW ACTION on the Users list: the fa-ticket icon, "Create and
download API key for this user" (source: mvc/app/views/OPNsense/Auth/user.volt). Any instruction
saying "edit user -> API keys -> +" describes the OLD UI and is wrong for 26.1. ApiKeyField::add()
mints key+secret as base64(random_bytes(60)) and stores key|crypt(secret,'$6$') -- the secret is
shown EXACTLY ONCE.
API gotchas: response root key is `dhcpv4`, NOT `dhcp4` (the config XML element name differs from
the API key name -- do not infer one from the other). camelCase actions map to snake_case URLs:
searchSubnetAction -> POST /api/kea/dhcpv4/search_subnet.
SEC-007 amended: ~/vr1-office1-creds/ now also holds the OPNsense API key/secret (0600).
NOT DONE: no WRITE through the API yet -- that is a live mutation and stays gated. And the
config.xml template is still in place: DO NOT push a full rendered config.xml now, it would
clobber API-managed state. Next step is reducing the template to a MINIMAL bootstrap (sshd +
root key + console) and moving DHCP/firewall/interfaces to the API.
Revert: read-only against the edge + docs; nothing to undo. To revoke the credential: GUI ->
System > Access > Users > root row > ApiKeys tab -> delete, then rm ~/vr1-office1-creds/opnsense-api.txt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) step 1: thin OPNsense REST API client + harness
...
First delivery under the D-113 ruling (stay on OPNsense, move config off hand-authored
config.xml and onto the documented REST API).
scripts/opnsense-api.sh -- [--dry-run] <GET|POST> <api-path> [json-body]
tests/opnsense-api/run-tests.sh -- offline harness, 21 PASS, no network, no real key
Gauntlet ALL GREEN (53 harnesses, was 52). repo-lint 0 fail.
Design decisions that the harness now enforces:
- THE SECRET NEVER REACHES ARGV. Credentials are read from a file and handed to curl via
--config on STDIN. `curl -u key:secret` would put the secret in argv, where any user on the
box can read it out of `ps`. Harness T9 stubs curl with an argv recorder and asserts the
secret is absent from argv and present on stdin -- so a future "simplification" to `curl -u`
turns the harness red. That case is the point of the file.
- The API host is NEVER inferred (hard rule 2): $OPNSENSE_API_HOST unset is a loud failure, not
a silent default of 10.10.0.1 (T3).
- --insecure is deliberate and scoped: the edge's self-signed cert is REGENERATED ON EVERY BOOT
(measured -- "Created web GUI TLS certificate" appears in the config revision trail after each
reboot), so pinning it is pointless. Acceptable ONLY on the private lab LAN leg; never to be
copied to a tenant-facing surface.
- Dry-run reads no credentials (T11), which is what lets the harness prove URL construction
offline.
NOT RUN against the edge yet, and blocked on one thing: the API is alive (lighttpd on 443
answers 401) but root has 0 API keys. Minting one is a GUI action (System > Access > Users >
root > API keys), deliberately NOT automated -- writing apikeys into config.xml by hand is the
exact anti-pattern D-113 just retired. Key lands in ~/vr1-office1-creds/opnsense-api.txt,
operator-only, never read into agent context (SEC-007 covers that directory).
Proof-of-path once the key exists: drive yesterday's DHCP subnet through the API and confirm it
round-trips. If it does, the template retires to a minimal bootstrap config (sshd + root key +
console + apikey) and DHCP/firewall/interfaces move to the API. If it cannot express it, that is
the signal to revisit D-113.
Revert: git rm scripts/opnsense-api.sh && git rm -r tests/opnsense-api/ (nothing live touched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113 ADOPTED: option (a2) -- stay on OPNsense, move config to the REST API
...
Operator ruling 2026-07-13. Two constraints surfaced during the discussion narrowed the field
decisively, and both are recorded so the reasoning is not re-run from the wrong premise:
1. GUI ACCESS IS A REQUIREMENT (operator). This eliminates VyOS and plain-Linux outright --
neither has a GUI -- and both had been the leading candidates until it was stated.
2. VyOS LTS binaries are subscription-gated (verified against vyos.net/get and the Software
Access Subscription page). Free VyOS means the ROLLING release, a moving target that fights
this repo's appendix-B version-pinning discipline. Real friction, independent of the GUI.
Field reduced to OPNsense / pfSense CE / OpenWrt / IPFire. pfSense rejected (same GUI-owned XML
lineage -- a migration that buys nothing). OpenWrt was the credible alternative: its UCI config
is text-first BY DESIGN, structurally unlike OPNsense's GUI-owned config.xml. It loses on cost:
a new image pipeline, a fresh bootstrap problem, and dnsmasq instead of Kea -- all to escape a
problem that is fixable in place.
The ruling rests on this: the bug class was never OPNsense. It was hand-authoring config.xml
when OPNsense ships a documented REST API with mature Ansible collections that model exactly
what we got wrong (dhcp_subnet, firewall rules, system settings as typed resources). NONE of
DOCFIX-191/192/193 is expressible through the API -- you cannot forget to enable sshd in a
format where sshd is a typed field with a default. (a2) keeps the GUI, keeps a working
routing+DHCP edge, keeps the libvirt fixes, and removes the defect source.
(a1) config.xml templating is now explicitly REJECTED: its cost compounds per feature and it
depends on an undocumented self-heal of an internal format (the 667-element finding).
Live-edge recon (read-only): lighttpd listening on 443/80, API answers 401 (alive), root has
0 API keys. Minting a key via the GUI is the bootstrap step -- hand-editing apikeys into
config.xml would be the exact anti-pattern being retired.
Revert: git revert this commit (docs only; nothing migrated, nothing built).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113 (PROPOSED): is OPNsense the right edge platform? -- and a confound in D-112
...
Operator asked, after a five-bug day, whether something is better suited than OPNsense for
the simulated edges. Written up with alternatives; NOT RULED.
Honest cost accounting: of the five bugs, TWO (DOCFIX-188 memory_unit, DOCFIX-190 ACPI) were
libvirt module bugs that would have hit any guest and whose fixes now protect every VR1 VM --
not OPNsense's fault. The other three (DOCFIX-191 no sshd/key, DOCFIX-192 no console,
DOCFIX-193 no DHCP) share ONE root cause: hand-authoring the appliance's internal, GUI-owned
config.xml. 2026-07-13 adds a fourth of the same kind -- a full-config push drops ~667
migration-populated elements including the only two firewall pass rules, and works ONLY
because OPNsense regenerates them on boot. We depend on an undocumented self-heal of an
internal format. That, not the boot bugs, is the ongoing cost.
THE CONFOUND, and the reason this is worth ruling rather than waving through: D-112 adopted
option (c) with the explicit rationale "C since it is the way people automate opnsense" --
i.e. the REST API (opn-cli, the Ansible collection). What we actually BUILT is config.xml
templating delivered over scp. That satisfies (c)'s letter (provision over the network, post
boot) but NOT the rationale it was ruled on. Every OPNsense-specific bug we are charging
against the platform is an artifact of hand-authoring XML -- precisely what the API exists to
avoid. So the fair comparison is not OPNsense vs VyOS; it is:
(a1) OPNsense + config.xml templating -- status quo; cost COMPOUNDS per feature
(a2) OPNsense + REST API -- what D-112's rationale actually called for
(b) VyOS -- blank-sheet best; discards a working edge
(c) plain Linux (nftables/Kea/FRR + cloud-init) -- max fidelity to our own tooling
(d) pfSense -- REJECT, same config-format lineage
Recommendation (operator rules): stay on OPNsense, and the real choice is a1 vs a2 -- I lean
a2. The hard part is done and measured working, and edges 2/3 reuse the module, so the cost
paid was one-time. But (a1) compounds, and (a2) is the honest reading of D-112's own ruling.
Rule BEFORE Stage 3: it builds two more edges from this module, so ruling later means
migrating three edges instead of one.
Revert: git revert this commit (docs only; nothing built or migrated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SEC-006/SEC-007: put the two known credential exposures on the ledger
...
Both were flagged only in changelog/session-ledger prose, never as ledger rows -- exactly
the failure mode the security-ledger header warns about ("never only a comment in a script
header, where the libvirt item lived for a week"). An exposure with no row has no owner, no
status, and no revoke trigger.
SEC-006: the NetBox API token pasted into agent chat context during the D-111 IPv6 work.
This one is BURNED, not merely due for rotation -- it exists in a transcript outside our
control, so it needs revoke + reissue. Operator ruling 2026-07-13: DEFERRED, revoke at
completion of this deployment. Recorded as OPEN and explicitly live-and-exposed until then,
so the deferral is visible rather than forgotten.
SEC-007: ~/vr1-office1-creds/ holds the Office1 edge root password, its bcrypt hash, and the
office1_svc SSH private key in plaintext (0700/0600). This is a ROTATION obligation, not a
delete-me: D-112(c) makes SSH the only management path to the edge, so the key must exist.
No change to any live system; documentation only.
Revert: git revert this commit (the rows are the only change).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SEC-005: store a GitBucket credential on the jumphost so pushes stop failing
...
The jumphost clone could not push (could not read Username for https://git.baldurkeep.com),
which left this branch 12 commits ahead of origin for two days -- all the D-112 / OPNsense
edge work existed only on this box. Now fixed: credential.helper store + a dedicated PAT at
~/.git-credentials (mode 600), written by the operator via a hidden prompt. The token was
never read into agent context; the credential is verified by USING it, never by printing it.
Two GitBucket behaviours measured and recorded, because both cost a cycle:
1. The git username is NOT the web-UI login. The operator signs in with an email; that is
rejected for git transport. Measured against info/refs?service=git-receive-pack:
jesse.austin@neumatrix.com -> 401, JANeumatrix (the git user.name) -> 401,
jesse.austin -> 200 CAN PUSH. Note GitBucket 401s PAT basic-auth on /api/v3/ across the
board while accepting it for git -- so validating a credential against the API gives a
FALSE NEGATIVE. Validate against the receive-pack endpoint.
2. credential.helper store DELETES the stored credential when the server rejects it (git
calls credential reject on a 401). The first bad-username push therefore emptied
~/.git-credentials to 0 bytes, and the follow-up diagnostics were authenticating with an
empty string -- which read as a corrupt file and misdirected the investigation. If the
file mysteriously empties: the credential was rejected, it did not fail to write.
Tooling consequence: validate BEFORE writing.
SEC-005 is OPEN by operator ruling: a long-lived, account-wide PAT now sits in plaintext on
the jumphost (GitBucket has no per-repo token scoping, so the control is rotation, not
scope). Revoke/rotate at v1 close, or immediately if the jumphost is shared or rebuilt.
Revert: git config --global --unset credential.helper && rm -f ~/.git-credentials,
then revoke the token in GitBucket -> Account Settings -> Applications.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
