| 2026-07-16 |
D-125 correction: uplink /24 is the ruled 172.30.2.0/24, not a HELD tfvar (no new NetBox object)
...
The initial D-125 records wrongly modeled the vcloud ISP /24 as a NEW,
NetBox-assigned HELD value (`var.vr1_dc0_uplink_cidr`) with an OPNsense WAN
re-address. That is factually wrong: bridge-in is single-NAT, so there is
exactly ONE simulated-ISP /24 for vr1-dc0, and D-115 already ruled it
(172.30.2.0/24, role edge, site vr1-dc0). NetBox models the prefix by role+site
and does not care which libvirt host runs the NAT, so the /24's IPAM identity is
UNCHANGED from Model A; only the libvirt realization moved (a vcloud NAT bridged
through vvr1-dc0). OPNsense WAN stays .2. (Caught reconciling the "create netbox
objects" request against dc-edge-wan-import.py, which already registers this /24.)
- opentofu/main.tf: module vr1_dc0_uplink cidr = "172.30.2.0/24" inlined (ruled
literal, matching the inner root's site-wan pattern) instead of the var.
- opentofu/variables.tf: REMOVED variable vr1_dc0_uplink_cidr (replaced with a
note explaining why it is a ruled literal, not a tfvar).
- Records corrected everywhere the false claim propagated: D-125 body + D-122
amendment, changelog-phaseC2-D125, model-a-fallback (B->A delta item 8 +
revert), session-ledger, dc-dc-deployment-workflow, phase-2 runbook.
Net effect: D-125 adds NO new NetBox object and NO new HELD gate. The only
NetBox work left for the DC substrate is D-124's transit /30 + rack IP.
Both roots validate; gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
docs(session-ledger): record D-125 push + the post-Model-B doc drift-reconciliation batch
...
Standing-ledger update (deliverable discipline): logs D-125 committed d34ec94 +
pushed, and the three doc-only reconciliations (8496c2e workflow-doc, 87f9d75
opentofu/README, 788911c teardown-rollback) with the repo-wide drift-scan
finding (remaining deleted-file/old-count refs are changelogs/decision-records/
tombstones, not live drift; deeper prose + Stage-4/5 sections still lag).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
docs(teardown-rollback): Model B reconciliation banner (two-root DC teardown)
...
runbooks/dc-dc-teardown-rollback.md was written 2026-07-10 for the SINGLE-ROOT
shape (OpenTofu creates DC node-VM domains directly on vcloud; maas-vm-host
registers the vcloud host). Under D-123 Model B that is operationally wrong for
the DCs and a mis-directed rollback is a hazard, so added a top-of-file
reconciliation banner (body not yet rewritten). Claims grep-verified:
- TWO roots / TWO state files: OUTER (opentofu/, vcloud) owns vvr1-dc0 + transit
+ the D-125 vcloud ISP uplink; INNER (opentofu/vr1-dc0-substrate/) owns the 9
node VMs + 6 planes + vr1-dc0-wan + edge + inner pool ON vvr1-dc0's libvirt.
- maas-vm-host now registers vvr1-dc0's INNER virsh -> Office1 region.
- Whole-DC site-down is ONE object: virsh destroy vvr1-dc0 (D-122/D-123).
- Teardown ORDER: inner root first (or virsh destroy vvr1-dc0), then outer;
D-061 "clean up MAAS record before destroying libvirt" still applies.
- Model B -> A revert is docs/model-a-fallback-plan.md, not this runbook.
Office1/mesh/pool steps unaffected. Doc-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
docs(opentofu/README): reconcile to Model B two-root + neutralize deleted config-ISO path
...
opentofu/README.md described a single-root pre-Model-B tree and, dangerously,
the DELETED OPNsense config.xml/config-ISO delivery as current. Added an
authoritative STATUS-RECONCILED banner (repo/ledger authoritative over the older
per-module prose) and inline SUPERSEDED/DELETED markers. Grep-verified:
- TWO ROOTS (D-123 Model B): outer opentofu/ + inner opentofu/vr1-dc0-substrate/,
bootstrap gate between; 9 node VMs (D-121/R-3); D-125 vcloud ISP uplink.
- Modules ADDED, missing from the Built list: site-wan (D-122), wan-bridge (D-125).
- INSTANTIATED (contradicts "not instantiated" prose): cloudinit-vm + base-image
(voffice1/vvr1-dc0/ubuntu_noble_base), node-vm (inner), dc-planes/site-wan/
wan-bridge/opnsense-edge (inner).
- DANGER neutralized: templates/opnsense-config.xml.tmpl,
scripts/opnsense-build-config-iso.sh, scripts/opnsense-render-config.sh are
DELETED (verified) -- a full config.xml push clobbers the live API-managed edge
(D-112/D-113(a2)). Banner + inline markers on the Built-list refs; deeper
2026-07-09 research/audit refs are framed historical by the banner.
- GitBucket out of scope (D-116) where listed.
Kept Stage-1 history (applied 2026-07-10, DOCFIX-179 provider mapping). Doc-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
docs: reconcile workflow-doc Stage-3 section against repo (post D-121..D-125)
...
The dc-dc-deployment-workflow.md Stage-3 (Phase 2) section + gap register had
drifted behind the rulings that landed through 2026-07-16. Repo wins; each
correction grep-verified against the tree (hard rule 2):
- Build/Gate/Owns rows: single-root -> D-123 Model B two-root
(outer->bootstrap->inner->maas-register); vvr1-dc0 IS the rack; D-125
bridge-in per-DC ISP egress; Owns += D-121/122/123/124/125.
- Authoring status: node sizing RULED (D-121/R-3 = 9/DC), config.xml retired
(D-113(a2) REST), both roots instantiated + validate.
- Replaced the stale "two blocking rulings landed 2026-07-15" note (D-121
"3 storage/16 VMs, 790 GiB, vault fork OPEN") with a current rulings summary:
9 nodes/DC (R-3), FIT 870/1024 (Model B), vault RULED (R-2), + D-123/124/125.
- Rack source no longer a missing-module gap: vvr1-dc0 stood up by
site-headend-install.sh --host-nodes.
- Gap register: D-113(a2) OPEN->DONE in both places (banner + line 838)
(template deleted, config_seed/config_iso_path retired). Stage-4
"node count/sizing undecided" -> RULED.
Bottom line now stated honestly: Stage-3 authoring complete; execution is
operator-gated (HELD NetBox addressing + deploy-time gates). Doc-only; no
code/behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap
...
OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.
- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
bridge (bridge={name}, schema measured from the provider, not guessed). No
subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.
Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).
NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening
...
Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
(108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
single Office1 transit leg); node/plane/wan/opnsense modules removed from
vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.
Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.
Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.
KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.
Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Review sweep A+B + R-3 storage-04: coherent record, whole-host calculator, R-3-compliant Model A fallback
...
Phase A (record amendments) + Phase B (scripts/dc-dc-whole-host-budget.py + harness) of the
Stage-3 adversarial-review sweep, plus the R-3 storage-04 addition (Model A layout now
3+2+4 = 9 nodes/DC). This commit is the R-3-COMPLIANT Model A state and becomes the re-cut
`model-a-fallback` anchor (the prior tag at 87a7a8a was R-3-stale -- 3 storage -- flagged by
the Model B design cross-check). NO cloud mutation; nothing applied.
- design-decisions.md: D-103/D-107/D-114/D-115/D-121/D-122/D-123/D-124 amendments (rulings R-1..R-4).
- scripts/dc-dc-whole-host-budget.py + tests/ (13/13): replaces the lost optc-calc.py (R3-F06).
- opentofu/main.tf: locals.vr1_dc0_nodes += storage-04 (R-3); tofu validate Success.
- security-ledger SEC-011 closed; session-ledger rulings + sweep progress; model-a-fallback plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
WIP: freeze surface for Stage-3 adversarial review -- SEC-010/011 + ledger
...
Plane-segregation review findings logged earlier this session, committed to
freeze the review surface (per the Stage-3 adversarial-review prompt, FIRST
ACTION). Commit is not push; nothing leaves the jumphost.
- security-ledger: SEC-010 (rack ip_forward/DC-local enforcement) + SEC-011
(node least-connectivity) as PRE-APPLY hardening rows.
- session-ledger: the 2026-07-16 plane-segregation review outcome + the
functional note (rack routes to an as-yet-unattached transit peer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-15 |
stage-close discipline: add the operating-skill sweep as part of definition-of-done
...
Per operator: the skill-update sweep must itself be part of every stage close, so it
is never skipped. At stage close (alongside the branch->main merge + branch retire),
sweep .claude/skills/openstack-cloud-ops/ for any new INVARIANT the stage established
and reconcile divergence from repo HEAD. Recorded in both the workflow doc's
Cross-cutting discipline and the skill's own VR1 deploy-loop stage-close note.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
skill: add the standing merge-at-close discipline + the DOCFIX-195 apex pointer
...
The operating skill delegates facts (stage status, IPs) to the repo by design, so
Stage-2-merged/D-120/etc. need no skill edit. But two things are the skill's own
domain and were missing:
- The new INVARIANT: branch-per-stage + MERGE AT STAGE CLOSE (merge commit,
operator-gated, retire the branch, next stage off post-merge main). Added to the
VR1 deploy loop as part of a stage's definition-of-done. Precedent: Stage 2 -> PR #1.
- A DOCFIX-195 pointer on the IPAM routing row: during VR1 the WORKING apex is
office1-netbox (netbox.baldurkeep.com is a read-only v1 reference, write-back
deferred). This exact ambiguity cost time early this session.
repo-lint 0 fail (L8 guide<->skill coupling pins clientdocs, not this skill -- unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
Stage 2 CLOSED+MERGED; open Stage 3 branch; standing merge-at-close discipline; drop stray temp/ files
...
Post-merge housekeeping after PR #1 (e468d2e) put VR1 Stage 1+2 on main.
- Workflow doc: the Stage 2 "branch merge point" note now records the merge as DONE
(PR #1, e468d2e) and GENERALIZES it into a standing rule in the Cross-cutting
discipline section -- "branch-per-stage + MERGE AT STAGE CLOSE": every stage
develops on its own branch and merging it to main (merge commit, operator-gated)
is part of that stage's definition-of-done, then the branch is retired. main is
trunk-of-record only for merged stages; the next stage branches off post-merge main.
- session-ledger: Stage 2 CLOSED+MERGED recorded; the PINNED merge ruling graduated
into the standing discipline; NEXT = Stage 3 on dc-dc-stage3-phase2-dc-substrate,
with D-119 selectors + creds-audit (SEC-009) in place.
- Removed temp/ (3 stray screenshot PNGs that rode onto main via 95f0634).
repo-lint 0 fail. No script/harness touched (gauntlet unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
Merge pull request #1 from OpenStack/dc-dc-stage1-phase0-first-exec
...
Dc dc stage1 phase0 first exec
|
C2 sub-task (ii): extend sandbox loop to ip-ranges/ip-addresses; load + verify D-120 bands
...
Closes the logged tooling gap and the D-120 Step 6 load, verified end to end. With
this, C2 is fully closed and Stage 2 close-out (C1..C5) is COMPLETE.
Tooling:
- prod-draft-dump.py: ENDPOINTS += ipam/ip-ranges, ipam/ip-addresses (ranges get a
synthetic start-end key). Verified live.
- sandbox-fidelity-check.py: KEY + EXPECTED_NEW cover the 3 D-120 ranges + 2 service
IPs; both-bounds gives endpoint/address correctness (pure delta -- the frozen draft
has none). Success banner now D-115/D-117/D-118/D-120. Harness +T11/T12 (12->14).
- netbox/d120-compose-bands.py: stdlib, UA-aware, SANDBOX_HOSTS-guarded, dry-by-default,
whole-plan preflight (parent 10.10.1.0/24 exists, endpoints in-parent, start<=end) so
it cannot half-write. Harness 16/16 pins the band values + guards.
Load (operator-approved --commit into office1-netbox, the VR1 apex; sandbox-local):
- created 3 ip-ranges (.2-.49 / .100-.200 / .201-.254) + 2 ip-addresses (.10 netbox,
.11 tailscale), 0 already present. Re-dump -> 3 ranges, 2 IPs. Fidelity check -> exit 0,
"EXACTLY the planned delta (D-115/D-117/D-118/D-120)."
D-120 Step 6 CLOSED (no apex write -- write-back stays deferred). Gauntlet ALL GREEN
(60); repo-lint 0 fail. Workflow C2 row + session-ledger updated: Stage 2 close-out
complete -> branch->main merge point reached (merge remains operator-gated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
creds-audit: enforce per-site credential consolidation (SEC-009); prevents the DC0/DC1 spread
...
SEC-009 established "all site secrets live in ~/<site>-creds/" but nothing enforced
it, and it had two blind spots: VM-minted secrets (no forcing function to consolidate
-- office1-netbox's sandbox NetBox token went un-consolidated until needed) and silent
sprawl (a loose *.env in ~ had no detector). Standing up DC0/DC1 multiplies both.
Built (operator-directed):
- creds-manifests/<site>.manifest -- declares exactly the secrets a site must hold,
each with mode + provenance (local, or <vm>:<path> for the VM-minted class).
vr1-office1.manifest seeded (11 entries); the live folder audits CLEAN.
- scripts/creds-audit.sh <site> -- verifies the live ~/<site>-creds/ against the
manifest (0700 folder, declared modes, present, non-empty), flags UNDECLARED files,
and scans the home root for SPRAWL. Reads NO secret values (metadata only; harness
T7 asserts no cat/head/tail/od/xxd). CREDS_ROOT/MANIFEST_DIR overrides for testing.
- tests/creds-audit/run-tests.sh (7/7) -- missing VM-secret, wrong mode, undeclared
file, home-root sprawl, non-0700 folder, metadata-only. Auto-discovered by gauntlet.
- dc-dc-phase3 GATE -- non-negotiable close-out: consolidate + manifest each secret AT
MINT TIME, and creds-audit $DC must be CLEAN before a DC is declared done.
- security-ledger SEC-009 ENFORCEMENT note.
Verified: harness 7/7; live creds-audit vr1-office1 -> CLEAN (also tool-confirms the
sandbox-token gap is closed); gauntlet ALL GREEN (59); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
C2 sub-task (i): sandbox re-verified faithful; fidelity-checker scope false-positive fixed
...
Re-ran the HARDENED fidelity check against office1-netbox (the VR1 IPAM apex,
DOCFIX-195) -- read-only dump via the consolidated sandbox token + an SSH tunnel.
Result: exit 0, "upstream draft + EXACTLY the planned delta (D-115/D-117/D-118)."
C2 sub-task (i) is CLOSED; the sandbox is proven faithful.
The hardened checker false-failed on 10.10.0.0/16 and 172.30.0.0/16 being
unscoped -- but those two role-container /16s are unscoped BY DESIGN
(d115-office-carve.py), matching the 27 unscoped role containers already in the
upstream apex. A full cross-check of all 44 delta prefixes against their intended
scope was 44/44 (every DC prefix on the correct DC site; no wrong-target binding).
Fix (netbox/sandbox-fidelity-check.py): assert scope == INTENDED per delta prefix
(EXPECTED_PREFIX_SCOPE; DC half generated by _dc_prefix_scopes so the CIDR set and
scope map cannot drift). Strictly stronger than the old "has a scope?" test -- still
catches a dropped scope, and now catches a WRONG-TARGET binding (a DC0 prefix bound
to vr1-dc1, the D-117 near-miss). NOT fixed by exempting the two CIDRs.
Harness moved to dump format (scope_site), T1 sets each prefix's scope from the
checker's own map, +T9 (wrong-target fails) +T10 (legit-unscoped passes). 10->12.
Gauntlet ALL GREEN (58); repo-lint 0 fail.
SEC-009: the sandbox NetBox token had been MISSED by the consolidation (lived only
at /root/netbox-secrets/api.token on office1-netbox); now consolidated to
~/vr1-office1-creds/vr1-netbox-sandbox.env (0600), copied off the VM without
printing. Ledger addendum recorded.
No NetBox state changed -- dump and check are read-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
DOCFIX-195: office1-netbox holds the VR1 IPAM apex role; apex write-back deferred; C2 redefined
...
Operator ruling 2026-07-15: office1-netbox (10.10.1.10) HOLDS THE IPAM APEX
ROLE during VR1 -- the authoritative NetBox every consuming system (OpenTofu,
MAAS, overlays, bundle) derives its values from. It is a working draft (edited
as the buildout proceeds), but it is the apex. netbox.baldurkeep.com takes NO
write during VR1; it is the eventual production apex, frozen as a read-only v1
reference draft. Merge-back is deferred to end-of-deployment (a maybe), NOT a
Stage 2 gate.
The repo encoded the opposite roles and defined close-out C2 (the last Stage 2
gate, and the branch->main merge trigger) as an operator-gated write UPSTREAM.
Under the ruling there is no apex write in Stage 2, so C2 is redefined:
office1-netbox holds the COMPLETE validated dataset -- carve + six-plane roles
+ both DCs + the D-120 compose-/24 child ranges + the two service IPs -- and it
is VERIFIED. C2 stays OPEN.
Finding surfaced while drafting (logged, not fixed): the D-120 objects are
ip-ranges/ip-addresses, object classes the sandbox loop does NOT cover
(prod-draft-dump.py ENDPOINTS and sandbox-fidelity-check.py KEY/EXPECTED_NEW
both stop at rirs/roles/regions/sites/aggregates/prefixes). So the hardened
fidelity check cannot verify the D-120 load; verifying it needs a tool
extension (+harness) or a recorded manual read-back. Confirmed no live system
config points at any NetBox URL -- documentation-only change.
Files: dc-dc-deployment-workflow.md (State + C2 row), dc-dc-netbox-buildout-scope.md
(sec 8.1 + 8.6), session-ledger.md (5 narrative spots + new deferred-item bullet;
machine-derived block re-seeded from the 2026-07-15 scan), design-decisions.md
(D-010 note + D-103 pointer), netbox/README.md (target banner), + changelog.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
D-120 ADOPTED + EXECUTED: re-IP Office1 services into the .2-.49 static band
...
office1-netbox .201->.10 and office1-tailscale .202->.11, on the wire, no
wipe. NetBox HTTP 200 at .10; tailscale still advertises 10.10.0.0/22.
Method (DC replication inherits it): MAAS refuses interface edits on a
Deployed machine and the only lift is Release->redeploy=WIPE, so the live
address is set on the guest (netplan + curtin cfg, reboot-durable) over
lxc exec (socket immune to the netplan apply), and MAAS-model drift is
accepted (reconciles at next teardown/redeploy). NetBox ALLOWED_HOSTS=*
so no restart; tailscale re-asserted automatically.
Swept SANDBOX_HOSTS .201->.10 in the three importers + test + the on-box
working copy; updated live-state docs; flipped D-120 to ADOPTED with an
Execution record; resolved the re-IP runbook's PENDING markers to the
measured method. Gauntlet ALL GREEN (58); repo-lint 0 fail / 1 legacy WARN.
Still owed: D-120 Step 6 (register the /24 child ranges + service IPs in
the sandbox, feed upstream) rides with C2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpGaiX6UvhSReDwN2ywstJ
|
D-120 PROPOSED: static-services addressing for compose /24s; re-IP plan drafted
...
Office1's persistent services (office1-netbox .201, office1-tailscale .202) sit in
MAAS's deployed-node band -- addressed like ephemeral compute nodes, by MAAS
auto-assign default, not by design. They are NOT in the DHCP dynamic range
(.100-.200), so no collision -- but no static-services sub-layout was ever ratified
for the /24 (D-115 left it "MAAS DHCP/PXE").
D-120 (PROPOSED) defines a functional band layout for EVERY MAAS-managed compose /24
(Office1 + the DCs, which inherit it):
.1 gateway (site containment VM)
.2-.49 STATIC site services (MAAS static-assign, outside the dynamic range)
.50-.99 reserved/spare
.100-.200 MAAS dynamic (unchanged)
.201-.254 deployed nodes (unchanged)
Proposed Office1 addresses: netbox .201->.10, tailscale .202->.11. Recommends
option (a): adopt + re-IP Office1 now, so the DC template is clean. Addresses AWAIT
operator ratification.
Re-IP plan: runbooks/dc-dc-office1-service-reip.md -- gated mutation on two running
services, honest PENDING VERIFICATION markers for the MAAS static-assign CLI and the
netbox-docker ALLOWED_HOSTS / tailscale re-assert (cannot confirm without the
operator-held box keys). Cascades to 20 repo refs incl. SANDBOX_HOSTS .201 in the
two pynetbox importers + their tests. NOT executed.
The layout belongs IN NetBox (the IPAM authority deciding where NetBox lives) --
recorded via the sandbox loop, not prose.
repo-lint 0 fail. No live services touched -- this is a proposal + plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Consolidate credential/env sprawl into ~/vr1-office1-creds/ (SEC-009); seed memory
...
Operator-approved. Three env files were loose in ~ outside the consolidated creds
folder -- .vr1-netbox.env, vr1-office1.env, and vr1-stage1.env. The last was mode
0664 (world-readable) and held TF_VAR_maas_api_key: a real MAAS-secret exposure,
not just untidiness. None created this session (mtimes 07-10..12).
Done (files outside the repo; recorded in the changelog):
- moved all three into ~/vr1-office1-creds/; chmod 600 on every sensitive file
(fixed the two 0664 files: vr1-stage1.env and tailscale-authkey.txt).
- no CONTENTS read -- key NAMES only, to identify each file's purpose.
- fixed a D-119 follow-up the in-repo sweep could not reach: vr1-stage1.env had
TF_VAR_dc1_pool_path/dc2 -> renamed to TF_VAR_vr1_dc0/vr1_dc1_pool_path.
Repo (committed):
- active path refs old ~/.vr1-netbox.env -> ~/vr1-office1-creds/vr1-netbox.env in
prod-draft-dump.py, the phase1 runbook Step 10b, as-built, and the ledger. Dated
changelog/incident records left as historical snapshots.
- as-built secrets table: the three env files + the convention. CORRECTED a
mid-session wrong assumption -- the SANDBOX token is NOT on vcloud; it is on
office1-netbox (/root/netbox-secrets/). vr1-netbox.env is the UPSTREAM token only.
- security-ledger SEC-009 (REMEDIATED) + the standing per-site consolidation convention.
- seeded agent memory (was EMPTY): the creds convention + the closed-test posture,
so neither is re-lost at compaction.
STANDING CONVENTION: per-site ~/<site>-creds/ (0700; files 0600); no loose env in ~;
~/vr1-dc0-creds/ and ~/vr1-dc1-creds/ follow the pattern from day one. Data-loss /
continuity control first; real cred hardening deferred to post-teardown (closed test).
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-14 |
C4 DONE: document the NetBox sandbox loop; add the upstream-write guard it exposed
...
C4 (Stage 2 close-out) -- the sandbox loop is now documented in two places, no
rationale duplicated: docs/dc-dc-netbox-buildout-scope.md section 8 (design: why a
sandbox, the fidelity check + its 2026-07-14 hardening, the write-guard table, the
WAF trap, C2 preconditions) and runbooks/dc-dc-phase1-office1-standup.md Step 10b
(the dump->seed->apply->verify->gated-write sequence, two-host/two-token separation).
Drafted by a subagent, corrected against this session's own fixes before placement.
The guard C4 exposed: roles-aggregates-import.py and dc-dc-prefixes-import.py had NO
hostname gate -- until today the WAF 403 was the only thing stopping an accidental
--commit to the apex, and the UA fix removed that accidental safety. Both now gate a
non-sandbox --commit behind --yes-write-upstream (SANDBOX_HOSTS allowlist), matching
d115-office-carve.py -- so all three write-capable importers behave identically. This
is defence-in-depth with the whole-plan preflights (which prevent a half-write).
Tests: prefixes-import 86->91 (refused without the flag + writes nothing; allowed
with it; sandbox host needs no flag). roles-aggregates 20->24. GAUNTLET ALL GREEN (58).
Stage 2: C1/C3/C4 DONE. C2 (production write) is the only item left -- gated, with
preconditions now documented (re-verify sandbox under the hardened check, run from
office1-netbox, --yes-write-upstream, approve each mutation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fix the WAF User-Agent gap: the two pynetbox importers could never write upstream
...
A C2 blocker, measured 2026-07-14: upstream netbox.baldurkeep.com 403s the default
Python User-Agent (references/platform-traps.md) -- curl->200, urllib/pynetbox->403.
The stdlib tools (prod-draft-dump/sandbox-seed/d115-office-carve) already send an
accepted UA, but roles-aggregates-import.py and dc-dc-prefixes-import.py built
pynetbox.api() with the DEFAULT UA and never overrode it. So they would 403 against
upstream -- they could only ever have written to the WAF-less sandbox, never to the
apex C2 must feed. The gap was invisible because every real run so far hit the sandbox.
Fix: get_nb() sets nb.http_session.headers["User-Agent"] = "curl/8.5.0" in both
(same value/rationale as the stdlib tools; pynetbox exposes its requests.Session).
Tests: prefixes-import 85->86 (the fake now has .http_session; the UA is asserted at
RUNTIME, not by source grep). roles-aggregates 19->20. GAUNTLET ALL GREEN (58).
Still open for C2 (logged in the changelog): pynetbox is NOT on vcloud, so the two
importers must run on office1-netbox (has pynetbox 7.0.0); the sandbox re-verify
needs the operator-held token; the upstream write stays operator-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119 OpenTofu rename APPLIED -- repo/state back in sync (operator-gated)
...
tofu apply: 11 added, 0 changed, 11 destroyed. tofu plan now reports "No changes".
Pre-apply LIVE re-check confirmed all 11 objects empty (0 leases, 0 volumes, no
domain attached) -- verify-before-mutate, not the earlier measurement. Fresh plan
was 11 add / 11 destroy, all "must be replaced" via the moved{} blocks.
Verified on the host: six vr1-dc0-* networks, three mesh-vr1-* links, vr1-dc0-pool
/vr1-dc1-pool; ZERO old dc1-*/dc2-*/mesh-dc* names remain. Office1 (office1-local/
office1-wan/wan/edge/voffice1) was NOT in the plan and is untouched -- edge and
voffice1 both reachable post-apply.
Applied now, while the objects are empty, so Stage 3's own tofu apply does not
silently bundle this 11-resource rename with real DC provisioning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Add the missing preflight-failure test; reconcile the stale tofu-sync ledger line
...
Two follow-ups a review caught:
1. Write-path fix #3 (whole-plan role preflight in dc-dc-prefixes-import.py) shipped
with NO test exercising it FAILING. Every existing test preseeds all roles, so
they ran the preflight only when it passes -- silently. A future edit could delete
the preflight and the harness would stay green: exactly the "green while the bug is
armed" failure this session has been killing. Added a test that runs main() with one
six-plane role missing and asserts nonzero exit AND that NOTHING was written (no
site, no prefixes). prefixes-import 82 -> 85.
2. The ledger's "Live state" bullet still claimed OpenTofu repo/state were IN SYNC
("tofu plan reports No changes"). False since D-119 -- plan is 11 to add / 11 to
destroy. Reconciled to state the gated destroying apply, contradicting nothing.
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger: write-path bugs 1-3 fixed; flag the sandbox re-verify owed before C2
...
The sandbox was declared "proven faithful" using the fidelity check BEFORE it was
hardened -- i.e. under the version that could false-green. That verdict must be
re-established under the hardened check before C2 feeds the carve upstream. Needs
the operator-held sandbox token; folds into the C2 step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
NetBox write-path hardening: the bugs that would have corrupted the apex at C2
...
Found by adversarial review during D-119; independent of the naming work. These
are what would have bitten during C2 (the production IPAM write Stage 2 needs).
1. roles-aggregates-import.py STILL died half-way through a production write. The
preflight added on 2026-07-13 -- after this same script committed 4 roles, 400'd
on the 5th and left the apex half-populated -- only covered role-NAME collisions.
The ARIN RIR lookup and validate_ula_48() both still ran BELOW the first write,
so an apex without ARIN, or a typo'd ORG_ULA_48, would commit 5 roles (and the
RIRs) and THEN die, with no rollback. The exact failure the preflight was added
to prevent, still armed one section lower. The RIR half of the preflight loop was
a literal no-op. Both checks moved INTO the preflight.
A preflight that does not cover every die() downstream of the first write is not
a preflight.
2. sandbox-fidelity-check.py could return a FALSE GREEN -- and it is the script we
have been citing as PROOF the sandbox is a faithful replica. It field-compared
only SHARED objects; the planned delta was checked merely as a SUBSET of
EXPECTED_NEW. With ZERO of the 36 DC prefixes created, extra={} -> unexpected={}
-> it printed "Nothing lost, nothing stray" and exited 0. EXPECTED_NEW was an
upper bound masquerading as an assertion. It is now BOTH bounds, and delta
prefixes are checked for scope and role (the 17-prefix scope-drop bug relocated
into the delta, where the old check was structurally blind).
3. dc-dc-prefixes-import.py died mid-loop on a missing role -- find_role() was
resolved lazily INSIDE the write loop. Against upstream as it stands TODAY (no
six-plane roles), a --commit creates the site + 4 prefixes then dies on the 5th:
a half-written datacenter. Now a whole-plan preflight resolves every role first.
4. The test fake was KINDER THAN REALITY at the failure point: fake_pynetbox.get()
returned matches[0] on a multi-match where real pynetbox RAISES. NetBox permits
duplicate prefixes (the importer's own docstring anticipates them vs vr0-dc0), so
this would blow up in production while the harness sailed through. It now raises.
TESTS: tests/sandbox-fidelity-check/ is NEW (10/10) -- the checker shipped with NO
harness at all, which is exactly why the false green survived; T3 reproduces it, and
T1 derives its expectation from the checker's own EXPECTED_NEW rather than re-typing
it. roles-aggregates-import 16->19, asserting BY POSITION that ARIN and the ULA are
validated before the first create (the old harness only checked the WORD "PREFLIGHT"
appeared -- green while the landmine was armed).
GAUNTLET ALL GREEN (58). repo-lint 0 fail. Zero NetBox writes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
C1 follow-ups: verify the RDNSS we never set; correct a revert line that did not revert
...
Two defects in the C1 changelog, both found on review:
1. The RA advertises `RDNSS 2602:f3e2:f01 :1` -- a field we never set; OPNsense
auto-defaulted it to the edge's own LAN v6 address. Same failure class we avoided
with mode=unmanaged (advertising a service that may not exist), one field over,
and it slipped in BECAUSE the field was left unset. CHECKED: the edge's Unbound
really does answer DNS on that v6 address (dig from voffice1 resolves; it works
with no v6 egress because Unbound queries upstream over v4). Correct -- but by
luck, not design. Set RDNSS explicitly when this RA config is reused on the
DC0/DC1 edges at Stage 3.
Bonus: this makes "services binding v6" EVIDENCE rather than inference -- a real
v6-bound service answering real queries.
2. The address-revert line said `--commit 10.10.0.1 lan :: 64`, which does NOT clear
the field -- it would SET the unspecified address. The script cannot express a
clear at all (4 args required, address validated). Shipping an unverified revert
is exactly what the delivery rule exists to prevent. Replaced with the honest
GUI clear + a note that a --clear flag would be the real fix.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 LAN IPv6 is LIVE and proven on the wire -- Stage 2 close-out C1 CLOSED
...
The last build item on Office1. Executed against the live edge, verified against
the kernel and the wire, not by {"result":"saved"}.
scripts/opnsense-set-interface-v6.sh --commit 10.10.0.1 lan 2602:f3e2:f01:1 64
POST /api/radvd/settings/addEntry {"entries":{...,"mode":"unmanaged"}}
POST /api/radvd/service/reconfigure
PROVEN: address on the kernel (ifconfig, not the config file); radvd advertising
2602:f3e2:f01:/64 AdvAutonomous/AdvOnLink with no M/O flags; voffice1
autoconfigured ...:5054:ff:fe6a:87e5 by SLAAC + an RA-learned route; edge->voffice1
GUA ping 0.0% loss. The neighbour MAC matches voffice1's known Kea lease hwaddr.
No collateral damage: v4 10.10.0.1 intact, Kea DHCPv4 still serving udp/67, the
compose-net route still good.
mode=unmanaged is DELIBERATELY not the model's default. `stateless` sets the RA
Other-config flag, pointing clients at a DHCPv6 server -- and Kea DHCPv6 is
enabled=0 with no daemon running. The default would have advertised a server that
does not exist. Field names, the request key (addBase('entries','entries')), the
mode values and the apply endpoint were all READ OFF THE EDGE, not recalled.
OPERATOR RULING -- the v6 NAT was NOT built. A v4-to-v6 NAT (a second OPNsense, or
nginx) was proposed to simulate the v6 the lab will not forward. Withdrawn: it
contradicts D-101 ("native GUA routing, no NAT66"), D-115's amendment says the
missing egress is the upstream's problem and not ours, and a translation box is
precisely the component Roosevelt will NOT have -- maximum delta for zero proof.
nginx is an L7 proxy, not NAT, and proves nothing about RA/ICMPv6/Ceph-over-v6.
Ruling: configure IPv6 internally now; configure the IPv6 edge AFTER deployment.
TRAP recorded: voffice1 has forwarding=1 + accept_ra=0 and STILL autoconfigured --
netplan/systemd-networkd processes RA in USERSPACE, independent of the sysctl.
WAN v6 leg stays DEFERRED: re-measured today, v6 internet is 100% loss from vcloud
while v4 is fine. Internet GUA reachability remains UNPROVABLE in VR1.
repo-lint 0 fail; opnsense-set-interface-v6 harness 40/40.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
