diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index a01a126..3ce597b 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -20,6 +20,33 @@ ## 1. Where the project IS +> ## >>> STANDING PIVOT, 2026-08-08: 10.12.0.0/16 COLLIDES WITH THE LIVE IPv4 CLOUD -- DC0 TO FULL DEPLOYMENT AS A CHECKPOINT, THEN TEARDOWN + REDEPLOY ON 10.13.0.0/16. <<< +> +> The VR1 build's `10.12.0.0/16` is already consumed by the still-online IPv4 single-DC cloud +> (vr0-dc0; the NetBox draft that seeded VR1 was that cloud's external export). Surfaced 2026-08-08 at +> the Headscale route step -- advertising `10.12.8.0/22` overlaps on the shared tailnet (no 4via6). +> **Operator plan: drive dc0 to FULL deployment as a clean checkpoint, then TEARDOWN + REDEPLOY on +> `10.13.0.0/16` -- never editing live infra.** Tailscale workstream STOPPED (do NOT approve +> `10.12.8.0/22`); **dc1 HELD entirely** (0 machines in vr1-dc1-region, all nodes powered off). +> +> **NOT YET RULED -- owed (GA-R5) BEFORE any dependent re-carve.** The re-IP is a **D-115 SUPERSESSION** +> (D-115, adopted, REJECTED 10.13 and set the NetBox Cloud role to `10.12.0.0/16`) AND it TERMINATES +> D-101's "vr1 inherits vr0-dc0 unchanged" (`lib-net.sh` ~124-134). The ruling must reconcile +> D-101/D-115/D-124/D-134 and confirm `10.13.0.0/16` is LIVE-free (existing cloud + tailnet). Draft +> mapping (octet-preserving `10.12.a.b -> 10.13.a.b`): `docs/audit/netbox-1013-subnetting-draft-20260808.md`. +> Sweep: `docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt`. +> +> **dc0 checkpoint scope (operator 2026-08-08): "activate + smoke-test"** -- networks + Octavia (1 test +> LB) + Designate (1 test zone) + wrap gates (cloud-assert BOM, controller backup, verify-live +> Ceph/geneve-over-v6); **Magnum/CAPI DEFERRED** to the redeploy. dc0 MEASURED 2026-08-08 (dc0 rack +> `juju status -m vr1-dc0` + openstack): 66 machines/162 units, ALL core services active; only octavia +> (configure-resources) + designate (nameservers) + ceph-rbd-mirror (cross-DC/dc1) blocked; +> tenant-resource-empty (networks/flavors/images=0). dc0 `.7` tailscale is up advertise-only +> (`100.64.0.57`, route UNAPPROVED, v6-disabled-runtime) -- inert, torn down at redeploy. +> +> **This 2026-08-08 pivot OVERTAKES the 2026-07-30 "proceed to juju deployment no matter what" +> directive below** -- dc0 now finishes as a CHECKPOINT, not as the final cloud. + > ## >>> STANDING OPERATOR DIRECTIVE, 2026-07-30: THE NEXT SESSION PROCEEDS TO THE JUJU DEPLOYMENT (STAGE 5). NO MATTER WHAT. <<< > > Verbatim: **"we have to continue to juju deployment next session no matter what"**. diff --git a/docs/audit/netbox-1013-subnetting-draft-20260808.md b/docs/audit/netbox-1013-subnetting-draft-20260808.md new file mode 100644 index 0000000..8b2b302 --- /dev/null +++ b/docs/audit/netbox-1013-subnetting-draft-20260808.md @@ -0,0 +1,281 @@ +# NetBox subnetting DRAFT -- VR1 DC-DC rebuild on 10.13.0.0/16 + +**DRAFT -- proposal, NOT ruled, NOT committed.** This is a planning artifact only. +It does not touch live NetBox, live infra, `lib-net.sh`, the overlays, or any other +authoritative surface. No `git commit`. It exists to let the operator rule the re-IP +(F1 pivot, below) with the full blast radius in view. + +- Author: background agent (Task #2), 2026-08-08. +- Method: read-only extraction from `scripts/lib-net.sh` (full), + `netbox/draft/vr1-office1-current-20260801.json`, `docs/design-decisions.md` (D-115, + D-134), and a repo-wide `grep` sweep for `10.12.` literals. Every value below is + measured from those files this session; none is inferred. +- Governing decisions named for the owed ruling: **D-101 / D-115 / D-124 / D-134** + (per `docs/audit/queued-findings-20260808-...-reip-pivot.txt` F1). + +--- + +## 1. Why (the pivot) + +`docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt` **F1** records +the operator pivot verbatim: the VR1 build's `10.12.0.0/16` **collides with the +still-online IPv4 single-DC cloud** (the NetBox draft that seeded VR1 was that cloud's +external export). It surfaced at the Headscale step -- advertising `10.12.8.0/22` +overlaps the live cloud on the shared tailnet, and Headscale has no 4via6, so routes +must be non-overlapping. Operator plan: drive dc0 to a full-deployment checkpoint, then +**teardown + redeploy on `10.13.0.0/16`** -- a fresh subnetting build, never editing +live infra. + +This draft is the addressing half of that redeploy. It is a **greenfield build on a +free /16**, so "re-IP" here means "author the new plan"; there is no live-object edit. + +--- + +## 2. Proposed mapping principle -- 1:1 octet-preserving shift + +**`10.12.a.b -> 10.13.a.b` for every DC (Cloud) address. Second octet `12 -> 13`; +octets 3 and 4 unchanged.** + +This preserves, exactly and by construction: +- the six-plane role layout and offsets for both DCs, +- dc1's `10.12.64.0/19 -> 10.13.64.0/19` /19 supernet (D-115), +- the D-134 utility octet map (`.4/.5/.6/.7`) and the `.4-.49` / `.50-.99` bands, +- the triple-HA VIP columns and octets (D-020), +- the FIP pool sizes and offsets, +- the keystone public VIP octet. + +Only the DC (Cloud-role) `10.12` space moves. Everything else stays (Section 6). + +--- + +## 3. Before / after tables + +### 3.1 DC0 planes (`vr1-dc0`; flat /22s, D-052/D-053 offsets) + +| Role | Before (10.12) | After (10.13) | +|-----------------|------------------|------------------| +| provider-public | 10.12.4.0/22 | 10.13.4.0/22 | +| metal-admin | 10.12.8.0/22 | 10.13.8.0/22 | +| metal-internal | 10.12.12.0/22 | 10.13.12.0/22 | +| data-tenant | 10.12.16.0/22 | 10.13.16.0/22 | +| storage | 10.12.32.0/22 | 10.13.32.0/22 | +| replication | 10.12.36.0/22 | 10.13.36.0/22 | + +### 3.2 DC1 planes (`vr1-dc1`; contiguous /22s inside the `10.12.64.0/19` supernet, D-115) + +| Role | Before (10.12) | After (10.13) | +|-----------------|------------------|------------------| +| (supernet) | 10.12.64.0/19 | 10.13.64.0/19 | +| provider-public | 10.12.64.0/22 | 10.13.64.0/22 | +| metal-admin | 10.12.68.0/22 | 10.13.68.0/22 | +| metal-internal | 10.12.72.0/22 | 10.13.72.0/22 | +| data-tenant | 10.12.76.0/22 | 10.13.76.0/22 | +| storage | 10.12.80.0/22 | 10.13.80.0/22 | +| replication | 10.12.84.0/22 | 10.13.84.0/22 | + +### 3.3 Gateways (`PLANE_GW`) -- unchanged posture, shifted value + +Per D-134 the `.1` gateway exists on the **provider-public** subnet ONLY; metal-admin +and the four data planes are `gw=none` (measured; MAAS returns `none`, `.1` is held by +NOTHING on both DCs). The VR0-only `vr0-dc0` arm additionally pins metal-admin `10.12.8.1` +-- that arm is NOT part of this rebuild (it is the live cloud; see OQ (a)/(c)). + +| Plane | Before | After | +|-----------------------------|-------------|-------------| +| vr1-dc0 provider-public gw | 10.12.4.1 | 10.13.4.1 | +| vr1-dc1 provider-public gw | 10.12.64.1 | 10.13.64.1 | +| all other planes, both DCs | none | none | + +### 3.4 VIP prefixes (triple-HA columns, D-020/D-052) + keystone public VIP + +| Item | Before | After | +|-------------------------------|------------|------------| +| dc0 VIP_PREFIX_PROVIDER | 10.12.4 | 10.13.4 | +| dc0 VIP_PREFIX_ADMIN | 10.12.8 | 10.13.8 | +| dc0 VIP_PREFIX_INTERNAL | 10.12.12 | 10.13.12 | +| dc0 KEYSTONE_VIP_DEFAULT | 10.12.4.50 | 10.13.4.50 | +| dc1 VIP_PREFIX_PROVIDER | 10.12.64 | 10.13.64 | +| dc1 VIP_PREFIX_ADMIN | 10.12.68 | 10.13.68 | +| dc1 VIP_PREFIX_INTERNAL | 10.12.72 | 10.13.72 | +| dc1 KEYSTONE_VIP_DEFAULT | 10.12.64.50| 10.13.64.50| + +VIP band `.50-.99` (`VIP_OCTET_MIN=50`, `VIP_OCTET_MAX=99`), `VIP_COUNT_EXPECT=13` +(11 + vault `.61` + designate `.62`, R11) -- all UNCHANGED. The 13 service VIP octets +(.50 keystone .. .62 designate) are octet-preserved on each of the three legs. + +### 3.5 D-134 utility bands + octet map (per plane) + +Each plane carries a `.4-.49` utility band and a `.50-.99` VIP band (NetBox ip-ranges). +The **D-134 utility octet map** (extended by D-129(iii)(a) for `.7`) rides the +**metal-admin** plane: + +| Octet | Role | dc0 before / after | dc1 before / after | +|-------|-------------------------|--------------------------|---------------------------| +| .4 | artifact mirror | 10.12.8.4 / 10.13.8.4 | 10.12.68.4 / 10.13.68.4 | +| .5 | juju controller | 10.12.8.5 / 10.13.8.5 | 10.12.68.5 / 10.13.68.5 | +| .6 | MAAS region | 10.12.8.6 / 10.13.8.6 | 10.12.68.6 / 10.13.68.6 | +| .7 | tailscale subnet router | 10.12.8.7 / 10.13.8.7 | 10.12.68.7 / 10.13.68.7 | + +(The `.4-.49` / `.50-.99` band ip-ranges exist on ALL six planes per DC, 24 ranges +total; each shifts by the same 12->13 rule.) + +### 3.6 FIP pools (D-003) -- NOTE: MAAS reserved ipranges, not NetBox objects + +| DC | Before | After | Size | +|----------|---------------------------|---------------------------|-----------| +| vr1-dc0 | 10.12.5.0 - 10.12.7.254 | 10.13.5.0 - 10.13.7.254 | 767 addrs | +| vr1-dc1 | 10.12.65.0 - 10.12.67.254 | 10.13.65.0 - 10.13.67.254 | 767 addrs | + +Each sits inside its DC's provider-public /22 and does not overlap the D-134 +`.4-.49` / `.50-.99` bands. **These pools are RESERVED ipranges on the MAAS provider +subnet (KI-P3-001), asserted by `scripts/phase-04-network-verify.sh` -- they are NOT +NetBox ip-range objects** (the office1 draft holds no FIP-pool range). The re-IP +therefore touches MAAS ipranges at build time; NetBox has a standing record gap here. + +### 3.7 MAAS rack-controller statics (D-124 / D-120 static band) + +| Host | Before | After | +|-----------|---------------|---------------| +| vvr1-dc0 | 10.12.8.2/22 | 10.13.8.2/22 | +| vvr1-dc1 | 10.12.68.2/22 | 10.13.68.2/22 | + +--- + +## 4. NetBox objects to re-scope (from `vr1-office1-current-20260801.json`) + +Counts of objects whose value lies in `10.12` and would shift to `10.13` under the 1:1 rule: + +| Object class | Count | Notes | +|---------------------|-------|----------------------------------------------------------------| +| ipam/prefixes | 13 | `10.12.0.0/16` Cloud + twelve plane /22s (6 dc0 + 6 dc1) | +| ipam/ip-ranges | 24 | D-134 `.4-.49` + `.50-.99` bands, 2 per plane x 12 planes | +| ipam/ip-addresses | 80 | 78 VIP legs (13 services x 3 legs x 2 DCs) + 2 rack `.2` statics| +| ipam/aggregates | 0 | `10.0.0.0/8` is the parent; it ALREADY covers 10.13 -- no change| +| ipam/vlans | 0 | none present in the draft | +| dcim/regions,sites | 0 | topology objects carry no v4 literal -- no change | + +**Two structural observations (feed OQ (b), not defects):** +- The dc1 **/19 supernet `10.12.64.0/19` has no NetBox prefix object** -- only its six + child /22s are present. If the plan wants the supernet represented, that is a new + object either way. +- The **FIP pools have no NetBox ip-range object** (Section 3.6) -- they live only as + MAAS reserved ipranges. + +--- + +## 5. Consumer inventory BEYOND NetBox (repo grep sweep, `10.12.` literals) + +The re-IP is not a NetBox-only change. `lib-net.sh` names its own twins-of-record +(`opentofu/variables.tf` `vr1_dc1_planes`; `overlays/vr1-dc*-vips.yaml`; +`provider-bundle-check.py`'s independent octet band). Grouped literal-bearing LIVE +surfaces (historical `asbuilt/` snapshots and `docs/audit/osd-carve-*` frozen captures +EXCLUDED -- those are verbatim records, not to be edited): + +| Surface class | Files | 10.12 literal hits | Key members | +|-------------------------------|-------|--------------------|-----------------------------------------------------------------------------| +| scripts/ | 27 | 246 | lib-net.sh (source of truth), phase-00-maas-standup, dc-rack-net, dc-snap-proxy | +| overlays/ | 3 | 77 | vr1-dc0-vips.yaml, vr1-dc1-vips.yaml (ratified VIP source, D-119 R9) | +| opentofu/ | 4 | 35 | variables.tf (`vr1_dc1_planes` -- lib-net.sh's named twin) | +| netbox/ importers + drafts | 11 | 257 | dc-dc-prefixes-import, dc-plane-apex-import, dc-util-hosts-import, d120-compose-bands, the draft JSONs | +| bundle.yaml | 1 | 16 | deploy input (VIPs) | +| tests/ harnesses + fixtures | 62 | 724 | dc-selector (re-parses the overlay), provider-bundle-check, render-baseline, phase-04 fixtures | +| runbooks/ | 22 | 194 | procedure text citing plane CIDRs | +| docs/ prose | 85 | 1337 | design-decisions, CURRENT-STATE, audit -- **prose record; update via new dated entries, do NOT rewrite** | + +Discipline note: each script change ships with its `tests//run-tests.sh` green, +`repo-lint` clean, and a changelog entry (per CLAUDE.md). The two-file twins +(`lib-net.sh` <-> `opentofu/variables.tf`, and `lib-net.sh` <-> `overlays/*-vips.yaml` +via `tests/dc-selector`) must change in the SAME commit. This inventory is scope +visibility for the ruling, NOT a per-line remediation plan. + +**Stale 10.13 reference already in-repo (pre-D-115):** `netbox/README.md:49` still shows +`VR1_DC1_V4_SUPERNET=10.13.0.0/19`, an importer default that D-115 superseded (dc1 became +`10.12.64.0/19`). Reusing 10.13 as the rebuild /16 is unrelated to that old /19 default, +but the stale line should be reconciled so the two 10.13 meanings do not confuse. + +--- + +## 6. What does NOT shift + +| Space | Value | Why it stays | +|-------------------------------|--------------------------------|-------------------------------------------------------------------| +| Rack transit /30s | 172.31.0.0/30 (dc0), 172.31.0.4/30 (dc1) | Not in 10.12; D-124 Scheme A. See caveat below. | +| Office1 LAN / compose | 10.10.0.0/22 (10.10.0.0/24, 10.10.1.0/24) | Office role, D-115; not part of the DC space. | +| Edge (sim ISP/WAN) | 172.30.0.0/16 (office1-wan 172.30.1.0/24) | Edge role, D-115. | +| Dev clouds | 10.16.0.0/16, 10.17.0.0/16 | Willamette / Roosevelt dev; unrelated. | +| Aggregate | 10.0.0.0/8 | Parent of BOTH 10.12 and 10.13 -- unchanged, 10.13 falls under it. | +| IPv6 (all) | 2602:f3e2::/36 GUA, fd50:840e:74e2::/48 ULA | This is a v4-only re-IP; v6 is region-hierarchical and untouched. | + +**Caveat -- transit ADDRESSES stay, transit ROUTES do not.** The 172.31.0.x /30 endpoint +addresses are unchanged, but the routes carried OVER them point at `10.12.*` DC +destinations today and must be re-pointed to `10.13.*` at rebuild. Flagged so "transit +unchanged" is not misread as "transit needs no work." + +--- + +## 7. Dependent surfaces beyond IPAM (for the redeploy plan, not this draft) + +- **Vault-issued cert SANs / IP SANs** on every API endpoint and VIP -- reissued against + the new addresses at redeploy (D-052 TLS). +- **MAAS DHCP ranges** on the provisioning planes -- re-declared on 10.13. (Currency + note: the measured dc1 DHCP range is `.201-.254`, not a recalled `.100-.200`.) +- **Tailscale subnet router advertised prefix** -- the whole reason for the pivot: + `10.12.8.0/22 -> 10.13.8.0/22` (metal-admin), and the **Headscale ACL** that references + the advertised metal-admin prefix per D-129(iii)(b) star model. +- **OVN / neutron provider network** definitions, and any external/provider network CIDR. +- **Charm config / bundle VIPs** (`bundle.yaml`, overlays) -- deploy inputs. +- **/etc/hosts and internal DNS** seeds (dc-node-etchosts and similar). + +--- + +## 8. OPEN QUESTIONS for the operator (DO NOT decide here) + +**(a) Confirm 10.13.0.0/16 is genuinely free -- LIVE, not just in a snapshot.** +The 10.13 scan in this draft was against `vr1-office1-current-20260801.json` (a dated +snapshot) -- absent there, NOT a measured live absence. Before committing, confirm +`10.13.0.0/16` is free of (i) the still-online IPv4 single-DC cloud, and (ii) anything +advertised on the shared tailnet (the exact class of overlap that caused the pivot). +10.14/10.15 also sit free between 10.13 and the 10.16/10.17 dev clouds, if a range choice +is wanted. + +**(b) Exact 1:1 shift, or regularize a wart while greenfield?** +The 1:1 shift preserves everything but also PRESERVES dc0's non-contiguous offsets +(`4/8/12/16` then jump to `32/36`, skipping `20/24/28`) and the resulting dc0-vs-dc1 +asymmetry (dc1 fits a /19; dc0 cannot). A greenfield /16 is the only cheap moment to +regularize dc0 into a contiguous, /19-shaped block symmetric with dc1 -- at the cost of +breaking octet-preservation and diverging further from the Roosevelt/VR0 layout D-101 +was built to inherit. Presented as an option with its cost; NOT recommended here. + +**(c) This re-IP needs a GA-R5 ruling BEFORE any dependent work.** +Governed by **D-101 / D-115 / D-124 / D-134** (F1). The number is deliberately NOT minted +here -- grep next-free `D-NNN` at ruling time. Two reconciliations the ruling must make +explicit: + +- **D-101 inheritance terminates.** `lib-net.sh`'s `vr1-dc0` arm exists because D-101 + rules VR1 DC0 *inherits VR0 DC0's v4 layout unchanged*, and the live 10.12 cloud IS + vr0-dc0. Moving VR1 to 10.13 ends that inheritance: **vr0-dc0 STAYS on 10.12 (live, + untouched); the `vr0-dc0` and `vr1-dc0` case arms genuinely split for the first time**, + and lib-net.sh's lines 124-134 comment ("Same PLANE values as vr0-dc0 ... INHERITS VR0 + DC0's v4 layout UNCHANGED") becomes FALSE and owes an update. +- **D-115 rejected 10.13 as "outside every allocated block."** D-115 (adopted) put the + NetBox **Cloud** role at `10.12.0.0/16` and explicitly superseded a `10.13.0.0/19` + recommendation because 10.13.x was unallocated. Its option (b) even floated a per-region + /16 (VR1 = `10.15.0.0/16`) and the operator chose role-based instead. Picking + `10.13.0.0/16` now REVIVES 10.13 as an allocation and must decide: does the NetBox + **Cloud** role/prefix move to 10.13.0.0/16 (leaving the live cloud's 10.12 as its own + record), does 10.13 become a NEW role (e.g. "Cloud -- VR1 rebuild") coexisting with the + 10.12 Cloud, and is this a D-115 amendment or a superseding decision? This is a genuine + fork in the IPAM apex, not a mechanical shift. + +--- + +## 9. Summary line + +**Proposed: octet-preserving `10.12.a.b -> 10.13.a.b` for the entire VR1 Cloud DC space +(both DCs), everything else held.** Remapped: **12 planes** (6 per DC), **13 prefixes**, +**24 ip-ranges**, **80 ip-addresses** (78 VIP legs + 2 MAAS rack statics), **2 FIP pools** +(MAAS ipranges), **2 keystone VIPs**, **1 provider gateway per DC**. Held: 172.31 transit +addresses (routes re-point), 10.10 office, 172.30 edge, 10.16/10.17 dev, all IPv6, the +10.0.0.0/8 aggregate. + +**DRAFT -- proposal, not ruled, not committed.** diff --git a/docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt b/docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt new file mode 100644 index 0000000..09b19f0 --- /dev/null +++ b/docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt @@ -0,0 +1,162 @@ +QUEUED FINDINGS -- session 2026-08-08 (dc0 tailscale install incident + the 10.12->10.13 re-IP pivot) +Sweep method: model read the session, grepped each candidate. FIRST SURFACE items lead. +Status authority is docs/CURRENT-STATE.md; this file is a sweep record, not status. + +================================================================================ +FIRST SURFACE (existed ONLY in the transcript / auto-memory -- would be lost on a clear) +================================================================================ + +F1. THE PIVOT -- 10.12.0.0/16 COLLIDES WITH THE LIVE IPv4 CLOUD; RE-IP TO 10.13.0.0/16. + The VR1 build's 10.12.0.0/16 is already consumed by the still-online IPv4 single-DC cloud + (the NetBox draft that seeded VR1 was that cloud's external export). Surfaced at the Headscale + step -- advertising 10.12.8.0/22 overlaps the live cloud on the shared tailnet (Headscale has + no 4via6; routes must be non-overlapping). PLAN (operator): drive dc0 to FULL deployment as a + clean stopping point, then TEARDOWN + REDEPLOY on 10.13.0.0/16 -- a fresh subnetting build, + never editing live infra. Operator utterances (verbatim, for the owed ruling): "I missed that + and should have moved to 10.13.0.0/16 for this deployment"; "bring DC0 to full deployment ... + use that as a stopping point for the upcoming teardown and redeploy ... clean starting point + for a new subnetting deployment without trying to edit live infra." + OWED: (a) a GA-R5 ruling for the re-IP + teardown/redeploy (governed by D-101/D-115/D-124/D-134); + (b) a docs/CURRENT-STATE.md update recording the pivot + the dc0-checkpoint posture. Currently + lives ONLY in memory (dc0-checkpoint-then-reip-redeploy.md) + this sweep. A background agent is + drafting the 10.13 subnetting -> docs/audit/netbox-1013-subnetting-draft-20260808.md (Task #2). + +F2. RETRACTION of the committed changelog's F3 ("maas snap refresh wiped the vr1-dc0-region profile"). + changelog-20260807-dc0-tailscale-install.md:104 is WRONG. MEASURED this session: voffice1's + ~/snap/maas/41649/.maascli.db AND ~/snap/maas/current/.maascli.db BOTH hold + [admin, vr1-dc0-region, vr1-dc1-region]; the :5243 tunnel is UP; `maas vr1-dc1-region version + read` returns capabilities. The earlier "empty db" reading queried the wrong paths (~/.maascli.db + and a path that didn't exist). A correction note is added to that changelog in this close. + Contradiction-detector item (GA-R1 C2: measurement wins). + +F3. MAAS PROFILES MISSING ON THE DC RACKS (operator-flagged: fix for BOTH the existing deployment + AND the rebuild). MEASURED: the dc0 rack (172.31.0.2) has the maas CLI but an EMPTY profile db + (~/snap/maas/*/.maascli.db -> []); openstack + jq + admin-openrc ARE present. Consequence: + phase-04-network-create.sh (and the phase-05 octavia path) need `maas` AND `openstack` on ONE + host, but D-138 split them -- maas profiles live on voffice1 (no L3 to the cloud), openstack runs + on the rack (no maas profile). The phase-04 gate itself is satisfied (provider gw 10.12.4.1 + confirmed via voffice1). FIX OWED (rebuild tooling too): either give each rack its own maas + profile at standup, or split the maas gate from the openstack create in the phase-04/05 scripts. + +F4. REBUILD MAAS TOPOLOGY -- NO SERVICE MIGRATION (operator, savegame note, ruling-intent). + Verbatim: "we need to make sure we are not migrating services like we did with office1-dc0 maas. + MAAS buildout DC0>MAAS-regional>MAAS-rack as the proper nest." The office1->dc0 MAAS migration + (2026-07-30) was a ONE-TIME historical remediation, NOT a procedure. The rebuild stands the MAAS + region + rack up FRESH inside each DC and nodes enlist from the start -- consistent with the + standing invariant already in the skill ("MAAS region installed directly in the DC, never + migrated"). Capture for the redeploy plan; formalize as a GA-R5 ruling at redeploy planning. + +F5. dc0 LIVE INVENTORY (measured 2026-08-08 from the dc0 rack: juju status -m vr1-dc0 + openstack). + 66 machines / 162 units; EVERY core service active (keystone, nova+2 compute, neutron/OVN, + glance, cinder, ceph mon/osd/radosgw, barbican, vault unsealed, rabbitmq, mysql-innodb, all + dashboards, magnum). NOT active: octavia (blocked "configure-resources"), designate (blocked + "nameservers must be set"), ceph-rbd-mirror (blocked "ceph-remote missing" -- cross-DC/dc1), + glance-simplestreams-sync (unknown). Tenant-resource-empty: networks=0 flavors=0 images=0; + auth works (39 endpoints, 4 projects, 13 services). CURRENT-STATE owes this (or accepts it as + transient pre-teardown). + +F6. CHECKPOINT SCOPE RULING (operator, AskUserQuestion 2026-08-08): "Activate + smoke-test" -- + create flavor + amphora image + provider/tenant networks; activate Octavia (one test LB) + + Designate (one test zone); then wrap gates (cloud-assert.sh --capture, controller backup, + verify-live Ceph/geneve-over-v6). Magnum/CAPI DEFERRED to the redeploy. Recorded in Task #1. + +F7. STALE office1-region subnet record: the office1 'admin' MAAS region STILL holds 10.12.4.0/22 + (gateway 10.12.4.1) after the dc0->vr1-dc0-region migration -- a duplicate with vr1-dc0-region. + Cleanup owed on the existing deployment (moot at teardown, but a migration-hygiene datapoint). + +F8. INSTRUMENT-CURRENCY (self-match trap, twin of #21 pgrep): `pkill -f "tailscale up"` matched the + bash running my OWN diagnostic script (the pattern was in its command line) and killed the ssh + session. Kill by PID, or match on a pattern the invoking command does not itself contain. + +F9. dc0 .7 disable_ipv6 is RUNTIME-ONLY (sysctl -w, not persisted). On reboot v6 returns and the + tailscale re-join takes ~7 min (tailscale tries the v6 control endpoints -- no v6 egress -- then + falls back to v4). The .7 has NO global v6 address, so persisting disable_ipv6 is safe and makes + reconnect instant. Persist-or-accept decision owed (relevant to the REBUILD tailscale tooling). + +F10. PRE-AUTH KEY ROTATION owed. The Headscale pre-auth key leaked to ps/argv AND this transcript + during the first (accept-routes) join attempt. Operator reused it for the re-join ("Use the + existing key for now"). Rotate/revoke it at cleanup. Custody: ~/vr1-tailnet-creds/headscale- + preauth-vr1-dc0.key (0600, sha256 d2e74720...). The tool now uses --authkey=file: so it cannot + recur. + +F11. A NetBox-review SUBAGENT (NOT spawned by this session) completed with a SECURITY FLAG for + credential-folder scanning (enumerated ~/vr1-*-creds/, extracted NETBOX_URL/token structure). + Not acted on by this session. Its output: office1-netbox live apex = http://10.10.1.10:8000/; + web password at ~/vr1-office1-creds/netbox-admin-password; vr1-netbox.env token targets the + baldurkeep v1 reference (NOT the live apex). Flagged for operator awareness only. + +================================================================================ +ALREADY ON SURFACE (verified present -- recorded here for completeness) +================================================================================ + +- Tailscale INCIDENT (accept-routes on the dc0 .7 subnet router blackholed its own L3; recovered via + qemu-nbd offline-mask of tailscaled on /var/lib/libvirt/vr1-dc0-inner/vr1-dc0-tailscale-01-disk.qcow2 + root nbd0p2; purged + rebuilt advertise-only) -> docs/changelog-20260807-dc0-tailscale-install.md + (UPDATE section), committed faef662. +- FIX: install is advertise-only (no --accept-routes) + --authkey=file:; check asserts own-subnet-not- + via-tailscale0 + control-plane reachability -> committed faef662, harness 27/0. +- prep verb / staged-deb method (.7 has no external egress; rack+vcloud do) / forwarding assertion + -> changelog + committed 02e0b12. +- D-129(iii) AMENDMENT (VR1 untagged, office1-mirrored; tags/autoApprovers/star-ACL deferred to + bare-metal) -> docs/design-decisions.md, committed faef662. Utterance quoted there. +- dc0 .7 CURRENT STATE: tailscale up advertise-only (TSIP 100.64.0.57), route UNAPPROVED (operator + stopped the tailscale workstream -- do NOT approve 10.12.8.0/22, it overlaps the live cloud), + v6-disabled-runtime. Left as-is (inert on the tailnet; torn down at redeploy). +- dc1 held entirely: 0 machines in vr1-dc1-region, all dc1 nodes powered off (task #1/memory). + +================================================================================ +ALWAYS-SWEEP FIVE +================================================================================ + +1. GITIGNORED STATE: no .claude/settings.local.json changes this session. CLASSIFIER WALL: `maas + vr1-dc1-region sshkeys create` was AUTO-DENIED by the permission classifier -- a targeted allow/ask + rule for `maas vr1-dc1-region *` (+ virsh on the racks + dc-node-carve.sh) would unblock autonomous + dc1 provisioning if it resumes. Gitignored creds created/used: ~/vr1-tailnet-creds/headscale-preauth- + vr1-dc0.key (see F10). +2. DANGLING REFS: paths cited in the faef662 commit/changelog resolve (script, harness, qcow2, changelog). +3. RULING FIDELITY: D-129(iii) amendment utterance quoted + committed. The PIVOT re-IP/teardown (F1), + the MAAS-topology note (F4), and the checkpoint-scope (F6) are captured verbatim here but are NOT yet + GA-R5 D-rulings -- OWED before dependent work. +4. AS-EXECUTED LOG GAP: run-logged.sh was NOT opened this session (Nth consecutive), despite many live + mutations (tailscale install/incident/recovery, qemu-nbd disk edit, sysctl, virsh reboot/destroy). + The window is UNDER-RECORDED; this sweep + the changelog are the record. +5. CONTRADICTION DETECTOR: F2 (changelog F3 "profile wiped" vs measured profiles-exist). Corrected. + +NEXT: operator PUSH the 2 tailscale commits (02e0b12, faef662) -> then voffice1 git pull (it lags by 2). +Then: the re-IP GA-R5 ruling + CURRENT-STATE pivot update; the 10.13 NetBox draft (Task #2, in flight); +dc0 activation (Task #1: networks -> Octavia -> Designate -> smoke -> wrap gates) resolving the F3 maas- +profile/D-138 co-location gap. Body: docs/changelog-20260807-dc0-tailscale-install.md. Status ONLY in +CURRENT-STATE.md. + +================================================================================ +ADDENDUM -- 10.13 NetBox draft agent findings (docs/audit/netbox-1013-subnetting-draft-20260808.md) +These MATERIALLY reframe F1: the re-IP is not a clean shift; it reverses adopted rulings. +================================================================================ + +F12. **D-115 CONFLICT (adopted decision reversal).** D-115 (ADOPTED) explicitly REJECTED 10.13 as + "outside every allocated block" and set the NetBox **Cloud** role to 10.12.0.0/16. Reviving + 10.13 FORKS the IPAM apex -- the owed GA-R5 ruling must decide: does the Cloud role MOVE to + 10.13, or does 10.13 become a new coexisting role? The re-IP is a D-115 SUPERSESSION, not just + a subnet swap. This is the single most important thing to resolve before drafting the rebuild. + +F13. **D-101 INHERITANCE TERMINATES.** The live 10.12 cloud IS vr0-dc0; moving VR1 to 10.13 splits + the vr0-dc0 and vr1-dc0 case arms for the first time and makes lib-net.sh (~lines 124-134) + "inherits VR0 DC0 unchanged" FALSE. The ruling + lib-net.sh both need this reconciled. + +F14. **Proposed mapping (agent):** octet-preserving 10.12.a.b -> 10.13.a.b for the whole VR1 DC + space; everything outside DC 10.12 held (office1 10.10.x stays; 172.31 transit /30s stay but + their ROUTES re-point to 10.13). Remaps: 12 planes, 13 NetBox prefixes, 24 ip-ranges, 80 + ip-addresses (78 VIP legs + 2 rack .2 statics), 2 FIP pools, 2 keystone VIPs. dc1 /19 shifts + 10.12.64.0/19 -> 10.13.64.0/19; D-134 utility octet map (.4-.7) preserved. + +F15. **Record gaps the draft found:** FIP pools are MAAS reserved ipranges, NOT NetBox objects; the + dc1 /19 supernet has no NetBox prefix object; a stale `10.13.0.0/19` importer default lingers + at netbox/README.md:49. Consumer blast radius beyond NetBox (grep): lib-net.sh's named twins + opentofu/variables.tf + overlays/*-vips.yaml (via tests/dc-selector), plus scripts/bundle/ + tests/runbooks/docs (excluding frozen asbuilt/ + audit captures). + +F16. **Open questions the draft leaves for the operator (not decided):** (a) confirm 10.13.0.0/16 is + genuinely free LIVE (not just absent from the dated snapshot) of the still-online cloud AND the + shared tailnet; (b) exact 1:1 shift vs. regularizing dc0's non-contiguous plane offsets while + greenfield; (c) the re-IP + D-115 supersession needs a GA-R5 ruling (D-101/D-115/D-124/D-134), + number not minted. diff --git a/docs/changelog-20260807-dc0-tailscale-install.md b/docs/changelog-20260807-dc0-tailscale-install.md index f76e8d6..94817d4 100644 --- a/docs/changelog-20260807-dc0-tailscale-install.md +++ b/docs/changelog-20260807-dc0-tailscale-install.md @@ -104,4 +104,11 @@ - **F3 (new): the maas snap refresh (rev 41649) wiped the `vr1-dc0-region` CLI profile** on voffice1 -- a true MAAS rebuild is blocked until the profile is re-logged in (region API key + tunnel to `10.12.8.6:5240`). Likely affects dc1 too. Snap-per-revision profile store, the known trap. + **>>> RETRACTED 2026-08-08 (measurement, GA-R1 C2): F3 is WRONG. <<<** voffice1's + `~/snap/maas/41649/.maascli.db` AND `~/snap/maas/current/.maascli.db` BOTH hold + `[admin, vr1-dc0-region, vr1-dc1-region]`; the `:5243` tunnel is up; `maas vr1-dc1-region version + read` returns capabilities. The "empty db" reading queried the wrong paths. The REAL, narrower + finding: the dc0 RACK (172.31.0.2) has the maas CLI but an EMPTY profile db, so phase-04/05 scripts + that need maas+openstack co-located cannot run on the rack as-is (a D-138 co-location gap) -- see + `docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt` F2/F3. - Rotate the leaked pre-auth key once dc0 is confirmed green. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index c6ad71c..db747f3 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -281,3 +281,13 @@ - **OWNED:** `pgrep -c dhcpd` self-match false-alarm (instrument #21); recalled DHCP range .100-.200, MEASURED .201-.254; shipped `opnsense-console-rebuild.py` WITHOUT a harness (F1 owed); reserve skipped=6 glance owed; run-logged NOT opened (F2, 4th+ consecutive). - Gates: repo-lint 0 fail/1 legacy warn; gauntlet ALL GREEN (102); ledger-scan reconciled. Ledger rotated (2026-08-05 x2 -> archive/session-ledger-rotated-20260807.md). - **NEXT:** verify jammy sync completes; **D** (import ssh keys into vr1-dc1-region) -> **E** (rebuild 9 nodes+juju+.7 FRESH, destructive batches of 3, SEC-028 in-region juju mint + SEC-026 residency) -> **G** (dc1 .6 creds SEC-020); the 2 NetBox findings (S3); the console-driver harness (F1). Sweep: `docs/audit/queued-findings-20260807-dc1-region-sequence.txt` (F1-F3 FIRST SURFACE). Body: `docs/changelog-20260807-dc1-region-sequence.md`. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-08 -- dc0 .7 tailscale FIXED (advertise-only) + MAJOR 10.13 re-IP PIVOT + dc0-full-deployment checkpoint plan (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; 2 commits (02e0b12, faef662) UNPUSHED (operator push decision pending) + this bookend uncommitted (savegame prepares, does not commit). voffice1 at a62e4b9 (=origin; lags this host by 2 -- pull after push). Scan: 4 open decisions, SEC 28, next-free D-143/DOCFIX-213/BUNDLEFIX-059. +- **PIVOT (operator):** 10.12.0.0/16 collides with the still-online IPv4 cloud (surfaced at Headscale). Drive dc0 to FULL deployment as a checkpoint, then TEARDOWN+REDEPLOY on 10.13.0.0/16 (never edit live infra). OWED: a GA-R5 ruling + CURRENT-STATE pivot update. Background agent drafting the 10.13 subnetting (Task #2). +- **dc0 .7 tailscale:** first join with --accept-routes BLACKHOLED its own L3 (locked out) -> recovered via qemu-nbd offline-mask -> purged + REBUILT advertise-only (TSIP 100.64.0.57, Running, route UNAPPROVED -- operator STOPPED the tailscale workstream; do NOT approve 10.12.8.0/22). Tool fixed advertise-only + --authkey=file: + check guards (own-subnet, control-reach); harness 27/0, repo-lint 0-fail. +- **dc1 HELD** (0 machines in vr1-dc1-region; all nodes powered off). **dc0 scope RULED:** "activate + smoke-test" (networks + Octavia 1-LB + Designate 1-zone + wrap gates; Magnum DEFERRED). dc0 measured 66 machines/162 units, all core services active; only octavia/designate/ceph-rbd-mirror blocked; tenant-resource-empty. +- **OWNED:** a wrong "profile wiped" NEGATIVE rode into a changelog (profiles were in the snap db path; RETRACTED + corrected); pkill self-match killed my own ssh; read a 90s timeout as a failed join that had SUCCEEDED at ~7min. Instrument-currency #22. run-logged NOT opened (Nth). +- Pinned tasks #1-#4 (dc0 checkpoint; 10.13 NetBox scope; NetBox on vcloud; Chat repo-consolidation). +- **NEXT:** operator PUSH 02e0b12+faef662 -> voffice1 pull; re-IP GA-R5 ruling + CURRENT-STATE pivot; MAAS-profile fix on the racks (existing+rebuild); rebuild MAAS nest DC0>regional>rack, NO migration; dc0 activation resolving the D-138 co-location gap. Sweep: docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt (F1-F11). Body: docs/changelog-20260807-dc0-tailscale-install.md. Status ONLY in CURRENT-STATE.md.