Newer
Older
openstack-caracal-dc-dc / docs / audit / queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt
QUEUED FINDINGS -- session 2026-08-08 (dc0 tailscale install incident + the 10.12->10.13 re-IP pivot)
Sweep method: model read the session, grepped each candidate. FIRST SURFACE items lead.
Status authority is docs/CURRENT-STATE.md; this file is a sweep record, not status.

================================================================================
FIRST SURFACE (existed ONLY in the transcript / auto-memory -- would be lost on a clear)
================================================================================

F1. THE PIVOT -- 10.12.0.0/16 COLLIDES WITH THE LIVE IPv4 CLOUD; RE-IP TO 10.13.0.0/16.
    The VR1 build's 10.12.0.0/16 is already consumed by the still-online IPv4 single-DC cloud
    (the NetBox draft that seeded VR1 was that cloud's external export). Surfaced at the Headscale
    step -- advertising 10.12.8.0/22 overlaps the live cloud on the shared tailnet (Headscale has
    no 4via6; routes must be non-overlapping). PLAN (operator): drive dc0 to FULL deployment as a
    clean stopping point, then TEARDOWN + REDEPLOY on 10.13.0.0/16 -- a fresh subnetting build,
    never editing live infra. Operator utterances (verbatim, for the owed ruling): "I missed that
    and should have moved to 10.13.0.0/16 for this deployment"; "bring DC0 to full deployment ...
    use that as a stopping point for the upcoming teardown and redeploy ... clean starting point
    for a new subnetting deployment without trying to edit live infra."
    OWED: (a) a GA-R5 ruling for the re-IP + teardown/redeploy (governed by D-101/D-115/D-124/D-134);
    (b) a docs/CURRENT-STATE.md update recording the pivot + the dc0-checkpoint posture. Currently
    lives ONLY in memory (dc0-checkpoint-then-reip-redeploy.md) + this sweep. A background agent is
    drafting the 10.13 subnetting -> docs/audit/netbox-1013-subnetting-draft-20260808.md (Task #2).

F2. RETRACTION of the committed changelog's F3 ("maas snap refresh wiped the vr1-dc0-region profile").
    changelog-20260807-dc0-tailscale-install.md:104 is WRONG. MEASURED this session: voffice1's
    ~/snap/maas/41649/.maascli.db AND ~/snap/maas/current/.maascli.db BOTH hold
    [admin, vr1-dc0-region, vr1-dc1-region]; the :5243 tunnel is UP; `maas vr1-dc1-region version
    read` returns capabilities. The earlier "empty db" reading queried the wrong paths (~/.maascli.db
    and a path that didn't exist). A correction note is added to that changelog in this close.
    Contradiction-detector item (GA-R1 C2: measurement wins).

F3. MAAS PROFILES MISSING ON THE DC RACKS (operator-flagged: fix for BOTH the existing deployment
    AND the rebuild). MEASURED: the dc0 rack (172.31.0.2) has the maas CLI but an EMPTY profile db
    (~/snap/maas/*/.maascli.db -> []); openstack + jq + admin-openrc ARE present. Consequence:
    phase-04-network-create.sh (and the phase-05 octavia path) need `maas` AND `openstack` on ONE
    host, but D-138 split them -- maas profiles live on voffice1 (no L3 to the cloud), openstack runs
    on the rack (no maas profile). The phase-04 gate itself is satisfied (provider gw 10.12.4.1
    confirmed via voffice1). FIX OWED (rebuild tooling too): either give each rack its own maas
    profile at standup, or split the maas gate from the openstack create in the phase-04/05 scripts.

F4. REBUILD MAAS TOPOLOGY -- NO SERVICE MIGRATION (operator, savegame note, ruling-intent).
    Verbatim: "we need to make sure we are not migrating services like we did with office1-dc0 maas.
    MAAS buildout DC0>MAAS-regional>MAAS-rack as the proper nest." The office1->dc0 MAAS migration
    (2026-07-30) was a ONE-TIME historical remediation, NOT a procedure. The rebuild stands the MAAS
    region + rack up FRESH inside each DC and nodes enlist from the start -- consistent with the
    standing invariant already in the skill ("MAAS region installed directly in the DC, never
    migrated"). Capture for the redeploy plan; formalize as a GA-R5 ruling at redeploy planning.

F5. dc0 LIVE INVENTORY (measured 2026-08-08 from the dc0 rack: juju status -m vr1-dc0 + openstack).
    66 machines / 162 units; EVERY core service active (keystone, nova+2 compute, neutron/OVN,
    glance, cinder, ceph mon/osd/radosgw, barbican, vault unsealed, rabbitmq, mysql-innodb, all
    dashboards, magnum). NOT active: octavia (blocked "configure-resources"), designate (blocked
    "nameservers must be set"), ceph-rbd-mirror (blocked "ceph-remote missing" -- cross-DC/dc1),
    glance-simplestreams-sync (unknown). Tenant-resource-empty: networks=0 flavors=0 images=0;
    auth works (39 endpoints, 4 projects, 13 services). CURRENT-STATE owes this (or accepts it as
    transient pre-teardown).

F6. CHECKPOINT SCOPE RULING (operator, AskUserQuestion 2026-08-08): "Activate + smoke-test" --
    create flavor + amphora image + provider/tenant networks; activate Octavia (one test LB) +
    Designate (one test zone); then wrap gates (cloud-assert.sh --capture, controller backup,
    verify-live Ceph/geneve-over-v6). Magnum/CAPI DEFERRED to the redeploy. Recorded in Task #1.

F7. STALE office1-region subnet record: the office1 'admin' MAAS region STILL holds 10.12.4.0/22
    (gateway 10.12.4.1) after the dc0->vr1-dc0-region migration -- a duplicate with vr1-dc0-region.
    Cleanup owed on the existing deployment (moot at teardown, but a migration-hygiene datapoint).

F8. INSTRUMENT-CURRENCY (self-match trap, twin of #21 pgrep): `pkill -f "tailscale up"` matched the
    bash running my OWN diagnostic script (the pattern was in its command line) and killed the ssh
    session. Kill by PID, or match on a pattern the invoking command does not itself contain.

F9. dc0 .7 disable_ipv6 is RUNTIME-ONLY (sysctl -w, not persisted). On reboot v6 returns and the
    tailscale re-join takes ~7 min (tailscale tries the v6 control endpoints -- no v6 egress -- then
    falls back to v4). The .7 has NO global v6 address, so persisting disable_ipv6 is safe and makes
    reconnect instant. Persist-or-accept decision owed (relevant to the REBUILD tailscale tooling).

F10. PRE-AUTH KEY ROTATION owed. The Headscale pre-auth key leaked to ps/argv AND this transcript
    during the first (accept-routes) join attempt. Operator reused it for the re-join ("Use the
    existing key for now"). Rotate/revoke it at cleanup. Custody: ~/vr1-tailnet-creds/headscale-
    preauth-vr1-dc0.key (0600, sha256 d2e74720...). The tool now uses --authkey=file: so it cannot
    recur.

F11. A NetBox-review SUBAGENT (NOT spawned by this session) completed with a SECURITY FLAG for
    credential-folder scanning (enumerated ~/vr1-*-creds/, extracted NETBOX_URL/token structure).
    Not acted on by this session. Its output: office1-netbox live apex = http://10.10.1.10:8000/;
    web password at ~/vr1-office1-creds/netbox-admin-password; vr1-netbox.env token targets the
    baldurkeep v1 reference (NOT the live apex). Flagged for operator awareness only.

================================================================================
ALREADY ON SURFACE (verified present -- recorded here for completeness)
================================================================================

- Tailscale INCIDENT (accept-routes on the dc0 .7 subnet router blackholed its own L3; recovered via
  qemu-nbd offline-mask of tailscaled on /var/lib/libvirt/vr1-dc0-inner/vr1-dc0-tailscale-01-disk.qcow2
  root nbd0p2; purged + rebuilt advertise-only) -> docs/changelog-20260807-dc0-tailscale-install.md
  (UPDATE section), committed faef662.
- FIX: install is advertise-only (no --accept-routes) + --authkey=file:; check asserts own-subnet-not-
  via-tailscale0 + control-plane reachability -> committed faef662, harness 27/0.
- prep verb / staged-deb method (.7 has no external egress; rack+vcloud do) / forwarding assertion
  -> changelog + committed 02e0b12.
- D-129(iii) AMENDMENT (VR1 untagged, office1-mirrored; tags/autoApprovers/star-ACL deferred to
  bare-metal) -> docs/design-decisions.md, committed faef662. Utterance quoted there.
- dc0 .7 CURRENT STATE: tailscale up advertise-only (TSIP 100.64.0.57), route UNAPPROVED (operator
  stopped the tailscale workstream -- do NOT approve 10.12.8.0/22, it overlaps the live cloud),
  v6-disabled-runtime. Left as-is (inert on the tailnet; torn down at redeploy).
- dc1 held entirely: 0 machines in vr1-dc1-region, all dc1 nodes powered off (task #1/memory).

================================================================================
ALWAYS-SWEEP FIVE
================================================================================

1. GITIGNORED STATE: no .claude/settings.local.json changes this session. CLASSIFIER WALL: `maas
   vr1-dc1-region sshkeys create` was AUTO-DENIED by the permission classifier -- a targeted allow/ask
   rule for `maas vr1-dc1-region *` (+ virsh on the racks + dc-node-carve.sh) would unblock autonomous
   dc1 provisioning if it resumes. Gitignored creds created/used: ~/vr1-tailnet-creds/headscale-preauth-
   vr1-dc0.key (see F10).
2. DANGLING REFS: paths cited in the faef662 commit/changelog resolve (script, harness, qcow2, changelog).
3. RULING FIDELITY: D-129(iii) amendment utterance quoted + committed. The PIVOT re-IP/teardown (F1),
   the MAAS-topology note (F4), and the checkpoint-scope (F6) are captured verbatim here but are NOT yet
   GA-R5 D-rulings -- OWED before dependent work.
4. AS-EXECUTED LOG GAP: run-logged.sh was NOT opened this session (Nth consecutive), despite many live
   mutations (tailscale install/incident/recovery, qemu-nbd disk edit, sysctl, virsh reboot/destroy).
   The window is UNDER-RECORDED; this sweep + the changelog are the record.
5. CONTRADICTION DETECTOR: F2 (changelog F3 "profile wiped" vs measured profiles-exist). Corrected.

NEXT: operator PUSH the 2 tailscale commits (02e0b12, faef662) -> then voffice1 git pull (it lags by 2).
Then: the re-IP GA-R5 ruling + CURRENT-STATE pivot update; the 10.13 NetBox draft (Task #2, in flight);
dc0 activation (Task #1: networks -> Octavia -> Designate -> smoke -> wrap gates) resolving the F3 maas-
profile/D-138 co-location gap. Body: docs/changelog-20260807-dc0-tailscale-install.md. Status ONLY in
CURRENT-STATE.md.

================================================================================
ADDENDUM -- 10.13 NetBox draft agent findings (docs/audit/netbox-1013-subnetting-draft-20260808.md)
These MATERIALLY reframe F1: the re-IP is not a clean shift; it reverses adopted rulings.
================================================================================

F12. **D-115 CONFLICT (adopted decision reversal).** D-115 (ADOPTED) explicitly REJECTED 10.13 as
     "outside every allocated block" and set the NetBox **Cloud** role to 10.12.0.0/16. Reviving
     10.13 FORKS the IPAM apex -- the owed GA-R5 ruling must decide: does the Cloud role MOVE to
     10.13, or does 10.13 become a new coexisting role? The re-IP is a D-115 SUPERSESSION, not just
     a subnet swap. This is the single most important thing to resolve before drafting the rebuild.

F13. **D-101 INHERITANCE TERMINATES.** The live 10.12 cloud IS vr0-dc0; moving VR1 to 10.13 splits
     the vr0-dc0 and vr1-dc0 case arms for the first time and makes lib-net.sh (~lines 124-134)
     "inherits VR0 DC0 unchanged" FALSE. The ruling + lib-net.sh both need this reconciled.

F14. **Proposed mapping (agent):** octet-preserving 10.12.a.b -> 10.13.a.b for the whole VR1 DC
     space; everything outside DC 10.12 held (office1 10.10.x stays; 172.31 transit /30s stay but
     their ROUTES re-point to 10.13). Remaps: 12 planes, 13 NetBox prefixes, 24 ip-ranges, 80
     ip-addresses (78 VIP legs + 2 rack .2 statics), 2 FIP pools, 2 keystone VIPs. dc1 /19 shifts
     10.12.64.0/19 -> 10.13.64.0/19; D-134 utility octet map (.4-.7) preserved.

F15. **Record gaps the draft found:** FIP pools are MAAS reserved ipranges, NOT NetBox objects; the
     dc1 /19 supernet has no NetBox prefix object; a stale `10.13.0.0/19` importer default lingers
     at netbox/README.md:49. Consumer blast radius beyond NetBox (grep): lib-net.sh's named twins
     opentofu/variables.tf + overlays/*-vips.yaml (via tests/dc-selector), plus scripts/bundle/
     tests/runbooks/docs (excluding frozen asbuilt/ + audit captures).

F16. **Open questions the draft leaves for the operator (not decided):** (a) confirm 10.13.0.0/16 is
     genuinely free LIVE (not just absent from the dated snapshot) of the still-online cloud AND the
     shared tailnet; (b) exact 1:1 shift vs. regularizing dc0's non-contiguous plane offsets while
     greenfield; (c) the re-IP + D-115 supersession needs a GA-R5 ruling (D-101/D-115/D-124/D-134),
     number not minted.