Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260806-step34-g3-probe.md
@JANeumatrix JANeumatrix 4 hours ago 9 KB substrate: add the .7 Tailscale subnet-router VM to both DC node maps
Raw Blame History

Changelog 2026-08-06 -- phase-03 Step 3.4: G3 domain-manager probe (named check + live PASS)

Session: continue Stage 5 (dc0) from the 2026-08-06 part-3 close. Branch dc-dc-stage5-preconditions. Under blanket approval the changelog is the review surface: each item states WHAT, WHY (evidence), and HOW TO REVERT.

Operator grounding this session (applied, not a ruling to adopt): (1) this is the D-121 HA scale-up deployment -- reconcile any single-DC/non-HA-era step or gate against the current HA census before grading it; (2) the standing IPAM ruling is IPv6-PRIMARY (v6 wherever possible, v4/dual-stack only where forced, per D-101/D-139); D-141's v4-active/v6-reserved is the NARROW, necessity-driven state of the container-hosted API-charm VIPs (juju LP #1723240), not a cloud-wide posture. Recorded here so the sequence's Decision B (Step 12 Ceph/geneve-over-v6) is framed as "verify the v6 target per-layer", not "defer v6".

Item 1 -- NEW gate tool: scripts/g3-domain-manager-probe.sh + its harness

WHAT: added scripts/g3-domain-manager-probe.sh and tests/g3-domain-manager-probe/run-tests.sh. The script is the appendix-C C.4 behavioral probe (phase-03 Step 3.4 stage 2, gate G3) turned into a NAMED EXECUTABLE CHECK: as a domain-scoped manager it proves PASS (create user + project in-domain, grant member + load-balancer_member), DENY (grant an unmanaged role manager -> refused; read admin_domain -> refused), and UNAFF (cloud-admin sees all domains). All fixtures live under a unique dated throwaway domain g3-probe-<UTC>; a trap tears the domain + contents down on ANY exit. Exit 0 PASS / 11 PASS-case miss / 12 DENY LEAK (security) / 14 precondition. Self-contained (sources no lib-*), so it pipes cleanly to a rack via bash -s.

WHY: hard rule 4 -- no tested artifact existed for this job (the probe was a manual runbook walk only); GA-R6 wants Step 3.4 to close on a named check whose output the commit cites; and Stage 5 is per-DC, so dc1's Step 7 reuses this identical probe. Grounded in the real policy: policies/domain-manager-policy.yaml:103 identity:create_grant = (domain_manager_grant AND is_domain_managed_role) OR cloud_admin ... -- so granting manager/Admin is refused and granting member/load-balancer_member is allowed, exactly what the probe asserts.

HARNESS: 12/12, every exit path (0/11/12/14) has a failing-direction fixture; teardown proven on both the happy and the failure path; all three groups proven to run. Writing it caught two real defects before delivery: (a) the happy case was run inside $(...) so its pass/fail accounting happened in a subshell and was LOST -- a genuine happy failure would have been invisible (fixed: account in the parent shell, capture output via a file); (b) a fake domain list emitted non-hex ids while the script counts hex (real keystone shape) -- the subshell fix immediately exposed it as a real happy-case exit 11. gauntlet ALL GREEN (100).

REVERT: git rm scripts/g3-domain-manager-probe.sh && git rm -r tests/g3-domain-manager-probe && bash scripts/run-tests-all.sh --record-manifest (drops the manifest line too).

Item 2 -- tests/HARNESS-MANIFEST recorded 99 -> 100

WHAT: bash scripts/run-tests-all.sh --record-manifest added g3-domain-manager-probe (line 39). The drift check FAILED the gauntlet until recorded (present-but-unpinned); the diff was exactly one addition, zero removals. REVERT: covered by Item 1's re-record.

Item 3 -- LIVE G3 probe on dc0: PASS, zero residue

WHAT: ran ssh vr1-dc0-rack 'bash -s' < scripts/g3-domain-manager-probe.sh (operator-approved mutation). Result: G3 PASS -- persona works and is bounded, 7 ok / 0 fail, exit 0. Capture: docs/audit/g3-dc0-probe-20260806.txt. Teardown VERIFIED clean by a follow-up read-only check (openstack domain list back to admin_domain/service_domain/Default; no g3-mgr/g3-user/g3-proj).

WHY: closes phase-03 Step 3.4 stage 2. Stage 1 (override loaded) was verified read-only the same session: juju resources keystone shows policyd-override attached (rev 2026-08-04) and all THREE keystone units carry PO: Unit is ready -- note the app-status AGGREGATE read "Application Ready" (no PO:), so verifying at the UNIT level is what prevented a false "override missing" report (assert-before-check).

REVERT: none required -- the probe self-cleans and touched only objects it created; no persistent cloud change. (The capture file is evidence; remove with git rm if unwanted.)

Item 4 -- CURRENT-STATE: Step 3.4 recorded RESOLVED

WHAT: section-1 phase-03 block -- item (b) "Step 3.4 ... STILL NOT RUN" -> RESOLVED with the capture cited; header count reconciled. The phase-03 exit gate now turns on the Horizon reachable/login-works item (D-044 cookie + D-075 root-redirect per-rebuild + the VR0 nginx-repoint vs VR1 metal-admin-VIP-over-tailnet access-model reconciliation = the sequence's Decision C), which is measured + ruled next. REVERT: git checkout HEAD~1 -- docs/CURRENT-STATE.md (or revert the single hunk).

Gates at delivery

  • gauntlet ALL GREEN (100 harnesses); g3 confirmed EXECUTED in-gauntlet (12 pass / 0 fail).
  • repo-lint 0 fail / 1 legacy warn (D-001..018 ASCII carve-out).
  • ledger-scan: decisions + SEC unchanged; no D/DOCFIX/BUNDLEFIX number consumed (new tooling).

Item 5 -- Decision C: phase-03 Horizon reconciled to VR1; Step 3.3 splits to its own gate

WHAT: measured (read-only, dc0 rack) that BOTH dashboard VIPs serve Horizon login over HTTPS 200 + csrftoken (metal-admin 10.12.8.58 AND provider 10.12.4.58), that D-044/D-075 are NOT applied this rebuild, and that the cert SAN on 10.12.8.58 already includes IP:10.12.8.58 (F-CV3). Concluded (operator-directed reframing): D-044 (Secure-cookie override) + the VR0 external-nginx repoint are artifacts of the VR0 plain-HTTP proxy leg and would only WEAKEN the cookie under the VR1 direct-HTTPS-over-tailnet access model. Operator: "We need to pull the tailscale steps forward so we can close out horizon properly." -> phase-03 does NOT close this session; Step 3.3 (Horizon) splits into its own gate row (GA-R6/E3), gated on the per-DC Tailscale build + the vault root CA on the operator workstation (NOT done) + a browser login over the tailnet. REVERT: revert the CURRENT-STATE phase-03 (c)-clause hunk.

Item 6 -- D-129(iii) amendment: per-DC Tailscale operator-access rulings (a)-(d), BOTH DCs

WHAT: recorded four GA-R5 rulings (2026-08-07) as a D-129(iii) amendment (advisor-corrected: these are D-129 implementation sub-decisions, NOT a new D-143 -- D-129(iii) already says ACL/ key-custody/SEC-row land at implementation): (a) dedicated VM at utility .7 (10.12.8.7 / 10.12.68.7); (b) STAR operator->DC only (the Headscale ACL / security boundary); (c) SINGLE router, HA scale-up PINNED; (d) SNAT ON now, source-IP preservation PINNED. Operator directive "Lets plan and push to both DC0 and DC1 in this step" -> both DCs. Also: corrected the D-107 citation defect (D-107 is airgap/mirror/NTP, rules nothing about Tailscale; D-129(iii) governs) as a DOCFIX-in-amendment; extended D-134's standing octet map to .7 (D-134 AMENDMENT 2026-08-07); updated the gap-21 register row (rulings made, build in progress) and CURRENT-STATE. WHY: GA-R5 requires rulings committed before dependent work (the build). The four-rulings-before- build precondition (gap-21) is now discharged. REVERT: revert the D-129/D-134 amendment blocks + the gap-21 + CURRENT-STATE hunks (no code, no cloud change -- records only).

Still owed (the build, next -- NOT done this commit)

  • scripts/site-tailscale.sh <site> (install/check) + harness; the .7 VM per DC (inner-tofu for_each); Tailscale tagged join + advertise + SNAT-on; Headscale star ACL + autoApprovers (written BEFORE first advertise) + fix the Office1 untagged-node defect; SEC row (key custody); browser login confirm over tailnet (needs the vault root CA on the workstation).
  • Headscale server version remains UNMEASURED (operator has no access this session) -> tag + autoApprovers support verified EMPIRICALLY at apply time.

Item 7 -- site-tailscale.sh tooling (committed 2929257) + substrate .7 VM added (both DCs)

WHAT: (a) committed scripts/site-tailscale.sh + tests/site-tailscale (15/15, gauntlet 101 ALL GREEN, manifest 100->101) -- the node-side subnet-router tool. (b) Added the .7 Tailscale VM to BOTH substrate node maps: opentofu/vr1-dc0-substrate/main.tf ("vr1-dc0-tailscale-01") and vr1-dc1-substrate/main.tf ("vr1-dc1-tailscale-01"), each 2 vCPU / 2 GiB / 25 GiB (Office1 tailscale shape), macs=[] (pin from virsh domiflist post-apply, pre-enlistment). for_each keyed -> expect 1 add / 0 change / 0 destroy per DC. CAPACITY RE-GATED 2026-08-07 (assert, not estimate): dc-dc-whole-host-budget.py at overhead 34/14 (the +2GiB/+2vCPU per DC over the region-era 32/12) -> "RAM 874/1024 = 85%, VERDICT FIT, 150 GiB headroom" (vCPU 258/256 = 101% is a sim overcommit, not a hard ceiling; disk fine). tofu validate PASS (both substrate roots + all modules). STILL OWED (gated, on voffice1 -- the inner substrate root runs there, D-128): tofu plan (verify 1 add/0/0) -> gated apply -> pin MACs -> MAAS enlist/commission/deploy -> carve .7 metal-admin + provider-public legs -> install tailscale. Then the Headscale-side join (deferred, access). REVERT: remove the "vr1-dc{0,1}-tailscale-01" map entries (for_each keyed -> plans 1 destroy each if already applied; a pure map removal if not yet applied).