Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260727-stage5-phase0.md

Session changelog 2026-07-27 -- grounding-audit merge + Stage-5 Phase 0

ONE changelog per session (GA-R2/D1). Under blanket approval this is the review surface: every item states WHAT, WHY (evidence), and HOW TO REVERT.

No logged window was opened for this session (scripts/run-logged.sh was offered; the operator approved the steps directly). This file plus docs/audit/stage5-phase0-20260727.txt are therefore the ENTIRE as-executed record of five live mutations. Reverts below are stated with that in mind.

Status authority for everything here is docs/CURRENT-STATE.md (GA-R1). This file is session-scoped scratch and is NOT citable as status or decision authority.


1. Merged dc-dc-stage5-grounding-audit to main

What. Merge commit 607813b, 2 parents (NOT squashed), 33 commits. Recorded on main by follow-up commit 6495cfb -- a --no-ff merge cannot carry the doc edit, so this is the Stage-4 shape (6f5701d recorded by 1023596).

Why. Operator direction, exact utterance: "Merge to main, then start Phase 0". The branch carried 14 GA-R5 rulings and the audit record; precondition work must branch off a main that contains them. No stage opened or closed.

Evidence. git log -1 --pretty=%P -> two hashes. git branch --merged main listed the branch before deletion. Post-merge ON main: gauntlet ALL GREEN (81 harnesses), repo-lint 0 fail / 1 standing warn.

Revert. git revert -m 1 607813b on main (then revert 6495cfb). Prefer this over a reset -- main is pushed and shared.

2. Retired the audit branch (local + origin)

What. Deleted dc-dc-stage5-grounding-audit from origin and locally on vcloud. Last commit was 39e8988.

Why. Contained in main after item 1, and leaving it alive would hand the voffice1 clone a FRESH stale remote-tracking ref during item 3 -- the exact condition that produced the 105-commit problem being fixed.

Evidence. Containment re-checked immediately before deletion; git branch -d (not -D) used, which refuses an unmerged branch. git ls-remote --heads origin afterwards -> main only.

Revert. git push origin 39e8988:refs/heads/dc-dc-stage5-grounding-audit. Nothing is lost meanwhile -- the commits are in main.

3. Advanced the voffice1 clone to main (precondition 0.1)

What. On voffice1:~/openstack-caracal-dc-dc: git fetch origin --prune then git switch main. Was 61c416e on dc-dc-g12-dc1-substrate, a branch deleted upstream; now 6495cfb.

Why. voffice1 is the D-128 Plane-2 host Stage 5 EXECUTES from, and it was 105 commits behind with both dc1 overlays ABSENT and bundle.yaml still the VR0 4-node hyperconverged layout -- readiness blocker 1, "you would deploy the wrong topology". R10 ruled this fix. A plain git pull could not work (tracked branch gone upstream); the shape is fetch + switch per finding L5-4.

Evidence / the safety argument. Both DCs' inner tfstate -- the substrate's state-of-record -- lives INSIDE that working tree, so this was not a routine checkout. PROVEN before the switch: every state artifact is git check-ignore-IGNORED, and origin/main tracks an identical file set at those paths (so no collision, and the one untracked-not-ignored file, vr1-dc1-substrate/.terraform.lock.hcl, is untracked on main too). AFTER: both tfstate sha256s BYTE-IDENTICAL to before (2e140b74..., aa91f103...); HEAD == origin/main asserted; both dc1 overlays now present; bundle.yaml now the 9-node role-separated layout.

Revert. ssh voffice1 'cd ~/openstack-caracal-dc-dc && git switch -c dc-dc-g12-dc1-substrate 61c416e'. The ignored state artifacts are untouched by either direction, which is the point of the check above.

4. Deleted two stale local branches on voffice1 (precondition 0.3)

What. dc-dc-g12-dc1-substrate (was 61c416e) and dc-dc-stage3-phase2-dc-substrate (was 57836b1). Three stale remote-tracking refs were pruned by item 3's --prune.

Why. Both tracked branches deleted upstream. The record named ONE stale local branch; there were two.

Evidence. git rev-list --count origin/main..<branch> -> 0 for both (fully contained, nothing unique lost). git branch -d used, which refuses if unmerged.

Revert. ssh voffice1 'cd ~/openstack-caracal-dc-dc && git branch dc-dc-g12-dc1-substrate 61c416e && git branch dc-dc-stage3-phase2-dc-substrate 57836b1'.

5. Installed the openstack client on voffice1 (precondition 0.2)

What. sudo apt-get install -y python3-openstackclient on voffice1. Landed python3-openstackclient 6.6.0-0ubuntu2, python3-openstacksdk 3.0.0-0ubuntu2, python3-novaclient 2:18.5.0-0ubuntu1. 68 packages newly installed, 0 upgraded, 0 removed.

Why. Readiness blocker 2: the client was absent on BOTH hosts while ten Stage-5/6/7 scripts invoke it. voffice1 is where D-128 runs them.

Evidence -- the method was measured, not assumed. The snap was REFUTED: openstackclients has NO Caracal channel (newest stable zed, 2023-03; latest/stable is xena, 2021), and docs/design-decisions.md:638 already records its home-only confinement trap. 6.6.0 IS the Caracal 2024.1 client, verified upstream rather than from memory. Verified behaviourally after install: openstack --version -> openstack 6.6.0; --help exit 0; server list fails CLEANLY on absent auth config rather than crashing.

Revert. ssh voffice1 'sudo apt-get purge -y python3-openstackclient python3-openstacksdk python3-osc-placement && sudo apt-get autoremove -y'. Scope the autoremove by reviewing /var/log/apt/history.log for this transaction first -- 68 packages were pulled and some (e.g. python3-pil, libopenjp2-7) may be shared with other consumers on that host.

Note this is now a PIN. Recorded in docs/CURRENT-STATE.md section 7, which carried no row for this component before.


Findings LOGGED, NOT ACTIONED (hard rule 1 -- outside Phase 0's ruled scope)

Full detail in docs/audit/stage5-phase0-20260727.txt.

  • P0-1 the gauntlet is HOST-DEPENDENT. 2/81 FAILED on voffice1 on the IDENTICAL commit that is ALL GREEN (81) on vcloud. Neither is a tree defect. opentofu-validate passes every sub-check and still reports FAIL because tofu fmt -check -recursive walks the FILESYSTEM and trips on a GITIGNORED file that exists only on voffice1. site-headend-install:36 asserts a --dry-run installed no snap by snapshotting the host without comparing to a pre-state -- on the real headend, where lxd/maas are installed by design, a false positive. Every GA-R6 stage close in this project cites a gauntlet figure measured on vcloud only.
  • P0-2 preflight P5 is HOST-BLIND. 34 findings on voffice1 vs 7 on vcloud, same 82-row matrix and commit. The matrix's jumphost role has no assertion about WHICH host is the jumphost, so those locations re-point to whatever host runs the checker. Surfaces here as false RED; the same hole yields false GREEN for a credential present on the wrong host under a matching name.
  • P0-3 opentofu/vr1-dc1-substrate/.terraform.lock.hcl is untracked on main while its dc0 twin IS tracked.
  • P0-4 no logged window opened for this session (see the header).
  • P0-5 lib-net.sh:37 declares a metal-admin gateway 10.12.8.1 that NO device holds -- measured 100% loss + INCOMPLETE ARP against a control ping to 10.12.4.1 at 0% loss. MAAS is right; the checker is wrong. This capture initially MIS-FILED it as covered by readiness item 3.7; it is not (3.7 is the VID-103/br-internal assertions). The dc1 arm carries the same shape at 10.12.68.1.

Question QUEUED for the operator (one GA-R5 exchange; no D-number assigned)

Which host is authoritative for the gates? R10 ruled preflight belongs on voffice1; R15(3) rules preflight must stop failing open. Executed together as ruled, and given P0-2, preflight on voffice1 becomes PERMANENTLY UNPASSABLE -- its 34 findings are host artifacts. Similarly R15(2) pins a harness MANIFEST, which catches renames but does nothing about P0-1, so R15(2) as ruled still leaves voffice1 red. GA-R3 resolves doubt DOWN to OPS; whether "which host is authoritative for a gate" clears the A1 Roosevelt test is the operator's call.

Standing consequence either way: Stage 5 executes from voffice1 and the gauntlet cannot be green there until P0-1 is resolved, so any GA-R6 close citing a gauntlet figure measured there inherits a known red -- the citation must name its host.