diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 517b05c..2833e65 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1591,6 +1591,29 @@ teardown, `juju status -m ` is the instrument; `juju models` counts are not.** **ENVIRONMENT IS NOW READY FOR `add-model` + `deploy`** except for the snap proxy, which is the one remaining build. + **SESSION SWEEP AT CLOSE: `docs/audit/queued-findings-20260731-stage5-deploy.txt` + (F1-F11, C1-C3, N1-N5). ELEVEN items were FIRST SURFACE** -- they existed only in the + transcript and would have been lost. The highest-consequence: **F1, that MAAS's own squid + ALREADY CONNECT-proxies `api.snapcraft.io` and would unblock the deploy today with nothing + built.** It was measured from the failing container itself, and **RULED AGAINST 2026-07-31 + (GA-R5), operator utterance verbatim: "No, the downsides are real and the upsides for + stability are more important then to just breeze past to keep the deployment going. We need + to complete the proxy work and use the one that will not be fragile."** [sic] The deciding + downsides: MAAS's squid applies NO destination restriction (pointing nodes at it hands the + node planes general HTTPS egress and makes D-107 true in letter, hollow in practice); its + config is MAAS-generated under a per-revision path so ACLs cannot be added and a snap + refresh can change it with no alarm; and it is the D-135 hidden-coupling shape while D-132's + region work is already touching MAAS. **Recorded because a future session WILL rediscover + that the MAAS proxy works and reach for it.** Also FIRST SURFACE: the failing apps are LXD + containers sourcing from METAL-ADMIN not the address `juju status` displays (an ACL on the + displayed address would have denied every client); ports 3128 AND 8000 already held by + MAAS's squid on both racks; the four reviewed bugs; the teardown instrument error; and that + a MAAS hostname rename on a `Deployed` machine is RECORD-ONLY. + **GA-R7 MEMORY REVIEW: one real violation found and corrected.** The + `multi-workstation-remote-control` memory asserted "never additions to `allow` for + mutations" -- an OPERATOR-POSTURE claim memory may not hold, and CONTRADICTED by a recorded + ruling (2026-07-30, "Add it to allow"). Re-pointed to an observation with the contradiction + recorded. The instrument-currency memory gained this session's two misreads. **memcached SCALED to 3 in `overlays/dc-ha-scaleup.yaml`** (operator-directed 2026-07-31); its exclusion comment is RE-POINTED rather than left stale. **`ceph-rbd-mirror` is PINNED, NOT SETTLED** -- gap register **item 22** carries four options and the recommendation diff --git a/docs/archive/session-ledger-rotated-20260731.md b/docs/archive/session-ledger-rotated-20260731.md new file mode 100644 index 0000000..54a8ff4 --- /dev/null +++ b/docs/archive/session-ledger-rotated-20260731.md @@ -0,0 +1,22 @@ +# Session-ledger summaries rotated out 2026-07-31 (GA-R4 rule 3 / F1) + +Moved VERBATIM from docs/session-ledger.md to hold the live file under its 300-line cap. +Each summary still points at its own archived full body; only the summaries moved. + +## SESSION CLOSE 2026-07-27 -- Phase 0, R15 gates, D-136, step 3 executed (bounded, GA-R4) + +- Merged the grounding-audit branch to `main` (`607813b`, recorded `6495cfb`), retired it; 29 + commits on `dc-dc-stage5-preconditions`. NO stage opened/closed. SEC 21; D 138/DOCFIX 205. +- PHASE 0: voffice1 off a 105-commit-stale retired branch (both DCs' inner tfstate lives there -- + proven gitignored, sha256 identical); `openstack` 6.6.0 (snap REFUTED: no Caracal channel). +- ALL THREE R15 GATES FIXED, each reproduced first: repo-lint PASSED over ZERO files on a typo'd + flag; preflight left "clear to deploy" on exits 127/126/130/3; the gauntlet pinned no names. +- RULINGS (GA-R5, quoted): no DC ordering + NOC deferred (F1); **D-136 ADOPTED option (D)**; v6 host + part mirrors the v4 octet (F4); dc1 FIP pool; node v6 = MAAS static; **D-101 GOVERNING RATIONALE**. +- STEP 3 EXECUTED both DCs: 12 v6 subnets, 24 bands + 2 FIP pools, apex 3->27 ranges / 4->160 + addresses (156 VIPs), node carve 108 links. D-134 bands + D-020/R11 VIPs were RULED-BUT-NEVER-BUILT, + now artifacts. 18 Ready unchanged. Found: gauntlet HOST-DEPENDENT, preflight P5 HOST-BLIND. +- OWNED: called step 3 "complete" twice while the node layer was unbuilt; mis-filed P0-5; built + the apex tool against a dump so it matched zero live. All corrected on-surface. +- NEXT: gate host-authority; `provider-bundle-check` ARITY gap (imminent); **voffice1 back to + `main` at merge**; then the renderer. Body: `changelog-20260727-stage5-phase0.md`. diff --git a/docs/audit/queued-findings-20260731-stage5-deploy.txt b/docs/audit/queued-findings-20260731-stage5-deploy.txt new file mode 100644 index 0000000..10a2eca --- /dev/null +++ b/docs/audit/queued-findings-20260731-stage5-deploy.txt @@ -0,0 +1,149 @@ +SESSION SWEEP 2026-07-31 -- Stage 5 dc0 deploy: prefer-ipv6 research through teardown +===================================================================================== +Method (ruled 2026-07-31): the session was read back in full, every candidate +enumerated, then each GREPPED against the repo. Items marked FIRST SURFACE had no hit +and would have been lost on a clear. FIRST SURFACE items lead the file. + +A phrase-exact grep produced false negatives on items whose CONTENT is recorded under +different wording; each candidate was re-grepped on distinctive TOKENS before being +classified. That correction is itself the instrument-error class this repo keeps hitting. + +-------------------------------------------------------------------------------------- +FIRST SURFACE -- would have been lost +-------------------------------------------------------------------------------------- + +F1. MAAS's OWN SQUID ALREADY CONNECT-PROXIES api.snapcraft.io, AND IT WOULD UNBLOCK + THE DEPLOY TODAY WITH NOTHING BUILT. + Measured from the rack, the controller VM AND one of the failing containers. + Machines already use 10.12.8.6:8000 for apt via 90curtin-aptproxy. So + `snap-https-proxy=http://10.12.8.6:8000` is a live, working option. + **RULED AGAINST 2026-07-31 (GA-R5).** Operator utterance, verbatim: "No, the + downsides are real and the upsides for stability are more important then to just + breeze past to keep the deployment going. We need to complete the proxy work and + use the one that will not be fragile." [sic] + The downsides that decided it, all measured or structural: MAAS's squid applies NO + destination restriction, so pointing nodes at it hands the node planes general + HTTPS egress and makes D-107's "nodes reach NO internet directly" true in letter + and hollow in practice; its config is MAAS-generated under a per-revision + /var/snap/maas/current/ path, so ACLs cannot be added and a snap refresh can change + it with no alarm (the SEC-012 power-key fragility class); and making it load-bearing + for snaps is the D-135 hidden-coupling shape exactly, while D-132's per-DC region + work is already touching MAAS. + RECORDED BECAUSE IT WILL BE REDISCOVERED: a future session WILL notice the MAAS + proxy works and reach for it. This is the record that it was found, measured, and + deliberately declined. + +F2. THE FAILING APPS ARE LXD CONTAINERS AND THEY SOURCE FROM METAL-ADMIN, NOT THE + ADDRESS `juju status` DISPLAYS. + `juju status` prints mysql-innodb-cluster/0 at 10.12.12.116. Measured INSIDE the + container: eth0 10.12.8.122/22, eth1 10.12.12.116/22, `ip route get 10.12.8.4` -> + src 10.12.8.122, NO default route. An ACL built on the displayed address would have + denied every client the proxy exists for. Generalisable: for a juju LXD container, + the displayed address is not necessarily the source address for a given destination. + +F3. PORTS 3128 AND 8000 ARE ALREADY HELD, WILDCARD-BOUND, BY MAAS's OWN SQUID ON BOTH + RACKS. noble's squid.conf pins `http_port 3128`, /etc/default/squid is EMPTY, and + conf.d loads before `deny all` but cannot unset an http_port -- so the PACKAGED unit + can only FATAL on these hosts. Any future rack-side proxy must use a dedicated + instance on another port. This is why the build uses 3129. + +F4. FOUR REAL BUGS IN scripts/dc-snap-proxy.sh, FOUND BY INDEPENDENT REVIEW, LOGGED NOT + FIXED (hard rule 1 -- the file was frozen for review): + BUG-1 dead `acl snap_probe src 127.0.0.1/32`: squid binds ${LISTEN}:3129 only, so + nothing can arrive on loopback. The probes are actually admitted by + `snap_clients` because LISTEN is inside CLIENT_CIDR, and the comment + misdescribes its own mechanism. Hazard: anyone narrowing CLIENT_CIDR to + exclude .4 breaks `check`, and the rule they would reach for would not help. + BUG-2 `| grep -q` under `set -o pipefail` -- INHERITED, present identically in + dc-mirror.sh:278,281 and dc-cache-proxy.sh:234,252,255. A THREE-SCRIPT SWEEP, + not a local fix. Direction is fail-closed (false MISS, never false OK). + BUG-3 the deny probe's `403` shape was measured from a PORT deny, never a DSTDOMAIN + deny. If squid answers differently the gate REFUSES PERMANENTLY -- and under + GA-R6 a gate that cannot PASS blocks a stage close as hard as one that cannot + fail. Requires a ONE-TIME apply-time confirmation before `check` is cited. + BUG-4 `dpkg -s squid` exits 0 for `Status: deinstall ok config-files`, so a purged + package reports as present. The existence-vs-content class again. + `[ -x /usr/sbin/squid ]` is shorter and strictly stronger. + +F5. TEARDOWN INSTRUMENT ERROR: `juju models` SUMMARY COLUMNS read 0 machines / 0 units + while `juju status -m vr1-dc0` simultaneously read 26 machines / 37 applications. + The summary zeroes during `destroying` and is NOT a progress signal. I reported it + as progress. Standing lesson: during a teardown, `juju status -m ` is the + instrument; `juju models` counts are not. + +F6. WHY THE TEARDOWN STALLED, AND WHY IT WAS TERMINAL: all 26 machine/container agents + were `stopped`, so NO hook could execute at all -- the destroy worker asks and + nothing answers. It is a fixed point, not backoff, and would have retried forever + (it reached attempt 30, flat ~19 minutes, app set byte-identical across a 12-minute + diff). Units already in `error` cannot run their teardown hooks. `--force` cleared + it and stranded NOTHING -- all nine read back Ready/owner=None, no manual MAAS + release was needed. + +F7. A MAAS HOSTNAME RENAME ON A `Deployed` MACHINE IS RECORD-ONLY. Measured: MAAS + accepted it (fqdn updated), while `juju status` and the running OS both kept the old + name. MAAS applies hostname at DEPLOY time via cloud-init and has no agent + reconciling a running host; juju captured the name at provisioning. NOTE THE + ASYMMETRY, which is the dangerous part: MAAS REFUSES interface changes on a Deployed + machine (loud) but ACCEPTS a hostname change (silent, record-only). + STILL UNVERIFIED: that a redeploy applies the new hostname. Post-deploy, check all + three -- MAAS record, `juju status`, and `hostname` on the node -- agree. + +F8. STORAGE-NODE IDENTITY WOULD HAVE BEEN MISLABELLED BY MACHINE ORDER. Resolved by + pinned boot MAC: `amused-corgi` is storage-01, not storage-03. Ordering by juju + machine number would have mislabelled three of the four storage nodes. + +F9. `ovn-central`'s charm default is `source: zed`, NOT caracal -- it points at a UCA + pocket the dc0 mirror does not carry. Repointing changes its RELEASE, not its URL, + so it is a separate question. LOGGED, untouched. + +F10. GITIGNORED STATE: no new `.claude/settings.local.json` rules were added this + session. The 2026-07-30 sweep's F1 record of the existing rules stands unchanged. + +F11. AS-EXECUTED LOG: NOT USED for this window. `run-logged.sh` opens an interactive + `script(1)` subshell, unusable from a non-interactive session. Every action is in + the commits with read-backs, and captures went to docs/audit/*. **This window is + therefore NOT covered by an as-executed log, and must not be read as if it were.** + +-------------------------------------------------------------------------------------- +ALREADY ON SURFACE (verified by grep; recorded for completeness) +-------------------------------------------------------------------------------------- +A1. The prefer-ipv6 research and both rulings -- docs/audit/stage5-prefer-ipv6-charm- + research-20260731.txt, D-101 RULING NOTES (a) and (b), CURRENT-STATE. +A2. The D-135 jammy-backports amendment and its ~1.0 GiB sizing -- D-135 AMENDMENT + 2026-07-31, CURRENT-STATE, changelog items 16-18. +A3. The UCA-in-DC ruling and its 15-app derived scope -- joint D-135/D-107 RULING NOTE, + overlays/vr1-dc0-machines.yaml header, CURRENT-STATE. +A4. Bundle re-runnability (explicit base on 56 apps) + invariant 12 -- bundle.yaml + header, provider-bundle-check.py, tests, CURRENT-STATE. +A5. The three deploy attempts and their causes -- docs/audit/stage5-dc0-deploy- + attempt{1,2,3}-20260731.txt. +A6. The snap proxy design, measurements and mutation results -- docs/audit/stage5-snap- + proxy-measurements-20260731.txt (549 lines), changelog item 19. +A7. Per-DC Tailscale as a standing standup requirement -- workflow gap register item 21. +A8. ceph-rbd-mirror pinned with options + recommendation -- gap register item 22. +A9. model-defaults durability + the owed dc-model-defaults.sh -- CURRENT-STATE. +A10. The commit-contamination incident (e57ad09 swept agent files via `git add -A`) -- + changelog item 19 and commit 364ff12's message. + +-------------------------------------------------------------------------------------- +CONTRADICTION DETECTOR -- measurement vs standing document +-------------------------------------------------------------------------------------- +C1. preflight P6's reminder text still quotes "50 apps / 97 relations" against a + MEASURED 56 apps / 108 relations. Recorded on 2026-07-31 in CURRENT-STATE; the + stale string in scripts/preflight.sh is STILL THERE. DOCFIX owed. +C2. G17's own text names `chronyc sources`, and chrony is NOT INSTALLED on the MAAS + jammy image. Recorded 2026-07-30; still unfixed. DOCFIX owed. +C3. The phase-4 runbook's dc0 deploy block (:553-557) omits overlays/vr1-dc0-machines + .yaml, which is NOT a no-op -- it carries ovn-chassis bridge-interface-mappings. + Recorded; DOCFIX owed. + +-------------------------------------------------------------------------------------- +DELIBERATELY NOT DONE +-------------------------------------------------------------------------------------- +N1. The snap proxy is NOT installed and NOTHING is proven end to end. No snap has been + fetched through it. The harness green is FIXTURE green. +N2. The dc-snap-proxy simplification plan is NOT applied. Applying it REQUIRES re-running + the mutation pass -- S8 invalidates M32-M39, M42-M47, M49-M52 as evidence. +N3. Tailscale: all four sub-decisions deferred to end-of-step by operator direction. +N4. BUG-2 not fixed -- a three-script sweep, out of scope under hard rule 1. +N5. dc-model-defaults.sh not built; only the apt-mirror value is set. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 0615797..253c711 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -164,24 +164,6 @@ owed, because the next session is directed straight at the juju deployment. Sessions from the 2026-07-27 Phase-0 close onward remain live below. -## SESSION CLOSE 2026-07-27 -- Phase 0, R15 gates, D-136, step 3 executed (bounded, GA-R4) - -- Merged the grounding-audit branch to `main` (`607813b`, recorded `6495cfb`), retired it; 29 - commits on `dc-dc-stage5-preconditions`. NO stage opened/closed. SEC 21; D 138/DOCFIX 205. -- PHASE 0: voffice1 off a 105-commit-stale retired branch (both DCs' inner tfstate lives there -- - proven gitignored, sha256 identical); `openstack` 6.6.0 (snap REFUTED: no Caracal channel). -- ALL THREE R15 GATES FIXED, each reproduced first: repo-lint PASSED over ZERO files on a typo'd - flag; preflight left "clear to deploy" on exits 127/126/130/3; the gauntlet pinned no names. -- RULINGS (GA-R5, quoted): no DC ordering + NOC deferred (F1); **D-136 ADOPTED option (D)**; v6 host - part mirrors the v4 octet (F4); dc1 FIP pool; node v6 = MAAS static; **D-101 GOVERNING RATIONALE**. -- STEP 3 EXECUTED both DCs: 12 v6 subnets, 24 bands + 2 FIP pools, apex 3->27 ranges / 4->160 - addresses (156 VIPs), node carve 108 links. D-134 bands + D-020/R11 VIPs were RULED-BUT-NEVER-BUILT, - now artifacts. 18 Ready unchanged. Found: gauntlet HOST-DEPENDENT, preflight P5 HOST-BLIND. -- OWNED: called step 3 "complete" twice while the node layer was unbuilt; mis-filed P0-5; built - the apex tool against a dump so it matched zero live. All corrected on-surface. -- NEXT: gate host-authority; `provider-bundle-check` ARITY gap (imminent); **voffice1 back to - `main` at merge**; then the renderer. Body: `changelog-20260727-stage5-phase0.md`. - ## SESSION CLOSE 2026-07-29 -- arity gate, renderer, ruling 3, chain audit, OSD carve LIVE (bounded, GA-R4) - Branch `dc-dc-stage5-preconditions`, 22 commits pushed. NO stage opened/closed. Scan unchanged: 3 decisions, SEC 21, D 138 / DOCFIX 205 / BUNDLEFIX 053. @@ -293,3 +275,22 @@ - **OWNED -- three instrument errors, all one shape:** a capture parsed at the wrong field, a `systemctl` poll racing an async start, and a charmhub query with a wrong field name returning a uniform "no config" across 13 charms. The last two were caught by the SHAPE of the answer being implausible, not by discipline; that detector is now in auto-memory. - Gauntlet **ALL GREEN (93)**; repo-lint 0 fail; creds-matrix 65/65. Permission allowlist +12 read-only rules (all pinned to check/verify/assert/plan subcommands). - **NEXT:** the prefer-ipv6 research, then re-deploy. Steps 1-3.5 and the controller are DONE and need no repeat. Body: `docs/changelog-20260730-dc0-node-carve.md` (11 items). Status ONLY in CURRENT-STATE.md. + +## ROTATED 2026-07-31 (GA-R4 rule 3 / F1 -- cap restored at this close) + +The oldest closed-session summary moved VERBATIM to +`docs/archive/session-ledger-rotated-20260731.md`. The live ledger stood at 295 lines and +this close's summary would have breached the 300-line cap. + +## SESSION CLOSE 2026-07-31 -- Stage 5 dc0: prefer-ipv6 ruled, deploy succeeded then torn down for a clean rebuild (bounded, GA-R4) + +- Branch `dc-dc-stage5-preconditions`, **45 commits** pushed. NO stage opened/closed. Scan: 3 decisions, **SEC 26**, D 139 / DOCFIX 207 / BUNDLEFIX 053. +- **8 RULINGS (GA-R5, all utterances quoted):** prefer-ipv6 dropped from the six, then from ALL thirteen; P5's five new findings accepted; **option D** (explicit base on 56 apps AND a clean model); jammy-backports synced; UCA points in-DC; snaps get an in-DC forward proxy; and the MAAS proxy declined in favour of the owned one. +- **THE RESEARCH INVERTED ITS OWN PREMISE.** Read from the charm ARTIFACTS: `prefer-ipv6` is NOT what makes HAProxy bind `:::port` -- that is gated on the kernel `disable_ipv6` sysctl, and pacemaker picks IPv6addr by family detection. Neither consults the option. vault ships no haproxy at all (`[::]:8200` hardcoded). +- **>>> THE BUNDLE DEPLOYED (attempt 3, `Deploy of bundle completed.`) <<<** after three failures each one layer deeper: unknown option; a missing `policies/overrides.zip` on the client host; and a bundle relying on `default-base` being deployable exactly ONCE. +- **FOUR ARTIFACT DEFECTS, each masked by the one before it** (`apt-get update --error-on=any` fails on the first bad source): backports 404, upstream UCA unreachable, `prefer-ipv6` fatal on containers with only link-local v6, and snaps with no path. Units tracked 22 -> 17 -> 4 -> 7-in-one-class. +- **THE v6 HALF IS CARVED AS ADDRESSES BUT NEVER MADE OPERATIONAL** -- all six v6 plane subnets carry ZERO allocatable ranges, the rack has NO global v6 at all, the mirror does not answer on v6, and nodes have no v6 default route. Operator: dual-stack was "only a safety net". DC1 carries the IPv6 build; DC0 folds it back. +- **TEARDOWN STALLED AND WAS FORCED.** All 26 agents were `stopped`, so no hook could run -- a fixed point, not backoff. `--force` cleared it; NOTHING was stranded. All nine nodes renamed to their ruled `vr1-dc0--NN` by pinned boot MAC and back to `Ready`. +- **SNAP PROXY BUILT AND INDEPENDENTLY REVIEWED, NOT APPLIED.** squid at the utility `.4:3129`, allowlisted, 53/53 mutations killed; review says the ruled design is HONOURED on all four counts and logs FOUR real bugs. Nothing proven end to end. +- **OWNED:** a red-lint push (`| tail` masked the exit code); a `git add -A` that swept an agent's files into an unrelated commit; two instrument misreads (a `juju models` summary read as progress, and stale unit logs read as current); and a wrong "immutable system_id" claim the operator corrected. +- **NEXT:** apply the review cuts, re-run the mutation pass (S8 invalidates it), confirm BUG-3's deny shape once, install the proxy, then `add-model` + spaces + `deploy`. Sweep: `docs/audit/queued-findings-20260731-stage5-deploy.txt` (11 FIRST SURFACE). Bodies: `docs/changelog-20260731-prefer-ipv6-research.md`. Status ONLY in CURRENT-STATE.md.