diff --git a/.claude/skills/openstack-cloud-ops/references/platform-traps.md b/.claude/skills/openstack-cloud-ops/references/platform-traps.md index 78dd56d..8d97a81 100644 --- a/.claude/skills/openstack-cloud-ops/references/platform-traps.md +++ b/.claude/skills/openstack-cloud-ops/references/platform-traps.md @@ -464,6 +464,35 @@ (`ip addr del`, `ip route del`) and re-apply from the unit you believe owns it. Stopping the unit and seeing everything still work is a false green. +**5c. `systemctl show` IS NOT AN EXISTENCE CHECK -- it fabricates a healthy-looking +oneshot for a unit that does not exist.** Measured 2026-08-02 on the dc0 rack against the +invented name `dc0-rack-net` (the real units are `dc0-rack-legs` and `dc0-node-dns`, +defined at `scripts/dc-rack-net.sh:16,22`): + +``` +systemctl show dc0-rack-net -p Type -p RemainAfterExit -p Result + Type= + UnitFileState= + RemainAfterExit=no + Result=success <-- success. For a unit that has never existed. + ExecMainStatus=0 +``` + +`show` answers from property DEFAULTS, so a typo'd or renamed unit returns +`Result=success` / `ExecMainStatus=0` and no journal entries -- which reads exactly like +a oneshot that ran cleanly and is now idle. On this project it produced a false +"inactive" reading that was reported before it was caught. + +Use an instrument that ERRORS on absence: +- `systemctl cat ` -- exits non-zero with `No files found for `. +- `systemctl list-unit-files ''` -- returns 0 entries. +- or, if you must use `show`, assert `UnitFileState` is NON-EMPTY before reading any + other property. The empty string is the only field that betrays the absence. + +Same class as the standing instrument-currency lesson: take unit names from the script +that defines them, never from memory, and prove your instrument can report absence +before you trust it to report health. + --- ## 6. MAAS IPAM -- two behaviours that silently invert an expectation diff --git a/.claude/skills/openstack-cloud-ops/references/script-authoring.md b/.claude/skills/openstack-cloud-ops/references/script-authoring.md index 4f7e11e..17fe6fc 100644 --- a/.claude/skills/openstack-cloud-ops/references/script-authoring.md +++ b/.claude/skills/openstack-cloud-ops/references/script-authoring.md @@ -218,6 +218,32 @@ is a TEST bug, not a code bug -- fix the test. Three 2026-07 "failures" were untruthful mocks or wrong assertions, each caught by running the harness before delivery. +**After adding harness cases, ASSERT THE CASE COUNT MOVED.** A suite that reports ALL PASS +with its OLD count has silently dropped what you just added. Measured 2026-08-02 while +extending `tests/dc-egress-check`: two new cases were written with `fok`/`fbad`, helper +names borrowed from a DIFFERENT harness (`tests/dc-node-v6-carve`). The suite printed +`fbad: command not found` to stderr, kept its count at 14, and still reported ALL PASS. +The two mutations those cases were meant to kill then "survived" -- which reads as the +tool being robust and was actually the tests not existing. Note the compounding: the +symptom of a non-existent test and the symptom of a passing test are the same symptom. +So the delivery check is arithmetic, not vibes -- record the count before, record it +after, and confirm the delta equals the number of cases you wrote. This is the same +family as the standing rule that a checker which cannot fail is not a gate: prove each +new assertion can FAIL (delete it, or feed it a failing fixture) *and* prove it RAN. + +**When two scripts probe the SAME endpoint, they must share the probe definition.** +Measured 2026-08-02: `dc-egress-check.sh`'s A4 snap-store probe and +`dc-snap-proxy.sh:215` both hit `api.snapcraft.io`, but only the latter sent +`Snap-Device-Series: 16` -- a header the store REQUIRES. Measured 400 without it BOTH +through the proxy and direct, 200 with it. The newer gate therefore reported a healthy +proxy as BROKEN on its first live run while the older script returned PASS with a real +payload, and the two gates disagreed about the same infrastructure. **A fixture that +mocks `curl` cannot catch a wrong REAL request** -- the fake replays whatever you told +it to, so the harness stays green in both worlds. Factor the probe into one function +(`lib-net.sh` or the owning script) and have the second caller use it; a duplicated +request line is a duplicated *contract*, and only one copy gets fixed when the endpoint +tightens. + ## Repo-record authoring traps (added 2026-07-27, both measured the hard way) **repo-lint L5 rejects a heading that LEADS with a D-number.** The rule is diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index db5fe43..9e4de8e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2904,6 +2904,102 @@ ULA), step 3 (node v6 re-carve WHILE v4 IS STILL PRESENT -- `dc-node-v6-carve.py` pivots on IPv4 existing); the step 4-6 exchange above; then Step 3.5 (`add-model vr1-dc0` + spaces gate + `apt-mirror` -- note the spaces gate reads MAAS subnets, which step 2 changes) and Step 4. + **>>> QUEUED-FINDINGS BACKLOG WORKED 2026-08-02 (session 2, post-`/clear`): THE DEPLOY + INPUT IS RE-STAGED, THE REGION DB IS PROVEN CLEAN, AND THE MIRROR'S EXIT CODE IS + SETTLED. <<<** Three read-only agents ran in parallel against the 2026-08-02 close sweep + (`docs/audit/queued-findings-20260802-stage5-edge-fold.txt`). Bodies: + `docs/changelog-20260802-queued-items.md`. **NOTE THE TWO F-REGISTERS:** everything below + cites the SWEEP register (F1-F6); `docs/runbook-fold-register.md` has its own F1-F12 with + different meanings and is untouched by this session. + **SWEEP F1 IS CLOSED -- the dc0 rack's staged deploy input now MATCHES the repo.** The + fix was one gated `scp` of `overlays/vr1-dc0-vips.yaml`, `ed19d989e80da8da` -> + `80d861560a6b3c52`. Verified after: `bundle.yaml` `4c8a78527a3100f6` and + `vr1-dc0-machines.yaml` `e3be85e41b419486` both still equal the repo; the `0600` + `vr1-dc0-octavia-pki.yaml` is UNCHANGED (`5fc117f188332042`, same mode, same mtime) and + was hashed, never read. **The repo copy's correctness was MEASURED, not inherited from + the bookend:** zero `fd50:` ULA legs remain and 39 GUA legs are present + (`2602:f3e2:f02:11` x13, `f02:20` x13, `f02:21` x13), so the 2026-08-01 conformance + audit's blocker C is CLOSED in the repo copy; and `tests/render-drift` names the file + explicitly, 4/4 PASS with a proof-of-teeth case. **A single-file copy was provably + sufficient because the WHOLE staging directory was enumerated first** + (`docs/audit/repo-stage-drift-dc0-20260802.txt`, 427 lines): 14 regular files, exactly + 1 diverged, **0 missing from the Step-4 deploy closure** -- incl. `policies/overrides.zip`, + the dependency a prior deploy died on, present at the repo digest -- and 0 rack-only + orphans. voffice1's clone already held the correct overlay, so the rack was the ONLY + stale host. + **SWEEP F6 IS CLOSED -- PASS. THE dc0 MAAS REGION DATABASE IS PROVEN UNCORRUPTED.** + `pg_dump` read every page of every table in `maasdb`: **23,878,796 bytes / 37,199 lines / + exit 0 / empty stderr**, with `-- PostgreSQL database dump complete` asserted in a + separate pass. A torn or bad-checksum page anywhere would have aborted it, so the + 2026-08-01 hard power cut left NO page-level corruption; the circumstantial evidence is + now backed by the definitive test rather than standing in for it. Scope, not overclaimed: + this proves pages are readable and rows decode, NOT that MAAS's application-level + invariants hold. **F6's OWN DIAGNOSIS WAS WRONG and that is the durable part** -- it + blamed snap confinement and prescribed a login shell; confinement does not reproduce as + `ubuntu` at all (it does as `_daemon_`, which is where the original reading came from and + was over-generalised). The real discriminators are **ROLE and TRANSPORT**: the role is + `maas`, not `ubuntu`; and over TCP (`-h localhost`) it is password-challenged + (`fe_sendauth`), which would have meant handling the `regiond.conf` secret, while over + the **UNIX SOCKET (`-h /tmp`) it authenticates with no credential at all**. Both identity + values are now MEASURED (`database_name: maasdb`, `database_user: maas`) -- `maasdb` had + come from F6's prose and had never been verified. Only non-secret field names were + grepped; the dump was STREAMED and nothing persisted, since the MAAS DB carries API keys + and password hashes. Capture: `docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt`. + **SWEEP F2 IS DIAGNOSED, NOT FIXED: READING R1 IS TRUE, R2 IS REFUTED.** debmirror itself + prints `All done.` and THEN exits non-zero; nothing runs after the message that could + fail. Confirmed at vendor source and **re-verified independently by this session** because + it is the load-bearing claim -- `debmirror 1:2.39ubuntu2`, `1615:say("All done.");` / + `1620: exit 1 if (!$ignore_small_errors);`, and the helper passes no such flag. The + journal shows `All done.` TWICE (one per leg) with the failing one followed 10 ms later by + its own error block in the SAME PID; between the last `All done.` and systemd's failure + line there are 6.7 ms and zero log lines. Proximate cause: ONE upstream download, + `jammy-backports/main/dep11/Components-amd64.yml.gz`, 500 read timeout at 300s. + **THE SWEEP'S PREMISE WAS WRONG:** every line F2 quoted as "its own log says the sync + succeeded" belongs to the SECOND debmirror (the UCA leg, which passed); the failing ubuntu + leg's evidence sat three lines below the last line quoted. **AND "check dc0 stays RED" + OVERSTATES IT -- THE GATE IS FLAKY, NOT STUCK.** Re-counted by this session because the + agent's denominator did not reconcile (it reported 7 of 14; and my OWN first count was + also wrong -- a `Starting` grep pattern that did not match the unit's actual description, + logged here as the instrument error it was). **MEASURED over the full journal window + 2026-07-23..08-02: 16 runs, 7 `Finished`, 9 `Failed with result 'exit-code'`.** The + agent's "four successes in the last four days" IS correct. **The stronger evidence is the + SHAPE, not the ratio: on FOUR separate days a failed run was followed within hours by a + clean one** -- 07-25 fail 06:20 then OK 06:23; 07-27 fail 00:54 then OK 08:43; 07-31 fail + 00:54 then OK 02:35; 07-31 fail 18:29 then OK 18:56. That is transience demonstrated, + not inferred. Two caveats kept: the earliest failure (07-23) was the HOME-under-systemd + defect fixed the same hour, so the lifetime ratio understates current health; and **the + two most recent runs (both 08-02) BOTH failed**, bracketing the edge outage -- last + success 08-01T00:04:19. NO + remedy applied: debmirror's exit code conflates "nothing mirrored" with "mirrored minus N + transient files", so any tolerance change alters what the gate ATTESTS -- an operator + decision, and relaxing it would be the 2026-07-27 false-green defect inverted. Capture: + `docs/audit/mirror-exitcode-diagnosis-dc0-20260802.txt`. + **DOCFIX-207 ASSIGNED (next-free DOCFIX is now 208):** `scripts/preflight.sh` P6 quoted + `plan: 50 apps / 97 relations`; the MEASURED figure is **56 apps / 108 relations** (this + document, section 1, and again 2026-07-31). The same figures were corrected in phase-01's + own Step 1.2 gate on 2026-07-10 and this copy was missed, so for three weeks the gate and + the reminder pointing at it disagreed. It is a reminder string, not an assertion, and no + harness asserts it. `tests/preflight` 43/43 PASS after. + **SWEEP F3/F4/F5 GRADUATED from the audit capture to their durable homes** (they lived + only in a sweep file, which is not where a session looks before acting): F3 -> + platform-traps **5c**, `systemctl show` fabricates `Result=success` / `ExecMainStatus=0` + for a unit that does not exist, with the instruments that DO error on absence; F4 + F5 -> + script-authoring -- assert the harness CASE COUNT moved, and two scripts probing one + endpoint must share the probe definition (a fixture that mocks `curl` cannot catch a wrong + REAL request). + **TWO NEW FINDINGS, LOGGED NOT FIXED (hard rule 1), both surfaced by the staging agent.** + **(1) A GATE VALIDATES A DIFFERENT ARTIFACT THAN THE DEPLOY CONSUMES, and it sits on the + Step-4 path:** `scripts/preflight.sh:144-145` folds `overlays/vr1-dc0-machines.yaml` into + the merged input P2 validates, while `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:527-528` + asserts that file does not exist and Step 4 does not pass it. DOCFIX-shaped; likely bundles + with the outstanding D-138 run-location DOCFIX. **(2) `docs/security-ledger.md` SEC-029(3) + records `~/repo-stage` as a NINE-file copy; it is FOURTEEN** -- the extra five are + `opnsense-*` files left by the 2026-08-02 edge rebuild. A re-stage planned off the + nine-file figure would omit five. + **LEDGER MACHINE-DERIVED BLOCK RE-SEEDED** -- it had been carrying the 2026-07-27 seed + (21 open SEC, next-free D-138) while the per-session summaries below it carried current + figures, which is the one drift that block exists to prevent. Now 28 open SEC / D-141 / + DOCFIX-208, each added SEC row verified against the register rather than inferred. - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). diff --git a/docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt b/docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt new file mode 100644 index 0000000..9fb42f2 --- /dev/null +++ b/docs/audit/maasdb-pgdump-integrity-dc0-20260802.txt @@ -0,0 +1,343 @@ +maasdb-pgdump-integrity-dc0-20260802.txt +======================================== +SWEEP FINDING F6 -- `pg_dump maasdb`, the one dc0 corruption test that was not +completed. Source: docs/audit/queued-findings-20260802-stage5-edge-fold.txt +lines 89-100. + +Date: 2026-08-02 +Host: vr1-dc0-maas (10.12.8.6, user ubuntu, ProxyJump vr1-dc0-rack) +Agent: read-only verification agent, jumphost vcloud +Purpose: force a FULL-TABLE READ of the MAAS region database after the + 2026-08-01 hard power-cut of the region VM. pg_dump takes only a + shared lock and writes nothing; that is why the test is sanctioned. + +RESULT: **BLOCKED** -- but NOT by the blocker F6 predicted. See "What changed". + The full-table read did NOT happen. The database's integrity is + therefore still UNPROVEN by this method. + +-------------------------------------------------------------------------------- +1. ORIENTATION (read-only look-around, sanctioned by the brief) +-------------------------------------------------------------------------------- + +COMMAND (verbatim): + ssh vr1-dc0-maas 'ls /snap/bin/ 2>&1 | head -50; echo "---which pg_dump---"; \ + which pg_dump 2>&1; echo "---snap list---"; snap list 2>&1' + +OUTPUT (verbatim, abridged only where marked): + createdb + createuser + maas + maas.power + pgbench + postgresql.archivecleanup + postgresql.basebackup + postgresql.config + postgresql.conftool + postgresql.createcluster + postgresql.createdb + postgresql.createuser + postgresql.ctl + postgresql.ctlcluster + postgresql.dropcluster + postgresql.dump + postgresql.dumpall + postgresql.isready + postgresql.lsclusters + postgresql.pgbench + postgresql.psql + postgresql.receivewal + postgresql.recvlogical + postgresql.renamecluster + postgresql.restore + psql + ---which pg_dump--- + <-- EMPTY: there is no bare `pg_dump` on PATH + ---snap list--- + Name Version Rev Tracking Publisher Notes + core24 20260410 1643 latest/stable canonical** base + maas 3.7.2-17972-g.35e297c4d 41649 3.7/stable canonical** - + postgresql 16.14 236 16/stable canonical** - + snapd 2.76.1 27591 latest/stable canonical** snapd + +MEASURED FACT 1: pg_dump is exposed as the snap wrapper `postgresql.dump` + (from the `postgresql` 16.14 snap, rev 236). There is NO bare `pg_dump` on + PATH. F6's literal wording `pg_dump maasdb` therefore has to be run as + `postgresql.dump maasdb`. Hunting a raw binary path is the prohibited + escalation and was not attempted. + +COMMAND (verbatim): + ssh vr1-dc0-maas 'bash -lc "id; echo ---LSCLUSTERS---; postgresql.lsclusters 2>&1; \ + echo ---DBLIST---; postgresql.psql -l 2>&1 | head -30"' + +OUTPUT (verbatim): + uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),24(cdrom),27(sudo),30(dip),105(lxd) + ---LSCLUSTERS--- + Ver Cluster Port Status Owner Data directory Log file + 16 main 5432 online _daemon_ /var/lib/postgresql/16/main /var/log/postgresql/postgresql-16-main.log + ---DBLIST--- + psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: role "ubuntu" does not exist + +MEASURED FACT 2 -- THE PIVOT: `postgresql.psql` RAN. Under `bash -lc` (a login + shell) the snap executed, was NOT rejected by the cgroup, did NOT complain + about HOME, did NOT need a library path, and successfully opened the unix + socket /tmp/.s.PGSQL.5432. The error came back FROM POSTGRES ITSELF, at the + authentication layer. Cluster 16/main is online, owned by `_daemon_`. + +MEASURED FACT 3: the cluster is up and reachable; the only thing missing is a + Postgres ROLE for the OS user `ubuntu`. + +-------------------------------------------------------------------------------- +2. IS THERE A LEGITIMATE, NON-PRIVILEGED ROUTE FOR `ubuntu`? -- NO +-------------------------------------------------------------------------------- + +COMMAND (verbatim): + ssh vr1-dc0-maas 'bash -lc "echo ---PGENV---; env | grep -E \"^PG\" || echo NONE; \ + echo ---PGFILES---; ls -l ~/.pgpass ~/.pg_service.conf 2>&1; echo ---HBA---; \ + ls -l /var/snap/postgresql/common/*/pg_hba.conf /var/snap/postgresql/*/pg_hba.conf \ + /etc/postgresql/16/main/pg_hba.conf 2>&1 | head"' + +OUTPUT (verbatim): + ---PGENV--- + NONE + ---PGFILES--- + ls: cannot access '/home/ubuntu/.pgpass': No such file or directory + ls: cannot access '/home/ubuntu/.pg_service.conf': No such file or directory + ---HBA--- + ls: cannot access '/var/snap/postgresql/common/*/pg_hba.conf': No such file or directory + ls: cannot access '/var/snap/postgresql/*/pg_hba.conf': No such file or directory + ls: cannot access '/etc/postgresql/16/main/pg_hba.conf': No such file or directory + +COMMAND (verbatim): + ssh vr1-dc0-maas 'bash -lc "find /var/snap/postgresql /var/lib/postgresql \ + -maxdepth 4 -name pg_hba.conf 2>/dev/null"' + +OUTPUT (verbatim): + (no output; exit 1 -- nothing found that `ubuntu` can see) + +MEASURED FACT 4: the `ubuntu` user has NO PG* environment, NO ~/.pgpass, NO + ~/.pg_service.conf, and cannot see pg_hba.conf. There is no pre-provisioned, + non-privileged database route for this user. + +NOTE ON WHY pg_hba DOES NOT MATTER HERE: the server's answer was + `role "ubuntu" does not exist`, which is a ROLE-EXISTENCE failure, not an + auth-METHOD failure. No pg_hba setting -- trust, peer, md5 -- can make a + non-existent role connect. The route does not exist regardless of its content. + +-------------------------------------------------------------------------------- +3. THE SANCTIONED TEST -- RUN ONCE, IN LOGIN-SHELL FORM +-------------------------------------------------------------------------------- + +The dump was streamed over ssh to the jumphost rather than written on the VM +(brief: prefer streaming). Jumphost free space checked first: `df -h /tmp +/home/jessea123` -> /dev/vda3 10T, 7.6T Avail, 20% used. Nothing was written to +the VM. pg_dump's OWN exit status was captured separately from the ssh exit +status and from any pipeline element. + +COMMAND (verbatim): + ssh vr1-dc0-maas 'bash -lc "postgresql.dump maasdb; echo PGDUMP_EXIT=\$? >&2"' \ + > /tmp/maasdb-pgdump-20260802.out 2> /tmp/maasdb-pgdump-20260802.err; \ + echo "SSH_EXIT=$?"; echo "--- STDERR (verbatim) ---"; \ + cat /tmp/maasdb-pgdump-20260802.err; echo "--- STDOUT BYTES ---"; \ + wc -c < /tmp/maasdb-pgdump-20260802.out + +OUTPUT (verbatim): + SSH_EXIT=0 + --- STDERR (verbatim) --- + pg_dump: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: role "ubuntu" does not exist + PGDUMP_EXIT=1 + --- STDOUT BYTES --- + 0 + +ASSERTIONS (content, not exit code): + pg_dump exit status .............. 1 (pg_dump's own $?, not ssh's) + ssh exit status .................. 0 (proves exit-code-only would MISLEAD) + dump BYTE COUNT .................. 0 (nothing was produced) + dump TAIL ........................ (none -- zero bytes; no + "-- PostgreSQL database dump complete" + completion marker was emitted) + + NOTE the trap this instrumentation caught: SSH_EXIT was 0. A test written as + `ssh host 'pg_dump maasdb > /dev/null' && echo PASS` would have printed PASS + against a ZERO-BYTE dump and a failed pg_dump. This is the same false-green + shape as sweep findings F3 and F4. + +NO ESCALATION WAS ATTEMPTED after this error, per the brief. Specifically NOT +tried: `sudo`, `sudo -u postgres`, `sudo -u _daemon_`, any raw binary path, +LD_LIBRARY_PATH manipulation, `snap run --shell`, `-U `, or +reading MAAS's regiond.conf to harvest the database user (that file holds the +password; the brief and CLAUDE.md's secrets rule forbid reading it). + +-------------------------------------------------------------------------------- +4. WHAT CHANGED -- F6'S PREMISE IS PARTLY SUPERSEDED +-------------------------------------------------------------------------------- + +F6 records that five prior attempts were "blocked every time by snap +confinement (`_daemon_` has HOME=/nonexistent; the cgroup is rejected; the raw +binary needs the snap's library path)", and prescribes the login shell as the +fix because "the snap cgroup is valid" there. + +Under `bash -lc` AS THE OS USER `ubuntu`, both `postgresql.psql` and +`postgresql.dump` executed as programs -- no cgroup rejection, no +HOME=/nonexistent, no missing library path -- and reached the PostgreSQL server +on its unix socket, which answered them. THE SNAP-CONFINEMENT BARRIER DESCRIBED +IN F6 DID NOT REPRODUCE IN THIS FORM. + +WHICH VARIABLE WAS DECISIVE WAS NOT ISOLATED. This run differed from the five +prior attempts in TWO ways at once: it used a login shell, AND it ran as +`ubuntu` rather than as `_daemon_` (F6's symptoms -- `_daemon_` has +HOME=/nonexistent, the cgroup is rejected -- indicate the prior attempts were +invoked as `_daemon_`). Only ONE form was run here, per the brief's one-shot +constraint. DO NOT record "the login shell fixed it" as established; the +decisive change may have been the invoking user. Same class as the repo's +standing instrument-currency lesson: do not attribute a cause the evidence does +not pin down. + +The remaining blocker is a DIFFERENT problem, one step further in: + + FATAL: role "ubuntu" does not exist + +i.e. the `ubuntu` OS user has no PostgreSQL role, and no credential or service +file provisioning one. This is an authorization gap, not a confinement gap. +Conflating the two is what would lead an operator to keep trying confinement +workarounds that cannot possibly help. + +-------------------------------------------------------------------------------- +5. WHAT IS STILL OWED +-------------------------------------------------------------------------------- + +The full-table read of the MAAS region database HAS NOT HAPPENED. F6 remains +OPEN. The circumstantial evidence against damage is unchanged and still good +(ext4 with no EXT4-fs errors, no orphan recovery, MAAS services active, zero +checksum/corrupt/invalid-page entries in the postgres logs, all ~95 writes read +back correctly) -- and it is still not a proof. Do not upgrade it to one. + +TO CLOSE THIS, an operator with authority to run a privileged command needs to +run the dump as a role that EXISTS. That is a human-approved action under +CLAUDE.md hard rule 3, outside a read-only agent's remit, and it needs the real +role name MEASURED (not guessed -- hard rule 2) rather than taken from this +file or from F6. Two facts already measured that whoever does it will need: +the cluster is 16/main on port 5432, owned by `_daemon_`, socket +/tmp/.s.PGSQL.5432; and the wrapper is `postgresql.dump`, not `pg_dump`. + +Also still unverified: the DATABASE NAME. `maasdb` came from F6's prose, never +from a measurement in this session -- `postgresql.psql -l` could not run. If a +future attempt errors with `database "maasdb" does not exist`, that is a NAME +error, not a corruption finding. + +-------------------------------------------------------------------------------- +6. STATE CHANGES MADE BY THIS TEST +-------------------------------------------------------------------------------- + +On the VM: NONE. Read-only throughout. Nothing written, no service + touched, no user or role created, no config changed. The + dump produced zero bytes and was streamed, not stored. +On the jumphost: two scratch files, /tmp/maasdb-pgdump-20260802.out (0 + bytes) and /tmp/maasdb-pgdump-20260802.err, plus this + capture. No repo file was edited and no git command run. +Credentials: none read, none printed, none copied. + +================================================================================ +RESOLUTION -- APPENDED BY THE MAIN SESSION, 2026-08-02, AFTER THE AGENT REPORT +ABOVE. THE AGENT'S BLOCKED RESULT IS LEFT VERBATIM; THIS IS WHAT FOLLOWED. +================================================================================ + +VERDICT: SWEEP F6 IS CLOSED -- PASS. The full-table read COMPLETED. + +7. WHAT THE AGENT'S BLOCK ACTUALLY WAS +-------------------------------------------------------------------------------- +Neither snap confinement (F6's stated cause) nor the login shell was the +operative variable. The agent proved the wrapper runs fine as `ubuntu`; the +real gap was that identity, and the fix was to name a role that exists. + +Two identity facts, MEASURED (they had been carried from F6's prose, never +verified -- the agent flagged exactly this): + + ssh vr1-dc0-maas 'sudo grep -E "^database_name|^database_user|^database_host|^database_port" \ + /var/snap/maas/current/regiond.conf' + database_host: localhost + database_name: maasdb + database_user: maas + + So `maasdb` IS the right database name -- now on measurement rather than + prose. The role is `maas`, not `ubuntu` and not `postgres`. + Only the non-secret field names were grepped; `database_pass` was NEVER read. + +8. THE TRANSPORT DECIDED IT, AND THIS IS THE REUSABLE PART +-------------------------------------------------------------------------------- + ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h localhost -d maasdb 2>&1 >/dev/null | head -3"' + Password: + pg_dump: error: connection to server at "localhost" (::1), port 5432 failed: + fe_sendauth: no password supplied + + `-h localhost` forces TCP, and over TCP role `maas` is password-challenged -- + which would have meant handling the regiond.conf secret. Over the UNIX SOCKET + the same role authenticates with NO credential at all: + + ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h /tmp -d maasdb 2>&1 >/dev/null | head -3"' + (no output -- empty stderr) + + DURABLE: on this VM, dump/inspect over the socket (`-h /tmp`), never `-h + localhost`. The socket path avoids touching the credential entirely. An + earlier reading that called this a confinement problem sent five attempts + down the wrong branch; the discriminator is TRANSPORT plus ROLE, not + confinement and not the login shell. + +9. THE FULL-TABLE READ -- ASSERTED ON CONTENT, NOT ON EXIT CODE +-------------------------------------------------------------------------------- +Run via `ssh vr1-dc0-maas 'bash -l -s' <