maasdb-pgdump-integrity-dc0-20260802.txt ======================================== SWEEP FINDING F6 -- `pg_dump maasdb`, the one dc0 corruption test that was not completed. Source: docs/audit/queued-findings-20260802-stage5-edge-fold.txt lines 89-100. Date: 2026-08-02 Host: vr1-dc0-maas (10.12.8.6, user ubuntu, ProxyJump vr1-dc0-rack) Agent: read-only verification agent, jumphost vcloud Purpose: force a FULL-TABLE READ of the MAAS region database after the 2026-08-01 hard power-cut of the region VM. pg_dump takes only a shared lock and writes nothing; that is why the test is sanctioned. RESULT: **BLOCKED** -- but NOT by the blocker F6 predicted. See "What changed". The full-table read did NOT happen. The database's integrity is therefore still UNPROVEN by this method. -------------------------------------------------------------------------------- 1. ORIENTATION (read-only look-around, sanctioned by the brief) -------------------------------------------------------------------------------- COMMAND (verbatim): ssh vr1-dc0-maas 'ls /snap/bin/ 2>&1 | head -50; echo "---which pg_dump---"; \ which pg_dump 2>&1; echo "---snap list---"; snap list 2>&1' OUTPUT (verbatim, abridged only where marked): createdb createuser maas maas.power pgbench postgresql.archivecleanup postgresql.basebackup postgresql.config postgresql.conftool postgresql.createcluster postgresql.createdb postgresql.createuser postgresql.ctl postgresql.ctlcluster postgresql.dropcluster postgresql.dump postgresql.dumpall postgresql.isready postgresql.lsclusters postgresql.pgbench postgresql.psql postgresql.receivewal postgresql.recvlogical postgresql.renamecluster postgresql.restore psql ---which pg_dump--- <-- EMPTY: there is no bare `pg_dump` on PATH ---snap list--- Name Version Rev Tracking Publisher Notes core24 20260410 1643 latest/stable canonical** base maas 3.7.2-17972-g.35e297c4d 41649 3.7/stable canonical** - postgresql 16.14 236 16/stable canonical** - snapd 2.76.1 27591 latest/stable canonical** snapd MEASURED FACT 1: pg_dump is exposed as the snap wrapper `postgresql.dump` (from the `postgresql` 16.14 snap, rev 236). There is NO bare `pg_dump` on PATH. F6's literal wording `pg_dump maasdb` therefore has to be run as `postgresql.dump maasdb`. Hunting a raw binary path is the prohibited escalation and was not attempted. COMMAND (verbatim): ssh vr1-dc0-maas 'bash -lc "id; echo ---LSCLUSTERS---; postgresql.lsclusters 2>&1; \ echo ---DBLIST---; postgresql.psql -l 2>&1 | head -30"' OUTPUT (verbatim): uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),24(cdrom),27(sudo),30(dip),105(lxd) ---LSCLUSTERS--- Ver Cluster Port Status Owner Data directory Log file 16 main 5432 online _daemon_ /var/lib/postgresql/16/main /var/log/postgresql/postgresql-16-main.log ---DBLIST--- psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: role "ubuntu" does not exist MEASURED FACT 2 -- THE PIVOT: `postgresql.psql` RAN. Under `bash -lc` (a login shell) the snap executed, was NOT rejected by the cgroup, did NOT complain about HOME, did NOT need a library path, and successfully opened the unix socket /tmp/.s.PGSQL.5432. The error came back FROM POSTGRES ITSELF, at the authentication layer. Cluster 16/main is online, owned by `_daemon_`. MEASURED FACT 3: the cluster is up and reachable; the only thing missing is a Postgres ROLE for the OS user `ubuntu`. -------------------------------------------------------------------------------- 2. IS THERE A LEGITIMATE, NON-PRIVILEGED ROUTE FOR `ubuntu`? -- NO -------------------------------------------------------------------------------- COMMAND (verbatim): ssh vr1-dc0-maas 'bash -lc "echo ---PGENV---; env | grep -E \"^PG\" || echo NONE; \ echo ---PGFILES---; ls -l ~/.pgpass ~/.pg_service.conf 2>&1; echo ---HBA---; \ ls -l /var/snap/postgresql/common/*/pg_hba.conf /var/snap/postgresql/*/pg_hba.conf \ /etc/postgresql/16/main/pg_hba.conf 2>&1 | head"' OUTPUT (verbatim): ---PGENV--- NONE ---PGFILES--- ls: cannot access '/home/ubuntu/.pgpass': No such file or directory ls: cannot access '/home/ubuntu/.pg_service.conf': No such file or directory ---HBA--- ls: cannot access '/var/snap/postgresql/common/*/pg_hba.conf': No such file or directory ls: cannot access '/var/snap/postgresql/*/pg_hba.conf': No such file or directory ls: cannot access '/etc/postgresql/16/main/pg_hba.conf': No such file or directory COMMAND (verbatim): ssh vr1-dc0-maas 'bash -lc "find /var/snap/postgresql /var/lib/postgresql \ -maxdepth 4 -name pg_hba.conf 2>/dev/null"' OUTPUT (verbatim): (no output; exit 1 -- nothing found that `ubuntu` can see) MEASURED FACT 4: the `ubuntu` user has NO PG* environment, NO ~/.pgpass, NO ~/.pg_service.conf, and cannot see pg_hba.conf. There is no pre-provisioned, non-privileged database route for this user. NOTE ON WHY pg_hba DOES NOT MATTER HERE: the server's answer was `role "ubuntu" does not exist`, which is a ROLE-EXISTENCE failure, not an auth-METHOD failure. No pg_hba setting -- trust, peer, md5 -- can make a non-existent role connect. The route does not exist regardless of its content. -------------------------------------------------------------------------------- 3. THE SANCTIONED TEST -- RUN ONCE, IN LOGIN-SHELL FORM -------------------------------------------------------------------------------- The dump was streamed over ssh to the jumphost rather than written on the VM (brief: prefer streaming). Jumphost free space checked first: `df -h /tmp /home/jessea123` -> /dev/vda3 10T, 7.6T Avail, 20% used. Nothing was written to the VM. pg_dump's OWN exit status was captured separately from the ssh exit status and from any pipeline element. COMMAND (verbatim): ssh vr1-dc0-maas 'bash -lc "postgresql.dump maasdb; echo PGDUMP_EXIT=\$? >&2"' \ > /tmp/maasdb-pgdump-20260802.out 2> /tmp/maasdb-pgdump-20260802.err; \ echo "SSH_EXIT=$?"; echo "--- STDERR (verbatim) ---"; \ cat /tmp/maasdb-pgdump-20260802.err; echo "--- STDOUT BYTES ---"; \ wc -c < /tmp/maasdb-pgdump-20260802.out OUTPUT (verbatim): SSH_EXIT=0 --- STDERR (verbatim) --- pg_dump: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: role "ubuntu" does not exist PGDUMP_EXIT=1 --- STDOUT BYTES --- 0 ASSERTIONS (content, not exit code): pg_dump exit status .............. 1 (pg_dump's own $?, not ssh's) ssh exit status .................. 0 (proves exit-code-only would MISLEAD) dump BYTE COUNT .................. 0 (nothing was produced) dump TAIL ........................ (none -- zero bytes; no "-- PostgreSQL database dump complete" completion marker was emitted) NOTE the trap this instrumentation caught: SSH_EXIT was 0. A test written as `ssh host 'pg_dump maasdb > /dev/null' && echo PASS` would have printed PASS against a ZERO-BYTE dump and a failed pg_dump. This is the same false-green shape as sweep findings F3 and F4. NO ESCALATION WAS ATTEMPTED after this error, per the brief. Specifically NOT tried: `sudo`, `sudo -u postgres`, `sudo -u _daemon_`, any raw binary path, LD_LIBRARY_PATH manipulation, `snap run --shell`, `-U `, or reading MAAS's regiond.conf to harvest the database user (that file holds the password; the brief and CLAUDE.md's secrets rule forbid reading it). -------------------------------------------------------------------------------- 4. WHAT CHANGED -- F6'S PREMISE IS PARTLY SUPERSEDED -------------------------------------------------------------------------------- F6 records that five prior attempts were "blocked every time by snap confinement (`_daemon_` has HOME=/nonexistent; the cgroup is rejected; the raw binary needs the snap's library path)", and prescribes the login shell as the fix because "the snap cgroup is valid" there. Under `bash -lc` AS THE OS USER `ubuntu`, both `postgresql.psql` and `postgresql.dump` executed as programs -- no cgroup rejection, no HOME=/nonexistent, no missing library path -- and reached the PostgreSQL server on its unix socket, which answered them. THE SNAP-CONFINEMENT BARRIER DESCRIBED IN F6 DID NOT REPRODUCE IN THIS FORM. WHICH VARIABLE WAS DECISIVE WAS NOT ISOLATED. This run differed from the five prior attempts in TWO ways at once: it used a login shell, AND it ran as `ubuntu` rather than as `_daemon_` (F6's symptoms -- `_daemon_` has HOME=/nonexistent, the cgroup is rejected -- indicate the prior attempts were invoked as `_daemon_`). Only ONE form was run here, per the brief's one-shot constraint. DO NOT record "the login shell fixed it" as established; the decisive change may have been the invoking user. Same class as the repo's standing instrument-currency lesson: do not attribute a cause the evidence does not pin down. The remaining blocker is a DIFFERENT problem, one step further in: FATAL: role "ubuntu" does not exist i.e. the `ubuntu` OS user has no PostgreSQL role, and no credential or service file provisioning one. This is an authorization gap, not a confinement gap. Conflating the two is what would lead an operator to keep trying confinement workarounds that cannot possibly help. -------------------------------------------------------------------------------- 5. WHAT IS STILL OWED -------------------------------------------------------------------------------- The full-table read of the MAAS region database HAS NOT HAPPENED. F6 remains OPEN. The circumstantial evidence against damage is unchanged and still good (ext4 with no EXT4-fs errors, no orphan recovery, MAAS services active, zero checksum/corrupt/invalid-page entries in the postgres logs, all ~95 writes read back correctly) -- and it is still not a proof. Do not upgrade it to one. TO CLOSE THIS, an operator with authority to run a privileged command needs to run the dump as a role that EXISTS. That is a human-approved action under CLAUDE.md hard rule 3, outside a read-only agent's remit, and it needs the real role name MEASURED (not guessed -- hard rule 2) rather than taken from this file or from F6. Two facts already measured that whoever does it will need: the cluster is 16/main on port 5432, owned by `_daemon_`, socket /tmp/.s.PGSQL.5432; and the wrapper is `postgresql.dump`, not `pg_dump`. Also still unverified: the DATABASE NAME. `maasdb` came from F6's prose, never from a measurement in this session -- `postgresql.psql -l` could not run. If a future attempt errors with `database "maasdb" does not exist`, that is a NAME error, not a corruption finding. -------------------------------------------------------------------------------- 6. STATE CHANGES MADE BY THIS TEST -------------------------------------------------------------------------------- On the VM: NONE. Read-only throughout. Nothing written, no service touched, no user or role created, no config changed. The dump produced zero bytes and was streamed, not stored. On the jumphost: two scratch files, /tmp/maasdb-pgdump-20260802.out (0 bytes) and /tmp/maasdb-pgdump-20260802.err, plus this capture. No repo file was edited and no git command run. Credentials: none read, none printed, none copied. ================================================================================ RESOLUTION -- APPENDED BY THE MAIN SESSION, 2026-08-02, AFTER THE AGENT REPORT ABOVE. THE AGENT'S BLOCKED RESULT IS LEFT VERBATIM; THIS IS WHAT FOLLOWED. ================================================================================ VERDICT: SWEEP F6 IS CLOSED -- PASS. The full-table read COMPLETED. 7. WHAT THE AGENT'S BLOCK ACTUALLY WAS -------------------------------------------------------------------------------- Neither snap confinement (F6's stated cause) nor the login shell was the operative variable. The agent proved the wrapper runs fine as `ubuntu`; the real gap was that identity, and the fix was to name a role that exists. Two identity facts, MEASURED (they had been carried from F6's prose, never verified -- the agent flagged exactly this): ssh vr1-dc0-maas 'sudo grep -E "^database_name|^database_user|^database_host|^database_port" \ /var/snap/maas/current/regiond.conf' database_host: localhost database_name: maasdb database_user: maas So `maasdb` IS the right database name -- now on measurement rather than prose. The role is `maas`, not `ubuntu` and not `postgres`. Only the non-secret field names were grepped; `database_pass` was NEVER read. 8. THE TRANSPORT DECIDED IT, AND THIS IS THE REUSABLE PART -------------------------------------------------------------------------------- ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h localhost -d maasdb 2>&1 >/dev/null | head -3"' Password: pg_dump: error: connection to server at "localhost" (::1), port 5432 failed: fe_sendauth: no password supplied `-h localhost` forces TCP, and over TCP role `maas` is password-challenged -- which would have meant handling the regiond.conf secret. Over the UNIX SOCKET the same role authenticates with NO credential at all: ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h /tmp -d maasdb 2>&1 >/dev/null | head -3"' (no output -- empty stderr) DURABLE: on this VM, dump/inspect over the socket (`-h /tmp`), never `-h localhost`. The socket path avoids touching the credential entirely. An earlier reading that called this a confinement problem sent five attempts down the wrong branch; the discriminator is TRANSPORT plus ROLE, not confinement and not the login shell. 9. THE FULL-TABLE READ -- ASSERTED ON CONTENT, NOT ON EXIT CODE -------------------------------------------------------------------------------- Run via `ssh vr1-dc0-maas 'bash -l -s' <