maasdb-pgdump-integrity-dc0-20260802.txt
========================================
SWEEP FINDING F6 -- `pg_dump maasdb`, the one dc0 corruption test that was not
completed. Source: docs/audit/queued-findings-20260802-stage5-edge-fold.txt
lines 89-100.
Date: 2026-08-02
Host: vr1-dc0-maas (10.12.8.6, user ubuntu, ProxyJump vr1-dc0-rack)
Agent: read-only verification agent, jumphost vcloud
Purpose: force a FULL-TABLE READ of the MAAS region database after the
2026-08-01 hard power-cut of the region VM. pg_dump takes only a
shared lock and writes nothing; that is why the test is sanctioned.
RESULT: **BLOCKED** -- but NOT by the blocker F6 predicted. See "What changed".
The full-table read did NOT happen. The database's integrity is
therefore still UNPROVEN by this method.
--------------------------------------------------------------------------------
1. ORIENTATION (read-only look-around, sanctioned by the brief)
--------------------------------------------------------------------------------
COMMAND (verbatim):
ssh vr1-dc0-maas 'ls /snap/bin/ 2>&1 | head -50; echo "---which pg_dump---"; \
which pg_dump 2>&1; echo "---snap list---"; snap list 2>&1'
OUTPUT (verbatim, abridged only where marked):
createdb
createuser
maas
maas.power
pgbench
postgresql.archivecleanup
postgresql.basebackup
postgresql.config
postgresql.conftool
postgresql.createcluster
postgresql.createdb
postgresql.createuser
postgresql.ctl
postgresql.ctlcluster
postgresql.dropcluster
postgresql.dump
postgresql.dumpall
postgresql.isready
postgresql.lsclusters
postgresql.pgbench
postgresql.psql
postgresql.receivewal
postgresql.recvlogical
postgresql.renamecluster
postgresql.restore
psql
---which pg_dump---
<-- EMPTY: there is no bare `pg_dump` on PATH
---snap list---
Name Version Rev Tracking Publisher Notes
core24 20260410 1643 latest/stable canonical** base
maas 3.7.2-17972-g.35e297c4d 41649 3.7/stable canonical** -
postgresql 16.14 236 16/stable canonical** -
snapd 2.76.1 27591 latest/stable canonical** snapd
MEASURED FACT 1: pg_dump is exposed as the snap wrapper `postgresql.dump`
(from the `postgresql` 16.14 snap, rev 236). There is NO bare `pg_dump` on
PATH. F6's literal wording `pg_dump maasdb` therefore has to be run as
`postgresql.dump maasdb`. Hunting a raw binary path is the prohibited
escalation and was not attempted.
COMMAND (verbatim):
ssh vr1-dc0-maas 'bash -lc "id; echo ---LSCLUSTERS---; postgresql.lsclusters 2>&1; \
echo ---DBLIST---; postgresql.psql -l 2>&1 | head -30"'
OUTPUT (verbatim):
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),24(cdrom),27(sudo),30(dip),105(lxd)
---LSCLUSTERS---
Ver Cluster Port Status Owner Data directory Log file
16 main 5432 online _daemon_ /var/lib/postgresql/16/main /var/log/postgresql/postgresql-16-main.log
---DBLIST---
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: role "ubuntu" does not exist
MEASURED FACT 2 -- THE PIVOT: `postgresql.psql` RAN. Under `bash -lc` (a login
shell) the snap executed, was NOT rejected by the cgroup, did NOT complain
about HOME, did NOT need a library path, and successfully opened the unix
socket /tmp/.s.PGSQL.5432. The error came back FROM POSTGRES ITSELF, at the
authentication layer. Cluster 16/main is online, owned by `_daemon_`.
MEASURED FACT 3: the cluster is up and reachable; the only thing missing is a
Postgres ROLE for the OS user `ubuntu`.
--------------------------------------------------------------------------------
2. IS THERE A LEGITIMATE, NON-PRIVILEGED ROUTE FOR `ubuntu`? -- NO
--------------------------------------------------------------------------------
COMMAND (verbatim):
ssh vr1-dc0-maas 'bash -lc "echo ---PGENV---; env | grep -E \"^PG\" || echo NONE; \
echo ---PGFILES---; ls -l ~/.pgpass ~/.pg_service.conf 2>&1; echo ---HBA---; \
ls -l /var/snap/postgresql/common/*/pg_hba.conf /var/snap/postgresql/*/pg_hba.conf \
/etc/postgresql/16/main/pg_hba.conf 2>&1 | head"'
OUTPUT (verbatim):
---PGENV---
NONE
---PGFILES---
ls: cannot access '/home/ubuntu/.pgpass': No such file or directory
ls: cannot access '/home/ubuntu/.pg_service.conf': No such file or directory
---HBA---
ls: cannot access '/var/snap/postgresql/common/*/pg_hba.conf': No such file or directory
ls: cannot access '/var/snap/postgresql/*/pg_hba.conf': No such file or directory
ls: cannot access '/etc/postgresql/16/main/pg_hba.conf': No such file or directory
COMMAND (verbatim):
ssh vr1-dc0-maas 'bash -lc "find /var/snap/postgresql /var/lib/postgresql \
-maxdepth 4 -name pg_hba.conf 2>/dev/null"'
OUTPUT (verbatim):
(no output; exit 1 -- nothing found that `ubuntu` can see)
MEASURED FACT 4: the `ubuntu` user has NO PG* environment, NO ~/.pgpass, NO
~/.pg_service.conf, and cannot see pg_hba.conf. There is no pre-provisioned,
non-privileged database route for this user.
NOTE ON WHY pg_hba DOES NOT MATTER HERE: the server's answer was
`role "ubuntu" does not exist`, which is a ROLE-EXISTENCE failure, not an
auth-METHOD failure. No pg_hba setting -- trust, peer, md5 -- can make a
non-existent role connect. The route does not exist regardless of its content.
--------------------------------------------------------------------------------
3. THE SANCTIONED TEST -- RUN ONCE, IN LOGIN-SHELL FORM
--------------------------------------------------------------------------------
The dump was streamed over ssh to the jumphost rather than written on the VM
(brief: prefer streaming). Jumphost free space checked first: `df -h /tmp
/home/jessea123` -> /dev/vda3 10T, 7.6T Avail, 20% used. Nothing was written to
the VM. pg_dump's OWN exit status was captured separately from the ssh exit
status and from any pipeline element.
COMMAND (verbatim):
ssh vr1-dc0-maas 'bash -lc "postgresql.dump maasdb; echo PGDUMP_EXIT=\$? >&2"' \
> /tmp/maasdb-pgdump-20260802.out 2> /tmp/maasdb-pgdump-20260802.err; \
echo "SSH_EXIT=$?"; echo "--- STDERR (verbatim) ---"; \
cat /tmp/maasdb-pgdump-20260802.err; echo "--- STDOUT BYTES ---"; \
wc -c < /tmp/maasdb-pgdump-20260802.out
OUTPUT (verbatim):
SSH_EXIT=0
--- STDERR (verbatim) ---
pg_dump: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: role "ubuntu" does not exist
PGDUMP_EXIT=1
--- STDOUT BYTES ---
0
ASSERTIONS (content, not exit code):
pg_dump exit status .............. 1 (pg_dump's own $?, not ssh's)
ssh exit status .................. 0 (proves exit-code-only would MISLEAD)
dump BYTE COUNT .................. 0 (nothing was produced)
dump TAIL ........................ (none -- zero bytes; no
"-- PostgreSQL database dump complete"
completion marker was emitted)
NOTE the trap this instrumentation caught: SSH_EXIT was 0. A test written as
`ssh host 'pg_dump maasdb > /dev/null' && echo PASS` would have printed PASS
against a ZERO-BYTE dump and a failed pg_dump. This is the same false-green
shape as sweep findings F3 and F4.
NO ESCALATION WAS ATTEMPTED after this error, per the brief. Specifically NOT
tried: `sudo`, `sudo -u postgres`, `sudo -u _daemon_`, any raw binary path,
LD_LIBRARY_PATH manipulation, `snap run --shell`, `-U <guessed-role>`, or
reading MAAS's regiond.conf to harvest the database user (that file holds the
password; the brief and CLAUDE.md's secrets rule forbid reading it).
--------------------------------------------------------------------------------
4. WHAT CHANGED -- F6'S PREMISE IS PARTLY SUPERSEDED
--------------------------------------------------------------------------------
F6 records that five prior attempts were "blocked every time by snap
confinement (`_daemon_` has HOME=/nonexistent; the cgroup is rejected; the raw
binary needs the snap's library path)", and prescribes the login shell as the
fix because "the snap cgroup is valid" there.
Under `bash -lc` AS THE OS USER `ubuntu`, both `postgresql.psql` and
`postgresql.dump` executed as programs -- no cgroup rejection, no
HOME=/nonexistent, no missing library path -- and reached the PostgreSQL server
on its unix socket, which answered them. THE SNAP-CONFINEMENT BARRIER DESCRIBED
IN F6 DID NOT REPRODUCE IN THIS FORM.
WHICH VARIABLE WAS DECISIVE WAS NOT ISOLATED. This run differed from the five
prior attempts in TWO ways at once: it used a login shell, AND it ran as
`ubuntu` rather than as `_daemon_` (F6's symptoms -- `_daemon_` has
HOME=/nonexistent, the cgroup is rejected -- indicate the prior attempts were
invoked as `_daemon_`). Only ONE form was run here, per the brief's one-shot
constraint. DO NOT record "the login shell fixed it" as established; the
decisive change may have been the invoking user. Same class as the repo's
standing instrument-currency lesson: do not attribute a cause the evidence does
not pin down.
The remaining blocker is a DIFFERENT problem, one step further in:
FATAL: role "ubuntu" does not exist
i.e. the `ubuntu` OS user has no PostgreSQL role, and no credential or service
file provisioning one. This is an authorization gap, not a confinement gap.
Conflating the two is what would lead an operator to keep trying confinement
workarounds that cannot possibly help.
--------------------------------------------------------------------------------
5. WHAT IS STILL OWED
--------------------------------------------------------------------------------
The full-table read of the MAAS region database HAS NOT HAPPENED. F6 remains
OPEN. The circumstantial evidence against damage is unchanged and still good
(ext4 with no EXT4-fs errors, no orphan recovery, MAAS services active, zero
checksum/corrupt/invalid-page entries in the postgres logs, all ~95 writes read
back correctly) -- and it is still not a proof. Do not upgrade it to one.
TO CLOSE THIS, an operator with authority to run a privileged command needs to
run the dump as a role that EXISTS. That is a human-approved action under
CLAUDE.md hard rule 3, outside a read-only agent's remit, and it needs the real
role name MEASURED (not guessed -- hard rule 2) rather than taken from this
file or from F6. Two facts already measured that whoever does it will need:
the cluster is 16/main on port 5432, owned by `_daemon_`, socket
/tmp/.s.PGSQL.5432; and the wrapper is `postgresql.dump`, not `pg_dump`.
Also still unverified: the DATABASE NAME. `maasdb` came from F6's prose, never
from a measurement in this session -- `postgresql.psql -l` could not run. If a
future attempt errors with `database "maasdb" does not exist`, that is a NAME
error, not a corruption finding.
--------------------------------------------------------------------------------
6. STATE CHANGES MADE BY THIS TEST
--------------------------------------------------------------------------------
On the VM: NONE. Read-only throughout. Nothing written, no service
touched, no user or role created, no config changed. The
dump produced zero bytes and was streamed, not stored.
On the jumphost: two scratch files, /tmp/maasdb-pgdump-20260802.out (0
bytes) and /tmp/maasdb-pgdump-20260802.err, plus this
capture. No repo file was edited and no git command run.
Credentials: none read, none printed, none copied.
================================================================================
RESOLUTION -- APPENDED BY THE MAIN SESSION, 2026-08-02, AFTER THE AGENT REPORT
ABOVE. THE AGENT'S BLOCKED RESULT IS LEFT VERBATIM; THIS IS WHAT FOLLOWED.
================================================================================
VERDICT: SWEEP F6 IS CLOSED -- PASS. The full-table read COMPLETED.
7. WHAT THE AGENT'S BLOCK ACTUALLY WAS
--------------------------------------------------------------------------------
Neither snap confinement (F6's stated cause) nor the login shell was the
operative variable. The agent proved the wrapper runs fine as `ubuntu`; the
real gap was that identity, and the fix was to name a role that exists.
Two identity facts, MEASURED (they had been carried from F6's prose, never
verified -- the agent flagged exactly this):
ssh vr1-dc0-maas 'sudo grep -E "^database_name|^database_user|^database_host|^database_port" \
/var/snap/maas/current/regiond.conf'
database_host: localhost
database_name: maasdb
database_user: maas
So `maasdb` IS the right database name -- now on measurement rather than
prose. The role is `maas`, not `ubuntu` and not `postgres`.
Only the non-secret field names were grepped; `database_pass` was NEVER read.
8. THE TRANSPORT DECIDED IT, AND THIS IS THE REUSABLE PART
--------------------------------------------------------------------------------
ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h localhost -d maasdb 2>&1 >/dev/null | head -3"'
Password:
pg_dump: error: connection to server at "localhost" (::1), port 5432 failed:
fe_sendauth: no password supplied
`-h localhost` forces TCP, and over TCP role `maas` is password-challenged --
which would have meant handling the regiond.conf secret. Over the UNIX SOCKET
the same role authenticates with NO credential at all:
ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h /tmp -d maasdb 2>&1 >/dev/null | head -3"'
(no output -- empty stderr)
DURABLE: on this VM, dump/inspect over the socket (`-h /tmp`), never `-h
localhost`. The socket path avoids touching the credential entirely. An
earlier reading that called this a confinement problem sent five attempts
down the wrong branch; the discriminator is TRANSPORT plus ROLE, not
confinement and not the login shell.
9. THE FULL-TABLE READ -- ASSERTED ON CONTENT, NOT ON EXIT CODE
--------------------------------------------------------------------------------
Run via `ssh vr1-dc0-maas 'bash -l -s' < <script>`; the dump was STREAMED
through awk and NOTHING was persisted -- the MAAS DB carries API keys and
password hashes, so writing it out would have created a new exposure needing
its own SEC row.
=== identity (measured, not assumed) ===
whoami=ubuntu SHLVL_login_shell=hBs
=== dump ===
BYTES=23878796
LINES=37199
TAIL3:
\unrestrict eBuavGsgHQrgevYDgajXmf0bU78hRnmApNLi3a0hnxxddTBoozIgRNjKEXtfHb6
PG_DUMP_RC=0
=== stderr ===
(empty)
Completion marker asserted in a separate pass, because an exit code alone is
exactly the weak green the agent warned about:
ssh vr1-dc0-maas 'bash -lc "postgresql.dump -U maas -h /tmp -d maasdb | tail -6"'
--
-- PostgreSQL database dump complete
--
\unrestrict 9JsfLYRgP1qz5Uasb2Xsx8KZlerohMVGHnFbjXld5fwPudHFz8KeG9zWmG21XPT
NOT A SECRET, stated because it looks like one: the `\unrestrict <token>` line
is psql's per-dump random nonce (the \restrict/\unrestrict meta-commands added
in the 2025 PostgreSQL security release). It DIFFERS between the two runs above
-- eBuav... vs 9Jsf... -- which is the proof that it is generated per dump and
carries nothing from the database.
10. WHAT THIS DOES AND DOES NOT ESTABLISH
--------------------------------------------------------------------------------
ESTABLISHES: pg_dump read EVERY page of EVERY table in `maasdb` and exited 0
with empty stderr and a well-formed terminator. A torn or checksummed-bad page
anywhere in the database would have aborted the dump. The 2026-08-01 hard power
cut left NO page-level corruption. The circumstantial evidence F6 listed (clean
ext4, no orphan recovery, no corrupt/invalid-page entries in the postgres logs,
~95 writes reading back correctly) is now backed by the definitive test rather
than standing in for it.
DOES NOT ESTABLISH: logical/application-level consistency of MAAS's own data
model. pg_dump proves the pages are readable and the rows decode; it does not
prove MAAS's invariants hold. Nothing observed suggests otherwise, and no such
claim is made here.
11. AGENT'S FALSE-GREEN FINDING -- CONFIRMED AND WORTH KEEPING
--------------------------------------------------------------------------------
The agent measured SSH_EXIT=0 while pg_dump exited 1 having produced 0 bytes.
So F6's own proposed form, `ssh host 'pg_dump maasdb > /dev/null' && echo PASS`,
would have printed PASS over a dump that never happened. Same family as sweep
F3 (`systemctl show` fabricating success for a non-existent unit) and sweep F4
(a suite reporting ALL PASS with cases that never ran). Asserting on BYTES plus
the completion marker is what makes this run a gate rather than a ritual.
