dc0 MAAS region LIVE; three creds minted, consolidated and REGISTERED (SEC-027) ...

Region: noble + MAAS 3.7.2 + PostgreSQL 16.14 snaps, byte-identical versions
to the Office1 region (checked, not assumed). regiond/rackd/apiserver ACTIVE,
dhcpd INACTIVE so there is no DHCP conflict with the Office1 rack still
serving that VLAN. API answers 200 on 10.12.8.6:5240 -- the address jujud
could not reach at Office1, which is the whole point of D-132's amendment.

Carve verified BOTH legs (the lesson of today): metal-admin 10.12.8.6 + v6
::6, provider-public 10.12.4.6 with gw 10.12.4.1.

Credentials, per operator direction: DB password, region admin password and
admin API key minted ON the VM, never printed, consolidated to
~/vr1-dc0-creds/ at 0600 with sha256 digests compared to prove the copies are
identical, API key format-verified parts=3. SEC-027 opened (open SEC 22->23),
recording the residual argv exposure honestly -- `maas init` has no stdin or
file option for --database-uri.

The creds MATRIX now expects them at BOTH DCs: 5 rows per DC, a new `region`
host-role token in the enum, 4 vm-secret-locations rows (definition-of-done
for a new mint site), dc0's manifest declares its three. dc1's three now FAIL
S2 BY DESIGN -- the forward register making dc1's absence detectable, which is
what D-137 exists for. Harness 65/65 PASS, repo-lint 0 fail.

Also: durable ssh aliases for DC nodes. Deployed nodes carry the
MAAS-registered vr1-office1-svc key, not the rack service key, so each hop
needs its own identity -- a bare `ssh -i <key> -J voffice1,<rack>` fails
because -i forces one key on every hop. No credential was copied anywhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Reissue backup custody: transferred to vcloud, and REGISTERED ...

Operator: 'Update the matrix as needed. Transfer the artifacts to the vcloud.
keep the pin that during the secrets workflow planning that the creds and certs
from this step need to be included.'

CUSTODY (Step 1.0-REISSUE.4, executed): both pre-reissue archives pulled from the
headend to the jumphost SEC-009 creds folders, sha256 compared and IDENTICAL at
both ends (e30cab7f...3487 dc0, 17c831da...6fe3 dc1), 0600, each listing 15
entries including the deploy overlay and the CA serial. Headend staging copies
removed; ~/octavia-pki/backups/ gone; no .reissue-* residue. The sha256 compare
happens BEFORE the delete because a truncated scp would otherwise leave a
verified-looking backup of nothing.

REGISTER: two octavia-reissue-backup rows (per-DC, jumphost, consolidated,
templated filename <site>-octavia-pki-<stamp>.tar.gz), the DERIVED manifests
updated to match (D-137 ruling 2 -- manifests are generated, not hand-authored),
and a new n-reissue-backup note. It is deliberately a SEPARATE id from
n-pki-backup: that one is the generation-time workspace archive, this one is
per-ROTATION and additionally carries the deploy overlay, which the generation
archive does not -- restoring only the workspace would leave the half that
actually reaches the charm unrecoverable.

Measured after: tier1 101 rows / SAME 5 pre-existing findings; tier2 the same 7,
with NO reissue-attributable finding -- and tier2 FOUND both archives at their
declared location, which is the point of registering them. Harness 65/65.

HAZARD RECORDED: these archives contain controller-ca.cert.srl, the CA's issuance
state. Restoring one over a workspace that has issued since rolls the serial
BACKWARDS and the next mint reuses a serial the estate already holds. Serials
burned 2026-07-30: ...250E (dc0), ...674DE (dc1).

PINNED: the note carries the operator's standing requirement that the creds and
certs from this step be included in the secrets-storage workflow planning, with
the full scope enumerated. The jumphost creds folder is the INTERIM home only.

gauntlet ALL GREEN (89); repo-lint 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

tfstate backup path built; E2/E3 closed live; a gate disagreement found and fixed ...

BACKUP PROVEN BY RESTORE, not by existence. Both PKI archives were extracted and compared
byte-for-byte against the live tree: 12/12 sha256 MATCH across both DCs, covering both encrypted
CA keys, both passphrases, both CA serials and both controller bundles. Archives are 600 inside
700 folders and differ between DCs, so per-DC independence survives into the backup.

E2/E3 CLOSED LIVE. Operator ran the chmod and the intermediate removal; octavia-pki.sh verify
now reads PASS 26/0 on BOTH DCs, workspaces 12 -> 10 files, and P5 on the headend fell 18 -> 8.

A GATE DISAGREEMENT, recorded as the pattern rather than the incident. verify read 26/0 while
creds-matrix E2 still reported the CA serial mode 664 on both DCs. The chmod list predated the
serial being declared, and verify's mode assertions covered six private files and three certs --
the serial was in neither. Two gates that can both see a file must not disagree about it. Fixed
in three places: verify now mode-checks the serial (integrity, not secrecy -- rewriting it forces
the next issuance to reuse a serial), the generator's chmod includes it, and T18 pins it. The
harness then caught its own fixture creating the serial at the inherited umask; baseline went red
until the fixture matched the corrected generator. 18/18.

TFSTATE BACKUP as dc-dc-phase2 step 13, 2 rows, notes key n-tfstate-backup. Register 99 rows;
schema, render-drift, mint-ref and notes checks clean.

Three deliberate differences from the PKI backup: tfstate is DYNAMIC so the serial is recorded
both sides and the step must be re-run after every apply (restoring a stale state is actively
dangerous -- tofu then believes everything created since does not exist); the pull block proves
RESTORE by decompressing to check the archive parses as JSON and reports a serial, because a
truncated gzip passes a size check; and mint-stage is stage3, which is REACHED, so P5 correctly
reports these EXPECTED-BUT-ABSENT until the step runs rather than deferring them.

Classification recorded honestly: the outer state is credential-bearing (DOCFIX-175, MAAS API key
in plaintext); the inner states show no credential-shaped key names, consistent with the inner
root using only libvirt over qemu+ssh. Strong evidence, NOT proof -- a name-based absence cannot
exclude material inside a value -- so they are registered and stored as if sensitive. Uncertainty
resolves toward more auditing, and an unaudited backup directory beside audited ones is how the
SEC-022 shadow stores happened.

Two of my own defects caught by the gates: repo-lint L9 rejected a hardcoded clone name in the
new step (the repo has been renamed once, D-110) -- now $REPO; and two mint-refs had drifted when
a later edit added lines above their anchors, which S4 cannot catch because it only checks
line <= EOF. Both re-anchored.

Also corrected: phase-01:605 said the overlay must be in the backup set. It is derivable from the
archived workspace by 1.0-GEN.d, so the line implied a gap that does not exist.

Known, not fixed: both live inner states are 664, group-writable, and they are the authority tofu
trusts. Tightening needs its own gated change plus proof the provider preserves the mode.

creds-matrix 65/65, octavia-pki 18/18, repo-lint 0 fail / 621 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Build the PKI backup path: three surfaces asserted a backup set that did not exist ...

Operator direction: back up to the per-DC jumphost creds folder BEFORE the cert cleanup, and
record that the pinned secrets-storage solution must carry a certificate/credential backup
procedure with this step folded into it.

THE GAP, MEASURED. phase-01:594 said the workspace must be "backed up securely", :605 said it
"MUST be in the per-DC backup set", and CURRENT-STATE said the inner tfstate should be added to
"the site backup set" -- while cloud-snapshot.sh, the only candidate, is a juju-layer capture
that mentions octavia, tfstate and terraform ZERO times. Both DCs' 10-year amphora trust roots
therefore sat in one place on one VM with no copy. Losing the issuing CA key means Octavia can
never sign another amphora certificate; losing the controller CA key means the controller
certificate can never be reissued, which F9 says it must be at Stage 7.

SHAPE: one gzipped archive per DC rather than a file-by-file tree copy -- 2 register rows instead
of ~20, atomic (a half-copied tree is the dangerous state), and sha256-verifiable. The live tree
on the headend stays what octavia-pki.sh verify asserts; the archive is recovery only.

COST STATED RATHER THAN GLOSSED: a second at-rest copy of both encrypted CA keys beside their
plaintext passphrases, so the archive's contents defeat encryption-at-rest. That is the trade
D-109 option (b) was refused for; the distinction is deliberate, because that ruling governed
where the authoritative artifact lives and which host the deploy reads. A recovery copy is not a
second source of truth.

Delivered: phase-01 step 1.0-GEN.e (build on the headend, pull to the per-DC creds folder,
compare sha256 against the source BEFORE removing the staging copy -- a truncated scp would
otherwise leave a verified-looking backup of nothing); 2 register rows, custody=consolidated
since the creds folder is the SEC-009 location; notes key n-pki-backup.

THE REGISTER CAUGHT AN INCOMPLETE CHANGE OF MINE: adding rows raised S2 EXPECTED-BUT-ABSENT
twice, because the per-site manifests are DERIVED from the matrix and I had not re-rendered them.
Rendered rows appended to both; S3 render drift clean, findings back to the pre-existing 7.

Standing forward requirement recorded in the register rather than as a runbook comment: the
pinned secrets-storage solution must include a documented process and procedure for certificate
and credential backup, and this step is one of the steps that must be folded into it. The
jumphost creds folder is the interim home only.

Still not backed up and out of scope here: the two inner terraform.tfstate files on the headend
-- gitignored, untracked, state-of-record for 20 VMs, and named as owed to the same backup set
that has just been shown not to exist.

Register 97 rows / 24 octavia rows. Gauntlet ALL GREEN (89), repo-lint 0 fail / 621 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

creds consolidation batch: SEC-021(b) dc0 power key, SEC-025 NetBox GUI, V2 ruled deferral ...

Operator asked whether vcloud holds a consolidated set of login creds for every account that
exists, and what is missing / needs creating / was never moved in. Audited, then cleared the
consolidation batch on operator direction ("clear the whole consolidation batch first").

The audit's own headline: creds-audit read CLEAN on all three sites while the matrix returned
13 findings -- and the two most consequential gaps were in NEITHER surface. They were found by
enumerating the accounts that ACTUALLY EXIST. Absence of a ROW is invisible to the register,
which generalises D-137's founding argument one level up.

VERIFIED POSITIVE (both previously only asserted):
- MAAS account set COMPLETE: all 6 live accounts accounted for; admin + operator passwords on
  vcloud, juju-vr1-dc0/dc1 random+unstored BY RULING with API keys present, MAAS +
  maas-init-node MAAS-internal.
- tier-3 V1 now MEASURES maas-admin-password byte-identical to the headend source-of-record,
  so SEC-020's stale-trap risk is clear as of this date.

DONE:
- dc0 SEC-012 power key consolidated to vcloud + .pub DERIVED (SEC-021(b) as written).
  MEASURED FIRST, because the naming looked like a defect and was not: the headend
  maas-virsh_ed25519 and the snap's id_ed25519 are the SAME key; it IS dedicated (distinct
  from the dc0 service key, which is what SEC-012 requires); and dc0 using the snap's default
  identity is SEC-016's RULED design. So no re-mint, no live power path touched.
- dc1 svc .pub backfilled to the headend store (public-key material, sha256-verified).
- NetBox web-GUI admin password consolidated -- a HUMAN login that had never left the VM it
  was minted on, the third instance of the SEC-009/SEC-020 miss class. SEC-025 OPENED for the
  at-rest exposure the copy CREATES, not the gap it closed. Open rows 20 -> 21.
- V2 taught the ruled-deferral state: SEC-006's standing "revoke at completion of this
  deployment" ruling is now ACKNOWLEDGED (still naming the credential live and exposed)
  instead of failing every run. Reissuing the token would have CONTRAVENED that ruling, so the
  register was what needed to change. Match demands the cased state word AND explicit operator
  ruling attribution, and cannot cross a ledger table cell -- T58/T59 lock the evasion out.
- Added --ledger (default unchanged): V2 read the register via --repo, which the harness must
  point at the real repo for S4, so V2 had shipped with ZERO harness cases. T56-T59 added.

Findings 13 -> 7. Matrix 82 rows, harness 60/60 (was 56), creds-audit CLEAN x3, gauntlet ALL
GREEN (81), repo-lint 0-fail. The register is NOT green and is not expected to be.

RESIDUAL, all recorded: dc0-edge-api x2 (the opnsense-api.txt re-mint is a live edge mutation,
deliberately EXCLUDED from this batch), S5 x3 (RULED by SEC-016 -- needs a ruled-exception
mechanism, operator decision), S6 conflation x1 (the SEC-020 defect), E4 x2 (Stage-5/6 rows).

LOGGED NOT ACTIONED (hard rule 1): no registered root/console credential at EITHER DC edge
(measured absence of row/manifest/SEC row; what those passwords ARE is UNKNOWN and
deliberately unprobed per hard rule 2 -- vector is the LAN-reachable GUI and serial console,
not SSH, which is key-only and proven); plus the two structural blind spots that let it hide
-- S5 compares only cardinality=per-DC while all six per-site rows are office1-only, and
vm-secret-locations declares no rack/edge/cloud/unit/client location though the checker
accepts them.

Revert: per-item reverts in docs/changelog-20260727-creds-consolidation.md. The headend, snap
and NetBox VM source copies were never modified, so every file move is reversible by deletion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

SEC-020: MAAS admin-account recovery -- consolidate `admin`, mint `operator`, creds-audit CLEAN x3 ...

Operator task: recover the MAAS admin accounts and save the creds retrievably.
The task premise was FALSIFIED by measurement before any mutation: the ledger's
2026-07-25 "MAAS web-GUI login does NOT exist / never minted" was wrong. The
`admin` password was minted 2026-07-13 by site-headend-install.sh:452 and is
MEASURED working (login 204 vs 400 wrong-password control). It was merely
root-only on voffice1 and un-consolidated -- the SEC-009 miss class, not a
missing account. Scope re-ruled by the operator after that finding:
"New account + consolidate only" -- no existing password rotated.

- NEW MAAS superuser `operator` for human GUI login; password generated on
  vcloud, set via stdin (never argv -- SEC-018 discipline; changepassword has
  no --password and createadmin --password would expose it).
- `admin` password CONSOLIDATED byte-identical (sha256-verified) to
  ~/vr1-office1-creds/. NOT rotated, so the VM copy REMAINS source-of-record;
  a future rotation must update both or the VM copy becomes a stale trap.
- juju-vr1-dc0/dc1 untouched BY RULING: their SEC-018/019 API keys are
  load-bearing for the blocked Stage-5 bootstrap.
- Manifest backfill -> creds-audit CLEAN on all three sites (was RED: dc0 3 +
  dc1 1 undeclared), closing the dc0 edge-keypair queued finding named in
  vr1-dc1.manifest:18-20.
- Root cause recorded: `admin` doubles as the automation identity (19 `maas
  admin` call sites in 4 scripts), so automation never needed the password and
  nothing forced it into the folder. Proposable as the next-free D-number
  [ARCH]; NOT assigned (GA-R3 doubt-resolves-DOWN).
- Falsified ledger claim struck+corrected; machine-derived block re-seeded
  (was stale on every line: SEC 12->16, next-free D 133->137, DOCFIX 200->204).

Evidence: docs/audit/maas-admin-recovery-20260725.txt (generated, no secret
values; scanned for password bytes = 0 hits with a planted-control check).
repo-lint 0 fail; gauntlet ALL GREEN (79).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

DC0 deploy-readiness handoff: per-env-key ruling implemented + netem ruled + prep checklist ...

Prep for a fresh session to run the Stage-3 Phase-2 DC0 build. Nothing applied;
the outer tofu apply remains the first (operator-gated) mutation.

Per-env-key (D-126, operator ruling 2026-07-16, option a): vvr1-dc0 gets its OWN
access key, not office1's. Minted ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 (creds-audit
CLEAN); added variable vr1_dc0_ssh_pubkey_path; vvr1-dc0 cloud-init now installs the
dc0 pubkey (voffice1 keeps office1's key -- only the vvr1_dc0 block changed). Re-
validated + re-planned: outer plan UNCHANGED at 5 add / 0 change / 6 destroy (the
cloud-init edit is create-only content, no new destroys/replaces of live infra).

netem (D-100 gap #11): RULED = placeholder for the rehearsal; final values deferred.

docs/dc0-deploy-readiness.md (NEW): the full readiness state (outer plan verified
safe -- vvr1-dc0 416GiB/108vcpu, 6 empty-plane relocation, 980GiB RAM headroom),
remaining prep (pre-apply / mid-deploy / non-blocking), fresh-session bootstrap, and
the gated A-E deploy sequence. Ledger points at it.

Gauntlet ALL GREEN (65 harnesses incl. the 2 new access harnesses); repo-lint 0-fail;
opentofu-validate clean (both roots + 12 modules). The dc0 private key + local tfvars
stay gitignored/off-repo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH

D-126: durable rootless agent access to VR1 site VMs (shell + forward) + DC creds prep ...

Establishes durable, rootless jumphost->site-VM access so the recurring
sudo-route friction stops (and generalizes to the DCs). Operator ruled Option A
(SSH via systemd --user / ssh_config ProxyJump), advisor-conferred.

Shell layer (FOUNDATION, EXERCISED): scripts/site-ssh-config.sh generates
~/.ssh/config.d/vr1-sites ProxyJump aliases (voffice1, office1-netbox,
office1-tailscale, office1-opnsense). All four aliases proven live from the
agent's session after the operator installed with the one-key office1 env key --
key read from IdentityFile, no creds path in any command (clears the guard hook).

Forward layer (convenience, reviewed-but-unexercised): scripts/site-forward.sh
holds a systemd --user SSH local-forward -> http://127.0.0.1:8000 for office1-netbox.

Secrets boundary: the key is operator-supplied via --key (a PATH, never read/baked);
generated fragments name that path and are written locally (not committed).

DC prep: per-env-key isolation (operator ruling) -- each DC gets a DEDICATED key in
its own ~/vr1-dc<N>-creds/ folder, minted at deploy. Folders pre-created (0700);
creds-manifests/vr1-dc{0,1}.manifest declare the keypairs. DC ssh aliases deferred
until their IPs are measured post-apply.

Also: D-124 dc0 rack tfvars written locally (opentofu/d124-rack.auto.tfvars,
gitignored) -- the independent `tofu plan` unblocker. vr1_dc1_rack_* NOT written
(no ruled dc1 addressing; variables absent) -- STOP flagged for an operator ruling.

Tests: site-forward 38/38, site-ssh-config 30/30; repo-lint 0-fail (1 legacy WARN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH