Reissue backup custody: transferred to vcloud, and REGISTERED ...

Operator: 'Update the matrix as needed. Transfer the artifacts to the vcloud.
keep the pin that during the secrets workflow planning that the creds and certs
from this step need to be included.'

CUSTODY (Step 1.0-REISSUE.4, executed): both pre-reissue archives pulled from the
headend to the jumphost SEC-009 creds folders, sha256 compared and IDENTICAL at
both ends (e30cab7f...3487 dc0, 17c831da...6fe3 dc1), 0600, each listing 15
entries including the deploy overlay and the CA serial. Headend staging copies
removed; ~/octavia-pki/backups/ gone; no .reissue-* residue. The sha256 compare
happens BEFORE the delete because a truncated scp would otherwise leave a
verified-looking backup of nothing.

REGISTER: two octavia-reissue-backup rows (per-DC, jumphost, consolidated,
templated filename <site>-octavia-pki-<stamp>.tar.gz), the DERIVED manifests
updated to match (D-137 ruling 2 -- manifests are generated, not hand-authored),
and a new n-reissue-backup note. It is deliberately a SEPARATE id from
n-pki-backup: that one is the generation-time workspace archive, this one is
per-ROTATION and additionally carries the deploy overlay, which the generation
archive does not -- restoring only the workspace would leave the half that
actually reaches the charm unrecoverable.

Measured after: tier1 101 rows / SAME 5 pre-existing findings; tier2 the same 7,
with NO reissue-attributable finding -- and tier2 FOUND both archives at their
declared location, which is the point of registering them. Harness 65/65.

HAZARD RECORDED: these archives contain controller-ca.cert.srl, the CA's issuance
state. Restoring one over a workspace that has issued since rolls the serial
BACKWARDS and the next mint reuses a serial the estate already holds. Serials
burned 2026-07-30: ...250E (dc0), ...674DE (dc1).

PINNED: the note carries the operator's standing requirement that the creds and
certs from this step be included in the secrets-storage workflow planning, with
the full scope enumerated. The jumphost creds folder is the INTERIM home only.

gauntlet ALL GREEN (89); repo-lint 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

tfstate backup path built; E2/E3 closed live; a gate disagreement found and fixed ...

BACKUP PROVEN BY RESTORE, not by existence. Both PKI archives were extracted and compared
byte-for-byte against the live tree: 12/12 sha256 MATCH across both DCs, covering both encrypted
CA keys, both passphrases, both CA serials and both controller bundles. Archives are 600 inside
700 folders and differ between DCs, so per-DC independence survives into the backup.

E2/E3 CLOSED LIVE. Operator ran the chmod and the intermediate removal; octavia-pki.sh verify
now reads PASS 26/0 on BOTH DCs, workspaces 12 -> 10 files, and P5 on the headend fell 18 -> 8.

A GATE DISAGREEMENT, recorded as the pattern rather than the incident. verify read 26/0 while
creds-matrix E2 still reported the CA serial mode 664 on both DCs. The chmod list predated the
serial being declared, and verify's mode assertions covered six private files and three certs --
the serial was in neither. Two gates that can both see a file must not disagree about it. Fixed
in three places: verify now mode-checks the serial (integrity, not secrecy -- rewriting it forces
the next issuance to reuse a serial), the generator's chmod includes it, and T18 pins it. The
harness then caught its own fixture creating the serial at the inherited umask; baseline went red
until the fixture matched the corrected generator. 18/18.

TFSTATE BACKUP as dc-dc-phase2 step 13, 2 rows, notes key n-tfstate-backup. Register 99 rows;
schema, render-drift, mint-ref and notes checks clean.

Three deliberate differences from the PKI backup: tfstate is DYNAMIC so the serial is recorded
both sides and the step must be re-run after every apply (restoring a stale state is actively
dangerous -- tofu then believes everything created since does not exist); the pull block proves
RESTORE by decompressing to check the archive parses as JSON and reports a serial, because a
truncated gzip passes a size check; and mint-stage is stage3, which is REACHED, so P5 correctly
reports these EXPECTED-BUT-ABSENT until the step runs rather than deferring them.

Classification recorded honestly: the outer state is credential-bearing (DOCFIX-175, MAAS API key
in plaintext); the inner states show no credential-shaped key names, consistent with the inner
root using only libvirt over qemu+ssh. Strong evidence, NOT proof -- a name-based absence cannot
exclude material inside a value -- so they are registered and stored as if sensitive. Uncertainty
resolves toward more auditing, and an unaudited backup directory beside audited ones is how the
SEC-022 shadow stores happened.

Two of my own defects caught by the gates: repo-lint L9 rejected a hardcoded clone name in the
new step (the repo has been renamed once, D-110) -- now $REPO; and two mint-refs had drifted when
a later edit added lines above their anchors, which S4 cannot catch because it only checks
line <= EOF. Both re-anchored.

Also corrected: phase-01:605 said the overlay must be in the backup set. It is derivable from the
archived workspace by 1.0-GEN.d, so the line implied a gap that does not exist.

Known, not fixed: both live inner states are 664, group-writable, and they are the authority tofu
trusts. Tightening needs its own gated change plus proof the provider preserves the mode.

creds-matrix 65/65, octavia-pki 18/18, repo-lint 0 fail / 621 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Build the PKI backup path: three surfaces asserted a backup set that did not exist ...

Operator direction: back up to the per-DC jumphost creds folder BEFORE the cert cleanup, and
record that the pinned secrets-storage solution must carry a certificate/credential backup
procedure with this step folded into it.

THE GAP, MEASURED. phase-01:594 said the workspace must be "backed up securely", :605 said it
"MUST be in the per-DC backup set", and CURRENT-STATE said the inner tfstate should be added to
"the site backup set" -- while cloud-snapshot.sh, the only candidate, is a juju-layer capture
that mentions octavia, tfstate and terraform ZERO times. Both DCs' 10-year amphora trust roots
therefore sat in one place on one VM with no copy. Losing the issuing CA key means Octavia can
never sign another amphora certificate; losing the controller CA key means the controller
certificate can never be reissued, which F9 says it must be at Stage 7.

SHAPE: one gzipped archive per DC rather than a file-by-file tree copy -- 2 register rows instead
of ~20, atomic (a half-copied tree is the dangerous state), and sha256-verifiable. The live tree
on the headend stays what octavia-pki.sh verify asserts; the archive is recovery only.

COST STATED RATHER THAN GLOSSED: a second at-rest copy of both encrypted CA keys beside their
plaintext passphrases, so the archive's contents defeat encryption-at-rest. That is the trade
D-109 option (b) was refused for; the distinction is deliberate, because that ruling governed
where the authoritative artifact lives and which host the deploy reads. A recovery copy is not a
second source of truth.

Delivered: phase-01 step 1.0-GEN.e (build on the headend, pull to the per-DC creds folder,
compare sha256 against the source BEFORE removing the staging copy -- a truncated scp would
otherwise leave a verified-looking backup of nothing); 2 register rows, custody=consolidated
since the creds folder is the SEC-009 location; notes key n-pki-backup.

THE REGISTER CAUGHT AN INCOMPLETE CHANGE OF MINE: adding rows raised S2 EXPECTED-BUT-ABSENT
twice, because the per-site manifests are DERIVED from the matrix and I had not re-rendered them.
Rendered rows appended to both; S3 render drift clean, findings back to the pre-existing 7.

Standing forward requirement recorded in the register rather than as a runbook comment: the
pinned secrets-storage solution must include a documented process and procedure for certificate
and credential backup, and this step is one of the steps that must be folded into it. The
jumphost creds folder is the interim home only.

Still not backed up and out of scope here: the two inner terraform.tfstate files on the headend
-- gitignored, untracked, state-of-record for 20 VMs, and named as owed to the same backup set
that has just been shown not to exist.

Register 97 rows / 24 octavia rows. Gauntlet ALL GREEN (89), repo-lint 0 fail / 621 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

SEC-020: MAAS admin-account recovery -- consolidate `admin`, mint `operator`, creds-audit CLEAN x3 ...

Operator task: recover the MAAS admin accounts and save the creds retrievably.
The task premise was FALSIFIED by measurement before any mutation: the ledger's
2026-07-25 "MAAS web-GUI login does NOT exist / never minted" was wrong. The
`admin` password was minted 2026-07-13 by site-headend-install.sh:452 and is
MEASURED working (login 204 vs 400 wrong-password control). It was merely
root-only on voffice1 and un-consolidated -- the SEC-009 miss class, not a
missing account. Scope re-ruled by the operator after that finding:
"New account + consolidate only" -- no existing password rotated.

- NEW MAAS superuser `operator` for human GUI login; password generated on
  vcloud, set via stdin (never argv -- SEC-018 discipline; changepassword has
  no --password and createadmin --password would expose it).
- `admin` password CONSOLIDATED byte-identical (sha256-verified) to
  ~/vr1-office1-creds/. NOT rotated, so the VM copy REMAINS source-of-record;
  a future rotation must update both or the VM copy becomes a stale trap.
- juju-vr1-dc0/dc1 untouched BY RULING: their SEC-018/019 API keys are
  load-bearing for the blocked Stage-5 bootstrap.
- Manifest backfill -> creds-audit CLEAN on all three sites (was RED: dc0 3 +
  dc1 1 undeclared), closing the dc0 edge-keypair queued finding named in
  vr1-dc1.manifest:18-20.
- Root cause recorded: `admin` doubles as the automation identity (19 `maas
  admin` call sites in 4 scripts), so automation never needed the password and
  nothing forced it into the folder. Proposable as the next-free D-number
  [ARCH]; NOT assigned (GA-R3 doubt-resolves-DOWN).
- Falsified ledger claim struck+corrected; machine-derived block re-seeded
  (was stale on every line: SEC 12->16, next-free D 133->137, DOCFIX 200->204).

Evidence: docs/audit/maas-admin-recovery-20260725.txt (generated, no secret
values; scanned for password bytes = 0 hits with a planted-control check).
repo-lint 0 fail; gauntlet ALL GREEN (79).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

G12 dc1: SEC-016 power key wired + commissioning 9/9 READY -- [V] leg complete ...

- SEC-016 dedicated dc1 MAAS->libvirt power key minted, authorized on the
  rack, installed in the region MAAS snap (per-host ssh config; dc0 SEC-012
  key untouched -- true per-DC isolation); manifest updated (creds-audit CLEAN)
- maas-node-power.sh --commit: power_type=virsh on all 9, each verified by a
  real query-power-state (proves the SEC-016 region->rack path)
- commissioned all 9 -> ALL 9 READY in ~3.5 min (no timeout, no SERVFAIL),
  shapes EXACT to D-121 Option C; dc0's two stacked faults pre-empted by
  pinned MACs + the dc-rack-net forwarder
- G12 [V] leg (dc1 build) COMPLETE; NEXT is close-out + operator-gated merge

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw

G12 dc1: edge bootstrap + v4 addressing COMPLETE (D-112c + D-113a2 proven on 26.7) ...

Edge phase for vr1-dc1, resuming the 2026-07-22 close handoff:
- D-112(c) console bootstrap: SSH + dc1 edge key materialized (payload
  needed util.inc/shell_safe() -- dc0 lesson iv the .b64 artifact lacked)
- key-only SSH verified (15.1-RELEASE-p1)
- D-113(a2) API key minted via vendor model; GET core/firmware/status
  exit 0, product_abi 26.7
- edge addressed: WAN 172.30.3.2/24 gw 172.30.3.1 (egress 0% loss),
  LAN -> 10.12.64.1/22 (ruled provider-public gw), API at new LAN
- interim reach leg removed; rack provider-public leg 10.12.64.2/22 live
- creds consolidated to ~/vr1-dc1-creds/ (creds-audit CLEAN); rack edge-key
  copy shredded -> SEC-015 (transient, remediated)

Two queued findings: bootstrap .b64 missing util.inc; opnsense-bootstrap-apikey.sh
scp transient post-restart-sshd failure (readiness-wait/retry candidate).

CURRENT-STATE G12 row + security-ledger SEC-015 + manifest updated (GA-R1/C1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw

D-126: durable rootless agent access to VR1 site VMs (shell + forward) + DC creds prep ...

Establishes durable, rootless jumphost->site-VM access so the recurring
sudo-route friction stops (and generalizes to the DCs). Operator ruled Option A
(SSH via systemd --user / ssh_config ProxyJump), advisor-conferred.

Shell layer (FOUNDATION, EXERCISED): scripts/site-ssh-config.sh generates
~/.ssh/config.d/vr1-sites ProxyJump aliases (voffice1, office1-netbox,
office1-tailscale, office1-opnsense). All four aliases proven live from the
agent's session after the operator installed with the one-key office1 env key --
key read from IdentityFile, no creds path in any command (clears the guard hook).

Forward layer (convenience, reviewed-but-unexercised): scripts/site-forward.sh
holds a systemd --user SSH local-forward -> http://127.0.0.1:8000 for office1-netbox.

Secrets boundary: the key is operator-supplied via --key (a PATH, never read/baked);
generated fragments name that path and are written locally (not committed).

DC prep: per-env-key isolation (operator ruling) -- each DC gets a DEDICATED key in
its own ~/vr1-dc<N>-creds/ folder, minted at deploy. Folders pre-created (0700);
creds-manifests/vr1-dc{0,1}.manifest declare the keypairs. DC ssh aliases deferred
until their IPs are measured post-apply.

Also: D-124 dc0 rack tfvars written locally (opentofu/d124-rack.auto.tfvars,
gitignored) -- the independent `tofu plan` unblocker. vr1_dc1_rack_* NOT written
(no ruled dc1 addressing; variables absent) -- STOP flagged for an operator ruling.

Tests: site-forward 38/38, site-ssh-config 30/30; repo-lint 0-fail (1 legacy WARN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH