# VR1 Stage 4 record -- Phase 3 "MAAS enlist / commission / deploy (per DC)"

**OPENED 2026-07-23** on branch `dc-dc-stage4-phase3-maas-deploy` (off post-merge `main`
after the Stage 3 / G12 merge). **CLOSED 2026-07-27.** 76 commits.

`docs/CURRENT-STATE.md` is the status authority for everything below; this record is
history and a pointer table. Nothing here may be cited as status (GA-R1).

## What the stage actually delivered

The stage's goal as authored was "bring the discovered node VMs to a deployed OS, ready
for Juju". That changed shape en route, by ruling: **MAAS-deploy is SKIPPED.** The deploy
model was ruled a **READY handoff** (2026-07-23) -- Juju provisions the nodes at Stage 5
per the phase-01 precondition -- and DOCFIX-200 rewrote the runbook's Step 4, which as
written would have broken Stage 5. So the delivered end-state is 18 nodes **Ready**,
carved, tagged, with both DCs' artifact paths serving.

- **Carve executed both DCs** (logged window `stage4-carve`, 2026-07-23): dynamic ranges
  .201-.254, 10 named plane fabrics, 90 NIC re-homes across 18 nodes each read back
  against its pinned MAC, 2 provider subnets moved off fabric-5 + 8 plane subnets created,
  6 spaces with 12 VLANs, 18 nodes' statics + br-ex OVS. All 18 stayed Ready.
  Rulings behind it: **D-133** (flat per-NIC carve for VR1; hardware-faithful bonds/trunks
  deferred to the next deployment) and **D-134** + its Option-B amendment (contiguous
  bands: .4-.49 utility, .50-.99 VIP, nodes .100-.200, dynamic .201-.254).
- **Placement tags** `openstack-vr1-dc0` / `-dc1` applied 9+9 (D-119 naming), and
  **verified live 2026-07-27** -- the 07-25 decision-recon had flagged them
  NEEDS-LIVE-VERIFY because the discovery capture carried no tag column.
- **jammy boot images synced** to the region (selection id=2).
- **Per-DC artifact delivery = D-135**, and this is the stage's most-amended decision.
  The rehearsal-scoped D-107 exception was REJECTED; mirrors were built. Then the
  2026-07-24 amendment split the two DCs, and the 2026-07-27 amendment made that split
  the deliberate experiment rather than an interim: **dc0 tests the full mirror
  (949G+342M), dc1 tests the apt caching proxy.** dc1's partial 330G mirror and its whole
  apparatus were removed at close.
- **Tooling shipped:** `dc-mirror.sh` (site-keyed check/install/sync), `dc-cache-proxy.sh`
  (and, at close, its own utility net layer), `creds-matrix.py` + the D-137 register,
  preflight **P5**, plus `lib-hosts.sh`'s VR1 arms and D-133 guards in the VR0 flow scripts.

## The three things a future session should read first

1. **The READY-handoff ruling (DOCFIX-200).** Stage 4 does NOT deploy an OS. Four surfaces
   still said "nodes Deployed" as late as 2026-07-27 and were corrected then; if a fifth
   turns up, that is the same defect, not a new fact.
2. **The two DCs' artifact paths are DIFFERENT BY RULING.** Checking dc1 as a mirror fails
   by design. `dc-cache-proxy.sh check dc1` is dc1's gate; `dc-mirror.sh check dc0` is dc0's.
3. **A checker that cannot fail is not a gate.** Both of this stage's late findings were
   false greens in controls that had been passing for days -- see below.

## False greens found in this stage's own controls (the pattern worth carrying forward)

This stage twice discovered that a control everyone trusted could not fail. Both were
found by *running* the control against reality rather than reading its output history:

- **`creds-audit` is declaration-based**, so an undeclared secret is structurally
  invisible. It read CLEAN on all three sites while the D-137 matrix returned 13 findings.
  Beyond that, the matrix itself cannot report the absence of a credential it has no ROW
  for -- which is how an unregistered root credential at both DC edges, and a human GUI
  login that had never left the VM it was minted on, both survived every audit. The fix
  that found them was to ENUMERATE what exists (`maas admin users read`) instead of
  auditing what is declared.
- **`dc-mirror.sh check` asserted that `last-sync.status` EXISTED**, printing its contents
  behind an unconditional `OK`. dc0 read `FAIL ... ubuntu=255` and dc1 read a four-day-stale
  `RUNNING`; both printed OK and the check exited PASS. Fixed at close (status word
  case-analysed; `RUNNING` is an explicit UNKNOWN cross-checked against the unit).

The D-137 six-lens committee audit (2026-07-26) is the same lesson at scale: it found
**8 false greens** in a register built four days earlier, four lenses converging on one
root cause (tier 2 keyed existence on host-role alone, ignoring site, so one site's
credential satisfied every site).

## Incidents

- **dc1 edge passed NO LAN traffic** after v4 addressing (2026-07-23): `configctl interface
  reconfigure` applies the address but does not regenerate the pf ruleset, so the automatic
  outbound NAT for the re-addressed LAN was never created. Self-masking -- dc0's earlier
  qga/plugin work had reloaded the filter as a side effect and hid it. Fixed live with
  `configctl filter reload`; **prevented in the tool at close** (2026-07-27 ruling), with the
  reload deliberately placed on a FRESH connection to the post-move address, because
  re-addressing the interface you arrived on drops the session during the reconfigure.
- **dc0 mirror's overnight incremental failed** on a transient upstream `500 read timeout`
  for `dists/jammy/Release` (2026-07-27). Content was intact; a re-run succeeded in 20s.
  Only visible because the check was fixed first.
- **A fabricated reference** ("D-193") appeared in 2026-07-24 analysis, was corrected to
  D-121, and was verified never to have reached the repo. Recorded because the cardinal sin
  is fabrication, and owning it is the control.

## Gates and rulings

Gate rows are in `docs/CURRENT-STATE.md` section 6. This stage opened **G17** (per-DC
artifact source reachable FROM A NODE) by operator ruling 2026-07-27, splitting the
node-side half of DoD bullet 5 out rather than closing it conditionally -- GA-R6 E3 forbids
a conditional close, and the READY-handoff ruling leaves nodes powered off, so no node-side
probe can run in this stage. G17's trigger is Stage 5 first boot; the phase-4 runbook now
owns its captures. DoD bullet 6 ("NTP from the DC's own OPNsense edge") was STRUCK by
DOCFIX-204 as unsatisfiable -- D-129(iv) had already ruled the MAAS hierarchy the time
authority, so a correctly-built DC could never have met it.

Decisions adopted or amended in this stage: **D-133**, **D-134** (+amendment), **D-135**
(+two amendments), **D-137** (five sub-rulings, all individually ruled), plus amendments to
D-104 (dedicated 10th controller VM), D-101 (address family CONFIRMED), and D-113 (the
set-interface reload). SEC rows opened: **SEC-017** through **SEC-025** (nine), of which
SEC-024 was opened and remediated the same day.

## Close-out set (GA-R6)

Executed 2026-07-27: every substantive in-stage item closed or split to G17; carve residue
cleaned (108 fabrics -> 17, cascade-checked); dc1 mirror removed after making the proxy
self-sufficient; GA-R2 consolidation (this record); skill sweep + dated snapshot
regeneration; GA-R7 memory review; final gauntlet **ALL GREEN (81)** + repo-lint 0-fail;
operator-gated merge to `main` as a merge commit; branch retired.

## Sessions (now `docs/archive/changelogs/`)

| file | scope | lines |
|---|---|---|
| changelog-20260723-caveman-optin-guard.md | caveman opt-in guard (pre-disable) | 46 |
| changelog-20260723-stage4-open.md | Stage 4 OPEN, D-133/D-134, carve both DCs, D-135 + dc-mirror.sh, dc1 edge pf incident | 249 |
| changelog-20260724-caveman-disable.md | caveman plugin DISABLED + SEC-017 + guardrails; DC1 proxy build start; D-135 amendment | 225 |
| changelog-20260724-stage5-committee-fork1.md | bundle-render committee, D-104 amendment (10th VM), BUNDLEFIX-052, validator spine | 294 |
| changelog-20260725-maas-admin-recovery.md | SEC-020 premise FALSIFIED, `operator` superuser minted, admin.pass consolidated; D-137 PROPOSED | 263 |
| changelog-20260726-d137-tier1.md | D-137 tiers 1-3 built, six-lens committee audit, 8 false greens remediated, preflight P5 wired, SEC-024 | 772 |
| changelog-20260727-creds-consolidation.md | consolidation batch (SEC-021(b)/SEC-025), V2 ruled-deferral, permission allowlist, mirror-gate fix, G17, DOCFIX-204, dc1 teardown, carve residue, set-interface-v4 reload | 499 |

Captures for this stage are in `docs/audit/` (23 added on the branch), including
`stage4-discovery-20260723.txt`, `stage4-carve-verify-20260723.txt`,
`stage4-mirror-gate-20260727.txt`, `dc1-mirror-teardown-20260727.txt`,
`stage4-carve-residue-cleanup-20260727.txt`, `creds-consolidation-audit-20260727.txt`,
and the D-137 committee/remediation set.
