Stage-3 / vr1-dc0 pre-deployment adversarial review -- 2026-07-16

Adjudicated findings from a four-charter adversarial review of the Stage-3 DC-substrate batch, run before vr1-dc0 is deployed for the first time. READ-ONLY: nothing was applied, mutated, or pushed. Every proposed fix is a PROPOSAL for the operator to gate individually; none were executed. Fixes are staged for a post-acceptance sweep, not applied mid-review.

Adjudicator: Code (main loop). Method: Layer-1 deterministic gate (facts) -> Code's own grounding of the decision-coherence register from primary text + the session transcript -> four independent charters (A1 author's-advocate, A2 prosecutor, A3 Roosevelt-hawk, A4 drift-archaeologist) in Round-1-independent + Round-2-cross-examination -> Code adjudicates. I am NOT a consensus engine: surviving dissent is recorded (section 4), not resolved away.

1. Review surface

• Repo: /home/jessea123/openstack-caracal-dc-dc (live jumphost clone). Branch dc-dc-stage3-phase2-dc-substrate, frozen at HEAD 87a7a8a (a WIP commit that froze the two uncommitted plane-review ledger files on top of the batch commit a48a60f).
• Baseline @{upstream} = 80502e9. Review range 80502e9..87a7a8a.
• Immutable artifact: docs/stage3-review-base.patch (3795 lines, 26 files, +3057/-275). Every finding cites against the repo files (line numbers as they stand at HEAD).
• IN scope: the D-121..D-124 rulings, the OpenTofu Stage-3 substrate (modules/site-wan, main.tf vr1_dc0 section, the 8 node VMs, the vvr1-dc0 rack, variables.tf), the two NetBox importers + harnesses, overlays/dc-ha-scaleup.yaml, scripts/site-headend-install.sh rack mode, the phase2 runbook rewrite, and the three changelogs.
• OUT of scope (operator-gated, correctly deferred): the live tofu apply, the NetBox --commit, the rack install, and the Stage-4 maas-vm-host wiring (DOCFIX-179).
• Undocumented-intent note (per the prompt's FIRST ACTION item 4): one design input exists as a claim but not as a file -- scratchpad/optc-calc.py, the whole-host capacity model that D-121's Option-C sizing rests on, is NOT in the tree (see R3-F06). No other design element was found to live only in prior-session discussion.
• A4 charter note: the A4 (drift-archaeologist) Round-1 pass died on an API stall mid-stream and was re-run as a standalone backfill; because Round-2 had already executed, A4 did not participate in cross-examination. Its lens (drift / status-table archaeology) was substantially covered by Code's own decision-coherence grounding and by A1. [A4 backfill: INTEGRATION PENDING -- see section 4a; this document is complete without it and will be annotated on its return.]

2. Layer-1 facts (mechanically proven; agents debate meaning, not these)

Verbatim tool output for every row below is captured under scratchpad/L1/ (ledger-scan.txt, repo-lint.txt, gauntlet.txt, tofu-validate.txt, tofu-plan.txt, ceph-optc-500.txt, state-addresses.txt, etc.); this table is the distilled register.

A2's external-authority citations (Ceph size=3-wants->=4-hosts guidance; MAAS rack-statelessness; CIS-style ip_forward hardening) are reproduced as the charter reported them; the underlying technical claims hold independently, but specific benchmark/control identifiers should be re-verified before being quoted as authoritative.

3. Findings register (ranked; severity, evidence, violation, action, contested?)

R3-F01 -- BLOCKING -- Two ADOPTED rulings rest on inferred/agent-authored operator rulings (count = 2)

The batch's single highest-value defect. An operator ruling is a VALUE; inferring it violates hard-rule-2, and here two of the four decisions carry an inferred/agent-authored ADOPTED status. The count is 2, not 1 -- the four charters converged on 1 (D-123 only) because Code's shared brief pre-asserted "D-121 is grounded via AskUserQuestion"; that biased them past the vault sub-ruling. Code verified the second case directly against the transcript (L1-13).

• (a) D-121 vault-HA backend v-a -- the code-consequential case. Status line design-decisions.md:3264: "Vault-HA backend sub-ruling: RESOLVED = (v-a) ... operator ruling". Its OWN body :3346 says "Operator sub-ruling needed." Transcript (L1-13, full question text read): vault backend was NEVER a presented choice -- the only AskUserQuestion asked "Node layout" (answered Option C) and "Proceed" (a delivery-workflow question answered "Do it all now"); neither is a vault ruling, and the operator uttered nothing about vault anywhere. So v-a is agent-authored, with no operator value. Worse, it is ENCODED in committed code: overlays/dc-ha-scaleup.yaml:67-80 re-declares vault-hacluster and re-adds [vault:ha, vault-hacluster:ha], REVERSING BUNDLEFIX-002 (which de-HA'd vault) -- a real change driven by an unratified ruling, not a status-quo path.
• (b) D-123 Model A -- the record case. Status design-decisions.md:3454: "ADOPTED Model A ... read from 'Yes, fire off those tasks' in response to the A/B question; flag if B was intended." The real utterance "Do it all now..." was the operator's answer to the AskUserQuestion "Proceed" question -- a DELIVERY-workflow choice (L1-13), NOT an A/B answer; Model A vs B was never a presented option. So the "A/B question" D-123 claims to read is one the operator was never asked. Corroborating inconsistency: the phase-2 runbook already labels D-123 PROPOSED (runbooks/dc-dc-phase2-tofu-dc-substrate.md:507 "D-123 (PROPOSED, recommend...)") while design-decisions.md marks it ADOPTED -- the record disagrees with itself about D-123's ratification. Refinement (A1, upheld in Round 2): Model A happens to equal the already-as-built D-103/D-114 node-placement seam, so -- unlike vault -- no NEW code rests on the Model-A inference (Model B would have been the change). The region+rack MAAS model and the D-124 rack block (main.tf:376-439, "# D-124", explicitly ADOPTED) are independently authorized.
• Legitimately ADOPTED on explicit authority: D-121 node layout = Option C (AskUserQuestion selection, :3397) + the 14-service scale-up ("this is the time we add the additional HA nodes"); D-122 (the operator's explicit "Ruling 2" + "Routed by fabric switches"); D-124 ("1. A 2. Confirmed", :3513).
• Violates: hard-rule-2 (no inferred value); hard-rule-1 (an inferred ruling drove the committed vault overlay). Action: demote BOTH the D-121 vault-HA axis and the D-123 Model-A axis to PROPOSED; put both one-line questions back to the operator ("vault HA backend: v-a MySQL-backed vs v-b etcd/Raft?" and "node containment: Model A vcloud-level group vs Model B nested-in-VM?"). D-121's "ADOPTED IN PART" then means what it says. Region+rack, Option C, and D-124 stand.
• Contested (severity): A1 argues MAJOR-not-BLOCKING for D-123 -- a status LABEL is not a value "entering a command", nothing is applied (L1-08), and apply is independently gated (L1-07). Adjudication: kept BLOCKING because (i) the prompt frames C2 as BLOCKING by design, (ii) the vault case DID drive committed code, and (iii) "ADOPTED" is the authority a downstream agent acts on. The dissent is real and recorded (section 4): as a matter of LIVE mutation risk, nothing is currently blocked; as a matter of decision-record integrity, both must be re-ruled before the record is trusted or apply proceeds.

R3-F02 -- MAJOR -- SEC-010: the metal-admin DC-LOCAL invariant is a ledger promise, not a committed artifact or a gate

Upheld under cross-examination (A3-F6 survived A2's "distro default is fine" refutation attempt). main.tf vvr1_dc0 cloud-init pins static IPs + one route but has NO net.ipv4.ip_forward=0 sysctl and NO host firewall on the transit leg. The rack straddles metal-admin (10.12.8.0/22, DC-local per D-052 :767 / D-100 :1946) and the office1<->dc0 transit (crosses fiber), so the "never crosses the fiber" invariant is preserved ONLY by Ubuntu's distro default -- which the deferred MAAS-rack snap install could flip. SEC-010 records this "Close BEFORE tofu apply", but a grep of scripts/ and the phase2 runbook finds NO mechanical gate; the only apply-block (L1-07) is an addressing gate, not security. Violates: D-052/D-100 (and general host-hardening guidance -- CIS-style benchmarks recommend ip_forward=0 on non-router multi-homed hosts; verify the exact control ID before quoting it). Action: add ip_forward=0 (v4+v6) + an nft/ufw transit-leg pin to the committed rack cloud-init as the ARTIFACT; wire a mechanical pre-apply gate (not a ledger note); carry the same pin onto voffice1 when its transit leg is wired (currently single-homed, main.tf:184). The pin is free -- a MAAS rack proxies at the application layer and needs no kernel forwarding.

R3-F03 -- MAJOR -- D-107's DR-independence claim is false as written (and false at Roosevelt too)

D-107 :2088: the per-DC mirror exists "so a DC can redeploy independently even if Office1 or the peer DC is down -- a DR requirement the drill exercises." Under the RULED region-on-Office1 + rack-per-DC model (D-123), a MAAS rack is stateless and cannot commission/deploy/power without the region, so an Office1 outage removes reprovisioning from BOTH DCs. And L1-10: the Phase-5 drill invokes NO MAAS -- it exercises workload failover, not redeploy. Two clauses fail.

• Nuance (A2, Round 2): the claim says "Office1 OR the peer DC is down." The peer-DC-down half is VALID (region up, mirror up -> the DC can redeploy). Scope the defect to the Office1-outage sub-case, not the whole claim.
• Elevation (A2, Round 2): buildout-design :157 mandates single-region PERMANENTLY, so this is unachievable at ROOSEVELT too -- not stale VR1 wording. A standby-region-per-DC mitigation would CONTRADICT the buildout's single-region decision.
• Violates: D-107 (its DR-independence + "drill exercises" clauses). Action (option a): amend D-107 to scope DR-independence to the in-DC artifact mirror + workload failover; add a WRITTEN, PERMANENT acceptance that node provisioning is unavailable during an Office1-region outage (day-1/2 dependency, not runtime -- running DC clouds are unaffected); delete/correct "a DR requirement the drill exercises". Do NOT reopen the topology.

R3-F04 -- MAJOR -- D-122's MAAS-controller bullet and single-object site-down wording were never superseded

• C1: D-122 :3441 "Each site runs its own MAAS controller (as voffice1 does)" -- a full region+rack -- was superseded by D-123's region-on-Office1 + rack-per-DC (operator-confirmed; matches buildout :157) but never marked. A downstream agent quoted the stale bullet back as if live. Action: append a verbatim supersession note pointing to D-123.
• C3: D-122 :3417 "virsh destroy against a single object" is void for DCs -- under Model A the containment VM does not contain the DC nodes (D-123 admits this :3482), and the Phase-5 drill already uses per-node group shutdown (L1-10), so the drill does NOT break. The stale claim also leaked into dc-dc-deployment-workflow.md:86. Action: amend D-122 AND workflow:86 to "destroy the vr1-dc0-* domain GROUP" for DCs (single-object destroy stays literal for Office1); note vvr1-dc0 at a DC is a MAAS rack headend, not a D-114 containment VM (rack mode runs no LXD/compose).
• Violates: D-123 (supersedes both). Topology operator-confirmed; record fix only.

R3-F05 -- MAJOR -- D-107 vs D-123 "node artifacts" wording (C4): reconcilable, but D-107 must be scoped

D-107 :2087-89 ("no node artifacts served from Office1"; "images including amphora ONLY from an in-DC mirror") reads against D-123 :3459 (rack "PROXIES OS images from the Office1 region"). The counter-reading HOLDS: the intra-MAAS region->rack commission/deploy image channel is architecturally distinct from D-107's supply-chain mirror (apt/snap/charmhub/registry/amphora served at/after juju deploy). Both stand once D-107 is scoped. Violates: NONE (a wording gap in D-107). Action: amend D-107 to scope "node artifacts" to the supply-chain classes, explicitly excluding the intra-MAAS provisioning-image proxy D-123 routes region->rack. Pairs with R3-F03.

R3-F06 -- MAJOR -- D-121's Option-C sizing rests on an uncommitted, now-absent model; the "reproducible" claim is false

scratchpad/optc-calc.py -- the whole-host validation that grounds Option C's 222 vCPU / 790 GiB / 77%-RAM ruling -- is NOT tracked and is ABSENT from disk (L1-12), yet the changelog claims "Resource model reproducible: python3 scratchpad/optc-calc.py". Violates: repo-authoritative discipline (CLAUDE.md); D-121 record integrity. Contested (severity, A1 Round 2): the Option-C ruling is an explicit operator selection (not gated on the calculator), and fit is independently checkable -- node literals are committed in main.tf locals and the disk dimension re-ran from the COMMITTED dc-dc-ceph-disk-budget.sh (L1-09 PASS). So the ruling is not "unverifiable"; the defect is reproducibility hygiene. Action: promote optc-calc.py into a committed, harnessed calculator (already logged as a follow-up) and re-run for Option C before the sizing is treated as measured; correct the changelog's "reproducible" claim until it is.

R3-F07 -- MAJOR -- D-121 records an Option-B disk validation for the ruled Option-C layout

D-121 :3382 records the disk-budget PASS for "Option B 4+4" while the RULED layout is Option C (3 storage/DC). Code re-ran for Option C (L1-09): PASS, 6.48 TiB margin -- capacity is fine, so this is a RECORD defect, not a capacity one. Action: re-record the Option-C validation (cite L1-09).

R3-F08 -- MINOR -- Option C's 3 OSD hosts at size=3 carry zero Ceph rebuild headroom (unrecorded)

D-121 flagged "size=3 has ZERO rebuild headroom" for the REJECTED Option A :3358 but not for the ADOPTED Option C, whose 3 storage hosts have the identical property. Walked back from MAJOR in Round 2 (A1/A2/A3 concur): with Charmed default min_size=2 the pool keeps SERVING on one host loss (the availability drill is clean), ceph-mon=3 sits on the CONTROL nodes (quorum untouched), and only the RE-REPLICATION sub-case lacks headroom -- an inherent size=3 economy that resolves at Roosevelt (>=4 storage hosts). Action: add an accepted-risk note to the Option-C record (a storage-node-loss drill will show degraded-not-self-healing; Roosevelt remedy = >=4 storage/DC).

R3-F09 -- MINOR -- D-115 reads PROPOSED to ledger-scan though it is ADOPTED by amendment (L1)

D-115's primary Status line retains the substring "Originally PROPOSED/OPEN", which trips ledger-scan's regex (L1-02), while the decision IS ratified by its 2026-07-13 amendment (:2727). So the office carve import + PR #1 merge rest on a legitimately ADOPTED decision -- NOT an unratified-executed one. Action: move "Originally PROPOSED/OPEN" out of the Status line (or harden ledger-scan to ignore an "originally/was PROPOSED" clause when ADOPTED is present). Sweep D-121..D-124 Status lines for the same machine-vs-human record divergence.

R3-F10 -- MINOR -- SEC-011 (node least-connectivity) is CONTESTED on Roosevelt-fidelity grounds

Code's own prior SEC-011 (storage nodes get provider-public + data-tenant legs they never bind) is challenged by A2 (Round 2, REFUTED) and A1 (WEAKENED): (1) in VR1 every plane is isolated L2 with no gateway/route out, so an unbound vNIC has NO exploitable reachability -- "attack surface" is nil today; (2) pruning would OVERRIDE the ADOPTED D-122 6-NIC ruling without a superseding decision AND INCREASE delta to Roosevelt, where D-052's planes are VLAN-trunked on bonded NICs (ALL planes present at every node; netplan/MAAS decides L3 binding), so uniform 6 isolated-L2 vNICs is the more faithful model. Adjudication: downgrade SEC-011 from "pre-apply hardening" to an OBSERVATION / operator's call; recommend amending the SEC-011 ledger row to record the Roosevelt-delta caveat. Surviving dissent (section 4).

R3-F11 -- MINOR -- Stale "DC1/DC2 + supernet unassigned" prose in the workflow doc (L3)

dc-dc-deployment-workflow.md Authoring-status cell still reads "DC1-first; DC2 hard-gated (D-101 supernet unassigned)" -- superseded by D-119 (code is vr1-dc0/vr1-dc1) and D-115 (vr1-dc1 supernet = 10.12.64.0/19 assigned). No gate depends on the stale reading (the second DC is sequenced out regardless). Action: update the cell; drop the "unassigned" clause.

R3-F12 -- OBSERVATION -- Transient Octavia N+1 amphora failover headroom is not modeled (C7b)

The whole-host validation is a steady-state sum; it does not visibly reserve the transient N+1 amphora placement headroom Octavia STANDALONE failover needs (a hard-won VR0 finding), and the Phase-5 drill exercises failover on two clouds sharing one host. Mitigated in this single-host sim (a hard-downed DC frees its RAM for the survivor) and unverifiable until R3-F06's model is committed. Action: when optc-calc.py lands, add a line on per-cloud transient amphora headroom (or state it is a Roosevelt-only concern for this sim).

R3-F13 -- OBSERVATION -- D-122 "6 NICs = baremetal-matched" is imprecise; Section-9 could add a cross-ref

(i) D-122 sizes nodes at "6 NICs, one per plane" but the Roosevelt realization (D-052) is untagged metal-admin + tagged VLAN subinterfaces trunked on bonded NICs, not 6 discrete physical NICs -- L2 isolation is behaviorally equivalent; a characterization nuance, no code change. (ii) A3 argued Section 9 omits the site-wan simulated-ISP and the virsh-destroy DR fault-injection; largely REFUTED in Round 2 (both have Roosevelt analogs -- a real circuit, a real facility-down drill -- so neither meets the register's "no analog" bar; the register is a build-step register and virsh-destroy is an operational step). At most add a clarifying cross-reference for the containment/virsh-destroy sim vehicle. Not a material incompleteness.

4. Minority report (surviving dissent -- recorded, not resolved away)

• R3-F01 severity (dissenter: A1). A1 holds D-123's inferred-status is MAJOR ledger-hygiene, not deploy-BLOCKING: a status label is not a value "entering a command" (hard-rule-2's literal scope), nothing is applied (L1-08), and apply is independently gated (L1-07). A1 further showed (Round 2, refuting A3-F1) that Model A = the already-as-built D-103/D-114 path, so no NEW code rests on that specific inference. Code's adjudication keeps R3-F01 BLOCKING on decision-record + governance grounds (and because the VAULT half DID drive committed code), but records A1's point: as pure live-mutation risk, nothing is currently blocked.
• R3-F10 SEC-011 (dissenter: A2). A2 refutes the "attack surface" framing outright: isolated-L2 planes have no exploitable reachability, and pruning increases Roosevelt delta. Code accepts the Roosevelt-delta point and downgrades SEC-011 accordingly, but records that A2 would go further and strike the finding as a non-issue; Code retains it as an OBSERVATION worth the operator's note.
• R3-F08 Ceph severity (dissenters: A1, A2, A3 concur down; A2-R1 held MAJOR). A2's Round-1 MAJOR ("opposite of the clean drill C is sold on") did not survive its own and others' Round-2 scrutiny (min_size=2 keeps serving; mon on control nodes). Recorded as the walk-back it was; final severity MINOR.

Convergence check (mandated). The three live charters converged on the C-register verdicts -- expected, because those items are text-provable quotations, not judgment calls. But they did NOT fully converge: each surfaced a distinct high-value finding (A2: external CIS/Ceph/MAAS citations; A3: the Section-9 completeness challenge + the baremetal NIC-realization nuance; A1: the uncommitted optc-calc.py), and Round-2 produced genuine dissent (above). So the charters separated adequately. The one place convergence WAS a failure mode: all three reported inferred-ruling count = 1 because Code's shared brief pre-asserted D-121's grounding -- a demonstration that shared priors create shared blind spots. Code's independent transcript check (L1-13) corrected the count to 2.

4a. A4 backfill integration (returned; corroborates)

The A4 drift-archaeologist backfill returned and independently reached inferred-ruling count = 2 (D-121 vault-(v-a) + D-123 Model A), corroborating R3-F01 by the harder path: A4 was tasked to scrutinize the vault Status-vs-body contradiction directly, and reached count = 2 WITHOUT the biasing "D-121 is grounded" hint the three Round-1 charters received. This confirms the section-4 convergence diagnosis -- the count-1 result was a shared-prior artifact, not a real ceiling. A4 did NOT participate in Round-2 cross-examination (its Round-1 died on an API stall; Round-2 had already run). Integration -- with one A4 claim verified-and-rejected (I check agent citations, not rubber-stamp them):

• A4's count = 2 corroborated. A4's F1 (D-123 Model A) + F2 (D-121 vault, Status :3264 vs body :3346, no vault utterance in the batch changelog) independently match R3-F01.
• A4's F7 REJECTED on direct check. A4 claimed the phase-2 exit gate cites the stale D-122 "each site runs its own controller" bullet (runbooks/dc-dc-phase2-tofu-dc-substrate.md:636). Verified: the runbook does NOT -- at :507 it cites "D-123 (PROPOSED, recommend...)" and the exit gate (:630-635) uses the rack-controller-per-DC (D-123) model. A4 miscited. NOTABLE side effect: the runbook already labels D-123 PROPOSED, which CONTRADICTS design-decisions.md's ADOPTED status and independently supports R3-F01 (the D-123 record is internally inconsistent about its own ratification). Added to R3-F01's evidence.
• A4 otherwise agrees across C1-C7 and L1-L5 (it frames C4 as "REFUTED -- both stand", the same disposition as R3-F05's "reconcilable, scope D-107"), and adds clean negative findings: no false DONE/as-executed claims, state reconciles with zero orphans, byte/lint hygiene clean.

5. Leads register

6. Go / No-Go on Stage 3

NO-GO for tofu apply as it stands -- on hard-gate grounds, NOT because the substrate design is unsound (it is sound: planes are isolated L2, state reconciles with no orphans, gauntlet green, Option C fits with margin, validate passes).

Blocking conditions to clear (each operator-gated, applied in a post-acceptance sweep):

1. Re-rule the two inferred axes (R3-F01): D-121 vault-HA (v-a vs v-b) and D-123 node containment (Model A vs B). Both are one-line operator questions. Until then the vault overlay and the Model-A record are unratified.
2. Ship the SEC-010 artifact + mechanical pre-apply gate (R3-F02). DC-LOCAL must be enforced, not promised.
3. Assign the rack transit/30 + rack IP in office1-netbox and feed the 4 tfvars (L1-07) so a plan can render and be reconciled against state. Wire voffice1's transit leg (or gate the rack route) so the route peer exists.

Conditional-GO once 1-3 are met AND the record amendments (R3-F03..F07, F09, F11) are applied and optc-calc.py is committed + re-run for Option C (R3-F06). The record corrections do not block the substrate build; they block treating the DECISIONS as coherent, which is the operator's stated concern. The MINOR/OBSERVATION items (F08, F10, F12, F13) are for the operator's judgment and need not gate the sweep.

Prepared read-only; not committed, not merged, not pushed. Presented for operator review. Fixes staged for an individually-gated post-acceptance sweep.