diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index ece3a09..0139f13 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -822,7 +822,17 @@ -> 5/0/0 dc1 substrate adds (G12 step A, saved-plan applied exact 2026-07-22) -> zero diff converged -> 0/1/0 office1 qga channel (G13 bundle, saved-plan applied exact 2026-07-23) -> zero diff converged -(`docs/audit/outer-plan-20260723-postqga-converged.txt`, this entry). +(`docs/audit/outer-plan-20260723-postqga-converged.txt`, this entry) +-> **RE-CONFIRMED ZERO DIFF 2026-07-27** by the Stage-5 grounding audit, and for +the FIRST TIME across ALL THREE roots in one session: the outer root on vcloud +(`docs/audit/outer-plan-20260727-stage5-audit.txt`, "No changes") AND both inner +roots on voffice1 (`vr1-dc0-substrate` and `vr1-dc1-substrate`, each "No changes", +exit 0). Validity of the inner pair was established FIRST by proving the voffice1 +clone's inner roots are byte-identical to `main` despite that clone being 105 +commits behind (`git diff --stat 61c416e..main -- opentofu/vr1-dc0-substrate/ +opentofu/vr1-dc1-substrate/ opentofu/modules/` -> EMPTY); had that diff been +non-empty the inner plans would have been UNMEASURED, not green. Full capture: +`docs/audit/stage5-live-measurement-20260727.txt`. ## 6. Open gates @@ -864,7 +874,8 @@ | Kernel (voffice1) | 6.8.0-136-generic | `ssh voffice1 'uname -r'` | voffice1 | | OPNsense edge | 26.7.1 (FreeBSD base 15.1) | MEASURED 2026-07-23 via the gated API (`GET core/firmware/status` -> product_version 26.7.1, capture `docs/audit/g13-close-20260723.txt`); updated 26.7 -> 26.7.1 in the G13 bundle. DC edges (vr1-dc0/dc1) remain 26.7 | office1-opnsense | | NetBox (Office1 apex) | 4.6.4 per as-built `docs/vr1-office1-as-built.md:44`; service UP verified (HTTP 302) this session | `ssh office1-netbox 'curl ... localhost:8000'` | office1-netbox | -| Juju | NOT part of VR1 substrate yet (arrives Stage 5). Last recorded VR0 value 3.6.25 -- historical, unverified here | n/a | n/a | +| Juju | **3.6.27 (rev 35621, `3/stable`) MEASURED 2026-07-27 on voffice1** -- the headend is the D-128 Plane-2 execution host and this is the client that will bootstrap the controller. Supersedes the 3.6.25 figure recorded 2026-07-24 (also at line 162, kept there as history): an in-channel patch refresh, which D-071 ADOPTED 2026-07-21 explicitly permits (patch-only jumps, in-channel-only refreshes), so this is policy-compliant drift and NOT an incident. The jumphost has NO juju client (measured ABSENT). | `ssh voffice1 'snap list'` (capture `docs/audit/stage5-live-measurement-20260727.txt`) | voffice1 | +| OpenStack client | **ABSENT ON BOTH HOSTS, MEASURED 2026-07-27** -- `command -v openstack` returns nothing on vcloud AND on voffice1, and voffice1's snap list carries only core24/juju/lxd/maas/postgresql/snapd. Ten Stage-5/6/7 scripts invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`), so every post-deploy step from Stage 5 Step 7 onward has no client to run. Recorded here because it is a measured Stage-5 precondition that no prior surface carried. | `command -v openstack`; `ssh voffice1 'command -v openstack; snap list'` | vcloud + voffice1 | The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so diff --git a/docs/audit/outer-plan-20260727-stage5-audit.txt b/docs/audit/outer-plan-20260727-stage5-audit.txt new file mode 100644 index 0000000..91a1cf9 --- /dev/null +++ b/docs/audit/outer-plan-20260727-stage5-audit.txt @@ -0,0 +1,30 @@ +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Refreshing state... [id=1ea36d3f-e7af-984c-8157-152724a0b85a] +module.vvr1_dc0.libvirt_cloudinit_disk.seed: Refreshing state... [id=ff281478c6083cc3] +module.vvr1_dc1.libvirt_cloudinit_disk.seed: Refreshing state... [id=41e9ec4b712038fc] +module.vr1_dc1_storage.libvirt_pool.dc: Refreshing state... [id=4a1df114-ee04-4c80-9233-cc0c140c8556] +module.office1_storage.libvirt_pool.dc: Refreshing state... [id=5f94194c-69c1-4b04-a85f-c18d87303a03] +module.vr1_dc0_storage.libvirt_pool.dc: Refreshing state... [id=7ce1101c-a89e-40ca-9263-5f572bee40a9] +module.mesh_vr1_dc0_office1.libvirt_network.link: Refreshing state... [id=8318548f-c3d6-4e06-bef4-fe3f11d68125] +module.mesh_vr1_dc1_office1.libvirt_network.link: Refreshing state... [id=38a20d2d-cd91-4604-a5f4-8e2a6609633c] +module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link: Refreshing state... [id=9cbc8589-9f40-48e6-872e-ef3abfe29a93] +module.vr1_dc0_uplink.libvirt_network.site_wan: Refreshing state... [id=f3500153-e4de-45f1-8854-9c92974a6094] +module.office1_network.libvirt_network.office1_local: Refreshing state... [id=8fdd2a97-417c-44d4-89e4-ae8d65594135] +module.voffice1.libvirt_cloudinit_disk.seed: Refreshing state... [id=a4694210c663c9ce] +module.vr1_dc1_uplink.libvirt_network.site_wan: Refreshing state... [id=4aad75c2-924f-410c-96bb-fa9217f8b4ea] +module.voffice1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso] +module.ubuntu_noble_base.libvirt_volume.base: Refreshing state... [id=/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2] +module.office1_opnsense.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/office1-opnsense-disk.qcow2] +module.vvr1_dc1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc1/vvr1-dc1-cloudinit.iso] +module.vvr1_dc0.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-cloudinit.iso] +module.voffice1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-disk.qcow2] +module.vvr1_dc1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc1/vvr1-dc1-disk.qcow2] +module.vvr1_dc0.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-disk.qcow2] +module.office1_opnsense.libvirt_domain.vm: Refreshing state... [name=office1-opnsense] +module.vvr1_dc1.libvirt_domain.vm: Refreshing state... [name=vvr1-dc1] +module.vvr1_dc0.libvirt_domain.vm: Refreshing state... [name=vvr1-dc0] +module.voffice1.libvirt_domain.vm: Refreshing state... [name=voffice1] + +No changes. Your infrastructure matches the configuration. + +OpenTofu has compared your real infrastructure against your configuration and +found no differences, so no changes are needed. diff --git a/docs/audit/preflight-20260727-stage5-audit.txt b/docs/audit/preflight-20260727-stage5-audit.txt new file mode 100644 index 0000000..95648b7 --- /dev/null +++ b/docs/audit/preflight-20260727-stage5-audit.txt @@ -0,0 +1,103 @@ +================ P1: repo lint ================ + [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 carve-out; NEW entries must be ASCII) + [FAIL] L10 pending change set modifies state-bearing surface(s) [docs/audit/: docs/audit/outer-plan-20260727-stage5-audit.txt, docs/audit/preflight-20260727-stage5-audit.txt] without touching docs/CURRENT-STATE.md -- GA-R1 rule 8 (C1): a status-changing commit updates CURRENT-STATE.md in the SAME commit + +FAIL: repo lint (1 fail, 1 warn) +================ P2: bundle invariants ================ + [ok] 11 charms bind public->provider-public; none on provider-vip + [ok] 11 clustered VIP(s) are provider-public/admin/internal triples, octet 50-60 + [ok] ovn-chassis bridge-interface-mappings: 2 well-formed MAC(s) (role-sep; VR0 set N/A) + [ok] 108 relations well-formed (explicit endpoints, all apps exist) + [ok] mysql-innodb-cluster num_units=3 (D-062) + [ok] keystone policyd-override wired in-bundle; zip content matches source (DOCFIX-071) + [ok] placement: role-separated (3 control/2 compute/4 storage); anti-affinity + role placement + counts OK + +PASS: Pattern A / D-052-D-053 bundle invariants (bundle.yaml) +================ P3: channel assert (charmhub) ================ + [WARN] barbican: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] barbican-vault: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] ceph-mon: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify squid/stable manually + [WARN] ceph-osd: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify squid/stable manually + [WARN] ceph-radosgw: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify squid/stable manually + [WARN] ceph-rbd-mirror: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify squid/stable manually + [WARN] cinder: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] cinder-backup: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] cinder-ceph: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] designate: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] designate-bind: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] glance: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] glance-simplestreams-sync: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] hacluster: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2.4/stable manually + [WARN] keystone: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] magnum: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] magnum-dashboard: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] memcached: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify latest/stable manually + [WARN] mysql-innodb-cluster: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 8.0/stable manually + [WARN] mysql-router: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 8.0/stable manually + [WARN] neutron-api: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] neutron-api-plugin-ovn: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] nova-cloud-controller: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] nova-compute: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] octavia: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] octavia-dashboard: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] octavia-diskimage-retrofit: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] openstack-dashboard: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] ovn-central: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 24.03/stable manually + [WARN] ovn-chassis: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 24.03/stable manually + [WARN] placement: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 2024.1/stable manually + [WARN] rabbitmq-server: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 3.9/stable manually + [WARN] vault: juju info failed ([Errno 2] No such file or directory: 'juju') -- verify 1.8/stable manually + +WARN: channel assert (33 pins, 0 fail, 33 warn) +================ P4: live pre-flight (MAAS/overlay/nodes) ================ + +=== Repo (informational) === +NOTE: REPO=/home/jessea123/openstack-caracal-dc-dc +NOTE: HEAD: 514f0aa Stage-5 grounding audit: land the scope charter before spawning any agent +WARN: working tree has 2 modified file(s) + +=== CHECK 0: octavia-pki overlay (no key material printed) === +FAIL: MISSING overlays/octavia-pki.yaml (gitignored secret; place it or regenerate via runbook Step 1.0-GEN before deploy) + +=== CHECK 1: bundle VIPs -- TRIPLE column .50-.60 (provider/admin/internal) === +PASS: vip: line count = 11 +PASS: aligned triple VIPs OK=11 bad=0 + +=== MAAS reachability gate (read-only) === +FAIL: MAAS unreachable: 'maas admin subnets read' failed + +Summary: 2 fatal, 1 warning +================ P5: credential matrix (D-137 tier 1 + tier 2 local) ================ +=== creds-matrix: tier 1 (STATIC) === +=== creds-matrix: tier 2 (EXISTENCE) === + [ok] S1 schema: 82 rows, all enums valid, site-keys region-qualified, no duplicate (id,site,host-role,filename) + [ok] S3 render: 3 source field(s) SKIPPED -- rendering them needs the declared path from creds-manifests/vm-secret-locations (ruling 3); the list now EXISTS but the source-field derivation is not wired + [ok] S3 render drift: rendered row fields (mode, source) match checked-in; header prose and non-jumphost rows are OUT OF SCOPE of this compare + [ok] S4 mint-ref: every script:/runbook: reference resolves to a real location + [ok] S4 provenance debt: 30 row(s) are mint-ref=operator-terminal -- NOT reproducible from the repo (research FINDING 1). Admitted by design; converting them is remediation, not a checker fix. + [ok] S7 notes: 34 note key(s) referenced, all resolve, none orphaned + [ok] E0 jumphost location '~/vault-init/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/issuing-ca/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/controller-ca/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/controller/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 8 remote location(s) SKIPPED -- rerun with --remote to include the headend shadow stores (SEC-022) and the region secrets dir (SEC-020) + [ok] E1 18 expected artifact(s) deferred as not-yet-minted per creds-manifests/stages-reached + [ok] E1 23 expected artifact(s) NOT JUDGED -- their role (headend/-, headend/vr1-dc0, headend/vr1-dc1, headend/vr1-office1, netbox/vr1-office1) has at least one location that could not be probed, so absence cannot be asserted over it + [ok] tier 3 (VALIDITY) NOT RUN -- pass --tier3 (with --tier2) to compare cross-copy sha256 provenance + [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=id_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=maas-virsh_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S5 ASYMMETRY: vr1-dc1 declares id=dcN-maas-power-key file=id_dcN_power on headend (custody=off-manifest-known) with no counterpart in vr1-dc0 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types (human via gui; service via api, cli-profile) -- ruling 5 requires one identity to serve one principal type + [FAIL] E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt' expected at jumphost/vr1-dc0, not found (mint-stage stage3, SEC-021) + [FAIL] E4 UNCHECKABLE: 2 row(s) have no declared location for their (role, site) and can never be verified -- add a location row or correct the matrix: capi-mgmt-kubeconfig 'config' (cloud/-); rbd-mirror-peer-token 'rbd-mirror-bootstrap-token' (unit/-) + +FAIL: creds-matrix tier 1 -- 82 row(s), 15 check group(s) clean, 7 finding(s) +================ P6: stage-2 reminders (NOT run here) ================ + - after 'juju add-model': bash scripts/juju-spaces-check.sh + - with sudo: bash scripts/osd-blank-check.sh + - phase-01 Step 1.2: juju deploy --dry-run (plan: 50 apps / 97 relations) + +PREFLIGHT: FAIL -- do NOT deploy +preflight rc=1 diff --git a/docs/audit/stage5-committee-raw-20260727.md b/docs/audit/stage5-committee-raw-20260727.md new file mode 100644 index 0000000..b2c9337 --- /dev/null +++ b/docs/audit/stage5-committee-raw-20260727.md @@ -0,0 +1,411 @@ +# Stage-5 grounding committee -- RAW LENS RETURNS (2026-07-27) + +Charter: `docs/audit/stage5-grounding-audit-scope-20260727.md`. +Branch: `dc-dc-stage5-grounding-audit`. + +This file is the VERBATIM record of what each read-only lens returned, kept +separate from the synthesis so the synthesis can never quietly reshape a +lens's claim. Findings are NOT yet verified here -- adversarial verification +happens in the synthesis, and a finding that is refuted stays in this file +with its refutation recorded, never deleted. + +Schema each lens was required to use: +`ID / CLAIM / EVIDENCE / BLOCKS-STAGE-5 / DISPOSITION`. +Every lens was also required to report any denied or prompted tool call as +UNMEASURED with the exact command, never folded into a negative finding. + +--- + +## LENS 3 -- DEPLOY-ARTIFACT CONFORMANCE (returned first) + +**ID: L3-1** +CLAIM: `preflight.sh` P2 validates only the bare dc0 base bundle; the artifact Stage 5 actually deploys (base + the three dc1 overlays, dc1 bands) is validated by no gate at all -- and both inputs return PASS, so P2's green is not evidence about the dc1 deploy. +EVIDENCE: `scripts/preflight.sh:33` -- `python3 scripts/provider-bundle-check.py bundle.yaml` (no `--overlay`, no `--dc`). Measured bare: `PASS: Pattern A / D-052-D-053 bundle invariants (bundle.yaml)` EXIT=0. Measured full dc1 input (`--overlay overlays/vr1-dc1-machines.yaml --overlay overlays/vr1-dc1-vips.yaml --overlay overlays/dc-ha-scaleup.yaml --dc vr1-dc1`): also PASS EXIT=0 (`109 relations well-formed`, `placement: role-separated (3 control/2 compute/4 storage)`). Nothing in `scripts/preflight.sh` ever passes an overlay or `--dc`. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L3-2** +CLAIM: `designate` is a SECOND app (beyond the known vault case) with full HA intent and no VIP anywhere in the tracked tree -- scaled to 3, `ha` bound, `designate-hacluster` related and set to `cluster_count: 3` -- and the checker passes it because its VIP loop `continue`s on a missing `vip`. +EVIDENCE: `bundle.yaml:830-847` (designate stanza: `ha: metal-internal`, no `vip:`); `bundle.yaml:940` `designate-hacluster: {... cluster_count: 1 ...}`; `bundle.yaml:1106` `- [designate:ha, designate-hacluster:ha]`; `overlays/dc-ha-scaleup.yaml:128-130` (`designate: num_units: 3`) and `:172` (`designate-hacluster: {options: {cluster_count: 3}}`). `grep -n 'vip' bundle.yaml` returns exactly 11 lines (200, 239, 295, 364, 400, 490, 576, 603, 657, 722, 780) -- none for designate, none for vault. `scripts/provider-bundle-check.py:133-135`: `vip = ...get("vip"); if not vip: continue`. D-136's recorded problem statement (`docs/design-decisions.md:4786-4789`) names only vault for this defect. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling + +**ID: L3-3** +CLAIM: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` Step 4 -- the Stage-5 deploy step -- names ZERO `--overlay` flags and instructs the opposite of the authored dc1 artifacts; following it verbatim deploys dc0's VIPs and dc0's machine tags into the dc1 model. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:203` -- "**DC1:** deploy the EXISTING `bundle.yaml` with NO VIP edits"; `:210` -- "**DC2:** BLOCKED until its VIP band exists ... `bundle.yaml`'s VIP lines need editing ... a genuine, reviewed bundle edit". Neither `overlays/vr1-dc1-vips.yaml`, `overlays/vr1-dc1-machines.yaml`, `overlays/dc-ha-scaleup.yaml` nor `overlays/dc-dc-ipv6-family-matrix.yaml` is mentioned anywhere in that runbook. dc1's band IS ratified and IS authored (`overlays/vr1-dc1-vips.yaml:16-46`, 10.12.64/68/72 .50-.60), so the "BLOCKED" line contradicts on-disk state; and the "edit bundle.yaml" instruction contradicts the recorded operator utterance at `docs/design-decisions.md:4930-4936` ("Base bundle and everything else is applied via overlay"). +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L3-4** +CLAIM: The 10th Juju-controller VM is UNAUTHORED, not merely un-applied -- no OpenTofu resource, no script, no runbook step exists for it anywhere; only prose describes it. +EVIDENCE: `opentofu/vr1-dc1-substrate/main.tf` locals declares exactly 9 node keys (`control-01..03`, `compute-01..02`, `storage-01..04`); `grep -in "controller" opentofu/vr1-dc1-substrate/*.tf` -> zero hits; module blocks are `inner_storage`, `vr1_dc1_planes`, `vr1_dc1_wan`, `vr1_dc1_opnsense`, `vr1_dc1_node` -- no controller module. `grep -rn "juju-controller-vr1-dc1" opentofu/ runbooks/ scripts/ bundle.yaml overlays/` -> zero hits; the string exists only in `docs/CURRENT-STATE.md:187`, `docs/design-decisions.md:2160-2161`, and audit files. `bundle.yaml:78` carries only the comment "The Juju CONTROLLER is a dedicated 10th VM (D-104 amendment) ... NOT a bundle machine". `docs/CURRENT-STATE.md:179-180`'s "10th node is ADDITIVE (`1 add/0/0`)" is therefore a projection with no resource behind it. +BLOCKS-STAGE-5: yes -- `dc-dc-phase4` Step 2 `juju bootstrap` has no target machine. | DISPOSITION: DOCFIX + +**ID: L3-5** +CLAIM: Per-role MAAS tags (Fork 3) are consumed by the machines block but authored nowhere in the repo; the only tag-creation code creates ONE tag, and the only runbook guidance still points at the retired DC0 literal `openstack`. Deploy fails hard at allocation (it does not silently mis-place). +EVIDENCE: Consumers -- `bundle.yaml:81-97` and `overlays/vr1-dc1-machines.yaml:23-39`: every machine constrains on TWO tags, `tags=openstack-vr1-dc1,`. Producers -- the only tag-create in the tree is `scripts/reenroll-hosts.sh:177-180` (`maas ... tags create name="$HOST_TAG"`, a single tag). Runbook -- `runbooks/dc-dc-phase3-maas-enlist-deploy.md:289-302` still says `lib-hosts.sh`'s `HOST_TAG="openstack"` is DC0's literal and shows `tag update-nodes ""`, singular. Nothing anywhere creates `control`/`compute`/`storage`. Juju passes `tags=` to MAAS as an allocation constraint; an unmatched tag yields no candidate machine. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L3-6** +CLAIM: `pre-flight-checks.sh` (preflight P4) asserts two carve facts that D-133 RULED cannot exist on ANY VR1 DC -- stronger than the known "VR0-frozen" framing: two checks contradict a RULED decision and can never pass on dc0 or dc1, and D-133's own affected-surfaces list omits the file. +EVIDENCE: `scripts/pre-flight-checks.sh:128` (`metal-internal VID=$mivid (want $METAL_INTERNAL_VID)`); `:143-144` (`$h metal-internal on '$ifname' (want $METAL_INTERNAL_IFACE)`). Values come from VR0 defaults `scripts/lib-net.sh:44-45` (`METAL_INTERNAL_VID="103"`, `METAL_INTERNAL_IFACE="br-internal"`) because `grep -n lib_net_select_dc scripts/pre-flight-checks.sh` returns zero hits. D-133 (`docs/design-decisions.md:4562-4566`): "no br-metal.103 / br-internal stack exists in VR1. VID 103 is a VR0-only fact." D-133's affected-surfaces list (`:4587-4592`) does not name `pre-flight-checks.sh`. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L3-7** +CLAIM: The dc1 arm of `lib-net.sh` UNSETS the VIP constants any DC-aware CHECK 1 would need, while leaving the octet bounds set -- so the data for the fix does not exist yet; and `VIP_OCTET_MAX` lives in `lib-net.sh`, not in `provider-bundle-check.py` (whose equivalents are `OCTET_LO`/`OCTET_HI`). +EVIDENCE: Measured -- `. scripts/lib-net.sh; lib_net_select_dc vr1-dc1` -> `VIP_PREFIX_PROVIDER=[UNSET] ADMIN=[UNSET] INTERNAL=[UNSET] MIN=[50] MAX=[60] COUNT=[UNSET]`. Source `scripts/lib-net.sh:157` unsets them; `:54-59` holds the VR0 values. `scripts/provider-bundle-check.py:41` uses separately-named `OCTET_LO, OCTET_HI = 50, 60`, so raising the band is a TWO-file change. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L3-8** +CLAIM: The base `bundle.yaml` -- documented as the vr1-dc0 source of truth -- carries **dc1's** compute MACs in `ovn-chassis` `bridge-interface-mappings`, and no dc0 counterpart overlay exists; the checker cannot catch it because the role-separated branch only asserts "well-formed MAC". +EVIDENCE: `bundle.yaml:458-464` -- comment "VR1 dc1: the 2 compute nodes' provider-public (enp2s0) MACs", values `br-ex:52:54:01:d1:04:02` / `br-ex:52:54:01:d1:05:02`, pinned to vr1-dc1 in `opentofu/vr1-dc1-substrate/main.tf:113-118`. dc0 uses `52:54:00:*` (`scripts/lib-hosts.sh:121-122`). `overlays/vr1-dc1-machines.yaml:7-8` states "The base bundle.yaml is the vr1-dc0 source of truth". Checker: `scripts/provider-bundle-check.py:167-172` only counts well-formed MACs under role_sep. +BLOCKS-STAGE-5: no for a dc1-first Stage 5 (values happen to be dc1's); yes for any dc0 deploy -- no local MAC match means `ovn-chassis` builds no br-ex mapping and provider egress is dead on dc0 compute, with no gate failing. | DISPOSITION: DOCFIX + +**ID: L3-9** +CLAIM: The D-101 two-overlay `vip:` collision is order-dependent in BOTH directions, measured: one order hard-fails, the other silently drops every v6 leg. +EVIDENCE: ipv6 overlay LAST -> EXIT=1, ten `[FAIL] vip not a triple: '{{KEYSTONE_VIP_PROVIDER_V4}} ...'` plus `[ok] 1 clustered VIP(s) are ... triples` (only octavia survives; the ipv6 overlay omits it -- `overlays/dc-dc-ipv6-family-matrix.yaml:140-152`). Reverse order (vips overlay last) -> v6 legs replaced wholesale, merged doc is a PASS with pure-v4 triples, no checker reports the loss. Merge semantics: `scripts/provider-bundle-check.py:52-58` `_deep_merge` -- `options.vip` is a scalar, REPLACED last-wins. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling + +**ID: L3-10** +CLAIM: `ceph-osd` is the only application carrying an app-level MAAS tag constraint, and it is the retired bare VR0 tag `openstack`. +EVIDENCE: `bundle.yaml:567` -- `constraints: arch=amd64 tags=openstack`. `grep -n 'tags=' bundle.yaml` returns ten lines: nine machines-block entries (`:81-97`, all `tags=openstack-vr1-dcN,`) plus `:567`. D-119 retires bare selectors (`scripts/lib-net.sh:161`). Live MAAS has no `openstack` tag. +BLOCKS-STAGE-5: no -- `ceph-osd` has explicit `to: ["5","6","7","8"]`, so the constraint is not used for selection. Latent reject if placement is removed. | DISPOSITION: DOCFIX + +**ID: L3-11** +CLAIM: `overlays/octavia-pki.yaml`'s absence is BY DESIGN -- a gitignored secret regenerated per deploy; preflight P4's failure on it is the intended pre-deploy prompt. +EVIDENCE: `.gitignore:40`; `git ls-files overlays/` returns only four tracked overlays; `scripts/pre-flight-checks.sh:64-71` fail message says "place it or regenerate via runbook Step 1.0-GEN before deploy"; `docs/design-decisions.md:5099` records it as an "Accepted residual". Residual doc defect: `README.md:35` still says "`overlays/` empty in git -- the only overlay (octavia-pki.yaml)", now four overlays stale. +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +**ID: L3-12** +CLAIM: A programmatic duplicate-key scan of `bundle.yaml` and all four overlays found ZERO duplicate keys at any nesting level -- the BUNDLEFIX-052 class is clean -- and the detector was self-tested to prove it can fail. +EVIDENCE: Custom `yaml.SafeLoader` subclass reporting repeated keys per node, run over `bundle.yaml` + `overlays/*.yaml`: `no-duplicate-keys` on all five. Self-test against a planted document: `SELF-TEST planted-dup findings: [(9, 'keystone'), (7, 'bindings')]` -- fires at both nesting levels. NOTE: no repo gate performs this scan; it exists only as this session's ad-hoc script, so the class can regress undetected (`provider-bundle-check.py:106` uses plain `yaml.safe_load`, which silently take-lasts). +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +**ID: L3-13** +CLAIM: Six applications added 2026-07-10 (D-106 designate / D-108 replication) have no row in the as-built version lock, so their channel pins have no lock authority behind them. +EVIDENCE: Set-difference of `bundle.yaml` applications against `runbooks/appendix-B-asbuilt-version-lock.md` rows: `in bundle not in appendix-B: ['ceph-rbd-mirror', 'cinder-backup', 'designate', 'designate-bind', 'designate-hacluster', 'designate-mysql-router']`; `in appendix-B not in bundle: []`. Every other application's channel matches appendix-B exactly. NOT verified against Charmhub -- see UNMEASURED. +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +**ID: L3-14** +CLAIM: Affirmative conformance -- the items checked that PASS, recorded so they are not re-litigated downstream. +EVIDENCE: D-121 Option C -- `bundle.yaml:80-97` gives 3 control / 2 compute / 4 storage; checker confirms `placement: role-separated ...; anti-affinity + role placement + counts OK` on the merged dc1 input; `nova-compute num_units: 2 to: ["3","4"]` (`:322-323`), `ceph-osd num_units: 4 to: ["5","6","7","8"]` (`:556-557`). D-108 replication bindings intact post-BUNDLEFIX-052 (`:545`, `:911-913`, relation `:1046`). D-100 provider-raw (`:456` `ovn-bridge-mappings: physnet1:br-ex`). D-133 flat carve -- no VLAN/VID construct anywhere in `bundle.yaml`. No application binds `public: provider-vip`. `bash tests/provider-bundle-check/run-tests.sh` -> `PASS=15 FAIL=0`, including T15 (dc1 VIPs vs dc0 bands FAILS), so the `--dc` arm is proven capable of failing. +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +**ID: L3-15** +CLAIM: Three artifact references in `bundle.yaml` comments point at files that do not exist, and one comment states the wrong plane CIDR. +EVIDENCE: `bundle.yaml:654` `--overlay overlays/vr0-dc0-testcloud.yaml` (no such file); `bundle.yaml:15`, `overlays/dc-ha-scaleup.yaml:11`, `overlays/dc-dc-ipv6-family-matrix.yaml:11` reference `overlays/${DC}-hostnames.yaml` (proposed only, `docs/dc-dc-deployment-workflow.md:257`, never authored); `bundle.yaml:35` says geneve overlay is on `10.12.12.0/22` but measured from `lib-net.sh` that is **metal-internal** and dc0's data-tenant is `10.12.16.0/22`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +### LENS 3 UNMEASURED +1. Juju overlay-merge semantics for the `machines:` block. `command -v juju` -> NOT on PATH on this jumphost. `provider-bundle-check.py:60-65` merges with a MIRROR of juju's documented behaviour, not juju itself, so the MODE-2 PASS is evidence about the mirror, not about juju. Not run: `juju deploy ./bundle.yaml --overlay ... --dry-run -m `. +2. Charm option existence for `prefer-ipv6`, `min-cluster-size`, and `hacluster`'s `cluster_count` semantics -- not fetched from Charmhub or charm source; the overlays carry their own VERIFY-AT-DEPLOY caveats. +3. Charmhub currency of the six unlocked channels in L3-13. No upstream fetch performed. +4. dc0's compute-node provider-public (`enp2s0`) MACs -- `lib-hosts.sh` records only dc0 BOOT MACs; no authored dc0 provider-NIC MAC found anywhere. What dc0's correct values ARE is unmeasured. +5. `preflight.sh` / `pre-flight-checks.sh` end-to-end execution -- not run by this lens; its P4 findings are read from source. +6. Live MAAS tag state -- used the measured list supplied in the brief; ran no `maas` command. + +--- + +## LENS 6 -- STAGES 6-7 PRE-MORTEM (returned second) + +**ID: L6-1** +CLAIM: `bundle.yaml` already carries live designate/designate-bind/designate-mysql-router/designate-hacluster blocks + 8 relations (DOCFIX-167), so Stage 5's `juju deploy` WILL deploy Designate -- which makes Stage 7 Step 5's gate ("the diff shows ONLY the new designate/... applications being added") unsatisfiable, and Stage 5 Step 4's own statement false. +EVIDENCE: `bundle.yaml:830,849,858,940` + `:1099-1106` (8 relations); `docs/dc-dc-deployment-workflow.md:531-542` (DOCFIX-167 CLOSED 2026-07-10). Contradicted by `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:215-217` ("bundle.yaml explicitly ships \"NO designate\" ... Designate reactivation is Stage 7's job (D-106), do not reactivate it mid-Stage-5") and `runbooks/dc-dc-phase6-designate-cos-magnum.md:359-370` GATE. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling -- (A) suppress designate for the Stage-5 deploy (subtractive overlay / rendered per-stage bundle), keeping Stage 7's incremental-add shape, or (B) accept Designate landing at Stage 5 and rewrite Stage 7 Steps 1/5 to "configure, not deploy". NOTE: D-106's own bootstrap order puts `os-public-hostname`+FQDN-SAN certs BEFORE Designate; option (B) inverts that order. + +**ID: L6-2** +CLAIM: `dc-dc-teardown-rollback.md` Step 2 now routes to a FALSE CLEAR -- it decides "nothing to clean up in MAAS" by reading a `vm_host`, but the VR1 DCs deliberately have NO MAAS pod (`power_type=virsh` per machine, D-103/D-123 amendments) while 18 nodes ARE enrolled; the check returns empty, the runbook says skip, and the containment VM gets destroyed out from under 18 live MAAS records. +EVIDENCE: `runbooks/dc-dc-teardown-rollback.md:160-176` ("If this returns nothing (expected for any VR1 site before Stage 4 has run for it): skip to Step 3, there is nothing to clean up at this layer yet") and `:101-110` ("this repo has not yet reached Stage 4 ... for any VR1 DC"). Both now false: `docs/CURRENT-STATE.md:47-48` (18 nodes READY); `docs/audit/stage4-discovery-20260723.txt:2-5` shows `vvr1-dc0 7chphy` / `vvr1-dc1 nmpcq4` are RACK CONTROLLERS, not vm-hosts. Related hazard already indexed but not referenced here: `runbooks/appendix-A-troubleshooting.md:781`. +BLOCKS-STAGE-5: yes -- Stage 5 is the first stage that can fail partway with a live cloud on top, and this runbook is the documented reach-for at that moment. | DISPOSITION: DOCFIX + +**ID: L6-3** +CLAIM: `dc-dc-teardown-rollback.md` Step 3's scoped-destroy command names five OpenTofu modules that exist in neither root, so the Path A rollback is not executable; its header also still declares the tree unvalidated for a reason no longer true. +EVIDENCE: `:192-206` targets `module.dc1_maas_vm_host`, `module.dc1_opnsense`, `module.dc1_node_01`, `module.dc1_node_02`, `module.dc1_storage`, `module.dc1_planes`, and admits at `:205-206` that a `-target` naming a nonexistent module errors. Actual outer modules: `vr1_dc0_storage`(35), `vr1_dc1_storage`(49), `office1_*`, `mesh_vr1_dc0_vr1_dc1`(125), `netem_...`(353), `vr1_dc0_uplink`(379), `vr1_dc1_uplink`(391), `vvr1_dc0`(410), `vvr1_dc1`(537). Inner: `vr1_dc0_planes`(34), `vr1_dc0_wan`(51), `vr1_dc0_opnsense`(60), `vr1_dc0_node`(136). Step 4's VERIFY (`:242-245`) greps vcloud `virsh` for DC domains that live INSIDE `vvr1-dc0`, and `grep -i dc1` now matches the SECOND DC. Header `:13-19` still says no `tofu` binary was ever available -- contradicted by `docs/CURRENT-STATE.md:857`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-4** +CLAIM: If `dc-ha-scaleup.yaml` is applied at Stage 5, `ceph-radosgw` and `designate` deploy at 3 units per DC -- but Stage 6's radosgw multisite procedure and the DOCFIX-165 script are single-unit-shaped, so realm/period membership lands on one of three gateways and Step 4's gate can false-green. +EVIDENCE: `overlays/dc-ha-scaleup.yaml:132-133`, `:128-129`, `:171-172`. Against `runbooks/dc-dc-phase5-dr-failover-drill.md:237-239` ("Restart the DC1 radosgw unit (`juju run ceph-radosgw/0 restart`)") and measured `bash scripts/dc-dc-radosgw-multisite.sh --help` -> `master-init --dc vr1-dc0|vr1-dc1 --model M --unit U` (single `--unit`, no all-units mode). Step 4's `--endpoints=http://` is unit-vs-VIP unspecified. +BLOCKS-STAGE-5: yes -- whether to apply the HA overlay is a Stage-5 deploy-time choice and scaling 1->3 afterwards is a live-cloud change. | DISPOSITION: ruling (apply HA overlay at Stage 5 or not) + DOCFIX (multi-unit radosgw path) + +**ID: L6-5** +CLAIM: Stage 7's two STOP conditions keyed on "dc2's literals are not yet assigned" are reversed by later rulings -- an operator halts Stage 7 for `vr1-dc1` on a gap that is closed, and the `dc2` fail-loud they cite is now a RETIRED-SELECTOR failure. +EVIDENCE: `runbooks/dc-dc-phase6-designate-cos-magnum.md:285-288` and `:536-539`. Reversed by `scripts/lib-net.sh:128-158` (real `vr1-dc1` arm) and `:160-163` (bare `$dc` RETIRED, D-119); `docs/CURRENT-STATE.md:781-783` (dc1 addressing RATIFIED 2026-07-21); `overlays/vr1-dc1-vips.yaml`. Same defect in `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:82-93` ("STOP here for DC2"). +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-6** +CLAIM: Stage 7 Step 10 presents COS-scrapes-the-edge as an unruled fork with "If out-of-scope, do nothing here" -- but D-129(i) was RULED in-scope 2026-07-21 and explicitly names this step's pin UNGATED. The DOCFIX-204 pattern exactly. +EVIDENCE: `runbooks/dc-dc-phase6-designate-cos-magnum.md:482-487`. Reversed by `docs/design-decisions.md:4228-4235` (utterance "In-scope, per-DC (Recommended)"; "the Step-10 pin ... is therefore UNGATED and lands at its pinned deployment step") and Status line `:4203`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-7** +CLAIM: Stage 6 Step 2's fallback wiring commands contradict the real charm metadata DOCFIX-167 recorded in `bundle.yaml`, and Step 2's governing DoD bullet is vacuous -- three wrong endpoint/placement shapes would be typed against a live cloud if that path is taken. +EVIDENCE: `runbooks/dc-dc-phase5-dr-failover-drill.md:153-158` deploys `cinder-backup` with `--to lxd:` and relates `cinder-backup:amqp rabbitmq-server:amqp`; `:161-162` relates `ceph-rbd-mirror:ceph-rbd-mirror ceph-mon:rbd-mirror`. Against `bundle.yaml:877-885` (cinder-backup is a SUBORDINATE requiring `ceph`/`juju-info`, no amqp; ceph-rbd-mirror requires `ceph-local`/`ceph-remote`) and actual relations `:1032-1033`, `:1046`. A subordinate takes no `--to`. DoD `:631` and delivery checkbox `:642-644` can never be ticked because `bundle.yaml:545` already reads `rbd-mirror: replication`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-8** +CLAIM: The Stage-6 runbook is written entirely in retired `DC1`/`DC2` labels and issues promote/demote commands against `--os-cloud dc1` / `--os-cloud dc2`, where `dc1` means the FIRST DC -- but under D-119 `vr1-dc1` is the SECOND DC. An inverted-target hazard on commands the runbook says must never be reordered. +EVIDENCE: `runbooks/dc-dc-phase5-dr-failover-drill.md:116-118`, `:371`, `:505`, `:585`. Against `scripts/lib-net.sh:160-163` (bare selectors RETIRED, "AMBIGUOUS ACROSS REGIONS") and `docs/dc-dc-deployment-workflow.md:143` (C3 done 2026-07-14, "**every runbook call site**"). Measured: `bash scripts/dc-dc-dr-drill.sh --help` -> `failover --dc vr1-dc0|vr1-dc1` -- the scripts were migrated, the runbook was not, so it disagrees with the tool it tells you to prefer. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-9** +CLAIM: Stage 6's entry condition requires Stage 5 to have delivered "Ceph-over-v6 and geneve-over-v6 verified", but Stage 5's Step 12 is contingent on an unapplied overlay and an unresolved Octavia IPv6 fork, and its text sanctions recording "blocked on Step 6" -- so Stage 5 can close without producing Stage 6's precondition. +EVIDENCE: `runbooks/dc-dc-phase5-dr-failover-drill.md:10-15`. Against `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:354-358` and `:290-297` ("Octavia's `lb-mgmt-net` IPv6 support is a real, open risk, not resolved ... Decide explicitly before this stage's ... gate is declared closed"). Register item 13: `docs/dc-dc-deployment-workflow.md:639-647`. +BLOCKS-STAGE-5: yes -- the Octavia lb-mgmt-net v4-vs-v6 fork is a Stage-5 deploy-shape choice; deciding after Octavia is deployed is a live-cloud change, and GA-R6 E3 forbids a conditional close. | DISPOSITION: ruling + +**ID: L6-10** +CLAIM: Both Stage 5 and Stage 7 issue `juju deploy --overlay "overlays/${DC}-hostnames.yaml"` against a file that does not exist and whose proposed name does not match the interpolation. +EVIDENCE: `ls overlays/` -> four files, no `*-hostnames.yaml`. Referenced as a live `--overlay` argument at `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:277-281`, `runbooks/dc-dc-phase6-designate-cos-magnum.md:361-364` and `:372-377`, `overlays/dc-ha-scaleup.yaml:9-13`. Proposed filenames conflict: `dc-dc-phase6:235-240` proposes `overlays/dc1-hostnames.yaml` / `dc2-hostnames.yaml` versus `${DC}` = `vr1-dc0`/`vr1-dc1` per D-119. +BLOCKS-STAGE-5: yes -- Stage 5 Step 6's apply command references it. | DISPOSITION: DOCFIX + +**ID: L6-11** +CLAIM: Gap-register item 17 is stale-OPEN in a way that produces a FALSE STOP: it states neither DC has an ISP-uplink network, but both uplinks are built and both D-125 egress gates PASSED. The skill's VR1 deploy loop says to read this register FIRST. +EVIDENCE: `docs/dc-dc-deployment-workflow.md:728-731`. Against `opentofu/main.tf:379`, `:391`, `opentofu/vr1-dc0-substrate/main.tf:51`, `docs/CURRENT-STATE.md:845` (D-125 PASS 2026-07-20) and the G12 row (D-125 PASS 2026-07-22 dc1). Same class: gap 15 (`:678-685`) still says "DC2 is still blocked at the network-selector layer" and that the per-DC artifact mirror "has no identified owning stage/module" though D-135 owns it; gap 11's netem sub-item (`:595-600`) predates the 2026-07-21 standing-placeholder ruling and the live apply on virbr5. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-12** +CLAIM: The gap register has TWO items numbered 19, and the second declares itself OPEN and a hard precondition of Stage 3 OpenTofu work -- a condition closed by D-119 on 2026-07-14. +EVIDENCE: `docs/dc-dc-deployment-workflow.md:787` (item 19 CLOSED via DOCFIX-176) and `:806` (a second item 19), whose `:835-837` reads "**Status: OPEN. It is Stage 2 close-out item C3 and the FIRST step of any Stage 3 OpenTofu work.**". Closed by `:143` and `scripts/lib-net.sh:95-166`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L6-13** +CLAIM: Register item 20 (durable host->site reach) is NOT satisfied on the record for EITHER DC -- the tool carries only a commented template row and no capture states the measured determination the DC-standup DoD requires. "No leg needed" is currently INFERRED, not measured. +EVIDENCE: `docs/dc-dc-deployment-workflow.md:867` ("Office1 CLOSED; DC rows DEFERRED-until-measured"); `scripts/site-baseleg.sh:45-48` -- `LEGS=( [office1]=... # [vr1-dc0]="|||" # MEASURE first )`, no dc row live or commented for dc1. Obligation: `runbooks/dc-dc-phase2-tofu-dc-substrate.md:748-752` DC-standup DoD item 1. Repo-wide grep of `docs/audit/*` and `docs/archive/stage-records/*` returns only the Office1 row. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX (record the measured verdict per DC, or add leg rows) -- the measurement is a session task, not a ruling. + +**ID: L6-14** +CLAIM: Stage 6 Step 9 and its DoD bullet stake the D-104 control-plane restore drill on `juju restore-backup`, a command with zero corroboration anywhere in this repo and no verified existence on the Juju 3.6 client the drill will use. +EVIDENCE: `runbooks/dc-dc-phase5-dr-failover-drill.md:443-456` and DoD `:624-625`. Repo-wide `grep -rn "restore-backup"` returns ONLY those two lines; `scripts/cloud-snapshot.sh:141` implements only `juju create-backup`. Juju is not installed here (`docs/CURRENT-STATE.md:872`), so the flag could not be verified. +BLOCKS-STAGE-5: no -- but verification is cheap the moment a Juju client exists at Stage 5. | DISPOSITION: log-only (queue the Stage-5 first-Juju-client verification) + +**ID: L6-15** +CLAIM: `runbooks/README.md`'s VR1 track table -- the surface the skill routes an operator to for stage selection -- carries two claims contradicted by the as-built. +EVIDENCE: `runbooks/README.md:56` ("with VR0's own testcloud host becoming DC1") against `docs/dc-dc-deployment-workflow.md:52-53` and `opentofu/main.tf:410,537`. `runbooks/README.md:67` lists Stage 2 as building GitBucket against `docs/dc-dc-deployment-workflow.md:85` ("**GitBucket is NO LONGER A GATE (D-116)**"). The VR0-vs-VR1 track separation itself is stated correctly at `:53-61`; no step routes an operator into the wrong track. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +### LENS 6 CHECKED, NO FINDING +- (d) Two-DCs-differ-by-ruling: no step in the phase-5/6 runbooks or the teardown runbook assumes an apt mirror at either DC (grep for mirror/apt/proxy/cache returns only `rbd mirror` / `ceph-rbd-mirror` hits). The D-135 asymmetry is correctly carried in the Stage-5 prerequisite block (`dc-dc-phase4:29-31,43-45`). The DR "mirror" vocabulary collision (ceph rbd-mirror vs apt mirror) is a naming hazard only. +- (g) Cross-DC sequencing: coherent as authored. `bundle.yaml:904-912` wires only this DC's `ceph-local` leg with `ceph-remote` bound but unrelated (deliberate, `docs/dc-dc-deployment-workflow.md:656-662`); the peer relationship is established by CLI bootstrap at Stage 6 Step 5 after both DCs exist. Netem is already live on virbr5, so Stage 6 Step 3's check is satisfiable where placed. NOTE: `ceph-rbd-mirror` will sit peerless from dc0's Stage-5 deploy until Stage 6 -- by design, but no runbook states the expected interim unit status, so an operator may read it as a Stage-5 failure. + +### LENS 6 UNMEASURED +- `juju restore-backup --help` -- NOT RUN, no Juju client on this host. Reported as L6-14 RISK, not as "the command does not exist." +- `juju bind ceph-mon rbd-mirror=replication` (`dc-dc-phase5:165`) -- subcommand and `=` syntax NOT verified; excluded from L6-7 rather than asserted wrong. +- `juju deploy ... --dry-run` merge behaviour of `overlays/vr1-dc1-machines.yaml` -- NOT RUN (no Juju, no model). The overlay itself flags this at `:20-23`. +- Reboot-durability of voffice1's dc0/dc1 transit addressing -- NOT probed; L6-13 is scoped to the missing RECORD, not to a claim that the path is fragile. +- `dc-dc-radosgw-multisite.sh`, `dc-dc-rbd-mirror.sh`, `dc-dc-dr-drill.sh` run with `--help` ONLY; runtime `lib_net_select_dc` behaviour under a real DC argument not exercised. + +--- + +## LENS 1 -- RECORD INTEGRITY / STATUS RECONCILIATION (returned third) + +LENS 1's own framing: every finding below is a **ruling-vs-surface contradiction** -- a RULED decision that a live Stage-5 surface reverses in text -- not a runtime failure. + +**ID: L1-1** +CLAIM: `scripts/lib-net.sh`'s `vr1-dc1` arm asserts the OpenStack-layer values are "NOT yet ruled/measured" and `unset`s them, contradicted by D-134-AMENDED (RULED 2026-07-23) plus the built `overlays/vr1-dc1-vips.yaml`, and hard-fails Stage 5 at its own Step 1 under `set -u`. +EVIDENCE: `scripts/lib-net.sh:128` (`vr1-dc1)` arm), `:152-158` -- comment "OpenStack-layer values (VIP bands, FIP pool, keystone VIP) and the VR0 metal fabric facts ... are NOT yet ruled/measured for vr1-dc1 ... UNSET so any use fails loud under `set -u`", then `unset METAL_INTERNAL_VID METAL_INTERNAL_IFACE` / `unset VIP_PREFIX_PROVIDER VIP_PREFIX_ADMIN VIP_PREFIX_INTERNAL VIP_COUNT_EXPECT` / `unset FIP_POOL_START FIP_POOL_END KEYSTONE_VIP_DEFAULT`. Against `docs/design-decisions.md:4637` (D-134 AMENDMENT, Status RULED 2026-07-23) and `overlays/vr1-dc1-vips.yaml`. Consumers Stage 5 Steps 7-9 run: `scripts/phase-03-admin-openrc.sh`, `phase-04-network-create.sh`, `phase-04-network-verify.sh`, `pre-flight-checks.sh`. Runbook mandates the selector for exactly these blocks: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:53-60`. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling +**GUARD -- DO NOT MECHANICALLY "POPULATE THE UNSET BLOCK".** The arm unsets TWO groups for TWO different reasons. `METAL_INTERNAL_VID` / `METAL_INTERNAL_IFACE` are **correctly** unset: D-133 (ADOPTED 2026-07-23, `docs/design-decisions.md:4549`) abolished the VLAN-103 / `br-internal` stack for VR1. Only the VIP/FIP/keystone group is superseded. The ruling required is *where the authority lives*: populate lib-net's dc1 arm from D-134-amended, or re-point the three `phase-0*` scripts at `overlays/vr1-dc1-vips.yaml`. NOT the same item as BLOCKER-1. + +**ID: L1-2** +CLAIM: Phase-4 Step 6's `juju deploy` names two overlay paths that do not exist and omits all three overlays that constitute the ruled dc1 deploy input; run literally it fails, and with the missing paths dropped it deploys dc0's 11 inline VIPs and no machine retag into dc1. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:276-282` names `--overlay overlays/octavia-pki.yaml` and `--overlay "overlays/${DC}-hostnames.yaml"`. `ls overlays/` returns exactly `dc-dc-ipv6-family-matrix.yaml dc-ha-scaleup.yaml vr1-dc1-machines.yaml vr1-dc1-vips.yaml`. The ruled deploy input is 3 overlays + `--dc vr1-dc1` (`docs/CURRENT-STATE.md:207-209`). `bundle.yaml` still carries 11 inline dc0 VIPs (`:200,239,295,364,400,490,576,603,657,722,780`). +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L1-3** +CLAIM: Phase-4 Step 4 and its delivery checklist instruct hand-editing `bundle.yaml`'s VIP lines for the second DC, which the 2026-07-25 ruling 3 reversed. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:210-214` ("`bundle.yaml`'s VIP lines need editing to DC2's real `.50-.60`-equivalent band ... a genuine, reviewed bundle edit") and `:374-375`. Against `docs/CURRENT-STATE.md:225-229` (ruling 3: bundle.yaml becomes VIP-free; every per-DC value arrives via overlay). The runbook's own `:212` maps DC2 = `vr1-dc1`, the exact DC Stage 5 deploys. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L1-4** +CLAIM: Phase-4 Step 2's `juju bootstrap` block carries no controller-tag constraint and still describes the target as a Stage-4 "MAAS-deployed machine" -- both reversed by ruled decisions; run as written it bootstraps onto one of the nine role nodes, the outcome the D-104 amendment explicitly rejected. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:136-137` and `:174` (`juju bootstrap ` -- no `--constraints`). Against `docs/design-decisions.md:2130-2133` (D-104 AMENDMENT ADOPTED 2026-07-24, "Dedicated 10th VM (Recommended)") and `:2158-2162` ("`juju bootstrap --constraints tags=juju-controller-vr1-dc1` targets it deterministically and no OpenStack `tags=` role constraint can cross-grab it"). Against DOCFIX-200: `docs/CURRENT-STATE.md:48`, and the same runbook's already-corrected Prerequisites at `:25-28`, which Step 2 was not swept with. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX +(Larger than the recorded fragment: Step 2 also still asserts a Deployed-node target, and the 10th VM is unbuilt, so the constraint has no machine to select today.) + +**ID: L1-5** +CLAIM: Phase-4 Steps 1/2 place the juju session on the jumphost, contradicting D-128's ruled two-plane origin -- juju 3.6.25 lives on `voffice1` and was REMOVED from vcloud, so Step 2's first mutation has no binary to run. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:119` ("**CHECK -- jumphost / wherever this session runs juju against this DC**"); `grep -n 'voffice1\|headend'` on that runbook returns only `:148` and `:156`, both about the MAAS key, never about where juju runs. Against `docs/CURRENT-STATE.md:161-162` ("Juju 3.6.25 on voffice1 (headend, D-128; removed from vcloud)") and D-128 Status `docs/design-decisions.md:4160`. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L1-6** +CLAIM: The phase-4 runbook is written wholly in the pre-D-117 `DC1`/`DC2` namespace, in which the DC Stage 5 deploys (`vr1-dc1`) reads as "DC2 -- NOT YET ASSIGNED / BLOCKED / STOP" while `vr1-dc0` reads as "DC1 -- needs NO VIP/CIDR changes"; D-117/D-119 retired that namespace and `lib_net_select_dc` mechanically rejects its tokens. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:59`, `:121`, `:72-79`, `:81-93` ("**DC2's bundle deploy is BLOCKED on this** ... STOP here for DC2 and only proceed with DC1"), `:127-131`, `:361-362`. Against D-117 Status (`docs/design-decisions.md:2996-2997`), D-119 Status (`:3221`), `scripts/lib-net.sh:159-161` (bare `dc0|dc1|dc2` returns 1, "RETIRED (D-119)"). Against "NOT YET ASSIGNED": `docs/dc-dc-netbox-buildout-scope.md:190` (DC2_V4_SUPERNET **ASSIGNED (D-115)** `10.12.64.0/19`); `scripts/lib-net.sh:128-138`. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX +(Two readings, both wrong, both destructive: read literally, Stage 5 STOPs on the DC it is meant to build; read by position, `vr1-dc1` is treated as "DC1" and gets dc0's `10.12.4/8/12.x` VIPs.) + +**ID: L1-7** +CLAIM: Gate row G3 reads OPEN with a standing FREEZE, contradicted by its own cited evidence file and by G11 -- and the FREEZE clause, if believed, blocks the DOCFIX batch this audit is queuing. +EVIDENCE: `docs/CURRENT-STATE.md:838` ("Batch 6 OPEN: ... PENDING only item 6 ...; FREEZE holds for un-gated surfaces"). Its own cited capture: `docs/audit/phase6-exit-runs-20260719.md:84` ("SIGNED 2026-07-19 ... PASS") and `:91-92` ("ALL SIX EXIT RUNS PASS ... **G3 + G11 CLOSED**"). Corroborated by `docs/CURRENT-STATE.md:846` and `:1249-1253`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX +(No choice required -- two surfaces already declare G3 CLOSED. The consequential half is the FREEZE clause, not the state token.) + +**ID: L1-8** +CLAIM: G17's closing condition omits the node time-source check that two other surfaces assign to G17, so G17 could close as written while the DoD bullet-6 replacement is never verified -- and the observation window is one-time. +EVIDENCE: `docs/CURRENT-STATE.md:852` (G17 [V] column) lists only the dc0 `curl -sI` and the dc1 proxy fetch -- no time check. Against `docs/dc-dc-deployment-workflow.md:206` ("Node-side reachability **and the node time source** are gate **G17**") and `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:44-45` ("both: `chronyc sources` shows the MAAS-served time source, NOT the DC edge (D-129(iv))"). The authority also self-contradicts on bullet 6: `docs/CURRENT-STATE.md:52` says "bullet 6 STRUCK" while `:342-347` says it "needs a DOCFIX re-expressing it as MAAS-hierarchy time verification". Runbook `:48-50`: "If first boot happens without these captures being taken, the observation window has passed". +BLOCKS-STAGE-5: no | DISPOSITION: ruling +(The choice: is MAAS-hierarchy time verification STRUCK, folded into G17's [V] text, or its own row? GA-R6 E3 forbids resolving it by conditional close.) + +**ID: L1-9** +CLAIM: Stage 5's stated entry gate -- "`preflight.sh` PASS" -- cannot be met as written, because preflight already exits 1 for reasons Stage 5 does not clear. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:186-192` ("Expect `PREFLIGHT: PASS`"); `docs/dc-dc-deployment-workflow.md:221` (Stage 5 Gate cell); runbook `:354-357`. Against `docs/CURRENT-STATE.md:999-1003`. `overlays/octavia-pki.yaml` confirmed absent this session. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling +(Requires an operator choice -- on what recorded basis Stage 5 proceeds against a red preflight, or which named sub-checks must go green first.) + +**ID: L1-10** +CLAIM: `docs/dc-dc-deployment-workflow.md` carries a full stage/build STATUS table plus a Stage-5 authoring-status cell (GA-R1 violation), whose rows assert dc1's addressing unassigned and its substrate unbuilt -- compounding L1-6, because the skill routes VR1 work to this doc BEFORE the runbooks. +EVIDENCE: `docs/dc-dc-deployment-workflow.md:871` ("## Where things stand today (2026-07-09)") with rows "Stages 3-7 NOT executed", "`tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED", "`opentofu/` (DC2 planes) | not done", "**DC1/DC2 still have NO uplink network**"; and `:224` ("**DC2 is hard-blocked pending its NetBox literals**"). Against `docs/CURRENT-STATE.md:39-55` and `scripts/lib-net.sh:128-138`. Routing: `SKILL.md:87`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX +(Mechanical fix = demote to pointers per GA-R1, not re-date. Deliberately scoped: NOT reporting the rest of that table, the register's OPNsense staleness, `opentofu/README.md:187`, or `docs/dc-dc-netbox-buildout-scope.md:155` -- cleanup under the GA-F06 bound. Checked and explicitly NOT a finding: the workflow doc's Stage-6 claim that `ceph-mon`'s `rbd-mirror` binding sits on `storage` is stale -- `bundle.yaml:544` reads `rbd-mirror: replication`, corrected by DOCFIX-167.) + +**ID: L1-11** +CLAIM: `opentofu/variables.tf` asserts dc1's planes are deferred pending a NetBox assignment, two variables above the ratified dc1 plane values in the same file. +EVIDENCE: `opentofu/variables.tf:34` against `:61-67` ("D-124 AMENDMENT 2026-07-21, addressing RATIFIED ... apex-VERIFIED, not invented"). +BLOCKS-STAGE-5: no (Stage 5 runs no tofu) | DISPOSITION: log-only + +**ID: L1-12** +CLAIM: Next-free numbers derive independently to D=138, DOCFIX=205, BUNDLEFIX=053, matching `ledger-scan.sh` exactly -- the section-9 self-inflation failure mode is NOT currently biting. +EVIDENCE: D header-authoritative: `grep -ohE '^## D-[0-9]+' docs/design-decisions.md` -> highest 137 (125 headers). Unfiltered token scans returned `DOCFIX-...201 202 203 204 777 888` and `BUNDLEFIX-...050 051 052 100 777`; the 777/888/100 values are deliberate harness fixtures occurring only in `tests/ledger-scan/run-tests.sh:84,91,92`. True highest DOCFIX 204 / BUNDLEFIX 052. +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +**ID: L1-13** +CLAIM: All four PROPOSED/OPEN decisions are correctly NOT gating Stage 5 -- but D-136's non-gating verdict applies to its *mechanism* only. +EVIDENCE: D-068 remainder is Q2 at Roosevelt Vault design time; `bundle.yaml:154` pins `channel: 1.8/stable` per the ruled Q1. D-131 sub-1/2 RULED and built; sub-4 pinned to next-deployment design time. D-132 Status PROPOSED/OPEN, operator-pinned to the NEXT DEPLOYMENT (`:4439`). D-136 Status PROPOSED, `docs/CURRENT-STATE.md:246` "Does NOT gate the dc1 deploy". HOWEVER D-136's own problem statement (`:4778-4788`) enumerates defects that ARE Stage-5 gating and tracked elsewhere: four disagreeing app-to-VIP lists, two overlays owning `vip:` for the same 10 apps under per-key REPLACE last-wins so the deploy input depends on overlay ORDER, and vault carrying no `vip:` while `dc-ha-scaleup.yaml` declares `cluster_count: 3`. +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +### LENS 1 UNMEASURED +1. `timeout 240 bash scripts/preflight.sh` -- **DENIED by the Claude Code auto-mode classifier**. Preflight's current live exit code and red set are UNMEASURED by this lens; L1-9 rests on the recorded measurement at `docs/CURRENT-STATE.md:999-1003`. **[RESOLVED BY THE AUDIT SESSION -- see `docs/audit/stage5-live-measurement-20260727.txt` section 3: preflight measured exit 1 with the red set EXACTLY the known one, nothing new. Root cause of the denial: the session allow-list has no `Bash(timeout *)` rule, so the command fell through to the classifier -- the same failed-to-MATCH class recorded on 2026-07-26, not a classifier override.]** +2. Gate rows G1-G13 and G16 (all CLOSED) not walked for evidence-path existence -- task scoped (c) to non-closed rows plus G17. Rows walked (G3, G14, G15, G17) all resolve, with one exception: G14's cell describes `overlays/octavia-pki.yaml` as a live in-clone exposure, but the file is absent from this working tree. Not raised as a finding -- it is a SEC-row description, not a Stage-5 decision input. +3. Live cloud state not probed -- out of lens scope; every claim above is repo-surface-vs-repo-surface. +4. `overlays/vr1-dc1-vips.yaml` / `vr1-dc1-machines.yaml` confirmed to EXIST but contents not parsed against D-134's bands -- lens 3's assertion to make. + +--- + +## LENS 2 -- AS-BUILT vs LIVE MEASUREMENT (returned fourth) + +**ID: L2-1** +CLAIM: The bundle's Ceph OSD device does not exist on any VR1 node in either DC -- `bundle.yaml` sets `osd-devices: /dev/vdb`, but every one of the 18 node VMs has exactly ONE block device (`vda`), and `modules/node-vm` is hard-coded to a single disk, so no second disk can appear without a module change. +EVIDENCE: `ssh voffice1 "ssh 172.31.0.2 'virsh -c qemu:///system domblklist vr1-dc0-storage-01'"` -> `vda vr1-dc0-storage-01-disk.qcow2` only. Identical single-`vda` for `vr1-dc0-storage-0{1..4}`, `control-01`, `compute-01`, and via `172.31.0.6` for `vr1-dc1-storage-01`, `storage-04`, `control-01`. MAAS agrees for all 18: `physicalblockdevice_set` length = 1, name = `vda` (590GB) on 18/18 rows. Contradicts `bundle.yaml:560` -- `osd-devices: /dev/vdb # libvirt-attached, MAAS-untracked, wiped pre-deploy` (a VR0 as-built comment; VR0 hosts had an attached vdb, VR1 nodes never did). Root cause in-config: `opentofu/modules/node-vm/main.tf:104-125` declares exactly one `devices.disks[]` with `target.dev = "vda"` from the single `libvirt_volume "disk"` (`main.tf:26`). `grep -rn "osd\|vdb\|extra_disk" opentofu/vr1-dc0-substrate/ opentofu/vr1-dc1-substrate/` returns only `disk_size_bytes` -- no OSD disk variable exists in either substrate root. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling (add an OSD volume to `modules/node-vm` and re-apply both substrates, vs re-point `osd-devices` at a directory/partition path, vs re-shape the storage role -- each has a different Roosevelt delta) + +**ID: L2-2** +CLAIM: D-134's `.4-.49` RESERVED band and `.50-.99` VIP band are reserved in PROSE ONLY -- MAAS holds ZERO reserved ipranges cloud-wide and reports the entire VIP band as allocatable on all 12 VR1 DC plane subnets. The tooling that creates these reservations refuses to run for any non-VR0 DC, so no VR1 path to create them exists. +EVIDENCE: `ssh voffice1 "maas admin ipranges read"` -> exactly three ranges, ALL `type=dynamic`: `10.10.1.100-.200`, `10.12.68.201-.254`, `10.12.8.201-.254`. Zero `type=reserved`. `maas admin subnet unreserved-ip-ranges ` for all 12 DC plane subnets shows the `.50-.99` band inside MAAS's allocatable pool on 12 of 12 (e.g. id=6 `10.12.8.0/22` UNRESERVED `10.12.8.5 - 10.12.8.99`; id=10 `10.12.64.0/22` UNRESERVED `10.12.64.3 - 10.12.64.99`). Only node statics and the `.201-.254` dynamic ranges are protected. Contradicts `docs/design-decisions.md:4654-4655` (D-134 AMENDMENT table: `.4-.49 RESERVED`, `.50-.99 VIP band`). Corroboration this is a VR1 gap not a design choice: `scripts/phase-00-maas-standup.sh:255` and its VR0 plane table (`:118-120`) reserve the API-VIP bands + D-003 FIP pool for VR0; none exist for either VR1 DC, and `:136` refuses outright (`FAIL: PLANES table is DC0-hardcoded ... refusing to plan another DC's scheme`). Downstream hard gate: `scripts/phase-04-network-verify.sh:100` fails if the FIP pool is not a reserved iprange; for vr1-dc0 that pool (`10.12.5.0-10.12.7.254`, `lib-net.sh:65-66`) is measured absent. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling (boundaries already ruled by D-134/D-058, but creating them is a live MAAS mutation on 12 subnets and needs gating; whether to carve the FIP pool now or at phase-04 is a sequencing choice) + +**ID: L2-3** +CLAIM: There is NO IPv6 anywhere in the VR1 DC substrate -- zero IPv6 subnets on any of the 12 DC plane fabrics and zero IPv6 addresses on all 18 nodes -- while D-101's 2026-07-25 RULING NOTE CLOSED the v4-only option for BOTH DC0 and DC1 in this deployment. +EVIDENCE: `maas admin subnets read | jq '.[]|select(.cidr|contains(":"))'` -> exactly ONE v6 subnet cloud-wide, `2602:f3e2:f01:100::/64` on `fabric-0` (the Office1 base net), space `undefined`. None of the 10 named plane fabrics nor the two metal-admin fabrics carries an IPv6 subnet. Zero IPv6 links across all 18 nodes; every node reports `default_gateways.ipv6 = NONE` (IPv4 gw `10.12.4.1` / `10.12.64.1`). Contradicts `docs/design-decisions.md:2034-2051` -- RULING NOTE 2026-07-25, operator verbatim "Dual stack to be used where IPv4 is required, IPv6 where IPv6 only makes sense" and "Dual stack deployment for DC0 and DC1. This is the deployment when the dual-stack is added"; Effect: "the v4-only phasing option ... is CLOSED, for BOTH DC0 and DC1 in this deployment." D-101's family matrix (`:2015-2016`) requires ULA on data-tenant, storage, replication and a ULA leg on metal-admin/metal-internal -- none has any MAAS v6 presence to bind against. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling (D-101's own "Remaining open item" at `:2032` is the un-assigned NetBox literals -- org ULA /48 and per-DC GUA carve. Those must be assigned and carved into MAAS before a dual-stack deploy; the alternative is re-opening the CLOSED v4-only phasing.) + +**ID: L2-4** +CLAIM: All three rack-side checkers emit a confident, definitive `FAIL` when invoked in the literal form the routing table gives (`bash scripts/.sh check ` from the repo root on the jumphost) -- they silently evaluate the LOCAL host instead of refusing. All three PASS when actually piped to their rack. A false RED on `dc-mirror.sh check dc1` is precisely what invites the `install` that CURRENT-STATE documents as destructive. +EVIDENCE: From the repo root on vcloud: `bash scripts/dc-mirror.sh check dc0` -> 15 MISS lines, `FAIL`, EXIT=1; `dc-cache-proxy.sh check dc1` -> 12 MISS, FAIL, EXIT=1; `dc-rack-net.sh check dc0` -> 8 MISS, FAIL, EXIT=1. Run correctly on the rack: `ssh voffice1 "ssh 172.31.0.2 'sudo -n bash -s -- check dc0'" < scripts/dc-mirror.sh` -> `OK 10.12.8.4/22 on virbr2`, `OK mirror answers http://10.12.8.4/ (200)`, `OK last-sync: OK 2026-07-27T08:43:46Z ubuntu=0 uca=0`, **PASS**; `dc-cache-proxy.sh` on dc1 -> **PASS**; `dc-rack-net.sh` -> **PASS on both racks**. The scripts state the requirement in prose but do not enforce it (`scripts/dc-rack-net.sh:8`). The copy-pasteable form is `SKILL.md:97`, which renders the bare local command as the backticked answer with the transport caveat later in the cell. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX (mechanical -- a host-identity precondition that REFUSES with a distinct exit code when the expected rack markers are absent, rather than reporting FAIL; the "unrecognised state refuses" rule applied to the RED side) + +**ID: L2-5** +CLAIM: The DC plane bridges are jumbo (MTU 9000) on both racks, but every MAAS VLAN -- including all 10 named plane fabrics -- records MTU 1500, and neither `bundle.yaml` nor any overlay carries a single MTU option; D-101's mandatory MTU budget is neither achieved nor declared anywhere the deploy will read. +EVIDENCE: `maas admin fabrics read` -> 17/17 VLANs report `mtu 1500` (all five dc0 plane fabrics, all five dc1, `fabric-4`, `fabric-142`, ...). On the racks: `ip -o link show` -> `enp1s0 mtu 1500` (rack transit leg), `virbr1..virbr6 mtu 9000` (the six plane bridges), same on both. `grep -n -i "mtu" bundle.yaml overlays/*.yaml` -> no output. Contradicts `docs/design-decisions.md:2025` (D-101 Tenant/MTU sub-policy: raise the underlay to jumbo end-to-end so tenant MTU stays 1500, or if pinned at 1500 tenant MTU drops to ~1444 and MUST be set consistently across ovn geneve, tenant-network MTU and amphora; "The measured underlay MTU is a Phase-0 gate -- do not assume jumbo"). Measured state is NEITHER branch: jumbo bridges, 1500 VLAN records, 1500 rack transit (so cross-DC replication is not jumbo end-to-end), no MTU on any charm. `scripts/dc-dc-mtu-geneve-budget.sh` exists but has never been run to a recorded verdict; invoked bare it correctly refuses (`FAIL: --underlay-mtu is REQUIRED`). +BLOCKS-STAGE-5: no (the deploy succeeds; the failure surfaces later as tenant/geneve blackholing -- D-101 names it "the classic nested-OpenStack failure mode") | DISPOSITION: ruling + +**ID: L2-6** +CLAIM: `scripts/lib-net.sh` declares a metal-admin gateway for both VR1 DCs that does not exist -- MAAS records no gateway on either metal-admin subnet, and nothing answers ARP or ICMP at either address. +EVIDENCE: `maas admin subnets read` -> `10.12.8.0/22 none`, `10.12.68.0/22 none` (control: `10.12.4.0/22 10.12.4.1`, `10.12.64.0/22 10.12.64.1`). `ping -c2 -W2 10.12.8.1` from the dc0 rack -> 100% loss, `ip neigh` FAILED; same for `10.12.68.1` from dc1. Control: both provider-public gateways answer 0% loss, ARP REACHABLE, and both racks' default route goes via them. Contradicts `scripts/lib-net.sh:37` (`PLANE_GW` includes `["10.12.8.0/22"]="10.12.8.1"`, inherited verbatim by the `vr1-dc0` arm at `:112-127`) and `:149` (the `vr1-dc1` arm's own `["10.12.68.0/22"]="10.12.68.1"`). The `10.12.8.1` value traces to VR0's real as-built (`phase-00-maas-standup.sh:119`); VR1's edges carry only a provider-public LAN leg, so metal-admin has no router in either DC. Consumer that would trip: `scripts/pre-flight-checks.sh:121-124` hard-fails on `gateway=$gw (want $exp)`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX (the vr1-dc1 arm should not carry a metal-admin gateway, and the vr1-dc0 arm's inheritance needs the same caveat treatment `lib-net.sh:120-126` already applies to `METAL_INTERNAL_VID`/`METAL_INTERNAL_IFACE`) + +**ID: L2-7** +CLAIM: The D-104-amendment 10th controller VM does not exist in either DC and no `juju-controller-vr1-dc*` MAAS tag exists -- measured confirmation of a recorded pending item. +EVIDENCE: `virsh list --all` on each rack -> exactly 9 node domains (all `shut off`) + 1 running edge per DC; no 10th VM. MAAS holds 20 machines (18 VR1 nodes + `office1-netbox` + `office1-tailscale`), none with a controller tag; measured tag set is the 5 already reported. +BLOCKS-STAGE-5: yes | DISPOSITION: log-only (already recorded as gated pending work; measured confirmation, do not double-count) + +### LENS 2 CONFIRMED AGAINST LIVE (no finding -- the record is RIGHT) +- 18 nodes, all `Ready`, 9 per DC, shapes EXACT to D-121 Option C: 3x`16cpu/65536MiB` + 2x`12cpu/49152MiB` + 4x`8cpu/24576MiB` per DC; 18/18 `power_type=virsh`. +- All 18 pinned boot MACs match `scripts/lib-hosts.sh:118-126` and `:141-149` exactly, cross-checked tofu-name -> MAC -> live `system_id` via `power_id`. +- `VIRSH_POWER_ADDRESS` matches live for all 18 (`qemu+ssh://jessea123@172.31.0.2/system` dc0 9/9, `...@172.31.0.6/system` dc1 9/9). `lib-hosts.sh:127` and `:150` correct. +- **17 fabrics -- CURRENT-STATE:349 verified exactly.** Survivors: 10 named plane fabrics, `fabric-4`/`fabric-142`, `fabric-0`/`fabric-1`, `fabric-2`/`fabric-141`, `fabric-3`. No `192.168.1.0/24`, no `fabric-5`. +- 6 spaces, 2 subnets each, 12 VLANs space-assigned, all `vid=0` untagged -- the D-133 flat carve holds. +- 7 interfaces per node, ZERO orphans, 18/18: `enp1s0..enp6s0` + `br-ex`. `enp2s0` correctly carries no link (D-100 provider-raw); `br-ex` is a bridge over `enp2s0` holding the provider-public static. +- D-134 statics perfect: one octet per node identical across all 6 planes, 6 links each, 18/18 -- control `.100/.101/.102`, compute `.120/.121`, storage `.150-.153`, both DCs. +- Rack legs + D-131 forwarder up on both racks (`dc-rack-net.sh check` PASS both). Forwarders live-probed: `dig @10.12.8.3` and `@10.12.68.3` NOERROR for `maas-internal`, `archive.ubuntu.com`, `streams.canonical.com`. +- Both artifact paths PASS on their own hosts. Both DC edges running and routing at 0% loss. +- Boot images: `ubuntu/jammy amd64/ga-22.04` + 5 jammy variants all `Synced`, plus noble. + +### LENS 2 UNMEASURED +1. Whether MAAS would in FACT allocate a Juju/LXD container address out of the unreserved `.50-.99` band (L2-2) -- behavioural, only observable during a real deploy. What IS measured is MAAS's own declaration that the band is unreserved on 12/12 subnets; no claim is made about allocation order. +2. The three `tofu plan` zero-diff captures -- not attempted by this lens. [DONE BY THE AUDIT SESSION: all three ZERO DIFF, see `docs/audit/stage5-live-measurement-20260727.txt` section 1.] +3. Node-side anything -- all 18 nodes powered off in `Ready`; gate G17's territory by construction. +4. `site-baseleg.sh check office1` -- outside this lens's bound. [DONE BY THE AUDIT SESSION: PASS.] +5. IPv6 state at the NetBox apex and on the two DC edges -- MAAS and nodes only were measured. L2-3 is scoped to MAAS + node reality and does NOT assert the apex is empty. +6. `maas` CLI reachability from the jumphost itself -- every MAAS read went through `ssh voffice1`; contributes nothing to the recorded "MAAS unreachable from the jumphost" preflight red. +7. dc1 rack bridge-to-network name mapping -- MTUs measured by index; only `virbr6`=metal-admin and `virbr4`=provider-public resolved by name. The 9000/1500 finding does not depend on the unmapped ones. +8. Whether `serial-console` / `pod-console-logging` tags are MEANT to be on the VR1 nodes -- measured that they are not; no ruling found requiring them, but not exhaustively searched. + +--- + +## LENS 7 -- ROOSEVELT-DELTA / GOAL ALIGNMENT (returned fifth) + +**ID: L7-1** +CLAIM: Stage 5's Juju work executes on the headend (voffice1), but D-137's absolute discovery bound declares `~/vault-init/*`, `~/octavia-pki/*` and `~/admin-openrc` on the **jumphost only** -- so every credential Stage 5's bundle deploy mints lands at a location the register is FORBIDDEN to visit, reproducing SEC-022 at the scale of the whole control plane. +EVIDENCE: `creds-manifests/vm-secret-locations:54-56,74-78` declare those five paths as `jumphost ... local`; the headend block `:83-92` contains only `~/vr1-dcN-creds/*`, `/root/maas-secrets/*`, `~/snap/maas/current/.maascli.db`, `~/.local/share/juju/*`, `/var/snap/maas/current/root/.ssh/*`, `/var/snap/maas/common/maas/secret` -- no vault-init, no octavia-pki, no admin-openrc. D-137 sub-ruling 3 (operator utterance "Declared locations only") makes that list ABSOLUTE: "it never walks outside the list". D-128 fixes Plane-2 execution on voffice1 (`docs/design-decisions.md:4176-4180`); SEC-018 (`docs/security-ledger.md:32`) already records the consequence. Jumphost has no juju: `docs/audit/preflight-20260727-stage5-audit.txt:49`. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX +[AUDIT-SESSION CONFIRMATION: `ssh voffice1 'command -v juju'` -> `/snap/bin/juju`; on vcloud `command -v juju` -> ABSENT. `~/admin-openrc`, `~/vault-init`, `~/octavia-pki` all ABSENT on voffice1 today. `~/.local/share/juju/` exists on voffice1 with `clouds.yaml`, `credentials.yaml`. Premise CONFIRMED.] + +**ID: L7-2** +CLAIM: The credential register carries ZERO rows for what Stage 5 actually mints; the vault/Octavia/openrc rows are `singleton` under `vr0-phase0N` mint-stages that `stages-reached` marks `pending`, so P5 emits a green "deferred as not-yet-minted" straight through the largest mint of the deployment, and the per-DC symmetry gate is structurally incapable of firing on them. +EVIDENCE: `creds-matrix.tsv` -- all 14 vault/Octavia-PKI/openrc/vault-ca rows are `cardinality=singleton`, `site-key=-`, `mint-stage` in {`vr0-phase01`,`vr0-phase02`,`vr0-phase03`}; `creds-manifests/stages-reached` marks all three `pending`, so flipping `stage5` to `reached` never makes them expected. The only 8 `mint-stage=stage5` rows are `dcN-maas-apikey` / `dcN-juju-maas-user` / `juju-client-clouds` -- already minted 2026-07-24 (SEC-018/-019), not new Stage-5 output. Symmetry gate: `scripts/creds-matrix.py:380` `if r.site_key != dc or r.cardinality != "per-DC": continue`. Measured green: `docs/audit/preflight-20260727-stage5-audit.txt:85` `[ok] E1 18 expected artifact(s) deferred as not-yet-minted`. Both operator branches fail closed-eyed: `runbooks/phase-02-vault-bringup.md:69,71` (`mkdir -p ~/vault-init`; `... | tee ~/vault-init/init.txt`) collides two DCs' unseal shares at one fixed path on an irreversible one-shot; `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:229-232`'s "Store them separately, clearly labeled by DC" puts material at a path `vm-secret-locations` does not list -- and `scripts/phase-03-admin-openrc.sh:31` still reads the fixed `CA="${CA:-$HOME/vault-init/vault-ca-root.pem}"`. +BLOCKS-STAGE-5: yes | DISPOSITION: DOCFIX + +**ID: L7-3** +CLAIM: The `.50-.60` VIP band, `provider-ext`/FIP pool and reserved-CIDR values that Stage 5 Steps 7-9 write into vr1-dc1 come from scripts that source `lib-net.sh` and NEVER call `lib_net_select_dc`, so they silently use **VR0/dc0's** 10.12.4/8/12/16/32/36 literals against a DC whose planes are 10.12.64-84. +EVIDENCE: measured over all 27 `lib-net.sh` consumers -- only 6 reference the selector at all. `scripts/phase-04-network-create.sh:24`, `phase-04-network-verify.sh:35`, `phase-03-core-verify.sh`, `phase-05-octavia-verify.sh`, `pre-flight-checks.sh`, `juju-spaces-check.sh:17`, `tenant-onboard.sh:183`, `tenant-acceptance.sh`, `tenant-offboard.sh` all source unconditionally with zero selector references and expose no `$DC` hook. `lib-net.sh:22` sets VR0 defaults; `:137-146` are dc1's divergent values. Safety is opt-in by construction -- `lib-net.sh:76-78` "sourcing this file with no further action continues to populate PLANE_CIDRS ... exactly as above (VR0/DC0's real, measured values)"; the dc1 arm's fail-loud `unset` guard (`:155-157`) only fires if the selector is CALLED. Remediation pattern exists and was applied to just three scripts (`scripts/carve-host-interfaces.sh:48-67`, DOCFIX-166). `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:119-125` has the operator call the selectors in their OWN shell -- a variable no child `bash scripts/...` process inherits, and which each script overwrites by re-sourcing. +BLOCKS-STAGE-5: yes (the vr1-dc1 arm; harmless for vr1-dc0 under D-101 value-for-value inheritance) | DISPOSITION: DOCFIX + +**ID: L7-4** +CLAIM: D-128 classifies the INNER `tofu` roots as Plane 2 -- "what we REHEARSE for Roosevelt" -- while every one of their five modules is libvirt-substrate creation that the shim register says has NO Roosevelt analog; a Roosevelt session greping design-decisions would treat `opentofu/vr1-dcN-substrate/` as transferable production IaC, which the workflow doc explicitly forbids. +EVIDENCE: `docs/design-decisions.md:4176-4180` (D-128 Plane 2) names "the INNER `tofu` root" alongside MAAS/NetBox/tenant tests as the rehearsed half; carried as a standing invariant at `SKILL.md:123-127`. Actual content of `opentofu/vr1-dc1-substrate/main.tf:24,33,46,57,142` = `dc-storage-pool`, `dc-planes`, `wan-bridge`, `opnsense-edge`, `node-vm`. 11 of 12 modules are `dmacvicar/libvirt`. `docs/dc-dc-deployment-workflow.md:157` on that same artifact: "Explicitly tagged shim: node-VM creation has no Roosevelt analog ... do not treat as reusable production IaC." The two boundaries differ by an entire tofu root. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L7-5** +CLAIM: The Section-9 shim register has not been maintained since 2026-07-09, and the workflow doc's exhaustive claim ("the ONLY deliberate no-Roosevelt-analog steps are the Section 9 shim register") is now FALSE -- at least six VR1-only mechanisms have accumulated outside it. +EVIDENCE: `docs/dc-dc-buildout-design.md:199-210` lists exactly four items; `docs/dc-dc-deployment-workflow.md:12-13` makes the exhaustive claim. Absent from the register, each measured: (i) D-128 Plane 1, self-declared "**No Roosevelt analog**" at `docs/design-decisions.md:4172`; (ii) the DC containment VM, "a deliberate simulation-only delta" at `docs/dc-dc-deployment-workflow.md:87`; (iii) `scripts/site-baseleg.sh:4-27`; (iv) `scripts/site-forward.sh:4-13`; (v) `scripts/dc-rack-net.sh:17-22` (the rack-legs half; the D-131 dnsmasq half at `:23-32` genuinely DOES transfer, and the script does not distinguish them); (vi) `scripts/osd-blank-check.sh:5,21-25`. Calibration counter-examples that ARE correctly tagged: `scripts/sudoers.d/netem-tc:25` and `scripts/maas-node-power.sh:23-25`. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L7-6** +CLAIM: Stage 5 cannot start phase-01 Step 1.0 without an unmade operator decision on whether the two DCs share one Octavia CA -- and the reuse branch would put a single amphora control-plane CA private key across two clouds that D-100 defines as independent, which is the wrong answer for hard tenant isolation. +EVIDENCE: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:204-208` ("this is a call the operator should make explicitly ... not silently default to reuse") -- correctly flagged, but UNMADE. Contrast the ruled position one step later at `:225-227`: "Per D-109: each DC's Vault is its OWN independent root CA -- no regional root-of-trust this phase." `overlays/octavia-pki.yaml` is a single unscoped repo-clone path holding two CA private keys plus a plaintext issuing-CA passphrase, in a repo SEC-004 records as currently PUBLIC. +BLOCKS-STAGE-5: yes | DISPOSITION: ruling + +**ID: L7-7** +CLAIM: The credential reproducibility debt is real and measured, but its sharp edge is VR1-PRESENT, not Roosevelt-future: SEC-007/-015 make edge SSH the only management path, so a jumphost rebuild locks the operator out of both DC edges TODAY -- while `creds-mint.sh` remains unbuilt and the checker reports the debt as `[ok]`. +EVIDENCE: measured on `creds-matrix.tsv`: **30 rows across 16 distinct ids** carry `mint-ref=operator-terminal`. Origin figure is different accounting -- `docs/audit/creds-creation-points-20260725.md` FINDING 1 counts 12 declared secrets with no mint command, from `grep -rnI "ssh-keygen" .` returning ZERO hits repo-wide. Gate reports it non-blocking: `preflight-20260727-stage5-audit.txt:77` `[ok] S4 provenance debt: 30 row(s) ... Admitted by design`. `scripts/creds-mint.sh` does not exist. ASSESSMENT -- minimum fix by SHAPE, not by inventing a command: (1) for the six SSH keypairs and the OPNsense root password/hash, record the mint invocation as a numbered runbook step and flip those rows' `mint-ref` from `operator-terminal` to `runbook::`; the S4 check already resolves `runbook:` references, so the debt becomes convergent WITHOUT a new tool. (2) Do the six edge/service/power keys FIRST -- their loss is unrecoverable-in-place. (3) `creds-mint.sh` is orthogonal: it prevents the NEXT unregistered mint, it does not retroactively make an existing key reproducible, so it should not gate this. +BLOCKS-STAGE-5: no | DISPOSITION: ruling + +**ID: L7-8** +CLAIM: Deploying Stage 5 with the apex incomplete is NOT now-or-never for the D-136 generator, but it IS now-or-never for the authority DIRECTION: the 33 VIPs become as-built Keystone endpoints and cert SANs, after which NetBox can only be back-filled to match the deploy, permanently inverting "NetBox is the IPAM apex". +EVIDENCE: D-136 (`docs/design-decisions.md:4761+`) is PROPOSED/unruled; its own option (C) argues hand-rendering first is what makes the front half "testable BY REPRODUCTION", so a generator is genuinely not now-or-never. What Stage 5 FORECLOSES: `overlays/vr1-dc1-vips.yaml:16-45` fixes 33 dc1 VIPs; once hacluster owns them and Keystone endpoints / Vault-issued SANs are registered against them they stop being an editable literal. Stage 5 has no wiring to the apex at all: `grep -n "D-136\|D-134\|apex\|netbox" runbooks/dc-dc-phase4-juju-bundle-per-dc.md` returns one incidental hit (`:211`). +BLOCKS-STAGE-5: no | DISPOSITION: log-only + +**ID: L7-9** +CLAIM: The substrate's declared INPUT values are the one deploy-artifact class the repo does not carry -- `*.auto.tfvars` is gitignored under a rationale written for STATE files -- so the runbooks-and-scripts-as-primary-deliverable property does not hold for `opentofu/`. +EVIDENCE: `.gitignore:75-76` `opentofu/**/*.auto.tfvars` + `.auto.tfvars.json`; `git ls-files | grep tfvars` returns nothing, and no `.example`/template is tracked. Two live tfvars sit untracked on disk (`opentofu/dc-dc-phase0.auto.tfvars`, `opentofu/d124-rack.auto.tfvars`), and the inner roots' tfvars live on voffice1. The ignore block's stated reason (`.gitignore:63-70`, DOCFIX-175) is about tfstate storing sensitive attributes in plaintext, and explicitly records "never pass secrets via `-var`" -- i.e. tfvars carry no secrets by this repo's own rule, so nothing in the rationale covers them. Consequence: the ZERO-DIFF `tofu plan` expectation depends on artifacts a fresh clone does not have. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +**ID: L7-10** +CLAIM: `lib-net.sh`'s vr1-dc1 arm still unsets `KEYSTONE_VIP_DEFAULT`/`VIP_PREFIX_*`/`FIP_POOL_*` as "not yet ruled/measured", but those values have since arrived in `overlays/vr1-dc1-vips.yaml`. +EVIDENCE: `scripts/lib-net.sh:152-157` vs `overlays/vr1-dc1-vips.yaml:16` `keystone: vip: "10.12.64.50 10.12.68.50 10.12.72.50"`. Sequencing gap, not a contradiction: the guard was correct when written and has not been closed. `KEYSTONE_VIP_DEFAULT` is consumed by the tenant-* family (`scripts/tenant-onboard.sh:19`) and phase-03/06. [Same underlying item as L1-1; L7-10 adds the consumer list.] +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX + +### LENS 7 UNMEASURED +1. Whether any repo timeout/wait/sizing constant was widened for depth-4 nested I/O and is now baked in without a VR1-only note. No constant found documented as nested-virt-calibrated; that is not evidence none exists. Unqualified candidates: `scripts/phase-05-amphora-pipeline.sh:43 RETROFIT_WAIT="${RETROFIT_WAIT:-30m}"` (runs at Stage 5 Step 9 on depth-4 nested compute; provenance of 30m not established) and `scripts/site-baseleg.sh:52-55 BRIDGE_WAIT=60`. +2. Live confirmation that the Stage-5 execution host is voffice1. [DONE BY THE AUDIT SESSION -- see the L7-1 confirmation note above.] +3. Whether `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:119` is a known DOCFIX or an unrecorded contradiction of D-128 -- the DOCFIX register was not grepped for an existing entry. + +--- + +## AUDIT-SESSION FINDINGS (measured by the session itself, not by a lens) + +**ID: S-1** +CLAIM: The `openstack` client is ABSENT on BOTH the jumphost and the headend, and TEN Stage-5/6/7 scripts require it -- so every post-deploy step from phase-03 onward cannot run anywhere today. +EVIDENCE: `command -v openstack` on vcloud -> ABSENT; `ssh voffice1 'command -v openstack'` -> ABSENT. `ssh voffice1 'snap list'` shows only core24, juju, lxd, maas, postgresql, snapd -- no openstack client. Consumers: `grep -ln 'openstack ' scripts/phase-0*.sh` -> `phase-03-admin-openrc.sh`, `phase-04-network-create.sh`, `phase-04-network-verify.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-amphora-pipeline.sh`, `phase-05-octavia-verify.sh`, `phase-06-bootstrap.sh`, `phase-06-capi-stack.sh`, `phase-06-mgmt-vm.sh`, `phase-06-net-setup.sh`. No lens found this and no repo surface records it. +BLOCKS-STAGE-5: yes (from Step 7 onward) | DISPOSITION: DOCFIX (install/record the client on the ruled Plane-2 host; the WHERE follows from D-128, so no new choice is required) + +**ID: S-2** +CLAIM: Juju on the headend is 3.6.27, not the 3.6.25 the record carries -- an in-channel patch refresh that D-071 permits but that leaves CURRENT-STATE's pin stale, and it is the version that will bootstrap the controller. +EVIDENCE: `ssh voffice1 'snap list'` -> `juju 3.6.27 rev 35621 3/stable`. Against `docs/CURRENT-STATE.md:161-162` ("Juju 3.6.25 on voffice1") and the section-7 pin row ("Last recorded VR0 value 3.6.25 -- historical, unverified here"). D-071 ADOPTED 2026-07-21 rules patch-only controller jumps and in-channel-only refreshes, so 3.6.25 -> 3.6.27 within `3/stable` is POLICY-COMPLIANT; the defect is only that the record was not updated. GA-R1 amendment C2: the measurement wins and corrects the document in the capture's commit. +BLOCKS-STAGE-5: no | DISPOSITION: DOCFIX diff --git a/docs/audit/stage5-live-measurement-20260727.txt b/docs/audit/stage5-live-measurement-20260727.txt new file mode 100644 index 0000000..7cbafa8 --- /dev/null +++ b/docs/audit/stage5-live-measurement-20260727.txt @@ -0,0 +1,237 @@ +===================================================================== +Stage-5 grounding audit -- LIVE MEASUREMENT SWEEP +Date: 2026-07-27 Branch: dc-dc-stage5-grounding-audit +Run by: the audit session itself (not a subagent), read-only throughout. +Charter: docs/audit/stage5-grounding-audit-scope-20260727.md +===================================================================== + +This file records MEASUREMENTS with their commands. Under GA-R1 amendment +C2, a fresh measurement WINS over the document and corrects it in the +capture's own commit. + +--------------------------------------------------------------------- +1. CONVERGENCE -- all three OpenTofu roots, ZERO DIFF +--------------------------------------------------------------------- +The standing expectation (CURRENT-STATE section 5) is ZERO DIFF, and ANY +diff is a STOP. All three roots were planned this session. + +OUTER (on vcloud): + $ tofu -chdir=opentofu plan -no-color + -> "No changes. Your infrastructure matches the configuration." + capture: docs/audit/outer-plan-20260727-stage5-audit.txt exit 0 + +INNER dc0 (ON voffice1 -- D-128 Plane 2, its tfstate lives there): + $ ssh voffice1 'cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate \ + && tofu plan -no-color' + -> "No changes. Your infrastructure matches the configuration." exit 0 + (tfvars present: d124-inner.auto.tfvars; 10 domains refreshed -- + 9 nodes + vr1-dc0-opnsense) + +INNER dc1 (ON voffice1): + $ ssh voffice1 'cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc1-substrate \ + && tofu plan -no-color' + -> "No changes. Your infrastructure matches the configuration." exit 0 + +VERDICT: the substrate is genuinely converged across all three roots. This +is the strongest single as-built test available and it passes cleanly. + +VALIDITY NOTE (important): the voffice1 clone is 105 commits behind main +(finding 5 below), so an inner plan run there uses that clone's config. It +was verified FIRST that the inner roots did not change: + $ git diff --stat 61c416e..main -- opentofu/vr1-dc0-substrate/ \ + opentofu/vr1-dc1-substrate/ opentofu/modules/ + -> EMPTY (no differences) +So the inner plans are valid convergence evidence. Had that diff been +non-empty, they would have been UNMEASURED, not green. + +--------------------------------------------------------------------- +2. GAUNTLET AND LINT +--------------------------------------------------------------------- + $ bash scripts/run-tests-all.sh + -> "GAUNTLET: ALL GREEN (81 harnesses)" exit 0 + + $ bash scripts/repo-lint.sh + -> 0 fail, 1 warn (L1 legacy non-ASCII carve-out in design-decisions.md) + This matches the expected result. NOTE: during the sweep repo-lint + reported "1 fail" -- L10 firing on this session's own UNCOMMITTED capture + files under docs/audit/. That is the rule WORKING (GA-R1 C1: a commit + touching a state-bearing surface must touch CURRENT-STATE.md in the same + commit), not a defect. It is recorded here because a future session will + see the same thing and should not treat it as a regression. + +--------------------------------------------------------------------- +3. PREFLIGHT -- red for EXACTLY the known reasons, nothing new +--------------------------------------------------------------------- + $ bash scripts/preflight.sh -> exit 1 + capture: docs/audit/preflight-20260727-stage5-audit.txt + + P1 repo lint ............ FAIL (this session's uncommitted captures; see 2) + P2 bundle invariants .... PASS (Pattern A / D-052-D-053) + P3 channel assert ....... WARN (33 pins, 0 fail, 33 warn) + P4 live pre-flight + CHECK 0 octavia-pki .. FAIL overlays/octavia-pki.yaml MISSING + (gitignored secret; known, expected) + CHECK 1 bundle VIPs .. PASS vip: line count = 11; triples OK=11 bad=0 + MAAS reachability .... FAIL 'maas admin subnets read' from the jumphost + (expected: the region is ON voffice1) + P5 credential matrix .... FAIL 82 rows, 15 groups clean, 7 findings + (matches the recorded residual 7 exactly) + +VERDICT: preflight was ALREADY red before this session and remains red for +the same reasons. NOTHING NEW has joined the set. P5's 7 findings match the +recorded residual exactly (dc0-edge-api x2, S5 x3, S6 conflation x1, +E4 uncheckable x2). + +CONSEQUENTIAL SIDE-FINDING -- see finding 6 below: CHECK 1 PASSING is not +the good news it looks like. + +--------------------------------------------------------------------- +4. PER-DC ARTIFACT PATHS -- both genuinely healthy +--------------------------------------------------------------------- +OWNED MEASUREMENT ERROR, recorded rather than quietly corrected: this +session first ran + $ bash scripts/dc-mirror.sh check dc0 + $ bash scripts/dc-cache-proxy.sh check dc1 +LOCALLY on vcloud. Both printed a comprehensive FAIL with every probe MISS +("apt-cacher-ng not installed", "nginx not active", utility address absent). +That output is INDISTINGUISHABLE from "the service was never installed". +It was wrong. Both scripts are designed to be PIPED TO THE RACK: + ssh -i -J voffice1 @ 'sudo bash -s' -- check \ + < scripts/dc-.sh +(header, scripts/dc-cache-proxy.sh:13-15). Run locally they check vcloud, +where of course nothing is installed. The scripts are NOT at fault. + +True exit codes were then measured (the first reading had captured `tail`'s +status through a pipe, not the script's): + $ bash scripts/dc-mirror.sh check dc0 >/dev/null 2>&1 ; echo $? -> 1 + $ bash scripts/dc-cache-proxy.sh check dc1 >/dev/null 2>&1 ; echo $? -> 1 +So they DO fail closed. + +The canonical piped invocation needs remote sudo, which is an `ask` rule and +the operator is away -- so it is UNMEASURED (see section 6). The FUNCTIONAL +question was measured instead, without sudo, from each rack host: + +dc0 rack (172.31.0.2, transit IP taken from scripts/lib-hosts.sh:127): + hostname vvr1-dc0; up 1 week 6 h; nginx ACTIVE + virbr2 10.12.8.2/22 10.12.8.3/22 10.12.8.4/22 ; virbr5 UP 10.12.4.2/22 + $ curl http://10.12.8.4/ -> 200 + $ curl http://10.12.8.4/ubuntu/dists/jammy/Release -> 200 + $ cat /var/lib/dc-mirror/last-sync.status + -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0 [attested current] + +dc1 rack (172.31.0.6, from scripts/lib-hosts.sh:150): + hostname vvr1-dc1; up 4 days 14 h; apt-cacher-ng ACTIVE + virbr4 UP 10.12.64.2/22 ; virbr6 10.12.68.2/22 10.12.68.3/22 10.12.68.4/22 + $ curl --proxy http://10.12.68.4:3142 http://archive.ubuntu.com/...jammy/Release + -> 200 + $ curl --proxy http://10.12.68.4:3142 http://ubuntu-cloud.archive.canonical.com/ + ubuntu/dists/jammy-updates/caracal/Release -> 200 + +VERDICT: dc0's full mirror and dc1's caching proxy are BOTH genuinely +serving, and dc0's sync is attested current. CURRENT-STATE's claim holds. +The D-135-amended per-DC split is intact (dc0 mirror / dc1 proxy). + +OBSERVATION, not a defect: on both racks the metal-admin bridge (virbr2 / +virbr6) reads DOWN while carrying its addresses. That is expected -- all 18 +node VMs are powered off in `Ready` under the READY-handoff ruling, so the +bridge has no attached running domain. It is noted because it is exactly the +kind of state that looks alarming to a fresh reader. + +--------------------------------------------------------------------- +5. OFFICE1 BASE LEG +--------------------------------------------------------------------- + $ bash scripts/site-baseleg.sh check office1 + ok leg 10.10.0.10/24 on virbr2 + ok route 10.10.1.0/24 via 10.10.0.20 exit 0 + +--------------------------------------------------------------------- +6. THE TWO CONSEQUENTIAL FINDINGS FROM THIS SWEEP +--------------------------------------------------------------------- +FINDING A -- THE OFFICE1 HEADEND CLONE IS 105 COMMITS STALE, AND STAGE 5 +EXECUTES FROM IT. + + $ ssh voffice1 'cd ~/openstack-caracal-dc-dc && git rev-parse --abbrev-ref HEAD' + -> dc-dc-g12-dc1-substrate + $ ... git log -1 --format="%H %ci %s" + -> 61c416e... 2026-07-22 21:38:06 +0000 + "G12 dc1 step B COMPLETE: transit live, rack enrolled, SEC-010 both ends" + $ ... git rev-list --left-right --count origin/main...HEAD + -> 105 0 [105 BEHIND origin/main, 0 ahead] + $ git diff --stat 61c416e..main | tail -1 + -> 133 files changed, 19381 insertions(+), 432 deletions(-) + +D-128 fixes Plane 2 (MAAS / NetBox / inner tofu / tenant tests) to EXECUTE +on voffice1, and Stage 5 is a Plane-2 activity: `juju bootstrap` and +`juju deploy` run from the Office1 headend. The deploy artifacts there are +therefore the PRE-RENDER ones. Measured: + + $ ssh voffice1 '... for f in ...; do [ -e $f ] && echo PRESENT || echo ABSENT' + PRESENT bundle.yaml + ABSENT overlays/vr1-dc1-vips.yaml + ABSENT overlays/vr1-dc1-machines.yaml + PRESENT overlays/dc-ha-scaleup.yaml + PRESENT scripts/provider-bundle-check.py + ABSENT scripts/creds-matrix.py + ABSENT creds-matrix.tsv + + bundle.yaml sha256 (first 16): vcloud/main 7d6261c55d50f362 + voffice1 40dc685eaccfa1d2 + voffice1's machines block: + machines: + "8": constraints: arch=amd64 tags=openstack + "9": constraints: arch=amd64 tags=openstack + "10": constraints: arch=amd64 tags=openstack + "11": constraints: arch=amd64 tags=openstack + +That is the VR0 4-node HYPERCONVERGED layout -- precisely the pre-render +state the 2026-07-24 committee identified as the deploy blocker. The +9-node role-separated render (machines "0"-"8", role+DC tags, nova-compute +3->2, ceph-osd on 4 storage) exists ONLY on vcloud/main. + +Consequence if unaddressed: an operator following the phase-4 runbook on +voffice1 today would either fail on the two absent overlay files or deploy +the WRONG TOPOLOGY. Note also that both retired branches still exist as +stale remote-tracking refs in that clone (it has never been pruned), while +ground truth from vcloud is: + $ git ls-remote --heads origin + -> only refs/heads/main and refs/heads/dc-dc-stage5-grounding-audit +so a naive `git pull` there operates on a branch that no longer exists +upstream. + +FINDING B -- A GA-R5 RULING FROM 2026-07-25 WAS NEVER EXECUTED, AND THE +CHECKER THAT WOULD HAVE CAUGHT IT PASSES ONLY BECAUSE IT WAS NOT. + +Ruling 3 of 2026-07-25 states that bundle.yaml becomes VIP-FREE (topology +only), every per-DC value arrives via overlay including dc0's, and dc0's +VIPs extract into `overlays/vr1-dc0-vips.yaml` in two commits. + + $ ls overlays/ + dc-dc-ipv6-family-matrix.yaml + dc-ha-scaleup.yaml + vr1-dc1-machines.yaml + vr1-dc1-vips.yaml + -> overlays/vr1-dc0-vips.yaml is ABSENT + $ grep -c 'vip:' bundle.yaml -> 11 + +So bundle.yaml still carries its 11 inline VIPs and the ruled extraction has +not happened. This reframes BLOCKER-1: preflight P4 CHECK 1 reports +"vip: line count = 11 ... OK=11 bad=0" and PASSES today ONLY BECAUSE the +ruled change was never made. BLOCKER-1 is LATENT, not live -- it becomes +live the moment ruling 3 is executed. A green CHECK 1 is therefore evidence +that a ruling is outstanding, not evidence of health. + +This is the "a checker that cannot fail is not a gate" family seen from a +new angle: here the checker CAN fail, but its current PASS encodes an +un-executed ruling rather than a good state. + +--------------------------------------------------------------------- +7. UNMEASURED (could not look -- NOT "nothing there") +--------------------------------------------------------------------- +- The canonical piped `dc-mirror.sh check dc0` / `dc-cache-proxy.sh check + dc1` invocation, which needs `ssh ... 'sudo bash -s'` (an `ask` rule; the + operator is away). The functional substitute in section 4 covers the gate + question -- the artifact source answering on its own address with an + attested-current sync -- but does NOT cover the units/timers/persistence + half that the real checker asserts. Re-run both on the operator's return. +- Everything gated behind jumphost sudo, tofu apply, any maas write verb, + juju bootstrap/deploy, and opnsense POST. None were attempted.