diff --git a/docs/audit/stage5-preflight-dc0-20260801.txt b/docs/audit/stage5-preflight-dc0-20260801.txt new file mode 100644 index 0000000..20afa7d --- /dev/null +++ b/docs/audit/stage5-preflight-dc0-20260801.txt @@ -0,0 +1,253 @@ +PREFLIGHT TARGET: DC=vr1-dc0 (override: DC=vr1-dc1 bash scripts/preflight.sh) + Every gate below is run against THIS DC; the verdict line repeats it. +================ P1: repo lint ================ + [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 carve-out; NEW entries must be ASCII) + +WARN: repo lint (0 fail, 1 warn, 652 files scanned) +================ P2: bundle invariants ================ + validating the MERGED vr1-dc0 deploy input: bundle.yaml --overlay overlays/vr1-dc0-vips.yaml --overlay overlays/vr1-dc0-machines.yaml --overlay overlays/vr1-dc0-octavia-pki.yaml --dc vr1-dc0 + [ok] all 56 application(s) carry an explicit base == default-base 'ubuntu@22.04/stable' (re-runnable after a partial deploy) + [ok] 11 charms bind public->provider-public; none on provider-vip + [ok] 13 clustered VIP(s) are provider/admin/internal, octet 50-99 (13 dual-family) + [ok] ovn-chassis bridge-interface-mappings: 2 well-formed MAC(s) (role-sep; VR0 set N/A) + [ok] 108 relations well-formed (explicit endpoints, all apps exist) + [ok] mysql-innodb-cluster num_units=3 (D-062) + [ok] 12 hacluster principal(s) all carry a VIP (R11) + [ok] 12 hacluster subordinate(s) declare cluster_count == principal num_units + [ok] keystone policyd-override wired in-bundle; zip content matches source (DOCFIX-071) + [ok] machines block: all 9 machine(s) tagged openstack-vr1-dc0, matching --dc vr1-dc0 + [ok] placement: role-separated (3 control/2 compute/4 storage); anti-affinity + role placement + counts OK + +PASS: Pattern A / D-052-D-053 bundle invariants (bundle.yaml) +================ P3: channel assert (charmhub) ================ + [ok] barbican 2024.1/stable (barbican) + [ok] barbican-vault 2024.1/stable (barbican-vault) + [ok] ceph-mon squid/stable (ceph-mon) + [ok] ceph-osd squid/stable (ceph-osd) + [ok] ceph-radosgw squid/stable (ceph-radosgw) + [ok] ceph-rbd-mirror squid/stable (ceph-rbd-mirror) + [ok] cinder 2024.1/stable (cinder) + [ok] cinder-backup 2024.1/stable (cinder-backup) + [ok] cinder-ceph 2024.1/stable (cinder-ceph) + [ok] designate 2024.1/stable (designate) + [ok] designate-bind 2024.1/stable (designate-bind) + [ok] glance 2024.1/stable (glance) + [ok] glance-simplestreams-sync 2024.1/stable (glance-simplestreams-sync) + [ok] hacluster 2.4/stable (keystone-hacluster, glance-hacluster, neutron-api-hacluster, nova-cloud-controller-hacluster, placement-hacluster, openstack-dashboard-hacluster, cinder-hacluster, octavia-hacluster, barbican-hacluster, magnum-hacluster, ceph-radosgw-hacluster, designate-hacluster) + [ok] keystone 2024.1/stable (keystone) + [ok] magnum 2024.1/stable (magnum) + [ok] magnum-dashboard 2024.1/stable (magnum-dashboard) + [ok] memcached latest/stable (memcached) + [ok] mysql-innodb-cluster 8.0/stable (mysql-innodb-cluster) + [ok] mysql-router 8.0/stable (vault-mysql-router, keystone-mysql-router, glance-mysql-router, ncc-mysql-router, placement-mysql-router, neutron-api-mysql-router, cinder-mysql-router, dashboard-mysql-router, octavia-mysql-router, barbican-mysql-router, magnum-mysql-router, designate-mysql-router) + [ok] neutron-api 2024.1/stable (neutron-api) + [ok] neutron-api-plugin-ovn 2024.1/stable (neutron-api-plugin-ovn) + [ok] nova-cloud-controller 2024.1/stable (nova-cloud-controller) + [ok] nova-compute 2024.1/stable (nova-compute) + [ok] octavia 2024.1/stable (octavia) + [ok] octavia-dashboard 2024.1/stable (octavia-dashboard) + [ok] octavia-diskimage-retrofit 2024.1/stable (octavia-diskimage-retrofit) + [ok] openstack-dashboard 2024.1/stable (openstack-dashboard) + [ok] ovn-central 24.03/stable (ovn-central) + [ok] ovn-chassis 24.03/stable (ovn-chassis, ovn-chassis-octavia) + [ok] placement 2024.1/stable (placement) + [ok] rabbitmq-server 3.9/stable (rabbitmq-server) + [ok] vault 1.8/stable (vault) + +PASS: channel assert (33 pins, 0 fail, 0 warn) +================ P4: live pre-flight (MAAS/overlay/nodes) ================ + +=== DC selection === +PASS: gating DC=vr1-dc0 (planes 10.12.4.0/22 .. 10.12.36.0/22; 10 node(s)) + +=== Repo (informational) === +NOTE: REPO=/home/jessea123/openstack-caracal-dc-dc +NOTE: HEAD: db15666 P8 host guard: && -> || (live false FAIL on voffice1); dc0 MAAS path restored +NOTE: working tree clean + +=== CHECK 0: per-DC octavia-pki overlay (no key material printed) === +PASS: overlay present with 5 lb-mgmt-* keys +PASS: overlay ASCII clean + +=== CHECK 1: bundle VIPs -- v4 triple or R2 dual-family sextet, .50-.99 (provider/admin/internal) === +PASS: vip: line count = 13 (from overlays/vr1-dc0-vips.yaml) +PASS: aligned VIPs OK=13 bad=0 (DC=vr1-dc0 bands 10.12.4/10.12.8/10.12.12) + +=== MAAS reachability gate (read-only) === +PASS: MAAS reachable (profile=vr1-dc0-region) + +=== CHECK 3: six planes resolved BY CIDR (id/vid/gw/dns) === + provider-public 10.12.4.0/22 id=3 vid=0 gw=10.12.4.1 dns=[] + metal-admin 10.12.8.0/22 id=1 vid=0 gw=none dns=["10.12.8.6"] + metal-internal 10.12.12.0/22 id=4 vid=0 gw=none dns=[] + data-tenant 10.12.16.0/22 id=5 vid=0 gw=none dns=[] + storage 10.12.32.0/22 id=6 vid=0 gw=none dns=[] + replication 10.12.36.0/22 id=7 vid=0 gw=none dns=[] +PASS: all six planes present (by CIDR) +PASS: metal-internal is UNTAGGED (vid 0) -- D-133 flat carve +NOTE: stale-NAME check is juju-side (run scripts/juju-spaces-check.sh after add-model) + +=== CHECK 2: data/storage NIC links BY CIDR (DC=vr1-dc0 role nodes; octet per D-134 band) === + == vr1-dc0-control-01 (677cta, octet .100) == + enp3s0 -> 10.12.12.0/22 10.12.12.100 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.100 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.100 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.100 type=physical + == vr1-dc0-control-02 (wpftxc, octet .101) == + enp3s0 -> 10.12.12.0/22 10.12.12.101 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.101 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.101 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.101 type=physical + == vr1-dc0-control-03 (6p8pbx, octet .102) == + enp3s0 -> 10.12.12.0/22 10.12.12.102 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.102 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.102 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.102 type=physical + == vr1-dc0-compute-01 (m3cgc7, octet .120) == + enp3s0 -> 10.12.12.0/22 10.12.12.120 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.120 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.120 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.120 type=physical + == vr1-dc0-compute-02 (t6kpe3, octet .121) == + enp3s0 -> 10.12.12.0/22 10.12.12.121 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.121 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.121 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.121 type=physical + == vr1-dc0-storage-01 (t7ymp6, octet .150) == + enp3s0 -> 10.12.12.0/22 10.12.12.150 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.150 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.150 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.150 type=physical + == vr1-dc0-storage-02 (fg6gxm, octet .151) == + enp3s0 -> 10.12.12.0/22 10.12.12.151 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.151 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.151 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.151 type=physical + == vr1-dc0-storage-03 (yws74f, octet .152) == + enp3s0 -> 10.12.12.0/22 10.12.12.152 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.152 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.152 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.152 type=physical + == vr1-dc0-storage-04 (6q4syf, octet .153) == + enp3s0 -> 10.12.12.0/22 10.12.12.153 type=physical + enp4s0 -> 10.12.16.0/22 10.12.16.153 type=physical + enp5s0 -> 10.12.32.0/22 10.12.32.153 type=physical + enp6s0 -> 10.12.36.0/22 10.12.36.153 type=physical +NOTE: vr1-dc0-juju-01 (arfr7p, octet .5) is D-134 utility-band infrastructure, not an OpenStack role node -- data-plane carve not asserted + +=== CHECK 4: DC=vr1-dc0 OpenStack role nodes -- status / power === + vr1-dc0-control-01 -> vr1-dc0-control-01 Ready power=off +PASS: vr1-dc0-control-01 Ready + vr1-dc0-control-02 -> vr1-dc0-control-02 Ready power=off +PASS: vr1-dc0-control-02 Ready + vr1-dc0-control-03 -> vr1-dc0-control-03 Ready power=off +PASS: vr1-dc0-control-03 Ready + vr1-dc0-compute-01 -> vr1-dc0-compute-01 Ready power=off +PASS: vr1-dc0-compute-01 Ready + vr1-dc0-compute-02 -> vr1-dc0-compute-02 Ready power=off +PASS: vr1-dc0-compute-02 Ready + vr1-dc0-storage-01 -> vr1-dc0-storage-01 Ready power=off +PASS: vr1-dc0-storage-01 Ready + vr1-dc0-storage-02 -> vr1-dc0-storage-02 Ready power=off +PASS: vr1-dc0-storage-02 Ready + vr1-dc0-storage-03 -> vr1-dc0-storage-03 Ready power=off +PASS: vr1-dc0-storage-03 Ready + vr1-dc0-storage-04 -> vr1-dc0-storage-04 Ready power=off +PASS: vr1-dc0-storage-04 Ready + +Summary: 0 fatal, 0 warning +================ P5: credential matrix (D-137 tier 1 + tier 2 local) ================ +=== creds-matrix: tier 1 (STATIC) === +=== creds-matrix: tier 2 (EXISTENCE) === + (host: voffice1) + [ok] S1 schema: 121 rows, all enums valid, site-keys region-qualified, no duplicate (id,site,host-role,filename) + [ok] S3 render: 5 source field(s) SKIPPED -- rendering them needs the declared path from creds-manifests/vm-secret-locations (ruling 3); the list now EXISTS but the source-field derivation is not wired + [ok] S3 render drift: rendered row fields (mode, source) match checked-in; header prose and non-jumphost rows are OUT OF SCOPE of this compare + [ok] S4 mint-ref: every script:/runbook: reference resolves to a real location + [ok] S4 provenance debt: 40 row(s) are mint-ref=operator-terminal -- NOT reproducible from the repo (research FINDING 1). Admitted by design; converting them is remediation, not a checker fix. + [ok] S7 notes: 39 note key(s) referenced, all resolve, none orphaned + [ok] E0 jumphost location '~/vr1-office1-creds/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '~/vr1-dc0-creds/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '~/vr1-dc1-creds/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '~/vault-init/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '~/tenant-*/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '/home/jessea123/openstack-caracal-dc-dc/opentofu/terraform.tfstate' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '/home/jessea123/openstack-caracal-dc-dc/opentofu/terraform.tfstate.backup' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '/home/jessea123/openstack-caracal-dc-dc/opentofu/terraform.tfstate.pre-*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 jumphost location '~/admin-openrc' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it + [ok] E0 18 remote location(s) SKIPPED -- rerun with --remote to include the headend shadow stores (SEC-022) and the region secrets dir (SEC-020) + [ok] E1 91 expected artifact(s) NOT JUDGED -- their role (headend/-, headend/vr1-dc0, headend/vr1-dc1, headend/vr1-office1, jumphost/-, jumphost/vr1-dc0, jumphost/vr1-dc1, jumphost/vr1-office1, netbox/vr1-office1, rack/vr1-dc0, rack/vr1-dc1, region/vr1-dc0, region/vr1-dc1) has at least one location that could not be probed, so absence cannot be asserted over it + [ok] E1/E3 existence: every expected artifact present and nothing undeclared, across 0 fully-probed role(s) + [ok] tier 3 (VALIDITY) NOT RUN -- pass --tier3 (with --tier2) to compare cross-copy sha256 provenance + [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc1 EXPECTED-BUT-ABSENT: 'maas-juju-api-key.txt' (id dc1-juju-apikey, SEC-028) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc1 EXPECTED-BUT-ABSENT: 'maas-juju-user-password' (id dc1-juju-user, SEC-028) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc1 EXPECTED-BUT-ABSENT: 'maas-region-admin-password' (id dc1-region-admin, SEC-027) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc1 EXPECTED-BUT-ABSENT: 'maas-region-api-key.txt' (id dc1-region-apikey, SEC-027) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S2 vr1-dc1 EXPECTED-BUT-ABSENT: 'maas-region-db-password' (id dc1-region-db, SEC-027) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=id_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=maas-virsh_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S5 ASYMMETRY: vr1-dc1 declares id=dcN-maas-power-key file=id_dcN_power on headend (custody=off-manifest-known) with no counterpart in vr1-dc0 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types (human via gui; service via api, cli-profile) -- ruling 5 requires one identity to serve one principal type + [FAIL] E4 UNCHECKABLE: 2 row(s) have no declared location for their (role, site) and can never be verified -- add a location row or correct the matrix: capi-mgmt-kubeconfig 'config' (cloud/-); rbd-mirror-peer-token 'rbd-mirror-bootstrap-token' (unit/-) + +FAIL: creds-matrix tier 1 -- 121 row(s), 19 check group(s) clean, 11 finding(s) +================ P7: Octavia amphora PKI ================ +=== octavia-pki verify: vr1-dc0 === + ok host: 'voffice1' is the declared headend, so its filesystem is the right one to measure + (expect CA label 'VR1 DC0'; provider VIP v4=10.12.4.57 v6=2602:f3e2:f02:11::57) + ok A1 workspace present: ~/octavia-pki/vr1-dc0 + ok A2 all 10 expected artifacts present + ok A3 private issuing-ca/passphrase.txt is 0600 + ok A3 private issuing-ca/issuing-ca.key.enc is 0600 + ok A3 private controller-ca/passphrase.txt is 0600 + ok A3 private controller-ca/controller-ca.key.enc is 0600 + ok A3 private controller/controller.key is 0600 + ok A3 private controller/controller.bundle.pem is 0600 + ok A3 cert issuing-ca/issuing-ca.cert.pem is 600 -- not group/world writable + ok A3 cert controller-ca/controller-ca.cert.pem is 600 -- not group/world writable + ok A3 cert controller-ca/controller-ca.cert.srl is 600 -- not group/world writable + ok A3 cert controller/controller.cert.pem is 600 -- not group/world writable + ok A4 issuing CA subject names this DC: contains 'VR1 DC0 Omega Cloud Octavia Issuing CA' + ok A5 controller CA subject names this DC: contains 'VR1 DC0 Omega Cloud Octavia Controller CA' + ok A6 issuing CA self-signature verifies + ok A7 controller CA self-signature verifies + ok A8 controller cert verifies against the CONTROLLER CA + ok A8 controller cert correctly does NOT verify against the issuing CA + ok A9 SAN carries 2 DNS names + ok A9 SAN carries this DC's provider v4 VIP (10.12.4.57) + ok A9 SAN carries this DC's provider v6 VIP (2602:f3e2:f02:11::57) + ok A12 DNS SANs are INERT -- os-public-hostname is set in no deploy artifact (B5 IP-only), so nothing resolves them; this assertion ARMS ITSELF when D-106 sets it + ok A12 DNS SANs are all in this DC's expected zone 'omega.dc0.vr1.cloud.neumatrix.local' + ok A13 controller cert CN is 'octavia-controller.omega.dc0.vr1.cloud.neumatrix.local' + ok A14 controller.key and controller.cert.pem carry the SAME public key (they are a pair) + ok A14 bundle carries exactly one CERTIFICATE block and one PRIVATE KEY block + ok A14 the bundle's CERTIFICATE block is byte-identical to controller.cert.pem + ok A14 the bundle's PRIVATE KEY block is byte-identical to controller.key + ok A15 controller cert carries keyUsage (critical: digitalSignature, keyEncipherment) and EKU (clientAuth, serverAuth) + ok A16 controller cert is valid and not expiring within 30 days + ok A10 overlay is 0600 + ok A10 overlay declares 5 lb-mgmt-* keys + ok A10 overlay is ASCII clean + ok A10 overlay is gitignored (F4) + ok A17 the overlay's controller cert and CA values decode byte-identically to this workspace's bundle and controller CA + ok A11 all 3 compared artifacts differ from vr1-dc1 -- per-DC independence holds + +octavia-pki verify (vr1-dc0): PASS -- 37 assertion(s), 0 failed + [ok] P7 octavia PKI verified for vr1-dc0, and its DNS SANs are in this DC's own zone +================ P8: substrate drift (outer tofu root) ================ + [warn] no outer-root state here -- substrate drift NOT evaluated (D-128: the + outer root lives on vcloud). Not a failure; not a pass either. +================ P6: stage-2 reminders (NOT run here) ================ + - after 'juju add-model': bash scripts/juju-spaces-check.sh + - with sudo: bash scripts/osd-blank-check.sh + - phase-01 Step 1.2: juju deploy --dry-run (plan: 50 apps / 97 relations) + +PREFLIGHT: FAIL (DC=vr1-dc0) -- do NOT deploy + +--- capture context (appended 2026-08-01T10:16:54Z) --- +Run ON voffice1 (D-128 Plane-2 host) at HEAD db15666, DC=vr1-dc0 MAAS_PROFILE=vr1-dc0-region. +EXIT 1. Exactly 11 [FAIL] lines, ALL of them the P5 set ruled-accepted 2026-07-30 (six) +and 2026-07-31 (five) -- compared BY IDENTITY, not by count. ZERO new P5 findings. +P3 PASS; P4 PASS; P7 PASS 37/0; P8 WARN (not the substrate host -- correct after the +db15666 guard fix; its subject measured clean on vcloud: tofu exit 0, 0 pending, 0 drift).