PREFLIGHT TARGET: DC=vr1-dc0 (override: DC=vr1-dc1 bash scripts/preflight.sh) Every gate below is run against THIS DC; the verdict line repeats it. ================ P1: repo lint ================ [WARN] L1 docs/design-decisions.md: 239 non-ASCII byte(s) (legacy D-001..018 carve-out; NEW entries must be ASCII) WARN: repo lint (0 fail, 1 warn, 627 files scanned) ================ P2: bundle invariants ================ validating the MERGED vr1-dc0 deploy input: bundle.yaml --overlay overlays/vr1-dc0-vips.yaml --overlay overlays/vr1-dc0-machines.yaml --overlay overlays/vr1-dc0-octavia-pki.yaml --dc vr1-dc0 [ok] 11 charms bind public->provider-public; none on provider-vip [ok] 13 clustered VIP(s) are provider/admin/internal, octet 50-99 (13 dual-family) [ok] ovn-chassis bridge-interface-mappings: 2 well-formed MAC(s) (role-sep; VR0 set N/A) [ok] 108 relations well-formed (explicit endpoints, all apps exist) [ok] mysql-innodb-cluster num_units=3 (D-062) [ok] 12 hacluster principal(s) all carry a VIP (R11) [ok] 12 hacluster subordinate(s) declare cluster_count == principal num_units [ok] keystone policyd-override wired in-bundle; zip content matches source (DOCFIX-071) [ok] machines block: all 9 machine(s) tagged openstack-vr1-dc0, matching --dc vr1-dc0 [ok] placement: role-separated (3 control/2 compute/4 storage); anti-affinity + role placement + counts OK PASS: Pattern A / D-052-D-053 bundle invariants (bundle.yaml) ================ P3: channel assert (charmhub) ================ [ok] barbican 2024.1/stable (barbican) [ok] barbican-vault 2024.1/stable (barbican-vault) [ok] ceph-mon squid/stable (ceph-mon) [ok] ceph-osd squid/stable (ceph-osd) [ok] ceph-radosgw squid/stable (ceph-radosgw) [ok] ceph-rbd-mirror squid/stable (ceph-rbd-mirror) [ok] cinder 2024.1/stable (cinder) [ok] cinder-backup 2024.1/stable (cinder-backup) [ok] cinder-ceph 2024.1/stable (cinder-ceph) [ok] designate 2024.1/stable (designate) [ok] designate-bind 2024.1/stable (designate-bind) [ok] glance 2024.1/stable (glance) [ok] glance-simplestreams-sync 2024.1/stable (glance-simplestreams-sync) [ok] hacluster 2.4/stable (keystone-hacluster, glance-hacluster, neutron-api-hacluster, nova-cloud-controller-hacluster, placement-hacluster, openstack-dashboard-hacluster, cinder-hacluster, octavia-hacluster, barbican-hacluster, magnum-hacluster, ceph-radosgw-hacluster, designate-hacluster) [ok] keystone 2024.1/stable (keystone) [ok] magnum 2024.1/stable (magnum) [ok] magnum-dashboard 2024.1/stable (magnum-dashboard) [ok] memcached latest/stable (memcached) [ok] mysql-innodb-cluster 8.0/stable (mysql-innodb-cluster) [ok] mysql-router 8.0/stable (vault-mysql-router, keystone-mysql-router, glance-mysql-router, ncc-mysql-router, placement-mysql-router, neutron-api-mysql-router, cinder-mysql-router, dashboard-mysql-router, octavia-mysql-router, barbican-mysql-router, magnum-mysql-router, designate-mysql-router) [ok] neutron-api 2024.1/stable (neutron-api) [ok] neutron-api-plugin-ovn 2024.1/stable (neutron-api-plugin-ovn) [ok] nova-cloud-controller 2024.1/stable (nova-cloud-controller) [ok] nova-compute 2024.1/stable (nova-compute) [ok] octavia 2024.1/stable (octavia) [ok] octavia-dashboard 2024.1/stable (octavia-dashboard) [ok] octavia-diskimage-retrofit 2024.1/stable (octavia-diskimage-retrofit) [ok] openstack-dashboard 2024.1/stable (openstack-dashboard) [ok] ovn-central 24.03/stable (ovn-central) [ok] ovn-chassis 24.03/stable (ovn-chassis, ovn-chassis-octavia) [ok] placement 2024.1/stable (placement) [ok] rabbitmq-server 3.9/stable (rabbitmq-server) [ok] vault 1.8/stable (vault) PASS: channel assert (33 pins, 0 fail, 0 warn) ================ P4: live pre-flight (MAAS/overlay/nodes) ================ === DC selection === PASS: gating DC=vr1-dc0 (planes 10.12.4.0/22 .. 10.12.36.0/22; 10 node(s)) === Repo (informational) === NOTE: REPO=/home/jessea123/openstack-caracal-dc-dc NOTE: HEAD: c58bf95 GA-R4 session close: F9 closed live, the reissue tool, P7, lib-identity NOTE: working tree clean === CHECK 0: per-DC octavia-pki overlay (no key material printed) === PASS: overlay present with 5 lb-mgmt-* keys PASS: overlay ASCII clean === CHECK 1: bundle VIPs -- v4 triple or R2 dual-family sextet, .50-.99 (provider/admin/internal) === PASS: vip: line count = 13 (from overlays/vr1-dc0-vips.yaml) PASS: aligned VIPs OK=13 bad=0 (DC=vr1-dc0 bands 10.12.4/10.12.8/10.12.12) === MAAS reachability gate (read-only) === PASS: MAAS reachable (profile=admin) === CHECK 3: six planes resolved BY CIDR (id/vid/gw/dns) === provider-public 10.12.4.0/22 id=7 vid=0 gw=10.12.4.1 dns=[] metal-admin 10.12.8.0/22 id=6 vid=0 gw=none dns=["10.12.8.3"] metal-internal 10.12.12.0/22 id=12 vid=0 gw=none dns=[] data-tenant 10.12.16.0/22 id=13 vid=0 gw=none dns=[] storage 10.12.32.0/22 id=14 vid=0 gw=none dns=[] replication 10.12.36.0/22 id=15 vid=0 gw=none dns=[] PASS: all six planes present (by CIDR) PASS: metal-internal is UNTAGGED (vid 0) -- D-133 flat carve NOTE: stale-NAME check is juju-side (run scripts/juju-spaces-check.sh after add-model) === CHECK 2: data/storage NIC links BY CIDR (DC=vr1-dc0 role nodes; octet per D-134 band) === == vr1-dc0-control-01 (nhg3nf, octet .100) == enp3s0 -> 10.12.12.0/22 10.12.12.100 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.100 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.100 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.100 type=physical == vr1-dc0-control-02 (ssyexn, octet .101) == enp3s0 -> 10.12.12.0/22 10.12.12.101 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.101 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.101 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.101 type=physical == vr1-dc0-control-03 (sk8c4d, octet .102) == enp3s0 -> 10.12.12.0/22 10.12.12.102 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.102 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.102 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.102 type=physical == vr1-dc0-compute-01 (dbcq8h, octet .120) == enp3s0 -> 10.12.12.0/22 10.12.12.120 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.120 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.120 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.120 type=physical == vr1-dc0-compute-02 (sgwfnb, octet .121) == enp3s0 -> 10.12.12.0/22 10.12.12.121 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.121 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.121 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.121 type=physical == vr1-dc0-storage-01 (kghggm, octet .150) == enp3s0 -> 10.12.12.0/22 10.12.12.150 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.150 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.150 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.150 type=physical == vr1-dc0-storage-02 (8mtpxq, octet .151) == enp3s0 -> 10.12.12.0/22 10.12.12.151 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.151 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.151 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.151 type=physical == vr1-dc0-storage-03 (sn6qda, octet .152) == enp3s0 -> 10.12.12.0/22 10.12.12.152 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.152 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.152 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.152 type=physical == vr1-dc0-storage-04 (xqqwdq, octet .153) == enp3s0 -> 10.12.12.0/22 10.12.12.153 type=physical enp4s0 -> 10.12.16.0/22 10.12.16.153 type=physical enp5s0 -> 10.12.32.0/22 10.12.32.153 type=physical enp6s0 -> 10.12.36.0/22 10.12.36.153 type=physical NOTE: vr1-dc0-juju-01 (7n87bt, octet .5) is D-134 utility-band infrastructure, not an OpenStack role node -- data-plane carve not asserted === CHECK 4: DC=vr1-dc0 OpenStack role nodes -- status / power === vr1-dc0-control-01 -> wired-thrush Ready power=off PASS: vr1-dc0-control-01 Ready vr1-dc0-control-02 -> ace-robin Ready power=off PASS: vr1-dc0-control-02 Ready vr1-dc0-control-03 -> real-filly Ready power=off PASS: vr1-dc0-control-03 Ready vr1-dc0-compute-01 -> keen-dove Ready power=off PASS: vr1-dc0-compute-01 Ready vr1-dc0-compute-02 -> superb-piglet Ready power=off PASS: vr1-dc0-compute-02 Ready vr1-dc0-storage-01 -> first-oryx Ready power=off PASS: vr1-dc0-storage-01 Ready vr1-dc0-storage-02 -> wise-stud Ready power=off PASS: vr1-dc0-storage-02 Ready vr1-dc0-storage-03 -> alert-cub Ready power=off PASS: vr1-dc0-storage-03 Ready vr1-dc0-storage-04 -> moral-salmon Ready power=off PASS: vr1-dc0-storage-04 Ready Summary: 0 fatal, 0 warning ================ P5: credential matrix (D-137 tier 1 + tier 2 local) ================ === creds-matrix: tier 1 (STATIC) === === creds-matrix: tier 2 (EXISTENCE) === (host: voffice1) [ok] S1 schema: 101 rows, all enums valid, site-keys region-qualified, no duplicate (id,site,host-role,filename) [ok] S3 render: 3 source field(s) SKIPPED -- rendering them needs the declared path from creds-manifests/vm-secret-locations (ruling 3); the list now EXISTS but the source-field derivation is not wired [ok] S3 render drift: rendered row fields (mode, source) match checked-in; header prose and non-jumphost rows are OUT OF SCOPE of this compare [ok] S4 mint-ref: every script:/runbook: reference resolves to a real location [ok] S4 provenance debt: 30 row(s) are mint-ref=operator-terminal -- NOT reproducible from the repo (research FINDING 1). Admitted by design; converting them is remediation, not a checker fix. [ok] S7 notes: 37 note key(s) referenced, all resolve, none orphaned [ok] E0 jumphost location '~/vr1-office1-creds/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '~/vr1-dc0-creds/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '~/vr1-dc1-creds/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '~/vault-init/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '~/tenant-*/*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '/home/jessea123/openstack-caracal-dc-dc/opentofu/terraform.tfstate' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '/home/jessea123/openstack-caracal-dc-dc/opentofu/terraform.tfstate.backup' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '/home/jessea123/openstack-caracal-dc-dc/opentofu/terraform.tfstate.pre-*' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 jumphost location '~/admin-openrc' NOT PROBED -- role 'jumphost' lives on 'vcloud' and this run is on 'voffice1'; probing it here would measure a different host's filesystem, so absence is not asserted over it [ok] E0 8 remote location(s) SKIPPED -- rerun with --remote to include the headend shadow stores (SEC-022) and the region secrets dir (SEC-020) [ok] E1 71 expected artifact(s) NOT JUDGED -- their role (headend/-, headend/vr1-dc0, headend/vr1-dc1, headend/vr1-office1, jumphost/-, jumphost/vr1-dc0, jumphost/vr1-dc1, jumphost/vr1-office1, netbox/vr1-office1) has at least one location that could not be probed, so absence cannot be asserted over it [ok] E1/E3 existence: every expected artifact present and nothing undeclared, across 0 fully-probed role(s) [ok] tier 3 (VALIDITY) NOT RUN -- pass --tier3 (with --tier2) to compare cross-copy sha256 provenance [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=id_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=maas-virsh_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape [FAIL] S5 ASYMMETRY: vr1-dc1 declares id=dcN-maas-power-key file=id_dcN_power on headend (custody=off-manifest-known) with no counterpart in vr1-dc0 -- a per-DC credential must exist at BOTH DCs in the same shape [FAIL] S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types (human via gui; service via api, cli-profile) -- ruling 5 requires one identity to serve one principal type [FAIL] E4 UNCHECKABLE: 2 row(s) have no declared location for their (role, site) and can never be verified -- add a location row or correct the matrix: capi-mgmt-kubeconfig 'config' (cloud/-); rbd-mirror-peer-token 'rbd-mirror-bootstrap-token' (unit/-) FAIL: creds-matrix tier 1 -- 101 row(s), 19 check group(s) clean, 6 finding(s) ================ P7: Octavia amphora PKI ================ === octavia-pki verify: vr1-dc0 === ok host: 'voffice1' is the declared headend, so its filesystem is the right one to measure (expect CA label 'VR1 DC0'; provider VIP v4=10.12.4.57 v6=2602:f3e2:f02:11::57) ok A1 workspace present: ~/octavia-pki/vr1-dc0 ok A2 all 10 expected artifacts present ok A3 private issuing-ca/passphrase.txt is 0600 ok A3 private issuing-ca/issuing-ca.key.enc is 0600 ok A3 private controller-ca/passphrase.txt is 0600 ok A3 private controller-ca/controller-ca.key.enc is 0600 ok A3 private controller/controller.key is 0600 ok A3 private controller/controller.bundle.pem is 0600 ok A3 cert issuing-ca/issuing-ca.cert.pem is 600 -- not group/world writable ok A3 cert controller-ca/controller-ca.cert.pem is 600 -- not group/world writable ok A3 cert controller-ca/controller-ca.cert.srl is 600 -- not group/world writable ok A3 cert controller/controller.cert.pem is 600 -- not group/world writable ok A4 issuing CA subject names this DC: contains 'VR1 DC0 Omega Cloud Octavia Issuing CA' ok A5 controller CA subject names this DC: contains 'VR1 DC0 Omega Cloud Octavia Controller CA' ok A6 issuing CA self-signature verifies ok A7 controller CA self-signature verifies ok A8 controller cert verifies against the CONTROLLER CA ok A8 controller cert correctly does NOT verify against the issuing CA ok A9 SAN carries 2 DNS names ok A9 SAN carries this DC's provider v4 VIP (10.12.4.57) ok A9 SAN carries this DC's provider v6 VIP (2602:f3e2:f02:11::57) ok A12 DNS SANs are INERT -- os-public-hostname is set in no deploy artifact (B5 IP-only), so nothing resolves them; this assertion ARMS ITSELF when D-106 sets it ok A12 DNS SANs are all in this DC's expected zone 'omega.dc0.vr1.cloud.neumatrix.local' ok A13 controller cert CN is 'octavia-controller.omega.dc0.vr1.cloud.neumatrix.local' ok A14 controller.key and controller.cert.pem carry the SAME public key (they are a pair) ok A14 bundle carries exactly one CERTIFICATE block and one PRIVATE KEY block ok A14 the bundle's CERTIFICATE block is byte-identical to controller.cert.pem ok A14 the bundle's PRIVATE KEY block is byte-identical to controller.key ok A15 controller cert carries keyUsage (critical: digitalSignature, keyEncipherment) and EKU (clientAuth, serverAuth) ok A16 controller cert is valid and not expiring within 30 days ok A10 overlay is 0600 ok A10 overlay declares 5 lb-mgmt-* keys ok A10 overlay is ASCII clean ok A10 overlay is gitignored (F4) ok A17 the overlay's controller cert and CA values decode byte-identically to this workspace's bundle and controller CA ok A11 all 3 compared artifacts differ from vr1-dc1 -- per-DC independence holds octavia-pki verify (vr1-dc0): PASS -- 37 assertion(s), 0 failed [ok] P7 octavia PKI verified for vr1-dc0, and its DNS SANs are in this DC's own zone ================ P6: stage-2 reminders (NOT run here) ================ - after 'juju add-model': bash scripts/juju-spaces-check.sh - with sudo: bash scripts/osd-blank-check.sh - phase-01 Step 1.2: juju deploy --dry-run (plan: 50 apps / 97 relations) PREFLIGHT: FAIL (DC=vr1-dc0) -- do NOT deploy