Newer
Older
openstack-caracal-dc-dc / docs / audit / stage5-unmeasured-register-20260727.md

UNMEASURED REGISTER -- Stage-5 grounding audit (2026-07-27)

Why this file exists. The operator asked whether the committee actually measured live state or took shortcuts. The honest answer is that it did some of both, and this register is the accounting: every UNMEASURED item the seven lenses declared, plus what a follow-up sweep closed, plus what remains genuinely open and why.

The founding principle it enforces, from the skill: "could not look" is never "nothing there" -- and its corollary, which this audit had to learn the hard way: a declared UNMEASURED is only honest if it was genuinely unmeasurable. Declaring an item unmeasurable that another lens measured in the same session is not scope discipline, it is a gap.


1. What went wrong, precisely

Two distinct failures, and they belong to different actors.

(a) The apex was never polled, by anyone. Lens 2's brief was as-built-vs-live, and NetBox is the IPAM apex per the skill's own routing table. It measured MAAS and the nodes and declared the apex out of scope. It did so HONESTLY -- its UNMEASURED item 5 reads: "IPv6 state at the NetBox apex ... I measured MAAS and the nodes only. D-101's literals may exist in NetBox and simply not be carved into MAAS -- L2-3's claim is scoped to MAAS + node reality and does not assert the apex is empty." That is the contract working: no overclaim. But the apex is a key system and it had permission and time to reach it. Under-scoped, not dishonest.

(b) The synthesis then used a two-day-old dump instead of polling live -- and that was mine, not an agent's. On being asked, I read netbox/draft/vr1-office1-current-20260725.json and treated it as apex state. Worse, I had already put a question to the operator (R2a, "which literals") built on D-101's stale "Remaining open item" prose, WITHOUT acting on lens 2's explicit warning that the literals might already exist. Trusting stale decision prose over an available measurement is exactly the failure this audit was convened to find.

The sharpest single instance of the general problem: lens 6 declared juju restore-backup unverifiable because "no Juju client exists on this host" -- while juju 3.6.27 was installed on voffice1 and lens 5 ran juju help against it successfully in the same session. One lens called impossible what another was measuring. That is not a permissions limit; that is a lens not looking.


2. CLOSED by the follow-up sweep (2026-07-27, after the operator's challenge)

# Item Lens that deferred it Measured result
U1 NetBox apex, LIVE L2 (#5) netbox/office1-record-dump.py run against the live apex: 139 prefixes, 103 IPv6, zero added / zero removed vs the 07-25 dump. The dump was accurate -- but that was luck of timing, not method. dc1's key prefixes all PRESENT: fd50:840e:74e2:320::/64, :330::/64, 2602:f3e2:f03:10::/64, f03:11::/64
U2 juju restore-backup exists? L6 (#1) IT DOES NOT. juju help restore-backup -> ERROR juju: "restore-backup" is not a juju command. See "juju --help". Did you mean: create-backup. This promotes L6-14 from a flagged RISK to a CONFIRMED DEFECT -- Stage 6 Step 9's D-104 restore drill and its DoD bullet are built on a command that does not exist. DOCFIX-204 class: unsatisfiable as written
U3 juju bind subcommand + <endpoint>=<space> syntax L6 (#2) EXISTS as documented: Usage: juju bind [options] <application> [<default-space>] [<endpoint-name>=<space> ...]. Lens 6 was right to EXCLUDE it from L6-7 rather than assert it wrong -- a correctly-handled uncertainty
U4 Is curl on the racks? L4 (#1) PRESENT on both: /usr/bin/curl on dc0 and dc1. L4-13's "could not look" hole is therefore theoretical, not live -- the artifact checkers' behavioural probe does run today
U5 dc0 compute provider-public MACs L3 (#4) vr1-dc0-compute-01 -> 52:54:00:1b:19:e6, vr1-dc0-compute-02 -> 52:54:00:18:ab:b4. Matches scripts/lib-hosts.sh:121-122 exactly, and CONFIRMS L3-8: bundle.yaml:458-464 carries dc1's 52:54:01:d1:* MACs in the file documented as dc0's source of truth
U6 Does the maas CLI work from the jumphost? L2 (#6) The maas binary is ABSENT on vcloud. So preflight P4's FAIL: MAAS unreachable: 'maas admin subnets read' failed is a MISDIAGNOSIS -- it is not a network or region problem, it is a missing client, the same class as the absent openstack client (S-1) and the absent juju (L4-2). The message actively misleads
U7 Do the pinned charm channels resolve on Charmhub? L3 (#3) -- "no upstream fetch performed", despite the session having Charmhub access granted They resolve. Run via juju info on voffice1 (the same tool the P3 gate uses): designate/designate-bind/cinder-backup all list 2024.1; hacluster lists 2.4; ceph-osd/ceph-mon/ceph-rbd-mirror/ceph-radosgw all list squid. This also proves L4-2's root cause and its fix: P3's 33 warns are purely the missing juju binary on vcloud. Run from the Plane-2 host, the lookups work
U8 dc1 edge egress OPEN right now? L5 (#2) MEASURED OPEN (earlier this session): streams.canonical.com/juju/tools/ -> 200, api.snapcraft.io reachable, archive.ubuntu.com -> 200, ping 1.1.1.1 0% loss, default route via 10.12.64.1. Probed with --noproxy so an apt-cacher hit could not fake it
U9 Stage-5 execution host / juju location L7 (#2) CONFIRMED: juju at /snap/bin/juju on voffice1, ABSENT on vcloud; ~/.local/share/juju/ holds clouds.yaml + credentials.yaml
U10 The three tofu plan captures L2 (#2) DONE: all three roots ZERO DIFF, with the inner pair's validity established first
U11 site-baseleg.sh check office1 L2 (#4) DONE: PASS
U12 preflight live exit code + red set L1 (#1) DONE: exit 1, red set exactly the known one. L1's denial was a missing Bash(timeout *) rule -- a failed-to-MATCH, not a classifier override
U13 creds-matrix --tier2 --remote --privileged L4 (#4) DONE: exit 1, same 7 findings, all three root-owned dirs read via sudo -n escalation, ZERO E3
U14 Canonical piped rack-checker invocations L4 (#1), audit session DONE by L2: dc-mirror dc0 PASS, dc-cache-proxy dc1 PASS, dc-rack-net PASS both

3. STILL OPEN -- and the honest reason for each

Genuinely unmeasurable today (a live artefact does not exist yet):

  • Juju's real overlay-merge behaviour for the machines: block, and any juju deploy --dry-run. Requires a bootstrapped controller and a model; juju controllers -> ERROR No controllers registered. provider-bundle-check's merge is a MIRROR of juju's documented behaviour, not juju itself.
  • Whether Juju applies an application-level tags= constraint to units placed by explicit to: machine ids (L5-14). No model to test against.
  • cloud-assert.sh end-to-end. No OpenStack model exists.
  • Anything node-side. All 18 nodes are powered off in Ready under the READY-handoff ruling -- G17's territory by construction.
  • Whether MAAS would in fact allocate out of the unreserved .50-.99 band. Behavioural, observable only during a real deploy. What IS measured is MAAS's own declaration that the band is unreserved on 12/12 subnets.
  • Whether ceph-osd accepts a directory/partition for osd-devices. Attempted this sweep: juju info ceph-osd does not expose config keys, so it needs a charm-config fetch. Moot for the decision -- R1 ruled option (a), a real second disk -- but recorded as still-unmeasured rather than quietly dropped.

Measurable, NOT yet done -- these are real remaining gaps, not excuses:

  • IPv6 state on the two DC edges. The OPNsense API GET path is available and permitted. Not attempted. Lower value than the planes (which ARE measured empty in MAAS), but it is part of "full state of the configuration" and should be closed.
  • repo-lint and the gauntlet ON voffice1 (L5 P2). Deliberately deferred: that clone is 105 commits behind, so running them there today measures a stale tree. Owed immediately AFTER the clone is advanced, which is precondition 0.1.
  • Reboot-durability of voffice1's dc0/dc1 transit addressing (L6 #4). A netplan read. Not attempted.
  • Provenance of RETROFIT_WAIT=30m and any other nested-virt-calibrated constant (L7 #1). Not attempted.

4. What this changes about the audit's conclusions

Nothing in the readiness verdict is reversed. Two things are STRENGTHENED and one question was withdrawn:

  • L6-14 upgraded from RISK to CONFIRMED DEFECT (U2). juju restore-backup does not exist; Stage 6's restore drill and its DoD bullet are unsatisfiable as written.
  • L3-8 CONFIRMED by measurement (U5): the base bundle really does carry dc1's compute MACs while documenting itself as dc0's source of truth.
  • A new finding: preflight's "MAAS unreachable" is a misdiagnosis (U6). Together with the absent juju (L4-2) and absent openstack (S-1), the pattern is that three separate preflight/deploy failures on this jumphost are all "the client is not installed", and each is reported as something else. That is its own DOCFIX and its own lesson: a tool-absence must not be reported as a target-unreachable.
  • R2a WITHDRAWN as never open -- the literals are assigned under D-111 and the apex carries them; the real work is propagation into lib-net.sh and MAAS.

5. The standing lesson for the next committee

Add to the subagent contract, alongside "report a denied call as UNMEASURED": before declaring an item unmeasurable, verify the constraint you are asserting. Lens 6 asserted "no Juju client exists on this host" without checking the host where D-128 says the client lives. The UNMEASURED escape hatch is load-bearing for honesty, and precisely for that reason it must not become a way to avoid looking.