diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 46d50e1..2a077cf 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2729,6 +2729,44 @@ measured plan is **56 apps / 108 relations**; `docs/dc-dc-buildout-design.md:12` records the repo was seeded at `8813efc` specifically so `bundle.yaml` + `phase-01..08` could be reused. **Provider capabilities must be READ (`references/opentofu-provider-docs.md`), never recalled.** + **>>> dc0 EDGE REBUILT 2026-08-02 AND EGRESS IS FULLY RESTORED: `dc-egress-check dc0` PASS + 8/8, exit 0. <<<** Capture `docs/audit/dc0-edge-rebuild-20260802.txt` (683 lines). Executed by + a gated agent. `tofu -replace` on both edge resources, **asserted on `tofu show -json` rather + than prose: exactly 2 non-no-op changes, both `delete,create`, both naming `vr1_dc0_opnsense`, + 0 not naming it -- and the POSITIVE half asserted too, because a plan replacing only the + DOMAIN would have reattached the corrupt qcow2 and passed a "nothing extra" check.** State + backed up first; the ARTIFACT verified rather than the log (a 2.2 GB volume "created in 4s" is + worth distrusting); convergence re-plan `No changes`. Bootstrap used a driver merging both + D-112(c) legs and **deliberately did NOT replay `~/d112c-bootstrap.php.b64`** -- confirmed + against the artifact that it lacks `require_once("util.inc")` and would have died in + `write_config()`. API key minted, `CORE_ABI 26.7` (third D-113(a2)-on-26.7 datapoint), + registered as **SEC-032**. Addressed WAN FIRST, **proved egress from the edge itself before + touching the LAN**, then LAN; and **`pfctl -s nat` explicitly verified to show real outbound + NAT for the new LAN network** -- the script only WARNs there, and that is precisely the + "addressed perfectly, forwards nothing" defect dc1 is sitting in right now. + **>>> MY AGENT BRIEF CARRIED A WRONG PATH, AND IT WAS THE ONE UNRECOVERABLE FAILURE MODE. <<<** + I cited the base image as `/var/lib/libvirt/vr1-dc0-inner/opnsense-26.7-nano.qcow2` -- that is + the MODULE DEFAULT. `d124-inner.auto.tfvars` overrides it to + `/home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2` **on voffice1**, because the provider + uploads from its own filesystem. **The apply DESTROYS the volume before recreating it, so a + wrong base path would have left no disk and no rollback point** (both edge disks have an empty + `backingStore`). The agent verified the real path before mutating. **This is the same + guess-instead-of-look failure the operator called out earlier the same day, committed by me + again, in a brief whose entire purpose was to stop the agent doing it.** + **THREE FINDINGS THAT NEED ATTENTION BEFORE STAGE 5.** **(1) The rebuilt edge runs dnsmasq + bound `udp4 *:67` on `vtnet0` with a STALE `dhcp-range 192.168.1.100-.199`, and `vtnet0` is now + `10.12.4.0/22` -- the segment carrying the juju controller `.5` and the MAAS region `.6`.** + MEASURED: running, bound, stale range, zero dhcpd/Kea blocks in config.xml. REASONED NOT + MEASURED: dnsmasq matches ranges to the interface subnet, so it is probably inert. NOT + ESTABLISHED: whether the destroyed edge was the same -- it cannot be checked now, so this is + NOT a claim that prior state was restored. **(2) SEC-021(a) IS ANSWERED: the off-jumphost copy + EXISTS** -- edge private key + API credential on the rack since 2026-07-20, undisclosed until + now, the dc0 analog of SEC-015. Both left in place (the key is structurally required; the + scripts have no ProxyJump path). Its `S2` finding stays red for a NEW reason -- the credential + exists, `vr1-dc0.manifest` just does not declare it; one `creds-matrix.py --render`. + **(3) A REBUILT EDGE INVALIDATES THE RACK'S `known_hosts`,** which would break every + `BatchMode=yes` script. No runbook says so, and neither first-build could have caught it -- + fold material. **>>> dc1 EDGE ASSESSED (read-only agent, 2026-08-02): IT FORWARDS BUT DOES NOT TRANSLATE, ITS CONFIG IS INTACT, AND THE VERDICT IS REPAIR NOT REBUILD. <<<** Capture `docs/audit/dc1-edge-assessment-20260802.txt` (643 lines). **DECISIVE MEASUREMENT:** diff --git a/docs/audit/dc0-edge-rebuild-20260802.txt b/docs/audit/dc0-edge-rebuild-20260802.txt new file mode 100644 index 0000000..f22d9bb --- /dev/null +++ b/docs/audit/dc0-edge-rebuild-20260802.txt @@ -0,0 +1,683 @@ +dc0-edge-rebuild-20260802.txt +============================= +REBUILD of the vr1-dc0 OPNsense edge, destroyed by UFS soft-update damage after +the 2026-08-01 05:48 hard power cut of its containment VM (full diagnosis: +docs/audit/dc0-edge-egress-outage-20260802.txt -- not re-derived here). + +Definition of done, set by the operator: `ssh vr1-dc0-rack 'bash -s -- check dc0' +< scripts/dc-egress-check.sh` exits 0. + +Written incrementally as each step completed. Every value below was LOOKED UP in a +script constant, a tfvars file, or a live object -- none was inferred. Where a value +in the task brief disagreed with the measured one, the measured one is used and the +divergence is called out. + +================================================================================ +SECTION 0 -- READ-ONLY BASELINE (before any mutation) +================================================================================ + +REPO / CLONE IDENTITY + jumphost (vcloud) HEAD : 95a6f5f branch dc-dc-stage5-preconditions + voffice1 clone HEAD : 22af620 (git status clean) + The two clones are at DIFFERENT commits. That matters only if it changes the + tofu root, so it was MEASURED rather than assumed -- sha256 of all seven files + the inner root consumes, both hosts: + + c81aa41b093af531f81a97254116f98c524b07a44f2e02240c03876bbbce9ab3 vr1-dc0-substrate/main.tf + f4020e606c1fc4d2a2dbf5430d0d38c518962ca3b4bd1b549de223a7e51250f2 vr1-dc0-substrate/variables.tf + 17de7c023fa1353f8f14eebf29b7bc30dd11f0bcfec1112d8b41c83fd1c3c081 vr1-dc0-substrate/versions.tf + cd117906d8f4baf526240e20eda53931746aaff2d2df8fa40bc7f75df30e7229 modules/opnsense-edge/main.tf + 7e38c5b88e6f5c83efd25f7487af87857ca37f52b17831cfb0e9e8a12f733712 modules/opnsense-edge/outputs.tf + b027fe2566e1e4aca89ddcade89051ea01b7b7de7470f9a2c61a2fe0cfb0e7f3 modules/opnsense-edge/variables.tf + f11b24565b04f42959706ad305416eced71310386ff78e029a882ac38382ec2d modules/opnsense-edge/versions.tf + + IDENTICAL on both hosts. The clone divergence therefore cannot affect this + root, and NO `git pull` was run on voffice1 (the operator is committing on this + branch concurrently; pulling would drag unrelated commits in mid-apply). + +INSTRUMENT TRAP CAUGHT AT THE FIRST PROBE -- recorded because it would have +produced a confident wrong negative: + `ssh vr1-dc0-rack 'virsh list --all'` -> EMPTY TABLE, and + `virsh dominfo vr1-dc0-opnsense` -> "error: failed to get domain" + That reads exactly like "the domain is gone". It is not. + `virsh uri` -> qemu:///session + The non-root default URI is the SESSION daemon, which owns nothing. With + `-c qemu:///system` all 12 domains are listed and the edge is Id 1 running. + EVERY virsh call in this capture carries `-c qemu:///system` for that reason. + +LIVE STATE, dc0 rack (vvr1-dc0), `virsh -c qemu:///system list --all`: + Id Name State + 1 vr1-dc0-opnsense running <- the broken edge (single-user) + 2 vr1-dc0-maas-01 running + 3 vr1-dc0-juju-01 running + - 9 role nodes shut off + +RACK NETWORK FACTS (measured, `ip -4 -o addr`, `ip route`, `virsh net-info`): + provider-public plane = libvirt net vr1-dc0-provider-public = bridge virbr5 + rack provider-public leg = 10.12.4.2/22 on virbr5 (permanent, dc-rack-net) + rack default route = default via 10.12.4.1 dev virbr5 <- the edge LAN + metal-admin = virbr2, carrying 10.12.8.2/.3/.4 (rack leg, node-DNS, utility) + WAN plane = libvirt net vr1-dc0-wan = bridge br-vr1-dc0-wan (D-125 bridge-in) + So the rack's A1 precondition for the egress gate is ALREADY satisfied; only the + edge itself is missing. + +TOFU ROOT (on voffice1, per D-128 -- inner roots run from the Office1 headend) + path : ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/ + tofu : OpenTofu v1.12.4, provider dmacvicar/libvirt v0.9.8 + state : 36 resources, of which the two edge resources are exactly + module.vr1_dc0_opnsense.libvirt_volume.disk + module.vr1_dc0_opnsense.libvirt_domain.vm + +>>> CORRECTION TO THE TASK BRIEF -- BASE IMAGE PATH. <<< + The brief cited the base image as + /var/lib/libvirt/vr1-dc0-inner/opnsense-26.7-nano.qcow2 + That is the module DEFAULT in variables.tf, and it is NOT the effective value. + `d124-inner.auto.tfvars` on voffice1 OVERRIDES it: + opnsense_base_path = "/home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2" + and variables.tf:27 explains why, in a note dated at the FIRST inner apply: + "Path ON THE EXECUTING HOST (voffice1, D-128 Plane 2) ... the remote + qemu+ssh provider UPLOADS volume content from ITS OWN filesystem -- a path + on vvr1-dc0 fails with 'Could not open URL for upload: failed to stat'." + This is load-bearing: the apply DESTROYS the volume before creating it, so a + base path that resolves only on the rack would have left the edge with no disk + at all. VERIFIED PRESENT ON voffice1 BEFORE ANY MUTATION: + -rw-r--r-- 1 jessea123 jessea123 2192703664 Jul 20 06:13 + /home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2 + sha256 3981821e3a3c7be49f6d96f405266806a619756d8816f8a9d92e7fdf5476627d + (The rack ALSO holds a copy at the default path -- same filename, staged by the + bootstrap step -- which is exactly why the wrong one is easy to cite.) + +BOOTSTRAP MATERIAL ALREADY ON THE RACK (from the 2026-07-20 original build) + ~/d112c-console.py 2481 B D-112(c) console driver v6 (materialize leg) + ~/d112c-bootstrap.php.b64 992 B the config-write payload + ~/scripts/{opnsense-api.sh,opnsense-mint-apikey.php, + opnsense-set-iface-v4.php,opnsense-set-interface-v4.sh} (Jul 20) + ~/opnsense-bootstrap-apikey.sh, ~/opnsense-mint-apikey.php (Jul 20) + python3 pexpect 4.9.0 present on the rack. + + The `.b64` payload was DECODED and read. Two things confirmed by reading it + rather than trusting the record: + (a) It carries only `require_once("config.inc")` -- NO `util.inc`. That is + exactly the DOCFIX-199 / dc1-lesson-(iv) defect: `write_config()` calls + `shell_safe()`, which lives in util.inc, so this payload dies at + "Call to undefined function shell_safe()" BEFORE writing. The runbook's + own warning ("Any surviving dc0 .b64 artifact on a rack predates this line + and is missing the include -- do not replay it verbatim") is CORRECT and + is now confirmed against the artifact itself. + (b) The public key it installs is + ssh-ed25519 AAAAC3...tzoc vr1-dc0-edge_svc (D-112c per-env) + which is BYTE-IDENTICAL to ~/vr1-dc0-creds/vr1-dc0-edge_ed25519.pub on the + jumphost (SHA256:2Gknnn3FsMtJov7MFBQlAjT6XJAd5EVz4OaCuY2ZLaU). The existing + private key therefore still opens the rebuilt edge -- no new keypair needed. + +>>> PRE-EXISTING CREDENTIAL FINDING, FOUND WHILE STAGING (logged, not actioned + beyond what the rebuild requires) -- SEE SECTION 6. <<< + `ls` (never a read) of vr1-dc0-rack:~/vr1-dc0-creds/ shows, dated 2026-07-20: + -rw------- 173 B opnsense-api.txt + -rw------- 432 B vr1-dc0-edge_ed25519 (the PRIVATE half) + SEC-021(a) records that the dc0 edge API credential is ABSENT from its recorded + jumphost location and asks for "a decision between RE-MINT and locating an + off-jumphost copy". THE OFF-JUMPHOST COPY IS THIS ONE. It has been on the rack + for 13 days. The dc1 build shredded its equivalents the same phase and opened + SEC-015 for the transient; the dc0 build never did, and nothing noticed -- + `creds-manifests/vm-secret-locations:123` DOES declare + `rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/*` as an audited location, so + this was visible to a `--remote` sweep and simply had not been reconciled. + CONSEQUENCE FOR THIS TASK: the 173 B key authenticates to a config that this + rebuild DESTROYS, so it is dead material after the apply and must be re-minted + regardless. It also blocks `opnsense-bootstrap-apikey.sh`, which refuses to + overwrite an existing output file (by design -- overwriting would strand a live + key). Handling is in Section 4. + +================================================================================ +SECTION 1 -- RE-PLAN AND CONTENT ASSERTION (step 1) +================================================================================ + +The stale plan the brief mentioned (/tmp/edge-rebuild.plan, Aug 2 03:36) was NOT +used. A fresh one was cut. Pre-mutation safety captures first: + + voffice1: cp -p terraform.tfstate ~/vr1-dc0-substrate-tfstate-pre-edge-rebuild-20260802.json + sha256 45e9aad95e37cdbf2840508173aa7cea56ddc469fdc922a57f0516742ec77b9c + (identical both files -- the backup is a real copy, verified not assumed) + rack: serial log BASELINE recorded so the post-rebuild boot can be asserted on + NEW content only. The domain's serial device has append="on", so the file + still holds the DAMAGED 08-01 boot; a grep of the whole file for a + clean-boot marker would match the old boot and look exactly like success. + /var/lib/libvirt/vr1/staging/vr1-dc0-opnsense-serial.log + size=477629 lines=21912 mtime=2026-08-01 05:49:39 + +PLAN (run on voffice1, D-128 Plane 2): + cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/ + tofu plan -replace=module.vr1_dc0_opnsense.libvirt_volume.disk \ + -replace=module.vr1_dc0_opnsense.libvirt_domain.vm \ + -out=/tmp/edge-rebuild-20260802.plan + + Plan: 2 to add, 0 to change, 2 to destroy. + +ASSERTION ON CONTENT, machine-read from `tofu show -json`, not from prose. +Prose was deliberately not the assertion surface: the refresh reports 20 resources +under "changed outside of OpenTofu", which is a DRIFT NOTE and not an action, and a +grep over prose blurs the two. + + $ jq '.resource_changes[] | select(.change.actions != ["no-op"]) | .address + " " + (.change.actions|join(","))' + module.vr1_dc0_opnsense.libvirt_domain.vm ACTIONS=delete,create + module.vr1_dc0_opnsense.libvirt_volume.disk ACTIONS=delete,create + + count of non-no-op changes ................................. 2 (expected 2) + count of non-no-op changes NOT naming vr1_dc0_opnsense ..... 0 (required 0) + +Both halves were asserted, not just the "nothing extra" half. Asserting only that +every action line names the edge would PASS on a plan that replaced the domain and +left the corrupt qcow2 in place -- which would reattach the damaged disk and make +the whole rebuild a no-op on the actual damage. So the positive half (exactly these +TWO addresses, each delete+create) is asserted as well. + +BASE IMAGE THE PLAN WILL ACTUALLY USE, read out of the plan rather than trusted: + $ jq '.resource_changes[] | select(.address|test("libvirt_volume.disk")) | .change.after.create.content.url' + /home/jessea123/vr1-dc0-images/opnsense-26.7-nano.qcow2 + -- the voffice1 path, confirmed present at 2192703664 B before the apply. This is + the check that made the brief's cited path worth correcting: the apply destroys + the volume BEFORE creating its replacement, so a source path that resolved only on + the rack would have left the edge with no disk at all and no way back. + +VERDICT: plan matches the intent exactly. Proceed to apply. + +================================================================================ +SECTION 2 -- APPLY (step 2) +================================================================================ + + $ tofu apply /tmp/edge-rebuild-20260802.plan (on voffice1) + module.vr1_dc0_opnsense.libvirt_domain.vm: Destroying... [name=vr1-dc0-opnsense] + module.vr1_dc0_opnsense.libvirt_domain.vm: Destruction complete after 0s + module.vr1_dc0_opnsense.libvirt_volume.disk: Destroying... [id=/var/lib/libvirt/vr1-dc0-inner/vr1-dc0-opnsense-disk.qcow2] + module.vr1_dc0_opnsense.libvirt_volume.disk: Destruction complete after 0s + module.vr1_dc0_opnsense.libvirt_volume.disk: Creating... + module.vr1_dc0_opnsense.libvirt_volume.disk: Creation complete after 4s + module.vr1_dc0_opnsense.libvirt_domain.vm: Creating... + module.vr1_dc0_opnsense.libvirt_domain.vm: Creation complete after 2s + Apply complete! Resources: 2 added, 0 changed, 2 destroyed. + +A 2.2 GB volume "created in 4s" is the kind of number worth distrusting, so the +ARTIFACT was measured rather than the log believed: + rack: -rw------- libvirt-qemu kvm 2198142976 Aug 2 06:14 + /var/lib/libvirt/vr1-dc0-inner/vr1-dc0-opnsense-disk.qcow2 + (the old disk was 2559705088 B and dated Aug 1 05:51 -- this is a new file of + base-image size, not the survivor.) + +DOMAIN AS BUILT (virsh -c qemu:///system, on the rack): + Id 4 vr1-dc0-opnsense running Autostart: enable 2 vCPU 2048 MiB + vnet14 network vr1-dc0-provider-public virtio 52:54:00:6d:a9:27 (LAN, idx 0) + vnet15 bridge vr1-dc0-wan virtio 52:54:00:ae:b2:89 (WAN, idx 1) + target org.qemu.guest_agent.0 state='disconnected' + -- D-127 autostart=true and D-129 expose_qga_channel=true both came back with + the module, as designed. The channel reads 'disconnected' because the + os-qemu-guest-agent PACKAGE was never installed on the DC edges (the G13 + retrofit was office1-opnsense only). PRE-EXISTING, not a rebuild regression. + Edge MACs are NOT pinned by the module (only the nine node VMs are), so both + edge MACs are new. Nothing keys on them: MAAS knows the nodes, not the edge. + +FIRST BOOT -- ASSERTED ON NEW SERIAL CONTENT ONLY +The serial device has append="on", so the file still contains the damaged 08-01 +boot. Grepping the whole file for a healthy-boot marker would match the OLD boot +and look exactly like success. Assertions were run on `tail -n +21913` only: + + ld-elf.so.1 .............. 0 (was: present, the failure) + UNREF FILE ............... 0 (was: 2533) + UNEXPECTED SOFT UPDATE ... 0 (was: 785) + SALVAGE? ................. 0 (was: 257) + libcrypto / libpython .... 0 / 0 + Error in early script .... 0 (was: 10-configd, 15-templates, 90-carp) + login: ................... 1 + OPNsense 26.7 ............ 1 + + ONE apparent hit needed chasing rather than dismissing: "Enter full pathname" + counted 1 in the "new" region. That is the single-user prompt -- the exact + symptom being rebuilt away. Cause, measured by reading the first line of the + region: the OLD log's final line had NO trailing newline, so `wc -l` counted + 21912 while a 21913th partial line existed; `tail -n +21913` therefore begins + mid-way through the OLD prompt, and the NEW boot starts on the very next + characters (`/boot/config: -S115200 -h -D`). An off-by-one in my instrument, + not a fault in the edge. Recorded because the alternative -- assuming it away -- + is the failure mode this repo keeps logging. + + Boot banner (new content): + *** OPNsense.internal: OPNsense 26.7 (amd64) *** + LAN (vtnet0) -> v4: 192.168.1.1/24 + WAN (vtnet1) -> + vtnet0 = LAN = provider-public, vtnet1 = WAN -- the same mapping the 2026-07-20 + dc0 build and the 2026-07-23 dc1 build both measured. WAN is address-less + because `vr1-dc0-uplink` carries an but NO block (measured at the + D-125 gate, 2026-07-20): the WAN static is set explicitly below, by design. + +================================================================================ +SECTION 3 -- D-112(c) CONSOLE BOOTSTRAP (step 3) +================================================================================ + +TOOLING STAGED FRESH FROM THE REPO, sha256-verified BOTH ENDS before use. The +rack's own copies date from 2026-07-20 and predate the sshd-readiness wait and +scp retry that were added on 2026-07-23; the Jul-20 copies were not used. + -> vr1-dc0-rack:~/repo-stage/scripts/ + c1bd904489fe6e819463f1c9db483ea054d15586acd05e08fd85285dbbf1b450 opnsense-bootstrap-apikey.sh + 2a85225dc6329524b86986c0c1ab3ccb1863b0fe0fd522fd7f6c88e1acca0938 opnsense-mint-apikey.php + 5dd78f323df0261b4f9bc32cdb8fbf5e7fcc411e58527d49742716ad8a4bb7ef opnsense-set-interface-v4.sh + ed3c4784e20046aab09b6041030176301bddb5442e2ff4cbb5911390d8bfdc5a opnsense-set-iface-v4.php + 8c2329f87fe72a939cfe48b7fece33685ff8f392949af96b490772453b35c905 opnsense-api.sh + (both PHP helpers staged BESIDE their drivers -- both scripts resolve them + from their own directory and fail at run time otherwise.) + +DRIVER: ~/d112c-console-dc0-rebuild.py on the rack (sha256 +6dbf985697815e030bdd29031a75b7142570e92558336a00d5194bdb9f78fa04, byte-identical +to the local copy). It merges the two legs the 2026-07-20 build ran as separate +artifacts, and it does NOT replay ~/d112c-bootstrap.php.b64 -- that artifact is +missing require_once("util.inc") and would have died inside write_config() at +"Call to undefined function shell_safe()" BEFORE writing anything. Confirmed on +the live edge: `grep -c shell_safe /usr/local/etc/inc/util.inc` -> 7. +Payload shipped as 10 base64 chunks of <=160 chars (lesson i: the serial console +wraps long lines). + +TRANSCRIPT (~/d112c-console-dc0-rebuild-20260802.log on the rack), key lines: + root@OPNsense:~ # grep -c authorizedkeys /conf/config.xml + 1 + root@OPNsense:~ # grep -c shell_safe /usr/local/etc/inc/util.inc + 7 + root@OPNsense:~ # php -l /tmp/b.php + No syntax errors detected in /tmp/b.php + root@OPNsense:~ # php /tmp/b.php + CONFIG-WRITTEN + AK-IN-CONFIG=152 + USER-MATERIALIZED + root@OPNsense:~ # configctl openssh restart + OK + AK-IN-CONFIG=152 matches the dc1 datapoint exactly (152). USER-MATERIALIZED is + the local_user_set() call -- lesson v: write_config() alone does NOT create + /root/.ssh/authorized_keys, which is the measured reason key auth "mysteriously" + fails after a clean config write. + + MINOR, RECORDED NOT FIXED: two diagnostic lines inherited verbatim from the v6 + driver use `2>&1`, and the edge root shell is tcsh, so both printed + "Ambiguous output redirect." (lesson iii). They are diagnostics only -- the + bootstrap itself is unaffected -- but the dc1 session made its equivalents + csh-safe and the dc0 rack copy still has not been. Queued as a finding. + +>>> A TRAP THAT WOULD HAVE READ AS A SCRIPT BUG, CAUGHT BY LOOKING FIRST <<< +The rack's ~/.ssh/known_hosts held the DESTROYED edge's host keys for BOTH +192.168.1.1 (3 entries) and 10.12.4.1 (1 entry), from 2026-07-20. The rebuilt +edge generates NEW host keys, so every script here -- all of which use +`-o BatchMode=yes` -- would have failed with REMOTE HOST IDENTIFICATION HAS +CHANGED and no interactive prompt to resolve it. Measured with `ssh-keygen -F` +BEFORE running anything, then: + cp -p ~/.ssh/known_hosts ~/.ssh/known_hosts.pre-edge-rebuild-20260802 + ssh-keygen -R 192.168.1.1 ; ssh-keygen -R 10.12.4.1 + ssh-keyscan -T 10 192.168.1.1 >> ~/.ssh/known_hosts (and 10.12.4.1 later) +This belongs in the runbook: a REBUILT edge invalidates the rack's known_hosts, +and the dc0/dc1 first-build records could not have caught it because on a first +build there is no prior entry. + +INTERIM REACH LEG (the factory LAN is 192.168.1.0/24 and the rack has no route): + sudo ip addr add 192.168.1.2/24 dev virbr5 (non-persistent; removed later) + +KEY-ONLY SSH PROVEN -- the D-112(c) exit condition: + ssh -i ~/vr1-dc0-creds/vr1-dc0-edge_ed25519 root@192.168.1.1 + uname -r -> 15.1-RELEASE-p1 + ifconfig -l -> vtnet0 vtnet1 lo0 enc0 pfsync0 pflog0 + ifconfig vtnet0 -> inet 192.168.1.1 netmask 0xffffff00 + ifconfig vtnet1 -> (no inet) + opnsense-version -> OPNsense 26.7 (amd64) + NO new keypair was minted: the pubkey in the surviving .b64 was verified + byte-identical to ~/vr1-dc0-creds/vr1-dc0-edge_ed25519.pub on the jumphost, so + the EXISTING private key opens the rebuilt edge. + +API KEY MINTED (D-113(a2), the vendor model -- no GUI click, no re-implemented crypto) + cd ~/repo-stage/scripts + OPNSENSE_SSH_KEY=$HOME/vr1-dc0-creds/vr1-dc0-edge_ed25519 \ + bash opnsense-bootstrap-apikey.sh 192.168.1.1 \ + $HOME/vr1-dc0-creds/opnsense-api-rebuild-20260802.txt + -> OK: minted for root -- key=80 chars, secret=80 chars, wrote 173 bytes (0600, + secret NOT printed) + Output went to a DATED path, not the canonical name, so the script's + refuse-to-overwrite guard was never worked around: a stale + ~/vr1-dc0-creds/opnsense-api.txt already sat on the rack (Section 0) and that + guard exists precisely to stop a live key being stranded. Nothing was deleted to + make a script run. + SMOKE TEST: `opnsense-api.sh GET core/firmware/status` -> exit 0, + {"product":{"CORE_ABI":"26.7","CORE_ARCH":"amd64",...,"CORE_PRODUCT":"OPNsens... + Third D-113(a2)-on-26.7 datapoint (after dc0 2026-07-20 and dc1 2026-07-23). + +CONSOLIDATED PER D-137, verified by SHAPE only -- no secret was ever read: + ~/vr1-dc0-creds/opnsense-api.txt on the JUMPHOST, 0600, 173 bytes + key= lines 1, secret= lines 1; key value 80 chars, secret value 80 chars + sha256 identical to the rack copy (ad59dca4...956378, compared not assumed) + This materially CHANGES SEC-021(a), which recorded that dc0 had no consolidated + edge API credential and therefore "no API management path from vcloud". It now + has one. See Section 6. + +================================================================================ +SECTION 4 -- ADDRESSING, BOTH LEGS, READ BACK (step 4) +================================================================================ + +Ordering follows the script's own warning: WAN FIRST, verified, LAN LAST -- the +LAN move relocates the very interface the session arrives on. + +WAN (dry run first, then --commit): + bash opnsense-set-interface-v4.sh --commit 192.168.1.1 wan 172.30.2.2 24 172.30.2.1 + interface 'wan' is device 'vtnet1' (measured from the edge's own config) + ipaddr: dhcp -> 172.30.2.2 | subnet: (none) -> 24 | gateway: (none) -> WAN_GW (172.30.2.1) + OK: saved and read back -- wan = 172.30.2.2/24 (3 interfaces intact) + OK: gateway WAN_GW = 172.30.2.1 is the default route for wan + ... + inet 172.30.2.2 netmask 0xffffff00 broadcast 172.30.2.255 + OK: 172.30.2.2/24 is live on wan (vtnet1). + The pf reload ALSO produced real automatic outbound NAT at this point -- + nat on vtnet1 inet from (vtnet0:network) to any -> (vtnet1:0) port 1024:65535 + which is the appendix-A "addressed perfectly, forwards nothing" defect NOT + happening. That check is a WARN-only report in the script, so it was read + deliberately rather than left to scroll past. + +EGRESS PROVEN FROM THE EDGE ITSELF, BEFORE THE LAN WAS TOUCHED. Deliberate +ordering: after the LAN moves, the reach path is gone, and a WAN gateway that did +not take is far harder to diagnose from the other side. + default 172.30.2.1 UGS vtnet1 + ping 172.30.2.1 -> 3 packets transmitted, 3 received, 0.0% packet loss + ping 1.1.1.1 -> 3 packets transmitted, 3 received, 0.0% packet loss + host archive.ubuntu.com -> 91.189.92.22 / 91.189.92.24 + fetch http://archive.ubuntu.com/ubuntu/dists/jammy/Release -> FETCH-OK + +LAN (dry run first, then --commit): + bash opnsense-set-interface-v4.sh --commit 192.168.1.1 lan 10.12.4.1 22 + interface 'lan' is device 'vtnet0' (measured from the edge's own config) + ipaddr: 192.168.1.1 -> 10.12.4.1 | subnet: 24 -> 22 | gateway: (none) [UNTOUCHED] + OK: saved and read back -- lan = 10.12.4.1/22 (3 interfaces intact) + === APPLY === -> the invocation was killed at the 240s timeout (EXIT=124) + EXPECTED AND DOCUMENTED: `configctl interface reconfigure lan` drops the session + it is issued over, and the foreground ssh then hangs on the dead interface -- + the identical behaviour the dc1 build recorded ("TaskStop-ped; config was + already saved+applied"). The config was saved and applied; what the timeout cost + was the script's own post-drop legs (filter reload + read-back), which were then + run explicitly over a FRESH connection to the new address: + + from the rack (permanent leg 10.12.4.2/22, no interim leg needed): + ping 10.12.4.1 -> 3 packets transmitted, 3 received, 0% packet loss + ip neigh -> 10.12.4.1 dev virbr5 lladdr 52:54:00:6d:a9:27 DELAY + (contrast the outage capture's `10.12.4.1 ... FAILED`) + on the edge: + ifconfig vtnet0 -> inet 10.12.4.1 netmask 0xfffffc00 broadcast 10.12.7.255 (/22) + ifconfig vtnet1 -> inet 172.30.2.2 netmask 0xffffff00 + netstat -rn -> default 172.30.2.1 UGS vtnet1 + configctl filter reload -> OK + pfctl -s nat (carp boilerplate removed): + nat on vtnet1 inet from (vtnet0:network) to any -> (vtnet1:0) port 1024:65535 + nat on vtnet1 inet from (lo0:network) to any -> (vtnet1:0) port 1024:65535 + nat on vtnet1 inet from 127.0.0.0/8 to any -> (vtnet1:0) port 1024:65535 + -- (vtnet0:network) is now 10.12.4.0/22, so the rack and everything on + provider-public NATs out of vtnet1. Read explicitly; not assumed from the + earlier WAN-stage output, because the LAN network changed underneath it. + + Interim leg removed: sudo ip addr del 192.168.1.2/24 dev virbr5 + virbr5 now carries only 10.12.4.2/22 -- the dc0 end-state exactly as before the + outage, and identical in shape to dc1's. + +BOTH LEGS, FINAL, READ FROM THE KERNEL: + WAN vtnet1 172.30.2.2/24 gw 172.30.2.1 (egress 0% loss) + LAN vtnet0 10.12.4.1/22 (the ruled provider-public gateway) + +================================================================================ +SECTION 5 -- THE GATE (step 5, the definition of done) +================================================================================ + +RUN 1 -- against repo HEAD's dc-egress-check.sh, 2026-08-02 ~06:40. FAIL, 7/8: + + == dc-egress-check dc0 == + edge LAN (gateway): 10.12.4.1 rack leg: 10.12.4.2 + snap proxy: http://10.12.8.4:3129 + OK A1 default route via 10.12.4.1 (this site's edge) + OK A2 edge 10.12.4.1 ANSWERS + OK A3 off-site ICMP anchor 1.1.1.1 answers + OK A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com) + OK A4 apt upstream (archive.ubuntu.com) reachable (HTTP 200) + OK A4 UCA upstream (ubuntu-cloud.archive.canonical.com) reachable (HTTP 200) + OK A4 juju agent stream (streams.canonical.com) reachable (HTTP 200) + FAIL A4 snap store VIA this site's proxy answered '400' -- unrecognised, not assumed healthy + RESULT: pass=7 fail=1 exit 1 + +The gate is LAYERED and reports the FIRST failure as the cause. A1, A2, A3 and +three of four A4 probes were green, so by the gate's own design the edge was NOT +the cause. The failure was ISOLATED rather than reasoned about -- three probes +from the rack: + + 1. THROUGH the proxy, no header (exactly what A4 sends): + http_code=400 curl_exit=0 + {"error-list":[{"code":"bad-argument","message":"Snap-Device-Series header is required."}]} + 2. THROUGH the proxy, WITH `-H 'Snap-Device-Series: 16'`: + http_code=200 curl_exit=0 + {"channel-map":[{"channel":{"architecture":"amd64","name":"stable",... + 3. DIRECT, no proxy, no header -- the control that separates store from proxy: + http_code=400 curl_exit=0 (identical error) + + Probe 3 is the decisive one: the 400 is produced with the proxy taken out of + the path entirely, so it is the STORE rejecting a malformed request, not the + proxy refusing a tunnel. A 400 in fact PROVES the tunnel end to end -- squid + admitted the CONNECT, TLS completed to api.snapcraft.io, and the store replied. + + Corroborated by the repo's own snap gate, which sends the header: + `dc-snap-proxy.sh check dc0` -> PASS, 16/16, including + OK CONNECT to the store API: curl exit 0, HTTP 200, channel-map + sha3-384 present + OK CDN tunnel allowed (curl exit 0; the CDN's own status was '403') + OK non-allowlisted destination is DENIED (403 at CONNECT) -- the allowlist is in force + + So: a DEFECT IN THE GATE, not in the edge. `dc-snap-proxy.sh:215,469` sends + `Snap-Device-Series: 16`; `dc-egress-check.sh`'s A4 `probe()` did not. The + assertion had never passed anywhere -- dc0 previously failed at A2 with A4 + SKIPPED, and dc1 has no ruled snap proxy and was failing A3/A4 for other + reasons -- so A4's snap branch had never once been exercised against a live + proxy. It was decoration until this run. + + The finding was LOGGED, NOT FIXED (hard rule 1; and a gate change needs a + failing-direction harness case + a changelog, which would have collided with + the operator's concurrent work on this branch). + +RUN 2 -- 2026-08-02 ~06:55. Between the two runs the operator INDEPENDENTLY +found and fixed the same defect in the working tree (`scripts/dc-egress-check.sh` +mtime 06:53:27, uncommitted), adding `-H 'Snap-Device-Series: 16'` to `probe()` +with a comment citing the same three measurements. Re-run against that: + + $ ssh vr1-dc0-rack 'bash -s -- check dc0' < scripts/dc-egress-check.sh + == dc-egress-check dc0 == + edge LAN (gateway): 10.12.4.1 rack leg: 10.12.4.2 + snap proxy: http://10.12.8.4:3129 + OK A1 default route via 10.12.4.1 (this site's edge) + OK A2 edge 10.12.4.1 ANSWERS + OK A3 off-site ICMP anchor 1.1.1.1 answers + OK A3 off-site TCP/HTTP anchor answers 200 (archive.ubuntu.com) + OK A4 apt upstream (archive.ubuntu.com) reachable (HTTP 200) + OK A4 UCA upstream (ubuntu-cloud.archive.canonical.com) reachable (HTTP 200) + OK A4 juju agent stream (streams.canonical.com) reachable (HTTP 200) + OK A4 snap store VIA this site's proxy reachable (HTTP 200) + + RESULT: pass=8 fail=0 + PASS: dc-egress-check dc0 + GATE-EXIT=0 + +>>> DEFINITION OF DONE MET: exit 0, 8/8. <<< +The two runs differ ONLY in the gate's own probe. No edge, rack, proxy or network +state changed between them -- the edge was already fully working at run 1, which +is what the A1/A2/A3 + three-upstream greens in run 1 already showed. + +CONVERGENCE RE-PLAN (the repo's standing post-apply evidence -- G16, step B, +step C and dc1 step C all cite one): + $ tofu plan (on voffice1) + No changes. Your infrastructure matches the configuration. + machine-asserted from `tofu show -json`: + non-no-op resource_changes: 0 + addresses: (empty) + State matches the built artifact. The 20 "changed outside of OpenTofu" drift + lines still appear on every plan of this root; they are notes, not actions, and + the JSON assertion is what distinguishes them. + +================================================================================ +SECTION 6 -- CREDENTIALS +================================================================================ + +MINTED : dc0 edge REST API key, via the vendor model (D-113(a2)), on the rebuilt + edge. Secret never printed, never read into context; verified by + length/format only. +CONSOLIDATED (D-137): ~/vr1-dc0-creds/opnsense-api.txt on the JUMPHOST, 0600, + 173 B, 1x `key=` (80 chars) + 1x `secret=` (80 chars), sha256-compared + to the source copy before the source was destroyed. +CLEANED: the dated rack copy `~/vr1-dc0-creds/opnsense-api-rebuild-20260802.txt` + was `shred -u`'d after consolidation -- a second copy of a LIVE secret + is an exposure I introduced, so I removed it (dc1 SEC-015 precedent). + +REGISTER STATE, run rather than assumed (`python3 scripts/creds-matrix.py --tier2` +on the jumphost, 121 rows, 20 check groups clean, 11 findings): + - `E1/E3 existence: every expected artifact present and nothing undeclared, + across 4 fully-probed role(s)` -- CLEAN. Previously CURRENT-STATE recorded + `E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt'`. The mint closed it. + - `[FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, + SEC-021)` -- STILL RED, and now red for a DIFFERENT REASON than when the + operator accepted it on 2026-07-30. Then, the credential did not exist. Now it + exists and `creds-manifests/vr1-dc0.manifest` simply does not declare it + (dc1's manifest:25 does declare its equivalent). The manifest is DERIVED from + the matrix via `creds-matrix.py --render`, so this is a mechanical + regeneration -- deliberately NOT run here: it rewrites a generated file across + all three sites while the operator has uncommitted work on this branch. + - The total (11) is NOT comparable to CURRENT-STATE's 2026-07-30 figure of 6: + that run had 101 rows, this one 121, the difference being the SEC-027/SEC-028 + dc1 forward-register rows added since. Quoting a delta across two different + row sets would be a fabricated number. + - NO test anywhere asserts the literal string `n-dc0-edge-api-absent` + (grepped `--include=*.py --include=*.sh`); the only occurrences are + `creds-matrix.tsv:76` and `creds-matrix-notes.md:123`. So nothing turns red + from the remediation -- but both are now factually STALE and their wording + needs REPLACING with the new invariant, never deleting. + +MATRIX ROW: `dc0-edge-api` ALREADY EXISTS at `creds-matrix.tsv:76`. No duplicate +was added -- the register was already expecting this credential, which is exactly +why its absence was detectable. + +LEDGER: SEC-032 added (next-free confirmed by grep at write time; SEC-031 was +taken by the operator's concurrent dc1-edge row). It records the dc0 analog of +SEC-015 -- edge private key + a now-dead API credential resident on the rack since +2026-07-20, undisclosed -- and notes that this ANSWERS SEC-021(a)'s open question +("locate an off-jumphost copy"). Nothing was actioned on it. + +================================================================================ +SECTION 7 -- WHAT WAS LEFT BEHIND, NAMED +================================================================================ + +ON THE RACK (vr1-dc0-rack): + ~/vr1-dc0-creds/vr1-dc0-edge_ed25519 LEFT. Pre-existing since 2026-07-20. + Structurally required by the current tooling: opnsense-bootstrap-apikey.sh + and opnsense-set-interface-v4.sh both SSH the edge DIRECT with no + ProxyJump, and the edge LAN is reachable only from the rack. Disclosed as + SEC-032; NOT silently kept. + ~/vr1-dc0-creds/opnsense-api.txt LEFT. The 2026-07-20 key, now DEAD + (the config holding its hash was destroyed by this rebuild). Left rather + than shredded because it is the physical evidence behind SEC-021(a) and the + operator has not ruled. Named in SEC-032 item (1). + ~/repo-stage/scripts/opnsense-* (5 files) LEFT. Non-secret repo scripts, + sha256-verified against the repo when staged. Same convention as the + 2026-07-30 juju staging already in that directory. They WILL go stale -- + re-verify hashes before any future use rather than trusting them. + ~/d112c-console-dc0-rebuild.py LEFT. Non-secret (embeds only the + PUBLIC key). Kept as the reproducible artifact for the next rebuild. + ~/d112c-console-dc0-rebuild-20260802.log LEFT. Non-secret console transcript, + the dc1 precedent for retaining one as an audit artifact. + ~/.ssh/known_hosts.pre-edge-rebuild-20260802 and ~/.ssh/known_hosts.old + LEFT. Public host keys only. + ~/d112c-bootstrap.php.b64 (2026-07-20) LEFT, UNUSED, AND DANGEROUS TO REPLAY + -- it is missing require_once("util.inc"). Section 3. + +ON voffice1: + ~/vr1-dc0-substrate-tfstate-pre-edge-rebuild-20260802.json the pre-apply state + backup. Delete once the rebuild is accepted. + /tmp/edge-rebuild-20260802.plan, .plan.json, /tmp/edge-rebuild-converge-20260802.plan + /tmp/edge-rebuild.plan -- the STALE plan named in the task brief, NOT used. + +REPO TREE (left dirty for the operator to commit, as instructed): + docs/audit/dc0-edge-rebuild-20260802.txt this file (new) + docs/security-ledger.md SEC-032 appended + (scripts/dc-egress-check.sh and tests/dc-egress-check/run-tests.sh were already + modified by the operator's concurrent session and were NOT touched here.) + +================================================================================ +SECTION 8 -- FINDINGS RAISED, NONE ACTIONED (hard rule 1) +================================================================================ + +F1. A REBUILT EDGE INVALIDATES THE RACK'S known_hosts, AND NO RUNBOOK SAYS SO. + Both edge management scripts use `-o BatchMode=yes`, so a changed host key + fails with REMOTE HOST IDENTIFICATION HAS CHANGED and no way to answer the + prompt. Neither the dc0 nor the dc1 build could have caught this: on a FIRST + build there is no prior entry. This is a REBUILD-only trap and belongs in the + D-112(c) chain and in appendix-A by symptom. + +F2. `dc-egress-check.sh`'s A4 snap probe omitted `Snap-Device-Series: 16` -- + found here, and independently fixed by the operator mid-session. Recorded + because the CLASS is what matters: the assertion had never been exercised + (dc0 failed earlier at A2, dc1 has no ruled proxy), so it was decoration on a + gate that shipped the same day with a 14/14 harness and a mutation pass. A + harness that mocks curl cannot catch a wrong REAL request. The durable rule: + when two scripts probe the same endpoint, they must share the probe + definition -- `dc-snap-proxy.sh:215` already had the correct one. + +F3. THE EDGE STILL SERVES DHCP ON THE LAN, WITH A STALE FACTORY RANGE. + Measured on the rebuilt edge AFTER the LAN move: + /usr/local/etc/dnsmasq.conf : interface=vtnet0 + dhcp-range=tag:vtnet0,192.168.1.100,192.168.1.199,86400 + dhcp-range=tag:vtnet0,::1000,::2000,constructor:vtnet0,slaac,64,86400 + sockstat -4 -l : nobody dnsmasq ... udp4 *:67 + vtnet0 is now 10.12.4.0/22 -- the segment carrying the juju controller + (10.12.4.5) and the MAAS region VM (10.12.4.6). Nothing in the addressing + toolchain regenerates dnsmasq.conf: `configctl interface reconfigure` and + `configctl filter reload` do not touch it, so the range still describes the + factory LAN. NOT CHANGED (out of scope; the edge has no ruled DHCP role and + changing it is a config decision). + WHAT IS MEASURED vs WHAT IS NOT, stated separately: + MEASURED: dnsmasq is running, bound to udp4 *:67, with a 192.168.1.x range + on vtnet0, and dhcpd/Kea blocks in config.xml count ZERO. + NOT MEASURED: whether it would actually answer a DHCP request on the new + subnet. dnsmasq matches a range to the interface's own subnet and + 192.168.1.0/24 is no longer on vtnet0, which SUGGESTS it is inert + -- but that is reasoning, not a measurement, and it is flagged as + such rather than reported as a fact. + NOT ESTABLISHED: whether the destroyed edge was in the same state. It very + likely was (it took the identical factory-boot-then-re-address + path), but it cannot be checked now and dc1 was out of scope for + this task, so this is NOT claimed as "restored prior state". + Worth an operator decision before Stage 5 puts nine nodes on that segment. + +F4. THE `.b64` BOOTSTRAP ARTIFACT ON THE dc0 RACK IS STILL THE BROKEN ONE. + DOCFIX-199 warned about it in prose; this session confirmed it against the + artifact. It remains on the rack, unmarked, next to a working driver. Prose in + a runbook does not disarm a file on a host -- either delete it or rename it to + something that cannot be replayed by mistake. + +F5. Two diagnostic lines in the console driver still use `2>&1` and print + "Ambiguous output redirect." on the edge's tcsh. Inherited from the v6 + driver; dc1 fixed its equivalents and the dc0 rack copy was never updated. + Cosmetic, but it is the documented tcsh trap firing in a live transcript. + +F6. `creds-manifests/vr1-dc0.manifest` does not declare `opnsense-api.txt` even + though the credential now exists and `creds-matrix.tsv:76` expects it. One + `creds-matrix.py --render` closes it. See Section 6 and SEC-032 item (2). + +================================================================================ +SECTION 9 -- WHAT WAS *NOT* RESTORED (scope, stated so it is not assumed) +================================================================================ + +The rebuilt edge is a FACTORY 26.7 nano plus SSH, an API key, and two static +addresses. It does NOT carry: + - os-qemu-guest-agent. The D-129 qga CHANNEL is present (module input + expose_qga_channel=true) and the domain XML shows it `state='disconnected'`, + but the PACKAGE was never installed on either DC edge -- the G13 retrofit was + office1-opnsense only. PRE-EXISTING, not a regression: the outage capture + records `qga: "QEMU guest agent is not connected"` on the OLD edge too. + - any interface-level IPv6. The dc0 edge's v6 config was blocked by SEC-021(a) + (no API credential) and had never been done. That blocker is now cleared. + - firmware 26.7.1. The office1 edge was updated to 26.7.1 in the G13 bundle; + both DC edges were and remain 26.7 (`CORE_ABI 26.7`, measured). + - any firewall rule beyond the vendor defaults, which is what the previous edge + also had -- D-113(a2) config is done over REST and none was applied to dc0 + beyond addressing. +Nothing was lost that the old edge had, on the evidence available; but the two +statements "the rebuild restored what was there" and "the rebuild produced a +working edge" are different claims, and only the second one is proven here. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index d604311..121aca7 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -82,3 +82,4 @@ | SEC-029 | 2026-07-30 | **Octavia PKI overlay resident on the DC RACK.** The overlay carries the Octavia controller CERT and its PRIVATE KEY, and `juju deploy` consumes it at deploy time. D-138 moved the cloud-facing juju client INTO the DC, and the deploy CANNOT run from `voffice1` at all -- there is no L3 path from the headend to the controller API on the provider plane. dc0's overlay was therefore copied to the dc0 rack 2026-07-30 (`~/repo-stage/overlays/vr1-dc0-octavia-pki.yaml`, `0600`, sha256 compared end to end and equal). dc1's is NOT yet copied. | 2026-07-30 Stage-5 bundle deploy; D-138; this session's changelog | operator ("Copy dc0's PKI overlay to the dc0 rack (Recommended)") | **OPEN -- residency, not a new key.** (1) This is a COPY of material that already exists on the headend (SEC-004 `n-overlay-in-clone`), not a new mint, so the rotation obligation remains the existing `octavia-pki.sh reissue` path -- **and that path must now update BOTH copies, or the rack will deploy a superseded cert.** (2) Per-DC isolation holds BY CONSTRUCTION: each rack receives ONLY its own DC's overlay, never the pair; a cross-DC copy would destroy the independence that `octavia-pki.sh verify`'s A11 assertion exists to prove. (3) `~/repo-stage/` on the rack is a STAGING directory created because the rack has NO REPO CLONE -- the same D-138 tooling gap as queued finding F1. If a clone is later placed on DC client hosts, this path moves and the `vm-secret-locations` row must move with it. | | SEC-030 | 2026-07-31 | **Local permission rules bypass the committed ask-gating -- ACCEPTED, KNOWN posture, recorded so it is explicit rather than implicit.** `.claude/settings.json` (committed team policy) deliberately puts `juju deploy`, `maas admin machine delete` and `maas admin * update/create/release` in **ask**, and the controller-destruction command in **deny**. `.claude/settings.local.json` (GITIGNORED, 286 allow rules accumulated across sessions, largely from interactive don't-ask-again approvals) contains `Bash(ssh *)`, `Bash(ssh voffice1 *)`, `Bash(ssh vr1-dc0-rack *)`, `Bash(ssh vr1-dc0-maas *)` and `Bash(python3 *)`. Because an `ssh ''` invocation matches on the OUTER ssh command, a destructive `maas admin machine delete` issued over ssh never reaches the ask rule; `python3 *` permits arbitrary execution outright. **CLAUDE.md hard rule 3 states the ask rules enforce per-mutation approval and are not to be worked around, so this is a real divergence between stated and enforced posture.** | 2026-07-31 permission carry-over review; continuation of `queued-findings-20260730-dc0-region-migration` F1 | operator ("Leave them as they are") | **OPEN -- ACCEPTED BY RULING, NOT AN OVERSIGHT.** (1) The operator was shown the five rules verbatim and the gating they bypass, and ruled to keep them: the mutation gate in practice is the operator's conversational approval of each presented command, not the rule engine. That has held all session. (2) **The compensating control is the PRESENTATION discipline** -- every mutation is still stated, justified as minimal, and approved before running. If that discipline lapses, nothing else catches it. (3) **NOT promoted to committed team policy** (ruling: "Promote a curated safe subset"): 36 read-only rules were promoted to `settings.json`; these five stay local-only so the bypass is NOT baked into team policy or carried to Roosevelt. (4) **`Read(//home/jessea123/**)` was deliberately EXCLUDED from promotion** -- it grants read of `~/vr1-*-creds/`, `~/vault-init/`, `~/as-executed/` and `~/tenant-*`, every one of which CLAUDE.md forbids reading into context. Promoting it would have put a secrets-read permission into team policy. (5) Four local wildcards were NARROWED before promotion because they permit mutation: `ip route *`, `ip neigh *`, `bridge fdb *`, `tc qdisc *` all allow add/del on the host running every DC node; only the `show`/`get` verbs were promoted. (6) A rebuild still loses the ~250 unpromoted local rules -- that half of F1 remains open by choice. | | SEC-031 | 2026-08-02 | **The vr1-dc1 OPNsense edge is currently an OPEN ROUTER serving its management GUI to the simulated ISP.** The 2026-08-01 hard power cut (in-place tofu resize bouncing the containment VM) destroyed the edge's FreeBSD base-system user database -- `/etc/master.passwd` (`pwd_mkdb: line #1 too long`) and `/etc/group` (`pw: group 'wheel' does not exist`) -- so `Configuring firewall.....failed.` on every boot since and NO pf ruleset is in force. MEASURED against a control, which is what makes this a finding rather than a worry: from vcloud, the healthy office1 edge (same 26.7 image, same build path) answers WAN ICMP with 100% loss and its GUI with `000`; the dc1 edge answers WAN ICMP at **0% loss** and its GUI **200 in 0.014s**. It also forwards without translating -- simultaneous tcpdump on both taps shows the same packet leaving the WAN leg with source still `10.12.64.2`. Bounded by lab topology: the "ISP" is the `vr1-dc1-uplink` libvirt NAT net on vcloud, not the internet, and D-125 egress isolation is unaffected. Discovered by the read-only dc1 edge assessment, `docs/audit/dc1-edge-assessment-20260802.txt`. | docs/audit/dc1-edge-assessment-20260802.txt | operator | OPEN -- closes when the edge's pf ruleset is restored (the repair regenerates the user DB from config.xml via the proven D-112(c) `local_user_set()` path, or donates it from the pristine `opnsense-26.7-nano.qcow2` in the same pool). Config.xml itself is INTACT -- fsck names every inode it deletes and `/conf/config.xml` is not among them. NOT rebuild-blocking; the exposure ends with the repair. | +| SEC-032 | 2026-08-02 | **dc0 edge credentials have been RESIDENT ON THE RACK, undisclosed, since 2026-07-20 -- the dc0 analog of SEC-015, which dc1 opened and remediated the same phase.** Found while staging the dc0 edge rebuild: `ls` (never a read) of `vr1-dc0-rack:~/vr1-dc0-creds/` shows `vr1-dc0-edge_ed25519` (the PRIVATE half, 432 B, 0600) and `opnsense-api.txt` (173 B, 0600), both dated 2026-07-20 -- i.e. left in place at the original edge build. dc1's build shredded both equivalents in the same phase and disclosed the transient as SEC-015; dc0's never did, and 13 days passed with nothing noticing. **THIS ALSO ANSWERS SEC-021(a), which asked for "a decision between RE-MINT and locating an off-jumphost copy": the off-jumphost copy is this one.** `creds-manifests/vm-secret-locations:123` already declares `rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/*` as an audited location, so this was visible to a `--remote` sweep and simply had not been reconciled -- the gap is reconciliation, not detection. **The 2026-07-20 `opnsense-api.txt` is now DEAD MATERIAL:** the rebuild destroyed the config.xml that held its hash, so it authenticates to nothing. It was deliberately NOT deleted (hard rule 1 -- a finding is logged, not actioned mid-step, and it is the physical evidence behind SEC-021(a)). The edge PRIVATE KEY was likewise left in place: `opnsense-bootstrap-apikey.sh` and `opnsense-set-interface-v4.sh` both SSH the edge DIRECT with no ProxyJump, and the edge LAN is reachable only from the rack, so rack residency is a structural requirement of the current tooling, not an oversight to be quietly cleaned up. **REMEDIATED IN PASSING:** SEC-021(a)'s access gap is closed -- a freshly minted key is now consolidated at `~/vr1-dc0-creds/opnsense-api.txt` on the jumphost (0600, 173 B, shape-verified 1x`key=`/1x`secret=` at 80 chars each, sha256-compared to the source, secret never printed). `creds-matrix.py` E1 for `dc0-edge-api` is consequently CLEAN where it previously read EXPECTED-BUT-ABSENT. | measured read-only during the dc0 edge rebuild, `docs/audit/dc0-edge-rebuild-20260802.txt`; SEC-015 (the dc1 precedent); SEC-021(a); `creds-manifests/vm-secret-locations:123` | operator | **OPEN -- three items, none actioned here.** (1) Rule on the dead 2026-07-20 `opnsense-api.txt` on the rack: shred (the dc1 precedent) or retain as SEC-021 evidence. (2) The `S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt'` finding is STILL RED and is now red for a DIFFERENT reason than when the operator accepted it on 2026-07-30 -- then the credential did not exist; now it exists and `creds-manifests/vr1-dc0.manifest` simply does not declare it. The manifest is DERIVED (`creds-matrix.py --render`), so this is a one-line regeneration, deliberately left to the operator rather than rendered inside a rebuild task. `creds-matrix.tsv:76`'s note key `n-dc0-edge-api-absent` and its `creds-matrix-notes.md:123` text are now factually stale for the same reason -- REPLACE the wording with the new invariant, never delete the row to go green (the note itself says so). (3) Decide whether rack-resident edge keys are the accepted steady state for BOTH DCs (making SEC-015's remediation the outlier) or whether the tooling should gain a ProxyJump path so the key never leaves the jumphost. Custody detail off-repo per D-069. | diff --git a/scripts/dc-egress-check.sh b/scripts/dc-egress-check.sh index e36cdd1..3846ddd 100755 --- a/scripts/dc-egress-check.sh +++ b/scripts/dc-egress-check.sh @@ -124,9 +124,19 @@ # Probed TWO ways on purpose: an ICMP-only probe would report a false outage on # a path that filters ICMP, and a TCP-only probe would miss a DNS-layer break. if [ "$EDGE_UP" -eq 1 ]; then + # DIAGNOSIS, added 2026-08-02. If A2 passed and A3 fails, the edge is UP and reachable + # but nothing comes back -- which is the signature of the edge FORWARDING WITHOUT + # TRANSLATING. Measured on vr1-dc1 that day: simultaneous tcpdump on both edge taps + # showed the same packet leaving the WAN leg with its LAN source address intact, so it + # was never masqueraded and no reply could return. A2 cannot see this (the edge does + # answer), and reporting it as a bare "anchor unreachable" sends the reader upstream + # chasing an outage that is not there. + NAT_HINT=" -- A2 PASSED, so the edge is up and reachable. An edge that answers but from \ +which nothing returns is the FORWARDS-WITHOUT-TRANSLATING signature (no outbound NAT); \ +confirm with tcpdump on both edge taps before looking upstream" ping -c2 -W3 1.1.1.1 >/dev/null 2>&1 \ && ok "A3 off-site ICMP anchor 1.1.1.1 answers" \ - || bad "A3 off-site ICMP anchor 1.1.1.1 does NOT answer" + || bad "A3 off-site ICMP anchor 1.1.1.1 does NOT answer${NAT_HINT}" C="$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 --noproxy '*' \ http://archive.ubuntu.com/ubuntu/dists/jammy/Release 2>/dev/null)" case "$C" in @@ -146,11 +156,17 @@ # asserts its upstream is REACHABLE. On 2026-08-01 the mirror served 200 all day # with its upstream gone, which is why both halves are needed. probe() { # $1 label, $2 url, $3 optional proxy + # `Snap-Device-Series: 16` is REQUIRED by the snap store API and is not optional + # politeness: measured 2026-08-02, the same URL returns 400 WITHOUT it both through the + # proxy AND direct, and 200 with it. This gate reported a healthy proxy as broken for + # exactly that reason -- a false FAIL in its own first live run, while the repo's + # dc-snap-proxy.sh (which sends it) returned PASS with a real payload. Harmless on the + # other probes, so it is sent unconditionally rather than special-cased. local lbl="$1" url="$2" px="${3:-}" code if [ -n "$px" ]; then - code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -x "$px" "$url" 2>/dev/null)" + code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H 'Snap-Device-Series: 16' -x "$px" "$url" 2>/dev/null)" else - code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 --noproxy '*' "$url" 2>/dev/null)" + code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 -H 'Snap-Device-Series: 16' --noproxy '*' "$url" 2>/dev/null)" fi case "$code" in 200|30[0-9]) ok "A4 $lbl reachable (HTTP $code)" ;; diff --git a/tests/dc-egress-check/run-tests.sh b/tests/dc-egress-check/run-tests.sh index 074268f..841e232 100755 --- a/tests/dc-egress-check/run-tests.sh +++ b/tests/dc-egress-check/run-tests.sh @@ -131,6 +131,30 @@ && ok "T14 A4 unrecognised HTTP code FAILS (A3 healthy, so only A4 can fail it)" \ || bad "T14 A4 unrecognised code not caught" "$OUT" +# T15 the snap probe must send `Snap-Device-Series: 16`. ADDED 2026-08-02 after the gate's +# FIRST LIVE RUN produced a false FAIL: the store API returns 400 without that header, both +# through the proxy and direct, and 200 with it -- while the repo's dc-snap-proxy.sh (which +# sends it) returned PASS with a real payload. The gate reported a healthy path as broken. +# Asserted on the SCRIPT here because the fixture curl cannot observe headers; the live +# behaviour is what the 2026-08-02 capture records. +# ASSERT ON THE CODE, not on any occurrence: a first cut grepped the whole file and passed +# on the COMMENT that explains the header, so deleting it from both curl invocations left +# the suite green. Count only curl lines that carry it -- there are two (proxied, direct). +t15n=$(grep -c "curl .*-H 'Snap-Device-Series: 16'" "$S") +[ "$t15n" -eq 2 ] \ + && ok "T15 both curl probes send the header the store API requires" \ + || bad "T15 expected 2 curl probes carrying Snap-Device-Series, found $t15n" + +# T16 when A2 PASSES and A3 fails, the output must name the forwards-without-translating +# signature. Measured on vr1-dc1 2026-08-02: the edge answered, packets left the WAN leg +# with their LAN source intact, and nothing returned. Without this, the reader is sent +# upstream chasing an outage that is not there. +mkfix "$TMP/t16" yes edge up down ok; run "$TMP/t16" check dc0 +{ printf '%s' "$OUT" | grep -q "A2 edge 10.12.4.1 ANSWERS"; } \ + && { printf '%s' "$OUT" | grep -qi "FORWARDS-WITHOUT-TRANSLATING\|no outbound NAT"; } \ + && ok "T16 edge up + anchor dead names the no-NAT signature, not a bare 'unreachable'" \ + || bad "T16 no-NAT diagnosis missing" + # T13 usage without an action REFUSES. OUT="$(bash "$S" 2>&1)"; RC=$? [ "$RC" -eq 2 ] && ok "T13 no action REFUSES (rc=2)" || bad "T13 no action not refused (rc=$RC)"