Nothing checks that two hosts run the same suite

The measurement

library_candidates/ in the watcher trees, 2026-08-30:

Four of the five are the NilPy corpus stack. Every Track T run on plexus silently omitted their jobs and reported GREEN. Fixed by hand with tools/install_lib_candidates.sh; both trees are at 25 now. Nothing stops it recurring, and nothing would have told us.

Why it is invisible, and why that is not a bug in the skip design

A skip is deliberately passlike (PASSLIKE = ("pass", "skip")), and testmgr.py argues the case well: a RED for an absent corpus is strictly worse than a SKIP, because it masks a future real red. That reasoning is correct and this ticket does not propose changing it.

The report is honest too — it prints its own banner:

COVERAGE: N job(s) DID NOT RUN on this box ... they are scored passlike, so they are invisible in the verdict above — a RED here speaks for the jobs that ran, not for the suite.

and skip_holes is in every runs-*.ndjson row. The instrument reports the hole. The gap is that nothing compares two hosts, and nothing consumes the count when a green is cited as proof. 1.4% of GREEN runs in the archive (44 of 3253) carry at least one hole.

Why it matters now, specifically

The owner ruled 2026-08-30 that self-host + all tests passed = proof, and the reasoning is that the compiler and target set are complex enough to constitute one. That is sound exactly to the degree the suite actually runs. A -O2 promotion citing a green from a box missing five corpora is citing a smaller suite than the reader will assume, and nothing in the citation says so.

What to build — three, in order of value

1. A parity check. tstate already knows what each host ran. Compare the job sets of the last full run per host and report any job present on one and absent on another. This finds the class, not the instance.
2. A proof-grade flag on the run row. skip_holes == 0 and tier == "full" is the property a promotion should cite. Name it once in the archive rather than have every consumer re-derive it — and re-deriving it is what nobody did.
3. A fetch-on-start for the watcher, or a loud refusal to start with an incomplete corpus. Prefer loud refusal at start, not per job: the per-job skip is correct behaviour and should stay.

What NOT to do

Do not make a corpus-absent skip red. testmgr.py already carries that argument and it is right. The fix is knowing which suite you ran, not failing runs for a box's fetch state.

A note for whoever takes this

The opt tier is disjoint from full, so -O3 is untested by full runs entirely — the report says so in a second banner. That means promoting a pass from -O3 to -O2 increases the coverage it gets, since the full tier then exercises it at the default level. Worth stating in the promotion ticket: the promotion is not only a speed change, it moves the pass into the suite that actually runs.

THE ACCEPTANCE CRITERION IS capability × job, NOT job (frankT + frank-user, 2026-08-30 evening)

A job-set diff — the obvious fix, and the one I agreed to — cannot see the second half of this bug. Measured tonight while clearing a red T guard.

tools/csmith_target_devtest.py asserted flatly that this box has no ILP32 oracle because gcc -m32 compiles but does not link. That is true on the box it was written on. On plexus gcc -m32 links, and probe_oracle("arm32") returns cc=['gcc','-m32'], kind=datamodel. So:

• both hosts run csmith-fuzz#arm32;
• one compares ILP32 checksums against an oracle, the other silently does not;
• the job list, the job count and the verdict are identical on both.

A job name is a promise, not a description of what ran (frank-user). csmith-fuzz#arm32 names the INPUT. Both hosts keep the promise. The artifact everyone would compare is the one place the difference is guaranteed to be absent.

The parity gap here is a toolchain capability, not a corpus, so no census of library_candidates — the thing that found the original 20-vs-25 — would ever have surfaced it. Guard fixed at 658f78ebc; the guard is not the bug.

The cheap implementation, and it is persistence rather than a new prober

Do not build a capability prober. probe_oracle already computes the vector and drops it after printing — the same shape as sync.sh proving a commit was on origin and discarding the sha, and as skip_summary counting coverage holes without naming them. Three instances in one day, and in all three the fix was persistence.

So: emit the vector into the runs-<host>.ndjson row that is already written, and the parity check becomes a diff of two rows rather than a new subsystem. The row construction is a single json.dumps in twatch.py (near skip_hole_jobs, which was added the same way and for the same reason), so the schema half is a few lines.

The one question I did NOT settle, because it has a real cost

Where the vector is computed, and what it costs per run. probe_oracle does actual compiles, and the fields that matter beyond it (does gcc -m32 link, which qemu-* exist, is fpc present and with which widestring manager) are each a small subprocess. Paying that once per run is cheap in absolute terms — and it lands on the box that is the binding constraint on sweep rate, which this repo has measured as what sets the median-8 commit gap.

Two shapes, and whoever takes this should choose deliberately rather than by default:

1. Compute once at watcher START, cache for the process lifetime, and stamp every row from the cache. Nearly free; goes stale if a package is installed mid-session, which is exactly how the original 20-vs-25 arose.
2. Compute per run. Always current, costs a handful of subprocesses per sweep on the constrained box.

My recommendation is (1) plus a re-probe whenever the clone is re-seeded, because the staleness window then matches the window in which the tree itself could change — but it is a cost decision on shared hardware and it is stated here rather than made in passing.

Deprioritised 2026-09-02 — the Track T tooling backlog was cut as a pile

This ticket is not being called wrong. It was moved as part of a pile, not judged individually, and nothing here disputes its finding.

Owner decision. 73 of the 74 open track: T tickets were filed between 2026-08-31 and 2026-09-02, 58 on one day. The pile was too large to work through and returned almost nothing, and a ticket nobody will fix does not sit neutrally — it stays in the ranker forever at zero value, which is the argument CLAUDE.md already makes for a terminal folder over a low prio.

Four were kept in the ranker on a purely structural test — an active umbrella or a hard blocked-by: edge from live work: umbrella-one-full-tier-run-with-no-red-tier, feature-t-freebsd-image-and-runner, and the two regression-test-core-* reds that block the umbrella.

Kept, not deleted, for two reasons: so the finding is not rediscovered and refiled from scratch by the next agent who trips over it, and so it can be pulled back if what it touches becomes load-bearing.

To revive it: move it to the owning lane's backlog, set status: backlog, and say in the ticket WHAT CHANGED to make it matter now. Restoring it because it reads well is how the pile comes back.