The quiet bench has produced nothing for two days, and never once on seven
The premise being checked
The owner's, and it is a good one: "if O gating is quiet and just waits for
results, we will have numbers." The machinery for that exists —
idle_bench in the watcher, BENCH_LEVELS, bench.tsv, 17178 rows. So the
question is only whether it is running.
Measured
Rows by host, whole archive:
| host | rows | status |
|---|---|---|
| borg | 11242 | retired |
| plexus | 3747 | last row 2026-08-28 |
| xeon | 2185 | retired |
| seven | 0 | never produced one |
seven.json → last_bench: {date: 2026-08-30T11:11:19Z, rc: 1, rows: 0}.
The failure is recorded and nobody read it. plexus.json → last bench
2026-08-28T21:10, two days ago.
So of the two live hosts, one has not benched in two days and the other has never benched at all.
Why, and the cause is a guard doing its job
BENCH_QUIET_LOAD_FRAC = 0.60 — start only when per-core load1 is under 60%,
with a 10 s cap on waiting. BENCH_CPU_WALL_MAX = 1.06 — discard any run whose
wall ran more than 6% ahead of its own child cpu time, because the process was
descheduled and the timing is contaminated. BENCH_EXTRA_TRIES = 5 spare
attempts. Run out and you get rc=1, rows=0.
Every one of those is the right call in isolation. A discarded run is a spike thrown away rather than averaged in, which is exactly the discipline that made the DCE row trustworthy while the min-of-3 sweep's zeros were not.
The trap is structural, not a bug in the guard: the box with idle time is the box that is sweeping, and sweeping is what fills the idle time. seven runs the watcher at load 6-13 on 24 threads. The bench waits 10 s for quiet, does not get it, burns its spare attempts on contaminated runs, and exits 1. A correct instrument, refusing to lie, producing nothing — and its refusal recorded in a field nobody reads.
What this means for the promotion rule, precisely
Two different things were being conflated by "we will have numbers":
- Correctness numbers accrue for free.
fullandoptrun anyway; a promotion's proof (self-host + all tests passed) arrives without anyone waiting. The owner's rule is unaffected. - Value numbers do not accrue at all. Does the pass still fire, does it
still pay is a bench question, the bench needs a quiet box, and no box is
quiet. This is the same distinction as
optdiffproving "not wrong" but never "works" — and it is where the missing half lives.
What to do — three, and the third is the real one
- Read
rcandrows.last_bench.rc == 1orrows == 0should surface intrackt.py health/twatch --statuslike any other red. A silent instrument failure is the most expensive kind, and this one has been silent since the box was added. - Give the bench a genuinely quiet window rather than an idle slot it will never win: pause sweeping for its duration, or schedule it when the fleet is deliberately parked. It is minutes, not hours.
- Prefer flag-shaped answers to margin-shaped ones. The transferable lesson from the same evening: the one row that survived a bad sweep survived because it was settled by a flag, not by a margin — decided by construction rather than a difference of means. Where a pass's effect can be made structural (does this call site emit the runtime call or not) the answer costs nothing and does not care about load. Not every question can be reshaped this way, but the ones that can should be, because the boxes will never be quiet.
Also, under the new charter
BENCH_LEVELS = ("-O0", "-O2", "-O3"). The charter
(decided/decide-the-o-level-charter) now defines -O1 debug-safe and
-O4 research. -O1 is a level users will actually select and should be
benched with the rest; -O4 should not ride every run, matching its slower
sweep cadence. Not urgent — nothing benches today — but it is the same edit.
The archive is smaller than it looks: 17178 rows, 90 on verifiable hardware
Checked when the owner said he would repair borg. Check this before spending a day on it, because two of the three facts point somewhere other than the name.
1. There is no hardware record for borg at all. meta/hosts.json records
xeon, plexus, seven — and no borg. Its 11242 rows, 65% of the whole
archive, carry hw_fp: None because the fingerprint field did not exist when
it last benched (2026-07-31). We do not know what CPU produced the largest block
of numbers we have.
2. borg.json says it was already retired INTO plexus — retired_at: 2026-08-12T07:46:31Z, retired_into: plexus, retired_kind: migration. xeon
was retired into plexus the same way on 2026-08-07, and plexus's own record says
renamed_from: xeon. So plexus IS the former xeon, and borg's identity was
administratively merged into it. 192.168.1.99, borg's address in
two-box-protocol.md, does not respond.
If borg is a distinct physical machine sitting somewhere, none of that stops it being repaired — but "restore borg and the baseline continues" is not true, and the tstate record will read as a duplicate identity unless it is un-retired deliberately.
3. The fingerprint over-partitions, and this is the fixable part. plexus's
fp changed three times on the same CPU — fc0640930141 →
9e379b78622c → 3f2b86ea7416 — and the only differences across those epochs
are:
| epoch 1 | epoch 2 | epoch 3 | |
|---|---|---|---|
| kernel | 7.0.0-28 | 7.0.0-29 | 7.0.0-30 |
| mem_total_kb | 63424944 | 63423856 | 63423848 |
A kernel upgrade and a few KB of reported RAM. Same CPU, same sockets, cores, threads, governor, turbo. Taken strictly, that means only 90 of plexus's 3747 rows are on the current epoch.
So the honest count for the whole archive is: 17178 rows, of which 90 are on a hardware fingerprint we can currently verify. It looks like a longitudinal dataset and behaves like a fortnight of one.
The fix is to split the key, not to loosen it. fp currently conflates
hardware identity (CPU, sockets, cores, threads, memory class) with
software environment (kernel, and whatever else moves on an apt upgrade).
Partition hard on the first, record the second as a column. A kernel bump should
annotate a row, not orphan it. Do that and most of plexus's history becomes
usable again; leave it and every routine upgrade silently discards the baseline.
A third box that is NOT sweeping remains the right structural fix for the guard problem above — whichever machine it turns out to be.
Deprioritised 2026-09-02 — the Track T tooling backlog was cut as a pile
This ticket is not being called wrong. It was moved as part of a pile, not judged individually, and nothing here disputes its finding.
Owner decision. 73 of the 74 open track: T tickets were filed between
2026-08-31 and 2026-09-02, 58 on one day. The pile was too large to work through
and returned almost nothing, and a ticket nobody will fix does not sit neutrally
— it stays in the ranker forever at zero value, which is the argument CLAUDE.md
already makes for a terminal folder over a low prio.
Four were kept in the ranker on a purely structural test — an active umbrella or
a hard blocked-by: edge from live work:
umbrella-one-full-tier-run-with-no-red-tier,
feature-t-freebsd-image-and-runner, and the two regression-test-core-* reds
that block the umbrella.
Kept, not deleted, for two reasons: so the finding is not rediscovered and refiled from scratch by the next agent who trips over it, and so it can be pulled back if what it touches becomes load-bearing.
To revive it: move it to the owning lane's backlog, set status: backlog,
and say in the ticket WHAT CHANGED to make it matter now. Restoring it because it
reads well is how the pile comes back.