← board

The quiet bench has produced nothing for two days, and never once on seven

The premise being checked

The owner's, and it is a good one: "if O gating is quiet and just waits for results, we will have numbers." The machinery for that exists — idle_bench in the watcher, BENCH_LEVELS, bench.tsv, 17178 rows. So the question is only whether it is running.

Measured

Rows by host, whole archive:

host rows status
borg 11242 retired
plexus 3747 last row 2026-08-28
xeon 2185 retired
seven 0 never produced one

seven.jsonlast_bench: {date: 2026-08-30T11:11:19Z, rc: 1, rows: 0}. The failure is recorded and nobody read it. plexus.json → last bench 2026-08-28T21:10, two days ago.

So of the two live hosts, one has not benched in two days and the other has never benched at all.

Why, and the cause is a guard doing its job

BENCH_QUIET_LOAD_FRAC = 0.60 — start only when per-core load1 is under 60%, with a 10 s cap on waiting. BENCH_CPU_WALL_MAX = 1.06 — discard any run whose wall ran more than 6% ahead of its own child cpu time, because the process was descheduled and the timing is contaminated. BENCH_EXTRA_TRIES = 5 spare attempts. Run out and you get rc=1, rows=0.

Every one of those is the right call in isolation. A discarded run is a spike thrown away rather than averaged in, which is exactly the discipline that made the DCE row trustworthy while the min-of-3 sweep's zeros were not.

The trap is structural, not a bug in the guard: the box with idle time is the box that is sweeping, and sweeping is what fills the idle time. seven runs the watcher at load 6-13 on 24 threads. The bench waits 10 s for quiet, does not get it, burns its spare attempts on contaminated runs, and exits 1. A correct instrument, refusing to lie, producing nothing — and its refusal recorded in a field nobody reads.

What this means for the promotion rule, precisely

Two different things were being conflated by "we will have numbers":

What to do — three, and the third is the real one

  1. Read rc and rows. last_bench.rc == 1 or rows == 0 should surface in trackt.py health / twatch --status like any other red. A silent instrument failure is the most expensive kind, and this one has been silent since the box was added.
  2. Give the bench a genuinely quiet window rather than an idle slot it will never win: pause sweeping for its duration, or schedule it when the fleet is deliberately parked. It is minutes, not hours.
  3. Prefer flag-shaped answers to margin-shaped ones. The transferable lesson from the same evening: the one row that survived a bad sweep survived because it was settled by a flag, not by a margin — decided by construction rather than a difference of means. Where a pass's effect can be made structural (does this call site emit the runtime call or not) the answer costs nothing and does not care about load. Not every question can be reshaped this way, but the ones that can should be, because the boxes will never be quiet.

Also, under the new charter

BENCH_LEVELS = ("-O0", "-O2", "-O3"). The charter (decided/decide-the-o-level-charter) now defines -O1 debug-safe and -O4 research. -O1 is a level users will actually select and should be benched with the rest; -O4 should not ride every run, matching its slower sweep cadence. Not urgent — nothing benches today — but it is the same edit.


The archive is smaller than it looks: 17178 rows, 90 on verifiable hardware

Checked when the owner said he would repair borg. Check this before spending a day on it, because two of the three facts point somewhere other than the name.

1. There is no hardware record for borg at all. meta/hosts.json records xeon, plexus, seven — and no borg. Its 11242 rows, 65% of the whole archive, carry hw_fp: None because the fingerprint field did not exist when it last benched (2026-07-31). We do not know what CPU produced the largest block of numbers we have.

2. borg.json says it was already retired INTO plexusretired_at: 2026-08-12T07:46:31Z, retired_into: plexus, retired_kind: migration. xeon was retired into plexus the same way on 2026-08-07, and plexus's own record says renamed_from: xeon. So plexus IS the former xeon, and borg's identity was administratively merged into it. 192.168.1.99, borg's address in two-box-protocol.md, does not respond.

If borg is a distinct physical machine sitting somewhere, none of that stops it being repaired — but "restore borg and the baseline continues" is not true, and the tstate record will read as a duplicate identity unless it is un-retired deliberately.

3. The fingerprint over-partitions, and this is the fixable part. plexus's fp changed three times on the same CPUfc06409301419e379b78622c3f2b86ea7416 — and the only differences across those epochs are:

epoch 1 epoch 2 epoch 3
kernel 7.0.0-28 7.0.0-29 7.0.0-30
mem_total_kb 63424944 63423856 63423848

A kernel upgrade and a few KB of reported RAM. Same CPU, same sockets, cores, threads, governor, turbo. Taken strictly, that means only 90 of plexus's 3747 rows are on the current epoch.

So the honest count for the whole archive is: 17178 rows, of which 90 are on a hardware fingerprint we can currently verify. It looks like a longitudinal dataset and behaves like a fortnight of one.

The fix is to split the key, not to loosen it. fp currently conflates hardware identity (CPU, sockets, cores, threads, memory class) with software environment (kernel, and whatever else moves on an apt upgrade). Partition hard on the first, record the second as a column. A kernel bump should annotate a row, not orphan it. Do that and most of plexus's history becomes usable again; leave it and every routine upgrade silently discards the baseline.

A third box that is NOT sweeping remains the right structural fix for the guard problem above — whichever machine it turns out to be.

Deprioritised 2026-09-02 — the Track T tooling backlog was cut as a pile

This ticket is not being called wrong. It was moved as part of a pile, not judged individually, and nothing here disputes its finding.

Owner decision. 73 of the 74 open track: T tickets were filed between 2026-08-31 and 2026-09-02, 58 on one day. The pile was too large to work through and returned almost nothing, and a ticket nobody will fix does not sit neutrally — it stays in the ranker forever at zero value, which is the argument CLAUDE.md already makes for a terminal folder over a low prio.

Four were kept in the ranker on a purely structural test — an active umbrella or a hard blocked-by: edge from live work: umbrella-one-full-tier-run-with-no-red-tier, feature-t-freebsd-image-and-runner, and the two regression-test-core-* reds that block the umbrella.

Kept, not deleted, for two reasons: so the finding is not rediscovered and refiled from scratch by the next agent who trips over it, and so it can be pulled back if what it touches becomes load-bearing.

To revive it: move it to the owning lane's backlog, set status: backlog, and say in the ticket WHAT CHANGED to make it matter now. Restoring it because it reads well is how the pile comes back.