← board

The job identifier is a selector, and it is being read as a label

The two jobs one string is doing

tools/testmgr.py, job_selector():

    # Prefer the first source it compiles (stable across renumbering); fall
    # back to the positional name for jobs that name no source
    return "%s#src:%s" % (job.target, srcs[0])

As a selector that is correct and the docstring's reason is a good one. A --job argument must survive somebody inserting a recipe line, and srcs[0] does while #00 does not.

As a label it is arbitrary. srcs[0] is decided by Makefile prerequisite order, so every $(COMPILER)-dependent job inherits that target's prerequisites at the head of its list, and a job about object emission gets named after a shell script.

The identifier is not an internal key. It is the row a human reads in a tstate report, the string a ticket gets titled after, the thing a coordinator hands a peer as an assignment, and the key a still_red comparison is made on.

Three instances in one report, from three sessions

devdocs/progress/tstate/reports/20260906T183724Z-6d04b14-seven.md:

printed identifier what it is about
test-emit-obj#src:tools/compiler_srchash.sh object emission; srchash appears nowhere in the failure
test-zlib#src:tools/compiler_srchash.sh the zlib corpus being absent
test-fpjson#src:tools/install_lib_candidates.sh a duplicate-definition warning in test/fpjson/testutils.pas

Three sessions reached it independently on 2026-09-06 from three different rows, which is what makes it positional rather than a coincidence of one recipe.

The evidence is already recorded, one field away

This is the part that changes what the fix is. testmgr.py already has:

So the routing evidence exists, is correct, and is already rendered — and the identifier does not use it. A reader who scrolls to the bullet is served. A reader who reads the row, or a coordinator who copies it into an assignment, is not.

The reason field does not rescue it either

The stored detail is a fixed-width tail of captured output, not a failure list. test-zlib's reads corpus absent: library_candidates/zlib. while sitting under a FIXED heading. tools-devtest#00's named three twatch_* progress lines — printed before each devtest runs, for files that may well have passed — cut mid-word, and it could never have named the devtest that actually failed, which sorts earlier in the glob. Absence from that field is uninformative in both directions, and a reader who takes it as "the failing sources" gets a confident wrong answer.

The fork, and the measurement that settles it

Do NOT change what job_selector() returns — that would break --job stability, which is the property it was written for.

The question is what the DISPLAYED identifier should be for a red, given that step_src is already sitting beside it:

The discriminator is a census nobody has taken: across one full tier, for how many multi-source jobs is srcs[0] actually the subject, and for how many reds is step_src non-empty? If step_src is usually populated, option one is strictly better and nearly free. If it is usually "" — a readelf assertion, a bare binary run — then the identifier cannot be repaired from it and option two is the honest answer. Take the census before choosing.

What must not happen

Do not reorder Makefile prerequisites so the interesting source sorts first. That makes the identifier right by accident, one Makefile edit away from being wrong again, and moves a real dependency order to satisfy a display string.

chore-t-split-lib-test-into-jobs-that-name-what-failed [T p45, low-prio] is about SPLITTING one bundled job so each piece names its own source. This is about the identifier every multi-source job already has. Fixing either does not fix the other.

The third witness — and the number that makes this worse than a broken name

frankH, reading the job listing for the full matrix (command elided; this file is scanned by the suite guard):

test-zlib#00 carries the SAME first source as the test-emit-obj row: tools/compiler_srchash.sh compiler/.pascal26.fixedpoint +4. Two different targets, two different failure modes, one identifier — so #src:tools/compiler_srchash.sh cannot be describing either of them. The neighbouring row in the same listing, lib-test#87, leads with test/lib_zlib.pas, which is a genuinely descriptive first source; so the identifier is right about a third of the time, which is worse than never, because the times it is right teach you to trust it.

That last clause is why this is a bug and not a cosmetic complaint, and it is CLAUDE.md's own rule arriving from the data: an 80%-accurate name is worse than a 0%-accurate one — the part you sample confirms it.

LAYER TWO: the REASON is not the thing either, and it is the more dangerous half

frankD, closing test-emit-obj#src:tools/compiler_srchash.sh (green at HEAD 2699f5769, fixed above the tree by fc000b076).

job_reason() returns the log's TAIL, by design, and its docstring gives a good argument for it: a signature list goes stale silently and then reports nothing for shapes nobody has met, while a tail is true for every shape. twatch.py:stub_reason() then cuts that to CASCADE_REASON_MAX = 200 for a cascade bullet. For this row the three surviving fragments were:

ok: $TMP [code=470952B …] | test-emit-obj: an i386 object's file-scope
initialisers run under a gcc -m32 main | ok: $TMP [code=186531B …]…

All three are success echoes. The middle one is the Makefile's own echo on the line after the assertion it names — it prints only once that check has PASSED. It became "the failing thing" in the srchash write-up because it was the only human-readable sentence in the string; the other two are compiler statistics that do not look like a subject. The real failure was the assertion immediately after the third fragment: i386 .text still has $abs absolute relocation(s).

So the write-up produced to teach "the row name does not name the failing thing" corrected name → subject and then made the identical error with reason → subject, one layer up — and it propagated to three sessions.

A REASON IS MORE DANGEROUS THAN A NAME. A name is obviously an identifier. A reason looks like content — and a truncated tail is the worst of the three, because it reads as a finished sentence about the subject while being a receipt for the last step that SUCCEEDED.

The rule is one sentence: read the last fragment as "everything up to here worked" and go to the recipe line AFTER it.

This layer is NOT a proposed patch. The tail is the right thing for job_reason() to return, and the untruncated 400-char form in tstate/<host>.json may well contain the error line the 200-char cut removed. The failure is in the READING. It belongs here because it is the same defect one level up — the identifier and the reason both describe something other than the defect — and because a fix to the identifier that leaves the reason un-annotated moves the trap rather than removing it. Written up in full in regression-test-debug-g-compiler-srchash-2.md, with the superseded table row marked inline so a grep-lander cannot take the old sentence.

A fourth case, different mechanism, same shape

test-zlib#src:tools/compiler_srchash.sh appeared under FIXED with the stored reason corpus absent: library_candidates/zlib. — traced by frankS to CORPUS_RE's character class swallowing a sentence-ending full stop, so the row self-skipped on a box that has had the corpus since 2026-08-29 and the skip was hiding a green. Not srcs[0], and worth counting anyway: four rows in one report where the printed identifier, the stored reason, or both are about something other than the defect.

Adjacent, and deliberately not folded in

frankD, on the row they just cleared: it can go green by accident — one extra unrelated local in PXXIoCheck moves code off -0x10 and the absolute relocation count goes 1 → 0 with the defect untouched. They confirmed this particular green is real by checking the load-bearing condition in the artefact (8b 45 f0 still at 28f22, store converted anyway, 0 absolute vs 587 PC-relative). A row whose verdict moves with unrelated stack layout is a different family from a row whose name is wrong — that is a guard whose expected value collides with an accidental one, not an identifier problem — so it is recorded here as a pointer and not merged in. It wants its own ticket against the assertion, phrased as a relation rather than a count.

2026-09-06, frankuser — the misnomer is PERSISTED IN SLUGS, not just printed

A fourth session hit layer 1 the same evening, and the new fact is that the bad name does not stay in a log line. It becomes the ticket slug, which is the search key.

Nine auto-filed regression tickets on disk are named #src:tools/compiler_srchash.sh, across seven distinct jobstest-debug-g (×2), test-cjson, test-fgl, test-lua, test-lua-cross (×2), test-emit-obj, test-sqlite-threads-aarch64. One script, seven unrelated subjects, because it sits at srcs[0] for everything that depends on $(COMPILER).

The cost, measured rather than argued. tools/compiler_srchash.sh was genuinely edited tonight (79264f396 — a bash shebang meant the stamp guard compared two absences and called them equal). Grepping the backlog for srchash afterwards returns nine regression tickets, and every one reads as a possible fallout from that fix. They are not: the newest was found 2026-09-06T04:48:59Z, seventeen hours before the change. Two tool calls to establish that, and the discriminator was a timestamp, not anything about the subject.

This is the specific way an 80%-accurate name is worse than a 0%-accurate one. A log line is read once by the person who ran the job and has the run in front of them. A slug is read by everyone afterwards, out of context, forever — and it is what grep matches, so it decides who finds the ticket as well as who is misled by it. The nine will still be answering the wrong question long after the run that minted them is gone.

Consequence for the fix, not a new ask: whatever layer 1 lands on, the slug generator should take it too. The archive is where a wrong name compounds, and renaming these nine afterwards is cheap only while they are still open.

The ticket's own line holds and this is an instance of it — right about a third of the time is what teaches you to trust it.

2026-09-06, frankH — the label manufactures a cluster that does not exist

Measured while looking for the reds behind [[umbrella-one-full-tier-run-with-no-red-tier]], and worth adding because it is a cost this ticket did not yet name: the label does not merely fail to identify a job, it invents a shared cause between unrelated ones.

Grouping today's 54 full runs on seven by red job name produced what looked like a six-member family, all naming tools/compiler_srchash.sh, going red together (test-emit-obj + test-zlib in 42 of 54, others joining):

test-zlib#src:tools/compiler_srchash.sh
test-emit-obj#src:tools/compiler_srchash.sh
test-debug-g#src:tools/compiler_srchash.sh
test-lua#src:tools/compiler_srchash.sh
test-lua-cross#src:tools/compiler_srchash.sh
test-fgl#src:tools/compiler_srchash.sh

Co-occurrence that tight reads as one condition with six faces, and the shared name says what it is. It is neither. The step those jobs share PASSES — the logs read self-host fixedpoint: verified — 1 round(s), ... (stamp read back; sources match it) — and each job then fails further along its own line for its own reason (FAIL 00040.c under test-c-conform for one, the gcc oracle build for another).

They share a name because a job is named after its FIRST source, and all of them depend on $(COMPILER), whose recipe's first prerequisite is tools/compiler_srchash.sh. One report (20260906T161630Z-29c4052-seven.md) has eleven jobs carrying it, five of them test-c-conformance*#shardN/6 — so the collision is not even confined to one suffix shape, and a reader filtering on #src: sees a subset without being told one exists.

Why the co-occurrence is the trap rather than the tell. Jobs sharing a label also share a prerequisite, so they genuinely do fail together — whenever the box is slow, or the tier is wide, the same set lights up. The correlation is real and its cause is scheduling, not a defect. So the usual defence ("check whether they move together") CONFIRMS the wrong reading here.

The discriminator is the log line, not the name and not the correlation: devdocs/dev/handbook-rationale.md's "a reason is more dangerous than a name", one level up — here the name was more dangerous than the reason, because the reason was three fields to the right in the same string and the name was in the summary. Reading the recipe text the ticket quotes is not enough either; that text is the same for all six.

Cost this time: one wrong root cause, caught before it was filed. The auto-filed [[regression-test-debug-g-compiler-srchash-2]] is one face of this and its own banner already says the job "names a MECHANISM rather than a subject" — which is this ticket, arrived at independently by the auto-filer's heuristic.

Correction to the section above, from frankuser's, landed the same hour

The section above argues "the step those jobs share PASSES" from the log line self-host fixedpoint: verified — ... (stamp read back; sources match it). Read against frankuser's finding, that sentence was not evidence of much: 79264f396 fixed a bash shebang in tools/compiler_srchash.sh that made the stamp guard compare two absences and call them equal. So for the runs I read, "sources match it" may have been a vacuous pass — the guard agreeing with itself about nothing, which is this handbook's "a guard that cannot fail is not a guard, and it prints PASS", sitting underneath a section about names.

The conclusion survives, by the other route, and that route is the one to quote. The six jobs fail at DIFFERENT later steps with different messages — FAIL 00040.c under test-c-conform for one, the gcc oracle build for another. That is what makes them unrelated. Their shared step's verdict is not load-bearing and should never have been the first thing offered.

Two readings that agree are not two sources unless they can fail differently, and "the shared step passed" and "the shared step is not the cause" fail together. The differing failure messages fail differently from both.

Quantified 2026-09-08: the src:tools/compiler_srchash.sh family, 18 jobs deep

The clearest live instance of this ticket, measured across the whole seven archive:

NEW-RED events carrying src:tools/compiler_srchash.sh 71
distinct JOB names they were attributed to 18
distinct shas 21
days 09-01 (14), 09-02 (6), 09-03 (1), 09-05 (38), 09-06 (7), 09-07 (1), 09-08 (4)
regression tickets auto-filed for it 12 — 10 in done/, 2 open, and both open ones are suffixed -2, i.e. REPEAT filings of a slug that already closed once

The 18: test-arm32, test-asm, test-c-abi-mixed-link, test-cjson, test-core, test-debug-g, test-emit-obj, test-fgl, test-lua, test-lua-cross, test-smoke, test-sqlite-threads-{aarch64,arm32,i386,x86_64}, test-threads, test-uforth, test-zlib.

That list is most of the tier, which is the whole argument. A defect that can present as test-zlib on Friday and test-arm32 on Monday is not a property of zlib or of arm32. The failing STEP is a guard prepended to the job — livesrc=$(tools/compiler_srchash.sh) compared against the srchash line of compiler/.pascal26.fixedpoint — so the job name in the identifier is whatever was running when the guard fired, and the ticket generator promotes it to the subject of a sentence.

The cost is not just noise, it is MISATTRIBUTION ONTO A SEAT. Each filing carries bad <sha>, last good <sha>, N commit(s) in range, so it points at whoever happened to land in that window. On 2026-09-08 the two new ones landed at 5506794317a7 — a Track P for-in change touching pasparser_stmt.inc and symtab.inc, which cannot affect a source-hash guard. The previous run's test-arm32 instance at b29428afe251 closed itself as "job green again" one run later.

What is NOT established here

The exact condition that fires on seven each time. One captured log shows stamp sources: deadbeef against a real tree hash, plus stamp 214 files, tree 215 — the stamp did not describe the tree at all. compiler/.pascal26.fixedpoint is gitignored and therefore per-checkout, written at build time, so the live-vs- stamp disagreement is consistent with the watcher's binary predating the tree it just pulled — the same hazard CLAUDE.md names for every seat ("rebuild after any sync touching compiler/** before you measure"), occurring inside the harness rather than in a seat.

Naming that mechanism would need seven's tree state at those moments, which this seat does not have. Recorded as a bounded observation so whoever takes it starts from the guard and the harness, not from eighteen innocent jobs.

The two cheapest rows in the archive, and they need no domain knowledge

frankS pointed out that the argument above is stronger than it was written, and checking it made it stronger again. The two consecutive srchash NEW-RED runs of 2026-09-08 landed on commits that provably cannot change the hashed file set:

sha what it changes files in the five hashed globs
5506794317a7 ONE line of ONE .md — sync.sh's PENDING-COMMIT fill-in 0
b29428afe251 tstate/runs-seven.ndjson + seven.json — the watcher's OWN bookkeeping 0

The stamp hashes exactly five globs — compiler/compiler.pas, compiler/*.inc, compiler/builtin/*.pas, lib/rtl/*.pas, lib/asmcore/*.pas. Neither commit touches one. So the stamp 214 files, tree 215 perturbation cannot have come from the commit range, and the identifier is not merely mislabelled — it is pointing at a window whose contents are incapable of causing the failure.

A tier that reds on a markdown one-liner has already told you the verdict is not about the commit, and the second row is better still: the watcher reddened on its own archive write. No knowledge of Pascal, of the for-in rules, or of what either job tests is needed to read either row.

This also closes off a plausible-sounding lead rather than opening one: a one-tracked-file commit looks like a clean probe for an off-by-one-file stamp comparison, and it is not, because the file it changes is outside the hashed set entirely. Whatever adds the 215th file is in the CHECKOUT or the harness — an untracked stray, a build artefact, a partially-written stamp — not in any commit. Start there.

2026-09-09 — a clean positive control for this ticket, unasked-for (frankuser)

Track T auto-filed test-sqlite-threads-aarch64#src:tools/compiler_srchash.sh with bad=e8020484be2c. That commit is:

 devdocs/progress/LOGBOOK.md | 1 +
 1 file changed, 1 insertion(+)

One line, in the logbook. It cannot compile, link, thread, or reach aarch64. This is the cleanest instance of the mislabel yet recorded, because the null hypothesis needs no argument: the commit is physically incapable of causing the failure attributed to it, so the label is wrong with certainty rather than on balance.

It is at least the third documented instance of the same shape. The two previously measured were also commits touching ZERO files in the five hashed globs — a docs one-liner and the watcher's own tstate write — against a measured 71 NEW-REDs carrying src:tools/compiler_srchash.sh across 18 distinct job names, 21 shas and 7 days.

Why this instance is worth more than the count. Every earlier one needed someone to establish that the named source was not implicated, which is work and which is why the class kept getting re-discovered. A docs-only commit needs nothing established. If this ticket ever wants a regression test for the mislabel itself, a commit touching only devdocs/ is the fixture — it is a positive control drawn from the right population that cannot be argued with.

Filed by the hourly fleet pass rather than by anyone working the ticket; no diagnosis added, no re-ranking. Recording the instance only, because the value here is that it accumulated without being hunted.