← board

Track T by default: the FAILING STEP named no owner. Line 53 of 47 is /tmp/next-test_multithreading26 | grep -q "multithreading test completed successfully". The job's own src (test/test_exception_unhandled.pas, 13 file(s)) is NOT used here on purpose: it is what the job compiles, not what broke, and guessing a lane from it is what sent three reds in one job to the wrong lane. This is a FALLBACK, not a finding — nothing says the defect is Track T's. Re-lane it before working it.

origin/master has advanced 3 commit(s) since this sha. Re-verify at current HEAD before acting — the callback is tagged to the sha that was tested, which may no longer be the state of the tree.

regression: test-core#src:test/test_exception_unhandled.pas@3 at 3d4801b6abc3 in step 53/47, /tmp/next-test_multithreading26 | grep -q "multithreading test completed successfully" (auto-filed by twatch)

Repro

tools/testmgr.py --tier native --job 'test-core#src:test/test_exception_unhandled.pas@3' at 3d4801b6abc305d803228ec381394834336b8b40

Range

The named sha 3d4801b6abc3 CANNOT be the cause — it touches no buildable file (docs / tickets / tstate only). It is the sha that was TESTED, i.e. the upper bound of an untested range; the cause is somewhere below it.

bad 3d4801b6abc3, last good 889bfcf73256, 1 commit(s) in range — the watcher narrows this by idle bisect; check tstate/TSTATE.md for the current range.

Log tail

Segmentation fault (core dumped)
(tail)
134]
ok: /tmp/testmgr-scratch-3028389/self-records26  [code=69400B  data=2792B  bss=43540B  procs=134]
ok: /tmp/testmgr-scratch-3028389/self-procs26  [code=69400B  data=2888B  bss=43496B  procs=137]
ok: /tmp/testmgr-scratch-3028389/self-string_compare26  [code=69400B  data=2960B  bss=43500B  procs=136]
ok: /tmp/testmgr-scratch-3028389/self_record_string_field26  [code=69400B  data=2900B  bss=43524B  procs=135]
ok: /tmp/testmgr-scratch-3028389/self-test_heap26  [code=69400B  data=2792B  bss=43504B  procs=134]
ok: /tmp/testmgr-scratch-3028389/self-test_multithreading26  [code=69288B  data=3988B  bss=43556B  procs=138]
ok: /tmp/testmgr-scratch-3028389/self-test_math_unit26  [code=224936B  data=6648B  bss=43576B  procs=675]
ok: /tmp/testmgr-scratch-3028389/self-fileio26  [code=110360B  data=5032B  bss=43788B  procs=248]
pascal26:5758: warning: bare own name 'TargetDisplayName' reads the result of parameterless function TargetDisplayName; write TargetDisplayName() for a recursive call, or Result to read the result
ok: /tmp/testmgr-scratch-3028389/pascal26-next.3029288.tmp  [code=9805592B  data=480952B  bss=103315116B  procs=3973]
ok: /tmp/testmgr-scratch-3028389/next-hello26  [code=65304B  data=2840B  bss=43492B  procs=134]
ok: /tmp/testmgr-scratch-3028389/next-bootstrap_features26  [code=69400B  data=2920B  bss=43508B  procs=134]
ok: /tmp/testmgr-scratch-3028389/next-records26  [code=69400B  data=2792B  bss=43540B  procs=134]
ok: /tmp/testmgr-scratch-3028389/next-procs26  [code=69400B  data=2888B  bss=43496B  procs=137]
ok: /tmp/testmgr-scratch-3028389/next-string_compare26  [code=69400B  data=2960B  bss=43500B  procs=136]
ok: /tmp/testmgr-scratch-3028389/next_record_string_field26  [code=69400B  data=2900B  bss=43524B  procs=135]
ok: /tmp/testmgr-scratch-3028389/next-test_heap26  [code=69400B  data=2792B  bss=43504B  procs=134]
ok: /tmp/testmgr-scratch-3028389/next-test_multithreading26  [code=69288B  data=3988B  bss=43556B  procs=138]
Segmentation fault (core dumped)

Stub ticket: signal only. Track T agent (face 2) enriches or a dev track takes it from the repro line.

Log

Does NOT reproduce on plexus, on EITHER compiler — 2026-09-02, frankZ

Run under the umbrella [[umbrella-one-full-tier-run-with-no-red-tier]], with the recipe's own flags:

binary commit runs failures
480d4584403c (HEAD) ad55e4dcc 40 0
stable_linux_amd64/default/pinned (v399) 2026-08-19 30 0

Two independently built compilers, byte-different programs, 70 runs, no failure. The tested sha 3d4801b6abc3 is also an upper bound rather than a cause — the ticket's own banner says it touches no buildable file.

This is half a finding and the residual belongs to Track T. "Not reproducible on plexus" does not answer "then why is it red on seven", and I cannot answer that from here: seven runs this job under full-matrix parallelism and I ran it solo. For a threading-adjacent program that is not a detail, it is the most likely difference — the same shape as test_multithreading, which needed no load to fail but whose rate moved with it.

What would settle it, and what I am NOT claiming to have done: the same job on seven, under load, at this sha. Left open and wired to the umbrella rather than resolved, because a red nobody can reproduce is still a red in the tier that decides whether a pin is green.

MEASURED: this is a FLAKE, and the tstate archive says so outright (frankZ, 2026-09-02)

The earlier entry on this ticket said the failure does not reproduce here on either compiler and named seven's full-matrix parallelism as the likeliest difference. That was half a finding. Here is the other half, and it does not depend on reproducing anything.

Track T's own markers alternate. Counting every tstate commit subject that names test_exception_unhandled:

NEW-RED-bearing commits : 10
FIXED-bearing commits   : 11
total markers           : 24
18 of the 24 markers land on 2026-09-01 alone

A regression cannot be fixed and re-broken 10 times in one day. That would need 10 fixes and 10 breakages, on one host, across shas that mostly touch docs and tickets. The only reading left is that the job's verdict is nondeterministic, and the NEW-RED / FIXED pairs are the watcher faithfully reporting a coin landing differently.

This is a second source that FAILS DIFFERENTLY from the local non-reproduction: 0-in-40 here is consistent with "rare flake" AND with "host-specific bug", and cannot separate them. The alternating markers separate them — a host-specific bug is stable on its host, and this is not stable on its host.

What changes because of it

Why it matters to the umbrella specifically

[[umbrella-one-full-tier-run-with-no-red-tier]] wants ONE full run with no RED in any tier. A test that flips on its own makes that a lottery rather than a consequence of fixing things: every arrival can be beaten by the fix rate and the run still comes back red. Recorded there too.