← board

Track A from the job NAME test-emit-obj, not from its source. This job names a MECHANISM rather than a subject — the source it was fed (test/c_obj_data_import.c) is what the mechanism was run ON, not what is being tested, so a lane guessed from it would be wrong by construction. The ranker reads frontmatter, so this line decides who works it; re-lane it if this job has changed what it covers.

origin/master has advanced 2 commit(s) since this sha. Re-verify at current HEAD before acting — the callback is tagged to the sha that was tested, which may no longer be the state of the tree.

regression: test-emit-obj#src:test/c_obj_data_import.c at e7a805d13a09 in step 11/11, if command -v gcc >/dev/null 2>&1; then \ printf '#include <stdio.h>\nint somebody_elses_global = 99;\nint read_it(void… (auto-filed by twatch)

Repro

tools/testmgr.py --tier full --job 'test-emit-obj#src:test/c_obj_data_import.c' at e7a805d13a0995bd8ec08e79e0604b32e4cc42f2

Range

The named sha e7a805d13a09 CANNOT be the cause — it touches no buildable file (docs / tickets / tstate only). It is the sha that was TESTED, i.e. the upper bound of an untested range; the cause is somewhere below it.

bad e7a805d13a09, last good 01b56f5f8f0f, 1 commit(s) in range — the watcher narrows this by idle bisect; check tstate/TSTATE.md for the current range.

Log tail

collect2: error: ld returned 1 exit status
(tail)
ok: /tmp/testmgr-scratch-2487013/cods_imp_x64.o  [code=108709B  data=1432B  bss=38160B  procs=478]
ok: /tmp/testmgr-scratch-2487013/cods_imp_386.o  [code=178091B  data=1360B  bss=38120B  procs=480]
ok: /tmp/testmgr-scratch-2487013/cods_exp_x64.o  [code=108806B  data=1432B  bss=38160B  procs=478]
ok: /tmp/testmgr-scratch-2487013/cods_exp_386.o  [code=178168B  data=1360B  bss=38120B  procs=480]
ok: /tmp/testmgr-scratch-2487013/cods_mat_x64.o  [code=108813B  data=1432B  bss=38176B  procs=478]
ok: /tmp/testmgr-scratch-2487013/cods_mat_386.o  [code=178304B  data=1360B  bss=38136B  procs=480]
ok: /tmp/testmgr-scratch-2487013/cods_only_x64.o  [code=108667B  data=1432B  bss=38164B  procs=477]
/usr/bin/ld: cannot find /tmp/testmgr-scratch-2487013/cods_imp_x64.o: No such file or directory
collect2: error: ld returned 1 exit status
test-emit-obj: data-import object FAILED to link

Stub ticket: signal only. Track T agent (face 2) enriches or a dev track takes it from the repro line.

Triaged 2026-09-04 by frankb-78 (Track A) — does not reproduce, and the log tail says harness

Not reproducible at HEAD. Ran the job's own two emit steps and both gcc links by hand at 7e271ff7d:

--emit-obj            test/c_obj_data_import.c  -> ok, 108709B
--emit-obj --target=i386 same source            -> ok, 178091B
gcc main.c cods_imp_x64.o && ./a.out            -> 99   (rc 0)
gcc -m32 main.c cods_imp_386.o && ./a.out       -> 99   (rc 0)

The failing sha does not contain the generator work landing that day. git merge-base --is-ancestor a090fa76d 01b56f5f8f0f is FALSE — the tested tree predates it. Nothing in the range touches compiler/** for C or emit-obj.

The log tail is self-contradictory in a way only the harness can explain. It prints ok: /tmp/testmgr-scratch-2487013/cods_imp_x64.o [code=108709B ...] — the compiler wrote the object and reported the exact byte count this repro reproduces — and then, in a later step of the SAME recipe:

/usr/bin/ld: cannot find /tmp/testmgr-scratch-2487013/cods_imp_x64.o: No such file or directory

A file the job created went missing between two steps of one recipe. That is not a code generation defect; the object was produced and measured. Re-laned to T on that evidence.

RESIDUAL QUESTION, AND IT HAS AN OWNER. "Not a compiler bug" is half a finding. The open half is why did $(TESTTMP) lose a file mid-job on seven — a scratch dir cleaned by a concurrent job, a tmpfs eviction, or a testmgr teardown racing its own recipe. Track T owns the scratch lifecycle and is the only lane that can see the run. This is a signal about the harness, not noise: whatever removed that object could remove any other, and every job that loses one reports as a red in the subject it was testing. Worth checking whether the other reds in the same run share the shape.

Not claimed: that the harness is broken. Claimed: it does not reproduce, the compiler wrote the object, and the cause is above the compiler.

Log

2026-09-06 (frankH) — three candidate mechanisms ELIMINATED, one real asymmetry fixed, residual still open

Taking frankb-78's residual question — why did $(TESTTMP) lose a file mid-job on seven — and reading the scratch lifecycle. I did not find the cause. What follows is eliminations plus one defect found on the way; the question still has no answer and still belongs to whoever can see a live run on seven.

Eliminated

1. The producer/consumer split is not it. testmgr splits a recipe into jobs and re-merges groups that share a /tmp token, and its own comment names the failure this would produce ("a standalone --job repro runs the consumer with a fresh scratch dir where the artifact never existed", test-core#555/#556, 2026-07-12). It cannot apply here: both lines name the artifact literally, in one target —

Makefile:28110  ./$(COMPILER) --emit-obj test/c_obj_data_import.c $(TESTTMP)/cods_imp_x64.o
Makefile:28123  gcc $(TESTTMP)/cods_imp_main.c $(TESTTMP)/cods_imp_x64.o -o ...

so tmp_re.findall sees /tmp/cods_imp_x64.o in both groups and the union-find merges them. The invisible-edge case that comment is really about (a .so found by soname, named nowhere) does not arise either.

2. sweep_orphan_tmp is not it. It only reaps a testmgr-scratch-<pid> whose pid is dead, and skips pid == os.getpid().

3. reap_stale is not it, on ordering. It is reached only from acquire_lock, which runs before the run has populated its scratch — so its rmtree cannot remove an artifact a recipe has already produced.

Fixed anyway: reap_stale would reap its OWN scratch

Elimination 3 is a fact about the CALLERS, not about the function, and the function is wrong on its own terms. The pid it acts on comes out of the lock file, and lock_state() says in its own docstring why that is not an identity: "pids get reused". A stale lock naming pid P plus a live run the kernel has handed P means $(TESTTMP)/testmgr-scratch-P is that run's RUN_TMP.

Three sibling cleanup paths, one predicate, and only two implementations of it:

kill_run()          if pid in (os.getpid(), os.getppid()): refuse
sweep_orphan_tmp()  if pid == os.getpid(): continue
reap_stale()        (nothing)                              <- fixed

kill_run's guard exists because it SIGKILLed itself proving the point. The rmtree beside it never got the same treatment.

And the symptom it would produce is character-for-character this ticket's: ok: <path> [code=...] from the compiler, then ld: cannot find <path> a few steps later, no error from either side, the red landing in whatever subject owned the file. That is not evidence it happened here — ordering says it did not — but it does mean this ticket's signature is reachable from the harness by more than one route, which is the part worth carrying forward.

Guarded, with tools/testmgr_reap_self_devtest.py as the positive control (picked up automatically by tools-devtest's glob). Ablation: remove the guard and the "our own scratch survives" row fails while the "a dead run's scratch is still reaped" row keeps passing — so the control is not a disabled reaper.

Residual, still unowned by this ticket

Nothing here explains a file vanishing mid-recipe on seven on 2026-09-04. The remaining candidates need the live box: a concurrent writer, the run's own --force path taken by a second testmgr, or something outside testmgr. The /tmp reaper is not a candidate for that date — it was still at 10 days then and only moved to 6h on 2026-09-06.