A saved partial is evicted by the next run of different work

The finding

resume_health()'s own docstring names this failure shape: "saved partials that never become carried runs is the exact silent-degradation shape: every individual log line reads fine and the mechanism does nothing." Its numbers now say that is what is happening. From the watcher clone, 2026-08-19T20:23:12Z:

saved_partials : 9
saved_jobs     : 1420
carried_runs   : 0      <- absent from the file, i.e. never incremented
superseded     : 9
discarded      : 0
no_report_on_abort : 0

Nine saves, nine supersedes, zero carries. 1420 decided jobs preserved and then thrown away, at a 100% loss rate. Shape 2 has never delivered once since it landed (e2449adc5).

The mechanism

RESUME_REL is a single path — .testmgr/resume.json — and every gate run passes a resume_key=(sha, tier) and calls resume_arg(). On a key mismatch that function does not merely decline to use the partial; it deletes it:

if (part.get("sha"), part.get("tier")) != key:
    ...
    bump_resume_stats(clone, superseded=1)
    drop_partial(clone)
    return None

So the observed cycle is:

1. pin verify on cabb5d598/full gets an idle slice, aborts partway, saves a partial worth a few hundred jobs;
2. a push arrives — the fast verdict runs newHEAD/native, calls resume_arg() with a different key, and drops the pin-verify partial;
3. the next idle slice restarts pin verify from zero.

Step 2 is not an edge case. It is the normal operation of the watcher, and it is guaranteed to happen between any two idle slices, because idle slices are separated by the pushes that end them.

Why the eviction looked right

The comment argues the drop is sound: "the watcher moved on to different work, and the old partial can never become valid again."

That is true for a HEAD-progression full tier — HEAD moves on and that exact sha is never tested again — and false for the case shape 2 was built for. Pin verify targets the PIN's sha, which does not move when HEAD does; it moves only when a pin lands. The ladder returns to that same (sha, tier) on every subsequent idle slice, which is precisely why a partial for it is worth keeping.

One true statement about HEAD progression, applied to a target that does not progress. The eviction is correct for the case it was written against and wrong for the case that motivated the feature.

Why it matters

This is the load-bearing half of bug-t-the-push-rate-starves-breadth-coverage-entirely. That ticket's re-measurement concluded breadth is queued behind an unfinishable pin verify — 21 contiguous minutes wanted, ~5-minute slices available, 100% of work discarded per abort — and recommended shapes 2 + 4. Shape 4 shipped and works (idle_yield yields the slot after 3 aborts; the live tstate shows it counting). Shape 2 shipped and does nothing, so the "with shape 2 it does" half of that recommendation is not in force. A 21-minute job still cannot complete in 5-minute slices.

The shape of a fix

Keep partials in a small keyed store rather than one slot: .testmgr/resume/ with a file per (sha, tier), or a single file holding a dict keyed the same way. A run then reads its own partial and leaves everyone else's alone.

Constraints:

• Bounded. Partials carry every decided job's dict; 1420 jobs is not small. Cap the count and evict oldest-first, and say what was evicted.
• The compiler-sha256 guard stays. load_resume() refuses a partial whose binary does not match, which is what makes a carried result attributable. That check is orthogonal to this and must not be relaxed to make carries happen.
• Genuinely dead partials should still go. A HEAD-progression sha really is never revisited, so its partial is garbage — but it should be evicted by age or capacity, not by "some other run started".
• carried_runs becoming non-zero is the acceptance test. The rate is the measurement, not the individual save (see the standing rule about reporting resume-stats rates rather than the one-off save).

The fix

.testmgr/resume.json (one slot) became .testmgr/resume/ (one file per key), via a new partial_path(clone, key) = <sha[:12]>-<tier>.json.

• resume_arg() is now read-only. It opens its own key's file and returns the path or None. The mismatch branch that called drop_partial() is gone — that branch was the bug, and its comment ("the watcher moved on to different work") was a true statement about HEAD progression applied to pin verify, whose target sha does not move. What survives of it is a payload check: a file whose contents disagree with its filename is declined, because the name is a convenience and the payload is the authority.
• drop_partial(clone, key) takes a key. The post-run call at the end of run_gate passes resume_key, so a finished run still clears its own partial — the retry-after-INFRA case the original comment was right about — and only its own. key=None still targets the legacy slot, which save_partial also unlinks on first write so it cannot shadow the store.
• gc_partials() bounds it at PARTIAL_CAP = 4, newest kept, oldest evicted by mtime, and it prints what it dropped. Four covers the live interleave (a pin verify, a breadth backfill, and the fast verdicts landing between their slices). An aged-out partial still bumps superseded: it is the same lost work an eviction was, and a silent GC would hide a regression behind a healthy-looking saved count.
• load_resume()'s compiler-sha256 guard is untouched. Nothing here relaxes attributability to manufacture a carry.

Guarded in tools/twatch_resume_devtest.py. The headline check is "A RUN OF DIFFERENT WORK DOES NOT EVICT IT — it is still there"; the rest cover coexistence of two partials, self-only drop, the payload-vs-filename check, the cap, newest-survive, and that an aged-out partial is counted. Non-vacuity proved by restoring the evict-before-read behaviour in a scratch copy: 4 checks go red, including the headline. PXX_TRACK=T make tools-devtest: 48 guards green.

What is proved and what is not. The devtest proves a partial now survives an interleaved run of different work — the mechanism that caused the 100% loss rate. It does not prove a carry: carried_runs can only be observed to leave zero by the live watcher completing a resumed slice. That number is the follow-up measurement, and it stays the acceptance test.

Log

• 2026-08-19 — resolved, commit 3674a3a60.

First live measurement, 2026-08-19 23:42 CEST — the eviction has stopped

The watcher restarted at 23:35 (pid 2214596 → 3171395), which is when the fix stopped being inert. Seven minutes later:

$ ls /home/neo/trackt-watch/.testmgr/resume/
d47acfee770c-full.json          # the keyed store EXISTS, one file per (sha, tier)

$ cat .testmgr/resume-stats.json
{"saved_partials": 13, "saved_jobs": 2600, "superseded": 11, ...}

Against the last pre-restart reading — {"saved_partials": 11, "saved_jobs": 1832, "superseded": 11} — two partials were saved and superseded did not move.

That is the fix, observed directly. Under the single slot every save was followed by the next run of different work deleting it, and the two counters climbed together: 9/9, 10/10, 11/11. They have now separated for the first time.

What this proves and what it does not. It proves partials survive the arrival of other work, which is the defect this ticket names. It does not prove a carry: carried_runs is still absent from the stats file, and it can only leave zero when the live watcher completes a slice it actually resumed from. That number remains the acceptance test, unchanged.

The before/after is in one log file, 40 minutes apart

/home/neo/trackt-watch.log happens to contain the same scenario twice — a full run preempted, its partial saved, then a different (sha, tier) starting — once under each code path. twatch: bye at line 712295 is the restart; everything above it is the old process, everything below the new one.

Before (line 700515), the defect in its own words:

twatch: kept 312 decided job(s) from the aborted full run — the next slice resumes instead of restarting
twatch: pin verify preempted by a push — will resume
twatch: dropping a partial for cabb5d5989f3/full — this run is 9b6d2c982d93/native

Three consecutive lines: saved, promised a resume, deleted it. The run that deleted it was a native gate at an unrelated sha, which is exactly the "next run of different work" in this ticket's title.

After (lines 717757-717760), same shape, opposite outcome:

twatch: kept 15 decided job(s) from the aborted full run — the next slice resumes instead of restarting
twatch: pin verify preempted by a push — will resume
twatch: 1 uncommitted tstate file(s) — ours by definition, publishing rather than pausing on them: plexus.json
twatch: testing 2e8b284343a5 (native, fast)

Same preemption of a full pin verify, same different-work native run starting immediately after — and no dropping line, because drop_partial() is now only reachable from the run that owns the key. .testmgr/resume/d47acfee770c-full.json is still on disk while that native run executes.

Same daemon, same log, same scenario, both verdicts. That is as controlled as a live-system measurement gets here, and it is why the counters separating (11→13 saved against a flat 11 superseded) is a fix and not a sampling artefact.

Still not a carry. The ladder has to come back to (d47acfee770c, full) and finish it before carried_runs can leave zero. Unchanged as the acceptance test.

ACCEPTANCE TEST MET, 2026-08-19 22:07Z — carried_runs is 1

{"saved_partials": 13, "saved_jobs": 2600, "superseded": 11,
 "carried_runs": 1, "carried_jobs": 15,
 "last_note": "resume: partial accepted — 15 job(s) already decided against this exact binary (1479b663dd15)"}

The whole loop, in one night, in the log:

Saved, survived the arrival of unrelated work, carried, consumed. The store is empty again afterwards, which is drop_partial() on the owning key doing its job — the one deletion path this fix left reachable.

Two details worth keeping:

The binary identity check earned its place. The partial was accepted against 1479b663dd15, the same compiler snapshot sha256 the aborted run logged at line 714946. Had anything rebuilt in between, the 15 decided jobs would have been decisions about a different binary, and the resume would have silently imported them. It refused nothing here because there was nothing to refuse — but that is the check being exercised, not the check being unnecessary.

Cost recovered is small and that is expected. 15 jobs of 2770. The abort landed 40 seconds into a run that takes ~21 minutes, so there was almost nothing to keep. The mechanism's value is not this number: it is that the fraction is now whatever the abort happened to leave, instead of always zero.

superseded stayed at 11 across the entire sequence — its pre-fix value, unmoved since. Closing the measurement: 9/9, 10/10, 11/11 before; 13 saved, 1 carried, 11 superseded after.