verify_pin builds with the previous pin, and files the result under the new one

The mechanism, and it is structural rather than a race

verify_pin():

print("twatch: verifying PIN %s (%s) at %s …" % (ver, sha[:12], tier))
set_phase(clone, host, "pin-verify", sha=sha[:12], tier=tier, pin=ver)
clone.checkout(sha)          # <- the pinned TREE
report, rc = run_gate(clone, tier, …)

clone.checkout(sha) brings the whole worktree to that sha — including stable_linux_amd64/**. And a pin commit is always a descendant of the tree it pins: you cut the pin at tree T, then commit the new binary as a child of T. So at T, stable_linux_amd64/default/pinned still holds v(N-1).

lib-test, demos and test-fpjson build with $(PXX_STABLE). They therefore build with the previous pin, every time, by construction.

Evidence — one run, three lines

twatch: verifying PIN v400 (67ae9a62d567) at full — the sha every other track builds on
self-host fixedpoint: verified — 1 round(s), 6c184b4bcc37 (stamp read back; sources match it)
testmgr: compiler snapshot /tmp/…/compiler/pascal26 (sha256 6c184b4bcc37)
testmgr: pin=399 sha256=954adef93a7b0e9e (lib-test and demos build with THIS, not HEAD)
  [   4/3846] FAIL    lib-test#60                     1.6s

The HEAD compiler is v400's bytes (6c184b4bcc37). $(PXX_STABLE) is v399 (954adef93a7b). grep -c 'pin=400' trackt-watch.log → 0.

report_pin_identity() — the function written so a lib-test red says which binary produced it — printed the correct answer, and nothing read it.

What it produced this time

The v400 verify recorded 5 reds:

lib-test#src:tools/crtl_reachability.py
lib-test#src:test/lib_synapse.pas
lib-test#src:test/lib_synapse_transitive_unit.pas
lib-test#src:test/lib_synapse_ssl.pas
tools-devtest#00

Those are v399's known defects, verified as such: v399 fails lib_synapse with cannot assign ShortString to Char and v400 builds it clean, measured both ways under srchash MATCH. The verify attributed v399's breakage to v400 in the record every other track reads.

Why this matters more than one bad record

It runs backwards through the history. Each pin verify is a judgement of the outgoing pin wearing the incoming pin's name:

• v400 at 67ae9a62d — 5 red — actually v399, and v399 is independently known-broken (two SIGSEGVs, the synapse trio, the 8b89a201d ModRM miscompile).
• v399 at 86c71828cd1e — "20 red, 19 new vs the v399 baseline" — actually v398, the pin that could not build C for i386 or arm32.
• v398 at c8e132a02b92 — "5 red, 4 new" — actually v397; annotated by hand as "NOT CORROBORATED … a load-shaped flake, do NOT revert on this count alone."

That last annotation may be the real cost. The reds were not flakes. They were true statements about a different binary, and a careful human read them as noise because they did not reproduce at HEAD — which is exactly what a previous-pin red would do.

This was already seen once and fixed at the wrong level

bug-t-pin-verify-records-positional-job-numbers-and-a-stale-version-label (now in done/) records:

"And its ver said v347 while the verified tree carried pin 346."

That is this bug, observed exactly. It was read as the label being stale and the label was corrected. The label is now right and the binary is still the old one, so the disagreement it noticed has been resolved in favour of the wrong side — the record is now confidently mislabelled instead of visibly inconsistent.

Fix — options, not a prescription

1. Restore the pinned binary after checkout. Check out the tree, then overwrite stable_linux_amd64/default/pinned (and the frozen builtin/) from the pin commit being verified. Most faithful to the question "is this pin good?", and the pin commit is a known sha.
2. Verify at the pin COMMIT, not the pinned tree. Simpler, one-line-ish, but it changes what is being tested — the pin commit's tree may carry other changes.
3. Refuse to verify when they disagree. Compare report_pin_identity()'s answer against ver and abort with a loud message rather than record. Cheapest, and turns a silent wrong record into a visible gap — the direction this repo consistently prefers.

Whatever is chosen, assert the two agree, because the evidence that they do not was printed in the log all along and read by nobody.

Positive control for any fix

A verify of v400 (67ae9a62d567) must log pin=400 sha256=6c184b4bcc37, and its lib_synapse jobs must be GREEN — v400 builds all three, measured. Under today's code the same verify logs pin=399 and fails them.

Second instrument, same defect, one hour later — and the gap is measured

Re-ranked 70 → 80 on this, not on the original finding. One instrument getting a pin identity wrong is a bug in that instrument; two independent ones within an hour is the identity itself being easy to get wrong, and everything that quotes a pin is exposed.

The second instance (coordinator, on umbrella-sizeof-is-one-answer): a pin ancestry table checking whether three closed tickets' pinned positive controls had gone stale. It resolved the pin as commit 766b99f98 — the chore(stable): pin v401 commit — rather than 07d196aa4, the tree v401 was built from. Caught before it moved a conclusion: none of the three controls sits in the gap, so all three were still VALID. Anything that did sit in the gap would have returned a false VALID.

That is not this ticket's mechanism — verify_pin checks out the tree and gets the old binary; the table read the commit and got a too-new tree — but it is the same root confusion, from the opposite end: the pin commit and the pinned tree are different objects and neither name says which one it is.

The gap is not incidental, and it is not small. Measured across the three pins cut on seven:

Never zero. The pin is cut at tree T, then stabilize-fast, pin, commit and a rebase loop against a busy origin all take minutes, during which the fleet lands work. So the pin commit is 2–4 commits ahead of the tree it describes, and that distance grows with fleet activity rather than shrinking.

Concretely for v401: dfff555e0, d6b222419 and 10916bd26 are inside the gap and are not in the pinned binary, but any check that resolves "the pin" to 766b99f98 will believe they are.

What this adds to the fix list. Alongside the three options above, the cheapest general remedy is naming: pin.log already records both — the git sha column is the TREE. Any consumer resolving "the pin" to a commit should be reading that column, and a helper that returns the pinned TREE for a version would make the right answer the easy one. Two instruments got this wrong in an hour by each deriving it independently.

Positive control for that helper, from the table above: given v401 it must return 07d196aa4, not 766b99f98, and git rev-list --count 07d196aa4..766b99f98 must be 4.

The gap is 17 commits at v407, and the ticket predicted the direction

Re-measured 2026-09-06 (frank-subcoord, on claiming this). The table above recorded 2–4 commits between the pin COMMIT and the pinned TREE across v399–v401, and predicted the distance "grows with fleet activity rather than shrinking". Confirmed, and by more than the original range suggests:

The v401 positive control this ticket specifies still passes exactly: git rev-list --count 07d196aa4..766b99f98 = 4.

What is inside the v407 gap and NOT in the pinned binary includes 2f1fe06b9 fix(P): a routine-local type section is parsed in PASS 2, so its specialize splice must move the spans — a compiler fix. Anything resolving "the pin" to 51901941e believes those 17 commits are in the binary 095ef4811a5b. They are not.

Caught live, in a request, in the direction that helps

frankuser queued a full tier in verify-requests.tsv at 51901941ef5d with the reason "pin v407 (binary 095ef4811a5b) was cut at this tree". It is not the tree — it is the pin COMMIT (chore(stable): pin v407 -- binary sha256 095ef4811a5b); pin.log's sha column gives the tree as 04559b9d6 (docs(roster): the night a correct measurement froze the fleet).

The error landed on the useful side. Because stable_linux_amd64 at the pin commit holds v407, that request measures the v407 binary — which is what anyone asking "is this pin good?" wants. verify_pin checks out the TREE and therefore builds every $(PXX_STABLE) job with v406 while filing the verdict under v407. So a hand-written request got the right answer by naming the object this ticket says is the wrong one, and the automated path gets the wrong answer by naming the right one.

That is empirical support for option 2 (verify at the pin COMMIT) over option 1, and it is worth more than the abstract argument: at the pin commit the pinned binary is vN by construction, with no restore step that can itself fail. The cost option 2 was charged with — "the pin commit's tree may carry other changes" — is now measured at 17 commits, so it is a real cost and not a rounding error. Neither option is free, and the choice is between building the right binary against a slightly newer tree (option 2) and reconstructing a state that never existed (option 1).

Three sessions confused these two objects in one evening. The naming remedy this ticket already proposes is the load-bearing part.

Landed 2026-09-06 (frank-subcoord): the refusal, and why only the refusal

What landed. verify_pin now compares report["pin"] — which testmgr has written since it was added, and which nothing read — against the version it is about to publish under. On a mismatch it publishes nothing and says why:

twatch: pin verify REFUSED to publish — it measured v406 and was about to file
the verdict under v407. The checkout is the pinned TREE 04559b9d6c5a, whose
stable_linux_amd64 still holds the previous pin; v407 is installed by the pin
COMMIT 51901941ef5d, 17 commit(s) later. Publishing nothing.

Plus three helpers, tested against this ticket's own controls: pin_version_str (pin.log writes v407, VERSION holds 407, twatch's ver carries the v — one identity, three spellings, and mixing them yields a None that reads as "no such pin"), pinned_tree_for, pin_commit_for. The last resolves by walking the VERSION file forward rather than matching a commit subject, because subjects are close to a convention and not one — v396 and v397 use an em-dash where v398 onward use --.

Controls, all four run: the real shape (measured v406, filing v407) refuses; identical identities pass; a spelling difference is not a mismatch; and a tier with no pin-built job does not refuse but announces that the check could not run, so an absent assertion is not mistaken for a passed one.

Why not the fix itself. Both routes are larger than they look, and neither can be verified from a seat that cannot run the daemon:

• Option 1 (check out the tree, restore stable_linux_amd64 from the pin commit) leaves the worktree carrying local changes. clone_head_back is a plain git checkout --quiet <branch>, which git refuses when it would overwrite them — so the naive version wedges the daemon rather than failing a run. It needs a force or a reset, on the fleet's one instrument.
• Option 2 (verify at the pin COMMIT) is a one-word change and is the state that actually existed — at the pin commit $(PXX_STABLE) is vN by construction. But the run's sha becomes the pin commit, so the archive row is keyed to it and the pin.log × tstate join in trackt.read_pin_log — which matches on the TREE — stops finding the verdict. The join has to move with it.

Consequence to state plainly: pin verify publishes nothing until one of those lands. That is the intended direction (a known unknown over a confident wrong answer, the same rule the INFRA/INVALID branch above already applies), but it is a live behaviour change and not a silent one.

The premise is measured now, not reasoned — nine pins, unanimous

frankH's challenge on landing the refusal was right: "verify that clone.checkout(<pinned tree>) really does bring back v(N-1) — your reasoning is sound, and it is also exactly the kind of premise that turns out to be right for a different reason or wrong in one case." Measured directly out of git, git show <tree>:stable_linux_amd64/default/VERSION for every pin from v399:

Nine of nine, exactly N-1. The mechanism is structural and there is no case where it happens not to bite. So the refusal now in place will fire on every pin verify until the artefacts are corrected, which is the intended behaviour and is why it was landed as a loud refusal rather than a silent skip.

The blocker on option 1 is removable (frankH), and option 2 should not be taken

frankH's answer to the worktree hazard: the dirt is ours and known — the verify put exactly stable_linux_amd64/** there — so clean it explicitly at the end of the verify rather than making clone_head_back tolerant:

git checkout HEAD -- stable_linux_amd64

HEAD --, not --: the latter restores from the INDEX and would re-apply staged content. Keeping clone_head_back strict matters, because a forceful one would discard state from any cause, and the whole reason option 1 was parked is that wedging the instrument beats a wrong verdict.

And option 2 is now argued down rather than merely ranked second: re-keying the archive row to the pin COMMIT breaks trackt.read_pin_log's join on the tree, which is a query that stays correct-looking, exhaustive and false — one word to write and a wrong answer nobody sees. Option 1 is the route.

Fixed 2026-09-06 — option 1, with the wedge measured rather than assumed

verify_pin now restores the pinned artefacts from the pin COMMIT after checking out the tree, and cleans that dirt before clone_head_back:

clone.checkout(sha)                                  # the pinned TREE
git checkout <pin commit> -- stable_linux_amd64      # ...make it vN's artefacts
<run the tier>
git checkout HEAD -- stable_linux_amd64              # our own dirt, only ours
clone_head_back(clone)                               # stays a plain checkout

HEAD --, never bare --: the restore writes the index too, and bare -- restores from the index, which would keep exactly what we are removing. clone_head_back stays strict on purpose — a forceful one would discard state from any cause, and a wedge we can diagnose beats state silently thrown away.

The wedge is conditional, and the guard's own control is what established that. The row asserting "without the cleanup, checking the branch back out FAILS" went RED on its first run: git refuses only when the target content DIFFERS from the local change, and the first fixture had master's artefacts identical to the restored ones. So verifying the CURRENT pin would usually not have wedged at all, and verifying one that a newer pin has overtaken would — pin_changed_mid_run is a tracked condition, so that case is real. The fixture now carries a newer pin on the branch tip and the control fires. The cleanup is kept: it is correct in both cases, and "usually does not wedge" is not a property to rely on in the fleet's one instrument.

Guard: tools/twatch_pin_identity_devtest.py, 24 rows, auto-enrolled by tools-devtest's tools/*devtest*.py glob. It builds a fixture repo shaped like a real pin (base → tree T → pin commit as T's child → a newer pin), and reproduces the defect before testing the fix: at the pinned tree, VERSION is the previous pin. An earlier draft drove verify_pin with a clone pointing at the real checkout and so ran git checkout against the actual working tree; it restored cleanly, which is the bad luck. It now uses the fixture and asserts the real checkout is untouched.

What is NOT yet established

No live pin verify has run with this. Seven's daemon is executing twatch.py from 2026-09-05 (tools/twatch_live_code.py --host seven), so none of this is live until a restart. The first correct row is what closes this ticket: twatch: pin verify — restored stable_linux_amd64 from the pin commit … in the daemon log, followed by a published pin-verify row whose pin matches the version it is filed under. Until then the backstop means the failure mode is a missing row, not a wrong one.

2026-09-06 — the defect filed a fresh live instance, and the "it looks fixed" reading has a named cause

Two rows on origin read to a careful peer as new-code behaviour, and neither is. Seven has not been restarted. Two instruments, failing differently:

1. tools/twatch_live_code.py --host seven exits 1 (STALE): seven publishes code_fp 7327e547732c, which resolves to 17854b85b (2026-09-05 19:45). Ten tools/twatch.py commits have landed since and are not live, this ticket's fix (a695174f0) and the 10->17 pin-row enrichment (703bebb14) among them.

2. The pin row that 1f8c2b3a2 filed at 2026-09-06T21:25:59Z is 10 keys with no code_fp — the old shape — and it is the defect itself, dated tonight:

   "pin": "v407",  "sha": "04559b9d6c5a…",  "pin_baseline": true
   $ git show 04559b9d6:stable_linux_amd64/default/VERSION   ->  406
   $ git log --ancestry-path 04559b9d6..origin/master -- stable_linux_amd64
     51901941e chore(stable): pin v407 -- binary sha256 095ef4811a5b
   
   

   Seven verified v406 and filed the verdict under the name v407.

So 1f8c2b3a2 does not close the unconfirmed step — it is a new instance of the defect. The step still needs a restart.

The trap that produced the misreading, which is worth more than the answer

c842de35a carries 17 keys and a code_fp, and that is exactly the shape this ticket's fix was going to add — so it reads as proof the fix is live. It is not. Ordinary run rows have carried 17 keys and a code_fp since 17854b85b, which is the code seven is already running; the enrichment in 703bebb14 was to the pin row, written by a different writer in twatch.py. Seven's last six rows: five are 17-key ordinary runs, and the one 10-key row is the pin verify.

A key count on an ordinary run row cannot separate old code from new. Only the pin row can, and it is the one that is still short. This is "the name is not the thing" in its usual clothes: the shape you were going to introduce already existed next door, so the sample that confirms it is drawn from the wrong population.