← board

Flaky async/multithreaded run-tests produce false NEW-REDs (no confirm-retry in reap())

Symptom

tstate/borg intermittently reports 0-in-range NEW-REDs for async / multithreaded run-tests under QEMU, always self-clearing on the next tick:

0-in-range (empty bisect window) + self-clearing + a run-test whose compile step is ok: and whose runtime is async/threaded under emulation is the signature of a scheduling/timing race in the emulated run, not a code defect.

Root cause

Scheduler.reap() (tools/testmgr.py:965) sets the verdict on the first process exit, with no retry:

rc = job.proc.poll()
if rc is not None:
    job.status = "pass" if rc == 0 else "fail"   # one shot — a single transient
                                                 # nonzero exit is permanent

The bench path already knows better: BENCH_EXTRA_TRIES = 5 (tools/testmgr.py:1408) re-runs a bench that lost its sample to contention. Regular pass/fail run-tests get no equivalent — so a job that spuriously exits nonzero once (descheduled thread, socket timing, QEMU scheduling jitter under a loaded full-matrix run) is declared RED, turns the whole run RED, and lands in tstate as a NEW-RED tied to whatever SHA happened to be under test. Triage cost is real even though it self-clears: every false NEW-RED is a bisect + a manual re-run + this exact investigation.

Fix direction (decide at pickup — do not mask real reds)

The invariant to preserve: a real red must stay red. A genuine failure reproduces every attempt; only a flake passes on retry. So confirm-on-failure is safe by construction — it costs re-runs only on jobs that were going to be RED anyway.

  1. Confirm-retry a failing run-test before declaring RED (preferred, narrow). On a nonzero exit, re-run that one job up to N times (N≈2–3); RED only if it fails every attempt; PASS the moment one attempt passes. Cheap — retries fire only on the already-failing minority, and a real red pays N× on one job, not on the suite. This is the exit-code analogue of BENCH_EXTRA_TRIES.
  2. Tag known-nondeterministic jobs (asyncecho, sqlite-threads-*, optdiff shards) as flake-prone and apply the retry only to them — smaller blast radius, but needs a maintained list and misses the next new async test.
  3. Report flaky as a distinct verdict (retried, passed-on-retry) so tstate records "flaked+recovered" instead of either hiding it or crying RED — keeps the signal without the false alarm. Compose with (1).

Recommendation: (1) + (3) — confirm-retry for correctness, a flaky verdict so the noise is still visible (a test that flakes 1-in-3 is worth knowing about) without being actionable-red. Keep it general (all run-tests), not a hand-maintained tag list.

Non-goals

Acceptance

Log