NilPy object reclamation — dict/list/instance/bound-method lifetime

The user's resolution of [[decide-uforth-exec-leak-strategy]] (2026-07-22): uforth stays untouched — it is a test case; improvement suggestions go to the uforth repo itself, not here. The compiler must not leak on Python code: "let leaks be due to application bugs, not compiler errors." So this is the work item, and it must land either way.

Current state

STALE — this section describes 2026-07-22, BEFORE the work. Slices 1-4 landed that same night; see "Progress" below for what is actually true. Kept because the ranked remainder is written against it, but read the Progress and NEXT sections first: a reader who stops here concludes NilPy objects are not refcounted at all, which has been wrong since 2026-07-23 (noted 2026-08-08, after exactly that misreading produced a wrong claim in a filed ticket).

• Variant SLOT lifetime is handled: scope-exit release, ARC-correct var_store/copy, promo-tier clears, aggregate-dest release, call-result boxing ownership (the five fixed layers of [[bug-a-runtime-variant-heap-grows-unbounded]]).
• CLASS INSTANCES (TPyDict, TPyList, TPyBytes, user classes) and bound-method boxes have NO lifetime management at all: a dict whose binding dies is never freed. xs = [] reassignment reclaims (special case); frame exit, dict-internal churn, and instance fields do not.

The concrete driver

uforth's exec_python_inline allocates per PYTHON-word call: env TPyDict {vm, push, pop, fpush, fpop} + ns TPyDict + 5 bound-method pairs + wrapper string → ~4-5 KB/call orphaned. 20k-iter doloop = 553 MB peak vs CPython's 24 MB (make bench-uforth tracks it). Per-op RSS probes and the reduction ladder live in the umbrella ticket.

Shape (not decided here — implementation's call, escalate if forked)

Refcount class instances like AnsiString handles (retain on bind, release on unbind/scope-exit, recursive release of variant-slot payloads/fields), OR scope-tied arena for instances that provably don't escape. Watch: cycles (vm ↔ words) — CPython solves with GC; a refcount-only scheme leaks cycles, which for uforth's env-per-call pattern is still ~all of the 553 MB, so refcounting is a legitimate first rung. The user's caution from FPC history (threaded ansistring leaks): threading × memory management is hard — the threadsafe heap lock discipline (heap-size-class allocator, PXXStr* lock protocol) applies to every new release path.

Gate: uforth doloop RSS bounded and near-CPython; test-nilpy green; self-host byte-identical; the vstr/vbox probes in the umbrella stay flat.

Design pass done (fable-abcnp, 2026-07-22)

Full design in devdocs/dev/nilpy-object-reclamation.md — written while the leak investigation was warm. Summary: rc in the existing heap-block header word ([-16], the AnsiString protocol), ownership rules mirroring the string ones (call results owned, lvalues retained — the layer-5 discrimination), recursive per-type finalizers via a VMT slot, cycles explicitly out of scope (FPC-grade contract), everything behind the NilPy-user gate (isNilPy AND CurrentUnitIdx<0 — the pyeval landmine). Five-slice ladder, slices 1-3 inert/additive, slice 4 (scope-exit release of NilPy tyClass locals) is the one that drains uforth's env-per-call and carries the risk. Verification set in the doc. Pick up at slice 1.

Progress (fable-a-n, night 2026-07-22/23)

Slices 1-4 LANDED (commits 0b39d0ea..HEAD): primitives + PXXObjAlloc construction route (whole NilPy compilation, uniform headers, PXX_OBJ_MAGIC population tag at [inst-8]); variant-slot ARC for VT_OBJECT/VT_BOUNDMETHOD (x86-64 emitters via reg-preserving obj blobs, portable helpers for cross); recursive finalizers (PXXObjFinalizeHook -> pylib PyObjFinalize; RAW magic for bound pairs which own +1 on recv; class layout kind 5 = variant fields); binding ARC + scope-exit release (owned = construction/call results via return-retain; borrows retain; field-store ARC pulled forward; PXXObjPlausible heap-envelope guard).

doloop RSS 595 -> 369 MB (as of the valgrind-profile night). Remaining tail to the <40 MB target:

• pyeval-side pinning (raw retains at PPyRec writes never released by pyeval's own storage; LclSet/globals lifetime)
• hidden desugar temps ('_py*', '') excluded from ARC — their construction refs still leak (needs a mid-body zero-init story before they can join)
• class-typed FIELDS not walked by the finalizer (kind for tyClass fields + release in PyObjFinalize/PXXRecordRelease) — field refs leak on instance death
• aarch64 inline EmitVariantClearA64/RetainA64 lack the object arms (leak-only asymmetry; arm32/i386/rv32/xtensa go through portable PXXVarClear and are covered); scope-exit tyClass release arm is x86-64 only so far
• d = None (RHS non-class) stores raw — old binding's ref leaks
• shared instance in two locals canary test (verification set) not yet written

Night follow-ups (2026-07-23, fable-a-n)

Landed since the slice-4 note: variant hidden-dest temp pre-call clear; mid-body tyClass watermark zero-init (hidden temps join ARC); refcounted pyeval closure objects (RAW2 magic + registry recycle stack, VT_PYCLOSURE=9 in all ARC arms); construction-in-arg spill to owning temp (pathIdx>=1). doloop 595 -> 413 MB; plain container/bound-method churn probes flat.

Night 2026-08-30/31 (frankS) — two landed fixes, and the NEXT list re-measured

Every item below was measured, not read. Probe shape: the same program at two loop counts, /usr/bin/time max RSS, CPython as the oracle for the printed value. Scratch probes, not tests, except where a test is named.

LANDED 1 — 1205cf286: every managed field of every NilPy class instance leaked, unless the program happened to build a container. PXXObjFinalizeHook is what PXXObjRelease calls at rc=0 to release an instance's children. All nine install sites are pylib/pyeval CONSTRUCTORS (pylist_new, pydict_new, pybound_new, bytes, the iterators), so a program that builds user-class instances and never a container ran with the hook nil: the block was freed and not one field released. 200k constructions of a class with one 2000-byte string field: 410 MB peak. Adding an unrelated dummy = [1] to the same program made it flat at 980 kB — the leak was gated on a feature the program did not use. Fixed from pylib's own initialization, the way pyeval's section already fixed the identical shape for PyIterCallHook. Byte-identical code size. Guard: test_nilpy_class_field_no_container_no_leak.npy (RSS ceiling 20 MB; positive control run — it fails pre-fix by 20x), and the .npy carries a warning that a container added to it anywhere disarms it.

LANDED 2 — ca8153b6c: a construction stored into a VARIANT was retained twice, on all three backends with an inline object arm. A NilPy construction is lowered through a conduit local (inst := PXXObjAlloc(size), VMT stamp, ctor, then LOAD_SYM), so the variant store — decided in CODEGEN — never saw the IR_CALL. The AST-level arms ask IRNodeYieldsOwnedRef and see the AN_CALL, which is why a tyClass local and a tyClass FIELD were already flat and only the variant slot leaked. Three backends, three different wrong answers: x86-64 listed the three call kinds, aarch64 listed none, i386 retained unconditionally (so i386 leaked an ordinary v = f() too). One predicate, IRNodeOwnsManagedObj, forwarded in compiler.pas beside its string twin; it can answer because the conduit is flagged where it is minted (SymIsCtorResultTemp). 400k o.w = Inner(i) into a variant field: 22932 kB -> 1044 kB. Guard: test_nilpy_object_in_variant_slot_survives_churn.npy, which is also the "shared instance in two locals" canary this ticket listed as unwritten — it reads keep.v and an a is b pair AFTER 50k constructions have churned through the same conduit, so it fails in the other direction too.

The NEXT list below is STALE in three places. Re-measured:

• Item 3 (class-typed FIELDS in the finalizer) is DONE and was already done. ClassFieldNeedsFinal admits isNilPy and tyClass (rtti_emit.inc:66-71), rtti_emit.inc:1465 stamps member kind 6, and PXXRecordRelease has the kind-6 arm (PXXObjRelease on the child). What made it look undone is LANDED 1: the walker was never reached.
• Item 5 (d = None rebind) is DONE for a local, and the FIELD case was never a missing release — o.w = None over a variant field measured flat once LANDED 2 removed the double retain. The AN_IDENT scalar-rebind arm is ir.inc:10082.
• Item 4 is HALF wrong and the surviving half is smaller than it reads. The aarch64 inline EmitVariantClearA64/RetainA64 really do lack the object arms (one lower-bound compare against VT_PROMO_BASE; tags 7..10 fall through) — but that is unreachable today, because NilPy does not build for aarch64 at all (bug-a-nilpy-on-cross-targets-four-remaining-walls; the wall has MOVED and that ticket's table is stale — it is now indirect call with more than 8 parameters at ir_codegen_aarch64.inc:3309, one of six separate > 8 refusals on that backend). The epilogue half — "scope-exit tyClass release is x86-64 only" — is TRUE (none of the five cross arms in EmitManagedLocalCleanupForTarget has a tyClass case) and is bounded at one object per scope, not per iteration: the loop case is covered by the IR-level rebind ARC, which is target-independent. Measured on arm32 (the one cross target NilPy reaches): x = Node(i) 400k times is FLAT, as is o.w = Node(i) into a variant field, and the churn test prints the right answer. Residual question, owned by this ticket: arm32/riscv32/xtensa take NO retain when boxing an object into a variant (VariantTagForTk then fill, no PXXObjRetain), which is right for a construction and wrong for a borrow — yet a borrow-then-drop probe (b.v = a in a callee, then 400 constructions of churn) printed correctly on arm32. It is balanced by the missing scope-exit release, so fixing either half alone turns a bounded leak into a UAF. They move together or not at all.

NEXT (ranked):

1. Literal-chain ownership: list/dict literals lower as Self-returning chains (Create.append(a).append(b)); the chain result IS the receiver, so receiver-position constructions cannot be ARC-spilled (test_nilpy_forin regression showed why). Fix in pyparser: hoist __py_t := Create + append statements (PyHoistHead exists), yield the temp IDENT as the expression. Closes arg-position literal leaks (exec(src, {...}) 15 MB / 20k probe; genexp-join wrapper build ~200 B/iter).
2. bug-n-pyeval-per-exec-leaks (see that ticket): ~24B/exec site-2 string + caller-side 64B with wrapper build.
3. Class-typed FIELDS in finalizer (kind for tyClass + release) — field refs leak on instance death.
4. aarch64 EmitVariantClearA64/RetainA64 object arms + non-x86 scope-exit tyClass release arm (leak-only asymmetry today).
5. d = None (RHS non-class) rebind leaks the old binding's ref.

RESOLVED 2026-08-31 (frankS) — the driver is met and exceeded, measured

make bench-uforth at 041204c7a, host plexus, uforth 07ffdb1:

The gate was "uforth doloop RSS bounded and near-CPython" against a starting point of 595 MB and a stated target of <40 MB. It is 16.7 MB, which is below CPython's own 24.8 MB, and flat across prelim and core too. 595 -> 413 -> 16.7.

Wall time is 0.68x CPython on that row and is NOT this ticket — it belongs to bug-o-uforth-blocktest-runs-slower-under-pxx-than-under-cpython. Saying so explicitly because a reader who wants this ticket to stay open will reach for that number.

What was verified, named exactly: make compiler/pascal26 byte-identical self-host at every step; gate.sh quick GREEN three times; nine named .npy canaries against the CPython oracle; the two new tests; an object-into-variant Pascal probe RUN on i386, aarch64, arm32 and riscv32; bench-uforth. NOT verified here: test-nilpy in full, which is full-tier and Track T's sweep, not this lane's gate (CLAUDE.md's per-fix loop supersedes this ticket's own Gate: line).

The residual tail, all of it, now has somewhere to live — none of it is carried by this ticket any more:

• bug-a-threadsafe-on-x86-64-leaks-every-managed-class-field-and-it-is-not-benign [A p55] — found underneath these fixes: both hold under --threadsafe, but PXX_TS_HARDLOCK gates PXXClassFinalize's managed-field pass off entirely, and plain Pascal goes 392 kB -> 398336 kB. Pascal-wide, not NilPy.
• bug-a-cross-backends-neither-retain-into-a-variant-nor-release-a-class-local-and-the-two-must-move-together [A p35] — item 4's epilogue half, plus the coupling that makes the obvious first move a UAF.
• bug-a-aarch64-has-no-stack-argument-passing-for-five-of-six-call-kinds [A p55] — what makes item 4's aarch64 half untestable. frankA is taking it.
• Item 1 (literal-chain ownership) is not carried forward as a ticket: its three named symptoms — arg-position dict/list literals, the genexp-join wrapper, exec(src, {...}) — all measure FLAT at HEAD over 10x the iterations, against pinned which leaks all three (16728 kB / 24536 kB at 200k). Attributed by control: they were already flat with ca8153b6c's arm reverted, so an earlier commit closed them, not tonight's. File a new ticket if a leaking shape is found; do not reopen this one on the strength of the prose above, which describes 2026-07-22.
• pyeval-side pinning: bug-n-pyeval-per-exec-leaks is already in done/, and its exec probe measures flat here.

Log

• 2026-08-02 — moved working/ -> unfinished/ by claude@xeon during T-queue triage. Last substantive commit 2026-07-23 (485c25c21, doloop RSS 369 MB after the concat + hidden-dest fixes), so the lock had been held 10 days with no agent on it. The remaining numbered items at the end of this ticket are real and unstarted. Owner fable-a-n left in place: this frees the lock, it does not reassign the work — re-claim to resume.
• 2026-08-31 — resolved, commit e8c32bb8b.