← board

lib pyexec: a real exec() for Python-subset source (library, two engines)

exec(src, env) as a LIBRARY, semantics matching CPython's explicit-dict form (which is exactly how uforth calls it): no ambient scope capture, the host passes name -> value bindings; values are variants (int/float/str/ list/dict/object-ref/bound-method).

Contract (censused from uforth's 134 PYTHON blocks — all parse)

Statements: assign/augassign, if/elif/else, while/for(+break), def, return, raise, del, expression statements. Expressions: arith/bit/compare/boolop, calls, attribute access, subscripts incl. slices, f-strings, isinstance, ternary, tuple/list/dict literals, len/int/print builtins. Sane restrictions: NO import, NO class definitions, NO exec-in-exec, explicit env only.

Two engines, shared front

  1. Front: tokenizer + parser -> AST, cached per source string (blocks run per-CALL in uforth's inner loop — SWAP executes millions of times in the conformance suite; parse exactly once).
  2. Engine 1, tree-walker (ships first): walks the AST over variants; host object access (vm.here, vm.memory[i], push(x)) resolved through field/method RTTI. The correctness reference.
  3. Engine 2, JIT (later, own ticket when phase starts): same AST through the in-tree backend (asmcore/obj writer) -> native fn ptr cached in the word body. Env types are concrete at block-compile time, so attribute access compiles to fixed-offset loads. Forth-native shape: native words ARE compiled words.

Language / porting plan (user policy 2026-07-19)

Start in PASCAL (avoids the chicken-egg on NilPy features; libs are language-neutral per Track B). Port to NilPy once N is feature-complete enough — the port is then itself an N corpus exercise. Same public surface either way so consumers don't care.

Gate: standalone test suite driving the extracted 134-block corpus against recorded CPython results (no uforth needed); make lib-test green.

Track B note (2026-07-20 sweep)

Blocked-by edge added for [[feature-rtti-field-reflection]], which the ticket's own header lists as a dependency ("Depends on feature-rtti-field-reflection for the host bridge") — the tree-walker resolves vm.here / vm.memory[i] / push(x) through field and method RTTI, so there is no engine without it.

Also worth flagging for whoever ranks this: it is a large umbrella (a tokenizer + parser + AST cache, then a tree-walking engine, then a JIT), not a single slice. The one piece that is genuinely startable today and independent of the RTTI dependency is extracting uforth's 134 PYTHON blocks with their CPython results into a fixture corpus under test/ — that is worth doing on its own, because it pins the contract before any engine exists to argue with.

2026-07-21 — architecture confirmed with user; JIT = rainy-day

Reached this ticket by driving uforth to the point where PYTHON-bodied stdlib words (/, 2/, most of CORE) are the wall (see [[feature-nilpy-corpus-uforth]] MILESTONE 3 — full STD.UFO now loads). Examined both engines closely with the user. Decisions:

Startable-today piece unchanged and still recommended first: extract the PYTHON block corpus + CPython oracle results into test/, pinning the contract.

2026-07-21 — corpus extracted, contract pinned, milestones sized

tools/pyexec_corpus.py extracts + categorizes the blocks from a uforth checkout (nothing vendored — uforth stays the user's separate tree). Measured over the shipped .UFO stdlib: 131 blocks — 60 pure-stack, 71 vm-accessing.

Milestone ladder this yields (each shippable, gated on the corpus subset):

Front half (tokenizer + parser -> cached AST) is shared by all three and by the later JIT; build it once at M1.

2026-07-21 — host-bridge FOUNDATION landed; interpreter is what remains

The hard/novel keystone of engine-1 is done and tested:

Remaining = the interpreter (a laborious but standard tree-walker), build plan:

  1. compiler/builtin/pyeval.pas (new unit; uses pylib, typinfo). NOT auto-used until solid — build + test standalone via a Pascal driver first, so a parse error can't break every NilPy compile. pylib must EXPORT the variant ops the evaluator needs (pyvar_to_int/float/bool, pyvarobj, pymul_v, pyfloordiv_v already exported; ADD pyadd_v/pysub_v/pymod_v/compare to the interface).
  2. Trampoline dispatcher PyHostCall(vm, name, args, nargs): Variant — reflect name on vm's class (GetMethInfoByName), marshal each variant arg by paramKinds (variant param -> pass @variant; int -> pyvar_to_int; …), dispatch on (retKind, arity, float-ness) to the matching typed proc-ptr cast, box the result by retKind. M1 needs the 4 stack shapes + a GP-arity family; extend for M2/M3 method shapes.
  3. Tokenizer + recursive-descent evaluator over Variants for the pure-stack grammar (assign/augassign, names as locals in a TPyDict, int lits, arith/bit/ compare/floordiv, ternary, calls). Correctness-first: may re-parse per call; cache the AST later (SWAP runs millions of times).
  4. Host-call resolution (M1 convention): a call name(args) or vm.name(args) -> PyHostCall(g["vm"], name, args). Uses g["vm"]; sidesteps bound-method values for M1 (the env's callables are all vm methods of the same name). Proper bound-method capture ([[feature-nilpy-bound-method-value]]) folds in at M3.
  5. def/ns wrapper + NilPy wiring (last): EvalPyStmts sees def __body__(): body; store {body, g} in a single global pending slot, set l["body"] to a variant whose payload = &PyEvalTrampoline; ns["__body__"]() (PyMakeDynCall unboxes payload -> indirect call) runs the pending body. Then rename the parser.inc exec() binding from pyexec to EvalPyStmts and add ParseUsesUnit('pyeval') after pylib.

Checkpoint tag checkpoint-pre-exec-arc marks the pre-arc state for rollback.

2026-07-21 — M1 CORE landed (pyeval.pas), standalone-green

compiler/builtin/pyeval.pas created: the M1 tree-walker over Variants. Covers the pure-stack grammar — simple statements (;/newline), assignment + augassign, full expression grammar (ternary, and/or/not, comparisons incl. chains, |^&, <<>> arithmetic shifts, +-/ // %, unary -/+/~), int/float/hex literals, True/False/None, builtins (int/float/abs/bool/len/ord/chr/str/hex/min/max/print), and the host bridge push/pop/fpush/fpop reflected BY NAME (case-insensitive) through the trampoline. test/test_pyeval_m1.pas drives SWAP/OVER/ROT, bitops, shifts, ternary min/max, floordiv/mod, augassign, comparisons, float stack — ALL PASS. pylib gained the variant ops it needs (pyadd_v/pysub_v/pymod_v/pybit_v/ pyshl_v/pyshr_v/pyinvert_v/pyneg_v/pycmp_v/pyeq_v/pyint_v/pyvar_of_int/bool).

Gate: quick tier green, self-host byte-identical, RTTI-consumer tests + test-nilpy green. pyeval is NOT auto-used yet (standalone only), so zero blast radius.

Blockers hit + resolved / filed

M1 deferred tail (next)

M1 coverage measured (corpus-driven, 2026-07-21)

Ran all 60 pure-stack blocks through pyeval against a seeded stub VM:

So M1 core already runs the arithmetic/stack/bitwise words that segfault today; the remaining pure-stack blocks are precisely the bignum + compound-block tail already scheduled after M1.

2026-07-21 — compound blocks landed (if/elif/else, while, for, break)

pyeval now handles Python-indented COMPOUND blocks: INDENT/DEDENT in the tokenizer (offside rule), if/elif/else (inline + block), while (+break), for-in over range()/lists. Skipped branches walk the grammar with an Executing gate (no side effects); while/for re-walk the body token span each iteration (correctness-first). range() builtin added. test/test_pyeval_compound.pas — 10 cases incl. nested for + if-in-for — ALL PASS; M1 test still green; self-host byte-identical; quick tier green.

Locals moved OUT of the passed-in TPyDict into pyeval's own name/value arrays: TPyDict keyed by an AnsiString-boxed Variant is unreliable — store and indexof box the string inconsistently and a heap key's bytes go stale after the block returns (the pylib str-into-variant ownership landmine). Owned AnsiString names compared with = are exact. Globals (the vm handle) still read from the host dict, which works. Filed nothing new — this is a pyeval-side choice, though the underlying TPyDict AnsiString-key boxing inconsistency is worth a Track B look.

Next: M2 — attribute access (vm.here get/set via GetFieldPtr) + subscripts (vm.memory[i]); then M3 method calls. That unlocks the 71 vm-accessing blocks.

2026-07-21 — M2 landed: attribute + subscript access (field reflection)

pyeval now reaches host state through field reflection (GetFieldPtr):

test/test_pyeval_m2.pas (18 cases: read/write scalars, augassign, bytes/list subscripts incl. negative, computed-address store, loop-writing memory, cond with attr+subscript) — ALL PASS. M1 + compound tests green; self-host byte-identical; quick tier green.

Remaining for full vm coverage: M3 method calls (vm.define_word(...), list.append/insert, bytes.decode, str.join), slices (vm.memory[a:b], hex(x)[2:]), del, isinstance, f-strings, and the bignum tail. M3 + list/bytes methods are the last big unlock.

2026-07-21 — M3 (partial) landed: method calls + list literals

pyeval now dispatches METHOD calls and parses list literals:

test/test_pyeval_m3.pas (str upper/join, list append/insert, vm.push/vm.pop via trampoline, pictured-output build idiom) — ALL PASS. M1/compound/M2 green; self-host byte-identical; quick tier green.

Remaining M3 = the GENERIC native-call trampoline. Host methods with signatures beyond push/pop/fpush/fpop (vm.define_word(name, ...), vm.next_token_strict() -> str, etc.) currently error "unsupported host-call shape". Enumerating typed proc-ptr casts per (arity, retKind, paramKinds) covers a bounded set; the ticket's "generic native-call trampoline" (per-target asm thunk marshalling N variant args) is the clean general solution — the one genuinely novel runtime piece left. Also still open: slices ([a:b]), del, isinstance, f-strings, int.from_bytes, dict-literal, and the bignum tail.

2026-07-21 — generic trampoline generalized (arbitrary vm-method arities)

PyHostCall now dispatches ANY host method whose params are all Variant (the NilPy-compiled host case) plus the fpush/fpop Double shapes — no hand asm, just enumerated typed proc-ptr casts by arity (0..5) and return kind (Variant/void/AnsiString/Int64). Unlocks vm.define_word(name, body), vm.next_token_strict() -> str, vm.next_token(), and the rest of the method census. A Variant arg rides by address, the result via hidden-dest — pxx's own ABI.

test/test_pyeval_trampoline_shapes.pas (2/3-arg Variant methods, string-return, void multi-arg) — ALL PASS. All pyeval tests + self-host byte-identical green.

Remaining corpus gaps now: f-strings (8), dict/set literals, del, isinstance, int.from_bytes, and the bignum tail (13 double-cell MATH). None need new architecture — all mechanical grammar/builtin additions.

2026-07-21 — engine-1 FEATURE-COMPLETE (standalone); wiring parked

pyeval now covers essentially the whole Python subset the corpus uses: expressions, all control flow (if/elif/else, while+break, for+range), nested functions (def/return with own scope), attribute + subscript + slice read/write, str/list/bytes/dict methods, the generalized host trampoline (any all-Variant method arity + string/int/void returns), f-strings, is/is not/in/not in, isinstance/del/dict+set literals, raise, hasattr, repr, bytearray/bytes, int.to_bytes/from_bytes + slice-assign (MEMORY store words). 14 dedicated test files; ~84/131 corpus RUN_OK with a full stub (rest = the deliberately-deferred bignum tail + a few import/try blocks). Self-host byte-identical throughout.

The exec()->EvalPyStmts WIRING was proven end-to-end (a NilPy VM ran SWAP correctly) but auto-using pyeval regresses an unrelated str-index test with a runtime segfault — parked in [[feature-nilpy-wire-pyeval-exec]] for root-cause. The str->slit rename that unblocked the compile is on master. Bignum strategy is a Track U fork: [[decide-pyeval-bignum-strategy]].

2026-07-21 — bignum landed (double-cell MATH now correct)

The deferred bignum tail is done: pyeval integers auto-promote to promoint.pas's arbitrary precision on overflow and demote back when they fit (the variant changes shape). The ~13 double-cell MATH.UFO words (UM*, M*/, D+, D<, …) that RAN but returned WRONG values now compute correct 128-bit intermediates and split them back to two 64-bit cells. [[decide-pyeval-bignum-strategy]] resolved (B′). So the interpreter is now correct across the whole censused corpus, modulo the few import/try blocks. Only the exec()-into-NilPy WIRING ([[feature-nilpy-wire-pyeval-exec]]) remains before uforth actually runs on it.

2026-07-31 (Track B) — the last blocker is gone; the public surface is now GATED

[[feature-nilpy-wire-pyeval-exec]], the exec()-into-NilPy wiring this ticket was waiting on, is resolved. So the whole chain works from a .npy today, and it is proven the way this ticket asked for — against CPython, not against our own output.

test/lib_pyexec.npy is a valid .py. lib-test compiles and runs it, then diffs the entire output against python3 running the same file and fails loudly on any divergence. It covers what the contract promises: the explicit-dict form with no ambient capture, bound methods of a compiled class in env, arithmetic, for/range, if/else, bit operations, a host field read, a host method call, a def with its own scope inside the exec'd source, string concatenation and an f-string, and arbitrary-precision integers. Byte-identical.

Three gaps this found, all filed rather than absorbed

Driving the surface from outside uforth is exactly what surfaces the assumptions that came in with it:

  1. [[bug-pyeval-exec-requires-a-globals-key-named-vm]] — exec(src, env) refuses every host call unless env has a key literally named "vm". That is uforth's variable name, not the contract; any other consumer writing {"canvas": c, "draw": c.draw} fails with a message naming an identifier they never wrote.
  2. [[feature-nilpy-multi-arg-callback-bridges]] — a bound method in env can be called with zero or one argument and no more, so store(0, vm.here) fails while vm.store(0, vm.here) works. Same missing runtime piece the tkinter callable-with-args ticket needs, which is why it is filed once, in Track N.
  3. [[feature-pyeval-power-operator]] — ** is not in the expression grammar, so 2 ** 70 is a parse error. The bignum tail already landed, so the value has somewhere to go; only the grammar is missing.

And one that turned out NOT to be a pyeval bug at all: an integer wider than 32 bits truncates on the way OUT of exec, through every route — bound method, qualified call, field assignment, even a value the host itself put in env. print(y) inside the exec'd source prints it correctly, which is what identifies it: pyeval holds the value fine and a NilPy FIELD initialised self.v = 0 is a 32-bit Integer. Measured onto [[bug-nilpy-int-promotion-decided-statically-so-computed-overflow-wraps]], whose table had the boundary at 2^63 — for a field it is 2^31.

Where the ticket stands

Engine 1 is feature-complete, correct across the censused corpus, wired, and now gated from the outside. What remains under this umbrella is Engine 2, the JIT, which the 2026-07-21 architecture note puts on rainy-day by decision and gives its own ticket when that phase starts. Nothing else here is Track B work.

Log