← board

Lazy / conditional CurTok.SVal materialization — cut per-token string allocation

The cost

FPC-symboled profile of the self-compile (build fpc -g -o... compiler.pas, perf record, post the label-clear fixes 984df81f/50549a81) is dominated by managed-string + heap churn, ~30% aggregate:

NewAnsiString 8.5%  ansistr_concat 6.9%  ansistr_setlength 6.7%
SysGetMem_Fixed 4.5%  SysFreeMem_Fixed 3.7%  AppendChar 3.4%
ansistr_decr_ref 2.8%  Move 2.4%  ... (SetCodePage/StringCodePage ~3% is
FPC-RTL-only overhead the pxx binary lacks)

Next (lexer.inc) materialises a fresh AnsiString into CurTok.SVal for every token the parser consumes (one NewAnsiString + heap alloc + later free each). Token-string materialisation was already made O(n) instead of O(n^2) per token in 0058a31e (SetLength+fill vs per-char AppendChar), but the per-token ALLOCATION remains.

The idea

Roughly half of Pascal tokens are operators / punctuation / keywords whose meaning is fully in CurTok.Kind (tkBegin, tkPlus, tkSemicolon, …) — their SVal is never read. Populate SVal only for the text-bearing kinds (tkIdent, tkString, tkChar, and the number kinds if any code reads their text); leave it empty (no alloc) otherwise. That removes a large fraction of the NewAnsiString/SysGetMem/SysFreeMem traffic at the top of the profile.

Why it is NOT a quick fix (the audit)

CurTok.SVal is read ~292× in parser.inc alone plus every other frontend that shares the lexer. Skipping materialisation for a kind that some path DOES read = silent miscompile. It must be gated on a proven-complete set of "SVal-bearing" kinds:

Gate

Self-host byte-identical at every opt level (the emitted compiler must not change), full make test, make test-opt, and cross-target output-equality on supported programs (i386/aarch64/arm32; riscv32 is a partial target — errors on unsupported nodes, exclude). Land only with T up (async matrix) OR after running the full cross-bootstrap locally, because it touches the shared Next on the hot path of every frontend.

Expected win

Upper bound is the NewAnsiString + matching getmem/freemem share attributable to non-text tokens — plausibly 5-10% of self-compile if ~half of tokens stop allocating. Measure before believing it (measured-not-speculative; several optimization candidates this campaign were rejected at 0-fire / OoO-hidden).

RESOLUTION 2026-07-11: REJECTED — measured 0 win on the shipping (frozen) binary

Prototyped in full (gated pool storage in LexAll/LexAppend to a proven SVal-bearing kind set + keep-everything inside inline asm..end regions + monotone SOffset preserved for the decl-order gate). Result on the default frozen-string self-host binary:

Root cause of the non-transfer: the motivating profile was of the FPC-built binary, whose RTL uses heap-allocated AnsiStrings (NewAnsiString/getmem/freemem per token). The self-host/pinned binary is built with frozen inline strings (PXX_MANAGED_STRING is never defined by default — -u only strips it for the explicit frozen bootstrap), so CurTok.SVal materialisation is a length store + short memcpy, no allocator traffic at all. The 30% alloc-churn share simply does not exist in the shipping binary. Managed-build timing could not be taken: the checked-in pascal26-managed seed is stale (errors "too many array constant elements" on HEAD source even unpatched — ordinary backward-compat reseed situation, not a regression).

Salvage: the SVal-read audit (useful for any future retry)

Exhaustive audit of all ~292 CurTok.SVal reads in parser.inc: keyword-kind SVal is LOAD-BEARING at these semantic sites, so any lazy scheme must keep text for this exact kind set:

Structural-keyword pool text is ~droppable, but the payoff only exists in FPC-hosted/managed builds, which are bootstrap/debug vehicles, not the perf path. Not worth the frontend-shared risk. Rejected per the campaign's measured-not-speculative rule.