← board

Dynamic compiler tables — kill the fixed array[0..MAX_*] ceilings (+ dynarray dogfood)

Problem

The compiler holds ~305 fixed parallel arrays array[0..MAX_*-1] in defs.inc. Two costs:

  1. Hard ceilings. Each MAX_* is a wall a big translation unit can hit (sqlite hit MAX_TOKENS; lua/sqlite will push MAX_AST, MAX_IR, MAX_SYMS, MAX_UFIELD, MAX_CTYPEDEF, MAX_CPREP_*, …). Each overflow is a manual bump + recompile + (because the bump changes the compiler's own bss) a stabilize/pin cycle.
  2. Static BSS bloat. These tables dominate the compiler's ~165 MB bss. Most of it is reserved for worst-case inputs and never touched. Bumping a cap (e.g. 512K→2M tokens, ×3 parallel arrays) quadruples that slice for every compile of every program, however small.

Proposal

Convert the largest / most overflow-prone tables from fixed array[0..MAX_*] to dynamic arrays that grow on demand (geometric, e.g. ×2 with an initial modest reserve). Keep the MAX_* as a sanity hard-cap if wanted, but allocate to fit.

Priority candidates (biggest + most overflow-prone first)

Smaller bounded tables (MAX_ARR_DIMS, MAX_CPREP_CONDS, MAX_GOTO_LABELS, …) can stay fixed — they are genuinely small and bounded.

Bonus — dynarray correctness dogfood

The compiler is the densest dynamic-array user we have. If Tokens[] et al. become managed dynarrays grown via SetLength, then self-hosting exercises dynarray growth/realloc on every compile, across every backend, with the byte-identical fixedpoint and the cross harness as oracles. Any latent bug in dynarray grow / managed-element handling / cross-target dynarray ABI would surface as a self-host or cross divergence. Free, brutal, deterministic coverage.

Landmines

Performance angle (2026-06-29)

Raised after the make benchmark run (commit 9eecff79 era):

User hypothesis: the speed cost is largely memory management — we reach for fixed array[0..MAX_*] static storage where a grow-on-demand dynarray belongs, and pay for it in a ~165 MB BSS that is touched/cache-thrashed and reserved worst-case on every compile.

Honest scoping (don't oversell): the dominant self-compile lever is still register allocation (no regalloc → ~2× baseline, per [[project_make_test_timing_analysis]]) — converting tables to dynarrays will not close the 2.96× gap on its own. Its perf wins are real but secondary: smaller resident set / better cache locality / faster process startup (less BSS to map+zero), plus killing the manual MAX_* bump+reseed treadmill. Treat perf as a bonus on top of the capacity+RAM+dogfood case above, and measure (wall-time self-compile + RSS + hello startup before/after) rather than assume.

Execution constraint — do this on a dev branch, NOT master

This is a big destabilizing overhaul that touches the compiler's hottest data structures. It breaks self-host byte-identical until it converges and needs a multi-gen reseed + re-pin. Unlike the usual Track-A "work on master" rule, the user has explicitly scoped this one to a separate git dev tree / branch: land it incrementally there, get make test + self-host fixedpoint + full cross

Acceptance

Log

Progress log — 2026-07-18 (agent opus-A): incremental-on-master approach PROVEN

Revised execution model — the "dev branch + multi-gen reseed" constraint is NOT needed for the incremental, one-family-at-a-time path. Converting a single parallel- array family in isolation lands on master byte-identical with NO reseed: the self-host gate is fixedpoint reproducibility (compile self twice → identical), not "same as before", and a deterministic static→dynamic swap keeps the fixedpoint. Proven twice below. The dev-branch caution still applies to a big-bang all-at-once rewrite; do it incrementally instead.

The pattern (see [[project_dynamic_compiler_arrays_pattern.md]] in agent memory): array[0..MAX_X-1] of T → array of T; EnsureXCapacity(need) (double from a base, grow ALL parallel arrays in lockstep) at the ONE append chokepoint; drop the overflow Error. Gate: rebuild fixedpoint cmp + a build-time-generated over-cap test + quick.

DONE:

REMAINING — priority (biggest BSS / most overflow-prone first):

  1. Tokens family (MAX_TOKENS=2M, 8 arrays incl. the large TRawToken record — the single biggest BSS consumer, ~100 MB, and the one sqlite already broke). LAND- MINE: audit for any @Tokens[i] / raw pointer held across a grow (realloc moves the buffer) — the token buffer is the most likely place code takes element addresses. Chokepoint = the lexer's token-append.
  2. Syms family (MAX_SYMS=131072, 31 parallel Sym* arrays) — chokepoint AllocSym (remember the Alloc*-resets-ALL-fields landmine: [[project_symtab_alloc_parallel_array_landmine]]).
  3. UField family (MAX_UFIELD=262144, 26 arrays, C-frontend heavy).
  4. Single buffers: Code (8 MB), Data (2 MB), CPrepChars (8 MB) — held-address audit matters most here.

Superseded [[feature-dynamic-compiler-arrays-ast-fixups]] (folded into this ticket). Note the seq-walk STACK OVERFLOW (~3500 chained statements SIGSEGVs the recursive AST/IR tree walk) is a SEPARATE problem — a stack-depth limit, not an array cap; needs an iterative worklist, out of scope here.

Update 2026-07-18 (cont.) — Tokens DONE (biggest win)

REMAINING: Syms (131072×31, chokepoint = the 5 Alloc* in symtab.inc), UField (262144×26), Code/Data/CPrepChars buffers.

Update 2026-07-18 (cont. 2) — Syms + UField DONE

REMAINING: Code/Data/CPrepChars byte buffers (held-address audit matters), label arrays (MAX_IR-sized), smaller MAX_ tables (CTypedef/CPrep*/DBG_VARS).