← board

Dynamic compiler tables — kill the fixed array[0..MAX_*] ceilings (+ dynarray dogfood)

Problem

The compiler holds ~305 fixed parallel arrays array[0..MAX_*-1] in defs.inc. Two costs:

  1. Hard ceilings. Each MAX_* is a wall a big translation unit can hit (sqlite hit MAX_TOKENS; lua/sqlite will push MAX_AST, MAX_IR, MAX_SYMS, MAX_UFIELD, MAX_CTYPEDEF, MAX_CPREP_*, …). Each overflow is a manual bump + recompile + (because the bump changes the compiler's own bss) a stabilize/pin cycle.
  2. Static BSS bloat. These tables dominate the compiler's ~165 MB bss. Most of it is reserved for worst-case inputs and never touched. Bumping a cap (e.g. 512K→2M tokens, ×3 parallel arrays) quadruples that slice for every compile of every program, however small.

Proposal

Convert the largest / most overflow-prone tables from fixed array[0..MAX_*] to dynamic arrays that grow on demand (geometric, e.g. ×2 with an initial modest reserve). Keep the MAX_* as a sanity hard-cap if wanted, but allocate to fit.

Priority candidates (biggest + most overflow-prone first)

Smaller bounded tables (MAX_ARR_DIMS, MAX_CPREP_CONDS, MAX_GOTO_LABELS, …) can stay fixed — they are genuinely small and bounded.

Bonus — dynarray correctness dogfood

The compiler is the densest dynamic-array user we have. If Tokens[] et al. become managed dynarrays grown via SetLength, then self-hosting exercises dynarray growth/realloc on every compile, across every backend, with the byte-identical fixedpoint and the cross harness as oracles. Any latent bug in dynarray grow / managed-element handling / cross-target dynarray ABI would surface as a self-host or cross divergence. Free, brutal, deterministic coverage.

Landmines

Performance angle (2026-06-29)

Raised after the make benchmark run (commit 9eecff79 era):

User hypothesis: the speed cost is largely memory management — we reach for fixed array[0..MAX_*] static storage where a grow-on-demand dynarray belongs, and pay for it in a ~165 MB BSS that is touched/cache-thrashed and reserved worst-case on every compile.

Honest scoping (don't oversell): the dominant self-compile lever is still register allocation (no regalloc → ~2× baseline, per [[project_make_test_timing_analysis]]) — converting tables to dynarrays will not close the 2.96× gap on its own. Its perf wins are real but secondary: smaller resident set / better cache locality / faster process startup (less BSS to map+zero), plus killing the manual MAX_* bump+reseed treadmill. Treat perf as a bonus on top of the capacity+RAM+dogfood case above, and measure (wall-time self-compile + RSS + hello startup before/after) rather than assume.

Execution constraint — do this on a dev branch, NOT master

This is a big destabilizing overhaul that touches the compiler's hottest data structures. It breaks self-host byte-identical until it converges and needs a multi-gen reseed + re-pin. Unlike the usual Track-A "work on master" rule, the user has explicitly scoped this one to a separate git dev tree / branch: land it incrementally there, get make test + self-host fixedpoint + full cross

Acceptance

Log

Progress log — 2026-07-18 (agent opus-A): incremental-on-master approach PROVEN

Revised execution model — the "dev branch + multi-gen reseed" constraint is NOT needed for the incremental, one-family-at-a-time path. Converting a single parallel- array family in isolation lands on master byte-identical with NO reseed: the self-host gate is fixedpoint reproducibility (compile self twice → identical), not "same as before", and a deterministic static→dynamic swap keeps the fixedpoint. Proven twice below. The dev-branch caution still applies to a big-bang all-at-once rewrite; do it incrementally instead.

The pattern (see [[project_dynamic_compiler_arrays_pattern.md]] in agent memory): array[0..MAX_X-1] of T → array of T; EnsureXCapacity(need) (double from a base, grow ALL parallel arrays in lockstep) at the ONE append chokepoint; drop the overflow Error. Gate: rebuild fixedpoint cmp + a build-time-generated over-cap test + quick.

DONE:

REMAINING — priority (biggest BSS / most overflow-prone first):

  1. Tokens family (MAX_TOKENS=2M, 8 arrays incl. the large TRawToken record — the single biggest BSS consumer, ~100 MB, and the one sqlite already broke). LAND- MINE: audit for any @Tokens[i] / raw pointer held across a grow (realloc moves the buffer) — the token buffer is the most likely place code takes element addresses. Chokepoint = the lexer's token-append.
  2. Syms family (MAX_SYMS=131072, 31 parallel Sym* arrays) — chokepoint AllocSym (remember the Alloc*-resets-ALL-fields landmine: [[project_symtab_alloc_parallel_array_landmine]]).
  3. UField family (MAX_UFIELD=262144, 26 arrays, C-frontend heavy).
  4. Single buffers: Code (8 MB), Data (2 MB), CPrepChars (8 MB) — held-address audit matters most here.

Superseded [[feature-dynamic-compiler-arrays-ast-fixups]] (folded into this ticket). Note the seq-walk STACK OVERFLOW (~3500 chained statements SIGSEGVs the recursive AST/IR tree walk) is a SEPARATE problem — a stack-depth limit, not an array cap; needs an iterative worklist, out of scope here.

Update 2026-07-18 (cont.) — Tokens DONE (biggest win)

REMAINING: Syms (131072×31, chokepoint = the 5 Alloc* in symtab.inc), UField (262144×26), Code/Data/CPrepChars buffers.

Update 2026-07-18 (cont. 2) — Syms + UField DONE

REMAINING: Code/Data/CPrepChars byte buffers (held-address audit matters), label arrays (MAX_IR-sized), smaller MAX_ tables (CTypedef/CPrep*/DBG_VARS).

Update 2026-08-21 (agent-A) — six more families, and a MEASUREMENT that contradicts the ticket's premise

Landed, each its own commit, each with the self-host fixedpoint byte-identical:

family arrays BSS freed
AsmDisProcAtPos 1 67.1 MB
CallFix + CodeRef + the DCE graph tables 5 33.5 MB
the per-ROUTINE family (Proc*, ProcParam*, PyCapName, InlineLocalTk) 90 38.8 MB
SymArrDimLo / SymArrDimSpan 2 6.3 MB
GlobFix 1 2.1 MB
IRSeqSpine 1 4.2 MB
Code 1 16.8 MB

Compiler BSS 246.8 MB -> 75.9 MB. (It was 146 MB when this ticket was parked; the rise since was ordinary growth plus ~28 MB I added earlier the same day for [[feature-emission-size-dce]], which is why that one is in the table.)

Three caps are gone with the reservations: MAX_GLOBFIX — the one measured to have been hit for real, by the --threadsafe self-host, 121 entries in — MAX_IR_SEQ_SPINE, and the internal call/code-reference tables. MAX_CODE and MAX_PROCS stay as hard caps on purpose.

The premise does not survive measurement

The ticket's performance framing was that the fixed tables are "touched/cache- thrashed and reserved worst-case on every compile". They are reserved worst-case; they are not touched. BSS is demand-zero — an untouched page never becomes resident — so the reservation was free, and replacing it with heap is not.

Max RSS, same inputs, pre-session binary vs HEAD:

compiling before after
hello.pas 24.2 MB 26.3 MB
a NilPy module 55.3 MB 66.2 MB
compiler.pas (self-compile) 453.1 MB 497.0 MB

Wall time is unchanged (self-compile 26.9s both ways), so the extra indirection per element does not show. But RSS went UP, by ~10%, which is the opposite of what the ticket predicted and the opposite of what a reader would assume from "BSS 246 MB -> 76 MB".

Why, exactly

Not the live data — that is the same bytes either way. It is the GROWTH: SetLength is allocate-copy-free, and the freed block cannot serve the next doubling, because the large-block free list is first-fit on size >= request and every subsequent request is BIGGER. So each table leaves behind the whole geometric series of its previous buffers — about one final-size worth of garbage per table, permanently unreusable by that table.

That is the root cause, it is in the allocator rather than in any table, and it costs every pxx program that grows a dynamic array, not just the compiler. Filed as [[feature-opt-dynarray-grows-in-place]]: give the scalar-array SetLength path the in-place-when-unique + geometric-headroom treatment the AnsiString path already has, three arms away in the same case statement.

What is left here

Procs (8.4 MB, an array of RECORDS holding managed strings — a different SetLength path), TokChars / LoadFileBuf / CPrepChars (8.4 MB each), Data (2.1 MB, ~20 unguarded append sites), the TemplateTokens / SpecializeTokens pair (1.8 MB each), the MAX_IR-sized label arrays.

Do the allocator ticket before any of them. Converting more tables now buys a smaller number in the bss= line and pays for it in resident memory; after in-place growth lands, the same conversions are close to free.

The trap, for whoever picks this up

MAX_PROCS was also being used as an INDEX bound in nine places — for pi := 0 to MAX_PROCS - 1 do ProcSigOff[pi] := -1 in rtti_emit.inc became a write off the end of a heap block the moment the table stopped being that long. The self-host gate did NOT catch it (it happens to sit in NilPy's signature emitter). Grep every MAX_<FAMILY> after converting a family, and treat each remaining use as a question: is this a capacity bound (fine) or an index bound (now wrong)?

Progress log — 2026-08-21 (agent-A): the REMAINING list above is STALE

Read the source before picking an item from it. Measured today:

family list above says actually
Tokens (MAX_TOKENS) remaining, #1 done — Tokens/TokPackRecords/Tok*Checks/CAttr* are array of, grown by EnsureTokCapacity
Syms (MAX_SYMS) remaining, #2 done — Syms + the ~30 Sym* arrays grown by EnsureSymCapacity
UField (MAX_UFIELD) remaining, #3 done — 26 arrays grown by EnsureUFieldCapacity
IR / AST done done

Still genuinely fixed: Data (MAX_DATA), Strs (MAX_STRS), CPrepChars (MAX_CPREP_CHARS), TokChars/LoadFileBuf (STRING_CAP, 8 MB each), LabelFixupPos/LabelFixupTarget (MAX_IR — 1 MB each), UCls* (MAX_UCLASS = 2048), Procs itself (deliberate, see EnsureProcCapacity's note). Code is already dynamic.

The conversion has a second half nobody ran, and it is where the remaining risk sits: deleting a cap does not delete the code that assumed it. Four sites had taken MAX_X to mean "a number the count can never reach" and were left behind — one of them an out-of-bounds stack write in IRVerify, which runs on every body. Filed and fixed as [[bug-a-dynamic-tables-left-their-fixed-size-shadows-behind]].

So: before converting the next family, grep for its MAX_ name across compiler/** and read every hit. A hit that is not the array declaration is either a real remaining cap (fine — LabelFixup* still is one) or a shadow that the conversion has just made wrong. That grep is now part of the pattern, not an afterthought.

2026-08-30 — RE-MEASURE (triage only, nothing applied): still genuine

Checked in the parked-ticket pass. No resume condition names another ticket: the five resolved slugs here are cited landed work, and the one open slug (feature-opt-dynarray-grows-in-place) is a pointer, not a blocker.

This ticket is an incremental conversion with its method already written down — before converting the next family, grep for its MAX_ name across compiler/** and read every hit, because deleting a cap does not delete the code that assumed it (the IRVerify out-of-bounds write is the worked example). That instruction is the ticket's real value and it is intact.

Re-priced: unchanged. Parked for want of an agent, not for want of a bridge.

2026-09-05 (frankH) — LoadFileBuf converted, and the interesting part is WHO runs it

LoadFileBuf was array[0..STRING_CAP-1] of Byte — 8 MB of BSS in every compiler this repo ships. It is now array of Byte, grown by LoadFile, with LoadFileCap never shrinking across loads. Measured on the self-host build: bss 106842604 -> 98454060, exactly the 8 MB.

The claim I nearly shipped, and what measuring it actually found. I wrote, first, that this removed a silent truncation: one sysread of at most STRING_CAP means a file over 8 MB is read short. Then I tested it — a 30 MB unit through the PINNED compiler, expecting a truncation error — and it compiled and ran correctly. That is impossible if LoadFile were the reader.

It is not. LoadFile is intercepted in the parser as a builtin (pasparser_stmt.inc, backed by PXXStrLoadFile), so a self-hosted pxx never executes the Pascal body at all. The 8 MB array was BSS that path allocates and never touches. The body is live only under the FPC-seeded cold-bootstrap compiler, where sysread is fpRead and nothing intercepts the call.

So the correctness half is real, narrow, and now has a before/after control. Two FPC seeds built from the same tree, one with the change stashed:

30 MB unit small unit
seed WITHOUT the change pascal26:88305: error: unterminated comment works
seed WITH the change compiles, prints 777 works

Line 88305 is where the 8 MB cut lands. The genuinely SILENT case is not source at all — it is {$R}: resources_emit.inc calls LoadFile to embed a file's bytes, so a resource over 8 MB was embedded SHORT with no diagnostic, the build succeeding and the blob simply wrong. A truncated Pascal source usually errors, but about whatever the cut leaves dangling, which misnames the fault.

It also stopped borrowing STRING_CAP. That constant is the TOKEN CHAR POOL's capacity and carries ~40 overflow checks; it was also sizing this unrelated read buffer, so a bump for one silently moved the other. Following this ticket's own method — grep the MAX_ name and read every hit — is what surfaced that: of the ~40 STRING_CAP hits, exactly one was about this buffer.

Method note for the next family. The grep this ticket prescribes found the shared constant, but it would NOT have found the builtin interception, because nothing in elfwriter.inc or defs.inc says the body is dead in self-host. The question that found that was "what would this be if it were false" — run the old binary on an oversized input and see whether it actually breaks. Add that to the method: before claiming a fixed table costs correctness, make the current compiler fail on it. A table can be pure BSS waste and no ceiling at all, and those two justify very different amounts of risk.

Still fixed after this: Data, Strs, CPrepChars, TokChars, LabelFixupPos/LabelFixupTarget, UCls*, and Procs deliberately.

2026-09-06 (frankH) — CPrepChars converted, and the cap was REAL this time

CPrepChars : array[0..MAX_CPREP_CHARS-1] of Char (8 MB) → array of Char grown geometrically from 256 KB in its single writer, CPStoreRange. MAX_CPREP_CHARS is deleted: it had exactly one code use, this table's bound.

bss 98454060 → 90066100, the 8 MB, on top of LoadFileBuf's.

The method step, and why it earned its place

The previous family (LoadFileBuf) taught that a MAX_ grep finds the constant and not the reachability, so the rule added there was: before claiming a fixed table costs correctness, make the current compiler fail on it. Applied here it paid immediately, and it paid in BOTH directions:

So the generalisation is not "grep the cap, then try a big input". It is two caps can be in range of one input, and which one you hit is a ratio. Macro COUNT and macro TEXT are independent axes; an input that maximises one tells you nothing about the other, and the diagnostic is the only thing that says which axis you actually tested. Read the error string, not the exit code. An unreachable ceiling and a ceiling you failed to aim at are the same rc=1.

The method, stated symmetrically — a probe's SUCCESS is not evidence either

The rule inherited from LoadFileBuf was make the current compiler fail on it, and the near-miss above sharpens its refusal half: a refusal from a DIFFERENT limit reads exactly like your answer, so read whose message it is.

The other half has no diagnostic at all and is the quieter of the two. After converting, the 13.8 MB input answers rc=0 — and rc=0 is not the proof. A pool that grew but silently truncated, or mis-deduplicated a stored range, also exits 0 and also emits a linkable binary, because nothing in that file's macros has to be used. The success face has the same structure as the refusal face: it answers, it does not error, and it is correct about something else (that the compiler did not crash).

So, for a reachability probe on a converted table:

Scope of the claim

Reachable with a synthetic input, measured. I have not shown a real header reaching 8 MB of macro text, and this ticket should not claim one. What the measurement does establish is that this was a ceiling and not pure BSS waste — which is the distinction that decides how much risk the conversion is worth, and it is exactly the distinction LoadFileBuf turned out to fail.

Verification

Two findings for the NEXT families (grep done, conversion not)

Constraint on the conversion pattern (from frank-coordinator's landmine)

This ticket's pattern is realloc that PRESERVES INDICES, and that is load bearing. Geometric growth is safe for zero-init sentinel columns like AliasEnumId (defs.inc, stores enum index plus one so an unwritten row reads 0 = NONE, written by one of six allocators, inert because AliasCount never decreases and no row is recycled). A free list or a compaction pass is not safe: it turns every allocator that does not write such a column into a stale read that fails open. Out of scope here — a free list for one of these tables is a different ticket with a different gate.

STALE-PARK, answered a second time — and the answer changed

progress.sh check reports STALE-PARK-HELD on this ticket. It is a false positive by the check's own note — the slug matched, not the question; blocked-by: is [] and the citations are prose references to landed work. The 2026-08-30 re-measure already adjudicated it once.

But re-reading it was not free of information, because one detail of that 2026-08-30 note is now stale: it recorded feature-opt-dynarray-grows-in-place as "the one open slug ... a pointer, not a blocker", and that ticket is now in done/, as is feature-emission-size-dce. Both cited dependencies have landed. That matters here rather than being bookkeeping: in-place dynarray growth is the thing that makes this ticket's doubling amortise, so the conversion pattern got cheaper after the note that dismissed the pointer was written.

Which is the actual lesson about the check: it fires on slug adjacency and it will keep firing, so it cannot be closed by being right once — but a report that is wrong about blocking was still right about staleness, and the ticket had a sentence that had quietly become untrue. Read it; do not act on it.

2026-09-06 (frankH) — Data converted, and it exposed a guard that could not fail

Data : array[0..MAX_DATA-1] of Byte (2 MB reserved in bss) → array of Byte, grown by a single helper DataEnsure(n) in util.inc (included after lexer.inc, where Error lives, and before every user). MAX_DATA deleted.

bss 90066100 → 87985348, the 2 MB.

The two tables were coupled, and the ticket's own method could not see it

The prescribed step is grep the MAX_ name across compiler/**. That found 26 MAX_DATA hits and zero connection to the string table — because the coupling is not an identifier, it is arithmetic between two independent constants.

Measured, on the pre-change compiler:

Confirmed by running it, not by the arithmetic: 52000 literals compile at data=2092792; 52200 answers error: data overflow; 66000 answered data overflow too. emit.inc's Error('string table overflow') was a guard that cannot fail, and it had no way to say so.

After the conversion the same 66000-literal input answers error: string table overflow. That is the positive control for the whole claim: the identical input moved from one cap's message to the other's, which is the only evidence that could distinguish "MAX_STRS was unreachable" from "I never aimed at it". Strs is therefore the next family and it is now load bearing, where before converting it would have changed nothing observable.

Generalisation for the remaining families: two tables can share a ceiling without sharing a constant. Grep finds shared identifiers; it cannot find a shared resource. Before converting a table, ask what else consumes the thing its cap is denominated in.

The one MAX_DATA use that was not a bound

symtab.inc had if base + n * esz >= MAX_DATA then Exit; — a policy threshold, not an array bound. Near the 2 MB reserve, the typed-const-array promotion silently gave up and left the array on the startup-store path, i.e. the ~29-bytes-of-code-per-element treatment that optimisation exists to delete, with no diagnostic. The function fails closed (Result := False default), so this read as "not eligible". Now DataEnsure(base - DataLen + n * esz): promotion no longer depends on how much .data the rest of the compile used first. Behaviour change, deliberate, in the direction the optimisation wants.

What the conversion actually broke, and how it was found

The first build segfaulted in round 2. Cause: writes into Data that never went through an overflow check at all, because with a fixed bss array they never needed one. A grown array is nil until asked.

The grep that finds these is not the MAX_ grep — it is every write to the table, and does an ensure dominate it. Scripted, not eyeballed, because the failure is silent for any run that stays under the initial 64 KB reserve: this class only crashes once the table is big, which is exactly the input nobody runs. Add to the method: after converting a table, enumerate its WRITE sites, not its CAP sites. The cap sites are the ones that were already thinking about the limit; the write sites are the ones that never had to.

DataEnsure also zeroes the newly grown region explicitly. Bytes at or past DataLen are read before being written — the static-array path aligns base up from DataLen and never writes the padding — and the old array was bss, i.e. zero by construction. SetLength is specified to zero new elements; the loop states the property rather than inheriting it, because a garbage alignment byte lands in .data silently and nothing in this compiler asserts on .data padding.

2026-09-06 (frankH) — Strs converted, immediately after the table that hid it

Strs : array[0..MAX_STRS-1] of TStrEntry → array of TStrEntry, doubled from 1024 in InternStr, its only writer. MAX_STRS deleted.

bss 87985348 → 86412492, the 1.5 MB (65536 × 24).

This is the first table here whose growth carries a managed field through the realloc — TStrEntry.Text is an AnsiString. Probed before converting rather than after: 5000 entries through 11 regrows, every earlier string compared back against its expected value, all intact.

Converted immediately after Data and not whenever, because until Data grew this cap could not fire at all. The proof is one input crossing three states: 66000 short literals answered data overflow before Data was converted, string table overflow after it, and now compile and run — 66000 lines, last qhp, which is exactly index 65999 in the generator's alphabet.

The write-site audit, second application

Strs has exactly one write block (emit.inc, three fields at [StrCount], immediately after the cap check) and every read is at an index below StrCount. That is what a clean conversion looks like, and it is worth recording as the contrast case: the audit is cheap when it finds nothing, and Data is why it gets run anyway.

Two stale citations repaired in the same commit

MAX_STRS was named in two comments that outlived it — MAX_UNITS' note and VisCacheVis', both recording that VisCacheVis used to be sized by the string-table cap. Both now say the constant is gone. The history stays because the point of those notes is the coupling, which was real and is the same mistake class this ticket keeps finding; but a comment citing a constant that no longer exists is a name with nothing behind it, and the next reader greps.

LANDMINE for whoever benefits from the lifted cap: InternStr is O(n²)

InternStr dedups by linear scan over the whole table, with an inner per-character compare, on every literal. Measured at HEAD (x86-64, warm):

literals wall
5000 1.29s
10000 2.48s
20000 6.30s
40000 25.69s

Doubling the input roughly quadruples the time by 40000. So removing the cap replaces a hard error with a practical time ceiling, which is strictly better — the old compiler did all the same scanning and then refused — but it means "the string table is unbounded now" is a claim about capacity, not about usability. A hash index over the pool is the fix and it is the natural next change; until it lands, do not quote the lifted cap as though 200k literals were practical. Nothing here is a regression: this cost is unchanged by the conversion, it is merely now reachable.

2026-09-06 (frankH) — InternStr hash index: the lifted cap made usable

Removing MAX_STRS swapped a hard error for a time ceiling. This closes it.

InternStr dedups through an open-chained index parallel to Strs (StrHashHead / StrHashNext, doubled at load factor 1) instead of scanning the whole table. Index identity is preserved — entries keep the numbers they would have had — so this changes only which entries get COMPARED, never which one wins: dedup guarantees at most one entry per bucket can match.

literals linear hashed
5000 1.27s 0.94s
10000 2.13s 1.30s
20000 6.60s 1.48s
40000 22.24s 1.68s
66000 (refused before Strs) 3.42s

Emitted output byte-identical at every size, which is the requirement: this is a performance change and any output difference would be a defect, not a feature.

The hash is djb2 masked to 24 bits inside the loop, so h * 33 stays under 2^29 and the result never depends on what signed overflow does — the compiler running this code is also the compiler being built by it.

The next cap in the chain, found by pushing past this one

200000 literals now answers error: fixup overflow — MAX_FIXUPS = 131072. Third time this ticket has hit a cap it was not aiming at, and the chain is now documented end to end for one input shape: data overflow → string table overflow → (time) → fixup overflow. Each conversion reveals the next ceiling, and the reveal is the only way any of them were known to be reachable.

MAX_FIXUPS sizes THREE parallel arrays — Fixups, FixupPCRel, FixupPicDelta — so it is the first family here where the ticket's "parallel arrays must grow in lockstep" landmine actually bites.

A DEFECT FOUND WHILE SCOUTING IT, not by this ticket's grep

Fixups is compacted in two places and only one of them is right.

FixupPCRel is initialised False per entry and set True on the i386 PIC path, so it is genuinely heterogeneous within one build; FixupPicDelta carries a per-site anchor delta. After a drop, every surviving fixup past i reads its neighbour's relocation flags — wrong addresses, silently.

Scope, corrected after probing — this is a LATENT invariant repair, not a bug with a demonstrated victim. The violation is certain: compiler.pas shifts the record and not the two arrays, and dce.inc states the rule. What is NOT established is that the arm ever runs.

I first wrote "common" here, sourced from emit.inc's comment calling a table-less sentinel "a documented answer, not a defect: a module that publishes no classes has no registry." That is a statement about design, and I read it as one about frequency. Measured since:

Exposure would additionally need a FixupPicDelta-bearing entry after the dropped index — i386 PIC, the class this repo's default instruments cannot see, since the dev loop, gate.sh quick and the pin all run on x86-64.

And the first repro was a comparison whose inputs did not exist: both compiles failed with "this object would define no linkable symbol", no .o was written, and cmp on two absent files printed DIFFERS. It was one step from being reported as the proof. The existence assertion is what caught it — the second repro built two 120972-byte objects, compared IDENTICAL, and then the precondition check showed the drop had not fired in it either, so that identity says nothing about the bug in both directions.

Fixing it anyway: two lines, no cost, and the comment-vs-code disagreement resolves cleanly in the comment's favour. Just not on a claim it cannot carry.

2026-09-06 (frankH) — the fixup family, and the chain terminates

Fixups / FixupPCRel / FixupPicDelta, all three array[0..MAX_FIXUPS-1], converted together through one helper FixupEnsure in util.inc. MAX_FIXUPS deleted. bss 86425036 → 83672548, 2752488 bytes.

All three grow in one procedure, and that is the design, not tidiness. They are parallel BY INDEX — the same rule the two compaction sites obey. A grow that moved one and not the others is the identical defect to a compaction that does, and this tree has already had one of those silently (a8bfcb695). Keeping the three SetLengths in one procedure means there is no site where you can grow the table without growing the columns, because there is only one site. This is also why it did not land in pieces: a partial landing here is silently wrong rather than loudly wrong.

The write-site audit found every [FixCount] write downstream of the single guard in EmitDataRef; the other writes are the two compactions, at indices already below FixCount.

The chain, complete — one input shape, five states

entries answer
52200 error: data overflow — MAX_DATA, 2 MB
66000 error: string table overflow — MAX_STRS, 65536
40000 22.24s — no cap, a time ceiling from the O(n²) intern
200000 error: fixup overflow — MAX_FIXUPS, 131072
500000 compiles, 14.04s, 992 MB peak RSS, runs: 500000 lines, last b6eF = index 499999

Each conversion revealed the next ceiling, and the reveal is the only way any of them were known to be reachable. The ticket's prescribed method — grep the MAX_ name — found each table's bound and could not have ordered them, because what ordered them was one oversized input asking the compiler which limit it would hit first. The terminal state is memory, not a constant.

What the orphaned comment turned out to know

Deleting MAX_STRS orphaned the paragraph that had justified raising it from 8192 to 65536 — a csmith --paranoid program needing 9426 distinct literals, refused at the old cap. Rewritten in place as history rather than dropped, because it explains the mechanism this ticket kept running into:

8192 was reachable and 65536 was not. At ≥40 bytes of Data per entry against a 2 MB MAX_DATA, the table could never exceed ~52108, so the new cap sat 13000 above anything Data could fund. The raise did not make the guard generous; it made it unreachable — and nothing in the raise's own evidence would have shown that, because the program that motivated it needed 9426 entries, comfortably inside both limits.

Raising a cap without checking the resource it is denominated in is how a guard stops being able to fire. That is the same failure as the MAX_STRS / MAX_DATA coupling recorded above, seen from the other end: the coupling was not introduced by anyone, it was introduced by a raise that only looked at one of the two numbers.

Remaining

TokChars (STRING_CAP) — still blocked behind splitting that constant from the shortstring type it also sizes (ast_syminfer.inc:151, ir.inc:2703). LabelFixupPos/LabelFixupTarget (MAX_IR), UCls* (MAX_UCLASS), and Procs deliberately.

2026-09-06 (frankH) — MAX_IR scouted and DELIBERATELY NOT converted

The next family by the old REMAINING list is LabelFixupPos/LabelFixupTarget (MAX_IR). Counted before choosing how to convert it, which is the step the fixup family taught, and the count says do not start.

MAX_FIXUPS (converted) MAX_IR
arrays sized by it 3, all parallel 10, several unrelated
cap-check sites 1 20 for LabelFixupCount alone (arm32 5, x86-64 8, aarch64 7)
uses that are not array bounds 0 7
total references 5 54

Three reasons this is a different job, not the next one in the rhythm:

  1. MAX_IR sizes ten tables that are not parallel to each other — LabelFixupPos, LabelFixupTarget, LabelFixupAnchor, LabelAddrFixPos, LabelAddrFixTarget, LabelPositions, WasmLabelBlock, WasmLabelStamp, WasmExcSlot, XtWideLabel. That is the STRING_CAP shape (one constant sizing two unrelated things) at five times the scale. The constant has to be split before anything is converted, and the split is the work.
  2. Seven uses are VALIDITY PREDICATES, not bounds — if (lblId >= 0) and (lblId < MAX_IR) in six backends plus wasm. Those ask is this label id plausible, and deleting the constant deletes the question. This is exactly the ticket's own worked example: four sites had taken MAX_X to mean "a number the count can never reach", one of them an out-of-bounds stack write in IRVerify. Here there are seven and they are spread across every backend.
  3. The structural answer is unavailable. Twenty cap sites means twenty Ensure calls, so lockstep between LabelFixupPos and LabelFixupTarget would be discipline at twenty sites rather than one procedure — the property the fixup family got for free because its old code had one check.

The cap-site count is knowable before the conversion and it decides whether the structural answer exists. Fixups had one check and its three columns could only be grown together; Data had 23 and needed 23 DataEnsure calls; MAX_IR has 20 across six backends and seven non-bound uses. Counting first is cheap and it is the difference between inheriting the good answer and promising to be careful twenty times.

Remaining, with the reason each is not next

2026-09-06 (frankH) — MAX_UCLASS scouted: it is a NUMBERING constant, not a bound

Counted, and it stops harder than MAX_IR. 51 arrays, 76 references, 2048 slots. Two of its uses put it in a different category from every constant this ticket has deleted:

1. It partitions a record-id space. symtab.inc:

if (rec < REC_UCLASS_BASE) or (rec - REC_UCLASS_BASE >= MAX_UCLASS) then Exit;
Result := UClsIsInterface[rec - REC_UCLASS_BASE];

MAX_UCLASS is the WIDTH OF A RANGE inside an encoded id, not the size of an array. cparser.inc reserves the top slot too (UClsCount >= MAX_UCLASS - 1). Growing the arrays does not make the encoding admit more classes — the id space has to be re-partitioned first, and every consumer of REC_UCLASS_BASE arithmetic is a consumer of that decision.

This is worth stating plainly because it is the exact distinction that came up tonight when frankS asked whether the constants I was deleting participate in a numbering scheme. The four I deleted (MAX_CPREP_CHARS, MAX_DATA, MAX_STRS, MAX_FIXUPS) were independent size bounds — nothing was defined in terms of them and nothing counted from them. MAX_UCLASS is the one in this file that is not, and a session working down the REMAINING list without checking would find that out after the conversion rather than before.

2. Four uses are INFINITE-LOOP GUARDS. pasparser_generic.inc (three) and pyparser.inc (one):

while (c >= 0) and (guard < MAX_UCLASS) do

The cap is being used as a number this walk can never legitimately exceed — which is precisely the reading the ticket's worked example warns about, except here it is load bearing rather than latent: deleting the constant deletes the termination guarantee of four ancestor walks. They would need their own bound, and choosing it is a separate question from how the tables are stored.

3. 51 arrays indexed by class id, which would be the fixup family's lockstep problem times seventeen.

Status of the group after tonight

Every remaining family is now scouted with numbers, and none of them is "grow the array":

family why it is not next
TokChars (STRING_CAP) the constant also sizes the SHORTSTRING TYPE — split it first
MAX_IR 10 unrelated tables, 20 cap sites, 7 validity predicates
MAX_UCLASS a range width inside an encoded record id, plus 4 loop guards
Procs deliberately fixed

All three live ones need a constant SPLIT or an encoding change before any storage change, and that is a different kind of work from what this ticket has been doing. The mechanical conversions are done. What is left should be filed and ranked as its own piece rather than continued as the sixth, seventh and eighth item in a rhythm — the rhythm is exactly how a while guard < MAX_UCLASS gets deleted by someone who has converted five tables successfully.

MAX_TEMPLATE_TOKENS — scouted 2026-09-06, CHEAP, and half-converting it is a trap

Routed here by the coordinator because bug-a-a-generic-body-takes-its-directive-state-from-the-specialization-site wants ~10 new parallel columns on these pools, and whether that is small work depends on whether they convert.

They do. This is the MAX_FIXUPS shape, not the MAX_IR shape:

MAX_TEMPLATE_TOKENS MAX_IR (refused)
arrays 6, in two parallel trios 10, unrelated
cap sites 2 20
validity predicates 0 7
refs 10 (8 are decls/comments in defs.inc) 54
files outside defs.inc 1 many 
TemplateTokens   TemplateSrcOff   TemplateSrcLen      { cap tested at :866, :976 }
SpecializeTokens SpecSrcOff       SpecSrcLen          { NO cap site at all }

One Ensure grows all six, and the ticket's ten new columns become ten SetLengths inside it rather than ten more arrays kept in lockstep by discipline. The generic-directive ticket should therefore wait for this conversion, not be written against the fixed tables.

The trap, and it is the reason this section exists

The Specialize trio has no cap site and does not need one today, because it is bounded implicitly through the Template pool's cap. SpecializeToBuffer runs while i < count, and subCount advances at most once per iteration — checked at all three Inc(subCount) sites, including the two that look like they might outrun i (i := HoistEnd[..]+1, i := gEnd+1) and the one that looks like a rewind (i := gSelf, where SelfSpecGroupEnd returns a position at or after i). All three advance i forward, so subCount <= count <= MAX_TEMPLATE_TOKENS with no guard anywhere.

So converting the Template trio alone silently un-bounds the Specialize trio. The moment count can exceed 65536, six arrays are written at subCount with no cap site to fail and nothing in the source naming the dependency. It would not error — it would write past a fixed array and surface somewhere else.

This is "after converting a table, enumerate its WRITE sites, not its CAP sites" with the teeth showing: a cap-site census finds 2, a write-site census finds 21, and the 21 belong to a table the 2 do not mention. It is also "two tables can share a ceiling without sharing a constant" — benign right up until someone moves the constant.

Convert all six together, and add the Specialize guard as part of the same change rather than trusting the implicit bound being removed.

Parked 2026-09-06

parked by frankH 2026-09-06 — not abandoned and not in progress. This session was redirected to owner-directed work (arm B of decide-how-a-type-carries-an-identity-its-kind-cannot-hold) and finished it; nothing of this ticket was started, so the tree carries no partial work for it. It read status:working owner:frankH at origin all day, which reads as ACTIVELY HELD and was over-claiming. Free to take; message frankH for context if any is wanted.

Before resuming: read the reason above, then the ticket body. If the reason does not tell you what would make this worth picking up again, establishing that is the first step -- a park is a handoff to a stranger who may be you.

Parked 2026-09-06 (frankH) — nothing started, no partial work in the tree

Parked for bookkeeping, not because it was attempted. This session was redirected to owner-directed work (arm B of [[decide-how-a-type-carries-an-identity-its-kind-cannot-hold]]) and finished it; no line of this ticket was written. The tree is clean and everything is pushed, so there is nothing half-applied to revert — which is the one thing check's "Track A ticket in unfinished/" rule is actually about.

It read status: working, owner: frankH at origin all day. That reads as actively held, so it was over-claiming rather than under-claiming, and it would have kept anyone else off it for nothing. Free to take.

Next step is unchanged from the ticket above.