← board

NilPy str counts characters, not bytes (phase 2)

The defect this closes

Measured at 8f1852f27, s = "héllo", t = "日本語":

expression CPython pxx
len(s) 5 6
s[1] é \xc3 — invalid UTF-8 on stdout
s.find("l") 2 3
s[::-1] olléh byte-reversed, invalid UTF-8
len(t) / t[0] 3 / 日 9 / \xe6
ord("€") 8364 TypeError: … string of length 3 found

The two rows that emit malformed UTF-8 are the urgent part; they are worse than any end state and worth fixing first.

Why this is a defect and tuple-mutability is not

devdocs/dev/nilpy-semantics-divergences.md accepts a mutable tuple because no working CPython program can observe it. Ordinary working code that slices or measures non-ASCII text observes every row above. Different side of that line.

What must NOT change

Pascal is already correct and must stay "wrong." FPC counts code units — Length on an AnsiString is bytes, and on a UnicodeString it is UTF-16 code units, so a non-BMP character counts 2 and s[1] can be half a surrogate pair. Being more correct than FPC would be a compat bug. This is why the substrate could not simply be changed: the two frontends have genuinely different correct answers, which is what the kind word exists to express.

Layout constraint you must obey

Every meaningful field lives in the low 32 bits of the meta word: BlockKind(8) | Flags(8) | KindData0(8) | KindData1(8), bits 32–63 reserved. Spending the upper half would permanently foreclose [[feature-a-shrink-managed-header-on-32-bit]], because a packed ILP32 header makes the meta word 32 bits wide. Consequently KindData0 holds a small encoding enum (0 = bytes, 1 = UTF-8, 2 = UCS-2, 3 = UCS-4), not a raw codepage — CP_UTF8 = 65001 does not fit in 8 bits, and the enum is the better field regardless.

Also rename the phase-1 offset constant PXX_HDR_KIND → PXX_HDR_META in this ticket's first commit. Nothing reads it yet, and this ticket re-pins anyway.

The work

  1. Stamp kinds at every materialisation site — the literal→managed conversion, concat, SetLength, and the pylib constructors. The static type is known at each: Pascal context → ByteString, NilPy context → TextString.
  2. Propagate through pylib. The ~79 pystr_* functions construct new blocks; every Result := … must carry the kind forward or the result silently degrades to the default. This is the bulk of the work and it is the part that is easy to under-estimate.
  3. Character-aware public positions for TextString: len, indexing, slicing, find/index/rfind, reverse, charlist, padding widths, ord/chr. Internal offsets stay bytes.
  4. Set the ASCII flag at construction when no byte is ≥ 0x80. Then len and indexing stay O(1) and byte-identical to today for the overwhelmingly common string; only genuinely non-ASCII text pays.

Keep the coordinate system coherent

Byte and character answers agree for in, count, split, ==, +, and — measured — offsets round-trip today: s[s.find("w"):] is correct across multi-byte characters, because UTF-8 is self-synchronising. That is why find must move to character offsets in the same change as indexing. Moving one and not the other breaks programs that currently work.

Static context wins

Kinds live on shared refcounted blocks, so a kind cannot be flipped at a boundary without copying. Where a static type exists it decides; the kind answers only where the static type is lost (variant, container element, generic/untyped parameter). A TextString reaching Pascal code is read as bytes, no copy; a ByteString reaching NilPy from a variant is treated as UTF-8 text.

Gate

Per-fix loop per item. .npy tests diffed against CPython with tools/pydiff.py covering every row of the table above, plus: an all-ASCII string (must stay byte-identical and O(1)), a find→slice round-trip across a multi-byte character, a Pascal AnsiString round-tripping through a variant into NilPy, and in/count/split/==/+ (which must not move). Watch for the O(n²) shape — while i < len(s): s[i] — on a non-ASCII string.

PARKED in unfinished/ — the foundation is LANDED and green, the semantics are not

Not blocked and not half-applied: the compiler change is complete, gated and pinned (v248), so nothing here is in a broken intermediate state. What remains is the semantic conversion, which is scoped below and must be done as one commit. Re-claim it; do not re-derive the survey.

2026-08-07 — the FOUNDATION landed; the semantic conversion did not

Split deliberately. What is in is complete and useful on its own; what is out would have been half a coordinate system, which is worse than none.

Landed

Verified: lit → ASCII, a UTF-8 é string → not-ascii, ascii + ascii → ASCII. Self-host fixedpoint in one round via the fast path — no layout changed, so make compiler/pascal26 is correct here, which is the narrow rule from devdocs/dev/fpc-optional-workflow.md working as documented.

The obstacle the next session needs to know about

pystr_at returns a Char. A NilPy s[i] lowers to it (pyparser.inc ~5180) and the result is typed tyChar, promoted to a str via pystr_ofchar where a string is needed. A Char is one byte, so it cannot carry a multi-byte character. Character indexing therefore needs a new string-returning entry point (pystr_at_s) and the lowering re-typed from tyChar to tyAnsiString — which ripples into NilPy's type inference. That is the structural work, not the UTF-8 arithmetic.

How CPython solves the "char" problem — and why that removes the objection

Python has no char type. s[i] returns a str of length 1, always. Measured against CPython 3.12:

result
type(s[0]).__name__, len(s[0]) str, 1
t[1] is t[1] for é (U+00E9) True — cached
u[0] is u[0] for 日 (U+65E5) False — freshly allocated
sys.getsizeof("a") 42 bytes

So CPython keeps a cache of the 256 latin-1 single-character strings and allocates only above U+00FF. Given a 42-byte str object, that cache is what makes for c in s and s[i] affordable at all.

This is the answer to the pystr_at obstacle above. The objection to returning a string instead of a Char is allocation cost per subscript — and CPython shows the standard fix: intern the single-character strings. For pxx that means the ~128 ASCII ones (our substrate is UTF-8, so latin-1 above $7F is already two bytes and less worth caching), which is exactly the population the PXX_FLAG_ASCII fast path is already about. On that path s[i] becomes a pointer to a shared block — cheaper than today's Char → pystr_ofchar promotion, which allocates.

Pleasingly, both reserved flags from the phase-2 foundation find their purpose here: PXX_FLAG_INTERNED marks a cached singleton, and PXX_FLAG_STATIC is what stops its refcount ever reaching zero. Neither was invented for this; they were reserved on general principle and the use arrived.

…but do NOT copy CPython here — keep the char TYPED (user, 2026-08-07)

CPython interns single-character strings because it has no static types: everything is an object, so s[i] must be one. We are a compiler and should not adopt a workaround for a constraint we do not have.

Any Unicode code point fits in 32 bits, so a scalar holds any character. The name already exists — FPC's system unit declares UCS4Char = type LongWord (with UCS4String), which is the same thing as Go's rune, Rust's char, C11's char32_t and CPython's own Py_UCS4. Use UCS4Char: it is the FPC-faithful spelling and costs nothing to adopt.

The ladder is not three sizes of one idea, and the middle rung is the trap:

type width holds
AnsiChar (tyChar today) 1 a UTF-8 code unit
WideChar (a value cast today, not a type) 2 a UTF-16 code unit — may be HALF a character
UCS4Char 4 a code point — always whole

Only the last is guaranteed to hold any character; WideChar needs a surrogate pair above the BMP, which is the same defect as Length(UnicodeString) counting code units.

So the shape of the fix changes, and gets cheaper. Widen the subscript result from tyChar to a code-point scalar rather than converting it to a string:

The remaining work is to make that promotion complete. NilPy requires s[i] to behave as a str everywhere — s[i] + "x", len(s[i]), type(s[i]).__name__, s[i] in d — and today pystr_ofchar is applied at specific sites (comparisons, pyparser.inc ~1971). That is contextual promotion in the frontend, where it already lives, not a type-system change.

The interned-1-char-string route is recorded above as the fallback if the promotion turns out not to be completable; it is no longer the recommendation.

Keep tyChar too — but know exactly when it is provable

tyChar does not go away, and not only for NilPy's sake: Pascal's Char is one byte unconditionally (FPC's AnsiChar), so the kind is load-bearing regardless. The question is only what NilPy's s[i] types as.

Use tyChar where the compiler can prove the source is single-byte, and tyUCS4Char otherwise. Be precise about which is which, because the obvious mistake is to reach for the ASCII flag:

source of s provable statically?
an all-ASCII string literal, or a concat of them yes → tyChar
anything from a variable, a call, input, a container no → tyUCS4Char

PXX_FLAG_ASCII is a RUNTIME fact and cannot drive a static type. It is still worth everything it costs — inside the tyUCS4Char subscript it gives an O(1) index and a one-byte load — but it is a runtime fast path, not a typing input. Do not write "we use tyChar when the string is ASCII"; that is only true for literals.

Adding the kind is small, and there is a precedent for its shape

Append tyUCS4Char at the tail of TTypeKind — ordinals 0–6 are frozen since first bootstrap and everything after is append-only, so a new kind at the end is the cheap, safe move. Storage is 4 bytes; TypeSize/TypeIsOrdinal carry it with no special-casing.

The precedent for making it a distinct kind rather than reusing tyUInt32 is _Bool, three lines above in the same enum: "the reason it is its own kind is CONVERSION, not layout". Same argument here — a UCS4Char converts to a string as its UTF-8 encoding, a UInt32 converts as decimal digits, and nothing downstream could tell them apart if they shared a kind. Only the conversion sites need to ask for it by name.

WideChar stays contained — do NOT complete it

It is a historical accident: a 16-bit code unit from the era when Unicode was 16 bits, so it cannot hold a character above the BMP without a surrogate pair. pxx already has it in the only defensible form — a boundary cast (__pxxWideCharToUTF8, with a surrogate-aware pair form) that exists so FPC-shaped source spelling WideChar(u) compiles. That containment is correct.

Do not promote it to a first-class type, a string element, or a subscript result. The ladder that matters is AnsiChar (byte) → UCS4Char (code point); WideChar is not a rung on it.

Why it must be all-or-nothing

Converting len() alone makes things worse: while i < len(s): s[i] would then mix a character count with a byte index, breaking code that works today. The byte model is at least internally consistent (measured: s[s.find("w"):] is correct across multi-byte characters, because UTF-8 is self-synchronising). So one commit must move the whole public coordinate system:

pystr_len, pystr_at(→ new str form), pystr_slice, pystr_slice_step, pystr_reverse, pystr_charlist, pystr_find/_from/_range, pystr_index, pystr_rfind, and the padding widths (ljust/rjust/center/zfill).

Not affected, because byte and character answers coincide — and this was measured against CPython with non-ASCII input, not assumed: in, count, split, partition, join, replace, ==, +, startswith/endswith without offsets all already agree on "héllo wörld". That is what bounds the conversion to the position-exposing functions above; do not re-verify it, and do not widen the list without a measurement.

The property that makes it safe when it happens

Gate every converted function on PXX_FLAG_ASCII: when set, byte position == character position and the function takes exactly today's code path. Every existing test uses ASCII, so the entire suite stays on the unchanged path and the new behaviour appears only where the old behaviour was wrong. That is the whole regression argument — build it that way from the start.

Access route, verified

pylib does not uses builtinheap today. Adding it was tried and works (a NilPy program built and ran with it). The exploratory edit was reverted to keep this commit purposeful, but the route is known-good — do not re-litigate it, and do not duplicate the header offsets into pylib instead.

2026-08-07 — the --no-unicode mode (user proposal, accepted with a sequence)

NilPy ends up with two string types, and the choice is partly per-target. A programmer — or a platform — may opt out of Unicode entirely and get plain byte text.

This costs almost nothing to build, because the foundation already fits it. The mode is not a second implementation: it changes which kind NilPy stamps on its literals, PXX_KIND_TEXTSTR → PXX_KIND_BYTESTR. Everything downstream dispatches on kind anyway, so the opt-out is one constant at the stamping sites.

The default must stay Unicode-on

Upward compatibility decides it (see this ticket's top): if code runs on CPython it must run on NilPy. A byte default makes ordinary CPython programs silently wrong, which is the defect being fixed. So Unicode is the default and --no-unicode is the opt-out — never the reverse.

ESP flips the default, and there is precedent. CLAUDE.md already justifies "ESP is not a Unix" — 33 PAL entry points refused even under IDF, so POSIX-shaped code meets a clear refusal rather than a wrong answer. "ESP is not Unicode" is the same move: a platform with an explicitly narrower contract rather than a silently different one. Serial I/O is bytes, and a 64 KiB static arena should not carry decode tables.

Contain the two-dialect risk

A switch that changes len() makes two languages, and code built the other way misbehaves quietly. Two cheap containments, both worth building WITH the flag and not after:

  1. Whole-program, not per-unit. No mixing inside one binary.
  2. Warn on a non-ASCII string LITERAL under --no-unicode. In that mode such a literal is almost certainly a mistake, so the mode polices itself at compile time instead of surprising someone at runtime.

Sequence — this is the part that matters

  1. The correct path first: tyUCS4Char + the runtime PXX_FLAG_ASCII fast path, so CPython-correct behaviour is what you get by default.
  2. Then the --no-unicode mode and the ESP default.
  3. Then the static-literal optimisation below.

Building (2) before (1) ships a mode that is fast and wrong by default, and a wrong default is far harder to withdraw than to never set.

The hardcoded-literal optimisation, with its caveat

An all-ASCII literal is statically provable, so s[i] on one is tyChar with no runtime check — this composes with the provability table above and is a pure win with no semantic change.

The user's "if they never mutate" caveat is load-bearing: the property belongs to the VALUE, not the variable. "hello"[i] is provable; s = "hello"; …; s[i] requires knowing s was not reassigned. Keep it conservative — literal-derived only, killed by any assignment that is not itself provable — or it quietly becomes dataflow analysis wearing the costume of a quick win.

2026-08-07 (later) — tyUCS4Char LANDED

Step 1 of the sequence above. The kind exists, is declarable, and converts.

FPC parity, verified against FPC 3.2.2 directly

FPC pxx
UCS4Char accepted as a type yes yes
SizeOf 4 4
Ord(c) the code point the code point
holds a value past the BMP yes yes

So the type surface is genuine Pascal compliance, which is the framing that justified doing it: pxx lacked a type FPC has.

…and one deliberate divergence, stated rather than blurred

FPC REJECTS 'h' + c — its UCS4Char = type LongWord is an integer type, so string-plus-integer does not compile. pxx converts it to the code point's UTF-8 encoding instead. That is a pxx extension, not parity, and it is the whole reason the kind is distinct rather than an alias for tyUInt32 — same storage, different conversion, exactly the argument defs.inc already makes for C99 _Bool. Consistent with the dialect stance: lax by default, FPC-strictness behind --strict-* flags if parity is ever wanted here.

What it took

Two traps worth keeping

  1. A one-character literal is tyChar. So 'h' + c had two ordinals and took the ARITHMETIC path — Chr(104+233), one wrong byte, silently. Both operands must be wrapped: the code point encodes, the byte only widens.
  2. Wrapping the operands is not enough — re-type the concat NODE. ('h' + c) + 'llo' left the inner node still claiming tyChar, so the outer concat read a 2-byte result as one byte and the whole expression came back 4 bytes instead of 6. Both are pinned in test/test_ucs4char.pas.

The trap that actually cost a gate run

The concat re-typing (trap 2 above) was first written as "if this is a tkPlus and either side is a string, the result is a string" — with no guard that a tyUCS4Char was involved at all. It therefore fired for every string concat in every frontend, and turned LOLCODE's SMOOSH into an infinite loop: the program printed its first four lines forever until it blew the stack.

Two things worth carrying from that:

One latent bug fixed on the way

IRVerify bounded valid kinds with a hardcoded Ord(tyBool8) — the last kind at the time — so every future kind appended to the enum would fail IR verification until someone found that line. Now Ord(High(TTypeKind)).

Still to do (unchanged from the plan)

Widen NilPy's s[i] from tyChar to tyUCS4Char and complete the pystr_ofchar promotion, then the character-aware len/slice/find as one commit, then --no-unicode, then the literal optimisation.