← board

A Unicode identifier is rejected by the lexer

Repro

_κ = 5
print(_κ + 1)



pxx:     pascal26:1: error: unexpected character: <?>
CPython: 6

Boundary

Non-ASCII in a string literal already works:

k = "κ appears in a string"
print(len(k) > 0)       # True in both

So the UTF-8 source path is fine and this is the identifier path only.

Where it bites

tinycss2/color4.py and color5.py name their CIE Lab constants with the conventional Greek letters:

_κ = 24389 / 27
f0 = x ** (1 / 3) if x > _ε else (_κ * x + 16) / 116

That is idiomatic for the domain — the spec itself writes κ and ε — so "just rename them" is not open to us: this is unmodified third-party source, and the whole point of the corpus ladder is compiling it as shipped.

Why the priority is low anyway

Two files, and both sit behind undefined variable (CodecInfo) in the same package, so nothing is unblocked by fixing this alone. It is filed because it is a genuine upward-compatibility break — Python 3 has allowed Unicode identifiers since PEP 3131 and this is working CPython code — and because it is cheap: the lexer already accepts these bytes inside a literal.

Gate

The repro prints 6. The unexpected character row leaves the ladder table.

Resolution (2026-08-27)

Fixed in compiler/pylexer.inc. Witness test/test_nilpy_a_unicode_identifier.npy (+ test/nilpy_uniids.py), registered in test-core as test_nilpy_uniid26, .expected generated by CPython. Red at pinned v382 (b12e5e27a) on line 13, green at HEAD.

The fix

Source is a byte string, so a UTF-8 identifier arrives as a run of bytes

= 128. Both identifier predicates now accept any such byte, start and continue alike, and the run is simply taken whole. Names are compared as bytes everywhere downstream, so two names match exactly when they are the same bytes — a promise a lexer that never decodes can actually keep, and enough for every program CPython accepts. The witness pins it from the string side too (getattr(g, "ω") finds the attribute, hasattr(g, "w") does not).

One predicate, not one patched arm

The byte set was written out six times in four different spellings across this file: the identifier scan, its continuation, the imaginary-suffix guard (1jj is not an imaginary), the two "not the tail of a longer identifier" guards in the f-string/bytes-prefix rewriter, and the trailing-dot-float lookahead. Teaching only the scan about UTF-8 would have left the other five disagreeing with it, which is the second path that stays broken (devdocs/dev/normalise-dont-special-case.md). All six now call PyIsIdentStart / PyIsIdentCont.

Stated honestly: the non-ASCII half of those other five is unreachable from valid Python — a name immediately followed by a quote, or by j, or by ., is a syntax error in CPython too — so their change buys consistency, not behaviour, and no .expected row can witness it. What the last four rows of the test pin is that their ASCII behaviour is unchanged.

Measured boundary

Every position works and matches CPython: bare name, def, class, method, attribute, keyword-argument name, f-string interpolation, dict key, a name with no ASCII prefix at all (κ, not just _κ), and across a module boundary both qualified and from-imported. A Unicode class name reaches codegen and __class__.__name__ prints it back. An ASCII lookalike (a vs å) stays a distinct name.

The filing file compiles past the row that filed it: tinycss2/color4.py was pascal26:431: error: unexpected character at v382 and is now pascal26:8: ... no unit named webencodings — the dependency wall the ticket predicted it sat behind. color5.py never had the row.

The diagnostic cost, and why it is accepted

Any byte >= 128 outside a literal is now an identifier byte, so a pasted smart quote moved from unexpected character: <?> to undefined variable (“hello”). That is laxer than CPython, which names invalid character '“' (U+201C). It is not a defect under Track N's upward-compatibility rule — no program CPython accepts and runs can observe it — and buying the better message means carrying a Unicode category table in the lexer purely to improve an error on source that is already invalid, which CLAUDE.md's compat table defers. The new message does at least show the offending characters, where the old one printed one unprintable byte. Recorded in devdocs/dev/nilpy-semantics-divergences.md.

Not done

The C frontend's lexer has its own copy of the same question (C23 allows UTF-8 identifiers). Deliberately untouched: parsers and lexers are duplicated per language on purpose (devdocs/dev/the-substrate-is-ast-and-ir-not-the-parser.md), and no C corpus file is waiting on it.

Log