__file__ for a module inside a package

What was measured, 2026-09-12

A two-file package, binary written beside the package root — the most favourable placement for the current convention:

# mypkg/inner.py
def where():
    return (__file__,
            os.path.dirname(os.path.abspath(__file__)),
            os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

Read the whole table, not the last column. With the package collapsed, pxx disagrees with CPython's shape on both idioms for a packaged module: one dirname should name the package directory and names its parent; two dirnames should name the root and name one above it. The current convention is exactly right for a TOP-LEVEL module (uforth, which is what it was measured on in August) and is off by one level for every module inside a package.

What it costs in the program that found it

lekkerzeilen/world.py:598:

def _home(root=None):
    return root or os.path.join(os.path.dirname(os.path.dirname(
        os.path.abspath(__file__))), "world")

Four sites in the runtime package use that shape (world.py ×3, gauges.py ×1) and zero use one dirname — the single one-dirname site in the tree is under tests/, which the closure does not compile. So the population in the one real packaged program we have is 4 of 4 against the current rule.

The failure is silent and it is the house shape: built() returns [], --starts prints open water: nowhere in particular, which is the point, exit code 0. Nothing raises. It is correct about open water.

The options

1. Keep the package directory in the virtual path — __file__ = <exe_dir>/<pkg>/<mod>.py. Both idioms then match CPython's shape with <root> := the executable's directory, which is the same substitution the August decision already makes, applied one level deeper. Cost: the August decision's payoff sentence — "dirname(abspath(__file__)) yields the executable's directory for EVERY module" — stops being true for a packaged module; it yields <exe_dir>/<pkg>. That sentence and the four lekkerzeilen sites cannot both be satisfied. Cost 2, and it is the real one: the resolved source path must NOT be emitted (defs.inc's own comment on CompiledUnitFile records that interning a unit path baked ExeDir into every binary and made our output argv0-dependent — bug-a-the-compilers-output-depends-on-argv0), so only a RELATIVE fragment may be emitted, and computing it needs a notion of the package root. __init__.py-walking from the main module's directory is the obvious rule and is not yet written.
2. Leave it, and change lekkerzeilen. The owner has licensed cheating on this corpus. One helper in world.py and one in gauges.py. Cost: it is idiomatic Python that works under CPython, so this is the "fix the program" side of the umbrella's own fork, and the next packaged program pays it again.
3. The application data root the August decision already deferred — a compile-time --data-root=<path> and/or a run-time override, default the executable's directory. Its own words: "Not built now (user): wait for the first program that needs it." This is that program. Cost: it does not fix the __file__ shape, so a program that computes paths from __file__ rather than asking for a data root still gets the off-by-one.

Options 1 and 3 are not exclusive.

Recommendation

Option 1. It makes the existing rule internally consistent rather than replacing it: the freezer frame is "paths hang off the executable's directory", and keeping the package directory is what makes that frame produce CPython's shape for a module that lives inside one. A frozen PyInstaller app does preserve package structure under its own root, so the convention we chose to imitate already does this. Option 3 is worth having afterwards and for its own reasons.

Why this is the owner's call and not mine

The August ticket is his decision and it states the rule in words this would change. The fork is not technical — both options are a few dozen lines — it is about which layout a compiled program's data is expected to follow. Stated with no implementation noun in it: do we want a compiled program to find its data the way the source tree is laid out, or the way the binary's own directory is laid out?

DECIDED 2026-09-22 (frankh-c0): option 1, and NOT escalated. Here is why.

I am taking this rather than sending it up, and the reason is the one CLAUDE.md predicts: "writing it usually reveals that an existing rule already decides it." This ticket did the hard part — it states the fork as a sentence about what we want, with no implementation noun in it — and stating it that well is what shows the fork is not a fork.

THE STATED FORK HAS AN ARM THAT IS NOT AN INTENT. The sentence is "do we want a compiled program to find its data the way the SOURCE tree is laid out, or the way the BINARY'S directory is laid out?" Option 1 does not choose the first arm. It keeps the binary's-directory frame exactly — <exe_dir> substitutes for <root> — and applies it at the level CPython uses. Under option 1 a packaged module's __file__ is <exe_dir>/<pkg>/<mod>.py, which is the August rule unchanged, read one level deeper. The two arms are not two intentions; one is the intention and the other is where a sentence written without packages in mind happens to land.

So what option 1 costs is not an intent. It is the August ticket's payoff sentence — "dirname(abspath(__file__)) yields the executable's directory for EVERY module" — which was true of every module that existed when it was written, because none of them were in a package. That is a claim with a date on it, not a decision.

AND THREE STANDING RULES DECIDE IT, none of which needed the owner:

1. NilPy is UPWARD compatible with CPython, one direction. Accepting what CPython rejects is a feature; code that runs correctly under CPython and wrongly under us is a bug. Four sites in one real program, silently wrong, is the bug side of that line and not a compat item.
2. "Real code compiling or running wrong is a bug." --starts prints open water: nowhere in particular and exits 0. Nothing raises.
3. The goal list names lekkerzeilen as a demo. Option 2 makes the demo depend on a patched copy of the program, which is the "fix the program" arm the umbrella's own fork exists to refuse.

WHAT I AM NOT DOING: I am not editing or overruling decide-nilpy-dunder-file-for-a-compiled-program. That is the owner's decision and it stands. This extends it to a case it does not mention — its own text never says "package" — and I will say so in its Log rather than rewriting its rule.

REVERSIBILITY, which is the actual test. This is a few dozen lines in two lowering sites plus a fixture. If the owner reads it and wants the collapsed path back, it is one revert. A big reversible change is mine; a small irreversible one would not be. Reported rather than asked.

WHAT WOULD MAKE ME WRONG: a real program that relies on dirname(abspath(__file__)) naming the executable's directory from inside a package. The population we have is 4 of 4 the other way, and the single one-dirname site in the tree is under tests/, which the closure does not compile. If someone finds such a program, that is evidence and this should be re-opened on it — not on the argument that the August sentence said so.

Option 3 (the data root) is not closed by this and remains worth having on its own terms; options 1 and 3 are not exclusive, as the ticket says.

MEASURED 2026-09-22: it is not an off-by-one, it is IMPORT-ORDER DEPENDENT

The ticket's table records one value for pxx, <root>/inner.py. There are two, and which one you get depends on the order of two import statements elsewhere in the program.

Two-file package, one module, two mains differing only in the order of their two imports:

# a_dotted_first.py            # b_rel_first.py
import mypkg.inner             import mypkg.sib      # sib does `from . import inner`
import mypkg.sib               import mypkg.inner
print(mypkg.inner.where())     print(mypkg.inner.where())

The same module in the same program reports two different __file__ values, and neither matches CPython. The mypkg_inner.py spelling is the worse of the two: it is a filename no file ever had, with an underscore where CPython has a separator, so basename is wrong as well as dirname.

Mechanism. The unit name is mangled dots-to-underscores at pyparser.inc:47335 (PyConsumeDottedModule) and again at pyparser.inc:47783 (PyPackageSubmoduleKey), so a dotted import registers the unit as mypkg_inner; a relative from . import inner passes the bare inner. Whichever spelling compiles the file FIRST wins, because the second one hits the resolved-path dedupe at pasparser_proc.inc:6886-6925 and becomes an alias without re-parsing — so __file__ is lowered exactly once, from whichever name got there first.

This is the first-wins hazard CLAUDE.md names, and it is a textbook instance: "a first-wins table is exposed only by the arrangement that puts the correct entry last." The ticket's original measurement took the relative arm, which produces the tidy-looking inner.py and reads as a clean off-by-one. The dotted arm produces a mangled name and does not read as an off-by-one at all. Both were reachable from the start; only one was measured.

What it changes. Nothing about the decision — option 1 fixes both arms, because both should produce mypkg/inner.py. What it changes is the ARGUMENT: this is no longer "our convention differs from CPython's by one directory level", which is the kind of thing a project may legitimately choose. It is "the same source produces different __file__ values depending on import order", which is not a convention at all. The fix must therefore be at the point where the fragment is COMPUTED, not a post-hoc adjustment of the mangled name — unmangling mypkg_inner is ambiguous anyway, since a module may legitimately contain an underscore.

Population and oracle, recorded so a re-run is comparable: one two-file package plus one sibling module, mypkg/{__init__,inner,sib}.py, built with compiler/pascal26 at 5f986d67044a, oracle CPython 3 on the same tree, executable and package both under one mktemp -d.

Log

• 2026-09-22 — decided; this names the commit that carried the decision, which is not always the one that carried the change — commit 4ecca09ec.