Syscall-only pthread shim for libc-free C libraries

• Type: feature
• Status: DONE (2026-07-06) — libc-free pthread shim + multithreaded SQLite green on x86-64; see the DONE block below. heap contract v151, I/O serialization v146). Track B (lib/crtl files) + C (cfront gaps). See "Entry point" below.
• Owner: Track A+B+C (combined)
• Unblocks: task-sqlite-libc-free-runtime-bringup
• Found / Opened: 2026-06-28, Track B sqlite/pthread discussion

DONE (2026-07-06, commits 9955810a + b48af4cf) — libc-free pthread + multithreaded SQLite

Shipped as a thin façade over the existing PAL (one thread layer, two consumers), NOT a raw-syscall reimplementation. Surface = exactly what SQLite THREADSAFE=1 (HOMEGROWN recursive mutex) uses + create/join: mutex, mutexattr (no-op), self/equal, create/join. No cond vars / TLS keys (SQLite references neither).

• lib/rtl/palpthread.pas — flat C-callable bridge over palsync (Drepper futex mutex) + palthread (clone/futex/gettid). pthread_mutex_t == TMutex (one futex word, zeroed = free).
• lib/crtl/include/pthread.h + lib/crtl/src/pthread.c — C veneer; create/join keep a tid->handle registry (POSIX returns only a pthread_t, but PalThreadJoin needs the 32-byte handle; the handle block stays opaque to C).
• cparser.inc pulls palpthread alongside pxxcio under --threadsafe.
• parser.inc — pthread removed from CHeaderDefaultSystem so <pthread.h> resolves to the pxx shim by default (real libpthread via --system-libs=pthread).

Two landmines hit: (1) <pthread.h> was force-resolved to glibc (pthread in the default-system-lib list) — the whole shim was silently bypassed until removed from that list; the giveaway was Fatal glibc error: pthread_mutex_lock.c + a libc.so.6 DT_NEEDED on a supposedly libc-free binary. (2) THREADSAFE=1 hung because sqlite's default (non-VxWorks) path uses PTHREAD_MUTEX_RECURSIVE; our mutex is non-recursive → deadlock on the first recursive enter. Fix: build with SQLITE_HOMEGROWN_RECURSIVE_MUTEX=1 so sqlite does recursion itself (owner-tid + nRef counter over a plain mutex). A shared connection across threads additionally needs SQLITE_OPEN_FULLMUTEX (serialized); the default THREADSAFE=1 mode is MULTITHREAD (per-connection, not serialized) — standard sqlite, not a shim bug.

Acceptance met: libc-free (no DT_NEEDED), 8-thread mutex-contention counter exact, SQLite THREADSAFE=1 opens :memory: and runs schema, AND real concurrent multithreaded SQLite (8 threads / 8000 inserts on one FULLMUTEX connection + 8 per-thread connections) passes repeatably. make test-sqlite-threads + test/csqlite_thread_test.c. Gate: make test + self-host byte-identical, test-threads, test-lua-cross 24/24 all green. x86-64 + i386 (both --threadsafe-capable targets tested green); cross targets deferred (the PAL atomics/clone are x86-64/i386 today — M5).

Motivation

Some C libraries, SQLite included when built with SQLITE_THREADSAFE=1 or 2, expect a pthread surface. The current PXX path can parse/import <pthread.h>, and Pascal thread tests can call the host libpthread.so.0, but libc-free C library bring-up should not need glibc's pthread implementation.

The goal is a constrained, PXX-owned, source-level pthread subset implemented with Linux syscalls. This is not a full glibc libpthread ABI clone.

For SQLite bring-up, keep the default path on SQLITE_THREADSAFE=0 until the current schema-parse corruption is fixed. This ticket is the later multithreaded SQLite path.

Scope

Initial Linux/x86-64 shim in lib/crtl:

• provide lib/crtl/src/pthread.c and keep lib/crtl/include/pthread.h compatible with the exposed subset;
• implement pthread_mutex_* with atomics plus SYS_futex;
• implement pthread_once with atomic state plus SYS_futex;
• implement pthread_cond_* with a futex sequence counter;
• provide pthread_self / pthread_equal;
• add pthread_key_* only if SQLite or another concrete target hits TLS paths;
• defer pthread_create / pthread_join until a target needs worker threads.

If/when thread creation is added, use clone/clone3 directly with a PXX-owned stack, child-tid futex join, and a small start trampoline. Keep cancellation, robust mutexes, scheduler attributes, signal details, and full glibc TLS semantics out of the first slice.

Acceptance

• A libc-free C test unity-includes the shim and passes mutex contention, pthread_once, and producer/consumer condition-variable checks without any DT_NEEDED on libpthread or libc.
• Existing external pthread import behavior remains available when --system-libs=pthread is selected.
• SQLite built with SQLITE_THREADSAFE=1 or 2 can initialize, open :memory:, execute a trivial schema statement, and close using the shim.
• Non-threadsafe SQLite (SQLITE_THREADSAFE=0) remains the recommended bring-up mode until the existing SQLite SQL execution bug is closed.

Risks / constraints

• Real preemptive threads require the rest of the runtime to be honest. PXX has some threadsafe refcounting already, but heap safety is a separate Track A contract and may depend on memory-management mode; see [[feature-threadsafe-heap-contract]].
• Current --threadsafe support is primarily x86-64; i386 and other targets need separate work before this can be generalized.
• ESP-IDF should use the platform's FreeRTOS/pthread-like layer instead of this Linux syscall shim.

Log

• 2026-06-28 — opened from Track B sqlite discussion. Decision: possible as a constrained syscall-only pthread subset, but keep immediate SQLite work on SQLITE_THREADSAFE=0 and do not attempt a full glibc pthread clone.

Part of the multithreading epic (2026-06-30)

Umbrella: [[meta-multithreading]]. Invariant: threading is opt-in/off-by-default; single-threaded self-build stays byte-identical; no libc (Linux syscalls only).

READY TO PICK — the libc-free PAL now exists (2026-06-30, Track A landed M1/M2)

The hard parts this ticket described as future work are already built and tested on x86-64 — the C pthread shim should reuse them, not reinvent. See devdocs/dev/threading.md. Concrete mapping:

Atomics: the compiler now has __pxxatomic_xchg/cas/add intrinsics — usable from C lowering if the shim wants them instead of inline asm.

Heap-safety dependency is de-risked: the heap is validated thread-safe under --threadsafe (test_thread_heap: 0 errors with the flag, SIGSEGV without). So a threadsafe-SQLite path compiles --threadsafe and the contract holds on x86-64. [[feature-threadsafe-heap-contract]] is still the formal contract doc, but the runtime guarantee exists.

Still Track B (owns lib/crtl/**). First slice unchanged: mutex + once + self/equal, libc-free, no DT_NEEDED on libpthread. cond var + create/join follow. x86-64 first (the atomics/clone intrinsics are x86-64 today).

Entry point for Track B/C (2026-07-02, ground = pinned v153)

Prereqs ALL LANDED — do not rebuild any of this:

• PAL (M1–M3, x86-64, libc-free, green in make test-threads): lib/rtl/palthread.pas (PalThreadCreate/Join, PalFutexWait/Wake, PalThreadSelf), lib/rtl/palsync.pas (TMutex = Drepper 3-state futex, TEvent, TRTLCriticalSection names, RunOnce), lib/rtl/palthreadobj.pas (TThread + Synchronize/Queue). Compiler intrinsics: __pxxclone, __pxxatomic_xchg/cas/add, __pxxrawsyscall.
• Heap contract (v151): hosted x86-64 --threadsafe = the only supported threads+alloc combo; every allocation family validated under concurrent churn (test_thread_heap_mixed). Doc: devdocs/dev/threading.md.
• MAX_PROC_PARAMS = 32 (v153): 17..32-param C function defs/calls work.

Build order (first slice = NO pthread_create, per Scope above):

1. lib/crtl/include/pthread.h subset + lib/crtl/src/pthread.c — mutex/once/cond/self/equal only. Two impl options: a. PREFERRED: bridge to the Pascal PAL via the pxxcio pattern (C extern binds a bodied Pascal proc, auto-pull — see devdocs/dev/c-linking-and-crtl-autopull.md and how <stdio.h> pulls pxxcio.pas). One PAL, two consumers (meta-multithreading rule). b. Raw __pxxatomic_* + __pxxrawsyscall(SYS_futex,...) directly in C.
2. Acceptance test: libc-free C test with mutex contention + pthread_once + cond producer/consumer, zero DT_NEEDED. No thread spawn in slice 1 → plain compile, no --threadsafe needed (contention exercised from the Pascal-thread side or single-thread semantics checks).
3. LATER slice (pthread_create/join): route through PalThreadCreate. Note the v151 gate: __pxxclone is a COMPILE ERROR without --threadsafe — intended; C tests that spawn must compile --threadsafe.

Lane rules: crtl impl files = Track B; cfront parse/lowering gaps = Track C (two known pre-filed: [[bug-c-cast-as-call-arg-parse-error]], [[bug-c-printf-without-stdio-include-varargs]]); anything in shared compiler internals (lexer/parser/ir/backends) = file a Track A ticket.