GitHub - fcmv/lucen: Parallelize Python loops by adding two comments - and it's guaranteed bit-identical to running sequentially. Same floats, same order. No rewrite, no locks, no wrong answers.

Lucen is a source-to-source compiler and automatic loop parallelizer for ordinary Python, driven by comment pragmas. Unlike existing Python parallel frameworks, it asks you to describe where parallelism is allowed rather than how to implement it, and it parallelizes only the loops it can prove are both safe and worthwhile. Its one guarantee has no tier and no opt-out: Lucen never produces an incorrect result.

Before, ordinary Python: for i in range(len(records)): scores[i] = score(records[i]) After, still ordinary Python: # LUCEN START for i in range(len(records)): scores[i] = score(records[i]) # LUCEN END 3.8x faster on 12 cores (CPython 3.14, CPU-bound map, measured). Bit-identical output, floats included. No multiprocessing code. No pools, no locks, no pickling errors to debug. No risk of adopting it: a loop Lucen cannot prove safe runs exactly as the sequential Python you wrote, and a structured report tells you why. Activation is one call at program start: import lucen lucen.activate() The pragmas are ordinary comments. A file with Lucen removed, uninstalled, deactivated, or never present runs identically to one where it never existed. We call this the Comment Invariant, and it is load-bearing: the worst case of adopting Lucen is the program you already had. It covers the other case: the loop already sitting in a codebase that nobody wants to restructure. No worker functions, no pool lifecycle, no serialization plumbing, no rewrite into a framework's shape. Two comments slide into existing code without a hiccup, and adoption is reversible by deleting them.

Contents: Guarantees | Install | Tutorial | Beyond the basics | Expert guide | How it works | Performance | Limitations | The honesty contract | Documentation | Contributing | License

The three guarantees

Never an incorrect result. Chunks write private slabs, audited for disjointness at join and committed in chunk order. Dict insertion order, float reduction bits, and mid-error container state are identical to sequential execution, bit for bit. A write conflict discards the parallel attempt and transparently re-runs your loop sequentially.

Never disruptive. Anything Lucen cannot prove safe runs as the sequential Python you wrote, and the reason lands in a structured fallback report instead of your stderr. Exceptions keep their type, their message, and the exact sequential-prefix state of your containers. Never silently pointless. A profitability gate (static pre-screen plus a runtime probe that does real work while measuring) refuses to parallelize loops that would lose to dispatch overhead, and reports that too. Parallelism you cannot observe is a bug here, not a shrug.

These are not aspirations. They are enforced by construction and verified by a cross-version test matrix: 7 interpreters x 8 workloads x 4 execution pathways, every cell bit-identical to plain Python. See BENCHMARK.md. Installation pip install lucen Python 3.9 or later. No required dependencies on 3.11+; on 3.9 and 3.10 the TOML parser dependency (tomli) installs automatically. From source (optional Rust acceleration core, needs a Rust toolchain): git clone https://github.com/fcmv/lucen cd lucen pip install -e ".[dev]" pytest On GIL builds 3.9 through 3.14, pip installs a native core (Rust, abi3, one binary per platform) that runs the orchestration hot loops (the write-set audit and the by-reference reduction folds). On free-threaded builds, where the abi3 core cannot load, pip installs a pure-Python wheel instead, so the install always succeeds; Lucen then runs its pure-Python fallback, which is fully supported and passes the identical test suite. Supported interpreters

Interpreter Status Native core

CPython 3.9 to 3.14 (GIL) Supported, tested per release yes

CPython 3.13t / 3.14t (free-threaded) Supported, tested pure-Python fallback

PyPy 3.11 Supported, tested on the fallback pure-Python fallback

GraalPy Best-effort, tested on the fallback pure-Python fallback

Tutorial: from zero to first speedup This walkthrough assumes nothing beyond basic Python.

Every step shows real commands and real output shapes. Step 1: install pip install lucen Step 2: start with a program that is too slow Save this as work.py. It scores 20,000 records with a CPU-heavy function, plain Python, no Lucen anywhere yet: import math

def score(x): acc = 0.0 for k in range(400): acc += math.sin(x * 0.001 + k) * math.cos(k * 0.5) return acc

def main(): records = list(range(20_000)) scores = [0.0] * len(records) for i in range(len(records)): scores[i] = score(records[i]) print(f"checksum: {sum(scores):.6f}")

if __name__ == "__main__": main() python work.py # takes roughly a second of pure compute Step 3: mark the loop Add two comment lines around the loop. Nothing else changes: # LUCEN START for i in range(len(records)): scores[i] = score(records[i]) # LUCEN END Run it again. Nothing happens. That is the point: pragmas are comments, and you have not activated anything. Your program is exactly as safe as before. Step 4: run it Run the file with lucen run. It rewrites the marked loops in the script you point at and then executes it, so the loop you just marked runs in parallel: lucen run work.py That is the whole story for a script you launch directly. When Lucen is instead embedded in a larger application you start yourself, activate the import hook once at startup. Activation installs the hook, so it must run before the module holding your marked loop is imported, which is why that loop lives in an imported module here: # app.py import lucen lucen.activate()

import work work.main() python app.py On a multi-core machine the loop now runs about 3 to 4 times faster, and the checksum is identical to the digit.

Not approximately identical. Identical. Two things to know, either way:

The if __name__ == "__main__": guard in work.py matters on Windows and macOS. Lucen uses process workers there, and Python re-imports the entry module inside each worker. Lucen detects a missing guard and falls back to sequential with a message telling you to add it, so the failure mode is slowness, not breakage. activate() is idempotent and safe to call once at program start.

Step 5: see what Lucen decided, without running anything lucen explain work.py work.py: 1 marked block(s) [gil interpreter assumed]

Block 1 (line 12) + Parallelized Backend: PROCESS (THREAD needs a free-threaded interpreter) (GIL interpreter assumed) Runtime-dependent (never reported statically): argument picklability, custom-callable well-formedness, pool availability -- see `lucen profile`.

explain is static and honest: facts are reported as facts, and anything knowable only at call time is never reported as a yes or no. Step 6: understand a refusal Change the loop body to depend on the previous element: # LUCEN START for i in range(1, len(scores)): scores[i] = scores[i - 1] + score(records[i]) # LUCEN END lucen explain work.py Block 1 (line 12) - Sequential Reason: cross-iteration dependency 'scores[i - 1]' (monotonic chain); ...

Your program still runs and still produces the correct answer. Lucen just refuses to parallelize what it cannot prove, and says so. This is guarantee number two working as designed. Step 7: understand "not worth it" Mark a trivial loop: # LUCEN START for i in range(len(xs)): ys[i] = xs[i] * 2 + 1 # LUCEN END At runtime Lucen probes the first chunk, measures roughly 50 nanoseconds per iteration, computes that process dispatch would cost more than it saves, and runs the whole thing sequentially at full speed.

The fallback report says: lucen fallback: PARALLEL_UNPROFITABLE (work.py:12): measured ~50 ns/iteration loses to dispatch overhead; ran SEQUENTIAL (calibrate=false overrides, spec 5.17)

If you believe the gate is wrong for your case, override it per block: # LUCEN START calibrate=false Step 8: read the fallback report programmatically import lucen lucen.activate()

import work work.main()

for record in lucen.get_fallback_report(): print(record.error, record.file, record.line, record.message) Nothing Lucen decides is hidden. Every downgrade has a reason string and a location, and lucen profile script.py shows what actually ran, per block, with timings. That is the whole naive workflow: mark, activate, read what it tells you. You never need to know what a slab or a wavefront is to get correct parallelism. Beyond the basics Tuning happens through pragma clauses. Every clause only ever trades a Lucen-held proof for a user-held assertion, or exactness for speed, never a different answer. A malformed clause is a loud import-time error with a did-you-mean suggestion, never a silent ignore. # LUCEN START calibrate=false, timeout=5.0, on_error=collect The full surface is fifteen clauses on # LUCEN START and two on # LUCEN TRUST. The complete reference, with every accepted form, is docs/pragmas.md; the summary:

Clause What it does

backend= Pin the backend: thread, process, sequential, with pool_size/chunks

calibrate= Control the profitability gate (false forces parallel)

grainsize= Level width for a recognized-DAG wavefront

affinity= CPU affinity: compact, scatter, or explicit(cores=[...])

nested= Policy for a block reached inside another parallel block

depend= Assert independence (none) or an acyclic order (expert)

skip_runtime_check= Disable the runtime write-set audit (expert, with depend=none)

trust= Waive the purity or pickle