Why and how we extended Polars with Rust expression plugins for fenic

fenic is a semantic DataFrame library. A PySpark-style API for building AI and LLM pipelines over messy, unstructured data. Its local engine is Polars. This post is about one specific problem we hit while building it, and the part of Polars that solved it: the expression plugin system. We ended up writing nine Rust plugins that extend Polars' expression engine. What follows is both why we went that route and how they're built, with the real code. tl;dr. The operations an AI pipeline needs over text (chunking, prompt templating, jq, fuzzy matching, markdown and transcript parsing, richer type casts) aren't in Polars. Doing them as Python UDFs is slow, and it breaks composition. Writing them as Polars expression plugins in Rust, via pyo3-polars, turns them into native expressions. They run in-engine over Arrow, they keep their declared types, and they compose with built-in ops in a single expression tree. If you're weighing a UDF against a plugin, this is the case for the plugin.

Why we built this What is fenic fenic is a DataFrame library for building AI and LLM pipelines, with an API modeled on PySpark. If you come from Polars, the important thing to know is that Polars is the core execution engine for fenic. Every DataFrame operation you write becomes a Polars expression, or a plan over pl.DataFrames. The semantic operators, the LLM map/extract/classify, the embeddings, the similarity joins, all sit on top of that same machinery. So in practice, what fenic can do is bounded by what we can express in Polars. That's a deliberate bet. Polars gives us a fast, columnar, Arrow-native engine with a real expression language and a lazy optimizer. We had no interest in rebuilding any of it. We wanted to add to it. What we wanted to do The workloads fenic targets are pipelines over unstructured text. Documents, chat logs, transcripts, scraped JSON, markdown. Concretely, that means row-level operations Polars has no native equivalent for:

Chunk a document into overlapping windows sized by token count, for embedding or retrieval. Render a prompt template per row. Real Jinja, with a struct of columns as the variables, ahead of an LLM call.

Query JSON columns with jq, and parse markdown into a structured AST. Fuzzy-match strings (six edit-distance metrics) for dedup and joins. Parse transcripts (SRT/WebVTT) into typed, timestamped cue records. Cast values into fenic's richer logical types (embeddings, markdown, typed structs) that Polars' physical dtypes don't model directly.

Each of these has to run over a whole column, produce a typed result, and slot into the middle of a larger DataFrame pipeline. Not sit off to the side. Where the obvious approaches fell short The reflexive way to add a custom operation to Polars is a Python UDF. map_elements for per-row work, map_batches for whole-Series work. For genuinely opaque, IO-bound work, like an LLM API call, that's still the right tool, and fenic uses map_batches for exactly that. But for the text operations above, UDFs cost too much on three axes. Speed. map_elements runs Python per row, under the GIL, with a Python-object round-trip for every value. For tokenizing or fuzzy-matching millions of rows, that's the bottleneck. Not the work itself. Composition. This is the one that actually hurt. Polars is fast because it plans and executes an expression tree as a whole. The moment one step is an opaque Python callback, the engine can't see through it. It becomes an optimization barrier, forces a materialization, and breaks the single-pass pipeline. Chain three UDFs and you've got three trips out of the engine and back. Types. A UDF's output type is something you assert loosely and hope holds. We needed operations that produce real, declared dtypes, like List<String>, Struct, fixed-size embedding arrays, so the rest of the plan can type-check against them. The alternatives were worse. Forking Polars to add native kernels means owning a fork forever. Doing the text work outside the DataFrame, preprocess then load, throws away the laziness and composition that are the whole reason to use Polars in the first place. We wanted these operations to be first-class citizens of the expression engine. Not neighbors of it. Why plugins Polars has a purpose-built answer for exactly this: expression plugins. You write a kernel in Rust, register it, and it becomes a normal pl.

Expr. To the engine, it's indistinguishable from a built-in. That checks every box the UDF route missed. It runs in-engine, in Rust, over Arrow buffers. Vectorized, parallelizable, eligible for streaming, with no Python and no per-row object round-trip. It returns a pl.Expr, so it composes. Plugins chain with each other and with native ops in one expression tree the engine plans as a whole. It declares its output dtype, so the plan stays typed through the custom step. And Rust gives us a mature ecosystem for the hot loops (jaq, minijinja, rapidfuzz, tiktoken, a markdown parser) without reimplementing any of it. The rule we settled on isn't "rewrite everything in Rust." Native Polars stays the fast path. A plugin only fills a gap Polars can't express: a tokenizer, a jq engine, a regex whose capture-group index is itself a column. Even the fuzzy matcher keeps just its six primitive kernels in Rust and composes the higher-order ratios from ordinary Polars expressions. pyo3-polars generates the FFI, the Arrow marshaling, and the keyword-argument bridge, so the cost of writing one is low. We wrote nine. What it bought us The end state is worth stating before the mechanics. Every one of fenic's text operations is now a native Polars expression. They run inside the engine over shared Arrow memory, they keep their types, and here's the part that matters most. They compose with native Polars ops in a single expression, with zero Python round-trips. Parsing a markdown document, filtering its AST with jq, indexing into the result, and casting it into a typed struct is one expression. Polars plans and executes it in a single pass, custom Rust and built-in list ops side by side. We'll come back to that exact example once the pieces are on the table. The rest of this post is how it's built, from the Python registration down to the Arrow memory, using nothing but the real code.

How it's built: an implementation walkthrough The mental model: two thin layers around a contract A Polars expression plugin is two small pieces of code with a well-defined contract between them:

Python side. Register a function so it looks like native Polars: expr.my_namespace.my_op(...).

Rust side. A function that takes &[Series], returns PolarsResult<Series>, and declares its output dtype.

Everything between them is generated for you by pyo3-polars. Moving Series across the FFI boundary via Arrow, marshaling keyword arguments, wiring the symbol lookup. You never touch the FFI by hand. Here's the entire Python surface for fenic's json.jq operator: # src/fenic/_backends/local/polars_plugins/json.py from pathlib import Path import polars as pl from polars.plugins import register_plugin_function PLUGIN_PATH = Path(__file__).parents[3] @pl.api.register_expr_namespace("json") class Json: """Namespace for JSON-related operations on Polars expressions.""" def __init__(self, expr: pl.Expr) -> None: self.expr = expr def jq(self, query: str) -> pl.Expr: return register_plugin_function( plugin_path=PLUGIN_PATH, function_name="jq_expr", args=self.expr, kwargs={"query": query}, is_elementwise=True, ) Two Polars APIs are doing the work:

@pl.api.register_expr_namespace("json") bolts a .json accessor onto every pl.Expr in the process. After import, pl.col("payload").json.jq(".name") is a legal expression anywhere Polars expressions are legal. register_plugin_function(...) returns a normal pl.Expr that, when the engine evaluates it, dlopens the compiled library at plugin_path, looks up the symbol named function_name, hands it the input Series, and reads back the result.

That's the whole idea. .json is not special-cased anywhere in Polars. It's a user-registered namespace, and fenic registers nine of them (json, jinja, markdown, chunking, tokenization, fuzz, regexp, dtypes, transcript). The plugin looks exactly like a built-in because, to the expression engine, there's no meaningful difference. (New to plugins? The Polars plugin docs and the pyo3-polars repo are the canonical references.)

Ok(values) if values.is_empty() => builder.append_null(), Ok(values) => { let strs: Vec<String> = values.iter().map(|v| v.to_string()).collect(); builder.append_values_iter(strs.iter().map(|s| s.as_str())); } // a failed query aborts the whole batch — it does NOT null the row Err(e) => return Err(PolarsError::ComputeError( format!("jq query execution failed: {e}. Query: '{}'", kwargs.query).into(), )), } } // malformed JSON is unreachable: the column is typed JsonType, so // upstream validation guarantees every non-null row is valid JSON. Err(e) => unreachable!("Invalid JSON: {s} ({e})"), } } else { builder.append_null(); } } Ok(builder.finish().into_series()) } Everything you need to understand the contract is in that snippet:

Signature. The #[polars_expr(...)] attribute macro (from pyo3_polars::derive) turns an ordinary Rust function into an exported, C-ABI symbol that Polars can dlopen and call.