Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,019 words · 3 segments analyzed
Polars is a library for transforming, analyzing, and visualizing data with a fast and expressive DataFrame API. It was first released by Ritchie Vink in 2020. Install Polars with all of its optional dependencies from the terminal: uv pip install "polars[all]" Import Polars in Python, and confirm which versions of Polars and its dependencies you have installed: import polars as pl pl.show_versions() Polars queries typically read data, transform it, and write the result back out. A complete query is often a single chain of method calls: fruit = pl.read_csv("fruit.csv") fruit.filter( (pl.col("weight") > 1000) & pl.col("is_round") ).write_parquet("fruit.parquet") Throughout this cheatsheet, df is a DataFrame, lf is a LazyFrame, o is a second DataFrame to combine with df, and e stands for any expression. So e.abs() means “call .abs() on an expression”, as in pl.col("x").abs(). Data Structures# Polars stores all of its data in either a Series or a DataFrame. Structure Description Series One-dimensional. Holds a sequence of values of the same data type. DataFrame Two-dimensional. Has rows and columns. One or more Series, all of the same length. LazyFrame Resembles a DataFrame but holds no data. A blueprint for generating a DataFrame. Unlike pandas, Polars DataFrames do not have a row index, and the API favors immutability and method chaining over in-place modifications. Create a Series by passing a name and a sequence of values: series = pl.Series("sales", [150.00, 300.00, 250.00]) Create a DataFrame from a dictionary of columns, where each value is a Series or a plain Python sequence. You can also use any of the pl.read_*() functions to create one from a file: df = pl.DataFrame({ "sales": series, "id": [41, 42, 43] }) Because there is no row index, add one explicitly as a column when you need it: df.with_row_index("id") Turn a DataFrame into a LazyFrame. Alternatively, start from a LazyFrame directly with any of the pl.scan_*() functions: lf = df.lazy() Eager and Lazy APIs# The eager API executes immediately, whereas the lazy API builds an optimized query plan first. The optimizer automatically applies predicate pushdown (filtering as early as possible) and projection pushdown (dropping columns that are never used). You move between the two representations with .lazy() and .collect(): .lazy() turns a DataFrame into a LazyFrame, and .collect() executes a LazyFrame and gives you a DataFrame back. Turn a DataFrame into a LazyFrame, and execute a LazyFrame to get a DataFrame: lf = df.lazy() df = lf.collect() Use the streaming engine to process data out-of-core, so that datasets larger than memory can still be handled: lf.collect(engine="streaming") Print the optimized query plan as text, or visualize it as a graph, to see what the optimizer decided to do: lf.explain() lf.show_graph() Execute the query and return per-node timings, which tells you where the time actually goes: lf.profile() Data Types# Polars implements most of the Apache Arrow memory specification, which is an efficient columnar format for flat and hierarchical data. Group Type Notes Numeric Decimal 128 bits, precision, scale Float32 Ranges ±3.4×10³⁸ Float64 Ranges ±1.8×10³⁰⁸ Int8 Ranges ±128 Int16 Ranges ±32,768 Int32 Ranges ±2.1×10⁹ Int64 Ranges ±9.2×10¹⁸ Int128 Ranges ±3.4×10³⁸ UInt8 Ranges 0–255 UInt16 Ranges 0–65,535 UInt32 Ranges 0–4.3×10⁹ UInt64 Ranges 0–1.8×10¹⁹ Temporal Date Days since Unix epoch Datetime Microseconds since epoch Duration Time duration / delta Time Time of day Nested Array Fixed-length sequence List Variable-length sequence Struct Multiple fields with names String String UTF-8 text, variable length Categorical Dict of Strings Enum Fixed dict of Strings Other Boolean True / False Binary Raw bytes Null Represents Null / None Inspecting Types# Get a dictionary of column names and data types, or just the list of data types: df.schema df.dtypes Print one row per column, including data types, which is useful for wide DataFrames where printing the DataFrame itself is unreadable: df.glimpse() Compute per-column summary statistics, including the number of nulls: df.describe() Report the in-memory size of the DataFrame in the unit you ask for: df.estimated_size("mb") Casting# Cast a column to another data type.
By default the cast is strict, so a value that does not fit raises an error: df.select(pl.col("id").cast(pl.UInt64)) Pass strict=False to cast without raising.
Values that overflow the target type become nulls instead: df.select(pl.col("id").cast(pl.Int8, strict=False)) Reading and Writing Data# Polars has four families of input and output functions, and which one you want depends on whether you are working eagerly or lazily: read_*() reads data into a DataFrame. scan_*() creates a LazyFrame, deferring the actual reading until you collect. write_*() writes a DataFrame to disk or to cloud storage. sink_*() streams data to disk or to cloud storage without holding it all in memory. Not every format supports all four operations: Format read scan write sink Avro ✓ ✓ Clipboard ✓ ✓ CSV ✓ ✓ ✓ ✓ Database ✓ ✓ Delta Lake ✓ ✓ ✓ ✓ Excel / ODS ✓ ✓ Iceberg ✓ ✓ ✓ IPC / Feather ✓ ✓ ✓ ✓ JSON ✓ ✓ NDJSON ✓ ✓ ✓ ✓ Parquet ✓ ✓ ✓ ✓ PyArrow Dataset ✓ Keyword arguments that many of these functions accept include schema_overrides, n_rows, row_index_name, storage_options, and compression. Scan files in cloud storage by passing a URI with a glob pattern, and use storage_options to supply credentials and region settings: pl.scan_parquet( "s3://bucket/*.parquet", storage_options={"aws_region": "us-east-2"} ) Stream a query straight to a partitioned Parquet dataset, writing one directory per distinct value of the key column: lf.sink_parquet(pl.PartitionBy("out/", key="x")) Transforming Data# Selecting Columns# Keep columns based on their name, data type, or position. Select columns by name: df.select("a", "b") Select the result of an expression, so that you can transform columns on their way out: df.select(pl.col("x") * 2) Give the result of an expression a name by using a keyword argument, which produces a new column: df.select(doubled=pl.col("x") * 2) Select columns whose names match a regular expression. The pattern must start with ^ and end with $: df.select(pl.col("^.*_color$")) Select every column: df.select(pl.all()) Use column selectors for more flexibility. They can be combined using the set operators |, &, -, ^, and ~. Import the selectors module, then select columns by data type or by name pattern. See also cs.string(), cs.contains(), and cs.first(): import polars.selectors as cs df.select(cs.numeric()) df.select(cs.starts_with("val")) Drop columns instead of keeping them.