Polars in Aggregate: Distributed On-Prem, a Spark Benchmark, and a Smarter Optimizer

Since the last edition of Polars in Aggregate in April:

• 13 releases have been published.
• 435 PRs were merged to open source Polars.
• 57 contributors submitted code to open source Polars. Thank you!

This post summarizes the biggest highlights of that work.

Highlighted Features

Polars On-Prem: Distributed on Your Own Kubernetes

Polars Distributed is now available on your own infrastructure. Deploy a Polars cluster with a single Helm command on any Kubernetes setup (EKS, AKS, GKE, or minikube), point your existing Polars code at the cluster, and run distributed queries without sending data to a managed service. The deployment ships with a query dashboard, advanced profiling that shows per-stage resource use in real time, and OpenLineage support for data lineage tracking.

Start here for free.

Polars vs Spark, Benchmarked

We published a full benchmark comparing Polars and Spark on the PDS-H benchmark (a TPC-H derivative at 1 TB scale). On a single node (128 vCPUs, 512 GB RAM), Polars averaged 6.4x faster, with one query finishing 38x quicker. In a distributed setting, Polars averaged 3.2x faster, ranging from 1.6x to 7.7x across individual queries.

Read the methodology and full results in the benchmark post.

A Smarter Query Optimizer

PRs: #27340, #27775

Two optimizer improvements landed back to back. 1.41 extended common subplan elimination (CSE, the optimizer’s deduplication of shared subqueries) to work at every nesting depth, so shared subplans inside cache nodes are now deduplicated instead of re-evaluated. Nested CSE is opt-in: set the environment variable POLARS_ALLOW_NESTED_CSPE=1 to enable it.

1.42 added contradictory-filter elimination: when a filter predicate can never be satisfied (for example value > 100 AND value < 50), the optimizer folds the entire query to an empty result at plan time, skipping the scan, filter, and any aggregation.

Both improvements require no code changes.

Faster Cloud and Parquet I/O

PRs: #27924, #27427

1.42 introduced an adaptive concurrency controller for reading Parquet and IPC from object stores (S3, GCS, Azure). It tunes the number of in-flight requests to observed bandwidth and latency automatically, delivering a 2x average improvement and up to 4x on I/O-bound queries.

In 1.41, Parquet footer decoding got a hand-written Thrift decoder to replace the auto-generated one. The speedup scales with the number of columns in the table:

Columns	Before	After	Speedup
100	2.12 ms	1.32 ms	1.61x
1000	11.66 ms	4.43 ms	2.63x
10000	117.4 ms	35.74 ms	3.29x

Both improvements are automatic for existing scan_parquet and read_parquet calls. 1.42 also added external object store support: you can register a builder for any non-native scheme (such as HDFS or a custom S3-compatible store) and scan it with the same scan_parquet API.

The Streaming Engine Keeps Growing

1.40 continued expanding the set of operations the streaming engine can handle. Streaming grouped AsOf joins are now supported, extending the streaming AsOf support that landed in 1.39 to joins with a by argument. This only runs in streaming when the input is sorted by (by, on); otherwise it falls back to the in-memory engine.

import polars as pl

pl.Config.set_engine_affinity("streaming")

prices = pl.LazyFrame({
    "ticker": ["AAPL", "AAPL", "MSFT", "MSFT"],
    "time": [1, 3, 1, 4],
    "price": [150.0, 152.0, 300.0, 302.0],
})
trades = pl.LazyFrame({
    "ticker": ["AAPL", "MSFT"],
    "time": [2, 3],
    "volume": [100, 200],
})

result = trades.join_asof(prices, on="time", by="ticker").collect()

# Output
# shape: (2, 4)
# ┌────────┬──────┬────────┬───────┐
# │ ticker ┆ time ┆ volume ┆ price │
# │ ---    ┆ ---  ┆ ---    ┆ ---   │
# │ str    ┆ i64  ┆ i64    ┆ f64   │
# ╞════════╪══════╪════════╪═══════╡
# │ AAPL   ┆ 2    ┆ 100    ┆ 150.0 │
# │ MSFT   ┆ 3    ┆ 200    ┆ 300.0 │
# └────────┴──────┴────────┴───────┘

Basic elementwise over() expressions can now run in the streaming engine, and more aggregate expressions (cov, corr, interpolate, skew, kurtosis, entropy) are natively lowered to streaming as well. We continue to recommend enabling the streaming engine by setting pl.Config.set_engine_affinity("streaming") at the start of your code.

Quality-of-life APIs

PRs: #27501, #27870, #26708

1.41 added LazyFrame.gather(), bringing index-based row selection into the lazy API so you no longer need to materialize a frame first:

import polars as pl

result = (
    pl.LazyFrame({"product": ["A", "B", "C", "D", "E"], "revenue": [100, 250, 80, 320, 150]})
    .gather([1, 3])
    .collect()
)

# Output
# shape: (2, 2)
# ┌─────────┬─────────┐
# │ product ┆ revenue │
# │ ---     ┆ ---     │
# │ str     ┆ i64     │
# ╞═════════╪═════════╡
# │ B       ┆ 250     │
# │ D       ┆ 320     │
# └─────────┴─────────┘

1.42 added DataFrame.is_sorted() and Expr.is_sorted(), matching the existing Series.is_sorted() interface with support for descending and nulls_last:

import polars as pl

df = pl.DataFrame({"a": [1, 2, 3], "b": [5, 4, 3]})

df.is_sorted("a")
# Output
# True

df.select(pl.col("b").is_sorted(descending=True))
# Output
# shape: (1, 1)
# ┌──────┐
# │ b    │
# │ ---  │
# │ bool │
# ╞══════╡
# │ true │
# └──────┘

Popular Blog Posts

• Fluent, not native: agents translating pandas to Polars

  If you work with LLMs or agents and want to use Polars, this post shows how well current models handle pandas-to-Polars migration, where they still fall short, and the new Polars skill we built to improve translation quality.

• Handling Schema Issues in Polars

  If schema mismatches across CSV inference, multi-file Parquet, Delta Lake, or Iceberg are causing headaches, this post explains how Polars handles each case and how to take explicit control when you need it.

And Much More

This is just a glimpse of what we’ve built.

If you want to stay up to date, you can follow us on the following platforms:

[LinkedIn] - [Twitter/X] - [Bluesky] - [GitHub]

And you can learn more about Polars Cloud at: https://cloud.pola.rs/