The fastest compute engine at any scale
The same query runs on one machine with open-source Polars, and unchanged across a cluster with Polars Distributed on Polars Nanook.
Polars total time against the next-fastest engine at each data size
10 GB · 16 vCPUs: 1.33× faster
100 GB · 16 vCPUs: 1.22× faster
100 GB · 192 vCPUs: 1.77× faster
1 TB · Polars Nanook 0.13: 1.36× faster
Built on industry-standard benchmarks
PDS-H is based on TPC-H, the standard benchmark for analytical databases. Its 22 queries filter, join and aggregate the sales data of a wholesale supplier. PDS-DS is based on TPC-DS and has 99 more complex queries on the data of a retailer. The scale factor (SF) sets the size of the data.
10 GB
About 10 GB of data as uncompressed CSV. We ran PDS-H at this size on a single machine.
100 GB
About 100 GB of data as uncompressed CSV. We ran PDS-H and PDS-DS at this size on a single machine.
1 TB
About 1 TB of data as uncompressed CSV. We ran PDS-H at this size on a cluster of 32 machines.
The fastest engine on a single machine
We ran open-source Polars 2.0, DuckDB 2.0 and DataFusion 54 on the same AWS machine, with 16 vCPUs and 32 GB of memory. Polars was the fastest on every benchmark. On a larger machine with 192 vCPUs, Polars was 1.77× faster than the next engine on PDS-H at 100 GB.
PDS-H at 10 GB
SF10 · 16 vCPUs, 32 GB · 22 queries, lower is better
- Polars 2.03.22 s
- DuckDB 2.04.27 s
- DataFusion 547.41 s
PDS-H at 100 GB
SF100 · 16 vCPUs, 32 GB · 21 queries, lower is better · Q18 excluded, DataFusion ran out of memory
- Polars 2.034.63 s
- DuckDB 2.042.12 s
- DataFusion 5478.04 s
PDS-DS at 100 GB
SF100 · 16 vCPUs, 32 GB · 97 queries, lower is better · Q67 and Q72 excluded, DataFusion timed out
- Polars 2.085.83 s
- DuckDB 2.098.59 s
- DataFusion 54198.48 s
When the data outgrows one machine
Polars Distributed runs each query across your cluster to handle terabytes of data. It is available in Polars Nanook. We ran PDS-H at 1 TB with Polars Nanook 0.13, PySpark, Trino and Databricks, each on the same cluster of 32 AWS machines. On the 19 queries all four finished, Polars Nanook was 5.48× faster than PySpark, 3.77× faster than Trino and 1.36× faster than Databricks.
PDS-H at 1 TB
SF1000 · 32 × m8i.xlarge · 19 queries, lower is better · Q8, Q9 and Q21 excluded, Trino ran out of memory
- Polars Nanook 0.13149.61 s
- Databricks 19203.84 s
- Trino 484564.28 s
- PySpark 4.0.1820.36 s
How we ran it
How we timed
At 10 and 100 GB, every query ran five times on a warm machine, each time in its own process with a 60-second timeout, and the fastest run counts. At 1 TB, PySpark / Trino / Nanook times are the fastest of three runs to avoid network stragglers skewing the results; Databricks times are from a single run.
Hardware
At 10 and 100 GB every engine ran on the same AWS c7a machines, reading Parquet from EBS. At 1 TB, every engine ran on the same 32 m8i.xlarge workers, reading from S3.
Out of memory
Some engines ran out of memory on some queries, so they could not finish the full benchmark. To still compare them, we gave those engines the benefit of the doubt and left those queries out for every engine, so each total covers the same work. The caption above each chart names the queries that were left out and why.
Run it yourself
The data generator settings, the queries and the run scripts are public, so you can repeat every run on your own hardware.
Frequently asked questions
On a 32-node cluster at 1 TB, PySpark 4.0.1 took 5.48× as long as Polars Nanook 0.13 to run the 19 PDS-H queries every engine finished.
In the single-machine benchmarks on this page, Polars 2.0 was faster than DuckDB on every run. On a 16 vCPU machine it was 1.33× faster than DuckDB 2.0 on PDS-H at 10 GB, 1.22× faster on PDS-H at 100 GB and 1.15× faster on PDS-DS at 100 GB. On a 192 vCPU machine it was 1.77× faster than DuckDB on PDS-H at 100 GB.
On a 32-node cluster at 1 TB, Trino 484 took 3.77× as long as Polars Nanook 0.13 to run the 19 PDS-H queries every engine finished. Trino ran out of memory on the other three, so those are left out for all engines.
PDS-H and PDS-DS, short for Polars Decision Support, are our open versions of TPC-H and TPC-DS. They keep the same data model, data generator and queries: 22 in PDS-H and 99 in PDS-DS. We rewrote the queries to match the original SQL and changed the run scripts so anyone can run them. Because of those changes, the results cannot carry the TPC name and may not be compared with officially published, audited TPC results.
The scale factor sets how much data the data generator produces. Scale factor 1 is about 1 GB as uncompressed CSV, so SF10 is about 10 GB, SF100 about 100 GB and SF1000 about 1 TB. The 10 and 100 GB runs read the data as Parquet, which is compressed, so the files on disk are smaller than that.
pandas could not run most of the PDS-H and PDS-DS queries at 10 GB and above, so its total time would cover only a small part of each benchmark. Rather than compare engines on a handful of queries, we left pandas out of the charts.
Moving from 16 to 192 vCPUs at 100 GB made Polars 3.8× faster on PDS-H and 2.2× faster on PDS-DS, more than any other engine we tested.
The 10 and 100 GB runs are in the pola-rs/polars-2.0-benchmark repository and the 1 TB runs in pola-rs/polars-benchmark, both on GitHub. They hold the data generator settings, the queries and the scripts that ran them, so you can repeat every run on your own hardware.
Open the 10 and 100 GB benchmarkOpen the 1 TB benchmark