Polars Nanook benchmarked against Databricks Photon

When we published our distributed benchmarks in June earlier this year, we compared Polars Nanook against open source Spark and observed a 3.2x speed-up. Today we are happy to announce a 6x speed-up.

A question we got at the time was how Polars Nanook compares to other accelerated distributions of Spark. In this blog post we compare Polars Nanook against the Databricks Photon engine and show that we beat Photon in 17 out of 22 queries. But first we cover where we came from and what improvements we made over this year.

Our distributed engine over the past year

We launched our distributed engine in September last year. Over the course of this year the team has worked very hard to improve the resilience and performance. In the past year alone we made over a 3x improvement on the PDSH¹ benchmark compared to last year’s version. The graph below shows our progress and the major improvements that have led to this success.

Cost-based planner (v0.6)

Initially the planner was rule-based, splitting the query plan into stages based on heuristics. In version 0.6 the cost-based planner became the default planner. It takes into account costs (shuffles, joins, …) and generates significantly more optimized query plans than before. This served as a strong foundation for many improvements down the line.

Work-stealing task scheduler (v0.9)

When there is data skew in the dataset or heavy I/O, some workers may finish earlier than others. This can lead to stalls and unoptimized resource usage, because some workers would be waiting on others. With work-stealing, our scheduler is able to steal tasks from workers and provide those tasks to workers that can start right away.

Runtime (bloom) filters (v0.13)

Since v0.13, Polars Nanook supports runtime bloom filters. Bloom filters are efficient data structures that can say whether some key or combination of keys does not exist in the dataset. This is especially useful for joins: joining a large table in a distributed manner often requires partitioning and shuffling these partitions over the network, which is expensive. In some cases a bloom filter can be constructed from the smaller table and sent to the reader of the larger table. There it can be applied before partitioning, significantly reducing the number of bytes over the network.

Comparison with Databricks

We ran Polars Nanook against Databricks on PDSH at scale factor 1000, directly reading Parquet from S3. This uses the same approach as in the earlier post². Polars Nanook is faster on 17 out of the 22 queries, with an overall speed up of 28%.

Overall Polars is 1.3 times faster than Databricks and 6 times faster than OSS Spark for distributed workloads.

For single node, Polars Nanook is much more efficient than Spark. Utilizing Polars for both single node and distributed workloads therefore speeds up your workloads and drastically simplifies your infrastructure needs.

Try it out now for yourselves

Polars Nanook is available in any Kubernetes distribution or managed in AWS with a generous free tier to test it on your pipelines. It is easy to set up and you’re up and running in no time. Sign up or ask for a demo with one of our engineers.