Announcements18 Nov 264 mins

Accelerating Polars + BigQuery pipelines with the polars-bigquery package

Google and Polars have launched polars-bigquery, a native I/O extension built on Google Cloud's Rust libraries that loads BigQuery data into Polars 3x to 4x faster than the previous standard approaches.

Tim Swena (Google)
Polars and BigQuery logos

Google and Polars have launched the polars-bigquery package, a native I/O extension designed for BigQuery. By wrapping Google Cloud’s official Rust libraries within Python, it speeds up Polars workflows far beyond traditional Python clients. Key features include:

  • Eager evaluation: The read_query method returns query results directly as a DataFrame.
  • Lazy evaluation: The scan_table method generates a LazyFrame, enabling column selection and row predicate pushdown to BigQuery.

Benchmark testing demonstrates a 3x to 4x speedup in query download performance versus former standard approaches for loading data into Polars DataFrames.

End-to-end query and DataFrame load time by result set size, comparing the polars-bigquery Rust plugin with the Python SDK Storage Read and REST JSON paths

To minimize query latency even further, the polars-bigquery package leverages key BigQuery capabilities like Arrow serialization alongside optional job creation. You can harness these performance enhancements directly via the eager read_query function.

Previous method, using pyarrow

import polars as pl
from google.cloud import bigquery

bq = bigquery.Client(project="your-project-id")

query = """
SELECT name, number, year
FROM bigquery-public-data.usa_names.usa_1910_2013
WHERE STARTS_WITH(name, 'T')
AND number > 10
AND year = 2000
"""

# %%timeit
# 2.15 s ± 76.3 ms per loop
# (mean ± std. dev. of 7 runs, 1 loop each)
pyarrow_table = bq.query(query).to_arrow()
bq_df = pl.from_arrow(pyarrow_table)

With polars-bigquery (3.9 times faster!)

import polars_bigquery

plbq = polars_bigquery.Client(quota_project_id="your-project-id")

query = """
SELECT name, number, year
FROM bigquery-public-data.usa_names.usa_1910_2013
WHERE STARTS_WITH(name, 'T')
AND number > 10
AND year = 2000
"""

# %%timeit
# 551 ms ± 136 ms per loop
# (mean ± std. dev. of 7 runs, 1 loop each)
plbq_df = plbq.read_query(query)

With Polars LazyFrames returned by the scan_table method, the speedup is technically unbounded due to the ability to push down row predicates and column selection to the BigQuery Storage Read API. For example, when filtering the usa_names public dataset, we get about a 3x speedup.

Previous method, no filter pushdown

import polars as pl
from google.cloud import bigquery

bq = bigquery.Client(project="your-project-id")

# %%timeit
# 2.6 s ± 115 ms per loop
# (mean ± std. dev. of 7 runs, 1 loop each)
bq_df = (
    pl.from_arrow(
        bq.list_rows("bigquery-public-data.usa_names.usa_1910_2013").to_arrow()
    )
    .select(pl.col("name"), pl.col("number"), pl.col("year"))
    .filter(
        pl.col("name").str.starts_with("T")
        & (pl.col("number") > 10)
        & (pl.col("year") == 2000)
    )
)

With polars-bigquery (2.9 times faster!)

import polars as pl
import polars_bigquery

plbq = polars_bigquery.Client(quota_project_id="your-project-id")

# %%timeit
# 884 ms ± 57 ms per loop
# (mean ± std. dev. of 7 runs, 1 loop each)
plbq_df = (
    plbq.scan_table("bigquery-public-data.usa_names.usa_1910_2013")
    .select(pl.col("name"), pl.col("number"), pl.col("year"))
    .filter(
        pl.col("name").str.starts_with("T")
        & (pl.col("number") > 10)
        & (pl.col("year") == 2000)
    )
    .collect()
)

What are the benefits of using polars-bigquery?

We created this connector for three main reasons:

  • To support filter pushdown from a Polars LazyFrame to the BigQuery engine, using the Polars IO plugin API.
  • To boost overall efficiency for users extracting Polars DataFrames from BigQuery, overcoming limitations inherent to legacy Python dependencies. With a streamlined, Polars-centric API, connector maintainers gain the agility to adopt higher-performance APIs in the Google Cloud Rust SDK as they launch, all without disrupting the Python interface.
  • To reduce dependency overhead when accessing BigQuery via Polars. Prior to this library’s release, integrating the two required the pyarrow package, which adds significant weight. As an example, the Linux wheel for pyarrow version 24.0.0 comes in at 52.0 MB. Given the overlapping features between Polars and pyarrow, incorporating pyarrow as a project dependency is a decision that requires careful consideration.

Comparison with pandas

The polars-bigquery connector shows a significant speedup over pandas workloads as well. Comparing the polars-bigquery read_query method against pandas-gbq’s read_gbq method shows a 3x to 11x speedup when querying a large table, such as bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2018.

Dataset rows pandas-gbq (rows/s) polars-bigquery (rows/s) Speedup
51,999 862.8 2,635.3 3.05x
103,998 1,597.1 5,366.3 3.36x
259,999 3,663.5 14,037.8 3.83x
519,998 6,220.3 23,913.2 3.84x
1,039,996 9,739.9 37,993.2 3.90x
2,599,998 14,410.1 80,748.1 5.60x
5,199,991 16,953.6 66,999.2 3.95x
10,399,999 19,925.3 165,593.2 8.31x
25,999,997 20,619.7 212,155.1 10.29x
51,999,999 21,133.7 233,918.6 11.07x

In addition to transitioning to Rust, several key BigQuery capabilities contribute to this performance boost:

  • Default LZ4_FRAME compression and Arrow serialization: Query results leverage Arrow serialization combined with default LZ4_FRAME compression to minimize data transfer overhead.
  • Direct Arrow support: Native Polars support eliminates the Arrow conversion step, unlike pandas-gbq which relies on numpy-backed arrays for numerous data types.
  • Persistent client object: The polars-bigquery connector uses a long-lived Client instance, maximizing resource reuse across queries compared to the global read_gbq function in pandas-gbq.
  • Optimized parallel processing: With the new maintain_order=False option, parallel downloads are more aggressive; the connector clears the work queue immediately without waiting for complete message processing, enabling faster incoming transfers.

What’s next for polars-bigquery?

We plan to close the loop on the user journey by adding support for:

Do you have other feature requests? Found a bug? We’d love to hear from you. File feedback with the team on our GitHub repository at github.com/pola-rs/polars-bigquery-client/issues.