Announcing Polars 1.44

By Thijs Nieuwdorp on Wed, 26 Aug 2026

Polars 1.44 is out. Some of the highlights:

It adds schema evolution to the Iceberg sink, paces cloud I/O to stay under the request rate an object store will still serve, and reworks the SQL layer to handle correlated subqueries.

Iceberg: schema evolution on sink, deletion vectors on scan

sink_iceberg can now evolve the target table’s schema, and scan_iceberg reads v3 deletion vectors without falling back to a PyIceberg scan.

PR: #28794, #28793, #28772, #28634

Appending a frame to an Iceberg table used to require an exact schema match, so a column added upstream meant evolving the table by hand before the write. 1.44 adds schema_mode to sink_iceberg (#28794).

What is schema evolution?

Iceberg tracks a table’s schema as versioned metadata rather than as a property of the files on disk. Adding, dropping, or renaming a column is recorded as a metadata commit instead of a rewrite of the data files. Older files keep their own layout, and columns they do not contain read back as null. That makes it safe for a writer to extend the schema mid-pipeline.

import polars as pl

# Table currently has columns: id, value
lf = pl.LazyFrame({"id": [3], "value": [30.0], "source": ["sensor-a"]})

# Write a small table with a new column "source"
lf.sink_iceberg(
    table,
    mode="append",
    schema_mode="merge",
    snapshot_properties={"pipeline": "ingest", "run-id": "42"},
)

# Read back the written result
pl.scan_iceberg(table).collect().sort("id")

# Output
# shape: (3, 3)
# ┌─────┬───────┬──────────┐
# │ id  ┆ value ┆ source   │
# │ --- ┆ ---   ┆ ---      │
# │ i64 ┆ f64   ┆ str      │
# ╞═════╪═══════╪══════════╡
# │ 1   ┆ 10.0  ┆ null     │
# │ 2   ┆ 20.0  ┆ null     │
# │ 3   ┆ 30.0  ┆ sensor-a │
# └─────┴───────┴──────────┘

schema_mode="merge" matches columns by name and adds the ones the table is missing, so the column source is added and the two pre-existing rows will have null for that column. Without it, the same append raises SchemaError: schema names in arrow_schema differ: .... schema_mode="overwrite" requires mode="overwrite". Together they replace the table with the frame: columns the frame does not have are dropped from the schema, and the existing rows are deleted in the same transaction that adds the new ones. Earlier snapshots stay in the table history, so the previous state remains reachable.

snapshot_properties (#28793) attaches arbitrary key-value metadata to the commit, which lands in the snapshot summary next to Iceberg’s own counters:

table.refresh().current_snapshot().summary

# Output
# {'added-records': '1', ..., 'pipeline': 'ingest', 'run-id': '42', 'total-records': '3'}

The sink also honours the write.object-storage.enabled table property (#28634), writing PyIceberg-compatible Murmur3 hashed prefixes that spread files across object-store partitions.

The native scan_iceberg reader picks up a table layout that previously forced a fallback to PyIceberg. It now reads v3 deletion vectors (#28772), the Puffin-based replacement for positional delete files in Iceberg v3, so a v3 table with row-level deletes no longer needs a PyIceberg scan.

An adaptive rate limiter for cloud I/O

Polars now learns how many requests per second an object store will serve and paces itself to respect it.

PR: #28591, #28885

A large scan over S3, GCS, or Azure issues a lot of concurrent requests. Past the store’s per-prefix request limit, those requests come back as 429 (Too Many Requests) or 503 (Service Unavailable) responses instead of data, which costs time in retries and turns into an error once the retry budget runs out. Polars 1.42 made the number of in-flight requests adapt to observed bandwidth and latency, but nothing reacted to the object store pushing back.

1.44 adds a rate limiter in the HTTP layer, one per object store and separately for reads and writes. It follows an AIMD (Additive Increase Multiplicative Decrease) control loop: raise the rate while requests succeed, cut it sharply as soon as they do not, and pace admitted requests against whatever rate that leaves.

The sawtooth is the steady state: the limiter never stops probing for more throughput.

From a cold start the rate climbs fast to find the ceiling, then cuts back on each throttling response and works its way up again in small steps. The probing never stops, because the ceiling itself moves: S3 repartitions a prefix that stays hot and then serves a higher request rate for it than it did at the start of the query. A limiter that settled on the first ceiling it found would hold a long scan at its cold-start rate for the rest of the run.

That also means a 429 or 503 in the logs are not a signal of failure. To find out whether the store can provide more bandwith, the limiter has to ask for more, and that happens through these error codes. What should drop to almost nothing is queries that fail on throttling, where the retry budget runs out before the rate comes back down.

The defaults are an initial 1,000 requests per second, a floor of 10, and a ceiling of 50,000. They target the large cloud object stores, so most pipelines need no tuning at all. Each bound is tunable for reads and writes separately through storage_options:

import polars as pl

# A bucket known to be rate-limited well below the default ceiling.
lf = pl.scan_parquet(
    "s3://my-bucket/data/**/*.parquet",
    storage_options={
        "rate_read_init": 500,
        "rate_read_ceiling": 5_000,
    },
)

The read keys are rate_read_init, rate_read_floor, and rate_read_ceiling, with rate_write_* equivalents for sinks. Unlike most storage_options entries, these take integers rather than strings. Lower the ceiling if queries still fail on throttling after the retries, and raise the init rate for a store that does not throttle at all, such as a local MinIO, where the limiter would otherwise open at 1,000 requests per second and climb while the store would have served more from the start.

Retries were retuned to match (#28885): the budget goes from 2 attempts to 8, the initial backoff from 100 ms to 250 ms, the maximum backoff from 15 s to 5 s, and the overall retry timeout from 10 s to 30 s. Any scan_* or sink_* against S3, GCS, or Azure gets all of this automatically, with no code change.

SQL: correlated subqueries

Subqueries that reference the outer query are now supported.

PR: #28494, #28927, #28929, #28881, #28876, #28864, #28854

The SQL interface rejected subqueries that referred to the query around them. 1.44 reworks how subqueries are parsed and planned, adds a decorrelation pass, and lands conformance test suites for aggregate semantics, set operations, ORDER BY, QUALIFY, and column naming.

What is a correlated subquery?

An ordinary subquery is evaluated once and its result reused. A correlated subquery references a column from the outer query, so conceptually it has to be re-evaluated for every outer row: “the average salary in this row’s department”. Executing it that way is a per-row loop, which is why engines instead decorrelate it, rewriting the query into an aggregation plus a join that produces the same answer in one pass.

import polars as pl

employees = pl.LazyFrame({
    "name": ["Alice", "Bob", "Carol", "Dan", "Erin"],
    "dept": ["eng", "eng", "sales", "sales", "sales"],
    "salary": [120, 95, 80, 110, 90],
})

pl.sql("""
    SELECT name, dept, salary
    FROM employees
    WHERE salary > (
        SELECT AVG(x.salary) FROM employees AS x WHERE x.dept = employees.dept
    )
    ORDER BY name
""").collect()

# Output
# shape: (2, 3)
# ┌───────┬───────┬────────┐
# │ name  ┆ dept  ┆ salary │
# │ ---   ┆ ---   ┆ ---    │
# │ str   ┆ str   ┆ i64    │
# ╞═══════╪═══════╪════════╡
# │ Alice ┆ eng   ┆ 120    │
# │ Dan   ┆ sales ┆ 110    │
# └───────┴───────┴────────┘

On 1.43 the same query raises SQLSyntaxError: subquery comparisons with '>' are not supported. The same works in the select list, which covers the common “attach a per-group statistic to every row” shape, in EXISTS, and in correlated IN (#28927):

pl.sql("""
    SELECT name FROM employees
    WHERE dept IN (
        SELECT x.dept FROM employees AS x WHERE x.salary > employees.salary
    )
    ORDER BY name
""").collect()

# Output
# shape: (3, 1)
# ┌───────┐
# │ name  │
# │ ---   │
# │ str   │
# ╞═══════╡
# │ Bob   │
# │ Carol │
# │ Erin  │
# └───────┘

Two more subquery gaps close alongside this. ANY and ALL comparisons against a subquery are desugared (#28929): = ANY becomes IN and <> ALL becomes NOT IN, including in the correlated case, while other operators combined with ANY/ALL over a correlated subquery raise a clear SQLInterfaceError: ANY/ALL with `>` and a correlated subquery is not currently supported. Functions registered on the SQL layer are now also visible inside a subquery (#28881), which previously resolved against an empty registry.

The decorrelation pass comes with its own optimizations. Equality predicates no longer go through a generic cross join (#28876), CTEs are cached in the SQL layer (#28864), and unqualified join predicates in a WHERE clause are lowered to real inner joins instead of a cross join plus a filter (#28854).

Faster without changing anything

Broader performance and planner work that needs no code changes on your side:

when/then/otherwise only evaluates the rows it keeps (#28498). Instead of evaluating both branches over every row and picking afterwards, Polars sets the rows a branch will not contribute to null in the columns that branch reads, so an expensive expression only does work on the rows that survive. That applies to elementwise branches, and a branch is skipped entirely when the condition is uniformly true or false. On a 5,000,000-row frame extracting a number with a regex in the then branch: 6.10x faster at 5% selectivity, 1.53x at 50%, unchanged at 100%.
```
import numpy as np
import polars as pl

N = 5_000_000
rng = np.random.default_rng(0)

df = pl.DataFrame({
    "flag": rng.random(N) < 0.05,   # 5% of rows take the then branch
    "text": ["abc-12345-xyz"] * N,
})

df.select(
    pl.when(pl.col("flag"))
    .then(pl.col("text").str.extract(r"-(\d+)-", 1).cast(pl.Int64))
    .otherwise(0)
    .alias("code")
)
```
DataFrame.filter rechunks the mask once (#28762) when its chunks do not line up with the frame’s, rather than paying for the mismatch once per column. Filtering a 40-column single-chunk frame with a 500-chunk mask is 1.63x faster.
Parquet scans keep more of what they already read. Partial file metadata survives a filter on the scan (#28737) instead of being re-read, one metadata entry is retained per source (#28661), and plan-time row estimates for multi-file scans are more accurate (#28380).
The streaming engine prunes more of the plan. It drops projections a filter does not need (#28713), and len() is pushed into the inputs of a concat (#28570).

Deprecations

The rechunk parameter is deprecated on every read and scan function (#28063), and Expr.rechunk() is deprecated outright (#28692), since rechunking inside a query is not well defined now that the streaming engine controls batching. Call DataFrame.rechunk() on the collected result instead.
struct.rename_fields() with a different number of names than the struct has fields now warns (#28672) rather than silently dropping the surplus. Use the new struct.drop() (#28666) to remove the trailing fields explicitly.

And More

The full list of changes is in the 1.44 release notes on GitHub.

[LinkedIn] - [Twitter/X] - [Bluesky] - [Reddit] - [GitHub]