Skip to content

Performance

Performance

Measured with just bench:

exact lookup, 200,000 rows throughput vs. map_elements
pycountry via Expr.map_elements 845k rows/s —
polars_pycountry, parallel=False 12.3M rows/s 14.5×
polars_pycountry, parallel=True 58M rows/s 69×
fuzzy lookup, 2,000 rows throughput vs. map_elements
pycountry via Expr.map_elements 130 rows/s —
polars_pycountry, fuzzy=True, serial 604k rows/s 4,663×
polars_pycountry, fuzzy=True, parallel 702k rows/s 5,419×

AMD Ryzen 9 3950X (16 cores / 32 threads), 31 GB RAM, Linux 6.18 (WSL2), Python 3.12.13, Polars 1.44.0. Median of six runs; the parallel=True exact figure varies by about ±13% between runs, because at 200,000 rows it completes in ~3 ms.

The two row counts differ on purpose. pycountry.countries.lookup is a dict probe, so 200,000 rows of it take a fraction of a second; search_fuzzy scans every country and every subdivision, in Python, with no cache, at ~8 ms per call — 200,000 rows of that would take about half an hour. Both sides are measured on the same count, so each ratio still compares like with like.

What the fuzzy number depends on

Fuzzy matching is memoized: each call resolves the distinct values in the column, not the rows. Real columns repeat themselves heavily, so the cost usually lands closer to the cardinality than to the row count — but it is worth knowing which regime you are in:

200,000 rows, fuzzy=True parallel=False parallel=True
14 distinct values 15.3M rows/s 21.7M rows/s
4,891 distinct values 182k rows/s 3.03M rows/s

Not part of just bench: the low-cardinality row draws from the same 14-value vocabulary the fuzzy benchmark uses, the high-cardinality row from every distinct ISO 3166-2 subdivision name. Median of five runs.

Above 100,000 rows the distinct values are resolved once up front, in parallel, and the rows are filled from the result. Without that each thread would build its own cache and re-run the same searches, which made parallel=True slower than parallel=False on a high-cardinality column.

Measure a release build. just bench builds one; a plain maturin develop is unoptimized and far slower on this workload, which measures the profile rather than the code.

Columns of 100k rows or more are split across rayon threads; pass parallel=False to force single-threaded. The threshold sits above the streaming engine's morsel size, so when Polars is already calling the plugin from several of its own worker threads each call stays single-threaded rather than nesting a fan-out inside it.

A caveat worth stating plainly: if your column has few distinct values, a dict built over Series.unique() plus replace_strict can still beat any per-row approach, including this one. This package wins on high-cardinality columns, on the fuzzy path, and on code you would rather not write.