Performance

Performance¶

200,000 URLs, measured with just bench:

_{AMD Ryzen 9 3950X (16 cores / 32 threads), 32 GB RAM, Linux 6.18 (WSL2), Python 3.12.13, Polars 1.43.}

Measure a release build. just bench builds one; a plain maturin develop is unoptimized and roughly 15× slower on this workload, which measures the profile rather than the code.

Columns of 100k rows or more are split across rayon threads; pass parallel=False to force single-threaded. The threshold sits above the streaming engine's morsel size, so when Polars is already calling the plugin from several of its own worker threads each call stays single-threaded rather than nesting a fan-out inside it.

The parallel figure scales with core count — 233× reflects 32 threads, and a 4-core laptop will land far below it. It is also by far the noisiest of the three, swinging ~20% run to run with thread scheduling while the single-threaded number holds within a couple of percent. The single-threaded number is the one to reason about when Polars is already saturating your cores.

A caveat worth stating plainly: if your column has far fewer distinct URLs than rows, a dict built over Series.unique() plus replace_strict can still beat any per-row approach, including this one. This package wins on columns with high cardinality, and on code you would rather not write.