Skip to content

polars-tldextract

Accurate URL domain parsing for Polars, as a native Rust expression plugin.

Splitting a hostname into subdomain / domain / public suffix is not a string operation. www.bbc.co.uk and blog.cloudflare.com look identical to a regex, but the registrable domain is bbc.co.uk in one and cloudflare.com in the other — "last two labels" is wrong half the time. Getting it right requires the Public Suffix List, and in Python that means tldextract — an excellent library, but a Python function. Inside Polars it can only be driven through Expr.map_elements, one interpreter round-trip per row.

This package implements the same algorithm in Rust and exposes it as ordinary Polars expressions. It is built to produce identical output to tldextract, not merely similar output — see Correctness.

import polars as pl
import polars_tldextract as tld

df = pl.DataFrame({
    "url": [
        "https://www.bbc.co.uk/news/technology",
        "github.com",
        "https://blog.cloudflare.com:443/page/2/",
        "127.0.0.1",
        None,
    ]
})

df.with_columns(
    tld.fqdn("url").alias("fqdn"),
    tld.registrable_domain("url").alias("registrable_domain"),
    tld.suffix("url").alias("suffix"),
)

┌─────────────────────────────────────────┬─────────────────────┬────────────────────┬────────┐
│ url                                     ┆ fqdn                ┆ registrable_domain ┆ suffix │
╞═════════════════════════════════════════╪═════════════════════╪════════════════════╪════════╡
│ https://www.bbc.co.uk/news/technology   ┆ www.bbc.co.uk       ┆ bbc.co.uk          ┆ co.uk  │
│ github.com                              ┆ github.com          ┆ github.com         ┆ com    │
│ https://blog.cloudflare.com:443/page/2/ ┆ blog.cloudflare.com ┆ cloudflare.com     ┆ com    │
│ 127.0.0.1                               ┆ 127.0.0.1           ┆ null               ┆ null   │
│ null                                    ┆ null                ┆ null               ┆ null   │
└─────────────────────────────────────────┴─────────────────────┴────────────────────┴────────┘

Install

pip install polars-tldextract
# or
uv add polars-tldextract

Prebuilt wheels cover Linux (glibc and musl, x86_64 and aarch64), macOS (Intel and Apple Silicon), and Windows (x64 and arm64). There is one wheel per platform rather than one per Python version, because the extension is built against the stable ABI. An sdist is published too, so anything else builds from source given a Rust toolchain — see CONTRIBUTING.md.

Compatibility
Python 3.10+ — one abi3 wheel covers all versions
Polars 1.37+ — the plugin FFI ABI is (0, 1) and unchanged across that range
Linux manylinux2014 and musllinux_1_2, x86_64 and aarch64
macOS x86_64 (10.12+) and arm64 (11.0+)
Windows x64 and arm64

If a future Polars release bumps the plugin ABI, this package fails loudly at load rather than miscomputing.

Where to go next

  • Usage — the six expressions, null semantics, and the .tld namespace.
  • The suffix list — the vendored snapshot, overriding it, and swapping it in a running process.
  • Performance — measured throughput against map_elements, and when not to reach for this.
  • Correctness — how parity with tldextract is proven.
  • API reference — every public expression and helper.
  • Architecture — why the publicsuffix crate's trie walk lines up with tldextract's.