Extending Python with Rust

Python is a good language for the program around the hot loop and a slow language for the hot loop itself. When one function spends its time walking a million characters, the usual fix is to write that function in a language with no interpreter in the way, and keep the rest of the program as it is. Rust is a good candidate: it produces a shared library with a plain C interface, and the same file can also be a Python module with a proper type system at the boundary.

This article takes one honest example, a word count over a generated text, and makes it fast twice. The first way uses nothing but cargo and the ctypes module of the standard library. The second uses PyO3 and maturin, which is what most projects do today. Every command was run on the machine described at the end, and every number is a real measurement from it.

The examples live in /tmp and in a virtual environment, because a Python project that installs into the system interpreter is a project that breaks the system. The page on virtual environments covers the idea, pip covers the installer, and the Rust side of the boundary is the subject of the lessons on calling C from Rust and cdylib and staticlib crates.

The function that is too slow

The example is a word count with no regular expression and no library: a text of a million and a quarter bytes is generated from eight words, and one function counts how often one of them appears. The count runs forty times, which is the shape of a program that calls the same helper for every row of a table.

The timing uses time.perf_counter, the clock with the highest resolution Python offers, and it measures only the loop, not the generation of the text.

import sys import time WORDS = 200_000 repeats = int(sys.argv[1]) if len(sys.argv) > 1 else 40 def make_text() -> str: words = ["rust", "python", "extension", "fast", "safe", "wheel", "abi3", "ctypes"] lines = [] for index in range(WORDS): lines.append(words[index % len(words)]) return " ".join(lines) def count_word(text: str, wanted: str) -> int: total = 0 for word in text.split(): if word == wanted: total += 1 return total text = make_text() start = time.perf_counter() result = 0 for _ in range(repeats): result += count_word(text, "rust") elapsed = time.perf_counter() - start print(f"{len(text)} bytes, {repeats} passes, {result} hits") print(f"pure python: {elapsed:.3f} s")

The environment is created once and activated, so every later command uses the interpreter inside it. The two exports are what the activate script does.

python3 -m venv /tmp/venv
export VIRTUAL_ENV=/tmp/venv
export PATH="/tmp/venv/bin:$PATH"
python count.py 40

1249999 bytes, 40 passes, 1000000 hits
pure python: 1.721 s

Forty passes take 1.7 seconds, so one pass takes about 43 milliseconds and the function is the whole program. The cost is in str.split: it builds a list of two hundred thousand strings on every pass, and the loop then throws all of them away. A faster Python would avoid the list, but the interpreter still visits every character, which is where the rest of the time goes.

A cdylib and ctypes

The shortest path from Rust to Python needs no crate. Cargo builds a cdylib, a shared library with a C interface; Python loads it with ctypes and calls a function by name. The whole contract is one signature written twice, once in Rust and once in the Python script.

The manifest declares the crate type and nothing else. The name of the library file follows the platform: libfastcount.so on Linux.

[package] name = "fastcount" version = "0.1.0" edition = "2024" [lib] crate-type = ["cdylib"]

The function takes a pointer and a length for the text and the same pair for the word, and returns the count. A C caller has no other way to pass a buffer, and Rust cannot check the pointer, so the whole body is an unsafe promise that the caller passed a valid slice.

use std::slice; #[unsafe(no_mangle)] pub extern "C" fn count_word( text: *const u8, length: usize, wanted: *const u8, wanted_length: usize, ) -> usize { let text = unsafe { slice::from_raw_parts(text, length) }; let wanted = unsafe { slice::from_raw_parts(wanted, wanted_length) }; text.split(|byte| byte.is_ascii_whitespace()) .filter(|word| *word == wanted) .count() } #[unsafe(no_mangle)] pub extern "C" fn offset() -> i64 { 5_000_000_000 }

split on a byte slice with is_ascii_whitespace gives the words without allocating, and the filter counts the ones that equal the wanted slice. The offset function exists for one of the mistakes at the end of the article.

The Python side sets argtypes and restype before the first call. Without them ctypes guesses 32-bit integers, which is the subject of the last section.

import ctypes import time lib = ctypes.CDLL("/tmp/ext/fastcount/target/release/libfastcount.so") lib.count_word.argtypes = [ctypes.c_char_p, ctypes.c_size_t, ctypes.c_char_p, ctypes.c_size_t] lib.count_word.restype = ctypes.c_size_t words = ["rust", "python", "extension", "fast", "safe", "wheel", "abi3", "ctypes"] text = " ".join(words[index % len(words)] for index in range(200_000)) data = text.encode() wanted = b"rust" start = time.perf_counter() result = 0 for _ in range(40): result += lib.count_word(data, len(data), wanted, len(wanted)) elapsed = time.perf_counter() - start print(f"{len(data)} bytes, 40 passes, {result} hits") print(f"ctypes: {elapsed:.3f} s")

Cargo puts the library in target/release, and the script points at it by absolute path so that it does not depend on the working directory.

cd /tmp/ext/fastcount
cargo build --release
cd /tmp/ext/count
python ctypes_count.py

   Compiling fastcount v0.1.0 (/tmp/ext/fastcount)
    Finished `release` profile [optimized] target(s) in 0.72s
1249999 bytes, 40 passes, 1000000 hits
ctypes: 0.066 s

Twenty-six times faster, and the function is the same one the interpreter ran. The call itself costs a few microseconds, which is invisible next to a pass over a megabyte.

The price of this way is the boundary. Only C types cross it: integers, floats and byte buffers. A Python string arrives as bytes, a list must be converted to an array by hand, a dictionary has no representation at all, and an error is a return code the caller must check, because there are no exceptions on this side. A wrong signature is worse than an error: it passes whatever the bytes mean and crashes the interpreter instead of raising. The next way keeps all of that in a crate.

PyO3 and maturin

PyO3 is the crate that implements the Python C API in Rust: it converts arguments, builds exceptions and writes the module initialisation function. maturin is the build tool that knows where Python keeps its headers and how a wheel is named, so the extension is built and installed with one command.

Three files make the project. The manifest names the crate, the crate type and the exact PyO3 version; the pyproject.toml tells pip and maturin how to build the package; the source file holds the functions and the module definition.

The version matters. PyO3 0.30.1 is the release that supports CPython 3.14, which is what this image has, so the project pins it. The wheel is built for CPython 3.14 only, because the abi3 feature is not enabled; with it the same wheel would load on every CPython from 3.9 up, at the cost of the functions that are not part of the stable ABI.

[package] name = "rustext" version = "0.1.0" edition = "2024" [lib] name = "rustext" crate-type = ["cdylib"] [dependencies] pyo3 = { version = "0.30.1", features = ["extension-module"] }

The [lib] name must equal the name of the module that Python imports, and the cdylib type is what makes a .so of it. The extension-module feature tells PyO3 not to link against libpython, because the interpreter that loads the module already provides it.

[build-system] requires = ["maturin>=1.16,<2.0"] build-backend = "maturin" [project] name = "rustext" requires-python = ">=3.9" classifiers = ["Programming Language :: Rust"] dynamic = ["version"]

The version is dynamic because cargo owns it: maturin reads Cargo.toml and writes the number into the wheel.

The source shows the whole module: the two counting functions, the prime counter used in the GIL section, the functions that take and return Python containers, the class, the error and the module definition. The attributes do the work that the C API does by hand: #[pyfunction] writes the conversion of the arguments, #[pyclass] and #[pymethods] build a Python type, and #[pymodule] writes the initialisation function.

use std::collections::HashMap; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; #[pyfunction] fn count_word(text: &str, wanted: &str) -> usize { text.split_whitespace().filter(|word| *word == wanted).count() } #[pyfunction] fn count_word_released(py: Python<'_>, text: String, wanted: String) -> usize { py.detach(move || { text.split_whitespace() .filter(|word| *word == wanted) .count() }) } #[pyfunction] fn count_heavy(text: String, wanted: String, passes: usize) -> usize { let mut total = 0; for _ in 0..passes { total += text .split_whitespace() .filter(|word| *word == wanted) .count(); } total } #[pyfunction] fn count_heavy_released(py: Python<'_>, text: String, wanted: String, passes: usize) -> usize { py.detach(move || { let mut total = 0; for _ in 0..passes { total += text .split_whitespace() .filter(|word| *word == wanted) .count(); } total }) } #[pyfunction] fn word_lengths(text: &str) -> Vec<usize> { text.split_whitespace().map(|word| word.len()).collect() } #[pyfunction] fn histogram(text: &str) -> HashMap<String, usize> { let mut counts = HashMap::new(); for word in text.split_whitespace() { *counts.entry(word.to_string()).or_insert(0) += 1; } counts } #[pyclass] struct Counter { total: usize, calls: usize, } #[pymethods] impl Counter { #[new] fn new() -> Counter { Counter { total: 0, calls: 0 } } fn add(&mut self, values: Vec<usize>) { self.total += values.iter().sum::<usize>(); self.calls += 1; } #[getter] fn total(&self) -> usize { self.total } #[getter] fn calls(&self) -> usize { self.calls } fn __repr__(&self) -> String { format!("Counter(total={}, calls={})", self.total, self.calls) } } fn primes_below(limit: u64) -> u64 { let mut count = 0; let mut candidate = 2; while candidate < limit { let mut divisor = 2; let mut prime = true; while divisor * divisor <= candidate { if candidate % divisor == 0 { prime = false; break; } divisor += 1; } if prime { count += 1; } candidate += 1; } count } #[pyfunction] fn count_primes(limit: u64) -> u64 { primes_below(limit) } #[pyfunction] fn count_primes_released(py: Python<'_>, limit: u64) -> u64 { py.detach(move || primes_below(limit)) } #[pyfunction] fn parse_sizes(values: Vec<String>) -> PyResult<Vec<u32>> { let mut sizes = Vec::new(); for value in values { match value.trim().parse::<u32>() { Ok(size) => sizes.push(size), Err(_) => return Err(PyValueError::new_err(format!("not a size: {value}"))), } } Ok(sizes) } #[pymodule] fn rustext(module: &Bound<'_, PyModule>) -> PyResult<()> { module.add_function(wrap_pyfunction!(count_word, module)?)?; module.add_function(wrap_pyfunction!(count_word_released, module)?)?; module.add_function(wrap_pyfunction!(count_heavy, module)?)?; module.add_function(wrap_pyfunction!(count_heavy_released, module)?)?; module.add_function(wrap_pyfunction!(word_lengths, module)?)?; module.add_function(wrap_pyfunction!(histogram, module)?)?; module.add_function(wrap_pyfunction!(parse_sizes, module)?)?; module.add_function(wrap_pyfunction!(count_primes, module)?)?; module.add_function(wrap_pyfunction!(count_primes_released, module)?)?; module.add_class::<Counter>()?; Ok(()) }

count_word takes &str, so PyO3 accepts any Python str and hands Rust a view of its UTF-8 buffer. That view has to be valid while the call runs, which PyO3 guarantees by borrowing from the Python object. The count_heavy pair takes String instead, which copies the text once and then lets the work run without borrowing the Python object.

maturin needs the virtual environment to install into, exactly like pip, and the two variables are the ones the activation script sets. maturin develop builds in debug by default; --release is not optional for a measurement.

import time import rustext words = ["rust", "python", "extension", "fast", "safe", "wheel", "abi3", "ctypes"] text = " ".join(words[index % len(words)] for index in range(200_000)) start = time.perf_counter() result = 0 for _ in range(40): result += rustext.count_word(text, "rust") elapsed = time.perf_counter() - start print(f"{len(text)} bytes, 40 passes, {result} hits") print(f"pyo3: {elapsed:.3f} s")

pip install maturin
cd /tmp/ext/rustext
maturin develop --release
cd /tmp/ext/count
python pyo3_count.py

Successfully installed maturin-1.16.0
🐍 Found CPython 3.14 at /tmp/venv/bin/python
🔗 Found pyo3 bindings
   Compiling pyo3 v0.30.1
   Compiling rustext v0.1.0 (/tmp/ext/rustext)
    Finished `release` profile [optimized] target(s) in 55.18s
📦 Built wheel for CPython 3.14 to /tmp/.tmpTLE8yI/rustext-0.1.0-cp314-cp314-linux_x86_64.whl
✏️ Setting installed package as editable
🛠 Installed rustext-0.1.0
1249999 bytes, 40 passes, 1000000 hits
pyo3: 0.219 s

The module is eight times faster than the interpreter and three times slower than the raw pointer. The difference is the boundary: every call validates the Python string and borrows its buffer before the loop starts, and the loop itself is the same. For a function that is called once with a large text that cost disappears; for a function called in a tight loop it is the price of the convenience.

The import is a plain import rustext because maturin installed the package into the environment that is active. Nothing else is needed: no sys.path edit, no ctypes call, no C header.

Lists, dicts and a class

A real extension passes more than a string. PyO3 converts the containers of both languages in both directions: a Python list to Vec, a Vec back to a list, a HashMap to a dict. The class in the module is a Rust struct with methods, exposed as a Python type with getters and a __repr__.

import rustext print(rustext.word_lengths("rust and python")) counts = rustext.histogram("a b a c a") print(dict(sorted(counts.items()))) counter = rustext.Counter() counter.add([1, 2, 3]) counter.add([10]) print(counter) print(counter.total, counter.calls)

The output shows the three shapes. word_lengths returns a list of integers, histogram returns a dict that the script sorts because the order of a Rust HashMap is not defined, and Counter is an object with two getters and a representation.

python data.py

[4, 3, 6]
{'a': 3, 'b': 1, 'c': 1}
Counter(total=16, calls=2)
16 2

A #[getter] turns a method into an attribute, which is why counter.total has no parentheses. The methods take &mut self, and PyO3 enforces the borrow at run time in the same way a RefCell does: two Python threads that hold the same object cannot both call a mutating method at once.

The conversion has a cost and it is worth knowing where it is: a list of a million integers becomes a Rust vector of a million values, and a dict becomes a hash map. For a large array the NumPy buffer protocol is faster than a list, and PyO3 supports it; the list is the shape that a script passes without thinking.

Errors as exceptions

A Rust function that can fail returns Result. PyO3 turns an Err into a Python exception and the whole call stack unwinds the way a Python programmer expects: PyValueError becomes ValueError, PyTypeError becomes TypeError, and any other error becomes an exception with the message it carries.

import rustext print(rustext.parse_sizes(["10", " 20 ", "30"])) try: print(rustext.parse_sizes(["10", "ten"])) except ValueError as error: print(f"caught: {error}") print(rustext.parse_sizes(["10", "ten"]))

python errors.py; echo "status $?"

[10, 20, 30]
caught: not a size: ten
Traceback (most recent call last):
  File "/tmp/ext/errors.py", line 8, in <module>
    print(rustext.parse_sizes(["10", "ten"]))
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
ValueError: not a size: ten
status 1

The traceback is the one Python writes for any exception, with the line of the call and the message from Rust in it. That is the difference from the ctypes version, where a failure is a return code and the caller decides whether to raise.

A panic is not an exception: PyO3 catches it at the boundary and raises pyo3_runtime.PanicException, which is a BaseException and therefore not caught by except Exception. A library should return an error rather than panic for that reason alone.

Releasing the GIL

A PyO3 function holds the global interpreter lock while it runs, so two Python threads that call it take turns even when the work is pure Rust. PyO3 offers a way out: the function asks for the Python token and calls detach, which releases the lock for the duration of the closure and takes it back afterwards. In older versions the same method was called allow_threads; in 0.30.1 it is detach.

The closure may not touch a Python object, which is why the detached functions take String and usize and not references into Python memory. The work is a prime count, a loop with no allocation, so the threads do not compete for the same cache lines.

The script times one call, then two threads that call the same function. With the lock held the two threads run one after the other; with detach they run at the same time, which is what the ratio in the output shows.

import threading import time import rustext limit = 600_000 def measure(function, threads): def worker(): function() workers = [threading.Thread(target=worker) for _ in range(threads)] start = time.perf_counter() for worker in workers: worker.start() for worker in workers: worker.join() return time.perf_counter() - start single = min(measure(lambda: rustext.count_primes(limit), 1) for _ in range(3)) print(f"one call {single:.3f} s") for name, function in (("held", lambda: rustext.count_primes(limit)), ("released", lambda: rustext.count_primes_released(limit))): two = min(measure(function, 2) for _ in range(3)) print(f"two threads, {name:8s} {two:.3f} s, {two/single:.2f}x the time of one call")

python gil5.py

one call                  0.158 s
two threads, held        0.319 s, 2.02x the time of one call
two threads, released    0.159 s, 1.00x the time of one call

Two threads with the lock take twice as long as one call; two threads without it take about the same time, a ratio of one, which means they overlapped. The machine is the limit: it does not give four cores worth of parallel work, so four threads would show a smaller gain and a longer run. The comparison that matters is the one between the two rows, and it is the same program on the same machine.

The call to detach is safe only when the closure does not need the interpreter. A function that reads a Python list, releases the lock and then touches the list again is a bug that the compiler cannot catch, because the borrow checker knows nothing about the GIL.

Shipping a wheel

maturin build --release writes a wheel into target/wheels, and that file is what a user installs. The wheel carries the compiled extension and, when the library needs one, a copy of the shared library it depends on.

On this image the build needs patchelf to repair the shared library, and maturin says so when it is missing. It is a Python package, so the same virtual environment can provide it.

pip install patchelf
maturin build --release
ls -1 target/wheels/

Successfully installed patchelf-0.19.3.0
🔗 External shared libraries to be copied into the wheel:
  /tmp/ext/rustext/target/maturin/librustext.so requires:
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1
🖨  Copied external shared libraries to package rustext.libs directory:
    /usr/lib/libgcc_s.so.1
📦 Built wheel for CPython 3.14 to /tmp/ext/rustext/target/wheels/rustext-0.1.0-cp314-cp314-musllinux_1_2_x86_64.whl
rustext-0.1.0-cp314-cp314-musllinux_1_2_x86_64.whl

The name says everything a package installer needs: the project, the version, the interpreter it was built for (cp314) and the platform (musllinux_1_2_x86_64, because this is Alpine with musl). A wheel built on glibc would be named manylinux and would not load here.

A fresh environment installs the file and imports the module, which is the whole test of a wheel.

python3 -m venv /tmp/venv2
/tmp/venv2/bin/pip install /tmp/ext/rustext/target/wheels/rustext-0.1.0-cp314-cp314-musllinux_1_2_x86_64.whl
/tmp/venv2/bin/python -c "import rustext; print(rustext.count_word('a b a', 'a'))"

Processing /tmp/ext/rustext/target/wheels/rustext-0.1.0-cp314-cp314-musllinux_1_2_x86_64.whl
Installing collected packages: rustext
Successfully installed rustext-0.1.0
2

The second environment has no maturin and no cargo; it has the wheel and the interpreter, which is what a user has. The lesson on cargo build covers the release profile that the wheel is built from, and the lesson on Cargo.toml covers the manifest read by maturin.

Three typical mistakes

The debug build. maturin develop without --release builds a debug library, and the difference is not a few percent. The same script, the same function, two builds:

maturin develop
python pyo3_count.py
maturin develop --release
python pyo3_count.py

1249999 bytes, 40 passes, 1000000 hits
pyo3: 4.773 s
1249999 bytes, 40 passes, 1000000 hits
pyo3: 0.214 s

The debug build is twenty-two times slower than the release build and three times slower than pure Python, which is the trap: a first measurement of an extension that looks disappointing is often a measurement of the debug profile.

The module name. The name in [lib], the name of the #[pymodule] function and the name Python imports must agree. A copy of the library under a different file name produces an import error whose second half is the only clue:

cp rustext.cpython-314-x86_64-linux-musl.so othername.so
python -c "import othername"

ImportError: dynamic module does not define module export function (PyInit_othername)

The file was loaded, so the import did not fail with ModuleNotFoundError; it failed because the file exports PyInit_rustext and Python looked for PyInit_othername. Renaming a module is a change of three names, not of one file.

The missing restype. A ctypes call returns a C int unless restype says otherwise, and a Rust i64 does not fit in it. The function below returns five billion, which a 32-bit integer cannot hold:

import ctypes lib = ctypes.CDLL("/tmp/ext/fastcount/target/release/libfastcount.so") print(lib.offset()) lib.offset.restype = ctypes.c_int64 print(lib.offset())

python restype.py

705032704
5000000000

The first call is wrong and raises nothing: the low 32 bits of five billion are 705032704 and that is what came back. The second call declares c_int64 and gets the real value. The same mistake with argtypes passes a Python integer as a 32-bit value, and the mistake with a pointer passes an address that was never a buffer.

The three timings

One table with the three versions of the same forty passes over the same text, all measured on the machine described below, all in release mode where a build mode exists.

The numbers are from this machine and move with it: a faster disk changes nothing here, a faster processor changes all three rows, and the ratio between them is what a reader should take away. The raw pointer wins because nothing crosses the boundary but an address; PyO3 loses some of its advantage on the conversion of the string on every call, and wins everything else that a real program needs.

The choice between the two ways is not about speed. A function with a C signature and no Python objects in it is finished faster with ctypes and cannot raise an exception, take a list or return a dict. A module that a Python programmer will import is written with PyO3, because the boundary behaves like Python: it converts, it raises and it releases the lock.

The commands were run with Python 3.14.10 and pip 26.2.2 in a virtual environment, maturin 1.16.0, pyo3 0.30.1, rustc 1.96.1 and cargo 1.96.1, on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev