Planet Python
Follow
Bob Belderbos: How Libraries Run Rust Inside Python (with PyO3)
Pydantic v2's data validation relies on a Rust extension called pydantic-core, built using PyO3. This post demonstrates how to create a custom JSON parser in Rust and expose it to Python. The process involves writing a Rust module, annotating it with PyO3 macros, and using maturin to compile and install it. The core of the example is a Rust enum that represents a JSON tree structure. Exposing a function to Python requires specific PyO3 annotations and types like Python and Bound. The most significant aspect is the conversion of Rust data structures back into Python objects. This "return trip" involves creating numerous Python objects, which can be more costly than the parsing itself for large JSON documents. This conversion is handled by implementing the IntoPyObject trait. Errors also need to be translated; Rust errors are converted into Python exceptions using the From trait. For scalar return values, the cost of crossing the boundary is negligible. However, for large structures, the conversion cost is substantial and can dominate performance. Optimizing this boundary conversion, potentially by returning lazy views, is crucial for successful Rust-to-Python ports. Profiling the boundary between Rust and Python is therefore essential.