DEV Community
Follow
I built a Python ORM with a Rust engine — here's how the GIL, PyO3, and asyncio actually cooperate
Yara-orm is a new asynchronous ORM designed for Python, offering a Django-style model interface but with a Rust-based engine for enhanced read performance. The core innovation lies in its dual-runtime architecture, combining Python's single-threaded asyncio event loop with Tokio's multi-threaded runtime in Rust. This setup aims to prevent blocking between Python's I/O operations and database operations.The Global Interpreter Lock (GIL) presents a challenge, requiring explicit handling in the Rust code when interacting with Python objects. Yara-orm manages this by only acquiring the GIL during parameter binding (Python to Rust) and row decoding (Rust to Python). The Rust database engine, including connection pooling and query execution, operates with the GIL released, allowing other Python tasks to run concurrently.Data passed between these runtimes is owned and must be Send to ensure safety across threads. Rust futures are converted into Python awaitables using pyo3-async-runtimes, seamlessly integrating with Python's asyncio loop. When a Rust future completes on a Tokio worker thread, its result is scheduled back to the asyncio loop for processing.This architecture ensures that the asyncio event loop remains unblocked by database I/O, and database I/O never holds the GIL. Performance optimizations are concentrated on the row decoding process, which occurs per row and is a significant factor in ORM speed. These optimizations include efficient type handling and dispatch mechanisms for faster data conversion.For SQLite, which uses a synchronous driver, a dedicated blocking thread pool is employed to maintain the decoupling principle. The choice between direct conversion with GIL holding and a serialization approach (like MessagePack) is presented as a trade-off. Yara-orm opts for direct conversion, believing it offers better performance for typical single-event-loop async Python services.The project is available for installation via pip and provides links to its GitHub repository and documentation. The author is also seeking insights from other developers who have built similar PyO3 bridges.