Eventual Rust in CPython

Welcome to LWN.net The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!

Emma Smith and Kirill Podoprigora, two of Python's core developers, have opened a discussion about including Rust code in CPython, the reference implementation of the Python programming language. Initially, Rust would only be used for optional extension modules, but they would like to see Rust become a required dependency over time. The initial plan was to make Rust required by 2028, but Smith and Podoprigora indefinitely postponed that goal in response to concerns raised in the discussion.

The proposal

The timeline given in their pre-PEP called for Python 3.15 (expected in October 2026) to add a warning to Python's configure script if Rust is not available when Python is built. Any uses of Rust would be strictly optional at that point, so the build wouldn't fail if it is missing. At this stage, Rust would be used in the implementation of the standard library, in order to implement native versions of Python modules that are important to the performance of Python applications, such as base64. Example code to accomplish this was included in the proposal. In 3.16, the configure script would fail if Rust is missing unless users explicitly provide the "--with-rust=no" flag. In 3.17 (expected in 2028), Python could begin strictly requiring Rust at build time — although it would not be required at run time, for users who get their Python installation in binary form.

Besides Rust's appeal as a solution to memory-safety problems, Smith cited that there are an increasing number of third-party Python extensions written in Rust as a reason to bring the proposal forward. Perhaps, if Rust code could be included directly in the CPython repository, the project would attract more contributors interested in bringing their extensions into the standard library, she said. The example in the pre-PEP was the base64 module, but she expressed hope that many areas of the standard library could see improvement. She also highlighted the Rust for Linux project as an example of this kind of integration going well.

Cornelius Krupp was apprehensive; he thought that the Rust-for-Linux project was more of a cautionary tale, given the public disagreements between maintainers. Those disagreements have settled down over time, and the kernel community is currently integrating Rust with reasonable tranquility, but the Rust for Linux project still reminds many people of how intense disagreements over programming languages can get in an established software project. Jacopo Abramo had the same worry, but thought that the Python community might weather that kind of disagreement better than the kernel community has. Smith agreed with Abramo, saying that she expected the experience to be "altogether different" for Python.

Steve Dower had a different reason to oppose the proposal: he wasn't against the Rust part, but he was against adding additional optional modules to Python's core code. In his view, optional extensions should really live in a separate repository. Da Woods called out that the proposal wouldn't bring any new features or capabilities to Python. Smith replied (in the same message linked above) that the goal was to eventually introduce Rust into the core of Python, in a controlled way. So, the proposal wasn't only about enabling extension modules. That didn't satisfy Dower, however. He said that his experience with Rust, mixed-language code, and teams forced him to disapprove of the entire proposal. Several other community members agreed with his disapproval for reasons of their own.

Chris Angelico expressed concern that Rust might be more susceptible to a "trusting trust" attack (where a compiler is invisibly subverted to introduce targeted backdoors) than C, since right now Rust only has one usable compiler. Sergey Davidoff linked to the mrustc project, which can be used to show that the Rust compiler (rustc) is free of such attacks by comparing the artifacts produced from rustc and mrustc. Dower agreed that Rust didn't pose any more security risk than C, but also wasn't sure how it would provide any security benefits, given that CPython is full of low-level C code that any Rust code will need to interoperate with. Aria Desires pointed to the recent Android Security post about the adoption of Rust as evidence that mixed code bases adopting Rust do end up with fewer security vulnerabilities.

Not everyone was against the proposal, however. Alex Gaynor and James Webber both spoke up in favor. Guido van Rossum also approved, calling the proposal a great development and saying that he trusted Smith and others to guide the discussion.

Stephan Sokolow pointed out that many people were treating the discussion as being about "Rust vs. C", but that in reality it might be "Rust vs. wear out and stop contributing". Paul Moore thought that was an insightful point, and that the project should be willing to put in some work now in order to make contributing to the project easier in the future.

Nathan Goldbaum is a maintainer of the PyO3 project, which provides Rust bindings to the Python interpreter to support embedding Python in Rust applications and writing Python extensions in Rust. He said that having official Rust bindings would significantly reduce the amount of work he has to do to support new Python versions. Another PyO3 maintainer, David Hewitt, agreed, going on to suggest that perhaps CPython would benefit from looking at the API that PyO3 has developed over time and picking "the bits that work best".

Raphael Gaschignard thought that the example Rust code Smith had provided would be a more compelling argument for adopting Rust if it demonstrated how using the language could simplify error handling and memory management compared to C code. Smith pointed out one such example, but concurred that the current proof-of-concept code wasn't a great demonstration of Rust's benefits in this area.

Gentoo developer Michał Górny said that the inclusion of Rust in CPython would be unfortunate for Gentoo, which supports many niche architectures that other distributions don't:

I do realize that these platforms are not "supported" by CPython right now. Nevertheless, even though there historically were efforts to block building on them, they currently work and require comparatively little maintenance effort to keep them working. Admittedly, the wider Python ecosystem with its Rust adoption puts quite a strain on us and the user experience worsens every few months, we still manage to provide a working setup.

[...]

That said, I do realize that we're basically obsolete and it's just a matter of time until some projects pulls the switch and force us to tell our users "sorry, we are no longer able to provide a working system for you".

Hewitt offered assistance with Górny's integration problems. "I build PyO3 [...] to empower more people to write software, not to alienate." Górny appreciated the thought, but reiterated that the problem here was Rust itself and its platform support.

Scaling back

In response to the concerns raised in the discussion, Smith and Podoprigora scaled back the goals of the proposal, saying that it should be limited to using Rust for optional extension modules (i.e. speeding up parts of the standard library) for the foreseeable future. They still want to see Rust adopted in CPython's core eventually, but a more gradual approach should help address problems raised by bootstrapping, language portability, and related concerns that people raised in the thread, Smith said.

That struck some people as too conservative. Jelle Zijlstra said that if the proposal were limited to optional extension modules, it would bring complexity to the implementation of the standard library for a marginal benefit. Many people are excited about bringing Rust to CPython, Zijlstra said, but restricting Rust code to optional modules means putting Rust in the place that it will do the least good for the CPython code. Several other commenters agreed.

Smith pushed back, saying that moving to Rust was a long-term investment in the quality of the code, and that having a slow, conservative early period of the transition would help build out the knowledge and experience necessary to make the transition succeed. She later clarified that a lot of the benefit she saw from this overly careful proposal was doing the groundwork to make using Rust possible at all: sorting out the build-system integration, starting to gather feedback from users and maintainers, and prototyping what a native Rust API for Python could look like. All of that has to happen before it makes sense to consider Rust in the core code — so even though she eventually wants to reach that state, it makes sense to start here.

At the time of writing, the discussion is still ongoing. The Python community has not reached a firm conclusion about the adoption of Rust — but it has definitely ruled out a fast adoption. If Smith and Podoprigora's proposal moves forward, it still seems like it will be several years before Rust is adopted in CPython's core code, if it ever is. Still, the discussion also revealed a lot of enthusiasm for Rust — and that many people would rather contribute code written in Rust than attempt to wrestle with CPython's existing C code.