Pre-requisites

You will need:

• The xcode command line tools from Apple
• Rust (instructions)

Here are the commands to install them:

• xcode-select --install
• curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

(those might change over time)

Install RustPython

First run:

cargo install --git https://github.com/RustPython/RustPython

If you want RustPython with ssl support, try:

cargo install --git https://github.com/RustPython/RustPython --features ssl

Go to Cargo’s bin directory with cd ~/.cargo/bin. Run ls, you should see the binary ~/.cargo/bin/rustpython.

If you type rustpython at the terminal, you should get the welcome message:

Setup PyCharm

In PyCharm, you can add an interpreter by using:

• virtual environment
• conda
• pipenv
• system interpreter
• poetry

The full docs are on this link. This blog post is for the two options: virtual environment and system interpreter.

System interpreter

Go to “Add Python Interpreter” -> System Interpreter -> Click on the browse icon.

Navigate to /Users/YOURUSERNAME/.cargo/bin - selecting rustpython as the interpreter gives this error:

There is an easy fix. Create a link/shortcut called python that points to rustpython.

cd ~/.cargo/bin
ln -s rustpython python

PyCharm is happy. Test out that things work by creating a python file in PyCharm and running

import sys
print(sys.executable)

Virtual environment

In the screen above, if you try to use the virtual option, you get this error.

However, you can still manually create the virtual environment. First, create a directory, cd into into it and run: ~/.cargo/bin/rustpython -m venv env --without-pip

This would create a virtual environement that looks like this.

Go the directory, open PyCharm, now you can set the interpreter from the virtual environement.

Setup Visual Studio Code

Install the Python Extension by Micorsoft. Create your Python file and press Command + Shift + P, then search for “Python: Select Interpreter”, add a new interpreter by using “Enter Interpreter Path” then browse your filesystem, go to ~/.cargo/bin/rustpython

You can repeat the same steps for creating a virtual environement and using that with Visual Studio Code.

This update is brought to you from @coolreader18’s terminal by asciinema 🍿.

get-pip.py

get-pip.py + pip install requests

pip installs itself and setup tools

What you need to do

(Update 2021-02-08): Everything has been merged to master!

    $ cargo install --git https://github.com/RustPython/RustPython --features ssl # or however you build
    $ curl https://bootstrap.pypa.io/get-pip.py -O
    $ rustpython get-pip.py
    ...lots of work...
    $ rustpython -m pip --version
    pip 21.0 from /home/.../.local/lib/rustpython3.9/site-packages/pip (python 3.9)

There’s undoubtedly still a lot that doesn’t work, so if there’s something simple-ish that errors inside of pip or a setup script, feel free to open an issue for it (note that C extensions won’t be supported anytime soon, and a ctypes implementation is in progress).

(Previous instructions):

This update is so fresh, things are not merged into the master or release branches yet. For now, to test this out, you have to checkout the pip-merg branch, and make sure to build with the ssl feature:

    $ cd RustPython
    $ git fetch --all
    $ git checkout pip-merg
    $ cargo install --path . --features ssl
    $ curl https://bootstrap.pypa.io/get-pip.py -O
    $ rustpython get-pip.py
    ...lots of work...
    $ rustpython -m pip --version
    pip 21.0 from /home/.../.local/lib/rustpython3.9/site-packages/pip (python 3.9)

Fix known compatibility bugs

Let’s find an incompatibility issue and fix it.

1. Look at the Lib/test directory of the project. There are many test_ prefixed files like test_unicode.py.
2. Try to open one of them. It might look just fine at a glance - but search for TODO: RUSTPYTHON in the files. There are tons of skipped, marked as an expected failure or commented out tests.
3. Alternatively, try looking at the regression test results to find skipped or expected failure tests; some of them have notes for a way to resolve the issue.
4. Choose one or two interesting bugs. Remove the test blocker - skip, expectedFailure or comments.
5. Try to fix them.

Here’s how you run a single unittest file:

$ RUSTPYTHONPATH=Lib cargo run --release Lib/test/test_unicode.py

Add a new unittest file

Because CPython unittest doesn’t work perfectly in RustPython, we are adding test files one by one. Here’s how:

1. Download the CPython source code with git clone https://github.com/python/cpython.git.
2. Check out a specific version of CPython. We test against CPython 3.8, so the most recent release of 3.8 (currently 3.8.7) is recommended.
3. Copy a file from CPython’s Lib/test
4. Commit the file without editing it. Specify the CPython version you copied from in the commit message.
5. Try to edit it until it runs without a crash or failure.
6. Commit the changes to make it run. This is the core contribution.

Because RustPython is not perfect, “try to edit it until it runs” doesn’t mean to make it run 100% of the tests successfully. The common methods to make the test file pass are:

1. It must at least be able to start to run the tests. Fix the test code or bug until it runs at least a single unit of the test. Typically, unimplemented stdlib or missing files in unittest can cause issues. Sometimes RustPython bugs cause issues too.
2. If any test can’t be loaded because of a SyntaxError, you’ll have to comment that part out.
3. If any test leads to a crash of RustPython, this code can’t be run. Mark the test with @unittest.skip('TODO: RUSTPYTHON') to skip it.
4. If any test runs but fails, this is an incompatibility issue. Mark the test as an expected failure with @unittest.expectedFailure.

We prefer the reversed order of above methods. The later, the more strict, so it’s easier to detect any progress or regression. When we temporarily disable parts of unittest due to RustPython caveats, we mark them to make it easier to find (and re-enable!) them later. Please see the examples below or search for TODO: RUSTPYTHON in the Lib/test directory to check actual usage.

Comment out:

# TODO: RUSTPYTHON
#
# def ...  # commented out tests

skip:

@unittest.skip("TODO: RUSTPYTHON")
def ...  # skipped tests

expectedFailure:

# TODO: RUSTPYTHON
@unittest.expectedFailure
def ...  # failed tests

Development guide

For a general introduction to RustPython development, please visit the RustPython development guide

When you write code in Python and run it, an interpreter, such as the RustPython interpreter, acts as the translator between you and your machine.

The interpreter has the job of turning your human code into bytecode that a Python virtual machine can run. Bytecode is an intermediate code between source code and machine code. This makes it portable across multiple hardware and operating systems. Bytecode “works” as long as you implement a virtual machine (vm) that can run it. There is a performance penalty for this flexibility. RustPython also has a vm that interprets the generated bytecode, other posts will go into the details of that vm but now let’s figure out how to turn code into bytecode.

What bytecode looks like

Seeing is believing. To see what bytecode looks like, you can use a Python module called dis. “dis” is short of for _dis_assembler. You can write source code then see how its bytecode looks like. Here is an example:

How RustPython turns your code into bytecode

Here are the main steps that RustPython currently goes through:

• parse the line of source code into tokens
• determine if the tokens have a valid syntax
• create an Abstract Syntax Tree (AST)
• compile the AST into bytecode

This list of steps introduces some new concepts like: tokens and abstract syntax trees. We’ll explain and demistify those.

Step 1: parsing source code into tokens

The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a tokenizer. Here is what happens if I run the tokenizer on the function that I created above.

$ python -m tokenize file.py

file.py has the function that I used in the previous example.

def add(x,y):
    return x+y

Tokenize output:

A picture IS worth a thousand words 😛 Those are the tokens. They are the basic “units” of the programming language. They are the keywords and operators that you typed. Even new lines and identation count.

If you want to sound fancy:

• The tokens are the basic “lexical components”
• The parsing process is called “lexical analysis”
• The thing that does this is a “lexer”

The code for the lexing stage lives in lex.rs of the parser crate.

If you want to dive into the details of lexical analysis, check out Python in a nutshell / Lexical structure

Step 2 : determine if the tokens are valid syntax

In the previous step, if you add random stuff to your function and tokenize it, it will work and still tokenize.

So don’t hate on the whole interpreter when you get error messages! or at least don’t hate on the tokenizer!

To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined “grammar” or set of rules. The official reference is on this link. There, you will find a machine readable file. You may read a book to know the rules of Python, but words are too “fluffy”, an algorithm that verifies if the rules are followed needs a very strict set of rules encoded in a file. This video explains the Python grammar and the file’s notation. As the presenter puts it, this is the spirit of the beast (Python) and it is only ~10KB 😭 (compare that to the size of the Python books you had to read!)

So, we have the rules or grammar of a programming language in a machine encoded format… now we need to write something that verifies that those rules were followed… This sounds like something that other people could use and like something that should exist as an open source project! 🤔

Sure enough, there is a whole Rust framework called LALRPOP. It takes the tokens generated by the lexer, verifies the syntax and turns the tokens into an AST (Abstract Syntax Tree). More information and a tutorial can be found in the LALRPOP book.

RustPython does one nice extra thing on top of LALRPOP. It masks the errors and provides you with safer, nicer errors. You can see the code for this in RustPython/parser/src/error.rs

Using RustPython to generate an AST

You can do:

 use rustpython_parser::{parser, ast};  
 let python_source = "print('Hello world')";  
 let python_ast = parser::parse_expression(python_source).unwrap();

Recap 🥴 🥵

As a recap, when you write a line of Python code and “run it”, here is what the RustPython interpreter does:

INPUT: your code (in file.py or interactive shell)
⬇️ parse the line of source code into tokens
⬇️ determine if the tokens are valid syntax
⬇️ create an Abstract Syntax Tree (AST)
⬇️ compile the AST into bytecode
OUTPUT: bytecode (in __pycache__/file.pyc or in memory)

The compiler is located in the rustpython-compiler crate. Keep an eye on the blog for a future post about the details or the compiler. In the meantime, check out the parser source code in rustpython-parser.