Adam Johnson - python

Python: introducing profiling-explorer

2026-04-03T00:00:00+01:00

I’ve made another package! Like icu4py, which I made in February, it was sponsored by my client Rippling. And like tprof, which I made in January, it’s a profiling tool!

profiling-explorer is a tool for exploring profiling data from Python’s built-in profilers, which are stored in pstats files. Here’s a screenshot of it in action, displaying profiling data from running a subset of Django’s test suite:

(Click to enlarge.)

The table copies the pstats interface, with functions in the right-hand column and stats about them in columns to the left. However, it has some differences, such as using improved column names and always displaying times in milliseconds, and it comes with these features:

Dark mode (sorry for the blinding screenshot!).
Click the calls, internal ms, or cumulative ms column headers to sort by that column.
Use the search box to filter by filename or function name.
Hover by a filename + line number pair to reveal the copy button, which copies the location to your clipboard for faster opening.
Click the callers or callees links on the right of a row (not pictured above) to see the callers or callees of that function.

For example, if I put ../django/db/migrations/ in the filter box, I can limit the table to just functions in Django’s migration system, which could be useful for optimizing that system:

(Click to enlarge.)

profiling-explorer was motivated by doing lots of optimization work, at Rippling, in Django, and elsewhere. I’ve often used Python’s pstats to inspect profiling data, but I’ve found its command line interface clunky and slow. Visualization tools like gprof2dot can help, but they don’t provide the immediacy of a table interface, where many numbers can be skimmed and compared at a glance.

Python’s built-in profilers

It’s also a great time for profiling in Python, as version 3.15 (expected October 2026) will introduce a new built-in profiler, bringing the total to three. Here’s a quick list explaining their history…

profile - introduced by Guido van Rossum in 1992 (commit), this module provides a simple profiler for Python code, using a tracing approach that records timings for every function call and return. However, because it’s implemented purely in Python, it adds a heavy overhead which slows down profiling and skews results, and as such it’s deprecated for removal in Python 3.17.
profiling.tracing (previously cProfile) - this module was built as a C re-implementation of profile by Armin Rigo in 2006 (commit). Being written in C, it has a much lower overhead than profile, and it has served developers for years. Still, it does make programs run ~50% slower, and since overhead is per-function, it can make oft-called functions look like more of a bottleneck than they really are.
profiling.sampling (“Tachyon”) - new in Python 3.15, thanks to Pablo Galindo Salgado (commit), this profiler uses a sampling approach, which means it periodically interrupts the profiled program to record the current call stack. This is a very low overhead approach, meaning programs can run at nearly full speed, and the skew from per-function overhead is gone. For compatibility, it uses the same pstats file format as the other profilers, allowing profiling-explorer to work with it out of the box. That said, it can also output several visualization formats, like flame graphs and heat maps, making it very versatile.

I find Tachyon a very exciting addition to Python, and I cannot wait to use it more. Previously, we’ve needed to turn to third-party sampling profilers, like py-spy, which is great but presents some difficulties, such as needing sudo permissions and ongoing maintenance to keep up with Python changes. Having a “batteries included” sampling profiler will make it a lot easier to accurately trace programs.

For more information, check out:
- the rewritten profiling documentation
- The Core.py podcast episode where Pablo and Łukasz Langa discuss the design and implementation.

Give it a try

Future hype aside, profiling-explorer is useful whatever (supported) Python version you’re currently using. To get started, first generate a pstats file for your program by running it under profiling.tracing, under its old name cProfile, with:

$ python -m cProfile -o <name>.pstats <program>

<program> can be a Python file, or -m with a module name, and further arguments are passed to the program.

For example, to profile Django’s system check command, you can run:

$ python -m cProfile -o check.pstats manage.py check

Or, to profile a set of tests under pytest, you can run:

$ python -m cProfile -o tests.pstats -m pytest example/tests/test_rocket.py

Once you have a pstats file, pass it to profiling-explorer. If you use uv, this is as easy as:

$ uvx profiling-explorer <name>.pstats

Otherwise, install profiling-explorer and invoke it directly:

$ profiling-explorer <name>.pstats

However you start it, you’ll see:

$ uvx profiling-explorer migrations.profile
profiling-explorer running at http://127.0.0.1:8099/
Press CTRL+C to quit.
Opening in web browser…

…and your browser will open the interface automatically.

Happy exploring, I hope you find some easy-to-fix bottlenecks!

Fin

Thanks again to Rippling for sponsoring this development. Several of us are already using profiling-explorer on their giant Django code base to find useful optimizations. If you’re looking for your next opportunity, and feel smart and ready to go after hard problems on day one, check out Rippling’s open roles.

May you always be improving,

—Adam

Python: introducing icu4py, bindings to the Unicode ICU library

2026-02-09T00:00:00+00:00

I made a new package! Thank you to my client Rippling for inspiring and sponsoring its development.

ICU (International Components for Unicode) is Unicode’s official library for Unicode and Globalization tools. It’s a de-facto standard for handling text in a locale-aware way, used by many major projects, including Chrome, Firefox, macOS, VS Code, and so on. ICU comes in two flavours: ICU4C (for C/C++) and ICU4J (for Java). (There’s also the newer ICU4X, built in Rust, but it’s a bit more of a work-in-progress.)

My new package, icu4py, exposes some ICU4C functionality in Python, translating the C++ API into a more Pythonic one. It currently supports boundary analysis (breaking text into words, sentences, etc.) and message formatting (using the ICU MessageFormat syntax). I’m open to adding more ICU features in the future, depending on demand.

Batteries included

Here’s a summary of the two features included in icu4py right now.

Boundary analysis

For example, ICU supports text boundary analysis: finding linguistic boundaries in text, such as words or sentences, based on per-language rules. These are useful for things like accurate word counts, or wrapping text for display. icu4py exposes this feature through “breaker” classes. For example, to split text into sentences using SentenceBreaker:

>>> from icu4py.breakers import SentenceBreaker
>>> text = 'You asked "Why?". We answered "Why not?"'
>>> breaker = SentenceBreaker(text, "en_GB")
>>> list(breaker)
['You asked "Why?". ', 'We answered "Why not?"']

Note the quoted sentences are kept within their outer ones.

Brilliant translation with `MessageFormat`

ICU also provides a flexible translation tool called MessageFormat. This allows translators to write patterns that capture the nuances of different languages, like varying pluralization rules. icu4py exposes this API through its MessageFormat class:

>>> from icu4py.messageformat import MessageFormat
>>> pattern = "{count,plural,one {# file} other {# files}}"
>>> fmt = MessageFormat(pattern, "en_GB")
>>> fmt.format({"count": 0})
'0 files'
>>> fmt.format({"count": 1})
'1 file'
>>> fmt.format({"count": 5})
'5 files'

In this case, our pluralization varies the output based on whether the count variable is one or any other number.

French has a subtle but important difference—it treats zero as singular, unlike English:

>>> pattern = "{count,plural,one {# fichier} other {# fichiers}}"
>>> fmt = MessageFormat(pattern, "fr_FR")
>>> fmt.format({"count": 0})
'0 fichier'
>>> fmt.format({"count": 1})
'1 fichier'
>>> fmt.format({"count": 2})
'2 fichiers'

This example shows an important point: the meaning of the one category is defined by the locale. In French, the “one” category matches both 0 and 1 because they both use the singular form.

Korean has no plural distinction at all—the same form is used regardless of count:

>>> pattern = "{count,plural,other {# 파일}}"
>>> fmt = MessageFormat(pattern, "ko_KR")
>>> fmt.format({"count": 0})
'0 파일'
>>> fmt.format({"count": 1})
'1 파일'
>>> fmt.format({"count": 5})
'5 파일'

ICU handles all these variations automatically based on the locale, so you don't need to know the rules for each language.

MessageFormat version 2 is in the works, complete with its own site. It’s in technical preview in ICU4C, and I aim to expose it in icu4py soon, tracked in Issue #17.

Backstory

Rippling is a global HR, Payroll, IT, and Finance platform, with over 20,000 customers all over the world. They work hard to make their product available in many languages and locales. Their Data Globalization team found that ICU Messageformat would present a great next step to handle complex localized messages in their product. Since they use Python (and Django), they needed to pick a Python library to integrate ICU.

This is where the challenge started. In a Slack thread, we surveyed the existing options, and found them all lacking, especially on two points:

None provided compiled wheels, a strong necessity for smooth installation among Rippling’s 1,000+ engineers.
They all depend on a system-based install of ICU, which could present different versions between environments.

In particular, we found these options:

pyicu - impressively maintained since 2007, but poorly documented, has a clunky C++-style API, and lacks some key Python modernization. For example, its C++ extensions don’t use multi-phase initialization, per PEP 489, which is required to support sub-interpreters, as Rippling may wish to adopt in the future.

Additionally, while working on this project, its self-hosted Gitlab went down, and it hasn’t been restored for some weeks. This does not bode well for long-term maintenance!
icupy - newer, modern, and better-maintained, but it still copies the C++ API directly, which is quite unwieldy.
pyseeyou - a pure-Python implementation of MessageFormat. While this library is appealing in terms of ease of installation, it performs all parsing and formatting in Python, making it fairly slow. Additionally, the package does not provide any locale-specific formatting.

After this survey, I opted to try my hand at creating a proof-of-concept package for calling the ICU4C’s C++ class MessageFormat from Python. Now, C++ is a beautifully haunted language, and I’m correspondingly uncomfortable with it. While I have developed some Python C extensions before, like time-machine, until this point I had not touched C++ since my university days.

Thankfully, we are living in the LLM era and this kind of “glue two things together” task is something that they excel at. Within a few hours, my mate Claude and I had a working prototype that could handle basic formatting:

from icupoc import MessageFormat

# English plurals
pattern = "{count, plural, one {# file} other {# files}}"
fmt = MessageFormat(pattern, "en_US")

print(fmt.format({"count": 1}))  # "1 file"
print(fmt.format({"count": 5}))  # "5 files"

(You can still see this prototype at icupoc.)

Happy with the PoC, Rippling gave me the go-ahead to build out a full ICU package.

Building

I continued to use Claude when building out icu4py proper, and I guess I was “agentic engineering” (or “vibe engineering”). While I used the LLM to generate code, I reviewed every line, making many edits before any step was ready. I also deployed my usual stack of tools to format code, check types, build robust documentation, etc.

Using an LLM essentially made this project feasible. A few years ago, I would have shied away from writing in C++ and it would probably have taken me a lot longer to get running. But instead of finding C++ big/scary, having many of the small details taken care of made the project fun!

Things weren’t all smooth sailing, though. Rippling have several “blessed” languages, including Rust, for which they have a monorepo with lots of tooling set up. Before deciding to build the open source package, I tried to build an internal Rust-based version. I made several different attempts to build Rust-to-C++ bindings, spending a silly amount of tokens and time going down rabbit holes to determine that various approaches don’t work or aren’t worth it. The LLM’s confidence and sycophany led me to trying ideas for a long time before cutting my losses. If I had known more about C++ or Rust, I think I wouldn’t have tried using Rust to begin with!

I also learned that LLMs are really poor at using all of their embedded knowledge at once. Even when writing the proof-of-concept, I prompted Claude to improve its own code and it came up with many changes (commit). That said, re-applying this trick more times made it adopt advanced C++ abstractions, completely unnecessarily. It seems that to get high quality code, you need to loop an LLM around a few times on the same code while still using your judgement to know when to stop.

Documentation was critical for getting good results. I often copied sections or whole pages from Python or ICU’s documentation straight into the LLM chat. For ICU’s auto-generated class documentation, I used Jina Reader to convert its dense HTML to Markdown. Docs are king, and even more so now.

That said, using LLM-generated code did free up my time to write and edit the documentation more than I usually would. I made sure to include sensible links to the relevant parts of ICU’s documentation, as well as great examples.

icu4c-builds

A large part of building icu4py was compiling ICU4C itself. It quickly became clear that I couldn’t build icu4py wheels that depended on the system ICU library because binding to C++ classes within a dynamic library is very tricky. Therefore, I built a second repository, icu4c-builds, which builds ICU4C in the exact setup I need for icu4py.

Getting icu4c-builds working across Linux, macOS, and Windows and multiple architectures was a challenge with many long-running failed builds. Claude was very useful here, though, since it has a lot of embedded knowledge around small details like compiler flags. At some points, I was just shovelling logs between GitHub Actions and Claude, wondering why I was even there.

Anyway, now the build system seems to be in a good place, and since ICU is pretty stable, the process is unlikely to break in the future.

Future work

Right now, icu4py only has bindings to two ICU modules: boundary analysis and message formatting. There are 40+ more to bind to, providing all kinds of features likely to be useful in globalizing Python apps. If there are modules that you’d like available, please open an issue!

I’m pretty confident that with a solid foundation to copy from, LLMs can churn through building further bindings pretty quickly. That said, I still think it’s key for icu4py’s success to design an API that makes sense in Python, for which I think human taste still prevails.

On the Django side, I think there could be a place for writing an integration for icu4py with Django’s existing internationalization and localization framework. While there’s a lot of overlap, ICU has a lot of extra features that could be useful.

Fin

Please check out icu4py for your text boundary analysis and translation needs!

Thanks again to Rippling for paying for this work. ⭐ Working with them is a fun challenge—even small changes can have a large impact, thanks to their massive scale. If you’re looking for your next opportunity, and feel smart and ready to go after hard problems on day one, check out Rippling’s open roles.

Peek-a-boo, I see you!

—Adam

Python: introducing tprof, a targeting profiler

2026-01-14T00:00:00+00:00

Profilers measure the performance of a whole program to identify where most of the time is spent. But once you’ve found a target function, re-profiling the whole program to see if your changes helped can be slow and cumbersome. The profiler introduces overhead to execution and you have to pick out the stats for the one function you care about from the report. I have often gone through this loop while optimizing client or open source projects, such as when I optimized Django’s system checks framework (previous post).

The pain here inspired me to create tprof, a targeting profiler for Python 3.12+ that only measures the time spent in specified target functions. Use it to measure your program before and after an optimization to see if it made any difference, with a quick report on the command line.

For example, say you’ve realized that creating pathlib.Path objects is the bottleneck for your code. You could run tprof like so:

Benchmark with comparison mode

Sometimes when optimizing code, you want to compare several functions, such as “before” and “after” versions of a function you’re optimizing. tprof supports this with its comparison mode, which adds a “delta” column to the report showing how much faster or slower each function is compared to a baseline.

For example, given this code:

def before():
    total = 0
    for i in range(100_000):
        total += i
    return total


def after():
    return sum(range(100_000))


for _ in range(100):
    before()
    after()

…you can run tprof like this to compare the two functions:

$ tprof -x -t before -t after -m example
🎯 tprof results:
 function         calls total  mean ± σ      min … max   delta
 example:before()   100 227ms   2ms ± 34μs   2ms … 2ms   -
 example:after()    100  86ms 856μs ± 15μs 835μs … 910μs -62.27%

The output shows that after() is about 60% faster than before(), in this case.

Python API

tprof also provides a Python API via a context manager / decorator, tprof(). Use it to profile functions within a specific block of code.

For example, to recreate the previous benchmarking example within a self-contained Python file:

from tprof import tprof


def before():
    total = 0
    for i in range(100_000):
        total += i
    return total


def after():
    return sum(range(100_000))


with tprof(before, after, compare=True):
    for _ in range(100):
        before()
        after()

…which produces output like:

$ python example.py
🎯 tprof results:
 function          calls total  mean ± σ      min … max delta
 __main__:before()   100 227ms   2ms ± 83μs   2ms … 3ms -
 __main__:after()    100  85ms 853μs ± 22μs 835μs … 1ms -62.35%

How it works

tprof uses Python’s sys.monitoring, a new API introduced in Python 3.12 for triggering events when functions or lines of code execute. sys.monitoring allows tprof to register callbacks for only specific target functions, meaning it adds no overhead to the rest of the program. Timing is done in C to further reduce overhead.

Thanks to Mark Shannon for contributing sys.monitoring to CPython! This is the second time I’ve used it—the first time was for tracking down an unexpected mutation (see previous post).

Fin

If tprof sounds useful to you, please give it a try and let me know what you think! Install tprof from PyPI with your favourite package manager.

May you hit your Q1 targets,

—Adam

Python: fix SyntaxWarning: 'return' in a 'finally' block

2025-08-29T18:13:00+01:00

Take this code:

import random


def d6() -> int:
    try:
        return random.randint(1, 6)
    except Exception:
        # Ignore issues in random number generation
        pass
    finally:
        # Fallback value chosen by a fair dice roll
        return 4


print("Your random number is:", d6())

Run it on Python 3.14+, and you’ll see:

$ python3.14 example.py
/.../example.py:12: SyntaxWarning: 'return' in a 'finally' block
  return 4
Your random number is: 4

Python reports a SyntaxWarning that indicates the return within the finally is dubious, likely to be an error, and may be removed from Python in the future.

And indeed, in this case, it is an error. d6() always returns its fallback value, 4, even though random number generation works. This is because a return in a finally block overrides the earlier return in the try block.

This confusion is why the SyntaxWarning was introduced, through PEP 765. The PEP introduced the warning for three statements within finally blocks:

return, as above.

break, as in:

for i in range(5):
    try:
        ...
    finally:
        break

…which warns like:

$ python3.14 example.py
/.../example.py:5: SyntaxWarning: 'break' in a 'finally' block
  break

continue, as in:

for i in range(5):
    try:
        ...
    finally:
        continue

…which warns like:

$ python3.14 example.py
/.../example.py:5: SyntaxWarning: 'continue' in a 'finally' block
  continue

To fix the warning, rewrite your code to avoid using return, break, or continue within a finally block. This may mean removing the statement, or the finally block entirely, or restructuring your code to avoid the need for it. For example, in the d6() function above, we could move the return 4 within the except block:

import random


def d6() -> int:
    try:
        return random.randint(1, 6)
    except Exception:
        # Ignore issues in random number generation
        # Fallback value chosen by a fair dice roll
        return 4

Even better, we could remove the try/except entirely, since random.randint() is unlikely to raise an exception in normal use:

import random


def d6() -> int:
    return random.randint(1, 6)

Fin

We are finally free of this confusing pattern!

—Adam

Python: capture stdout and stderr in unittest

2025-08-29T16:49:00+01:00

When testing code that outputs to the terminal through either standard out (stdout) or standard error (stderr), you might want to capture that output and make assertions on it. To do so, use contextlib.redirect_stdout() and contextlib.redirect_stderr() to redirect the respective output streams to in-memory buffers that you can then inspect and assert on.

For example, say we wanted to test this function:

def print_silly(string: str) -> None:
    # Alternate the case of each letter
    sillified = "".join(
        c.upper() if i % 2 == 0 else c.lower() for i, c in enumerate(string)
    )
    print(sillified)

We could write a test using redirect_stdout() like so:

from contextlib import redirect_stdout
from io import StringIO
from unittest import TestCase

from example import print_silly


class PrintSillyTests(TestCase):
    def test_print_silly(self):
        with redirect_stdout(StringIO()) as buffer:
            print_silly("What a lovely day!")

        self.assertEqual(buffer.getvalue(), "WhAt a lOvElY DaY!\n")

redirect_stdout() takes a file-like object to which it will redirect output, and then returns that object. That allows us to use a StringIO() as an in-memory buffer to capture the output, which we can call getvalue() on to retrieve the captured text.

redirect_stderr() works the same way, but for stderr.

Simplify capturing streams

To simplify capturing both stdout and stderr, you can copy in this extra decorator/context manager made with contextlib.contextmanager():

from contextlib import contextmanager, redirect_stderr, redirect_stdout
from io import StringIO
from typing import Generator, TextIO


@contextmanager
def capture_output() -> Generator[tuple[TextIO, TextIO]]:
    """
    Capture both stdout and stderr into StringIO buffers.

    Source: https://adamj.eu/tech/2025/08/29/python-unittest-capture-stdout-stderr/
    """
    with (
        redirect_stdout(StringIO()) as out,
        redirect_stderr(StringIO()) as err,
    ):
        yield out, err

Use it like so (here it has been placed in a testing module):

from unittest import TestCase

from example import print_silly
from testing import capture_output


class PrintSillyTests(TestCase):
    def test_print_silly(self):
        with capture_output() as (out, err):
            print_silly("What a lovely day!")

        self.assertEqual(out.getvalue(), "WhAt a lOvElY DaY!\n")
        self.assertEqual(err.getvalue(), "")

I like this pattern as it’s clear without being too long.

Assert on log messages

If the output you’re checking comes through the standard library’s logging module, rather than directly through print(), it’s better to use TestCase.assertLogs() context manager instead of redirecting stdout or stderr. For example, if we had this function that logged a message:

import logging

logger = logging.getLogger(__name__)


def log_silly(string: str) -> None:
    sillified = "".join(
        c.upper() if i % 2 == 0 else c.lower() for i, c in enumerate(string)
    )
    logger.info(sillified)

We could test it like so:

from unittest import TestCase

from example import log_silly


class LogSillyTests(TestCase):
    def test_log_silly(self):
        with self.assertLogs(level="INFO") as cm:
            log_silly("Max. My Name's Max. That's My Name.")
        self.assertEqual(
            cm.output,
            ["INFO:example:MaX. mY NaMe's mAx. ThAt's mY NaMe."],
        )

In pytest

If you use pytest to run your tests, the above pattern will work, but you may also wish to use its built-in capsys fixture. Within a unittest class, you can use it with a custom fixture in your test class that attaches the capsys fixture to the TestCase instance:

from unittest import TestCase

import pytest

from example import print_silly


class PrintSillyTests(TestCase):
    @pytest.fixture(autouse=True)
    def pytest_setup(self, capsys):
        self.capsys = capsys

    def test_print_silly(self):
        print_silly("What a lovely day!")

        captured = self.capsys.readouterr()
        assert captured.out == "WhAt a lOvElY DaY!\n"
        assert captured.err == ""

Otherwise, within regular pytest tests, in functions or plain classes, you can use the capsys fixture directly as a function argument:

from example import print_silly


def test_print_silly(capsys):
    print_silly("What a lovely day!")

    captured = capsys.readouterr()
    assert captured.out == "WhAt a lOvElY DaY!\n"
    assert captured.err == ""

Avoid the tools in `test.support`

If you search Python’s documentation or other sources, you may find test.support.captured_stdout() and the similar captured_stderr(). These work like wrappers around redirect_stdout() and redirect_stderr(), creating the StringIO buffers for you:

from test.support import captured_stdout

with captured_stdout() as stdout:
    print("Hello, World!")
assert stdout.getvalue() == "Hello, World!\n"

However, you should avoid using these tools in your own code. The test.support module is an internal part of Python’s own test suite, with no guarantees of stability or backwards compatibility. Additionally, it may not be available in all Python distributions or implementations—for example, it’s not available in the Python distributions from uv.

Fin

May you capture all the output that you expect,

—Adam

Python: check a package version with importlib.metadata.version()

2025-07-30T18:33:00+01:00

Sometimes, it’s useful to branch the behaviour of your code based on the version of a package that you have installed. This may be to support an upgrade in your project, or for your own package to support different versions of a dependency.

Some Python packages provide a top-level version attribute that gives the version number as a tuple. This is often called __version__, though in the case of Django, it is called django.VERSION, which can be used like this:

import django

if django.VERSION >= (5, 2):
    # Use new features in Django 5.2
    ...
else:
    # Do some backport
    ...

Such version attributes are not a standard though, so not all packages have them. They require extra work from package maintainers to provide: for example, setuptools requires extra dynamic metadata configuration. And in the worst cases, a __version__ attribute may have been forgotten about by new package maintainers, leaving it out of sync with the actual package version.

Instead of using such attributes, you can instead use importlib.metadata.version(), added in Python 3.8, to get a package version as a string:

In [1]: from importlib.metadata import version

In [2]: version('django')
Out[2]: '5.2.4'

This is available for all installed packages as it reads the metadata files left by the package installer.

For example, the previous example can be rewritten like this:

from importlib.metadata import version

if version("django").startswith("5.2."):
    # Use new features in Django 5.2
    ...
else:
    # Do some backport
    ...

It can be a bit annoying to work with strings, though. In this case, you can install packaging and use its packaging.version.Version class to parse the string into a Version object:

In [1]: from importlib.metadata import version

In [2]: from packaging.version import Version

In [3]: Version(version('django'))
Out[3]: <Version('5.2.4')>

Version objects can be compared with each other, so you can use them like this:

from importlib.metadata import version
from packaging.version import Version

if Version(version("django")) >= Version("5.2"):
    # Use new features in Django 5.2
    ...
else:
    # Do some backport
    ...

Fin

May you never need package-version-branching, and when you do, may it be short-lived.

—Adam

Python: fix BrokenPipeError when piping output to other commands

2025-07-20T00:00:00+01:00

If you’ve written a Python script that outputs a lot of data, then piped that output into another command that only reads part of it, you might have encountered a BrokenPipeError. For example, take this script:

for i in range(10_000):
    print(f"This is line {i} ")

When piped into head -n3, which only reads the first 10 lines, it raises a BrokenPipeError:

$ python example.py | head -n3
This is line 0
This is line 1
This is line 2
Traceback (most recent call last):
  File "/.../example.py", line 2, in <module>
    print(f"This is line {i} ")
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored on flushing sys.stdout:
BrokenPipeError: [Errno 32] Broken pipe

This happens because head stops reading its input after the first 10 lines and closes its input stream, which is the pipe from Python. The operating system sends a SIGPIPE signal to the Python process, indicating that it can no longer write to the pipe because it’s closed. Python translates this signal into a BrokenPipeError when it tries to write the next line to the closed pipe.

The exception occurs a second time as well, when Python shuts down, as it tries to flush the output stream. That’s reported in the final two lines, where the exception recurred but is ignored because it occurs during deletion of the sys.stdout object at shutdown.

To fix these BrokenPipeErrors, the Python documentation has a recommendation. Applying this to our example, we get:

import os
import sys

try:
    for i in range(10_000):
        print(f"This is line {i} ")

    sys.stdout.flush()
except BrokenPipeError:
    # Python flushes standard streams on exit; redirect remaining output
    # to devnull to avoid another BrokenPipeError at shutdown
    devnull = os.open(os.devnull, os.O_WRONLY)
    os.dup2(devnull, sys.stdout.fileno())

Here’s what’s new:

The output loop is wrapped in a try / except PipeError block.
The try branch now flushes sys.stdout, to trigger any BrokenPipeErrors that might occur during the loop.
The except branch handles the BrokenPipeError by redirecting the remaining output to /dev/null. It does this by opening the os.devnull special file (/dev/null on Unix-like systems, or NUL on Windows) for writing, and then using the arcane system call os.dup2() to redirect the standard output stream to this file descriptor.

Python’s documentation also suggests sys.exit(1) in the except block, but I didn’t include that because I think it’s unnecessary. Exiting with a non-zero exit code indicates an error, but there’s not really an error when stopping early. Many programs use a zero exit code when finishing early (cat, grep, rg) while others use exit code 141 (like yes or git log) It doesn’t seem common to use exit code 1, though.

After these changes, the example exits cleanly:

$ python example.py | head -n3
This is line 0
This is line 1
This is line 2

…and we can check the exit code using the pipestatus variable in Bash/Zsh:

$ echo $pipestatus 0 0

This variable lists the exit codes of all commands in the last one, and here they’re both 0.

As a context manager

It’s a bit fiddly to wrap the output section of your script in such a try / except block. Here‘s a version that bundles the logic into a context manager:

import os
import sys
from contextlib import contextmanager
from typing import Generator


@contextmanager
def handle_broken_pipe() -> Generator[None]:
    """
    Prevent BrokenPipeError when the output stream is closed early, such as
    when piping to head.

    https://adamj.eu/tech/2025/07/20/python-fix-brokenpipeerror/
    """
    try:
        yield
        sys.stdout.flush()
    except BrokenPipeError:
        # Python flushes standard streams on exit; redirect remaining output
        # to devnull to avoid another BrokenPipeError at shutdown
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())

Use it like so:

with handle_broken_pipe():
    for i in range(10_000):
        print(f"This is line {i} ")

I like this approach because it separates the error handling into a neat package. Also, if you put the context manager on your outer main() function, you can scatter output statements throughout your code without worrying about BrokenPipeErrors.

What about `stderr`?

The above code only handles stdout being closed. It’s also possible, though less common, to pipe stderr to a process that closes it early. For example, we can modify our example to write to stderr:

import sys

for i in range(10_000):
    print(f"This is line {i} ", file=sys.stderr)

With some Zsh redirection, we can pipe stderr to head like so:

$ python example.py 2> >(head -n3)
This is line 0
This is line 1
This is line 2

This time, there’s no exception displayed because Python would print it to stderr, but that stream has been closed early. Still, we can see a failure exit code:

$ echo $? 120

It’s possible to adapt the previous example to handle stderr as well, but I am not sure if it’s a great idea. stderr being closed early is less of a “normal” case, as it prevents normal error reporting from working, so perhaps it’s better to exit with a non-zero status code (at least).

(I’m not sure where the value 120 comes from.)

Fin

May your only broken pipes be virtual!

—Adam

Python: sharing common tests in unittest

2025-05-30T00:00:00+01:00

A neat testing pattern is writing common tests in a base class and then applying them to multiple objects through subclassing. Doing so can help you test smarter and cover more code with less boilerplate.

unittest doesn’t have a built-in way to define a base class of tests that should only be run when subclassed, but there a few ways to achieve it. We’ll explore a couple of approaches here.

Example code

In all the examples below, we’ll be testing these two classes which have a common interface:

class Armadillo:
    def speak(self) -> str:
        return "Hrrr!"


class Okapi:
    def speak(self) -> str:
        return "Gronk!"

With a deleted base class

This approach creates a base class, uses it, and then hides the base class with del:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests(TestCase):
    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests):
    animal_class = Okapi


del BaseAnimalTests  # Hide base class from test discovery

The del is needed to prevent unittest from collecting and running BaseAnimalTests itself. Without it, unittest would run it, and it would fail because it does not define the required animal_class attribute.

This approach is my preferred one. It might be a little surprising for readers to find a del, but I think the comment is explanation enough.

Running the tests, we see just the two concrete test classes executing:

$ python -m unittest -v
test_speak (tests.ArmadilloTests.test_speak) ... ok
test_speak (tests.OkapiTests.test_speak) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

With a test class mixin

This approach puts the common tests in a mixin class that does not inherit from unittest.TestCase:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests:
    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests, TestCase):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests, TestCase):
    animal_class = Okapi

No del is needed here: unittest will not collect BaseAnimalTests because it does not inherit from unittest.TestCase. But this approach has at least a couple of drawbacks:

If any base class forgets to inherit from both BaseAnimalTests and unittest.TestCase, it will not be collected by unittest, and its tests will not run. This may be hard to notice. Even a defence like enforcing 100% coverage on your tests, as Ned Batchelder implores we use, may not help, since subclasses may not contain any tests of their own.
Type checkers will raise errors about undefined methods in the base class, for example:
```
$ mypy --check-untyped-defs tests.py
tests.py:11: error: "BaseAnimalTests" has no attribute "assertIsInstance"  [attr-defined]
tests.py:12: error: "BaseAnimalTests" has no attribute "assertGreater"  [attr-defined]
Found 2 errors in 1 file (checked 1 source file)
```
I previously blogged about writing type-checked mixin classes, with the conclusion being that they should inherit from their intended base class. That means returning to the first approach, where the base class is a subclass of unittest.TestCase.

With pytest: using the `test` attribute

If you use pytest to run your unittest classes, you can use a __test__ attribute to prevent collection of a specific class:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests(TestCase):
    __test__ = False  # Hide from test discovery

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.__test__ = True  # Enable test discovery for subclasses

    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests):
    animal_class = Okapi

__test__ controls whether pytest should collect a class, or not. pytest respects it as a lightly-documented feature, originally copied from the historical nose runner. The approach above hides the base class but exposes the subclasses, through some automatic configuration in __init_subclass__.

Pytest collects and runs only the subclasses, as expected:

$ pytest -v
===== test session starts ======
...
collected 2 items

test_example.py::ArmadilloTests::test_speak PASSED                                                                                                                                     [ 50%]
test_example.py::OkapiTests::test_speak PASSED                                                                                                                                         [100%]

====== 2 passed in 0.00s =======

This approach is a little bit more complex than the previous two, and it only works with pytest. Still it is nice that pytest gives you that control. If you have a lot of common test classes, the technique could be wrapped up into a class decorator.

Fin

I think it would be neat if unittest gained some functionality here, like a TestCase decorator to prevent collection of a specific class but not its subclasses. Until that hypothetical future, though, I think del is the way to go.

May your common tests work towards the common good,

—Adam

Python: a quick cProfile recipe with pstats

2025-05-20T00:00:00+01:00

Python comes with two built-in profilers for measuring the performance of your code: cProfile and profile. They have the same API, but cProfile is a C extension, while profile is implemented in Python. You nearly always want to use cProfile, as it’s faster and doesn’t skew measurements as much.

By default, cProfile’s CLI profiles a command and displays its profile statistics afterwards. But that can be a bit limited, especially for reading large profiles or re-sorting the same data in different ways.

For more flexibility, cProfile can instead save the profile data to a file, which you can then read with the pstats module. This is my preferred way of using it, and this post covers a recipe for doing so, with a worked example.

The recipe

First, profile your script:

$ python -m cProfile -o profile <script> [args]

Replace <script> with the path to your Python file, and [args] with any arguments you want to pass to it. cProfile will run your script under its profiling machinery, saving the results to a file called profile, as specified by the -o option.

Second, view the profile file using pstats:

$ python -m pstats profile <<< $'sort cumtime\nstats 1000' | less

The pstats CLI provides a REPL for interacting with profile files, based on its Stats class. The CLI is oddly undocumented, but its help command lists the available commands.

The above command passes several commands to pstats in a string. The string uses the $ syntax, a Bash feature for C-style strings, allowing \n to represent a newline, passing two commands:

sort cumtime: Sort the output by cumulative time, largest first. This means the time spent in a function and all its callees.
stats 1000: Show the first 1,000 lines of the profile.

The output is passed to less, a common pager, allowing you to scroll through the results. Press q to quit when you’re done!

Profile a module

If you’re running a module instead of a script, add -m like:

$ python -m cProfile -o profile -m <module> [args]

Replace <module> with the name of the module you want to profile, and [args] with any arguments you want to pass to it.

Multiple profiles

If you’re profiling code before and after, consider using different profile file names instead of just profile. For example, for checking the results of some optimization, I often use the names before.profile and after.profile, like:

$ python -m cProfile -o before.profile example.py

$ git switch optimize_all_the_things

$ python -m cProfile -o after.profile example.py

Alternative sort orders

To sort by other metrics, swap cumtime in sort cumtime for one of these values, per the Stats.sort_stats() documentation:

time: internal time—the time spent in the function itself, excluding calls to other functions.

This is useful for finding the slowest functions in your code.
calls: number of calls to the function.

This is useful for finding functions that are called many times and may be candidates for optimization, such as caching.

A Djangoey example

Here’s a worked example showing how to apply this recipe to a Django management command. Say you are testing a database migration locally:

$ ./manage.py migrate example 0002
Operations to perform:
  Target specific migration: 0002_complexito, from example
Running migrations:
  Applying example.0002_complexito... OK

While it did pass, it was unexpectedly slow. To profile it, you would first reverse the migration to reset your test database:

$ ./manage.py migrate example 0001
...

Then you could apply the recipe to profile the migration.

First, stick the cProfile command in front of the migration command:

$ python -m cProfile -o profile ./manage.py migrate example 0002
Operations to perform:
  Target specific migration: 0002_complexito, from example
Running migrations:
  Applying example.0002_complexito... OK

Then, run the second pstats command to view the results:

$ python -m pstats profile <<< $'sort cumtime\nstats 1000' | less

This opens less with a long table, starting:

Welcome to the profile statistics browser.
profile% profile% Mon May 19 23:52:37 2025    profile

         213287 function calls (206021 primitive calls) in 1.150 seconds

   Ordered by: cumulative time
   List reduced from 3576 to 1000 due to restriction <1000>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   425/1    0.001    0.000    1.150    1.150 {built-in method builtins.exec}
       1    0.000    0.000    1.150    1.150 ./manage.py:1(<module>)
       1    0.000    0.000    1.150    1.150 ./manage.py:7(main)
       1    0.000    0.000    1.109    1.109 /.../django/core/management/__init__.py:439(execute_from_command_line)
   ...

The header tells us how many function calls were made, how many were primitive calls, and how long the code took to run. Then there’s the table of all function calls, limited to 1,000 entries.

Since we’re sorting by cumtime, cumulative time spent in each function, the first line shows the total time spent in all functions. That exec is cProfile running your code, and the later lines represent the top-level wrappers from Django.

Generally, it’s best to find the first listed function within your code base. In this profile, you would search for ``example/ and find this entry:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
    1    0.000    0.000    1.005    1.005 /.../example/migrations/0002_complexito.py:4(forward)
...

One call to the forward() function in the migration file took 1.005 seconds, nearly all of the 1.150 seconds total runtime. That’s a bit suspicious!

Right above that entry, you might also spot the time spent running queries:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
   13    0.000    0.000    1.007    0.077 /.../django/db/backends/utils.py:78(execute)
   13    0.000    0.000    1.007    0.077 /.../django/db/backends/utils.py:88(_execute_with_wrappers)
   13    0.000    0.000    1.007    0.077 /.../django/db/backends/utils.py:94(_execute)
   13    0.000    0.000    1.007    0.077 /.../django/db/backends/sqlite3/base.py:354(execute)
   13    1.006    0.077    1.006    0.077 {function SQLiteCursorWrapper.execute at 0x1054f7f60}
...

This stack of functions all show 13 calls, with a cumulative time of 1.007 or 1.006 seconds. They represent Django’s database backend wrappers, which eventually pass the query to Python’s SQLiteCursorWrapper.execute(), which is displayed differently because it’s implemented in C.

So, we can tell that the migration ran 13 queries in total, and at least one of them was slow and ran by forward(). At this point, you might look at the source of forward() to see if you can find the slow query. But first, you might want to re-display the profile to show only the forward() function and its callees (the functions it called), which might shed some light on what it was doing.

To show only forward() and its callees, you can use the pstats callees command. This takes a regular expression to match the function names you want to show:

$ python -m pstats profile <<< $'sort cumtime\ncallees \\bforward\\b' | less
Welcome to the profile statistics browser.
profile% profile%    Ordered by: cumulative time
   List reduced from 3576 to 1 due to restriction <'\\bforward\\b'>

Function
called...
ncalls  tottime  cumtime
/.../example/migrations/0002_complexito.py:4(forward)
  ->       1    0.000    0.000  /.../django/db/backends/utils.py:41(__enter__)
1    0.000    0.000  /.../django/db/backends/utils.py:44(__exit__)
1    0.000    1.005  /.../django/db/backends/utils.py:78(execute)
1    0.000    0.000  /.../django/utils/asyncio.py:15(inner)
1    0.000    0.000  {method 'create_function' of 'sqlite3.Connection' objects}

profile%
Goodbye.

(Output wrapped.)

This has revealed:

forward() only calls execute() once, so there’s only one slow query.
There’s also a call to SQLite’s create_function(). It’s fast, rounding down to 0.000 seconds, but perhaps may be something to do with the slow query.

Okay, time to look at the source:

def forward(apps, schema_editor):
    import time

    schema_editor.connection.connection.create_function(
        "sleep",
        1,
        time.sleep,
    )
    with schema_editor.connection.cursor() as cursor:
        cursor.execute("SELECT sleep(1)")

Ah, it’s a deliberate pause that I added to show you this example. Well, that solves that mystery.

Fin

May you cook up some great profiles with this recipe!

—Adam

pre-commit: install with uv

2025-05-07T00:00:00+01:00

pre-commit is my favourite Git-integrated “run things on commit” tool. It acts as a kind of package manager, installing tools as necessary from their Git repositories. This makes it fairly easy to set up: all you need to install is pre-commit itself, and it takes things from there.

That said, pre-commit’s install guide is not the friendliest, particularly to developers who don’t use Python. The guide only covers installation with Python’s default installer tool, Pip, and the rather unconventional zipapp alternative. Both of these methods are a bit annoying as they require a working Python installation and virtual environment, entailing manual upgrades of those tools too.

Enter uv (pronounced “you-vee”), a new Python environment manager. Since its release last year, it has made the Pythonsphere go wild, seeing massive adoption. That’s because it simplifies a lot of workflows, including managing development tools like pre-commit. Once you have uv, it can manage Python versions and virtual environments for you, swiftly and smoothly.

I now recommend you install pre-commit using uv’s tool mechanism, using this command:

$ uv tool install pre-commit --with pre-commit-uv

Running it, you’ll see output describing the installation process:

$ uv tool install pre-commit --with pre-commit-uv
Resolved 11 packages in 1ms
Installed 11 packages in 8ms
...
Installed 1 executable: pre-commit

This will put the pre-commit executable in ~/.local/bin or similar (per the documentation). You should then be able to run it from anywhere:

$ pre-commit --version
pre-commit 4.2.0 (pre-commit-uv=4.1.4, uv=0.7.2)

The install command also adds pre-commit-uv, a plugin that patches pre-commit to use uv to install Python-based tools. This drastically speeds up using Python-based hooks, a common use case. (Unfortunately, it seems pre-commit itself won’t be adding uv support.)

With pre-commit installed globally, you can now install its Git hook in relevant repositories per usual:

$ cd myrepo

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

$ pre-commit run --all-files
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Using pre-commit with uv 0.7.2 via pre-commit-uv 4.1.4
check for added large files..............................................Passed
check for merge conflicts................................................Passed
trim trailing whitespace.................................................Passed

Upgrade pre-commit

To upgrade pre-commit installed this way, run:

$ uv tool upgrade pre-commit

For example:

$ uv tool upgrade pre-commit
Updated pre-commit v4.1.0 -> v4.2.0
 - pre-commit==4.1.0
 + pre-commit==4.2.0
Installed 1 executable: pre-commit

This command upgrades pre-commit and all of its dependencies, in its managed environment. For more information, see the uv tool upgrade documentation.

Fin

May you commit early and often,

—Adam

Adam Johnson - python

Python: introducing profiling-explorer

Python’s built-in profilers

Give it a try

Fin

Python: introducing icu4py, bindings to the Unicode ICU library

Batteries included

Boundary analysis

Brilliant translation with MessageFormat

Backstory

Building

icu4c-builds

Future work

Fin

Python: introducing tprof, a targeting profiler

Benchmark with comparison mode

Python API

How it works

Fin

Python: fix SyntaxWarning: 'return' in a 'finally' block

Fin

Python: capture stdout and stderr in unittest

Simplify capturing streams

Assert on log messages

In pytest

Avoid the tools in test.support

Fin

Python: check a package version with importlib.metadata.version()

Fin

Python: fix BrokenPipeError when piping output to other commands

As a context manager

What about stderr?

Fin

Python: sharing common tests in unittest

Example code

With a deleted base class

With a test class mixin

With pytest: using the __test__ attribute

Fin

Python: a quick cProfile recipe with pstats

The recipe

Profile a module

Multiple profiles

Alternative sort orders

A Djangoey example

Fin

pre-commit: install with uv

Upgrade pre-commit

Fin

Brilliant translation with `MessageFormat`

Avoid the tools in `test.support`

What about `stderr`?

With pytest: using the `test` attribute