Adam Johnson - python

Python: use re.prefixmatch() instead of re.match() from Python 3.15

2026-08-16T00:00:00+01:00

Take this validation function:

import re

TRAIN_NUMBER_RE = re.compile(r"\d{6}")  # six digits


def is_valid_train_number(value: str) -> bool:
    return bool(TRAIN_NUMBER_RE.match(value))

It looks reasonable, and it works for the intended cases:

>>> is_valid_train_number("345071")
True
>>> is_valid_train_number("ABC123")
False

But, woah, it also accepts garbage suffixes:

>>> is_valid_train_number("345071-in-abbey-wood")
True

That is a bug, totally not what the author intended. re.Pattern.match() (and its module-level shortcut, re.match()) only anchors at the start of the string. It happily reports a match as soon as it finds 345071 at the beginning, regardless of what comes after.

If you want to check that the entire string conforms to the pattern, you need fullmatch() instead:

def is_valid_train_number(value: str) -> bool:
    return bool(TRAIN_NUMBER_RE.match(value))

…then you’ll see:

>>> is_valid_train_number("345071-in-abbey-wood")
False
>>> is_valid_train_number("345071")
True

So, there’s a bit of name confusion with match(). In many other regex implementations, like JavaScript, PCRE, and Ruby, the “match” method name means “match this pattern somewhere in the string”. Python has this behaviour as search(), and instead its match() means “match this pattern as a prefix of the string”, which programmers may not even realize until they hit a bug like the above.

Introducing `re.prefixmatch()`

Python 3.15 (expected October 2026) adds re.prefixmatch() and re.Pattern.prefixmatch(), as exact synonyms for re.match() and re.Pattern.match() respectively. The new name describes what the function actually does: match a prefix of the string, not the whole thing, and not “anywhere”. Hopefully, this will clear up the name confusion.

So now you could run:

>>> TRAIN_NUMBER_RE.prefixmatch("345071-in-abbey-wood")
<re.Match object; span=(0, 6), match='345071'>
>>> re.prefixmatch(r"\d{6}", "345071-in-abbey-wood")
<re.Match object; span=(0, 6), match='345071'>

Note that the behaviour is unchanged and the garbage suffix is still accepted. prefixmatch() doesn’t fix the validation bug discussed above—fullmatch() is still the solution. The new name only makes the prefix behaviour obvious at the call site, so you can spot such bugs when reading the code, or authors can make clear they really do intend to match a prefix.

Alongside the new name, Python’s documentation now describes match() as “soft deprecated” in favour of prefixmatch():

Soft deprecated since version 3.15: match() has been soft deprecated in favor of the alternate prefixmatch() name of this API which is more explicitly descriptive. Use it to better express intent. The norm in other languages and regular expression implementations is to use the term match to refer to the behavior of what Python has always called search().

…along with a link to a longer discussion on why.

Soft deprecated means that the old name, match(), does not emit a warning, nor are there any plans to add a warning or remove the method. Existing code will continue to work without issue, but you should prefer the new name prefixmatch() going forwards, when your code supports Python 3.15+.

Potentially, linters will grow to enforce the new prefixmatch() name over match(), to make the behaviour obvious, but it’s too early for that at time of writing, as Python 3.15 isn’t even out yet.

Fin

May your method names be precise and your regexes concise,

—Adam

Python: fix SyntaxWarning: invalid octal escape sequence

2026-08-14T00:00:00+01:00

Take this code:

path = "C:\477_data"

If we run this with Python 3.11+ in development mode (to enable deprecation warnings), we will see:

$ python3.11 -X dev example.py
/.../example.py:1: DeprecationWarning: invalid octal escape sequence '\477'
  path = "C:\477_data"

On Python 3.12+, we instead get a SyntaxWarning which happens at compile time, even without development mode. We can see this by compiling the file with py_compile rather than running it:

$ python3.12 -m py_compile example.py
example.py:1: SyntaxWarning: invalid octal escape sequence '\477'
  path = "C:\477_data"

So, what does this warning mean?

Python strings support octal escape sequences: a backslash followed by one to three octal digits (0–7), such as \120, which represents the character with that codepoint. Three octal digits can encode any value from 0 to 0o777 (511 in decimal), but octal escapes were only ever intended to cover single bytes, 0 to 0o377 (255 in decimal), the range used by earlier character sets like Latin-1.

\477 is one such over-large escape: 0o477 is 319 in decimal, above that 0o377 limit. In a str, Python doesn’t clamp or wrap the value, it uses it as a full Unicode codepoint, however large:

>>> ord("\477")
319
>>> "\477"
'Ŀ'

That Ŀ is U+013F LATIN CAPITAL LETTER L WITH MIDDLE DOT, an obscure character. It’s probably not what the developer intended, which is why Python introduced this warning. Curiously, octal escape sequences in bytes literals behave differently, since a byte can only hold values 0–255: Python masks the value down to fit, so b"C:\477_data" becomes b"C:?_data" (319 & 0xFF is 63, the code point for ?), an equally surprising, silent, result.

This warning was added in Python 3.11. From the release notes:

Octal escapes in string and bytes literals with values larger than 0o377 (255 in decimal) now produce a DeprecationWarning. In a future Python version, they will raise a SyntaxWarning and eventually a SyntaxError. (Contributed by Serhiy Storchaka in gh-81548.)

And, as promised, Python 3.12 upgraded it, from the 3.12 release notes:

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError. (Contributed by Victor Stinner in gh-98401.)

This is the same version-by-version escalation that “regular” invalid escape sequences went through, as covered in a previous post, one Python version later. The eventual promotion to SyntaxError hasn’t happened yet, even in Python 3.15 (due October 2026).

To fix this warning, try to understand the author’s intention when writing it. In most cases, the backslash wasn’t intended to start an octal escape sequence, for example when writing out a Windows path or a regular expression, so you can double the backslash to make it a literal backslash:

path = "C:\\477_data"

In rare cases, if a genuine octal escape was meant, you can substitute the rendered character.

Fin

There is no escape,

—Adam

Python: fix TypeError: NamedTuple() got an unexpected keyword argument

2026-08-13T00:00:00+01:00

Take this code, defining a small NamedTuple with a keyword argument per field:

from typing import NamedTuple

Point = NamedTuple("Point", x=int, y=int)

Run it on Python 3.13 or 3.14, and you’ll see:

$ python3.14 example.py
/.../example.py:3: DeprecationWarning: Creating NamedTuple classes using keyword arguments is deprecated and will be disallowed in Python 3.15. Use the class-based or functional syntax instead.
  Point = NamedTuple("Point", x=int, y=int)

And on Python 3.15+, it’s broken:

$ python3.15 example.py
Traceback (most recent call last):
  ...
TypeError: NamedTuple() got an unexpected keyword argument 'x'

What’s up with that?

NamedTuple has always had two documented ways to define fields: the class-based syntax, and a functional syntax taking a list of (name, type) pairs. The keyword-argument style above was never one of them, it worked only as a side effect of the old implementation happening to accept **kwargs. This form was therefore deprecated and removed in Python 3.15, with NamedTuple’s signature now locked to positional-only.

From the release notes:

The undocumented keyword argument syntax for creating NamedTuple classes (for example, Point = NamedTuple("Point", x=int, y=int)) is no longer supported. Use the class-based syntax or the functional syntax instead. (Contributed by Bénédikt Tran in gh-133817.)

The fix

Switch to the class-based syntax, which is the more readable option, and gives you a normal class body to add methods or docstrings to later:

from typing import NamedTuple


class Point(NamedTuple):
    x: int
    y: int

Or, if you’d rather keep a one-liner, use the documented functional syntax, a list of (name, type) pairs, instead of keyword arguments:

from typing import NamedTuple

Point = NamedTuple("Point", [("x", int), ("y", int)])

Auto-fix this problem with Ruff

Ruff’s pyupgrade-derived rule convert-named-tuple-functional-to-class (UP014) rewrites both functional forms, the list-of-tuples version and this keyword-argument version, to class syntax automatically. As of Ruff 0.16, released July 2026, UP014 is one of the hundreds of rules Ruff now enables by default, so there’s nothing to switch on, just run:

$ ruff check --fix example.py
Found 2 errors (2 fixed, 0 remaining).

(If you’re on an older Ruff, add --select UP014, or the broader --select UP, to opt in explicitly.)

…turning our original example straight into:

from typing import NamedTuple


class Point(NamedTuple):
    x: int
    y: int

Fin

May you name your tuples well,

—Adam

Python: introducing emojet, a fast emoji lookup library

2026-08-12T00:00:00+01:00

New package just landed!

emojet is an emoji library for Python: it converts between emoji and their names, in both directions, plus the searching and lookup functions that go with that. It covers the core API of the emoji package, a library that been available for this job since 2014, using the same names and the same data. The difference is that emojet does the work in Rust, running 3.5x faster for conversion, 70 times faster for deconversion, and using about 40% less memory.

Use emojize() to convert names to emoji:

>>> import emojet
>>> emojet.emojize("Python is fun :thumbs_up:")
'Python is fun 👍'

…and demojize() to deconvert emoji to their names:

>>> emojet.demojize("Python is fun 👍")
'Python is fun :thumbs_up:'

Names come in 14 languages, plus the English aliases that GitHub and Slack use:

>>> emojet.emojize("Python ist toll :daumen_hoch:", language="de")
'Python ist toll 👍'
>>> emojet.emojize("Python is fun :thumbsup:", language="alias")
'Python is fun 👍'

See the documentation for the full API.

The motivation

Part of my work for my client Rippling is speeding up their project’s startup time. I use several techniques to achieve this, as covered in my previous post on optimizing startup time. I’ve spent a lot of time finding slow-to-import third-party packages and choosing what to do with them, like deferring their imports, or upgrading to a faster version.

The emoji package stood out from profiling data: it took 22ms to import while providing minimal functionality. 22ms is not the slowest package by far, but it is still noticeable on a trace. Further profiling revealed that most of its import time was spent loading and parsing a 520 KB JSON file containing the emoji characters and names.

Initially, I deferred the emoji imports, like:

-from emoji import emojize

 def show_message(text):
+    from emoji import emojize
     print(emojize(text))

This pushes the 22ms cost out of most processes, at the cost of loading the package later, after initialization is complete. But the idea of a faster rewrite smouldered away at the back of my mind, based on these motivations:

The data rarely changes (on Unicode annual updates), perfect for baking into a binary format.
The API is fairly simple.
String manipulation is often much faster in a lower-level language.

Anyway, I have been trying out Claude more recently, thanks to a free subscription that Anthropic gifted me due to my open source contributions. I decided to fire Claude at the task and whaddaya know, bot did good, and after a bunch of iteration and minor touchups, we have emojet.

Benchmarkation

The emojet repo bundles a benchmark script for comparison with the original emoji package. Here are some of the results comparing emojet 1.0.0 against emoji 2.15.0, on Python 3.14 (macOS, ARM):

Benchmark	emoji	emojet	Speedup
`import`	20.57 ms	869.94 µs	23.7x
`import` + first `demojize()`	24.45 ms	911.13 µs	26.8x
`demojize()`	345.12 µs	4.94 µs	69.9x
`emojize()`	29.23 µs	8.33 µs	3.5x

Memory use is also lower: importing and calling demojize() once uses about 17.4 MB of memory with emojet versus 29.2 MB with the emoji package.

You can check these numbers yourself. The benchmark script uses PEP 723 inline script metadata, so uv will install the latest releases of both packages into a temporary environment for you:

uv run scripts/benchmark.py

How it works

emojet is a Rust extension module, built with PyO3 and maturin, with all of the emoji data compiled into static tables:

A perfect hash function over the 5,316 English names and aliases, so emojize() gets a collision-free, single-probe lookup.
A static trie over the code points of all 5,225 emoji sequences, which drives the scanner behind demojize() and friends.
Per-language name tables, indexed directly, with no loading step.

Nothing is parsed at import time. The tables sit in the read-only data section of the extension module, so the operating system pages them in on demand, and they’re shared between processes. That’s where the import-time win comes from: the remaining ~900 µs is mostly the dynamic loader doing its job.

The scanners work on UTF-8 bytes end to end, writing each result into a single buffer rather than joining a list of pieces. They also skip runs of ASCII text with a byte-per-character test, since almost no emoji starts with an ASCII character—and most text is mostly ASCII. That’s why demojize() on a mixed string sees a bigger speedup than emojize(), which has to hunt for delimiters.

The data itself is generated by scripts/generate_data.py, which follows the emoji package’s own pipeline: Unicode’s emoji-test.txt and emoji-variation-sequences.txt, CLDR annotations for translated names, and GitHub’s gemoji database for aliases. The generated file is checked in, so installing or building emojet downloads nothing. 1.0.0 ships Unicode 17.0.0 data, matching emoji 2.15.0.

Fin

If your project has emoji conversion needs, please give emojet a try and let me know how it goes.

One emojo at a time,

—Adam

Python: tprof 1.3.0: now with less overhead

2026-08-08T00:00:00+01:00

Back in January, I introduced tprof, a targeting profiler for Python 3.12+ that measures the time spent in specific functions, rather than your whole program. As a reminder, here’s the basic usage, specifying a target function with -t and a script to run:

$ tprof -t lib:maths ./example.py
...
🎯 tprof results:
 function    calls total  median ± σ     min … max
 lib:maths()     2 610ms 305ms ± 2ms 304ms … 307ms

Today, it’s my pleasure to announce tprof 1.3.0, a big release with the following changes.

Way less overhead

tprof previously stored each recorded call as a Python int inside a dict of lists, then used the statistics module to crunch the numbers at report time. These data structures were easy to write in an initial version and get the project going, but it added meaningful overhead to every monitored call.

The new release replaces all of that with per-thread C structures: a stack of entry times and a flat array of durations per target, storing raw nanosecond int64_t values instead of boxed Python ints. On Python 3.13+, it also reads timestamps with the new PyTime_PerfCounterRaw() instead of calling time.perf_counter_ns().

Most importantly, sys.monitoring callbacks are now disabled entirely for code that isn’t a target, so non-target functions run at full speed.

The results, benchmarking a trivial function on Python 3.13 (overhead versus an unprofiled call in parentheses):

	Before	After
Target call	469ns (+439)	170ns (+141)
Non-target call	440ns (+410)	28ns (+0)
Memory per 1M calls	34.8 MiB	8.0 MiB
Report per 1M calls	~430ms	~3ms

In short:

Target functions have about a 3× reduction in overhead.
Non-target functions have none at all.
Memory use is 4× lower.
Generating the final report is roughly 100× faster.

Comparing against a saved baseline

tprof’s existing comparison mode (-x/--compare) is great for comparing two functions side-by-side in one run, such as “before” and “after” versions while you’re mid-refactor. But sometimes the “before” version only exists on another Git branch. For that, this release adds --json and --baseline.

Run with --json <path> to write the statistics to a file (or - for stdout) as well as printing the usual report:

$ git checkout main
$ tprof -t lib:maths --json before.json ./example.py

Then, after checking out your change, pass that file to --baseline to get a “delta” column comparing each function’s median against the saved run:

$ git checkout my-optimization
$ tprof -t lib:maths --baseline before.json ./example.py
...
🎯 tprof results:
 function    calls total  median ± σ     min … max     delta
 lib:maths()     2 592ms 296ms ± 2ms  294ms … 297ms -3.11%

Median, not mean

The headline statistic in tprof’s report is now the median rather than the mean, since it’s more robust to the outliers that profiling data tends to contain, like garbage collection pauses and cold caches. Comparison mode deltas, and the new baseline deltas above, are computed from medians too. The median is computed exactly, in C, via quickselect over a scratch buffer at report time, so this change doesn’t add any recording overhead—only a small amount of extra work when the report is generated.

Programmatic access to results

The tprof() context manager / decorator now yields a list of FunctionStats objects, populated once the profiled block ends, so you can use the results in code rather than only reading the printed report:

from lib import maths

from tprof import tprof

with tprof(maths) as results:
    maths()

(function_stats,) = results  # unpack the single result for maths()
print(f"{function_stats.name} took {function_stats.median_ns}ns")

Each FunctionStats has name, calls, total_ns, min_ns, max_ns, median_ns, and stdev_ns, matching the fields in the --json output.

Fin

If you’re already using tprof, this release should make it easier to get correct results. And if you haven’t tried it yet, install it from PyPI and give it a go.

More tea, prof?

—Adam

Python: how time-machine is O(1) where freezegun is O(n)

2026-08-03T00:00:00+01:00

time-machine is my library for mocking the current date and time in Python tests. Its headline advantage over freezegun, the library that inspired it, is speed.

Back in 2021, I benchmarked the two libraries at two project sizes, and found time-machine 100 to 200 times faster. That post asserted that freezegun’s work grows with the number of imported modules, whilst time-machine’s does not.

Five years on, following many time-machine optimizations, including in Friday’s time-machine 3.3.0, let’s re-benchmark the two libraries to empirically measure their runtime complexity and examine their code to explain why.

The benchmark

I wrote a benchmark script to compare the two libraries, listed in full at the end of this section. The variable here is the number of attributes across all imported modules.

To roughly simulate production code, the benchmark generates modules as bare types.ModuleType instances and stuffs them straight into sys.modules. Each module object has 25 attributes, and one in ten of them has module-level references to the date and time functions, as if written with from datetime import date, datetime and from time import time.

import datetime as dt
import time


def make_modules(count):
    modules = {}
    for i in range(count):
        name = f"generated_module_{i}"
        module = types.ModuleType(name)
        for j in range(ATTRIBUTES_PER_MODULE):
            setattr(module, f"attribute_{j}", f"value_{j}")
        if i % 10 == 0:
            # As if: from datetime import date, datetime
            #        from time import time
            module.date = dt.date
            module.datetime = dt.datetime
            module.time = time.time
        modules[name] = module
    return modules

The benchmark then measures the time taken to mock and unmock time with each library, with these test skeletons:

TARGET = dt.datetime(2020, 1, 1)


def freezegun_test():
    with freezegun.freeze_time(TARGET, tick=True):
        pass


def time_machine_test():
    with time_machine.travel(TARGET):
        pass

A sweep then walks the module counts, installing the generated modules, timing both functions with timeit, and cleaning up:

for count in COUNTS:
    generated = make_modules(count)
    sys.modules.update(generated)
    try:
        total = len(sys.modules)
        freezegun_time = measure(freezegun_test)
        time_machine_time = measure(time_machine_test)
    finally:
        for name in generated:
            del sys.modules[name]
    results[count] = (total, freezegun_time, time_machine_time)

measure() uses timeit.Timer.autorange() to pick a loop count, repeats five times, and takes the fastest per-call time. The whole sweep then runs five times over, keeping the fastest time seen at each size. Keeping the fastest times removes noise from our results, as nearly all slowdowns are due to external reasons.

Here’s the whole thing, with inline script metadata so that uv run benchmark.py is all you need to run it:

benchmark.py source

"""Benchmark freezegun against time-machine as imported modules grow.

Generates module objects, installs them in sys.modules, and times a
mock-and-unmock cycle for each library. One in ten generated modules has
module-level references to the date and time functions, as if written with
``from datetime import date, datetime`` and ``from time import time``.
"""

# /// script
# requires-python = ">=3.11"
# dependencies = ["freezegun==1.5.5", "time-machine==3.3.0"]
# ///

from __future__ import annotations

import datetime as dt
import sys
import time
import timeit
import types
from collections.abc import Callable

import freezegun
import time_machine

TARGET = dt.datetime(2020, 1, 1)
COUNTS = [0, 500, 1000, 2000, 4000, 8000, 16000]
ATTRIBUTES_PER_MODULE = 25
ROUNDS = 5


def make_modules(count: int) -> dict[str, types.ModuleType]:
    modules = {}
    for i in range(count):
        name = f"generated_module_{i}"
        module = types.ModuleType(name)
        for j in range(ATTRIBUTES_PER_MODULE):
            setattr(module, f"attribute_{j}", f"value_{j}")
        if i % 10 == 0:
            # As if: from datetime import date, datetime
            #        from time import time
            module.date = dt.date
            module.datetime = dt.datetime
            module.time = time.time
        modules[name] = module
    return modules


def freezegun_test() -> None:
    with freezegun.freeze_time(TARGET, tick=True):
        pass


def time_machine_test() -> None:
    with time_machine.travel(TARGET):
        pass


def measure(func: Callable[[], None], repeat: int = 5) -> float:
    """Seconds per call, taking the fastest of several timed batches."""
    timer = timeit.Timer(func)
    number, _ = timer.autorange()
    return min(t / number for t in timer.repeat(repeat=repeat, number=number))


def sweep() -> dict[int, tuple[int, float, float]]:
    """Time both libraries at each module count, once each."""
    results = {}
    for count in COUNTS:
        generated = make_modules(count)
        sys.modules.update(generated)
        try:
            total = len(sys.modules)
            freezegun_time = measure(freezegun_test)
            time_machine_time = measure(time_machine_test)
        finally:
            for name in generated:
                del sys.modules[name]
        results[count] = (total, freezegun_time, time_machine_time)
    return results


def main() -> None:
    # Sweep several times over, keeping the fastest time seen for each size.
    # A single pass takes a couple of minutes, over which a machine can drift
    # in and out of being busy, which would otherwise bend the results.
    best = sweep()
    for _ in range(ROUNDS - 1):
        for count, (total, freezegun_time, time_machine_time) in sweep().items():
            _, best_freezegun, best_time_machine = best[count]
            best[count] = (
                total,
                min(best_freezegun, freezegun_time),
                min(best_time_machine, time_machine_time),
            )

    print(f"{'generated':>10} {'total':>10} {'freezegun':>13} {'time-machine':>13}")
    for count, (total, freezegun_time, time_machine_time) in best.items():
        print(
            f"{count:>10,} {total:>10,} "
            f"{freezegun_time * 1e6:>10,.1f} µs {time_machine_time * 1e6:>10,.1f} µs"
        )


if __name__ == "__main__":
    main()

Here are the results, recorded on Python 3.15 on my M1 MacBook with freezegun 1.5.5 and time-machine 3.3.0. Every total includes the 259 modules that are there before any are generated: those imported by the benchmark itself, plus the ones that freezegun imports, such as asyncio, the first time it freezes.

Generated	Total modules	freezegun	time-machine
0	259	1,406.7 µs	1.5 µs
500	759	2,484.3 µs	1.5 µs
1,000	1,259	3,736.1 µs	1.5 µs
2,000	2,259	5,757.4 µs	1.5 µs
4,000	4,259	10,428.1 µs	1.5 µs
8,000	8,259	20,429.3 µs	1.5 µs
16,000	16,259	40,971.3 µs	1.5 µs

freezegun’s times climb in a straight line: about 1.4 ms of fixed cost plus 2.5 µs per module. time-machine stays at 1.5 µs whatever you throw at it, with no trend as the modules pile up.

So the ratio is not a fixed “N times faster”—it’s a function of your project’s size:

259 modules, a bare interpreter: 940 times faster.
1,259 modules, about the size of the small Django project from the 2021 post: 2,490 times faster.
16,259 modules, a big project with a lot of dependencies: 27,300 times faster.

Per start/stop cycle, freezegun at 16,000 generated modules costs 41 ms. A suite with 2,000 time-mocking tests spends 82 seconds doing nothing but find-and-replace. The same suite with time-machine spends 3 milliseconds.

Why freezegun is O(n): the sweep

freezegun replaces the date and time functions with fakes, then goes looking for every copy of the originals that other modules made when they imported them. That search is a loop over sys.modules, in freeze_time.start() (source):

for mod_name, module in list(sys.modules.items()):
    if mod_name is None or module is None or mod_name == __name__:
        continue
    elif mod_name.startswith(self.ignore) or mod_name.endswith(".six.moves"):
        continue
    elif not hasattr(module, "__name__") or module.__name__ in ("datetime", "time"):
        continue

    module_attrs = _get_cached_module_attributes(module)
    for attribute_name, attribute_value in module_attrs:
        fake = fakes.get(id(attribute_value))
        if fake:
            setattr(module, attribute_name, fake)
            add_change((module, attribute_name, attribute_value))

Every time you freeze time, every loaded module gets visited. fakes is a dict keyed by id() of the real objects, so a module that imported datetime.datetime has that name rebound to FakeDatetime.

_get_cached_module_attributes() gathers the attributes to check, with a little caching to speed things up: it remembers which of a module’s names held date and time objects last time, and reuses that list if the module’s contents hash the same. Even on a hit, though, computing that hash means listing and hashing every attribute name of every module, every time. Therefore, the work per freeze stays proportional to the number of module-level attributes across all loaded modules.

THus we can say freezegun has O(n) runtime complexity, where n is the number of module-level attributes across all loaded modules. The larger your project, the longer it takes, proportionally.

This slow process is also leaky for (at least) these two reasons:

It doesn’t discover some kinds of references, like class attributes, default arguments, closures, and C extensions that hold references to the original functions. These continue to yield the true time, even while most calls to the same function name are mocked.

The fake objects have observably different types:

>>> import datetime as dt, time, freezegun
>>> dt.datetime.__name__, time.time.__name__
('datetime', 'time')
>>> with freezegun.freeze_time("2020-01-01"):
...     print(dt.datetime.__name__, time.time.__name__)
...
FakeDatetime fake_time

This difference can subtly break code that checks or records types.

Why time-machine is O(1): the swap

time-machine takes a different approach: it swaps what the date and time functions do, a bit like unittest.mock but for built-in functions.

In CPython, a built-in like time.time is a PyCFunctionObject. Its m_ml member points at a PyMethodDef struct, and that struct’s ml_meth member is the C function pointer that actually gets called.

That pointer is writable, so time-machine’s patch() overwrites it, saving the original first (source):

if (state->time_time->m_ml->ml_meth != _time_machine_time) {
    original_time = state->time_time->m_ml->ml_meth;
}
state->time_time->m_ml->ml_meth = _time_machine_time;

That’s the whole technique, repeated for the ten functions that read the clock, such as datetime.now() and time.time_ns(). Ten pointer writes, no matter how big your project is.

Thus we can say time-machine has O(1) runtime complexity: its runtime is constant, no matter how many modules are loaded or how many attributes they have.

This process also fixes the two sources of leakiness in freezegun discussed above:

The swap is done at the C layer, so every reference to the function object, no matter where it lives, sees the new behaviour.

The types of date and time functions are unchanged:

>>> import datetime as dt, time, time_machine
>>> dt.datetime.__name__, time.time.__name__
('datetime', 'time')
>>> with time_machine.travel("2020-01-01"):
...     print(dt.datetime.__name__, time.time.__name__, time.time())
...
datetime time 1577836800.0

Migrate today with time-machine’s migration CLI

If the numbers above look temptign to shave seconds or minutes from your test suite, time-machine makes it easier to migrate with its migration CLI that does some of the work for you. Run it with uvx, pointed at your test files:

$ uvx --from 'time-machine[cli]' python -m time_machine migrate tests/test_delorean.py
Rewriting tests/test_delorean.py

Given this file:

import datetime as dt

from freezegun import freeze_time


@freeze_time("1955-11-05 01:22")
def test_delorean():
    assert dt.date.today().isoformat() == "1955-11-05"


def test_clock_tower():
    with freeze_time("1955-11-12 22:04"):
        assert dt.datetime.now().hour == 22

…the tool rewrites the import, the decorator, and the context manager:

 import datetime as dt

-from freezegun import freeze_time
+import time_machine


-@freeze_time("1955-11-05 01:22")
+@time_machine.travel("1955-11-05 01:22", tick=False)
 def test_delorean():
     assert dt.date.today().isoformat() == "1955-11-05"


 def test_clock_tower():
-    with freeze_time("1955-11-12 22:04"):
+    with time_machine.travel("1955-11-12 22:04", tick=False):
         assert dt.datetime.now().hour == 22

Note tick=False appearing where freezegun didn’t pass tick. freezegun freezes time by default, whilst time-machine lets it tick, so the tool spells out the old behaviour to keep your tests passing. Once migrated, dropping tick=False where you can is worthwhile—time that advances is more realistic, although it means writing assertions against ranges rather than exact values.

The CLI does partial replacements, so it can leave a file in a broken state, such as an unused freezegun import next to a call it couldn’t rewrite. Run it from a clean commit and lean on your linters to find the leftovers. Ruff’s F401 and F821 rules catch most of them.

Fin

O(1) is where the heart is,

—Adam

Python: time-machine 3.3.0 lets your tests time-travel even faster

2026-07-31T00:00:00+01:00

time-machine is my library for mocking the current date and time in Python tests. I’ve just released version 3.3.0 with a decent number of changes, so here’s a lovely summary for you.

Despite doing one simple-to-describe task, time-machine has ended up being quite a big library, thanks to the various complexities of reading time and API changes and optimizations in Python. I’m grateful for all the help I’ve received through open source contributions, it really makes it more manageable.

Python 3.15 support

Python 3.15 is currently in beta, due in October 2026, but time-machine 3.3.0 already supports it (wheels pending the release candidate). I try to get time-machine ready for new Python versions as soon as possible because it is becoming more widely used, and projects like Fedora want to test all the packages they support against the new Python version early.

This release required a bit more work than usual. Historically, time-machine never needed to mock datetime.date.today(), because CPython implemented it by calling cls.fromtimestamp(time.time()), which time-machine already patched. Python 3.15 added a fast path that reads the system clock directly (CPython PR #130980), so time travel stopped affecting it. time-machine now mocks date.today() directly to restore the expected behaviour.

Thanks to Miro Hrončok and Karolina Surma for the report in Issue #610, Lumír ‘Frenzy’ Balhar for the fix in PR #618, and Maurycy Pawłowski-Wieroński for review.

Faster patched functions

When I first wrote time-machine, I knew I needed to write a C extension, but I wanted to avoid writing too much of it because I found it pwetty scawy. This meant that during time travel, the patched-in C functions would actually call into Python to compute the datetime value to return. This behaviour made things easy to implement, but it meant extra overhead on basic calls like time.time().

Now, with the power of LLM assistance and a rigorously built-up test suite, I can bravely read and write more C code, so I was able to convert those functions from Python to C, in PR #643. The results are much more verbose, but faster.

As an example, here’s how the intermediary function calculating a result for datetime.datetime.now() looked in Python:

def now(tz: dt.tzinfo | None = None) -> dt.datetime:
    return dt.datetime.fromtimestamp(time(), tz)

…and in C:

/* datetime.datetime.now() */

static PyObject *
_time_machine_now(
    PyTypeObject *type, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)

{
    PyObject *tz = Py_None;
    Py_ssize_t nkwargs = (kwnames != NULL) ? PyTuple_GET_SIZE(kwnames) : 0;
    for (Py_ssize_t i = 0; i < nkwargs; i++) {
        PyObject *name = PyTuple_GET_ITEM(kwnames, i);
        if (PyUnicode_CompareWithASCIIString(name, "tz") != 0) {
            PyErr_Format(
                PyExc_TypeError, "now() got an unexpected keyword argument '%U'", name);
            return NULL;
        }
        tz = args[nargs + i];
    }
    if (nargs + nkwargs > 1) {
        PyErr_Format(
            PyExc_TypeError, "now() takes at most 1 argument (%zd given)", nargs + nkwargs);
        return NULL;
    }
    if (nargs == 1) {
        tz = args[0];
    }

    // cls.fromtimestamp(traveller_time, tz)
    PyObject *timestamp = _time_machine_traveller_time();
    if (timestamp == NULL) {
        return NULL;
    }
    PyObject *stack[3] = {(PyObject *)type, timestamp, tz};
    PyObject *result = PyObject_VectorcallMethod(
        str_fromtimestamp, stack, 3 | PY_VECTORCALL_ARGUMENTS_OFFSET, NULL);
    Py_DECREF(timestamp);
    return result;
}

Yeah… it takes a lot of extra work in C to even verify the arguments before even calling cls.fromtimestamp().

At least the result is worth it, with a ~30% speedup, as demonstrated with this benchmark using IPython’s %timeit magic:

In [1]: %%timeit
   ...: import time, time_machine
   ...: with time_machine.travel(0):
   ...:     for _ in range(1_000_000):
   ...:         time.time()

Result before:

418 ms ± 4.49 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Result after:

299 ms ± 9.38 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This change also unblocked a couple of bug fixes:

The DeprecationWarning from datetime.utcnow() on Python 3.12+ is no longer hidden while time travelling. time-machine now emits a copy of the warning itself from its patched utcnow() function.
Calling now() or utcnow() on a datetime subclass now returns an instance of that subclass, rather than a plain datetime.datetime instance. This was hard to fix with the Python-level intermediary function, because it did not receive the calling type as a parameter, but the C-level intermediary function does.

Subinterpreter support

Python 3.14 shipped concurrent.interpreters, which allows multiple Python interpreters to run in one process, allowing a new model of parallelization. Back in 2021, for version 2.2.0, I moved time-machine’s C module to multi-phase initialization, which was “future-proofing” for the day CPython supported subinterpreters. However, it was not sufficient, and time-machine would error in an isolated subinterpreter when another interpreter was time-travelling.

The fixes in this version fix those errors, enabling each interpreter to run its own time travel independently of the others. Hopefully this change unblocks experimentation with using subinterpreters to run tests in parallel.

There are probably still some edge cases to fix, but at least the basic functionality is there and tested.

Migration CLI

I built time-machine as an optimized successor to freezegun (it’s way faster). To help you migrate test suites from freezegun to time-machine (a current client task!), time-machine ships with a Migration CLI that rewrites your code as best it can.

The CLI gained some new capabilities in this release.

First, it now handles calls that pass tick, which it previously left alone:

-@freeze_time("2023-01-01", tick=True)
+@time_machine.travel("2023-01-01", tick=True)
 def test_ticking():
     ...

(Where tick isn’t passed, the tool adds tick=False, matching freezegun’s frozen-by-default behaviour.)

Thanks to George-Cristian Birzan for the report in Issue #609 and tanren for the fix in PR #636.

Second, it also rewrites decorators on async functions, another case it used to skip:

-@freeze_time("2023-01-01")
+@time_machine.travel("2023-01-01", tick=False)
 async def test_async():
     ...

Thanks to George-Cristian Birzan for the report in Issue #608 and Sanjay Santhanam for the fix in PR #640.

Third, from-imports that pull in freeze_time alongside other names now get split, keeping those other names importing from freezegun:

-from freezegun import freeze_time, FakeDate
+import time_machine
+from freezegun import FakeDate

Fourth, a bug fix: relative imports to modules that happen to be called freezegun name are now left alone. Previously, a line like this would get rewritten to import time_machine, even though it imports from your a local module:

from .freezegun import freeze_time

This might happen if you have a wrapper module, which probably needs its own consideration for migration, so the CLI should leave it alone.

Wheels

Two changes to what ships on PyPI:

No more wheels for 32-bit Linux and Windows. I realized these platforms are vanishingly rare at this point, and many larger projects have dropped their 32-bit support, so I’m following suit.
No more wheels for free-threaded Python 3.13, since cibuildwheel 4.0.0 dropped support for building them. Free-threaded Python 3.14+ is unaffected.

Wheels are also now built with frame pointers enabled, per the PEP 831 change in Python 3.15. Frame pointers are a debugging feature for low-level languages like C, which allow debuggers to reconstruct the call stack of a program. Python has decided to enable frame pointers by default and encourage the ecosystem to do so as well, to make debugging easier for us all, so time-machine is joining the party.

Fin

Please upgrade today and let me know how it goes!

Are your tests still running?

—Adam

Python: fix ValueError: day of month directive '%d' may not be used without a year directive

2026-07-29T00:00:00+01:00

Take this code, parsing the date of an annually recurring event that only has a month and day, such as a work anniversary:

from datetime import datetime


def parse_anniversary(date_str: str) -> datetime:
    parsed = datetime.strptime(date_str, "%d/%m")
    return parsed.day, parsed.month

With Python 3.13 or 3.14, run it on January 1st, and it will log a warning, but still parse the date:

>>> parse_anniversary("01/01")
/.../example.py:7: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguous
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
  parsed = datetime.strptime(date_str, "%d/%m")
(1, 1)

Note the returned value’s year is 1900.

But, parse February 29th, and the warning is followed by a ValueError:

>>> parse_anniversary("29/02")
/.../example.py:7: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguous
and fails to parse leap day...
Traceback (most recent call last):
...
ValueError: day is out of range for month

That ValueError isn’t new: without a year, strptime() has always filled in 1900 behind the scenes, which isn’t a leap year, so parsing “29 February” has always failed, on every Python version. The new part is the warning, telling us this whole approach is on its way out.

On Python 3.15+, any call to strptime() with a %d in the format string, without a year, will raise a ValueError immediately, no matter the input:

>>> parse_anniversary("01/01")
Traceback (most recent call last):
...
ValueError: Day of month directive '%d' may not be used without a year directive. Parsing dates involving a day of month without a year is ambiguous and fails to parse leap day. Add a year to the input and format. See https://github.com/python/cpython/issues/70647.

From the release notes:

strptime() now raises ValueError when the format string contains %d (day of month) without a year directive. This has been deprecated since Python 3.13. (Contributed by Stan Ulbrych and Gregory P. Smith in gh-70647.)

It seems like a fair trade-off to break this usage completely, rather than let it continue to hide around in code bases and only fail in the rare case that it is handed February 29th. But it does mean any such call sites need fixing…

The fix

Add a year to both the format string and input. If you do have a specific year, use it, otherwise add a leap year (2000 works) and ignore the year in the returned value:

from datetime import datetime


def parse_anniversary(date_str: str) -> datetime:
    date_str = f"{date_str}/2000"  # Add leap year to input
    parsed = datetime.strptime(date_str, "%d/%m/%Y")
    return parsed.day, parsed.month

This will work fine on all Python versions for all dates, including February 29th:

>>> parse_anniversary("01/01")
(1, 1)
>>> parse_anniversary("29/02")
(29, 2)

Fin

May all of your directives be unambiguous,

—Adam

Python: fix TypeError: NotImplemented should not be used in a boolean context

2026-07-28T00:00:00+01:00

Take this class, which implements __eq__() and derives __ne__() from it:

class Lemming:
    def __init__(self, name):
        self.name = name

    def __eq__(self, other):
        if not isinstance(other, Lemming):
            return NotImplemented
        return self.name == other.name

    def __ne__(self, other):
        return not self.__eq__(other)

This technique was common on Python 2, before Python 3.0 started deriving __ne__() automatically from __eq__() (release note).

Now compare a Lemming with something that isn’t one:

>>> p = Lemming("Lenny")
>>> p != "Lenny"
False

Ah, woops, that’s wrong. A Lemming is not equal to the plain string "Lenny", so != should return True.

If we run the same code on Python 3.9–3.13 with warnings enabled, we can see why:

$ python -W default example.py
/.../example.py:12: DeprecationWarning: NotImplemented should not be used in a boolean context
  return not self.__eq__(other)

And on Python 3.14+, it’s no longer just a warning:

>>> p != "Lenny"
Traceback (most recent call last):
  ...
TypeError: NotImplemented should not be used in a boolean context

What’s going on?

other is a string, not a Lemming, so self.__eq__(other) hits the isinstance() check and returns the special singleton NotImplemented, rather than True or False. __ne__() then does not NotImplemented, feeding that singleton into a boolean context.

NotImplemented isn’t False, None, or empty, so like most objects, it’s truthy by default. That means not NotImplemented evaluates to False, and that False becomes the (wrong) result of our != comparison. This “worked” only by accident: NotImplemented is meant to be returned from a comparison method, to tell Python “try something else”, not to be evaluated as a bool itself.

The warning was added in Python 3.9, per the release note:

Using NotImplemented in a boolean context has been deprecated, as it is almost exclusively the result of incorrect rich comparator implementations. It will be made a TypeError in a future version of Python. (Contributed by Josh Rosenberg in bpo-35712.)

Five years later, Python 3.14 delivered on that promise and turned the warning into a TypeError, per the release note:

Using NotImplemented in a boolean context will now raise a TypeError. This has raised a DeprecationWarning since Python 3.9. (Contributed by Jelle Zijlstra in gh-118767.)

So rather than silently giving the wrong answer, our Lemming example now fails loudly, right at the point where the mistake happens.

The fix

The fix for the above Lemming class is to remove the __ne__() method entirely:

-    def __ne__(self, other):
-        return not self.__eq__(other)

Then we’ll see:

>>> p = Lemming("Lenny")
>>> p != "Lenny"
True

Since Python 3, object provides a default __ne__() that calls __eq__() and correctly inverts its result, whilst still propagating NotImplemented onwards so Python can try the other object’s reflected method, or fall back to identity comparison. There is rarely a need to write a __ne__() method by hand anymore, unless you have specific custom behaviour that isn’t the inverse of __eq__().

Another case

This bug isn’t confined to hand-written classes. For years, a “clever” one-liner for dropping None values from a list was:

>>> values = [1, None, 2, None, 3]
>>> list(filter(None.__ne__, values))
[1, 2, 3]

This looks like it filters out None, and on Python 3.8 and earlier, it does, but only because two “wrongs” made a right:

None.__ne__(other) returns NotImplemented for any other value that isn’t None itself.
filter() then checks the truthiness of that return value directly, and NotImplemented is truthy, so non-None items pass through. For None itself, None.__ne__(None) correctly returns False, so it’s filtered out.

As above, this technique triggers a warning from Python 3.9, and raises TypeError from Python 3.14.

The fix is to write the check explicitly in a generator expression or list comprehension, as appropriate:

>>> [x for x in values if x is not None]
[1, 2, 3]

This technique is also about 15 times faster than the filter() version, since it avoids the overhead of calling a method (None.__ne__()) for every item in the list.

Fin

May your code be modernized and swift,

—Adam

Python: inspect interleaved unittest.mock calls with attached mocks

2026-07-27T00:00:00+01:00

When you use a unittest.mock mock, you often make assertions on the calls it received, such as through the mock_calls list. But sometimes you want to assert on the order of calls across multiple mocked functions, for example to check that steps in a process happen in the right sequence.

To do this, use attached mocks, which let you collect calls from multiple mocks into a single timeline. A parent mock has multiple child mocks attached to it, and its mock_calls list records calls from all of its children together, interleaved in the order they actually happened. Sometimes that attachment happens automatically, and sometimes you have to set it up yourself.

Say we have a Kettle class, and a function that fills one by opening the lid, adding water, and closing the lid again:

class Kettle:
    def open(self) -> None: ...

    def add_water(self, litres: float) -> None: ...

    def close(self) -> None: ...


def fill(kettle: Kettle, litres: float) -> None:
    kettle.open()
    kettle.add_water(litres)
    kettle.close()

To test that fill() calls those three methods in order, pass in a mock Kettle:

from unittest import TestCase, mock

from example import Kettle, fill


class FillTests(TestCase):
    def test_fill_order(self):
        kettle = mock.create_autospec(Kettle, instance=True)

        fill(kettle, 1.5)

        assert kettle.mock_calls == [
            mock.call.open(),
            mock.call.add_water(1.5),
            mock.call.close(),
        ]

mock.create_autospec() builds a mock that matches Kettle’s shape, checking arguments against the real method signatures. instance=True makes it stand in for an instance of the class, rather than the class itself.

The method mocks, like open(), are created on access and automatically attached to the parent mock, kettle. That’s how kettle.mock_calls contains all three calls in order, even though we never explicitly attached them, and a single assertion can check the whole sequence.

This all works the same if you use spies rather than plain mocks, by adding the wraps argument, as covered in my previous post. That often makes a better test, since the test would then exercise the real behaviour of the wrapped object as well as recording the calls.

Separate functions need `attach_mock()`

Automatic attachment only helps when the calls you care about are already methods on one object. Say fill_kettle() instead calls three separate module-level functions:

def open_lid() -> None: ...


def fill_water(litres: float) -> None: ...


def close_lid() -> None: ...


def fill_kettle(litres: float) -> None:
    open_lid()
    fill_water(litres)
    close_lid()

open_lid, fill_water, and close_lid are entirely separate functions, so mocking them separately gives you three separate mocks, each with its own independent mock_calls, and no shared parent to collect them together. This is where Mock.attach_mock() comes in: use it to attach each mock to a bare parent mock.

from unittest import TestCase, mock

import example
from example import fill_kettle


class FillKettleTests(TestCase):
    def test_fill_kettle_order(self):
        with (
            mock.patch.object(example, "open_lid", autospec=True) as open_lid_mock,
            mock.patch.object(example, "fill_water", autospec=True) as fill_water_mock,
            mock.patch.object(example, "close_lid", autospec=True) as close_lid_mock,
        ):
            parent = mock.Mock()
            parent.attach_mock(open_lid_mock, "open_lid")
            parent.attach_mock(fill_water_mock, "fill_water")
            parent.attach_mock(close_lid_mock, "close_lid")

            fill_kettle(1.5)

            assert parent.mock_calls == [
                mock.call.open_lid(),
                mock.call.fill_water(1.5),
                mock.call.close_lid(),
            ]

parent doesn’t mock anything itself, and so is never called. It exists only to collect calls from its attached children. Once attached, calls to any of the three mocks also appear on parent.mock_calls, prefixed with the name each was attached under, interleaved in the order they actually happened.

Fin

May you not get too attached to your mocks,

—Adam

Adam Johnson - python

Python: use re.prefixmatch() instead of re.match() from Python 3.15

Introducing re.prefixmatch()

Fin

Python: fix SyntaxWarning: invalid octal escape sequence

Fin

Python: fix TypeError: NamedTuple() got an unexpected keyword argument

The fix

Auto-fix this problem with Ruff

Fin

Python: introducing emojet, a fast emoji lookup library

The motivation

Benchmarkation

How it works

Fin

Python: tprof 1.3.0: now with less overhead

Way less overhead

Comparing against a saved baseline

Median, not mean

Programmatic access to results

Fin

Python: how time-machine is O(1) where freezegun is O(n)

The benchmark

Why freezegun is O(n): the sweep

Why time-machine is O(1): the swap

Migrate today with time-machine’s migration CLI

Fin

Python: time-machine 3.3.0 lets your tests time-travel even faster

Python 3.15 support

Faster patched functions

Subinterpreter support

Migration CLI

Wheels

Fin

Python: fix ValueError: day of month directive '%d' may not be used without a year directive

The fix

Fin

Python: fix TypeError: NotImplemented should not be used in a boolean context

The fix

Another case

Fin

Python: inspect interleaved unittest.mock calls with attached mocks

Separate functions need attach_mock()

Fin

Introducing `re.prefixmatch()`

Separate functions need `attach_mock()`