What is needed to support the free-threaded build?

[ngoldbaum]

See also #1649

Using the latest version of Cython, PyAV builds and passes all tests on free-threaded Python 3.13.5 on my Mac if I force the GIL to be disabled. If I additionally run the tests many times in a per-test thread pool using pytest-run-parallel, I do see some failures in test_logging, but I think that's due to use of global logging state rather than real thread safety issues:

Details

=================================================== ERRORS ==================================================== ___________________________________ ERROR at call of test_threaded_captures ___________________________________

def test_threaded_captures() -> None:
    av.logging.set_level(av.logging.VERBOSE)

    with av.logging.Capture(local=True) as logs:
        do_log("main")
        thread = threading.Thread(target=do_log, args=("thread",))
        thread.start()
        thread.join()

  assert (av.logging.INFO, "test", "main") in logs

E AssertionError: assert (32, 'test', 'main') in []

tests/test_logging.py:31: AssertionError
____________________________________ ERROR at call of test_global_captures ____________________________________

def test_global_captures() -> None:
    av.logging.set_level(av.logging.VERBOSE)

    with av.logging.Capture(local=False) as logs:
        do_log("main")
        thread = threading.Thread(target=do_log, args=("thread",))
        thread.start()
        thread.join()

  assert (av.logging.INFO, "test", "main") in logs

E AssertionError: assert (32, 'test', 'main') in []

tests/test_logging.py:44: AssertionError
________________________________________ ERROR at call of test_repeats ________________________________________

def test_repeats() -> None:
    av.logging.set_level(av.logging.VERBOSE)

    with av.logging.Capture() as logs:
        do_log("foo")
        do_log("foo")
        do_log("bar")
        do_log("bar")
        do_log("bar")
        do_log("baz")

    logs = [log for log in logs if log[1] == "test"]

  assert logs == [

        (av.logging.INFO, "test", "foo"),
        (av.logging.INFO, "test", "foo"),
        (av.logging.INFO, "test", "bar"),
        (av.logging.INFO, "test", "bar (repeated 2 more times)"),
        (av.logging.INFO, "test", "baz"),
    ]

E AssertionError: assert [(32, 'test',...test', 'baz')] == [(32, 'test',...test', 'baz')]
E At index 0 diff: (32, 'test', 'bar') != (32, 'test', 'foo')
E Left contains one more item: (32, 'test', 'baz')
E Use -v to get more diff

tests/test_logging.py:62: AssertionError
_________________________________________ ERROR at call of test_error _________________________________________

def test_error() -> None:
    av.logging.set_level(av.logging.VERBOSE)

    log = (av.logging.ERROR, "test", "This is a test.")
    av.logging.log(*log)
    try:

      av.error.err_check(-errno.EPERM)

tests/test_logging.py:79:

---

av/error.pyx:385: in av.error.err_check
cpdef int err_check(int res, filename=None) except -1:

---

raise cls(code, message, filename, log)
E av.error.PermissionError: [Errno 1] Operation not permitted

av/error.pyx:424: PermissionError

During handling of the above exception, another exception occurred:

def test_error() -> None:
    av.logging.set_level(av.logging.VERBOSE)

    log = (av.logging.ERROR, "test", "This is a test.")
    av.logging.log(*log)
    try:
        av.error.err_check(-errno.EPERM)
    except av.error.PermissionError as e:

      assert e.log == log

E AssertionError: assert None == (16, 'test', 'This is a test.')
E + where None = PermissionError(1, 'Operation not permitted').log

tests/test_logging.py:81: AssertionError
---------------------------------------------- Captured log call ----------------------------------------------
ERROR libav.test:test_logging.py:77 This is a test.
***************************************** pytest-run-parallel report ******************************************
All tests were run in parallel! 🎉
=========================================== short test summary info ===========================================
PARALLEL FAILED tests/test_logging.py::test_threaded_captures - AssertionError: assert (32, 'test', 'main') in []
PARALLEL FAILED tests/test_logging.py::test_global_captures - AssertionError: assert (32, 'test', 'main') in []
PARALLEL FAILED tests/test_logging.py::test_repeats - AssertionError: assert [(32, 'test',...test', 'baz')] == [(32, 'test',...test', 'baz')]
PARALLEL FAILED tests/test_logging.py::test_error - AssertionError: assert None == (16, 'test', 'This is a test.')
================================== 286 passed, 13 skipped, 4 errors in 7.62s ==================================

It looks like both test_logging.py and test_timeout.py have tests that create Python threads, but I don't see anything like a parallel processing stress test using multiprocessing or threading. I also don't see any thread safety guarantees in the docs.

What are the thread safety guarantees that you're interested in guaranteeing? Presumably it should be safe to simultaneously read from streams in multiple threads, but then is there any sort of protection from someone abusing threads to simultaneously read and mutate a libav stream? If not, is that also an issue on the GIL-enabled build?

Ultimately I'd like to add support for the free-threaded build following the guidance in the Cython documentation:

https://cython.readthedocs.io/en/latest/src/userguide/freethreading.html

I'd also like to enable free-threaded wheel builds in the cibuildwheel configuration.

[ngoldbaum]

https://ffmpeg.org/developer.html#Code-behaviour makes a pretty strong statement about thread safety in the libav/ffmpeg codebase. Looking at the release notes, it looks like the library started to be thread-safe by default in ffmpeg 6.0.

[ngoldbaum]

I have a branch that sets the freethreading_compatible compiler directive for all the extension modules and turns on the cpython-freethreading option in cibuildwheel. I also ran the tests workflow on my fork, which succeeded. You can download the wheel files the action workflow created from the workflow summary page.

I looked closer at the cython code it looks like things are already in pretty good shape. There are probably some spots that need critical sections --- there are a few descriptors that can be mutated, that sort of thing.

It's not clear to me to what extent the maintainers want to worry about pathological failure modes. It's totally valid just to say that it's up to users to add locking if they want to do shared multithreaded mutation. The concern with adding locking for theoretical thread safety issues is it might introduce unforeseen scaling issues for read-only sharing.

I'll probably stop for now until I get some feedback. I'm very happy to work on this, just need some guidance about how the maintainers would like this to proceed.

[WyattBlue]

Cython now builds 3.14 with freethreading support automatically

[ngoldbaum]

Cython now builds 3.14 with freethreading support automatically

You do still need to declare support in the extension, see https://cython.readthedocs.io/en/latest/src/userguide/freethreading.html#status.

I just installed the PyAV 3.14t wheel, and it looks like the GIL is getting re-enabled at runtime:

>>> import av
<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'av._core', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

What is happening automatically is cibuildwheel 3.1 builds 3.14t wheels without any special configuration. That's probably what happened here.

I'll go ahead and send in a PR to declare free-threaded support and a separate PR to enable 3.13t wheels.