Add support for abi3.abi3t tag from PEP 803

[ngoldbaum]

This updates the tags logic to handle abi3t from PEP 803. Also adds tests to validate everything.

[ngoldbaum]

@pfmoore, thank you! I also pushed an accompanying fix to my setuptools PR that allows setuptools to correctly support compressed ABI tags without incorrectly returning compressed tags from cpython_tags here. See the newest commit in pypa/setuptools#5193.

[pfmoore]

Just to be 100% clear here, when you returned abi3.abi3t, you did mean that the environment will work with either an abi3 or an abi3t wheel, didn't you?

You seem now to be saying that free threaded builds only support abi3t.

I think one of us is confused here 🙁

[ngoldbaum]

You seem now to be saying that free threaded builds only support abi3t.

Take a look at the test I added that checks the output of tags.cpython_tags((3, 16), ["cp316"], ["platform"] and note how there are abi3t tags for both 3.15 and 3.16.

https://github.com/pypa/packaging/pull/1099/changes#diff-c2a8158c2ea75e325dc5d418fe408374d215b771f40be4a907dca380ad4a6701R1060-R1080

It turns out setuptools doesn't currently support compressed ABI tags unless I make packaging produce them, which is why I thought that was necessary initially. I also had to slightly patch how setuptools checks whether tags are compatible by calling tags.parse_tags. See the change I made to the assert here: https://github.com/pypa/setuptools/pull/5193/changes#diff-5f0d2f159393c18a35b50b6a97212ace4cb44ef75df0c85dd28edb23dce9a782L356

[pfmoore]

OK, some thoughts. But I'm getting way out of my depth here, and I'm concerned that we're not going off the PEP here but rather trying to define behaviours that aren't clearly stated in the PEP. It's quite likely we need to get the PEP clarified before making decisions here.

For example, why is cp315-abi3t so low in the priority order in the cp316 example? Surely a cp315-abi3t wheel should be preferred over a cp310-abi3 one? Or is the ordering only implemented in sys_tags() and the low-level functions don't need to yield their values in preference order? This is where I don't understand the packaging implementation well enough to know what's acceptable - but I'm 99.9% sure that pip calls cpython_tags directly and relies on the values being returned in "most preferred first" order...

I can't comment on the setuptools point. But I do know that pip won't work if packaging returns a compressed tag.

I feel that this might need to be brought back to Discourse. At the very least, I think we need more eyes on the problem.

[ngoldbaum]

For example, why is cp315-abi3t so low in the priority order in the cp316 example?

Because it simply didn't occur to me that sys_tags is in priority order. But also it says it right there in the docstring:

packaging/src/packaging/tags.py

Lines 654 to 660 in cd1520c

	def sys_tags(*, warn: bool = False) -> Iterator[Tag]:
	"""
	Returns the sequence of tag triples for the running interpreter.
	The order of the sequence corresponds to priority order for the
	interpreter, from most to least important.
	"""

I adjusted the logic in my latest commit.

I feel that this might need to be brought back to Discourse. At the very least, I think we need more eyes on the problem.

I agree, let's ask Petr to clarify the tag priority order for the GIL-enabled build, since it's ambiguous in PEP 803 right now.

[ngoldbaum]

And after trying to draft a message about this, I think I finally understand what needs to happen here.

Installers should only install wheels on the free-threaded build with an abi3t tag, that means packaging should only produce abi3t tags on the free-threaded build.

I was confused in my original implementation, because I thought that packaging was responsible for generating compressed tag sets, but that's wrong. Build tools do that.

I was also confused by the fact that setuptools didn't handle compressed tag sets in abi tags. I'll raise this issue with the setuptools maintainers separately from PEP 803 support.

I agree that this is a little "sparely" described in the PEP. I'll post a followup to discourse about this.

[pfmoore]

Given the clarification in the Discourse thread, I'd strongly recommend adding some sort of test(s) here to validate that if you change the Python implementation from 316 to 316t, the list of tags remains unchanged except for abi3 changing to abi3t and cp316 changing to cp316t. And probably vice versa, for good measure 🙂

[ngoldbaum]

I think the last push responds to all comments. Thanks for helping me to understand this better!

[henryiii]

This looks reasonable to me.

[henryiii]

I think this just needs a note in the docs with a versionchanged.

[ngoldbaum]

Good point, I updated the docstrings.

I also somehow messed up the git history on my branch and accidentally rebased a bunch of commits from main. Rather than trying to sort that out, I just squashed the history from this PR into a single commit. I know that's annoying to deal with as a reviewer but I hope it's not too bad here given the small diff outside of the tests. There are no subtle changes from earlier rounds, just expanded docstrings.

[stonebig]

pip-26.1 is soon , can this be ready in time ?

[ngoldbaum]

@henryiii do any other packaging maintainers need to give this a once-over?

I spoke with @notatallshaw about this on the PyPA Discord and he said the pip release isn't yet scheduled but it should happen towards the end of the month.

So as long as there's a packaging release with the code from
this PR merged sometime in the next couple weeks, pip's normal vendoring process should pick this up.

No other work besides updating the vendored packaging is needed in pip unless there's a hole in our testing somewhere.

[henryiii]

I was hoping a few of the others who engaged earlier would look it over and approve, but if not, I'll merge.

[notatallshaw]

I've not been closely following this, either here or on the DPO thread, but I'd just push back a little on urgently merging/deploying. The packaging -> release -> pip -> release pipeline might seem frustrating slow, but given the impact the cost of three months is almost always worth getting something right in the first place compared to having a version of pip out there that implements something wrong.

That said, this isn't a review of this PR, if there's a consensus this behavior is correct, excited to see it land.

[pfmoore]

Having both _abi3_applies and _abi3t_applies seems odd - it obscures the logic a little and suggests that it's possible for both to apply in some circumstances (which isn't true). It might be worthwhile to refactor this to simplify things, but I guess that can be a follow-up PR.

[pfmoore]

I do agree with @notatallshaw though - this shouldn't be merged in a rush to get into pip 26.1, it should be merged when it's ready to go. Having to do a follow-up release and an emergency pip release just because someone found a last minute logic flaw would suck.

Ideally I'd say wait until after pip 26.1, and give it some time to be tested before releasing it in pip 26.2, but the reality is that no-one tests pip before release, so that's probably pointless 🙁

[ngoldbaum]

but given the impact the cost of three months is almost always worth getting something right in the first place compared to having a version of pip out there that implements something wrong.

I think most of the complexity in PEP 803 is on the build side rather than the installer side. Having a version of pip with support for the abi3t ABI tag available alongside 3.15.0b1 will make testing builds much easier than if we had to wait three months.

[henryiii]

I think this is quite simple, and follows the accepted PEP, so there's not an issue merging, and it's important to have it ready for 3.15.0b1 so we can start testing builds. Unless it's fine to make a patch release of pip bumping packaging to a version that supports this before 3.15.0b1 is out.

If there was potential issue, yes it can be delayed, but I think we are in agreement that this follows the PEP?

[pfmoore]

Yes, it follows the PEP as I understand it. Agreed having it in the pip shipped with Python 3.15b1 makes sense. Although how much use it will get probably depends on the state of build backend support (because that affects availability of free-threaded wheels), which I assume is in progress as well.

[henryiii]

I plan to have scikit-build-core ready. Though might depend a bit on upstream CMake.

[ngoldbaum]

My setuptools PR is still a draft because it depends on this PR but otherwise is ready for review. I still haven't heard from any of the setuptools maintainers. I'm a little worried that setuptools won't make a new release for quite a while.

I'm also going to look at whether the CFFI maintainers are OK with merging in the code in the cffi-buildtool project and/or better documenting how to do CFFI builds with any build backend, which might end up unblocking projects that use CFFI and rely on setuptools. Right now setuptools is the only documented way to build CFFI extensions in the CFFI docs.

@mgorny will be taking on the work needed for meson-python and possibly meson. There's a WIP branch here. It still needs to be updated to match the spec in the final PEP 803 implementation.

I have WIP branches of maturin and PyO3 that can produce abi3t.abi3 wheels. See this post on Discourse for more details. PyO3 and maturin support probably can't be added until 3.15.0b1 at the absolute earliest because PyO3 has a policy of not enabling stable ABI wheels until after beta 1. My PyO3 branch also needs bindings for the critical section API in the stable ABI. Technically it's possible to get PyO3 working without critical sections, but making everything thread safe will require a different locking strategy in the limited API and I'd really like to avoid that if possible.

@henryiii if you would like an extra hand anywhere with scikit-build-core, please feel free to ping me!

All that to say, things are moving.

I'm also not worried about the pace of support in build backends. With build backends it's a little less of a tall order to require pre-release versions of build dependencies to enable ecosystem-wide experimentation. The build only needs to happen once per architecture. For installers it's a bigger problem if the installer refuses to install a wheel, since there are going to be many more installs than builds. I'm glad things look on-track to have a compatible pip available with 3.15.0b1.

[pradyunsg]

It sounds like it would be good to land & release this in time for 3.15b1; which I personally would be onboard with. :)