markdown does not import on ArchLinux

[mitya57]

I am forwarding here the bug I have received in retext-project/retext#458:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    import markdown
  File "/usr/lib/python3.7/site-packages/markdown/__init__.py", line 67, in <module>
    __version__ = _get_version()
  File "/usr/lib/python3.7/site-packages/markdown/__init__.py", line 64, in _get_version
    return str(packaging.version.Version(v))
AttributeError: module 'pkg_resources._vendor.packaging' has no attribute 'version'

I looked at ArchLinux PKGBUILD file and found out that they are patching setuptools to use system versions of modules such as packaging, instead of vendored versions:

https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python-setuptools#n26

Now I realized that pkg_resources.extern.packaging is a non-documented module, and we cannot be sure that it exists on all platforms or that it will not be removed in the future. So probably we should switch to the official version of packaging or at least do something like this:

try:
    import packaging
except ImportError:
    from pkg_resources.extern import packaging

[eli-schwartz]

Actually, markdown imports just fine on archlinux... We do not package markdown 3.1 at all, my assumption is because this bug makes it impossible to build any new version of it.

Ping @keenerd

[mitya57]

Ah, so the reporter of retext-project/retext#458 was using setuptools from ArchLinux with markdown 3.1 from PyPI.

So this (upstream) bug is valid, but the bug I asked them to file against distro package is invalid. Sorry for the confusion.

[mitya57]

Another option may be using pkg_resources.parse_version() which is public and documented API:

https://setuptools.readthedocs.io/en/latest/pkg_resources.html#parsing-utilities

https://github.com/pypa/setuptools/blob/3adda966e2e76ad59cf79fd8bdeab77fb11d308f/pkg_resources/__init__.py#L127-L131

[eli-schwartz]

That seems reasonable. pkg_resources is kind of slow to import just for that, but markdown.core will import it either way (and is itself imported in __init__) so this won't add anything new.

[waylan]

Another option may be using pkg_resources.parse_version() which is public and documented API

My recollection is that that won't work as packaging normalizes the input to a valid version while pkg_resources does not. But I may be remembering that incorrectly. Or it could be that the differences only existed in older python versions. The documentation is not clear for either on how that works. I think I passed both some random (likely) invalid versions and compared the output, but again, my memory is fuzzy on that.

If it works, then by all means.

[waylan]

I just checked the source code and using pkg_resources.parse_version() won't help. All it does is import and use packaging. So ArchLinux has broken a documented API in pkg_resources. This is an upstream problem. I'm not inclined to do anything. ArchLinux can fix this themselves.

As an aside, we are not going to call pkg_resources.parse_version() as it uses LegacyVersion as a fallback. We do not want that behavior. That is the different behavior I couldn't remember in my previous comment.

[eli-schwartz]

The docs say it should return a PEP 440 version, PEP 440 says versions should be normalized, and as of today pkg_resources.parse_version() is a convenience wrapper for, essentially:

import packaging
return packaging.version.Version(...)

Arch Linux most definitively does not break parse_version... that function correctly imports packaging, but does not do it via pkg_resources.extern.

[mitya57]

I just checked the source code and using pkg_resources.parse_version() won't help. All it does is import and use packaging. So ArchLinux has broken a documented API in pkg_resources.

You are wrong — ArchLinux did not break documented API. They replace the import on top of the file, so parse_version works, but uses external version of packaging module rather than the bundled version.

[mitya57]

As an aside, we are not going to call pkg_resources.parse_version() as it uses LegacyVersion as a fallback. We do not want that behavior.

That is understandable. Are you fine with one of the original proposals then — either adding a dependency on packaging, or using the fallback?

[eli-schwartz]

@mitya57 Also note the comments above that, explaining exactly why we implemented the precise solution which we did. :)

setuptools is broken and breaks pip when devendored, because its vendoring solution is inferior to the one pip uses. In fact, the Version object which pkg_resources returns cannot be compared to objects created by packaging.version.Version (unless you are on Arch Linux, I guess)

[waylan]

Are you fine with one of the original proposals then — either adding a dependency on packaging, or using the fallback?

Presumably if pkg_resources is installed at all, then we can be sure one of packaging or pkg_resources._vendor.packaging will be available. That being the case, the fallback should be fine. There is no need to list packaging as a dependency.

So ArchLinux has broken a documented API in pkg_resources.

You are wrong — ArchLinux did not break documented API.

Right. Sorry but I get really annoyed when third party packaging systems change stuff. Maybe they didn't break the documented API, but they broke the library as it was intended by its devs. That's not the way to address it IMO. Take the issue upstream and get the source changed, which would be better for everyone (I agree with ArchLinux's reasons for the change, just not the way they affected the change). If upstream refuses, then just deal with it or write your own library. Don't pass some modified thing off as something its not. So, yeah, they broke the library.

[eli-schwartz]

Right. Sorry but I get really annoyed when third party packaging systems change stuff. Maybe they didn't break the documented API, but they broke the library as it was intended by its devs.

While I'm sympathetic to the frustration this heisenbug has caused, I still don't think I'd go so far as to name it an intention of the devs to have extern be available. And if it was so clear-cut that it was indeed such an intention, I guess Arch Linux would probably not have done all this.

Take the issue upstream and get the source changed, which would be better for everyone (I agree with ArchLinux's reasons for the change, just not the way they affected the change).

We did! There is an upstream bug report. Arch Linux ended up implementing a workaround to make things work, especially pip (because pip being broken is a large issue to have), because pip had already been broken for weeks and there didn't seem to be a solution pending upstream.

Actually this workaround is no longer necessary for pip to work, although the underlying setuptools issue remains.

If upstream refuses, then just deal with it or write your own library.

The first seems harsh, and the second doesn't seem to make much sense. Are you suggesting that if Arch Linux packaged something called "setuptools-ng" or "pyarchtools", which provided egg-info metadata claiming to be an implementation of the pkg_resources/setuptools public API suitable for use by projects such as pip... that you would be okay with this, even though it doesn't claim to provide the internal API of pkg_resources (and therefore it is just as incompatible with markdown 3.1)?

[eli-schwartz]

Anyway, thank you for considering to try both packaging and pkg_resources.extern.packaging