Implement PEP 440 by using the packaging library

Author: dstufft

CHANGES.txt

@@ -2,7 +2,6 @@
 CHANGES
 =======
 
----
 6.0
 ---
 
@@ -24,6 +23,9 @@ CHANGES
   case-insensitively, but not case-sensitively, should rename those files in
   their repository for better portability.
 
+* Implement PEP 440 within pkg_resources and setuptools. This will cause some
+  versions to no longer be installable without using the ``===`` escape hatch.
+
 ---
 5.8
 ---

docs/pkg_resources.txt

@@ -594,7 +594,7 @@ Requirements Parsing
 
         requirement  ::= project_name versionspec? extras?
         versionspec  ::= comparison version (',' comparison version)*
-        comparison   ::= '<' | '<=' | '!=' | '==' | '>=' | '>'
+        comparison   ::= '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
         extras       ::= '[' extralist? ']'
         extralist    ::= identifier (',' identifier)*
         project_name ::= identifier
@@ -646,13 +646,10 @@ Requirements Parsing
     The ``Requirement`` object's version specifiers (``.specs``) are internally
     sorted into ascending version order, and used to establish what ranges of
     versions are acceptable.  Adjacent redundant conditions are effectively
-    consolidated (e.g. ``">1, >2"`` produces the same results as ``">1"``, and
-    ``"<2,<3"`` produces the same results as``"<3"``). ``"!="`` versions are
+    consolidated (e.g. ``">1, >2"`` produces the same results as ``">2"``, and
+    ``"<2,<3"`` produces the same results as``"<2"``). ``"!="`` versions are
     excised from the ranges they fall within.  The version being tested for
     acceptability is then checked for membership in the resulting ranges.
-    (Note that providing conflicting conditions for the same version (e.g.
-    ``"<2,>=2"`` or ``"==2,!=2"``) is meaningless and may therefore produce
-    bizarre results when compared with actual version number(s).)
 
 ``__eq__(other_requirement)``
     A requirement compares equal to another requirement if they have
@@ -681,10 +678,7 @@ Requirements Parsing
 ``specs``
     A list of ``(op,version)`` tuples, sorted in ascending parsed-version
     order.  The `op` in each tuple is a comparison operator, represented as
-    a string.  The `version` is the (unparsed) version number.  The relative
-    order of tuples containing the same version numbers is undefined, since
-    having more than one operator for a given version is either redundant or
-    self-contradictory.
+    a string.  The `version` is the (unparsed) version number.
 
 
 Entry Points
@@ -967,7 +961,7 @@ version
     ``ValueError`` is raised.
 
 parsed_version
-    The ``parsed_version`` is a tuple representing a "parsed" form of the
+    The ``parsed_version`` is an object representing a "parsed" form of the
     distribution's ``version``.  ``dist.parsed_version`` is a shortcut for
     calling ``parse_version(dist.version)``.  It is used to compare or sort
     distributions by version.  (See the `Parsing Utilities`_ section below for
@@ -1541,40 +1535,12 @@ Parsing Utilities
 -----------------
 
 ``parse_version(version)``
-    Parse a project's version string, returning a value that can be used to
-    compare versions by chronological order.  Semantically, the format is a
-    rough cross between distutils' ``StrictVersion`` and ``LooseVersion``
-    classes; if you give it versions that would work with ``StrictVersion``,
-    then they will compare the same way.  Otherwise, comparisons are more like
-    a "smarter" form of ``LooseVersion``.  It is *possible* to create
-    pathological version coding schemes that will fool this parser, but they
-    should be very rare in practice.
-
-    The returned value will be a tuple of strings.  Numeric portions of the
-    version are padded to 8 digits so they will compare numerically, but
-    without relying on how numbers compare relative to strings.  Dots are
-    dropped, but dashes are retained.  Trailing zeros between alpha segments
-    or dashes are suppressed, so that e.g. "2.4.0" is considered the same as
-    "2.4". Alphanumeric parts are lower-cased.
-
-    The algorithm assumes that strings like "-" and any alpha string that
-    alphabetically follows "final"  represents a "patch level".  So, "2.4-1"
-    is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
-    considered newer than "2.4-1", which in turn is newer than "2.4".
-
-    Strings like "a", "b", "c", "alpha", "beta", "candidate" and so on (that
-    come before "final" alphabetically) are assumed to be pre-release versions,
-    so that the version "2.4" is considered newer than "2.4a1".  Any "-"
-    characters preceding a pre-release indicator are removed.  (In versions of
-    setuptools prior to 0.6a9, "-" characters were not removed, leading to the
-    unintuitive result that "0.2-rc1" was considered a newer version than
-    "0.2".)
-
-    Finally, to handle miscellaneous cases, the strings "pre", "preview", and
-    "rc" are treated as if they were "c", i.e. as though they were release
-    candidates, and therefore are not as new as a version string that does not
-    contain them.  And the string "dev" is treated as if it were an "@" sign;
-    that is, a version coming before even "a" or "alpha".
+    Parsed a project's version string as defined by PEP 440. The returned
+    value will be an object that represents the version. These objects may
+    be compared to each other and sorted. The sorting algorithm is as defined
+    by PEP 440 with the addition that any version which is not a valid PEP 440
+    version will be considered less than any valid PEP 440 version and the
+    invalid versions will continue sorting using the original algorithm.
 
 .. _yield_lines():
 
@@ -1629,10 +1595,12 @@ Parsing Utilities
     See ``to_filename()``.
 
 ``safe_version(version)``
-    Similar to ``safe_name()`` except that spaces in the input become dots, and
-    dots are allowed to exist in the output.  As with ``safe_name()``, if you
-    are generating a filename from this you should replace any "-" characters
-    in the output with underscores.
+    This will return the normalized form of any PEP 440 version, if the version
+    string is not PEP 440 compatible than it is similar to ``safe_name()``
+    except that spaces in the input become dots, and dots are allowed to exist
+    in the output.  As with ``safe_name()``, if you are generating a filename
+    from this you should replace any "-" characters in the output with
+    underscores.
 
 ``safe_extra(extra)``
     Return a "safe" form of an extra's name, suitable for use in a requirement

pkg_resources.py

@@ -73,6 +73,14 @@
 except ImportError:
     pass
 
+# Import packaging.version.parse as parse_version for a compat shim with the
+# old parse_version that used to be defined in this file.
+from setuptools._vendor.packaging.version import parse as parse_version
+
+from setuptools._vendor.packaging.version import (
+    Version, InvalidVersion, Specifier,
+)
+
 
 _state_vars = {}
 
@@ -1143,13 +1151,14 @@ def safe_name(name):
 
 
 def safe_version(version):
-    """Convert an arbitrary string to a standard version string
-
-    Spaces become dots, and all other non-alphanumeric characters become
-    dashes, with runs of multiple dashes condensed to a single dash.
     """
-    version = version.replace(' ','.')
-    return re.sub('[^A-Za-z0-9.]+', '-', version)
+    Convert an arbitrary string to a standard version string
+    """
+    try:
+        return str(Version(version))  # this will normalize the version
+    except InvalidVersion:
+        version = version.replace(' ','.')
+        return re.sub('[^A-Za-z0-9.]+', '-', version)
 
 
 def safe_extra(extra):
@@ -2067,7 +2076,7 @@ def yield_lines(strs):
 # Distribution or extra
 DISTRO = re.compile(r"\s*((\w|[-.])+)").match
 # ver. info
-VERSION = re.compile(r"\s*(<=?|>=?|==|!=)\s*((\w|[-.])+)").match
+VERSION = re.compile(r"\s*(<=?|>=?|===?|!=|~=)\s*((\w|[-.*_!+])+)").match
 # comma between items
 COMMA = re.compile(r"\s*,").match
 OBRACKET = re.compile(r"\s*\[").match
@@ -2079,67 +2088,6 @@ def yield_lines(strs):
     re.VERBOSE | re.IGNORECASE
 ).match
 
-component_re = re.compile(r'(\d+ | [a-z]+ | \.| -)', re.VERBOSE)
-replace = {'pre':'c', 'preview':'c','-':'final-','rc':'c','dev':'@'}.get
-
-def _parse_version_parts(s):
-    for part in component_re.split(s):
-        part = replace(part, part)
-        if not part or part=='.':
-            continue
-        if part[:1] in '0123456789':
-            # pad for numeric comparison
-            yield part.zfill(8)
-        else:
-            yield '*'+part
-
-    # ensure that alpha/beta/candidate are before final
-    yield '*final'
-
-def parse_version(s):
-    """Convert a version string to a chronologically-sortable key
-
-    This is a rough cross between distutils' StrictVersion and LooseVersion;
-    if you give it versions that would work with StrictVersion, then it behaves
-    the same; otherwise it acts like a slightly-smarter LooseVersion. It is
-    *possible* to create pathological version coding schemes that will fool
-    this parser, but they should be very rare in practice.
-
-    The returned value will be a tuple of strings.  Numeric portions of the
-    version are padded to 8 digits so they will compare numerically, but
-    without relying on how numbers compare relative to strings.  Dots are
-    dropped, but dashes are retained.  Trailing zeros between alpha segments
-    or dashes are suppressed, so that e.g. "2.4.0" is considered the same as
-    "2.4". Alphanumeric parts are lower-cased.
-
-    The algorithm assumes that strings like "-" and any alpha string that
-    alphabetically follows "final"  represents a "patch level".  So, "2.4-1"
-    is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
-    considered newer than "2.4-1", which in turn is newer than "2.4".
-
-    Strings like "a", "b", "c", "alpha", "beta", "candidate" and so on (that
-    come before "final" alphabetically) are assumed to be pre-release versions,
-    so that the version "2.4" is considered newer than "2.4a1".
-
-    Finally, to handle miscellaneous cases, the strings "pre", "preview", and
-    "rc" are treated as if they were "c", i.e. as though they were release
-    candidates, and therefore are not as new as a version string that does not
-    contain them, and "dev" is replaced with an '@' so that it sorts lower than
-    than any other pre-release tag.
-    """
-    parts = []
-    for part in _parse_version_parts(s.lower()):
-        if part.startswith('*'):
-            # remove '-' before a prerelease tag
-            if part < '*final':
-                while parts and parts[-1] == '*final-':
-                    parts.pop()
-            # remove trailing zeros from each series of numeric parts
-            while parts and parts[-1]=='00000000':
-                parts.pop()
-        parts.append(part)
-    return tuple(parts)
-
 
 class EntryPoint(object):
     """Object representing an advertised importable object"""
@@ -2292,7 +2240,7 @@ def from_location(cls, location, basename, metadata=None,**kw):
     @property
     def hashcmp(self):
         return (
-            getattr(self, 'parsed_version', ()),
+            self.parsed_version,
             self.precedence,
             self.key,
             _remove_md5_fragment(self.location),
@@ -2338,11 +2286,10 @@ def key(self):
 
     @property
     def parsed_version(self):
-        try:
-            return self._parsed_version
-        except AttributeError:
-            self._parsed_version = pv = parse_version(self.version)
-            return pv
+        if not hasattr(self, "_parsed_version"):
+            self._parsed_version = parse_version(self.version)
+
+        return self._parsed_version
 
     @property
     def version(self):
@@ -2447,7 +2394,12 @@ def from_filename(cls, filename, metadata=None, **kw):
 
     def as_requirement(self):
         """Return a ``Requirement`` that matches this distribution exactly"""
-        return Requirement.parse('%s==%s' % (self.project_name, self.version))
+        if isinstance(self.parsed_version, Version):
+            spec = "%s==%s" % (self.project_name, self.parsed_version)
+        else:
+            spec = "%s===%s" % (self.project_name, self.parsed_version)
+
+        return Requirement.parse(spec)
 
     def load_entry_point(self, group, name):
         """Return the `name` entry point of `group` or raise ImportError"""
@@ -2699,7 +2651,7 @@ def scan_list(ITEM, TERMINATOR, line, p, groups, item_name):
 
         line, p, specs = scan_list(VERSION, LINE_END, line, p, (1, 2),
             "version spec")
-        specs = [(op, safe_version(val)) for op, val in specs]
+        specs = [(op, val) for op, val in specs]
         yield Requirement(project_name, specs, extras)
 
 
@@ -2708,26 +2660,23 @@ def __init__(self, project_name, specs, extras):
         """DO NOT CALL THIS UNDOCUMENTED METHOD; use Requirement.parse()!"""
         self.unsafe_name, project_name = project_name, safe_name(project_name)
         self.project_name, self.key = project_name, project_name.lower()
-        index = [
-            (parse_version(v), state_machine[op], op, v)
-            for op, v in specs
-        ]
-        index.sort()
-        self.specs = [(op, ver) for parsed, trans, op, ver in index]
-        self.index, self.extras = index, tuple(map(safe_extra, extras))
+        self.specifier = Specifier(
+            ",".join(["".join([x, y]) for x, y in specs])
+        )
+        self.specs = specs
+        self.extras = tuple(map(safe_extra, extras))
         self.hashCmp = (
             self.key,
-            tuple((op, parsed) for parsed, trans, op, ver in index),
+            self.specifier,
             frozenset(self.extras),
         )
         self.__hash = hash(self.hashCmp)
 
     def __str__(self):
-        specs = ','.join([''.join(s) for s in self.specs])
         extras = ','.join(self.extras)
         if extras:
             extras = '[%s]' % extras
-        return '%s%s%s' % (self.project_name, extras, specs)
+        return '%s%s%s' % (self.project_name, extras, self.specifier)
 
     def __eq__(self, other):
         return (
@@ -2739,29 +2688,13 @@ def __contains__(self, item):
         if isinstance(item, Distribution):
             if item.key != self.key:
                 return False
-            # only get if we need it
-            if self.index:
-                item = item.parsed_version
-        elif isinstance(item, string_types):
-            item = parse_version(item)
-        last = None
-        # -1, 0, 1
-        compare = lambda a, b: (a > b) - (a < b)
-        for parsed, trans, op, ver in self.index:
-            # Indexing: 0, 1, -1
-            action = trans[compare(item, parsed)]
-            if action == 'F':
-                return False
-            elif action == 'T':
-                return True
-            elif action == '+':
-                last = True
-            elif action == '-' or last is None:
-                last = False
-        # no rules encountered
-        if last is None:
-            last = True
-        return last
+
+            item = item.version
+
+        # Allow prereleases always in order to match the previous behavior of
+        # this method. In the future this should be smarter and follow PEP 440
+        # more accurately.
+        return self.specifier.contains(item, prereleases=True)
 
     def __hash__(self):
         return self.__hash
@@ -2777,16 +2710,6 @@ def parse(s):
             raise ValueError("Expected only one requirement", s)
         raise ValueError("No requirements found", s)
 
-state_machine = {
-    #       =><
-    '<': '--T',
-    '<=': 'T-T',
-    '>': 'F+F',
-    '>=': 'T+F',
-    '==': 'T..',
-    '!=': 'F++',
-}
-
 
 def _get_mro(cls):
     """Get an mro for a type or classic class"""