pathspec v1 changes

[cpburnz]

Is your feature request related to a problem? Please describe.

Hi, I'm the author of pathspec, and recently released v1. The biggest change is GitWildMatchPattern (a.k.a. "gitwildmatch") was split into two different implementations. (1) GitIgnoreBasicPattern (a.k.a. "gitignore") is new and properly implements the patterns in gitignore's documentation. (2) GitIgnoreSpecPattern is the previous GitWildMatchPattern implementation and is used by GitIgnoreSpec to implement Git's edge-cases.

PathSpec

The form PathSpec.from_lines('gitwildmatch', ...) is now deprecated because it does not fully match the behavior of the gitignore docs or Git itself. There's 3 possible replacements depending on the exact behavior you want:

1. To keep the exact behavior as before, use PathSpec.from_lines(GitIgnoreSpecPattern, ...). This is the non-deprecated equivalent to using "gitwildmatch". I discourage this use because it's a hybrid between Git's behavior and the gitignore docs.

   from pathspec.patterns.gitignore.spec import GitIgnoreSpecPattern

2. To follow the gitignore docs, use PathSpec.from_lines('gitignore', ...). The change for using this form is "foo/*" will no longer match files in subdirectories, and the exact pattern "/" is equivalent to "**" (match everything) rather than being a no-op.

3. To follow Git's implementation, use GitIgnoreSpec.from_lines(...). The change for using this is Git allows including files from excluded directories which directly contradicts the gitignore docs.

black/src/black/files.py

Line 249 in c3cc5a9

return PathSpec.from_lines("gitwildmatch", lines)

black/tests/test_black.py

Lines 2517 to 2519 in c3cc5a9

	gitignore = PathSpec.from_lines(
	"gitwildmatch", ["exclude/", ".definitely_exclude"]
	)

black/tests/test_black.py

Line 2693 in c3cc5a9

gitignore = PathSpec.from_lines("gitwildmatch", [])

GitIgnorePatternError

pathspec.patterns.gitwildmatch.GitWildMatchPatternError is now a deprecated alias of pathspec.patterns.gitignore.GitIgnorePatternError. I recommend switching to GitIgnorePatternError.

black/src/black/files.py

Line 14 in c3cc5a9

from pathspec.patterns.gitwildmatch import GitWildMatchPatternError

black/src/black/__init__.py

Line 27 in c3cc5a9

from pathspec.patterns.gitwildmatch import GitWildMatchPatternError

[cobaltt7]

PRs are welcome!

[cpburnz]

@cobaltt7 I'm happy to submit a pull request. However, I need to know whether behavior 1, 2, or 3 should be used first.

[tzachshabtay]

Hey so we started getting an error importing black, we're suspecting it's due to the latest pathspec release as it's not pinned apparently, can you confirm this is the cause for this error?
(Note that we're currently running with black 23.9.1, will updating to latest expected to resolve this or do we need to wait for a fix?)

(Edit: for now we worked around the error by pinning pathspec to be under 1.0.0)

The error:

    from black import FileMode, format_str
  File \"src/black/__init__.py\", line 33, in <module>
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/__init__.py\", line 30, in <module>
    from .gitignore import (
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/gitignore.py\", line 28, in <module>
    from pathspec._backends.agg import (
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/_backends/agg.py\", line 23, in <module>
    from .hyperscan.gitignore import (
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/_backends/hyperscan/gitignore.py\", line 24, in <module>
    from pathspec.patterns.gitignore.spec import (
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/patterns/__init__.py\", line 6, in <module>
    from .gitignore import basic as _
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/patterns/gitignore/basic.py\", line 12, in <module>
    from pathspec import util
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/util.py\", line 208, in <module>
    on_error: Optional[Callable[[OSError], None]] = None,
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 243, in inner
    return func(*args, **kwds)
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 316, in __getitem__
    return self._getitem(self, parameters)
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 433, in Optional
    return Union[arg, type(None)]
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 243, in inner
    return func(*args, **kwds)
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 316, in __getitem__
    return self._getitem(self, parameters)
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 421, in Union
    parameters = _remove_dups_flatten(parameters)
  File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 215, in _remove_dups_flatten
    all_params = set(params)
TypeError: unhashable type: 'list'

Thanks

[cpburnz]

@tzachshabtay This looks like an unexpected bug with the pathspec type hints affecting only Python 3.9.0 and 3.9.1. 3.9.2 fixed an incompatibility between typing.Callable and collections.abc.Callable.

…

-------- Original Message --------

On Wednesday, 01/07/26 at 13:06 Tzach Shabtay ***@***.***> wrote: tzachshabtay left a comment [(psf/black#4944)](#4944 (comment)) Hey so we started getting an error importing black, we're suspecting it's due to the latest pathspec release as it's not pinned apparently, can you confirm this is the cause for this error? (Note that we're currently running with black 23.9.1, will updating to latest expected to resolve this or do we need to wait for a fix?) The error: from black import FileMode, format_str File \"src/black/__init__.py\", line 33, in <module> File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/__init__.py\", line 30, in <module> from .gitignore import ( File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/gitignore.py\", line 28, in <module> from pathspec._backends.agg import ( File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/_backends/agg.py\", line 23, in <module> from .hyperscan.gitignore import ( File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/_backends/hyperscan/gitignore.py\", line 24, in <module> from pathspec.patterns.gitignore.spec import ( File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/patterns/__init__.py\", line 6, in <module> from .gitignore import basic as _ File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/patterns/gitignore/basic.py\", line 12, in <module> from pathspec import util File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/pathspec/util.py\", line 208, in <module> on_error: Optional[Callable[[OSError], None]] = None, File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 243, in inner return func(*args, **kwds) File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 316, in __getitem__ return self._getitem(self, parameters) File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 433, in Optional return Union[arg, type(None)] File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 243, in inner return func(*args, **kwds) File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 316, in __getitem__ return self._getitem(self, parameters) File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 421, in Union parameters = _remove_dups_flatten(parameters) File \"/home/ubuntu/.asdf/installs/python/3.9.0/lib/python3.9/typing.py\", line 215, in _remove_dups_flatten all_params = set(params) TypeError: unhashable type: 'list' Thanks — Reply to this email directly, [view it on GitHub](#4944 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAQHBWZFPMNVDQXIHPXSR2L4FVDLJAVCNFSM6AAAAACQ4OCFP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMRQGEYDENRUGY). You are receiving this because you authored the thread.Message ID: ***@***.***>

[cpburnz]

@tzachshabtay pathspec v1.0.2 now fixes this problem.

[cobaltt7]

@cobaltt7 I'm happy to submit a pull request. However, I need to know whether behavior 1, 2, or 3 should be used first.

Sorry, missed that part.
I think (3) would be the best solution, but for the sake of minimizing unwanted changes we may want to stick with (1). How much would changing to (3) affect, do you think? If we do decide to switch to (3), there should be a clear call-out in the changelog. Also, our next release does happen to be a major release (26.1.0), which would be a good time to make this change.

[cpburnz]

@cobaltt7 The change for using (3) is it will allow files to be included even if the parent directly is excluded. For example:

>>> from pathspec import GitIgnoreSpec
>>> spec = GitIgnoreSpec.from_lines(['*', '!libfoo/'])
>>> list(spec.match_files(['ignore.txt', 'libfoo/__init__.py']))
['ignore.txt', 'libfoo/__init__.py']

So, despite the directory libfoo/ being excluded by pattern !libfoo/, the file libfoo/__init__.py is still included by pattern *. Included here means Git ignores it.

Using (1) or (2) will have file libfoo/__init__.py excluded instead.

You can verify Git's behavior by creating the file tree:

ignore.txt
libfoo/__init__.py

And .gitignore:

*
!libfoo/

Running git check-ignore -nv $(find *) will result in:

.gitignore:1:*  ignore.txt
.gitignore:2:!libfoo/   libfoo
.gitignore:1:*  libfoo/__init__.py

[cobaltt7]

Huh, interesting. I would've assumed that libfoo would be excluded with that .gitignore.

The only time we use pathspec is specifically to read patterns from .gitignore (the --exclude options use regex) so I do think we should change this to follow Git's behavior.