-W option and PYTHONWARNINGS env variable does not accept module regexes

[coldfix]

BPO	34624
Nosy	@ncoghlan, @vstinner, @blueyed, @tiran, @nedbat, @mpaolini, @kernc, @tirkarthi, @coldfix, @kevinoid
PRs	bpo-34624: Allow regex for module passed via -W or PYTHONWARNINGS #9358 bpo-42272: fix misleading warning filter message/module docs #23172

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-20.13:04:35.484>
created_at = <Date 2018-09-10.23:36:34.750>
labels = ['type-feature', 'library', '3.11']
title = '-W option and PYTHONWARNINGS env variable does not accept module regexes'
updated_at = <Date 2022-03-12.09:00:18.988>
user = 'https://github.com/coldfix'

bugs.python.org fields:

activity = <Date 2022-03-12.09:00:18.988>
actor = 'kevinoid'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-20.13:04:35.484>
closer = 'coldfix'
components = ['Library (Lib)']
creation = <Date 2018-09-10.23:36:34.750>
creator = 'coldfix'
dependencies = []
files = []
hgrepos = []
issue_num = 34624
keywords = ['patch', 'security_issue']
message_count = 16.0
messages = ['324959', '324960', '324997', '325538', '328648', '328650', '336646', '348202', '373358', '373536', '373669', '408305', '408496', '408498', '408499', '408959']
nosy_count = 11.0
nosy_names = ['ncoghlan', 'vstinner', 'blueyed', 'christian.heimes', 'nedbat', 'mpaolini', 'kernc', 'xtreak', 'coldfix', 'kevinoid', 'Yongjik Kim']
pr_nums = ['9358', '23172']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34624'
versions = ['Python 3.11']

[coldfix]

Hi,

This command does not report a warning, while it should:

python -c 'import warnings; warnings.warn("This should show up")' -Wi -W'default:::.*'

If the regex .* is replaced by __main__ it works as expected.

Same applies for regexes in PYTHONWARNINGS and for the message part of the argument.

The reason can be found in Lib/warnings.py:144 (def _setoption):

  module = re.escape(module)

This point-blank escape makes me think that it was intended that no regexes can be passed to message/module. On the other, the documentation reads as if it should be supported.

Specifically, the -W option is documented in 1. While this page lists only basic examples, it refers to 2 and 3 for more details. 2 states that message/module are regexes. 3 seems to be written to specifically address the syntax of the PYTHONWARNINGS and the -W option and explicitly lists an example with a regex.

I would welcome if we could remove re.escape to make the implementation fit the documentation, or are there any downsides to this?

Best regards, Thomas

[coldfix]

Very sorry, the example command above should read:

python -Wi -W'default:::.*' -c 'import warnings; warnings.warn("This should show up")'

[tirkarthi]

The original escape was done with 2a862c6 which was in 2000. The doc example you are referring to at [1] was added with https://bugs.python.org/issue31975 and there doesn't seem to be a test involving regular expression as I can see from the PR. Adding Nick to the issue since he had committed the doc and might help here.

[1] https://docs.python.org/3/library/warnings.html#describing-warning-filters

Thanks

[coldfix]

Thanks for your response. I have opened a PR at 1 that would remove the re.escape such that the implementation matches the documentation if you decide that this is fine.

Personally, I would go even further and always add the (\..*)?$ suffix (instead of only $) to make it easier to match all modules in a package which is probably the most important use-case for the case of packages. What do you think?

[nedbat]

Another option is to make clear in the docs that the command line does not accept regular expressions, but only literal module names, and raise an error if a non-importable name (for example with asterisks in it) is specified.

And while we are here: the docs show "error:::mymodule[.*]" which doesn't even make sense as a regex!

[coldfix]

Hi, thanks for the consideration!

Is there any reason to introduce different behaviour than with filterwarnings here? This increases the number of things to remember - and except for the dot regex syntax doesn't interfere with module names, so there is no big drawback here either.

More importantly, my main use case for filterwarnings would be to select/ignore warnings based on module *or package* name. With the current interpretation as exact module name, you have to list all submodules in advance, which is quite inhibiting.

I have fixed the [.*] bogus example in the PR.

Best, Thomas

[ncoghlan]

I think the only reason I didn't mention this discrepancy in my doc updates is because I wasn't aware there *was* a discrepancy.

The weird syntax was then just an incorrect amalgamation of "optional argument" notation with an improperly escaped regex suffix.

[blueyed]

Note that "module" might also be the filename (with ".py" removed).

I've just created bpo-37634 - might be worth including in your PR if it makes sense to only document this.

[mpaolini]

hello Thomas,

do you need any help fixing the conflicts in your PR?

even if Lib/warnings.py changed a little in the last 2 years, your PR is still good!

[coldfix]

Hi, I have rebased this on master and fixed the minor conflict. Let me know if there is anything else I can do.

Best, Thomas