Percent-signs (%) in .pypirc should not be interpolated

[tlevine]

BPO	20120
Nosy	@jaraco, @pitrou, @bitdancer, @ambv, @webknjaz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2016-05-04.16:13:21.715>
created_at = <Date 2014-01-04.08:54:35.372>
labels = ['type-bug', 'library']
title = 'Percent-signs (%) in .pypirc should not be interpolated'
updated_at = <Date 2020-05-03.10:15:05.723>
user = 'https://bugs.python.org/tlevine'

bugs.python.org fields:

activity = <Date 2020-05-03.10:15:05.723>
actor = 'webknjaz'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2016-05-04.16:13:21.715>
closer = 'jaraco'
components = ['Distutils']
creation = <Date 2014-01-04.08:54:35.372>
creator = 'tlevine'
dependencies = []
files = []
hgrepos = []
issue_num = 20120
keywords = []
message_count = 24.0
messages = ['207275', '207277', '207278', '207289', '207295', '207298', '207352', '253344', '256376', '256423', '256424', '258835', '258839', '259031', '259086', '264680', '264753', '264754', '264829', '264831', '264833', '268919', '278233', '367967']
nosy_count = 8.0
nosy_names = ['jaraco', 'pitrou', 'r.david.murray', 'nagylzs', 'lukasz.langa', 'python-dev', 'webknjaz', 'tlevine']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20120'
versions = ['Python 3.5', 'Python 3.6']

[tlevine]

This works fine in Python 2.7, but it fails in Python 3.3.

[tlevine@wildebeest mailfest-scoreboard]$ python3 --version
Python 3.3.3
[tlevine@wildebeest mailfest-scoreboard]$ python3 setup.py register sdist
running register
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    scripts=['scoreboard']
  File "/usr/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.3/distutils/dist.py", line 929, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.3/distutils/dist.py", line 948, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.3/distutils/command/register.py", line 45, in run
    self._set_config()
  File "/usr/lib/python3.3/distutils/command/register.py", line 71, in _set_config
    config = self._read_pypirc()
  File "/usr/lib/python3.3/distutils/config.py", line 83, in _read_pypirc
    current[key] = config.get(server, key)
  File "/usr/lib/python3.3/configparser.py", line 790, in get
    d)
  File "/usr/lib/python3.3/configparser.py", line 391, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
  File "/usr/lib/python3.3/configparser.py", line 440, in _interpolate_some
    "found: %r" % (rest,))
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%rest-of-my-password'

Here are the relevant files.

[tlevine@wildebeest mailfest-scoreboard]$ cat ~/.pypirc
[distutils]
index-servers =
pypi

[pypi]
username:tlevine
password:yh^%#rest-of-my-password
[tlevine@wildebeest mailfest-scoreboard]$ cat setup.py
#!/usr/bin/env python3

from distutils.core import setup

setup(name='mailfest-scoreboard',
      version='0.0.1',
      description='Score mailfest participants',
      author='Thomas Levine',
      author_email='_@thomaslevine.com',
      url='https://github.com/tlevine/mailfest-scoreboard',
      scripts=['scoreboard']
     )

[tlevine]

The relevant section distutils/config.py seems no different in Python 2.7, so now I see this as a bug in configparser.

[tlevine]

Hmm now it looks to me like this is the intended behavior.
http://hg.python.org/cpython/file/ea0aa3e32ab5/Lib/test/test_configparser.py#l1541

Switching the single percent sign (%) to two (%%) in .pypirc makes it work. Maybe we can make a nicer error message though.

[bitdancer]

It could be an issue of which configparser is/was used.

[pitrou]

In 3.x, the ConfigParser class is 2.x's SafeConfigParser, and the parsing rules are different (stricter, it seems).

It's probably a won't fix, I'd say.

[bitdancer]

"More rational" is probably closer to the truth :).

Yeah, it's probably a won't fix, but it would be nice to have Łukasz's input.

[ambv]

Yes, that's wontfix but we can make the error message more friendly so I'll leave this open.

The .pypirc parser should have been RawConfigParser from the start, the interpolation mechanism is obscure enough it's probably not used. Unfortunately this "probably" means we cannot easily switch it to RawConfigParser now.

[nagylzs]

When I uploaded via "setup.py register" and created a new user, then distutils saved the password in the wrong format! So maybe the password read part is a wontfix, but then the save part must be fixed.

[jaraco]

This same issue applies to setuptools (https://bitbucket.org/pypa/setuptools/issues/442/setuptools-1832-cannot-access-pypi-if-pypi).

I've created a test that captures the failure, but I struggled to find a solution that doesn't break across Python versions.

Thomas suggested guiding the user to provide two percent signs instead of one, but that breaks on Python 2, which will interpret the value as two percent signs.

As suggested, using RawConfigParser does address the issue on both Python 2 and 3, but that of course could break compatibility if interpolation was expected.

I have found that if one uses SafeConfigParser on Python 2 and 3, and requires the user to use two percent signs to represent one, it does seem to work on both Python 2 and 3.

[tlevine]

The error message could just say. something to the effect of ".pypyrc works differently in Python 2 and Python 3. If you want a percent sign in Python 3, you have to escape it with another percent sign".

Suppose someone is already using a configuration file that specifies a value as having two literal percent signs in a row, like a password that actually contains two percent signs. I see no way of fixing the present bug without breaking this situation, and I presume that this is why Antoine suggested that this is a wont fix.

[jaraco]

It doesn't make sense to give guidance to use one percent sign on one Python and two on another because .pypirc applies to all Python versions. I'm sure it's not uncommon for one to use different versions of Python for various package releases.

I'm planning to use RawConfigParser in Setuptools. If there's no fallout from this change, I suggest that Python 3 should do the same for distutils in bugfix releases, as this behavior is a regression over Python 2.

[tlevine]

I noticed something else that might be interesting.
If I delete my .pypirc and have setuptools create it,

python3.5 -c 'import setuptools; setuptools.setup()' register

the .pypirc is created with the un-escaped percent sign and without error.
I receive the error when I try to use the .pypirc next time.

If a different config parser is being used in the writing, perhaps we could at least use the same one in both places.