configparser.InterpolationMissingOptionError is not very intuitive

[PCManticore]

BPO	21159
Nosy	@freddrake, @warsaw, @rbtcollins, @PCManticore, @ambv, @serhiy-storchaka
Files	issue21159.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2016-06-04.17:10:07.252>
created_at = <Date 2014-04-05.10:35:12.240>
labels = ['type-bug', 'library']
title = 'configparser.InterpolationMissingOptionError is not very intuitive'
updated_at = <Date 2016-06-04.17:10:07.250>
user = 'https://github.com/PCManticore'

bugs.python.org fields:

activity = <Date 2016-06-04.17:10:07.250>
actor = 'lukasz.langa'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2016-06-04.17:10:07.252>
closer = 'lukasz.langa'
components = ['Library (Lib)']
creation = <Date 2014-04-05.10:35:12.240>
creator = 'Claudiu.Popa'
dependencies = []
files = ['36751']
hgrepos = []
issue_num = 21159
keywords = ['3.4regression']
message_count = 24.0
messages = ['215589', '215603', '215622', '227788', '227794', '237658', '247619', '248544', '248545', '248563', '248564', '248565', '248566', '249790', '252860', '252862', '252936', '252938', '252939', '252940', '257662', '261711', '264678', '267277']
nosy_count = 8.0
nosy_names = ['fdrake', 'barry', 'rbcollins', 'Claudiu.Popa', 'lukasz.langa', 'lrowe', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21159'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[PCManticore]

The error message from the title is not very intuitive from a user point of view. For instance, in the following traceback, only the first one gives a meaning to this error, through the leaked KeyError, thus the user knows that home_dir1 is invalid and missing. But the second message with "Bad value substitution" doesn't emphasize the actual problem and you have to stare at it for a couple of moments to figure out what's happening and why. Maybe changing the "Bad value substitution" with a "Missing value" would suffice. This came up in bpo-19546.

Traceback (most recent call last):
  File "C:\Python34\lib\configparser.py", line 410, in _interpolate_some
    v = map[var]
  File "C:\Python34\lib\collections\__init__.py", line 789, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "C:\Python34\lib\collections\__init__.py", line 781, in __missing__
    raise KeyError(key)
KeyError: 'home_dir1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\configparser.py", line 1204, in __getitem__
    return self._parser.get(self._name, key)
  File "C:\Python34\lib\configparser.py", line 773, in get
    d)
  File "C:\Python34\lib\configparser.py", line 374, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
  File "C:\Python34\lib\configparser.py", line 413, in _interpolate_some
    option, section, rest, var)
configparser.InterpolationMissingOptionError: Bad value substitution:
        section: [Paths]
        option : my_dir
        key    : home_dir1
        rawval : /lumberjack

[bitdancer]

Another problem with that error message is that it seems confusing as to where the bad value *reference* came from, given that rawval doesn't seem to include any interpolation point.

[ambv]

Those exception messages are really old. I agree we should fix them, we have to think about possible compatibility breakage though. I'll look into it.

[ambv]

So the diff would look like this (please find it attached). It does two things:

• changes messages on InterpolationMissingOptionError and InterpolationDepthError to be more helpful to the end user
• fixes rawval to actually hold the raw value of an option and not the "rest after substitution" like before

Fred, R. David, I have two questions for you:

• the reliable method of getting the arguments of both exceptions is the .args attribute; this didn't change but the message did. It's unlikely anybody would be parsing the old message string, but even if there was I'm inclined to treat any code doing so as broken by design. Do you agree?

• without the diff, the rawval argument in those exceptions holds a value that exposes internal implementation and is not generally useful for user code. It wasn't exposed directly as an attribute in those exceptions (section, option and reference are). That being said, fixing this is a change in logic of sorts. Do you see any danger of third-party code relying on the old behaviour?

Actually, I have a hypothetical third question:

• Should I make sure that those exceptions are unpicklable by older releases of Python 3?

I'm asking because if there's no such expectation, we could add .rawval as a direct attribute to InterpolationMissingOptionError, and introduce reference to InterpolationDepthError (currently the exception message can say that "option O in section S contains an interpolation key that cannot be substituted" but it doesn't say which interpolation key).

Anyway, the first two questions are most important because they basically decide whether we can change the exceptions at all at this point. I'm inclined to say "yes", Python 3 did that with a number of exceptions both built-in and in the standard library.

[bitdancer]

We never promise that the messages won't change (they are not part of the API), so that part isn't a problem. We do try to be backward compatible there when it comes to args beyond the message text. I don't think unpickleability is an issue; at least there isn't any discussion of it in PEP-3161, and I don't remember any around the implementation.

I think this is fine. I don't think we should worry about someone who is actually using rawval/rest pulled out of the exception args. You could put a porting note in What's New, but my guess is that the chances of anyone being inconvenienced by this are pretty near to zero.

[PCManticore]

I find the new error messages clear and straight to the point. It would be nice if this would get into 3.5.

[rbtcollins]

LGTM - lukasz, do you want to commit this, or would you like someone else to if you're too busy? Looks like we should patch this in 3.4/3.5./3.6 at this point.

[python-dev]

New changeset 267422f7c927 by Robert Collins in branch '3.4':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/267422f7c927

New changeset 1a144ff2d78b by Robert Collins in branch '3.5':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/1a144ff2d78b

New changeset fb4e67040779 by Robert Collins in branch 'default':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/fb4e67040779

[rbtcollins]

I've applied this since it seems Lukasz was busy. Thanks for the patch Lukasz!

[serhiy-storchaka]

You forgot to merge default with 3.5.