traceback: add a new thin class storing a traceback without storing local variables

[gvanrossum]

BPO	17911
Nosy	@gvanrossum, @rhettinger, @terryjreedy, @ncoghlan, @pitrou, @vstinner, @rbtcollins, @ned-deily, @ericsnowcurrently, @vadmium, @1st1, @Martiusweb
Files	traceback.patch traceback2.patch: Better documentation, fixed edge cases linecache_1.patch linecache_2.patch frame_1.patch tb_stats_1.patch issue17911-1.patch issue17911-2.patch issue17911-3.patch issue17911-4.patch qualname_in_exceptions.patch: Prints qualnames in function call exceptions. issue17911-5.patch issue17911-6.patch issue-19711-8.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-03-16.22:17:38.516>
created_at = <Date 2013-05-05.21:31:26.143>
labels = ['type-feature', 'library', 'expert-asyncio']
title = 'traceback: add a new thin class storing a traceback without storing local variables'
updated_at = <Date 2016-06-29.05:00:26.293>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2016-06-29.05:00:26.293>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2015-03-16.22:17:38.516>
closer = 'rbcollins'
components = ['Library (Lib)', 'asyncio']
creation = <Date 2013-05-05.21:31:26.143>
creator = 'gvanrossum'
dependencies = []
files = ['30895', '30901', '37700', '37769', '37770', '37771', '37782', '37862', '37870', '38007', '38034', '38324', '38325', '38335']
hgrepos = []
issue_num = 17911
keywords = ['patch']
message_count = 77.0
messages = ['188467', '188530', '188548', '188554', '188556', '188616', '188620', '188648', '188719', '192672', '192690', '192728', '192909', '193272', '193294', '193296', '193333', '220381', '220390', '231743', '231792', '233895', '233995', '234297', '234298', '234299', '234319', '234338', '234712', '234740', '234741', '234765', '234769', '234777', '234781', '234782', '234788', '234795', '234796', '234797', '234804', '234812', '234820', '235365', '235428', '235444', '235446', '235455', '235456', '235472', '235480', '235483', '235517', '235569', '235570', '235574', '235578', '235658', '235720', '235872', '237166', '237217', '237220', '237224', '237225', '237228', '237229', '237230', '237231', '237232', '237233', '237237', '237339', '237842', '238169', '238248', '269464']
nosy_count = 16.0
nosy_names = ['gvanrossum', 'rhettinger', 'terry.reedy', 'ncoghlan', 'pitrou', 'vstinner', 'rbcollins', 'ned.deily', 'python-dev', 'eric.snow', 'martin.panter', 'mahmoud', 'yselivanov', 'adaptivelogic', 'martius', 'xonatius']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17911'
versions = ['Python 3.5']

[gvanrossum]

For Tulip I may have to extract tracebacks and store the extracted data, in order to avoid cycles created by the frames referenced in the traceback. I'll be extracting the traceback every time an exception is raised, and in most cases I won't be using the extracted info, because usually the exception is caught (or logged) by user code. But it's very important to ensure that if the user code doesn't catch or log it, I can log a traceback, and I won't know that this is the case until a destructor is called, which may be quite a bit later. (Reference: http://code.google.com/p/tulip/source/browse/tulip/futures.py#38)

Unfortunately it looks like the public APIs do a lot more work than needed. Ideally, there'd be something similar to _extract_tb_or_stack_iter() that doesn't call linecache.getline() -- it should just return triples of (filename, lineno, functionname), and enough structure to tell apart the __cause__, __context__, and __traceback__ (the first two possibly repeated). Given this info it would be simple enough to format and log the actual traceback, and storing just this would take less space and time than computing the lines of the fully-formatted traceback.

[pitrou]

I think instead we may want to add a finalize() or close() method on frame objects which would clear all local variables (as well as dereference the globals dict, perhaps), after having optionally run a generator's close() method (if the frame belongs to a generator).

If I'm not mistaken, it should allow breaking reference cycles, and remove the need for complex traceback processing, which Twisted currently also does: http://twistedmatrix.com/trac/browser/trunk/twisted/python/failure.py#L89

Note that generator cleanup through the frame has a patch in bpo-17807.

[gvanrossum]

I'm not snubbing my nose at something that breaks these types of cycles more easily, but I still think that the abstractions offered by the traceback module could be improved.

[gvanrossum]

On Mon, May 6, 2013 at 6:06 AM, Antoine Pitrou <report@bugs.python.org> wrote:

Note that generator cleanup through the frame has a patch in bpo-17807.

Sadly that doesn't help with my issue. I applied path gen3.patch and
ran various versions of the code I have. There's still something
holding on to the Task or the exception.

[pitrou]

On Mon, May 6, 2013 at 6:06 AM, Antoine Pitrou
<report@bugs.python.org> wrote:
> Note that generator cleanup through the frame has a patch in
> bpo-17807.

Sadly that doesn't help with my issue.

It won't. It's just a building block for the change I've proposed here.

[ncoghlan]

Antoine - I like your idea, but I think it's a separate issue. I agree with Guido that exposing some lower level non-formatting APIs in the traceback module would be helpful.

I see Guido's suggestion here as similar to the change we just made in the dis module to expose a dis.get_instructions iterator. That change makes it much easier to work with the disassembler output programmatically, whereas the module was previously too focused on displaying text with a specific format.

My current thoughts: define a "TracebackSummary" with the following fields:

exc_type
exc_repr
stack_summary
cause
context

stack_summary would be a list of (filename, lineno, functionname) triples as Guido suggested (probably a named tuple)

cause and context would be either None or a reference to an appropriate TracebackSummary object

Basically, the summary should contain all of the information needed to create the formatted traceback output, without actually doing any formatting (aside from calling repr() on the exception)

[gvanrossum]

That sounds great, except I'd expect the exception type and str(), since a formatted traceback uses the str() of the exception, not its repr().

Personally I don't think the tuples need to be named, but I won't hold up progress either. :-)

[pitrou]

You may want two pieces of stack: the piece of stack that is above and including the catch point, and the piece of stack that is below it. The former is what gets traditionally printed in tracebacks, but the latter can be useful as well (see bpo-1553375 for a related feature request).

[pitrou]

Ok, I've spinned off the frame clearing suggestion to bpo-17934.

[adaptivelogic]

I've been looking into this as an easy piece to bite off. If I understand Guido correctly, he'd like to defer or suppress the linecache call when getting the tb summary. The problem with deferring is that you need to access f_globals for the loader to work correctly when the module source is a non-file import source. If we keep a reference to f_globals for each line in the traceback, we can defer this to later (ideally we'd just keep the __loader__ value, but that would require changing the linecache interface as well).

My inclination would be to have another keyword argument to _extract_tb_or_stack_iter (terse=False or verbose=True - either style works). In terse mode, no source lines would be available, and the formatted output would be the same as if the source wasn't available at all. This would work, although the traceback module is structured so that I'd need to pass it through quite a few wrapped iterator calls.

I'm not sure how free a hand I have when it comes to refactoring the internal implementation. I'm not fond of the extractor callbacks - I'd prefer a generator-based approach on the lines of:

def _tb_iter(tb, limit):
    i = 0
    while tb is not None:
        if limit is not None and limit < i:
            break
        yield tb.tb_frame, tb.tb_lineno
        tb = tb.tb_next
        i += 1

def _extract_frame_iter(frames, terse=False):
    ...
    for frame, lineno in frames:
    ...

[gvanrossum]

In terms of how much freedom you have about changing the internal, I'd check how long ago they were changed. "Internal" APIs that have been stable for many versions tend to have illicit external uses -- but internal APIs that were introduced recently (e.g. in 3.2) are usually safe to use -- nobody is going to make too much of a stink if you break their code.

As for saving f_globals, if you're going to save an extra pointer anyways, why not just save the frame pointer?

[pitrou]

If we keep a reference to f_globals for each line in the traceback, we
can defer this to later (ideally we'd just keep the __loader__ value,
but that would require changing the linecache interface as well).

Or you could add a *new* linecache interface.
Keeping __loader__ sounds much less hackish than keeping f_globals to me.