Fix pydoc crashing on unicode strings

[cben]

BPO	1065986
Nosy	@loewis, @rhettinger, @cben, @vstinner, @benjaminp, @merwok, @akitada, @bitdancer, @taschini
Files	PYDOC-UNICODE.diff: Path to Lib/pydoc.py to use unicode when needed issue1065986.patch issue1065986-2.patch issue1065986-3.patch issue1065986-4.patch issue1065986-5.patch issue1065986-6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-01-05.20:42:28.609>
created_at = <Date 2004-11-14.01:36:55.000>
labels = ['type-bug', 'library']
title = 'Fix pydoc crashing on unicode strings'
updated_at = <Date 2014-01-05.22:14:27.082>
user = 'https://github.com/cben'

bugs.python.org fields:

activity = <Date 2014-01-05.22:14:27.082>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2014-01-05.20:42:28.609>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2004-11-14.01:36:55.000>
creator = 'cben'
dependencies = []
files = ['6363', '25380', '31793', '31832', '32721', '32738', '33316']
hgrepos = []
issue_num = 1065986
keywords = ['patch']
message_count = 31.0
messages = ['47289', '47290', '47291', '47292', '158138', '158141', '158142', '159140', '159141', '159160', '159167', '159452', '169209', '175304', '175305', '175313', '175329', '197886', '197992', '198186', '198201', '203482', '203486', '203487', '203530', '203548', '207342', '207364', '207399', '207400', '207403']
nosy_count = 13.0
nosy_names = ['loewis', 'ping', 'rhettinger', 'cben', 'vstinner', 'benjamin.peterson', 'eric.araujo', 'akitada', 'r.david.murray', 'mu_mind', 'python-dev', 'ness', 'taschini']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1065986'
versions = ['Python 2.7']

[cben]

The pydoc module currently only outputs ASCII and
crashes with UnicodeEncodeError when printing a unicode
string (in contexts where it prints the str rather than
a repr, e.g. docstrings or variables like
__credits__). The most ironic example of it is that
since patch 1009389 was committed, pydoc.py pydoc
crashes on its own __credits__!

This patch changes pydoc help functions to return
unicode strings only when needed; it returns ASCII
strings if all characters are from ASCII. Therefore
there should be no compatibility problems.

For output, all pager functions were changed to encode
to the locale's preferred encoding and HTML output was
changed to always use UTF-8.

cgitb.py, DocXMLRPCServer.py and/or
SimpleXMLRPCServer.py seems to rely on pydoc to some
degree. I didn't touch them, so they might still be
broken in this respect.

[loewis]

Logged In: YES
user_id=21627

This is a too major change so short before the 2.4 release,
so postponing it 2.5.

[ping]

Logged In: YES
user_id=45338

I'm so sorry this has caused so much trouble.
The silly moose comment is my fault; it can be removed.

[rhettinger]

Logged In: YES
user_id=80475

I believe this was fixed. Feel free to re-open is something
is unresolved.

[ness]

Hello,

[this is my first bug report, so I'm sorry if I'm not adhering to some conventions]

in what versions of python is this supposed to be fixed? Consider:

% python
Python 2.7.2+ (default, Nov 30 2011, 19:22:03) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydoc import pager
>>> from locale import getpreferredencoding
>>> expr = u'\u211a'
>>> pager(expr) # error
>>> pager(expr.encode(getdefaultencoding())) # works

The error is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pydoc.py", line 1318, in pager
    pager(text)
  File "/usr/lib/python2.7/pydoc.py", line 1332, in <lambda>
    return lambda text: pipepager(text, os.environ['PAGER'])
  File "/usr/lib/python2.7/pydoc.py", line 1359, in pipepager
    pipe.write(text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u211a' in position 0: ordinal not in range(128)

Best,
Tom

[bitdancer]

It is fixed in Python3. Apparently Raymond was wrong about it having been fixed earlier (or perhaps he was referring to the unicode being removed from the pydoc __credits__ string).

[ness]

I see. Thank you.

On 12.04.2012 16:08, R. David Murray wrote:

R. David Murray<rdmurray@bitdance.com> added the comment:

It is fixed in Python3. Apparently Raymond was wrong about it having been fixed earlier (or perhaps he was referring to the unicode being removed from the pydoc __credits__ string).

----------
nosy: +r.david.murray

---

Python tracker<report@bugs.python.org>
<http://bugs.python.org/issue1065986\>

---

[taschini]

Shouldn't this be reopened for Python 2.7 ?

[bitdancer]

I don't think so. We aren't promising unicode support in pydoc in 2.x, and it is too late to add it.