getpass.getpass() fails with non-ASCII characters in prompt

[Arfrever]

BPO	21169
Nosy	@loewis, @bitdancer, @serhiy-storchaka, @kushaldas
Files	getpass_test_python2 getpass_test_python3 issue21169.patch: Stops the breakage of nonascii chars in non utf-8 environment issue21169_v2.patch: Version2 of the patch with a test update. issue21169_v3.patch: Updated patch with discovering of currect locale and corresponding test case. issue21169_v4.patch: New patch with actual test case :) issue21169_v5.patch: Newer version of the patch with stream.encoding issue21169_v6.patch: New patchset. issue21169_v7.patch: Another patch with docs update and one line code comment.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-04-14.14:32:58.696>
created_at = <Date 2014-04-07.11:23:55.654>
labels = ['easy', 'type-bug', 'library']
title = 'getpass.getpass() fails with non-ASCII characters in prompt'
updated_at = <Date 2014-04-14.14:32:58.695>
user = 'https://bugs.python.org/Arfrever'

bugs.python.org fields:

activity = <Date 2014-04-14.14:32:58.695>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-14.14:32:58.696>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2014-04-07.11:23:55.654>
creator = 'Arfrever'
dependencies = []
files = ['34747', '34748', '34766', '34767', '34775', '34776', '34807', '34810', '34815']
hgrepos = []
issue_num = 21169
keywords = ['patch', 'easy']
message_count = 19.0
messages = ['215697', '215778', '215779', '215828', '215829', '216002', '216006', '216009', '216013', '216041', '216045', '216050', '216051', '216052', '216054', '216077', '216078', '216081', '216082']
nosy_count = 6.0
nosy_names = ['loewis', 'Arfrever', 'r.david.murray', 'python-dev', 'serhiy.storchaka', 'kushal.das']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21169'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[Arfrever]

getpass.getpass() fails with non-ASCII characters in prompt.

The attached example scripts (for Python 2 and 3) contain non-ASCII unicode prompt (Polish "hasło" == English "password") written in UTF-8.
Python-2 version fails always. Python-3 version fails in non-UTF-8 locale.

$ ./getpass_test_python2
Traceback (most recent call last):
  File "./getpass_test_python2", line 5, in <module>
    getpass.getpass(u"Hasło: ")
  File "/usr/lib64/python2.7/getpass.py", line 71, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/usr/lib64/python2.7/getpass.py", line 128, in _raw_input
    prompt = str(prompt)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0142' in position 3: ordinal not in range(128)
$ LC_ALL="en_US.UTF-8" ./getpass_test_python3
Hasło: 
$ LC_ALL="C" ./getpass_test_python3
Traceback (most recent call last):
  File "./getpass_test_python3", line 5, in <module>
    getpass.getpass("Has\u0142o: ")
  File "/usr/lib64/python3.4/getpass.py", line 78, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/usr/lib64/python3.4/getpass.py", line 138, in _raw_input
    stream.write(prompt)
UnicodeEncodeError: 'ascii' codec can't encode character '\u0142' in position 3: ordinal not in range(128)
$

[kushaldas]

Here is a patch which stops the breakage in getpass for python3.

[kushaldas]

Version 2 of the patch with a test update.

[kushaldas]

Updated patch with discovering of currect locale and corresponding test case.

[kushaldas]

New patch with actual test case :)

[serhiy-storchaka]

I don't think this is a bug. Any text output operation can fail when outputs unencodable string. You should use a stream with proper encoding and/or error handler.

[loewis]

I agree that it is not a bug if the device where the prompt is shown simply does not support the characters; on Unix, this includes cases where the locale does not support the characters.

Arfrever: when you say that it fails in Python 3 in a non-UTF-8 locale, which specific locale was that that it failed in?

[serhiy-storchaka]

$ LC_ALL=en_US.iso88591 ./python -c "print('\u20ac')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

$ LC_ALL=en_US.iso88591 ./python -c "input('\u20ac')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

$ LC_ALL=en_US.iso88591 ./python -c "import getpass; getpass.getpass('\u20ac')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 78, in unix_getpass
    passwd = _raw_input(prompt, stream, input=input)
  File "/home/serhiy/py/cpython/Lib/getpass.py", line 138, in _raw_input
    stream.write(prompt)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

[Arfrever]

Martin v. Löwis: In this case, device support non-ASCII characters, but Python's getpass module forgets to properly encode string. Message 215697 contains example with C locale.

[kushaldas]

Here is a new patch which uses stream.encoding instead getting the encoding from the locale as suggested by David. It also contains the new test.

[kushaldas]

New patchset with updated test, now sending ascii stream into the call as argument.