print fails on unicode '\udce5' surrogates not allowed

[AtlePedersen]

BPO	13717
Nosy	@pitrou, @ezio-melotti

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-01-05.20:23:12.091>
created_at = <Date 2012-01-05.20:12:52.320>
labels = ['type-bug', 'invalid', 'expert-unicode']
title = "print fails on unicode '\\udce5' surrogates not allowed"
updated_at = <Date 2012-01-08.21:41:17.625>
user = 'https://bugs.python.org/AtlePedersen'

bugs.python.org fields:

activity = <Date 2012-01-08.21:41:17.625>
actor = 'Atle.Pedersen'
assignee = 'none'
closed = True
closed_date = <Date 2012-01-05.20:23:12.091>
closer = 'ezio.melotti'
components = ['Unicode']
creation = <Date 2012-01-05.20:12:52.320>
creator = 'Atle.Pedersen'
dependencies = []
files = []
hgrepos = []
issue_num = 13717
keywords = []
message_count = 4.0
messages = ['150684', '150685', '150686', '150910']
nosy_count = 3.0
nosy_names = ['pitrou', 'ezio.melotti', 'Atle.Pedersen']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13717'
versions = ['Python 3.1']

[AtlePedersen]

I've made a short program to traverse file tree and print file names.

for root, dirs, files in os.walk(path):
        for f in files:
                hex = ' '.join(["%02X"%ord(x) for x in f])
                print('file is',hex,f)

This fails with the following file:

file is 67 72 DCE5 6B 61 6C 6C 65 6E 2E 6A 70 67 2E 68 74 6D 6C Traceback (most recent call last):
  File "/home/atle/bin/findpictures.py", line 16, in <module>
    print('file is',hexa,f)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce5' in position 2: surrogates not allowed

I don't really understand the issue, but this works with Python 2, and fails using 3.1.4 (gentoo: dev-lang/python-3.1.4-r3)

Same code using Python 2.7.2 gives:
('file is', '67 72 E5 6B 61 6C 6C 65 6E 2E 6A 70 67 2E 68 74 6D 6C', 'gr\xe5kallen.jpg.html')

[ezio-melotti]

On Python 3, os.walk() uses the surrogateescape error handler. If the filename is in e.g. iso-8859-* and the filesystem encoding is UTF-8, decoding '\xe5' will then result in '\udce5', and '\udce5' can't then be printed because it's a lone surrogate.

See also http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables

[pitrou]

The file tree contains a file which has an undecodable character in it. It ends up mangled as specified in PEP-383.
Priting such filenames is not directly supported (since they have invalid characters in them), but you can workaround it in several ways, for example escaping all non-ASCII chars: print(ascii(f)).

(note that opening the file will still work fine; only outputting the filename without special care will fail)

Python 2 is different since it doesn't attempt to decode filenames at all, it just treats them as opaque bytes.

[AtlePedersen]

Just wanted to say thanks for very fast response, and informative information.

I respect your decision to close the bug as invalid. But my five cent is that it still feels like a bug, something that shouldn't happen. Especially since it's part of a very basic function, and very unpredictable for inexperienced Python programmers.

I do understand your headache. I've had my share of character set issues in my time.

But thanks again for the quick reply, and suggested workarounds, which will work well for me and my situation.