Allow changing difflib._file_template character encoding.

[josephoenix]

BPO	2052
Nosy	@terryjreedy, @ezio-melotti, @bitdancer, @berkerpeksag, @serhiy-storchaka
Files	issue2052.diff issue2052_html5.diff issue2052_html5_v2.diff issue2052_v2.diff issue2052_v3.diff issue2052_v4.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/berkerpeksag'
closed_at = <Date 2015-03-14.23:19:45.915>
created_at = <Date 2008-02-08.21:21:53.520>
labels = ['type-feature', 'library']
title = 'Allow changing difflib._file_template character encoding.'
updated_at = <Date 2015-03-14.23:19:45.914>
user = 'https://bugs.python.org/josephoenix'

bugs.python.org fields:

activity = <Date 2015-03-14.23:19:45.914>
actor = 'berker.peksag'
assignee = 'berker.peksag'
closed = True
closed_date = <Date 2015-03-14.23:19:45.915>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2008-02-08.21:21:53.520>
creator = 'josephoenix'
dependencies = []
files = ['35244', '35245', '35273', '38474', '38483', '38484']
hgrepos = []
issue_num = 2052
keywords = ['patch']
message_count = 15.0
messages = ['62208', '62209', '62211', '116949', '117234', '184726', '184751', '184755', '218479', '218726', '237383', '238050', '238097', '238104', '238105']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'josephoenix', 'ezio.melotti', 'r.david.murray', 'python-dev', 'berker.peksag', 'serhiy.storchaka', 'hashimo']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2052'
versions = ['Python 3.5']

[josephoenix]

When passed unicode strings, difflib.HtmlDiff.make_file and make_table
fail with a UnicodeEncodeError. Also, the html outputted by make_file
seems to be hardcoded to use charset=ISO-8859-1 (line 1584 of difflib.py)

[josephoenix]

Oops, please close this. Apparently was fixed in 2.5.1, and I'm just
behind.

[josephoenix]

After installing 2.5.1, the UnicodeEncodeError is gone, but the charset is
still hardcoded in difflib._file_template. So, I guess this is still a
separate bug.

[BreamoreBoy]

difflib._file_template is still hard-coded in py3k SVN. I'm unsure as to whether this is a feature request, a behaviour issue or not an issue at all, can someone please advise, thanks.

[bitdancer]

I believe that charset is the standard default for html, which would make this a feature request.

[terryjreedy]

In 3.2, it is line 1629:
content="text/html; charset=ISO-8859-1" />

That charset was only standard for Western European documents limited to that charset. Now, even such limited-char docs often use 'utf-8' (python.org does). The result of putting an incorrect charset designation in an html file is that the browser will not display the file correctly.

For instance, I tried an input sequence containing line 'c\u3333', which displays in IDLE as  'c㌳'. The string from HtmlDill.make_file() must be written to a file opened with encoding='utf-8', not the above or equivalent. Firefox then reads the three bytes of the utf-8 encoding as three separate characters and displays 'c㌳'. To check:
>>> 'c㌳'.encode().decode(encoding='Latin-1')
'cã\x8c³'

To me the clear implication of "returns a string which is a complete HTML file containing a table showing line by line differences with inter-line and intra-line changes highlighted." is that the resulting file will display correctly. The current template charset prevents that, changing to 'utf-8' results in a file that displays correctly (tested). So the current behavior and the code that causes it is to me clearly a bug. I would like to fix it before 2.7.4 comes out.

[terryjreedy]

After thinking about it more, the real problem is that the charset setting must match the chars used and how they re encoded, so no one setting is right for all uses. An alternative to changing the default in existing versions is to at least document what it is and explain how to work around it with .replace -- for instance output.replace('ISO-8859-1', 'utf-8'). I agree that adding a parameter (charset=xxx) is a new feature.

[ezio-melotti]

I haven't looked at the code, but if an HTML page is generated it should probably be updated to use HTML5 and <meta charset="utf-8">.