difflib.HtmlDiff produces different output from difflib.ndiff

[SilentGhost]

BPO	27709
Nosy	@loewis, @terryjreedy, @j, @tirkarthi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-08-08.15:40:57.110>
labels = ['type-bug', 'library']
title = 'difflib.HtmlDiff produces different output from difflib.ndiff'
updated_at = <Date 2018-09-22.18:00:18.520>
user = 'https://bugs.python.org/SilentGhost'

bugs.python.org fields:

activity = <Date 2018-09-22.18:00:18.520>
actor = 'xtreak'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-08-08.15:40:57.110>
creator = 'SilentGhost'
dependencies = []
files = []
hgrepos = []
issue_num = 27709
keywords = ['patch']
message_count = 2.0
messages = ['272173', '272196']
nosy_count = 6.0
nosy_names = ['loewis', 'terry.reedy', 'eli.bendersky', 'SilentGhost', 'jlwing', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27709'
versions = ['Python 3.5', 'Python 3.6']

Linked PRs

• gh-71896: warn in doc about trailing newlines ignored by HtmlDiff #153930
• [3.15] gh-71896: Add a docs warning about trailing newlines ignored by difflib.HtmlDiff (GH-153930) #153965
• [3.14] gh-71896: Add a docs warning about trailing newlines ignored by difflib.HtmlDiff (GH-153930) #153966
• [3.13] gh-71896: Add a docs warning about trailing newlines ignored by difflib.HtmlDiff (GH-153930) #153968

[SilentGhost]

msg264842 in bpo-26945 reports an odd results of HtmlDiff.make_file, digging into it I've noticed couple of things: different output to one generated directly from difflib.ndiff and underlying issue further up the stack that generates same faulty output with a slightly modified files.

The patch I'm attaching fixes the issue with different outputs, from make_file and ndiff, by not removing EOL characters, I had to re-generate expected test output file.

[SilentGhost]

The degenerate behaviour appears ultimately due to the autojunk heuristic. Given that autojunk defaults to True and there isn't any way to change via most user-facing functions, I don't think there is an easy resolution that can be found here. I'm going to withdraw the earlier patch, since it feels like a half-measure which enable correct behaviour by accident.

[Lenormju]

As described in the #118150 issue and handled there, yes the autojunk behavior is surprising at times.
But there is also another issue here (reproduced using 3.13, 3.14 and 3.15):

import difflib

A = ["same\n", "diffme\n"]
B = ["same", "diffme\n"]   # line 0 differs only by trailing newline

print(difflib.SequenceMatcher(None, A, B).get_opcodes())  # [('replace', 0, 1, 0, 1), ('equal', 1, 2, 1, 2)]
print(list(difflib.ndiff(A, B)))  # ['- same\n', '?     -\n', '+ same', '  diffme\n']
print(list(difflib.unified_diff(A, B, lineterm='')))  # [..., '-same\n', '+same', ' diffme\n']
print(list(difflib.context_diff(A, B, lineterm='')))  # [..., '! same\n', ..., '! same', '  diffme\n']
print(difflib.HtmlDiff().make_file(A, B))  # no line is marked as changed/added/removed (classes diff_add/diff_chg/diff_sub)

A_stripped = [elem.rstrip() for elem in A]
B_stripped = [elem.rstrip() for elem in B]

print(difflib.SequenceMatcher(None, A_stripped, B_stripped).get_opcodes())  # [('equal', 0, 2, 0, 2)]
print(list(difflib.ndiff(A_stripped, B_stripped)))  # ['  same', '  diffme']
print(list(difflib.unified_diff(A_stripped, B_stripped, lineterm='')))  # []
print(list(difflib.context_diff(A_stripped, B_stripped, lineterm='')))  # []
print(difflib.HtmlDiff().make_file(A_stripped, B_stripped))  # still no line is marked as changed/added/removed

The HtmlDiff is the only one that does end-of-line stripping from the whole difflib. To me, this is a bug because it hides actual differences.
It is caused by the _tab_newline_replace method being called before _mdiff.

[Lenormju]

Petr Viktorin just told me we should NOT fix the issue (which dates back tot he original implementation in 2004, 22 years ago, cf e064b41). Instead, I will add a warning about it, and an automated test to prevent unexpected regression (if someone would fix the bug).