Incorrect word wrapping with Unicode quotes

[PeteX]

• VSCode Version: 1.13.0
• OS Version: Ubuntu 16.04

When word wrapping is enabled, the editor should only break lines at space characters. (More precisely, I suppose it should break lines at any of the Unicode space characters which are not specifically designated as non-breaking.) Unfortunately at the moment it also breaks lines between alphabetic characters and adjacent Unicode quotes. For example, it will not break don't because it only contains an ASCII quote. However, it will break don’t because it contains U+2019, the right single quotation mark.

Steps to Reproduce:

1. Turn on word wrap in a buffer, with the "editor.wordWrap" configuration option.
2. Enter a word with a Unicode quote, for example by copying don’t from the paragraph above.
3. Enter more text to the left of the word don’t until it wraps.
4. Observe that it first wraps by moving the t onto the next line, leaving the rest of the word behind.

[alexdima]

It is possible to configure the wrapping rules using these "hidden" editor options:

Here are their default values:

"editor.wordWrapBreakBeforeCharacters": '([{‘“〈《「『【〔（［｛｢£¥＄￡￥+＋',
"editor.wordWrapBreakAfterCharacters": ' \t})]?|&,;¢°′″‰℃、。｡､￠，．：；？！％・･ゝゞヽヾーァィゥェォッャュョヮヵヶぁぃぅぇぉっゃゅょゎゕゖㇰㇱㇲㇳㇴㇵㇶㇷㇸㇹㇺㇻㇼㇽㇾㇿ々〻ｧｨｩｪｫｬｭｮｯｰ’”〉》」』】〕）］｝｣',
"editor.wordWrapBreakObtrusiveCharacters": '.',

If the defaults are incorrect, a PR is most welcome in https://github.com/Microsoft/vscode/blob/ba9a4314e1e25cbd135a6f0d5ea855e8f437d104/src/vs/editor/common/config/editorOptions.ts#L2079

Additionally, here is the word wrapping algorithm if you think any improvements can be made https://github.com/Microsoft/vscode/blob/ba9a4314e1e25cbd135a6f0d5ea855e8f437d104/src/vs/editor/common/viewModel/characterHardWrappingLineMapper.ts#L78

[PeteX]

Thanks for the information about the hidden options, that lets me stop the editor splitting on apostrophes.

I suppose wrapping source code is different to wrapping plain text. With plain text you really only want to split at white space. (You can I suppose hyphenate words if you really want to be clever, but I feel that's more appropriate behaviour for a word processor than a text editor.) With source code there may be other logical places to split, such as before the opening bracket in a function call.

I am puzzled to know why the Unicode quote marks are in the break characters, though. Although quite a few languages will accept Unicode source code now, I don't know of any that use Unicode quotes as a language feature. Because of this, the only place I would expect them to turn up is in strings. As these strings are typically natural language text, it's unhelpful to break at apostrophes, just as it is with my Markdown file.

Another thing I've noticed with editing English text is that the editor doesn't collapse white space around a line break. This means that if you stick to the old typewriter convention where you follow a period with two spaces, one of these spaces can end up as an indentation on the following line. For example:

Lorem ipsum.  Lorem ipsum.

can get wrapped as:

... Lorem ipsum.
 Lorem ipsum.`

Obviously this is much less important than splitting at apostrophes but I thought I'd mention it as I'm talking about word wrap generally.

I'm curious to know what you think about this. Would the changes I'm talking about cause problems in other areas? It's difficult to know if I've thought of all the possibilities. Anyway thanks again for the information about the hidden editor options, that solves the immediate problem.

[alexdima]

I think not collapsing whitespace is ok given our primarily being a source code editor. The wrapping algorithm crams as much whitespace as possible on a line, but if the whitespace would overflow, it overflows over to the next line. This is a somewhat conscious choice, as this might be a string in a source code where it would be unexpected for whitespace to disappear. I know this is sometimes at odds with what one would expect from a text editor, but I think it is a decent trade-off and I hope it is not too annoying when authoring .md or .txt files

---

👍 I think it is safe to remove ‘ from the default wordWrapBreakBeforeCharacters. PR most welcome!

[PeteX]

Here is a pull request. I've only done this once before so I hope I've put it together right.

Regarding the white space, I tend to agree with you that for source code, it's most helpful if it wraps to the start of the next line. As such I imagine the best approach would be to make it configurable, so it can be one thing for source code buffers, and something different for Markdown and similar things. I'm not sure when I'd get time to do such a big change, but would you consider a pull request that made it work in that way?

Thanks for your comments and encouragement.

[alexdima]

Thank you for the PR, I just merged it in ❤️ !

I would suggest we live with the current whitespace handling. The wrapping algorithm can be tweaked to allow for overflowing whitespaces, but:

• there is currently no language option that distinguishes "source code" languages from "plain text" languages (like markdown), so we would need to first introduce a mechanism that a language can be described as "textual" to turn this tweak on.
• there is a large likelihood that there will be horizontal scrollbars for the whitespace that overflows. Since the wrapping algorithm is not perfect (i.e. does not use actual font rendering/layouting rules), it is possible that in some cases the wrapping points lead to text overflowing horizontally. In such cases it is desirable to get horizontal scrollbars, and that is the current behaviour. We will then need to distinguish the overflowing whitespace case from the overflowing text due to font issues case, which could be tricky.

Given the extra effort to cover the overflowing whitespace, and that markdown and plaintext are important scenarios for a code editor, but no the core scenario (editing source code), I would suggest we live with the current implementation.

[PeteX]

Thanks for merging the PR. I don't really have time to work on the white space wrapping at the moment, even if we had an achievable design, so let's not worry about that. If I ever get the urge to try and improve it, I'll come back with suggestions.