Misaligned unicode from tree.draw() in jupyter notebook

[brianzhang01]

Repro: https://github.com/brianzhang01/msprime_unicode/blob/master/msprime_unicode.ipynb

I have also observed this issue on Google Colab. The ASCII output is aligned fine (shown in notebook), and the SVG works too (not shown).

[brianzhang01]

So apparently Unicode character width can either be 1 or 2 Latin characters of width. It seems like Jupyter notebook sets all Unicode characters to be rendered as width 2 by default. There may or may not be a way to alter this.
https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html#unicode-formatting

[brianzhang01]

One approach would be to add an option into the Unicode drawer that converts regular ASCII characters into full-width versions: https://stackoverflow.com/questions/4622357/how-to-control-padding-of-unicode-string-containing-east-asia-characters

[hyanwong]

I have had problems in the past with unicode font substitution. But I agree that having format="unicode" working in a Jupyter notebook would be nice, and a reason to keep the ascii format until this is working (although it's pretty easy to display SVG in a juptyer notebook, so I feel it isn't that strong an argument)

[jeromekelleher]

I think this is a browser/font issue - when I view notebooks you've linked to they all look fine. My guess is that on Macs the browser doesn't use a proper fixed-width font when it sees unicode chars, but does for some reason with ascii. I'm guessing that you're using a Mac @brianzhang01?

[jeromekelleher]

Here's what the trees look like for me on Vivaldi Linux:

[jeromekelleher]

Firefox linux:

[jeromekelleher]

Would you mind trying this on a few browsers on Macs @brianzhang01, so we can get a handle on how much of a problem this is? I know that these trees don't render properly on Android either.

If it is a browser font issue, I don't think there's much we can do about it, and all we can do is recommend that people use SVG trees in browser environments.

[hyanwong]

If it works on Jerome's browser I strongly agree that it is likely to be a font substitution issue on Macs, which also occurs e.g. in the html docs and sometimes in the terminal too.

[brianzhang01]

Yes, I am using Mac OS X Sierra. Agreed that this is likely a Mac font issue.

The misalignment with the linked GitHub notebook is an issue for me on both Chrome and Safari. Interestingly, on Chrome and Safari, the trees here show up fine: https://msprime.readthedocs.io/en/stable/tutorial.html.

If I copy either the GitHub or msprime documentation Unicode tree into Sublime Text, it's also misaligned there.

Having a Markdown code environment makes things a bit better, but there are still some slight alignment issues.

I will try to do a few more tests like iOS and Firefox on Mac OS X.

[jeromekelleher]

It is almost certainly a font spacing issue - sadly, proper monospaced fonts seem to be a rare thing outside of Linux. It'd be good if you could collect together a few more examples @brianzhang01, thanks.

[brianzhang01]

I was at a public library with Windows (10 I think, but could be 8), so I tested there and also got issues that seemed identical to the ones on Mac. The GitHub notebook preview was off on Google Chrome, and the Markdown was off in Google Chrome and Internet Explorer. (IE wasn't able to show the GitHub notebook preview, since I think GitHub no longer supports IE.) Below are two photos of Chrome.

I also get misalignment on iOS Safari; here is a screenshot.

[jeromekelleher]

Thanks very much @brianzhang01, this is helpful. I think these are all clearly monospace font issues. Keeping the ASCII tree drawer will make it a lot harder to add new features to the text rendering engine (and therefore much less likely to happen). I think the only thing we can do here really is change all the trees shown in the docs to SVG, and put in a note about this know browser alignment issue in the docs.

Does this sounds reasonable to you?

[brianzhang01]

A few notes:

• The new SVG features sound exciting, and I already noticed the labelling of mutations.
• I'll see if I can invest a bit of time into a fix for the Unicode alignment issue. I posted this link earlier, which I think is one reasonable approach. The issue is not really the existence of monospace fonts. It's that some East Asian Unicode characters require double-width to display. On Linux, perhaps only this subset of characters is shown in double-width, but on Mac and Windows, they show what are known as ambiguous width characters in double-width as well. The proposed solution is to add an option that changes all characters, such as digits 0-9 and the space character, to double-width.
• I'm having trouble intuiting why it would be easier to add certain features to the Unicode version rather than the ASCII version. Is it more just the burden of maintaining two text formats?
• I mentioned this earlier, but strangely, the trees in the docs (msprime, tskit) show up aligned for me in Chrome and Safari on Mac. So I'm not sure if we need to change the docs to SVG, but we could if we think it's a more compelling demo.

[jeromekelleher]

I'll see if I can invest a bit of time into a fix for the Unicode alignment issue.

This would be great, thanks @brianzhang01. If the issue is to do with the ambiguous width characters then we should definitely try to fix it. I had always assumed that the issue was that spaces took up slightly less space than the other chars, leading to this misalignments.

I'm having trouble intuiting why it would be easier to add certain features to the Unicode version rather than the ASCII version. Is it more just the burden of maintaining two text formats?

Basically, yes. I'm in the middle of rewriting the Text drawing code at the moment, and it is greatly simplified by not having to worry about ascii (bytes) and unicode (str) issues. Using plain ASCII characters in unicode output, rather than the box drawing characters wouldn't be very hard though.

I mentioned this earlier, but strangely, the trees in the docs (msprime, tskit) show up aligned for me in Chrome and Safari on Mac. So I'm not sure if we need to change the docs to SVG, but we could if we think it's a more compelling demo.

This is weird. It would be good to get to the bottom of this problem!