Terminal does not render true color

[Tyriar]

See microsoft/vscode#18544

[cnsumner]

Can't wait for this. I've been hacking away at hterm trying to get truecolor support but I feel like this is a way better option.

[Tyriar]

Implementing this will need to be done here https://github.com/sourcelair/xterm.js/blob/365a9f949cc1acfb6c4649ee1d85da3bbc60f928/src/InputHandler.ts#L1276

The tricky part is determining how we store the colors against the character(s) and then how to render it (probably inline style attributes). Something like #450 would probably simplify the adding of the attributes.

[cnsumner]

I may take a look at it when I get some time, but I can't say I'm too familiar with control sequences.

I took a look at http://invisible-island.net/xterm/ctlseqs/ctlseqs.html but couldn't seem to find anything about true color or 24-bit color. Am I missing something, or is there some other resource that covers this?

[jerch]

@cnsumner xterm currently cant display true color but has support for one of the proposed escape sequences. You can read more about this here: https://gist.github.com/XVilka/8346728
BTW this is partly specified in the ECMA-48 standard.

[Tyriar]

Put together a messy proof of concept to get a better idea of what we need to do here https://github.com/Tyriar/xterm.js/tree/484_truecolor

Things that need to be done:

• The branch currently adds another number to each character representing it's 'truecolor' in the format 0x1RRGGBB (fg) or 0x0RRGGBB (bg), which fits in a 32 bit integer

• It needs to support both fg and bg colors, and be able to mix truecolor and regular color

• curTrueColor is super ugly

• Support inverse

• Color flags need to be cleared when cells are reused

• It might be time to rethink how characters are encoded, the current format is:

  [attribute, character, width]
  

  • attribute: A 32-bit integer that holds ascii fg, ascii bg and flags (bold, underline, blink, inverse, invisible), not all bits are used
  • char: Either the empty string (for 0 width chars that pad wide chars) or a single character unicode string
  • width: The width of the char, currently can be 0, 1 or 2 but this could include more to support tabs properly Support rendering of tab characters in terminal #734

• The eventual solution should be compact for memory and also quick to access, sticking with an array, binary flags and bit shifting for data is probably the best idea.

[jerch]

@Tyriar
Yeah memory usage will explode if it is not optimized for low consumption. Maybe it can be packed together with width, this field is unlikely to get any bigger than 16 consuming only 4 bytes (tab defaults to 4 or 8 normally). This leaves room for one RGB definition* (assuming width is a 32bit integer). Not sure how many bits are free in attribute, maybe there can go the second RGB definition*. The 256 color definitions could end up in one of those since they can't be defined simultaneously with RGB on one cell.
(*) A cell will need room for two RGB definitions for foreground and background.

[Tyriar]

It looks like attributes currently consumed 9 bytes for bg, 9 bytes for bg and 5 bytes for flags, leaving plenty to squeeze width in at the end. The RGB definitions need 24 bytes (8*3) each.

Here's the (slightly complicated) space saving approach I think you're suggesting:

[char, attr1, attr2]

Where attr1 is (29 bits):

• width: 4 bits
• is 256 color or truecolor flag: 1 bit
• bg color: 24 bits

And attr2 is (30 bits):

• flags: 5 bits
• is 256 color or truecolor flag: 1 bit
• fg color: 24 bits

---

A simpler approach would use a number specifically for attributes:

[char, attr, truecolorBg, truecolorFg]

Where attr is:

• width: 4 bytes
• flags: 5 bytes
• 256 bg color: 9 bytes
• 256 fg color: 9 bytes

---

I want so badly to encapsulate these details as they really complicate things, I wonder if typescript will let us put getters on an array somehow. Using an object was 20x slower than an array based on a micro benchmark I did on jsperf, I know they're not always reliable but this is kind of what I expected as overhead to create an object based on a prototype.

Whatever we do, the format needs to be documented very well. Right now you have to figure it out yourself by looking at InputHandler.charAttributes.

[Tyriar]

Some more thoughts on memory saving:

• If a character has a width of > 1, we don't really need to store any data at all for the following characters. The "empty characters" are there as a workaround and would probably work fine being null.
• Normally a set of characters is styled, meaning attributes could be shared between a sequence of characters.
• Attributes could be cached and each char uses a pointer to one of the cached values,

[jerch]

Yes I think on most JS engines arrays with index access are still faster than objects and their property resolving. V8 got some extra optimizations for objects though. Construction and GC wise an array is faster due to the much simpler memory layout (this disadvantage can be omitted by reutilizing existing objects).

I have no clue about the right layout here, problem is that any higly packed stuff will raise runtime penalty due to the needed transformations. (This is one reason not to use bitfields in C if memory is no concern - the needed shift operations will pollute the instruction cache leading to cache misses - the code runs much slower).
Imho only tests will show what works best here.

[Tyriar]

@jerch interesting, I have 4 number version for simplicity right now in #756.

So memory here is expected to be around: 5 * (1000 scrollback) * (8 bytes on 64-bit machine) * 80 = 3.2mb (before overhead of js array). The actual width of a typical VS Code terminal would see that double if we keep the current format. Seems pretty steep now that I look at it, especially since users will often have higher than 1000 scrollback and multiple active terminals. The trailing whitespace which really doesn't add much at all and could just as easily be null which would probably be a pretty huge win.

The attribute/color set cache idea also seems promising, as long as there's a good way for fast access. A lightweight splay tree-like structure would be ideal (fast access for attributes uses most frequently).

It might also be worth looking into reusing line data arrays when they're trimmed instead of just discarding them (eg. here).

[jerch]

Yeah and with the engine overhead in mind it grows even bigger - everything in an array or object is stored as a reference, basically a pointer to the content. Dense js-arrays have a slight advantage here, they are formed as a C like array somewhere down the path (well actually ArrayLists). Sparse arrays and others objects are build with some tree magic to solve the property resolution.
So for 64bit the reference thing adds 5 * 8 more bytes per cell (it basically doubles your calculation above - 6.4mb). The content can add even more bytes beside the "real content", but this depends on the actual implementation of the Integer and String type.

Maybe a TypedArray can solve this. There the integer stuff can reside directly at the index position in memory without an additional indirection. asm.js basically operates only on those.

[jerch]

Here is a quick micro benchmark to compare TypedArray and js-Array for creation and data insertion: http://jsbench.github.io/#af3ea1056a70df2aa6775699d174df0d

The TypedArray runs 10 times faster with FF and Chrome on my machine. This might be due to the preallocation, have not tested this with js-Array since it is discouraged. Not sure if the JITs spot the fact, that the value just increases by 1 every 8 bytes and even optimize this out. Then the benchmark is useless at all.

The access is slightly worse for a TypedArray (maybe due to Number conversion), memory consumption is much better with a TypedArray.

Downside of a TypedArray is the static block of memory when it comes to spare data (tricks to avoid data at all wont work here, the memory is "lost" anyways) or resizing with content wrapping. It is hard to get it right and likely to involve copying big chunks of data over and over.