Add tokenization and prompting API to GPT models

[JulienVig]

Fixes #646

• Adds tokenization support, allowing to choose any pre-trained tokenizer from Transformers.js
• Implement text generation with a trained model
• Extend text processing with shuffling and left padding
• Create an example of language model training in docs/examples/wikitext.ts
• I encountered a lot of issues and limitations which I listed in Improve and rework GPT-tfjs #654

[JulienVig]

@tharvik @peacefulotter I'd be happy to hear your take on how to integrate tokenization and prompting into the Disco architecture.
The initial commit is a POC of a pipeline from text training data (so not already preprocessed) to textual output (cf docs/example/wikitext.ts)
I wanted to make the tokenizer as a trainingInformation field in the infamous TaskProvider but it seems that complex objects (like functions) are removed from the task object when communicated between the server and the client...
My next step is to migrate away from gpt-tokenizer to use Transformers.js such that we can specify which pre-trained tokenizer we want to use as a string in the trainingInformation

[peacefulotter]

Congrats on the LLM POC milestone!

Firstly, I am wondering if having a custom tokenizer is even really relevant in the first place. Is there really a use case for it in Disco?

If so, instead of passing the tokenizer object, you could pass a string representing the tokenizer to use. Then an object could map tokenizer ids / names to the corresponding instance.

[peacefulotter]

Secondly, since it appears tokenization needs to be done on the fly (before or during training) and not as a completely separate step. It could be done as a preprocessing step like Hugo planned to do on his branch

[JulienVig]

@peacefulotter yes I want to do both points you mentioned! The second one is already implemented, the tokenization is part of the preprocessing.
And for the first I want to migrate to Transformers.js for that reason, it allows loading different pre-trained tokenizer from a string (rather than an import like gpt-tokenizer)

[tharvik]

I'd be happy to hear your take on how to integrate tokenization and prompting into the Disco architecture.

I don't like Task, it's the goto place for every context variable in Disco. here are some questions before adding it to task

• is it applicable to every model? ie, can images be tokenized?
• would someone want to change the tokenizer of wikitext?

after a bit more thinking, I don't think it really matters for now, as long as #639 is not done

I wanted to make the tokenizer as a trainingInformation field in the infamous TaskProvider but it seems that complex objects (like functions) are removed from the task object when communicated between the server and the client...

indeed, functions can't be represented by msgpack/json so it gets removed. it'll be nice to have a serialization process to ensure that we get function back from the network (a string such as gpt-tokenizer/davinci in the serialized object that get mapped to a known function)

[JulienVig]

Transformer.js requires Node >= 18, I'm waiting on #653 to be merged

[tharvik]

great work! a bit of nitpicking here and there but nothing really vital

[tharvik]

I've created a new models/tokenizer.ts file

LGTM!

a few more comments (I'm a never ending stream of critics, please stop me)