discojs-core/models: add gpt

[tharvik]

add GPT model, tracked in #641

• add text loader from Hugo
• add "wikitext" task, the LLM reference
• add models.GPT, wrapping @peacefulotter code
• do not add real tokenization as Add tokenization support to Disco LLMs #646 takes care of that
• add /datasets in place of /example_training_data
  • a script takes care of getting the data, it should be replaced by submodules & LFS, see put ressources in repo instead of remotely #648

this is a prototype that is not being tested (as we need tokenization to get meaning out of it), only that the training of the model is reducing loss

[martinjaggi]

amazing work, thanks!

i just left minor comments

[martinjaggi]

add a comment what tokenizer is used (type, and pretrained tokenizer downloaded from ...)

[martinjaggi]

and also a comment on what is to expect (like if you train alone for 5mins or 5h, you should expect this train/test loss value roughly, and that the resulting model will somehow sound english?)

[tharvik]

there isn't really tokenization happening: gpt-tokenizer is unable to be compiled in the webapp (probably due to the lack of #632). currently, it takes wikitext as a stream of characters and pass it trough GPT.convertCharDataset to have the correct shape (don't know what its purpose is but it was required to make it work).

on the duration of training, I can only say that the loss is reducing. I hope that it'll write some correct english but I've no test to prove it.

hopefully, #646 will help us make it work and to general try it out.

[peacefulotter]

I think running gpt-micro on Shakespeare for 1 epoch should already produce some "shakespearesque" output

[martinjaggi]

comment somewhere on where the tokenizer came from? (from the task definition maybe?)

[peacefulotter]

I wonder if the tokenizer should be part of the task definition or be abstracted away. Is there really a use case for Disco where one might change this parameter? (Or even has the knowledge about what a tokenizer is)

[martinjaggi]

comment that 2 bytes will be one token id? BTW is it much more mem overhead to use an array (or stream) of int (to represent the token stream input) than the 2 bytes here?

[peacefulotter]

I a not sure if my code is wrong but it is supposed to pull the exact number of bytes it needs for one batch. So chunkSize is the number of bytes in the read buffer not number of integers (4 bytes)

[JulienVig]

Great work thank you Valérian! I left some questions and opinions but you can proceed with the merge