add llamafile code snippet #1088
Conversation
not-lain
requested review from
SBrandeis,
gary149,
Wauplin,
julien-c,
pcuenca and
ngxson
as code owners
|
(launching the CI) |
Co-authored-by: Julien Chaumond <[email protected]>
| const LinuxAndMacCommand = () => { | ||
| const snippet = [ | ||
| "# Load and run the model:", | ||
| `wget https://huggingface.co/${model.id}/resolve/main/${filepath ?? "{{LLMAFILE_FILE}}"}`, |
There was a problem hiding this comment.
is LLMAFILE_FILE correct? Should it be LLAMAFILE_FILE? Where is it defined?
There was a problem hiding this comment.
Thanks a lot for raising this @pcuenca
this was the most confusing aspect of this PR, i found {{GGUF_FILE}} as well but it was not defined anywhere else in the repo, would love to get some inputs on this one.
maybe it's some kind of a jinja pattern of some sorts, if so let me know what other changes that needs to be made.
There was a problem hiding this comment.
simplest would be to get one example file ending with .llamafile (if we have the list of files here, i don't remember) – not sure we need to implement the same selector as for GGUF @mishig25 – maybe overkill here
There was a problem hiding this comment.
for 🦙📁 the file needs to have .llamafile somewhere in the filename but it might not need to end with the file_extension.
an example of this might be https://huggingface.co/Mozilla/Meta-Llama-3-70B-Instruct-llamafile/tree/main
for heavy files you might find that the cat[number] might be either at the end or right before the file_extension.
There was a problem hiding this comment.
Ah yes, I didn't realize we were planning to use a similar method as for gguf, makes sense.
There was a problem hiding this comment.
Looking at llamafile model repos on the hub: https://huggingface.co/models?library=llamafile, I only see a few from Mozilla which have more than 2-3 llama files.
Unless implementing a selector is trivial, I'd recommend that we pick up the first file/ or just name xyz.llamafile as we did in really early GGUF snippets.
There was a problem hiding this comment.
Yes^
(BTW we only detect *.llamafile files as library=llamafile in the Hub. That's fine IMO, let's keep things simple)
| const LinuxAndMacCommand = () => { | ||
| const snippet = [ | ||
| "# Load and run the model:", | ||
| `wget https://huggingface.co/${model.id}/resolve/main/${filepath ?? "{{LLMAFILE_FILE}}"}`, |
There was a problem hiding this comment.
Looking at llamafile model repos on the hub: https://huggingface.co/models?library=llamafile, I only see a few from Mozilla which have more than 2-3 llama files.
Unless implementing a selector is trivial, I'd recommend that we pick up the first file/ or just name xyz.llamafile as we did in really early GGUF snippets.
fixes : #871 and #848
as mentioned llamafile can work on both files ending with a
.llamafileextension andGGUFones as well.this pr will support only the
.llamafileextension files.more details can be found in this slack thread (internal)