Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FAQ (please read before opening a new issue) #98

Open
hauntsaninja opened this issue Apr 5, 2023 · 0 comments
Open

FAQ (please read before opening a new issue) #98

hauntsaninja opened this issue Apr 5, 2023 · 0 comments

Comments

@hauntsaninja
Copy link
Collaborator

hauntsaninja commented Apr 5, 2023

Usage help

Check out this awesome tokeniser app https://tiktokenizer.vercel.app/ built by Diagram!

Check out the OpenAI cookbook!

In particular, the following are great examples of using tiktoken:

Other useful resources:

Support for other languages

Publishing an official NPM package

I have no current plans to do this, but it is likely that in the future OpenAI will provide an official package. Discuss at #22

Publishing an official Rust crate

In the future when I have time, I will publish the core BPE logic as its own crate. Discuss at #24

I'm short on time currently, but in the longer term I want to explore ways to make maintaining downstream bindings easier. Apologies to the people who've reached out about this — y'all are great, sorry for being a bottleneck.

Packaging

Issues with pyinstaller

I've never used pyinstaller, but people report --hidden-import=tiktoken_ext.openai_public --hidden-import=tiktoken_ext helps make things work. See #43 for more details.

Wheels for other platforms

I'm reluctant to publish wheels for additional platforms, particularly if I can't easily build or test code for them. I'm unlikely to do this unless there's significant interest. Please add reactions to the following issues if you'd like to see those wheels. No promises, but I'll reopen any issue that gets >10 reactions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant