Skip to content

Releases: Systemcluster/kitoken

0.10.1

20 Dec 07:12
@Systemcluster Systemcluster
v0.10.1
e47fe20
This commit was signed with the committer’s verified signature.
Systemcluster Chris
GPG key ID: 5AEFFB51A0D6720D
Learn about vigilant mode.
Compare
Choose a tag to compare
0.10.1 Latest
Latest

Crates.io NPM PyPI

This release adds support for conditional normalization to match the recent changes in Hugging Face Tokenizers.

Web version of the tokenizer is available on kitoken.dev.

Assets 2
Loading

0.10.0

17 Dec 00:03
@Systemcluster Systemcluster
v0.10.0
e8213f7
This commit was signed with the committer’s verified signature.
Systemcluster Chris
GPG key ID: 5AEFFB51A0D6720D
Learn about vigilant mode.
Compare
Choose a tag to compare
0.10.0

Crates.io NPM PyPI

Initial release.

Web version of the tokenizer is available on kitoken.dev.

Assets 2
Loading