NEWS
tok 0.1.5
tok 0.1.4 (2024-09-04)
- Updated libR-sys to fix mac oldrel notes. (#18)
- Report rust version during compilation.
tok 0.1.3 (2024-07-06)
tok 0.1.2 (2024-06-27)
- Added support for authentication. (#7)
- Added support for training tokenizers from scratch. (#8)
- Added support for serializing tokenizers. (#8)
- Implemented support for configuring padding and truncation. (#9)
- Added support for decoders, post_processors and normalizers. (#10)
- Added support for
get_vocab_size(). (#10)
- Updated extendr version to avoid non-API function calls. (#11)
tok 0.1.1 (2023-08-17)
- Dependencies are now vendored to avoid versioning problems on CRAN. (#5)
tok 0.1.0 (2023-07-06)