NEWS

---

tok 0.2.2.9000

• Lowered minimum supported rust version (MSRV) from 1.91 to 1.77.2, at request of CRAN (#29).

tok 0.2.2 (2026-04-22)

• Updated upstream tokenizers library to 0.22.2.
• Update extendr-api to 0.9.0.
• Changed maintainer to Tomasz Kalinowski.

tok 0.2.1 (2025-09-30)

• Fixed tiny documentation issue for normalizers.

tok 0.2.0 (2025-08-27)

• Updated upstream tokenizers to 0.20.3
• Update extendr-api to 0.8.1

tok 0.1.5

• add message translation in FR (#19, @cregouby)

tok 0.1.4 (2024-09-04)

• Updated libR-sys to fix mac oldrel notes. (#18)
• Report rust version during compilation.

tok 0.1.3 (2024-07-06)

tok 0.1.2 (2024-06-27)

• Added support for authentication. (#7)
• Added support for training tokenizers from scratch. (#8)
• Added support for serializing tokenizers. (#8)
• Implemented support for configuring padding and truncation. (#9)
• Added support for decoders, post_processors and normalizers. (#10)
• Added support for get_vocab_size(). (#10)
• Updated extendr version to avoid non-API function calls. (#11)

tok 0.1.1 (2023-08-17)

• Dependencies are now vendored to avoid versioning problems on CRAN. (#5)

tok 0.1.0 (2023-07-06)

• Initial CRAN submission.