Package: tok 0.2.2.9000

Tomasz Kalinowski

tok: Fast Text Tokenization

Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.

Authors:Tomasz Kalinowski [ctb, cre], Daniel Falbel [aut], Regouby Christophe [ctb], Posit [cph]

tok_0.2.2.9000.tar.gz
tok_0.2.2.9000.zip(r-4.7-x86_64)tok_0.2.2.9000.zip(r-4.7-arm64)tok_0.2.2.9000.zip(r-4.6-x86_64)tok_0.2.2.9000.zip(r-4.6-arm64)tok_0.2.2.9000.zip(r-4.5-x86_64)
tok_0.2.2.9000.tgz(r-4.6-x86_64)tok_0.2.2.9000.tgz(r-4.6-arm64)tok_0.2.2.9000.tgz(r-4.5-x86_64)tok_0.2.2.9000.tgz(r-4.5-arm64)
tok_0.2.2.9000.tar.gz(r-4.7-arm64)tok_0.2.2.9000.tar.gz(r-4.7-x86_64)tok_0.2.2.9000.tar.gz(r-4.6-arm64)tok_0.2.2.9000.tar.gz(r-4.6-x86_64)
tok_0.2.2.9000.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
tok/json (API)

# Install 'tok' in R:
install.packages('tok', repos = c('https://mlverse.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/mlverse/tok/issues

On CRAN:

Conda:

rustcargo

6.34 score 48 stars 2 packages 19 scripts 526 downloads 20 exports 2 dependencies

Last updated from:f925ad65e3. Checks:15 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-arm64OK218
linux-devel-x86_64OK242
source / vignettesOK270
linux-release-arm64OK235
linux-release-x86_64OK226
macos-release-arm64OK241
macos-release-x86_64OK419
macos-oldrel-arm64OK253
macos-oldrel-x86_64OK557
windows-devel-arm64OK377
windows-devel-x86_64OK343
windows-release-arm64OK375
windows-release-x86_64OK316
windows-oldrel-x86_64OK331
wasm-releaseOK208

Exports:decoder_byte_levelencodingmodel_bpemodel_unigrammodel_wordpiecenormalizer_nfcnormalizer_nfkcpre_tokenizerpre_tokenizer_byte_levelpre_tokenizer_whitespaceprocessor_byte_leveltok_decodertok_modeltok_normalizertok_processortok_trainertokenizertrainer_bpetrainer_unigramtrainer_wordpiece

Dependencies:cliR6