Package: tokenizers.bpe
Type: Package
Title: Byte Pair Encoding Text Tokenization
Version: 0.1.6
Authors@R: c(person('Jan', 'Wijffels', role = c('aut', 'cre', 'cph'), email = 'jwijffels@bnosac.be', comment = "R wrapper"), 
    person('BNOSAC', role = 'cph', comment = "R wrapper"), 
    person('VK.com', role = 'cph'), 
    person('Gregory Popovitch', role = c('ctb', 'cph'), comment = "Files at src/parallel_hashmap (Apache License, Version 2.0"), 
    person('The Abseil Authors', role = c('ctb', 'cph'), comment = "Files at src/parallel_hashmap (Apache License, Version 2.0"),
    person('Ivan Belonogov', role = c('ctb', 'cph'), email = 'xbelonogov@gmail.com', comment = "Files at src/youtokentome (MIT License)"))
Maintainer: Jan Wijffels <jwijffels@bnosac.be>
Description: Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.
URL: https://github.com/bnosac/tokenizers.bpe
License: MPL-2.0
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.3.2
Depends: R (>= 2.10)
Imports: Rcpp (>= 0.11.5)
LinkingTo: Rcpp
NeedsCompilation: yes
Packaged: 2026-06-13 21:42:29 UTC; jwijf
Author: Jan Wijffels [aut, cre, cph] (R wrapper),
  BNOSAC [cph] (R wrapper),
  VK.com [cph],
  Gregory Popovitch [ctb, cph] (Files at src/parallel_hashmap (Apache
    License, Version 2.0),
  The Abseil Authors [ctb, cph] (Files at src/parallel_hashmap (Apache
    License, Version 2.0),
  Ivan Belonogov [ctb, cph] (Files at src/youtokentome (MIT License))
Repository: CRAN
Date/Publication: 2026-06-13 22:00:02 UTC
