MerchantryTidbits

llm-inference / library

Tokenizers

Capability: Tokenizers

Use it when

  • fast Hugging Face tokenization for training and RAG
  • encode decode batches of text efficiently

What it solves

context-window-overflow

Install

pip install tokenizers

Invoke

# see https://github.com/huggingface/tokenizers
pip install tokenizers

Alternatives

No reviewed alternatives recorded yet.