Tokenization
Why models break text into subwords, how Byte Pair Encoding builds a vocabulary from scratch, and what naive word splitting gets wrong.
–––
Topics / Tokenization
Tokenization
1 post.
Why language models split text into subwords, how Byte Pair Encoding builds a vocabulary, and how tokenizers handle unfamiliar text.