# Byte-pair encoding

tags
NLP

The process of byte-pair encoding can be summarized as follow:

• Each character is a token
• Find pairs that occur most often
• Create a new token that encoded those common pairs
• Repeat the process until target vocabulary size is reached

The output of this process is both a vocabulary and a set of merging rules for tokens to be used to process more data.