What term refers to the smallest unit of text processed by an LLM?

The term that refers to the smallest unit of text processed by a large language model (LLM) is "token." Tokens can represent various structures in text, such as words, sub-words, or even individual characters, depending on the tokenization strategy employed. In natural language processing, tokenization breaks down text into these smaller, manageable units that the model can analyze and generate.

Tokenization is a crucial preprocessing step in the workings of language models because it allows them to handle different languages and nuances effectively. For instance, some models may use byte pair encoding or another method that produces sub-word tokens, which help in managing rare words by breaking them down into more frequently occurring components.

While "word," "character," and "phrase" are all relevant terms in text processing, they do not encompass the full range of what tokens can represent. A word may seem like a logical choice at first glance, but it does not account for situations where common prefixes or suffixes are treated separately, which can enhance the model's ability to generate and understand language. Similarly, a character is too granular, while a phrase encompasses multiple words, thus not reflecting the granularity that tokens provide.