Greed is All You Need: An Evaluation of Tokenizer Inference Methods (2403.01289v2)

Abstract: While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment.

• The paper presents a comprehensive evaluation of seven tokenizer inference methods across four tokenization algorithms and varying vocabulary sizes.
• It finds that simple greedy inference methods achieve high morphological alignment and robustness compared to more complex methods.
• The research implies that decoupling vocabulary creation from inference and leveraging innovative designs like SaGe can enhance overall tokenizer performance.

Evaluating Tokenizer Inference Methods: A Controlled Analysis

Introduction

NLP systems routinely convert raw text into sequences of subword tokens using algorithms like Byte-Pair Encoding (BPE), WordPiece, or UnigramLM. Although much attention has been devoted to optimizing these tokenization algorithms, the process of inferring the sequence of tokens from these vocabularies—a critical component known as the inference method—has remained under-explored. In a paper, a comprehensive analysis of seven tokenizer inference methods was performed across four different algorithms (BPE, UnigramLM, WordPiece, and SaGe) and three vocabulary sizes. This research unveiled surprising findings about the efficacy of these methods and outlined their implications for future developments in the field.

Investigation into Inference Methods

Subword tokenization plays a pivotal role in how text data is represented for NLP models. The paper put under the microscope not just the well-known tokenizer vocabularies but also the associated inference methods, which dictate how the text is broken down into the tokens provided by these vocabularies. The inquiry centered on:

• Greedy inference methods, which iteratively choose one token at each step based on certain criteria (e.g., longest prefix/suffix or token).
• Merge rules-based inference methods, where word character sequences are iteratively merged according to predefined rules.
• Likelihood-based inference methods, which utilize token likelihoods to find the most probable segmentation of a word.

Their performance was measured using a variety of intrinsic evaluations that ranged from aligning with morphological segmentation, cognitive plausibility, to information-theoretical considerations.

Benchmarking Results and Insights

The findings from this rigorous evaluation showed that greedy inference methods, which are relatively simple in approach, performed remarkably well across a variety of metrics. This was particularly evident in their ability to align with morphological standards, suggesting a latent prowess in handling complex word forms. Among the evaluated tokenizers, SaGe—a newly introduced tokenizer—demonstrated superior performance in morphological alignment, suggesting its sophisticated mechanism was advantageous for capturing the subtleties of word structure.

In terms of vocabulary-size influence, the paper illuminated how certain inference methods scaled with vocabulary adjustments, providing key insights into their robustness and utility across different dataset magnitudes.

Implications and Future Directions

The implications of these findings are manifold:

• Decoupling Tokenization and Inference: The paper underscores the potential benefits of decoupling vocabulary creation from the inference method, advocating for the flexibility to choose the most suitable inference method depending on the task.
• Greedy Methods’ Surprising Efficacy: The success of greedy inference methods calls for a reassessment of their role in tokenizer design, potentially encouraging their adoption in scenarios where complex tokenization algorithms were previously thought necessary.
• Advancements in Tokenizer Design: The standout performance of SaGe offers promising directions for future tokenizer designs, particularly for applications requiring nuanced understanding of language morphology.

In conclusion, by providing a comprehensive analysis of tokenizer inference methods, this paper paves the way for more informed choices in tokenizer selection and design. It not only highlights the often-overlooked importance of inference methods but also opens the door for future investigations that could lead to even more efficient and effective NLP systems. The ongoing evolution of tokenization strategies, as evidenced by this research, is crucial for the advancement of LLMs and their applications, offering a richer understanding of language processing at both theoretical and practical levels.