Tokenization is an understudied and often neglected component of modern LLMs. Most
published works use a single tokenizer for all experiments, often borrowed from another …

We propose task-adaptive tokenization as a way to adapt the generation pipeline to the
specifics of a downstream task and enhance long-form generation in mental health. Inspired …

Adam has been shown to outperform gradient descent in optimizing large language
transformers empirically, and by a larger margin than on other tasks, but it is unclear why this …

Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, despite
being devised initially as a compression method. BPE appears to be a greedy algorithm at …

Self-supervised pre-training of language models usually consists in predicting probability
distributions over extensive token vocabularies. In this study, we propose an innovative …

Tokenization-the practice of converting strings of characters from an alphabet into
sequences of tokens over a vocabulary-is a critical step in the NLP pipeline. The use of …

To analyze the scaling potential of deep tabular representation learning models, we
introduce a novel Transformer-based architecture specifically tailored to tabular data and …

Tokenization is a foundational step in Natural Language Processing (NLP) tasks, bridging
raw text and language models. Existing tokenization approaches like Byte-Pair Encoding …

Abstract Language models can greatly benefit from efficient tokenization. However, they still
mostly utilize the classical Byte-Pair Encoding (BPE) algorithm, a simple and reliable …

We present TMMLU+, a new benchmark designed for Traditional Chinese language
understanding. TMMLU+ is a multi-choice question-answering dataset with 66 subjects from …