More than three billion years of evolution have produced an image of biology encoded into
the space of natural proteins. Here we show that language models trained on tokens …

We present a unified transformer, ie, Show-o, that unifies multimodal understanding and
generation. Unlike fully autoregressive models, Show-o unifies autoregressive and …

Verifiers or reward models are often used to enhance the reasoning performance of large
language models (LLMs). A common approach is the Best-of-N method, where N candidate …

The burgeoning interest in developing Large Language Models (LLMs) with up to trillion
parameters has been met with concerns regarding resource efficiency and practical …

We observe an empirical phenomenon in Large Language Models (LLMs)--very few
activations exhibit significantly larger values than others (eg, 100,000 times larger). We call …

Scaling laws are useful guides for derisking expensive training runs, as they predict
performance of large models using cheaper, small-scale experiments. However, there …

Robust and effective scaling of models from small to large width typically requires the
precise adjustment of many algorithmic and architectural details, such as parameterization …

Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model
size as a function of the compute budget, but these laws yield substantially different …

Underpinning the past decades of work on the design, initialization, and optimization of
neural networks is a seemingly innocuous assumption: that the network is trained on a\textit …

Training language models becomes increasingly expensive with scale, prompting numerous
attempts to improve optimization efficiency. Despite these efforts, the Adam optimizer …