A major factor in the recent success of large language models is the use of enormous and
ever-growing text datasets for unsupervised pre-training. However, naively training a model …

This paper studies the theoretical framework of the alignment process of generative models
with Reinforcement Learning from Human Feedback (RLHF). We consider a standard …

We propose controlled decoding (CD), a novel off-policy reinforcement learning method to
control the autoregressive generation from language models towards high reward …

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback
(RLHF) in this technical report, which is widely reported to outperform its offline counterpart …

Diffusion models excel at capturing the natural design spaces of images, molecules, DNA,
RNA, and protein sequences. However, rather than merely generating designs that are …

Best-of-N (BoN) is a popular and effective algorithm for aligning language models to human
preferences. The algorithm works as follows: at inference time, N samples are drawn from …

Inference-time alignment enhances the performance of large language models without
requiring additional training or fine-tuning but presents challenges due to balancing …

Large language models are typically fine-tuned to align with human preferences, but tuning
large models is computationally intensive and complex. In this work, we introduce $\textit …

Generative modeling of discrete data underlies important applications spanning text-based
agents like ChatGPT to the design of the very building blocks of life in protein sequences …

Let $ p $ denote a generative language model. Let $ r $ denote a reward model that returns
a scalar that captures the degree at which a draw from $ p $ is preferred. The goal of …