We study how in-context learning (ICL) in language models is affected by semantic priors
versus input-label mappings. We investigate two setups-ICL with flipped labels and ICL with …

While the successes of transformers across many domains are indisputable, accurate
understanding of the learning mechanics is still largely lacking. Their capabilities have been …

With infinitely many high-quality data points, infinite computational power, an infinitely large
foundation model with a perfect training algorithm and guaranteed zero generalization error …

The surprising discovery of the BYOL method shows the negative samples can be replaced
by adding the prediction head to the network. It is mysterious why even when there exist …

In-context learning has been recognized as a key factor in the success of Large Language
Models (LLMs). It refers to the model's ability to learn patterns on the fly from provided in …

In this paper, we propose the use of self-supervised pretraining on a large unlabelled data
set to improve the performance of a personalized voice activity detection (VAD) model in …

Theoretically understanding the success of modern neural networks remains challenging. In
the direction of theoretically understanding fully connected Multilayer Perceptrons (MLPs) …