This paper shows that masked autoencoders (MAE) are scalable self-supervised learners
for computer vision. Our MAE approach is simple: we mask random patches of the input …

We propose ADIOS, a masked image model (MIM) framework for self-supervised learning,
which simultaneously learns a masking function and an image encoder using an adversarial …

We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM
uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of …

Pre-training models on large scale datasets, like ImageNet, is a standard practice in
computer vision. This paradigm is especially effective for tasks with small training sets, for …

Masked image modeling (MIM) has achieved promising results on various vision tasks.
However, the limited discriminability of learned representation manifests there is still plenty …

Masked AutoEncoder (MAE) has recently led the trends of visual self-supervision area by an
elegant asymmetric encoder-decoder design, which significantly optimizes both the pre …

In this work, we explore self-supervised visual pre-training on images from diverse, in-the-
wild videos for real-world robotic tasks. Like prior work, our visual representations are pre …

To date, most existing self-supervised learning methods are designed and optimized for
image classification. These pre-trained models can be sub-optimal for dense prediction …

This paper is concerned with self-supervised learning for small models. The problem is
motivated by our empirical studies that while the widely used contrastive self-supervised …

Recent advances in self-supervised learning (SSL) have largely closed the gap with
supervised ImageNet pretraining. Despite their success these methods have been primarily …