As the deep learning revolution marches on, self-supervised learning has garnered
increasing attention in recent years thanks to its remarkable representation learning ability …

Visual tasks vary a lot in their output formats and concerned contents, therefore it is hard to
process them with an identical structure. One main obstacle lies in the high-dimensional …

This paper enhances image-GPT (iGPT), one of the pioneering works that introduce
autoregressive pretraining to predict the next pixels for visual representation learning. Two …

Autoregressive language modeling (ALM) has been successfully used in self-supervised pre-
training in Natural language processing (NLP). However, this paradigm has not achieved …

Current self-supervised methods can primarily be categorized into contrastive learning and
masked image modeling. Extensive studies have demonstrated that combining these two …

Inspired by the masked language modeling (MLM) in natural language processing tasks, the
masked image modeling (MIM) has been recognized as a strong self-supervised pre …

The development of autoregressive modeling (AM) in computer vision lags behind natural
language processing (NLP) in self-supervised pre-training. This is mainly caused by the …

In recent years, the rise of generative self-supervised learning (SSL) paradigms has
exhibited impressive performance across visual, language, and multi-modal domains. While …

The vision community has started to build with the recently developed state space model,
Mamba, as the new backbone for a range of tasks. This paper shows that Mamba's visual …

This paper presents a new self-supervised video representation learning framework,
ARVideo, which autoregressively predicts the next video token in a tailored sequence order …