As humans, we understand events in the visual world contextually, performing multimodal
reasoning across time to make inferences about the past, present, and future. We introduce …

As humans, we navigate a multimodal world, building a holistic understanding from all our
senses. We introduce MERLOT Reserve, a model that represents videos jointly over time …

Even from a single frame of a still image, people can reason about the dynamic story of the
image before, after, and beyond the frame. For example, given an image of a man struggling …

Visual understanding goes well beyond object recognition. With one glance at an image, we
can effortlessly imagine the world beyond the pixels: for instance, we can infer people's …

What makes a video task uniquely suited for videos, beyond what can be understood from a
single image? Building on recent progress in self-supervised image-language models, we …

Visual commonsense reasoning task aims at leading the research field into solving
cognition-level reasoning with the ability to predict correct answers and meanwhile …

Commonsense is defined as the knowledge that is shared by everyone. However, certain
types of commonsense knowledge are correlated with culture and geographic locations and …

Visual reasoning and question-answering have gathered attention in recent years. Many
datasets and evaluation protocols have been proposed; some have been shown to contain …

Natural language rationales could provide intuitive, higher-level explanations that are easily
understandable by humans, complementing the more broadly studied lower-level …

We present Video-LLaMA, a multi-modal framework that empowers Large Language Models
(LLMs) with the capability of understanding both visual and auditory content in the video …