Multimodal foundation models: From specialists to general-purpose assistants
Neural compression is the application of neural networks and other machine learning
methods to data compression. Recent advances in statistical machine learning have opened …
methods to data compression. Recent advances in statistical machine learning have opened …
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts
Although Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit
impressive skills in various domains, their ability for mathematical reasoning within visual …
impressive skills in various domains, their ability for mathematical reasoning within visual …
SEED-Bench: Benchmarking Multimodal Large Language Models
Multimodal large language models (MLLMs) building upon the foundation of powerful large
language models (LLMs) have recently demonstrated exceptional capabilities in generating …
language models (LLMs) have recently demonstrated exceptional capabilities in generating …
Capsfusion: Rethinking image-text data at scale
Large multimodal models demonstrate remarkable generalist ability to perform diverse
multimodal tasks in a zero-shot manner. Large-scale web-based image-text pairs contribute …
multimodal tasks in a zero-shot manner. Large-scale web-based image-text pairs contribute …
What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?
Recent advancements in GPT-4V have displayed remarkable multi-modal capabilities in
processing image inputs and following open-ended instructions. Despite these …
processing image inputs and following open-ended instructions. Despite these …
Binding touch to everything: Learning unified multimodal tactile representations
The ability to associate touch with other modalities has huge implications for humans and
computational systems. However multimodal learning with touch remains challenging due to …
computational systems. However multimodal learning with touch remains challenging due to …
Instruct-Imagen: Image generation with multi-modal instruction
Abstract This paper presents Instruct-Imagen a model that tackles heterogeneous image
generation tasks and generalizes across unseen tasks. We introduce multi-modal instruction …
generation tasks and generalizes across unseen tasks. We introduce multi-modal instruction …
How many unicorns are in this image? a safety evaluation benchmark for vision llms
This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different
from prior studies, we shift our focus from evaluating standard performance to introducing a …
from prior studies, we shift our focus from evaluating standard performance to introducing a …
Uncovering What Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly
Video anomaly understanding (VAU) aims to automatically comprehend unusual
occurrences in videos thereby enabling various applications such as traffic surveillance and …
occurrences in videos thereby enabling various applications such as traffic surveillance and …
Llm evaluators recognize and favor their own generations
Self-evaluation using large language models (LLMs) has proven valuable not only in
benchmarking but also methods like reward modeling, constitutional AI, and self-refinement …
benchmarking but also methods like reward modeling, constitutional AI, and self-refinement …