A simple recipe for contrastively pre-training video-first encoders beyond 16 frames
Understanding long real-world videos requires modeling of long-range visual
dependencies. To this end we explore video-first architectures building on the common …
dependencies. To this end we explore video-first architectures building on the common …
Upop: Unified and progressive pruning for compressing vision-language transformers
Real-world data contains a vast amount of multimodal information, among which vision and
language are the two most representative modalities. Moreover, increasingly heavier …
language are the two most representative modalities. Moreover, increasingly heavier …
TESTA: Temporal-spatial token aggregation for long-form video-language understanding
Large-scale video-language pre-training has made remarkable strides in advancing video-
language understanding tasks. However, the heavy computational burden of video …
language understanding tasks. However, the heavy computational burden of video …
Membridge: Video-language pre-training with memory-augmented inter-modality bridge
Video-language pre-training has attracted considerable attention recently for its promising
performance on various downstream tasks. Most existing methods utilize the modality …
performance on various downstream tasks. Most existing methods utilize the modality …
Enhancing video-language representations with structural spatio-temporal alignment
While pre-training large-scale video-language models (VLMs) has shown remarkable
potential for various downstream video-language tasks, existing VLMs can still suffer from …
potential for various downstream video-language tasks, existing VLMs can still suffer from …
Balance act: Mitigating hubness in cross-modal retrieval with query and gallery banks
In this work, we present a post-processing solution to address the hubness problem in cross-
modal retrieval, a phenomenon where a small number of gallery data points are frequently …
modal retrieval, a phenomenon where a small number of gallery data points are frequently …
Mclf: A multi-grained contrastive learning framework for asr-robust spoken language understanding
Enhancing the robustness towards Automatic Speech Recognition (ASR) errors is of great
importance for Spoken Language Understanding (SLU). Trending ASR-robust SLU systems …
importance for Spoken Language Understanding (SLU). Trending ASR-robust SLU systems …
Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers
Recent vision-language models have achieved tremendous progress far beyond what we
ever expected. However, their computational costs are also dramatically growing with rapid …
ever expected. However, their computational costs are also dramatically growing with rapid …
Few-shot Action Recognition with Captioning Foundation Models
Transferring vision-language knowledge from pretrained multimodal foundation models to
various downstream tasks is a promising direction. However, most current few-shot action …
various downstream tasks is a promising direction. However, most current few-shot action …
Bayes risk ctc: Controllable ctc alignment in sequence-to-sequence tasks
Sequence-to-Sequence (seq2seq) tasks transcribe the input sequence to a target sequence.
The Connectionist Temporal Classification (CTC) criterion is widely used in multiple …
The Connectionist Temporal Classification (CTC) criterion is widely used in multiple …