Enhancing multimodal cooperation via sample-level modality valuation

Y Wei, R Feng, Z Wang, D Hu - Proceedings of the IEEE …, 2024 - openaccess.thecvf.com
One primary topic of multimodal learning is to jointly incorporate heterogeneous information
from different modalities. However most models often suffer from unsatisfactory multimodal …

Multimodal representation learning by alternating unimodal adaptation

X Zhang, J Yoon, M Bansal… - Proceedings of the IEEE …, 2024 - openaccess.thecvf.com
Multimodal learning which integrates data from diverse sensory modes plays a pivotal role
in artificial intelligence. However existing multimodal learning methods often struggle with …

Multimodal fusion on low-quality data: A comprehensive survey

Q Zhang, Y Wei, Z Han, H Fu, X Peng, C Deng… - arXiv preprint arXiv …, 2024 - arxiv.org
Multimodal fusion focuses on integrating information from multiple modalities with the goal of
more accurate prediction, which has achieved remarkable progress in a wide range of …

Dual-branch dynamic modulation network for hyperspectral and LiDAR data classification

Z Xu, W Jiang, J Geng - IEEE Transactions on Geoscience and …, 2023 - ieeexplore.ieee.org
Deep learning algorithms that can effectively extract features from different modalities have
achieved significant performance in multimodal remote sensing (RS) data classification …

Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing

X Lin, S Wang, R Cai, Y Liu, Y Fu… - Proceedings of the …, 2024 - openaccess.thecvf.com
Abstract Face Anti-Spoofing (FAS) is crucial for securing face recognition systems against
presentation attacks. With advancements in sensor manufacture and multi-modal learning …

Test-time Adaptation against Multi-modal Reliability Bias

M Yang, Y Li, C Zhang, P Hu, X Peng - The Twelfth International …, 2024 - openreview.net
Test-time adaptation (TTA) has emerged as a new paradigm for reconciling distribution shifts
across domains without accessing source data. However, existing TTA methods mainly …

C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation

F Huo, W Xu, J Guo, H Wang… - Proceedings of the IEEE …, 2024 - openaccess.thecvf.com
Abstract Existing Knowledge Distillation (KD) methods typically focus on transferring
knowledge from a large-capacity teacher to a low-capacity student model achieving …

Quantifying and enhancing multi-modal robustness with modality preference

Z Yang, Y Wei, C Liang, D Hu - arXiv preprint arXiv:2402.06244, 2024 - arxiv.org
Multi-modal models have shown a promising capability to effectively integrate information
from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations …

Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion

Z Gao, X Jiang, X Xu, F Shen, Y Li… - Proceedings of the …, 2024 - openaccess.thecvf.com
As a fundamental problem in multimodal learning multimodal fusion aims to compensate for
the inherent limitations of a single modality. One challenge of multimodal fusion is that the …

A variational expectation-maximization framework for balanced multi-scale learning of protein and drug interactions

J Rao, J Xie, Q Yuan, D Liu, Z Wang, Y Lu… - Nature …, 2024 - nature.com
Protein functions are characterized by interactions with proteins, drugs, and other
biomolecules. Understanding these interactions is essential for deciphering the molecular …