Sight and hearing are two senses that play a vital role in human communication and scene
understanding. To mimic human perception ability, audio-visual learning, aimed at …

In the context of Audio Visual Question Answering (AVQA) tasks, the audio and visual
modalities could be learnt on three levels: 1) Spatial, 2) Temporal, and 3) Semantic. Existing …

Audio-visual learning has been a major pillar of multi-modal machine learning, where the
community mostly focused on its $\textit {modality-aligned} $ setting, $\textit {ie} $, the audio …

As a newly emerging task, audio-visual question answering (AVQA) has attracted research
attention. Compared with traditional single-modality (eg, audio or visual) QA tasks, it poses …

In recent years, Audio Visual Scene-Aware Dialog (AVSD) has been an active research task
in the multimodal dialogue community and has also been a core part of the Dialog System …

Abstract We present\(\mathbb {MST} _\mathbb {MIXER}\)–a novel video dialog model
operating over a generic multi-modal state tracking scheme. Current models that claim to …

In this report, we present our submissions to the DSTC10 Audio Visual Scene Dialog
(AVSD) challenge. We investigated variants of an encoder-decoder model, including those …

There have been many attempts to build multimodal dialog systems that can respond to a
question about given audio-visual information, and the representative task for such systems …

In the realm of video dialog response generation, the understanding of video content and
the temporal nuances of conversation history are paramount. While a segment of current …

Generating dialogue grounded in videos requires a high level of understanding and
reasoning about the visual scenes in the videos. However, existing large visual-language …