Multimodal language processing in human communication

J Holler, SC Levinson - Trends in Cognitive Sciences, 2019 - cell.com
The natural ecology of human language is face-to-face interaction comprising the exchange
of a plethora of multimodal signals. Trying to understand the psycholinguistic processing of …

A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

S Nyatsanga, T Kucherenko, C Ahuja… - Computer Graphics …, 2023 - Wiley Online Library
Gestures that accompany speech are an essential part of natural and efficient embodied
human communication. The automatic generation of such co‐speech gestures is a long …

Taming diffusion models for audio-driven co-speech gesture generation

L Zhu, X Liu, X Liu, R Qian, Z Liu… - Proceedings of the IEEE …, 2023 - openaccess.thecvf.com
Animating virtual avatars to make co-speech gestures facilitates various applications in
human-machine interaction. The existing methods mainly rely on generative adversarial …

Gesturediffuclip: Gesture diffusion model with clip latents

T Ao, Z Zhang, L Liu - ACM Transactions on Graphics (TOG), 2023 - dl.acm.org
The automatic generation of stylized co-speech gestures has recently received increasing
attention. Previous systems typically allow style control via predefined text labels or example …

Generating holistic 3d human motion from speech

H Yi, H Liang, Y Liu, Q Cao, Y Wen… - Proceedings of the …, 2023 - openaccess.thecvf.com
This work addresses the problem of generating 3D holistic body motions from human
speech. Given a speech recording, we synthesize sequences of 3D body poses, hand …

Listen, denoise, action! audio-driven motion synthesis with diffusion models

S Alexanderson, R Nagy, J Beskow… - ACM Transactions on …, 2023 - dl.acm.org
Diffusion models have experienced a surge of interest as highly expressive yet efficiently
trainable probabilistic models. We show that these models are an excellent fit for …

Speech gesture generation from the trimodal context of text, audio, and speaker identity

Y Yoon, B Cha, JH Lee, M Jang, J Lee, J Kim… - ACM Transactions on …, 2020 - dl.acm.org
For human-like agents, including virtual avatars and social robots, making proper gestures
while speaking is crucial in human-agent interaction. Co-speech gestures enhance …

Learning hierarchical cross-modal association for co-speech gesture generation

X Liu, Q Wu, H Zhou, Y Xu, R Qian… - Proceedings of the …, 2022 - openaccess.thecvf.com
Generating speech-consistent body and gesture movements is a long-standing problem in
virtual avatar creation. Previous studies often synthesize pose movement in a holistic …

Impacts of face coverings on communication: An indirect impact of COVID-19

GH Saunders, IR Jackson… - International Journal of …, 2021 - Taylor & Francis
Objective To understand the impact of face coverings on hearing and communication.
Design An online survey consisting of closed-set and open-ended questions distributed …

Learning individual styles of conversational gesture

S Ginosar, A Bar, G Kohavi, C Chan… - Proceedings of the …, 2019 - openaccess.thecvf.com
Human speech is often accompanied by hand and arm gestures. We present a method for
cross-modal translation from" in-the-wild" monologue speech of a single speaker to their …