A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

S Nyatsanga, T Kucherenko, C Ahuja… - Computer Graphics …, 2023 - Wiley Online Library
Gestures that accompany speech are an essential part of natural and efficient embodied
human communication. The automatic generation of such co‐speech gestures is a long …

Learning to listen: Modeling non-deterministic dyadic facial motion

E Ng, H Joo, L Hu, H Li, T Darrell… - Proceedings of the …, 2022 - openaccess.thecvf.com
We present a framework for modeling interactional communication in dyadic conversations:
given multimodal inputs of a speaker, we autoregressively output multiple possibilities of …

Can language models learn to listen?

E Ng, S Subramanian, D Klein… - Proceedings of the …, 2023 - openaccess.thecvf.com
We present a framework for generating appropriate facial responses from a listener in
dyadic social interactions based on the speaker's words. Given an input transcription of the …

Style‐controllable speech‐driven gesture synthesis using normalising flows

S Alexanderson, GE Henter… - Computer Graphics …, 2020 - Wiley Online Library
Automatic synthesis of realistic gestures promises to transform the fields of animation,
avatars and communicative agents. In off‐line applications, novel tools can alter the role of …

From audio to photoreal embodiment: Synthesizing humans in conversations

E Ng, J Romero, T Bagautdinov, S Bai… - Proceedings of the …, 2024 - openaccess.thecvf.com
We present a framework for generating full-bodied photorealistic avatars that gesture
according to the conversational dynamics of a dyadic interaction. Given speech audio we …

Gender stereotypes in virtual agents

P Nag, ÖN Yalçın - Proceedings of the 20th ACM International …, 2020 - dl.acm.org
Visual, behavioral and verbal cues for gender are often used in designing virtual agents to
take advantage of their stereotypical effects on the users. However, recent studies point …

Let's face it: Probabilistic multi-modal interlocutor-aware generation of facial gestures in dyadic settings

P Jonell, T Kucherenko, GE Henter… - Proceedings of the 20th …, 2020 - dl.acm.org
To enable more natural face-to-face interactions, conversational agents need to adapt their
behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal …

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

T Kucherenko, D Hasegawa, N Kaneko… - … Journal of Human …, 2021 - Taylor & Francis
This paper presents a novel framework for speech-driven gesture production, applicable to
virtual agents to enhance human-computer interaction. Specifically, we extend recent deep …

SRG3: Speech-driven Robot Gesture Generation with GAN

C Yu, A Tapus - 2020 16th International Conference on Control …, 2020 - ieeexplore.ieee.org
The human gestures occur spontaneously and usually they are aligned with speech, which
leads to a natural and expressive interaction. Speech-driven gesture generation is important …

Dyadic Interaction Modeling for Social Behavior Generation

M Tran, D Chang, M Siniukov, M Soleymani - arXiv preprint arXiv …, 2024 - arxiv.org
Human-human communication is like a delicate dance where listeners and speakers
concurrently interact to maintain conversational dynamics. Hence, an effective model for …