Our experience of the world is multimodal-we see objects, hear sounds, feel texture, smell
odors, and taste flavors. Modality refers to the way in which something happens or is …

This paper surveys the current state of the art in Natural Language Generation (NLG),
defined as the task of generating text or speech from non-linguistic input. A survey of NLG is …

A robot that can carry out a natural-language instruction has been a dream since before the
Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot …

We introduce a new dataset for joint reasoning about natural language and images, with a
focus on semantic diversity, compositionality, and visual reasoning challenges. The data …

In this paper, we address referring expression comprehension: localizing an image region
described by a natural language expression. While most recent work treats expressions as a …

We introduce a novel framework for image captioning that can produce natural language
explicitly grounded in entities that object detectors find in the image. Our approach …

Navigation guided by natural language instructions presents a challenging reasoning
problem for instruction followers. Natural language instructions typically identify only a few …

We present an approach named JSFusion (Joint Sequence Fusion) that can measure
semantic similarity between any pairs of multimodal sequence data (eg a video sequence …

We introduce an effective and efficient method that grounds (ie, localizes) natural sentences
in long, untrimmed video sequences. Specifically, a novel Temporal GroundNet (TGN) is …

In this paper, we address the temporal moment localization issue, namely, localizing a video
moment described by a natural language query in an untrimmed video. This is a general yet …