Incorporating physics into data-driven computer vision
Many computer vision techniques infer properties of our physical world from images.
Although images are formed through the physics of light and mechanics, computer vision …
Although images are formed through the physics of light and mechanics, computer vision …
On the opportunities and risks of foundation models
AI is undergoing a paradigm shift with the rise of models (eg, BERT, DALL-E, GPT-3) that are
trained on broad data at scale and are adaptable to a wide range of downstream tasks. We …
trained on broad data at scale and are adaptable to a wide range of downstream tasks. We …
Physically grounded vision-language models for robotic manipulation
Recent advances in vision-language models (VLMs) have led to improved performance on
tasks such as visual question answering and image captioning. Consequently, these models …
tasks such as visual question answering and image captioning. Consequently, these models …
On the binding problem in artificial neural networks
Contemporary neural networks still fall short of human-level generalization, which extends
far beyond our direct experiences. In this paper, we argue that the underlying cause for this …
far beyond our direct experiences. In this paper, we argue that the underlying cause for this …
3d neural scene representations for visuomotor control
Humans have a strong intuitive understanding of the 3D environment around us. The mental
model of the physics in our brain applies to objects of different materials and enables us to …
model of the physics in our brain applies to objects of different materials and enables us to …
Physgen: Rigid-body physics-grounded image-to-video generation
We present PhysGen, a novel image-to-video generation method that converts a single
image and an input condition (eg., force and torque applied to an object in the image) to …
image and an input condition (eg., force and torque applied to an object in the image) to …
Learning multi-object dynamics with compositional neural radiance fields
We present a method to learn compositional multi-object dynamics models from image
observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and …
observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and …
Algorithms and systems for manipulating multiple objects
Robot manipulation of multiple objects is an important topic for applications including
warehouse automation, service robots performing cleaning, and large-scale object sorting …
warehouse automation, service robots performing cleaning, and large-scale object sorting …
Dynamic visual reasoning by learning differentiable physics models from video and language
In this work, we propose a unified framework, called Visual Reasoning with Differ-entiable
Physics (VRDP), that can jointly learn visual concepts and infer physics models of objects …
Physics (VRDP), that can jointly learn visual concepts and infer physics models of objects …
RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks
Modeling and manipulating elasto-plastic objects are essential capabilities for robots to
perform complex industrial and household interaction tasks (eg, stuffing dumplings, rolling …
perform complex industrial and household interaction tasks (eg, stuffing dumplings, rolling …