Mobile devices with passive depth sensing capabilities are ubiquitous, and recently active
depth sensors have become available on some tablets and AR/VR devices. Although real-
time depth data is accessible, its rich value to mainstream AR applications has been sorely
under-explored. Adoption of depth-based UX has been impeded by the complexity of
performing even simple operations with raw depth data, such as detecting intersections or
constructing meshes. In this paper, we introduce DepthLab, a software library that …

We demonstrate DepthLab, a playground for interactive augmented reality experiences
leveraging the shape and depth of the physical environment on a mobile phone. Based on
the ARCore Depth API, DepthLab encapsulates a variety of depth-based UI/UX paradigms,
including geometry-aware rendering (occlusion, shadows, texture decals), surface
interaction behaviors (physics, collision detection, avatar path planning), and visual effects
(relighting, 3D-anchored focus and aperture effects, 3D photos). We have open-sourced our …