Depth image upsampling guided by high resolution color images
Image Guided Depth Upsampling using Anisotropic Total Generalized Variation
Computer Vision and Pattern Recognition
Summary
Depth cameras capture how far away things are but with blurry details. The authors created a new method to make these depth images sharper and more detailed by using clearer color pictures to guide the enhancement. Their approach uses a math technique to carefully fill in missing depth details without blurring edges. They also made new data to test this method using real camera measurements and found it works faster and more accurately than older methods.
What this means in practice
- •For depth camera developers: Improve low-resolution depth outputs of devices by integrating high-resolution color cues to produce sharper 3D measurements.$Commercial implications: Enables manufacturing of depth cameras with enhanced resolution suitable for consumer or industrial products.
- •For robotics system engineers: Create better 3D environment maps by upsampling depth data guided by high-res images for navigation and object recognition.
Authors
David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Rüther, Horst Bischof
Abstract
In this work we present a novel method for the challenging problem of depth image upsampling. Modern depth cameras such as Kinect or Time of Flight cameras deliver dense, high quality depth measurements but are limited in their lateral resolution. To overcome this limitation we formulate a convex optimization problem using higher order regularization for depth image upsampling. In this optimization an anisotropic diffusion tensor, calculated from a high resolution intensity image, is used to guide the upsampling. We derive a numerical algorithm based on a primaldual formulation that is efficiently parallelized and runs at multiple frames per second. We show that this novel upsampling clearly outperforms state of the art approaches in terms of speed and accuracy on the widely used Middlebury 2007 datasets. Furthermore, we introduce novel datasets with highly accurate groundtruth, which, for the first time, enable to benchmark depth upsampling methods using real sensor data.