Event camera method reconstructs 3d surface shape under moving lights
PIE-PS: Photometric Stereo from Physical Irradiance Event Streams
Computer Vision and Pattern Recognition
Summary
Getting detailed 3D shapes of objects by looking at how light moves across them is hard when both the object and lights are changing quickly. The researchers use special event cameras that detect tiny changes in light very fast, but these signals are tricky because they depend on settings the camera doesn't tell you. They developed a new approach that groups pairs of these light-change events and uses a smart graph-based method to figure out the surface shape more accurately. Their system works better than previous methods on both fake and real scenes.
What this means in practice
- •For robotic vision teams: Reconstruct accurate 3D object surfaces in dynamic lighting using fast event cameras and this new processing method.
- •For augmented reality developers: Create detailed and responsive surface models in AR systems by integrating real-time event camera data under moving illumination.
Authors
Xiangze Meng, Guangyu Li, Jing Li, Di Mei, Songchen Ma, Mingkun Xu, Rui Ma
Abstract
Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range. These properties are useful for photometric stereo under moving illumination, but raw events are sparse and depend on an unknown contrast threshold. We start from the event trigger model and derive a physical relation between adjacent events, light motion, and surface normals. This relation gives a direct physics-only solver, but the solver needs the threshold, enough events at each pixel, and independent per-pixel optimization. To address these limits, we introduce PIE-PS, a learning-based framework for dense surface normal reconstruction from raw event streams and known lighting. We form Physical Irradiance Events (PIEs) by pairing two adjacent events at the same pixel with their corresponding light directions. Each PIE provides a Physical Irradiance Event Feature (PIEF), defined as the signed event rate. PIEF does not require the unknown contrast threshold. To share spatial and temporal context across nearby PIEs, we introduce PIE-GNN, which treats each PIE as a graph node and encodes it with its light-pair geometry. Since the reliability of PIE observations can vary with local appearance, illumination geometry, and sensor noise, Reliability-Grading Attention (RGA) predicts reliability weights to down-weight unreliable PIEs. Pixel aggregation then produces dense normals. Experiments on synthetic and real data show that PIE-PS outperforms prior event-based photometric stereo methods and the direct solver baseline.