Spatiotemporal flux probing captures fast videos with few photons
Spatiotemporal Flux Probing for Single-Photon Videography
Computer Vision and Pattern Recognition
Summary
Capturing videos in very dark or low-light places is hard because cameras only get a few photons, or particles of light. Existing methods group photons nearby in space and time to get clearer images, but they fail when photons are very sparse. The authors show that looking at the whole pattern of photon arrivals over space and time contains the information needed to recover detailed motion and changes in brightness. They developed a new way to directly estimate the important features of the video from photon data, enabling faster and clearer video with fewer photons, and even zoom in on specific movements.
What this means in practice
- •For security camera operators: Capture clearer videos of fast-moving targets in extremely low light environments using far fewer photons than current technology requires.$Commercial implications: Enables novel single-photon video products for nighttime surveillance with reduced hardware complexity and power needs.
- •For robotics engineers: Improve robot vision systems to detect and track specific motions under low-light conditions by exploiting velocity-selective video reconstruction.
Authors
Jerry Yan, Matteo Forlivesi, Bowen Tan, Andrew Xie, Siddharth Somasundaram, Sotiris Nousias
Abstract
We address the problem of recovering high-speed videos from dynamic scenes under extreme photon sparsity. Existing methods rely on aggregating photon detections in local spatiotemporal windows to improve signal-to-noise ratio; however, this local grouping discards global structure and fails in low-light regimes where photon detections are sparse in space and time. In this work, we show that the information needed to recover both motion and illumination is encoded in correlations over the full space-time pattern of photon arrivals. Building on this insight, we develop a spatiotemporal flux probing theory and an algorithm that estimates the Fourier coefficients of the underlying intensity directly from the photon stream. We demonstrate that our approach (1) recovers fast motion and temporal illumination dynamics with substantially fewer photons than prior methods, (2) enables velocity-selective videography that automatically refocuses video onto specific detected motions, and (3) generalizes across sensing modalities including single-photon, event, and spike cameras.