Flow matching completes hidden curvy objects in images accurately
ROCOCO: Robust Outline Completion of Occluded Curvilinear Objects
Computer Vision and Pattern RecognitionMachine Learning
Summary
Sometimes curved lines or shapes in pictures are partially blocked by other things, making it hard to see the full shape. The authors developed a way for computers to predict and fill in the missing parts of these curved shapes without needing prior knowledge of where the blockage is. Their method uses a special learning technique that looks at the visible parts and guesses the hidden sections accurately. Tests show their approach works better than many existing methods, especially when no information about the blocked areas is given.
What this means in practice
- •For autonomous vehicle engineers: Improve object detection by reconstructing obscured road markings and signs from partial views.
- •For medical image analysts: Fill in missing segments of curved anatomical structures such as blood vessels hidden by noise or artifacts.
Authors
Ken Zheng, Jaimyn Drake
Abstract
We propose a learning-based generative model for completing curvilinear features under occlusion by environmental obstacles. To this end, we adopt a flow matching-based approach that supports the inpainting of occluded regions with the appropriate curvilinear features. To eliminate the reliance on privileged information about the occlusion a priori, we also train a UNet-based mask prediction model, which can identify regions for inpainting given only the occluded observation. The components of our approach are validated via experiments using procedurally-generated test cases, and the complete pipeline is validated using large public datasets for curvilinear objects. Against off-the-shelf inpainting baselines, ROCOCO is the strongest method by a wide margin when no occlusion mask is given, and is competitive with mask-supervised inpainting when one is, achieving best on full-image error and second only to LaMa on the occluded region, while using a completer of $7.7$M parameters against LaMa's $51$M and Stable Diffusion's $860$M.