Summary
Semantic segmentation helps computers understand different parts of a scene in an image, but it struggles when images are taken in the dark or poor lighting. The authors created VesperMUSE, a new system that improves this by combining regular camera images with thermal images and surface details, helping the system see better in low light. They also improved low-light images using a physics-based method, then combined these multiple sources using a special attention mechanism that picks out the most useful pieces of information. This makes it easier for machines like robots and autonomous vehicles to understand scenes more reliably even when lighting is bad.
What this means in practice
- •For autonomous vehicle developers: Enhance scene understanding algorithms to navigate safely in low-light or nighttime environments using combined thermal and visual data.$Commercial implications: This enables automotive companies to build more reliable driver-assist and self-driving systems that work effectively in poor lighting.
- •For security system integrators: Improve surveillance camera systems by integrating thermal and RGB images for clearer object recognition under challenging lighting.
Abstract
Semantic segmentation is critical for the reliable navigation and scene understanding of autonomous systems, yet its performance degrades severely in low-light environments due to noise, artifacts, colour distortion, and the loss of structural details. To address this challenge, this work introduces VesperMUSE, a unified multi-modal framework designed to achieve robust image semantic segmentation across diverse illumination conditions. The architecture addresses the light deficiency problem on multiple levels: proposing a novel physics-informed synthetic low-light adaptation algorithm, approaching Low-Light Image Enhancement (LLIE) through a thermal-guided paradigm, and finally aiming to recover the intrinsic scene details for semantic segmentation through a multi-modal fusion approach. Following enhancement, the framework dynamically integrates the primary RGB features with auxiliary thermal representations and surface normal maps highlighting geometric discontinuities and material boundaries. At the core of this integration is a highly scalable, modality-agnostic self-attention mechanism that extracts both global context and fine local features from an arbitrary number of auxiliary inputs. By combining and projecting these hierarchical multi-modal representations into a unified component, the attention module allows the network to selectively attend to relevant cross-modal cues without requiring any architectural modifications when different sensor configurations are used. Extensive evaluations demonstrate that VesperMUSE successfully recovers missing visual cues, providing a flexible, state-of-the-art (SOTA) solution for machine perception in extreme and unstructured real-world environments.