Hard region supervision boosts video object segmentation accuracy
Hard-Region Supervision: #1 on the Waymo Open Dataset 2D Video Panoptic Segmentation Leaderboard
Computer Vision and Pattern Recognition
Summary
Segmenting and tracking every object in video frames across multiple cameras is hard because mistakes happen in challenging areas. The authors improved an existing system by teaching it to focus more on these tricky regions during training without changing its structure at run-time. They also combined multiple models and linked identities across cameras to boost accuracy. Their method won first place on a major video segmentation challenge by a clear margin.
What this means in practice
- •For autonomous vehicle engineers: Improve multi-camera perception systems to better detect and track objects over time and across viewpoints for safer driving decisions.$Commercial implications: Enables higher quality video panoptic segmentation in self-driving cars, improving object recognition and tracking crucial for commercial autonomous vehicle platforms.
- •For video surveillance operators: Enhance long-term tracking of individuals and objects across multiple camera feeds by focusing training on challenging segmentation areas.
Authors
Jinghan Yang
Abstract
We describe our winning entry to the Waymo Open Dataset 2D Video Panoptic Segmentation Challenge. The task asks for a semantic class at every pixel of every frame and, for countable objects, an identity that holds across 100 frames and across five overlapping cameras. We build on DVIS++, a cascade of a segmenter, a tracker, and a refiner, as our baseline. We propose hard region supervision (HRS) to improve the baseline. In particular, we use the baseline to define the hard region as where it makes mistakes, and design a loss and an auxiliary prediction head for this region. The auxiliary head is used only in training and removed at test time, so at inference the model trained with HRS has the same architecture as the baseline. In addition, we propose three test-time steps that further improve the results: a two-model ensemble, a merge of the segmenter's output into the final panoptic map, and cross-camera identity linking. On the challenge test set, our entry reaches 0.3547 wSTQ, 0.2071 wAQ, and 0.6075 mIoU, ranking first on all three metrics. It is 3.6 wSTQ points ahead of the second entry and 2.4 points ahead of our DVIS++ baseline.