Unified Video Dense Prediction from Disjoint Data
2026-07-23 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created UniD, a single video model that can predict eight different scene properties like depth, materials, and human parts all at once, even though existing datasets only label some tasks separately. They use a clever training method where expert models for each task teach the unified model without needing overlapping data or costly extra labeling. Their approach relies on a pretrained diffusion model to help the unified model handle differences between datasets, enabling it to perform well on new combinations of tasks it hasn’t seen before. UniD matches or beats specialized models and works better over time and across tasks.
Scene understandingSemantic segmentationSurface normalsDiffusion modelsMulti-task learningKnowledge distillationPseudo-labelingDepth estimationTemporal consistencyDomain adaptation
Authors
Yihong Sun, Seoung Wug Oh, Jiahui Huang, Bharath Hariharan, Joon-Young Lee
Abstract
Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials-all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling. Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency. Code and video results are available at https://unid-video.github.io/.