Orthogonal knowledge distillation improves open-vocabulary event localization in videos
If You Hear It, Help Find It: Orthogonal Knowledge Distillation for Open-Vocabulary Audio-Visual Event Localization
MultimediaComputer Vision and Pattern RecognitionMachine LearningSound
Summary
Finding specific events in videos by using both sound and images can be tricky because sometimes the audio or video signals are more reliable than the other. The authors studied how to best combine these signals by treating one as the main guide and the other as a helpful sidekick during training. Their method keeps the main and side information separate but connected, which helps the system recognize events better, especially when asked about things it hasn’t seen before. This approach improved the accuracy of locating events in videos compared to previous methods.
What this means in practice
- •For video content analysis teams: Improve automatic detection and timing of events in videos using both audio and visual clues with better handling of new event categories.
- •For security monitoring professionals: Enhance surveillance systems to more accurately identify specific audio-visual events over time, including unfamiliar sounds or visuals.
Authors
Yi Xu, Cheng Chen, Wenzhuo Lei
Abstract
Open-vocabulary audio-visual event localization (OV-AVEL) grounds a text-queried event in time from video, audio, and language. The supervision sources available to this task can differ in temporal-boundary reliability: on OV-AVEBench, our configured visual teacher gives more reliable boundary cues than the configured audio teacher, although the latter is a strong pretrained audio model and remains semantically informative. This is a setting-specific diagnostic rather than a universal ranking of vision and audio. We formulate the resulting challenge as supervision placement: which teacher signals may shape the localization decision, and which should remain auxiliary. Based on this view, we propose OV-OrthKD, a reliability-aware asymmetric distillation framework. Visual feature transfer shapes a decision-aligned representation, audio feature transfer enriches a complementary auxiliary subspace, a text prototype anchors seen/unseen category semantics, and an orthogonality loss limits directional overlap between the two teacher-specific projections. The student continues to use both modalities through query-aware fusion at inference, while the default training recipe keeps audio-teacher supervision off the segment-logit path. On OV-AVEBench, OV-OrthKD achieves 0.816 segment AP and improves F1@0.5 over the official fine-tuning baseline by 2.7 points overall and 3.4 points on unseen categories. Path-assignment, role-swap, corruption, and transfer analyses consistently support supervision placement as a task-specific design axis for OV-AVEL.