GradCO improves text-based searches for people by spotting behavior differences
GradCO: Behavior-Aware Gradient-Conflict Offloading for Text-Based Person Anomaly Search
Computer Vision and Pattern RecognitionInformation Retrieval
Summary
Finding people based on descriptions that include both what they look like and how they behave is tricky. The authors created GradCO, a method that helps computers better separate different behaviors of the same person while still recognizing who they are. GradCO uses special components to handle appearance and behavior clues separately, reduce confusion when training, and combine these clues smartly. Tests show GradCO works better than older methods on several benchmark sets.
What this means in practice
- •For security monitoring teams: Improve surveillance systems to find individuals based on detailed behavior descriptions alongside appearance using text queries.
- •For video analysis developers: Develop software that retrieves and distinguishes individuals by fine behavior differences from textual inputs in crowded scenes.
Authors
Zhiheng Fu, Zixu Li, Zhiwei Chen, Jiale Huang, Fangxu Liu, Jinhe Bi, Yupeng Hu
Abstract
Text-based Person Anomaly Search (TPAS) retrieves a target pedestrian using natural-language descriptions of appearance and normal or anomalous behavior. Unlike conventional text-based person retrieval, TPAS must preserve identity cues while distinguishing fine-grained behaviors. It presents two challenges: same-identity, different-behavior hard negatives cause gradient conflict during granular alignment, while strong appearance cues can obscure weak behavioral evidence during aggregation. We propose GradCO (behavior-aware Gradient-Conflict Offloading), comprising three complementary components. Behavior-Aware Mixture-of-Experts (B-MoE) forms appearance, normal-behavior, and anomalous-behavior streams whose roles are learned from retrieval objectives without expert-specific semantic labels. Conflict Offload Learning (COL) uses pair-conditioned offload anchors in similarity computation to reduce the conflicting-gradient burden on the main text representations. Hierarchical Intent Alignment (HIA) aggregates expert similarities and features using query-conditioned weights. Experiments on five TPAS/TIPR benchmarks show GradCO significantly surpasses SOTA methods and validate our design.