Vision language model improves player identity tracking in sports broadcasts
Native Association: Confidence-Aware Human Perception in the Wild with a Foundation VLM
Computer Vision and Pattern Recognition
Summary
Figuring out which player is which on a sports broadcast is hard because regular systems detect players and their numbers separately, often mixing up identities when players overlap. The authors used a large vision-language model fine-tuned to recognize all details about each player at once, ensuring correct linking between players and their jersey numbers. This method greatly reduces mistakes and works well across multiple sports, even knowing how confident it is about its guesses and zooming in automatically when needed.
What this means in practice
- •For sports video analysts: Identify players and their numbers reliably in broadcast footage without manual correction, improving automated sports analytics accuracy.
- •For live sports broadcast teams: Automatically track and label players in real time with confidence scores, enabling smarter camera zooming and viewer information overlays.$Commercial implications: Enables enhanced live broadcast features and automated highlight generation that can be marketed to sports networks and streaming services.
Authors
Igal Dmitriev, Ofir Liba
Abstract
Extracting who is where, on which team, wearing which number from a broadcast frame is typically done by stitching a detector, an OCR engine, and classifiers together -- and the stitching step swaps identities under occlusion. We make association native instead: a 0.77B vision-language model (Florence-2) is fine-tuned to emit all per-person attributes as one grammar-constrained sequence, with each attribute generated inside its owner's block. Output is therefore schema-valid on every frame by construction, and no post-hoc binding step exists to attach a correctly read number to the wrong player: residual misassociation is pure perception error, $\approx4\times$ rarer than zero-shot-prompted frontier APIs' (0.057 vs. 0.21-0.24). On a frozen multi-sport test set, this single pass reaches 0.95 detection F1 (APIs: 0.65-0.75). A single extra forward pass yields a per-field confidence that supports a reject option (jersey precision $0.71\rightarrow0.96$ at half coverage) and routes a training-free zoom-and-re-read for small players. Surprisingly, once the grammar is learned, further parameter-efficient tuning yields no measurable gain under the adaptation configurations we test; the identical recipe on WIDER-Attribute reaches 93.1 mAP given-box, yields the first detection-coupled end-to-end results under its standard test protocol (84.5 mAP), and reproduces the same tuning result. In this regime, the gains live in the structure, not in added weights.