Semreward-vl improves video descriptions for child behavior assessments
SemReward-VL: Semantic Reward-Guided Video-Language Adaptation for Developmental Behavior Assessment
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Videos show how children behave during developmental tests, but the written records usually describe only outcomes, not the details. The authors created SemReward-VL, a system that learns to describe specific behaviors seen in these videos by comparing generated descriptions to clinical results. It uses a vision-language model and a language model to judge how well descriptions match the recorded outcomes and focuses learning accordingly. While it improves accuracy on many test items, it still struggles with timing, duration, and age-related behavior interpretation.
What this means in practice
- •For clinical software developers: Create tools that generate detailed behavior descriptions from developmental screening videos to aid clinical documentation.$Commercial implications: The paper enables products that automatically summarize child behavior in videos for healthcare providers to improve patient records.
- •For speech and language therapists: Assist therapists by providing semantically accurate video summaries aligned with clinical outcomes for developmental behavior tracking.
Authors
De Jiang, Shuo Zhang, Kehong Yuan, Hongen Liao
Abstract
Developmental screening videos show how children perform specific behaviors, but clinical records usually contain outcomes rather than descriptions of what happened. We present SemReward-VL, which learns to describe item-specific behavior from these outcomes. A vision-language model generates a description, and a frozen language model scores its agreement with the clinical outcome, relevance to the item, abstention on unrelated video-item pairs, and clarity. Group relative policy optimization (GRPO) updates LoRA adapters using this semantic reward. On 13,379 videos covering 41 items, the method improves aggregate accuracy and the number of items with recall above 0.5. Errors remain in temporal direction, duration, and age-specific interpretations of behavior.