Selective fine-tuning improves player arousal detection in gameplay videos

Emoception: Selective Affective Layer Fine-Tuning of Video Vision Transformers for Player Arousal Change Recognition From Gameplay Footage

Human-Computer InteractionArtificial IntelligenceComputer Vision and Pattern Recognition

Summary

This paper looks at how to tell when a video game player is feeling excited or calm by analyzing gameplay videos. The authors created a method called SALFT that changes only a small part of a video analysis model, making it faster and less costly to train. They showed that this method works just as well as changing the entire model, with much less work. They also came up with a way to see what parts of the video the model focuses on, helping people understand how the model makes decisions.

What this means in practice

  • For game development teams: Detect player emotional changes from gameplay videos with efficient model updates to improve game design feedback.
  • For user experience analysts: Analyze emotional responses in gameplay footage while minimizing computing resources to better understand player engagement.

Authors

Yi Xia, Ibrahim Khan, Mury Fajar Dewantoro, Wenwen Ouyang, Ruck Thawonmas

Abstract

This article proposes Selective Affective Layer Fine-Tuning (SALFT), an efficient adaptation framework for Video Vision Transformers in player arousal recognition from gameplay. To bypass computationally expensive full fine-tuning, SALFT introduces a selection criterion based on the L2-norm change in layer parameters after brief adaptation, directly measuring representational shifts and providing a more stable basis than gradient-based alternatives. Evaluated via five-fold cross-validation on the Arousal Video Game AnnotatIoN dataset, SALFT achieves performance comparable to full fine-tuning across all games without statistically significant degradation ($p>0.05$), while updating only $\approx$8% of parameters (over 92% reduction). Notably, in one game, SALFT consistently outperforms both full fine-tuning and the best baseline across all metrics and folds, reaching the theoretical minimum p-value (p=0.0625, exact two-sided Wilcoxon signed-rank test). In addition, we introduce an interpretability method to trace attention patterns, enhancing model transparency. These results establish SALFT as an effective and efficient approach for affective game computing.