Papers for

human computer interaction engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multimodal voice activity predicts turn-taking for social robot mediation

Multimodal Voice Activity Projection for Social Robot Mediation: Expected Behavior and Deployment Constraints

Abstract: Turn-taking prediction is especially relevant for social robots that act as mediators in human-human interaction, where the expected action is often not to speak, but to orient, wait, avoid interruption, or prepare a balanced intervention. This paper presents Multimodal Voice Activity Projection (MM-VAP) as a human state-aware perception layer for future robot mediation behavior. The model estimates the future evolution of the conversational floor from synchronized audio-visual evidence and derives turn-taking events such as Hold, Shift, Shift prediction, Backchannel prediction, and overlap-related states. The approach uses VA-related pretrained audio-visual encoders, LoRA adaptation, inter-speaker attention, and zero-shot event inference from future voice activity projections. Experiments on NoXi, NoXi+J, and Haru EDR support the feasibility of this formulation, especially for floor management events that can be connected to gaze preparation, active listening, and conservative intervention. Finally, the paper defines the expected robot output interface and discusses the main deployment constraints, including real-time inference, preprocessing latency, multimodal synchronization, and input-quality monitoring.

Wed 23 SeptRobotics

The gist

Knowing when to speak or stay quiet is important for robots that help people talk to each other. The authors developed a system that uses sound and video to guess who will speak next and whether someone will keep talking or let another person speak. This helps the robot behave politely by waiting, listening, or gently joining the conversation. They tested their method on several datasets and discussed how to make it work in real time with good input quality.

Open → 2609.28317v1

SignRefine improves sign language video clarity using keypoint data

SignRefine: Adapting Foundational Video Models for Sign Language Generation

Abstract: Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign language research, we present NVSign, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.

Tue 8 SeptComputer Vision and Pattern Recognition

The gist

Making videos of sign language that show clear hand and facial movements is very hard. Most video-generating tools are trained on regular videos with speech, so they create confusing signs with mistakes. The authors developed SignRefine, a new model that starts with an existing video tool and then focuses on fixing hands and faces based on simple movement signals. They also created a big collection of sign language videos to train their system. SignRefine produces clearer and more understandable signing videos that people who use sign language prefer.

Open → 2609.08496v1