Papers for

human-computer interaction developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Human movement prediction improves by learning forward and backward representations

BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations

Abstract: Learning semantically meaningful human movement representations from 3D pose sequences is essential for human behavior modeling. Existing self-supervised methods derive supervision from artificial assumptions, e.g., invariance to augmentation within a certain range or recoverability of masked regions at a certain temporal scale, introducing heuristic hyperparameters that make the learned semantics sensitive to their settings. We propose BAL, a self-supervised framework that instead exploits supervision intrinsic to human movement: the natural ordering structure of everyday behavior. BAL autoregressively predicts in a semantically abstract latent space, analogous to how humans anticipate others' actions rather than their precise joint configurations. Bidirectional autoregressive prediction, both forward and backward in time, fully leverages these ordering constraints while enabling representation extraction using both past and future context at inference. To suppress representation collapse, BAL decomposes each prediction target into a global component capturing sequence-level semantics and a local component capturing context-independent movement. We prove that the local targets retain nontrivial angular diversity unless the representations are completely collapsed, preserving a meaningful training signal. Experiments on large-scale datasets demonstrate that BAL's representations are highly effective for motion captioning, forecasting, and interpolation, outperforming baselines on most evaluation metrics.

Fri 13 NovComputer Vision and Pattern RecognitionMachine Learning
The gist
Understanding and modeling human movement is important for many applications, but existing methods often rely on tricky assumptions that can make learning inconsistent. The authors propose BAL, a new approach that learns human motion by predicting what happens next and what happened before in an abstract form instead of raw joint positions. This method uses the natural order of movements and prevents the model from collapsing into meaningless patterns by splitting predictions into overall motion and small local changes. Their experiments show that this approach produces better representations for tasks like describing movements, predicting future poses, and filling in missing motions.
Open → 2611.18559v1

Crowdsourcing multiple robot actions per state boosts real-world learning

CrowdMAPS: Crowdsourcing Multiple Actions per State for Real-World Imitation Learning

Abstract: Real-world robot imitation learning is often limited by the cost of collecting physical demonstrations, which typically yield only one action label per visited state. We present CrowdMAPS, a web-based crowdsourcing paradigm for augmenting real-robot datasets with Multiple Actions per State (MAPS). After or during standard robot data collection, CrowdMAPS distributes visited robot states to remote annotators through a web-based 3D interface, where they specify alternative valid actions without requiring access to physical robot hardware. This expands robot datasets along a complementary "width" dimension--by increasing the number of actions per visited state--rather than only by collecting more trajectories. We evaluate CrowdMAPS using keyframe-centric goal-pose annotations supplied by non-expert annotators and train imitation policies on the resulting MAPS data. Across five real-world manipulation tasks, increasing MAPS width from one to ten actions per state improves sample efficiency and increases average success rates by 35.6 percentage points for keypose policies and 33.0 percentage points for trajectory policies. These results show that software-mediated human annotations can augment physical robot data to afford denser supervision, establishing CrowdMAPS as a scalable digital layer for real-robot data collection.

Wed 11 NovRobotics
The gist
Robots learn by watching examples, but collecting these examples with different ways to act in the same moment is expensive and slow. The authors created CrowdMAPS, a web tool where many people can suggest different valid actions for the same robot situation without needing the robot itself. This extra diversity of actions helps robots learn better and succeed more often in real tasks. Their tests showed a big improvement, over 30% higher success rates, when using many actions per state rather than just one.
Open → 2611.15078v1

Mapping computer use at multiple levels reveals user and agent differences

Comparing Computer Use Behavior Across Multiple Levels of Abstraction

Abstract: Interaction traces record how people and agents use computers, and comparing them can reveal usability barriers, differences in expertise, and human--agent similarity. Yet traces are recorded as low-level, interface-specific actions, making comparisons difficult across tasks, interfaces, and input modalities. Existing approaches either compare these actions directly, reduce traces to aggregate outcomes, or summarize them at higher-level goals, obscuring where behavior differs. We present MAP (Multi-level Action Parsing), a hierarchical semantic representation and annotation pipeline that maps traces into a common vocabulary across linked levels of abstraction. MAP compares traces by sequence, recurring patterns, and effort, localizing differences by type and abstraction level. A technical evaluation across three computer use datasets shows high agreement with manual annotations and broad coverage of behavior. Applying MAP shows that sighted and blind users increasingly converge at higher levels despite navigation differences, while agents omit corrective, help-seeking, and recovery behaviors present in human traces.

Mon 19 OctHuman-Computer Interaction
The gist
Comparing how people use computers is hard because computer actions are often recorded in very detailed, specific ways. The authors created a new method called MAP that groups these detailed actions into a common, simpler language at different levels, making comparison easier and clearer. They showed MAP works well by testing it on several datasets and found interesting differences: blind and sighted users act more similarly at higher task levels, while artificial agents skip helpful human behaviors like asking for help or fixing mistakes.
Open → 2610.26703v1

Tool measures body language and speech in video and audio recordings

NICE Toolbox: Nonverbal Interpersonal Communication Exploration Toolbox

Abstract: Understanding human communication is a long-standing goal in psychological research. People express themselves through many verbal and nonverbal channels: posture, gesture, gaze, facial expression, and speech. Measurement and annotation of these signals has traditionally required time-consuming manual work by trained human labellers. The Nonverbal Interpersonal Communication Exploration (NICE) Toolbox is a free, open-source Python framework that automatically measures these signals by combining existing state-of-the-art computer vision and speech-processing methods into a single software package. It includes support for multi-camera systems, interchangeable detection algorithms, 2D and 3D visualization of detector outputs, evaluation on the user's custom data, and export to third-party software. With the release of this tool, we aim to make data exploration of interpersonal communication signals scalable, reproducible, and accessible for psychological research.

Thu 15 OctComputer Vision and Pattern Recognition
The gist
People communicate using gestures, facial expressions, tone of voice, and more, but studying these signals is usually slow and needs experts to label them. The authors created the NICE Toolbox, a free software package that automatically detects these nonverbal and verbal cues from videos and audio. It combines several advanced methods into one tool that works with multiple cameras and shows results in both 2D and 3D views. This makes it easier and faster for researchers to analyze how people interact without doing all the manual coding.
Open → 2610.21786v1

4D facial expression dataset enables deeper emotion intensity analysis

Capturing Dynamics: The 4D Facial Expression Intensity Dataset

Abstract: The estimation and analysis of facial expression intensity play a crucial role in affective communication and human-computer interaction. Previous research has primarily focused on detecting and estimating facial expression intensity from frame-level 2D representations. However, this limitation restricts a comprehensive understanding of real-world facial expressions, as they are inherently 3D and temporally continuous. This paper investigates the perception of facial expression intensity by introducing the 4D Facial Expression Intensity Dataset (4DFEID). We employ a parametric face model and compile a total of 2,869 mesh sequences with controlled geometric variations, generating 4D data instances with diverse peak intensities and identity attributes. Using a Likert scale, we collect more than 90,000 subjective intensity perception ratings via a crowdsourcing platform. We explore various architectures and aggregation methods to establish baselines for episode intensity estimation on the new dataset, revealing that spatial-temporal graph models consistently outperform traditional frame-aggregation methods. In contrast to existing datasets that rely on 2D static imagery, the proposed 4D-FEID dataset provides the community with a unique and vital resource for investigating the perception of facial expression intensity through the use of dynamic 3D stimuli. By offering high-fidelity, spatio-temporally coherent facial data, 4D-FEID establishes a new foundation for research into more nuanced and naturalistic expression analysis, thereby addressing a gap in the current landscape of affective computing and human-computer interaction studies. The dataset is available at link.

Fri 2 OctComputer Vision and Pattern Recognition
The gist
Understanding how strong facial expressions are is important for how computers can read human emotions. Past work mostly looked at flat 2D pictures, but real faces change in 3D over time. The authors created a large collection of 3D face animations showing different expression strengths and asked people to rate how intense they seemed. They tested different ways for computers to estimate these intensities and found models that consider both space and time worked best. This dataset helps researchers build better systems to analyze natural facial expressions more accurately.
Open → 2610.02647v1

EyeTAG improves gaze tracking by modeling eye movement patterns

EyeTAG: Eye Trajectory-Aware Gaze Estimation

Abstract: Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0$^\circ$ on Gaze360 and performs on par with the strongest baseline on EVE (2.56$^\circ$ vs. 2.58$^\circ$). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.

Thu 1 OctComputer Vision and Pattern RecognitionMachine Learning
The gist
Tracking where a person is looking can be shaky when their head and eyes move naturally, causing errors in measurements. The authors show that by not only looking at each eye image separately but also considering how the gaze changes over time, their method called EyeTAG gets smoother and more accurate estimates. EyeTAG uses a special way to keep track of the recent movement of the eyes and combines this with visual information to predict gaze direction. This reduces errors and helps avoid false jumps in where the system thinks someone is looking.
Open → 2610.00922v1

Vista helps machines understand mixed emotional signals in context

VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict

Abstract: Conflicting emotional cues can be individually valid: a subdued voice may reflect a blocked goal while a smile satisfies a social obligation. Their interpretation depends on what the event means to the person. We introduce VISTA (Value-Informed Semantic Trust Arbitration), a learned seven-field appraisal interface that conditions modality arbitration on concerns, event relations, and expression conditions while retaining a joint-evidence residual. A log-odds decomposition separates emotion expectation from cue diagnosticity, motivating an interface that lets appraisal change how evidence is interpreted. With a shared Qwen2.5-Omni-7B backbone and matched training examples and steps, VISTA reaches 64.5% conflict accuracy on CA-MER, improving on modality gating by 2.5 percentage points on conflict and 0.2 on consistency. Shuffling appraisal across scenes or removing its decision connection reduces this benefit. A common frozen-backbone probe reaches 0.600 macro CCC for appraisal readout, compared with 0.505 for emotion-only fine-tuning. Evaluations across five benchmarks connect recognition under increasing conflict with appraisal readout and downstream decision use. Together, the analyses and experiments support scene-specific appraisal as an intermediate representation that helps interpret conflicting emotional evidence.

Tue 29 SeptArtificial Intelligence
The gist
Sometimes different parts of a person’s expression can show conflicting emotions, like a quiet voice but a smile. The authors developed VISTA, a method that helps computers figure out what these mixed signals mean by considering what’s important to the person and the situation. This makes emotion recognition better, especially when signals don’t agree. VISTA uses a shared model that improves understanding by combining clues with an interpretation of value and expectations about feelings.
Open → 2609.37324v1

Multimodal dataset improves attention tracking with brain and eye signals

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

Abstract: Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

Fri 25 SeptHuman-Computer InteractionMachine LearningSound
The gist
People use where they look, head movements, and seeing cues to focus on someone talking in noisy places, but most studies only use brain waves to follow attention. The authors created a new dataset called MAESTRO that records brain activity, eye gaze, pupil size, video from a wearer’s perspective, and head motion all at once with multiple people talking. They found that combining these different signals helps track who a person is listening to better than just using brain waves alone. This dataset and code are publicly available for others to build better listening technology.
Open → 2609.31898v1

Latent tokens improve visual reasoning and segmentation accuracy

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models

Abstract: Reasoning segmentation aims to interpret implicit textual queries and enable fine-grained visual perception, which is critical for applications such as human-computer interaction and embodied agents. Existing methods typically generate explicit Chain-of-Thought (CoT) by multimodal large language models (MLLMs) before localizing the target. Although intuitive, such explicit verbal reasoning introduces substantial attention interference: redundant textual tokens disrupt attention during perception-token generation and also increase the effective distance between visual tokens. To address this issue, we propose LIRSeg, which fully replaces explicit CoT with a compact set of learnable latent tokens for reasoning segmentation. LIRSeg is trained in two stages: spatial alignment grounds the latent tokens in object-relevant visual evidence, and GRPO further optimizes them with segmentation rewards. To make these compact latent tokens more informative, we introduce three complementary mechanisms from an information perspective: extreme-advantage sampling for selecting informative training signals, decoupled exploration-stability updates for learning complementary representations, and latent diversity amplification for preventing representational collapse. Extensive experiments on benchmarks demonstrate that LIRSeg consistently improves both segmentation accuracy and reasoning efficiency. Compared with the VisionReasoner baseline, LIRSeg achieves absolute gIoU improvements of 4.9% on ReasonSeg, 7.1% on MUSE, and 4.7% on MMR, while achieving a approximately 16x reduction in reasoning tokens. Code is available in supplementary materials.

Fri 25 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Figuring out where things are in images based on tricky questions is hard because explaining the reasoning aloud can make the computer’s attention messy and slow. The authors designed a new way that skips talking through each reasoning step and instead uses small hidden tokens inside the model to represent the reasoning process. By teaching these tokens to connect well with objects and improve over time, their method boosts accuracy and works much faster than previous ones. This approach helps computers better understand and pinpoint objects in images without getting distracted by extra words.
Open → 2609.30783v1

Lightweight neural network recognizes hand-drawn shapes in real time

A Lightweight Convolutional Neural Network for Real-Time Recognition of Hand-Drawn Geometric Shapes

Abstract: Recognizing hand-drawn geometric shapes is a foundational sub-problem of sketch recognition, with applications in education, human-computer interaction, and diagram digitization. This paper presents the design, implementation, and evaluation of a desktop application that recognizes four basic hand-drawn geometric shapes, circle, square, rectangle, and triangle using a compact Convolutional Neural Network (CNN). A dataset of 2,000 labeled 28x28-pixel shape images was collected independently and released publicly. The classifier consists of three convolutional blocks (16, 32, and 64 filters) with max-pooling, an in-model data-augmentation stage (random horizontal flip, rotation, and zoom), a dropout-regularized dense layer of 128 units, and a 4-way linear output layer, totaling 97{,}956 trainable parameters. The network is trained with the Adam optimizer on a sparse categorical cross-entropy objective computed directly on logits. On an 80/20 train-validation split, the model achieves 94.80% training accuracy and 96.01% validation accuracy with a validation loss of 0.1437. A Tkinter-based graphical interface allows a user to draw a shape with the mouse and receive an immediate class prediction with a confidence score. We situate this system within the broader sketch and shape-recognition literature, compare its accuracy against related hand-drawn shape classification studies, and discuss the limitations inherent to a small, single-contributor dataset. The complete source code, trained model, and per-class datasets are released publicly to support reproducibility.

Mon 21 SeptComputer Vision and Pattern RecognitionMachine Learning
The gist
Recognizing simple hand-drawn shapes is useful for things like teaching and making diagrams on computers. The authors built a small neural network that can tell if a drawing is a circle, square, rectangle, or triangle almost instantly. They trained it on a small collection of 2,000 images they made and tested it to get over 95% accuracy when guessing the shapes. They also made a simple program where you can draw a shape and it will tell you what it thinks it is right away. The authors shared all their code and data so others can use or improve their work.
Open → 2609.24384v1

Multimodal dataset captures humour styles and emotions in video actors

MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions

Abstract: Computational recognition of verbal humour remains a challenging task, requiring an understanding of language, delivery style, emotions, and cultural context. Most existing approaches focus on binary classification and lack datasets that capture psychological dimensions of humour alongside variations in expression. We introduce MultiHuSE, a multimodal dataset comprising 2,407 high-definition videos of 50 demographically diverse actors performing 1,463 text samples across four psychological humour styles (affiliative, aggressive, self-enhancing, and self-deprecating), as well as neutral content. A subset is additionally annotated for underlying emotions. The dataset uniquely captures multiple actor interpretations of the same texts, enabling systematic analysis of expressive diversity. Baseline experiments show that multimodal fusion outperforms unimodal approaches (80.1% vs. 77.4% accuracy) in humour style classification, with particularly strong gains for affiliative humour (66% to 74%). While text provides the strongest individual signal, fusion models deliver meaningful improvements. We hope that MultiHuSE provides empirical support for psychological theories linking humour and emotion, while also opening new avenues for research in human communication, well-being, and AI-driven interaction. The dataset is available for academic use under an End-User Licence Agreement.

Thu 10 SeptComputation and LanguageComputer Vision and Pattern RecognitionMultimedia
The gist
Understanding humor by computers is hard because it involves language, emotions, and how something is said. The authors created a large video dataset called MultiHuSE with many actors showing different types of humor styles using the same texts. This dataset also includes emotions related to the humor and shows how the same joke can be expressed differently. They found using video, audio, and text together helps better identify humor styles than using just one type of information.
Open → 2609.11322v1