Papers for

assistive technology engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Content descriptions improve in Android app accessibility data sets

Trained on What Screen Readers Cannot Read: Accessibility Debt in the GUI Corpora Behind Computer-Use Agents

Abstract: Computer-use agents learn to operate mobile interfaces from corpora of captured screens and accessibility trees. The same trees are what a screen reader consumes, so an agent trained on them inherits whatever labelling gaps the underlying applications carry. This paper measures one of those gaps, content-description coverage, in two Android corpora: Rico, collected in 2017, and AndroidControl, collected in 2024. Across 17,373,614 interface elements, 17.74% of the Rico elements that need a content description under our coding rule carry one, against 57.27% in AndroidControl. The difference of 39.5 percentage points holds under five alternative coding rules and when whole apps are resampled. It describes two datasets that also differ in app selection, capture method and platform version; it does not measure change over time. Input controls are the least labelled role in both corpora (2.20% and 27.21%, or 30.99% in AndroidControl if hint text counts as a name). Descriptions in the older corpus are also five times more likely to be placeholders (10.25% against 1.95%), so the difference is not an artefact of developers typing filler. Resolving 26,205 task actions to the elements they touched shows that, among tappable graphics with no text, demonstrations operate elements described 9.0 points more often than the others on the screens they visit, about 3.5 points of it from the choice of element within a screen. That is a selection effect inside the benchmark. A two-coder reliability check (kappa = 0.91 and 0.96) and the full pipeline accompany the paper.

Wed 11 NovHuman-Computer InteractionArtificial Intelligence
The gist
Many apps on phones have parts that screen readers use to help people with vision difficulties. The authors looked at two large collections of Android app screens, one from 2017 and one from 2024, to see how many app elements had helpful descriptions for those screen readers. They found the newer collection had more descriptions, but many controls still lacked them, which makes it harder for screen readers to work well. This gap also affects AI agents that learn how to use apps from these descriptions, inheriting the same missing information.
Open → 2611.16107v1

Audio visual segmentation improves by aligning sounds with image regions

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Abstract: Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.

Fri 6 NovComputer Vision and Pattern Recognition
The gist
Audio-Visual Segmentation (AVS) is about finding and outlining objects in videos that are making sounds by looking at both the pictures and the sounds together. The authors found that previous methods had trouble because the audio and visual information often don’t match well or get mixed up. They created a new technique that better aligns the sounds with the right parts of the image using contrastive learning and prototypes that represent categories. Their method performs better than others on common tests and works well even in different situations.
Open → 2611.09191v1

Large language models struggle to verbalize math formulas accurately

How Well Can LLMs Verbalize Mathematical Formulas? A Multi-Domain Evaluation

Abstract: Large Language Models (LLMs) have shown strong capabilities across natural language tasks. But their ability to generate accurate written verbalizations of mathematical formulas has not been systematically evaluated. Such verbalization is important for text-to-speech systems supporting audio-based learning and visually impaired users. We evaluated five state-of-the-art LLMs across six scientific domains and constructed Read-Aloud Math, a balanced multi-domain corpus with LaTeX formulas, manually validated simple and extended transcriptions, multiple variants, textual context, domain labels, and formula complexity levels. Using multi-reference BLEU, GPT-5.2 achieved the highest score for simple transcriptions (75.9% without context, 64.9% with context), although without context it did not significantly outperform Grok 4.1 Fast or Claude Sonnet 4.5. Claude Sonnet 4.5 achieved the highest score for extended transcriptions without context (36.5%), and Grok 4.1 Fast with context (48.8%). For extended transcriptions, Statistics and Mathematics scored highest across the two context settings, while Physics scored lowest. Formula length was significantly negatively associated with BLEU for simple transcriptions without context, but not with context, whereas BLEU increased significantly with formula length for extended transcriptions in both settings. Grouping symbols were most frequently involved in transcription differences, while variable names, capitalization, and Greek letters were associated with lower BLEU scores. Overall, LLM-based formula verbalization is promising but remains sensitive to transcription type, textual context, and formula characteristics.

Sat 17 OctComputation and LanguageArtificial Intelligence
The gist
It can be hard for computers to read math formulas out loud in a clear, accurate way. This paper tests how well popular language models do at turning math formulas into words across different science subjects. The results show some success, but accuracy depends a lot on the formula's length, complexity, and extra context. Certain math symbols and variable names cause more mistakes. Overall, the models can help but still make errors in verbalizing math.
Open → 2610.24917v1

Brain signals aligned with text enable imagined speech decoding

Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech

Abstract: Imagined speech decoding from EEG offers a promising route toward naturalistic brain-computer interfaces. Traditional closed-set classifiers treat semantic categories as independent labels and do not explicitly exploit relationships among linguistic intents, limiting output flexibility. We adapt a multimodal contrastive learning framework to imagined speech, aligning EEG trials with language representations in the embedding space of a frozen pretrained text encoder. An EEG encoder is trained with a composite objective combining semantically reweighted EEG-sentence alignment and text-prototype-based class discrimination, encouraging semantic structure while preserving separability among 39 intent categories. We evaluate the approach on the Chisco imagined speech dataset across five subjects, reporting Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2). We also investigate maximum EEG-text cosine similarity (MaxCosine) as an intrinsic indicator of decoding reliability and probe semantic structure beyond category labels using five evaluation-time semantic axes and a qualitative Semantic Gallery Probe (SGP) with previously unseen candidate sentences. The contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier while mapping EEG into a frozen language embedding space. MaxCosine identified subsets of held-out trials with higher Top-1 and Top-5 accuracy across subjects. Evaluation-time semantic axes showed mean agreement above the 50% chance reference across all five dimensions, and the SGP qualitatively illustrated direct comparison with unseen candidate sentences. Together, these findings suggest that EEG-language alignment can retain comparable fixed-intent decoding while supporting similarity-based retrieval, intrinsic reliability ranking, and evaluation of broader semantic structure.

Wed 14 OctMachine Learning
The gist
Decoding what people imagine saying based on their brain signals is tricky because traditional methods treat each spoken idea as completely separate. The authors developed a new way to link brain activity to the meaning of words by training a system that matches brain patterns with text embeddings from a language model. This method performed as well as existing classifiers but also lets the system judge how confident it is and compare new imagined phrases to known meanings. Their tests involved multiple subjects and showed promises for more flexible and reliable brain-computer communication.
Open → 2610.20766v1

MindVLM turns brain activity into personalized image descriptions

MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection

Abstract: Translating visual brain activity into natural language offers a powerful window into human cognition and neural semantic representation. Most prior brain-captioning approaches project fMRI signals into language space using paired supervision. However, the widely used Natural Scenes Dataset (NSD) was built based only on image-fMRI pairs, while existing methods rely on MS-COCO captions as indirect textual supervision. Critically, during fMRI acquisition, participants observe only cropped image regions, whereas the associated MS-COCO captions describe full scenes. This mismatch creates a fundamental gap between neural representations and textual supervision, introducing an inherent semantic loss in NSD-based brain captioning. To mitigate this limitation, we reformulate brain captioning as Neural-Grounded Visual Captioning, where language is generated directly from visually grounded neural evidence rather than text supervision. We propose MindVLM, a framework that maps fMRI signals into subject-aware visual tokens and leverages a frozen vision language model (VLM) to produce interpretable and personalized descriptions. Instead of point-wise regression, MindVLM focuses on extracting semantically relevant visual cues from neural activity, naturally accommodating partial observability and inter-subject variability. Extensive experiments on NSD show that MindVLM consistently improves caption quality across multiple VLM backbones, including LLaVA-v1.5, Qwen2.5-VL, and Gemma3, beyond intrinsic model priors. Despite requiring neither text supervision nor an LLM during training and operating under an image-grounded inference paradigm distinct from conventional captioning methods, MindVLM remains competitive with supervised approaches, suggesting that subject-specific neural grounding provides an efficient, scalable, and interpretable path toward faithful brain-to-language generation.

Thu 8 OctComputational Engineering, Finance, and Science
The gist
Translating what people see in their brains directly into words is hard because brain data and text descriptions don’t always match. The authors found that existing methods use captions describing whole images, but brain scans often reflect only parts of an image someone is looking at. To fix this, they created MindVLM, which uses brain signals to select visual cues before generating captions, making the output more personalized and accurate. This method works without relying on direct text training and performs well compared to traditional captioning systems.
Open → 2610.12673v1

Neuromotor hierarchy network improves muscle signal decoding accuracy

Neuromotor Hierarchy Network: Physiological Inductive Biases for Robust Generalization in sEMG Decoding

Abstract: Surface electromyography (sEMG) provides a wearable, noninvasive interface to neuromuscular activity for movement decoding and human-computer interaction. Population-scale decoding remains difficult because the relationship between sEMG and neuromuscular activity varies across users and sessions, while task-relevant dynamics span channels and multiple timescales. Learning waveform-to-output mappings from task labels leaves the distinction between recording variability and coordinated motor activity implicit. We introduce the Neuromotor Hierarchy Network (NHN), which learns a compact latent neuromotor state from task supervision to represent task-relevant neuromuscular coordination. NHN constructs this latent state through a hierarchy inspired by neuromotor organization.It adapts recording statistics while preserving relative intensity.Its spatiotemporal encoder uses parameter-efficient channel interactions and modulates features with multi-timescale history. The resulting features yield candidate activations of learned motor primitives, which are temporally integrated and continuously weighted to form the state. Theoretical analysis characterizes the efficiency, temporal behavior, and optimization of NHN's core mechanisms. We evaluate the architecture for both continuous hand-pose estimation on emg2pose and touch-typing recognition on emg2qwerty. On emg2pose, NHN reduces user-averaged angular error by 0.52% to 2.84% across all three generalization splits in both Regression and Tracking relative to Hadidi et al.'s best task-specific variants, using 48.42% to 48.51% fewer parameters. On emg2qwerty, NHN reduces beam-search character error rate by 19.40% zero-shot and 30.42% after fine-tuning relative to SplashNet-Upscale, using 65.86% fewer parameters. Physiology-guided inference of a latent neuromotor state supports parameter-efficient sEMG decoding.

Tue 6 OctMachine Learning
The gist
Decoding muscle signals from the skin is tricky because signals change between people and over time. The authors introduce a new neural network called the Neuromotor Hierarchy Network that learns important muscle coordination patterns to better interpret these signals. Their method uses a layered design mimicking how muscles work together and adjusts to different recording conditions. Tests show it achieves more accurate and efficient movement and typing predictions than earlier methods.
Open → 2610.07713v1

LiteEMG-FM improves real-time muscle signal recognition for devices

LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing

Abstract: Electromyography (EMG) signals vary substantially across individuals, body regions, recording sessions, and sensing hardware, limiting the generalization of models for assistive devices and human-computer interaction. Existing time-series foundation models are also computationally expensive for real-time wearable deployment and often fail to capture EMG-specific time-frequency characteristics. We present LiteEMG-FM, an efficient hybrid CNN-Transformer foundation model for practical EMG sensing. Pretrained on 16 diverse upper- and lower-limb EMG datasets, LiteEMG-FM learns representations that generalize across users and datasets. For resource-constrained deployment, we implement a hierarchical wake-up architecture in which a lightweight, always-on 1D-CNN filters rest and non-target activity and activates LiteEMG-FM only for valid gestures. We evaluate full inference offloading, split inference, and full on-device processing, characterizing their trade-offs in latency, power consumption, and memory footprint. Across diverse evaluation settings, LiteEMG-FM outperforms state-of-the-art time-series foundation models and supervised baselines, particularly under zero-calibration cross-participant and data-scarce conditions. These results demonstrate that LiteEMG-FM is an effective, efficient, and deployable foundation model for EMG applications.

Thu 1 OctMachine Learning
The gist
Muscle electrical signals change a lot depending on who you are, where on the body you measure, and the device used. This makes it hard to build devices that understand these signals well for everyone. The authors created LiteEMG-FM, a smart model trained on many different muscle signal datasets, which works well across people and devices. It also uses a clever method to save battery by only turning on fully when needed. This model performs better than previous ones, especially when there isn’t much training data or no user-specific setup.
Open → 2610.02497v1

Removing timing clues improves brain-to-text speech decoding

Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text

Abstract: We find that major reported improvements in decoding words from non-invasive brain recordings are largely reproducible without any brain data. In the influential work of d'Ascoli et al. (2025), time series of brain activity from subjects perceiving continuous speech are segmented into fixed-length windows starting at each word. A neural network then generates predictions for all of the words in a sentence together. Neighbouring windows partially overlap, implicitly revealing the interval between words. Since these intervals indicate the duration of the words spoken, and different words tend to have different durations - for example, "the" is much shorter than "supercalifragilisticexpialidocious" - the neural network can improve its predictions of words without relying on the underlying brain activity. Consistent with this, the method reaches 22.0% balanced accuracy on synthetic signals containing no brain information, compared with 22.3% on real brain recordings. To prevent the network from learning this shortcut, we make a single, simple change. Instead of jointly encoding all windows in a sentence, we process each independently. As a result, the neural network achieves better performance by learning underlying word-specific information from brain recordings. This makes two existing strategies become much more effective than before. Both aggregating predictions from distinct neural responses to the same word and using a pretrained LLM as a linguistic prior now substantially improve results. On our perceived speech benchmark, this simple recipe (SimpleB2T) achieves a word error rate of 36.6% with five observations per word, approaching past invasive speech decoding performance, albeit under different conditions. The results in this work expose an important shortcut in brain-to-text decoding and show that removing it leads to a simple and considerably more effective strategy.

Wed 30 SeptMachine Learning
The gist
Decoding words directly from brain signals when hearing speech is very challenging. The authors found previous methods partly guessed words by noticing the time gaps between spoken words, not the brain activity itself. By processing each word’s brain data independently, this timing shortcut is removed, making the decoding rely more on real brain signals. This change notably improves the accuracy of predicting heard words, bringing non-invasive brain decoding closer to invasive methods.
Open → 2609.40359v1

Audio visual system predicts speaker turns to isolate voices online

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

Abstract: In face-to-face, real-time communication, a talking agent must track the target speaker through natural pauses, turn-taking, and backchannels, often amid background cross-talk. Most target speaker extraction (TSE) studies, however, rely on simulated mixtures with full or sparse overlap and ignore the turn-taking of real conversations. We therefore introduce, to our knowledge, the first benchmark for online audio-visual TSE (AV-TSE), built from intact dyadic interactions with independent third-party interference. Observing that anticipating upcoming activity from semantic, acoustic, and facial cues benefits online AV-TSE, we propose an LLM-based target-speaker voice activity projection (TS-VAP) module. Unlike conventional VAP with separated speaker channels, it forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-latency separator. We further combine this predictive context with historical and synchronous speaker context. Experiments show that TS-VAP consistently improves streaming extraction across multiple AV-TSE backbones, and that further combining historical, synchronous, and predictive context yields nearly 1 dB gain on real AV conversations. Project page: https://jjjjiaozi.github.io/TS-VAP/.

Fri 25 SeptSound
The gist
Listening to one person speaking clearly during a conversation is hard when others talk at the same time. The authors created a new test setup using real two-person talks with unrelated background noise, rather than artificial mixtures. They designed a way to predict when a person will start or stop talking by using language understanding and facial cues. This prediction helps separate the right voice from all sounds quickly while the conversation happens. Their experiments show that guessing future talk times helps improve the clarity of the chosen speaker’s voice.
Open → 2609.30774v1

BranchShine-CR improves multilingual phonetic transcription accuracy efficiently

BranchShine-CR: Compact Multilingual IPA Transcription with Self-Conditioned CTC and Consistency Regularization

Abstract: We introduce BranchShine-CR, a 25M-parameter model for multilingual transcription into the International Phonetic Alphabet (IPA). It combines log-mel features, a rotary-position E-Branchformer encoder, intermediate self-conditioned connectionist temporal classification (CTC), and consistency regularization across augmented views. On 16,646 shared IPApack++ test utterances, it achieves 4.47% IPA character error rate, a 22.3% relative reduction from ZIPA-CTC-NS, with approximately one-twelfth as many parameters while being trained from scratch. BranchShine-CR also outperforms a similarly sized NeMo Conformer baseline across all 41 dataset language labels. Ablation studies indicate the individual components synergetically acting in model performance contribution. These findings support compact IPA recognition capabilities under limited compute budget, for applications in low-resource on-device pronunciation assessment.

Thu 24 SeptMachine Learning
The gist
Transcribing many languages into a universal set of speech sounds called IPA is hard, especially on small devices. The authors introduce BranchShine-CR, a compact machine-learning model that understands many languages and transcribes their sounds more accurately than previous models with fewer parameters. It uses a combination of new techniques to stay accurate while being lightweight. This makes it easier to add detailed speech recognition into small devices even when computing resources are limited.
Open → 2609.29069v1

Audio visual speech recognition struggles outside broadcast settings

AVSRBench: A Multi-Condition AVSR Benchmark

Abstract: While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true generalization or just domain adaptation. To investigate this gap, we evaluate three AVSR architectures across six conditions: controlled broadcast speech, fixed-grammar utterances, hyper-articulated Lombard speech, read speech from professional lipspeakers and non-professional speakers, and spontaneous multi-party video conversations. We find that visual-only performance deteriorates rapidly beyond broadcast domains, and audio-video fusion mainly benefits Lombard speech environments. Visual understanding degrades sharply at 90° profile views, with multimodal systems relying largely on acoustic fallback. Additionally, speaker articulation proves more critical than minor camera shifts, and LLM-based architectures suffer from poor out-of-domain generalization. Our work highlights a significant generalization gap in current AVSR research. To address this, we also introduce RoomReader-AV as a new benchmark for AVSR and release a unified data preprocessing pipeline to make comprehensive multi-condition evaluation accessible.

Wed 9 SeptComputer Vision and Pattern RecognitionMultimedia
The gist
Speech recognition that uses both sound and lip movements works very well on TV broadcast speech, but the authors found it struggles when used in everyday situations like casual conversations or unusual speaking styles. They tested different systems in a variety of conditions and found that visual-only recognition fails quickly outside broadcast video, and even combined audio-visual systems mostly rely on sound when the speaker is not facing the camera. This shows current speech recognition technology does not generalize well to real-world video settings. The authors also offer a new dataset and tools to help researchers better test their systems in diverse conditions.
Open → 2609.10366v1