assistive technology developers

Abstract: Multimodal AI, powered by Large Language Models (LLMs) and Vision-Language Models (VLMs), is transforming assistive technologies by enabling simultaneous processing of visual and textual data. This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide who face persistent challenges in navigating indoor and outdoor environments due to limited spatial awareness and insufficient environmental cues. Existing navigation aids often lack comprehensive 3D scene understanding, relying on constrained route-based strategies that hinder user autonomy. In this paper, we introduce a novel end-to-end framework that integrates LLMs, VLMs and digital twin technologies to deliver a spatially cognitive navigation support for visually impaired and neuro-divergent users. Our system captures video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds from monocular RGB sequences in real-time. Our custom post-processing algorithm ensures accurate point cloud alignment across multiple viewpoints without requiring predefined reference points. This enhances the capabilities of SpatialLM to produce structured 3D representations, including architectural elements and oriented object bounding boxes. The enriched spatial data is then processed by a locally deployed LLM, which interprets 3D contexts to generate detailed scene descriptions and precise distance measurements between users and surrounding objects. We evaluated our approach across diverse video scenarios featuring various perspectives, looped walking views and captured in multiple environments. The evaluation results demonstrate consistent accuracy in 3D scene interpretation and object localisation, underscoring the potential of our system as a transformative assistive navigation solution that combines advanced visual perception with spatial reasoning

Abstract: Braille literacy among blind school-age children has fallen sharply, in part because producing illustrated braille pages still requires specialized software and trained labor. We present Tact, a browser-based pipeline that converts a spoken or typed story idea into printable braille with a matching raised tactile illustration, without an account or mandatory cost and with an offline-capable path. The paper documents the engineering history of the system: its sighted-operator ethical model; hardware rationale for consumer fused-deposition modeling; physical braille geometry and printer calibration; local, hosted, and fallback language-model paths; a deterministic Grade 1 braille translator; verified page layout and pagination; a 93-shape hand-drawn tactile illustration library; and synthesized sound design for a voice-first interface. We report engineering verification, ethical commitments, limitations, and the work required before the system is ready for real blind and low-vision readers.

Abstract: Electroencephalography (EEG) is a fundamental tool for understanding the brain's electrical activity related to human motor activities. Brain-Computer Interface (BCI) uses such electrical activity to develop assistive technologies, especially those directed at people with physical disabilities. However, extracting signal features and patterns is still complex, sometimes delegated to machine learning (ML) algorithms. Therefore, this work aims to develop a ML based on the Random Forest algorithm to classify EEG signals from subjects performing real and imagery motor activities. The interpretation and correct classification of EEG signals allow the development of tools controlled by cognitive processes. We evaluated our ML Random Forest algorithm using a consumer and a research-grade EEG system. Random Forest efficiently distinguishes imagery and real activities and defines the related body part, even with consumer-grade EEG. However, interpersonal variability of the EEG signals negatively affects the classification process.

Abstract: P300 brain-computer interface (BCI) spellers can provide hands-free communication for people with severe motor impairments. Modern pipelines combine multiple individually promising components, often assuming that 'more-is-better'. We tested this assumption using a four-component full-factorial experiment varying the inclusion of Euclidean Alignment (EA), xDAWN spatial filtering, subject calibration, and language model priors on a public P300 dataset. Performance was evaluated using accuracy, repetitions, and information transfer rate (ITR) with mixed-effects models. Results show that the value of components is conditional rather than additive. Calibration was the strongest singular contributor, while EA compensated for its absence in zero-calibration settings. Adding independently useful components could also reduce performance, revealing component anti-synergy. Contrary to conventional wisdom, LM support was not universally beneficial: its effect depends strongly on the strength of the underlying EEG pipeline, while results from a larger LM showed a similar pattern. Together, these findings challenge maximal 'all-on' pipeline design and highlight the value of selecting spatial and language-support components according to the quality of available EEG evidence.

Abstract: Non-invasive speech decoding remains constrained by the low signal-to-noise ratio of neural recordings, which makes fine-grained reconstruction of phonemes or individual words difficult. Motivated by neuroscientific evidence that high-level semantic representations are distributed across cortical regions and evolve over slower temporal scales, we hypothesize that semantic content may provide a more suitable target for non-invasive decoding than low-level acoustic or lexical features. We introduce Brain2Semantics2Text, a method that reconstructs text through an intermediate semantic embedding space. Our model maps sentence-level MEG responses into a semantic manifold and then inverts the predicted embeddings into natural language. This semantic bottleneck enables recovery of high-level meaning without word-level alignment. We describe the core principles of the approach, its implementation, and the strategies used to mitigate the challenges of learning a reliable neural-to-semantic mapping. Finally, we compare against prior non-invasive Brain2Text methods and show improved sentence-level results.

Abstract: We describe a participatory design approach toward developing Deaf-centric text-to-speech (TTS) technologies. While TTS is growing rapidly in the mainstream, it has received little attention to date in the deaf and hard of hearing (DHH) technology space. Critical problems have remained unaddressed for DHH users, including the ability to manipulate tone, emotions and delivery via non-auditory means. Verifying that the generated speech matches intent and is appropriate for a given situation without having to listen to it is another challenge. Respecting cultural and identity factors in the generated speech is also important. This work explores the design space with DHH participants through two focus groups, three co-design sessions, and four one-on-one early-stage design evaluation sessions. Participants included people both familiar and unfamiliar with TTS, as well as DHH content creators. We describe key findings, design ideas, results, and implications for future Deaf-centric TTS development. We also identify unmet technology requirements that pose barriers to adoption of Deaf-centric TTS technology.

Abstract: P300 spellers offer people with severe motor impairment, such as ALS, an effective communication channel and remain one of the most established surgery-free alternatives to intracortical interfaces. Advanced language models have made spellers faster and more robust, yet the hardware beneath them is under-studied. We present a hardware-agnostic, real-time P300 acquisition pipeline built on BrainFlow and Lab Streaming Layer (LSL) that runs unchanged across consumer- and research-grade EEG headsets, with permutation tests of signal separability. Using a standard 6 x 6 row/column paradigm, we piloted five configurations: a custom dry system, a custom wet/gel system, Emotiv Flex, Emotiv EPOC X, and Muse 2. The custom systems and EPOC X showed weak or inconsistent signal separability, Muse 2 had the highest acquisition reliability despite limited centro-parietal coverage, and Flex showed the most promising signal. In 20 further Flex sessions varying subject, timing, and phrase length (131 target characters), a peak-amplitude permutation test and a cross-validated xDAWN decoder both detected a significant target response under two channel-exclusion policies, with decoder AUC reaching about 0.72 after 15 repetitions. Character accuracy depended heavily on evaluation methodology: in-sample majority voting reached 94.7%, whereas character-held-out accuracy was 31.3% with evidence accumulated across repetitions, about three times that of held-out majority voting. These analyses indicate that Flex captured a detectable, if still weak, P300 under the studied conditions, while broader participant-level validation and improved decoding remain necessary.

Abstract: This paper presents a marker-free eye-gaze estimation approach using a single 2D camera, such as an integrated laptop webcam. The gaze-related features are estimated from iris localization and head pose estimated by using depth from defocus. A variational Bayesian multinomial logistic regression framework is used as mapping from the estimated features to the position of regard, based on an 8-dimensional feature vector of head-pose and iris-displacement parameters. No external marker is needed. Experiments were conducted by estimating the gaze of people watching a computer screen at different distances and compared against five existing methods. The obtained scores demonstrate the effectiveness of the proposed approach.