Natural language descriptions reveal gaps in accessible 3d modeling for blind makers
Beyond Being Accurate: Implications for Using Natural Language Descriptions for Accessible 3D Modeling with~Blind and Low Vision Makers
Human-Computer Interaction
Summary
People who are blind or have low vision often can’t use 3D modeling tools because those tools rely on seeing shapes. The authors studied how natural language descriptions could help by describing 3D models so blind users can edit them. They found that descriptions good enough to understand models aren’t enough for making changes, because editing needs knowing details users might not ask about. This led to a new way to evaluate and improve descriptions so they better support blind users creating 3D models.
What this means in practice
- •For assistive technology developers: Design description systems that accurately support blind users in editing 3D models by addressing identified failure types in natural language descriptions.
- •For voice interface designers: Create conversational interfaces for 3D modeling tools that guide blind users through spatial features beyond simple descriptions to enable effective model editing.
Authors
Gene S-H Kim, Nicholas Oskiper, Sean Follmer, Megan Hofmann
Abstract
People who are blind or have low vision (BLV) are largely excluded from 3D modeling tools that rely on visual representations of geometry. Natural language descriptions offer a potential path to accessibility, but it is unclear whether language alone can support the deep spatial understanding that editing, not just browsing, requires. Advances in generative AI offer the promise of automatic model description, yet descriptions suited to BLV modelers are structurally distinct from alt-text and from existing 3D model captioning datasets, and the domain-specific requirements for generating them are not well understood. We address this through two coordinated studies: an autoethnographic pilot in which the blind first author worked with 12 sighted participants to develop a taxonomy of descriptive strategies, and a Wizard-of-Oz main study that applied those strategies as idealized scripts with 10 BLV participants. We identify a systematic use-to-editing gap: descriptions that support accurate but incomplete mental models are insufficient for specifying modifications, because editing requires knowing the state of features the user did not know to ask about. We show that this gap produces four failure types (accuracy, completeness, expression, and navigation) that accuracy-focused evaluation frameworks cannot detect. We propose a two-phase evaluation framework that diagnoses the failure type a description system exhibits and guides developers toward targeted design interventions, thereby providing a methodological standard for assessing whether description-based systems genuinely support BLV modelers as creators.