Compact vision language model improves spatial awareness for low vision users
Small yet Assistive: Spatially-Aware Post-Training for Low Vision
Computer Vision and Pattern RecognitionComputation and Language
Summary
Many people with vision problems need detailed descriptions to navigate safely, but current models either run slowly or lack important spatial information. The authors created a small yet effective AI model that runs on phones and gives better directions, distances, and hazard warnings for low vision users. They trained it in stages to keep both general description quality and specialized navigation help. Their model runs offline on smartphones and improves several vision-related tasks compared to previous smaller models.
What this means in practice
- •For mobile app developers: Integrate a compact vision-language model that provides spatially detailed and hazard-aware descriptions for blind or low-vision users directly on smartphones without internet.
- •For assistive technology manufacturers: Build lightweight navigational aids offering precise spatial and textual information for visually impaired users that run offline on portable devices.$Commercial implications: Enables standalone low-vision assistance devices that improve user safety with detailed environment descriptions without relying on cloud connectivity.
Authors
Rishabh Choudhary, Shreyansh Raj, Umesh Goyal, Shubh Kashyap, Shrestha Kumar, Sushovan Jena, Komal Kumar, Hisham Cholakkal, Aditya Nigam
Abstract
An estimated 1 billion people worldwide live with vision impairment, yet current vision-language models (VLMs) produce descriptions too vague for safe navigation by blind and low-vision (BLV) users. Large VLMs can generate high-quality audio-description-compliant narrations but cannot run on mobile devices; small VLMs offer competitive latency but lack spatial detail, directional cues, and hazard awareness for navigational assistance. We present Smol-VL-BLV, a compact VLM for blind and low-vision users that closes this gap using a 500M decoder transformer model and two post-training mechanisms: (1) teacher-student distillation and (2) Group Relative Policy Optimization (GRPO) with a composite BLV reward targeting directional language, metric distances, and hazard detection. Because multi-stage post-training can induce catastrophic forgetting, we add a lightweight finetuning stage after the last stage GRPO finetuning to recover general descriptive quality while preserving BLV-specific spatial grounding. Our best model substantially outperforms the baseline across various benchmarks, including tasks: VQA, BLV captioning, OCR, and latency. Compared with the baseline for relative improvement, it improves the Spatial score gain of 19.3%, and the Social score gain of 14.8%. It also increases OCR-Bench by 101.5%, and raises TextVQA accuracy by 44.2%. These results show that BLV-focused post-training improves both accessibility-specific spatial grounding and general visual-text reasoning. Deployed on a mid-range Android smartphone via Mixed-Precision Quantization, the model remains approx. 450 MB and runs entirely on-device, offline and without network dependency, generating descriptions with latency dependent on host hardware capabilities. Our model, dataset, and code is publicly released at https://smol-vl-blv.github.io/Smol-VL-BLV-website/