LangLoc: "Tell Me What You See"
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed LangLoc, a system that figures out exactly where someone is inside a building just by reading their natural language description of the surroundings. First, it finds the right room or scene using a smart language and image feature comparison. Then, it guesses the exact spot and direction by checking what objects should be visible from various points on a floor plan. Finally, it asks yes/no questions to clear up any confusion and pinpoint the location more accurately. They also created a large dataset with over 13,000 descriptions tied to precise indoor locations to test their method.
Indoor localizationNatural language processing3D environmentScene retrievalGraph Attention Network (GATv2)CLIP featuresRay castingBayesian inferencePose estimationDialog system
Authors
Shaurya Kishore Panwar, Roham Zendehdel Nobari, Shirley Feng Yi Lau, Abu Bakr Rahman Shaik, Manuel Günther, Marc Pollefeys, Daniel Barath
Abstract
We tackle fine-grained indoor localization from natural language: given a free-form description of one's surroundings, estimate the observer's 2D position and heading within a known 3D environment. Language queries are lightweight, privacy-preserving, and need no camera - yet prior work stops at coarse scene retrieval and cannot resolve an intra-scene pose. We close this gap with LangLoc, a three-stage pipeline that (i) retrieves the correct scene via a dual-branch GATv2 encoder with CLIP semantic features, surpassing the previous best by 8 percentage points in Top-1 recall; (ii) estimates position and heading by scoring a dense floor grid through ray-cast object visibility, reaching a median error of 0.95 m; and (iii) resolves residual ambiguity through a Bayesian dialog module that asks targeted yes/no questions and updates a pose posterior until the location is pinpointed. To support this task we contribute a benchmark of $13{,}000{+}$ pose-indexed natural-language descriptions over $1{,}300{+}$ indoor 3D scans.