Map aggregation for autonomous driving improved with large language models
From Fragments to Global Maps: Learning Vectorized Map Aggregation with Large Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Building detailed road maps for self-driving cars is tricky because the maps come from lots of small, messy pieces collected as a car moves. The authors created a system that uses a large language model, like those used in text processing, to smartly combine these small map pieces into one clean, big map. Their system learns from simulated errors to handle different map-making mistakes without needing special tuning for each sensor. This approach works better than previous rule-based methods on popular autonomous driving datasets.
What this means in practice
- •For autonomous vehicle developers: Create more accurate global road maps by automatically combining partial local map predictions using the language-model-based method.
- •For mapping software engineers: Improve robustness of map merging pipelines with a data-driven model that adapts to different map prediction errors without manual tuning.
Authors
Ziwei Li, Yi-Tang Chen, Xiaoqi Wang, Wenbin He, Han-Wei Shen, Liu Ren
Abstract
Large-scale vectorized HD maps provide structured road information that is essential for perception, localization, and planning in autonomous driving. Constructing such maps requires aggregating noisy, fragmented, and overlapping local predictions collected along a vehicle trajectory into a coherent global map. Existing aggregation methods typically rely on hand-crafted rules for fragment association and refinement. However, a fixed set of thresholds cannot effectively handle variations in road structures and prediction errors, often requiring detector-specific tuning or manual adjustment. To address this limitation, we propose MapMergeLLM, a data-driven framework that formulates vectorized map aggregation as conditional sequence generation with a large language model. Given serialized local vectorized maps, our model directly predicts the aggregated global map polylines. To reduce dependence on any particular upstream detector, we train the model on synthetic local maps generated from clean vector maps using corruptions that simulate representative prediction errors. We further introduce a coordinate tokenizer with geometry-aware pretraining to precisely represent map coordinates. In addition, we propose a line-level association loss that explicitly supervises correspondences between local observations of the same map element. Experiments on Argoverse2 and nuScenes using multiple recent upstream detectors demonstrate that MapMergeLLM substantially outperforms heuristic and optimization-based aggregation baselines without detector-specific retraining.