PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model
2026-07-20 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors present PGN, a system that helps a robot understand language instructions and predict what actions to take by looking at a sequence of images. They first train parts of the model to connect vision and language, then fine-tune it using expert navigation examples while keeping some parts fixed to keep training stable. Their tests show the system can closely match expert actions offline, but they have not yet checked how well it performs in real-time navigation. Future work will evaluate how well it handles mistakes and completes goals during actual navigation.
Vision-Language NavigationMultimodal Large Language ModelVisual-Language AlignmentAction PredictionTemporal SamplingQ-FormerLoRA AdaptersTeacher-Forced EvaluationNormalized Action MatchDeepSpeed ZeRO
Authors
Li Xian, Mingxi Li, Yizheng Wang, Yiming Shen, Qi Chen, Zhuoling Xiao
Abstract
Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN requires visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B. Training proceeds in two stages. First, PGMM aligns a frozen EVA-ViT-G/14 vision encoder with the frozen language backbone by training a Q-Former and a two-layer MLP projector. Second, PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters. The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER). These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.