Fpga system localizes and renders scenes for robots with low power
Localization and Neural Splatting on the Edge: An FPGA Scene Memory for Robots
RoboticsComputer Vision and Pattern Recognition
Summary
Small robots and vehicles need to remember their surroundings and figure out where they are without using big, power-hungry computers. The authors designed LENS, a system that runs on a low-power FPGA chip to quickly locate a camera and produce images of nearby views. They adapted two main techniques to work well on limited hardware and measured its speed, power, and image quality directly on the device. LENS performs better per watt and faster in some tasks than a popular edge GPU, making it useful for robots that need efficient, on-device scene understanding.
What this means in practice
- •For robotics engineers: Run accurate scene localization and novel view rendering on low-power embedded hardware to improve autonomous robot navigation.
- •For edge device developers: Implement efficient 3D scene memory and rendering pipelines on FPGA/ARM platforms to reduce power while maintaining quality and speed.
Authors
Que Haohua, Mingkai Liu, Handong Yao
Abstract
Resource-constrained robots and autonomous vehicles need scene memories that localize a camera and render nearby views without a desktop GPU. Scene-coordinate regression (SCR) and 3D Gaussian splatting (3DGS) fit because they avoid an SfM stage, but ACEsplat-style pipelines are GPU-centered, and moving them to an edge FPGA is hard because SCR is pose-sensitive and 3DGS rendering is irregular and capacity-sensitive. We present LENS, an edge scene-memory acceleration system on the AMD KR260 that brings an ACEsplat-style workload to a low-power FPGA/ARM platform. Rather than proposing a new scene model, LENS co-designs the two dominant kernels with the board data path: an INT16 SCR accelerator that preserves pose accuracy, and a bit-exact fixed-point ordered 3DGS renderer that rewrites alpha transmittance as a log-sum-exp recurrence and removes provably useless streamed Gaussians. All LENS latency, power, and quality results are board-measured. The SCR engine preserves DSAC* localization accuracy at 25.42 ms, the on-board renderer reaches 34.11 dB on Cambridge against ACEsplat's 33.20, novel-view synthesis on RealEstate10K reaches 27.84 dB against 26.11, and all ten Wayspots scenes reach the ACEsplat rendering target. LENS wins performance per watt over a Jetson Orin NX on every deployment task, by 1.9x to 5.7x at about a quarter of the power, and is faster in absolute latency on localization and Wayspots rendering. On-device training reaches ACEsplat-level quality but not the GPU's time or energy, marking the remaining boundary for a future training accelerator.