World-model improves autonomous excavation scooping performance
From Prediction to Decision: World-Model-Guided Action Selection for Continuous Pile Excavation
Robotics
Summary
When a wheel-loader digs a pile of material, each scoop changes the shape of the pile, making future scoops different. The authors propose a system called WAM that predicts how scooping will change the pile’s shape and how much material will be loaded. WAM picks the best scooping action from several options and keeps updating its plan as the pile changes. Tests show that WAM scoops more efficiently than other approaches and can run fast enough on real machines to control them autonomously.
What this means in practice
- •For construction equipment operators: Automate wheel-loader excavation by selecting scooping actions that maximize load while adapting to terrain changes in real time.
- •For autonomous heavy machinery developers: Integrate an efficient world-model-guided decision loop to improve autonomous earth-moving task success rates and operation speed.
Authors
Ailing Zhang, Fan Gao, Song Zhang, Kawa Leong, Ziyu Wu, Yafei Wang
Abstract
Wheel-loader excavation is a sequential decision problem in which every scoop changes the terrain available to subsequent actions. A practical world model must predict action consequences accurately, rank candidates in real time, and operate inside the closed loop of a full-size machine. We present the World-Action Model (WAM), which proposes multiple scoops, rejects geometrically inadmissible candidates, jointly predicts signed terrain change and loaded volume, executes the candidate with the largest predicted load, and replans from the newly observed terrain. On 32 geometry-disjoint MinSlope test episodes, adding world-model ranking to matched diffusion proposals reduces the mean scoop count from 651.8 to 540.6 (17.1%), preserves 32/32 completion, and improves every paired episode. In a complete-system comparison, WAM completes 32/32 episodes versus 29/32 for an independently trained soft actor-critic policy. Comparisons of input representations, spatial support, and five architectures identify an accurate and efficient physics-structured predictor. We further evaluate the interface on event-disjoint full-size-loader data and deploy the complete perception-proposal-prediction-selection-execution loop for autonomous excavation. The ROS2/TensorRT implementation processes five candidates in 72.4 ms on a Jetson AGX Orin. The simulation results establish decision-level gains, while the physical experiments demonstrate real-world closed-loop feasibility.