Reinforcement learning improves packing boxes safely and efficiently
SHORE: Stability-Hybrid On-Policy Reinforcement Learning for Online Bin Packing
RoboticsMachine Learning
Summary
Packing boxes in a container can be tricky because boxes might fall if stacked unstably, and you don't know what boxes will come next. The authors present a method called SHORE that learns to predict the stability of placing each box and uses that prediction to pack more boxes safely and tightly. This method learns faster and achieves better space use than earlier approaches, and it was also tested on a real robot. SHORE balances the risk of unstable packing against fitting more boxes, improving how well the space is used.
What this means in practice
- •For warehouse automation teams: Implement packing policies that dynamically assess stack stability to improve space usage and reduce packing time in online box sorting.
- •For robotic system integrators: Integrate learned stability prediction models to enhance robotic bin packing performance without costly real-time search or simulation.
- •For logistics software developers: Develop smarter packing algorithms for logistics platforms that can handle unpredictable box arrivals and optimize space utilization.$Commercial implications: Enables shipping and warehouse software companies to offer improved packing optimization features reducing shipping costs with minimal latency.
Authors
Chang Shi, Ann Majewicz Fey
Abstract
Online 3D bin packing with a processing buffer requires choosing which box to pack, its orientation, and its placement while keeping the stack stable under unknown future arrivals. A collapse ends the episode and forfeits the remaining volume. Existing methods use geometric rules or hard stability masks, limiting how risk can influence the joint choice of box, orientation, and placement. We propose Stability-Hybrid On-Policy Reinforcement Learning (SHORE), which combines a learned stability probability of a placement with an on-policy estimate of its return conditional on stability. A 3D U-Net learns stability from executed outcomes and simulated alternatives, revealing outcomes of actions underrepresented in early rollouts. A discrete on-policy actor--critic uses stability predictions in both its action values and policy prior, with gates that limit the effect of an untrained or changing model. In deployment, SHORE does not require additional search or simulation overhead. In Isaac Gym, SHORE reaches 73\% space utilization by 0.4M environment steps, while the same learner without stability modeling reaches that level at about 1M. On the expanded 60-box set, SHORE achieves 0.734 space utilization versus 0.588 for the GOPT state-of-the-art baseline, with a higher placement-failure frequency (1.03\% vs. 0.63\%), trading more risk for greater packed volume. The results show faster learning and higher utilization on the more demanding realistic box arrival set. Ablations suggest that simulated alternatives provide useful early supervision, while executed outcomes alone sustain later performance. We also demonstrate the method on a real robot to verify the simplicity of integrating the method to real bin packing pipelines.