Large language models optimize robot control without revealing secrets
Privacy-Preserving Prompted Policy Search for Robotic Control
Robotics
Summary
Making robots learn better control often needs sharing private details about how they work, which risks exposing important information. The authors designed a method that hides these private control details when asking large language models (LLMs) to help improve robot actions. Their approach also breaks down feedback into detailed parts so the LLM can improve policies more efficiently and keeps the information size manageable. They tested their method on various robot control tasks and showed it works better than previous LLM-based and standard learning methods.
What this means in practice
- •For robotics engineers: Improve robot control policies using cloud AI without exposing proprietary strategy data.
- •For autonomous vehicle teams: Enhance decision-making algorithms in driving systems securely while keeping sensitive control parameters private.
Authors
Ali Irshayyid, Feng Lin, Chong Li, Jun Chen
Abstract
Large language models (LLMs) have recently demonstrated promising capabilities as in-context policy optimizers for Reinforcement Learning (RL), enabling policy search driven by both numerical reward signals and natural language reasoning. However, deploying such methods in practice requires transmitting raw policy parameters and rewards history to cloud-based LLM APIs, exposing proprietary control strategies to third-party service providers. To address this issue, this paper introduces Privacy-Preserving Prompted Policy Search (PP-ProPS), a framework that enables LLM-guided policy optimization while keeping policy and environmental parameters confidential. PP-ProPS encodes policy parameters and reward values using secret client-side transformations before they are included in each API request, ensuring that the LLM provider observes only encoded policy parameters and scaled reward information. Furthermore, unlike Vanilla ProPS, the proposed framework does not require the true optimal episodic return to be known or disclosed to the LLM. Beyond protecting the optimization data, PP-ProPS improves the search process in two ways. First, it provides the LLM with individual reward components instead of only a single total return, offering more informative feedback about each candidate policy. Second, it uses a bounded history that prevents the prompt from growing indefinitely, improving search with high-dimensional policies and supporting the use of open-weight LLMs. The proposed PP-ProPS is evaluated on both continuous and discrete control problems spanning Multi-Joint dynamics with Contact (MuJoCo) locomotion, classic control, highway driving, and robotic arm manipulation. Compared to Vanilla ProPS, the proposed PP-ProPS outperforms ProPS in seven of the ten evaluated tasks, and surpasses conventional RL methods including PPO, SAC, and TRPO, in five of the six tasks.