Homomorphic advantage operator stabilizes encrypted reinforcement learning
Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints
Artificial IntelligenceCryptography and Security
Summary
Reinforcement learning uses trial and error to teach computers how to make decisions, but keeping the data private during this process is hard. The paper shows that applying fully homomorphic encryption, which lets computers work on encrypted data, causes errors that make learning unstable. The researchers created a new method called the Homomorphic Advantage Operator that stops these errors from growing, keeping the learning process stable and accurate. They tested this method in different scenarios and found it worked well, even when extra noise was added to the learning.
What this means in practice
- •For cloud ai engineers: Securely train reinforcement learning models on confidential data in the cloud without losing learning stability due to encryption errors.
- •For logistics optimization teams: Run encrypted reinforcement learning for routing and scheduling problems while maintaining data privacy and stable model training under noisy conditions.
Authors
Abid Mohamed Nadhir, Ahmad Al Hanbali, Beggas Mounir
Abstract
Privacy-preserving machine learning presents significant deployment challenges on the cloud for intelligent systems with confidential data. Fully Homomorphic Encryption (FHE) offers a compelling solution for secure computation, preserving data confidentiality of cloud computations. However, applying FHE to reinforcement learning (RL) requires replacing non-linear operations with polynomial approximations, which diverge catastrophically due to a unique recursive error phenomenon known as the Bellman drift. This article introduces the Homomorphic Advantage Operator (HAO), a stabilization framework designed to prevent polynomial approximation divergence in FHE-based deep RL. HAO adapts the zero-mean centering projection from advantage-based value estimation directly to temporal-difference (TD) targets. This linear projection annihilates the uniform state-value baseline that drives the Bellman drift, maintaining per-state action rankings while requiring zero additional non-linear multiplicative depth and avoiding expensive ciphertext bootstrapping. The proposed HAO framework was evaluated using a three-tier experimental methodology, including a tabular Markov Decision Process (MDP), an encrypted CartPole environment using real CKKS cryptographic operations, and a 20-node logistics routing benchmark with dense continuous features. The results demonstrate that the proposed HAO strictly bounds network pre-activations within the safe polynomial approximation domain. The proposed HAO RL agents achieved 0% boundary breaches across all random seeds used, whereas regularization alone (L2 weight decay and gradient clipping) breached the bound on 3 of 5 seeds and the unstabilized baseline did so in 83.8% of episodes. Finally, HAO agents improve optimal policy accuracy by 18.0 percentage points in tabular domains and remain stable when DP-SGD-style Gaussian noise is added to the clipped gradients.