Federated reinforcement learning attacks depend on timing of manipulation
When to Intervene? State-Aware Sparse Manipulation in Federated Reinforcement Learning
Machine Learning
Summary
Federated reinforcement learning lets many agents train a shared decision-making model without centralizing data, but attackers can try to corrupt this process. The authors found that when an attacker intervenes during an agent's decision path matters a lot for how harmful the attack is. They designed a new attack method that picks critical moments based on uncertainty and uses fewer disruptions to mislead the training. This shows that timing is an important factor to consider when defending distributed learning systems.
What this means in practice
- •For distributed ai engineers: Design defenses for federated RL systems that monitor and mitigate attacks by focusing on the timing of malicious interventions in decision sequences.
- •For security analysts: Assess vulnerability of decentralized RL deployments by evaluating how timing of attacks affects overall policy corruption effectiveness.
Authors
Shutong Zheng, Sijia Chen
Abstract
Federated reinforcement learning (FRL) enables distributed agents to collaboratively train decision-making policies, but its decentralized training process also exposes global policy learning to Byzantine manipulation. Existing poisoning attacks primarily focus on how to construct malicious updates, while trajectory-level intervention timing remains largely implicit. In sequential decision making, however, where an intervention is applied can alter subsequent trajectories and learning signals. Through controlled experiments, we find that changing the selected trajectory states materially alters attack efficacy even when the malicious-update construction is fixed. We therefore identify when as a distinct attack dimension and introduce the Viability-constrained Behavioral Steering Attack (V-BSA), which uses local policy uncertainty to select sparse intervention states and applies envelope-constrained behavioral steering. Across discrete-action benchmarks, V-BSA achieves substantial degradation against robust aggregators and ensemble defenses with only a fraction of the interventions used by dense poisoning, while revealing task- and aggregation-dependent boundaries. Overall, our results highlight intervention timing as a distinct dimension of sequential robustness in FRL. The code is available at https://github.com/Yodeesy/V-BSA