Safe multi-agent training improves teamwork and safety in control tasks

SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning

Machine LearningArtificial Intelligence

Summary

When multiple agents work together, their actions must meet shared safety rules, but it's hard to tell which agent causes a problem. The authors introduce SafeDiSTL, a method that figures out each agent’s role in joint safety issues by measuring how sensitive these safety violations are to their actions. Using a logical way to track safety over time, it breaks down joint problems into individual contributions. This helps train teams to avoid unsafe actions while working well together in tasks like carrying loads or controlling traffic.

What this means in practice

  • For robotics engineers: Train teams of robots to coordinate safely during cooperative tasks such as payload transport with better individual action crediting for safety.
  • For traffic system operators: Optimize networked traffic control policies to increase throughput while ensuring joint safety constraints among multiple controllers.

Authors

Hongrui Zheng, Aditya Singh, Rahul Mangharam

Abstract

Cooperative multi-agent tasks often impose constraints on a shared objective or collective safety. A team level violation cost function can measure this constraint outcome, but does not distinguish which agents' actions most strongly influence that violation. We present SafeDiSTL, a constrained multi-agent reinforcement learning framework that allows for credit assignment even with joint constraint measurements. We use Signal Temporal Logic (STL) robustness to measure episode-level violation through running-maximum cost increments. Smooth full-episode robustness and differentiable dynamics provide action sensitivities, which allocate each cost increment across agents at the same decision time. These sensitivities account for delayed effects when weighting agents' shares of each cost increment. We study the framework in cooperative payload transport, connected navigation, and networked traffic control tasks. SafeDiSTL achieves 1.6-5.1x the safe-success rate in transport, improves navigation safe-success by 11.1%, and delivers 50-68% more throughput in traffic when compared to constrained baselines. Results demonstrate a strong improvement of the learned policy in dealing with shared constraint while retaining the performance of unconstrained policies.