AI agents cannot always act safely under tight time limits
AI Safety Considerations for Agents With Limited Time to Act
Artificial IntelligenceMachine Learning
Summary
AI systems sometimes need to make decisions quickly in complex environments where they cannot see everything. The researchers show that in such situations, even a perfect AI cannot guarantee it will always act safely. They use two example scenarios to prove this limitation. This means that to ensure AI safety, both the AI system and the specific environment it operates in must be considered together.
What this means in practice
- •For autonomous vehicle engineers: Design safety measures that explicitly account for the partial and rapid decision-making environments faced by self-driving cars.
- •For robotics system developers: Create AI robots with safety proofs tailored to their operational environments and time limits to reduce accident risks.
A theory result. No direct application yet.
Authors
Leo Zeitler, Jack Richings, Victoria Nockles
Abstract
In the wake of the increasingly public discussion about AI alignment, recent work has tried to propose specific AI architectures that behave safely. However, the proposed arguments that seemingly demonstrate proved alignment mostly neglect the environment the agent needs to act in. We discuss theoretical bounds for agent-agnostic safety guarantees in environments that can only be partially observed and within which an action is required within limited time. We introduce two realistic scenarios, one with an infinite state space and one with signal mixture. In these scenarios, we prove that even a perfect agent cannot guarantee safe behaviour. It will be argued that for any proof of AI safety or alignment, the environment and associated safe actions need to be specifically considered together with the agent.