AI summaryⓘ
The authors developed a system called the Ethical Decision Head (EDH) for self-driving cars, which uses deep reinforcement learning to make ethically informed driving choices based on two moral theories: Utilitarianism (minimizing total harm) and Kantian ethics (following strict rules). They trained the system using human feedback on simulated accident scenarios. Their findings showed that while the system could reliably follow Kantian rules, it learned a surprising behavior under the Utilitarian approach: people preferred self-sacrifice over reducing overall casualties, which led the model to adopt that preference. This suggests that machines trained with human feedback learn human behaviors as experienced rather than ideal ethical principles.
Autonomous VehiclesReinforcement LearningEthical Decision MakingUtilitarianismKantian EthicsCARLA SimulatorProximal Policy OptimizationHuman PreferenceReward ModelingRLHF
Authors
Thomas Mbrice, Ammar Ali, Sami Mian, Khai Hern Low, Eric Chen, Arshia Aghajani, Wolf Schäfer, Amin Shirangi
Abstract
As autonomous vehicles (AVs) approach Level 4 and Level 5 operational capability [SAE International, 2018], their on- board decision systems must handle not only safety-critical locomotion but also their subsequent moral weight. This paper details the Ethical Decision Head (EDH), a deep re- inforcement learning (RL) framework that encodes ethical reasoning as a differentiable reward signal, enabling a pol- icy gradient agent to learn morally-aligned driving behavior in scenarios whose state representation is aligned with the CARLA simulation environment [Dosovitskiy et al., 2017]. Two normative frameworks are instantiated and evaluated: a Utilitarian framework minimizing total casualties and a Kan- tian framework enforcing course maintenance as a categori- cal imperative. The EDH is trained via Proximal Policy Op- timization (PPO) [Schulman et al., 2017] against a Bradley- Terry reward model [Bradley and Terry, 1952] learned from pairwise human preference annotations over 200 collision- imminent scenarios. Results reveal an asymmetry in the learnability of normative ethical frameworks under human su- pervision. The Kantian condition, which reduces to a con- stant prediction task under the codebook, serves as a pipeline control: it confirms training stability and rules out infrastruc- ture failure as an explanation for the utilitarian result. The Utilitarian agent learned something more unsettling: human raters rewarded self-sacrifice over casualty minimization, and the model learned that preference faithfully. This divergence between what humans prescribe in theory and what they re- ward in practice suggests that RLHF does not learn ethics as philosophers define it, but as humans live it.