Child safety risks from AI often come from use not models
Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents
Artificial IntelligenceComputers and Society
Summary
AI systems are used around children in schools, games, and social media, but checking if these systems are safe usually focuses only on the AI models themselves. The authors looked at over 1,300 reported incidents involving children and AI and found that most harms happened because of how the systems were used or managed, not because the AI models were faulty. These harms included things like damaging children’s dignity or causing psychological harm. The authors suggest safety efforts should include better rules about who can use the AI, how consent is handled, design features that limit harm, and clear reporting when problems occur.
What this means in practice
- •For ai product managers: Develop safety policies that go beyond model accuracy to include deployment context and governance to protect child users.
- •For user experience designers: Create age-appropriate interaction designs with built-in consent checks and safety constraints informed by harm patterns in children.
Authors
Junfeng Jiao, Kevin Chen, Shrestha Mishra, Abhejay Murali, David Atkinson, Shriti Raj, Saleh Afroogh
Abstract
AI systems are increasingly present in children's everyday environments: education, entertainment, social networks, and toys. Their safety is typically assessed at the model level, through benchmarks, red-teaming, and content filtering. However, most AI harm taxonomies lack child-specific evidence and under-specify developmental harm and the stakeholder asymmetries unique to minors. Building on prior taxonomies, we construct a child-centered harm taxonomy mapped to sociotechnical failure modes. We curate and code 1331 publicly reported child-related AI incidents, annotating internal and external harm types and the failure mode implicated in each incident. Our analysis reveals which harms are most prevalent in public reporting, how individual-level and population-level harms relate, and which failures are most often coded alongside harm. We find that harm concentrates in representational, dignity-related, and psychological categories, and co-occurs most often with deployment-context, governance, and misuse failures rather than model defects. Harm to an individual child is also frequently linked to population-level harm, reaching children who never interacted with the system. These findings suggest that model-level evaluations, while necessary, cannot by themselves anticipate the harm pathways most often documented for children. We conclude with actionable implications emphasizing mitigations beyond model-level safeguards such as content filtering: age assurance, consent verification for depiction, safety-by-design UX constraints, incident disclosure norms, and auditing regimes targeting developmental risk. Privacy-minimized annotations, a codebook, and documentation are available at https://github.com/The-Responsible-AI-Initiative/KidAID.