Ethics flags lead authors to revise papers not research in AI conferences
Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions
Computers and Society
Summary
AI conferences flag papers for ethical concerns to encourage safer research. The authors show that flagged papers are more often changed in how they are written or presented, not in the actual research methods. Many authors accept ethical criticisms while their work is reviewed but revert changes after rejection if they don't see the problems as valid. The study suggests showing previous ethics flags on resubmissions to hold authors accountable.
What this means in practice
- •For conference organizers: Improve peer review policies by disclosing prior ethics flags during resubmissions to enhance accountability over research changes.
- •For academic journal editors: Design editorial guidelines to better track ethical concerns consistently across multiple submissions of the same research.
Authors
Kento Nishi, Alec Laprevotte, Isaiah Bullock, Mfoniso Andrew
Abstract
Selective AI conferences have recently begun enforcing ethics flags and related review requirements, with the goal being to steer research towards safer and more responsible practices before publication. But do these requirements actually steer research as intended? In this paper, we show that authors more often revise how projects are presented following ethics flags than redirect their underlying research agendas. We first study the longitudinal effects of ethics flags by following rejected and withdrawn ICLR submissions with ethics flags into later public resubmissions, tracking manuscript changes after the ICLR review ends, when the original reviewers no longer oversee the project. We qualitatively code these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and directly interview authors about their rebuttal processes and resubmission decisions. We find that authors often concede concerns during rebuttal when reviewers can update their assessments, but drop those concessions after rejection when they do not regard the criticism as a sound reason to change the research. Interview participants describe publication changes as separate from changes to research direction, calling review an "editorial process" that shapes "what stories get seen" and, in another case, saying peer reviews are "mostly to filter out papers." Authors more readily change what they publish than what they study or build; we therefore recommend policy changes, especially disclosure of prior ethics flags upon resubmission so accountability carries over.