Researchers use AI coding tools mainly for data tasks and check results informally
How Researchers Use and Verify AI Coding Assistants: Tasks and Validation Practices in Scientific Programming
Software EngineeringHuman-Computer Interaction
Summary
Many researchers now use AI tools to help write code for their scientific projects, especially for handling data, making charts, and fixing errors. The authors looked at over 500 examples from researchers describing how they use these tools and how they check if the AI's code works. Most people just run the code themselves rather than use formal tests or ask others to review it. Less experienced coders trust the AI more, while experienced ones trust their own skills instead. The study shows that researchers mostly rely on their own judgment to decide if AI-generated code is correct.
What this means in practice
- •For software development teams: Design coding assistant tools that offer built-in checks tailored to typical research programming tasks, improving user confidence and correctness.
- •For data analysis teams: Adapt AI coding assistants to support informal evaluation methods common in data-related programming, providing validation cues during code creation.
A survey. It maps existing work.
Authors
Gabrielle O'Brien, Reed Milewicz, Nasir Eisty
Abstract
Generative AI has entered research programming, yet there is little evidence about which tasks researchers hand to it or how they decide whether its code is correct. We draw on 527 free-text responses to a 2025 survey of researchers who write code, most of them at U.S. universities. In each response, a researcher recounts a single task from their own work, the way they used an AI tool for it, and what they did to assess the result. We coded the task and the evaluation strategies reported, and related both to programming experience, research area, and confidence ratings. Use was concentrated in five tasks: data handling, visualization, debugging, mathematical/scientific computing, and statistical analysis. Evaluation was informal and individual. Over half of accounts described running the generated code, while automated tests and review by another person were rare. Use cases and evaluation strategies varied little with programming experience, but confidence did: Less experienced programmers trusted the AI more than themselves, and experienced programmers the reverse. Evaluation confidence was not associated with the strategies reported. Its strongest correlates were confidence in the tool and in oneself. Validating AI contributions to scientific code rested largely on individual judgment, outside shared infrastructure for testing or review. Interfaces could support task-appropriate evaluation rather than leave it to the user.