Machine learning notebooks show how developers check code without errors
Characterizing Feedback Statements in Machine Learning Jupyter Notebooks
Software Engineering
Summary
Many machine learning programs run in Jupyter notebooks, where people write code and check results interactively. The authors call certain code lines 'feedback statements'—these show information or run checks to catch problems early. They studied over a1 million such statements from public notebooks and created a system to classify them by purpose and project stage. This work highlights that most checks are for exploring data rather than preventing crashes, revealing that many errors go unnoticed by usual crash-focused methods.
What this means in practice
- •For machine learning engineers: Develop tool features that better surface and support exploratory feedback statements to catch silent errors in notebooks during model development.
- •For data platform developers: Design notebook analysis tools that consider notebook source as a crucial factor when studying or improving ML developer practices.
- •For software quality assurance teams: Create validation frameworks that extend beyond crash detection by using assertion patterns identified in ML notebooks to reduce undetected silent failures.$Commercial implications: Enables development of commercial software QA tools targeting silent errors specific to ML pipelines in notebooks.
Authors
Arumoy Shome, Lu\is Cruz, Diomidis Spinellis, Arie van Deursen
Abstract
Machine learning development in Jupyter notebooks is iterative and feedback-driven. Practitioners author statements that reveal information about program execution and use it to decide what to do next. We call these feedback statements and identify two forms: exploratory statements that display values for visual inspection, and validation statements that enforce conditions programmatically through assertions. Many ML failures do not surface as exceptions and thus escape the crash-based analyses that dominate prior work on ML notebooks. This study examines what practitioners check to catch failures that would otherwise pass silently, by characterizing feedback statements that encode the practitioner's mental model of what the code should do and what could go wrong. We mine 297,851 public Python Jupyter notebooks from GitHub and Kaggle and extract 1,092,780 feedback statements. We sample 816 statements through proportional stratified sampling from semantic clusters obtained from CodeBERT embeddings, and apply grounded theory and open coding to label and analyze each one. We contribute a taxonomy of feedback statements in ML notebooks, organized along the functional intent of the statement and the ML pipeline stage in which it appears. The taxonomy reveals that feedback is overwhelmingly exploratory, and that the two platforms host qualitatively different modes of ML work. Mapping our taxonomy to an existing crash taxonomy shows that it captures defensive practices against silent failures that crash analysis cannot observe. Our findings indicate that notebook source should be treated as a confounder in studies of ML developer practice, surface opportunities for notebook tooling, and motivate empirical study of silent ML failures. We release the corpus of 1,092,780 feedback statements and the codebook to support replication and tooling research.