Notrix helps analyze hundreds of machine learning notebooks together
Notrix: Understanding Machine Learning Solutions Across Computational Notebooks at Scale
Human-Computer Interaction
Summary
It can be hard to understand how hundreds of data science notebooks solve a problem because each notebook is different and full of scattered code. The authors created Notrix, a tool that looks at hundreds of notebooks at once, organizing their code cells into machine learning steps and grouping similar notebook workflows. This makes it easier for people to explore and learn from many notebooks quickly without having to open each one. In tests, people using Notrix answered questions about large notebook collections faster, more accurately, and with less stress.
What this means in practice
- •For data science teams: Explore common solution patterns across many machine learning notebooks in a project to speed up collaboration and code reuse.
- •For machine learning engineers: Identify diverse workflows and isolate specific notebooks for closer study without opening all files individually, enabling efficient debugging and learning.
Authors
Xiaotian Su, Hongxin Fu, Xiaoyu Zhang, April Yi Wang
Abstract
Computational notebooks make problem-solving visible, but typically only one notebook at a time. Meanwhile, in data science platforms like Kaggle, one competition can accumulate hundreds of notebooks. Effective collection-level analysis requires characterizing recurring solution patterns across all notebooks, as well as isolating specific notebooks for closer examination and learning. However, standard notebooks provide no common basis for this. Their workflows are nonlinear, cells declare no intent, and identical code can serve different ends, leaving hundreds of notebooks as separate documents. In this paper, we present Notrix, an interactive visual analytics tool for profiling hundreds of notebooks as one collection. Inspired by a formative study (N = 11), Notrix classifies every cell into one of thirteen machine learning (ML) stages, turning each notebook into a stage sequence, and clusters those sequences by structure rather than by code. To keep the representation constant as the scope narrows from the whole collection to a single cell, Notrix features three coordinated views---Workflow, Structural Matrix, and Detail---that appear at all four levels of granularity. In a within-subject study (N = 17) using two Kaggle collections of over 400 notebooks each, we observed participants answered questions about all notebooks more accurately with Notrix (median 88% vs. 50%) while opening 80% fewer notebooks per minute. Notably, four of the fourteen answered it without opening a single notebook (interaction logs, N = 14). Participants also reported significantly lower mental demand, temporal demand, and stress with Notrix (Holm-Bonferroni adjusted).