Systematic review framework improves machine learning model selection
A Framework for the Systematic Review of ML Assets in AI Registries
Machine Learning
Summary
Choosing the right machine learning models and datasets is often done in an informal way without clear evidence or records. This paper shows how methods used to review scientific papers can help organize and document the search for machine learning assets systematically. The authors create a framework that treats these assets like important data, making it easier to find, compare, and select them with clear reasoning. This approach aims to make machine learning system building more transparent and reliable.
What this means in practice
- •For software engineers: Use a structured, reproducible process to select ML models and datasets for building reliable AI software.
- •For data platform teams: Organize and align multiple AI registries to improve discoverability and integration of ML assets across the company.
A position paper. It proposes an approach and reports no results.
Authors
Alexandra González, Quim Motger, Xavier Franch, Silverio Martínez-Fernández
Abstract
Background: Modern software systems increasingly rely on Machine Learning (ML) assets (i.e., pre-trained models, datasets, benchmarks) for building, evaluating, and integrating ML-based systems. However, current exploration, selection and reuse practices of ML assets are not supported by systematic retrieval methodologies comparable to those used in traditional evidence synthesis. Consequently, in practice, ML asset selection is often presented as a settled design decision, supported by informal justification rather than a traceable, evidence-based, and updatable selection process. Aims: This paper explores how systematic review methods can support ML asset retrieval. In doing so, we aim to make their selection transparent and reproducible, grounded in explicit evidence, and ultimately better suited to its intended use. Method: We analyze established systematic review practices from scientific literature and adapt their phases (i.e., planning, conducting, and documenting) to Artificial Intelligence (AI) registries, treating ML assets as first-class units of analysis. The resulting framework integrates registry-aware search strategies, cross-registry schema alignment, and dependency-driven ML asset exploration. Results: We conceptualize ML asset retrieval as a systematic and reproducible process rather than an ad hoc activity, and propose a framework for structured ML asset discovery. \textbf{Conclusions:} This work illustrates how systematic review principles can be extended beyond scientific literature to support evidence synthesis over evolving AI registries.