Modellakefishing speeds up finding models in huge public model collections
ModelLakeFishing: Efficient Retrieval over Million-Scale Model Lakes
Information RetrievalDatabases
Summary
Finding the best machine learning model for a new dataset can be really slow when there are millions of models to check. The paper presents ModelLakeFishing, a tool that quickly finds good models by organizing past performance data into a graph and using smart search methods. This way, it narrows down millions of models to a small set and reranks them efficiently. Their tests show that ModelLakeFishing finds top models nearly as well as checking all models but much faster.
What this means in practice
- •For machine learning engineers: Quickly select high-performing models from large public or private model collections for new datasets without exhaustive evaluation.
- •For data platform teams: Integrate efficient model retrieval systems into platforms hosting millions of reusable models to speed up model discovery and deployment.
Authors
Xiaoyang Liu, Zhengyuan Dong, Renée J. Miller
Abstract
Open model lakes may contain millions of reusable models, making it costly to identify suitable models for a new dataset. We present ModelLakeFishing, a model-retrieval framework for queries specifying a target dataset, prediction task, and evaluation metric. It consolidates metadata and historical evaluations into a model-dataset-task evidence graph, learns model and query embeddings with a relation-aware graph encoder, and indexes model embeddings using Hierarchical Navigable Small World (HNSW) search. At query time, HNSW retrieves 1,000 candidates without scoring every model, after which a training-side task prior reranks candidates for the requested metric and returns the top 10. We evaluate on a lake of 3,016,439 models and 247,803 observed model-dataset performance pairs using three root-aware splits that hold test performance edges out of representation learning and retrieval. ModelLakeFishing achieves a mean eligible-query gold@10 of 0.2968, recovering the observed-best model in the top 10 for 29.68% of eligible queries and retaining 93.47% of the gold@10 of an exhaustive baseline using the same scoring and reranking procedure. Given precomputed query embeddings, retrieval and reranking take 0.747 ms median and 1.102 ms at the 95th percentile. These results demonstrate efficient retrieval over million-model lakes from sparse relational evidence.