Kernel autoresearch discovers new machine learning models that generalize
Kernel Autoresearch for Open-Ended Model Discovery
Machine Learning
Summary
Designing good kernels helps machine learning models learn better, but creating valid and useful kernels automatically is hard. The authors created a system called Kernaut that lets computer programs invent new kernels safely and keeps only the best and most different ones. These new kernels work well on tasks they haven't seen before and can be understood and improved by humans. The approach beats some existing complex models on real problems, like predicting enzyme behaviors.
What this means in practice
- •For machine learning developers: Create reusable kernels automatically that improve model predictions on unseen tasks and specialized scientific problems.
- •For biotech data teams: Generate interpretable kernel models from enzyme kinetics data that predict related biological systems with lower error than current methods.
Authors
Richard Cornelius Suwandi, Feng Yin, Kevin Murphy
Abstract
Kernels encode the inductive bias of a wide range of machine learning models, yet automated kernel design faces a fundamental dilemma. A fixed grammar of base kernels and operators guarantees validity but limits the search to structures expressible by those building blocks. Conversely, unrestricted programs remove this limitation but no longer guarantee validity. In our stress tests, 22-58% of LLM-generated kernels that pass numerical checks on random inputs fail when evaluated at different scales or dimensions. We propose Kernel Autoresearch (Kernaut), which treats kernel design as open-ended model discovery. Coding agents write kernels as programs, while construction contracts ensure that every accepted kernel is valid. A quality-diversity archive retains high-performing kernels with distinct behaviors, and novelty screening steers agents toward functionally new candidates. Our experiments demonstrate that the discovered kernels encode reusable inductive biases that generalize to unseen tasks. On held-out black-box optimization families, a discovered kernel outperforms a meta-learned deep kernel trained on the same episodes. Furthermore, kernels discovered from ten enzyme-kinetic rate laws achieve lower error than tuned ARD and deep kernel baselines on five unseen mechanisms. The discovered kernels are also interpretable programs that human researchers can refine: a human-refined version of one further reduces the held-out predictive error by 5.7% and optimization regret by 7.8%.