OrionMoE delivers adaptive tabular prediction with compact expert model
OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
Machine LearningArtificial Intelligence
Summary
Tabular data models predict outcomes like labels or categories based on structured data tables. Existing models often treat all features the same way, which can limit their ability to adapt to different datasets. The authors created OrionMoE-Small, a model that uses a smart way to focus on different parts of the data selectively, adapting to features and the overall dataset context. This model is trained on generated data and can make predictions without needing to learn from new data each time. It works well especially when data is imbalanced or has many features.
What this means in practice
- •For data science teams: Deploy models that adaptively allocate computation across heterogeneous tabular datasets without retraining on new data.
- •For financial analysts: Improve prediction accuracy on small or high-dimensional tabular datasets where class imbalance is common.
Authors
Mohamed Bouadi, Nassim Bouarour, Aditya Tanna, Vinay Kumar Sankarapu
Abstract
Tabular foundation models enable in-context learning by predicting unseen observations from a labeled support table without task-specific parameter updates. Despite recent advances, most existing approaches rely on dense architectures that apply the same transformations across features and datasets. We introduce OrionMoE-Small, a compact tabular foundation model that combines distribution-aware representations with dataset-conditioned sparse mixture-of-experts computation. Built on a dual-axis Transformer architecture, OrionMoE-Small selectively activates experts along the feature axis, with routing decisions informed by both individual feature representations and the broader dataset context. This design enables adaptive allocation of computation across heterogeneous tabular problems while retaining a shared computational pathway across tasks. OrionMoE-Small is pretrained entirely on synthetic classification tasks generated from a structured causal prior and performs downstream prediction without gradient-based adaptation. Experiments across multiple tabular benchmarks demonstrate competitive performance against substantially larger foundation models, with particularly strong results on small-context and high-dimensional datasets. Further analysis reveals favorable relative performance under class imbalance and highlights the importance of feature composition.