Transformer based meta model improves long sequence system prediction
In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation
Machine LearningArtificial Intelligence
Summary
Predicting how systems behave often requires making a model from lots of training data, which can be slow and complicated. The authors study a new approach where a single powerful model is trained ahead of time to directly learn from past examples and new inputs together, without building a separate system model each time. Their method uses a special Transformer design to handle very long sequences of data and can estimate how uncertain its predictions are. They tested their approach on difficult benchmarks and showed it can match traditional methods and even adapt well when conditions change unexpectedly.
What this means in practice
- •For control engineers: Generate detailed synthetic training data using a meta model to improve system models when real data is limited.
- •For automation system developers: Deploy a pretrained Transformer meta model for accurate system predictions directly from input-output sequences, even with long historical data.
Authors
Matteo Rufolo, Dario Piga, Filippo Pura, Marco Forgione
Abstract
Traditional system identification typically involves two steps: initially, training data is processed to estimate a model of the system of interest; then, the model is used to predict outputs for new input queries. In contrast, recent in-context learning approaches learn a meta model that maps directly from training data and query inputs to outputs in a zero-shot fashion, bypassing explicit model estimation. This approach leverages a meta model trained offline, on large sets of synthetically generated data, to emulate the behavior of a system identification algorithm. In this work, we advance this meta-learning framework for system identification in three directions. First, we develop a unified probabilistic meta-identification architecture that combines a Transformer-based encoder-decoder with tailored mechanisms for handling long training sequences, enabling calibrated predictions over long horizons. Second, we show how a pre-trained meta model can act as an uncertainty-aware synthetic data generator for classical system identification, providing task-specific data that regularize parametric models when real data are scarce. Third, we systematically evaluate the framework on standard nonlinear system identification benchmarks and on a challenging out-of-distribution scenario, demonstrating competitive accuracy with state-of-the-art methods and strong generalization, including cases where input-only adaptation is sufficient to handle significant distribution shifts.