Benchmark improves retrieving relevant time series for fault diagnosis
READ-Bench: Benchmarking Historical Instance Retrieval for Time-Series Diagnosis
Artificial Intelligence
Summary
Finding past examples of similar faults helps diagnose problems in systems using time-series data, but it’s hard when faults look visually different. The authors created READ-Bench, a benchmark that tests how well methods retrieve relevant historical cases based on actual fault types, not just data shape. They show that adding a small amount of labeled examples to guide reranking improves results more than fancy representation methods or language models. Combining classic methods with their best reranking approach leads to better diagnosis case retrieval across many datasets.
What this means in practice
- •For industrial maintenance teams: Retrieve past fault cases accurately from sensor data to support diagnosing machine problems using fewer labeled examples.
- •For software developers for monitoring tools: Improve fault detection software by integrating a combined search and reranking method that boosts retrieval of diverse but relevant historical fault patterns.$Commercial implications: Allows selling enhanced diagnostic software with more precise case retrieval using a novel reranking approach.
Authors
Gerardo Pastrana, Haojun Li, Dhruv Mehta, Anoushka Vyas, Sina Khoshfetrat Pakazad, Henrik Ohlsson, John Paparrizos
Abstract
Time-series diagnostic systems rarely rely on retrieving relevant historical cases, and when they do, retrieval is evaluated only indirectly through downstream prediction. We introduce READ-Bench, a benchmark for historical-case retrieval across 12 diagnostic datasets, centered on multivariate time series, that defines relevance by shared fault or event type rather than signal shape, so visually different traces of the same fault count as relevant while similar-looking traces of different faults do not. Treating retrieval as a base retriever followed by a reranker, we evaluate classical distances, symbolic retrievers, self-supervised and foundation-model embedders, and their fusion, plus label-aware and language-model rerankers, under one protocol that varies supervision, pollution, and corpus scale with significance testing. Under a common channel-independent interface, pretrained representations offer no statistically detectable advantage over strong classical and symbolic baselines for search alone. The decisive factor is a small amount of resolved-case supervision at reranking, namely a Gaussian-process reranker that propagates a few neighbor labels in embedding space, which helps far more than more sophisticated representations or language-model reasoning and holds under pollution and at full corpus scale. Guided by these findings, we fuse a normal-residual-scored embedder with a dynamic time warping leg via reciprocal-rank fusion, then rerank with the Gaussian-process reranker, improving NDCG@10 over its own search stage on all 12 datasets, by +0.11 from reranking and +0.16 over the strongest single base retriever.