Claim calibration clarifies reproducibility in recommender systems studies

Calibrating Reproduced Claims in Recommender Systems

Information Retrieval

Summary

When researchers try to repeat or reproduce results in recommender systems, they sometimes find different numbers or outcomes but still agree on which methods are better. The authors introduce the idea of "claim calibration" to clearly state what parts of an original research claim are truly supported by follow-up studies and under what conditions. They tested this approach using five pairs of original and follow-up research papers to show that full agreement is rare and that follow-up studies often confirm only parts of original claims. They also propose a way to report these findings in a structured profile to improve clarity.

What this means in practice

A position paper. It proposes an approach and reports no results.

Authors

Alan Said

Abstract

Reproduction studies can produce mixed outcomes. Reported values may differ while the ordering of the compared methods remains the same, a result may hold only under some experimental conditions, or a released implementation may fail to reproduce a result that the model can still reach. The terms repeatability, reproducibility, and replicability describe how a follow-up study relates to the original experiment, but not which parts of the original claim are supported by the new results. We introduce \emph{claim calibration} as a way of stating the strongest claim supported by a follow-up study, together with the conditions under which it holds and the parts that remain untested. We apply this perspective to five original--follow-up paper pairs from recommender-systems research. The cases show that agreement in numerical values, method rankings, statistical results, and overall conclusions does not always coincide, and that follow-up studies often support only part of the original claim. Based on these observations, we propose a Claim Evidence Profile for reporting the original claim, its scope, the reproduction target, the reported results, the calibrated claim, and the parts of the original claim that remain unresolved.
View PDFOpen arXiv