Aggregating article predictions improves news publisher reliability assessment
From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference
Computation and Language
Summary
Determining if a news publisher is reliable usually involves experts looking at many factors about the publisher. This paper tries a new approach by first judging individual news articles and then combining these judgments to decide if the whole news source is trustworthy. The authors tested this method on nearly 20,000 political articles from over 400 publishers and found that combining article-level checks made the overall predictions more accurate. They also looked at whether political bias affected mistakes in their predictions.
What this means in practice
- •For news platform teams: Automatically evaluate and flag the reliability of news publishers by aggregating predictions from many articles to improve trust signals.
- •For social media moderation teams: Enhance content moderation by incorporating aggregated reliability scores for news sources to better identify potential misinformation.
Authors
John Bianchi, Manuel Pratelli, Fabio Pinelli, Marinella Petrocchi
Abstract
Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standards at source. When this process is translated into a computational approach, the problem is often formulated at the level of individual articles, with models being trained on a set of pre-labelled articles and their performance being evaluated in a test phase. In this work, we investigate news source reliability inference as a source-level prediction problem. We propose a two-stage framework in which transformer-based language models first estimate the reliability of individual articles and subsequently aggregate article-level predictions to infer the reliability of previously unseen publishers. To approximate realistic deployment conditions, we enforce a strict publisher-disjoint evaluation protocol, ensuring that no publisher appears in both training and test sets. Experiments on 19,476 political news articles from 439 English-language publishers labeled with NewsGuard reliability ratings show that aggregation substantially improves robustness and performance, increasing accuracy from approximately 0.60 at the article level to 0.69 at the publisher level. Finally, we analyze how prediction errors vary across political orientations, revealing statistically significant associations between political leaning and misclassification patterns. Overall, our findings show that publisher reliability can be inferred from aggregated textual signals alone, supporting scalable and content-based approaches to automated news source assessment.