Papers for

ai system testers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multi model ensembles show limits and performance drop points

The Universe of Universes: Benefit Yield Functions, Implosion Thresholds, and Infrastructure-Aware Optimization in Multi-LLM Systems

Abstract: We introduce the Universe of Universes (UoU) framework, which treats the full ecosystem of major large language models (LLMs) as a structured retrieval corpus and proposes a compositional Automated Reasoning (AR) and Machine Learning (ML) architecture for cross-model retrieval-augmented generation. The central contribution is the formal characterization of the Benefit Yield Function (BYF), the marginal performance gain per additional model added to an ensemble, and the identification of the implosion threshold θ*: the ensemble size at which BYF crosses zero and aggregate performance begins to degrade. Existing LLM ensemble and mixture-of-agents systems treat models as responders and aggregate outputs, but do not study performance as a function of ensemble size N across the full model universe. Benchmark research confirms performance plateaus at the individual model level; model collapse literature establishes that iterative training on AI-generated outputs degrades individual model distributions. Neither body of work formalizes the ensemble-level implosion threshold, models Epistemic Hereditary Drift (EHD) at the ecosystem level, or treats AI manufacturing velocity as a co-variable of θ*. The framework has direct implications for DoD multi-model AI acquisition policy and the emerging science of testing AI-enabled systems.

Mon 14 SeptMachine LearningArtificial IntelligenceMultiagent Systems
The gist
Sometimes, combining many large language models (LLMs) to work together can improve how well they perform, but adding too many can actually make things worse. The authors introduce a way to measure the benefit of adding each new model and identify a specific point where adding more models starts to harm performance. They also explore why this happens at the ecosystem level and how fast creating new AI models influences that tipping point. Their framework helps understand and optimize large AI systems using multiple language models.
Open → 2609.15314v1

Compression changes vision language model scores without changing answers

BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering

Abstract: Quantization makes vision language models (VLMs) cheaper to store and run by using fewer bits to represent their weights. While unchanged answers on visual question answering (VQA) after compression are an expected behavior, they can still hide changes in the underlying scores (log probabilities). For example, a model may still answer yes after compression, even as the score gap between yes and no shrinks. We introduce BEFORE THE FLIP to measure these hidden changes. Our method compares the score change caused by compression with the change caused by replacing the image's internal representations, or image tokens, with one fixed average token. We then increase the precision of one weight group at a time to identify where extra bits help, and test whether choosing different groups for each question offers benefits beyond shuffled controls. Among 8,277 LLaVA questions where image token replacement measurably affects the scores, 4-bit compression shifts the yes or no score gap farther toward the replacement output than 8-bit compression. Qwen shows the same pattern, but with a smaller difference. Yet only 265 of 9,000 LLaVA answers change at 4 bits. In a separate study of 1,024 calibration questions, choosing weight groups separately for each question does not outperform both shuffled controls at any tested storage budget. These findings show that compression can alter the scores behind unchanged answers, but do not establish a reliable benefit from adjusting precision for each question.

Mon 7 SeptComputer Vision and Pattern Recognition
The gist
Compressing vision language models to use fewer bits can keep their answers the same but still change the confidence scores behind those answers. The authors created a way to measure these hidden score shifts by comparing compression effects to changes caused by replacing image parts. They found that lower-precision compression shifts scores more but rarely changes the actual answers given. Trying to adjust bit precision for each question did not consistently improve results. This suggests that compression can quietly alter how models weigh options without flipping their final choices.
Open → 2609.06922v1