Self-critique rounds shape hypothesis diversity in multi-agent discovery
Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation
Artificial Intelligence
Summary
When many AI systems work together to generate scientific ideas, it's hard to tell how much they improve and how different their ideas really are. This paper looks at how much letting the AI critique its own ideas changes what it comes up with and how the way we judge if two ideas mean the same thing affects the diversity we measure. The authors found that one round of self-critique causes big changes, but more rounds add little extra change. They also showed that different ways of deciding if ideas are equivalent can wildly change how diverse the ideas seem.
What this means in practice
- •For ai development teams: Improve evaluation of AI-generated scientific hypotheses by tuning the number of self-critique iterations and choosing better equivalence measures for hypothesis comparison.
- •For biomedical knowledge engineers: More accurately assess the diversity and refinement of AI-generated biological hypotheses using an LLM-based judge that captures meaningful mechanistic differences.
Authors
Ji Young Byun, Anthony Hu, Jesse Rogers, Roujia Wang, Manasa Kesapragada, Falgun Shah
Abstract
Multi-agent systems built on large language models (LLMs) are increasingly applied to scientific discovery and hypothesis generation. Both the effect of refinement and the diversity of the delivered set are hard to interpret before experimental ground truth exists, and both are typically reported by deciding whether pairs of generated hypotheses describe the same underlying mechanism. We study two evaluation questions that rest on this pairwise equivalence judgment: (1) how much self-critique changes delivered hypotheses beyond run-to-run variability, and (2) how the equivalence rule used to group hypotheses affects measured diversity. Across four proprietary instances, we hold opening hypotheses fixed, rerun the downstream workflow with 0, 1, and 5 critique rounds, and score matched hypothesis pairs with an LLM-as-a-judge. Relative to matched same-depth reruns, moving from 0 to 1 round produces 34.5 percentage points (pp) of additional mechanism-level divergence, whereas 1 to 5 rounds adds 1.3 pp. We then compare three equivalence rules: term frequency--inverse document frequency (TF--IDF) similarity, dense embeddings, and the same LLM-as-a-judge. We construct controlled hypothesis pairs that either preserve the causal explanation through wording or biological-terminology changes, or replace one component of the causal chain while holding the rest fixed. All three rules are invariant to meaning-preserving edits, but when the initiating event is replaced, the LLM-as-a-judge identifies 83% of valid pairs as different mechanisms, versus 0% for TF--IDF and 8% for embeddings; varying only the rubric that defines same mechanism moves this figure from 38% to 96%. Together, these results show that pairwise equivalence judgments are a measurement choice: how mechanism equivalence is defined affects both the estimated effect of self-critique and the measured diversity of generated hypotheses.