LLM graph agents risk repeating evidence without gaining insight
GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents
Artificial IntelligenceComputation and Language
Summary
Large language models (LLMs) that explore knowledge graphs may confuse retracing the same evidence paths with finding new support. The authors introduce GraphEcho, a test to check if these agents mistake repeated visits for more proof. Their experiments show that while some improvements reduce revisits, they might miss important distinct information. This work helps understand how well graph-exploring agents use evidence versus just exploring efficiently.
What this means in practice
- •For knowledge graph engineers: Improve graph traversal algorithms to avoid repeated evidence paths and enhance distinct source coverage in LLM-based graph agents.
- •For scientific claim verification teams: Reduce repetitive evidence gathering by LLM agents when verifying scientific claims, balancing exploration efficiency and accuracy.
Authors
Sikun Wang, Yixi Zhou, Lei Fan, Fan Zhang
Abstract
A large language model (LLM) agent can follow more graph paths without acquiring more independent evidence. GraphEcho tests whether agents mistake these repeated encounters for additional corroboration. The benchmark varies path counts and evidential origins while holding evidence content fixed, and evaluates both judgments and active exploration. Controlled synthetic experiments reveal model-dependent judgment shifts, but redundant supporting paths increase the share of repeated walks across all evaluated frozen agents. Provenance-aware post-training (PAPT) reduces revisits and improves synthetic accuracy, yet covers fewer distinct sources. On scientific claims, it continues to reduce repetition while accuracy declines. These findings expose a gap between efficient exploration and effective evidence use: an agent can learn to stop repeating itself while overlooking information it needs. GraphEcho provides a controlled way to evaluate both what graph agents conclude and whether their exploration reaches distinct evidential sources.