Papers for

document analysis software teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Long context models focus better by ignoring nearby background noise

The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence

Abstract: Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: https://xiaoyuyoung.github.io/LYRA/

Tue 22 SeptMachine LearningArtificial Intelligence
The gist
Large language models look at lots of information to find helpful facts far away in text, but they often get distracted by lots of nearby irrelevant details. The authors found this is less about distance and more about overwhelming nearby background. They created a new method called LYRA that helps models pay more attention to useful information even if it's far away, without losing track of where things appear in the text. Tests show LYRA helps models work better on various long-text tasks, and they also made a new test to measure how well models handle distant information despite distracting nearby data.
Open 2609.26718v1