Silver evidence improves decision making from noisy text in AI learning
From Noisy Retrieval to Decision Evidence: Silver Evidence Demonstrations for In-Context Learning
Computation and Language
Summary
Sometimes when AI tries to learn by looking at examples, it can get confused by extra or distracting information in long texts. The authors show a way to help AI focus by creating shorter, clear summaries of the important parts of each example before making a decision. This method helps AI make better choices and give better explanations, especially when the original information is long and messy. Their tests show this works well for several different tasks.
What this means in practice
- •For natural language processing teams: Improve AI model accuracy on tasks involving long, noisy documents by replacing retrieved passages with generated evidence summaries in example prompts.
- •For policy analysis teams: Enhance automated decision-making in synthetic policy retrieval by generating compact evidence summaries that reduce distracting information.
Authors
Masoud Monajatipoor, Fazlolah Mohaghegh, Hamid Reza Hassanzadeh, Akash Uttamrao Bhoite, Robert E. Tillman
Abstract
RAG and retrieval-augmented few-shot ICL can struggle when decision-relevant evidence is buried in long, noisy context. Retrieved passages and demonstrations may contain redundant or distracting information. Although expert-labeled development examples may identify the text supporting each decision, these annotations are typically used only as answer-side rationales rather than as supervision for filtering noisy retrieval. We propose silver-evidence demonstrations, which replace raw retrieved passages in ICL examples with generated, target-conditioned evidence summaries. For each labeled development example, an LLM generates silver evidence without access to the expert label or gold evidence; this evidence is then paired with the expert label and gold evidence where available. At test time, a single structured call generates silver evidence from retrieved passages and then predicts the final label and decision evidence. Across EI 2.0, SciFact, synthetic policy retrieval, and HICRIC/IMR data, our full method improves over equally sized gold-only ICL (label-only for policy) on three of four datasets and improves evidence F1 on all annotated datasets. It also achieves higher mean label accuracy than InstructRAG-style ICL on all four datasets. These results suggest that replacing demonstration-side retrieved passages with compact, target-conditioned silver evidence is particularly useful for decisions based on long and noisy text, where it reduces distracting context while preserving decision-relevant signals and expert supervision.