Papers for

software teams building ai assistants

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Maple framework reveals varied effects of knowledge extraction methods

MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

Abstract: LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques such as activation steering, decoding adjustments, and prompting modify inference, potentially reinforcing existing predictions or surfacing less accessible answers. Yet their evaluations centered on aggregate accuracy, which obscures diverse effects by collapsing recovered failures and disrupted successes into a single score. We introduce MAPLE, a unified framework for characterizing how KEE techniques affect parametric knowledge extraction. Its diagnostic metrics disentangle these effects at the surface, probability, and beyond-sampling levels by measuring answer recovery and disruption, gold-token rank shifts on unrecovered failures, and recovery of answers missed by fixed-budget sampling. We evaluate six KEE techniques spanning decoding, activation steering, and prompting across three model families: OLMo2, Qwen2.5, and Llama3. Our analysis reveals distinct effects and trade-offs across techniques, highlighting the value of MAPLE for understanding and comparing KEE beyond aggregate accuracy.

Wed 14 OctMachine LearningComputation and Language
The gist
Large language models (LLMs) store a lot of information, but it's not always clear how well this information comes out when you ask them questions. Researchers developed MAPLE, a way to better understand how different techniques can help get information out of these models by measuring various detailed effects beyond just getting the right answer. MAPLE looks at how some methods help recover missed answers or cause disruptions in others, providing a clearer picture of each technique’s strengths and weaknesses. They tested MAPLE on several popular LLMs and found important differences between techniques that simpler scores would miss.
Open → 2610.19827v1