Maple framework reveals varied effects of knowledge extraction methods
MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation
Machine LearningComputation and Language
Summary
Large language models (LLMs) store a lot of information, but it's not always clear how well this information comes out when you ask them questions. Researchers developed MAPLE, a way to better understand how different techniques can help get information out of these models by measuring various detailed effects beyond just getting the right answer. MAPLE looks at how some methods help recover missed answers or cause disruptions in others, providing a clearer picture of each technique’s strengths and weaknesses. They tested MAPLE on several popular LLMs and found important differences between techniques that simpler scores would miss.
What this means in practice
- •For llm engineers: Assess specific techniques that improve information retrieval accuracy from large language models by comparing nuanced effects beyond aggregate correctness.
- •For software teams building ai assistants: Improve responses by selecting knowledge extraction methods based on detailed evaluations of how they affect answer recovery and disruption.
Authors
Minsuh Joo, Youna Kim, Choonghyun Park, Soo Kyung Kim, Hyunsoo Cho
Abstract
LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques such as activation steering, decoding adjustments, and prompting modify inference, potentially reinforcing existing predictions or surfacing less accessible answers. Yet their evaluations centered on aggregate accuracy, which obscures diverse effects by collapsing recovered failures and disrupted successes into a single score. We introduce MAPLE, a unified framework for characterizing how KEE techniques affect parametric knowledge extraction. Its diagnostic metrics disentangle these effects at the surface, probability, and beyond-sampling levels by measuring answer recovery and disruption, gold-token rank shifts on unrecovered failures, and recovery of answers missed by fixed-budget sampling. We evaluate six KEE techniques spanning decoding, activation steering, and prompting across three model families: OLMo2, Qwen2.5, and Llama3. Our analysis reveals distinct effects and trade-offs across techniques, highlighting the value of MAPLE for understanding and comparing KEE beyond aggregate accuracy.