When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
2026-08-17 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how different types of natural language explanations (NLEs) affect the performance of language models when used as examples in prompts. They compared human-written, self-generated, and externally generated explanations across several tasks and models. Their findings show that adding explanations usually helps classification tasks, with externally generated explanations often working well. However, for math problems, the results depend more on the model and explanation source. They also found that filtering explanations based on how truthful they are sometimes helps but can also hurt performance depending on various factors.
Natural Language ExplanationsIn-Context LearningFew-Shot PromptingLanguage ModelsFaithfulness MetricsClassification BenchmarksMath ReasoningPrompt EngineeringModel Robustness
Authors
Mahdi Dhaini, Adam Dejl, Juraj Vladika, Volkan Özer, Barbara Plank, Gjergji Kasneci
Abstract
Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL). However, it remains unclear how different types of NLEs compare in their effects on downstream model performance in explanation-augmented prompting. Therefore, we provide a comparative evaluation across six benchmarks and four instruction-tuned models, studying how NLE source (human-written when available, self-generated explanations, generated by an external LLM) and NLE selection (random vs faithfulness-based filtering) affect downstream utility of NLEs when used in ICL settings. Our extensive evaluation shows that, on classification-style benchmarks, adding NLEs to few-shot prompts often improves accuracy over few-shot prompting without explanations; among NLE sources, externally generated LLM-NLEs often provide strong downstream utility and remain competitive with human rationales where both are available, whereas self-NLEs are more sensitive to the selection strategy. On math reasoning, the effects are more model- and source-dependent. We further show that faithfulness-based selection of self-NLEs yields small average gains overall, but can improve or reduce performance depending on the metric, task, and model. Different faithfulness metrics can disagree substantially, affecting which self-NLE examples are selected and their downstream predictive utility. Robustness tests with randomly swapped and out-of-distribution rationales indicate partial robustness, suggesting that semantic alignment contributes to performance gains. Overall, our results provide insights for selecting and reporting explanations that influence model behavior in practical prompting pipelines.