Small language models differ in reasoning paths beyond answer accuracy
The Path Matters: Evaluating Small Language Models Beyond Answer Accuracy in KGQA
Computation and LanguageArtificial Intelligence
Summary
Answering questions using knowledge graphs is complicated because it involves searching and reasoning through linked data. The authors studied how small language models perform when asked to follow these reasoning paths step-by-step rather than just giving final answers. They found that some models can give the right answers but take different or incorrect paths to get there, while others may follow the paths faithfully but have lower answer accuracy. This suggests it's important to check not just if an answer is right, but also if the reasoning path used is correct.
What this means in practice
- •For natural language processing teams: Assess small language models’ ability to follow correct reasoning paths on knowledge graphs beyond just answer accuracy for better model selection.
- •For conversational ai developers: Enhance question-answering systems by integrating stepwise path evaluation to improve trustworthiness of answers backed by knowledge graphs.
Authors
Eduin E. Hernandez, Sergio A. Diaz, Luis F. Garcia, Nurassyl Askar, Stefano Rini
Abstract
Small language models (SLMs) are increasingly paired with knowledge graphs (KGs), yet end-to-end KG question answering conflates graph access, search, navigation, reasoning, and answer generation. This coupling makes it difficult both to determine whether an SLM can faithfully execute the reasoning path implied by a question and to attribute failures to navigation rather than to other stages of the pipeline. We isolate this capability by employing the THESEUS navigation and traceability framework and using frozen, off-the-shelf SLMs as local action policies. At each hop, the environment exposes the legal outgoing graph actions, and the model selects one executable graph action and decides whether to stop, without task-specific parameter updates, model-controlled beam search, or free-form answer generation. This controlled setting allows us to evaluate terminal-answer accuracy with Hits@1 together with path fidelity, using Path Edit Distance (PED) as the primary trajectory metric. Across the Kinship and MQuAKE-ST KGQAs, similarly sized local models differ substantially in answer accuracy and path fidelity, with the two metrics sometimes favoring different models. This model-dependent behavior also extends to prompting, as a single demonstrated trajectory can improve or degrade navigation depending on the model. These results motivate evaluating SLM graph reasoning beyond endpoint accuracy alone.