COMPREHEND improves detailed prompt understanding for large language models

COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy

Computation and Language

Summary

Understanding the types of questions people ask large language models (LLMs) and how well these models handle them is important but challenging. The authors propose COMPREHEND, a method that organizes prompts into very detailed categories with clear sentence descriptions. This helps users see exactly where the model performs well or poorly, beyond broad groupings. Their method showed better accuracy and informative descriptions compared to previous approaches. COMPREHEND also produces clearer reports that help predict model performance on different prompt types.

What this means in practice

  • For llm developers: Generate precise prompt categories to better understand and improve model strengths and weaknesses.
  • For benchmark evaluators: Produce detailed reports predicting model response quality on specific prompt types using fine-grained prompt hierarchies.

Authors

Sivaraaman Balakrishnan, Prakriti Shetty, Jeremiah Hong, Hamed Zamani, Haw-Shiuan Chang

Abstract

Understanding what kind of prompts users typically ask and how well an LLM performs on these prompt categories is crucial for LLM practitioners. The existing methods satisfy the demand by hierarchically clustering the user prompts and synthesizing the cluster names using an LLM to build a taxonomy. However, the cluster names from these methods are usually too general for humans to understand the detailed prompt distributions and identify the exact strengths and weaknesses of LLMs. In this study, we propose COMPREHEND, which automatically constructs an ultra-fine-grained entailment hierarchy for the prompts. The entailment hierarchy replaces the cluster names in the taxonomy with sentences to provide more precise cluster descriptions for more fine-grained categorization. Compared with EvalTree, a state-of-the-art taxonomy construction method, COMPREHEND improves an entailment/faithfulness metric from 22.6 to 41.7 and an informative metric from 16.3 to 39.1 in WildBench. Our experiments also show that the weakness report generated based on our entailment hierarchy is much more informative than EvalTree in terms of predicting the response scores in the benchmark. All the code is released in https://github.com/JackNapier20/COMPREHEND_public.