Pos signatures improve demonstration choice for dependency parsing

POS-Signature Demonstration Retrieval for In-Context Dependency Parsing

Computation and Language

Summary

When computers try to understand the relationships between words in sentences, they learn better if given good examples first. The authors found that picking examples based on parts of speech, rather than just meaning, helps language models do a more accurate job. They tested this on English sentences and showed it worked better than some other ways of choosing examples. They also shared their tools and results for others to use.

What this means in practice

Tested on one dataset.

Authors

Muhammad Azam Afridi, Rajab Ali, Marco Passarotti

Abstract

Demonstrations for in-context learning are often retrieved by semantic similarity, which may be a poor proxy for relevance in structured prediction. We study demonstration selection for Universal Dependencies parsing with a POS-signature policy that represents sentences with weighted part-of-speech statistics and retrieves three length-compatible, gold-parsed examples from the labeled training split. We evaluate two open-weight language models without fine-tuning on all 2,077 sentences of the UD English Web Treebank test set, comparing the policy with shot- and format-matched fixed and semantic three-shot baselines, as well as zero-shot, chain-of-thought, and critique-refinement systems. POS-signature retrieval yields the highest labeled attachment score among the tested prompting systems for both models (72.4 and 73.2), outperforming semantic retrieval by 1.9 and 2.7 points; paired-bootstrap 95% confidence intervals exclude zero for both contrasts. Chain-of-thought and refinement comparisons vary across models. These exploratory results favor the tested POS-signature retrieval approach on English-EWT. We release the prompts, archived outputs, and evaluation code.