Goldsmith improves annotation definitions using expert examples and AI edits
Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
Computation and Language
Summary
Annotation projects often start without clear guidelines or enough examples. The authors present Goldsmith, a method that uses a few expert-annotated examples to create a clear annotation definition that can be improved automatically. It tests candidate definitions on these examples, scores mistakes, and uses a large language model to suggest edits that reduce those errors. Goldsmith outperforms other prompt-improvement methods and helps improve downstream annotation tasks with limited expert input.
What this means in practice
- •For machine learning engineers: Create reusable and optimized annotation guidelines from limited expert examples to improve training data quality.
- •For data labeling teams: Enhance annotation accuracy and scalability by using AI-assisted definition optimization combined with human review for complex text tasks.
Authors
Yihan Li, Hanyi Zhang, Xiaoxi Jiang, Man Guo
Abstract
Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model. We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition. Goldsmith treats this definition as a trainable textual object. Candidate definitions are run on the same gold examples and scored with an executable structured loss, while the output schema, formatting, retrieval, repair, judging, and human review remain in an external harness. A large language model (LLM) editor converts the highest-loss failures into textual-gradient revisions, which are accepted only when the measured loss decreases. In prompt-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols. The resulting definition also improves downstream annotation when combined with retrieval, score-based routing, and human review across typed span, pair-level relation, and fixed-trigger event-argument tasks. These results show that scarce expert supervision can support both task-definition learning and scalable annotation.