Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents
Artificial Intelligence
Summary
The authors studied how language model agents can extract information about datasets from conference papers. They compared a simple fixed process to more advanced agents that use reflection and memory to improve their work. Their focus was on how the agents behaved during the task, like retrying steps and recovering from mistakes, rather than just the number of datasets found. They also explored an improved agent that could use better tools and choose them dynamically. The study helps understand when adding these smart features really changes the workflow and helps fix problems.
Authors
Lujia Zhang, Xingzhou Chen, Hongwei Feng
Abstract
Large language model (LLM) agents are increasingly used for complex information-extraction tasks, yet it remains unclear whether agentic components such as reflection and memory lead to observable and controllable improvements over fixed LLM workflows. We study this question through conference-paper dataset extraction, where a system must identify datasets mentioned in scholarly PDFs and produce structured records. We compare a fixed workflow baseline with reflective agent variants and specify an optimized agent condition (S2) that extends the same task with richer PDF tools and dynamic tool selection. Our evaluation emphasizes process-level behavior--including tool execution, retries, reflection, memory use, runtime, and failure recovery--while treating extraction coverage and field completeness as secondary outcome measures. The paper characterizes when agentic mechanisms change system behavior, whether these changes improve task completion, and how the observed failure modes motivate an optimized agent design under the same evaluation harness.