Fine-tuned large language models improve writing revision detection accuracy
From Granular Revision Operations to Meaningful Revision Units: Evaluating LLMs for Revision Boundary Detection
Computation and LanguageArtificial Intelligence
Summary
Detecting when a writer makes a meaningful change can be tricky because small edits can look like many separate changes. The authors tested if large language models (LLMs) can better identify these meaningful chunks in student writing revisions. They found that LLMs with simple prompts don’t beat basic methods, but adapting and fine-tuning a specific LLM made it better at grouping edits correctly. This means LLMs could help understand how students revise their drafts, but only when carefully trained for the task.
What this means in practice
- •For educational software developers: Improve software that tracks student writing changes by using fine-tuned LLMs to better group related edits into meaningful revisions.
- •For document collaboration tool teams: Enhance revision tracking features by integrating task-adapted LLMs to detect coherent edit segments in collaborative document drafts.
Authors
Yu Tian, Andrew Potter, Katerina Christhilf, Motahareh Darvishpour Ahandani, Jessica Early, Steve Graham, Danielle S. McNamara
Abstract
Revision traces provide valuable evidence about students' writing processes, but their usefulness for learning analytics depends on how individual revisions are represented. Automated draft-comparison methods often produce granular edit operations that can fragment a single purposeful revision into multiple analytic units. This study evaluates whether LLMs can identify meaningful revision unit boundaries in structured revision operation data and whether they provide value beyond simple non-LLM baselines. Using 113 matched draft--revision pairs from undergraduate writing, expert annotation yielded 4,344 candidate boundaries. We compared zero- and few-shot GPT-5.5 and base Qwen3-32B, parameter-efficient fine-tuning of Qwen3-32B, and majority and proximity-based baselines. Despite receiving revision context and task instructions, no prompted LLM condition outperformed the proximity heuristic (macro-F1 = .825). In contrast, fine-tuned Qwen3-32B using the two context representation achieved the highest macro-F1 (.859), identifying more same-unit relationships while maintaining precision comparable to the heuristic. Deterministic post-processing substantially improved the prompted models but added little benefit to the strongest fine-tuned model. These findings suggest that LLMs can support revision boundary judgment when task-adapted, but general purpose prompting alone may not outperform transparent structural heuristics.