Large language models often break markdown boundaries in generated text
LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text
Software EngineeringComputation and LanguageMachine Learning
Summary
Large language models (LLMs) are used to create Markdown text for computers to read and process, but sometimes the models mess up the special formatting boundaries while keeping the content correct. The authors created LatentMD, a test set and tool that can spot when the Markdown formatting is broken even though the content is right. They tested nearly 38,000 generated examples across 9 models and found that over a third had correct content but broken formatting boundaries. The mistakes mostly happen because of confusing similar delimiter marks and are not fully fixed by giving hints in prompts.
What this means in practice
- •For software engineers: Detect and fix boundary errors in Markdown generated by language models when building tools that render or extract code.
- •For automation developers: Improve reliability of automated pipelines that process LLM-generated Markdown by verifying fence-boundary correctness separately from content correctness.
Authors
Sungjune Lee, Myungjoo Kang
Abstract
Large language models (LLMs) increasingly generate Markdown that is consumed by renderers, agents, code extractors, and structured downstream pipelines. Yet existing evaluations often conflate content quality with format adherence, leaving Markdown boundary failures under-measured. We introduce LatentMD, a benchmark and evaluation protocol for diagnosing CommonMark-level fence-boundary failures in LLM-generated Markdown. LatentMD separates content correctness from boundary correctness, enabling detection of outputs that are content-correct but boundary-broken. The benchmark contains 4,179 prompts and a CLI for scoring arbitrary model outputs. Across 9 LLMs and roughly 37,600 generations, we find that Markdown boundary failures are widespread: 38.0% of valid main-grid outputs are content-correct but boundary-broken, with substantial boundary breakage under unspecified prompts and in a small human-authored validation set. Ablations show that failures are driven primarily by same-family symmetric-delimiter collisions rather than nesting alone, are only partially mitigated by prompt hints, and generalize to Python triple-quote docstrings while JSON remains robust as an asymmetric-delimiter control. LatentMD provides a reproducible diagnostic target for parser-sensitive LLM evaluation.