Large language models often misformat markdown boundaries causing parsing issues
LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text
Software EngineeringComputation and LanguageMachine Learning
Summary
Many computer programs now use large language models (LLMs) to write text in Markdown, a format that helps organize content for displays and software. The authors found that while the information in these model-generated texts is often correct, the special markdown boundaries that tell programs where sections start and end are frequently broken. This can cause problems when other tools try to read or use the text. They created a new test called LatentMD to specifically measure these boundary mistakes separately from content errors. Their tests showed these formatting issues happen often and are hard to fix just by giving better instructions to the model.
large language modelsMarkdownCommonMarkboundary failuresparsingcontent correctnessformat adherencesymmetric delimiterstriple-quote docstringsJSON
Authors
Sungjune Lee, Myungjoo Kang
Abstract
Large language models (LLMs) increasingly generate Markdown that is consumed by renderers, agents, code extractors, and structured downstream pipelines. Yet existing evaluations often conflate content quality with format adherence, leaving Markdown boundary failures under-measured. We introduce LatentMD, a benchmark and evaluation protocol for diagnosing CommonMark-level fence-boundary failures in LLM-generated Markdown. LatentMD separates content correctness from boundary correctness, enabling detection of outputs that are content-correct but boundary-broken. The benchmark contains 4,179 prompts and a CLI for scoring arbitrary model outputs. Across 9 LLMs and roughly 37,600 generations, we find that Markdown boundary failures are widespread: 38.0% of valid main-grid outputs are content-correct but boundary-broken, with substantial boundary breakage under unspecified prompts and in a small human-authored validation set. Ablations show that failures are driven primarily by same-family symmetric-delimiter collisions rather than nesting alone, are only partially mitigated by prompt hints, and generalize to Python triple-quote docstrings while JSON remains robust as an asymmetric-delimiter control. LatentMD provides a reproducible diagnostic target for parser-sensitive LLM evaluation.