Multimodal AI detection improves spotting AI text in math papers
Multimodal AI Detection for Math-Heavy Scientific Text
Computation and Language
Summary
As AI systems get better at writing parts of scientific papers, especially those heavy in math, it becomes important to tell AI-generated content from human writing. The authors found that current tools relying only on raw text struggle with math content. They showed that using the original LaTeX code or images of the formatted paper helps detect AI writing much more accurately. They also created a new tool, SnapJudge, which uses screenshots of papers and detects AI-generated math sections better than existing methods.
What this means in practice
- •For scientific publishers: Automatically identify AI-written math-heavy sections in scientific papers during submission review processes.
- •For academic conference organizers: Enhance screening tools to verify originality of math proofs and prose in submitted papers by leveraging image-based detection.
Authors
Yixuan Huang, Danish Pruthi, Bhuwan Dhingra
Abstract
As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We release our data, models and code at https://nlplab.cs.duke.edu/snapjudge.