Pattransbench offers detailed benchmark for patent translation accuracy

PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation

Computation and LanguageArtificial Intelligence

Summary

Translating patents between Chinese, English, Japanese, and Korean can be tricky because patents contain highly technical language and varied formats. The authors created PatTransBench, a large collection of patent texts carefully chosen to test translation systems on both short and long texts across these languages. They also developed a new way to evaluate translation quality using multiple measures, including how well meaning stays consistent across languages. Their tests show that some translation accuracy problems are hidden if only one evaluation method is used. This benchmark helps others test and improve patent translation tools more reliably.

What this means in practice

Authors

Xiaowei Su, Xusen Cheng

Abstract

We present PatTransBench, a 12,000-item Chinese-English/Japanese/Korean patent machine translation (PatMT) benchmark with a four-dimensional evaluation protocol. The benchmark comprises 6,000 short-text (sentence/claim) and 6,000 long-text (paragraph) pairs, evenly split across three language pairs and both translation directions, built via IPC-section-stratified sampling and dualthreshold quality gating from upstream patent-family parallel corpora. The evaluation protocol spans literal n-gram overlap (BLEU, ROUGE-L, chrF), cross-lingual semantic textual similarity (CLSTS), 38-probe-language consistency testing, and serving performance under concurrency. A case study on PatMT2-7B and its fp8/AWQ4/AutoRound-4bit variants reveals direction-dependent quantization effects masked by aggregate scores and non-monotonic language-failure patterns across quantization schemes that are invisible to single-metric evaluation. We plan to release the benchmark, protocol, and case-study results upon publication to support reproducible PatMT evaluation.
View PDFOpen arXiv