Lüshi 1840 trains classical Chinese model with verified historical cutoff

Lüshi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model

Computation and Language

Summary

Language models that only learn from texts before a certain year help us study how language and knowledge changed over time. The authors created Lüshi-1840, a language model trained on classical Chinese books written before 1840, carefully selecting and cleaning these texts despite unreliable dates in existing collections. They tested whether the model could recognize or predict events after 1840 and found it struggles more compared to modern models, confirming it mostly learned only older language. This work shows it is possible but challenging to build language models with a true historical knowledge cutoff.

What this means in practice

  • For digital humanities teams: Use Lüshi-1840 to study changes in classical Chinese language and historical texts up to 1840 with a vetted dataset and model.
  • For machine learning engineers: Develop language models that isolate knowledge to pre-specified historical periods by applying the authors’ data cleaning and auditing approach.

Authors

Alex Liu

Abstract

Language models trained only on text from before a given year are useful for studying hindsight and historical language, but the open collections of pre-modern Chinese carry no publication dates and unreliable dynasty labels. We present Lüshi-1840, a 983M-parameter model trained from scratch on 12,262 books (1.28B characters) selected as composed before 1840. A first corpus, dated by a common heuristic, in effect ended at 1644 while containing twentieth-century material. We describe the re-dating and cleaning pipeline that replaced it and an audit with markers the pipeline did not use, which still finds later material. On the model side we measure surprisal, relative to a modern reference, on annals compiled after the cutoff. For events after 1840 against events up to 1820 the ratio is 1.38 (95% interval 1.33-1.44), against about 1.00 for modern general-purpose models. In a series of 294M models, adding the later court records lowers the ratio from 1.33 to 1.16, and the two models diverge from the 1820s, where the added records begin. The probe tracks the coverage of sources and does not establish a single cutoff year; none of our probes exposes the audited contamination. Model, book list and scripts will be released.