Optimizing hyphenation patterns with smarter computer tuning methods
The Art of Hierarchical Competing Patterns: Gaussian Process Optimization of Hyphenation
Computation and LanguageDigital LibrariesMachine Learning
Summary
Breaking words correctly for line breaks in text is important but often relies on experts manually tuning rules. The authors show that using a mathematical search method called Gaussian-process Bayesian optimization can automatically find better tuning settings. Their approach improves accuracy and reduces the size of the rule sets needed, working well across many languages and scripts. This makes generating these rules less dependent on trial and error and more consistent. The results suggest computer-aided tuning can help maintain quality while simplifying the process.
hyphenationtypesettingpattern generationhyperparameter optimizationGaussian processBayesian optimizationF1 scoretrie data structuremachine learningtext processing
Authors
Ondřej Sojka, Petr Sojka
Abstract
Hyphenation patterns remain a compact and widely deployed solution for word breaking in typesetting systems, text processors, and web rendering engines, but their generation still depends on manually tuned patgen program parameter profiles. We formulate patgen profile selection as a black-box hyperparameter optimization problem and evaluate Gaussian-process Bayesian optimization for this task. The search objective combines a precision-oriented F_{1/7}-score with an explicit trie size-accuracy trade-off using a normalized trie-size penalty. We evaluate the method on 17 hyphenated word-list datasets covering 14 languages and multiple scripts. Against two strong hand-tuned profiles regenerated from the same 8/10 training split and evaluated on the same 1/10 held-out test split, the GP-optimized profiles improve F_{1/7} on 16 of 17 datasets and reduce trie size on all 17. The median optimized/baseline trie ratio is 0.407. A dataset-level sign test gives p = 1.37e-4; a separate budget-matched comparison on five representative datasets shows that systematic search is competitive and usually improves over the best hand-tuned profile under the fixed comparison objective. The results show that model-based optimization can make pattern generation more reproducible and less dependent on expert trial-and-error while keeping the accuracy-compactness trade-off explicit.