Machine translation works better when search effort varies by sentence
When is more search useful? An Empirical Study of Search Utility in Quality-Aware Decoding for Machine Translation
Computation and Language
Summary
Translating text with machines can be improved by trying multiple ways to say the same thing, then picking the best one. But spending the same amount of time searching for all sentences is wasteful because some benefit a lot from more tries and others don’t. The authors studied how to tell which sentences improve more with extra attempts and how much extra work is needed. They found that signals from the translations already made help decide if searching more will be helpful. This can save effort by focusing extra work only on sentences that really benefit.
What this means in practice
- •For machine translation engineers: Assign more search effort selectively to sentences likely to gain quality, improving translation accuracy without extra cost on all inputs.
- •For natural language processing engineers: Use translation candidate pool features during decoding to guide resources dynamically, making quality improvements more efficient for diverse text inputs.
Authors
Javier Garcia Gilabert, Rico Sennrich, André F. T. Martins
Abstract
Quality-Aware Decoding (QAD) improves machine translation through test-time scaling, generating multiple candidate translations and selecting the final output with a quality-aware decoder. Usually the same candidate budget N is spent on every source sentence, assuming that all inputs require the same amount of search. We show that this assumption does not hold and that the quality gain obtained from additional candidates varies across sentences. We study search utility through three quantities: total scaling gain, how much quality a sentence can gain from a much larger candidate pool; marginal scaling gain, the expected gain from generating one additional batch of candidates; and the sampling saturation point, the candidate budget required to realize most of the total scaling gain. We investigate two sources of information for predicting these quantities: signals available before candidate-set expansion, and statistics computed from the candidate pool accumulated during search. We find that sentence-level signals correlate with total scaling gain but are less informative about the sampling saturation point. In contrast, properties of the current candidate pool provide useful signals of marginal scaling gain. Finally, we show that these signals can identify sentences that obtain larger gains when additional search is selectively applied.