Summary
This paper studies a process that turns spoken conference talks into question-answer pairs, checking how cleaning up the transcripts and filtering the pairs affects their usefulness. The authors found that while cleaning steps improved the quality of the question-answer sets according to experts and language models, this did not make the sets better at helping retrieve the correct information. In fact, one cleaning step slightly reduced retrieval accuracy despite improving quality ratings. Overall, having the corpus greatly improved retrieval compared to no corpus, but better quality scores didn’t mean better performance downstream.
What this means in practice
- •For medical data teams: Use question-answer pairs generated from conference speech to enhance retrieval of domain knowledge in Korean medical sessions.
- •For finance analytics teams: Ingest refined question-answer corpora from finance conference talks to support improved information retrieval in financial domains.
Abstract
Practitioners building training corpora with a pipeline of language models keep a preprocessing or refinement (post-process) stage when switching it on raises the corpus quality. We ablate a production pipeline that turns conference speech into question-answer pairs. The preprocessing stage rewrites the transcript, and the refinement stage filters the generated pairs. Switching each on and off gives four corpora from the same 40 Korean medical and finance sessions. On the corpus itself, quality rises as the stages switch on. Four LLM judges and six human domain experts rank the four corpora the same way, and the fully refined corpus leads by +0.18 and +0.22 on a 1-5 scale. We then ask whether that gain carries downstream. We supply each corpus to six models at inference and measure retrieval accuracy. The ranking does not hold. Enabling the transcript rewrite lowers retrieval accuracy by 0.061, the only stage effect that survives correction for multiple comparisons. That stage also reads the whole session, so it almost doubles the cost of one. With both stages on, accuracy does not change. An equivalence test we declared in advance confirms this. The corpus itself is clearly valuable. It lifts retrieval accuracy from 0.231 to 0.748 in every model and domain we test. More than a third of its pairs carry knowledge that neither of two frontier models produces on its own. On our data, the null does not come from weak corpora. A rising score on a rubric of this kind did not predict what the corpus holds for a model.