Conditional retrieval improves matching social survey questions accurately
Conditional information Retrieval for Longitudinal Survey Harmonisation
Computation and LanguageComputers and SocietyInformation Retrieval
Summary
Matching questions from different social surveys is hard because similar words don’t always mean the same thing. The authors designed a two-step approach that first finds the main idea behind each question and then uses that idea to better match questions between surveys. Their method works well even when there isn’t a lot of labeled data to learn from. They tested it on real social science data and showed it outperforms simpler techniques. This approach could help combine survey results more reliably over time.
What this means in practice
- •For survey data analysts: Identify matching questions across different long-term surveys to enable better data comparison and longitudinal research.
- •For social science data platforms: Improve integration tools that combine survey datasets by using concept-aware retrieval to align similar questions.$Commercial implications: Enables commercial platforms to offer enhanced survey data harmonisation features for clients in research and policy sectors.
Authors
Wing Yan Li, Zeqiang Wang, Michelle Lin, Jon Johnson, Suparna De
Abstract
Identifying conceptually equivalent questions across different surveys is a challenging information retrieval task where surface-level textual similarity is insufficient. Two questions may be lexically similar yet assess entirely different sociological sub-concepts, or lexically different yet conceptually equivalent. We introduce a conditional information retrieval framework that addresses this by decomposing the task into two steps: concept extraction, which identifies the underlying sociological concept of a survey question using a fine-tuned generative model, and concept-aware retrieval, which augments question representations with the extracted concept as an explicit conditioning signal for finer-grained alignment. To reflect the realistic data scarcity conditions of specialised social science annotation, we use CLOSER as a case study. Our framework consistently outperforms baselines by 2-3 points in both MAP and Recall. We further analyse fine-tuning strategies including SFT, DPO, and dynamic DPO, finding that SFT delivers the most stable gains while dynamic DPO interacts differently with instruction-tuned versus distilled reasoning backbones. Finally, we demonstrate that a model fine-tuned on a general social science thesaurus (ELSST) transfers robustly to task-specific survey data, offering a practical solution under data scarcity.