Shared autoregressive context can skew relationships in synthetic data
Shared Autoregressive Context Can Distort Relationships in Synthetic Data
Machine LearningComputation and Language
Summary
When large language models generate multiple fake survey answers all at once, the earlier answers can influence later ones in a way that makes some relationships between data points seem stronger than they really are. The authors show this effect by comparing single-answer versus multi-answer generation approaches on European Social Survey data, finding that producing several answers in one go distorts the correlations more. They also found that hiding earlier answers reduces this problem but makes individual answers less accurate. This means how you ask the model to generate data affects the quality of the synthetic data’s patterns.
What this means in practice
- •For synthetic data engineers: Design request methods that balance batch size and context use to avoid distorting variable relationships in synthetic datasets.
- •For social science data teams: Evaluate synthetic data not only by individual accuracy but also by how well it preserves the relationships needed for specific analyses.
Authors
Thomas S. Robinson
Abstract
Large language models can generate several records within one autoregressive completion, making earlier answers available as context for later records. This paper shows that such shared-completion batching can distort relationships among variables in the resulting synthetic data, using controlled tests on synthetic survey respondents. In a matched experiment on 2,000 European Social Survey profiles, generating ten rather than one respondent per request increases mean absolute error in within-country correlations by 48-58% for Qwen3.8-27B and 114-127% for Llama-3.3-70B-Instruct across three seeds, holding profiles, examples, questions and decoding parameters fixed. The distortion primarily reflects exaggerated relationship strength, while retaining substantial agreement with the human ordering of correlations. Controlled interventions establish answer history as a causal channel: re-pairing the same preceding values, with profiles and marginal distributions fixed, changes correlations among subsequently generated responses. Hiding preceding answers reduces correlation error in the tested settings but worsens marginal accuracy. Exploratory corrections across social-attitude, health and economic data likewise show that lower correlation error can coexist with worse marginal distributions and regression estimates. Request construction is therefore part of the data-generating process, and synthetic-data validity must be evaluated against the analyses the generated data are intended to support.