Bidirectional training improves AI preference learning from mistakes
Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures
Artificial IntelligenceComputation and Language
Summary
Current methods to teach AI from mistakes usually treat bad answers simply as wrong for the original prompt. The authors point out that some mistakes actually follow a different, related instruction better than the original one. They propose a new way, called Bidirectional Preference Synthesis, which trains AI by pairing each response with both its original prompt and a generated similar prompt where the response is correct. This approach helps AI understand better when answers are right or wrong depending on the prompt, improving its ability to follow instructions in multiple languages and tasks.
What this means in practice
- •For machine learning engineers: Train AI models to better recognize when responses are correct under varied prompts by using bidirectional preference pairs without extra reward modeling.
- •For multilingual chatbot developers: Improve multilingual instruction-following capabilities in chatbots by incorporating synthesized reverse preference data to clarify prompt-response correctness.
Authors
Junbo Wang, Lidong Lu, Zhuoqun Li, Guiping Jiang, Xiangyu Wu, Tinghai Zhang, Tong Lu
Abstract
Correction-based offline preference pipelines commonly treat model failures only as rejected responses under the original prompt. This supervision is incomplete for boundary failures: responses that violate the given instruction yet coherently satisfy a nearby intent or constraint setting. We introduce Bidirectional Preference Synthesis (BPS), a data-construction method for standard Direct Preference Optimization (DPO) that makes this missing prompt dependence explicit. For each validated boundary failure, BPS keeps the conventional forward pair under the original prompt and adds a reverse pair under a synthesized achieved prompt, so the same response is rejected where it is wrong and chosen where it is right, without changing the DPO objective, training a reward model, or requiring online sampling. On Qwen3-4B-Instruct-2507, BPS preserves original-side pairwise ranking while raising achieved-side ranking accuracy from 6.8% to 62.3% on held-out crossed anchors, with a similar shift under a Kimi-K2.6 cross-teacher probe. A blind human audit supports the intended reverse preference direction, and downstream evaluations show the clearest separation from Forward-DPO in multilingual multi-turn instruction following, with consistent capability-retention patterns on agentic, tool-use, and code checks.