Candidate-independent attention improves permutation robustness in decision models
Permutation-Robust Decision Modeling with Candidate-Independent Block-Causal Attention
Machine LearningArtificial Intelligence
Summary
Decision models often rank or score many choices that can appear in different orders. The authors found that standard methods can mistakenly treat the order of choices as important, causing inconsistent results. They introduced a new way, called candidate-independent block-causal attention, which keeps each choice separate and ignores the order they appear in. This makes the model's decisions more stable without losing quality.
What this means in practice
- •For machine learning engineers: Build more stable decision-making components in generative AI systems that handle variable candidate orders.
- •For ai model developers: Improve evaluation consistency of ranked candidate sets in large language models by reducing order bias.
Authors
Guy Amit
Abstract
Decision models often score a variable-sized set of candidate actions encoded in a single sequence. This setting is increasingly relevant for System 1 components inside generative systems, where candidates may be proposed or ordered differently across runs. Standard causal cross-encoding is expressive, but it can make a candidate's score depend on serialization order rather than on the underlying decision problem. We introduce candidate-independent block-causal attention, which preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions. We compare this architecture with standard causal attention and complementary invariant baselines across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones. Candidate-independent attention consistently reduces permutation sensitivity while retaining competitive decision quality; ablations indicate that candidate isolation is the primary source of the effect, with position resetting completing the intended symmetry. A larger Qwen3-4B study further examines the behavior of the proposed architecture with substantially more training data. Code is available at the \href{https://github.com/guyAmit/ci-decision-models}{\textcolor{blue}{project repository}}, and the \href{https://huggingface.co/Guy-Amit/qwen3-4b-ci-decision-4096-poc}{\textcolor{blue}{Qwen3-4B model artifact}} is available on Hugging Face.