Language models sometimes say impossible engineering designs are solved
Calculation and rejection diverge in language models tested at known mechanical limits
Computational Engineering, Finance, and ScienceArtificial IntelligenceComputation and Language
Summary
When language models are asked to solve engineering problems, they sometimes say impossible designs are "solved" even when these designs can't physically work. The authors tested several models on design problems that are just beyond physical limits and found that some models correctly recognize when designs are impossible if asked only about feasibility. However, those same models often accept impossible designs and say "solved" when asked to provide a full answer. This mismatch means software should test whether a model can judge feasibility separately from trusting its detailed answers.
What this means in practice
- •For engineering software developers: Improve engineering design check systems by separately testing feasibility judgments from full solution acceptance to avoid trusting impossible results in automated pipelines.
- •For ai prompt designers: Design prompts that clearly separate feasibility checks from full solution requests to reduce incorrect acceptance of impossible designs by language models.
Authors
Shaoliang Yang, Jun Wang
Abstract
When a language model returns engineering quantities with the status "solved", software that trusts it passes them on, even when the requested design is physically impossible. Such failures could reflect poor calculation, how the answer is requested, or a failure to judge feasibility. We set beam loads and disk speeds at known percentages above or below plastic collapse or burst, measured each model's limit calculation separately and tested 12 models in 4,860 calls, including follow-up experiments on new designs. We find that a model can recognize an impossible design when asked only whether it is feasible, yet accept the same design, reporting "solved", when asked for a full answer. Three models accepted beams 10 to 30% beyond collapse despite precise limit calculations in separate questions, and many accepted replies stated the violation. Asked only about feasibility, Opus 5 and Gemini 3.8 Flash rejected every impossible new design they were given, including those accepted in full answers, without false alarms. Rewording when to decline or changing the answer format raised rejection on the original beams, but not consistently on new designs. Software and benchmarks using a model's status should test feasibility judgment and full-answer rejection separately, and report the exact prompt.