Models struggle more on hard NP-complete problems than easy P twins
Did the Problem Actually Get Harder? Testing LLMs on Paired P and NP-Complete Problems Hardened by Adversarial Multi-Agent Rewriting
Computation and Language
Summary
Some hard computer problems (called NP-complete) look very similar to easier problems (in P), differing by a small detail. The authors created a new test set where each hard problem is paired with an easy problem that looks almost the same but is rewritten to remove any hints. They found that AI models still make more mistakes on the hard problems, which shows the models really struggle with problem difficulty, not just with familiar wording. However, some of the performance gap comes from how problems are phrased, not just their complexity.
What this means in practice
- •For software engineers: Identify if AI-generated code fails specifically on truly hard problems versus easier variants even with rewritten problem statements.
- •For ai evaluation teams: Use this adversarial benchmark to better measure large language models’ reasoning abilities beyond memorized or familiar problem wording.
Authors
Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Zabir Al Nazi, Mahbub E Sobhani, Shubhashis Roy Dipta
Abstract
Many NP-complete (NPC) problems have a polynomial-time (P) twin that differs in a single structural detail, such as 3- versus 2-colouring a graph. A model that reasons rather than recalls should notice which side of this boundary a problem falls on, yet existing benchmarks either fix the wording of each task or ignore its complexity class, so they cannot separate the two. We introduce P2NPCBench, a benchmark of 12 problem families in which every NPC problem is paired with a P twin posed over the same surface. To remove the textbook cues that enable recall, we build it with AgentGAN, an adversarial multi-agent framework in which a generator rewrites each statement until a panel of frozen code models fails, while an executable validity gate keeps every rewrite answer- and class-preserving. The result is 2,114 hardened statements with hand-verified reference human solvers. Evaluating 12 proprietary and open-source models, we find that every model scores lower on NPC statements than on their P twins (35.4% vs. 63.8% pass@1), and that 95.1% of model-family pairs fail more often on the NPC side. A tenfold time limit raises NPC accuracy only from 28.5% to 31.9%, so failures stem from wrong or crashing programs rather than slow ones. Hardening also narrows the class gap from 24.8 to 14.6 points, showing that part of the apparent gap reflects familiarity with textbook phrasing rather than complexity class.