Deeper auditing can sometimes reduce reliable CAD model evaluation
DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?
Software EngineeringArtificial Intelligence
Summary
When checking if computer-generated CAD models behave correctly after changes, doing more checks on each model isn’t always better. This is because spending too much time on fewer models means missing out on testing many different examples. The authors studied how this tradeoff plays out depending on what causes uncertainty in the evaluation. They found that if differences between tasks or randomness dominate, fewer deep checks are better; but if model behavior varies a lot within a single task and generating models is costly, deeper checks help more. Their work helps decide how to best use limited testing time for trustworthy results.
What this means in practice
- •For cad software developers: Determine efficient testing strategies to balance thoroughness and coverage when evaluating generative CAD model behavior under limited budgets.
- •For quality assurance teams: Optimize resource allocation during automated CAD model auditing to improve confidence in model reliability without excessive inspection.
Authors
Hongye Yang, Zhihao Xie, Shengjun Xiong, Boxiao Huang
Abstract
Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as a direct route to more reliable evaluation. Under a fixed budget, however, auditing each program more thoroughly reduces the number of tasks and independent generations that can be evaluated, which can ultimately make model-level estimates less accurate. We study this phenomenon and the conditions under which it arises. We decompose behavioral evaluation into three evidence levels: task templates, stochastic generations, and within-program edits. We define an average failure risk that is invariant to audit depth, and combine three-level variance with measured execution costs to analyze the tradeoff between deeper edit auditing and broader independent coverage. Experiments across two CAD environments and five generation systems show that the value of deeper auditing depends on where evaluation uncertainty originates. When template heterogeneity or generation stochasticity dominates, additional edit checks can increase total estimation error; when within-program state variation is large and generation is expensive, deeper auditing is more valuable. Variance and cost estimates from calibration predict the direction of this change and provide a diagnostic basis for allocating evidence on held-out tasks. These results show that the thoroughness of program inspection can diverge from the reliability of model evaluation, and they help determine whether the next unit of budget should be spent on a new task, a new generation, or additional edit checks.