Qiushi Engine improves research task success on challenging benchmark
Qiushi Engine on AstaBench E2E-Bench-Hard
Artificial IntelligenceSoftware Engineering
Summary
Solving complex research tasks usually requires designing experiments, writing and running code, analyzing results, and writing reports. The Qiushi Engine, a system that uses an AI model called DeepSeek, was tested on a hard benchmark set with 40 tasks like these. It completed 10% of tasks fully, which is more than three times better than previous AI attempts. The system met most requirements but had trouble with repeated tests and some details like checking external code and metrics. The report explains how Qiushi Engine works and where it could improve.
autonomous agentsexperimental designcode implementationbenchmarkreport generationAI modelDeepSeekreproducibilityablation studytask completion rate
Authors
Wenhao Li, Shuxing Yang, Fujia Chen, Jincheng Mi, Yuang Pan, Rui Zhao, Zichen Li, Junyao Wu, Shenzhan Hong, Yaqi Li, Yize Wang, Kaihao Zhu, Taowen Deng, Junjie Yang, Hongsheng Chen, Yihao Yang
Abstract
This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a research question through experimental design, code implementation, actual execution, result analysis, and report delivery. Qiushi Engine is model-configurable; this evaluation selected DeepSeek deepseek-v4pro-preview as the model backend. The official AstaBench leaderboard records a score of 0.816 and an average benchmark cost of USD 15.209 per task, while the full-precision local recomputation is $81.59 \pm 1.87$. Four tasks satisfied every rubric item, yielding a full-task completion rate of 4/40 = 10% -- 7 percentage points above, and about 3.3 times, the approximately 3% best rate reported for AstaBench's official agents. Across 507 required rubric items, 416 were satisfied (82.1%). Official scoring archives and 40 Meta-Trace records show sustained production and verification of reports, code, and experimental artifacts; the principal gaps lie in repeated runs, external dependencies, specified metrics, and ablation studies. The report explains the benchmark, system workflow, aggregate results, representative cases, and limits of interpretation.