Qiushi Engine improves autonomous research task completion rates
Qiushi Engine on AstaBench E2E-Bench-Hard
Artificial IntelligenceSoftware Engineering
Summary
Researchers tested Qiushi Engine, a computer program that carries out complex research tasks from design to reporting, on a tough 40-task benchmark. Using a specific AI model, Qiushi Engine completed 10% of tasks perfectly, better than the previous best 3%. It successfully met 82% of the detailed requirements across all tasks. The system showed strengths in producing reliable reports and code but struggled with repeated testing, using outside tools, and including some specific analysis steps. The report explains how the system works, overall results, example tasks, and the challenges it still faces.
What this means in practice
- •For software developers: Use Qiushi Engine to automate complex end-to-end research workflows that include experiment design and code execution.
- •For data science teams: Integrate Qiushi Engine for systematic generation and verification of experimental analysis and results reporting.
Authors
Wenhao Li, Shuxing Yang, Fujia Chen, Jincheng Mi, Yuang Pan, Rui Zhao, Zichen Li, Junyao Wu, Shenzhan Hong, Yaqi Li, Yize Wang, Kaihao Zhu, Taowen Deng, Junjie Yang, Hongsheng Chen, Yihao Yang
Abstract
This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a research question through experimental design, code implementation, actual execution, result analysis, and report delivery. Qiushi Engine is model-configurable; this evaluation selected DeepSeek deepseek-v4pro-preview as the model backend. The official AstaBench leaderboard records a score of 0.816 and an average benchmark cost of USD 15.209 per task, while the full-precision local recomputation is $81.59 \pm 1.87$. Four tasks satisfied every rubric item, yielding a full-task completion rate of 4/40 = 10% -- 7 percentage points above, and about 3.3 times, the approximately 3% best rate reported for AstaBench's official agents. Across 507 required rubric items, 416 were satisfied (82.1%). Official scoring archives and 40 Meta-Trace records show sustained production and verification of reports, code, and experimental artifacts; the principal gaps lie in repeated runs, external dependencies, specified metrics, and ablation studies. The report explains the benchmark, system workflow, aggregate results, representative cases, and limits of interpretation.