CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
2026-06-22 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors address the problem of lacking good training data for teaching language models to perform terminal tasks. They create CLI-Universe, a system that carefully builds and verifies tasks by combining different skills and checking them in real environments. This process filters out unclear or easy tasks, leaving only challenging and accurate examples. Using a dataset from CLI-Universe, the authors fine-tune a large language model, achieving strong results compared to bigger models trained on less reliable data. This shows that well-made, verified tasks improve learning efficiency for terminal-agent models.
Large Language ModelsTerminal AgentsTask SynthesisExecutable VerificationDockerFine-tuningTraining DataCapability TaxonomyData EfficiencyQwen3-32B
Authors
Zhanbo Hua, Yifan Yao, Weihao Xie, Yongchi Zhao, Minghao Liu, Ruizhi Qiu, Zhewei Huang, Zun Wang, Yiyan Ji, Yunhai Ye, Letian Zhu, Xinping Lei, Han Li, Zhiyuan Ma, Zili Wang, Zhaoxiang Zhang, Jiaheng Liu
Abstract
While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.