Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training
2026-08-03 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors developed a method called CoPES to make training large language models that use tools more efficient when computing resources are limited. Instead of adjusting all model parameters at once, CoPES breaks the problem into smaller pieces and optimizes them together, using less memory and time. When tested on math and question-answering tasks, CoPES performed better than standard methods under the same resource limits. This shows a better balance between memory use and training speed for improving language models after their initial training.
large language modelsevolution strategiespost-trainingcooperative coevolutionGPU memoryparameter subspacereinforcement learningtool-using agentsvalidation accuracyLoRA
Authors
Zhiyuan Wang, Shengcai Liu, Jiahao Wu, Ning Lu, Hui Ouyang, Shaofeng Zhang, Haoze Lv, Ke Tang
Abstract
Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive. Evolution strategies (ES) enable memory-efficient full-parameter post-training without backpropagation and can eventually match the performance of gradient-based reinforcement learning (RL). However, resource-constrained settings typically offer only a few GPUs, so the high GPU-hour requirements of ES translate into prohibitively long training times. To address this, we introduce Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that decomposes the full parameter space into lower-dimensional subspaces and searches over them cooperatively to improve optimization efficiency. We post-train a Qwen3.5-4B tool-using agent for the math task and evaluate it on five benchmarks of varying difficulty. Under the GPU-hour budget of full-parameter GRPO's best validation checkpoint, CoPES recovers 92% of GRPO's validation-accuracy gain, versus 67% for standard ES, while its theoretical GPU memory requirement is less than one-eighth that of full-parameter GRPO. It consistently outperforms standard ES and LoRA-based GRPO on all evaluated pass@k metrics across the five benchmarks. Additional experiments further show the advantage of CoPES on the question-answering task. These results demonstrate an improved trade-off between memory requirements and training time for agentic LLM post-training under resource constraints. The code is open-sourced in https://github.com/MetaronWang/CoPES