MoMHa optimizes language model setups for accuracy safety and tokens
MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens
Artificial Intelligence
Summary
Usually, people try to make large language models as accurate as possible, but this paper shows that the code around the model—called the harness—is also very important. The authors introduce MoMHa, a system that tweaks the harness to improve three things at once: accuracy, safety (not doing unsafe tasks), and how many tokens it uses. They tested MoMHa on many kinds of tasks and models, finding it works better than other ways of tuning the harness. The system helps make AI responses safer and more efficient without losing accuracy.
What this means in practice
- •For machine learning engineers: Improve large language model deployment by automatically tuning harness code for better accuracy, safety, and token cost balance without retraining models.
- •For ai safety teams: Develop safer AI systems by optimizing harnesses to reduce unsafe outputs while maintaining high accuracy across varied tasks.
Authors
Subhojyoti Mukherjee, Md Mehrab Tanjim
Abstract
Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model that constructs prompts, routes calls, and parses outputs, is a first-class design surface whose quality is inherently multi-objective: an accurate harness that refuses no unsafe request, or that consumes an order of magnitude more tokens, is not a good harness. We present Meta-Harness, a system that casts harness design as search over three per-domain objectives (accuracy, behavioural safety, and token cost) solved by an agentic proposer (Claude Code) with full filesystem access to prior harness source, execution traces, and scoring artifacts. Our central finding is that a singlephase joint-reward proposer (MoMHa) outperforms every alternative, including a two-phase "accuracy then tokens" ablation, scalar-only feedback, and an accuracy-only baseline. We evaluate on seventeen domains: seven synthetic capability suites, seven real-world public benchmarks (HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath), and three U-SafeBench-derived user-specific safety domains, using a 12-model fleet spanning four families. On the synthetic track MoMHa achieves a joint mean of 0.482 versus 0.198-0.422 for ten baselines, winning $7 / 10$ per-domain columns; on the real-world track it scores 0.461 versus 0.377 for the strongest baseline (DSPy), winning 5/7 columns, demonstrating that harness strategies transfer to unseen benchmarks without retraining on 8 of 12 target models. MoMHa attains the highest measured behavioral safety composite (U-SafeBench, 0.781) and uses 95 fewer tokens per example than the two-phase alternative. We will release all harness code, evaluation infrastructure, and crossmodel logs.