From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix
2026-09-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors address the challenge of managing many AI language models in a limited GPU setup by combining over 200 applications onto one improved model. They identify different types of errors related to following instructions, calling functions, and task handling, then train separate experts to fix each issue. By carefully merging these experts, their approach improves model quality and efficiency without mixing up goals. This consolidated model handles half of their platform's traffic with lower costs compared to much larger models. Overall, the authors show how targeted training can maintain high performance while reducing resource use.
Large Language Models (LLMs)Data ResidencyGPU Resource ManagementInstruction FollowingFunction CallingModel EnsembleGRPO (Gradient Regularized Policy Optimization)SLERP (Spherical Linear Interpolation)Model EvaluationOffline Benchmarks
Authors
Olga Tsymboi, Dmitrii Stoianov, Ramil Latypov, Danil Taranets, Daniil Dryabin, Mikhail Gashkov, Viktor Zelenkovskiy, Aleksandr Fida, Gleb Alektorov, Nikita Gulyakov, Arthur Babkin, Aleksandr Medvedev, Pavel Gein, Anatolii Potapov
Abstract
Data-residency constraints force enterprises to self-host LLMs, but continuous adoption of newer models without decommissioning their predecessors expands the serving fleet, fragmenting a finite GPU pool. We consolidate traffic from over 200 internal applications onto a single model by closing quality gaps identified through production error analysis along three axes: instruction following, function-calling, and internal task distribution. Quality is tracked by offline benchmarks stratified to production traffic and scored by deterministic verifiers or calibrated LLM judges. Rather than optimising all objectives jointly, which introduces cross-domain reward interference, we train a separate GRPO expert per axis and merge them via two-stage SLERP. Each expert's reward exposes a distinct failure mode, namely semantic collapse, over-calling, and verbosity hacking, each requiring a domain-specific fix. In non-reasoning mode the recipe surpasses a ${\sim}7\times$ larger by total parameters baseline on the in-house Arena with 69.6 to 65.8, instruction following with 0.85 to 0.83, and function-calling with 0.79 to 0.77, while lifting general dialogue benchmarks. The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.