Jev decision models reduce latency and cost in edge services
Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration
Distributed, Parallel, and Cluster ComputingNetworking and Internet Architecture
Summary
Handling natural language requests can slow down services because language models take time to decide what to do. The authors show that replacing large language models with Jev decision models in edge computing environments can make these decisions faster. Their tests show Jev reduces the time clients wait to get responses and saves money on API calls without losing accuracy. This method works well on tasks like optical character recognition, especially for new requests.
What this means in practice
- •For edge computing engineers: Cut response delays and API costs by replacing large language models with Jev decision models in edge service workflows.
- •For cloud service operators: Optimize multi-node edge services like OCR by integrating Jev’s decision API to reduce latency and fees compared to standard language models.
Authors
Delong Li, Xu Wang, Haochen Gong, Rui Lang, Guangsheng Yu
Abstract
Natural-language service requests can require a language-model decision before execution starts, consuming part of the request's latency budget. We integrate Jev's decision-oriented application programming interface (API) into edge service orchestration to reduce this overhead while retaining service completion. The integration extracts four bounded intent fields and applies a shared validator, admission policy, and scheduler, accounting for decision waiting throughout the request timeline. We compare Jev with a short, structured-output DeepSeek deployment using live API measurements followed by modeled execution, and then a real two-node optical character recognition (OCR) service with self-hosted Qwen and rule-based references. Across three consecutive measurement blocks, Jev reduces median client decision latency by 15.9-26.5%. In eight paired OCR conditions, Jev matches DeepSeek's correct, on-time completion count in seven and exceeds it in one. Without caching, median end-to-end latency on requests completed correctly by both systems is 11.1-25.3% lower; API fees per correct completion are 69.0-70.6% lower. Repeated-request caching largely removes the latency difference. The results demonstrate that decision-model substitution can lower both response latency and API fees in the tested service path, and identify fresh interpretation as the main opportunity for latency savings.