Workflow control improves fairness and speed under shared api limits
Dispatch Control for Priority-Based, Multi-Tenant Workflows under Multi-Resource External Rate Limits
Distributed, Parallel, and Cluster Computing
Summary
Many software systems run tasks that use shared external services with strict limits on how much they can be used. When one task hits a usage limit and retries, it can block others unfairly. The authors created a method called Backpressure that carefully checks if each task can proceed without breaking the limits and makes tasks wait without blocking others. This method makes sure different priority groups get fair access and speeds up important tasks significantly.
What this means in practice
- •For workflow engineers: Control API call dispatch in workflows to improve fairness and reduce latency under shared rate limits among tenants and priorities.
- •For cloud platform operators: Manage external API rate limits across multiple customers and priorities without needing to observe or reconfigure caps.
Authors
Stanislav Kosorin, Trever Schirmer, Leonard Wossnig, David Bermbach
Abstract
Workflow engines orchestrate pipelines of activities, which often call external APIs, on a fixed pool of worker slots, the threads that run them. When an API enforces a hard rate limit shared by all tenants and priorities, a throttled call that retries after a backoff keeps its worker slot and starves other tenants. Large Language Model provider APIs are particularly complex: their per-organization caps are low and bind independently on requests and tokens per minute. The operator can neither reconfigure the cap nor observe saturation before a request fails, so classical fair-queuing assumptions do not hold. We present Backpressure, a worker-side admission gate. It checks each call's cost atomically against shared admission state and enforces fairness over requests and tokens. A denied call releases its worker slot and waits on a durable timer, so waiting no longer blocks execution capacity. We compare against a naive retry, the AI gateway LiteLLM, and the built-in throttle of the workflow engine Temporal. On a workload with three priority classes, Backpressure raises fairness (Jain's index over each queue's fraction of its entitled share) from 0.60 to 0.89, where 1.0 is ideal. It also cuts high-priority tail latency roughly 8x. When tokens are the binding limit, all request-based baselines serve every priority equally, whereas Backpressure keeps the priority order and reaches a dominant-resource fairness of ~0.99. Admission costs one round trip to shared state, adds no measurable latency when the cap does not bind, and fails open if shared state is unavailable.