A Scalable Pipeline for LLM-Teacher Distillation Labeling: Work-Stealing Job Scheduling and Memory-Aware GPU Concurrency
2026-08-17 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors created a system to help label huge collections of text automatically using large language models (LLMs), which is too big for humans to do by hand. They designed a way for GPU workers to share and steal tasks efficiently, even if some workers fail, using a simple storage method to keep track of work without conflicts. They also made sure the system adjusts the number of tasks each GPU handles based on its memory size. To measure the system's accuracy and cost, the authors had the model relabel datasets with known correct answers. Their experiments showed better speed and reliability than older methods, and all their code and tests are available for others to try.
Large Language ModelsLabelingGPU Workload ManagementWork StealingConcurrencySQLiteRelabeling BenchmarkInstruction TuningSentiment AnalysisFault Tolerance
Authors
Ravi Satya Durga Prasad Yenugula
Abstract
Labeling large text corpora with LLM teachers has become a practical route to training data at scale. At millions of items, hand-labeling every batch is not feasible, and two questions dominate: what label quality a teacher buys per dollar, and how to keep a fleet of GPU workers busy under skewed, failure-prone workloads. We present a simple, reproducible pipeline that addresses both. First, a work-stealing ring pool: each worker owns a queue, drains it first, and then steals from ring successors, with exactly-once task claims via atomic conditional writes and crash tolerance via stale-claim sweeping. The claim protocol requires only a compare-and-set primitive from its storage layer; we implement it on a single SQLite file, which makes the reference implementation dependency-free and the experiments reproducible on one machine. Second, a memory-aware concurrency rule that sizes per-node parallelism by how many model copies fit on the GPU, so the same code runs safely across device sizes. Third, a relabeling benchmark methodology in which the teacher relabels a public dataset that already has gold labels, so quality reduces to an agreement measurement and cost follows from measured throughput. Under skewed load the pool sustains up to 3.4 times the throughput of static sharding while matching it at zero skew, loses 0 of 2,000 tasks when half the workers are killed mid-run (static sharding loses 953), and yields measured quality and cost points for an instruction-tuned teacher on irony and sentiment tasks. All experiments run on public data and commodity hardware; code, tests, and run logs are released.