Papers for

cloud platform teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

High-speed kv compression cuts delays in large language model serving

Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving

Abstract: KV cache compression reduces the communication volume of disaggregated large language model serving, but its practical benefit depends not only on compression ratio and model quality, but also on how efficiently the compression pipeline executes. This requirement becomes increasingly important as modern serving systems adopt high-bandwidth interconnects, where inefficient compression can offset a substantial portion of the saved communication time. This paper presents \system, a high-throughput KV cache compression data path built on top of \kvserve. \system improves compression execution in two complementary ways. First, it fuses the distribution transformation and low-bit quantization stages into a KV-aware \tilelang kernel, eliminating intermediate tensor materialization and redundant memory accesses. Second, it replaces the original lossless compression backend with an open, extensible, and searchable backend based on \lc, enabling high-performance compression operator pipelines. The fused kernel achieves up to $5.9\times$ speedup, while the new lossless backend increases compression throughput from below 90~GB/s to over 500~GB/s. Across the evaluated workloads, \system reduces job completion time by up to $2.04\times$. End-to-end evaluation shows that these optimizations allow KV compression to retain meaningful communication benefits under modern high-bandwidth network.

Tue 3 NovDistributed, Parallel, and Cluster Computing
The gist
Large language models often need to share key-value (KV) data quickly across computers, but sending all this data can slow things down. The authors found that just compressing the data isn't enough; the compression itself has to be very fast to really help. They created a system called system that speeds up compression by combining steps and using a better toolkit. As a result, it greatly reduces the time taken to handle KV data and improves overall processing speed with fast networks.
Open → 2611.04382v1

Self refining prompts improve streaming text to SQL accuracy significantly

Self-Refining Prompt Induction for Streaming Text-to-SQL

Abstract: Text-to-SQL assistants for streaming engines must handle dialect-specific types, windowing rules, and temporal semantics, including constructs poorly handled by generic prompts. We present Self-Refining Prompt Induction (SRPI), an offline method that converts generation failures into reusable dialect instructions while keeping model weights fixed. A Refiner revises the Generator's prompt using reference-conditioned feedback from three LLM judges; a sampled regression guard checks earlier successes, and a separate Compressor reduces prompt length. Inference uses the frozen prompt without retrieval or query repair. We evaluate cross-domain and cross-model transfer on Confluent Cloud Flink SQL using a controlled synthetic workload seeded by 30 expert-written examples, with 335 induction examples and 257 questions from held-out domains. The same induced prompt improves mean judge scores across seven Generators by 7.1-19.2 percentage points, including an increase from 85.3% to 97.5% for the induction model. A separate compression study reduces prompt length by 52.7% with a 0.19-percentage-point score decrease. An assistant using the induced prompt is in early access, with initial user feedback to inform subsequent refinement.

Sat 31 OctComputation and LanguageArtificial IntelligenceDatabases
The gist
Understanding and converting natural language questions into correct SQL queries for streaming data is tricky because streaming SQL has many special rules. The authors developed a method that improves the instructions given to language models by learning from their mistakes without changing the model itself. This approach increased accuracy across different models and domains using a refined prompt. They also reduced the prompt's size while keeping performance high. Their method is already being tested in a real assistant tool.
Open → 2611.00960v1

MemSidecar improves agent memory scheduling for long tasks

Who Schedules Agent Memory? A Sidecar Approach to Long-Horizon Tasks

Abstract: As large language model agents increasingly undertake complex, long-horizon tasks, memory has become an important component for supporting sustained decision-making and action. Existing methods primarily focus on memory storage, updating, and retrieval, but often overlook how memory scheduling affects task execution: the main agent must identify memory needs, initiate retrieval, and wait for results while advancing the task, increasing both scheduling burden and execution time. To address this problem, we propose MemSidecar, which decouples memory scheduling from task execution through two independently selectable modes. Hooks directly delivers bound memory to the main agent when execution events match predefined conditions. Foresight uses an independent agent to predict potential queries, retrieve results in parallel, and store them in a cache, delivering matching and valid results when actual requests arrive. Hooks and Foresight achieve scores of 41.23% and 44.34% on MCP-Persona, respectively, while effectively reducing execution costs and latency and improving the efficiency of long-horizon agent execution.

Sat 31 OctArtificial Intelligence
The gist
Big language model agents often need to remember information to handle long and complex tasks. The authors noticed that the way these agents schedule memory retrieval slows them down and adds to their workload. They created MemSidecar, which separates memory scheduling from task execution using two modes: Hooks delivers needed memory right when it's needed, and Foresight predicts memory needs ahead of time to fetch data early. These approaches help the agents work faster and more efficiently on long tasks.
Open → 2611.00779v1

Agents develop shared rules to coordinate access in coding tasks

SCARCE: Emergent Institutions in Long-Horizon Multi-Agent Coding Systems

Abstract: As autonomous agents become widely deployed, coordinating access to shared computing resources and services will become increasingly important. Recent incidents have already revealed unplanned coordination between agents, highlighting the safety implications of these interactions. Yet prior studies of agent negotiation often use stylized games and simulated bargaining, leaving open how these behaviors develop during sustained, long-horizon tasks. We propose SCARCE (Shared Compute ARena for Coding agEnts), a framework in which agents solve sequences of coding tasks under individual deadlines while negotiating access to a shared execution resource. This negotiation determines when and for how long they can inspect programs, compile code, and run tests. We find that agents spontaneously develop informal institutions: persistent rules and commitments grounded in reputation, reciprocity, and precedent. These arrangements, which we call institutional behaviors, carry across tasks and influence resource allocation, with agents sometimes foregoing immediate advantages to honor earlier agreements. We develop a 21-category taxonomy of negotiation strategies and emergent norms and use it to analyze eight models and audit more than 1,500 trajectories across different environmental settings. Among the models tested, newer frontier models exhibit more institutional behaviors than older or lightweight models. Alongside cooperation, we observe exclusionary coalitions and rare divide-and-conquer strategies. SCARCE provides a controlled setting for proactively and systematically auditing how these relationships emerge, shape access to resources, and affect the safety and coordination of multi-agent systems.

Thu 15 OctMultiagent SystemsArtificial Intelligence
The gist
When computer programs called agents work together on tasks that need shared computing time, they need to figure out how to take turns fairly. The authors created a setup where agents solve coding problems but must agree on when to use a shared computer resource. They found that over time, agents create informal rules and promises that help them cooperate better, sometimes even giving up short-term gains to keep trust. This study helps us understand how artificial agents might safely and effectively share resources in the real world.
Open → 2610.21029v1

Benchmark tests multi database agents for finding and using data correctly

DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents

Abstract: Most text-to-SQL benchmarks provide the target database. In a large data catalog, however, choosing the sources is part of the problem: an agent must find every database needed for a request, determine how their results fit together, and execute the resulting workflow. We present DSABench to evaluate these capabilities jointly. Its 815 human-reviewed tasks draw on Spider, BIRD, and LiveSQLBench; 618 require several databases to contribute, and 105 allow more than one interpretation of the required sources. Executable contracts define source participation, cross-source dependencies, and source-local execution, while RouteEnv lets agents inspect public catalog evidence without revealing the gold annotations. In our routing-only study across seven model backends, the highest RA-Set score is 60.55%, compared with a highest RA-Graph score of 34.66% when coordination requirements are included. The gap becomes more pronounced when execution is required. Across four backends, median non-privileged Route+Exec-Graph success ranges from 6.69% to 7.98% without clarification and from 11.44% to 13.25% with one clarification; the best observed clarified configuration reaches 16.99%. On ambiguous tasks, clarification improves exact branch-source recovery by 39.8-43.2 percentage points. These findings illustrate why retrieving plausible databases or producing correct local SQL is not enough: an agent must carry the right information through the entire multi-database workflow.

Sat 10 OctArtificial Intelligence
The gist
When computers answer questions by searching databases, they often assume they already know which databases to use. This paper shows that in real situations with many databases, the computer first needs to figure out which databases to use, how to combine their answers, and then run the full process correctly. The authors created DSABench, a new benchmark with complex tasks requiring multiple databases, to test how well computer agents handle this challenge. Their tests reveal big gaps between just finding databases and correctly combining and executing data from them, and that asking clarifying questions helps improve results on tricky tasks.
Open → 2610.14700v1

Fixed-budget probes fail to find practical expert groupings in multi-source training

From Transfer Probes to Expert Grouping: Evidence from LoRA Post-Training

Abstract: A platform that post-trains one base model for many business lines often has more lines than experts it can maintain, so lines must share experts, yet lines seldom come with a taxonomy saying which should share. We ask whether transfer probes (train on one source, measure paired changes on all sources) can discover a grouping when the standard one is unknown. To test probes against a reference, we use seven public sources that do have a label tree (five mathematics subfields and two kinds of programming, standing in for business lines) and treat the tree's grouping as a label reference that the evaluator holds and the method does not. With four LoRA experts under fixed per-source exposure and per-expert budgets, we compare a purely probe-derived grouping and an uninformed uniform default (whose tied routing serves every line from one expert) against this reference. Trained once for one interval from a shared warm-up state and routed by a rule fixed by each allocation, neither reached the prespecified practical gain of 2 percentage points, and both had lower overall utility than the reference ($-2.43$ and $-2.54$ pp; 95\% intervals below zero). In a post hoc comparison, the probe-derived grouping was indistinguishable from the uniform default ($+0.11$ pp, $[-1.25,+1.54]$). In fixed-budget LoRA post-training, a grouping built from initial-state single-source probes thus did not deliver the advantage predicted from the probes once carried to a later training state, a finite budget, and specified routing.

Sat 10 OctMachine Learning
The gist
When training one AI model for many tasks, it helps to group tasks that can share experts to save resources. The authors tested if training on one task and checking its effect on others (called transfer probes) can reveal good groupings automatically. They found that using these probes to assign tasks to shared experts did not perform better than simply spreading tasks evenly, and both were worse than known groupings. This suggests that in practical settings with limited training time, probe-based grouping may not offer improvement.
Open → 2610.14437v1

Mixture of experts models improve reliability by robust routing training

Distributionally Robust Mixture-of-Experts Training

Abstract: Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability problem: when routing is imperfect, load-balanced models may send tokens to experts that are insufficiently trained for the assigned inputs. We propose Distributionally Robust MoE Training (DRMoET), a drop-in objective that treats layer-wise experts as endogenous robustness groups and optimizes high-loss routing outcomes rather than merely equalizing traffic. DRMoET updates a per-layer expert distribution by an entropy-regularized softmax rule on EMA-smoothed, activation-weighted expert losses, strengthening plausible non-top routing paths while preserving standard MoE computation. Under the FLAME-MoE recipe at 746M-total and 10.3B-total scales, DRMoET improves downstream averages over both standard FLAME-MoE and auxiliary-loss-free balancing. At 10.3B total parameters and 67B training tokens, DRMoET improves the seven-task average from 0.6625 to 0.6767, while the auxiliary-loss-free baseline achieves 0.6431. Mechanistic analyses show lower expert-loss variance with nearly unchanged mean loss, 4.3% lower excess loss under forced mid-$k$ misrouting, and improved domain-expert specialization. These results position routing robustness-not only utilization balance-as a practical objective for reliable sparse MoE scaling. Project page and code are available at: https://drmoet.github.io/.

Mon 5 OctMachine LearningArtificial Intelligence
The gist
Mixture-of-Experts (MoE) models split work among multiple smaller parts called experts, but sometimes tokens get sent to the wrong expert, causing errors. The authors found this routing problem affects model reliability and proposed a new training method called Distributionally Robust MoE Training (DRMoET) to make routing more robust. Their method adjusts how tokens are assigned to experts based on expert performance, improving overall results across multiple tasks. This helps MoE models scale up more reliably without just balancing workload equally.
Open → 2610.07207v1

UBTree speeds up language model decoding with new parallel token drafting

UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding

Abstract: Speculative decoding accelerates language model inference by verifying multiple draft tokens in a single target-model pass. Recent parallel drafters have achieved breakthrough performance in frontier production models, but their effectiveness deteriorates as the entropy of target distributions increases due to insufficient draft diversity. To overcome this bottleneck without sacrificing parallelism, we introduce UBTree, a parallel drafter that couples a Unigram proposer with a Bigram selector to construct drafting Trees. The unigram proposer is trained with the standard cross-entropy objective to generate candidate tokens independently for each position, while a lightweight bigram selector predicts transition scores between adjacent candidate pairs. Unlike the proposer, the selector is trained with a renormalized KL objective on high-temperature data. This tree-native training broadens the supervision beyond the greedy path, encouraging plausible alternative branches that improve the chance of accepting additional tokens during tree verification. Across seven standardized benchmarks with Qwen3-4B and Qwen3-8B, UBTree achieves an average speedup of $5.84$--$6.94\times$ over autoregressive decoding and outperforms DARTree in all 28 comparisons. Production-scale evaluation further demonstrates UBTree's advantage over frontier baselines such as DSpark.

Wed 30 SeptComputation and Language
The gist
Language models write text one word at a time, which can be slow. The paper presents UBTree, a method that guesses many possible next words in parallel using two models—a unigram model to suggest candidates for each position, and a bigram model to pick pairs of words that fit together better. This approach creates a tree of possible word drafts that can be quickly checked and accepted, making the process faster. The authors tested UBTree on standard benchmarks and found it speeds up decoding more than previous methods without losing accuracy.
Open → 2609.39972v1

Memory efficient gpu clustering method speeds up large scale k means

GEM-KMeans: Memory-Efficient and Accurate Clustering on Massive Scale with GPU Optimization

Abstract: Memory-efficient scaling on clustering problems without sacrificing statistical accuracy is of central interest for large-scale data analysis and machine learning problems. Nonnegative low-rank (NLR) matrix factorization for $K$-means is a scalable clustering method, which connects to semidefinite relaxations with optimal average-case exact recovery guarantees. However, a direct GPU implementation of NLR requires multiple large factor-sized buffers and substantial data movements that are essentially memory-bound. In this paper, we introduce GEM-KMeans, a spectrally normalized yet mathematically equivalent NLR formulation that fuses the gradient update, nonnegative projection, and sufficient statistics for normalization and iterate movement into a matrix-multiplication epilogue. Instead of retaining three massive factor-sized arrays, our IO-aware GPU implementation materializes only one single factor with small tile-reduction arrays as additional storage in the High Bandwidth Memory (HBM). We derive explicit memory costs and spectrally normalized smoothness bounds for optimizing the clustering objective function. Accurate clustering is demonstrated at massive scales on synthetic and real datasets, where performance gains of GEM-KMeans over existing GPU-accelerated Lloyd's algorithms involve data-dependent runtime tradeoffs.

Mon 28 SeptMachine LearningHardware ArchitecturePerformance
The gist
Clustering large datasets efficiently and accurately is important but challenging because of heavy memory use. The authors created GEM-KMeans, a new way to do a math step called nonnegative low-rank factorization for k-means clustering that reduces memory needs on GPUs. This method cleverly combines several computations to keep just one main data array in fast GPU memory, cutting down movement and storage. Tests on large real and synthetic datasets show it works well and can be faster than standard GPU-based methods depending on the data.
Open → 2609.36074v1

Efficient attention method speeds up large context processing

SANTA++: Sampling Attention through Representative Keys

Abstract: Attention often concentrates on a small subset of tokens in the context, but which subset matters changes from one query to the next. To exploit this changing structure, we introduce SANTA++, a training-free stochastic attention method that uses representative keys for memory-efficient selection without scanning the entire key-value (KV) cache. Cached keys are organized into teams, and the query scores one representative from each team to decide which teams to sample. We compute exact attention scores within the sampled teams and reweight each team's contribution by the inverse of its inclusion probability. This importance sampling correction estimates attention over the full cache, with a sampling budget that lets us trade memory reads for accuracy. Remarkably, with 32 or 64 sampled teams, SANTA++ uses 16% to 22% of dense attention's KV reads and retains 94% to 99% of the dense-attention baseline's scores on LongBench v2 and HELMET's retrieval-augmented generation subset, and 85% to 91% on RULER, with Qwen2.5-7B-Instruct at 32K context. With 31 sampled teams, our GPU implementation delivers a $1.69\times$ attention speedup over the dense FlashAttention baseline at 32K context. By reducing the number of cache entries read, SANTA++ in principle complements architectures with compressed KV representations, such as multi-head latent attention. Our kernels are available at: https://github.com/OPUSLab/santapp-kernel-demo.git.

Mon 28 SeptMachine LearningComputation and Language
The gist
Processing long sequences of information in AI models can be slow because they have to look at everything at once. The authors introduce SANTA++, which cleverly picks smaller representative parts to pay attention to without having to check everything. This approach keeps results very close to full processing but reads much less data, making it faster and more efficient. Their tests show it works well on tasks requiring understanding of very long context.
Open → 2609.35629v1

Relational AI personas boost user engagement and goal diversity

AI Persona, Service Consumption, and User Intent Entropy: Field Experimental Evidence from an LLM Platform

Abstract: Problem definition: Firms deploying large language model services must decide how their AI communicates, not just what it can do. We examine how a relational persona - warmer, more empathetic and more engaging than a non-relational persona - affects service consumption and the evolution of user objectives. Methodology/results: In a randomized field experiment with 9,586 newly registered users, we hold the underlying model and service capabilities constant. The relational persona increases interactions (sessions, +8.1%; duration, +10.6%; chat rounds, +24.2%; intent entropy, +5.8%) and outputs (files, +12.3%; distinct goals, +12.1%). Effects vary by entry intent. First-session effects are insignificant for Task Execution users. Socialization and Knowledge Exploration users show similar increases in chat rounds: Socialization increases intent entropy without more outputs, whereas Knowledge Exploration increases outputs without higher intent entropy. Modeling intent dynamics as a transition process, we find higher intent transition entropy for Socialization (+11.8%) but higher intent continuation probability for Knowledge Exploration (+12.8%), suggesting greater conversational breadth and persistence, respectively. In subsequent use, the relational persona increases aggregate chat rounds and outputs across all entry intents. Session count rises by 12.2% for Task Execution and 49.4% for Socialization, but not significantly for Knowledge Exploration. Effects on session count and intent entropy strengthen over time, whereas output effects remain stable. Managerial implications: AI persona is an operational design lever, not merely a presentation feature. Because more interactions do not uniformly generate more outputs, firms should evaluate interactions and outputs separately and consider matching persona to user intent, especially when added interactions consume costly computing resources.

Sun 20 SeptHuman-Computer Interaction
The gist
Choosing how an AI talks to users can change how people use it, not just what it does. The authors found that when an AI sounds warmer and more empathetic, users interact more, explore more topics, and produce more work. Different kinds of users respond differently: some chat more broadly, others stick with their goals longer. This means companies should think carefully about the AI’s personality to match what users want.
Open → 2609.23274v1

Distributed database testing tool finds new query processing bugs

Distribution-Aware Distributed Database Testing (Extended Version)

Abstract: Distributed database management systems (DDBMSs) introduce new challenges for assessing their reliability due to distribution-specific characteristics that affect query execution and optimization. Existing testing approaches, largely designed for centralized DBMSs, often fail to explore diverse distributed execution behaviors and suffer from low executability of generated test queries, thereby limiting their effectiveness in bug detection. We propose DAT (Distribution-Aware Testing), a novel automated approach for detecting query-processing bugs related to distribution strategies and distributed optimizations in DDBMSs, by systematically leveraging distribution-aware information throughout the testing pipeline. DAT builds on a set of techniques that capture diverse combinations of logical schemas and data distribution strategies, and performs guided query mutation to trigger a wide range of distributed query execution behaviors and optimizations, while improving query executability via historical feedback. We implement our approach in a tool, DistRanger, and evaluate it on four widely used production DDBMSs. It uncovers 31 previously unknown bugs, including 28 related to distributed query processing and optimization, and outperforms state-of-the-art testers.

Wed 16 SeptDatabases
The gist
Distributed databases split data across many machines, making it harder to test if they work correctly. Existing tests often miss problems because they don’t cover how data is spread out and handled. The authors created DAT, a new method that understands how data is distributed and changes queries to explore more cases. Their tool, DistRanger, tested four popular distributed databases and found 31 unknown bugs, mostly involving how queries are processed and optimized across machines.
Open → 2609.18501v1