Papers for

data center infrastructure teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

AMD GPU features keep some speed gains across real workloads and sharing

Capability Gain from ISA to Shared Execution on AMD MI300A and MI250X

Abstract: Each new GPU generation is showcased by its peak instruction rates, yet users hardly ever run instructions directly: they reach the hardware's performance through vendor libraries, applications and devices shared with other jobs. How much of a theoretical speedup survives these layers is rarely measured, so a team deciding whether to move to a new generation, or to adopt FP8 or 2:4 sparsity, lacks adequate guidance. AMD's MI300A (CDNA3) adds new FP8 matrix instructions and 2:4 structured-sparse instructions that the predecessor, the MI250X (CDNA2), does not have. We analyze this theoretical speedup gap on AMD's MI300A (CDNA3) against the MI250X (CDNA2) by defining a new metric, capability gain. Capability gain is the fraction of a feature's instruction-level speedup that it keeps at a higher level, such as a library, an application or a shared device. We trace it from ISA microbenchmarks through hipBLASLt and hipSPARSELt to LLM inference and mixed-precision HPL, and onto a device shared by up to eight concurrent streams. FP8 keeps most of its speedup through the library, completing an $8192^3$ GEMM \genCapPkgEndToEndGainEightK{} faster than the MI250X with FP16, but unfused quantization turns it into a per-layer slowdown in an LLM, and 2:4 sparsity keeps at most about half of its speedup in hipSPARSELt. Under queue sharing, FP8's GEMM capability gain rises above 1. How fairly streams share the device depends on the number of hardware queues, not on the numeric precision, yet shared LLM serving gains nothing from FP8.

Wed 4 NovDistributed, Parallel, and Cluster Computing
The gist
The team studied how new AMD GPU features that speed up certain calculations actually perform in real-world use, like in libraries, AI tasks, and when the device runs many jobs at once. They found that some features, like FP8 math, mostly keep their speed advantages in basic math libraries but not always in AI language models because of extra steps. Another feature, 2:4 sparsity, loses about half of its speed benefit in sparse math libraries. How well speed improvements show up when sharing the GPU depends more on hardware queues than number types. The study helps users understand when upgrading hardware or using new math tricks will really speed up their work.
Open → 2611.07198v1

Large language models quantized with better accuracy using weight decomposition

Quantization of Large Models with Weight Outliers and Sparsity

Abstract: Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contribution decomposes the original trained weights into a quantized matrix, a low-rank component, and a sparse matrix that accounts for outlier rows in the weights. Due to analytical intractability and the discrete structure of the decomposition, a simulated annealing search is employed to explore candidate decompositions while avoiding poor local minima. Numerical tests demonstrate the effectiveness of this approach across a broad suite of large language models, including Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen2.5 7B, evaluated on natural language generation benchmarks using 4-bit NF4 and FP4 quantizers. The novel method can markedly outperform direct quantization while incurring minimal memory overhead.

Fri 30 OctMachine Learning
The gist
Storing big AI language models usually needs a lot of memory because their 'weights' are high precision numbers. The authors found a new way to shrink these weights into fewer bits while keeping the model's performance good. They break down the weights into simpler parts including a sparse part to handle unusual values, and use a clever search method to find the best breakdown. This method works well on several popular large language models and allows smaller memory use without hurting how well the AI talks and writes.
Open → 2611.00286v1

Priority aware scheduler improves serverless function startup and balance

Priority-Aware Scheduler for Serverless Functions Provision and Load Balancing

Abstract: Serverless computing has emerged as a pivotal cloud deployment paradigm due to its efficient utilization of on-demand resources. However, the persistent cold start problem continues to hinder system performance. Current strategies often employ oversimplified preloading/unloading mechanisms that inadequately exploit invocation patterns, failing to optimize the critical trade-off between startup latency and resource efficiency. Furthermore, existing approaches disregard practical constraints of memory-limited multi-worker environments, exacerbating function eviction and load imbalance issues. To address these limitations, we present SPES-Ext, a rule-based priority-aware scheduler designed for memory-constrained distributed worker nodes. Our methodology stems from two key observations: (1) serverless systems exhibit concentrated invocation patterns enabling behavioral predictability, and (2) these patterns permit function categorization for granular instance management. This insight posits that characteristic-driven prioritization enables optimal scheduling rules, leading to our system design, which integrates three coordinated components: predictive function provisioning, eviction optimization, and dynamic load balancing. Experimental validation demonstrates the superiority of SPES-Ext over state-of-the-art solutions: 80% reduction in $75^{th}$-percentile cold starts, 83.92% decrease in memory waste, and 52.49% improvement in worker load variance.

Mon 26 OctDistributed, Parallel, and Cluster ComputingSoftware Engineering
The gist
Serverless computing lets programs run in the cloud without managing servers, but starting these programs slowly (cold starts) hurts performance. The authors noticed that serverless functions are called in patterns that can predict which ones to keep ready and which to remove. They created a scheduling system called SPES-Ext that uses these patterns and prioritizes functions to reduce cold starts, save memory, and balance workload better across workers. Their tests show big improvements in startup speed, memory use, and load distribution compared to previous methods.
Open → 2610.35073v1

Holistic resource management boosts cloud workload performance under congestion

Holistic Buoyancy-Driven Cloud Native Resource Management

Abstract: Efficient resource management is a challenge in multi-tenant cloud environments as workloads compete for shared resources such as last-level cache and memory bandwidth, whose congestion can change dynamically as workload composition and resource demands evolve. Existing resource management approaches often rely on static resource requirements or prior workload profiling, limiting their ability to respond to changing resource contention as required in the cloud. In this paper, we leverage workload buoyancy, a runtime framework for capturing workload sensitivity to individual resources and application-level performance, to enable adaptive scheduling and resource management. We propose a holistic approach that incorporates buoyancy into workload scheduling, node-local resource allocation, and workload rescheduling throughout the workload lifecycle. We implement our approach as a Kubernetes-native prototype and evaluate it on a multi-node cluster using diverse cloud workloads under varying levels of resource contention. In a high-congestion scenario, our approach increases the percentage of workloads satisfying their SLOs from 7.74% to 87.8% on average compared with the Kubernetes baseline, while reducing resource congestion, the severity of SLO violations, and cluster fragmentation using limited rescheduling.

Mon 19 OctDistributed, Parallel, and Cluster Computing
The gist
Managing computing resources in cloud environments is tough because many applications share limited hardware like memory and cache, which can get crowded and slow down programs. The authors propose using a technique called workload buoyancy that measures how sensitive each program is to different resources during runtime. This helps the system dynamically adjust scheduling and resource allocation to reduce congestion and improve performance. Their approach, built on Kubernetes, significantly increased the number of programs meeting their performance goals even in stressful conditions.
Open → 2610.26160v1

Analog memory stores multiple vectors in single array for ai models

Superposed analog memory for vector storage in neural networks

Abstract: The growing memory demands of artificial neural networks pose a major challenge for energy-efficient computing, particularly for models that rely on large collections of high-dimensional weight and embedding vectors. Conventional memory architectures store these vectors individually, requiring many memory cells per vector and limiting achievable storage density. Here we demonstrate an analog memory architecture that encodes multiple vectors in a distributed superposition within a single memory array. Individual vectors are recovered through in-memory matrix-vector multiplication, allowing the degree of superposition to be tuned to balance storage density against recall fidelity. This approach introduces a new scaling dimension for analog memory, in which effective storage density can be increased without a corresponding increase in the number of physical memory cells. The distributed representation also provides resilience to device-level noise. We implement the architecture on a prototype phase-change memory chip and experimentally evaluate it using embedding tables from deep learning recommendation models and large language models. The results demonstrate inference accuracy comparable to full-precision software baselines, establishing analog superposition as a promising route towards dense, energy-efficient memory for artificial intelligence.

Mon 19 OctEmerging TechnologiesHardware Architecture
The gist
Artificial intelligence models need a lot of memory to store their data, which can be energy-consuming. The authors show a new kind of analog memory that can hold many data vectors all mixed together in one place and still recover each one accurately. This means more data can fit in the same physical space, saving energy and resisting noise. They tested this on special memory chips using examples from language and recommendation models and found it worked as well as traditional digital methods.
Open → 2610.26118v1

GPU powered batch policy matching speeds up rule processing

BitPCE: Batched Multi-Dimensional PolicyResolution with GPU Bitmap Algebra

Abstract: Policy controllers must determine which workloads match rules spanning several label dimensions. Hash sets, compressed bitmaps, and GPU set-processing primitives address the underlying set operations. However, intermediate results, repeated scope evaluations, dependent policy updates, and host transfers can dominate the cost of a heterogeneous batch. We study this problem as batched intersection-of-unions (IoU). Our design exploits independent scope--word tasks, reusable endpoint expressions, and policy dependencies. The challenges are preserving wildcard, empty-group, and exclusion semantics, recognizing equal scopes, and retaining unchanged recipients affected by source edits. BitPCE addresses them with register-resident GPU evaluation, canonical scope compilation, and dependency-complete candidate selection under a stated rendering contract. IP-only changes reuse cached memberships. We establish correctness and word-work bounds. On a four-vCPU A10G host, resident GPU execution is 3.46--36.60$\times$ faster than the best tested native CPU path, including CRoaring, and 1.14--2.22$\times$ faster than an adapted operand-parallel GPU reduction. Gains narrow at the host-output boundary: on an eight-vCPU host, including upload and compact identifier output, resolution takes 16.3 ms versus 19.3 ms for CPU at 100K objects and 500 scopes. Sparse labels and small batches can favor CPU execution. Scope reuse, candidate selection, and incremental transfers are measured as components; complete update-to-enforcement latency remains unevaluated.

Sun 18 OctDistributed, Parallel, and Cluster ComputingCryptography and Security
The gist
Matching workloads to rules often requires checking many labels and rules, which can be slow and complicated. The authors study this challenge and create BitPCE, a method that runs on GPUs to quickly find which workloads match which rules in batches. They use smart techniques like caching and dependency tracking to avoid doing the same work repeatedly. Their method runs faster than CPU-only solutions, especially when handling large batches and many labels, but small or sparse cases might still favor CPUs.
Open → 2610.24974v1

Optical network control benefits from fast probabilistic physical layer model

Probabilistic Optical-Layer Surrogate for Risk-Aware Control-Plane Decision Making

Abstract: In this paper, we propose a generative surrogate for the optical layer. Current control-plane decisions in optical transport networks are made using analytical impairment models, deterministic learned predictors, or quantile-regression-based uncertainty estimates. These approaches expose only a limited, static summary of the optical layer at any given time. This forces the control plane to rely on conservative fixed margins that leave capacity underutilized. Our proposed surrogate is a conditional generative model trained on traces from optical simulators with validated physical models. The surrogate produces a full conditional distribution over physical-layer metrics such as OSNR, Q-margin, nonlinear penalty, and feasibility. This output allows the controller to estimate violation probability and tail risk at arbitrary thresholds at runtime. This is different from existing approaches because no retraining is required for each decision. We present the first case study in fast path feasibility estimation using a generative surrogate for the optical layer. We compare the surrogate against simulators, deterministic learned predictors, and quantile-regression estimators. We evaluate the surrogate along nine research questions (RQ1--RQ9) covering prediction accuracy, inference latency, uncertainty quality, controller integration, cross-emulator generalization, dynamic reconfiguration under span degradation, topology scalability, GNPy-validated generalization, and finally, inference latency on real-world live hardware traces. The core contribution is an optical-layer control abstraction where physical-layer behavior is exposed as a fast probabilistic query mechanism instead of an offline simulation output or a deterministic feasibility rule.

Thu 15 OctDistributed, Parallel, and Cluster Computing
The gist
Making decisions about how to send data through complex optical networks is tricky because the physical environment can vary and cause errors. Current methods use fixed safety margins or deterministic predictions that can waste network capacity. The authors trained a model that can quickly provide not just single predictions but full probability distributions about signal quality and possible failures. This allows network controllers to better estimate risks and make smarter choices without slowing down. Their approach was tested on real and simulated data, showing it works faster and with useful uncertainty information.
Open → 2610.21069v1

Mixed precision approach improves storage efficiency in large models

Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization

Abstract: Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.

Wed 14 OctMachine Learning
The gist
Large AI language models called mixture-of-experts activate only a few parts at a time but must keep all parts stored, which takes a lot of space. Simply ranking parts to decide their storage precision doesn't fully capture how important their accuracy is. The authors present NatScale, a new method that adjusts how much storage precision each part gets based on a calibrated measure of its error impact, while respecting their importance order. Testing on several models shows this method better preserves model behavior with less storage than other ways of allocating precision.
Open → 2610.20353v1

Gpu placement optimizer improves cloud efficiency with fewer migrations

OrigaMIG: MIG-Aware VM Placement with a Neighborhood-Restricted BILP and Live Migration

Abstract: The extensive use of GPUs in cloud computing, accelerated by the spread of large language model (LLM) services, and the growing need for multitenancy have driven the development of innovative solutions for efficient GPU resource management. Multi-Instance GPU (MIG) technology from NVIDIA enables shared GPU usage in cloud data centers by providing isolated instances, which are offered as MIG-backed virtual GPUs (vGPUs). However, MIG placement rules often lead to fragmentation and suboptimal resource allocation. In this work, we formally model the MIG-aware virtual machine (VM) placement as a binary integer linear programming (BILP) problem aimed at maximizing request acceptance, consolidating resources, and reducing migration overhead. Building upon this formulation, we propose OrigaMIG, a MIG-aware placement optimizer. OrigaMIG places each arriving request at once with a lightweight greedy rule and calls the solver only when a request cannot be placed or the GPUs of a class become fragmented. Each call solves the BILP on a small neighborhood of physical machines (PMs), keeps the rest of the data center fixed, and executes the solution as live migrations. We compare OrigaMIG with the default placement and with two state-of-the-art MIG-aware policies in simulation, across six loads, eleven variants of the workload and the hardware, and data centers of up to 4096 PMs. On 256 A100 and A30 GPUs in 128 PMs, OrigaMIG keeps fewer PMs active than the stronger policy in 29 of 30 paired runs while migrating 29% to 49% fewer VMs, and it uses the least energy per admitted GPU-hour at every load. Against the default placement, it keeps up to 12.5% fewer PMs active and admits up to 7.8 percentage points more of the requested GPU memory. On small data centers, it stays within 4.1% of the optimal number of active PMs.

Mon 5 OctDistributed, Parallel, and Cluster Computing
The gist
Cloud data centers use GPUs to run large AI models, but sharing these GPUs efficiently is hard because of strict rules about how they can be divided and used. The authors modeled this sharing problem mathematically and created OrigaMIG, a tool that decides where to place GPU tasks to use fewer machines and move fewer tasks around. OrigaMIG reacts quickly to requests and only adjusts small parts of the data center as needed. In tests, it saved energy and improved GPU usage compared to common methods.
Open → 2610.06646v1

Multi chiplet gpus boost speed and cut energy use drastically

Divide and conquer: Scalable performance and energy in MCM GPUs

Abstract: Multi-chip-module (MCM) GPUs offer a promising path to scale compute capability beyond monolithic designs by integrating multiple chiplets on a common package. However, the impact of disaggregation on performance scalability and energy consumption remains underexplored. The design space grows rapidly across dimensions such as SMs per chiplet, chiplet count, and interconnection network. The inter-chiplet network is particularly critical, as it determines whether additional compute resources translate into performance gains. This limited understanding leaves industry and research without clear guidance on the performance and energy trade-offs of MCM GPU scaling. In this work, we investigate whether distributing compute and memory capability across multiple chiplets offers a more scalable alternative to concentrating resources. We quantify their effects on performance, energy, and efficiency and examine how inter-chiplet topology influences scalability at different system sizes. Our results demonstrate that a 16 chiplet Torus configuration with 256 SMs delivers a remarkable $2.40\times$ performance improvement over a state-of-the-art MCM architecture with the same compute capability, while simultaneously reducing energy consumption by $4.45\times$. These substantial gains provide evidence that disaggregation is a first-order architectural factor and will be critical to unlocking the performance and energy-efficiency potential of next-generation GPUs.

Fri 2 OctHardware Architecture
The gist
GPUs, which help computers run fast graphics and calculations, usually rely on one big chip. The authors study a design where many smaller chips work together on one package. They found that spreading the work across multiple chips with the right connection pattern can make the GPU run much faster and use less energy. This approach, called disaggregation, could be key for future better and more efficient GPUs.
Open → 2610.03061v1

QuantForge improves four-bit quantization for large language models

QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantization

Abstract: Four-bit post-training quantization can reduce the memory demands of large language models, but preserving accuracy under strict MXFP4 W4A4 requires coordinating several design choices. Coordinate transforms change block-encoding errors, which in turn affect the residuals propagated through the network. The useful algorithmic decomposition is therefore not fully known before search. LLM-driven program evolution offers a way to explore these choices, but performance scores alone do not explain which design should change next. We introduce QuantForge, a PTQ discovery system that records competing explanations, selects controls that distinguish them, and checks that successor code implements the resulting conclusions. This residual compilation guides program revisions while retaining useful programs even when their original explanations are rejected. Remeasuring the revised program reveals the next error to address. This process discovers HiRes, a fixed MXFP4 quantizer that shapes coordinates, refines legal code assignments, and recovers errors along attention and MLP paths. Each stage acts on residuals measured after the preceding stage has executed. Across seven tasks, HiRes achieves the lowest seven-model Robust Fit (0.09300) and the lowest quantized Fit-7 at 32B. In matched-budget comparisons of LLM-driven program evolution, each with 240 evaluator calls, QuantForge reaches a held-out transfer target in six of eight runs, compared with three each for textual memory and reflection memory, and one for score-only evolution, despite evaluating fewer new programs. These results show that QuantForge improves the discovery of transferable PTQ algorithms by turning controlled evidence into subsequent program changes.

Mon 28 SeptMachine Learning
The gist
Reducing the memory needed by large language models usually harms their accuracy, especially when using very tight four-bit quantization. The authors created QuantForge, a system that helps discover better quantization methods by carefully analyzing errors and revising algorithms step-by-step. Their system guided the discovery of HiRes, a new quantizer that better preserves accuracy by adjusting how errors and residues propagate through the model. Tests on multiple tasks show that HiRes achieves more accurate results compared to other quantizers at the same memory budget.
Open → 2609.34680v1

PackServe improves large language model request scheduling efficiency

PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale

Abstract: Request scheduling is a key challenge in large-scale clusters serving agentic large language model (LLM) workloads. An effective scheduler must preserve key-value cache (KVC) reuse across long, shared prefixes, meet token-level latency service-level objectives (SLOs), and minimize GPU resource footprint. Existing schedulers struggle to reconcile these requirements: request consolidation can sacrifice cache locality and increase prefill/decode interference, compromising both SLO attainment and resource efficiency. We present PackServe, a scheduler designed to reduce resource costs while meeting latency SLOs for agentic LLM serving. PackServe uses compact white-box models to predict latency under prefill/decode interference. Guided by these predictions, it packs requests onto fewer serving instances while preserving KVC reuse and SLO constraints, trading available latency headroom for improved per-instance throughput. Evaluation on 64 NVIDIA H20 GPUs shows that PackServe uses up to 16.8% and 24.6% fewer GPU-hours than state-of-the-art schedulers under 30-ms and 50-ms TPOT targets, respectively, while meeting the target TPOT objectives. PackServe has also been deployed in our production cluster comprising over 1000 GPUs, where it reduces the resource footprint by 34.7% compared with the original production scheduler.

Sun 27 SeptDistributed, Parallel, and Cluster Computing
The gist
Scheduling many requests for large language models (LLMs) is hard because it needs to reuse cached work, meet strict timing goals, and save GPU resources. The authors created PackServe, a scheduler that predicts processing delays and smartly groups requests on fewer GPUs while keeping speed and cache benefits. This reduces GPU usage significantly compared to previous methods, saving energy and cost without slowing down response times. PackServe is already used in a big production system with over 1000 GPUs.
Open → 2609.33224v1

Extender reduces memory needs for transformer attention with new channel

The Extender: A Log-Structured Transformer

Abstract: We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual $\mathbf{h}$, a superposition channel. The Extender adds a concatenation channel $\mathbf{x}$: each layer $\ell$ emits both a residual update $δ_\ell$ which is added to $\mathbf{h}$, and a much smaller extension $ε_\ell$ which is appended to $\mathbf{x}$. While both the FFN and $\mathbf{q}$ see $\mathbf{h}$, the attention $\mathbf{kv}$ projections take only $\mathbf{x}$ as input. As a result, the fully extended $\mathbf{x}$ contains the complete input for the $\mathbf{kv}$ projections of all layers, reducing the persistent attention memory footprint from $2Ld_{model}$ to $\sum|ε_\ell|$. We find that with $|ε_\ell|=32$, the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters, and exceeds Transformer accuracy on long-context (RULER) workloads, again at 924M parameters. For our 1664-wide, 924M model, the Extender's persistent attention memory footprint is $104\times$ smaller than MHA. The memory savings grow with model width.

Sat 26 SeptMachine LearningArtificial Intelligence
The gist
Transformers are a type of AI model that process information in layers, usually passing one big summary through all layers. The Extender adds a second, smaller channel that keeps track of extra details from each layer. This makes it possible to use much less memory when looking at long sequences of information. The authors found that the Extender works as well as, or better than, standard Transformers especially on tasks needing longer context, while drastically cutting memory use.
Open → 2609.32759v1

Multi-asic switches boost network performance with circuit-switched indirection

The Power of Indirection: Scaling Switches Beyond Silicon Boundaries

Abstract: The slowdown of Moore's law and the area limit of monolithic integration have made chiplet-based designs inevitable across many domains, including network ASICs. However, combining multiple network ASICs together poses a fundamental challenge: maintaining sufficient inter-ASIC bandwidth to match the performance of an idealistic single-ASIC design. Providing full bandwidth is prohibitively expensive as it requires valuable forwarding capacity, while reducing inter-ASIC bandwidth creates severe performance bottlenecks. We propose a novel multi-ASIC switch architecture that introduces a circuit-switched indirection layer in front of the ASICs. This layer flexibly remaps ingress ports across ASICs, localizing traffic and minimizing inter-ASIC communication based on observed patterns. Our system, Fastroute, combines packet and circuit switching to deliver performance comparable to a single-ASIC switch while reducing inter-ASIC bandwidth requirements. This frees up capacity for external network interfaces, allowing Fastroute to outperform traditional non-oversubscribed multi-ASIC designs. Our hardware prototype demonstrates the system's functional feasibility by evaluating it on an LLM training workload. By reducing bandwidth and power overhead, Fastroute bridges the gap between silicon fabrication limits and soaring application demands. It provides an efficient transition to multi-ASIC switches, enabling bandwidth and radix demand to be met without waiting for the next ASIC generation.

Fri 25 SeptNetworking and Internet Architecture
The gist
Chips used in networks are hitting physical limits, so designs now combine multiple smaller chips (ASICs). But connecting many chips together is tricky because connecting them fully is expensive, and less connection slows down the whole system. The authors created a new design that puts a flexible layer in front of these chips to smartly redirect traffic, cutting down on the hard-to-connect parts and speeding things up. Their system, called Fastroute, mixes packet and circuit switching to keep performance high while saving on connection costs, proven with a hardware prototype running AI training workloads.
Open → 2609.31092v1

Pattern matching improves online resource allocation in cloud networks

Pattern-Aware Virtual Network Embedding Optimization for Cloud Data Centers

Abstract: The network virtualization (NV) technology has enabled the sharing of multiple resources among virtual networks (VNs) in cloud data centers. One of the key challenges is to allocate resources in real-time for virtual network request (VNR), which is known as online virtual network embedding (VNE). However, the existing online VNE methods do not exploit the multi-dimensional complementary relationship among diverse VNRs, resulting in the fragmentation and waste of substrate resources. In this paper, we propose the pattern matching based online VNE approach by constructing appropriate matching rules among observed patterns to maximize resources utilization. We devise the clustering based VNRs quantization method and conduct rigorous study on the pattern combination filtering problem. Then, we utilize the column generation to solve it and construct the pattern matching rules. Based on the rules, we propose an online pattern matching VNE algorithm with linear worst-case complexity. Evaluation on a 106-server testbed using Alibaba production cluster trace dataset shows that our algorithm achieves close-to-offline performance and more accepted workloads that outperforms traditional designs by 25%-30%.

Fri 18 SeptNetworking and Internet Architecture
The gist
Allocating resources for virtual networks in cloud data centers is tricky because many requests come in all the time and use resources differently. Existing methods don't fully take advantage of how these requests complement one another, causing resource waste. The authors propose a method that groups similar virtual network requests and matches patterns to use resources more efficiently. Their approach runs quickly in real time and performs almost as well as offline methods.
Open → 2609.21302v1

Fast method commits many power generators under grid limits

Fast Relax-and-Round Unit Commitment with Topological Constraints

Abstract: Recent developments in the US knowledge economy have created a significant growth in datacenter loads, with two major consequences for power generation. First, to compensate for growth in load, datacenters are encouraged to bring their own generating units. Second, in search of the remaining pools of dispatchable generation, utilities are increasingly turning to subtransmission and distribution level generating assets. Coupled with increasing loads and the resulting tighter grid conditions, both trends are likely to create a need to commit a large number of localized generating units under grid constraints. We propose an extension of Relax-and- Round Unit Commitment (RRUC) that is capable of committing generating units for larger problems faster than conventional methods, while staying within intertemporal and spatial MVA constraints. We demonstrate the performance of RRUC using synthetic congestible test systems ranging from 100 to 20,000 buses. RRUC consistently finds low cost solutions, independent of the problem size, and its run time increases sub-quadratically in the number of buses. RRUC can solve the 100 bus system in less than a second and the 20,000 bus system 7 minutes. In contrast, a leading state of the art solver cannot find a feasible solution to the 100 bus system in 15 minutes.

Fri 11 SeptMathematical Software
The gist
With more power needed for data centers, managing many local generators on the grid is tricky. The authors extended a technique called Relax-and-Round Unit Commitment to quickly decide which generators to turn on. Their method works well even for very large grid models, solving problems much faster than current solvers while respecting grid safety rules. This helps utilities plan generation more efficiently amid growing and shifting power demands.
Open → 2609.13480v1

Py-kvcache improves large language model caching performance with nvme ssds

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

Abstract: Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.

Thu 10 SeptDistributed, Parallel, and Cluster ComputingMachine Learning
The gist
Long conversations with artificial intelligence models can be slow because they need to remember everything said before. The authors studied how saving and reusing parts of these conversations, called key-value states, can speed up responses. They built a tool called py-kvcache that stores these states on fast NVMe disk drives and cleverly starts loading them before they're needed, making the process faster. They found that py-kvcache can speed up the time to start answering by up to twice compared to earlier systems, but using external cache depends on the setup and hardware.
Open → 2609.11744v1