Papers for

cloud storage operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Chiron uses llm advice to improve cache speed for ai workloads

Chiron: Cache Management Empowered by LLM Intelligence for AI Workloads

Abstract: AI clusters keep data in remote object stores and rely on local caches to stop GPUs from waiting for data. A cache performs well only when its prefetching and eviction policies match the access pattern, yet these patterns differ across AI workloads and change between phases of one workflow. Existing methods either need code changes, which compromises user transparency, or use low-end heuristics, which cannot follow such changes. Large language models (LLMs) can infer access behavior without user hints, but direct LLM control of cache actions suffers from long reasoning delay, high token cost, and poor decision quality. We present Chiron, a cache system based on advisory switching control. An LLM asynchronously diagnoses the access pattern and recommends a policy. A deterministic harness gates this advice, executes expert-designed policies locally, and returns cache feedback for review. We implement Chiron atop JuiceFS and evaluate it on nine AI workloads. It reduces job completion time by 14.3-41.2% over the best baseline on popular AI workloads.

Wed 4 NovDistributed, Parallel, and Cluster ComputingMachine Learning
The gist
Many AI programs need to quickly access data stored far away, so they use local caches to save time. But picking the right way to manage these caches is tricky because different AI tasks use data differently and change over time. The authors introduce Chiron, which uses large language models (LLMs) to guess what kind of data access will happen next and suggest how to manage the cache. Instead of relying only on the LLM's advice, Chiron checks and applies expert rules to keep things fast and efficient. Their tests show Chiron can speed up AI tasks by a noticeable amount compared to older methods.
Open → 2611.06179v1

Serialization invariant checks improve data integrity verification by agents

Same Data, Different Bytes: Serialization-Invariant Integrity Checks for Agent-Published Datasets

Abstract: Agent networks require reliable ways for requesting agents to verify where data is located, how it can be accessed, and whether the data received matches what the data owner made available. Checksum mismatches can reflect either altered content or serialization differences that preserve data equivalence under specified rules. Data identification mechanisms for agents, such as DataFacts, describe where data can be accessed and provide checksums for checking received data against published references. We propose a serialization-invariant identification method that assigns the same identifier to data representations equivalent under explicit rules. We use DataFacts as a case study, retaining its existing checksum alongside the proposed identifier. In a controlled agent study, we compare the proposed configuration with ordinary-tool and checksum baselines across equivalent re-encodings, altered content, and unavailable references. The study includes three model tiers from each of two providers. For equivalent re-encodings, the proposed configuration increased expected-outcome rates by 80.4 percentage points over the ordinary-tool baseline and by 70.2 percentage points over the checksum baseline. The proposed configuration also increased evidence-supported rejection of altered content by 3.5 percentage points over the checksum baseline. We contribute a serialization-invariant identification method with explicit equivalence rules, an integration approach illustrated through DataFacts, and an empirical evaluation against ordinary-tool and checksum baselines.

Sat 17 OctCryptography and SecurityArtificial IntelligenceMultiagent Systems
The gist
Sometimes the same data can be stored in different ways that look different to computers but actually mean the same thing. This can confuse software agents that check if data they received is the same as what was originally shared. The authors created a method that treats these different but equivalent versions as the same, so agents can verify data more accurately. They showed in tests with multiple data models that their method catches real data changes better and ignores harmless formatting differences. This helps agents trust the data they get without being tricked by simple re-encodings.
Open → 2610.24916v1

IAPRepair improves erasure-coded storage repair speed with in-network aggregation

IAPRepair: In-Network Aggregation Enhanced Proactive Repair for Erasure-Coded Storage System

Abstract: Erasure-coded storage provides fault tolerance with substantially lower storage overhead than full replication, but repairing lost or at-risk blocks requires intensive cross-node data transfer. Existing reactive repair schemes start only after a failure, while proactive schemes can move data before failure but commonly treat migration, reconstruction, and network aggregation as loosely coupled operations. As a result, receiver?side bottlenecks, heterogeneous available bandwidth, and limited programmable-switch state continue to constrain repair paral?lelism. This paper presents IAPRepair, an in-network aggre?gation enhanced proactive repair framework for erasure-coded storage. IAPRepair jointly constructs each repair batch, assigns reconstruction providers and replacement nodes according to normalized transmission loads, schedules migration around the remaining receive capacity, and selectively enables in-network aggregation for reconstruction blocks that would otherwise over?load healthy nodes. The selective design reduces receiver-side traffic while retaining migration parallelism and respecting a configurable switch-resource budget. We implement IAPRepair with a Tofino programmable switch and 16 storage nodes, and evaluate it using both a prototype testbed and large-scale simu?lations. Across coding parameters, block sizes, node populations, and multiple STF-node scenarios, IAPRepair reduces repair time by at least 47.83% compared with the evaluated state-of-the-art methods. The results demonstrate that coordinating proactive repair decisions with in-network processing is an effective way to improve repair efficiency under bandwidth heterogeneity.

Thu 8 OctDistributed, Parallel, and Cluster Computing
The gist
Erasure-coded storage keeps data safe with less extra space than full copies, but fixing lost data pieces needs lots of data moving between computers. Previous methods either wait for failures or separate tasks like moving and rebuilding data, causing delays. The authors created IAPRepair, a system that plans repairs ahead, balances data loads, and uses special network switches to combine data during repair. This approach reduces network traffic and speeds up repair times by nearly half compared to current methods. Their tests show it works well across various setups and network conditions.
Open → 2610.11114v1

Optimal data conversion bandwidth found for special fault tolerant codes

Optimal Conversion Bandwidth for MDS Convertible Codes in the Split Regime with $r^F<k^F<r^I$

Abstract: Erasure codes are widely used in distributed storage systems to provide fault tolerance. An $[n,k]$ erasure code encodes $k$ data symbols into $n$ coded symbols and distributes them across $n$ storage nodes. Once the code parameters are fixed, the achievable fault tolerance is also fixed. However, the failure rates of storage nodes may vary over time, and dynamically adapting the code parameters to these variations can substantially reduce storage overhead. Motivated by this observation, Maturana and Rashmi introduced convertible codes, which allow an $[n^I,k^I]$ initial code with redundancy $r^I=n^I-k^I$ to be transformed into an $[n^F,k^F]$ final code with redundancy $r^F=n^F-k^F$, while preserving the required code properties. Convertible codes whose initial and final codes are both MDS codes are called MDS convertible codes. They are of particular interest because MDS codes provide the maximum erasure tolerance for a given amount of storage overhead. Several works have established lower bounds and constructions for the bandwidth cost of MDS convertible codes in the split regime. However, the tight bound in the parameter range $r^F<k^F<r^I$ remains unknown. In this work, we establish a family of new rank inequalities for stable MDS convertible codes with linear conversion procedures and derive an improved lower bound on the bandwidth cost consisting of three cases for this remaining range. We prove that the bound is tight by presenting three explicit constructions, one for each case.

Tue 6 OctInformation Theory
The gist
Distributed storage systems protect data by splitting and encoding it across multiple nodes. Sometimes, the system needs to change how much extra data it stores to handle different failure conditions. The paper focuses on a special kind of code, called MDS convertible codes, which can switch between configurations while keeping data safe. The authors solved an open problem about the best possible bandwidth needed to switch these codes in a tricky parameter range, and they also provide exact ways to achieve this efficiency.
Open → 2610.08349v1