Hardware-Transparent I/O Governance in Disaggregated Heterogeneous Storage
Distributed, Parallel, and Cluster ComputingOperating SystemsPerformance
Summary
The authors address two scheduling problems in storage clusters used by databases and block storage: keeping performance steady across different hardware types and enforcing overall I/O limits when some storage nodes get lots of traffic. They introduce the I/O Resource Manager (IORM), a scheduler used in Oracle Exadata that uses a hardware-aware model to normalize I/O costs, a rate limiter that allows brief bursts but respects long-term limits, and a feedback controller that balances load across nodes. Their tests show IORM maintains performance limits accurately, scales well with many users, and quickly recovers after node failures. The authors also share practical insights from using IORM in production.
Authors
Rajarshi Chowdhury, Akshay Shah, Sue K. Lee
Abstract
Shared-nothing disaggregated storage clusters that serve both latency-sensitive databases and opaque block-volume workloads face two governance problems unsolved by existing schedulers: maintaining consistent performance across heterogeneous hardware generations, and enforcing global I/O limits when access patterns skew to a subset of storage nodes. We present the I/O Resource Manager (IORM), a multi-stage distributed scheduler deployed in production within Oracle Exadata Exascale. IORM combines three mechanisms: a hardware-aware cost modeler that normalizes I/O accounting using datasheet-derived fixed costs to make limits invariant across hardware generations; a quantum-based rate limiter with bounded carry-forward credits that accommodates database micro-bursts while enforcing long-term SLOs; and a distributed adaptive feedback controller that redistributes unused entitlements across the cluster to resolve topological access skew. Beyond design, we share operational lessons from production deployment. On an 8-node test cluster running up to 100 concurrent tenant volumes, IORM converges within 5\% of provisioned limits under extreme sequential skew, scales without inter-tenant interference, and recovers full throughput within 15 seconds of a storage-node failure.