data center network architects

Abstract: This paper examines input/output buffered crossbar switches under combined packet and cell data. Our switch architecture uses input buffering with Virtual Output Queues to avoid Head of Line Blocking. The switch fabric is a crossbar with no speedup. We use a modified iSlip [McKeown] crossbar scheduler geared towards packet data, called piSlip. Cell ports are largely unmodified from standard iSlip behavior. For packet output ports, we introduce changes to the grant pointer which minimizes output latency caused by packet reassembly. Our model uses several output states, including packet cut-through. From simulation results, we show that piSlip with virtual cut-through offers latency characteristics significantly better than unmodified iSlip with similar packet port interfaces. Simulation further shows that piSlip and iSlip have similar maximum and average buffering requirements.

Abstract: Ring All-Reduce is widely used within large-scale distributed training for exact computation of the aggregate gradient. For a system with $N$ workers, its normalized per-worker communication is $2(N-1)/N$. In this work we present a communication-efficient Ring All-Reduce (CERAR) protocol for ``approximate'' gradient aggregation. CERAR partitions each local gradient into $c$ components and crucially relies on linear encoding and decoding operations. It performs $L=c+N-2$ communication rounds over an $N$-worker ring, yielding normalized communication rate $1+(N-2)/c$ and normalized storage $1+N/c$. We present an explicit Vandermonde-based construction, whose approximation error can be made arbitrarily close to zero with communication and storage rates approaching one with increasing $c$. However, this limit is achieved through ill-conditioned encoding matrices. Accordingly, we give a multiplicative perturbation construction whose error is $O(ε)$, while the relevant condition numbers are $O(ε^{-r_\star})$, where $r_\star=\lceil c/N\rceil-1$. This naturally motivates a condition-number-constrained optimization formulation for trading off the competing objectives and obtaining numerically stable practical designs. We present experiments on multi-GPU clusters with low-bandwidth and high-bandwidth interconnects. Our results demonstrate clear benefits in the low-bandwidth setting, even for moderate parameter length. We expect corresponding improvements even in the high-bandwidth setting for experiments with much higher parameter lengths.