Nested parallel computing architecture enables million processor coordination
Nested Parallel von Neumann Architecture and Nested BSP
Distributed, Parallel, and Cluster ComputingHardware Architecture
Summary
Computing with millions of processors is not about making one processor stronger but about coordinating many processors as if they were one. The authors introduce Nested BSP, a way to organize parallel tasks in layers where each group works, waits, exchanges data, and repeats evenly across all layers. They also propose extending the classic computer design to a Nested Parallel von Neumann Architecture, where all parts are peers connected by a unified bus. This helps build very large-scale AI computers that behave like one machine, not many separate ones.
What this means in practice
- •For hardware architects: Design large-scale AI computer systems with many processors coordinated as peers under a unified parallel-nested architecture.
- •For parallel software developers: Develop large parallel applications based on Nested BSP principles to manage multi-layered synchronization and communication among millions of processors.
A position paper. It proposes an approach and reports no results.
Authors
Heng Liao
Abstract
Large-scale AI computing is no longer a contest of ''one stronger processor,'' but of how an army of processors under one command can still be one computer. This paper offers two interlocking extensions. First, extend BSP to Nested BSP. The Turing machine describes computation as a single tape, in sequence. A million processors need not a longer tape, but a battle plan nested layer within layer: at every layer, parallel work, barrier, exchange and aggregate, then the next phase. Every ``parallel advance'' inside a layer repeats the same four steps. Nested BSP extends classic BSP by nesting it recursively, a computing paradigm for million-scale parallelism, under one rule: every node at every layer is a peer. Second, extend von Neumann to the Nested Parallel von Neumann Architecture, and Unified Bus is its interconnect. Von Neumann taught us how to build one stored-program computer. The false extrapolation of eighty years was that wiring many computers into a network yields one larger computer. A second habit ran deeper: nearly every design assumes a master that commands and slaves that obey---host over device, CPU over accelerator, center over edge. The Nested Parallel Architecture extends that idea rather than discarding it. Two nesting dolls must fit: Nested BSP in software, and the Nested Parallel von Neumann Architecture from package to autonomous zone, joined by one memory-semantic bus end to end, with full peer equality: physically sparse, logically tight. It pairs with Huawei's $τ$ Scaling law: $τ$ governs how each layer folds time, while the Architecture governs how the nested parallel computer stands, layer by layer, peer by peer. In summary, the paper extends BSP to Nested BSP and extends von Neumann to the Nested Parallel von Neumann Architecture. $τ$ folds time, peer-equal parallelism nests layer by layer---many processors, still one computer.