Accelerated token arrays enable fast analysis of massive network data
Analyzing 10 Petabit/s Network Data with Accelerated Associative (Token) Arrays
Networking and Internet ArchitectureCryptography and SecurityDistributed, Parallel, and Cluster ComputingMathematical SoftwarePerformance
Summary
As computer networks grow bigger and more important, it gets harder to analyze their traffic quickly while keeping data private. The authors show that mathematical tools called associative arrays, when sped up using powerful graphics processors (GPUs), can process vast amounts of network data efficiently. They tested their approach on large clusters of CPUs and GPUs, achieving performance that could handle networks transmitting data at speeds of 10 petabits per second. This work shows a promising way to keep up with analyzing very large and complex networks.
What this means in practice
- •For network operators: Analyze large-scale network traffic quickly and privately using accelerated associative arrays implemented on GPU clusters.
- •For cloud infrastructure teams: Deploy GPU-accelerated associative arrays to monitor and diagnose anonymized network data at petabit speeds across distributed nodes.
Authors
Jeremy Kepner, Hayden Jananthan, LaToya Anderson, William Arcand, David Bestor, William Bergeron, Chansup Byun, Alex Bonn, Daniel Burrill, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Piotr Luszczek, Peter Michaleas, Lauren Milechin, Julie Mullen, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee, Alex Pentland
Abstract
As networks expand and become an ever more critical infrastructure to modern society the need to analyze these networks with the highest regard for privacy is essential to ensure their proper function. Depending on the level of the network layer to be analyzed, sources and destinations can be any combination of physical, logical, or persona/agentic endpoints, which requires the ability to handle diverse data. Invaluable to these analyses are mathematical tools that enable sophisticated mathematical algorithms to be expressed succinctly while achieving scalable vertical (within a compute node), horizontal (across compute nodes), and temporal (over different generations of hardware) performance. Associative (token) array mathematics and corresponding libraries is one approach that can meet these requirements. Accelerating these libraries with GPUs enables the analysis of the largest networks. The MIT/IEEE/Amazon Anonymized Network Sensing Graph Challenge provides a venue for highlighting the applicability of accelerated associative arrays for these types of problems. The D4M associative library has been implemented in a number of languages. This work benchmarks a prototype Matlab D4M GPU accelerated implementation of the Anonymized Network Sensing challenge across a wide range of CPU and GPU hardware. Scalable performance is demonstrated within and across CPU cores, CPU nodes, and GPU nodes. Horizontal scaling across multiple nodes was linear. Running on hundreds of GPU nodes simultaneously achieved a sustained processing rate sufficient to potentially analyze a 10 Petabit/s network.