Simple method selects useful data fields for AI analysis
SAIVE: Selecting AI Valuable Entities
Databases
Summary
Large companies collect huge amounts of data from computers and networks, but not all information is useful for AI analysis. The authors developed SAIVE, a simple way to pick which pieces of data are likely to help AI work better by looking at a small sample of the data. They base their method on a mathematical pattern called Zipf-Mandelbrot distribution to filter out unhelpful data without needing experts. This helps make AI analysis more efficient and focused.
What this means in practice
- •For enterprise data teams: Improve AI analysis speed and quality by automatically identifying the most informative data fields in large telemetry datasets.
- •For network operations teams: Filter network sensor logs to focus on the data fields most likely to show meaningful behavior shifts for faster anomaly detection.
Authors
Inna Voloshchuk, Hayden Jananthan, Jeremy Kepner
Abstract
Data lakes store large amounts of telemetry, with logs from network sensors, hosts, and applications containing possibly hundreds of fields for every event. Large enterprises are then left with data lakes that cannot be analyzed efficiently with AI. Aggregate analysis looks at persistent shifts in behavior over time. Many of the fields and columns in data lakes are not useful as they do not contain information that is sufficiently diverse or concentrated to support AI analysis. SAIVE is a simple method for examining a few rows in a large table and applies a histogram of histograms filtering criterion to select the fields that for AI analysis is more likely to yield useful results. This paper provides a principled foundation for the SAIVE heuristics by assuming of a Zipf-Mandelbrot power-law distribution of the underlying data. Constraining the Zipf-Mandelbrot exponent alpha to a reasonable range provides a a practical, cheap, expert-free filter for selecting AI valuable entities in large data sets.