Sampling strategies affect accuracy of web security studies
You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements
Cryptography and Security
Summary
Web security researchers often study large lists of websites to find security problems, but checking every site can be too expensive. Instead, they look at a smaller sample, but it hasn’t been clear if common ways of picking these samples give a true picture. The authors studied different sampling methods and found that picking the top popular sites misses many important details and can bias results. They suggest using a probability-based approach that gives more reliable estimates and works well even when you don’t know how common a security issue is.
What this means in practice
- •For web security teams: Choose probability-based sampling to get more accurate estimates of vulnerabilities when testing website security at scale.
- •For internet measurement engineers: Implement adaptive probability sampling to improve reliability of large-scale web measurements despite unknown issue prevalence.
- •For cybersecurity product developers: Design vulnerability scanning tools that incorporate adaptive sampling strategies to balance cost and coverage efficiently.$Commercial implications: Enables new scanning tools that deliver cost-effective, unbiased security assessments by using adaptive sampling methods.
Authors
Xuenan Zhang, Yuqing Yang, Giancarlo Pellegrino
Abstract
Web measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling. Despite its widespread use, sampling remains largely guided by convention---most commonly \emph{Top $N$} domain selection---rather than evidence, and its influence on the validity and generalizability of security findings has received little systematic evaluation. Consequently, it remains unclear whether common sampling strategies introduce systematic bias, distort observed vulnerability rates, or limit comparability across studies. In this work, we undertake, to the best of our knowledge, the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions. Through a comprehensive literature review and large-scale measurements of 500k Tranco and 24.8M Common Crawl hosts, we perform a comparative evaluation of datasets and sampling strategies. We show that, while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web. Instead, probability-based strategies yield stable, unbiased estimates for prevalence and many impact objectives. Hybrid sampling provides no advantages over pure probability sampling, as its deterministic prefix consistently contributes negatively to accuracy. Building on these results, we provide data-backed guidance for future studies, proposing to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.