Papers for

web data crawlers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Email address presentation varies widely on computer science homepages

Beyond "[at]": A Large-Scale Measurement of Email Address Presentation on Computer Science Scholars' Homepages

Abstract: Publishing an email address makes a scholar reachable, but also exposes it to automated collection. Yet how email addresses are presented on the Web has not been measured systematically across their content, form, and location. We measure it across 32,413 homepage records of computer-science scholars listed in CSRankings with MailScope, which preserves raw and rendered HTML, extracts addresses with a three-stage cascade (rules, then large language models, then a crawling agent), and labels each address. Among 19,517 reachable homepages, we find an address for 17,712 scholars (90.8%). Our measurement yields four main findings. (i) Presentation is diverse: only 53.8% publish plain content, the others spread over 23 subtypes of symbol substitution, code, and semantic description, and 5.5% of scholars' addresses appear only after rendering. (ii) Presentation follows hosts rather than individuals: scholars on one host share a content class in 67.9% of pairs, compared with 38.8% across hosts of one institution, whereas research output and institution rank are barely related to presentation. (iii) Content and form are coupled through code-based links, and 6.2% of scholars with non-plain selected presentations have plain-text copies of the same address elsewhere. (iv) Plain addresses are more common behind passed bot challenges, and 36.1% of redirecting homepages no longer lead to scholars. Human assessment yields 94.4% weighted exact-set agreement on 143 judgeable pages, complemented by three 500-page review rounds with human verification. Presentation is diverse but clusters by shared host, and by our label-based estimate plain-string matching misses nearly half of the addresses.

Mon 26 OctInformation Retrieval
The gist
Publishing email addresses on personal web pages helps people reach scholars but makes the addresses vulnerable to being collected by automated tools. This study looked at over 32,000 computer science scholars' homepages and found that almost half use ways to hide or change their displayed email addresses to protect them. The way email addresses are shown is more influenced by the website host than by individual scholars or their institutions. The authors also found that simple text searches miss nearly half of all email addresses due to these variations.
Open → 2610.35499v1