betaFeedback

WorksComputer Science

Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls

Abstract

Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing. Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.5-1.7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82-87% of tracker organizations observed by participants, compared with 58-68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.

Introduction

Tracking is pervasive on the Web, raising concerns about users’ privacy and the collection of their browsing data. Understanding the extent of Web tracking is important for assessing these risks and evaluating protections. Yet measuring what people actually encounter while browsing remains difficult. Most large-scale studies instead rely on automated crawlers [1, 19, 27, 52]. A crawler, however, is not a neutral observer: its browser configuration, automation framework, and browsing behavior can affect how websites respond and what tracking activity it records [2, 10, 31]. Consequently, automated measurements can differ substantially from human browsing [55]. A recent comparison with study participants found that an automated crawl missed fingerprinting on 45% of the websites where participants encountered it, a gap that the authors attributed to authentication, bot detection, and scripts that run only after user interaction [4].

Researchers have sought to reduce these discrepancies by simulating human interactions [21], using realistic browser configurations [10, 31], and reducing detectable automation artifacts [33]. More recently, LLM-driven browsing agents have emerged that interpret Web pages and decide how to navigate and interact with them [23, 56, 57]. Unlike traditional crawlers that follow predefined navigation rules, these agents can make context-dependent decisions about which links to follow and which elements to interact with, potentially resembling aspects of human browsing. These differences raise two questions: how does agentic browsing affect the tracking activity that crawlers record, and how do the resulting measurements compare with what people encounter when browsing the Web?

Answering these questions requires distinguishing three related but easily conflated aspects of Web tracking measurement: website interventions, the challenges and denials crawlers encounter; tracking visibility, the third-party tracking and fingerprinting a crawler records; and human agreement, how those observations compare with what people encounter on the same pages. A crawler can improve one of these properties without the others: it may avoid every intervention and still trigger little tracking, or it may record more tracking than any visitor encounters. We therefore investigate agentic crawlers across these three dimensions, comparing their observations with those of non-agentic crawlers:

  • RQ1 (Website interventions): How do the website interventions that agentic and non-agentic crawlers encounter differ, particularly beyond landing pages, and which configuration components account for the differences?

  • RQ2 (Tracking visibility): On matched URLs, how does the third-party tracking and browser fingerprinting that agentic and non-agentic crawlers observe differ, and what factors account for these differences?

  • RQ3 (Human agreement): How well do agentic and non-agentic crawlers recover the tracking and fingerprinting activity that participants observed, and what additional activity do they record?

To address these RQs, we study crawling on 2,000 popular websites (Figure 1), comparing four non-agentic and two agentic crawlers. The non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox. Their stealth variants incorporate browser fingerprint spoofing, with the Chromium variant also using Patchright, a modified version of Playwright designed to reduce automation detection. The two agentic crawlers are an agent built on the browser-use framework [7] and SuperScraper, a research crawler we built that perceives pages ends_mid=1 only through screenshots and acts through operating-system input. Because each crawler chooses its own pages, a non-agentic crawler revisited every URL that the agents had visited, which lets us compare crawlers on identical pages. Five targeted variants of the Chromium and SuperScraper crawlers isolate the effects of specific configuration choices. An additional interaction-logging crawl records the browser-use agent’s clicks and subsequent third-party script activity. Finally, browsing data from 26 participants on a subset of the sites provide a human reference.

Study design overview showing website corpus, crawlers and participants, targeted variants and logging, and research questions

Figure 1. Study design overview.

Our results reveal several ways in which crawler design affects measurement outcomes. First, website denials are similar across configurations on landing pages but increase on subsequent pages for browsers driven by unmodified Playwright. Replacing Playwright with Patchright largely eliminates this escalation, whereas browser fingerprint spoofing and simulated mouse movement do not. Yet Patchright does not increase recorded tracking, demonstrating that avoiding interventions and increasing tracking visibility are distinct. Second, on matched URLs, the agentic crawlers record trackers from 1.5–1.7 times as many organizations as non-agentic crawls. This difference is particularly pronounced on consent-managed pages. The interaction-logging crawl shows that our browser-use agent frequently accepted consent and that confirmed consent actions were followed by additional third-party scripts. Finally, the agentic crawlers recover 82–87% of the tracker organizations observed by participants on shared URLs, compared with 58–68% for non-agentic crawlers. However, only 48–49% of the agents’ own observations also appear in the participant visits. Thus, greater tracking visibility does not necessarily translate into closer agreement with human browsing.

In summary, our contributions are:

  • A systematic comparison of agentic and non-agentic Web crawlers. We evaluate crawler configurations across 2,000 websites, examining how navigation strategies, automation frameworks, browsing behavior, and browser environments affect website interventions and tracking visibility.

  • An evaluation against human browsing. We compare agentic and non-agentic crawler observations with participants’ visits to shared URLs, distinguishing the recovery of participant-observed tracking from additional activity absent from those visits.

  • Practical guidelines and reusable artifacts for Web tracking measurement. We derive recommendations for selecting, configuring, and validating crawlers according to different measurement objectives. To support reproducibility, we will release SuperScraper, our crawler configurations, measurement code, analysis code, and per-page crawl data.

Related Work

Validity of crawl-based measurement. Studies of tracking and fingerprinting rely on instrumented crawls [1, 19, 42] and on heuristics or classifiers applied to recorded browser API calls [6, 27, 34]. A growing body of work questions how well crawlers represent real browsing: automated crawls differ systematically from real users’ telemetry [55], common practices such as Selenium automation change what websites serve [10], crawler design and configuration alter measurement outcomes [2, 31], and internal pages differ from landing pages in structure and third-party activity [5, 50]. Instrumentation choices, such as recording JavaScript inside the browser runtime rather than through page-level hooks, also shape results and their reproducibility [15, 30]. Closest to our work, Annamalai et al. found that an automated crawl of the pages that 30 participants had browsed missed 45% of the websites on which the participants encountered fingerprinting [4]. We extend this comparison to agentic crawlers and separate the reasons why a crawler records more or less than people do.

Bot detection as a measurement threat. Websites detect automation through browser fingerprints [29, 51], behavioral signals [26], and network characteristics [28], often through commercial services [3], and because anti-abuse systems fingerprint clients to classify them [8], the fingerprinting that a crawler records can depend on whether a site considers it suspicious. Measurement crawlers have therefore adopted human-like interaction [21] and been audited for detectable artifacts [33], and honeypots have characterized automated traffic [36]. Gundelach et al. measured blocking on the homepages of the Tranco top 10K under four Playwright configurations and found it concentrated among particular infrastructure providers [22]. We show that homepage measurements hide most of the differences between configurations, which emerge on later pages, and that reducing blocking does not by itself increase the tracking recorded.

LLM browsing agents and consent. Browsing agents act on DOM-derived representations, screenshots, or both [11, 16, 23, 38, 56, 57], and they are evaluated on task completion rather than on what they are exposed to while browsing. Recent work studies agents from the website’s perspective, distinguishing them from humans and from one another through fingerprints and interaction traces [20, 39, 53] and proposing defenses against agent-driven scraping [37]. Agents also make consent decisions: in controlled settings they frequently accept cookie banners on the user’s behalf [48], and accepting consent increases third-party cookies [47] and propagates consent signals to many vendors [40]. We measure what instrumentation records during agents’ visits to live websites and quantify how their consent clicks change the scripts that load.

Study Design

We evaluate eight main crawler configurations: four non-agentic and four agentic. The latter correspond to two agentic crawlers, each run with and without a residential proxy. We also conduct same-URL crawls, test targeted configuration changes, and record agent interactions to investigate differences between crawlers. Browsing data from human participants provide a reference for evaluating their observations. Figure 1 summarizes the study design.

Sites and Crawler Configurations

Sites. We constructed a 2,000-site corpus by selecting 1,000 domains from the Tranco top 10K [35] and 1,000 from the CrUX global top list [18], both obtained in July 2026. We exclude adult, gambling, advertising, malicious, and non-user-facing domains (Appendix § 8). For smaller experiments, we selected a 450-site informative subset for measurement feasibility where SuperScraper’s crawl successfully retrieved at least one same-site HTML page and recorded instrumented JavaScript calls from scripts associated with at least one known tracker organization. Using a fixed seed, we sampled with approximately equal quotas across strata based on list rank, bot-protection indicators such as detected Web Application Firewall (WAF) vendors and requests to known bot-management or CAPTCHA services, and observed tracker counts, redistributing shortfalls from smaller strata. This subset is used for the SuperScraper variants, interaction logging, and participant assignments.

Non-agentic crawlers. We evaluate four non-agentic crawler configurations: Chromium, Firefox, Chromium-Stealth (Chromium-S), and Firefox-Stealth (Firefox-S). All four explore same-site links using a first-in, first-out queue, starting each site in a fresh, headed browser context. We use Playwright [41] to control Chromium and Firefox. For Chromium-S, we use Patchright [32], a modified version of Playwright designed to reduce automation artifacts, together with a BrowserForge [14] desktop profile and browser-property overrides. For Firefox-S, we use a generated profile and the same overrides, retaining Playwright after encountering compatibility issues with Firefox-specific alternatives such as Camoufox [9]. We disable Firefox’s built-in tracking protections in both Firefox configurations to improve comparability with Chromium.

We selected Patchright as a representative, reproducible stealth framework based on its public availability, adoption, and active maintenance [43] at the time of data collection. We prioritized maintainability over maximizing evasion, as detection vendors can identify and target stealth frameworks’ fingerprints [21, 53]. Additional execution details and browser preferences appear in Appendix § 7.

Agentic crawlers. We evaluate two agentic crawlers: browser-use, built on browser-use 0.13.4 [7], and our screenshot-driven research crawler SuperScraper (designed strictly as a browsing agent for tracking measurement rather than content extraction). Both use Qwen3-VL-8B [46], which we instruct to visit ten distinct pages per site, favor internal links, and dismiss obstructing overlays. The browser-use agent receives DOM-derived page text and indexed interactive elements, can request screenshots, and controls Chromium through direct Chrome DevTools Protocol (CDP) commands rather than Playwright. In contrast, SuperScraper perceives pages through screenshots annotated by an OmniParser-based pipeline [38] and interacts through operating-system mouse and keyboard input with randomized timing; no browser automation framework controls its navigation (Appendix § 7). We run both agents with and without Webshare residential proxies [54] to isolate the effect of network origin on tracking visibility and website interventions.

Data Collection

Main crawls. We run each of the eight configurations on the 2,000-site corpus, targeting up to ten distinct pages per site. Each crawler explores sites independently, so the resulting crawls may visit different URLs.

Same-URL crawls. Because crawlers independently select which links to follow, they may explore different URLs within the same website. Therefore, to compare agentic and non-agentic crawlers on the same URLs, we adapt the URL-revisiting approach of Annamalai et al. [4], conducting two non-agentic Chromium crawls, each revisiting the URLs recorded by one of the agents. Each URL is loaded as a separate one-page task in a fresh browser context. Landing pages are visited first, with internal URLs revisited a median of 8–12 hours later. These crawls hold the requested URLs fixed, but not prior navigation, consent state, dwell time, or interactions (§ 3.2.1).

Targeted variants. To isolate crawler configuration choices, we evaluate three Chromium variants on the 2,000-site corpus: replacing Playwright with Patchright, using a BrowserForge-generated profile with Playwright, and replacing queued-URL loads with visible-link clicks via GhostCursor [45]. On the 450-site subset, we evaluate two SuperScraper variants: one without behavior simulation and another using Playwright to launch the browser while retaining operating-system input for navigation. We compare the Chromium variants against their main crawl. Because SuperScraper’s main crawl was conducted substantially earlier, we use a separate unmodified SuperScraper crawl on the same sites as the reference for its variants (Table 1).

CollectionDatesSitesPagesPg/siteMiss. (%)Obs. (s)
Main crawls
Chromium9/23–261,9879,8054.9314
Chromium-S9/27–302,0009,8444.949
Firefox10/12–161,9888,7254.4415
Firefox-S10/12–151,9818,8704.5316
browser-use7/19–311,97112,5596.4519
browser-use (proxy)8/21–9/41,98213,0616.6417
SuperScraper7/15–251,9885,8372.92926
SuperScraper (proxy)8/21–281,9966,0903.13224
Same-URL crawls
Same-URL (browser-use)8/13–141,94413,6547.038
Same-URL (SuperScraper)8/61,9906,8593.446
Targeted variants and reference
Chromium → Patchright10/5–71,99612,4776.336
Chromium + BrowserForge10/5–101,9999,7044.9213
Chromium + click navigation10/2–51,9975,4972.8419
SuperScraper (reference)10/18–194501,8044.02524
SuperScraper − behavior sim.10/5–64501,9134.32413
SuperScraper + Playwright launch10/5–64501,9184.3124
Participant sessions
Participants9/11–10/184154,35310.587

Table 1. Collection periods (2026) and coverage. Pages: first attempt per requested URL; Miss. (%): pages without response capture; Obs. (s): median time from page start to the last recorded event.

Interaction-logging crawl. To examine agent interactions and subsequent third-party script loading, we ran an additional browser-use crawl on the 450-site subset, using the same model and browsing task as the main crawl. We obtained usable interaction logs from 445 sites. We classify consent-related clicks as accept, reject, settings, or weak accept (e.g., “OK”). A consent click is considered confirmed if a consent-state change occurs in the same tab within 15 s, indicated by a TCF (Transparency and Consent Framework) useractioncomplete event, Google Consent Mode update, or first write of a known consent cookie. We count newly observed third-party script domains within 10 s of each click, comparing consent-related and ordinary clicks. Further details on logging and validation appear in Appendix § 7.

Participants. With IRB approval and informed consent, we recruited 26 adults (18 through CloudResearch Connect [13], 8 local university students), compensating each with $10. We assigned each participant 15 sites from the 450-site subset and asked them to browse naturally, visiting at least ten distinct subpages per site. Participants used Chrome with browsing data cleared and other extensions disabled, alongside a custom-built measurement extension that collected the same logging data as the crawlers (Appendix § 7). The resulting logs contain 4,353 pages across 415 sites, including 368 assigned sites (and 6 additional unassigned sites) with recorded landing-page visits. Not all participants completed every assigned site or reached the requested ten subpages per site. We pool participants’ observations as a descriptive human reference, rather than an estimate of population-level browsing behavior or individual browsing patterns.

Collection period. Data collection spans July–October 2026. Table 1 summarizes the collection periods and coverage. Because the crawls were conducted at different times, we examine repeated collections to assess temporal variation (Table 7).

RepeatDaysLandingInternalTrackers
Same-URL crawls (internal pages)7——0.95
Chromium → Patchright11−0.3−6.40.99
Chromium + BrowserForge14−0.2+0.30.98
Chromium + click navigation9+0.1+0.51.00
SuperScraper + behavior sim.13+0.2+0.00.97
SuperScraper + Playwright launch13−1.6+1.30.93
SuperScraper, repeated91+3.8+0.01.01

Table 7. Drift between repeated collections: change in page denial (pp, variant minus reference) and tracker ratio on matched pages. Internal-page changes for Patchright and the Playwright launch are configuration effects. Bold: interval excludes zero (one for ratios).

Instrumentation and Data Preparation

We use OpenWPM-derived JavaScript instrumentation [19] across all crawls and participant browsing sessions. The instrumentation is byte-identical across Chromium-based configurations and adapted for Firefox with the same hooks. It records calls to fingerprinting-relevant APIs, their arguments (except for getContext), and calling-script URLs, but not return values. We also record visit starts, intervention signals, bot-management headers, and eligible HTTP 2xx responses (details in Appendix § 8). SuperScraper and participants use a Chrome extension with debugging-interface response capture, browser-use uses CDP, and the non-agentic crawlers use their respective automation frameworks.

We match pages by site and normalized requested URL, retaining the first eligible attempt (Appendix § 8). However, we noticed that response capture is missing on approximately 85% of SuperScraper crawled landing pages (accounting for nearly all of its 29–32% overall missing rate in Table 1), and even nominally captured pages contained little usable network data. We therefore exclude SuperScraper’s landing pages from network-derived comparisons, while retaining separately recorded intervention signals and JavaScript calls. Appendix § 8 provides further details on data preparation, capture failures, and sensitivity checks.

Metrics

Website interventions. We measure website interventions using page denials, hard blocks, and challenge markers. We mark a page as denied when we record a first-party HTTP 403 (Forbidden) or 429 (Too Many Requests) intervention event whose URL matches the requested page. The denial rate is the fraction of attempted pages classified as denied. This metric captures recorded 403/429 denials; blocking conveyed through other status codes or page content is not counted as a page denial. A hard block is any recorded first-party 403 or 429 response during a visit, including responses to requests other than the requested page. A challenge marker is a page element associated with a known challenge or bot-management provider (e.g., a CAPTCHA), classified as visible when displayed to the visitor and passive when hidden (e.g., invisible bot checks). We define escalation as the mean across sites of each site’s denial rate on pages 2–KK minus its landing-page rate, considering sites with at least KK recorded pages (KK=6 unless noted).

Tracking visibility. We count tracker organizations per page as the distinct owners of third-party domains appearing in captured responses and classified by DuckDuckGo Tracker Radar [17] as advertising, analytics, session-replay, or social tracking. A matched pair of visits is consent-managed if a consent-management platform (e.g., OneTrust or Didomi) appears in either visit. For fingerprinting, our strict heuristic flags a page when an attributable script performs at least one of five behaviors: canvas text drawing followed by extraction, offline audio rendering followed by sample readback, WebRTC candidate gathering without media, extensive font enumeration, or querying the unmasked WebGL vendor or renderer. Because WebGL activity depends on browser launch settings (§ 3.2), we also report a WebGL-free heuristic that excludes WebGL vendor and renderer queries. We further use a legacy fingerprinting heuristic based on Engelhardt and Narayanan [19] as a sensitivity check (Appendix § 8). Because fingerprinting also serves defensive purposes such as fraud and bot detection [8], and our heuristics cannot establish a script’s intent, we treat fingerprinting signals as observed behavior rather than as evidence of privacy-invasive tracking.

Human agreement. On pages visited by both a crawler and participants, we compare tracker organizations within each page and pool these observations across pages. Recall is the fraction of participant observations also recorded by the crawler; precision is the fraction of crawler observations also recorded by participants. We additionally report the shares of their pooled union observed only by participants, only by the crawler, or by both.

Comparisons and Analysis Methods

We compare crawls and participant observations on matched pages, defined as pages with the same normalized requested URL attempted by both. Network-based comparisons additionally require response capture in both visits. We report paired mean differences with equal weight per page, in percentage points (pp) for rates, and ratios of summed tracker-organization counts. We compute 95% confidence intervals using 2,000 whole-site bootstrap resamples to account for dependence among pages from the same site. For crawler contrasts, we use site-level sign-flip tests with 10,000 random sign flips per test and apply Holm’s step-down correction [24] to adjust pp-values within two groups of tests spanning crawler contrasts and outcome metrics (Table 5). As sensitivity checks, we also examine unblocked pairs (both visits capture the page’s HTML, with neither recording a hard block or visible challenge) and landing-page pairs (neither visit has preceding navigation within the site). Additional statistical details and assumptions appear in Appendix § 8.

Contrast (A − B)PagesPage deniedHard blockPagesTrackers/pageStrictWebGL-free
Stealth vs. base configuration
Chromium-S − Chromium7,109−4.2 [−5.5, −3.0]−5.1 [−6.5, −3.8]6,757−0.27 [−0.34, −0.19]−10.9 [−12.6, −9.4]−4.8 [−6.0, −3.7]
Firefox-S − Firefox7,247+0.4 [0.0, 0.9]+0.3 [−0.2, 0.8]6,946+0.03 [−0.02, 0.08]−0.5 [−1.1, 0.2]−0.4 [−1.0, 0.1]
Browser engine
Firefox − Chromium7,121+0.4 [−0.2, 0.9]+0.1 [−0.5, 0.8]6,821−0.29 [−0.34, −0.23]−8.7 [−10.2, −7.3]−2.2 [−3.0, −1.3]
Agentic vs. non-agentic crawlers
browser-use − Chromium3,464−3.5 [−4.7, −2.4]−4.8 [−6.1, −3.5]3,141+2.05 [1.78, 2.33]−7.3 [−9.3, −5.3]+3.1 [1.7, 4.6]
SuperScraper − Chromium2,565−1.9 [−2.8, −1.0]−4.0 [−5.2, −2.7]572+2.42 [1.83, 3.04]−8.1 [−10.1, −6.0]+3.2 [1.6, 4.8]
browser-use − Chromium-S3,376−0.9 [−1.7, −0.3]−0.4 [−1.2, 0.3]3,008+2.87 [2.58, 3.18]+5.7 [4.1, 7.4]+9.6 [8.3, 11.0]
SuperScraper − Chromium-S2,526−1.1 [−2.0, −0.3]−0.7 [−1.7, 0.3]525+2.16 [1.62, 2.75]+6.1 [4.4, 8.0]+9.8 [8.2, 11.4]
The two agentic crawlers
browser-use − SuperScraper2,8590.0 [−0.6, 0.5]−0.2 [−1.1, 0.5]883−0.09 [−0.38, 0.19]+0.3 [−1.2, 1.9]+0.3 [−1.2, 1.9]
Agentic crawlers vs. their same-URL crawls
browser-use − same-URL crawl12,089−0.6 [−1.1, 0.0]−2.8 [−3.8, −1.8]11,435+1.99 [1.72, 2.26]−3.0 [−4.8, −1.2]+1.0 [−0.6, 2.6]
SuperScraper − same-URL crawl5,815−2.6 [−3.5, −1.8]−4.0 [−5.3, −2.6]3,759+2.95 [2.53, 3.36]−1.7 [−3.8, 0.4]+2.8 [1.0, 4.6]
Residential proxy vs. none
browser-use: proxy − none5,108+0.7 [0.3, 1.2]+1.0 [0.5, 1.5]4,734+0.01 [−0.17, 0.18]−0.6 [−1.6, 0.3]−0.6 [−1.6, 0.3]
SuperScraper: proxy − none2,631+2.2 [1.4, 2.9]+2.4 [1.5, 3.3]635−0.41 [−0.75, −0.07]+1.0 [−0.3, 2.2]+1.0 [−0.3, 2.2]

Table 5. Table 5: All planned contrasts on matched pages (A − B; pp unless noted; 95% intervals); values compare within rows only. Trackers: organizations per page where both visits captured responses (SuperScraper: internal pages). Bold: Holm-adjusted p<0.05p < 0.05 within its family (planned: hard block, trackers, strict; added: page denial, WebGL-free). Strict fingerprinting reflects the effects in § 3.2.2.

Results

We present findings for each research question. Table 5 in Appendix § 9 reports crawler contrasts and Holm-adjusted tests.

Website Interventions

Page denials. On the 1,902 landing pages attempted by all eight configurations, denial rates were similar for non-agentic (6.0–8.1%) and agentic crawlers (5.3–8.1%); 86% of these sites recorded no denial in any configuration (Table 6). Differences emerged on subsequent pages (Figure 2). Among sites with at least six pages per crawler, Chromium’s denial rate increased from 0.2% on landing pages to 10.4% on pages 2–6, an escalation of +10.1 pp [8.3, 12.1]. Escalation was smaller for Firefox (+4.9 pp), Firefox-S (+4.4 pp), and Chromium-S (+1.2 pp), and absent for browser-use (−0.4 pp) and SuperScraper (−1.7 pp). Five- and eight-page analyses and hard-block measurements showed similar patterns (Table 6). Thus, similar denial rates on landing pages do not imply similar access during continued browsing.

Page denial by position in the visit, on sites with at least six recorded pages

Figure 2. Page denial by position in the visit, on sites with at least six recorded pages (bands: 95% CIs).

ConfigurationLanding (%)Escalation, denialHard
Den.HardVis.Pass.K=5686n
Chromium6.011.00.38.6+8.9+10.1+12.6+4.6879
Chromium-S6.58.30.52.2+1.2+1.2+1.6+0.3868
Firefox7.110.90.16.5+5.2+4.9+6.3+1.8701
Firefox-S8.111.80.13.6+5.2+4.4+5.9+2.5697
browser-use5.37.50.517.5−0.4−0.4−0.2−0.6842
browser-use (proxy)6.59.40.514.2−0.2−0.3−0.9−1.4844
SuperScraper5.57.90.518.1−2.1−1.7−1.7−1.5344
SuperScraper (proxy)8.111.00.420.3−2.7−2.6−2.9−3.4364

Table 6. Interventions by configuration. Landing: signals on the 1,902 shared landing pages (%; vis./pass.: visible/passive challenges). Escalation: later-page minus landing-page rate (pp) on sites with ≥K pages; n: sites at K=6. Bold: interval excludes zero.

Targeted variants. Configuration changes help explain this pattern (Table 2). Although Chromium and Chromium-S use the same link-selection strategy, Chromium-S encountered fewer denials on 5,122 matched internal pages (2.4% versus 8.5%), despite shorter median intervals between requests (9.9 versus 14.2 s). Replacing Playwright with Patchright alone reduced Chromium’s escalation from +10.1 to +1.8 pp and matched internal-page denial by 6.4 pp. BrowserForge alone (+9.3 pp) and click navigation with simulated mouse movement (+9.8 pp) retained escalation similar to Chromium; Firefox-S likewise retained Playwright and showed escalation similar to Firefox.

ConfigurationInterventions (pp)VisibilityInstrument (%)
EscalationΔ denialTrackersUncertainWebGL
Chromium (reference)+10.1ref.ref.8.322.4
→ Patchright+1.8−6.40.993.714.3
+ BrowserForge+9.3+0.30.989.324.2
+ click navigation+9.8+0.51.009.118.5
SuperScraper (reference)−5.0ref.ref.9.51.1
− behavior simulation−0.8+0.00.9710.00.9
+ Playwright launch−0.4+1.30.9310.345.4

Table 2. Targeted configuration variants and their effects on website interventions, tracking visibility, and instrumentation. SuperScraper variants use a separate unmodified run on the same 450 sites as their reference.

The two agents had similar denial rates on matched pages (difference: 0.0 pp [−0.6, 0.5]). Launching SuperScraper’s browser through Playwright while retaining operating-system navigation did not create escalation (−0.4 pp), although it raised matched internal-page denial by 1.3 pp [0.0, 2.9]. Together, these variants suggest that navigation through the automation framework contributes more to website detection than how the browser is launched or whether human-like interaction is simulated.

Cloudflare-served pages and passive challenges. Denials were particularly concentrated on Cloudflare-served internal pages: Chromium was denied on 21.6% of these pages, versus 4.3% elsewhere. Among Cloudflare-served pages, denial was lower for Chromium-S (1.7%) and the agents (0.5–1.0%) than for Firefox and Firefox-S (14.6–15.1%). Yet passive challenge markers appeared on 17–24% of agents’ matched pages versus 2–11% of non-agentic pages, despite the agents’ lower denial rates. Participants also encountered more passive markers (9–12%) than Chromium (6.1%).

Takeaway. How browsing is automated matters more for avoiding website interventions than whether an agent directs navigation. Passive challenge markers are not reliable indicators for actual access denial.

Tracking Visibility

We compare tracker organizations identified in captured third-party responses and scripts exhibiting fingerprinting behavior.

Third-Party Tracking

Tracker counts. On 11,435 matched pages, browser-use recorded 6.00 tracker organizations per page versus 4.01 for its non-agentic same-URL crawl (ratio 1.50 [1.42, 1.58]). On 3,759 matched internal pages, SuperScraper recorded 7.24 versus 4.29 (ratio 1.69 [1.57, 1.81]); its landing pages are excluded from network comparisons (§ 2.3). Comparisons of the main crawls on shared URLs; where both crawlers maintain stateful multi-page sessions rather than loading internal URLs in fresh contexts; also favored the agents (ratios 1.7–2.3 across all 16 pairs of agentic and non-agentic configurations). The two agents, despite different interfaces and collection mechanisms, recorded nearly identical counts on their shared pages (ratio 0.99 [0.95, 1.03]). On unblocked pages, the agents recorded more organizations across advertising, analytics, session replay, and social tracking; the difference also remained when counting only organizations whose scripts made instrumented JavaScript calls.

Beyond denials. On matched pages where neither crawler was blocked, the agents still recorded 1.47 (browser-use) and 1.62 (SuperScraper) times as many tracker organizations as their same-URL crawls. Conversely, switching Chromium to Patchright reduced matched internal-page denial by 6.4 pp without increasing tracker counts (ratio 0.99). Avoiding denials therefore does not, by itself, explain the agents’ additional tracking.

Consent-managed pages. On matched pages where either crawler detected a consent-management platform, the agents recorded 1.80–2.16 times as many tracker organizations as their same-URL crawls, versus 1.29–1.42 times on other pages (Figure 3b). Consent-managed pages comprised 28% of matched pages but accounted for 62–65% of the additional tracker organizations.

Figure 3. (a) Each paired contrast (A vs. B on matched pages): change in page denial against the tracker-organization ratio. (b) Agentic contrasts on pages with and without a consent-management platform. Bars: 95% CIs.

Consent interactions. To examine what happens after consent actions, we instrumented a separate browser-use crawl on 445 sites. It made 273 consent-related clicks across 140 sites: 201 explicit accepts, 47 weak accepts, 23 settings openings, one rejection, and one unclassified action. On the 69 sites where confirmed consent clicks could be compared with ordinary page-loading clicks, the former were followed by a mean of 9.2 new third-party script domains within 10 s, versus 1.0 (site-level means; Wilcoxon signed-rank p<0.001p < 0.001).

The largest increases followed a reload or navigation after consent (Table 3), and newly observed domains were predominantly advertising-related (Table 9).

ClickPage afterwardsClicksSitesMeanAny (%)
Confirmed consentreloaded323112.194
Confirmed consentnavigated222111.182
Confirmed consentunchanged31251.735
Ordinarynavigated5,0883990.519
Ordinaryreloaded8032060.413
Ordinaryunchanged4,4703750.12

Table 3. Interaction-logging crawl (browser-use, 445 sites): new third-party script domains within 10 s of a click, grouped by the page’s subsequent behavior. Clicks followed by another click within 10 s are excluded. Any: clicks followed by at least one new domain (%).

Functionbrowser-useSuperScraper
Per page%Per page%
Supply-side platforms and exchanges+0.6030+0.9030
Platform ads, pixels and analytics+0.4322+0.5920
Ad verification and measurement+0.2110+0.4214
Demand-side and retargeting+0.168+0.196
Content recommendation+0.074+0.155
Identity and audience data−0.02−1+0.021
All other organizations+0.5527+0.6923

Table 9. The agents’ additional trackers by ad-tech function: organizations recorded only by the agent minus only by its same-URL crawl, per matched page, and share of the total (%). Functions assigned manually for the 45 largest contributors.

Visit duration and interaction. On matched pages, the agents had longer observation times than their same-URL crawls (median 19.6 and 23.7 s versus 7–8 s) and made three to eight times as many interaction-related calls. However, a Chromium click-navigation variant also had longer observation times (20.9 versus 13.4 s) and made about twice as many interaction-related calls, without recording more tracker organizations (ratio 1.00 [0.99, 1.02]).

Takeaway. Agents’ greater tracking visibility is not simply a consequence of avoiding denials or spending more time on pages. The tracking recorded depends on which interactions occur, with consent-related actions providing one concrete explanation for the additional activity.

Browser Fingerprinting. Differences across vectors

Unlike tracker counts, fingerprinting did not consistently increase with agentic crawling. On matched pages, offline audio fingerprinting was more frequent among agents (+2.5–4.4 pp), but WebGL renderer queries occurred on fewer than 0.5% of agents’ pages versus 10–22% for non-agentic crawlers (Figure 4a). Excluding WebGL, the agents recorded only 1.0–3.2 pp more fingerprinting than Chromium and their same-URL crawls, despite recording 1.5–2.2 times as many tracker organizations.

Figure 4. (a) Selected fingerprinting vectors, weak indicators, and instrumentation diagnostics, computed on each configuration’s own pages. (b) Strict (filled) and WebGL-free (hollow) fingerprinting prevalence on 1,902 shared landing pages; 95% CIs.

Sources of the differences. The agents’ same-URL crawls recorded WebGL renderer queries on 16–17% of those URLs, indicating that URL selection alone does not explain the agents’ near-absence of WebGL queries. In two same-day SuperScraper variants on 875 matched pages, WebGL queries occurred on 1.3% with normal browser launch and no behavior simulation, versus 44.8% with Playwright launch, while WebGL-free fingerprinting remained near 21%. The same Google advertising hosts ran scripts in both variants but queried the renderer only under Playwright (Appendix § 8) – consistent with defensive scripts [8, 29] probing WebGL only when browser launch flags or CDP attachment expose automation signals [32, 33]. Browser launch is therefore implicated, although the variants also differed in behavior simulation.

Instrumentation further complicates the comparison. The agents recorded 9.6–9.8 pp more WebGL-free fingerprinting than Chromium-S, versus approximately 3 pp more than Chromium (Table 5). Yet 83% of Chromium-S’s pages contained uncertain JavaScript call sequences, versus 8% for Chromium. Neither Patchright nor BrowserForge alone reproduced this problem (Table 2), suggesting interference from their combination when BrowserForge’s prototype-spoofing wrappers and OpenWPM’s hooks run together under Patchright’s modified execution context, stripping call arguments.

Consequences for comparisons. Under the strict heuristic, Chromium recorded fingerprinting on 7–8 pp more matched pages than either agent. Excluding WebGL changed the direction: both agents recorded approximately 3 pp more than Chromium, a reversal also visible on the 1,902 shared landing pages (Figure 4b).

Takeaway. A crawler can appear to encounter less fingerprinting because of differences in browser execution or instrumentation, rather than because the websites it visits perform less fingerprinting.

Comparison with Human Browsing

Website interventions. On participant-shared internal pages, participants were denied on at most 0.4% of pages and the agents on none. The three configurations using standard Playwright (Chromium, Firefox, and Firefox-S) were denied on 3.9–7.2% of these pages, compared with 1.7% for Chromium-S. On matched landing pages, denial rates were 3.0% for participants, 3.5% for agents, and 4.3–7.8% for non-agentic crawlers. Thus, the later-page escalation observed with standard Playwright was absent from participant visits.

Tracker agreement. On pages shared with participants, the agents recovered 82% (browser-use) and 87% (SuperScraper) of participant-observed tracker organizations, versus 58–68% for non-agentic configurations (Figure 5). However, only 48–49% of the agents’ observations were also seen in participant visits, compared with 65–70% for non-agentic crawlers. The higher recall therefore came with lower precision, yielding similar pooled F1 scores across all six crawlers (0.61–0.66). Because SuperScraper’s landing pages lack network capture (§ 2.3), its comparison uses only internal pages. Restricting all six crawlers to internal pages preserves the pattern (recall 85–87% versus 57–68%; precision 49–50% versus 74–75%).

Figure 5. Tracker-organization agreement with participants. Pooled recall (solid) and precision (light) on shared pages with network capture in both visits; 95% CIs.

Consent-related differences. On consent-managed pages, the agents recorded 1.83–2.00 times as many tracker organizations as participants, whereas non-agentic crawlers generally recorded fewer (0.61–0.96 times). The agents’ excess persisted on other pages (1.61–1.62 times). These observations are consistent with the consent-related activity in § 3.2.1, but participant consent decisions were not logged, and consent state may persist across pages.

Visit duration. Participants’ median observations lasted 6.8 s versus 26.6 s for browser-use and 5.0 versus 23.6 s for SuperScraper on shared pages. Truncating each agent’s observations at the corresponding participant’s observation duration reduced the agent-to-participant tracker ratio from 1.71 [1.54, 1.90] to 0.95 [0.84, 1.06] for browser-use, and from 1.76 [1.49, 2.08] to 1.28 [1.08, 1.50] for SuperScraper. At these durations, additional trackers were no longer evident for browser-use but persisted for SuperScraper.

Fingerprinting agreement. On pages shared with participants, agents recorded canvas extraction and offline audio on 14–17% of pages, versus 5–7% for participants and 6–10% for non-agentic crawlers. Conversely, WebGL renderer queries appeared on at most 0.5% of agent pages versus 17–20% for participants. The agents’ aggregate strict fingerprinting prevalence (21%) resembled participants’ (20–23%), despite these vector-level differences. Only Chromium-S and the two Firefox configurations recorded WebGL-free prevalence close to participants’ (8–9% for both), although Chromium-S’s instrumentation limitations complicate that comparison § 3.2.2.

Takeaway. The agents recovered more participant-observed trackers but also recorded many additional ones, while similar aggregate fingerprinting rates concealed differences across vectors. Thus, greater tracking visibility or similar aggregate prevalence does not necessarily imply closer agreement with human browsing.

Discussion

Different outcomes require different controls. Our findings show why crawler quality cannot be reduced to a single measure of human likeness or detection avoidance. Automation frameworks affect whether a crawler can continue through a site; browsing duration and interactions influence the tracking it records; and browser execution and instrumentation affect which fingerprinting behaviors can be observed. These effects need not move together. Patchright reduced denials without increasing tracker counts, whereas the agents recovered more participant-observed trackers but also recorded substantially more activity absent from participant visits. A crawler that performs well on one measurement objective therefore need not be suitable for another.

Choosing and validating crawlers. Crawler choice should follow the question being measured (Table 4). For studies of tracking during ordinary browsing, the visit protocol should reflect the target users’ navigation, observation duration, and consent choices. Non-agentic crawlers may make those choices easier to standardize, but are not inherently representative of human visits. For studies seeking tracking activity that interaction can activate, agents can provide broader exploratory coverage, provided that their actions are logged and their observations are not presented as typical user exposure.

ObjectiveRecommended Crawler & Parameters
Interaction-triggered TrackingAgentic (direct protocol control). Autonomous exploration with overlay dismissal; log clicks across all frames and monitor consent-state APIs; capture network via browser protocol rather than extensions.
Typical Human Tracking ExposureNon-agentic (stealth driver) or Constrained Agent. Bound page dwell time to human medians; enforce a fixed consent policy matching target users rather than auto-accepting banners.
Multi-page Access (low blocking)Non-agentic (stealth driver) or Agentic (protocol/OS-input). Avoid unpatched automation drivers; crawl multiple internal pages per site; omit property spoofing and simulated mouse movement.
Browser FingerprintingNative-launch Browser + Unmodified Profile. Launch browser natively rather than via automation frameworks; avoid combining driver patches with fingerprint-spoofing overrides.
EssentialsMatch on normalized requested URLs; verify first-page capture completeness (notably with browser extensions); interleave runs or repeat reference crawls to bound temporal drift.

Table 4. Crawler selection and configuration parameters by measurement objective.

Fewer denials do not guarantee complete measurements: in Chromium-S, arguments needed for fingerprinting classification were often missing. Likewise, identical JavaScript hooks do not guarantee comparable results if browser launch settings change which API calls scripts make. Fingerprinting studies should therefore report vector-level measurements and test instrumentation in every browser configuration, rather than relying only on aggregate prevalence. Repeated reference crawls can help assess temporal variation, but do not remove every difference between collections conducted at different times.

Implications for the agentic Web. These findings also matter when agents browse on users’ behalf rather than as measurement instruments. In our interaction-logging crawl, the browser-use agent made 201 explicit consent-accept clicks and only one explicit rejection while following an instruction to dismiss obstructing overlays. Confirmed consent actions were followed by additional third-party script loading, and our agents recorded more tracker organizations than participants on shared pages. These observations do not establish that consent explains all of the difference or that other agents would act similarly. They do show how a seemingly minor interaction policy can affect the tracking recorded during delegated browsing. Agent frameworks should therefore make consent handling an explicit, user-controllable choice, rather than an incidental consequence of clearing obstacles to navigation. For websites, the low later-page denial rates of our agents do not establish that agentic navigation is inherently harder to detect: Patchright produced a similar reduction without using an agent, and passive challenge markers did not track denial rates consistently.

Limitations. We study one open model with one framework per agent design, and our crawls without a proxy ran from a single US vantage point, where consent banners are less common than in the EU, so consent effects may be larger there. The intervention results reflect stealth and detection tools at one point in time. SuperScraper reached fewer pages per site than the other crawlers and lacks network data on most landing pages, and only the browser-use agent’s consent actions were logged. The participant reference is small, pooled, and descriptive, with short visits and unrecorded consent decisions. Page denial is a lower bound on blocking, our heuristics cannot separate tracking from defensive fingerprinting [8], and the drift over the three-month collection is bounded in § 2.2.

Ethics

Our human-browsing study was reviewed and approved by our institution’s IRB. All 26 participants provided informed consent and received $10 for their participation. Participants used a browser extension to record Web activity for the study. The collected data included JavaScript API calls, visited URLs, and eligible HTTP responses. Participants used a separate browser profile from their personal profile and received explicit instructions describing the data collected. The extension was configured to initiate recording on domains in the 2,000-site study corpus, including eligible third-party resources loaded during those visits. We did not solicit demographic information. We will release participant observations only in aggregate, not individual browsing records. Our automated crawlers accessed publicly available pages without credentials, visiting at most 10 pages per site per configuration sequentially within a single browser session (yielding a low per-site request rate with median inter-page intervals of 9.9–26.6 s; § 3.1 and § 3.2) and never running concurrent sessions against the same domain. Following established Web tracking measurement methodology [1, 19, 31], our crawlers did not enforce robots.txt exclusions, as our objective is to measure the tracking and interventions encountered during ordinary interactive browsing rather than to index content. Agent interactions with website controls, including consent banners and a single-attempt challenge handler, occurred in isolated, stateless sessions.

Conclusion

Our findings show that agentic crawling does not uniformly improve Web-tracking measurements. Agents recorded more third-party tracking and recovered more participant-observed trackers, but also observed activity absent from human visits. Their lower denial rates could largely be reproduced with a patched automation framework, while browser configurations and instrumentation complicated fingerprinting comparisons. These findings demonstrate that website interventions, tracking visibility, and agreement with human browsing are distinct measurement objectives. We provide practical guidance for selecting and validating crawlers according to these objectives rather than assuming that any one crawler type is inherently better.

Crawler and Study Details

Non-agentic crawlers. The queue described in § 2.1 starts at the site’s root URL. For each URL, the crawler waits for document load or navigation failure, checks challenges, scrolls to the bottom, waits briefly, captures a screenshot, and queues unseen fragment-free HTTP(S) links on allowed hosts, including hosts reached through redirects. It stops after ten URLs, an empty queue, a failure limit, or the per-site time budget; a watchdog restarts unresponsive drivers. The stealth overrides cover navigator.webdriver, platform, hardware concurrency, device memory, notification permissions, and WebGL vendor/renderer strings. Firefox preferences disable Enhanced Tracking Protection and its social-tracking, cryptomining, fingerprinting, and email-tracking lists, bounce-tracking protection, query-parameter stripping, first-party isolation, network-state partitioning, and fingerprinting resistance. They allow all cookies, full referrers, DNS prefetching, and speculative connections; retain storage, service-worker, Beacon, and WebRTC APIs; and send neither Global Privacy Control nor Do Not Track. No non-agentic run used a proxy.

Browser-use agent. browser-use uses headed Chromium. Its task specifies ten distinct subpages and, beyond the shared instructions (§ 2.1), avoiding repeated actions; a progress tool lists visited URLs. The driver caps steps and stops on stalled progress, errors, or the time limit.

SuperScraper. SuperScraper runs a closed perception–decision–action loop in an ordinary Chrome profile (Figure 6). In each iteration, it reads the URL from the address bar and captures a screenshot, which an OmniParser-based pipeline [38] annotates with labeled bounding boxes through OCR, icon detection, and captioning. Qwen3-VL-8B [46], at temperature 0, receives the annotated screenshot, the element descriptions, the goal, the visited URLs, and the last ten actions, and it outputs one action in a format constrained by a GBNF grammar: a short rationale, a tool (click, double-click, move, type, enter, drag, scroll, wait, or CAPTCHA), and its arguments. Actions are executed with operating-system input, using HumanCursor [25] trajectories for pointer movement and drags and PyAutoGUI [44] for keys, wheel events, and some clicks, with randomized timing: the mouse button is held for 0.2–0.5 s, inter-key intervals last 0.05–0.2 s, waits last 1–3 s, and a scroll consists of 6–14 wheel events. After a URL change, with probability 0.5, the agent also moves to a random content position, scrolls down one to three times, and returns to the top. The CAPTCHA action makes one model call specialized for the challenge type and is attempted once per challenge. A site visit ends after ten distinct URLs, five consecutive steps without a URL change, or 60 iterations, and between sites the agent clears browsing data and restarts Chrome. Because screenshot-based bounding-box grounding occasionally clicks non-navigational elements or spends multiple steps scrolling and dismissing overlays without changing the URL, SuperScraper hits the five-step stall limit more often than DOM-indexed agents, resulting in fewer recorded pages per site (2.9–3.1 in Table 1). The extension (§ 2.3) injects at document start in every frame; the agent never sees its output.

SuperScraper’s perception–decision–action loop

Figure 6. SuperScraper’s perception–decision–action loop (steps 1–5, as labeled).

Cost of agentic crawling. Both agents ran the model locally on one NVIDIA RTX 4000 Ada Generation GPU with 20 GB VRAM, and every step of a visit waits for a model call. In the main runs, the recorded part of a site visit lasted a median of 3.6 minutes for browser-use and 2.3 minutes for SuperScraper, versus 1.1–2.3 minutes for the non-agentic configurations, and in the interaction-logging crawl, the median agent step took 12.8 s and a site required a median of 39 steps. Estimated token usage was approximately 5,000 tokens per step for SuperScraper and 12,000 for browser-use. While SuperScraper operated with a 16k-token context window, browser-use required at least 22k tokens. Actual usage varies with website content and complexity.

Interaction-logging crawl. The interaction-logging crawl (§ 2.2) retained the main run’s agent version and used fresh profiles and a 600 s site limit, ending 68% of visits before task completion. Logging reuses the agent’s CDP connection, and CDP auto-attachment extends it to cross-origin iframes, which often host consent banners. Capture-phase listeners in every page and frame record clicks and form submissions together with their target elements, and 99.6% of the recorded page clicks matched an action in the agent’s step history. A click is classified as consent-related when its target lies inside a known consent-management container or frame or carries consent-related text in one of several languages. Script requests are recorded with their CDP initiator stacks. Windows that overlap the agent’s next click are excluded (5.7% of clicks at 10 s), and the within-site comparison tests per-site means of confirmed consent clicks against those of ordinary clicks with a Wilcoxon signed-rank test; the result holds against all ordinary clicks (8.96 versus 0.42 new domains, 71 sites) and against those without a page load (8.44 versus 0.11, 63 sites) and, on the 16–21 sites whose 20 s windows did not overlap the next click, also for 20 s windows (all p≤0.003p \le0.003). After doubleclick.net, the domains most often first observed after confirmed consent were adtrafficquality.google (45% of sites, versus 8% after ordinary clicks), googlesyndication.com (35% versus 7%), facebook.net (34% versus 10%), pubmatic.com (31% versus 4%), criteo.net (25% versus 4%), adnxs.com (23% versus 4%), and the identity services id5-sync.com (18% versus 4%) and crwdcntrl.net (14% versus 2%). Fingerprinting after a click was classified per script with the strict heuristic on the calls in the 10 s window, so a script that draws before the click and extracts after it is missed in both click groups. Without manual labels, we assessed the consent classifier against two independent signals: 68% of the clicks it labeled as consent matched the agent’s stated goal, which mentioned cookies or consent, 83% of the accept clicks did so, and the share of likely consent clicks that it missed implies a recall of about 79%. Confirmed clicks are a conservative subset, since unconfirmed consent clicks that were followed by a reload still brought a mean of 4.0 new domains.

Participants. Participants were asked to browse naturally, visit at least ten distinct subpages per assigned site, and spend a reasonable time on each. They had one week, could take breaks, and could not use VPNs, proxies, or incognito mode, or clear browsing data during the study. Logs also include sites reached beyond their assignments (§ 2.2).

Data Preparation and Metrics

Records and matching. Captured 2xx bodies exclude media, fonts, octet streams, PDFs, archives, and responses with declared length above 256 KiB. Intervention logs include 403, 404, 429, and 5xx statuses, connection failures, and challenge/provider DOM markers with visibility flags. We recompute first-party status from URLs, assign visits to site tasks using session order and root-URL visits, discard other-site documents (mostly frames), and retain the first remaining attempt. Normalization lowercases scheme/host, removes fragments/default ports, merges HTTP/HTTPS, leading www., and trailing slashes, but retains query strings. Matching uses requested URLs, not redirect destinations.

Corpus filtering (§ 2.1) uses StevenBlack hosts lists [49], Cloudflare family-filtering DNS [12], and domain-name heuristics. The 25 public-suffix entries (e.g., github.io, blogspot.com, myshopify.com) cover their subdomains: 465/74,791 pages (0.6%). Excluding them changes headline tracker ratios by less than 0.01 and escalation by less than 0.1 pp.

SuperScraper’s response capture. Response capture is missing on 1,684 of SuperScraper’s 1,988 landing pages (84.7%), versus 0.9% of its internal pages, and on 95% of the landing pages of its proxy run; even the 304 landing pages with nominal capture record little activity (median observation span 0 s). JavaScript activity was nevertheless recorded on 93% of the affected pages, and 6.4% of them carried page-denial signals, comparable to the 5–8% of the other crawlers. Launching the same extension-equipped browser through Playwright reduced missing landing-page capture to 2%, whereas the normally launched variant without simulated interaction missed 97%. The participants, who used the same extension in an already running browser, lost capture on 24% of their 374 landing pages (368 assigned plus 6 unassigned sites) and on 7% of their internal pages, so the extension is least reliable on the first page that a fresh browser or tab loads, and SuperScraper, which restarts Chrome before every site, exposes every landing page to this weakness: the extension must start and attach to the debugging interface before initial navigation. Under the exclusion rule in § 2.3, including the 304 landing pages with nominal capture would lower its tracker ratio to its same-URL crawl from 1.69 to 1.64 and its tracker difference to Chromium from +2.42 to +0.84 organizations per page. We did not substitute the Playwright-launched variant for the main run, because the launch path itself changes denial and WebGL activity (§ 3.1 and § 3.2.2).

Fingerprinting diagnostics. The raw logs contain WebGL calls in every arm, including 57 SuperScraper and 93 browser-use visits, so the collector does not suppress them, and SuperScraper’s normally launched and Playwright-launched variants ran byte-identical instrumentation in the same extension. On the 875 pages (430 sites) that both same-day variants recorded with JavaScript activity, the Playwright launch raised WebGL renderer queries from 1.3% to 44.8% and strict fingerprinting from 21.0% to 46.4%. On the 236 of these pages where the Playwright-launched variant recorded fingerprinting-like API calls by scripts from pagead2.googlesyndication.com or ep1.adtrafficquality.google (which commonly perform anti-fraud and automated-traffic checks), the normally launched variant recorded JavaScript from the same hosts on 216 pages and canvas calls on 199 pages, but no WebGL call. In the stealth bundle, Patchright alone left 3.7% of pages with uncertain sequences and the BrowserForge profile alone 9.3% (Chromium: 8.3%), and Firefox-S, which applies the overrides through Playwright, left 4.4% (Firefox: 4.8%); device-metadata reads were unchanged by Patchright alone (13.9% versus 14.5% of pages) but fell to 4.3% in Chromium-S, whereas WebGL renderer queries already fell with Patchright alone (14.3% versus 22.4%). Firefox (§ 2.3) records less strict fingerprinting than Chromium (−8.7 pp [−10.2, −7.3], or −2.2 pp without WebGL), and it never triggers the device-metadata vector, which relies on APIs that Firefox does not implement.

Trackers and consent platforms. Tracker Radar [17] categories map to advertising (Advertising, Ad Motivated Tracking, Action Pixels), analytics (Analytics, Audience Measurement, Third-Party Analytics Marketing, Tag Manager), session replay, and social (Social Network, Social – Share, Social – Comment); a domain without a listed owner counts as its own organization. We treat OneTrust, Usercentrics, SFBX, CookieYes, Civic Computing, IAB Europe, Sirdata, ConsentManager, Digital Data Solutions, Cookie Information, Ringier, Sourcepoint, TrustArc, Didomi, Cybot, Osano, Termly, and iubenda as consent-management platforms.

Fingerprinting. Calls are grouped per visit by full script URL, so calls without a usable script URL cannot form sequences. The canvas vector requires text drawing followed within 500 ms by an extraction other than JPEG or WebP, with getImageData covering at least 16 × 16 pixels; the font vector requires more than 20 font-family settings and more than 20 text measurements, or more than 20 families passed to FontFaceSet.check; and the WebGL vector requires getParameter(37445) or getParameter(37446). The weak vectors are GPU-surface queries, a request for the debug-renderer extension without the unmasked query, local-font access, and device metadata (at least two of client hints, media-device enumeration, voices, storage estimate, Intl options, and hardware concurrency

with device memory). A sequence is uncertain when an extraction lacks a recorded draw or timestamp, or when a getImageData or getParameter call lacks its arguments. The legacy definition implements Englehardt and Narayanan’s canvas, canvas-font, WebRTC, and audio rules [19].

Inference. Whole-site bootstrap samples retain all matched pages. Sign-flip tests swap arms consistently within sites, assuming site-level exchangeability rather than guaranteed randomization. Weighting sites equally instead of page changes none of the headline contrast signs (§ 2.5); escalation already uses equal site weights (§ 2.4).

Additional Results

Tables 5 to 9 give planned contrasts, intervention-depth checks, collection drift, dwell checks, and additional trackers’ ad-tech functions, respectively.

Collection drift. Same-URL crawls one week apart have a tracker ratio of 0.95 on shared internal pages; Chromium variants 9–14 days after their reference yield 0.98–1.00, and SuperScraper variants yield 0.93–0.97 relative to its unmodified reference 13 days later. Across three months, repeated SuperScraper on the 450-site subset yields 1.01 [0.96, 1.07] on matched pages with unchanged internal denial. Landing denial changes by at most 1.6 pp across two weeks but rises 3.8 pp [1.8, 5.8] across three months (Table 7). These checks support the comparison choices in § 2.5.

Populationbrowser-useSuperScraper
RatioPagesRatioPages
Both active at 2 s, first 2 s2.23 [2.09, 2.38]9,9992.65 [2.45, 2.85]3,266
Both active at 5 s, first 5 s2.43 [2.26, 2.62]8,8012.55 [2.39, 2.76]2,397
Both active at 10 s, first 10 s1.64 [1.53, 1.76]3,9221.89 [1.76, 2.04]1,303
Both active at 20 s, first 20 s1.04 [0.98, 1.11]6401.43 [1.34, 1.53]197
Agent span ≤\le same-URL crawl span1.04 [0.95, 1.14]1,9321.10 [1.03, 1.17]481
Internal pages only1.48 [1.40, 1.58]9,8041.69 [1.57, 1.81]3,759

Table 8. Dwell robustness: tracker-organization ratio, agent over its same-URL crawl (95% intervals). *Both active at TT: pairs still recording at TT, counting organizations first seen by TT.

References

  1. [1]Gunes Acar, Christian Eubank, Steven Englehardt, Marc Juarez, Arvind Narayanan, and Claudia Diaz. 2014. The Web Never Forgets: Persistent Tracking Mechanisms in the Wild. In ACM CCS. 674–689.
  2. [2]Syed Suleman Ahmad, Muhammad Daniyal Dar, Muhammad Fareed Zaffar, Narseeo Vallina-Rodriguez, and Rishab Nithyanand. 2020. Aphonias or Epiphanies? How Crawlers Impact Our Understanding of the Web. In WWW. 271–280.
  3. [3]Babak Amin Azad, Oleksii Starov, Pierre Laperdrix, and Nick Nikiforakis. 2020. Web Runner 2049: Evaluating Third-Party Anti-bot Services. In DIMVA. 135–159.DOI
  4. [4]Meenatchi Sundaram Muthu Selva Annamalai, Emiliano De Cristofaro, and Igor Bilogrević. 2025. Beyond the Crawl: Unmasking Browser Fingerprinting in Real User Interactions. In WWW. 3896–3907.
  5. [5]Waqar Aqeel, Balakrishnan Chandrasekaran, Anja Feldmann, and Bruce M. Maggs. 2020. On Landing and Internal Web Pages: The Strange Case of Jekyll and Hyde in Web Performance Measurement. In ACM IMC. 680–695.
  6. [6]Pouneh Nikkhah Bahrami, Umar Iqbal, and Zubair Shafiq. 2022. FP-Radar: Longitudinal Measurement and Early Detection of Browser Fingerprinting. Proceedings on Privacy Enhancing Technologies 2022, 2 (2022), 557–577.
  7. [7]Browser Use. 2026. browser-use: Make Websites Accessible for AI Agents. https://github.com/browser-use/browser-use.
  8. [8]Elie Burstsein, Artem Malyshev, Tadek Pietraszek, and Kurt Thomas. 2016. Picasso: Lightweight Device Class Fingerprinting for Web Clients. In ACM SPSM. 93–102.
  9. [9]Camoufox. 2026. Camoufox. https://camoufox.com/.
  10. [10]Darion Cassel, Su-Chin Lin, Alessio Bargigli, William Wang, Andrew Zhang, Lujo Bauer, Hsu-Chun Hsiao, Limin Jia, and Timothy Libert. 2022. OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers. Proceedings on Privacy Enhancing Technologies 2022, 1 (2022), 227–252.DOI
  11. [11]Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yuantao Li, Jianing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In ACL. 9313–9332.
  12. [12]Cloudflare. 2020. Introducing 1.1.1.1 for Families. https://blog.cloudflare.com/introducing-1-1-1-1-for-families.
  13. [13]CloudResearch. 2026. Connect: Participant Recruitment. https://www.cloudresearch.com/products/participant-recruitment/.
  14. [14]dajro. 2026. BrowserForge: Intelligent Browser Header & Fingerprint Generator. https://github.com/dajro/browserforge.
  15. [15]Nurullah Demir, Matteo Große-Kampmann, Tobias Urban, Christian Wressnegger, Thorsten Holz, and Norbert Pohlmann. 2022. Reproducibility and Replicability of Web Measurement Studies. In WWW. 533–544.
  16. [16]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. In NeurIPS. 28091–28114.arxiv.org/abs/2306.06070
  17. [17]DuckDuckGo. 2026. Tracker Radar. https://github.com/duckduckgo/tracker-radar. US snapshot, commit 6253f5a (July 2026).
  18. [18]Zakir Durumeric. 2026. Chrome (CrUX) Top Million Websites. https://github.com/zakird/crux-top-lists.
  19. [19]Steven Englehardt and Arvind Narayanan. 2016. Online Tracking: A 1-million-site Measurement and Analysis. In ACM CCS. 1388–1401.
  20. [20]Iliana Fayolle, Sihem Bouhniche, Samuel Pélissier, Pierre Laperdrix, Clémentine Maurice, and Walter Rudametkin. 2026. On the Internet, Nobody Knows You’re an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting. arXiv:2606.30119.arxiv.org/abs/2606.30119
  21. [21]Daniel Gößhen, Hugo Jonker, Stefan Karsch, Benjamin Krumnow, and David Roefs. 2021. HLISA: Towards a More Reliable Measurement Tool. In ACM IMC. 380–389.
  22. [22]Ralf Gundelach, Michael Mühlhauser, and Dominik Herrmann. 2026. Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research. In ARES. 275–296. arXiv:2606.14525.arxiv.org/abs/2606.14525
  23. [23]Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. In ACL. 6864–6890.
  24. [24]Sture Holm. 1979. A simple sequentially rejection multiple test procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70.DOI
  25. [25]HumanCursor. 2026. HumanCursor: Human-like Cursor Movement. https://github.com/riftnoslake/HumanCursor.
  26. [26]Christos Iliou, Theodoros Kostoulas, Theodora Tsikrika, Vasilis Katos, Stefanos Vrochidis, and Ioannis Kompatsiaris. 2021. Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics. Digital Threats: Research and Practice 2, 3 (2021), 1–26.DOI
  27. [27]Umar Iqbal, Steven Englehardt, and Zubair Shafiq. 2021. Fingerprinting the Fingerprinters: Learning to Detect Browser Fingerprinting Behaviors. In IEEE S&P. 1143–1161.arxiv.org/abs/2008.04480
  28. [28]Ghalia Jarad and Kemal Bıçakcı. 2026. When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints. arXiv:2602.09606.arxiv.org/abs/2602.09606
  29. [29]Hugo Jonker, Benjamin Krumnow, and Gabry Vlot. 2019. Fingerprint Surface-Based Detection of Web Bot Detectors. In ESORICS. 586–605.DOI
  30. [30]Jordan Jueckstock and Alexandros Kapravelos. 2019. Visible8: In-browser Monitoring of JavaScript in the Wild. In ACM IMC. 393–405.
  31. [31]Jordan Jueckstock, Shaown Sarker, Peter Snyder, Aidan Beggs, Panagiotis Papadopoulos, Matteo Varvello, Benjamin Livshits, and Alexandros Kapravelos. 2021. Towards Realistic and Reproducible Web Crawl Measurements. In WWW. 80–91.
  32. [32]Kaliiiiiiiiiii-Vinzyu. 2026. Patchright: Undetected Version of the Playwright Testing and Automation Library. https://github.com/Kaliiiiiiiiiii-Vinzyu/patchright.
  33. [33]Benjamin Krumnow, Hugo Jonker, and Stefan Karsch. 2022. How Gullible Are Web Measurement Tools? A Case Study Analysing and Strengthening Open-WPM’s Reliability. In ACM CoNEXT. 171–186.
  34. [34]Pierre Laperdrix, Natalia Bielova, Benoit Baudry, and Gildas Avoine. 2020. Browser Fingerprinting: A Survey. ACM Transactions on the Web 14, 2 (2020), 8:1–8:33.arxiv.org/abs/1905.01051
  35. [35]Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczynski, and Wouter Joosen. 2019. Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation. In NDSS.
  36. [36]Xingao Li, Babak Amin Azad, Amir Rahmati, and Nick Nikiforakis. 2021. Good Bot, Bad Bot: Characterizing Automated Browsing Activity. In IEEE S&P. 1589–1605.DOI
  37. [37]Xinfeng Li, Tianze Qiu, Yingbin Jin, Lixu Wang, Hanqing Guo, Xiaojun Jia, Xiaofeng Wang, and Wei Dong. 2026. WebCloak: Characterizing and Mitigating Threats from LLM-Driven Web Agents as Intelligent Scrapers. In IEEE S&P. 436–455.DOI
  38. [38]Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203.
  39. [39]William Lugooloboi, Samuelle Marro, Jabez Magomore, Joss Wright, and Chris Russell. 2026. Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces. arXiv:2605.14786.
  40. [40]Célestin Matte, Natalia Bielova, and Cristiana Santos. 2020. Do Cookie Banners Respect my Choice? Measuring Legal Compliance of Banners from IAB Europe’s Transparency and Consent Framework. In IEEE S&P. 791–809.
  41. [41]Microsoft. 2026. Playwright. https://playwright.dev.
  42. [42]Nick Nikiforakis, Alexandros Kapravelos, Wouter Joosen, Christopher Kruegel, Frank Piessens, and Giovanni Vigna. 2013. Cookieless Monster: Exploring the Ecosystem of Web-Based Device Fingerprinting. In IEEE S&P. 541–555.DOI
  43. [43]Patchright Contributors. 2026. Patchright v1.61.2 – PyPI Release History. https://pypi.org/project/patchright/1.61.2/. Released July 5, 2026.
  44. [44]PyAutoGUI. 2026. PyAutoGUI Documentation. https://pyautogui.readthedocs.io/en/latest/.
  45. [45]Python Ghost Cursor. 2026. Python Ghost Cursor. https://pypi.org/project/python-ghost-cursor/.
  46. [46]Qwen Team. 2025. Qwen3-VL Technical Report. arXiv:2511.21631.
  47. [47]Ali Rasaai, Shivani Singh, Devashish Gosain, and Oliver Gasser. 2023. Exploring the Cookieverse: A Multi-Perspective Analysis of Web Cookies. In PAM. 623–651.
  48. [48]Ryota Saito, Takuya Kataiwa, Minato Takashima, and Tetsushi Ohki. 2026. Invisible Consent: Browser Automation by LLM Agents is Not Privacy-aware. In CHI EA. 414:1–414:9.DOI
  49. [49]StevenBlack. 2026. Hosts: Curated Host Blocklists. https://github.com/stevenblack/hosts.
  50. [50]Tobias Urban, Martin Degeling, Thorsten Holz, and Norbert Pohlmann. 2020. Beyond the Front Page: Measuring Third Party Dynamics in the Field. In WWW, 1275–1286.DOI
  51. [51]Antoine Vastel, Walter Rudametkin, Romain Rouvoy, and Xavier Blanc. 2020. FP-Crawlers: Studying the Resilience of Browser Fingerprinting to Block Crawlers. In MADWeb.DOI
  52. [52]Yash Vekaria, Yohan Beugin, Shaoor Munir, Gunes Acar, Natalia Bielova, Steven Englehardt, Umar Iqbal, Alexandros Kapravelos, Pierre Laperdrix, Nick Nikiforakis, Jason Polakis, Franziska Roessner, Zubair Shafiq, and Sebastian Zimmeck. 2025. SoK: Advances and Open Problems in Web Tracking. arXiv:2506.14057.
  53. [53]Ethan Wang, Zubair Shafiq, and Yash Vekaria. 2026. FP-Agent: Fingerprinting AI Browsing Agents. arXiv:2605.01247.arxiv.org/abs/2605.01247
  54. [54]Webshare. 2026. Webshare Proxies. https://www.webshare.io/.
  55. [55]David Zebe, Sarah Bird, Camila Oliveira, Walter Rudametkin, Ilana Segall, Fredrik Wollén, and Martin Lopatka. 2020. The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing. In WWW, 167–178.DOI
  56. [56]Bouyan Zheng, Boyu Gou, Jiying Kil, Huan Sun, and Yu Su. 2024. GPT-4V(ision) is a Generalist Web Agent, if Grounded. In ICML. 61349–61385.
  57. [57]Shuyuan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In ICLR.arxiv.org/abs/2307.13854

Paper details

vlnc:W_nb85qs3d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents