betaFeedback

WorksComputer Science

Who sets the consent signal? Platform defaults and crawl coverage in measuring opt-outs from AI training

Abstract

Web data used in social research and in the training of AI models increasingly carry consent signals, but who sets these signals is rarely measured. This paper attributes opt-outs from AI training to the party that set them. The complete lists of Polish, Czech, Slovak, Hungarian and German domains recorded by Common Crawl, with 3,445,751 reachable sites, were crawled in October 2026 together with a random sample of 10,000 Slovak registry domains. Each signal was matched against vendor templates and identical files. Four exploratory hypotheses were examined with logistic models. Between 0.70% and 3.82% of the listed sites opt out. Platform and hosting defaults set 15% to 78% of the opt-outs. Settings active by default reach 60% to 95% of a vendor's sites, while settings that owners switch on reach at most 2.1%. Most of the higher Czech and Slovak rates compared with Germany are statistically attributable to platforms. Lists derived from Common Crawl miss about a third of the Slovak opt-outs because they exclude sites that block its crawler. The results describe delegated consent, a signal that a platform sets on behalf of its users, which measurements of consent on the web must consider.

Introduction

Research that uses web data rarely asks who produced the signals it records. Many rules of the digital economy rely on consent signals that individuals are expected to set themselves, such as cookie choices, privacy preference signals and, most recently, opt-outs from the training of artificial intelligence (AI) models. Research on consent management shows that such signals are often produced by intermediaries whose interfaces and defaults determine what most users declare [31, 56, 76]. The same pattern may apply to the content on which AI models are trained. The European Union permits text and data mining of lawfully accessible content unless the rightholder has reserved this use “by machine-readable means” [21 Art. 4(3]. In legal terms this opt-out is a reservation of rights. The AI Act obliges providers of general-purpose AI models to identify and respect it [22 Art. 53(1)(c]. In practice the opt-out signal is a line in a robots.txt file or a metadata tag. Content management systems, e-commerce platforms, hosting providers and content delivery networks often write such lines for thousands of sites at once [37 p. 5].

The rule assumes that an opt-out expresses the decision of the rightholder. This paper uses the informed choice of the rightholder as a normative benchmark. It introduces the concept of delegated consent for a consent signal that a platform or another intermediary sets on behalf of its users. Based on research on defaults and consent intermediaries, platformisation, measurement frames and digital inequality (§ 2), it formulates four expectations. Vendor defaults should produce many of the opt-outs of a national web. Differences between countries should follow their platform markets. Lists derived from crawls should undercount opt-outs, while popular sites should opt out more often than ordinary ones. These expectations are stated as four exploratory hypotheses and examined with descriptive analyses and logistic models.

Existing measurements cannot test these hypotheses because they use lists of top sites ranked by popularity, the domains included in AI training corpora or curated sets of news publishers ([5, 24, 33, 45, 46]). Such samples overrepresent large, professionally managed sites, which opt out far more often than the rest ([46]). Dinzinger and Granitzer (2024) trace robots.txt files to content management systems, but no study separates the settings of vendors from the files written by site owners or compares countries that share the same legal rule.

This paper measures five national web zones that share the Directive but differ in language, in their national implementation of Article 4(3) and in their platform markets: Poland, the Czech Republic, Slovakia, Hungary and Germany. The complete lists of domains that Common Crawl recorded for these zones were crawled in October 2026, which gave 3,445,751 reachable sites. Because such lists cover only part of each zone, a random sample of 10,000 domains drawn from the public list of the Slovak registry serves as a benchmark. Every signal that can express an opt-out from AI training was collected and classified by its form, its scope and its provenance, which records whether a vendor or the site owner set it. The Tranco list (Le Pochat et al. [43]) was used to compare ranked and unranked sites. Four research questions guide the analysis:

RQ1 How common are opt-outs from AI training on national webs?

RQ2 In which legal and technical form are they expressed?

RQ3 Who sets the opt-out signal: the site owner or a platform or hosting provider?

RQ4 How do the domain list and the popularity of sites affect the measured rate?

The paper makes three contributions to the study of consent on the web. First, it makes a methodological contribution to research with web data. It shows how consent signals can be attributed to the party that set them. It also measures how lists derived from Common Crawl exclude the sites that block its crawler, a coverage error in the sense of the total error framework for digital traces ([66]), which extends work on the quality of digital behavioural data ([13, 14] (Weils et al. 2025)) to the frames from which web data are drawn. Second, it develops the concept of delegated consent and extends research on defaults and consent intermediaries from privacy to the use of web content for AI. Where a dominant platform sets an opt-out by default, that single decision produces most opt-outs of a national web, about three quarters on the Czech and Slovak lists. Third, it provides a classification of the signals by form and provenance with a public dataset ([41]).

Conceptual framework

The opt-out of Article 4(3) assumes that each rightholder decides whether to opt out. Four strands of research suggest that practice departs from this model in predictable ways. The result is delegated consent, a signal that expresses the setting of an intermediary unless the user changes it.

The first strand concerns defaults and consent intermediaries. Behavioural research shows that defaults influence choices strongly [36] because the existing arrangement “tends to stick” [71 p. 176]. Studies of platform settings ask whether users configure them actively or show “settings blindness” [27]. Research on privacy consent offers the closest parallel. Field studies of consent notices find that preselected options, nudging and the position of a notice change how users respond [76]. Consent management platforms spread quickly across the web [31], most of their notices fail minimal legal requirements [56] and some register consent before the user has chosen [51]. In social research, consent to share digital traces is itself selective and determines who enters the data [25]. Earlier machine-readable preference signals such as P3P, Do Not Track and Global Privacy Control show that the effect of a signal depends on whether its recipients accept it as the choice of the user [32]. Where a platform writes robots.txt for its customers, an opt-out set by default should therefore persist on most of its sites, while an opt-out that owners must switch on should remain rare.

The second strand concerns platform and infrastructural governance. Platforms set rules for their users through technical settings [26]. The platformisation of the web extends platform infrastructure into external websites [30], while producers that depend on a platform adapt their practices to its rules [55]. Infrastructure studies describe how a few providers come to determine practices across large parts of the web [57]. Shops on a hosted e-commerce platform are such dependent producers, because the platform writes their robots.txt. Platform markets are often regional, as e-commerce platforms serve the language areas in which they operate. Differences between countries should then follow the geography of these markets.

The third strand concerns the frames used to measure the web. Research on digital behavioural data shows that the measurement process, including the choice of platform and data source, explains much of the variation in results [65, 78]. Total error frameworks for digital traces separate errors of representation, such as coverage, from errors of measurement [14, 66]. Content collected by scraping is systematically incomplete when pages cannot be reached [13]. The conditions of access also change what researchers receive [34], while top lists used in web research are unstable and biased towards particular kinds of sites [64]. Common Crawl, the main source of web data for training AI models, collects only pages that its crawler is allowed to fetch [4]. Lists of domains derived from it therefore exclude the sites that block its crawler. Because blocking a crawler is itself an opt-out, such lists should undercount the behaviour that they are used to measure. Popularity lists in turn overrepresent large, professionally managed sites, which opt out more often [46].

The fourth strand concerns inequality in the use of digital technologies. Research on the second level of the digital divide shows that differences in skills and usage persist once access is common [29, 63, 77]. Digital inclusion is gradual and actors with fewer resources are more often excluded from its benefits [59]. Applied to AI training, the capacity to opt out should depend on resources, so professionally managed sites should opt out more often than ordinary ones. Protective behaviour online, however, depends less on literacy than on attitudes such as privacy cynicism [61]. Popularity is moreover only an imperfect indicator of resources. Four hypotheses follow:

H1 Sites of platforms that set an opt-out by default opt out far more often than other sites, while an opt-out that owners must switch on remains rare.

H2 Differences between countries shrink once the platform of each site is taken into account.

H3 Sites missing from the lists derived from Common Crawl opt out more often than listed sites because of opt-outs that name the crawler of Common Crawl.

H4 Sites on a popularity list opt out more often than unranked sites, also when platform and country are taken into account.

H1 and H2 test delegated consent and concern RQ3, while H3 and H4 concern RQ4. RQ1 and RQ2 are descriptive and provide the basis for testing them. The hypotheses were formulated after a first descriptive analysis of the data and are therefore exploratory. The analyses examine whether the observed patterns persist under adjustment and alternative specifications. H2 is examined as a decomposition of country differences and H3 as an estimate of coverage bias.

Technical context

The legal rule, its national implementations and the related case law are summarised in Appendix § 10. Three kinds of signals can express a reservation. The Robots Exclusion Protocol [39] instructs crawlers by name, so a robots.txt file can disallow known AI crawlers such as GPTBot or CCBot, the crawler of Common Crawl. The TDM Reservation Protocol (TDMRep) expresses a reservation in a file, an HTTP header or a meta tag [42]. Content Signals, introduced by Cloudflare in 2025, add a line such as Content-Signal: ai-train=no to robots.txt [1].

Most websites do not write their own robots.txt. Content management systems, e-commerce platforms, hosting providers and content delivery networks generate or modify it. Site owners are often not the rightholders of the content they publish [37 p. 10]. Vendors set opt-outs in robots.txt in different ways. Cloudflare offers a managed robots.txt that customers switch on [10] and since July 2025 has asked new domains whether to block AI crawlers in its network [9]. Squarespace offers a setting that is off unless the owner selects it (Squarespace, 2026). The Polish platform IdoSell documents a switch that a merchant must set to yes for robots.txt to admit AI robots [35]. Figure 1 summarises who writes the file that an AI crawler reads, with the shares of opt-outs measured in § 6.3.

Diagram showing a Rightholder and Site owner, with the site owner choosing among Vendor default (15–78% of opt-outs), Vendor option (0.6–13% of opt-outs), Shared file (5–32% of opt-outs), and Individual file (13–58% of opt-outs), which lead to robots.txt and files and an AI crawler; a network block not in robots.txt also leads to the AI crawler.

Figure 1. Who sets the opt-out signal that an AI crawler reads from the 4.08 million hosts of the Tranco list

Related work

Longpre et al. [46] audit the domains of three AI training corpora and find restrictive robots.txt rules on 26.5% of the top 2,000 domains by token count against 3.4% of a random sample of the corpus domains. Liu et al. [45] study popular sites on the Tranco list, survey artists and show that platform features such as the blocking option of Squarespace are relevant for creators. Bouchaud and Ramacchotti [5] find GPTBot fully blocked on 10.6% of the top one million sites in the Chrome browser data. Hoffmann et al. [33] fetch robots.txt, ai.txt and llms.txt Dinzinger and Granitzer (2024); Dinzinger et al. (2024) use Common Crawl data, find a TDMRep file on only 45 hosts and estimate that at least 10% of robots.txt files are generated by content management systems or tools for search engine optimisation. Fletcher [24] reports that 48% of the top news sites of ten countries blocked the crawlers of OpenAI by the end of 2023. Steinacker-Olsztyn et al. [69] find that 60.0% of reputable news sites with a robots.txt disallow at least one AI crawler against 9.1% of misinformation sites, a gap that popularity alone does not explain. Studies of compliance by crawlers [38, 45, 47] measure what crawlers do, whereas this study measures what sites declare. Policy analysis describes how infrastructure providers influence this practice [37].

None of these studies uses complete national domain lists with a registry benchmark, separates vendor settings from files written by owners or compares countries under the same directive.

Data and methods

Domain lists and registry sample

For each zone the domains recorded in the Common Crawl crawls of July to September 2026 were imported and every domain on the list was crawled. The Polish list was extended with nine earlier crawls and with link discovery. The lists cover 13.4% of the registered domains in .de, 18.3% in .hu, 22.7% in .cz, 23.7% in .sk and 31.0% in .pl [12, 15, 54, 68] (Council of Hungarian Internet Providers 2026). Because the lists were crawled completely, their estimates carry no sampling error but may be biased by coverage. To measure this bias, 10,000 domains were drawn at random from the list of all 493,505 registered .sk domains, which the Slovak registry publishes [68]. Of these, 2,254 were on the Common Crawl list and kept their results, while the other 7,746 were crawled on 25 October 2026 with the same crawler. The other registries do not publish their lists. The unit of analysis is the registrable domain, the domain that a registrant can register under the zone. The main denominator is the set of reachable domains, which answered at least one request.

Collection and classification

For each domain the robots.txt file, the headers and meta tags of the home page, the TDMRep file and the files llms.txt and ai.txt were collected. The crawler honoured robots.txt, applied the Crawl-delay of each site up to ten seconds and used no login. A site counts as opting out if it shows a signal in one of three tiers of legal form. Tier 1 is a protocol built for text and data mining (TDMRep). Tier 2 is a machine-readable directive or signal: a robots.txt group that names a generative AI crawler from a curated list and disallows the whole site, the line Content-Signal: ai-train=no or a noai tag. Tier 3 is a statement in natural language in robots.txt comments, llms.txt or ai.txt without a machine-readable equivalent. Groups for all crawlers that disallow the whole site name no AI use and are not counted, but their prevalence is reported. The scope of each opt-out is training only or training together with search and assistants, following the values of Content Signals and the classes of the blocked crawlers.

The provenance records who wrote the robots.txt of the site. A vendor default is a file that matches the template of a platform or hosting provider that writes the opt-out unless the owner changes it (Shoptet, IdoSell and the German hosting provider PixelX). A vendor option is a template whose opt-out the owner switches on (the managed robots.txt of Cloudflare and the setting of Squarespace). The setting of each template was established from vendor documentation, archived files and the files themselves (§ 6.3). A shared file matches no template and is identical byte for byte on at least two sites with an opt-out, which indicates a framework, plugin or copied list. All other opt-outs are classified as individual files written for one site.

Validation

The classification was compared with blind labels on a stratified holdout sample of 300 sites (60 per zone), weighted back to the population by stratum. A fresh sample of 49 statements in natural language was drawn after the rules for that tier were revised. Recall was checked by labelling sites without an opt-out that contain words an opt-out would need. Labels were produced blind by a language model following a written coding guide, without access to the classifier output. Terms of service were examined in a separate random sample of 8,000 sites per zone with a link from the home page to a terms, regulations or legal notice page on the same site. Up to two such pages per site were fetched with the same rules as the main crawl and scanned for opt-outs from text and data mining or AI training. All flagged pages and 20 unflagged pages per zone that mention these topics were labelled blind.

Statistical analysis

Because the lists were crawled completely and a few provenance sources carry many sites, results for the lists are reported as descriptive values of the listed population. Ratios of rates in Figure 6 carry Katz log intervals as a descriptive measure of their precision. Inference is confined to the registry sample, which carries Wilson score intervals for proportions. The hypotheses are examined with logistic regressions of the presence of an opt-out on reachable sites. Model 1 contains the country of the site as its only predictor. Model 2 adds the platform identified from the robots.txt template (Shoptet, IdoSell, Squarespace or none). Model 3 also adds the use of Cloudflare and the Tranco rank in three bands, with unranked sites as reference. Model 4 uses the Slovak registry sample and relates opting out to absence from the list, the platform and Cloudflare. Model 5 repeats it with opt-outs that do not name CCBot as the outcome. Intervals for Models 4 and 5 come from a cluster bootstrap over 535 providers of name servers (999 resamples), which group sites by hosting independently of the outcome. Shoptet shops are identified by the paths of their cart and filters, IdoSell shops by a comment of their recommendation system and Squarespace sites by the header of its file. The identification thus uses no opt-out line and every platform includes sites without an opt-out. The 2,478 sites of PixelX are excluded from the models, because this provider is recognised only by the file that carries the opt-out. Templates without a setting for AI crawlers count as no platform. Because odds ratios of nested logistic models are not comparable [53], differences between countries are compared across models as average marginal effects in percentage points against Germany, with linear probability models as a check. Since the platform is identified from the file that also carries the opt-out, the models measure associations and do not identify causal effects. All variables are part of the published dataset.

Figure 6. Opt-out rate by Tranco rank and zone

Results

Prevalence of opt-outs (RQ1)

Figure 2 shows the share of reachable sites with an opt-out in each zone, divided by provenance, with the number of reachable sites under each bar. Of the 3,445,751 reachable sites on the lists, 45,837 publish an opt-out signal, from 0.70% in .de to 3.82% in .cz. The last bar gives the Slovak registry sample, in which 4.30% of the 7,787 reachable sites opt out, with a 95% confidence interval (CI) of 3.87% to 4.78%. Its four provenance groups correspond to 101, 47, 31 and 156 sites. Because of rounding the segments may not add up to the total.

Figure 2. Opt-out rate by zone and provenance in per cent of reachable sites

Legal and technical form (RQ2)

Crawler directives in robots.txt and Content Signals are the primary mechanism of more than 93% of opt-outs in every zone. Crawler directives dominate in .pl, .hu and .de, while Content Signals account for 81% of opt-outs in .cz and 75% in .sk. TDMRep is the primary mechanism for at most 1.73% of opt-outs and TDMRep files with an opt-out were found on 1 to 173 domains per zone. Statements in natural language without a machine-readable signal account for 0.41% to 2.47% of opt-outs, except in .sk, where an ai.txt file deployed by a vendor on many sites raises the share to 4.51%. Terms of service, which tier 3 does not cover, were examined separately. Opt-outs in terms pages concern an estimated 0.005% (.sk) to 0.092% (.pl) of reachable sites. The labels confirmed 54 of the 59 flagged pages. Even in .de, where 67.8% of sites link to a terms page, opt-outs found only in terms pages would add about 5% to the opt-outs, with at most 12% at the upper bound of the interval. Most crawler directives address individual crawlers by their names. Among sites that opt out with a crawler directive, 18% (.hu) to 53% (.pl) also block CCBot. Between 20% (.pl) and 42% (.sk) of them name the crawlers of a single operator. Opt-outs limited to training make up 93% in .cz and 86% in .sk, where Content Signals set ai-train=no only. In the other zones they make up 45% to 75%, because many files also block AI search and assistant crawlers.

Provenance of opt-outs (RQ3)

On the lists vendor defaults account for 78% of opt-outs in .cz and 71% in .sk but only for 15% to 20% in the other zones. Vendor options account for 13% of opt-outs in .pl, mainly through the managed robots.txt of Cloudflare. In the other zones they account for at most 3%. Individual files are the most frequent provenance in .pl, .hu and .de with 42%, 51% and 58% of opt-outs, as well as in the Slovak registry sample with 47%. Figure 2 shows the same groups in per cent of reachable sites.

Figure 3 shows the share of each vendor’s sites that carry the opt-out, which indicates how the vendor settings work. Of the 2,558 Polish IdoSell shops, identified by the paths of their robots.txt, 95.2% carry the block of AI crawlers, which agrees with a setting that is active unless the merchant changes it. Of the shops whose robots.txt has the structure of the Shoptet template, 60.3% in .cz, 62.4% in .sk and 67.0% in .hu carry Content-Signal: search=yes, ai-train=no. About 1% carry a line that permits all uses and the others have no Content-Signal line. The files of the three groups are otherwise identical, so the line is the value of a platform setting with at least three states. The few Shoptet shops in .pl and .de show similar shares (Figure 3). Archived copies in Common Crawl show that the line appeared on 23 of 25 sampled shops between their captures of 7 to 18 August and 4 to 17 September 2026, which agrees with a single release of the platform. The help pages of Shoptet describe three settings for search, AI answers and AI training, each with the values allow, disallow or not set. Their default allows search, leaves AI answers unset and disallows training (Shoptet, 2026), which produces exactly the observed line. A trial shop opened on 5 November 2026 showed these preselected values in its administration. The platform writes a signal only when its value is set [79], so the opt-out line is a default that owners can change. Its position in the part of the file that the platform writes agrees with this [8]. Of the sites served through Cloudflare, only 0.32% to 2.09% use its managed robots.txt. Of the live Squarespace sites 0.19% use its setting for AI crawlers. Both agree with settings that customers switch on. Settings that are active by default thus reach 60% to 95% of a platform’s sites, while settings that owners switch on reach at most 2.1%. The hosting provider PixelX serves its file to every hosted site without a robots.txt of its own. Its sites are recognised only by this file, so the share of its sites that carry the opt-out cannot be observed.

Figure 3. Share of each vendor’s sites that carry the opt-out on a logarithmic scale

Figure 4 shows how concentrated the sources of opt-outs are on the lists. For each zone it gives the share of reachable sites with an opt-out and the share that would remain without the largest source of opt-outs. A source is a template, a shared file or, for individual files, the site itself. The comparison assumes that the sites of the largest source would publish no opt-out without it. In .cz and .sk one template produces 78% and 70% of all opt-outs. Without it, the share of sites with an opt-out would fall from 3.82% to 0.84% and from 2.94% to 0.88%, within the range of the other zones without their largest source (0.58% to 1.68%). The three largest sources together account for 19% (.de) to 81% (.cz) of the opt-outs. Individual files remain upper bounds for files written by owners, because 43% of them were labelled as generated. When files are compared after removing sitemap lines, the domain of the site, case and blank lines, the share of reachable sites with an individual file falls to 0.23% to 0.44%. Files written by owners themselves are therefore fewer than Figure 2 suggests.

Figure 4. Opt-out rates with and without the largest source of opt-outs in each zone

Effect of the sampling frame (RQ4)

The registry sample shows that the lists derived from Common Crawl undercount opt-outs (Figure 5). Within the registry sample, the 2,010 reachable sites on the Common Crawl list opt out at 2.94%, the same rate as the full .sk list after rounding. The 5,777 reachable sites that are not on the list opt out at 4.78% (95% CI 4.26-5.36%). The difference comes from sites that block CCBot. They make up 2.75% of the sites off the list against 0.35% of the sites on it (Fisher’s exact test, p<10−12p < 10^{-12}). Common Crawl honours robots.txt, so a site that blocks its crawler rarely enters its index and therefore rarely enters a list derived from it. Opt-outs that do not name CCBot occur at similar rates on and off the list (2.59% and 2.03%, p=0.15p = 0.15). The lists also differ in their provenance mix. In the registry sample vendor defaults account for 30% of opt-outs and individual files for 47%, against 71% and 22% on the .sk list. Vendor options are more frequent in the registry sample because 46 of its 47 sites with the managed robots.txt of Cloudflare are off the list, as this file blocks CCBot. The same pattern appears within the Common Crawl lists. In .cz, .sk and .hu sites recorded in all three crawls opt out more often than sites recorded once, mainly because of platform shops (4.95% against 1.04% in .cz). On the part of the Polish list drawn from the same three crawls as the other lists, 1.32% of sites opt out. Of the domains found through links 7.78% opt out, as do 3.40% of those from earlier crawls. The coverage bias of the list is thus 1.36 percentage points, about a third of the opt-outs in the zone. The estimates for the five lists are therefore likely lower bounds for the share of sites with an opt-out in each zone and overstate the share of vendor defaults. The Polish rate is not directly comparable with the others.

Figure 5. Opt-out rates in the Slovak registry sample on and off the Common Crawl list

Popularity bias (RQ4)

Figure 6 compares reachable sites on the Tranco list with unranked sites and gives the ratio of their rates under each zone. Ranked sites are 0.78% to 1.20% of each zone but opt out 3.6 to 13.0 times more often than unranked sites. The ratio is lowest in .cz and .sk because the Shoptet shops are rarely ranked. Without template files it lies between 6.5 and 15.9. In every zone except .cz the rate is highest in the top 100,000 and 31.3% of the 131 German sites in the top 10,000 opt out. Popular sites and the national web are therefore different populations.

Tests of the hypotheses

Table 1 reports the models for the five lists as descriptive values. Sites of platforms that opt out by default have far higher odds of opting out than sites without a known platform. The odds of Squarespace sites, whose setting the owner must switch on, are only 2.8 times higher in Model 2. Of the sites matched by the templates, 94.8% of IdoSell shops and 60.9% of Shoptet shops

Model 1Model 2Model 3
Odds ratios: country (reference DE)
CZ6.771.471.39
HU1.781.551.46
PL3.622.952.47
SK5.161.561.48
Odds ratios: platform (reference none)
Shoptet181223
IdoSell1,1891,554
Squarespace2.773.34
Odds ratios: other predictors
Cloudflare3.55
Tranco 100k to 1M8.28
Tranco 10k to 100k23.6
Tranco top 10k58.5
Average marginal effects against DE (percentage points)
CZ3.240.330.28
HU0.450.380.33
PL1.501.240.97
SK2.360.380.34
Reachable sites3,443,2733,443,2733,443,273
McFadden R2R^20.0540.2420.272

Table 1. Logistic models of opting out on the five lists

opt out against 1.8% of the sites with a Squarespace file, which agrees with H1. These shares differ from Figure 3, which identifies IdoSell shops by the paths of their files and counts only live Squarespace sites, while the Squarespace template also matches files of unpublished sites. Adding the platforms reduces the average marginal effect of the Czech zone against Germany from 3.24 to 0.33 percentage points and that of the Slovak zone from 2.36 to 0.38 points. In this decomposition platforms account for 90% and 84% of these differences (92% and 85% in linear probability models), as H2 expects. The Polish effect falls only from 1.50 to 1.24 points, since individual files and the managed robots.txt of Cloudflare account for more of its opt-outs. Adjustment for Cloudflare and the Tranco rank in Model 3 changes the country effects little. Ranked sites have 8.3 to 58.5 times the odds of unranked sites in Model 3, against 8.4 to 35.7 times in a model with the Tranco rank alone. The gap thus persists after adjustment and widens for the top bands, which agrees with H4.

Table 2 reports the models of the registry sample with bootstrap intervals. Sites missing from the list have 2.40 times the odds of opting out once platform and Cloudflare are taken into account (Model 4). When opt-outs that name CCBot are excluded from the outcome, the odds ratio falls to 1.21 with an interval that includes one (Model 5). The difference between the frames is thus due to sites that block the crawler of Common Crawl, as H3 expects. This result is close to definitional, since a crawler that is blocked cannot record the site, but it measures a coverage bias that changes the observed rate in the Slovak zone by almost half.

Model 4 (any opt-out)Model 5 (not naming CCBot)
Missing from the list2.40 (1.27 to 3.23)1.21 (0.73 to 1.65)
Shoptet84.9 (51.1 to 122)253 (115 to 463)
Cloudflare8.73 (2.32 to 12.5)8.10 (unstable)
Sites with an opt-out335169
McFadden R2R^20.2420.415

Table 2. Logistic models of opting out in the Slovak registry sample

Robustness and validation

The order of the zones stays the same when crawler directives that name none of the seven major AI operators are excluded and when blocks of other AI crawlers are counted. Counting blocks of all crawlers raises the shares to between 1.19% and 4.03%. Opt-outs whose only signal is a block of CCBot, which also collects data for research and archiving, make up 0.5% to 7.2% of the opt-outs. Without them the rates change by at most 0.15 points (.pl: 1.93% against 2.08%). On the part of the Polish list drawn from the same three crawls as the other lists, the Polish odds ratio in Model 3 falls from 2.47 to 1.30. Its marginal effect falls from 0.97 to 0.21 points, so most of the remaining Polish difference comes from the domains added to its list. A model of the sites without a known platform gives country effects close to those of Model 3. In the registry sample, 40 reachable domains (0.5%) are delegated to parking services. Without them the rate off the list is 4.81% against 4.78% with them. Parking pages hosted by ordinary registrars cannot be identified from the name servers. The holdout sample contained 121 sites that the classifier marked as opting out and 179 that it did not, because the strata oversampled sites with an opt-out. The labels agreed on the presence of an opt-out for 299 of the 300 sites, with one false positive and no missed opt-out. Precision is thus 120 of 121 (99.2%, Clopper-Pearson 95% CI 95.5% to 100%), with full agreement on tier. The rules for statements in natural language were revised after the holdout showed a false positive in this tier. On a fresh sample the blind labels confirmed 40 of 49 such statements (weighted 89%). The recall check scanned 20,000 sites without an opt-out per zone and labelled 100 sites in total from the 7.49% that contained relevant words. It found one missed opt-out, a statement in an ai.txt file. With the upper 95% Clopper-Pearson bound of 5.45% for one miss in 100, missed opt-outs concern at most 0.41% of reachable sites against 1.33% with an opt-out, which bounds recall at 76% or more. Under the revised rules, which detect this statement, the bound of 3.62% for no miss gives 83% or more. The check does not cover AI crawlers missing from the curated list.

Discussion

Theoretical implications

The results agree with the four exploratory hypotheses, with qualifications that follow from the design. An opt-out that a vendor sets by default reaches 60% to 95% of its sites, while one that owners must switch on reaches at most 2.1% (Figure 3). This difference of more than an order of magnitude agrees with the stickiness of defaults (H1). The Czech and Slovak zones show nearly the same pattern under different national texts, because the same platform serves both markets. In a decomposition, 84% to 90% of their difference from Germany is attributable to platforms (H2). Elsewhere individual files are the most frequent provenance, so the influence of vendors depends on the structure of each platform market. Five countries do not allow the effect of the national texts to be separated from other national differences. The results therefore show that platform markets account for most of the observed differences without testing the law itself. The coverage bias of the frame (H3) is due to sites that block CCBot. Ranked sites opt out more often also after adjustment for platform and country (H4).

Taken together, the results describe delegated consent, since the opt-out was designed as an individual decision of each rightholder but in practice operates through infrastructure. This repeats a pattern known from privacy, where consent management platforms and their defaults determine most of the signals that users send [31, 56]. One difference concerns the direction of the default. Consent interfaces often favour the collection of data [50, 76], while the vendors studied here set defaults that favour the rightholder. A decision of one vendor applies to thousands of sites at once and works in both directions. The sites of platforms that write no opt-out signal remain open to AI crawlers without any decision of their owners. Shops on hosted platforms, whose robots.txt is part of the platform, are dependent producers in the sense of research on platformisation [30, 55]. One setting designed and preselected by a single platform produces about three quarters of the opt-outs on two national lists. The mechanism described for the settings of platform users [27] thus also applies to a rule of copyright law. In this sense the providers of content management systems, e-commerce platforms and hosting services take part in the governance of the data used to train AI models.

Delegated consent may also have economic and cultural consequences, which the data do not measure. A change of one default can make thousands of small online shops visible or invisible to AI assistants. The defaults of a few platforms also change how much Czech, Slovak and Hungarian content remains available for training models in smaller languages.

The popularity gap agrees with research on digital inequality, in which actors with more resources use digital options more fully [29, 59, 63]. The data measure neither resources nor preferences, however. Popularity may also reflect the value of content to AI developers or its exposure to public debate. Owners of small sites may decide not to opt out because they want to remain visible in AI search. Protective behaviour online follows attitudes such as privacy cynicism more than literacy [61], so the absence of an action does not by itself indicate a lack of capacity. The gap shows that the opt-out is exercised unevenly, while its causes require data on site owners. For most small sites on hosted platforms the decision lies in practice with the platform, which can protect owners who would not act themselves but also decides for them.

Methodological implications

Research that uses web data should treat consent signals as attributable to the party that set them. Matching files against the templates of platforms and grouping identical files separate the settings of vendors from the decisions of owners and can be applied to other signals, such as llms.txt files or cookie banners. Delegated consent is in this sense a measurement error, since a signal attributed to a site may express the setting of a vendor. Crawl archives in turn are not neutral samples of the web. Like pages that scraping cannot reach [13], sites that refuse a crawler are missing from every dataset built on it, a coverage error in the sense of the total error framework [66, 78]. Studies that measure opting out, or the composition of training data, from lists derived from Common Crawl are therefore likely to underestimate refusal and to overstate the weight of large platforms. A random sample from a registry, even of one zone, shows the size of this bias.

Implications for platform design and policy

Whether a default counts as the decision of the rightholder is legally open [28 p. 17]. With vendor defaults at 15% to 78% of the opt-outs, the answer decides the status of a large part of the opt-out. Platforms can choose among three designs of the opt-out setting. A default with a disclosure informs owners that the platform opts out for them and how to change it. A prompt during onboarding, such as the question that Cloudflare asks new domains, records an explicit decision. No default leaves the decision to owners, most of whom do not take it, as the low uptake of vendor options shows. The first design maximises the reach of the opt-out, the second its attribution to an informed choice and the third the autonomy of the owner. Authorities that monitor the opt-out should report vendor defaults separately from individual opt-outs and rely on random samples drawn from registries, because lists derived from Common Crawl exclude the sites that block its crawler.

Limitations and future research

The data describe the state of October 2026. The five lists probably give lower bounds for their zones. The Polish list was built from more sources than the others and is not directly comparable with them. Only the Slovak registry permits a random sample, so the size of the frame bias in the other zones is unknown. Within the registry sample, domains on the list were crawled between five and sixteen days before the others. Parked domains could be identified only where they are delegated to parking services. Domains that redirect to the same site were not merged. Blocking of AI crawlers in the network of a content delivery network leaves no trace in the collected files, so the influence of intermediaries is if anything underestimated. Platforms are identified from robots.txt, so shops that replaced the file of their platform are counted as sites without a platform and the platform effects are likely overstated. Provenance relies on known templates and byte identity, which miss templates with local changes and may group short files written independently. The setting of PixelX is known only from its files. The defaults of Shoptet and IdoSell are documented by their help pages, a trial shop and the shares of their shops that carry the opt-out. The validation used labels of a language model, whose risks for annotation are documented [44]. The rules for natural language were revised after the holdout was labelled, which the fresh sample addresses. Tranco rank measures popularity, which is only an imperfect indicator of the resources of site owners. The classification makes no statement on legal validity, while compliance by crawlers is outside the scope of the study.

Future research can build on the dataset in five directions. Surveys or interviews of site owners on hosted platforms would show whether they know the setting that determines their opt-out and would answer the question of whose choice it is. Repeated waves of the same collection would show how opt-outs spread and how quickly vendors change their defaults. Changes of a default, such as the introduction of Content Signals by a platform, offer natural experiments on its effect. Random samples from the registries of the other zones would measure the frame bias beyond Slovakia. Experiments with sites under the control of researchers could test which forms of opt-out are respected by crawlers in practice.

Conclusion

The opt-out from AI training is designed as a decision of each rightholder. On the national webs of Poland, the Czech Republic, Slovakia, Hungary and Germany, however, it is largely delegated consent set by platform infrastructure. Opt-outs are rare and are expressed almost entirely through robots.txt and Content Signals. Where a dominant platform sets an opt-out by default, as on the Czech and Slovak lists, its setting produces most of the opt-outs of a national web, so national differences follow regional platform markets. The Slovak registry sample shows that lists derived from Common Crawl exclude the sites that block its crawler, so they undercount opt-outs and overstate the share of vendor defaults. Popular sites opt out far more often than the rest, so the opt-out is used unevenly. The design of platform defaults, the monitoring of the opt-out and the interpretation of Article 4(3) therefore have to take into account who sets the signal and which frame is used to measure it.

Ethics statement

The study collected only files that any visitor of a website can request: robots.txt, the home page, linked terms pages and three files at fixed locations. The crawler followed robots.txt, applied the Crawl-delay of each site up to ten seconds, used no login or form and identified itself with a fixed user agent. No human participants took part in the study. Domains of natural persons can be personal data, as can other content collected from the web [62], so the data are processed for research purposes only. From the public list of the Slovak registry only domain names and name servers were kept. Processing for scientific research follows the safeguards of Article 89 of the General Data Protection Regulation, in particular data minimisation and the pseudonymisation of the released data. As no human participants were involved, the study did not undergo review by an ethics committee. The released dataset contains parsed fields without texts of third parties or addresses of terms of service pages.

Declaration of conflicting interests

The author declares no potential conflicts of interest with respect to the research, authorship or publication of this article.

Funding

The author received no financial support for the research, authorship or publication of this article.

Data availability statement

The parsed signals and the classification of each listed site are available as version 1.0 of October 2026 on Zenodo under a CC BY 4.0 licence at https://doi.org/10.5281/zenodo.23686009 ([41]). The repository also contains the results of the Slovak registry sample and of the terms of service sample, with domain names replaced by keyed hashes. Raw texts of third parties, headers that reveal server versions, addresses of terms of service pages and fields of the registry other than the domain are withheld for legal and data protection reasons. The popularity ranking used is Tranco list N26LW ([43]). The code used for the collection and the analysis is available from the author on request.

Regulatory context, national implementation and case law

Regulatory context

Article 4 of the Directive on copyright in the Digital Single Market ([21]) permits anyone, including commercial developers of AI models, to reproduce lawfully accessible works for text and data mining unless their use has been “expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online” (Art. 4(3)). The AI Act requires providers of general-purpose AI models to identify and comply with these reservations ([22], Art. 53(1)(c)). No harmonised standard defines an appropriate opt-out ([18, 48]), while courts disagree on whether statements in natural language suffice ([28, 58]). The five countries implemented Article 4(3) with different wording, from a requirement of machine-readable form with metadata in Poland to no requirement of form in Slovakia.

National implementation and case law

Article 7(1) of the Directive protects Articles 3, 5 and 6 against contractual override but not Article 4, so the terms of a website may also exclude the general exception. The five countries implemented the Directive with different wording (Table A1), a divergence that was noted soon after the deadline ([49]). German law makes a reservation for online works effective only in machine-readable form. The Polish provision, added in 2024 ([75, 75]), requires a machine-readable format together with metadata. The Czech and Hungarian texts require machine-readable means, the Hungarian one as an advance objection. The Slovak text states no requirement of form, although it has to be read in conformity with the Directive.

ProvisionCondition for content available online
DE § 44b(3) UrhG [72]“nur dann wirksam, wenn er in maschinenlesbarer Form erfolgt” (effective only if made in machine-readable form)
PL art. 263 ust. 2 pr. aut. (Ustawa o otwartych danych, 2021; Ustawa o prawie autorskim, 2025)express reservation “w formacie przeznaczonym do odczytu maszynowego” (in a machine-readable format) as defined in the 2021 open data act “wraz z metadanymi” (with metadata)
CZ § 39c(2) z. 121/2000 Sb. (Autorský zákon č. 121/2000 Sb., 2000)“výslovně vyhradil vhodným způsobem” (expressly reserved in an appropriate manner) and online “strojově čitelnými prostředky” (machine-readable means)
HU § 35/A(1)(b) Szjt. [70]advance objection in an appropriate manner “géppel olvasható formában” (machine-readable form)
SK § 51c(1) z. 185/2015 Z. z. (Autorský zákon č. 185/2015 Z. z., 2015)“ak takéto použitie nie je výslovne vyhradené” (if not expressly reserved), no requirement of form

Table A1. National provisions on reserving text and data mining

The Code of Practice states that adherence to it does not constitute compliance with Union law on copyright (Recital (a)) and keeps reservations expressed by other appropriate means effective (European Commission, AI Office, 2025, Measure 1.3(2)). In December 2025 the Commission opened a consultation on reservation protocols ([19]). No such list had been published by October 2026. In Like Company v Google Ireland (C-250/25) a Hungarian court asked the Court of Justice whether the training of a language model involves reproduction and whether Article 4 applies to it ([7]). In Kneschke v LAION the Regional Court of Hamburg remarked in an obiter dictum that disclaimers in natural language may be machine-readable ([40]), a view that Quintais [58] calls “contrary to Recital 18”. On appeal, the Higher Regional Court of Hamburg called the question “umstritten” (disputed), described the requirement as “technikoffen” (open to any technology) and listed robots.txt and the TDM Reservation Protocol among the forms under discussion ([28 p. 19]). It attributed a reservation by an image agency to the photographer, with the agency acting as a licensee within a contract ([28 p. 17]). The Federal Court of Justice scheduled the hearing of the

revision for 3 September 2026 and had not decided by October 2026 [6]. In a case on a news monitoring service, the District Court of Amsterdam found a robots.txt that excluded only certain AI crawlers insufficient [60 para. 4.33]. The Budapest Court of Appeal treated robots.txt as a machine-readable objection in a case on the indexing of a website by a search engine [23, 52].

References

  1. [1]Allen, W. (2025). Giving users choice with Cloudflare’s new Content Signals Policy. Cloudflare Blog. https://blog.cloudflare.com/content-signals-policy/ (24 September 2025)
  2. [4]Baack, S. (2024). A critical analysis of the largest source for generative AI training data: Common Crawl. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2199-2208). https://doi.org/10.1145/3630106.3659033
  3. [5]Bouchaud, P., & Ramaciotti, P. (2025). Web crawler restrictions, AI training datasets & political biases. arXiv. https://doi.org/10.48550/arXiv.2510.09031
  4. [6]Bundesgerichtshof. (2026). Verhandlungstermin am 3. September 2026 um 9:00 Uhr in Sachen I ZR 281/25 (Erstellen eines Datensatzes für KI-Training). Pressemitteilung Nr. 085/2026. https://www.bundesgerichtshof.de/SharedDocs/Pressemitteilungen/DE/2026/2026085.html
  5. [7]Case C-250/25. (2025). Request for a preliminary ruling from the Budapest Környéki Törvényszék (Hungary) lodged on 3 April 2025, Like Company v Google Ireland Limited (Case C-250/25). Official Journal of the European Union, C/2025/3039. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62025CN0250 (Pending in October 2026)
  6. [8]Cetl, M. (2026). Jak na Shoptetu nastavit, co si o vás AI přečte. Custela blog. https://www.custela.com/blog/shoptet-a-ai-viditelnost (Published 19 September 2026)
  7. [9]Cloudflare. (2025). Cloudflare just changed how AI crawlers scrape the Internet-at-large: Permission-based approach makes way for a new business model. Cloudflare press release. https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/ (1 July 2025)
  8. [10]Cloudflare. (2026). robots.txt setting. Cloudflare Docs. https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ (Last updated 22 October 2026)
  9. [12]CZ.NIC. (2026). Domain statistics of the .cz zone. https://stats.nic.cz/fred_domains_latest (1,617,067 domains on 23 October 2026)
  10. [13]Dahlke, R., Kumar, D., Durumeric, Z., & Hancock, J. T. (2025). Quantifying the systematic bias in the accessibility and inaccessibility of web scraping content from URL-logged web-browsing digital trace data. Social Science Computer Review, 43(5), 1071-1086. https://doi.org/10.1177/08944393231218214
  11. [14]Daikeler, J., Fröhling, L., Sen, I., Birkenmaier, L., Gummer, T., Schwalbach, J., … Lechner, C. (2025). Assessing data quality in the age of digital social research: A systematic review. Social Science Computer Review, 43(5), 943-979. https://doi.org/10.1177/08944393241245395
  12. [15]DENIC. (2026). Domain statistics for .de. https://www.denic.de/fileadmin/denic-domainstatistics/denic-domainstatistics-DE_en.csv (18,266,124 domains on 30 September 2026)
  13. [18]EUIPO. (2025). Executive briefing: The development of generative artificial intelligence from a copyright perspective. European Union Intellectual Property Office. https://www.euipo.europa.eu/en/publications/genai-from-a-copyright-perspective-2025 (May 2025. Full study: doi:10.2814/3893780)
  14. [19]European Commission. (2025). Commission launches consultation on protocols for reserving rights from text and data mining under the AI Act and the GPAI Code of Practice. https://digital-strategy.ec.europa.eu/en/consultations/commission-launches-consultation-protocols-reserving-rights-text-and-data-mining-under-ai-act-and (Consultation open 1 December 2025 to 23 January 2026)
  15. [21]European Parliament and Council of the European Union. (2019). Directive (EU) 2019/790 of the European Parliament and of the Council of 17 April 2019 on copyright and related rights in the Digital Single Market and amending Directives 96/9/EC and 2001/29/EC. https://eur-lex.europa.eu/eli/dir/2019/790/oj (Official Journal of the European Union, L 130, 17.5.2019, pp. 92-125)
  16. [22]European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). https://eur-lex.europa.eu/eli/reg/2024/1689/oj (Official Journal of the European Union, L, 2024/1689, 12.7.2024)
  17. [23]Fővárosi Ítélőtábla. (2024). Judgment of 3 December 2024, 9.Pf.20.353/2024/6-II. https://eakta.birosag.hu/anonimizalt-hatarozatok (Budapest Court of Appeal, case on the indexing of a website by a search engine, para. [51]. Anonymised text in the court decision database (search 20.353/2024))
  18. [24]Fletcher, R. (2024). How many news websites block AI crawlers? Reuters Institute for the Study of Journalism, Factsheet. https://doi.org/10.60625/risj-xm9g-ws87
  19. [25]Gaughan, C., Cernat, A., Gibson, R., Cantijoch, M., & Batista-Navarro, R. (2026). Who consents to sharing their tweets with researchers? A comparative analysis of selection bias in linked survey and social media data. Social Science Computer Review. (Advance online publication) https://doi.org/10.1177/08944393261421114
  20. [26]Gorwa, R. (2019). What is platform governance? Information, Communication & Society, 22(6), 854-871. https://doi.org/10.1080/1369118X.2019.1573914
  21. [27]Haji, R., & Stock, W. G. (2021). User settings for advertising optimization on Facebook: Active customer participation or settings blindness? Telematics and Informatics, 59, 101548. https://doi.org/10.1016/j.tele.2020.101548
  22. [28]Hanseatisches Oberlandesgericht Hamburg. (2025). Urteil vom 10. Dezember 2025, 5 U 104/24 (Kneschke v LAION). https://www.itm.nrw/wp-content/uploads/2025/12/5-u-104-24.pdf (5. Zivilsenat. Anonymised full text. Revision pending at the Bundesgerichtshof, I ZR 281/25)
  23. [29]Hargittai, E. (2002). Second-level digital divide: Differences in people’s online skills. First Monday, 7(4). https://doi.org/10.5210/fm.v7i4.942
  24. [30]Helmond, A. (2015). The platformization of the web: Making web data platform ready. Social Media + Society, 1(2). https://doi.org/10.1177/2056305115603080
  25. [31]Hils, M., Woods, D. W., & Böhme, R. (2020). Measuring the emergence of consent management on the web. In Proceedings of the ACM Internet Measurement Conference (pp. 317-332). https://doi.org/10.1145/3419394.3423647
  26. [32]Hils, M., Woods, D. W., & Böhme, R. (2021). Privacy preference signals: Past, present and future. Proceedings on Privacy Enhancing Technologies, 2021(4), 249-269. https://doi.org/10.2478/popets-2021-0069
  27. [33]Hoffmann, F., Goergens, T., Khosla, S., & Bajpai, V. (2026). From robots.txt to ai.txt: Mapping the evolution of web permissions in the age of AI. ACM SIGCOMM Computer Communication Review, 56(2), 29-41. https://doi.org/10.1145/3831956.3831960
  28. [34]Hölzl, J., Keusch, F., & Collins, J. (2026). Where you are is what you get! Inconsistencies of digital trace data across download locations. Social Science Computer Review. (Advance online publication) https://doi.org/10.1177/08944393261469494
  29. [35]IdoSell. (2026). Pozycjonowanie SEO i SEM w sklepie. IdoSell Help. https://pomoc.idosell.com/wyglad-i-konfiguracja/seo-w-sklepie/pozycjonowanie-seo-i-sem-w-sklepie (Last modified 5 February 2026)
  30. [36]Johnson, E. J., & Goldstein, D. (2003). Do defaults save lives? Science, 302(5649), 1338-1339. https://doi.org/10.1126/science.1091721
  31. [37]Keller, P. (2026). Divergent Mechanisms, Elusive Vocabularies: The state of machine-readable rights reservations. Open Future, Policy Brief 10. https://openfuture.eu/wp-content/uploads/2026/01/260115_Divergent-Mechanisms-Elusive-Vocabularies.pdf (15 January 2026)
  32. [38]Kim, T., Bock, K., Luo, C., Liswood, A., Posorlay, C., & Wenger, E. (2025). Scrapers selectively respect robots.txt directives: Evidence from a large-scale empirical study. In Proceedings of the ACM Internet Measurement Conference (IMC ’25) (pp. 541-557). https://doi.org/10.1145/3730567.3764471
  33. [39]Koster, M., Illyes, G., Zeller, H., & Sಸ್ಸman, L. (2022). Robots exclusion protocol (RFC 9309). Internet Engineering Task Force. https://doi.org/10.17487/RFC9309
  34. [40]Landgericht Hamburg. (2024). Urteil vom 27.09.2024, Az. 310 O 227/23 (Kneschke v LAION e.V.). https://openjur.de/u/2495651.html
  35. [41]Lawenda, M. (2026). Reservations against AI training on five European national webs: Parsed signals, October 2026 (Version 1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.23686009
  36. [42]Le Meur, L. (2024). TDM Reservation Protocol (TDMRep): Final community group report. W3C Text and Data Mining Reservation Protocol Community Group. https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/ (10 May 2024, not on the W3C Standards Track)
  37. [43]Le Pochat, V., Van Goethem, T., Tajalizadehkhoob, S., Korczyński, M., & Joosen, W. (2019). Tranco: A research-oriented top sites ranking hardened against manipulation. In Network and Distributed System Security Symposium (NDSS). https://doi.org/10.14722/ndss.2019.23386
  38. [44]Lin, H., & Zhang, Y. (2026). Navigating the risks of using large language models for text annotation in social science research. Social Science Computer Review, 44(3), 403-427. https://doi.org/10.1177/08944393251366243
  39. [45]Liu, E., Luo, E., Shan, S., Voelker, G. M., Zhao, B. Y., & Savage, S. (2025). Somesite I used to crawl: Awareness, agency and efficacy in protecting content creators from AI crawlers. In Proceedings of the ACM Internet Measurement Conference (IMC ’25) (pp. 78-99). https://doi.org/10.1145/3730567.3732913
  40. [46]Longpre, S., Mahari, R., Lee, A., Lund, C., Oderinwale, H., Brannon, W., . . . Pentland, S. (2024). Consent in crisis: The rapid decline of the AI data commons. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track (pp. 108042-108087). https://doi.org/10.52202/079017-3431
  41. [47]Lopez-Fonseca, G., Rodriguez, D., Bechtold, S., & Del Alamo, J. M. (2026). Do generative AI assistants respect robots.txt? Tracing web access beyond visible answers. arXiv. https://doi.org/10.48550/arXiv.2607.14447
  42. [48]Lucchi, N. (2025). Generative AI and copyright: Training, creation, regulation. European Parliament, Policy Department for Justice, Civil Liberties and Institutional Affairs. https://www.europarl.europa.eu/thinktank/en/document/IUST_STU(2025)774095 (Study requested by the JURI Committee, July 2025, PE 774.095)DOI
  43. [49]Margoni, T., & Kretschmer, M. (2022). A deeper look into the EU text and data mining exceptions: Harmonisation, data ownership, and the future of technology. GRUR International, 71(8), 685-701. https://doi.org/10.1093/grurint/ikac054
  44. [50]Mathur, A., Acar, G., Friedman, M. J., Lucherini, E., Mayer, J., Chetty, M., & Narayanan, A. (2019). Dark patterns at scale: Findings from a crawl of 11K shopping websites. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 81. https://doi.org/10.1145/3359183
  45. [51]Matte, C., Bielova, N., & Santos, C. (2020). Do cookie banners respect my choice? measuring legal compliance of banners from IAB Europe’s Transparency and Consent Framework. In 2020 IEEE Symposium on Security and Privacy (pp. 791-809). https://doi.org/10.1109/SP40000.2020.00076
  46. [52]Mezei, P. (2025). The multi-layered regulation of rights reservation (opt-out) under EU copyright law and the AI Act: for the benefit of whom? SSRN preprint 5064018, version 3. https://doi.org/10.2139/ssrn.5064018
  47. [53]Mood, C. (2010). Logistic regression: Why we cannot do what we think we can do, and what we can do about it. European Sociological Review, 26(1), 67-82. https://doi.org/10.1093/esr/jcp006
  48. [54]NASK. (2026). Liczba aktywnych nazw domeny .pl. https://www.dns.pl/statystyki/liczba_aktywnych_nazw_domeny_pl (2,745,034 names on 24 October 2026)
  49. [55]Nieborg, D. B., & Poell, T. (2018). The platformization of cultural production: Theorizing the contingent cultural commodity. New Media & Society, 20(11), 4275-4292. https://doi.org/10.1177/1461444818769694
  50. [56]Nouwens, M., Liccardi, I., Veale, M., Karger, D., & Kagal, L. (2020). Dark patterns after the GDPR: Scraping consent pop-ups and demonstrating their influence. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-13). https://doi.org/10.1145/3313831.3376321
  51. [57]Plantin, J.-C., Lagoze, C., Edwards, P. N., & Sandvig, C. (2018). Infrastructure studies meet platform studies in the age of Google and Facebook. New Media & Society, 20(1), 293-310. https://doi.org/10.1177/1461444816661553
  52. [58]Quintais, J. P. (2025). Generative AI, copyright and the AI Act. Computer Law & Security Review, 56, 106107. https://doi.org/10.1016/j.clsr.2025.106107
  53. [59]Ragnedda, M., Ruiu, M. L., & Addeo, F. (2022). The self-reinforcing effect of digital and social exclusion: The inequality loop. Telematics and Informatics, 72, 101852. https://doi.org/10.1016/j.tele.2022.101852
  54. [60]Rechtbank Amsterdam. (2024). Judgment of 30 October 2024, ECLI:NL:RBAMS:2024:6563 (DPG Media et al. v Howardshome). https://uitspraken.rechtspraak.nl/details?id=ECLI:NL:RBAMS:2024:6563
  55. [61]Respi, C., Gui, M., Scaduto, G., Serini, M., Pizzul, D., Gerosa, T., & Lutz, C. (2026). Lower cynicism, not higher literacy, promotes protective behavior: Exploring the “privacy exception” in the digital inequality framework. Social Science Computer Review, 44(5), 1085-1106. https://doi.org/10.1177/08944393251341201
  56. [62]Rohatgi, A., & Park, T. J. (2025). Privacy in the public: Analysing the EU framework to outline approaches for regulating AI personal data scraping. Computer Law & Security Review, 57, 106150. https://doi.org/10.1016/j.clsr.2025.106150
  57. [63]Scheerder, A., van Deursen, A., & van Dijk, J. (2017). Determinants of Internet skills, uses and outcomes. A systematic review of the second- and third-level digital divide. Telematics and Informatics, 34(8), 1607-1624. https://doi.org/10.1016/j.tele.2017.07.007
  58. [64]Scheitle, Q., Hohlfeld, O., Gamba, J., Jelten, J., Zimmermann, T., Strowes, S. D., & Vallina-Rodriguez, N. (2018). A long way to the top: Significance, structure, and stability of Internet top lists. In Proceedings of the Internet Measurement Conference 2018 (pp. 478-493). https://doi.org/10.1145/3275852.3278574
  59. [65]Schneck, A., & Przepiorka, W. (2025). Meta-dominance analysis: A tool for the assessment of the quality of digital behavioural data. Social Science Computer Review, 43(5), 980-1000. https://doi.org/10.1177/08944393241261958
  60. [66]Sen, I., Flöck, F., Weller, K., Weis, B., & Wagner, C. (2021). A total error framework for digital traces of human behavior on online platforms. Public Opinion Quarterly, 85(S1), 399-422. https://doi.org/10.1093/poq/nfab018
  61. [68]SK-NIC. (2026). List of registered .sk domains in live status. https://sk-nic.sk/subory/ domains.txt (493,505 domains in the list generated on 25 October 2026) Squarespace. (2026). Request that AI models exclude your site. Squarespace Help Center. https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site (Updated 8 January 2026)
  62. [69]Steinacker-Olsztyn, N., Gosain, D., & Dao, H. (2026). Is misinformation more open? A study of robots.txt gatekeeping on the web. In Proceedings of the ACM Web Conference 2026 (WWW ’26). (Also arXiv:2510.10315) https://doi.org/10.48550/arXiv.2510.10315
  63. [70]Szerzői jogi törvény. (1999). 1999. évi LXXVI. törvény a szerzői jogról, § 35/A. Magyar Közlöny. https://njt.hu/jogszabaly/1999-76-00-00 (Section 35/A inserted by 2021. évi XXXVII. törvény, in force 1 June 2021)
  64. [71]Thaler, R. H., & Sunstein, C. R. (2003). Libertarian paternalism. American Economic Review, 93(2), 175-179. https://doi.org/10.1257/000282803321947001
  65. [72]Urheberrechtsgesetz. (2021). Gesetz über Urheberrecht und verwandte Schutzrechte (Urheberrechtsgesetz), § 44b. BGBl. I 1965, S. 1273, as amended. https://www.gesetze-im-internet.de/urhg/__44b.html (Section 44b inserted by the Act of 31 May 2021, BGBl. I S. 1204)
  66. [75]Ustawa z dnia 26 lipca 2024 r. (2024). Ustawa z dnia 26 lipca 2024 r. o zmianie ustawy o prawie autorskim i prawach pokrewnych, ustawy o ochronie baz danych oraz ustawy o zbiorowym zarządzaniu prawami autorskimi i prawami pokrewnymi. Dz.U. 2024 poz. 1254. https://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id=WDU20240001254 (In force 20 September 2024. Consolidated text of the copyright act: Dz.U. 2025 poz. 24)
  67. [76]Utz, C., Degeling, M., Fahl, S., Schaub, F., & Holz, T. (2019). (un)informed consent: Studying GDPR consent notices in the field. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (pp. 973-990). https://doi.org/10.1145/3319535.3354212
  68. [77]van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2014). The digital divide shifts to differences in usage. New Media & Society, 16(3), 507-526. https://doi.org/10.1177/1461444813487959
  69. [78]Weiß, B., Leitgöb, H., & Wagner, C. (2025). Conceptualizing, assessing, and improving the quality of digital behavioral data. Social Science Computer Review, 43(5), 927-942. https://doi.org/10.1177/08944393251367041
  70. [79]Zahálka, T. (2026). Robots.txt v Shoptetu: jak ho správně nastavit pro SEO. Blog. https://tomaszahalka.cz/co-je-robots-txt/ (Published 20 August 2026, modified 7 September 2026)

Paper details

vlnc:W_prv4wrya
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents