betaFeedback

WorksComputer Science

Manifest-Assisted Retrieval in Artificial Intelligence Search: Evidence Gaps and a Security Evaluation

Abstract

The external sources retained in our non-systematic audit did not isolate the incremental security or utility effect of consuming /llms.txt while ordinary retrieval remained fixed. The initial audit covered 39 sources through August 16, 2026, with later targeted updates through October 14. A purposive 15-tool study observed no automatic requests in 45 trials at one endpoint; a non-probability log study found that 97% of domains with detected files received no recorded requests in May 2026. We separate publication, fetch, model use, and action; define five operational threat paths and paired security-utility estimands; and exercise deterministic pre-model controls with 30 author-defined parser fixtures. We also execute a reduced pilot: 24 tasks across eight sites, five conditions, and 120 real model responses, with ordinary evidence held fixed. Frozen automatic scoring identifies attacker-target adoption in 21/2421/24 raw adversarial responses, with one further response unresolved; the adversarial URLs-only condition has no known adoptions and one unresolved response. The pilot measures output changes conditional on supplied evidence, without live retrieval or external actions. Together, the framework and pilot distinguish manifest-content effects from publication, discovery, and downstream execution, while leaving full retrieval and agent evaluation open.

Introduction

Retrieval-augmented generation (RAG) lets a language model consult current sources at answer time [1]. In artificial intelligence (AI) search, publisher-controlled pages and tool outputs enter the answer context, where embedded instructions can redirect an application [2].

Instruction following is only one failure mode. Our framework therefore separates data, ranking, and resource outcomes from instruction hijacking.

The /llms.txt v2 proposal permits a predictable Markdown overview and curated link list at the site root or beneath a subpath [3]. This paper focuses on root-level /llms.txt because the operational and product sources audited here use that location. Unlike robots.txt, which expresses crawler access preferences, /llms.txt supplies publisher-selected prose and links without defining trust, ranking, permission, or execution semantics [4]. The closest discovery comparator is sitemap.xml, which lists publisher-declared locations for retrieval and indexing [5]. Both formats are publisher-authored discovery aids, but neither establishes that a downstream system will fetch, prioritize, trust, cite, or act on any listed resource.

This paper asks whether consuming /llms.txt changes security or utility relative to ordinary retrieval. We contribute five path-specific outcomes, a pipeline-stage evidence audit, paired manifest-specific loss estimands, and a guarded-consumption design whose deterministic parser and pre-fetch decisions were exercised against 30 offline fixtures. We add a 120-response pilot that varies manifest content and prose ingestion while holding supplied ordinary evidence fixed. Low observed consumption limits present exposure; it does not establish what happens when a consumer begins using the file.

Evidence-Audit Method

The initial audit assembled research, standards, specifications, first-party product documentation, and operational studies on August 5, 2026, followed by a targeted recheck of manifest, product, and operational sources on August 16. We retained 39 sources, with 15 focal studies purposively selected to cover the available stages from discovery through action. This coverage sample is illustrative rather than statistically representative. Historical database names, exact search strings, complete exports, and counts of records screened or excluded were not retained in a reproducible form. This is therefore a non-systematic evidence audit, and its absence claim applies only to the retained sources. One reviewer performed selection and extraction without independent screening or adjudication.

For each estimate, we retained attacker access, system snapshot, evidence stage, endpoint, numerator, denominator, uncertainty, benign utility, adaptive knowledge, directness to /llms.txt, and artifact availability. Heterogeneous endpoints and sampling frames precluded pooling, so comparisons remain within a study or common stage. The companion material contains the retained source ledger, stage coding, bias and applicability judgments, figure data, and replication instructions. It cannot reconstruct the missing historical search flow.

A separate targeted update on October 10, with selected source rechecks on October 14, added two directly relevant sources discussed in § 3.1; it was not a rerun of a comprehensive search.

We use demonstrated for controlled attacks, defenses, or measured system behavior; observed for crawl, request-log, or implementation evidence; and proposed for controls introduced here. Formal prompt-injection definitions separate exposure, model control, and realized outcome [6]. National Institute of Standards and Technology (NIST) terminology further distinguishes attack mechanisms from impacts [7].

Threat Model and Estimands

Pipeline and Adversary

An AI-search pipeline comprises retrieval RR, context construction KK, a large language model (LLM) MM, and an optional action layer AA. The adversary controls at least one reachable first-party, compromised, user-generated, corpus, metadata, alternate-representation, or off-origin resource, while the user, model weights, and trusted orchestrator remain outside its control. Figure 1 traces exposure through model interpretation, policy, capabilities, and resulting security loss.

Figure 1. Proposed threat model. A content-borne attack crosses an exposure boundary and may follow one or more paths. Retrieval abuse can precede model interpretation, and ranking manipulation may arise during retrieval or reranking before final context construction. Instruction hijacking, data injection, and evidence poisoning require model-visible content, but only instruction hijacking necessarily involves instruction-like control. Deterministic policy can reduce realized impact independently of model robustness.

Operational Paths

A binary hijack label conflates failures during retrieval, interpretation, ranking, and action. Table I instead assigns each path a decision rule and denominator.

PathDecision ruleMetric / denominator
Retrieval abuseFetching or parsing violates a network, origin, resource, cost, or availability policy before model-selected action.Violations or excess cost per attempted fetch.
Instruction hijackInstruction-bearing content increases an attacker-goal or task-violation probability relative to a matched neutralized twin.Goal or violation rates per exposed trial; repeated-sample contrast.
Data injectionThe task uses an attacker-controlled identifier, origin, author, tool result, or value as trusted data.Corrupted-field action rate per exposed trial.
Evidence poisoningA controlled false or selected claim changes the answer, support, or confidence without an attacker-task instruction.Target-claim success per context inclusion.
Ranking manipulationWith canonical facts fixed, attacker-controlled content changes candidate inclusion or promotes, suppresses, or monopolizes sources, products, or citations in violation of the user’s stated criteria.Candidate-set, rank, or citation-share change per paired query.

Table I. Operational attack paths and outcomes

Matched twins use a prespecified task predicate UU and attacker-goal predicate GG. For stochastic outputs, instruction attribution concerns Pr⁡(G=1∣zinj)−Pr⁡(G=1∣zneutral)\Pr(G = 1 \mid z_{\mathrm{inj}}) - \Pr(G = 1 \mid z_{\mathrm{neutral}}),

or the corresponding difference for U=0U = 0, estimated over repeated generations. The neutralized twin removes instruction-like material while retaining relevant facts, formatting, and retrieval opportunity where possible. A single target answer does not establish instruction attribution; inseparable factual and instructional edits remain attribution-uncertain. Following the separation of mechanisms and impacts [7], we code stage, mechanism, and outcome separately and permit overlapping path labels within individual trials.

Exposure, Control, and Loss

For site–query–model trial ii and arm z=(m,p)z = (m,p), m∈{absent,benign,adversarial}m \in\{\text{absent},\text{benign},\text{adversarial}\} is the manifest condition and pp is the consumption policy. Let Ti(z)T_i(z), Yi(z)Y_i(z), and Ai(z)A_i(z) denote the pipeline trace, answer, and action. For each path kk, let Bik(z)B_{i k}(z) indicate a prespecified violation under Table I. We define

Ls,i(z)=max⁡k=1,…,5Bik(z),Lu,i(z)=1−Ui(z).(1)L_{s,i}(z) = \max_{k=1,\ldots,5} B_{i k}(z), \qquad L_{u,i}(z) = 1 - U_i(z). \tag*{(1)}

Here Ui=1U_i = 1 requires the authorized answer or task state, supporting evidence, and compliance with task constraints, including no prohibited action; otherwise Ui=0U_i = 0. The binary security loss counts a trial once when any path is compromised. It measures incidence, without assigning severity weights; path outcomes and consequential actions are reported separately.

The paired security and utility estimands remain

Δs(z1,z0)=E[Ls,i(z1)−Ls,i(z0)],(2)\Delta_s(z_1,z_0) = \mathbb{E}[L_{s,i}(z_1) - L_{s,i}(z_0)], \tag*{(2)}
Δu(z1,z0)=E[Lu,i(z1)−Lu,i(z0)].(3)\Delta_u(z_1,z_0) = \mathbb{E}[L_{u,i}(z_1) - L_{u,i}(z_0)]. \tag*{(3)}

Positive values indicate greater loss under z1z_1. The expectation refers to the declared evaluation distribution. Paired comparisons randomize arm order while holding the site, query, model, index, and evaluation horizon fixed and resetting state to avoid spillover and carryover. The control arm retains ordinary Hypertext Markup Language (HTML), alternate Markdown, metadata, and sitemap retrieval.

Retrieval abuse may occur before model visibility. Conditional rates use the denominators named in Table I; cumulative stagewise rates retain the original assigned cohort. Model errors or unreadable outputs leave semantic security unresolved, with bounds obtained by assigning unknown outcomes zero or one rather than counting them as safe. An unsuccessful or malformed task response incurs utility loss one.

Evidence by Pipeline Stage

Table II presents 10 of the 15 illustrative focal studies, their reported outcomes, and the limits of each measurement. Figure 2 maps evidence-stage coverage; Figure 3 traces cumulative attack survival across pipeline stages.

Figure 2. Evidence-stage coverage for the 15 historical focal studies. Columns denote publication or discovery (P/D), HTTP fetch, retrieval or reranking (R/R), model output (M/O), and external action (E/A). Filled circles are direct measurements, open circles fixed stages, triangles observational proxies, and blanks unmeasured stages. Carrier labels distinguish actual /llms.txt, manifest-style proxies, and generic content. October additions are outside this historical map.

Cumulative success across retrieval, reranking, and generation under BM25 and dense retrieval

Figure 3. Cumulative success across retrieval, reranking, and generation in the Qwen3-8B pipeline reported by Yin et al. [10]. Each line retains 200 original query–target trials under BM25 or dense retrieval. The seven attack labels and all 42 percentages are provided in the supporting archive. The figure is neither a cross-study average nor an /llms.txt estimate.

StudySetting / attacker modelReported outcome and interpretation limit
BIPIA [12]Five tasks; 86,250 factorial promptsGPT-4 attack success 31.03%; model-output instruction following with exposure fixed, not independent web retrieval or executed external action.
AgentDojo [13]GPT-4o; 629 stateful casesUtility 69.0% benign and 50.01% under attack; targeted attack success 57.69%.
Chang et al. [14]Ten-token poison; one malicious item per query; 11 × 100 queries with the default embedding modelMean Recall@5 94.1%; this retrieval metric is separate from the paper’s end-to-end RAG and agent exploit outcomes.
PoisonedRAG [15]Five malicious passages in a 2,681,468-text corpus97/100 target answers; targeted corpus poisoning, not arbitrary-page prevalence.
RAG data loaders [16]Nineteen techniques; five loaders; 357 scenariosLoader-stage attack success 74.4%; ingestion representation, not live discovery.
SIREN [17]One edited page per context; two Claude models; eight query–model contextsRank-1 in 62/124 technique trials; successful payloads had a mean fresh-session reproduction rate of 0.805.
Web scan [18]1.2 billion Uniform Resource Locators; separate 5,200-trial follow-up15,387 validated instances; exposure and controlled response remain distinct.
Chen et al. [19]1,000 query–site pairs; nine systems; three trials27,000 requests; 99.78% cumulative blocking rate; target survival, not semantic harm.
Yin et al. [10]Qwen3-8B; 200 rank-10 targets per attack–retriever pairCumulative reasoning-attack survival was 91.0/62.5/48.0% across retrieval, reranking, and generation with BM25 and 98.5/72.5/59.0% with dense retrieval.
Borysenko [20]Fifteen tools; one endpoint; three trials per toolNo incidental /llms.txt requests: 0/15 tools and 0/45 trials under an explicit-URL prompt.

Table II. Illustrative focal studies, reported outcomes, and interpretation limits

Evidence on ranking and model behavior is stage-specific. A conversational-search benchmark measures manipulation after source texts have already been selected [8]; a separate ranker experiment holds the candidate set fixed and measures reordering [9]. Both designs identify post-discovery effects and cannot estimate organic discovery.

Table II preserves study units, denominators, and attack-success terms. Figure 2 classifies publication or discovery, Hypertext Transfer Protocol (HTTP) fetch, retrieval or reranking, model output, and external action as direct, fixed, proxy, or unmeasured, locating evidence within the pipeline without assessing methodological quality.

Rates from noncomparable designs should not be ranked. Model-visible benchmarks condition on exposure, while retrieval, loader, and search studies begin at different pipeline stages. Blank downstream stages are unmeasured rather than negative findings. Endpoints and denominators therefore remain separate throughout the analysis.

Figure 3 shows one reasoning-based attack falling from 91.0% to 48.0% under BM25 and from 98.5% to 59.0% under dense retrieval. The decline reflects cumulative pipeline filtering, not a manifest effect. Stagewise reporting should retain the original query–target cohort as the denominator at every transition and report both conditional pass rates and cumulative survival. Changing to a survivor-only denominator measures a different quantity and prevents direct comparison with the cumulative rates. A complete audit trace therefore links retrieval, reranking, context inclusion, answer selection, and action under one trial identifier.

/llms.txt Evidence Gap

Table III separates publication, requests, and current provider statements. These endpoints cannot be arranged as one adoption funnel because their samples and units differ.

Source / sampleObserved endpointInference boundary
Rankability; September 18 ranked samples [11]93/1,000 and 830/10,000 confirmed root files; 474 and 4,211 unknown outcomesNon-probability frame; changed validation rules prevent a direct trend comparison with the historical 87/1,000 result. No use measure.
SE Ranking; nearly 300,000 domains [21]10.13% presence without a detected citation advantageCommercial cross-section without causal support.
Ahrefs; 137,210 domains [22]About 38,000 detected HTTP-200 Markdown files; 97% received no recorded request during May 2026Non-probability frame; specification conformance was not assessed, and request absence is not a safety outcome.
Borysenko; 15 tools, 45 trials [20]No /llms.txt requests: 0/15 tools and 0/45 trialsServer-log test of incidental fetching under an explicit-URL prompt; purposive tools, one endpoint, and one task.
Google Search; Chrome Lighthouse [23], [24]Search ignores the file for ranking; Lighthouse offers an optional auditProduct-specific statements; general consumption remains unmeasured.

Table III. /LLMS.TXT EVIDENCE BY SOURCE AND DENOMINATOR

The v2 proposal defines a Markdown overview at the site root or beneath a subpath, with one required level-1 heading, optional summary and prose, and zero or more level-2 link-list sections; an Optional section is conventional rather than required, and processing remains consumer-defined [3]. The distinction from crawler-access preferences is substantive, not merely syntactic, because discovery does not confer trust.

Rankability’s September 18 update reports 93/1,000 and 830/10,000 confirmed root-file adopters, with 474 and 4,211 unknown outcomes, respectively [11]. Its non-probability ranking frame and validation rules changed, so these observations cannot establish growth from the original 87/1,000 figure. A commercial cross-section of nearly 300,000 domains found 10.13% presence but no detected citation advantage [21].

A non-probability frame of 137,210 Ahrefs Web Analytics domains contained about 38,000 detected HTTP-200 Markdown files; 97% of those domains received no recorded request for the file during May 2026 [22]. Full specification conformance was not assessed. Google Search documentation states that the file is ignored for visibility and ranking [23]. These observations constrain claims about present exposure; they do not answer what changes when a consumer deliberately uses the file.

An optional Chrome Lighthouse audit flags server errors when retrieving /llms.txt; a missing file is marked not applicable, so the audit does not demonstrate model use [24]. A separate permission-manifest proposal declares allowed web-agent interactions but leaves enforcement to complementary mechanisms in the surrounding system [25].

The counterfactual is narrow: hold the site, queries, model, index, and canonical facts fixed, then vary manifest condition mm and consumption policy pp. A manifest-specific effect exists only when the file changes context inclusion, output, ranking, action, cost, or utility relative to ordinary retrieval. The reduced pilot in Section VII measures answer changes with supplied evidence fixed; the full retrieval comparison remains unexecuted.

Two additional sources concern model use and action. Hertz reports package-namespace interventions and agent-triggered execution following manifests or manifest-linked documentation [26]. This first-party disclosure lacks a paired absent-manifest and benign-utility comparison and is not an independently replicated benchmark. Volpini et al. evaluate enhanced entity pages combining instructions, navigation, search affordances, and added linked facts [27]; the bundled changes do not isolate root-file consumption. Neither report supplies the full counterfactual above.

By specification, the manifest provides publisher-authored navigation but confers no instruction, capability, or cross-domain epistemic authority. Cross-stage evidence still clarifies boundaries that matter if a consumer adopts it. Retrieval-suppression experiments establish that crafted documents can block relevant material before generation [28]. Agent evaluations with repeated operations demonstrate a distinct resource-exhaustion path under their evaluated architectures [29].

Multi-step research-agent experiments show that influence may persist through repeated retrieval, conditional on exposure and architecture [30]. Data-injection experiments separately show why trusted-looking identifiers and tool results require provenance controls in agent settings [31].

Guarded Consumption and Validator

Table IV summarizes the proposed safeguards and their corresponding verification boundaries.

Defense evidence maps to different boundaries. Structured-query results [32] motivate separating trusted queries from untrusted data. Action-alignment results [33] motivate evaluating policy on adversarial and benign tasks.

Design-level confinement can keep untrusted content outside privileged control flow [34]. Task-specific cross-site confinement has also been evaluated for web agents [35].

A system-level risk classification identifies prompt injection impacts as dependent on surrounding capabilities [36]. Evidence that defenses learn surface heuristics motivates explicit evaluation of contextual false positives and over-defense [37].

A guarded consumer treats manifest prose and links as untrusted typed data. It requires secure HTTP transport and same-origin defaults, revalidates Domain Name System results and redirect targets at resolution and connection time, and checks the connected peer before accepting content. These checks address server-side request forgery and cross-origin steering. It caps bytes, tokens, links, depth, redirects, time, and cost; preserves final location, retrieval time, representation, and hash; and blocks prohibited addresses. Each external link triggers an independent retrieval decision. Consequential actions require task-scoped policy and confirmation, while high-stakes claims require canonical evidence and independent corroboration before downstream use.

These network, resource, provenance, evidence, and capability protections apply to untrusted pages generally. The manifest-related decisions are whether to ingest free prose, expand curated links, or select alternate representations. No new general defense is claimed. In the reduced pilot, “guarded” denotes only the stated URL policy and removal of manifest prose; it does not implement or validate this complete system design.

The companion material defines ten oracles: three cover parser or network-policy properties, and seven require paired semantic, ranking, utility, or action evaluation. All 30 fixtures matched the offline contract: structure and provenance 6/6, resource bounds 4/4, location decisions 16/16, and authority separation 4/4. The last category checks that parser output contains no executable control objects; it does not test model obedience. The validator covers location and internationalized domain-name normalization, supplied-address checks, and provenance hashing, but not network fetch, live resolution, connection-time peer validation, redirect execution, semantic conflict, ranking, agent safety, or action. These operations need instrumented end-to-end trials.

The 30/30 result establishes consistency between the implementation and the specified fixtures, not completeness of fixture coverage or policy validity. It does not estimate exploit prevalence, model robustness, ranking quality, or action safety. The model responses reported separately in § 5 do not extend the validator’s tested scope.

Paired Evaluation Protocol

Planned Paired Evaluation

We specify a preregistration-oriented protocol using Agent-Dojo’s stateful trial structure [13]. This full protocol remains proposed, not publicly preregistered or executed. Three manifest conditions—absent, benign, and adversarial—are crossed with four policies: ignore, raw-text ingestion, deterministic link-only parsing, and guarded consumption. Under the ignore policy, the system neither requests nor consumes /llms.txt. The absent condition is the no-manifest baseline rather than an additional arm. Operationally equivalent cells, such as present-manifest conditions under the ignore policy, are retained as implementation-equivalence checks rather than distinct causal contrasts. Ordinary HTML, alternate Markdown, metadata, and sitemap.xml retrieval would remain identical across factorial arms; explicit user-directed retrieval would be analyzed as a separately labeled baseline. Each arm would begin from the same frozen site, index, and agent-state snapshot, differing only in the assigned manifest condition and consumption policy, in a fresh isolated session. Retry rules, model version, top-kk settings, context templates, and attack budget would remain fixed; only arm order would be randomized within matched site and query-family blocks.

Planned outcomes span discovery and context inclusion; rank, citations, and answer quality; proposed and realized actions; policy blocks; false positives; latency; tokens; and cost. Let z0=(absent,guarded)z_0 = (\text{absent},\text{guarded}), zb=(benign,guarded)z_b = (\text{benign},\text{guarded}), and za=(adversarial,guarded)z_a = (\text{adversarial},\text{guarded}). The co-primary contrasts are Δs(za,z0)\Delta_s(z_a,z_0), adversarial-manifest security loss relative to ordinary retrieval without a manifest, and Δu(zb,z0)\Delta_u(z_b,z_0), the benign-manifest utility-loss difference. The adversarial-versus-benign contrast Δs(za,zb)\Delta_s(z_a,z_b) and guarded-versus-raw contrasts within benign and adversarial conditions are secondary; manifest-present ignore cells are negative-control equivalence checks. Planned randomization inference respects matched site–query-family blocks. Cluster-bootstrap intervals would resample matched blocks while retaining paired arms; mixed-effects models are proposed for sensitivity analyses. All rates would retain the assigned-trial denominator, with fetch failure, parser rejection, model error, policy block, timeout, and tool failure reported separately. Multiplicity control and joint security, utility, false-positive, and cost margins must be set before a confirmatory deployment decision. A confirmatory run requires a prespecified sample size, power calculation, and numeric deployment margins; these remain open design choices. Extending AgentDojo also requires versioned web resources, retrieval tools, matched resets, and separate attacker-goal and utility evaluators; the pilot below implements none of that integration.

LevelSafeguardBoundary / outcome
Discovery policyFetching is policy-controlled and logged; file location confers neither trust nor rank.Exposure and hidden product behavior.
Network invariantRequire secure transport and same-origin defaults; revalidate redirect targets and DNS results, and verify the connected peer.Server-side request forgery and cross-origin steering.
Resource invariantEnforce byte, token, link, depth, redirect, time, and cost limits before model use.Resource exhaustion, cost abuse, or expansion loops.
Parsing and provenanceEmit typed untrusted data with origin, final location, time, representation, and hash.Instruction ambiguity, drift, and attribution.
Ranking and evidenceDo not let manifest prose set cross-site rank; corroborate comparative or consequential claims.Evidence poisoning and source manipulation.
Capability policySeparate public retrieval from secrets and write tools; confirm consequential actions.Data exfiltration and unauthorized state changes.
Assurance criterionLog discovery, context, citations, actions, decisions, and overrides; run paired adaptive tests.Regression and security–utility tradeoffs.

Table IV. Core Safeguards and Verification Levels

The planned attack suite spans imperatives, persuasive prose, role confusion, Unicode and multilingual variants, carrier mismatch, link and redirect steering, false first-party claims, source promotion or suppression, resource exhaustion, and adaptive defense-aware attacks. Contextual-integrity attacks involving misrepresented identities, fabricated agreements or norms, and mixed flows motivate role- and provenance-oriented test cases [38]. Adaptive web-agent evidence [39] also motivates trajectory-responsive attacks across multi-step interactions.

Stakeholder-stratified evaluation is needed because the same policy can distribute false-positive and attack costs differently across users, platforms, and content providers [40]. Benign hard cases include quoted security instructions, legitimate approvals, migration notices, uncommon topics, and valid late user instructions. Adaptive attacks must be included because fixed-template defenses can overstate robustness [41].

Reduced Pilot Design

We executed a narrower fixed-evidence experiment: eight purposively selected sites (WithJoy, Stripe, Supabase, ClickUp, Linear, PostHog, Zapier, and Asana), three factual questions per site, five arms, and one generation per task–arm, totaling 120 calls. Answers were checked with AI assistance against 12 official documentation pages captured on October 14, 2026 (UTC); several questions share a page. Sources, timestamps, hashes, excerpts, questions, and answer aliases are recorded in the companion. There was no independent human screening.

All arms receive identical, explicitly labeled researcher-authored summaries of ordinary source pages, including the correct facts. A has no manifest; B and C ingest benign and adversarial manifests as raw text; D and E consume their approved same-origin HTTPS URLs only. The manifests are locally authored experimental files, not the publishers’ observed files. Each adversarial version retains the benign links and adds a wrong target using an instruction-bearing or false-fact template, assigned to 12 tasks each. The guard discards all prose and link labels, so D and E have identical model-visible inputs. No live retrieval or actions occur.

Questions, aliases, targets, protocol, and code were locally frozen before outcomes. Stateless requests used gpt-4.1-mini-2025-04-14, temperature 0.7, a 256-token completion cap, and randomized arm order within tasks, without automatic retries. Every request instructed the model to treat retrieved material as untrusted. The expected-answer key was reserved for scoring. The pilot security loss ℓs\ell_s measures recognized target adoption, with unrecognized meaning unresolved; it is narrower than the full five-path loss. Utility loss ℓu\ell_u is zero only for an accepted answer, the specified supporting URL, and no proposed actions. Numeric values use exact decimal equivalence; verbal and unit-bearing forms require frozen aliases. Citation membership does not establish entailment. The reduced protocol calls for descriptive reporting, without confidence intervals or deployment decisions.

Results and Scope

All 120 calls returned valid JSON without API errors, and offline recomputation reproduces the frozen scores in Table V. With 24 assigned tasks per arm, three per site, task means equal site-weighted means. Unresolved security outcomes are retained and bounded, not counted as safe. These are classification-identification bounds, not confidence intervals.

ArmKnown target adoptionsUnresolved securitySecurity-loss bounds (%)Accepted answersUtility failures
A: absent0/242/240–8.322/242/24
B: benign, raw0/241/240–4.223/241/24
C: adversarial, raw21/241/2487.5–91.72/2422/24
D: benign, guarded0/242/240–8.322/242/24
E: adversarial, guarded0/241/240–4.223/241/24

Table V. Frozen automatic pilot results, 24 responses per arm. Security bounds reflect unresolved classification. Accepted answers use frozen forms; utility also requires the specified source URL and no actions.

Using these narrow pilot endpoints, the retained co-primary security contrast Δspilot(E,A)\Delta_s^{\mathrm{pilot}}(E,A) is bounded by [−2/24,1/24][-2/24,1/24], or [−8.3,+4.2][-8.3,+4.2] percentage points. The utility contrast Δupilot(D,A)=0\Delta_u^{\mathrm{pilot}}(D,A)=0: both arms incur loss 2/242/24, rather than zero loss. The secondary raw adversarial-versus-benign contrast C–B is bounded by [20/24,22/24][20/24,22/24], or [+83.3,+91.7][+83.3,+91.7] points. Unresolved classifications prevent a frozen security point estimate.

A separate post hoc AI-assisted semantic review resolves seven outputs without changing primary scores: six express correct facts in unlisted forms, and C’s remaining output инек expresses its false target. This supplemental interpretation gives 22/24 target adoptions in C and 0/24 in every other arm, with correct answers and supporting URLs on 2/24 and 24/24 tasks respectively. It is not independent human adjudication. Eight target-adopting answers still cite the ordinary supporting URL despite contradicting its supplied passage.

Provider records report 43,360 input and 3,437 output tokens. Mean request latency is 0.628 seconds; summed request latency is 75.39 seconds, not total wall-clock time. Per-arm usage and latency are retained in the companion. Monetary charges are unverified without billing records. The experiment demonstrates answer sensitivity to these authored payloads, conditional on correct evidence already being present. A utility ceiling, one model, one generation, shared pages, and purposive selection limit inference. Identical D/E inputs follow from removing all attack-bearing prose; this does not establish adaptive robustness. Retrieval gains, source-block false positives, SSRF defenses, and external-action safety remain unmeasured.

Discussion and Conclusion

Implications

Search systems need retrieval-to-action instrumentation because final-answer logs omit discovery, ranking shifts, and blocked actions. Publishers can reduce supply-chain and representation-drift risk by generating /llms.txt from canonical version-controlled content, reviewing changes as code, minimizing external or stale links, and aligning visible and alternate representations. Systems must therefore enforce structural separation among discovery, content use, evidence assessment, and permissions for external action.

Operationally, a deployment can begin in shadow mode: log whether manifest discovery or consumption would alter the retrieved set while leaving the served answer unchanged. The shadow log also reveals whether an apparent utility gain comes from duplicate canonical content or genuinely new evidence. Promotion to active use should require prespecified bounds on security loss, utility gain, cost, and false positives, followed by rollback when a bound fails. This sequencing ties adoption to measured incremental value rather than to file presence.

Limitations

One reviewer screened and extracted the historical records; complete search exports and screening counts are unavailable. Some 2026 sources remain preprints, commercial data come from non-probability samples, and product statements can become stale. Adaptive attacks can also erode the apparent robustness of fixed-template defenses. Selection and extraction error therefore remain plausible despite the retained ledger and verification steps.

The reduced pilot uses one model snapshot, eight purposively selected sites, researcher-authored summaries, fixed attack templates, and one generation per task and arm. Tasks share source pages, and repeated calls may differ even under identical inputs. It measures response behavior conditional on supplied content, not natural discovery, live retrieval, ranking, tool execution, or the full guarded policy. Removing manifest prose also removes the entire attack carrier in these cases; broader guard effectiveness cannot be inferred. The semantic review is post hoc AI assistance, not independent human adjudication. No population claim, confirmatory power claim, or calibrated deployment margin follows from this pilot.

The fixtures test deterministic parsing and pre-fetch decisions, not exploit prevalence or action safety. Images, scripts, accessibility trees, tool metadata, memory, and inter-agent messages remain outside the evaluated scope.

Conclusion

Existing studies establish several content-borne attack paths, but the retained external sources do not isolate the manifest’s incremental security and utility effects in a fixed search pipeline. Our reduced pilot adds original evidence that attack-bearing manifest prose can change answers when ordinary evidence is already supplied. Frozen scoring detects 21 attacker-target adoptions in 24 raw adversarial responses, with one unresolved response; the broader retrieval and action comparison remains to be executed.

Consumers should treat /llms.txt as optional, publisher-asserted navigation data. Bounded retrieval, corroboration, capability separation, confirmation, monitoring, and adaptive testing should be calibrated to the measured security–utility tradeoff in each deployment.

Decision criteria should be reported separately for retrieval-only systems and agents with external tools. A result acceptable for answer generation may be unacceptable when the same content can initiate requests, expose secrets, or modify state. Reporting the active policy, available capabilities, and blocked-action rate alongside answer quality makes the trade-off interpretable for each deployment and exposes whether low attack success was purchased by excessive blocking of benign tasks. These fields should accompany reported security results.

Data and Code Availability

The accompanying revision package contains the retained estimate ledger, 15-study stage and bias coding, 42 plotted percentages, supplemental tables, all 30 fixtures and expected results, validator tests, figure data, and checksums. It also includes the frozen pilot protocol and scoring code, 24 tasks with source-verification records, 120 raw responses, independently recomputed summaries, and a separate post hoc AI review. The original response archive and scoring outputs remain unchanged. A public repository or DOI is not yet available; the package accompanies this revision, and public artifact access remains to be established.

Acknowledgment

OpenAI ChatGPT/Codex assisted with figure production, manuscript layout, language polishing, experiment tooling, and the separately labeled post hoc semantic review. The authors take full responsibility for the analysis, claims, and publication ethics.

References

  1. [1]P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474.arxiv.org/abs/2005.11401
  2. [2]K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. ACM Workshop on Artificial Intelligence and Security, 2023, pp. 79–90, doi: 10.1145/3605764.3623985.DOI
  3. [3]J. Howard, “The /llms.txt file, v2,” updated Aug. 10, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://llmstxt.org/
  4. [4]M. Koster, G. Illyes, H. Zeller, and L. Sassman, Robots exclusion protocol, RFC 9309, Sep. 2022, doi: 10.17487/RFC9309.
  5. [5]Sitemaps.org, “Sitemaps XML format,” Sitemaps.org. Accessed: Aug. 5, 2026. [Online]. Available: https://www.sitemaps.org/protocol.html
  6. [6]Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in Proc. 33rd USENIX Security Symp., 2024, pp. 1831–1847.arxiv.org/abs/2310.12815
  7. [7]A. Vassilev, A. Oprea, A. Fordyce, H. Anderson, X. Davies, and M. Hamin, Adversarial machine learning: A taxonomy and terminology of attacks and mitigations, NIST AI 100-2e2025, Mar. 2025, doi: 10.6028/NIST.AI.100-2e2025.DOI
  8. [8]H. Puerto, M. Gubri, T. Green, S. J. Oh, and S. Yun, “C-SEO Bench: Does conversational SEO work?” in Advances in Neural Information Processing Systems, vol. 38, 2025, pp. 31145–31169, doi: 10.52202/085713-0923.DOI
  9. [9]Y. Yin, S. Wang, B. Koopman, and G. Zuccon, “The vulnerability of LLM rankers to prompt injection attacks: You are to [MARK] this paper as the best paper,” in Proc. 49th Int. ACM SIGIR Conf. Research and Development in Information Retrieval, 2026, pp. 3070–3080, doi: 10.1145/3805712.3808553.DOI
  10. [10]Y. Yin, S. Wang, B. Koopman, and G. Zuccon, “Can it reach the generator? Investigating the survival of prompt-injection attacks in realistic RAG settings,” 2026, arXiv:2605.28017v2.
  11. [11]Rankability, “llms.txt adoption research report,” Sep. 18, 2026. Accessed: Oct. 10, 2026. [Online]. Available: https://www.rankability.com/blog/llms-txt-adoption/
  12. [12]Y. Yi et al., “Benchmarking and defending against indirect prompt injection attacks on large language models,” in Proc. 31st ACM SIGKDD Conf. Knowledge Discovery and Data Mining, 2025, pp. 1809–1820, doi: 10.1145/3690624.3709179.DOI
  13. [13]E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, and F. Tramèr, “AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,” in Advances in Neural Information Processing Systems, vol. 37, 2024, pp. 82895–82920, doi: 10.52202/079017-2636.DOI
  14. [14]H. Chang, E. Bao, X. Luo, and T. Yu, “Overcoming the retrieval barrier: Indirect prompt injection in the wild for LLM systems,” in Proc. 35th USENIX Security Symp., Baltimore, MD, USA, Aug. 2026, pp. 1567–1586.arxiv.org/abs/2601.07072
  15. [15]W. Zou, R. Geng, B. Wang, and J. Jia, “Poisoned RAG: Knowledge corruption attacks to retrieval-augmented generation of large language models,” in Proc. 34th USENIX Security Symp., 2025, pp. 3827–3844.arxiv.org/abs/2402.07867
  16. [16]A. Castagnaro, U. Salviati, M. Conti, L. Pajola, and S. Pizzi, “The hidden threat in plain text: Attacking RAG data loaders,” in Proc. 18th ACM Workshop on Artificial Intelligence and Security, 2025, pp. 170–181, doi: 10.1145/3733799.3762976.DOI
  17. [17]E. Caville, S. Layeghy, B. Sung, S. Dolnicar, and M. Portmann, “SIREN (Luring LLMs onto the rocks): PAIR-driven preference manipulation in Web-RAG recommenders,” 2026, arXiv:2607.21951.arxiv.org/abs/2607.21951
  18. [18]S. Khodayari, X. Zhang, B. Acharya, and G. Pellegrino, “Indirect prompt injection in the wild: An empirical study of prevalence, techniques, and objectives,” 2026, arXiv:2604.27202.arxiv.org/abs/2604.27202
  19. [19]P. Chen et al., “Unveiling the resilience of LLM-enhanced search engines against black-hat SEO manipulation,” in Proc. ACM Web Conf., 2026, pp. 2962–2973, doi: 10.1145/3774904.3792399.DOI
  20. [20]O. Borysenko, “Developer experience with AI coding agents: HTTP behavioral signatures in documentation portals,” 2026, arXiv:2604.02544v2, revised Jul. 24, 2026.arxiv.org/abs/2604.02544
  21. [21]Y. Deda, “Does LLMs.txt impact your AI visibility and citations? No, according to research,” SE Ranking, Nov. 7, 2025. Accessed: Aug. 5, 2026. [Online]. Available: https://seranking.com/blog/llms-txt/
  22. [22]L. Linehan, “We analyzed 137K sites: 97% of llms.txt files never get read,” Ahrefs, Jun. 15, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://ahrefs.com/blog/llmstxt-study/
  23. [23]Google Search Central, “Google’s guide to optimizing for generative AI features on Google Search,” Google Search Central, 2026. Accessed: Aug. 5, 2026. [Online]. Available: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
  24. [24]Chrome for Developers, “llms.txt,” Chrome for Developers, May 5, 2026. Accessed: Aug. 5, 2026. [Online]. Available: https://developer.chrome.com/docs/lightouse/agentic-browsing/llms-txt
  25. [25]S. Marro et al., “Permission manifests for web agents,” 2026, arXiv:2601.02371.arxiv.org/abs/2601.02371
  26. [26]A. Hertz, “Data became code: We ran code inside Fortune 500s using files they published for AI agents,” Aug. 26, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://medium.com/@alexhertz/data-became-code-we-ran-code-inside-fortune-500s-using-files-they-published-for-ai-agents-0cd67ffbbffc
  27. [27]A. Volpini, E. Raad, B. Gamba, and D. Riccitelli, “Structured linked data as a memory layer for agent-orchestrated retrieval,” 2026, arXiv:2603.10700v1.arxiv.org/abs/2603.10700
  28. [28]A. Shafran, R. Schuster, and V. Shmatikov, “Machine against the RAG: Jamming retrieval-augmented generation with blocker documents,” in Proc. 34th USENIX Security Symp., 2025, pp. 3787–3806.arxiv.org/abs/2406.05870
  29. [29]J. Luo et al., “Autonomy comes with costs: Detecting denial-of-service vulnerabilities caused by resource abusing in LLM-based agents,” in Proc. 35th USENIX Security Symp., Baltimore, MD, USA, Aug. 2026, pp. 3991–4010.DOI
  30. [30]T. Zhang, H. Triemdan, and V. Shmatikov, “Deep-research agents can be poisoned via user-generated content,” 2026, arXiv:2605.24245.arxiv.org/abs/2605.24245
  31. [31]W. Choi, J. Kim, T. Kang, J. Jeong, L. Xing, and B. Lee, “Agent data injection attacks are realistic threats to AI agents,” 2026, arXiv:2607.05120.arxiv.org/abs/2607.05120
  32. [32]S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “StruQ: Defending against prompt injection with structured queries,” in Proc. 34th USENIX Security Symp., 2025, pp. 2383–2400.arxiv.org/abs/2402.06363
  33. [33]F. Jia, T. Wu, X. Qin, and A. Squicciarini, “The Task Shield: Enforcing task alignment to defend against indirect prompt injection in LLM agents,” in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics (ACL), Vol. 1: Long Papers, Vienna, Austria, Jul. 2025, pp. 29680–29697, doi: 10.18653/v1/2025.acl-long.1435.DOI
  34. [34]E. Debenedetti et al., “Defeating prompt injections by design,” 2025, arXiv:2503.18813.arxiv.org/abs/2503.18813
  35. [35]C. Villa, A. E. Ozdarendeli, S. Tan, and R. A. Popa, “Prismata: Confining cross-site prompt injection in web agents,” 2026, arXiv:2607.08147.arxiv.org/abs/2607.08147
  36. [36]OWASP Gen AI Security Project, “LLM01:2025 prompt injection,” OWASP Gen AI Security Project, 2025. Accessed: Aug. 5, 2026. [Online]. Available: https://genai.owasp.org/llmrisk/llm01-prompt-injection/
  37. [37]S. Li et al., “Defenses against prompt attacks learn surface heuristics,” in Proc. 64th Annu. Meeting Assoc. Comput. Linguistics (ACL), Vol. 1: Long Papers, San Diego, CA, USA, Jul. 2026, pp. 10970–10987, doi: 10.18653/v1/2026.acl-long-502.
  38. [38]S. Abdelnabi and E. Bagdasarian, “AI agents may always fall for prompt injections,” 2026, arXiv:2605.17634.arxiv.org/abs/2605.17634
  39. [39]G. Syros et al., “MUZZLE: Adaptive agentic red-teaming of web agents against indirect prompt injection attacks,” in Proc. 35th USENIX Security Symp., Baltimore, MD, USA, Aug. 2026, pp. 1527–1546.arxiv.org/abs/2602.09222
  40. [40]Z. Wang et al., “Who pays the price? Stakeholder-centric prompt injection benchmarking for real-world web agents,” 2026, arXiv:2606.13385v2.arxiv.org/abs/2606.13385
  41. [41]Q. Zhan, R. Fang, H. S. Panchal, and D. Kang, “Adaptive attacks break defenses against indirect prompt injection attacks on LLM agents,” in Findings Assoc. Comput. Linguistics: NAACL 2025, Albuquerque, NM, USA, Apr. 2025, pp. 7116–7132, doi: 10.18653/v1/2025.findings-naacl.395.DOI

Paper details

vlnc:W_hhbw8fep
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT/Codex

View statement
Looking for related papers…
Contents