betaFeedback

WorksComputer Science

Can LLMs Give Good Product Advice? A Framework for Diagnosing Failures

Abstract

Evaluating product advice from a language model requires assessing whether the recommendation meets the user's needs and identifying where the model falls short when it does not. However, matching a product mentioned in an online reply cannot answer either question on its own: several products may be suitable, and the recorded answer may itself be poor advice. When shown a community-listed alternative alongside their own sampled car recommendation, six models prefer the alternative in 1.9 to 6.5 percent of eligible disagreements, consistently across both source-blind presentation orders. We propose a diagnostic framework that separates the user's stated needs, the model's added assumptions, candidate generation, filtering, and final choice, with 19 possible failure modes and the evidence needed to assess them. We apply this framework to product-advice data and to a separate product-selection task with externally assigned relevance labels. The experiments distinguish naming a community-listed product from selecting it and show how answer extraction and candidate-list construction affect the outcome.

Introduction

Consider a parent asking a language model for a family-car recommendation. The car must seat six people and cost at most $20,000; either an SUV or a minivan is acceptable, and lower running costs are preferred. Recommending a five-seat car is a clear error; choosing one suitable minivan over another may be a reasonable trade-off. The task is to give advice that fits the request, not necessarily to name one uniquely correct car.

Our goal is to answer two questions about product advice from a large language model (LLM): can it give objectively good advice, and where are its weaknesses when it falls short? By good advice we mean a recommendation that satisfies the user’s firm requirements, rests on accurate product facts, and makes a defensible trade-off among the user’s preferences. Seating capacity can be checked directly; other judgments depend on priorities that may leave several acceptable choices.

We diagnose advice by separating its stages, from interpreting the request to selecting a product (Figure 1). Explicitly listed products form the candidate pool. Changing “SUV or minivan” to “SUV only” calls for a different correction from choosing a five-seat car after correctly recording the six-seat requirement. Our framework defines 19 possible failure modes and the evidence needed to assess each. It examines the request and the written response, so it does not require access to a model’s internal computation.

Structured reasoning process for product advice, showing Request, Evidence, Criteria, Options, and Choice connected by arrows

Figure 1. A structured reasoning process for product advice. The request and background knowledge yield facts and beliefs that inform constraints and preferences. Constraints filter the candidate pool; preferences guide ranking to select a recommendation. This flow helps locate shortcomings in LLM advice.

Assessing quality requires adequate product evidence. Human responses to the same request provide a narrower comparison: we call a product name taken from such a response and used as a comparison target a reference. If a reply names a Toyota Sienna, a model’s Honda Odyssey is a mismatch; judging its suitability still requires vehicle facts and the user’s needs. Community references are neither exhaustive nor guaranteed good recommendations. Advice benchmarks address multiple perspectives through community preferences or expert rubrics ([14, 15, 4]), and prior work distinguishes matching a recorded product from being useful to a user ([33]).

We analyze saved recommendations from seven models. Listing options tests whether a reference enters the candidate pool; choosing from a supplied list tests selection with those options available. We compare choices with and without the user’s research dataset with relevance labels. Naming a reference does not ensure selecting it, and measured agreement depends on answer extraction and list construction. These comparisons locate behavioral differences; judging quality requires checking whether the products satisfy the request. We propose a procedure for assessing the quality and self-consistency of LLM product advice and locating weaknesses, with reference-based tests when direct quality judgments are unavailable (Section 3). We define a structured representation of advice and 19 possible failure modes, specifying the evidence needed to distinguish them (Section 4). We apply these measurements to saved product recommendations and supplied-choice experiments, identifying both selection behavior and evaluation artifacts that affect its interpretation (Sections 3 and 5).

Related work

Advice and reference validity. Conversational recommenders elicit preferences [11]; LLMs support zero-shot recommendation [8]. Evaluation combines component tests and user judgments [10], with interactive evaluation exposing limits of recorded-item matching [33]. AdvisorQA and ComPO use community feedback and preferences [14, 15]; ShoppingReasoningBench and ShoppingComp use expert-authored tasks and rubrics [4, 31]. Complementing ShoppingReasoningBench’s taxonomy and expert-validated rubric evaluation, we use coverage and selection stages to examine community-reference and extraction effects, candidate-construction artifacts, and generation–comparison disagreement.

Constraints, candidates, and ranking. Constraint-based recommenders match user requirements to product properties [5]. Candidate generation and ranking are established recommendation stages [2], with retrieval bottlenecks in LLM systems [3]. LLM recommendations also exhibit position and popularity effects [9, 17]. ESCI supplies external product-search relevance labels [28]. We connect these stages to the explicit constraints and evidence needed to assess a product choice.

Behavioral and factual assessment. Check-List tests capabilities beyond aggregate accuracy [29]; FActScore checks atomic claims against reliable sources [23]. Process supervision evaluates intermediate steps [18], but written chains of thought can misrepresent what drives a response [16, 32]. We use observable structure to select checks and test interventions, without treating the response as a trace of internal computation.

Diagnosing advice quality and reference agreement

The diagnostic task takes a user’s request and a model’s response as inputs. We want to establish whether the advice meets the user’s needs and identify where it falls short. We first specify the evidence available for judging quality, then inspect the response’s components, and finally test a suspected weakness by changing the relevant part of the task.

Advice quality and reference agreement

For the family-car request, checking the recommended configuration’s seating capacity and price directly tests the firm requirements. Assessing the running-cost trade-off also requires relevant product facts and the user’s priorities. If the configuration or its price is unknown, the corresponding requirement cannot yet be judged. This is the evidence needed for a quality assessment, regardless of whether anyone else named the same car.

When such judgments are unavailable, a reference provides a narrower comparison. In our community data, it is a product name extracted from an online reply to the same request; the recorded names form the reference set SS. Top-reference agreement measures how often the final pick matches the highest-weighted entry; set-reference agreement accepts any entry. For example, if replies name a Toyota Sienna and a Honda Odyssey, with the Sienna ranked first, choosing either counts toward set agreement, but only the Sienna counts toward top-reference agreement.

Reporting both shows how reference choice affects agreement with the same model outputs. Sparse feedback may not establish a reliable preference for the top entry, while a single mention does not establish suitability. Section 3.3.2 reports both scores alongside a direct comparison between the model’s own recommendation and a community-listed alternative.

Experimental setup

Models and data. We analyze saved outputs from GPT-5, Claude Opus, Llama-70B, Qwen-80B, Nemotron-49B, Llama-8B, and Mistral-14B. The evidence. Appendix § 12 details the product-matching and answer-extraction rules, sample aggregation, experimental comparisons, and uncertainty estimation used in the retrospective reanalysis.

Reddit generation corpus contains 800 car, 800 laptop, and 226 headphone requests from early 2019. The five-domain choice probe additionally uses Stack Exchange Software Hardware Recommendations. The five-domain supplied-choice experiments cover four models. The main enumeration table covers Llama-70B; supplementary analyses add six-model coverage and GPT-5.4 generated-pool choice results. Appendix § 10 gives experiment-specific counts and Appendix § 11 gives model identifiers and prompts. The main text emphasizes car-domain diagnostics while retaining cross-domain summaries and the ESCI product-search study. Further results and experimental details are reported in the corresponding appendices.

Dataset statistics. Table 1 summarizes the requests used in the experiments. The Reddit rows describe the shared seven-model generation cohorts; each model produced three responses per request. The Stack Exchange rows describe the software and hardware cohorts used in § 5.2. Community replies are counted separately from generated responses: they include all stored top-level comments on Reddit and answers on Stack Exchange, even these without recommendations.

DomainQueriesNamesRepliesScore
Cars8001,68152
Laptops80073741
Headphones22648131
SE software2,4452,34911
SE hardware1,2661,39711

Table 1. Dataset statistics. Names: distinct reference strings; Replies: median replies/query; Score: median stored reply score. Full statistics: Appendix § 10.

The median request has five, four, and three stored top-level replies for cars, laptops, and headphones, respectively, versus one answer in each Stack Exchange domain. Median reply scores are only one or two. These counts describe the amount of observed community feedback.

Community references. The Reddit records contain product strings extracted from top-level comments, weighted by max(comment score, 1) per mention. This process can include comparisons, negative mentions, and malformed names. We therefore call the raw records reference mentions. Stack Exchange references use consensus product extraction. Catalog validation checks product identity; endorsement and suitability require separate evidence. Appendix § 12 details the product-matching and answer-extraction rules, sample aggregation, experimental comparisons, and uncertainty estimation used in the retrospective reanalysis.

Experiments

We first test whether recommendations satisfy explicit user constraints, independently of community references. We then test whether models prefer a supplied community alternative to their own initial recommendation and examine how reference granularity affects agreement scores.

Objective constraint satisfaction

The saved free-form car corpus contains responses from all seven models: three responses to each of 800 shared requests. The audit covers 149 requests with explicit drivetrain or transmission requirements, totaling 3,129 responses. The check uses constraints extracted from the request text shown to the models and a shared vehicle-fact table, without using the community recommendation as ground truth. Deterministic rules identify candidate requirements and exclusions, retaining their source spans for review. Product facts are reused across recommendations of the same vehicle.

We checked primary recommendations against drivetrain and transmission requirements using saved US Environmental Protection Agency (EPA) records, Canadian fuel-consumption records, and manufacturer specifications. Ambiguous vehicle identities, configurations, or unsupported facts remain unknown; unverified catalog attributes are not treated as ground truth. Shared identity rules account for the stated year, variant, and purchase market. We reviewed all surfaced candidate conflicts in their full responses, distinguishing un retracted violations from disclosed trade-offs and corrected recommendations. Support concerns only the audited configuration fields; budget and other attributes remain outside this check. Appendix § 12 details the evidence, review, and uncertainty estimates.

Results. Table 2 records 34 confirmed conflicts and 356 supported primary configurations; 2,739 answers remain in the other/unknown category. Mistral-14B has 17 confirmed conflicts among 447 answers (3.8%); the other open models have three to six each. No conflict was confirmed for GPT or Claude under this audit. A separate 44-query headphone audit finds 795 supported connection checks, no confirmed conflicts, and 129 unknowns across 924 answers (Appendix § 12).

ModelSupportedConflicts (%)Other
GPT-5800 (0.0)367
Claude Opus440 (0.0)403
Llama-70B626 (1.3)379
Qwen-80B573 (0.7)387
Nemotron-49B593 (0.7)385
Llama-8B365 (1.1)406
Mistral-14B1817 (3.8)412

Table 2. Conservative configuration checks on car advice. Each row retains 447 answers (149 requests, three generations). Supported counts have evidence for the primary recommendation’s audited drivetrain/transmission. Conflicts are counts (%) of 447. Other includes unresolved and separately classified cases. Conflicts are minimum detected violations; zeros do not establish error-free advice. Intervals: Table 9.

Discussion. Even a narrow specification check identifies concrete failures to meet explicit requirements. However, exact product identity and market-specific evidence limit how many answers can be assessed. The supported and conflict cases cover only 390 of 3,129 car answers, with no query/sample position assessable for all seven models. The counts therefore establish minimum detected violations, while a comparative compliance ranking requires broader evidence coverage. Constraint compliance does not capture every preference or trade-off. We next examine what community references contribute to the evaluation and what their limitations are.

Comparing to the reference

A recommendation can differ from a community reference without being worse. We ask whether the same model, shown its own recommendation and a community-listed alternative, prefers the alternative for the original request. This measures agreement between initial generation and comparative judgment under different prompts.

Source-blind comparison. For each of 150 shared car requests, we fix a community-listed alternative before generating three initial recommendations per model. Whenever a valid recommendation differs from that alternative, we ask the same model to compare the two in a fresh context, using the full request but omitting the initial explanation and both options’ sources. We present the pair in both orders, given reported position biases in LLM judging ([36]), allowing ties or abstention rather than forcing a choice. Appendix § 13 provides the alternative-selection rule, prompts, and response formats.

Results. Table 3 shows that the community-listed alternative wins in both orders for 1.9–6.5% of eligible initial recommendations across all six models with completed runs, after the product-identity sensitivity check. GPT-5.4 and Llama-70B each have 26 such trials (6.5% and 6.4%), while Claude, Qwen, and Nemotron have rates of 4.1–4.5%. Ministral-14B was unavailable on the tested endpoints and is not included. Temperature-zero initial generation yields 5.3% for GPT and 3.1% for Claude (Appendix § 13.§ 1).

ModelCountRate [95% CI]
GPT-5.426/4026.5 [3.0, 10.3]
Claude Opus 4.816/3924.1 [1.5, 7.3]
Llama-70B26/4056.4 [2.9, 10.6]
Qwen-80B18/4004.5 [2.2, 7.4]
Nemotron-49B13/3174.1 [1.2, 7.7]
Llama-8B7/3671.9 [0.5, 3.6]

Table 3. Preference for the community alternative. Counts: community choices in both source-blind orders / eligible trials, from three initial generations on 150 shared car requests per model. Rates are percentages with 95% query-cluster bootstrap intervals. Eligibility and identity exclusions: Appendix § 13.

A low community-preference rate can coexist with unstable choices. Llama-8B consistently favors its original recommendation in only 59.1% of eligible trials, compared with 78.5–85.8% for the other models. In 94/367 trials (25.6%), it instead chooses the first-listed option in both presentations, changing its product choice when the order changes. Its low rate of community preference (1.9%) thus does not establish stronger initial advice.

A restricted laptop replication yields 44/400 both-order preferences for GPT-5.4 (11.0%) and 41/403 for Opus 4.8 (10.2%) after the identity sensitivity check (Appendix § 13.§ 2). Its references are community mentions concentrated in MSI and Dell; cohort differences preclude a controlled comparison with cars. The rates describe comparative preferences on each model’s eligible recommendations. Assessing advice quality requires judging the products against the user’s needs.

Agreement with stored references. As a separate descriptive comparison, Table 4 reports how often the seven models’ saved recommendations match the top reference or any name in the reference set, as defined in § 3.1. With outputs and matcher fixed, accepting any reference entry raises agreement throughout, but all-entry agreement remains only 14.7–27.4% across models and domains. Several mechanisms could contribute: model failures (§ 4), community biases toward popular or familiar products and favored brands, and different weightings of the user’s preferences. For example, a request for a “large and cheap” car does not specify the relative importance of space and price. The community may favor a roomier, costlier car, while the model favors a smaller, cheaper one, even when both satisfy the hard constraints. This trade-off mismatch need not indicate an error by either. Incomplete references and extraction errors can lower measured agreement. The scores alone cannot distinguish these explanations. Appendix § 14 adds a software extraction-scope sensitivity.

ModelCarsLaptopsHeadphones
GPT-56.8 / 24.810.3 / 26.06.6 / 22.6
Claude Opus6.5 / 24.09.6 / 24.87.2 / 24.6
Llama-70B6.0 / 24.510.0 / 27.48.3 / 21.5
Qwen-80B5.2 / 23.25.0 / 19.05.5 / 17.1
Nemotron-49B6.0 / 22.08.6 / 23.34.9 / 17.7
Llama-8B5.6 / 21.39.0 / 24.05.2 / 16.1
Mistral-14B5.5 / 20.85.1 / 18.43.7 / 14.7
Queries800800226

Table 4. Reference granularity changes agreement. Top-entry / all-entry mention agreement (%), using identical outputs, the same matcher, and all three samples. Missing picks remain in the denominator. Query-bootstrap intervals are in Appendix § 14.

A structured framework for advice

Evidence, criteria, and decisions

We first distinguish the kinds of information needed to assess a recommendation. In the family-car example, six seats are required, lower running costs are preferred, and an unrequested assumption about driving in snow is neither. Keeping these roles separate makes it possible to ask where an unsupported restriction entered the answer. A statement supplied by the user, an assumption added by the model, and a product choice must be checked against different evidence.

We represent a response to request qq by A(q)=(F,B,C,P,I,Ifiltered,R)A(q) = (F, B, C, P, I, I_{\mathrm{filtered}}, R). FF contains facts explicitly stated or directly supported by the request. BB contains additional beliefs, separating inferences BinfB_{\mathrm{inf}} with evidence in FF from assumptions BassumeB_{\mathrm{assume}} introduced as background priors. CC contains hard constraints; PP contains soft preferences and their stated priorities. II is an explicit candidate set, IfilteredI_{\mathrm{filtered}} the subset retained after applying CC, and RR a ranking or recommendation over that subset. The final pick is p(q)p(q).

This decomposition describes the reasoning tasks involved in giving product advice: understand the request, identify relevant products, apply the requirements, and weigh the alternatives. Making these steps explicit lets us assess different capabilities separately. For example, a model may preserve the user’s requirements yet fail to name a suitable product, or name one but reject it when choosing. This emphasis on explicit intermediate structure is related to chain-of-thought prompting ([34]) and decomposed prompting ([13]), which organize problem solving into reasoning steps and modular subtasks, respectively.

The distinction between inference and assumption prevents a common logical substitution. “Winters are harsh” supports asking about weather-related needs; it does not establish that all-wheel drive is a hard requirement. Likewise, “SUV or minivan” should not silently become “SUV only.” These are interpretable failures even when the request admits several final products. The framework makes their evidential requirements explicit. The representation is limited to observable content: the model may consider additional candidates, preferences, or priors without stating them. Prompts requesting structured output can expose these fields, but they may change the recommendation.

Failure taxonomy

The taxonomy contains 19 modes in six categories: fact extraction, belief attribution, constraint and preference formation, candidate construction, filtering, and ranking. Appendix § 15 gives the complete inventory and the evidence each diagnosis requires. For example, filtering a satisfying option out and retaining a violating option are distinct from failing to generate either option in the first place.

We implement a prototype set of automated checks, which we call verifiers. The selected implementation combines 15 non-generative heuristics with four generative judge checks. The non-generative checks include lexical coverage, citation-tag patterns, constraint patterns, and catalog comparisons. When the evidence required for a check is missing, its outcome is unassessable rather than a model error.

Worked example: a wrong constraint value (3b). Suppose a request requires a vehicle for at least six passengers under $20,000. The response records both requirements correctly in FF, but changes the minimum capacity to five in CC. This is failure 3b: a correctly extracted requirement changes during constraint formation, before candidate generation or filtering. The prototype verifier flags this repairable mistake without deciding which car is best. Appendix G gives the numerical check, its reproduced scores, and its limitations.

The experiments below distinguish the behavior of these checks from the effects of a prompt intervention on the final recommendation. Root-check flags identify responses for inspection; product matches measure agreement with the recorded references. Estimating the prevalence of all 19 failures would additionally require validating the checks against independently assessed responses.

Dependency-guided diagnosis

An upstream error can change the evidence used by later checks: a misrecorded seating requirement can affect both the derived constraint and the filtering decision. Figure 2 organizes the taxonomy into a proposed diagnostic ordering. Its five root checks concern missing facts (1a), invented facts (1b), changed valence or attribution (1c), priors presented as user-derived (2b), and inadequate candidate coverage (4a). After addressing a flagged issue, we revisit the downstream checks using the updated response. The arrows specify this inspection order; a check can be assessed directly whenever its required evidence is available.

Proposed diagnostic dependency graph for the 19 failure modes

Figure 2. Proposed diagnostic dependency graph for the 19 failure modes. The five highlighted root nodes are entry checks; arrows indicate downstream checks to revisit after an intervention. Codes follow Appendix F.

Experiments

Root-check profiles. We revisit the archived root-check experiment on structured car advice. Table 5 uses the same 100 requests across all seven models and the first saved generation for each request. The query-side checks use the title and the 1,800-character body available to the original generation prompt. We report each verifier’s explicit firing

Model1a1b1c2b4a
GPT-545737415
Claude Opus5185302
Llama-70B70593021
Qwen-80B638842339
Nemotron-49B679621925
Llama-8B707121420
Mistral-14B579933542

Table 5. Root-check flags on structured car advice. Counts out of 100 shared requests per model, one generation each. Checks measure lexical fact coverage (1a), token/number support (1b), polarity patterns (1c), assumption tags (2b), and candidate-count shortfalls (4a).

flag, since the checks use different score thresholds. Appendix G gives the operational rules and the reconciliation with the archived scores. The profiles vary by check. The assumption-tag check (2b) never fires for Claude or Llama-70B, but fires on 41 GPT and 35 Mistral responses. The candidate-count check (4a) flags two Claude responses and 42 Mistral responses. By contrast, lexical fact checks fire frequently across models: 45–70 responses for 1a and 59–99 for 1b. Paraphrases can trigger these lexical checks, while correct tagging alone does not establish that a belief is justified. The profiles identify different responses and fields to inspect. Appendix G compares two checks with exploratory assistant annotations.

Fact anchoring. Does supplying the fact list improve the final recommendation? An archived Llama-70B experiment prefills FF with request sentences on 400 paired car requests, with three generations per condition. Reference agreement is 22.9% without anchoring and 22.0% with it (−0.9 points; 95% paired query-bootstrap interval [−2.8, +0.8]). An input-length-matched sensitivity also finds no improvement. The model can rewrite the supplied list, so the targeted representation and final answer both need checking. Appendix § 16 gives the protocol and controls. The root checks reveal diagnostic profiles across models: Claude has fewer candidate-count flags than Llama-70B but more token-support flags. This variation motivates component-specific follow-up. Fact anchoring also fails to improve reference agreement, highlighting the need to verify both the targeted representation and the downstream effect of an intervention. These findings concern car advice; validated failure rates and differences between domains remain open.

Candidate coverage and selection

Coverage and selection capture two critical failure points: omitting all suitable products from the pool, or failing to choose one when available. Either can leave the user without suitable advice despite correct reasoning elsewhere. We measure these stages against recorded reference products.

Candidate coverage is the fraction of requests whose explicit pool contains at least one reference, addressing failure 4a (inadequate candidate coverage). Conditional selection measures how often a reference is chosen when the pool contains one. For a reference such as the Honda Odyssey, we test its inclusion if absent and inspect filtering (5a–5b) or ranking (6a) if present but unchosen. This follows the candidate-generation and ranking distinction [3]. Diagnosing a failure still requires evidence about product suitability, constraints, and preferences.

For choices restricted to the pool, reference agreement requires both coverage and selection. If choices can name products outside the pool, those matches must be counted separately.

Candidate coverage

Question and test. Does the model generate reference products, and does coverage vary by domain? We supply the request without community product names and ask for three lists of 20 plausible products. We merge names in first-mention order and measure whether a recorded reference appears among the first 10 or 40 candidates. This enumeration experiment includes Llama-70B on

171 car, 176 laptop, and 49 headphone requests. Appendix § 17 gives the protocol and extends coverage to six models on shared subsets of 160, 164, and 49 requests, respectively.

Results. At 40 candidates, reference coverage is 95.5% for laptops, 76.6% for cars, and 57.1% for headphones (Table 6). Thus, even after three enumeration calls, a recorded reference is absent from 42.9% of headphone pools, compared with 4.5% of laptop pools. The six-model appendix analysis also shows higher coverage for laptops than headphones in every model, with different point-estimate orderings across domains. The appendix includes catalog- and source-reference sensitivities. Extending a list necessarily preserves or increases coverage; whether the model then selects a reference is a separate question.

DomainnnCoverage (K=10K = 10)Coverage (K=40K = 40)Selection % (ncn_c)
Cars17159.676.626.5 (131)
Laptops17685.895.547.0 (168)
Headphones4949.057.139.3 (28)

Table 6. Llama-70B coverage and selection. Coverage counts reference-containing pools; selection averages three choices on ncn_c covered queries at K=40K = 40 (invalid: zero). Extensions and controls: Appendix § 17.

Discussion. Why might a product be missing? One possibility is weak or absent product knowledge; another is that the model knows the product but the request does not make it sufficiently likely to be generated. A finite list also excludes lower-ranked products, and the model may judge a community reference unsuitable. Coverage therefore measures what is brought into the explicit pool, not everything the model knows.

An archived open-model likelihood-lift experiment scores reference names with and without the request. In 156 matched Llama-70B car requests, 28/69 omitted target references (40.6%) receive positive lift: context can raise a product’s likelihood without bringing it into the pool. Prompt and serving differences limit this association. Appendix § 17 reports the three-model, five-domain results and controls.

Selection

Question and tests. When a reference is available, how often does the model select it, and how does this vary across models and domains? We first ask Llama-70B to choose three times from each generated pool, measuring agreement only where a reference is present. To compare four models across five advice domains, we also supply one recognizable community reference and four popular nonmatching distractors. We present the same shuffled options with a “best in general” instruction and with the request and a “best for this user” instruction. This controls reference availability and tests the effect of the request-specific prompt; both the request and instruction change. Appendix I reports both conditions and paired intervals.

Results across models and domains. Within its own reference-containing pools, Llama-70B selects a reference on 26.5% of car, 47.0% of laptop, and 39.3% of headphone choices (Table 6). GPT-5.4 selects a reference in 30.9% of covered car choices and 24.0% of covered headphone choices (Appendix § 17); these rates condition on different model-generated pools. Laptop coverage is therefore near-complete, yet selection agrees on fewer than half of eligible choices. In the supplied five-option test, the request-specific prompt increases agreement in all 20 model–domain cells. Agreement is 46.0–62.5% across the Reddit domains, compared with 91.3–96.7% on Stack Exchange (Appendix § 18, Table 29).

ModelCarsLaptopsHeadphonesSE softwareSE hardwareESCI
GPT-559.454.054.096.794.670.2
Claude Opus62.558.551.596.394.473.6
Llama-70B57.060.551.094.491.367.4
Llama-8B56.951.146.093.792.358.6
Metadata control86.264.388.497.397.9—
Queries1,2929021982,4451,266500

Table 29. Selection with the reference supplied. Percentage choosing the community reference with the user’s request (five options), or the Exact-labeled ESCI product (four options). Models share queries within each column. The metadata control uses reconstructed popularity ranks unavailable in the model prompts. The two tasks use different reference criteria and candidate pools.

Interpreting the domain differences. The supplied lists admit a shortcut: the inserted reference is often less popular than the four distractors. Choosing the least-popular option in reconstructed pools, without reading the request, exceeds every model’s point estimate in each domain (Table 29). This control diagnoses the benchmark using construction metadata unavailable in the model prompts. The observed domain differences therefore cannot establish that models handle user preferences better in one domain than another. In a balanced-pair test, most of the real-versus-shuffled request gain comes from transitions between non-selection and choosing the reference (Appendix § 18).

External relevance check. To test selection against assigned relevance labels, we use ESCI product-search queries ([28]). Each prompt contains one Exact-labeled product and three Irrelevant-labeled products, shuffled without labels, with titles truncated to 120 characters. On a custom sample of 500 US queries, Claude and GPT-5 select the Exact product in 73.6% and 70.2% of cases; Llama-70B and Llama-8B score 67.4% and 58.6% (uniform choice: 25%). Thus even the two closed models miss the product on 26.4% and 29.8% of queries. Appendix § 19 gives sampling details, uncertainty estimates, and the Qwen-80B result (67.8%). Truncated titles, label noise, and invalid outputs can contribute to misses, and the saved hit flags do not distinguish them.

Conclusion. High reference coverage does not ensure reference selection. Selection scores vary across models and domains, but the community test’s shortcut limits their interpretation; ESCI provides a separate relevance-label check. The free-form versus enumeration comparison and its extraction audit are retained in Appendix § 17, since their different answer interfaces prevent attributing the score difference to better advice.

Discussion

We investigated product advice from LLMs, distinguishing objective quality from community agreement and decomposing advice into 19 possible failure points. Coverage and selection deserve separate evaluation: either can prevent a suitable product from reaching the final recommendation.

Even frontier models sometimes prefer an alternative to their initial advice: GPT-5.4 and Claude Opus 4.8 favor the community-listed product in some comparisons in both source-blind orders (Table 3). This reveals a mismatch between generation and comparative choice. Root checks flag omitted user facts and unsupported additions, with different profiles across models (Table 5). These flags need inspection because lexical checks can flag valid paraphrases. Supplying a fact list did not improve reference agreement, so repairs must be checked at both the targeted stage and the final answer. Reference coverage and selection vary across domains, with high coverage sometimes accompanied by low selection agreement (Table 6).

The next step is to improve advice through targeted interventions: catalog-backed generation for missing candidates, or better filtering and ranking when suitable candidates are rejected. Validating the diagnostic checks and testing models on verified suitable candidates can help choose the intervention. Its effect should be assessed against verified constraints and independent suitability judgments, using comparable answer interfaces and inference budgets.

Limitations

Reference and extraction validity. Raw community mentions are neither verified endorsements nor an exhaustive set of acceptable answers. Catalog-based extraction misses unlisted products and can select a mentioned alternative instead of the intended recommendation. A prior automated endorsement check on 250 catalog-validated entries does not validate every raw entry used here. Independent human validation remains necessary for claims about acceptable disagreement.

Observable structure and verifier scope. The taxonomy describes the explicit response. The prototype combines heuristics and generative judges and has not been validated as a 19-way detector on representative human-labeled responses. Format restrictions can affect model performance [30]; structured prompting can therefore alter the output being examined. Accordingly, no claim about the dominant failure mode is based on its uncalibrated firing counts.

Temporal and population scope. The main Reddit requests date to 2019, while the models and some catalog entries are newer. The matching reanalysis does not enforce historical availability, so it cannot certify advice that was feasible when the request was posted. The corpora are English-language, platform-specific, and concentrated in a US-oriented product space. Reference extraction also selects for recognizable products; neither the queries nor the chosen models are a random sample of all advice.

Experimental controls and saved outputs. Enumeration uses more calls than direct answering and changes the prompt. Its asymmetric answer extraction prevents treating the laptop score difference as an established recommendation gain. Query lift uses generic popular distractors, admits a popularity-rank shortcut, and changes the task instruction. ESCI uses truncated titles, a custom surface filter, and no same-pool lexical baseline. Its archived hit flags, and the lift records, cannot recover raw parsing failures. These constraints limit causal interpretation while leaving the reported protocol-level measurements reproducible.

The source-blind comparison preserves the request and counterbalances option order, but changes the task from generating an answer to comparing supplied names. Reference names are often less specific than initial recommendations, so year, trim, and configuration information need not be balanced. The identity check removes explicit duplicate or ambiguous pairs. Product facts, historical availability, community endorsement, and the models’ explanations require independent validation before a preference reversal can be interpreted as a correction of bad advice. The identity exclusions are post-hoc and the frozen-cohort result is reported alongside them.

Uncertainty and reproducibility. Query-level intervals do not include generation-seed variance, and multiple exploratory comparisons do not establish a complete model ranking. Model identifiers are recorded, but full served version and timestamp provenance is incomplete for some historical runs. The manuscript combines an offline reanalysis of historical outputs with the separately archived full-request comparison run. Neither supplies independent human judgments of recommendation quality.

Ethical considerations

Public user text. The study uses public Reddit and Stack Exchange requests and responses. The working data contain raw text, which may include identifying details even without dedicated username fields.

Consumer and exposure risks. Community popularity can favor well-known products without satisfying an individual’s budget or needs, and optimizing it can reinforce exposure imbalances.

Compute and assistance. The reported model experiments use inference; enumeration requires generation calls as well as choices. The full-request car comparison contains 4,895 responses for GPT and Claude (September 27, 2026), including reconsideration controls, and 4,834 responses for four open models (October 10). GPT-5.4 car and headphone enumeration/choice and Qwen ESCI use 1,820 responses collected on October 10. Laptop source-blind comparisons, including parser recovery, use 1,254 GPT responses (812,486 tokens) and 1,269 Opus responses (1,317,156 tokens). Balanced-pair selection uses 1,128 GPT/Llama responses (463,570 tokens), 564 Opus responses (371,275 tokens; October 11), and 564 Qwen responses (238,619 tokens; October 12). GPT-5.4 laptop enumeration and choice uses 1,056 responses (798,996 tokens; October 11), and Opus headphone enumeration and choice uses 294 responses (218,820 tokens; October 12). The temperature-zero initial-generation check uses 826 responses (517,099 reported tokens; October 13). Objective and reference audits use saved answers, and the paired transition analysis uses the 2,256 balanced-pair responses. These analyses require no further evaluated-model inference. Generative assistants supported code, analysis, and manuscript revision. The authors remain responsible for checking the final experimental claims, citations, data handling, and submission disclosures.

References

Experiment coverage and data provenance

Table 7 preserves the full dataset statistics summarized in Table 1. Table 8 maps each experiment to its domains, models, and query counts.

DomainRequestsDistinct ref. namesReplies (total)Replies per queryRef. names per queryScore per reply
Cars8001,6815,1955 [3, 7]5 [4, 8]2 [1, 3]
Laptops8007373,0924 [3, 5]4 [3, 5]1 [1, 2]
Headphones2264819683 [2, 6]3 [3, 5]1 [1, 2]
SE software2,4452,3494,6631 [1, 2]1 [1, 2]1 [1, 2]
SE hardware1,2661,3972,0531 [1, 2]1 [1, 1]1 [1, 3]

Table 7. Full dataset statistics. The final three columns give medians [25th, 75th percentiles]. Reference names are case- and whitespace-normalized; aliases and extraction errors remain. Reply scores are stored Reddit or original Stack Exchange scores, excluding accepted-answer bonuses; they are not separate upvote counts.

ExperimentDomainsModelsQueries per cell
Reference-granularity reanalysisCars, laptops, headphonesAll seven800 / 800 / 226
Objective configuration auditCarsAll seven149 shared; 3 saved generations each
Objective connection auditHeadphonesAll seven44 shared; 3 saved generations each
Matched enumeration and choiceCars, laptops, headphonesLlama-70B171 / 176 / 49
Recovered paired coverageCars, laptops, headphonesSix; no Mistral160 / 164 / 49
Generated-pool choiceCars, laptops, headphonesGPT-5.4171 / 176 / 49; plus 171 archived car queries
Generated-pool choiceHeadphonesOpus 4.849; same requests and protocol
Audited hardware enumeration and choiceSE hardwareLlama-70B234 archived; 194 with specific-product references
Teacher-forced likelihood liftAll fiveLlama-8B, Llama-70B, Qwen-80B1,292 / 902 / 198 / 2,445 / 1,266
Likelihood lift joined to pool inclusionCarsLlama-70B artifacts156 matched; 577 archive sensitivity
Query-conditioned supplied choiceAll fiveGPT-5, Claude, two Llamas1,292 / 902 / 198 / 2,445 / 1,266
External-reference choiceESCI US custom sampleGPT-5, Claude, two Llamas, Qwen500
Source-blind own-answer comparisonCars, full requestsSix; Ministral unavailable150 shared; 3 initial generations each
Temperature-zero initial generationCars, full requestsGPT-5.4, Opus 4.8Same 150; 1 initial generation each
Source-blind laptop comparisonRestricted laptop cohortGPT-5.4, Opus 4.8150; 3 initial generations each
Root-check profilesCars, structured adviceAll seven100 shared; first generation
Assistant root-check pilotCars, structured adviceAll seven20 shared; first generation
Software extraction sensitivitySE softwareAll seven400 shared; 3 saved generations each
Headphone source/identity sensitivityHeadphonesSix coverage; three choice runs49 fixed requests
Balanced-pair selectionCarsGPT-5.4, Opus, Llama-70B, Qwen-80B94 shared; 3 conditions, 2 orders
Fact-anchored promptCars, structured adviceLlama-70B400 paired; 361 input-length matched
Raw comment-support countCars reference corpusNo model generation1,465

Table 8. Coverage of the results in this manuscript. Corpus size and evaluated sample size are distinct. The uniform-slot baseline uses the same enumeration pools as its chooser comparison.

Reddit requests come from r/whatcarshouldIbuy, r/SuggestALaptop, and r/HeadphoneAdvice, with the main generation corpus drawn from January–April 2019. Eligibility required a personal recommendation request and at least three weighted raw reference entries. Reference-support and lift analyses use larger available reference corpora where specified. Stack Exchange data were drawn from 2024 dumps. The documented extraction funnel attempted 3,000 software and 2,352 hardware questions and retained 2,445 and 1,266 with a consensus reference. Retention is conditional on successful extraction, and the extractor models overlap with evaluated models.

Reproduction, prompts, and model identifiers

Evaluation procedure. The experiment-specific appendices specify the cohorts, prompts, scoring rules, aggregation, and uncertainty estimates. Agreement and enumeration analyses use saved generations and choices, with query-level resampling that keeps each request’s repeated samples together. The model/domain extensions follow the corresponding protocols, with any changes and remaining validation limits stated alongside their results.

Models. The model labels correspond to the following requested serving identifiers:

  • Llama-8B: meta/llama-3.1-8b-instruct.

  • Llama-70B: meta/llama-3.3-70b-instruct.

  • Qwen-80B: qwen/qwen3-next-80b-a3b-instruct.

  • Nemotron-49B: nvidia/llama-3.3-nemotron-super-49b-v1.

  • Mistral-14B: mistralai/ministral-14b-instruct-2512.

  • GPT-5: openai/gpt-5.4.

  • Claude Opus: anthropic/claude-opus-4-8.

These identifiers document the requested endpoints. Generation timestamps and endpoint-version histories are not complete for all saved records.

Free-form generation. Free-form generation uses a single user message with the template:

You are a [product] recommendation advisor. Read the user’s request and give one practical recommendation. Start with your top recommendation as a specific [product] name, then briefly explain why it fits. You may mention alternatives after the top recommendation. User’s request: Title: [title] [body]

It requests three samples at temperature 0.7, nominal seeds 42–44, with body truncation at 1,800 characters. Output limits are 4,096 tokens for the two frontier models and 1,100 for the open models. The reanalysis uses the raw saved responses.

Enumeration and selection. Enumeration and selection each receive one user message. In both, [body] is truncated to 1,500 characters and [product] denotes the domain’s product category. The enumeration prompt is:

A user asks for a [product] recommendation: Title: [title] [body] List 20 DISTINCT specific [product] options that could plausibly fit this request – cast a wide net, include less obvious ones, and give exact product names. One per line, name only, no commentary.

The chooser receives:

User [product]-shopping request: Title: [title] [body] Below is the candidate list. Pick the SINGLE best recommendation. Reply with EXACTLY one line: CHOICE: <number> Candidates: [numbered candidate list] Output only one line.

Appendix § 17 specifies sampling, output limits, pool construction, and choice parsing.

Query-lift prompts. Both conditions request CHOICE: <number> on the first line and allow an explanation. The query-blind condition asks “Which of these is the best [product] in general?”; the conditioned one asks “A user asks: [query]. Which of these is the best [product] for this user?” Both append the same numbered list. Thus query content and the decision instruction change together.

ESCI prompt. The single user message requests CHOICE: <number> on the first line, then asks “A shopper searched for: [query]. Which product best matches?” followed by numbered truncated titles. Appendix § 19 specifies filtering, candidate sampling, title truncation, and parsing. The historical records retain hit flags rather than raw responses or selected product IDs.

Historical analyses. Earlier analyses used different reference sets, answer extractors, sample aggregations, and candidate pools. Their numbers are not pooled with this reanalysis. In particular, unpaired historical root scores, the asymmetric body-class comparison, and title-only concession aggregates do not support the main claims here.

Reanalysis protocol and uncertainty estimation

Objective-check scope and sources. The retrospective audit retains 149 shared car requests with an explicit drivetrain or transmission requirement and all 3,129 saved answers (447 per model). These requests form the audited subset of the 800-query corpus; the remaining 651 are outside its scope. All 149 request annotations were reviewed against the generation-visible text. One missing 4WD requirement was corrected for all seven models. Automatic transmission includes CVT and dual-clutch transmission; AWD and 4WD share one category. All 279 nonempty extracted-name groups were screened for source-name identity, and the final shared rules were applied to every answer. US EPA records, Canadian fuel-consumption data, and manufacturer specifications supply the configuration evidence. Year ranges, body types, engines, turbo status, and recognized variants restrict each name match. Uncovered years, ambiguous families, and unverified trim tokens remain unresolved. Shared US-source rules require a US purchase market established from the query; dollars or miles alone are insufficient. The request review identified 55 US, 13 Canadian, six other, and 75 unspecified markets. Individually reviewed cases can use separately retained regional evidence. Source rows, identity maps, request annotations, and response hashes are archived with the final run.

Review and outcome accounting. The initial assistant review covered 159 full query/response pairs: all 89 surfaced candidate/correction cases and a fixed sample of 70 apparent passes (ten per model). Source-evidence review covers one further unresolved case, 130 cases reclassified as supported, and five apparent supports reclassified as unknown. Complete-answer checks cover all these cases. The apparent-pass sample is an audit, not a random estimate of correctness. All surfaced flags received a disposition. A confirmed conflict requires a source-backed violation of a requested field that remains in the recommendation; self-retractions, disclosed trade-offs, and ambiguous alternatives are kept separate. The 356 supported configurations comprise 33 individually reviewed supports and 323 conservative source-rule matches (Table 9). This support concerns the primary recommendation’s audited fields. It does not certify the remaining statements, price, availability, or overall suitability. The other/unknown category includes unresolved identities and markets, unspecified configurations, corrected or rejected flags, and disclosed trade-offs; none is counted as supported.

ModelReviewed supportRule supportConflict minimum (%; 95% CI)Other/unknown
GPT-59710.00 [–]367
Claude Opus5390.00 [–]403
Llama-70B8541.34 [0.00, 3.36]379
Qwen-80B3540.67 [0.00, 2.01]387
Nemotron-49B5540.67 [0.00, 1.79]385
Llama-8B2341.12 [0.00, 2.91]406
Mistral-14B1173.80 [1.79, 6.27]412

Table 9. Evidence coverage and uncertainty for the objective audit. Each model has 447 answers from the same 149 requests. The two support columns partition the supported counts in Table 2. Intervals use 2,000 whole-query bootstrap resamples (seed 42), retaining all three generations together. They describe the detected conflict statistic and exclude uncertainty about conflicts hidden among unknown cases. Zero-event bootstrap intervals are withheld (–); zero confirmed conflicts does not establish a zero population rate.

The assessable query/sample intersection across all seven models is empty. We therefore retain every answer in the denominator and report minimum confirmed conflicts, without ranking conditional rates on different assessable subsets. This is retrospective assistant adjudication, not independently annotated gold data. Source-certificate review reclassifies 130 cases as supported and five as unresolved because their year/trim scope exceeds the evidence, a net increase of 125 supported answers. These five cases are evidence-scope abstentions, not detected drivetrain errors. The cohort and confirmed-conflict counts are fixed throughout the audit.

Headphone connection audit. A request-only screen identified 44 of the 226 shared headphone requests with explicit connection requirements in the generation-visible text: 25 wireless, 11 Bluetooth, and eight wired. All seven models contribute three saved answers per request, giving 924 answers.

One assistant reviewed the complete responses for product identity, final recommendations, alternatives, and accessory bundles. Model-blinded verification of manufacturer evidence covered 49 distinct wired capability gaps and 33 exact-product nonwired gaps, establishing 52 product/capability facts; 42 ambiguous nonwired gaps remained outside that pass. The evidence map and full-response review of 118 potentially affected answers were frozen before scoring all 924 answers.

Manufacturer evidence supports the requested connection in 795 answers; 129 remain unknown, with no confirmed conflicts (Table 10). Ten requests are assessable across all models and generations, with all 210 answers supported. Fiftyfour unknown answers use unverified HD4.50BT or HD4.50BT ANC abbreviations. Although the search considered positive and explicit negative capability statements, no exact-product negative was established; missing evidence remains unknown. Supported counts measure evidence coverage, so an increase in support under a broader source map is not a change in model performance.

ModelSupportedConflictsUnknownSupported % [95% CI]
GPT-51250794.7 [88.6, 99.2]
Claude Opus1280497.0 [93.9, 99.2]
Llama-70B11002283.3 [72.0, 93.2]
Qwen-80B1270596.2 [91.7, 100.0]
Nemotron-49B9903375.0 [65.2, 84.1]
Llama-8B10602680.3 [71.2, 88.6]
Mistral-14B10003275.8 [67.4, 83.3]

Table 10. Headphone connection-capability audit. Counts retain all 132 answers per model (44 requests, three responses). Supported percentages reflect evidence coverage. Intervals use 10,000 request-cluster resamples (seed 20261012), stratified by the two annotation batches of 20 and 24 requests. Zero conflicts does not establish zero population risk or a model-quality ranking.

Annotation uses two batches of 20 and 24 requests; the 20-request batch was selected by a fixed hash order before answer review. The bootstrap retains these batches as strata. Annotations come from one assistant, not independent human validation; intervals exclude annotation and source uncertainty. The check covers only the top actionable recommendation’s connection capability. Price, sound quality, comfort, other requested features, and historical availability remain unassessed. The attribute differs from the car audit and cannot establish comparative domain difficulty.

Reanalysis protocol. We recomputed the Reddit agreement and matched enumeration results from saved generations with one declared product matcher: the token-boundary matcher with the alias and variant-suffix rules specified in Appendix E. The answer extractor scans a recommendation section preferentially, then the response, using a catalog vocabulary purged of fragments and designated family wildcards. It can still mistake a mentioned alternative for the final pick. For reference sensitivity we score against the first raw reference entry and against all raw entries, keeping the same outputs and denominator. All three saved samples contribute equally within each query. Unextractable outputs remain in the denominator and score zero. The protocol was specified retrospectively.

Comparisons and uncertainty. Enumeration coverage is measured at nested depths on the same queries; its final choices and the free-form baseline use the same three-sample mean. Enumeration consumes additional generation calls, so this is not a cost-matched efficiency comparison. Query lift is paired within query, with the query-blind prompt as control. ESCI uses uniform four-way choice as a reference level. We use 2,000 deterministic query-bootstrap resamples for the agreement and enumeration tables and 10,000 for the extraction and baseline sensitivity checks, with Wilson intervals for ESCI’s binary rates. Intervals describe query variation conditional on the saved outputs, not variance across generation seeds.

Full-request source-blind comparison

Sample and fixed alternative. The prespecified protocol samples 150 car requests from 1,247 eligible source requests using seed 20260927. Eligibility requires a nonempty full request and an unambiguous catalog-recognizable community mention. The first such entry in the stored reference order supplies one fixed alternative per request, selected before model generation. The original answer can itself match another community mention; eligibility for the comparison concerns only the fixed alternative.

Generation and comparison. All six models receive the same title, full body, and historical-date instruction. Initial generations use temperature 0.7, seeds 42–44, and a 2,048-token output limit. Each must return JSON with one recommendation and an explanation, or abstain. Initial abstentions, invalid responses, and recommendations matching the fixed alternative are ineligible; Table 11 accounts for all 450 initial generations per model. Each eligible trial has two fresh, source-blind comparisons, at temperature zero and a 1,024-token output limit. They present the same two name strings in opposite orders, without the initial explanation or conversation history. The allowed choices are A, B, EQUAL, NEITHER, and INSUFFICIENT. Scoring maps the selected letter back to its supplied name; no judge interprets the explanation to determine the winner. Invalid or missing choices cannot count as a reference win and remain in the denominator. All scheduled calls completed without terminal API failures; unsuccessful parsing was retained rather than retried.

ModelInvalidAbstainMatchExcl.EligibleBad comp.Ref. / presentations
GPT-5.481390402077/804 (9.6%)
Claude Opus 4.8143365392153/784 (6.8%)
Llama-70B130257405670/810 (8.6%)
Qwen-80B53366400158/800 (7.2%)
Nemotron-49B9116215317270/634 (11.0%)
Llama-8B3711251036726120/734 (16.3%)

Table 11. Source-blind comparison eligibility and per-presentation results. Each model starts with 450 initial generations (150 requests, three seeds). The first five numeric columns partition these generations: invalid initial responses, abstentions, parser-detected reference matches, post-hoc identity exclusions, and retained eligible trials. Invalid comparisons (“Bad comp.”) remain in the denominator of two presentations per retained trial. The final column counts reference choices in individual presentations, including order-sensitive choices, with percentages in parentheses.

Complete prompts and response formats. The initial generation receives a single user message below; bracketed fields are replaced with the stored request date, title, and full body. No system message is added.

You are a car recommendation advisor. Read the user’s request and give one practical recommendation that fits the stated needs.

Return only a JSON object with “recommendation” (one specific car make and model, or null if there is insufficient information) and “reason” (a brief explanation, at most 120 words). Do not list alternatives in the recommendation field.

Treat the request as made on [request date]. Use only vehicles available by that date. User’s request: Title: [title] [full body]

Thus, the initial response has the form

{"recommendation": "[car name]", "reason": "[explanation]"},

with JSON null replacing the car-name string for abstention.

Each source-blind comparison also receives just one user message, without the initial response or conversation history:

Treat the request as made on [request date]. Use only vehicles available by that date. User’s request: Title: [title] [full body]

Which is the better recommendation for this request? The options are unlabeled as to source. If they are equally suitable, neither is suitable, or the information is insufficient, report that rather than force a preference. A. [first car name] B. [second car name]

Return only JSON with “choice”: “A”, “B”, “EQUAL”, “NEITHER”, or “INSUFFICIENT”, and “reason”: a brief explanation (at most 120 words).

The comparison response has the form

{"choice": "[allowed choice]", "reason": "[explanation]"}.

One ordering places the initial recommendation first and the fixed alternative second; the other reverses them. Source labels are absent to avoid explicitly cueing deference to the community or defense of the model’s earlier answer. The execution order is deterministically shuffled within each trial; all prompts and raw responses are saved.

Identity sensitivity check. For GPT-5.4 and Claude, inspection of all eligible initial-name/reference pairs found three Claude trials comparing “Volkswagen Golf GTI” with “Volkswagen GTI”; both blind responses identify the names as equivalent. Two other Claude recommendation fields contain explicit alternatives: “Mazda MX-5 “Miata (or Mazda3 for practicality)” and “2019 Mazda3 Sedan (or Hatchback).” The original parser missed the parenthesized disjunctions. We conservatively exclude all five trials, including the within-model body-style alternative, leaving 392 Claude trials. None is a reference win under either the both-orders or per-presentation metric; the numerators do not change. No GPT-5.4 trial is excluded. Both retained cohorts cover 137 of the 150 sampled requests, not necessarily the same 137 requests.

These exclusions form a post-hoc sensitivity analysis. Before exclusions, the primary results are 26/402 (6.5%) and 16/397 (4.0%); per-presentation results are 77/804 (9.6%) and 53/794 (6.7%). The checked figures in the main text use 402 and 392 as trial denominators for these two models. The check verifies saved prompts, parsing, option-order mapping, and outcome counts, and retains all 42 positive trial records with their explanations. It does not certify those explanations as correct product facts or independently validate recommendation quality.

The same conservative identity check excludes seven Llama-70B, six Qwen, five Nemotron, and 10 Llama-8B trials: 22 overlapping name/family/trim identities, two explicit alternative configurations, three “Subaru dealer” responses, and one conflicting “Subaru CX-5” make/model string. Overlaps include Miata versus MX-5 Miata, BMW 3-Series versus 328i, and abbreviated Jetta, Bolt, or 4Runner names. None is a both-orders reference win. The frozen primary counts are 26/412, 18/406, 13/322, and 7/377; after exclusions they are 26/405, 18/400, 13/317, and 7/367. Their 64 positive trials span 10, 13, seven, and six requests, respectively. All 106 positive trials across the six models are retained with their explanations in the derived audit.

Eligibility also differs across models. Nemotron has 91/450 invalid initial responses (20.2%) and 16 abstentions; Llama-8B has 37 invalid responses (8.2%) and 11 abstentions. The other four models have 5–14 invalid initial responses each (1.1–3.1%). These losses include JSON/schema violations and multiple-name fields, limiting the comparison’s coverage.

Uncertainty and archived controls. Intervals use 5,000 bootstrap resamples of eligible request IDs with seed 20260927, retaining each request’s initial generations and both orders together. They describe variation across these requests conditional on the saved outputs, not independent evidence from each presentation. The GPT/Claude car collection also includes two reconsideration conditions with the original answer visible and an otherwise matched no-alternative reconsideration control. These saved follow-ups are not used in the source-blind headline. In particular, the control’s raw change count can include year- or trim-only changes and must not be read as a rate of changes in the recommended car model.

Collection and audit. The GPT/Claude car collection requests openai/openai/gpt-5.4 and azure/anthropic/claude-opus-4-8. Returned model identifiers are recorded for every response. Primary and identity-screened results use the same saved responses; the latter apply only the five exclusions listed above. The earlier title-only concession analysis is not pooled with this run: cohort, generation, date instruction, and response format differ.

The open-model car collection was recorded on October 10, 2026, using the same frozen cases, prompts, seeds, and token limits. It evaluates Llama-3.3-70B-Instruct, Qwen3-Next-80B-A3B-Instruct, Llama-3.3-Nemotron-Super-49B-v1, and Llama-3.1-8B-Instruct through a hosted inference endpoint. The requested Ministral-14B-Instruct-2512 version was unavailable; no replacement version was substituted. The failed public-endpoint probes are infrastructure records and are not scored. This collection contains 1,800 initial responses and 3,034 blind comparisons, with no conversational revision controls. Collection dates and serving implementations differ from the GPT/Claude collection. We checked every prompt and scored outcome against the prespecified protocol and report primary and identity-screened results separately.

Temperature-zero initial recommendations

Protocol. The three-sample experiment uses initial recommendations at temperature 0.7; a sampled answer need not be the model’s preferred recommendation. We therefore generated one initial answer at temperature zero for each of the same 150 car requests, using GPT-5.4 and Opus 4.8 with nominal seed 42. We retained the full requests, dates, fixed alternatives, prompts, token budgets, parser, and requested model routes. Valid disagreements received both source-blind orders at temperature zero. The check used 414 GPT and 412 Opus calls, all successful on their first transport attempt. Malformed outputs were retained. Temperature zero removes deliberate sampling randomness but does not guarantee deterministic API serving.

Eligibility and review. Of 150 initial GPT answers, three fail the inherited name schema and 15 match the reference, leaving 132 disagreements. For Opus, eight initial answers fail parsing, one abstains, and 10 match, leaving 131. The identity sensitivity excludes two further Opus pairs: Golf GTI versus GTI (duplicate labels), and GTI versus an unspecified Golf (family/variant overlap). Neither is a both-order reference win. The retained denominator is 129, including one invalid comparison pair; valid-pair sensitivity gives 4/128 (3.1%). GPT has no invalid comparison pair. All 300 initial-name pairs were reviewed, together with the first 10 requests per model, all 11 positive cases, and every malformed output. Parsing and identity screening were not treated as objective-quality judgments.

Results and interpretation. The alternative wins in both orders for 7/132 GPT requests (5.3%; 95% interval 2.3–9.1%) and 4/129 Opus requests (3.1%; 0.8–6.2%; Table 12). These point estimates are lower than the temperature-0.7 rates of 6.5% and 4.1%, but the behavior remains observable without deliberately sampling the initial answer. This supports a limited claim about disagreement between generation and comparison, not the objective superiority of the alternative. The different initial products, eligibility sets, and collection dates preclude attributing the rate difference solely to temperature. Some positive-case explanations introduce unstated priorities or questionable product claims; choosing the reference does not verify the reasoning. Per presentation, the reference wins 24/264 GPT and 22/258 Opus comparisons; order sensitivity remains possible.

ModelInitial temp.Initial answersEligibleBoth ordersRate % [95% CI]
GPT-5.40.7450402266.5 [3.0, 10.3]
GPT-5.4015013275.3 [2.3, 9.1]
Opus 4.80.7450392164.1 [1.5, 7.3]
Opus 4.8015012943.1 [0.8, 6.2]

Table 12. Temperature-zero robustness of both-order community preference. Both orders counts requests/trials choosing the fixed community alternative in both presentations, after identity screening. The 0.7 rows use three initial generations per request; the zero rows use one. Rates divide by eligible disagreements; intervals use 5,000 whole-query bootstrap resamples (seed 20260927). Initial products, eligible cohorts, and collection dates differ across temperatures.

Primary, identity-screened, and valid-pair analyses use the same saved responses without further generation. Review was by one assistant, not independent human annotation.

Restricted laptop replication

Cohort and protocol. We extended the source-blind comparison to GPT-5.4 and Claude Opus 4.8 on the same 150 laptop requests, sampled with seed 20261010 from 201 eligible requests among 961 stored posts. Eligibility required a full dated request and an exact catalog manufacturer/model-series name also present in a stored comment. The first eligible reference in stored order was fixed before generation. Broad families and underspecified names were excluded. Of the 150 fixed references, 80 name MSI, 57 Dell, nine Lenovo, and four ASUS products, on requests posted from January 1 to March 8, 2019.

The prompts retain the full request and historical-date instruction. Three initial generations use seeds 42–44 and temperature 0.7; each eligible disagreement receives both source-blind orders at temperature zero. Initial agreement uses manufacturer/model-series identity. The outcome is preference for the reference in both orders per initial generation, without voting across the three generations.

Parser and identity audit. For GPT, review of 18 initially excluded names recovered 16 single-product names containing configuration details: three matched the reference, and 13 received their 26 missing comparisons. The 450 initials then comprise 46 same-series matches, two exclusions, and 402 disagreements. A separate identity sensitivity

ModelInitial temp.Initial answersEligibleBoth ordersRate % [95% CI]
GPT-5.40.7450402266.5 [3.0, 10.3]
GPT-5.4015013275.3 [2.3, 9.1]
Opus 4.80.7450392164.1 [1.5, 7.3]
Opus 4.8015012943.1 [0.8, 6.2]

excludes two probable Dell G5 aliases, leaving 400 trials across 142 requests.

For Opus, review recovered 17 single-product initials and their 34 missing comparisons, leaving 433 valid initials: 23 same-series matches and 410 disagreements. The remaining 17 comprise 14 multiple/invalid names, two format errors, and one terminal transport failure. Seven further uncertain identities are excluded only in the sensitivity analysis, leaving 403 trials across 143 requests. Six malformed comparison responses remain in the parser-corrected denominator; five remain after identity exclusions. GPT has no malformed comparisons. Neither initial answers nor completed comparisons were regenerated during recovery.

Results and interpretation. After identity exclusions, GPT prefers the reference in 44/400 trials (11.0%) and Opus in 41/403 (10.2%; Table 13). On 368 commonly eligible query/seed trials across 136 requests, Opus has 38 both-order preferences and GPT 37: a paired difference of +0.27 percentage points (95% CI [−5.98, +6.16]), without established model separation. The models generated different original alternatives, so even this paired comparison concerns the joint generation/comparison procedure. Opus’s identity-filtered valid-pair rate is 41/398; allowing every unresolved comparison pair that could favor the reference gives 42/403.

ModelAnalysisPreferences / trialsRate (%; 95% CI)
GPT-5.4Frozen parser43 / 38911.05 [6.57, 15.97]
GPT-5.4Parser-corrected44 / 40210.95 [6.52, 15.83]
GPT-5.4Identity sensitivity44 / 40011.00 [6.55, 15.88]
Claude Opus 4.8Frozen parser42 / 39310.69 [6.44, 15.25]
Claude Opus 4.8Parser-corrected43 / 41010.49 [6.37, 14.88]
Claude Opus 4.8Identity sensitivity41 / 40310.17 [6.09, 14.64]

Table 13. Preference for community-mentioned laptop alternatives. Both-order preferences per eligible initial generation, on the same restricted 150-request cohort. Rows within each model are successive analyses of one run. Intervals use 5,000 whole-query bootstrap resamples (seed 20260927); malformed comparisons remain in the eligible denominator.

Assistant review covered the fixed 20-request samples, valid initial/reference pairs, positive trials, and their source contexts. References include criticized products, retracted suggestions, and affiliate-style links. Some positive comparisons impute configurations absent from the series-only reference. No post-hoc endorsement filter changes the frozen cohort. These are preferences for source-mentioned alternatives of uneven specificity, not independent evidence that the original advice was wrong. Differences from cars cannot be attributed to domain alone.

Collection. Including the additional comparisons after parser recovery, the GPT replication used 1,254 successful calls to openai/openai/gpt-5.4. The Opus replication used 1,269 successful calls to azure/anthropic/claude-opus-4-8, plus the terminal initial failure. Product facts in explanations and historical availability remain unverified.

Agreement reanalysis and extraction coverage

Support for the top reference. In the larger car reference corpus, 1,202 of 1,465 requests (82.0%) have a top reference name extracted from just one stored top-level comment. Thus, the highest stored reference weight often comes from a single comment. Counting distinct endorsing users would require author IDs and endorsement judgments, which are unavailable.

Agreement scoring. The comparison uses all 800/800/226 saved query IDs in each Reddit domain for every model, with three outputs per query. We verified that query sets agree across models. The extraction vocabulary excludes entries whose model names contain only generic/function words, designated plural family entries, and short alphabetic line prefixes shared by multiple numbered models. Reference strings themselves remain raw for the reference-granularity comparison. For matching, we split letter–digit boundaries, lowercase names, replace non-alphanumeric characters with spaces, and remove a final s from tokens longer than three characters. A match requires contiguous whole-token containment in either direction. Containment ending in a numeric token is rejected if the longer name continues with a single-letter variant suffix, distinguishing T14 from T14s. If neither name is a contiguous token subsequence of the other, an alias match can apply: both names must share the first token, and the shorter name must have at least two further tokens, at least one containing a digit, all present in the longer name after its first token. These rules can award family-level credit and can conflate abbreviated sibling products; the reported identity sensitivities assess some of these cases. For each query, we average the three binary hit outcomes before averaging across queries. No-pick outputs remain zero in both scores. The paired set-minus-top comparison changes no generation, extractor, query inclusion rule, or sample aggregation. Table 14 also reports extraction success, which varies materially by domain and model. A missing extraction can occur even when the response contains a useful recommendation.

DomainModelTop [95% CI]Set [95% CI]Difference [95% CI]Parsed (%)
CarsGPT-56.8 [5.2, 8.5]24.8 [22.0, 27.6]18.1 [15.6, 20.6]95.3
CarsClaude Opus6.5 [5.0, 8.1]24.0 [21.4, 26.8]17.6 [15.1, 20.2]93.5
CarsLlama-70B6.0 [4.6, 7.5]24.5 [21.7, 27.0]18.5 [16.2, 20.9]96.7
CarsQwen-80B5.2 [3.8, 6.8]23.2 [20.6, 26.0]18.0 [15.6, 20.4]97.5
CarsNemotron-49B6.0 [4.6, 7.4]22.0 [19.5, 24.6]16.1 [13.9, 18.4]97.5
CarsLlama-8B5.6 [4.2, 7.0]21.3 [18.8, 24.0]15.7 [13.5, 18.1]95.8
CarsMistral-14B5.5 [4.2, 6.8]20.8 [18.5, 23.3]15.4 [13.2, 17.5]98.1
LaptopsGPT-510.3 [8.5, 12.3]26.0 [23.4, 28.6]15.7 [13.6, 18.0]81.1
LaptopsClaude Opus9.6 [7.8, 11.4]24.8 [22.2, 27.4]15.2 [13.0, 17.5]80.6
LaptopsLlama-70B10.0 [8.2, 12.0]27.4 [24.6, 30.4]17.3 [15.0, 19.8]73.1
LaptopsQwen-80B5.0 [3.7, 6.4]19.0 [16.6, 21.5]14.0 [11.9, 16.1]72.6
LaptopsNemotron-49B8.6 [6.9, 10.2]23.3 [21.0, 25.8]14.8 [12.6, 17.0]73.5
LaptopsLlama-8B9.0 [7.3, 10.7]24.0 [21.5, 26.6]15.0 [12.9, 17.2]70.6
LaptopsMistral-14B5.1 [3.9, 6.5]18.4 [16.1, 20.8]13.2 [11.3, 15.2]81.7
HeadphonesGPT-56.6 [3.8, 9.9]22.6 [17.8, 27.4]15.9 [11.7, 20.6]87.3
HeadphonesClaude Opus7.2 [4.3, 10.3]24.6 [19.5, 29.8]17.4 [12.7, 22.1]91.7
HeadphonesLlama-70B8.3 [5.2, 11.5]21.5 [16.8, 26.4]13.3 [9.4, 17.1]86.1
HeadphonesQwen-80B5.5 [2.8, 8.6]17.1 [12.7, 21.5]11.7 [8.3, 15.6]90.9
HeadphonesNemotron-49B4.9 [2.8, 7.2]17.7 [13.7, 21.8]12.8 [9.3, 16.8]89.7
HeadphonesLlama-8B5.2 [2.9, 7.7]16.1 [12.5, 20.1]10.9 [7.5, 14.3]81.6
HeadphonesMistral-14B3.7 [1.9, 5.8]14.7 [11.2, 18.4]11.1 [8.3, 14.0]96.2

Table 14. Agreement uncertainty and extraction coverage. Percentages with 95% query-bootstrap intervals; differences are paired set-minus-top percentage points. Parsed is the fraction of saved outputs yielding a catalog-extracted pick. Every sample remains in the agreement denominator.

Scope of the rescore. Reference endorsement, final-pick extraction, and historical availability remain separate validation questions. The older catalog-validated top-three metric used a different reference target. The former Stack Exchange baseline counted mentions in a response window; Table 14 uses one extracted pick on the Reddit domains. The software analysis below examines extraction scope; Appendix § 18 separately reports supplied-choice results.

Software extraction-scope sensitivity. A separate recovery uses 400 shared Software Recommendations requests and three saved responses from each of seven models (8,400 answers). Its 651 stored reference entries were traced to source answers and the two consensus extractors. We compare literal reference-name occurrence in the extracted opening recommendation label with occurrence anywhere in the complete answer. All 1,200 answers per model remain in the denominator, including unresolved opening labels. Full-answer occurrence exceeds opening-label occurrence by 17.3–25.3 points (Table 15). On the 912 query/sample positions parsed for every model, gaps remain 16.8–25.2 points. Excluding 18 entries with generic names leaves 17.1–25.0-point gaps on the full denominators. These sensitivities support an extraction-scope effect, while composite recommendations, aliases, and retractions still require semantic judgment. The parser was developed on disclosed subsets and received assistant spot checks; this is not an independent human extraction benchmark. The source-order cohort, edited source text, and incomplete serving provenance also limit interpretation.

ModelOpening (%)Full answer (%)Difference (pp)Unparsed
GPT-524.3 [20.3, 28.4]45.9 [41.3, 50.5]21.7 [18.4, 25.3]5
Claude Opus20.8 [17.1, 24.8]44.7 [40.1, 49.4]23.8 [20.2, 27.7]48
Llama-70B23.2 [19.3, 27.2]40.4 [35.9, 44.8]17.3 [14.2, 20.6]2
Qwen-80B20.9 [17.3, 24.8]41.9 [37.3, 46.5]21.0 [17.7, 24.5]36
Nemotron-49B17.5 [14.3, 20.8]37.4 [33.2, 41.8]19.9 [16.9, 23.1]60
Llama-8B14.6 [11.8, 17.6]33.0 [28.9, 37.3]18.4 [15.5, 21.5]126
Mistral-14B14.2 [11.3, 17.3]39.4 [35.0, 43.9]25.3 [21.9, 28.8]59

Table 15. Software reference-name occurrence by extraction scope. Percentages and paired percentage-point differences with 95% intervals from 4,000 query-cluster resamples (seed 20261010). Unparsed counts unresolved opening labels among 1,200 answers per model. A name occurrence can be an alternative, component, or rejection; these are not semantic recommendation-accuracy scores.

Failure taxonomy and assessment requirements

Table 16 lists all 19 conceptual modes and the evidence needed to establish each diagnosis. For example, diagnosing an unavailable candidate requires checking product availability beyond the local catalog.

CodeIntended failure modeEvidence required for assessment
1aA relevant fact is omittedRequest spans, recovered facts, and a relevance criterion.
1bA fact is inventedThe asserted fact and evidence that the request does not support it.
1cValence or attribution is changedWho expressed the preference, its target, and its polarity.
2aAn inference is unsupportedThe inferred belief and the facts claimed to license it.
2bA prior is presented as user-derivedProvenance of the belief and an explicit distinction between evidence and assumption.
2cA prior contradicts the requestA contradiction between an added belief and a supported user fact.
3aA constraint is omittedAn explicit or justified requirement and its representation in CC.
3bA constraint has the wrong valueThe source value, units, and represented value.
3cHard and soft criteria are confusedLanguage establishing requirement versus preference; ambiguous cases may remain unknown.
3dInfeasibility is not acknowledgedEvidence that the specified constraints cannot jointly be met in the relevant option space.
3eAn allowed class is prematurely excludedA request allowing alternatives, and evidence that a class was removed without justification.
4aCandidate coverage is inadequateAn explicit pool and a declared target or breadth criterion; a small pool alone is a proxy.
4bA candidate is nonexistent or unavailableReliable product identity and availability at the evaluation place and date.
4cA candidate belongs to the wrong tierThe requested tier and relevant product attributes, with units and temporal context.
5aA satisfying candidate is discardedThe original and retained pools plus assessable constraints showing a false negative.
5bA violating candidate is retainedThe retained option and an assessable violated hard constraint.
6aRanking conflicts with stated prioritiesThe stated preferences, relevant attributes, and a defensible comparison rule.
6bThe final pick contradicts the explanationAn identifiable pick and a contradiction in the response’s own explanation.
6cChoice varies under equivalent conditionsRepeated, matched inputs and valid parsed selections; variation alone need not be an error.

Table 16. The 19-mode diagnostic taxonomy. The last column specifies the evidence needed for each diagnosis.

Illustrative use. Consider a constructed request for a six-seat family vehicle, with a fixed budget, allowing either an SUV or a minivan. If the extracted constraints require an SUV, mode 3e directs attention to the lost disjunction. If the constraints preserve both classes but the pool contains only SUVs, pool construction merits inspection; an SUV-only list is not itself an error if its options satisfy the request. If a six-seat candidate is removed despite satisfying the stated criteria, the error lies in filtering. Each diagnosis requires different evidence.

Prototype implementation versus intended checks

Table 17 documents the selected prototype checks. Fifteen checks are non-generative; four call a generative judge. Selection on small constructed fixtures does not establish precision or recall on natural responses. Several checks expect explicit Step headings and are not general free-form extractors.

CodeSelected implementationScope or qualification
1aRegular-expression coverageLimited to canonical query patterns.
1bToken overlapMeasures lexical overlap; semantic support requires separate assessment.
1cPolarity and target lexiconsLimited sentiment and attribution coverage.
2aStructural citation checkInference validity requires checking the cited evidence.
2bStrict provenance-tag patternMeasures explicit labeling of assumptions.
2cGenerative judgeRequires judge calibration and records of its outputs.
3aExpanded constraint patternsLimited to recognized constraint forms.
3bValue-pattern comparisonSensitive to parsing, units, and scope.
3cModal-language patternsA proxy for hard/soft distinctions.
3dInfeasibility-flag phrasesJoint catalog feasibility requires separate assessment.
3eMulti-class patternsLimited to recognized alternative-class expressions.
4aCandidate-count thresholdSelected recognizer is vehicle-specific; cross-domain firing is not a valid breadth estimate.
4bGenerative judgeCatalog absence and real-world nonexistence must be distinguished.
4cCatalog price-tier comparisonDepends on accurate identity, price metadata, and the requested market.
5aBroader constraint/mention extractionRequires both pools and assessable constraints.
5bConstraint patterns and mention scanSame prerequisites; unknown attributes cannot count as violations.
6aGenerative judgeRequires a specified interpretation of preference trade-offs.
6bGenerative judgeTargets explanation–choice consistency.
6cHedge-word countNot a measurement of cross-rollout agreement.

Table 17. Selected prototype implementations. The final column states the scope and limitations of each heuristic or judge-based check.

Worked example: numerical check for failure 3b. In the constructed example in § 4.2, the request requires at least six passengers under $20,000. The response records both requirements correctly in FF, but its hard-constraint section CC specifies at least five passengers and price at most $20,000.

The implemented 3b verifier extracts numbers by dimension from the request and the response’s labeled hard-constraint section. It checks whether a represented value is within 10% of the requested value. Here the budget matches, but five differs from six by 16.7%, so the verifier flags the response and reports a score of 0.5: one of two dimensions matches. Replacing five with six makes the score 1.0 and clears the flag. We reproduced both outcomes with the implementation.

The numerical tolerance is a prototype heuristic. Compliance must be assessed against the user’s exact requirement. The check also depends on recognizing the wording and section headers.

Root-check experiment. The five non-generative root checks are rerun on the original Llama-70B root experiment’s first 100 request IDs, using the same IDs for every model and the first saved generation. The historical analysis used the first 100 saved responses per model; their request order differs, so those cohorts were not fully paired. We reproduce those historical car scores separately, then align the IDs and restrict the query body used by each check to the generator’s 1,800-character limit. Table 5 reports the resulting matched profiles. No generation is dropped for a verifier failure. The counts describe this fixed sample and one generation per request. The 1a check fires when any recognized query token is absent from the parsed fact list. The 1b check fires when any fact item has less than 50% content-token overlap or contains a number absent from the query. The 1c check searches for limited target/polarity patterns. The 2b check requires a recognizable Step 2B section and explicit prior tags on its items. The 4a check flags fewer than five recognized vehicle names, except when the query already names at least two vehicles. It uses the parsed post-filter pool when available and otherwise scans the response. Its input can therefore differ from the initial candidate pool. Missing parseable facts cause 1a and 1b to return a non-firing score, and queries without recognized targets can also pass 1c without a substantive assessment. Conversely, harmless paraphrases can trigger the lexical checks. A non-firing flag therefore does not certify that the corresponding reasoning step is correct. We do not restore the historical 100% laptop/headphone 4a rates: the selected recognizer is vehicle-specific, making those figures out-of-domain extraction artifacts rather than candidate-coverage measurements.

Exploratory checker–assistant comparison. We selected 20 of the shared 100 car requests by a fixed hash, retaining the first response from all seven models (140 answers). A request-only fact inventory and rubric were frozen before answer annotation. One assistant read complete responses under opaque IDs; its labels were frozen before model keys or checker outputs were opened. Paraphrases counted as supported, and labeled assumptions were distinguished from claims about the user’s stated facts. Agreement on definite cases is 69.6% for 1a and 35.8% for 1b (Table 18). Disagreements include supported paraphrases, equivalent number formats, and changed relationships hidden by reused vocabulary. One readable fact list is missed by the legacy parser, so agreement can also conceal a missing check. Of 88 assistant-labeled stage omissions, 17 are fully recovered elsewhere in the answer, 69 retain a whole-answer omission, and two have uncertain recovery. All 280 saved checker outputs were reproduced exactly. This small, subjective audit motivates detector validation; it does not establish population failure rates or a model ranking. The post-hoc input-length sensitivity check excludes the 39 requests with bodies longer than 1,800 characters, without conditioning on outcomes. Table 19 reports the remaining 361 requests. Intervals use 2,000 paired query resamples with fixed, hash-derived seeds and retain the three generations together. They quantify variation across requests conditional on the saved outputs, excluding uncertainty from collection dates or serving implementations.

RootBoth flagChecker onlyAssistant onlyBoth clearUnknownAgreement (%)
1a63172533296/138 (69.6)
1b24772201744/123 (35.8)

Table 18. Lexical checks versus exploratory assistant labels. Counts among 140 responses. Roots 1a and 1b concern omissions and unsupported additions in the explicit user-facts stage. Agreement excludes unknown assistant labels. These single-assistant labels are not independently validated human gold.

Fact-anchoring experiment. The fact-anchoring experiment takes the first 400 eligible car requests and generates three Llama-70B responses per request using seeds 42–44, temperature 0.7, and a 1,100-token output limit, as in the structured baseline. The prefilled fact list is formed from the title and body, capped jointly at 2,500 characters, split on sentence boundaries and newlines, retaining the first 12 segments longer than 12 characters. The title and up to 4,000 body characters are also included in the request, versus 1,800 body characters in the baseline. The model can revise or omit the prefilled fact list in its response.

We match all 400 request IDs, verify the stored references agree across conditions, and apply the same purged car catalog, final-pick extractor, and name matcher to both sides. Each query’s score is the mean of its three binary sample scores; an unextractable pick contributes zero. The all-entry agreement difference is −0.9 points, with a paired query-bootstrap interval of [−2.8, +0.8]. The post-hoc input-length sensitivity check excludes the 39 requests with bodies longer than 1,800 characters, without conditioning on outcomes. Table 19 reports the remaining 361 requests. Intervals use 2,000 paired query resamples with fixed, hash-derived seeds and retain the three generations together. They quantify variation across requests conditional on the saved outputs, excluding uncertainty from collection dates or serving implementations.

Among the 1,083 anchored outputs in this subset, 82 lack a parseable returned fact list, compared with none in the baseline. The absence may reflect using the supplied list without restating it; it cannot establish that the facts were ignored. Because the lexical checks can return non-firing scores when this section is absent, a reduction in flags would not demonstrate a successful fact repair. Extraction rates across all 400 requests are 97.0% and 96.1%, and the matched-subset rates are reported in Table 19.

Missingness semantics. For an option-level constraint check, an unrecognized product or missing

MetricBaselineAnchoredDifference [95% CI]
Top-entry agreement5.14.6-0.5 [-1.5, +0.6]
All-entry agreement22.220.2-1.9 [-4.0, -0.2]
Pick extracted97.096.1-0.9 [-2.1, +0.2]

caption: Fact anchoring on input-length-matched requests. Llama-70B, 361 paired car requests, three generations per condition. Both prompts receive the same request content; the anchored prompt additionally supplies the fact-list scaffold. Rates are percentages; differences are anchored minus baseline percentage points with 95% paired query-bootstrap intervals. Missing picks remain in the agreement denominator.

:::

attribute yields unknown. For the claim that at least one reference option satisfies a constraint, the result is true if any is known to satisfy it, false only if all are assessable and violating, and unknown otherwise. We exclude the legacy body-class comparison from the empirical claims because its treatment of unresolvable community options was asymmetric.

Candidate-coverage details and controls

Recovered coverage across models. Table 20 recovers six-model results from the saved enumeration pools. We intersect the original 171/176/49 request IDs with availability in all six models, yielding 160/164/49 shared IDs; the exclusions arise from missing Llama-8B records. Raw reference lists agree across models. Duplicate records must agree exactly, and each candidate is matched separately, preventing a match from spanning two candidate names. The archived files preserve parsed pools, with incomplete raw-generation and serving-identifier records. The collections differ in timing and serving provenance.

ModelCars (n=160n = 160)Laptops (n=164n = 164)Headphones (n=49n = 49)
GPT-580.0 [73.8, 86.2]89.0 [84.1, 93.9]69.4 [55.1, 81.6]
Claude Opus78.8 [71.9, 85.0]91.5 [87.2, 95.7]75.5 [63.3, 87.8]
Llama-70B75.6 [68.8, 81.9]95.7 [92.1, 98.2]57.1 [42.9, 69.4]
Qwen-80B72.5 [65.6, 79.4]92.7 [88.4, 96.3]59.2 [44.9, 71.4]
Nemotron-49B73.8 [66.9, 80.6]93.3 [89.0, 97.0]49.0 [34.7, 63.3]
Llama-8B70.0 [63.1, 76.9]92.1 [87.8, 96.3]53.1 [38.8, 65.4]

Table 20. Recovered six-model reference coverage at K=40K = 40. Percentages with 95% query-bootstrap intervals on shared request IDs within each domain. All models use the same raw reference lists and candidate-wise matcher. KK counts parsed entries, which can include aliases, configurations, or malformed list items. No Mistral pools are available for this comparison.

Laptop coverage exceeds headphone coverage for all six models. The highest coverage point estimates belong to GPT-5 for cars, Llama-70B for laptops, and Claude for headphones.

Reference-filtering sensitivity. Inspection of the saved references reveals fragments such as “Hyundai has” and “Sony is,” alongside legitimate names and broad families. As a symmetric sensitivity check, we retain only references matching the existing purged catalog, applying one rule to every model. At least one reference survives for 146/160 car, 160/164 laptop, and 43/49 headphone requests. We keep all original queries in the denominator, assigning zero coverage when no reference survives. Table 21 reports the resulting rates. The reduction is largest for headphones, at 12.2–18.4 percentage points across models. The filter can remove valid unlisted names, and family-level matches remain. Reference endorsement and suitability require independent validation.

ModelCarsLaptopsHeadphones
GPT-576.988.457.1
Claude Opus77.591.561.2
Llama-70B71.295.740.8
Qwen-80B68.892.140.8
Nemotron-49B68.192.734.7
Llama-8B67.592.138.8

Table 21. Catalog-filtered reference coverage at K=40K = 40. Percentages on the unchanged 160/164/49 denominators from Table 20. Only the reference lists change; generated pools and matching rules are fixed.

Generated-pool choice across models. An archived GPT car chooser run covers all 171 car requests; we verify its displayed pools against the saved enumeration records. GPT-5.4 enumeration and choice on the same 171 car and 49 headphone requests uses the same prompts, seeds, token limits, and first-mention pool order, with 1,320 calls. Both the historical number parser and a strict first-line parser agree for every GPT-5.4 selection. Table 22 distinguishes the archive from the October 2026 collection. GPT-5.4 coverage rates are 80.7% and 65.3%, but reference selection conditional on coverage is 30.9% and 24.0%. Conditional rates use each model’s own generated pools.

DomainModel / collectionQueriesCoverage@40CoveredSelection when covered
CarsLlama-70B (archive)17176.6 [70.2, 83.0]13126.5 [18.8, 34.4]
CarsGPT-5 (archive)17180.7 [74.9, 86.0]13833.3 [25.6, 41.6]
CarsGPT-5.4 (Oct. 2026)17180.7 [74.9, 86.0]13830.9 [23.4, 37.9]
HeadphonesLlama-70B (archive)4957.1 [42.9, 69.4]2839.3 [22.6, 57.1]
HeadphonesGPT-5.4 (Oct. 2026)4965.3 [53.1, 77.6]3224.0 [9.4, 38.5]
LaptopsLlama-70B (archive)17695.5 [92.0, 98.3]16847.0 [40.1, 54.6]
LaptopsGPT-5.4 (Oct. 2026)17690.3 [85.8, 94.3]15924.9 [18.9, 31.7]
HeadphonesOpus 4.8 (Oct. 2026)4975.5 [63.3, 87.8]3735.1 [18.9, 51.4]

Table 22. Coverage and generated-pool selection across models. Rates are percentages with 95% query-bootstrap intervals; Covered counts requests whose pool contains a reference. Selection averages three choices per covered request. The dated collection was gathered October 10–12, 2026; archive denotes saved outputs from the enumeration corpus. All collections share request IDs within a domain.

GPT-5.4 laptop enumeration and choice uses 1,056 calls on the same 176 requests. Coverage is 159/176 (90.3%), with reference selection in 119/477 covered choice slots (24.9%). All 528 choices agree under the historical and strict parsers. The same catalog filter gives 157/176 coverage (89.2%) and 119/471 conditional selections (25.3%). Broad family references and unconstrained product dates remain; these rates describe reference agreement on model-specific pools. Opus enumeration and choice uses 294 calls on the same 49 headphone requests, covering 37/49 (75.5%) and selecting a reference in 39/111 covered choice slots (35.1%). All 147 choices agree under both parsers; source/identity sensitivities follow below.

Applying the same catalog filter to the October 2026 GPT-5.4 collection changes coverage to 78.9% for cars and 53.1% for headphones, with conditional selection of 30.9% (135 covered requests) and 25.6% (26 requests). Pools from either collection can contain products introduced after the original posting date because the prompts leave availability unconstrained.

Headphone source and identity sensitivities. An assistant audit traced all 197 stored references on the same 49 headphone requests to 101 complete source comments. It labeled 52 exact endorsements, 18 conditional endorsements, 24 family-level endorsements, 30 unendorsed mentions, 71 fragments/nonproducts, and two unresolved identities. Exact endorsements remain for 28 queries; the broad rule retaining exact, conditional, and family-level endorsements covers 40. Labels were frozen before per-model scoring. Missing or filtered references remain unassessable source cases, not objective model failures.

With the unchanged matcher, eligible-only broad coverage is 47.5–72.5%, below laptop coverage for all six models (Table 23). The individual intervals for the domain gaps exclude zero, but omit annotation uncertainty and are not multiplicity-adjusted. A separate blinded review of 153 historical positive identity matches found 62 supported, 22 unresolved, and 69 rejected matches. Restricting broad coverage to supported identities yields 15, 20, 13, 12, 10, and 11 covered queries out of 49 in the table’s model order. This is a veto sensitivity of existing positive matches; absent aliases and unextracted source recommendations are not recovered.

ModelRaw /49Exact /49Broad /49Broad /40 (%)Laptop gap (pp)
GPT-534122460.0 [44.4, 74.4]29.0 [13.7, 45.5]
Claude Opus37172972.5 [57.9, 85.7]19.0 [5.1, 34.0]
Llama-70B2882050.0 [35.3, 65.8]45.7 [29.8, 60.6]
Qwen-80B2972050.0 [34.2, 65.0]42.7 [26.9, 58.9]
Nemotron-49B2471947.5 [32.4, 63.4]45.8 [29.5, 61.4]
Llama-8B2681947.5 [32.5, 63.4]44.6 [27.7, 59.8]

Table 23. Headphone coverage under source-reference filtering. The first three numeric columns count covered queries among the same 49. Broad /40 is coverage on source-eligible queries, with 95% query-bootstrap intervals; Laptop gap subtracts it from raw laptop coverage on 164 requests. Intervals use 4,000 resamples (seed 20261010), independently sampling domains and pairing model rows within each domain. Laptop references lack an equivalent source audit.

The same source rules also change conditional selection (Table 24). We use Llama-70B’s archived displayed pools and GPT-5.4’s October 2026 pools from Table 22, not the GPT archive in the six-model coverage table. Three choices remain per query; source rules change both matching choices and covered denominators. These ratios are sensitivities, not ordered bounds or model-quality rankings. The raw-reference rows reproduce the original results. Applying the identical source labels and identity rules to Opus gives 28/49 broad-reference coverage and 21/84 conditional selections (25.0%). Requiring supported identity matches leaves 17/49 covered requests and 3/51 matching choices (5.9%); four name pairs were reviewed before aggregation.

RunReference ruleCoverageSelected / covered slotsSelection (%; 95% CI)
Llama-70BRaw28/4933/8439.3 [22.2, 57.0]
Llama-70BExact8/496/2425.0 [0.0, 60.0]
Llama-70BBroad20/4920/6033.3 [13.3, 54.3]
Llama-70BBroad + identity13/499/3923.1 [0.0, 50.0]
GPT-5.4Raw32/4923/9624.0 [9.9, 39.4]
GPT-5.4Exact14/496/4214.3 [0.0, 35.7]
GPT-5.4Broad26/496/787.7 [0.0, 20.0]
GPT-5.4Broad + identity16/496/4812.5 [0.0, 31.3]
Opus 4.8Raw37/4939/11135.1 [20.0, 51.3]
Opus 4.8Exact15/496/4513.3 [0.0, 33.3]
Opus 4.8Broad28/4921/8425.0 [10.3, 42.3]
Opus 4.8Broad + identity17/493/515.9 [0.0, 20.0]

Table 24. Headphone selection under the same source-reference rules. Each run retains 49 queries and three recorded choices per query. Conditional selection divides matching choices by three times that run’s covered-query count. Intervals use 4,000 paired query-cluster resamples (seed 20261010), recomputing the denominator each time. Exact and broad references survive on 28 and 40 queries; Broad + identity additionally requires a supported positive match.

The stored references are incomplete even after filtering. For one request, all three GPT choices name Sony WH-1000XM2, which a source comment endorses, but the stored complete reference names only WH-1000XM3. The fixed no-addition filter therefore leaves genuine source-endorsed choices uncounted. No independent human annotation, complete alias recovery, or suitability judgment is supplied by this audit. Llama’s archive contains 147 valid picks but lacks historical raw responses and transport records; GPT’s 294 successful enumeration and choice calls retain full logs. The Opus run also retains all 294 prompts and responses and underwent a fixed ten-query raw audit and supplemental identity review. Collection date and model-specific pools remain different across runs.

Hardware coverage and selection audit. We recover 234 archived Llama-70B Hardware Recommendations queries, the complete predetermined subset selected by the original hash-bucket rule from 1,266 reference-bearing questions. Every chooser pool matches its saved enumeration prefix. The archive retains 701 of 702 planned choices; the missing choice contributes zero. A generative assistant reviewed all 296 reference entries against their full source answers, independently of pool and choice outcomes. Of these, 235 identify an endorsed specific product, 49 name families or ranges, and 12 are unendorsed, truncated, compound, or unresolved. These annotations assess source endorsement, not product suitability, and are not independent human gold. Candidate existence, historical availability, and complete configurations were not validated. Raw generations and historical serving identifiers are unavailable, preventing an independent choice-parser audit. Intervals use 2,000 query-cluster resamples with seed 42, retaining choices together and recomputing the covered denominator. They exclude reference and matcher uncertainty.

On the 194 queries with a specific-product reference, conservative name-equivalence matching gives coverage of 21/194 (10.8%) at K=10K = 10 and 36/194 (18.6%) at K=40K = 40. Among the 36 covered queries, 31/108 planned choices select a reference (28.7%); excluding the missing choice gives 31/107 (29.0%). Table 25 separates exact lexical matching, audited name equivalence, source-explicit identifiers, and family-level credit. Across all stored references, the historical liberal matcher instead gives 64/234 coverage (27.4%), illustrating why reference quality and identity rules matter.

Reference and identity ruleCoverage at K=40K = 40Conditional selection
Specific; strict lexical22/194: 11.3 [7.1, 16.2]19/66: 28.8 [12.0, 49.3]
Specific; audited equivalence36/194: 18.6 [13.1, 24.5]31/108: 28.7 [15.1, 44.4]
Specific; source-explicit identifiers37/194: 19.1 [13.6, 25.0]34/111: 30.6 [17.2, 45.9]
Specific or named family51/226: 22.6 [17.4, 28.1]45/153: 29.4 [18.2, 42.5]

Table 25. Audited Llama-70B hardware coverage and selection. Cells give counts, percentages, and 95% query-bootstrap intervals. Selection uses three planned choices per covered query. Reference eligibility and identity rules differ across rows; family credit does not establish exact-product identity.

This single-model subset provides descriptive reference agreement, not a controlled domain comparison or an objective failure rate. Some requests concern components, reuse, or software rather than a new hardware purchase, and some already name products.

GPT-5.4 collection audit. The October 2026 GPT-5.4 runs preserve every request payload, raw response, and returned model identifier. We inspected the first ten frozen cases in each coverage run, including all enumerations and choices, and sampled archived pools across the six models and three domains. The inspection was performed by a generative assistant. We verified the prompts, shared request IDs, reference lists, and parser agreement before calculating scores and catalog sensitivities. Intervals use 2,000 query resamples with seed 42, keeping all three choices together within each request.

Enumeration and choice protocol. Enumeration requested three lists at temperature 0.9, nominal seeds 17–19, and a 500-token output limit, requesting 20 names per list. The line parser removes leading bullets or numbering and surrounding spaces, asterisks, and backticks, then discards text after a colon or a whitespace-delimited dash. It retains names of 3–59 characters unless they begin with here, sure, or note, taking at most 25 names per response. The three lists are concatenated in sample order and deduplicated case-insensitively, retaining first occurrences. Actual list lengths and identities can differ from the requested 20 distinct names. The chooser receives up to the first 40 names, with three choices at temperature 0.7, nominal seeds 42–44, and a 64-token output limit. It maps the first integer in the response to the corresponding candidate if that integer is in range; otherwise the choice is invalid. Hosted endpoints may not guarantee deterministic seeds.

The paired reanalysis intersects saved Llama-70B chooser records with the free-form evaluation IDs, yielding 171/176/49 queries. Their reference lists agree exactly across the two artifacts. The mean displayed pool sizes are 36.4/36.2/34.8. A pool shorter than KK is evaluated in full, and a failed or invalid choice contributes zero to the three-sample mean. We score each selected name directly against the same raw reference set used for the free-form output. Numerical choices have a different parsing interface from free text. The extraction

asymmetry documented below contributes to the measured score difference.

Free-form extraction sensitivity. The saved laptop protocols differ by 16.9 percentage points (Table 26), but that difference is not reliable evidence that enumeration improves the recommendation. The free-text extractor recovers a product in only 75.0% of laptop samples; all saved numbered choices in this matched comparison are valid. Moreover, 84 of the 89 net additional matched samples occur on the 54 requests with at least one unextractable free-form answer. An explicit “Dell XPS 13” can fail catalog extraction while receiving credit when supplied directly as a chooser’s selected name. Thus a recognizable recommendation can be penalized on only one side of the comparison.

DomainnnCoverage@10Coverage@40Free formEnum. + choiceChange [95% CI]
Cars17159.676.625.320.3−5.1-5.1 [−10.9-10.9, +0.8+0.8]
Laptops17685.895.528.044.9+16.9+16.9 [+7.6+7.6, +26.1+26.1]
Headphones4949.057.122.422.40.0 [−10.9-10.9, +12.2+12.2]

Table 26. Candidate coverage and observed final-choice scores. Llama-70B on matched queries; all values except nn are percentages or percentage-point differences. Coverage@KK measures reference presence among the first KK enumerated options. Both final-choice scores average three samples; failed choices score zero. Intervals on the paired change are 95% query-bootstrap intervals. Additional enumeration calls and different answer-extraction interfaces prevent interpreting the change as an isolated improvement in recommendation.

Among the other 122 laptop requests, where all three free-form answers are extractable, the observed gap is 1.4 points, with a 95% query-bootstrap interval of [−9.6, 12.3]. This selected subset is a sensitivity check, not a corrected estimate of the full-sample effect. A shared, validated identity-extraction procedure is needed before interpreting the full-sample difference as a gain.

Within-pool selection controls. Uniformly selecting a displayed slot gives mean reference agreement of 7.8% on cars, 11.6% on laptops, and 3.2% on headphones under these exact pools and matching rules. The LLM chooser’s point estimates are higher in all three domains. These task-specific baselines count every matching slot.

A stronger within-pool baseline selects the first enumerated name, without an additional chooser call. It obtains 25.1% on cars, 36.4% on laptops, and 16.3% on headphones. Chooser-minus-first differences are respectively −4.9-4.9, +8.5+8.5, and +6.1+6.1 points, with 95% paired query-bootstrap intervals [−11.7-11.7, 1.9], [−0.2-0.2, 17.0], and [−6.8-6.8, 19.0]. All three intervals include zero. Both sides use direct product-name matching, so this comparison avoids the free-text catalog gate. The first option itself comes from a query-conditioned generation.

The sensitivity calculations use 10,000 paired query-bootstrap resamples with seeds derived from stable hashes. They reuse the same saved responses without new model calls.

The request body supplied to enumeration and choice is truncated at 1,500 characters, compared with 1,800 for free-form generation. A cost- and format-controlled follow-up should use identical query content, equal inference budgets, and the same answer interface. The present comparison measures the full saved protocols.

Teacher-forced likelihood lift. Factual question-answering studies link recall to entity popularity and pretraining support [19, 12]. Our product-name scores probe a related distinction, but do not establish whether a model knows a product’s attributes. The archived open-weight experiment scores a reference name as a continuation. For answer tokens a1,…,aTa_1,\ldots,a_T, define ℓ(a∣x)=T−1∑t=1Tlog⁡p(at∣x,a<t)\ell(a \mid x) = T^{-1}\sum_{t=1}^{T}\log p(a_t \mid x,a_{<t}). Lift is ℓ(a∣xq)−ℓ(a∣x0)\ell(a \mid x_q)-\ell(a \mid x_0), where xqx_q includes the request and x0x_0 does not. Both conditions score the identical token sequence under the model’s chat template. The car prompts are:

Without request: “Recommend one specific car. Reply with the product name only.”

With request: [request text], followed by “Recommend one specific car for this request. Reply with the product name only.”

The product category changes with the domain. The request is the title and body truncated jointly to 900 characters. The reference is the first stored name recognized by the purged catalog or Stack Exchange lexicon. Evaluation uses bfloat16 and teacher-forces answer tokens without scoring an end-of-sequence token. The model identifiers are meta-llama/Llama-3.1-8B-Instruct, unsloth/Llama-3.3-70B-Instruct, and Qwen/Qwen3-Next-80B-A3B-Instruct.

Table 27 reports all three models across five domains. The existing cluster runs were recovered without new inference. We checked that every saved answer matches the corresponding input and verified the paired score differences. Intervals quantify variation across requests, conditional on the archived model scores rounded to three decimals; they do not include checkpoint or numerical-precision variability.

DomainModelQueriesMedian lift [95% CI]Lift > 0 (%)Lift > 0.5 (%)
CarsLlama-8B1,2921.21 [1.09, 1.34]76.966.0
CarsLlama-70B1,2920.44 [0.21, 0.77]55.349.5
CarsQwen-80B1,2921.61 [1.42, 1.77]73.267.0
LaptopsLlama-8B9020.85 [0.75, 0.94]74.861.5
LaptopsLlama-70B9021.42 [1.19, 1.70]68.560.1
LaptopsQwen-80B9021.84 [1.66, 2.00]83.673.2
HeadphonesLlama-8B1981.02 [0.83, 1.22]79.865.7
HeadphonesLlama-70B1980.83 [0.26, 1.08]59.153.0
HeadphonesQwen-80B1981.16 [1.02, 1.37]81.373.7
SE softwareLlama-8B2,4453.24 [3.14, 3.36]93.790.0
SE softwareLlama-70B2,4453.82 [3.64, 3.98]84.681.4
SE softwareQwen-80B2,4456.25 [6.04, 6.44]97.195.9
SE hardwareLlama-8B1,2661.49 [1.41, 1.56]92.482.5
SE hardwareLlama-70B1,2661.96 [1.74, 2.13]80.674.4
SE hardwareQwen-80B1,2662.19 [2.08, 2.30]93.487.6

Table 27. Effect of the request-specific prompt on reference-name likelihood. Lift is the paired change in mean log-probability, in nats per answer token. Medians retain the archived upper-median convention, with 95% intervals from 2,000 query-bootstrap resamples. Fractions are descriptive counts over the same queries; the 0.5 threshold is the original probe’s summary threshold. Positive lift indicates increased name likelihood.

This probe measures prompt-conditioned name likelihood. Inferring product knowledge would require additional evidence about the model’s knowledge of its attributes. Both the request and the phrase “for this request” change, so lift is not an isolated effect of the request text. References may already appear in the request, and a likelihood decrease can reflect an unsuitable reference. The probe does not rank all competing products or reproduce multi-item enumeration.

Linking likelihood lift to pool omissions. We joined the saved Llama-70B car scores to enumeration pools by exact reconstructed request text and query ID. The primary cohort is the 156-request overlap with the fixed 171-query enumeration sample; 15 requests lack a score under the historical lift eligibility rule. The target is the first catalog-valid stored reference, which is the first raw reference for 102/156 requests. Target omission therefore differs from omission of every stored reference. Candidate-wise matching at K=40K = 40 reproduces all 156 existing any-reference coverage labels.

Included targets have 2.28 more nats/token mean lift than omitted targets (95% CI 1.06–3.56; Table 28). Yet positive lift occurs for 28/69 omitted targets, including 9/25 requests where every stored reference is absent. The association also appears among 74 requests with complete inputs within the 900-character limit and no full target-name mention in either prompt (difference 2.52, CI 0.87–4.19), and in the separate 577-request archive overlap (2.09, CI 1.44–2.72).

Pool outcomeQueriesMean lift [95% CI]Positive lift (%)
Target present871.75 [0.87, 2.66]58 / 87 (66.7)
Target absent69−0.52 [−1.42, 0.32]28 / 69 (40.6)
All references absent (subset)25−0.76 [−1.95, 0.49]9 / 25 (36.0)

Table 28. Reference-name likelihood lift and explicit pool inclusion. Llama-70B artifacts on 156 matched car requests, with coverage at K=40K = 40. The last row is a subset of target-absent requests. Lift is measured in nats per target token; mean intervals use 2,000 query-bootstrap resamples (seed 20261010). The final column gives counts and percentages with positive lift. Scores and pools come from different prompts and serving implementations.

The protocol fixed these cohorts and sensitivity checks before computing the association. Assistant review covered 12 fixed raw cases and all 67 disagreements with exact surface matching. Some references are incidental or critical mentions. No outcome-dependent reference cleaning was applied. The local scorer is unsloth/Llama-3.3-70B-Instruct; the historical pool-serving alias lacks a common checkpoint fingerprint. Scoring truncates requests at 900 characters, versus title plus 1,500 body characters for enumeration. These differences make the result a descriptive association between contextual name likelihood and pool inclusion; it does not establish product knowledge, suitability, or a causal explanation of omissions.

Query-conditioned choice details

Table 29 summarizes supplied-choice results, including the separate ESCI test described in Appendix § 19.

The payload selects the first reference string recognizable by the purged catalog vocabulary (Reddit) or product lexicon (Stack Exchange). Query text is the title and body truncated jointly to 900 characters. Distractors are the first four nonmatching names among the first 60 entries of a popularity-ordered list. Options are shuffled once per query and kept fixed across the two prompt conditions within that run.

Both prompt conditions use temperature zero and a 200-token response budget. Parsing first seeks an explicit CHOICE: <number>, case-insensitively and allowing an optional # before the number. If that number is absent or out of range, the first in-range integer anywhere in the response is used. Invalid responses count as misses. Stored records contain query text, the reference, its position, and two hit flags, but not the complete option list or raw responses. The shuffle uses Python’s process-dependent string hash, so its nominal seed does not guarantee identical positions across different model runs. We can reproduce the reported hit rates and paired intervals; individual choice and parsing decisions remain partly untraceable.

Table 30 reports both prompt conditions across all five domains; Table 32 gives paired intervals for their differences. Reference selection increases all 20 model–domain cells.

ModelCarsLaptopsHeadphonesSE softwareSE hardware
GPT-55.0→59.45.0 \to59.416.4→54.016.4 \to54.037.9→54.037.9 \to54.01.8→96.71.8 \to96.72.4→94.62.4 \to94.6
Claude Opus7.4→62.57.4 \to62.514.9→58.514.9 \to58.533.3→51.533.3 \to51.51.3→96.31.3 \to96.36.2→94.46.2 \to94.4
Llama-70B7.8→57.07.8 \to57.027.7→60.527.7 \to60.528.8→51.028.8 \to51.03.5→94.43.5 \to94.416.1→91.316.1 \to91.3
Llama-8B9.0→56.99.0 \to56.920.0→51.120.0 \to51.118.2→46.018.2 \to46.02.5→93.72.5 \to93.710.0→92.310.0 \to92.3
Queries1,2929021982,4451,266

Table 30. Query-conditioned choice in a supplied five-option pool. Percentage choosing the reference without →\to with the request. The instruction changes from “best in general” to “best for this user.” Models share query identities within each domain; options are fixed across prompt conditions within each run.

DomainGPT-5Claude OpusLlama-70BLlama-8B
Cars54.4 [51.5, 57.5]55.2 [52.2, 58.2]49.2 [46.1, 52.2]47.9 [44.9, 51.0]
Laptops37.6 [33.9, 41.5]43.7 [39.6, 47.6]32.8 [28.5, 36.9]31.2 [27.2, 35.3]
Headphones16.2 [7.1, 25.8]18.2 [10.1, 26.8]22.2 [13.1, 31.3]27.8 [19.2, 35.9]
SE software94.9 [93.9, 95.8]95.0 [94.1, 95.8]90.9 [89.7, 92.0]91.1 [89.9, 92.4]
SE hardware92.3 [90.8, 93.7]88.2 [86.3, 89.9]75.2 [72.7, 77.6]82.3 [80.2, 84.5]

Table 32. Paired query-lift estimates. Percentage-point change from query-blind to conditioned selection, with 95% query-bootstrap intervals. All estimates use paired saved hit flags. They do not measure uncertainty in labels, distractor construction, or parsing.

A candidate-construction control. We reconstructed the unordered option sets using the saved references, the current popularity-ordered list, and the four-distractor rule specified above. Reference identities and query sets agree across all four models and with the payload. The control selects the option with the largest index in the popularity list; it uses no request text and no fitted parameters. Because the four distractors come from the head of that list, this rule often identifies the inserted reference. There are only five or six distinct distractor tuples per domain. The control is invariant to option order, but the archived records do not establish that every historical option list exactly matches this reconstruction. This diagnostic is analogous to hypothesis-only checks for annotation artifacts in natural-language inference [6]: it tests whether construction cues support prediction without the intended input.

Table 31 rules out treating the high conditioned hit rates as sufficient evidence of distinguishing subtle preferences. It does not establish that the models used the same shortcut. A separate token-overlap control, choosing the option sharing most normalized tokens with the request and averaging uniformly over ties, scores only 31.3%, 30.3%, 34.5%, 39.5%, and 54.6% in the table’s domain order. Thus this simple lexical rule does not explain the models’ high rates either.

DomainRank control (%)Best model (%)Difference [95% CI]
Cars86.262.5+23.7 [+20.7, +26.7]
Laptops64.360.5+3.8 [+0.2, +7.3]
Headphones88.454.0+34.3 [+26.3, +42.4]
SE software97.396.7+0.6 [-0.3, +1.6]
SE hardware97.994.6+3.3 [+2.0, +4.7]

Table 31. A query-free shortcut in reconstructed choice sets. Reference-selection rates for the least-popular-option control and the highest query-conditioned model point estimate in each domain. Differences have 95% paired query-bootstrap intervals over the saved query sets (10,000 resamples). The control uses construction metadata unavailable in the model prompts. Intervals are exploratory and not corrected for selecting the best model or multiple comparisons.

The historical prompt comparison changes both the instruction and the request, with the reference supplied in both conditions. The balanced-pair test holds the instruction fixed and varies request content on frequency-balanced candidate pairs.

Balanced-pair request control. We test 94 car requests with GPT-5.4, Llama-70B, Claude Opus 4.8, and Qwen-80B, using the same instruction with the real request, no request, or a deterministically shuffled request. Each query has one stored reference and one alternative from its archived Llama candidate pool, presented in both orders: 564 calls per model and 2,256 in total. Pairs share a coarse catalog category and have corpus-reference frequencies within a factor of two. The reference is more frequent in 47 pairs and less frequent in 47, so a frequency-only rule selects it 50% of the time. Six source-only category exclusions from an initial 100-query sample were fixed before outcomes, without replacements. Generic model-family opinions and source mentions do not establish verified suitability.

Real-request reference agreement is 45.7% for GPT and 29.3% for Llama, versus 15.4% and 5.9% under shuffled requests (Table 33). Paired real-minus-shuffled differences are +30.3 points (95% CI [19.1, 41.5]) and +23.4 points ([13.3, 34.0]), using 5,000 whole-query bootstrap resamples (seed 20261013). Opus gives 44.7% agreement with real requests and 13.3% with shuffled requests, a paired increase of 31.4 points ([20.7, 42.0]). Qwen gives 36.7% versus 9.0%, an increase of 27.7 points ([17.0, 38.3]). Without a request, GPT, Llama, and Opus always report insufficient information; Qwen does so 179/188 times and reports equal suitability nine times. Shuffling often changes the relevant product category and elicits rejection of both options.

ModelRequestReferenceAlternativeEqualNeitherInsufficientInvalid
GPT-5.4Real8665142120
GPT-5.4Absent00001880
GPT-5.4Shuffled29271012110
Llama-70BReal55451718521
Llama-70BAbsent00001880
Llama-70BShuffled11135107484
Opus 4.8Real84492717110
Opus 4.8Absent00001880
Opus 4.8Shuffled25261012070
Qwen-80BReal69561630152
Qwen-80BAbsent00901790
Qwen-80BShuffled1723413563

Table 33. Balanced-pair selection outcomes by request condition. Counts out of 188 presentations per row (94 requests, both option orders), at temperature zero and seed 42. Equal, neither, and insufficient-information responses are distinct from choosing an option; invalid outputs remain in the denominator.

The models also differ in willingness to choose: real-request decisive rates are 151/188 for GPT and 100/188 for Llama. Conditional on choosing either option, reference agreement is 86/151 (57.0%; 95% CI [46.2, 67.4]) and 55/100 (55.0%; [41.8, 68.0]), respectively. Thus, the unconditional difference cannot establish a selection-accuracy advantage.

Opus chooses an option in 133/188 presentations and selects the reference in 84/133 decisive choices (63.2%; [51.7, 73.7]). Qwen selects the reference in 69/125 decisive choices (55.2%; [43.6, 66.9]). These outcomes measure request-sensitive choices, rejection, and abstention; it retains coarse categories, imperfect source references, and conditioning on an archived Llama pool.

All payloads, option orders, serving identifiers, and outcome mappings were checked. Assistant review covered all 144 responses on a fixed 12-query audit and all five malformed outputs, followed by all 72 responses each for Opus and Qwen on the same audit queries and all five malformed Qwen outputs. A symmetric parser accepts one whole outer JSON fence; malformed content is not repaired. The initial calls use openai/openai/gpt-5.4 and nvidia/meta/llama-3.3-70b-instruct; an earlier route attempt failed. The Opus collection uses azure/anthropic/claude-opus-4-8 with unchanged pairs, donor requests, and scoring. Two Opus responses require outer-fence removal; both report insufficient information without a request. The Qwen collection uses nvidia/qwen/qwen3-next-80b-a3b-instruct. Its five malformed outputs contain unescaped quotes and remain invalid.

Separating product switches from non-selection.

A retrospective reanalysis pairs each real-request outcome with its shuffled-request outcome for the same query and option order. The change in the reference-selection indicator partitions exactly into three net contributions: switches between the alternative and the reference; transitions between a valid non-selection (equal, neither, or insufficient information) and the reference; and transitions between an invalid output and the reference. Each transition into the reference contributes +1, and its reverse contributes −1, using all 188 presentations per model as the denominator. This accounting retains non-selections and parser failures, without conditioning on a changed set of decisive answers.

The net non-selection contribution is 22.9–28.7 percentage points, whereas product-to-product switches contribute 0.0–4.3 points (Table 34). All product-switch intervals include or touch zero. Net contributions can conceal changes in both directions: Qwen switches from alternative to reference in seven presentations and from reference to alternative in seven. The overall agreement increase thus mainly reflects transitions between non-selection and selecting the reference. It does not demonstrate improved ranking among suitable products, especially when shuffled requests change the relevant product category.

ModelTotal Δ\DeltaProduct switchNon-selectionInvalid
GPT-5.430.3 [19.1, 41.5]1.6 [-3.2, 6.4]28.7 [18.6, 39.4]0.0 [0.0, 0.0]
Llama-70B23.4 [13.3, 34.0]0.5 [-1.6, 2.7]22.9 [13.3, 33.0]0.0 [0.0, 0.0]
Opus 4.831.4 [20.7, 42.0]4.3 [0.0, 9.0]27.1 [16.5, 37.2]0.0 [0.0, 0.0]
Qwen-80B27.7 [17.0, 38.3]0.0 [-4.8, 4.8]26.6 [18.1, 36.2]1.1 [0.0, 2.7]

Table 34. Decomposition of real-minus-shuffled reference agreement. Net contributions in percentage points, with descriptive 95% intervals from 5,000 paired whole-query resamples (seed 20261013); each resample retains both option orders and all models. The three contributions sum to the total before rounding. Intervals are not adjusted for multiple comparisons. Non-selection includes equal, neither, and insufficient; invalid outputs remain in the denominator. These are reference-agreement changes, not correctness scores or causal mediation estimates.

The analysis uses all 2,256 saved responses. Validation reproduces the condition counts and total-effect intervals. An assistant checks all 48 real/shuffled responses on three hash-selected requests and all ten malformed responses. Declared choices determine the outcome; incorrect product assertions in explanations are not treated as evidence of suitability. The transition analysis neither adjudicates those assertions nor provides independent human validation.

ESCI selection protocol

We filter to US records with small_version==1, pooling the official train and test splits. Queries must begin with a lowercase ASCII letter or digit, contain two to six whitespace-separated tokens, and contain neither # nor !. Each retained query must have at least one Exact product and at least three Irrelevant products with available titles. These surface restrictions define the sampling frame.

We sample query IDs using Python’s random.Random(0) over sorted eligible IDs, then use a query-ID seed to choose one Exact and three Irrelevant products and shuffle their titles. Each title is truncated to 120 characters. Models are prompted at temperature zero with a 120-token output limit. The parser first seeks an explicit CHOICE: <number>, then falls back to an in-range number. An invalid selection is scored as a miss. The artifacts contain the same 500 query IDs across models and respectively 368, 351, 337, and 293 hits for Claude, GPT-5, Llama-70B, and Llama-8B. Qwen-80B uses the identical queries, candidate titles, ordering, and prompt (collected October 10, 2026): 339/500 Exact-label matches (67.8%). All 500 outputs are valid under both products and preserving raw selections are necessary to identify the remaining error sources.

the historical and strict first-line parsers. This run retains raw responses and returned model identifiers; the other four model records retain only hit flags.

Table 35 reports intervals for each hit rate. The point estimates span 15 percentage points across models, and Llama-70B exceeds Llama-8B by 8.8 points. Claude and GPT-5 miss the Exact product on 132/500 and 149/500 queries, respectively. The GPT, Claude, and two Llama records retain hit flags rather than raw responses, preventing a retrospective separation of reasoning errors, invalid outputs, title truncation, and label noise. The Qwen run permits inspection of raw choices. In its first ten frozen cases, the query “helen butler” has a license-plate lamp as its Exact-labeled option, illustrating that not every miss can safely be called a selection error.

ModelHit rate (%)95% CI
Claude Opus73.6[69.6, 77.3]
GPT-570.2[66.0, 74.0]
Llama-70B67.4[63.2, 71.4]
Qwen-80B67.8[63.6, 71.7]
Llama-8B58.6[54.2, 62.8]

Table 35. Externally labeled supplied choice. Exact-reference hit rate on the same 500 ESCI queries, with 95% Wilson intervals. Uniform choice is 25%. Higher means greater agreement with the designated label.

The larger Llama wins on 80 discordant queries and loses on 36 against the smaller model; a two-sided exact paired test gives p≈5.4×10−5p \approx5.4 \times10^{-5}. The GPT-5–Llama-70B contrast has 47 wins and 33 losses (p≈0.146p \approx0.146). These exploratory comparisons provide evidence for the larger Llama’s advantage, while the GPT-5–Llama-70B difference remains inconclusive.

These results describe the specified sampling frame; they do not establish that the surface filters causally reduce label noise. Auditing the displayed

Artifact attribution and reproducibility details

Artifact origins

The Reddit requests and community answers originate from contributors to r/whatcarshouldIbuy, r/SuggestALaptop, and r/HeadphoneAdvice. The Stack Exchange material originates from contributors to Software Recommendations and Hardware Recommendations, distributed through Stack Exchange’s data dumps.1 § 3.2 and Appendix § 10 document the cohorts and filtering. The externally labeled shopping subset derives from Amazon’s Shopping Queries Dataset (Reddy et al., [28]); Appendix § 19 specifies its construction and evaluation scope.

The evaluated open-model families are credited to Meta (Meta, [21]), Qwen (Qwen, [27]), NVIDIA (NVIDIA, [25]), and Mistral AI (Mistral AI, [24]). The hosted frontier models are from OpenAI (OpenAI, [26]) and Anthropic (Anthropic, [1]). These references identify the model creators; the requested serving identifiers and their historical provenance limits are documented in Appendix § 11. In particular, the Mistral gateway alias differs from the official Ministral-3-14B-Instruct-2512 model-card name. The archived Llama-70B likelihood probe used the Unsloth distribution unsloth/Llama-3.3-70B-Instruct, which credits Meta’s original model.2 This checkpoint should be distinguished from the hosted Llama route.

Software and evaluation settings

Preprocessing and evaluation use Python, NumPy (Harris et al., [7]), and pandas (McKinney, [20]), with Requests for HTTP clients and PyArrow for Parquet data.3 The recorded full-request, open-model car, and temperature-zero client records report Python 3.8.3, NumPy 1.24.4, pandas 1.0.5, and Requests 2.24.0. The laptop coverage/selection client record reports pandas 1.5.3 and Requests 2.32.4 with the same Python and NumPy versions. Both profiles record PyArrow 16.1.0. The final headphone connection audit separately records Python 3.8.3 and NumPy 1.24.4, with no GPU use. These are per-run client/analysis settings; they do not describe the providers’ model-serving environments.

Product normalization, alias matching, reference extraction, and choice parsing follow the task-specific rules described in Appendices § 12–§ 14, including normalization scope, malformed-output handling, denominators, and uncertainty estimation. Bootstrap resampling uses NumPy; the experiment-specific appendices give the resampling unit, replicate count, and seed or deterministic seed derivation. Prompts, generation limits, temperatures, and ordering controls are reported in Appendix § 11 and the corresponding experiment appendices. These are inference experiments, with no model training on the evaluation queries.

The archived teacher-forced likelihood implementation uses PyTorch and Hugging Face Transformers (Wolf et al., [35]).4 The model is loaded in bf​loat16 with automatic device placement and evaluated without gradients, one sequence at a time. Prefixes use the tokenizer’s chat template with a generation prompt. Prefix and answer tokens are encoded separately with add_special_tokens=False; the mean answer-token log probability is computed using float32 log-softmax, and archived scores are rounded to three decimals.

Compute resources

The open-model names indicate nominal scales of 8B and 70B parameters for Llama, 80B total and 3B active for Qwen3-Next, 49B for Nemotron, and 14B for Ministral. Exact parameter counts for the hosted GPT and Claude models are unavailable in the retained service records. The recorded local client/analysis host is Linux x86-64 with 20 logical CPUs and approximately 31 GiB of RAM. Hosted generation runs use remote inference services; this host specification does not describe their accelerators. The archived likelihood probes ran on a Slurm-managed cluster.

The Ethical considerations section reports response counts and available token usage for the listed hosted-inference runs, including the 826-response temperature-zero check. These counts measure the evaluated workload; they are not estimates of provider-side GPU-hours, energy consumption, or total project compute.

Human involvement, data release, and AI assistance

No human annotators were recruited and no participant study was conducted. The experiments reuse public posts and saved model outputs. Source and response reviews described as assistant audits are AI-assisted checks, not independent human annotations. Public availability of the underlying posts does not establish individual consent for every downstream use; we make no claim of individual consent or institutional ethics-board approval or exemption.

No dataset, raw user-text collection, or model weights are being released with this submission. This release status does not certify that the working data are anonymized or cleared for redistribution. Any future data release requires the privacy, licensing, attribution, and removal-process review, including screening for identifying and potentially objectionable content. The present study does not establish demographic representativeness beyond the platform, language, domain, and sampling scope documented in § 3.2, Appendix § 10, and Limitations.

Generative AI assistants provided substantive support for experimental code, analysis and source checks, literature lookup, and manuscript drafting and revision. This assistance is distinct from evaluating LLMs as research objects. The authors are responsible for verifying the final evidence, citations, claims, and disclosures.

References

  1. [1]Anthropic. 2026. Introducing Claude Opus 4.8. Model announcement, 28 May 2026.
  2. [2]Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for YouTube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 191–198. ACM.DOI
  3. [3]Zhe Dong, Fang Qin, Manish Shah, and Yicheng Wang. 2026. Diagnosing and mitigating retrieval bottlenecks in LLM-based cold-start recommendation. Preprint, arXiv:2606.29947.arxiv.org/abs/2606.29947
  4. [4]Shuxian Fan, Seonwoo Min, Youna Hu, Botao Xia, Jayakrishnan Unnikrishnan, Rowan Musselman, Yi-fan Gao, Qingyu Yin, Priyanka Nigam, and Bing Yin. 2026. Shopping reasoning bench: An expert-authored benchmark for multi-turn conversational shopping assistants. Preprint, arXiv:2606.12608.arxiv.org/abs/2606.12608
  5. [5]Alexander Felfernig, Gerhard Friedrich, Dietmar Jannach, and Markus Zanker. 2015. Constraint-based recommender systems. In Recommender Systems Handbook, pages 161–190. Springer US.DOI
  6. [6]Suchin Gururangan, Swabha Sweyamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. Association for Computational Linguistics.arxiv.org/abs/1803.02324
  7. [7]Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, and 7 others. 2020. Array programming with NumPy. Nature, 585(7825):357–362.
  8. [8]Zhangkui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley. 2023. Large language models as zero-shot conversational recommenders. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 720–730. ACM.DOI
  9. [9]Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval (ECIR). arXiv:2305.08845.arxiv.org/abs/2305.08845
  10. [10]Dietmar Jannach. 2023. Evaluating conversational recommender systems: A landscape of research. Artificial Intelligence Review, 56(3):2365–2400.DOI
  11. [11]Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A survey on conversational recommender systems. ACM Computing Surveys, 54(5):1–36.DOI
  12. [12]Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 15696–15707. PMLR.arxiv.org/abs/2211.08411
  13. [13]Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.02406
  14. [14]Minbeom Kim, Hwanhee Lee, Joon suk Park, Hwaran Lee, and Kyomin Jung. 2025. AdvisorQA: Towards helpful and harmless advice-seeking question answering with collective intelligence. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ArXiv:2404.11826.arxiv.org/abs/2404.11826
  15. [15]Sachin Kumar, Chan Young Park, Yulia Tsvetkov, Noah A. Smith, and Hannaneh Hajishirzi. 2025. ComPO: Community preferences for language model personalization. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 8246–8279.
  16. [16]Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, and 11 others. 2023. Measuring faithfulness in chain-of-thought reasoning. Preprint, arXiv:2307.13702.arxiv.org/abs/2307.13702
  17. [17]Jan Malte Lichtenberg, Alexander Buchholz, and Pola Schwöbel. 2024. Large language models as recommender systems: A study of popularity bias. In Workshop on Generative Information Retrieval (Gen-IR) at SIGIR 2024.arxiv.org/abs/2406.01285
  18. [18]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2305.20050
  19. [19]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannēah Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822. Association for Computational Linguistics.
  20. [20]Wes McKinney. 2010. Data structures for statistical computing in Python. In Proceedings of the 9th Python in Science Conference, pages 56–61.DOI
  21. [21]Meta. 2024a. Llama-3.1-8B-Instruct. Model card. Accessed 13 October 2026.DOI
  22. [23]Sewon Min, Kalpesh Krish­na, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle­moyer, and Hannah Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100. Association for Computational Linguistics.
  23. [24]Mistral AI. 2025. Ministral-3-14B-Instruct-2512. Model card. Accessed 13 October 2026.
  24. [25]NVIDIA. 2025. Llama-3.3-Nemotron-Super-49B-v1. Model card. Accessed 13 October 2026.
  25. [26]OpenAI. 2026. Introducing GPT-5.4. Model announcement, 5 March 2026.
  26. [27]Qwen. 2025. Qwen3-Next-80B-A3B-Instruct. Model card. Accessed 13 October 2026.
  27. [28]Chandan K. Reddy, Luís Márquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping queries dataset: A large-scale ESCI benchmark for improving product search. Preprint, arXiv:2206.06588.arxiv.org/abs/2206.06588
  28. [29]Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912. Association for Computational Linguistics.arxiv.org/abs/2005.04118
  29. [30]Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of format restrictions on large language model performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1218–1236. Association for Computational Linguistics.
  30. [31]Huaixiao Tou, Ying Zeng, Yuemeng Li, Cong Ma, Muzhi Li, Minghao Li, Weijie Yuan, He Zhang, and Kai Jia. 2025. ShoppingComp: Are LLMs really ready for your shopping cart? Preprint, arXiv:2511.22978.arxiv.org/abs/2511.22978
  31. [32]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36. ArXiv:2305.04388.arxiv.org/abs/2305.04388
  32. [33]Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. 2023. Rethinking the evaluation for conversational recommendation in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10052–10065, Singapore. Association for Computational Linguistics.DOI
  33. [34]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837.arxiv.org/abs/2201.11903
  34. [35]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.DOI
  35. [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuihan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc.

Paper details

vlnc:W_du4uf4ea
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents