Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark]

PVLDB Reference Format:

Jingru Yang, Honglu Zhang, Jiayi Mi, Ju Fan, Zheng Chen, and Xiaoyong Du. Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning. PVLDB, 20(1): XXX-XXX, 2027.

doi:XX.XX/XXX.XX

PVLDB Artifact Availability:

The source code, data, and/or other artifacts have been made available at https://github.com/hinsonver/CoRe-Benchmark.

Introduction

Data cleaning methods detect and repair erroneous values and are commonly evaluated on a small set of public benchmark tables [2, 10, 22, 52]. Large language models (LLMs) have opened new approaches to data management [17], including few-shot error detection and imputation competitive with specialized systems [37]. Recent cleaners use LLMs to edit values, derive rules or write cleaning programs [41, 42, 64, 72]. These methods, however, can use information beyond the table being cleaned.

Public cleaning benchmarks contain familiar hospitals, films, breweries and articles. Pretrained models can retrieve entity facts from their parameters [34, 48], while public tabular datasets can carry signals of prior exposure [6, 58]. A correct repair can therefore combine benchmark-row memory, knowledge of the entity (entity knowledge), knowledge of English word forms (spelling knowledge), and evidence within the table. Scores on familiar entities need not predict repair quality on unfamiliar data.

Example 1.1. Movies contains two distinct recovery problems. A Rotten Tomatoes slug can stand in place of an IMDb identifier. For American Ultra, recovering tt3316948 from american_ultra requires the film’s mapping; a fictitious slug does not determine its random identifier. Year ranges, however, retain the same candidates after renaming. DeepSeek-V4-Pro’s repairs decline in both cases (see Figure 1), although the year candidates remain available.

Vector (PDF)

Controlled tabular evaluations already test recognition and completion of dataset instances (KnowTable [58]) and adherence to counterfactual database values (ContraTable [63]). Cleaning requires deciding which cells to change and which source should supply their values. It also scores each edit against a reference, which may itself contain a systematic conversion.

We introduce CoRe (Cleaning or Recalling) to compare repairs on dirty/reference pairs under controlled entity replacements. Each benchmark is compared with fresh real records, a word-based surrogate with fictitious English-word names, and a syllable-based surrogate with meaningless names. The original and its surrogates share rows and error positions, while fresh records carry the same kinds of errors. Paired substitutions test entity familiarity and word form. Fresh records test performance beyond the benchmark, while a separate continuation probe tests for benchmark-row memory.

We transfer the benchmarks’ error mechanisms to new records and renamed identity values, retaining error positions and non-identity values in the paired surrogates. This is particularly important for Movies, where errors often reflect differences between two legitimate sources. Fictitious names must also hide the entity without themselves looking erroneous. We build 21 controlled tables: nine each for Hospital and Movies across three seeds, and three for Beers. Their surrogates contain about 41,000 fictitious values, assessed by 506 acceptance checks. We evaluate four commercial and four open-weight LLMs, alongside Baran [32] and the program-generating cleaner TabClean [64], using about 23,000 prompts. Our main findings are as follows.

Movie identifiers lose their mappings after renaming, but year ranges retain the same candidates. On Movies, DeepSeek-V4-Pro’s year accuracy falls from 91% on real films to 18% on word names and 13% on syllable names, even though the candidates are unchanged (Figure 1). Across the four commercial models, Movies’ identity-column F − W gaps are +0.17 to +0.42 at construction seed 0 (Table 7). On Hospital, three commercial models lose 0.59 to 0.72 repair F1 after syllable replacement on identity columns, which contain entity names and locations.

Model	Hospital Scores O	Hospital Scores F	Hospital Scores W	Hospital Scores S	Hospital Nonpaired F−W	Hospital Paired W−S	Hospital Paired O−S	Movies Scores O	Movies Scores F	Movies Scores W	Movies Scores S	Movies Nonpaired F−W	Movies Paired O−W
DeepSeek-V4-Pro	0.88	0.95	0.50	0.16	+0.45†	+0.34†	+0.72†	0.36	0.45	0.04	0.01	+0.41†	+0.32†
Qwen3.8-Max	0.90	0.93	0.55	0.21	+0.38†	+0.33†	+0.68†	0.30	0.44	0.02	0.00	+0.42†	+0.27†
GLM-5.3	0.78	0.83	0.55	0.19	+0.28†	+0.37†	+0.59†	0.11	0.20	0.04	0.01	+0.17†	+0.07†
GPT-6.1-Sol	0.94	0.98	0.71	0.62	+0.28†	+0.09	+0.32†	0.12	0.37	0.05	0.04	+0.32†	+0.07
Qwen2.5-7B	0.22	0.23	0.07	0.02	+0.16	+0.05	+0.20†	0.01	0.02	0.00	0.00	+0.02	+0.01
Llama-3.1-8B	0.31	0.21	0.02	0.00	+0.19†	+0.02	+0.30†	0.04	0.04	0.01	0.01	+0.04	+0.03†
Qwen2.5-14B	0.44	0.42	0.19	0.04	+0.23†	+0.15	+0.40†	0.06	0.04	0.01	0.01	+0.03	+0.05†
Qwen2.5-32B	0.55	0.48	0.31	0.03	+0.17	+0.28†	+0.52†	0.03	0.03	0.00	0.00	+0.03	+0.03

The screened continuation probe finds no positive evidence of benchmark-row memory in its tested models (§ 6). Hospital and Movies retain positive condition contrasts across construction seeds and, for DeepSeek-V4-Pro, with labeled examples and error positions. Hospital’s O − S gap grows across the three Qwen2.5 model sizes. On Hospital and Beers, whole-table Baran and TabClean retain high scores under renaming, with repeated records and formatting patterns available for repair (§ 7–§ 8). The supplementary report² gives the overview with intervals.

Yet a familiar fact can be the wrong value to preserve. Under explicit table-evidence instructions, commercial models rewrite 0.000–0.404 of assigned directors of known films toward real-world credits, compared with 68%–100% under the standard prompt. Qwen2.5-14B still rewrites more than 60% under the explicit instruction. Hospital and brewery city rewrites are less frequent (§ 9).

Reference errors can also change which edits receive credit. Rayyan contains systematic date and page conversions (§ 10). A repair that matches a title-linked PubMed ISSN receives no credit because the reference contains a spreadsheet conversion. On TabClean’s input, 434 of 461 credited repairs (94%), after excluding development rows, reproduce the reference date transformation.

Stratified PubMed field checks examine these references separately, with field-specific criteria and measured title-match coverage.

We make the following contributions:

• We propose CoRE, a controlled evaluation protocol that transfers benchmark errors to paired name surrogates and relates repair scores to the recovery evidence retained in the input (§ 3).

• We construct controlled tables for three benchmarks and provide construction and evaluation code, numerical witnesses and replay materials (§ 4 and § 11.2).

• We characterize both knowledge-dependent repair and directed rewriting of assigned values across eight LLMs, with separate continuation and information-condition checks (§ 6–§ 9).

• We identify systematic reference transformations in Rayyan and trace how they receive repair credit (§ 10).

Section 2 provides background and related work. Sections 3–5 describe the protocol, constructions and evaluation settings. Sections 6–10 present the analyses, followed by recommendations and limitations in Section 11.

Background and Related Work

Cleaning Benchmarks

Cleaning systems draw on several kinds of evidence. Rules repair constraint violations or apply fixing patterns [11, 13, 62], while KATARA uses knowledge bases and crowdsourcing [12]. Learned detectors use a small set of labels [20, 33, 39]; Baran learns context models for repair [32], and HoloClean combines signals through probabilistic inference [53]. Experimental studies compare detection and repair tools [2, 40] and their effects on machine-learning models [1, 29]. ActiveClean integrates progressive cleaning with statistical model training [26]. BART controls detectability and repairability during error generation [3].

We select public dirty/reference pairs from Raha and Baran for their complementary error profiles (Table 1). Hospital’s damaged names contrast with Movies’ source-specific mappings and retained candidates. Beers adds a formatting contrast: its errors retain the information needed for repair. Rayyan’s spreadsheet transformations instead expose what reference matching rewards. For cleaning we use 1,000-row tables. Movies and Beers retain all source-replaced and city-error rows, respectively, plus sampled benchmark rows. Flights and Tax serve only the continuation probe.

Dataset	Entities	Source	Error cells	Kind of error	Use in this paper
Hospital	US hospitals	CMS registry	509 (219)	one character replaced by x	four conditions
Movies	films	IMDb and Rotten Tomatoes	1,965 (379)	the other source’s value; formatting rules	four conditions
Beers	craft beers and breweries	CraftCans.com	565 (127)	state glued to city; alcohol % suffix	formatting control
Rayyan	biomedical articles	PubMed	948	spreadsheet conversions (§ 10)	quality analysis
Flights	flight times	38 web sources	not used for cleaning		memorization probe
Tax	personal tax records	—	not used for cleaning		memorization probe

Identity columns name or locate an entity: provider number, name, address, city, ZIP code, county and phone in Hospital; identifier, title, director, writers and actors in Movies; and beer, brewery and city in Beers. We distinguish their recovery from repairs of other values, such as film years and formatting errors.

LLM-Based Cleaners

Few-shot prompting can match specialized systems on studied detection, imputation and matching tasks [37]. Subsequent work includes table-tuned and preprocessing models [28, 71] and entity matchers [47, 59]. LLM cleaners label errors (ZeroER [41]), iterate on repairs (IterClean [42]), derive ontological or semantic rules (LLMclean [4], Cocoon [72]), standardize data as agents (Clean-Agent [49]), compile curation pipelines (SEED [9]), or generate cleaning programs (TabClean [64]). We study direct value editing and generated programs under controlled entity replacements.

Knowledge and Benchmark Evaluation

LLMs on enterprise data. Enterprise annotation and integration studies examine domain context, schemas, table width and sparsity [15, 25]. Bodensohn et al. also observe a bias toward familiar SQL despite explicit SIGNAL documentation [5].

Memorization and contamination. LLMs can memorize training text, with stronger effects as models grow or data repeat [7, 8]. Detection methods include benchmark completion [18], likelihoods of canonical and shuffled orderings under exchangeability [45], and least-likely token probabilities [57]. Benchmark-specific records and surveys document these issues [55, 68], while mitigation restricts exposure or refreshes questions [23, 65]. For tables, Bordt et al. [6] find strong memorization of Iris and Wine; Adult’s signals concentrate in headers and initial rows. KnowTable [58] compares completion and existence queries over real, column-resampled, swapped and obfuscated variants, with non-neural baselines and statistical tests. Our comparison instead follows dirty/reference cell pairs: we transfer their error mechanisms to renamed entities and relate repair scores to the candidates and patterns left in the input. The continuation probe separately uses structural baselines to distinguish predictable cells from evidence of row memory.

Knowledge and counterfactual evaluation. Models store facts about entities, especially popular ones [24, 34, 48]. Entity substitutions in QA reveal conflicts between parametric and contextual answers [31, 67, 69], and counterfactual or low-probability variants expose dependence on familiar task instances [35, 36, 66]. DisentQA separates contextual and parametric answers [38]. In data management, ContraTable [63] pairs original and counterfactual databases to compare answer faithfulness across lookup, joins and temporal reasoning. A cleaner must both change errors and preserve assigned values. Our conflict test measures directed rewrites of reference-consistent cells toward familiar facts, alongside arbitrary edits (§ 9.2). Fictional ontologies and pseudowords also limit factual shortcuts in reasoning tasks [54, 56]. We contrast word and syllable names for cleaning.

Benchmark quality. Label errors can change model comparisons, particularly on mislabeled subsets and at higher error prevalence [43]. Spreadsheet conversion can also silently alter values [73]. Rayyan lets us trace credited repairs back to reference transformations and examine what the score rewards.

The CoRe Protocol

CoRe evaluates repair under four input conditions. Within a setting, the prompt, information access and scoring are fixed. Paired surrogate replaces identity values in the same dirty/reference rows; fresh records provide a separate comparison on other records. Figure 2 also shows two checks beyond recovery: whether a cleaner preserves an assigned director, and whether a repair agrees with the scoring reference. These use the conflict and reference inputs in § 9 and § 10.

Vector (PDF)

Four Conditions and Their Comparisons

Besides the original (O), CoRe builds three tables with the same schema and size:

• Fresh (F) uses real records outside the benchmark: newer registry records for Hospital, or other films and beers. Their entities may remain familiar to a model.

• Word-based surrogate (W) replaces identity values one to one with fictitious names made of English words and random numeric identifiers, retaining non-identity values and error positions.

• Syllable-based surrogate (S) uses syllable names in the same construction, retaining generic terms such as hospital and street.

Figure 3 illustrates naming and error transfer on Hospital rows.

Let $s_O$, $s_F$, $s_W$, $s_S$ denote a method’s repair F1 in a fixed column group. Table 2 defines three paired contrasts: O − W, W − S and O − S. They measure score changes under entity and word-form replacement; the first two sum to the third. We interpret these differences alongside the error profiles and information-access controls, since renaming changes candidate values and construction regularities as well as familiarity. F − W compares fresh real records with word-based surrogates and is reported separately. O − S also changes record difficulty; the continuation probe examines benchmark-row memory separately.

Contrast	Interpretation and comparison assumption
O−W	Paired entity-renaming contrast; replacement changes familiarity, word forms and factual associations between fields.
W−S	Word-form contrast; interpretation depends on the difficulty of the two naming styles and any regularities they introduce.
O−S	Paired O–S contrast, combining changes in entity familiarity and word form.
F−W	Fresh records versus word-based surrogates; entity familiarity changes with record difficulty and, where recipes cannot transfer exactly, error composition.
O−F	Original–fresh comparison; benchmark exposure and table difficulty both change. Memorization is tested separately.

What Makes an Error Recoverable?

Recovery depends on the target convention and the information shown to the cleaner (Table 3). The reference specifies the target convention, which is not explicitly stated in the zero-shot prompt. A locally reversible error admits a conversion to the specified convention (R1). A constrained error retains candidates or regularities that narrow the choices (R2), without necessarily selecting a unique source value. An underdetermined error lacks a required mapping or replacement value (R3). We describe these properties for each error type and information setting, not from a model’s success rate. These properties can overlap.

Case	Examples in the benchmark conditions
R1: locally reversible	Beers city/state split and percentage suffix; Movies thousands separators and integer ratings with an added 0.
R2: constrained	Movies year ranges; Hospital name typos with a matching candidate; syllable names with a restricted construction vocabulary.
R3: underdetermined	Random fictitious identifiers; omitted film credits; source-specific ratings and durations not recoverable from the other source.

Where rules or candidates remain available, paired contrasts test their use after renaming. When the required mapping is absent, lower scores instead expose the recovery evidence removed by construction. A repeated record or labeled example can supply a previously missing value (Section 8.1); a candidate list need not resolve which source takes priority. Report §4 gives the criteria and error-type coverage. The S score includes repairs using local rules and construction regularities.

Chunking, Prompting and Scoring

The main input contains 20 rows from different entities, preventing copying from another record of the same hospital or brewery while retaining cross-entity patterns. The same entity grouping gives O/W/S identical chunks, 50 per table. A zero-shot prompt requests JSON corrections with row, column and value fields, leaving correct cells unchanged. Other information settings are defined in Section 5.

Let $E$ be the cells whose dirty and reference strings differ, $P$ the cells effectively edited after no-ops are removed, and $R \subseteq P \cap E$ those set to the reference value. Then

and detection F1 uses $P \cap E$ in place of $R$. Edit and repaired-value comparisons ignore case and normalize whitespace, while $E$ uses original strings. Thus Movies’ five case-only title errors remain in $E$ but cannot earn repair credit. Identity columns are reported separately from the others. Normalization and reply handling follow Report §2; the exact prompt is in Report §7.

Building the Controlled Tables

The controlled tables retain the benchmark’s error mechanisms while varying entities and word forms. Error generators can vary detectability and repairability [3]; here, we start from the observed error profile and transfer it to renamed values or fresh records. Surrogates preserve non-identity cells and error positions, whereas fresh error counts depend on whether recipes and rules apply to the new records. Naming rules hide identity cues while retaining expected word forms. Table 4 summarizes the constructions, with full recipes and checks in Report §1.

Dataset	Entities	Errors	Source	Entities	Errors	Values renamed	Seeds
Hospital	45 hospitals	509 (219)	CMS registry, July 2026 release	34 hospitals (33 also in O)	509 (219)	296	3
Movies	1,000 films	1,965 (379)	IMDb datasets, Sept. 2026	1,000 films	2,195 (371)	6,018	3
Beers	421 breweries	565 (127)	Wikidata, Sept. 2026	94 breweries	565 (127)	1,712	1

Hospital

Hospital’s characteristic corruption replaces a character by x: 508 of 509 error cells follow this form, usually replacing every occurrence of that character. F uses the July 2026 CMS registry in the benchmark schema. Its 34 hospitals include 33 from O, but all quality measures have changed, giving new rows about largely the same entities. We retain the original per-column error counts and position-weighted substitution rule, with newly drawn error locations.

Surrogates rename each distinct identity value consistently and transfer the original errors by position (Figure 3). S uses an inventory of 24 syllables; W uses English-word combinations. Generic suffixes preserve column conventions, and numeric identifiers become random digits. Column-wise replacements preserve repeated-value relationships, but random ZIP codes and phones are not constrained by the retained State values. Thus Hospital also changes factual associations between fields; Report §1 details these relationships and the archived-edit checks. The restricted syllable inventory supplies a regularity examined in § 8.3.

Movies

Movies joins IMDb and Rotten Tomatoes records. Its 190 source-replaced rows, together with two formatting rules, produce the errors in Table 5. A film can have a Rotten Tomatoes slug instead of an IMDb identifier, a year range instead of a year, or credits that add or omit people. For example, American Ultra has american_ultra in place of tt3316948. Such differences require recovering the benchmark’s preferred source value, rather than merely fixing spelling.

Column	Errors	In the original	Fresh table	Surrogates
identifier	190	Rotten Tomatoes slug (135 plain, 28 numbered, 27 with the year)	slug of the fresh title, forms at the original’s rates	slug of the dirty fictitious title, same form
year	190	range “Y−1 Y Y+1”	same range	the original’s cell
title	17	12 edit recipes (capitalization, & for and, accent, punctuation, subtitle, article)	the same edits on a fresh title	the same edits on the fictitious title
director	43	115 names added, 1 dropped, 17 respelled	copied name by name	copied name by name
writers	129	123 names added, 51 dropped, 20 respelled; 5 cells emptied	copied name by name	copied name by name
duration	187	Rotten Tomatoes format (105) or empty (80)	drawn at the original’s rates	the original’s cell
rating	261	score pair (165) or empty (22); “.0” added (74)	drawn at the original’s rates	the original’s cell
rating count	768	thousands separators on every count ≥ 1,000	the same rule	the original’s cell
genre	180	Rotten Tomatoes genre vocabulary	copied slot by slot	the original’s cell

F matches each original film to another feature film in the September 2026 IMDb snapshot, using the same release year (or within two years) and the nearest logarithmic vote count among eligible films with at least 1,000 votes. The corresponding 190 rows receive source-style errors. People and title errors are transferred as recipes over names and edits; genre errors retain the other source’s vocabulary. A recipe that cannot apply to its assigned cell is moved to another source-style row able to take it.

Some recipes remain inapplicable: four director and four writer errors stay clean, leaving 371 identity errors in F versus 379 in O/W/S. Formatting rules also change counts: the separator rule affects all 1,000 fresh rating counts, compared with 768 in O. F therefore has 2,195 errors at seed 0 versus 1,965 in O/W/S. Table 2 specifies the nonpaired F comparison.

Surrogates rename the 6,018 distinct identity values one to one. Titles require special care: retaining a real title’s function words, numbers and punctuation can reveal the film. We generate title shapes independently at the original’s aggregate rates, exclude real title words from W, and require generated titles to support their transferred edit recipes. Fictitious IMDb identifiers and slugs are then formed consistently with those titles.

Recovery evidence varies even within a column. Of 105 duration errors expressed in hours and minutes, unit conversion agrees with the reference in 50 and disagrees in 55. A locally available conversion can therefore differ from the preferred source value. Year ranges instead retain useful candidates, whose use we examine in § 7.

Beers

All Beers errors retain the information needed for formatting repair. They append a percentage sign to a fractional alcohol value, or glue a state to a city and empty the state field. Removing the suffix or splitting the location restores the benchmark’s convention. F uses 1,000 beers from 94 Wikidata breweries [60], excluding original beers and transferring the original error forms and counts.

Surrogates preserve generic terms and consistent relationships between brewery and city names. A one-word city stays one word, avoiding the appearance of a glued city/state value.

Acceptance Checks and Construction Seeds

Acceptance checks cover structure, error counts and strength, naming leakage, fresh-source requirements and deterministic rebuilding. The tables pass 505 of 506 checks. Hospital W at seed 0 has fewer errors replacing three or more characters: 6.4%, versus 16.9% in O. The 10.5-point difference exceeds the eight-point tolerance by 2.5 points; error positions are unchanged. Report §1 gives the thresholds and Report §7 the distributions.

Naming and leakage account for 263 checks: one-to-one replacements, exclusion of real values, and title-shape comparisons against chance-level matching. Deterministic rebuilding checks that the same rules yield identical tables.

Hospital and Movies have three construction seeds: surrogates redraw names and applicable errors; F redraws errors on fixed clean records. Seed ranges capture construction variation, whereas cluster intervals describe one fixed construction. Main tables report seed 0. Hospital W at seeds 1 and 2 passes the strength check, with positive F − W and W − S contrasts for all eight cleaners. Table 12 gives construction ranges; Report §7 gives the passing-seed contrasts.

Model	Hospital			Movies
	F−W	W−S	O−S	F−W	O−W
DeepSeek	.34–.45	.33–.36	.66–.72	.39–.44	.31–.32
Qwen3.8	.34–.38	.33–.38	.68–.69	.38–.42	.26–.28
GLM-5.3	.28–.29	.34–.37	.57–.61	.14–.19	.07–.08
GPT-6.1	.28–.29	.09–.10	.32–.34	.32–.35	.07
Qwen-7B	.16–.22	.02–.07	.19–.20	.01–.03	.00–.01
Llama-8B	.14–.24	.01–.04	.30	.03–.04	.03
Qwen-14B	.23–.31	.12–.17	.37–.42	.03–.06	.05
Qwen-32B	.17–.30	.27–.28	.49–.52	.02–.03	.01–.03

Experimental Setup

Models. Commercial requests disable thinking and set temperature 0 for DeepSeek-V4-Pro [14] and Qwen3.8-Max [51]. GLM-5.3 [70] uses low reasoning effort and temperature 0; GPT-6.1-Sol [44] uses low reasoning effort with sampling parameters left at API defaults. Settings stay fixed across conditions within each model. Locally, vLLM [27] greedily decodes Qwen2.5 Instruct at 7B,

14B and 32B (AWQ) [50], and Llama-3.1-8B-Instruct [19]. OLMo-2-7B instruction/base variants [61] support auxiliary continuation with a public pretraining corpus. Table labels shorten model names; Qwen-7B/14B/32B denote Qwen2.5. Eight of GLM’s 500 Movies replies remain incomplete with no parsed corrections. Report §2 gives budgets and retries; Report §7 covers usage and sensitivity.

Methods and information. The aligned setting adds 20 labeled dirty/clean tuples and error positions to the main chunks. Baran [32] uses the same aligned inputs with three seeds, plus whole tables with perfect detection or Raha [33]. We disable Baran’s Wikipedia-revision value models and Raha’s knowledge-base detector. TabClean runs DeepSeek-V4-Pro on whole tables with repeated records; scoring excludes 150 labeled development rows.

Scale. The 1,000-row inputs retain Movies’ source-style and Beers’ city-error rows (Section 2.1; Report §1). Fixed paired O/W/S chunks and Hospital/Movies construction-seed repetitions support recovery-mechanism comparisons (Section 4.4). Report §2 details about 23,000 cleaning, aligned, conflict and continuation prompts; Rayyan input versions follow Section 10.

Statistics. Repair intervals use 50,000 bootstrap reweightings [16, 46]. Hospital and Beers multiply independent entity and reply weights; Movies reweights replies. O/W/S share paired weights, while F uses independent reply weights. Weighted counts are pooled before computing F1. Two-sided tests use centered bootstrap differences, with Holm correction [21] within each result table (40 contrasts in Table 7). Chunk wins/losses are descriptive where entities recur; Movies sign tests form a separate eight-model family.

Conflict intervals resample whole replies 50,000 times, retaining equal entity weights (Section 9.2); all-or-none outcomes are reported as counts and rates. Rayyan uses 5,000 within-stratum resamples of saved binary checks with fixed type weights, conditional on title matches and field criteria (Section 10.3). Intervals use the 2.5th and 97.5th percentiles; Report §7 details the clustering.

Testing Benchmark Memorization

The continuation probe covers DeepSeek-V4-Pro, the four open-weight cleaners and both OLMo-2 variants. It asks for the next row after a header and ten consecutive rows, and tests whether dirty context elicits a clean reference value. Report §2 details trial counts and context settings.

Screening predictable continuations. Repeated values and record order make many continuations predictable. A cell is hard if its value is absent from the context column and missed by a full-table screen: column modes, frequent successors, low-cardinality functional dependencies and previous-value ±1. Counts subtract the target vote; dependency eligibility stays fixed. The screen uses records beyond the prompt (Report §2). On Hospital, screening lowers DeepSeek’s context-absent accuracy from 47% to 2.1% (3/144).

The highest hard-cell accuracy is 32/375 (0.085; Table 6). On Beers, many replies name other real beers of the contextual brewery, not the next beer in the file. OLMo-2 base also has low accuracy (at most 0.011); an infini-gram [30] search of its public corpus finds only Hospital fragments in a GitHub issue and tool documentation, and none of the other tables.

Clean-value leakage. For clean references absent from the corresponding context column, the hard-cell screen retains 601 erroneous cell-trials for DeepSeek, which supplies the clean reference in three; the four open instruction cleaners do so in 1 of 2,008. This additional screen uses the full clean table and clean predecessor—oracle information beyond the dirty prompt.

Model	Anonymous clean	Anonymous dirty	Named clean	Named dirty	Error rows	Clean value leaked
DeepSeek-V4-Pro	0.080	0.070	0.085	0.065	—	3 / 601
OLMo-2-7B-Instruct	0.027	0.019	—	—	0.030	0 / 187
OLMo-2-7B (base)	0.011	0.011	—	—	—	1 / 92
Qwen2.5-7B	0.040	0.028	0.040	0.046	0.029	0 / 449
Llama-3.1-8B	0.042	0.019	0.014	0.009	0.016	1 / 490
Qwen2.5-14B	0.027	0.061	0.041	0.049	0.030	0 / 532
Qwen2.5-32B	0.005	0.037	0.013	0.028	0.029	0 / 537

Original versus fresh. All four commercial cleaners score lower on O than F on Hospital (−0.03 to −0.07) and Movies (−0.09 to −0.25; Table 7). Some open models have the opposite sign, including Llama’s +0.10 on Hospital, without positive evidence from the continuation probe. As defined in Table 2, O − F also includes difficulty changes in addition to exposure. § 7 compares repair on these real records with the paired surrogates.

Finding 1. The screened continuation results provide no positive evidence attributable to benchmark-row memory in the tested settings.

Entity Familiarity and Repair Performance

Table 7 reports the eight-model condition scores and contrasts. The error profiles in Section 3.2 distinguish missing entity mappings from candidates and rules retained after renaming.

Hospital: Damaged Identity Values

Replacing Hospital’s entity names with syllables reduces DeepSeek-V4-Pro’s identity-column repair F1 from 0.88 to 0.16. Qwen3.8-Max and GLM-5.3 have similarly large losses, giving a 0.59–0.72 range for these three models. Their W − S contrasts are +0.33 to +0.37, in addition to positive O − W contrasts. F − W is also large (+0.28 to +0.45), with higher scores on fresh real records than on word-based names. GPT-6.1-Sol’s higher S score follows the construction regularities examined in § 8.3.

Detection remains stronger than repair (Table 8). For example, DeepSeek detects S identity errors with F1 0.73 but repairs them with 0.16, while its non-identity repair scores remain between 0.91 and 0.95 across conditions. Name candidates can help resolve typos; recovering an arbitrary identifier requires its mapping. Open models show smaller O − S differences, from +0.20 to +0.52 at seed 0.

Model	Hospital O	Hospital W	Hospital S	Movies O	Movies W	Movies S
DeepSeek	1.0/.88	.95/.50	.73/.16	.56/.36	.41/.04	.47/.01
Qwen3.8	.98/.90	.97/.55	.72/.21	.56/.30	.57/.02	.52/.00
GLM-5.3	.98/.78	.97/.55	.72/.19	.26/.11	.12/.04	.08/.01
GPT-6.1	.95/.94	.71/.71	.65/.62	.17/.12	.11/.05	.09/.04

Finding 2. Syllable replacement lowers identity-column repair F1 by 0.59–0.72 for DeepSeek-V4-Pro, Qwen3.8-Max and GLM-5.3 on Hospital. Detection remains stronger than repair.

Movies: Candidates and Mappings

Year ranges retain useful candidates. In the 190 source-style rows, the year is “Y−1 Y Y+1” and the middle candidate matches 167 references (88%). DeepSeek’s year accuracy is 91% on real films, but only 18% on W and 13% on S, where it mostly chooses the first year (Figure 4). Renaming shifts its choices from the middle toward the first candidate; the prompt does not specify which candidate to select. Report §7 separates the 167 middle-reference cases from the other 23. The other commercial models show the same pattern: 171–176 correct years on O, versus 24–48 on surrogates. Duration and genre repairs also decline after renaming (Table 10).

Vector (PDF)

Column	O	F	W	S
identifier	78 / 190	75 / 190	0 / 190	0 / 190
title	0 / 17	4 / 17	1 / 17	0 / 17
director	18 / 43	27 / 39	7 / 43	2 / 43
writers	1 / 129	27 / 125	3 / 129	1 / 129
year	173 / 190	183 / 190	35 / 190	25 / 190
duration	90 / 187	101 / 187	42 / 187	41 / 187
genre	44 / 180	70 / 167	3 / 180	1 / 180
rating	30 / 261	73 / 280	9 / 261	9 / 261
rating count	2 / 768	26 / 1000	24 / 768	90 / 768

Rating-count formatting moves in the opposite direction. Its dirty and reference values are identical across O/W/S, yet DeepSeek repairs 2 of 768 counts on O and 90 on S by removing commas. All 90 S repairs occur in seven chunks. The rule remains locally available, but its use varies across chunks and naming conditions.

Identifiers require entity mappings. All commercial models score at most 0.05 on Movies’ surrogate identity columns. Their F − W gaps are +0.17 to +0.42, while the paired O − W contrasts are smaller (Table 7). DeepSeek replaces 78 of 190 real slugs with the correct IMDb identifier and none on surrogates (Table 10). A fictitious slug does not determine its random identifier, so the repairs on real films use mappings absent after renaming.

Without identifiers, F − W remains +0.26 to +0.33 across the four commercial models, whereas O − W is +0.04 to +0.15. Original people-list errors often contain legitimate Rotten Tomatoes credits, making the reference’s source choice difficult even when the film is known. GPT-6.1-Sol rarely guesses identifiers on O (10 of 190); its O − W of +0.07 is not significant after correction, whereas its F − W gap is.

Open models repair at most 14 of 379 identity errors on O and three on W (Table 9). These near-zero results provide little evidence about their response to entity renaming on Movies.

Model	Hospital W/S	Hospital Chunks	Movies O/W	Movies Chunks	$p$
DeepSeek	104/28	35:1	97/11	35:3	<.001
Qwen3.8	116/37	39:2	94/7	37:2	<.001
GLM-5.3	117/32	39:2	25/8	16:3	.022
GPT-6.1	120/100	19:6	24/10	13:6	.334
Qwen-7B	11/4	8:4	1/0	1:0	1
Llama-8B	5/1	4:0	12/3	9:1	.086
Qwen-14B	35/7	23:2	14/3	10:0	.012
Qwen-32B	63/7	35:0	7/1	6:0	.094

GLM’s O − W remains significant after excluding incomplete replies: +0.06 versus +0.070 on all chunks.

Finding 3. Movies’ commercial-model F − W gaps are +0.17–+0.42 at seed 0, with surrogate identity repair F1 at most 0.05. Year repair also deteriorates despite retained candidate evidence.

Beers: Formatting Errors

On Beers, we compare city/state repairs for real and fictitious names (Table 11). At seed 0, seven models show no significant positive O − S effect; their contrasts range from −0.08 to +0.04, including GPT’s −0.03 in the opposite direction. Qwen3.8-Max instead repairs 122 of 127 cities on O, 108 on W and 82 on S, giving a significant O − S difference of +0.24 (adjusted $p = 0.042$). W repairs more cities than S in 15 chunks and fewer in four. F is descriptive because its brewery concentration and alcohol distribution differ (§ 4.3).

Model	O	F	W	S	O−S	Cities repaired
DeepSeek-V4-Pro	0.27	0.24	0.11	0.28	−0.01	20 / 18 / 8 / 22
Qwen3.8-Max	0.94	0.61	0.82	0.69	+0.24†	122 / 67 / 108 / 82
GLM-5.3	0.94	0.88	0.89	0.94	+0.00	120 / 110 / 116 / 117
GPT-6.1-Sol	0.96	0.89	0.98	1.00	−0.03	127 / 115 / 127 / 127
Qwen2.5-7B	0.00	0.10	0.04	0.05	−0.05	0 / 7 / 4 / 5
Llama-3.1-8B	0.18	0.27	0.11	0.14	+0.04	23 / 28 / 14 / 17
Qwen2.5-14B	0.16	0.05	0.11	0.19	−0.03	13 / 4 / 15 / 20
Qwen2.5-32B	0.03	0.00	0.04	0.11	−0.08	2 / 0 / 3 / 10

Finding 4. Qwen3.8-Max has a positive O − S repair-F1 difference of +0.24 on Beers’ formatting errors; no significant positive effect appears in the other seven models.

Stability over Construction Seeds

Table 12 reports construction variation separately from cluster intervals. Hospital’s O − S contrast varies by at most 0.06 per model, and W − S by at most 0.05. F − W is more sensitive to the fresh error draw, with ranges reaching 0.11 for DeepSeek and 0.13 for open models. Movies’ commercial-model gaps remain positive across seeds, including +0.39 to +0.44 for DeepSeek.

Information Available for Repair

Aligned Information and Repeated Records

The aligned setting adds labeled target-value examples and error positions without enlarging the main chunks. DeepSeek-V4-Pro improves across conditions but retains F − W gaps of +0.22 on Hospital and +0.58 on Movies (Figure 6). Hospital’s rounded W and S scores differ by only 0.01, compared with the larger W − S difference in the zero-shot setting.

Vector (PDF)

Baran with the same chunks, labels and positions scores 0.27–0.32 on Hospital and zero on Movies identity columns. With whole-table perfect detection, it reaches 0.91–0.93 and 0.25–0.27, respectively, with little condition variation (Table 13). Whole-table access exposes repeated records that can supply values missing from the main chunks. DeepSeek still leads Baran in every aligned Hospital and Movies condition, but its advantage narrows on surrogates.

Method	O	F	W	S
Hospital
DeepSeek, aligned	0.97	0.98	0.76	0.75
Baran, aligned	0.29	0.32	0.27	0.28
Baran, whole table	0.91	0.93	0.93	0.93
TabClean	1.00	1.00	0.96	0.99
Movies
DeepSeek, aligned	0.64	0.67	0.09	0.07
Baran, aligned	0.00	0.00	0.00	0.00
Baran, whole table	0.27	0.26	0.26	0.25
TabClean	0.00	0.00	0.03	0.00
Beers
DeepSeek, aligned	1.00	1.00	1.00	1.00
Baran, aligned	0.19	0.77	0.20	0.22
Baran, whole table	0.94	1.00	0.94	0.96
TabClean	1.00	1.00	1.00	1.00

Model Scale

Within Qwen2.5, Hospital’s $O - S$ grows from +0.20 at 7B to +0.40 at 14B and +0.52 at 32B, in the same order at every construction seed (Figure 5; Table 12). S repair F1 stays low at 0.02, 0.04 and 0.03; the increase is mainly in scores on the original hospital names.

Vector (PDF)

Recoverable Syllable Names

GPT-6.1-Sol repairs 100 of Hospital’s 219 identity errors with 105 edits, compared with 28–37 repairs for the other commercial models. The constructor’s 24-syllable inventory is represented in every chunk and permits a unique restoration for 95 corrupted names. GPT restores 74 of them; the other commercial models restore 8–12 and open models at most one (Table 14). The inventory is used by our analysis, not supplied to the cleaner. On digits, which it does not constrain, GPT repairs 6 of 97. Its S advantage is concentrated on names recoverable from the restricted vocabulary.

Model	Unique (95)	Ambig. (11)	Generic (14)	Digits (97)
DeepSeek-V4-Pro	8	0	13	5
Qwen3.8-Max	12	0	12	11
GLM-5.3	10	0	12	8
GPT-6.1-Sol	74	4	14	6
Open models	0–1	0	0–3	0–6

Program-Generating Cleaner

TabClean [64] generates cleaning programs. Its identity-column scores stay near perfect on Hospital and Beers, but near zero on Movies (Table 13). For example, it repairs “San Francisco CA” into city and state, both retained in the input. Movies’ source mappings and omitted credits require external evidence.

Preserving Assigned Values

Movies often pairs legitimate values from different sources, so a familiar credit alone does not identify the task’s target. We next fix a value to preserve and test whether a familiar name draws the cleaner toward the real-world fact.

Constructing Knowledge Conflicts

Starting from S, we assign target cells another real director or location in both dirty and reference tables. Correctness here means preserving the assigned target, although other cells in the row may still contain errors. Table 15 illustrates the assignment of Stanley Kubrick to Taken, whose real director is Pierre Morel.

Version	Name	Year	Director
Exact name	Taken	2008	Stanley Kubrick
One-letter name	Takun	2008	Stanley Kubrick
Surrogate control	Drikel Joujou	2008	Stanley Kubrick

Each target has an exact real name, a one-vowel-changed name and its syllable control. Only the name differs across versions; attributes, other values, error positions and chunks are fixed. Hospital and Beers receive internally consistent locations. Movies also updates related director occurrences consistently, preventing another column from supplying the old director as a candidate. Report §2 specifies these cross-column replacements.

The common pool contains 57 films, 24 hospitals and 48 breweries known to DeepSeek-V4-Pro. Main chunks contain three targets and 17 other-entity fillers, with one row per target. Supplementary Hospital/Beers conditions use three rows per target. Every model receives the same pool, but rates use its own subset passing separate factual queries: 52–57 films, 22–24 hospitals and 7–48 breweries. The factual screen for films accepts a credited full name or a surname of at least four letters; the location screen requires matching city and state. Rates are conditional on the fixed pool.

The standard prompt asks for cleaning while preserving correct cells, without specifying source priority. The table-evidence prompt explicitly favors input-table evidence over real-world knowledge. Both use the same chunks and known subsets.

Measuring Rewrites

A directed rewrite changes the assigned Director or City to the paired entity’s real-world fact. Director outputs must contain a full credited name; cities must match the normalized real city. The one-letter-name variant instead measures restoration of the exact real name. These events are scored separately from any change to the target cell. Output parsing and matching normalization are specified in Report §2.

Let $E_m$ contain entities passing model $m$’s factual queries, $s$ be target rows per entity, and $z_{ej}$ indicate a matching rewrite in row $j$ of entity $e$. We report

For Movies, $s = 1$, giving rewritten known films divided by known films. For $s = 3$, row indicators are first averaged within entities. Targets without a matching correction, including unusable replies, remain in the denominator. Surrogate controls use the same known subset and paired fact. Reply-cluster intervals follow Section 5.

Results

Explicit table priority. With the table-evidence prompt, GPT does not replace any assigned director with a real-world credit (0/57). DeepSeek still rewrites 12 of 57 directors; GLM and Qwen3.8-Max rewrite 10/57 and 23/57, respectively. Rates for Qwen2.5 at 14B and 32B remain above 0.60 (Table 16). These updates contradict the prompt’s explicit instruction to use table evidence.

Model	Exact-name target Standard	Exact-name target Evidence	Surrogate control Standard	Surrogate control Evidence
DeepSeek-V4-Pro	39/57	12/57	2/57	0/57
Qwen3.8-Max	56/57	23/57	14/57	3/57
GLM-5.3	57/57	10/57	10/57	1/57
GPT-6.1-Sol	53/57	0/57	7/57	0/57
Qwen2.5-7B	7/52	5/52	0/52	0/52
Llama-3.1-8B	18/56	15/56	2/56	2/56
Qwen2.5-14B	38/55	35/55	2/55	0/55
Qwen2.5-32B	37/53	32/53	0/53	0/53

Director rewrites. Commercial models know the same 57 films and rewrite 68%–100% of their assigned directors under the standard prompt (Figure 8). DeepSeek, for example, rewrites 39/57. Controls also receive Director edits, but none matches the paired film’s real credit (Table 16).

Vector (PDF)

Within Qwen2.5, conditional rates rise from 0.135 at 7B to 0.691 at 14B and 0.698 at 32B. Their known subsets differ, as shown in Table 16. Llama knows 56 films and rewrites 18 (0.321).

Director versus city. Standard-prompt city rewrites reach at most 0.042 on Hospital and 0.140 on Beers across models. DeepSeek also restores 91% of one-letter-changed film names (Figure 7). This may correct spelling without changing the assigned attribute.

Vector (PDF)

Benchmark Quality

Rayyan contains systematic date and page conversions in its reference values. A cleaner can receive credit for reproducing these conversions but none for an externally matched ISSN correction.

Systematic Reference Transformations

Rayyan’s audit pair has 1,000 rows and eleven columns, differing in 948 cells. Its values exhibit patterns characteristic of spreadsheet conversion [73]. All 722 date differences map input m/d/yy to

$d/\operatorname{int}(yy)/\operatorname{mm}$, with a two-digit input month in the final position: for example, 1/1/13 becomes 1/13/01. Similarly, 31 of 32 pagination differences turn page ranges into month-name strings, such as 70–6 to Jun-70.

What the Reference Rewards

On the audit input, DeepSeek-V4-Pro’s zero-shot output scores F1 0.002. It changes dirty ISSN Mar–22 to 0003–3022, matching a title-matched PubMed record, but the reference is 22–Mar. This correction receives no repair credit.

TabClean’s distributed Rayyan input has 960 difference cells, rather than the audit’s 948. Both are 1,000-row, eleven-column tables with the same date-conversion pattern. We score the archived output against its actual input pair and exclude 150 development rows. The remaining 850 rows contain 808 errors; 498 effective edits yield 461 credited repairs, giving $F_1 = 2 \times461/(498 + 808) = 0.71$ after rounding. Of these credited repairs, 434 (94%) reproduce the reference date transformation. Report §3 specifies the actual inputs, development indices and output.

Stratified PubMed Checks

We sample up to 80 cells per error type, including all volume/issue, pagination and ISSN differences, for 279 cells. Automatic title matching selects the highest-similarity PubMed candidate among up to five results at a normalized similarity of at least 0.9. This yields 152 matched cells associated with 121 PMIDs: 54.5% overall and 47.5%–61.3% across types (Table 17). Query construction and retrieval provenance appear in Report §3.

Check	$N_h$	$n_h$	$m_h$	Reference fail $b_h/m_h$	Dirty pass $d_h/m_h$
Year	722	80	44	43/44	42/44
Changed words	107	80	38	38/38	—
Vol./issue	75	75	46	28/46	17/46
Pages	32	32	18	18/18	17/18
ISSN	12	12	6	6/6	0/6

Field-specific checks. The date field article_jcreated_at is checked for agreement with PubMed’s publication year; the dataset does not document its exact semantics. Volume and issue checks measure presence, using reference −1 and dirty empty strings as absence. Two reference volumes of 1, for example, pass presence while differing numerically from saved PubMed values 31 and 21. Pagination matches the page string, and ISSN accepts print or electronic identifiers. For encoding differences, a nonempty set of reference-side changed words must all occur in the case-folded PubMed word set, with accents retained. Encoding’s dirty side is unassessed.

Let $N_h$ be the error-cell population of type $h$, $m_h$ its title-matched count, and $b_h$ the matched references failing that field’s check. The weighted reference-check failure rate is

Weighting the five field checks by the 948-cell audit composition gives a reference-check failure rate of 95.2% (95% interval: 91.2–97.8%; Table 17). Each within-type proportion uses title-matched cells. Dates account for 722 of 948 error cells and most of the weight. The interval conditions on the saved title matches and field checks; matching uncertainty and dependence between cells sharing a PMID are outside this estimate.

Among the 44 title-matched date cells, 42 dirty values agree with PubMed’s publication year, whereas 43 reference values do not. These sampled publication-year checks differ from TabClean’s credited-conversion count of 434/461.

Reference Issues in Other Benchmarks

Hospital’s reference places Helen Keller Memorial Hospital in Jefferson County, although its city, Sheffield, is in Colbert County. Three commercial models correct some of these cells and are penalized as false edits. Counts and duplicate writer credits in Movies are detailed in Report §3.

Finding 7. On Rayyan, 434/461 of TabClean’s credited repairs (94%), after excluding 150 development rows, reproduce the reference date transformation. An externally matched ISSN correction earns no repair credit.

Lessons and Recommendations

Distinguish missing mappings from retained candidates. An identifier may lose its mapping after renaming, while a year range retains its candidates (Section 3.2). Report formatting repairs separately once the target convention is established: a local conversion and a source-specific value pose different recovery problems (Findings 2–4). If a renamed identifier has no mapping, supply a candidate list or retrieval source before comparing a method’s ability to recover it. Where candidates remain, report which candidates the method selects, as in the year comparison, rather than interpreting all score losses as missing external knowledge.

Align labels, error positions and supporting records. Compare chunk-based and whole-table methods under aligned labels and error positions (Finding 5). Report supporting-record access separately: another record of the same entity may supply a replacement absent from the chunk. Equal detection information does not imply equal evidence for recovering values.

Set source priority and test value preservation. Movies requires selecting a preferred source, whereas the conflict test requires retaining an assigned director. Count factual rewrites separately from other edits: a real credit can still be the wrong update for that task (Finding 6). For cross-source errors, name the preferred source explicitly: a Rotten Tomatoes credit can be legitimate yet differ from an IMDb reference. For assigned values, state that the table takes priority and include unchanged target cells in evaluation.

Inspect the edits rewarded by the reference. Rayyan credits date conversion but rejects an externally matched ISSN. Compare fields with independent records under explicit criteria, and report the scored input and development exclusions (Finding 7). Separate mechanical conversions from changes in field content. An external year check also requires the date field’s semantics, while a weighted summary requires the match coverage and criteria for each field. Inspecting high-credit edits is especially useful when a single transformation accounts for most repaired cells.

Applying CoRe to Other Benchmarks

Applying CoRe requires profiling errors and reference conventions, selecting a compatible fresh source, and building the two naming styles with acceptance checks. Report F1 contrasts alongside error-type counts and candidate selections, as in the Movies year analysis. Record which field relationships survive identity replacement: geographic codes can retain familiar meanings after names change. Report §4 describes the adaptation procedure.

Artifact and Result Replay

The statistical core replays condition contrasts, conflict rates and Rayyan checks offline from Hospital/Beers tables, Movies numerical witnesses and re-scored continuation counts. Authorized local validation checks historical movie inputs, replies and scoring-code hashes before re-scoring. The repository README gives the commands; Report §5 maps each analysis to its inputs, scoring code and outputs; Report §6 gives the extended displays.

Limitations

The datasets diagnose recovery mechanisms and reference transformations in entity-rich public tables, rather than estimate their prevalence across cleaning workloads. Transfer to private data also depends on size, sparsity, task complexity and the availability of supporting records. Our comparisons change name forms as well as familiarity, and F1 introduces record-difficulty differences (Table 2). Beers F has different brewery concentration and alcohol distributions, while Hospital’s restricted syllables provide the regularity observed for GPT (Section 8.3). The study covers eight commercial and open-weight cleaners on English tables concerning mainly US entities, with decoding settings in Report §2; it does not cover Claude or Gemini.

The intervals describe variation across entities and replies within fixed constructions. New backgrounds, regrouped prompts and repeated API calls introduce other sources of variation. Report §7 gives the entity–reply structure and shows how repair F1 changes when one hospital’s rows are omitted from the saved outputs.

Conclusion

Remaining Movies lowers DeepSeek-V4-Pro’s year accuracy from 91% to 18% even though the candidate years remain in the input. Identifier repairs decline too, but their mappings are no longer available. On Hospital, GPT repairs 74 of 95 errors with a unique restoration under the constructor’s restricted syllable inventory. The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately. The probe finds no positive evidence of row memory in its tested models.

The assigned-value and reference analyses reveal errors that conventional repair scores can miss. Some models replace assigned directors with real credits even after an explicit table-evidence instruction. On Rayyan, 434/461 of TabClean’s credited repairs reproduce a reference date transformation, while an externally matched ISSN correction earns no credit. For evaluation, these cases call for tests with a specified source priority and inspection of the edits counted as correct. The constructions and scoring code support comparisons with other data cleaners.

References