betaFeedback

WorksComputer Science

Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do

Abstract

The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.

Introduction

A benchmark does more than rank algorithms: it defines what a research community counts as progress. Meaningful progress requires the benchmarks tasks to resemble the problems encountered in practice [25, 16]. TSC is applied in diverse domains such as clinical monitoring [4], human activity recognition [1], and satellite remote sensing [23], and a realistic benchmark should reflect this breadth. Diversity alone, however, does not make a benchmark representative: each task must also be posed as it would be in practice, with labels and evaluation protocols that correspond to a real-world problem.

The UCR archive has become the de facto standard benchmark for univariate TSC. Successive generations of distance-based, ensemble, convolutional, deep and representation-learning methods are compared on it [6, 19, 28, 20], as are recent pretrained models [12, 11, 18]. Yet many of the archives datasets are knowingly not reflective of real world applications [5]. It admits several ordered non-temporal sequences, such as image outlines or spectra, and is biased towards image-derived datasets and datasets shaped by research interest [5]. While reasonable for a diverse and reproducible testbed, these choices are at odds with the archive’s common use as a benchmark of realistic TSC.

We revisit the UCR archive from a provenance perspective, tracing each of its 128 datasets to its source to examine how each task was derived from the original data. 1 We identify two recurring design choices that weaken the link between archive performance and real-world TSC: artificial task construction and train/test leakage. We further raise open questions about pre-segmented and aligned inputs, ordered but non-temporal sequences, and multivariate recordings decomposed into several univariate datasets. We argue that the community-wide adoption of UCR in its current form as the benchmark for univariate TSC, together with the absence of more realistic alternatives, calls for action towards evaluation settings that reflect real-world TSC problems.

Background and Related Work

Time Series Classification Archives. The UCR archive [5] and its multivariate counterpart, the UEA archive [3], are the standard benchmarks for TSC. Together they comprise 158 tasks, 128 univariate and 30 multivariate, drawn from image outlines, spectroscopy and simulated processes alongside ECG, EEG, motion capture and industrial sensor recordings. Recent collections follow the same pattern at larger scale: MONSTER [7] adds substantially larger datasets, and Multiverse [21] consolidates the UEA archive, MONSTER and several domain-specific collections into 133 multivariate problems. The bake-off series [2, 20] imposed rigor on how comparisons over these archives are run, but likewise takes the task collection as given. Prior critiques target properties of the archive’s tasks: Hu et al. [15] document the simplifications of the pre-segmented setting, Dhariyal et al. [8] show that tabular classifiers are competitive on many tasks, and Zhang et al. [29] show that temporal order is often irrelevant and the methods can emphasize on a fixed specific position rather than patterns. Each identifies a symptom without asking why the archive contains such tasks in the first place.

Benchmark Construction. Recent work on AI evaluation distinguishes a benchmark as an instrument for separating algorithms from a benchmark as a proxy for downstream applicability, and warns against reading aggregate scores on convenience-driven collections as general capability [25]. Neighboring fields have responded with explicit curation: TabArena [10] admits only datasets that originate from real prediction problems and excludes derived or artificial tasks; RelBench [26] defines tasks directly on relational databases with temporally valid splits. TSC has no equivalent. Existing critiques diagnose properties of particular datasets; but we are not aware of any analysis for the origin of the tasks these datasets were collected for nor whether they represent realistic TSC applications. Our provenance analysis is a first step towards such an examination.

A Provenance Audit of UCR: Two Ways Its Tasks Fail as Benchmarks

We assess all 128 UCR datasets through three stages: 1) the source study that produced the data, 2) the construction of the labelled task from the data, and 3) the construction of the train/test split. Appendix A details the protocol.

Two questions decide a dataset’s standing as a benchmark task and organize this section. A dataset fails on task construction (§ 3.1) if the source study posed no real task or the archive constructed the label itself. It fails on leakage (§ 3.2) if the test set shares series or subjects with the training set. We further flag 65 datasets where the subjects or session identity are underspecified, so the leakage counts below are lower bounds. Whether the series index is physical time is recorded but not judged, and we return to it in Section 4. Figure 1 summarizes the outcome of our realism provenance.

Artificial Task Construction

For 52 of the 128 UCR datasets, the classification target is not a real-world prediction problem. Some labels differ from the source question or were constructed post hoc where no classification task was posed. Other source tasks were built only to exercise algorithms, with no application behind the label.

While such constructed labels are best understood as pretext tasks in the sense of self-supervised learning, where pretext accuracy is a known-unreliable proxy for downstream performance [17], UCR reports the pretext task as the benchmark result itself. We identify four mechanisms by which such labels are constructed:

Metadata as label (13): the archive derives classes from a recording attribute. ItalyPowerDemand labels demand curves by season (October–March vs. April–September); ECGFiveDays labels one subject’s ECG segments by which of two recording dates they come from.

Distribution of archive datasets over realism criteria and leakage status

Figure 1. We show the archive datasets distribution over the different realism criterion. Our audit identifies only 38/128 UCR datasets as real temporal tasks; about a quarter of these are prone to split leakage. We indicate the proportion of datasets failing our leakage criteria (white boxes) and with no leakage found (black boxes) within each category, although most splits cannot be checked (53 of the 76 real-task datasets are Unclear).

Instance identity as label (14): the archive uses the recording’s source as the class, regardless of what the source study asked. In CinCECGTorso, the 2007 challenge concerned infarct location and extent [24], but the archive labels each ECG by which of four subjects produced it; the Pig* datasets turn a bleeding-detection study [13] into a 52-way subject-identification problem.

Task replacement (13): the archive replaces the source’s label with a different task, or defines classes by its own rule where the source posed none. AllGestureWiimoteX/Y/Z were collected for user identification but are benchmarked as gesture recognition [14]; HouseTwenty takes a smart-meter recording of one home [22] and asks whether a window is the whole-house signal or the sum of two appliances, a question the study never posed.

Toy source tasks (17): the data were simulated or staged by method developers, and the classes were chosen to demonstrate a method rather than to answer any external question. ShapeletSim embeds one of two shapes in noise so that shapelet methods are guaranteed to separate them [2]; GunPoint has two actors draw a gun or point at UCR, staged in 2003 by the archive’s curators [5].

These mechanisms are not exclusive. For example, in GunPointMaleVersusFemale, the recording was staged by the curators, and the archive then labels it by which of the two actors is performing rather than by the gun/point distinction the staging was designed around. Appendix § 8 lists them all with their original label (if one exists) and the constructed label, and marks the overlaps.

For a benchmark, the consequence is that accuracy on these tasks is a pretext score with no downstream target to validate it against: a method can rank first archive-wide by exploiting recording artifacts, subject identity or generator mechanics, none of which are the capabilities a practitioner deploying a TSC model needs.

Train/Test Leakage

For 41 of the 128 UCR datasets, the test set is not independent of the training set. This happens in two ways: identical series appear in both train and test splits, or the split itself introduces leakage by grouping (or not grouping) the subjects. Additionally in 65 datasets, the information needed to check source separation is missing. To our knowledge, no systematic leakage check was run by the curators or the donors. We flag two main concerning setups: Duplication (17 datasets): at least one test series is value-identical to a training series; in PowerCons this affects 37% of the test set (67/180), in Crop 16% (2753/16800), in Worms 13% (10/77). Same-label exact matches permit a classifier to reproduce a training label without demonstrating generalisation to a new case (see Figure 2 and Table 5).

MiniRocket accuracy with the original training set and after removing duplicates

Figure 2. MiniRocket accuracy with the original training set (with duplicates, grey) and after removing the duplicates (blue). On several datasets, duplication inflates test accuracy.

DatasetArchiveTest = train%Acc. dup.Acc. restMixed
PEMS-SFUEA79 / 17345.71.0000.681–
PowerConsUCR67 / 18037.21.0000.991–
CropUCR2,753 / 16,80016.40.9920.722–
WormsTwoClassUCR10 / 7713.01.0000.746–
WormsUCR10 / 7713.01.0000.701–
LibrasUEA19 / 18010.61.0000.907–
InsectEPGSmallTrainUCR15 / 2496.01.0000.991–
EigenWormsUEA7 / 1315.31.0000.935–
ElectricDevicesUCR208 / 7,7112.70.3650.75254
ToeSegmentation1UCR4 / 2281.81.0000.960–
InlineSkateUCR6 / 5501.11.0000.471–
FreezerSmallTrainUCR27 / 2,8500.91.0000.967–
PLAIDUCR5 / 5370.91.0000.930–
ShapesAllUCR2 / 6000.31.0000.923–
SwedishLeafUCR2 / 6250.31.0000.961–
StrawberryUCR1 / 3700.30.0000.9861
MedicalImagesUCR1 / 7600.11.0000.791–
FaceAllUCR1 / 1,6900.11.0000.806–
MixedShapesRegularTrainUCR1 / 2,4250.01.0000.971–
FreezerRegularTrainUCR1 / 2,8500.01.0001.000–

Table 5. Exact cross-split matches and MiniRocket accuracy, retained from the original generated results. Counts refer to test series, not pairs. Mixed counts duplicate groups with conflicting labels; – denotes none.

Shared Sources (24 datasets): whether a split is valid depends on the task. Class recognition, whose task is to predict a class for subjects unseen in training, requires the split to be grouped by subject; instance recognition, whose task is to re-identify a known subject, requires the test set to contain new recordings of that subject. On these 24 datasets one of the two rules is violated: in SemgHandMovementCh2 (class recognition) the grasps of all five subjects appear on both sides of the split; in SemgHandSubjectCh2 (instance recognition) each of the five subjects was recorded in a single session, and the repetitions of that session are split by a stratified shuffle.

These pose trustworthiness challenges, as scores should state which evaluation setting they represent in benchmarking. Further details can be found in Appendix § 9.

Open Questions

Do ordered sequences translate to temporal axes? The UCR archive encompasses multiple ordered datasets that span domains like spectrography and image contours. While one can argue that they provide meaningful tests for sequence models, temporal notions as warping, trends, periodicity and regime changes have no direct counterparts on a spatial or spectral axis. Treating both kinds of data as one benchmark therefore assumes a transfer the archive never tests[2, 20].

Zhang et al. [29] provide indirect evidence: applying a single fixed permutation to every series, destroying local order while leaving each value at a fixed position, barely changes accuracy on roughly a third of the 112 equal-length UCR datasets, so those tasks are solved from aligned values rather than from order, a strategy unavailable in deployment for TSC.

Should decomposed multivariate recordings count as separate datasets? Extracting one channel from a multivariate recording is a useful test of a univariate model. Counting several channels as independent datasets is a different decision: they share subjects, labels, acquisition and the underlying phenomenon [5, 27]. Archive-wide comparisons, including rank-based tests, treat datasets as independent samples of TSC problems; derived channels violate that assumption, and one source problem receives several correlated votes in the aggregate score.

How much of the problem does preprocessing solve? Every UCR series is a pre-extracted instance, and in many datasets instances are further aligned to a common reference before release, such as image outlines to a canonical starting point. The archive therefore never asks where in a stream a class-defining pattern occurs or how long it lasts. Hu et al. [15] argued that this pre-segmented setting is far removed from how classification is deployed on streams, yet the archive and the methods evaluated on it have continued to treat it as given.

The consequence is that, once patterns sit at near-fixed positions, time steps become features. This offers an explanation for two recent findings: tabular classifiers are competitive on many UCR tasks [8], and disrupting temporal order often does not meaningfully affect predictive performance [29]. Whether performance on such tasks says anything about a method’s ability to locate patterns in time, a key capability required in deployment, remains an open question.

Conclusion

The UCR archive made univariate TSC evaluation accessible and reproducible, and they remain a reasonable instrument for comparing algorithms. Our audit, however, shows that archive-wide performance should not be interpreted uncritically as performance on real-world TSC problems. Constructed targets, train/test dependencies, preprocessing choices, non-temporal sequences and repeated source problems each break that link, and they are common enough in UCR that an aggregate ranking cannot be read as a measure of deployment readiness. We argue for evaluation that complements the archives with provenance-aware valid task selection, deployment-relevant splits and capturing the realism of deployment environments to close the gap of explored caveats. Progress requires both better scores and benchmarks that measure the intended capability.

Acknowledgements

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. This work was supported by the European Union’s Horizon Europe research and innovation program under grant agreement No 101214398 (ELLIOT). L.P. acknowledges funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under SFB 1597 (SmallData), grant number 499552394; F.H. acknowledges the financial support of the Hector Foundation.

Provenance Audit Protocol

We describe how the six audit fields of Figure 3 were filled for the 128 UCR datasets, which sources were admitted as evidence, how the checks on the data were run, and how agentic systems were used in the setup.

The six audit fields across source study, archive task, and evaluation

Figure 3. The six audit fields, two per stage of a dataset’s life. Green marks the value that passes, red a value that fails or flags leakage, grey a value the verdict does not read.

The six fields

Each dataset is followed through three stages of its life, with two fields per stage (Figure 3). Fields (i) and (v) are free text and carry the evidence; the other four take a value from a fixed vocabulary, and only those four feed the counts in § 3.

(i) Source publication. The paper, thesis, competition or data release the series come from, with the passages read.

(ii) Original task. Whether the source study posed a classification task of its own: real task (a stakeholder outside the paper needed the prediction), toy task (data staged or generated to demonstrate a method), or none (the source collected data but posed no classification target).

(iii) Data origin. What the series index is: temporal (sampled over physical time), ordered, non-temporal (wavelength, contour arc-length, spatial position, entity index), or synthetic (a generator’s step).

(iv) Label origin. How the archive’s class relates to (ii): source task (the same target, possibly coarsened), metadata (a recording attribute such as season, date, lead or device), instance identity (which subject, sensor or actor produced the series), or redefined (a target the archive defined where the source posed none or a different one).

(v) Split provenance. Who built the default train/test split and by what rule, where this is documented at all.

(vi) Split leakage. Whether the test set is independent of the training set: Separated (no duplicates, and the unit a deployed model would face differs between the sets, as documented for the split or by construction because each series is a distinct event or object with no known grouping above that unit), Concern (at least one test series is value-identical to a training series, or the unit a deployed model would face appears on both sides), Unclear (the information needed to check separation of that unit is missing), Related periods (different days or windows of the same ongoing process), or Generated (synthetic tasks, not assessed by source separation).

A dataset fails on task construction (§ 3.1) when (ii) is toy task or none, or (iv) is metadata, instance identity or redefined. It fails on leakage (§ 3.2) when (vi) is Concern. Unclear is never counted as a pass on leakage. The four datasets whose original task or label origin is undocumented are counted with the real tasks. Field (iii) is recorded for every dataset but does not enter either verdict (§ 4).

Admissible evidence

Four kinds of evidence were admitted, and every value in the released audit carries a tag saying which kind supports it.

  • [S] Source text. A passage of the source publication, the archive’s own briefing document [5], the donor’s documentation, or the dataset page on timeseriesclassification.com, cited by section and page and quoted where the claim is decisive. Dead donor links were recovered through the Internet Archive where possible; a link that points to an unrelated resource is itself recorded as evidence of weak provenance.

  • [D] Shipped data. A property computed on the archive’s own files: the duplicate check of Appendix A.4, and one-line sanity checks used during drafting (sign of the values, whether the series length equals a count of entities in the source, whether split sizes reconcile with the source’s arithmetic).

  • [R] Inference. A conclusion the auditor drew from [S] or [D] rather than read in a source, such as identifying which of several challenges a vaguely described dataset comes from.

  • [I] Intended deployment. An inference about intended deployment, such as which unit a deployed model would face.

Where the sources do not answer a question, the field is set to unknown or undocumented and the record says what is missing. A fluent but unsupported account of a dataset’s origin was treated as a defect, not a filled field.

Language-model assistance

The audit followed a five-stage pipeline: scripted scaffolding, agentic drafting, agentic review, an agentic fact-check of the released CSV against the records, and human verification. The prompts and protocols for each stage are released together with the audit.

Scaffolding. A script scrapes the archive page of each dataset and fills the identity fields (split sizes, series length, number of classes, archive type tag). All judgment fields are left as placeholders.

Drafting. An agent follows the archive’s source links, reads the source publication, fills fields (i) to (v) according to the evidence rules of Appendix A.2, and runs the data checks. The protocol forbids filling fields from prior knowledge, requires the load-bearing sentence to be quoted with its location, and instructs the agent to lower the confidence field and state what is missing rather than infer unstated values.

Review. A second, independent agentic pass, run under a separate protocol, fetches and reads every listed reference before inspecting the draft. For each decisive claim it records one of confirmed with quote, quote mismatch, not found, contradicted, or omitted by draft, re-derives the field values from the verified evidence, and re-runs the duplicate check.

Fact-check. Two independent agents (Claude Opus and GPT Astra) re-derived every verdict and fact column of the released CSV from the records, keeping the existing value where a record is silent, and a third agent reconciled the two.

Human verification. The authors decided every verdict on which the agents disagreed or which the reconciling agent rated as low-confidence. For contested datasets, the authors re-read the cited sources themselves, and corrected records were re-checked.

Leakage checks

Duplicates. For every dataset we loaded the shipped train and test files with their original lengths and missing values, and hashed each series as a byte string of its 64-bit values with missing values mapped to a fixed sentinel, so that identical padding hashes identically. Two series are duplicates when their hashes agree. We report, per dataset, the number of test series whose values appear in the training set, the number of distinct series overall, the duplicate-group size histogram, and whether any duplicate group carries more than one label.

Group overlap. Duplicates are not the only way a test set can depend on the training set. When the same subject, device or recording session contributes series to both sides of the split, a model can pass by recognizing the source rather than the class. We marked a split as leaking when the archive, the source or the released data establish such sharing, as clean when they document that it is absent, or when each series is a distinct event or object with no known grouping above it, and as unknown otherwise.

Release

The repository https://github.com/automl/ucr-provenance-audit contains the audit CSV with a column guide, one YAML record per dataset, the drafting, review and fact-check protocols, the duplicate-check script with its output, and the browser page at https://automl.github.io/ucr-provenance-audit/.

Artificial Task Construction

Tables 1–4 list all 52 datasets that fail task construction, because their label differs from the source task or the source task is a toy, grouped by the mechanism described in § 3.1. There are 57 entries for 52 datasets; five datasets fall under two mechanisms and appear in both tables. The column “Source target or purpose” states the question the source study posed, or notes that no classification target was documented. A derived label does not by itself make a task invalid; the tables record what is being evaluated and who defined it.

Metadata as Label

DatasetBenchmark targetSource target or purposeLabel by
ItalyPowerDemandCold season (Oct–Mar) vs warm season (Apr–Sep)None; utility demand recordsCurators
PowerConsWarm vs cold seasonNone; EDF consumption recordsCurators
ChinatownWeekend vs weekdayNone; Melbourne pedestrian countsCurators
DodgerLoopDayDay of weekUnsupervised event detection; no class targetCurators
DodgerLoopGameWhether a Dodgers home game took place that dayUnsupervised detection of game traffic events; schedule used as validation ground truthCurators
DodgerLoopWeekendWeekend vs weekdayAs DodgerLoopDayCurators
ECGFiveDaysWhich of two dates, five days apartNone documented; one subject’s clinical ECGCurators
GunPointOldVersusYoung†Recording session: 2003 vs 2018Gun-draw vs pointing actionCollectors (= curators)
SemgHandGenderCh2Sex of five subjectsSix hand-grasp typesCurators
TwoLeadECGWhich of two ECG leadsNone used; one long-term ECG recordCurators
RockFour rock typesASTER spectral-library metadata; no defined taskCurators
WormsWorm strain: wild type or one of four mutantsUnsupervised discovery of behavioural motifs; no class targetCurators
WormsTwoClassWild type vs mutantAs WormsCurators

Table 1. Task-construction cases in the current audit: metadata. None means no classification target is documented, not that the source data had no scientific purpose. † also listed under another mechanism.

Instance Identity as Label

DatasetBenchmark targetSource target or purposeLabel by
PigAirwayPressureWhich of 52 pigsBefore vs during induced bleedingCurators
PigArtPressureWhich of 52 pigsAs PigAirwayPressureCurators
PigCVPWhich of 52 pigsAs PigAirwayPressureCurators
CinCECGTorsoWhich of four peopleInfarct extent and locationCurators
MelbournePedestrianWhich of ten sensor locationsPedestrian monitoring; no class targetCurators
SemgHandSubjectCh2Which of five subjectsSix hand-grasp typesCurators
Yoga†Which of two actors, from silhouettesActor discrimination, posed by the collectors as a demo taskCollectors (= curators)
GunPointMaleVersusFemale†Actor sex; confounded with identity (two actors)Gun-draw vs pointing actionCollectors (= curators)
MoteStrainWhich source stream: humidity vs temperatureUnsupervised stream correlation tracking; no class targetCurators
FreezerRegularTrainWhich of two freezers in one houseREFIT load monitoring; no class targetCurators
FreezerSmallTrainAs FreezerRegularTrain; smaller training splitAs FreezerRegularTrainCurators
DatasetBenchmark targetSource target or purposeLabel by
FaceAll{sup}†Which of 14 graduate students, from head profileHead profiles recorded at UCR; no applied problemCollectors (= curators)
FacesUCR{sup}†As FaceAll; rotation-aligned, different splitAs FaceAllCollectors (= curators)
HapticsWhich of five users entered the passgraphNone documented; passgraph entry on a touchscreenUnknown

Table 2. Task-construction cases in the current audit: identity. None means no classification target is documented, not that the source data had no scientific purpose. † also listed under another mechanism.

Task Replacement, Derivation or Subsetting

DatasetBenchmark targetSource target or purposeLabel by
AllGestureWiimoteXWhich of ten gesturesUser identity; gesture was a protocol factorCurators
AllGestureWiimoteYAs AllGestureWiimoteXAs AllGestureWiimoteXCurators
AllGestureWiimoteZAs AllGestureWiimoteXAs AllGestureWiimoteXCurators
EarthquakesWhether a large event follows a window; curator thresholdsContinuous seismic monitoring; no class targetCurators
HouseTwentyWhole-house aggregate vs summed dryer/washer channelsREFIT load monitoring; no class targetCurators
ECG5000Five automatically assigned beat typesNo clinician beat labels used for this recordCurators
ToeSegmentation1Normal vs acted abnormal gaitCMU motion descriptions; no defined taskCurators
ToeSegmentation2As ToeSegmentation1As ToeSegmentation1Curators
ChlorineConcentrationThree undocumented classesNone; EPANET simulation built to test missing-value imputationCurators
BeetleFlyBeetle vs fly, a pair carved out of 70 classesShape retrieval over 70 MPEG-7 classes; no applied problemCurators
BirdChickenBird vs chicken, a pair carved out of 70 classesAs BeetleFlyCurators
FiftyWordsWhich of 50 word typesWord spotting by word identity in historical manuscriptsCurators
WordSynonymsWhich of 25 curator-defined word groupsAs FiftyWordsCurators

Table 3. Task-construction cases in the current audit: replacement. None means no classification target is documented, not that the source data had no scientific purpose.

Toy Source Tasks

DatasetBenchmark targetSource target or purposeLabel by
BMEBell at begin, none, or at end of a central plateNone; unpublished generator (Grenoble LIG/AMA)Generator
UMDBell pointing up, absent, or pointing downAs BMEGenerator
CBFCylinder, bell or funnel shape in noiseNone; demo signal for local feature extractionGenerator
SyntheticControlSix control-chart patternsNone; no process measured, generated to test a similarity methodGenerator
TwoPatternsOrdered pair of up/down steps in noiseNone; designed for thesis experimentsGenerator
MallatEight generated signal variants; classes undocumentedNone; wavelet test signalGenerator
ShapeletSimWhich of two shapes is embedded in noiseNone; simulator built so that shapelet methods winGenerator
SmoothSubspaceWhich five-step window carries the signalCluster identity, to test a kk-means variantGenerator
TraceFour feature/fault pairsSixteen simulated plant transient/failure classesGenerator
GunPointGun-draw vs pointingStaged at UCR as a similarity-search benchmarkCollectors (= curators)
GunPointAgeSpanGun-draw vs pointing, pooled over actors and yearsGunPoint re-staged for the 2018 archive releaseCollectors (= curators)
GunPointMaleVersusFemale{sup}†`Actor sex; confounded with identity (two actors)As GunPointAgeSpanCollectors (= curators)
GunPointOldVersusYoung{sup}†`Recording session: 2003 vs 2018As GunPointAgeSpanCollectors (= curators)
Yoga{sup}†`Which of two actors, from silhouettesActor discrimination, posed by the collectors as a demo taskCollectors (= curators)
FaceAll{sup}†`Which of 14 graduate students, from head profileHead profiles recorded at UCR; no applied problemCollectors (= curators)
FacesUCR{sup}†`As FaceAll; rotation-aligned, different splitAs FaceAllCollectors (= curators)
SymbolsSix classes; three symbols documentedCopying experiment with 13 participants; no publicationCollectors (= curators)

Table 4. Task-construction cases in the current audit: toy source task. None means no classification target is documented, not that the source data had no scientific purpose. {sup}†` also listed under another mechanism.

Split Provenance

The 41 UCR datasets assigned the Concern label, comprise 17 with at least one exact cross-split match and 24 more with a documented or data-verified shared source but no exact match.

Train/Test leakage or inconsistent labels

Table 5 reports the datasets with the duplicates and split matches. A test series whose exact values already appear in training can be answered by recall rather than by generalisation, so the two accuracy columns split each test set in two: acc. dup. covers the test series that have an identical training counterpart, acc. rest the remainder. The gap between them is large and one-directional. MiniRocket answers the repeated cases almost perfectly while scoring far lower on the rest of the same test set. Two datasets run the other way, ElectricDevices (0.365 against 0.752) and Strawberry (0.000 against 0.986), because some repeated series carry conflicting labels and no classifier can satisfy every copy.

Split Concerns Against Evaluation

Table 6 examines whether each split supports the intended evaluation. A new instance means a case from a new source at the relevant level: for example, a new patient, a new physical sample or a fresh recording of a familiar person. Another window from the same recording is a different case, but does not provide a new recording occasion. We assign one verdict per dataset:

DatasetRelevant new caseVerdictWhat the split tests and why a concern can rise
ACSF1New appliancesUnclearThe 100/100 split matches the source’s Test Protocol 1.0, which trains on the first measurement session and tests on the second session of the same appliances. The archive does not say which protocol was used, and the series length does not match the documented one-hour sessions. Whether the test presents familiar appliances again or appliances the model has never seen therefore cannot be established. A concern arises only if the result is presented as performance on new appliances.
AdiacNew independently collected samplesUnclearThe cases are outlines of tiny biological specimens. We do not know whether samples grown from the same biological source, or photographed on the same microscope slide, occur in both sets. They could share growth or imaging conditions. The records do not let us check whether the test samples were collected independently.
AllGestureWiimoteXNew peopleUnclearThe data contain gestures from ten people. The archive shuffled and sampled recordings, but does not say that all recordings from a person were kept in one set. We therefore do not know whether test users are new. This matters if the intended application is to recognise gestures from people the model has never seen.
AllGestureWiimoteYNew peopleUnclearSame as AllGestureWiimoteX.
AllGestureWiimoteZNew peopleUnclearSame as AllGestureWiimoteX.
ArrowHeadNew physical objectsUnclearEach case is an outline of an arrowhead. The source assigned outlines to a 36/175 split at random. We cannot check whether multiple views or representations of the same object were kept together.
BMENew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
BeefNew physical samplesUnclearThe data are spectral measurements of beef samples. It is not documented whether repeated measurements of one physical sample were kept in the same set. Without that information, performance on new samples cannot be distinguished from performance on further measurements of familiar samples.
BeetleFlyNew source imagesUnclearThe cases are outlines extracted from images of beetles and flies. We do not know how the original images were assigned to training and testing. If different versions of one image occur in both sets, the model could recognise a familiar shape. Such shared images have not been verified.
BirdChickenNew source imagesUnclearSame as BeetleFly.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
CBFNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
CarNew vehicles or recordingsUnclearThe outlines were extracted from traffic videos. Several frames can show the same vehicle under very similar conditions. The documentation does not establish whether all frames of a vehicle or video were kept in one set, so performance on new vehicles or capture occasions cannot be checked.
ChinatownNew days or later periodsRelated periodsA test day may be close to a training day and share its season or local events. These are different days, so this is not automatically leakage. No source says whether the split holds out a later period that follows all the training days.
ChlorineConcentrationNew simulation runsConcernThe audit describes cases taken at nearby times in one simulated system. A training case and test case can therefore describe almost the same system conditions. The result tests more cases from that simulation. It does not show whether the model works on a new simulation run; similarity between cases would need a separate check.
CinCECGTorsoNew recording occasionsConcernThe task is to identify four people from their ECG measurements. The archive defines each class as one person, so training and test share those people, and very likely one recording each. The same identities are expected in an identification task, but new recordings are not tested. The model may recognise characteristics of the recording as well as the person.
CoffeeNew physical samplesUnclearThe cases are spectral measurements of coffee. We do not know whether a sample was measured more than once and whether those measurements were kept together. If one measurement is in training and another in testing, the test sample is familiar. The missing sample information prevents us from checking this possibility.
ComputersNew appliances or householdsUnclearThe source measured each physical appliance over many days. We do not know whether all days from an appliance or household were kept in one set. If the same appliance appears in both sets, the model may recognise its usual pattern. That would be a narrower test than recognising appliance types in new homes.
CricketXNew peopleUnclearThe data contain repeated gestures, but the files do not say which person produced each case. We cannot check whether all gestures from a person were kept in one set. A test gesture might come from someone already represented in training. Performance on people the model has never seen is therefore not established.
CricketYNew peopleUnclearSame as CricketX.
CricketZNew peopleUnclearSame as CricketX.
CropNew fields, scenes or seasonsConcernThe cases are pixel sequences from one scene and season. Nearby pixels may describe the same field and growing conditions. The value check also finds 2,753 of 16,800 test cases identical to training cases (Table 5). This makes the test different from evaluating new fields, scenes or seasons.
DiatomSizeReductionNew biological sourcesUnclearThe sources explain size variation within each class by successive generations of a clone, but do not say how the split was drawn or whether training and test come from the same clone. If they do, a new specimen from a familiar line is a narrower test than a specimen from an independently collected source. The intended application should make clear which of these is relevant.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
DistalPhalanxOutlineAgeGroupNew patientsUnclearThe cases are finger-bone outlines extracted from X-rays. We do not know whether one patient contributed several images or whether those images were kept together. If one patient appears in both sets, the model may benefit from familiar anatomy. Patient identifiers are needed to check performance on new patients.
DistalPhalanxOutlineCorrectNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
DistalPhalanxTWNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
DodgerLoopDayNew days or later periodsRelated periodsDays from one traffic detector were assigned at random. Training and test days can be part of the same baseball event or share similar traffic conditions. The test contains different days, but does not specifically evaluate a later period or another detector. This relationship should be considered when interpreting the score.
DodgerLoopGameNew days or later periodsRelated periodsSame as DodgerLoopDay.
DodgerLoopWeekendNew days or later periodsRelated periodsSame as DodgerLoopDay.
ECG200New recording sessions of a known patientConcernEach case is a heartbeat. The source thesis drew all 200 beats from one Holter recording of a single patient, so training and test beats share the patient and the recording session. For patient-specific monitoring a new case is a new recording session of that patient, which the split cannot test.
ECG5000New patients or recording occasionsConcernThe source record identified by the audit contains heartbeats from one patient during one long recording. Training and test beats therefore share that patient and recording. This can test more beats from that record, but it cannot establish performance on another patient or after the patient is recorded again.
ECGFiveDaysIndependent recordingsConcernThe label says which of two recording dates produced a heartbeat. Training and test beats come from those same two recordings of one person. The model may distinguish conditions specific to those recordings. The test does not show that the distinction holds when further independent recordings are collected.
EOGHorizontalSignalNew people or recording occasionsUnclearThe eye-movement samples come from six people. The audit describes splitting individual movements rather than keeping all movements from a person together. We cannot establish performance on unfamiliar users. Sharing users can suit a personalised system, but its test should explain whether they were measured again on a separate occasion.
EOGVerticalSignalNew people or sessionsUnclearSame as EOGHorizontalSignal.
EarthquakesNew periods or event groupsRelated periodsThe time windows are described as non-overlapping, so shared time samples are not an established problem here. However, windows from one earthquake sequence can be assigned to both sets. They may share the same broader event. The split does not specifically test later periods or separately collected event groups.
ElectricDevicesNew appliances or householdsConcernThe source kept each household in one set, but the shipped files differ in size from the published split, so it is unverified that the archive split keeps this. The value check also finds 208 test cases identical to training cases (Table 5). Also, 54 of its repeated-value groups carry conflicting labels. Most matches are two-valued on/off days, so they need not mean shared households.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
EthanolLevelNew bottle typesSeparatedThe archive paper says that a bottle type is never used in both training and testing [5]. The test therefore checks measurements involving unfamiliar bottle types. This is a documented source separation. It does not, by itself, establish that every other recording condition is different between the sets.
FaceAllNew photography occasionsConcernThe task identifies people, so those identities are expected in both sets. The concern is whether test photographs were taken on a different occasion. The audit does not establish this. Similar photos taken under the same conditions may make identification easier than recognising the person in newly collected photographs. The value check also reports one test case identical to training data (Table 5).
FaceFourNew source images or sessionsUnclearThe available record does not adequately describe the people, source photographs or capture sessions. We cannot check whether related images or acquisitions occur in both sets. No specific overlap is established by this absence of documentation.
FacesUCRNew photography occasionsUnclearThe task uses reprocessed photographs from the same collection as FaceAll. Identifying familiar people is expected, but the audit does not establish that test photos were taken on a separate occasion. Similar lighting, pose or camera conditions could help recognition without showing that it works in newly collected photographs.
FiftyWordsNew manuscript pagesUnclearThe word images come from one writer’s manuscripts. Words on the same page can share handwriting and scanning conditions. We do not know whether whole pages were kept in one set. The test may therefore contain more words from familiar pages, rather than words from pages the model has never seen.
FishNew physical specimensUnclearThe cases are outlines derived from fish images. The description does not establish how physical specimens and any repeated views were assigned to the two sets. New-specimen evaluation therefore cannot be verified; repeated views are a possibility to check, not an observed overlap.
FordANew vehicles or recording occasionsUnclearThe files do not identify the vehicle, test rig or recording occasion behind each case. We therefore cannot check whether these sources were kept apart. The split may be appropriate, but the available information does not show that testing uses vehicles or recording occasions absent from training.
FordBNew conditions and recordingsUnclearThe audit describes test data collected under different, noisier operating conditions. This tests a change in conditions; it does not demonstrate shared recordings. We still do not know which vehicles or recording occasions produced the cases, so we cannot check whether those sources are different between the sets.
FreezerRegularTrainNew physical freezersConcernThe source contains ongoing measurements of two freezers in one house. Pieces of the same recording can share appliance settings and operating conditions. The audit does not show that testing was kept for a later period or a new recording occasion. The value check separately reports 1 of 2,850 test cases matching training data.
FreezerSmallTrainNew physical freezersConcernSame as FreezerRegularTrain. This smaller-training-set variant has 27 of 2,850 test cases matching training data in the value check.
FungiNew biological strainsConcernThe source measured fungal strains repeatedly. The audit reports that testing includes another measurement of a strain used in training. A model could recognise that familiar strain. This is a narrower test than identifying the species of a newly collected strain, for which all measurements of a strain should be kept together.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
GestureMidAirD1New peopleSeparatedThe archive description says that no person occurs in both sets [27]. The test therefore uses people absent from training. There is disagreement in the records about which participant group is training, but both describe keeping the groups separate. No shared-person concern is identified by that description.
GestureMidAirD2New peopleSeparatedSame as GestureMidAirD1.
GestureMidAirD3New peopleSeparatedSame as GestureMidAirD1.
GesturePebbleZ1New peopleConcernThe same four users performed gestures on two occasions. Training uses the first occasion and testing the second [5]. Every test user is also in training, so the score can rest on per-user motion signatures and does not measure recognition for new wearers of the watch; GesturePebbleZ2 is the subject-disjoint version.
GesturePebbleZ2New peopleSeparatedTraining uses two people and testing the other two [5]. The model therefore sees unfamiliar users in the test. This differs from measuring familiar users again. The description establishes separation of people, although it does not establish separation of every other possible recording condition.
GunPointNew people or recording occasionsConcernAll 200 series come from one recording session of two actors, so training and test necessarily share that session and the split cannot test newly collected recordings. Which actor is on which side is not recorded. Having only two actors also limits how broadly results can be interpreted.
GunPointAgeSpanNew people or recording occasionsConcernThe archive pools actions from two actors recorded in two years [5]. A check of the shipped files finds every actor and year combination in both sets, so the same two people and recording sessions appear on both sides. The score therefore concerns more actions from that collection. It does not establish performance on unfamiliar people or measurements collected on a new occasion.
GunPointMaleVersusFemaleNew people or recording occasionsConcernThe two classes distinguish the two actors, and recordings from both years are pooled [5]. A model can recognise these particular people to predict the label. The audit reports no new recording occasion in testing. The result also does not establish the same distinction for other people.
GunPointOldVersusYoungNew recordings for each age conditionConcernThe label distinguishes the original recording years [5]. Both sets use actions recorded on those occasions. Differences in equipment or recording conditions could help the model along with age-related changes. Further recordings under each age condition would be needed to check whether the result reflects age rather than those occasions.
HamNew physical samplesUnclearEach physical ham supplies several image measurements. We do not know whether all measurements of a ham were kept in one set. The test could therefore contain another measurement of a familiar ham. Not finding very similar pairs in the original check does not prove that different hams were used.
HandOutlinesNew patientsUnclearThe outlines come from hand X-rays. We do not know whether one patient contributed several images or whether those images were kept together. The test could contain a new image of a familiar patient. Patient identifiers or an explicit patient grouping rule are needed to check this.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
HapticsNew recording occasionsUnclearThe labels identify users from repeated measurements. We do not know whether the test measurements were collected on occasions separate from training. Familiar identities are expected, but the model might also recognise conditions specific to one recording. Identification when a user is measured again remains unverified.
HerringNew physical specimensUnclearThe cases are outlines derived from fish ear-bone images. The description does not establish how physical specimens and any repeated views were assigned to the two sets. New-specimen evaluation therefore cannot be verified; repeated views are a possibility to check, not an observed overlap.
HouseTwentyNew householdsConcernThe cases are pieces of one household’s ongoing recording, so training and testing share the same home, its appliances and its routine. The test measures recognition within that home; performance in another home, which load monitoring needs, is not tested.
InlineSkateNew recording occasionsConcernSix of the 550 test series are value-identical to training series (Table 5). If the classes are individuals, training and test cycles may also come from the same treadmill run and share its person, equipment and conditions, but no source confirms this. The test does not show whether recognition works after collecting another recording. This interpretation also needs confirmation of what the original class labels identify.
InsectEPGRegularTrainNew insects or recordingsUnclearThe cases are pieces of long insect recordings. No insect or recording identifier ships with the data, so we cannot check whether pieces of one insect were kept in one set. Pieces in both sets could share the plant, sensor setup and nearby feeding behaviour, but this is not established. Unlike InsectEPGSmallTrain, which shares its test set, this training set has no exact matches with the test data.
InsectEPGSmallTrainNew insects or recordingsConcernThe cases are pieces of long insect recordings. No insect or recording identifier ships with the data, so whether pieces of one insect occur in both sets is not established. The value check finds 15 of the 249 test series identical to series in this small training set (Table 5), so a model can score them by recall.
InsectWingbeatSoundNew insect groups or recordingsUnclearThe source recorded insects in groups inside cages. Cases from one group can share its insects and recording conditions. The files do not identify those groups, so we cannot check whether a whole group was kept in one set. Shared groups are possible, but have not been verified.
ItalyPowerDemandNew days or later periodsRelated periodsTraining and test days can share the same season and similar weather. Different days are tested, so this is not automatically leakage. However, the split does not specifically test a later period or another grid.
LargeKitchenAppliancesNew appliances or householdsUnclearSame as Computers.
Lightning2New recorded eventsUnclearEach series is one recorded lightning event, but no source documents how the 121 records were divided or whether events of one flash or storm sit on both sides.
Lightning7New recorded eventsUnclearEach series is one recorded lightning event. The source evaluated by random cross-validation over all 143 records [9]; who built the fixed 70/73 split, and whether strokes of one flash sit on both sides, is not documented.
MallatNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
MeatNew physical samplesUnclearThe source measured each of 60 physical samples twice. We do not know whether the two measurements of a sample were kept together. If one goes into training and the other into testing, the model sees another measurement of a familiar sample. The sample assignment is needed to check whether this happened.
MedicalImagesNew patients or imagesConcernThe files do not identify the patient or original image behind each series. Different series might therefore come from one patient or image. We cannot check whether those sources were kept apart. Missing identifiers leave this question unanswered; they do not prove that sources were shared. Separately, the value check reports one test case identical to training data (Table 5).
MelbournePedestrianNew days or later periodsRelated periodsTraining and test days come from the same ten city sensors and the same year 2017; the assignment rule is undocumented. Training and test days can share local events, seasons and familiar sensors. The test checks more days from that collection. It does not specifically check a later period or a sensor that was absent from training.
MiddlePhalanxOutlineAge-GroupNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
MiddlePhalanxOutlineCorrectNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
MiddlePhalanxTWNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
MixedShapesRegularTrainNew images or physical objectsConcernThe cases are outlines from shape images. We do not know whether several views or processed versions of one original object can occur in both sets. If they do, the model may recognise a familiar shape. The source-image information needed to check this is not described in the audit. Separately, the value check reports one test case identical to training data (Table 5).
MixedShapesSmallTrainNew source images or objectsUnclearSame uncertainty about the original images as MixedShapesRegularTrain. The one value match mentioned there concerns MixedShapesRegularTrain, not this variant.
MoteStrainIndependent recordingsConcernThe audit traces cases to nearby pieces of a small number of sensor recordings, with the recording defining the class. Training and testing therefore share those original recordings. The model may recognise their particular conditions. The test does not show that the distinction works across recordings collected independently.
NonInvasiveFetalECGThorax1Independent recordingsUnclearThe original audit thinks the labels identify recordings and that training and test heartbeats come from those same recordings. However, the source mapping is not confirmed. If correct, testing checks more beats from familiar recordings. This remains a possible concern until the original sources and labels are confirmed.
NonInvasiveFetalECGThorax2Independent source recordingsUnclearSame as NonInvasiveFetalECGThorax1.
OSULeafNew physical specimensUnclearThe cases are outlines derived from leaf images. The description does not establish how physical specimens and any repeated views were assigned to the two sets. New-specimen evaluation therefore cannot be verified; repeated views are a possibility to check, not an observed overlap.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
OliveOilNew physical samplesUnclearThe cases are spectral measurements of oils. The source measured 60 oils in duplicate; the archive ships 60 spectra and does not say whether they are one per oil or both replicates of 30 oils. Looking for similar sequences cannot tell us which sample produced them. The test’s use of genuinely new samples therefore remains unverified.
PLAIDNew physical appliancesConcernThe source recorded individual appliances several times. We do not know whether all recordings of an appliance were kept in one set. Familiar appliances might occur in testing, but this is not established by the missing information. Separately, the value check finds five test cases identical to training cases.
PhalangesOutlinesCorrectNew X-rays or patientsUnclearSeveral bone outlines can come from one hand X-ray. Unless kept together, one image can supply outlines to both sets, letting the model benefit from familiar image conditions. The audit does not describe such grouping. Image or patient identifiers are needed to establish whether this possible overlap actually occurs.
PhonemeNew speakers or spoken recordingsUnclearOne spoken word can supply several phoneme segments. We do not know whether complete word recordings or speakers were kept in one set. Segments in both sets could share a voice and recording conditions. The test’s use of new speakers or separately recorded words remains unverified.
PickupGestureWiimoteZNew recording occasionsUnclearThe labels identify people, with several gesture repetitions per person. We do not know whether those repetitions were recorded on separate occasions or how the occasions were assigned to the sets. The test may check more repetitions from a familiar recording. Identification after recording the person again is not established.
PigAirwayPressureNew recording occasionsConcernThe task is to identify pigs. Training uses the first pieces of each pig’s before/after pressure recordings; testing uses later pieces of the same recordings [5]. The same pigs are expected in an identification task. The concern is that the model is tested on familiar recordings, rather than after measuring those pigs again.
PigArtPressureNew sessions of known animalsConcernSame as PigAirwayPressure.
PigCVPNew sessions of known animalsConcernSame as PigAirwayPressure.
PlaneNew physical models or capture sessionsUnclearThe photographs show die-cast plane models, 30 shape samples per class; the source does not say how many physical models were photographed. The description does not establish how physical models or photography sessions were assigned to the two sets. The test could therefore contain another view of a familiar model; that possibility is not verified from the metadata.
PowerConsNew days or later periodsConcernNearby days can share weather and routines, and no source says whether a later period was kept for testing. Separately, 67 of 180 test cases exactly match training cases (Table 5). These matches are a concrete issue beyond the possible relationship between neighbouring days.
ProximalPhalanxOutlineAgeGroupNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
ProximalPhalanxOutlineCorrectNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
ProximalPhalanxTWNew patientsUnclearSame as DistalPhalanxOutlineAgeGroup.
RefrigerationDevicesNew appliances or householdsUnclearSame as Computers.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
RockNew physical samplesUnclearThe original check found strongly similar pairs of sequences. That alone does not show that the same physical rock was measured twice. We need sample identifiers to know whether the two sets share rocks. Until then, repeated measurements are a possibility, rather than confirmed overlap.
ScreenTypeNew appliances or householdsUnclearSame as Computers.
SemgHandGenderCh2New peopleConcernThe original audit reports the same five people in both sets, checked using labels from the related person-identification dataset. Recognising a familiar person could help predict their gender label. The test checks further measurements of these people, rather than establishing performance on people absent from training.
SemgHandMovementCh2New people or recording occasionsConcernThe audit reports repeated grasps by the same people in the same recording sessions across both sets. These repetitions share the person and electrode placement. The test does not show whether recognition works for a new person or after removing and attaching the electrodes again. Those are separate evaluation settings.
SemgHandSubjectCh2New recording occasionsConcernThe task identifies people, but each person was recorded on one occasion according to the audit. Both sets contain repetitions from that occasion. The model might recognise that recording setup along with the person. The test does not establish identification after measuring the person again on another occasion.
ShakeGestureWiimoteZNew sessions of known usersUnclearSame as PickupGestureWiimoteZ.
ShapeletSimNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
ShapesAllNew source images or objectsConcernThe cases are outlines derived from shape images. The documentation does not establish whether repeated views or versions of one source object were kept in one set. Independence at the original image or object level remains unknown; no particular shared object is verified here. Separately, the value check reports two test cases identical to training data (Table 5).
SmallKitchenAppliancesNew appliances or householdsUnclearSame as Computers.
SmoothSubspaceNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
SonyAIBORobotSurface1New recording occasionsConcernThe audit describes short pieces from one continuous robot recording per surface. Training and testing therefore share the robot run and its recording conditions. The test checks more pieces from those runs. It does not show whether surface recognition works after collecting another recording on a separate occasion.
SonyAIBORobotSurface2New recording occasionsConcernSame as SonyAIBORobotSurface1.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
StarLightCurvesNew starsSeparatedThe source-level record describes one separate star per series. On that description, the test contains stars absent from training, rather than another piece of a familiar star’s recording. No shared-star concern is identified. Possible shared observing conditions or processing procedures were not separately checked.
StrawberryNew physical samplesConcernThe test set is described as the source study’s independently collected test data. However, the value check finds one sequence present in both sets with different labels (Table 5). Thus, the original split description does not resolve this inconsistency in the released data.
SwedishLeafNew physical specimensConcernThe cases are outlines derived from leaf images. The description does not establish how physical specimens and any repeated views were assigned to the two sets. New-specimen evaluation therefore cannot be verified; repeated views are a possibility to check, not an observed overlap. Separately, the value check reports two test cases identical to training data (Table 5).
SymbolsNew peopleUnclearThe source contains several symbol-drawing attempts per person. We do not know whether all attempts from a person were kept in one set. A test attempt could therefore come from a person already represented in training. This possible sharing needs participant identifiers to check; performance on new people is not established.
SyntheticControlNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
ToeSegmentation1New people or recording occasionsConcernThe source contains repeated motion recordings from a small group of actors. The original audit suspects that actors are shared between the sets, but the files lack actor identifiers for checking this. We therefore cannot establish performance on new people or new recording occasions, and should not present actor overlap as verified. Separately, the value check finds four test cases identical to training data (Table 5).
ToeSegmentation2New people or recording sessionsUnclearSame uncertainty about the actors and recording occasions as ToeSegmentation1. The four value matches mentioned there concern ToeSegmentation1, not this dataset.
TraceNew simulated eventsSeparatedEach series is described as a separate simulated event, so testing uses other generated events rather than another piece of one recording. No shared-event issue is identified by that description. Whether the generation process creates further dependencies would need its settings or code to check.
TwoLeadECGIndependent recordingsConcernThe label identifies which ECG lead produced the signal. Training and test beats come from one patient’s recording. The model may recognise differences specific to those leads during that measurement. The test does not show that the same distinction works when another patient or recording is used.
TwoPatternsNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
DatasetRelevant new caseVerdictWhat the split tests and why a concern remains
UMDNew generated examplesGeneratedThis is a generated task rather than measurements from people, devices or specimens. The corresponding source-separation question is therefore not assessed here. Its split would instead need to be examined through the generation procedure; synthetic origin alone is not evidence of leakage.
UWaveGestureLibraryAllNew days or recording occasionsUnclearThe original system recognises gestures for familiar users, so sharing users can be appropriate. The missing information is whether all recordings from a day were kept together. Without this, the test could contain gestures recorded on a day already represented in training, rather than gestures collected when the user returns.
UWaveGestureLibraryXNew sessions or daysUnclearSame as UWaveGestureLibraryAll.
UWaveGestureLibraryYNew sessions or daysUnclearSame as UWaveGestureLibraryAll.
UWaveGestureLibraryZNew sessions or daysUnclearSame as UWaveGestureLibraryAll.
WaferNew wafers or production runsConcernThe archive ships the six sensor traces of each wafer as separate cases and splits them without regard to wafer or run. Matching every case to the raw files shows 690 wafers and 99 of 104 production runs on both sides; every training case has another sensor of its wafer in test. The test therefore does not measure performance on new wafers or runs.
WineNew physical samplesUnclearWe do not know how many physical wines produced the spectral measurements or whether one wine was measured repeatedly. The two sets might contain measurements of the same wine. Not finding obvious similar pairs does not settle this question. Physical-sample identifiers are needed to check the split.
WordSynonymsNew manuscript pagesUnclearSame as FiftyWords.
WormsNew worms or recordingsConcernThe original record describes each case as a separate worm recording. However, the value check finds 10 of 77 test cases identical to training cases (Table 5). We therefore need worm and recording identifiers to resolve the disagreement between that source description and the released values.
WormsTwoClassNew worms or acquisitionsConcernSame as Worms.
YogaNew filming occasionsConcernThe task identifies two actors from video frames. No test frame is identical to a training frame, but every training frame has a near-twin in test (median distance 2.09 versus 23.8 for random pairs), so both sets come from one pool of frames. The model may recognise familiar filming conditions along with the actor. The test does not establish identification in footage collected on another occasion.

Table 6. UCR split-audit categorization (128 datasets): Concern 32.03%; Unclear 50.78%; Separated 5.47%; Related periods 5.47%; Generated 6.25%.

  • Concern (41 datasets) identifies either a relevant source unit shared across the split that the archive, its description paper, the source study or the released data establish, or at least one test series value-identical to a training series. Documented source sharing can let accuracy benefit from familiar recording or sample characteristics, while same-label exact matches permit memorisation.

  • Unclear (65 datasets) means that the information needed to check separation is missing.

  • Separated (7 datasets) means that the relevant sources differ between the sets, as documented or by construction when each series is a distinct event or object with no known grouping above that unit, supporting evaluation at that level. Separate sessions of familiar users, for example, do not establish performance on new users.

  • Related periods (7 datasets) identifies different days or windows from the same ongoing process, which may share seasons, events or operating conditions. Random assignment does not establish performance in a later period.

  • Generated (8 datasets) marks tasks not assessed here by patient, sample or recording separation. Their independence must instead be checked from how the examples were generated.

Overall, 106 of 128 datasets (82.8%) either receive a Concern label because of a documented or data-verified shared source or an exact cross-split match, or lack sufficient information to verify source separation. Each verdict applies to the setting described in its row and qualifies what the score can be taken to demonstrate. Rows marked “same as” share the named explanation.

References

  1. [1]Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L. Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), pages 437–442, 2013.
  2. [2]Anthony Bagnall, Aaron Bostrom, James Large, and Jason Lines. The great time series classification bake off: An experimental evaluation of recently proposed algorithms. extended version, 2016. URL https://arxiv.org/abs/1602.01711.
  3. [3]Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018, 2018. URL https://arxiv.org/abs/1811.00075.
  4. [4]Gari D. Clifford, Chengyu Liu, Benjamin Moody, Li-wei H. Lehman, Ikaro Silva, Qiao Li, Alistair E. W. Johnson, and Roger G. Mark. AF classification from a short single lead ECG recording: The PhysioNet/Computing in Cardiology challenge 2017. In Computing in Cardiology (CinC), volume 44, pages 1–4, 2017.DOI
  5. [5]Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The UCR time series archive, 2019. URL https://arxiv.org/abs/1810.07758.
  6. [6]Angus Dempster, François Petitjean, and Geoffrey I. Webb. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34: 1454–1495, 2020. URL https://arxiv.org/abs/1910.13051.
  7. [7]Angus Dempster, Navid Mohammadi Foumami, Chang Wei Tan, Lynn Miller, Amish Mishra, Mahsa Salehi, Charlotte Pelletier, Daniel F. Schmidt, and Geoffrey I. Webb. MONSTER: Monash scalable time series evaluation repository, 2025. URL https://arxiv.org/abs/2502.15122.
  8. [8]Bhaskar Dhariyal, Thach Le Nguyen, and Georgiana Ifrim. Back to basics: A sanity check on modern time series classification algorithms. In Advanced Analytics and Learning on Temporal Data (AALTD), ECML-PKDD Workshops, 2023. URL https://arxiv.org/abs/2308.07886.
  9. [9]Damian Eads, Daniel Hill, Sean Davis, Simon Perkins, Junshui Ma, Reid Porter, and James Theiler. Genetic algorithms and support vector machines for time series classification. In Applications and Science of Neural Networks, Fuzzy Systems, and Evolutionary Computation V, volume 4787 of Proc. SPIE, pages 74–85, 2002. doi: 10.1117/12.453526.DOI
  10. [10]Nick Erickson, Lennart Prurucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. In 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Track on Datasets and Benchmarks, 2025. URL https://arxiv.org/abs/2506.16791.
  11. [11]Vasilii Feofanov, Songkang Wen, Shifeng Xie, Simon Roschmann, Marius Alonso, Hongbo Guo, Romain Ilbert, Malik Tiomoko, Quentin Bouinot, Zeynep Akata, Lujia Pan, Jianfeng Zhang, and Ievgen Redko. Mantis: Lightweight foundation model for time series classification. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2502.15637.
  12. [12]Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MOMENT: A family of open time-series foundation models. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 2024. URL https://arxiv.org/abs/2402.03885.
  13. [13]Mathieu Guillaume-Bert and Artur Dubrawski. Classification of time sequences using graphs of temporal constraints. Journal of Machine Learning Research, 18(121):1–34, 2017. URL http://jmlr.org/papers/v18/15-403.html.
  14. [14]Jože Guna, Iztok Humar, and Matevz Pogačnik. Intuitive gesture based user identification system. pages 629–633, 07 2012. ISBN 978-1-4673-1117-5. doi: 10.1109/TSP.2012.6256373.DOI
  15. [15]Bing Hu, Yanping Chen, and Eamonn Keogh. Classification of streaming time series under more realistic assumptions. Data Mining and Knowledge Discovery, 30(2):403–437, 2016. doi: 10.1007/s10618-015-0415-0. URL https://link.springer.com/article/10.1007/s10618-015-0415-0.DOI
  16. [16]Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. Reduced, reused and recycled: The life of a dataset in machine learning research. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021.arxiv.org/abs/2112.01716
  17. [17]Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.arxiv.org/abs/1901.09005
  18. [18]Jaris Küken, Shi Bin Hoo, Martin Mraz, Frank Hutter, and Lennart Prurucker. TimEE: End-to-end time series classification via in-context learning, 2026. URL https://arxiv.org/abs/2607.07500.
  19. [19]Matthew Middlehurst, James Large, Michael Flynn, Jason Lines, Aaron Bostrom, and Anthony Bagnall. HIVE-COTE 2.0: a new meta ensemble for time series classification. Machine Learning, 110:3211–3243, 2021. URL https://arxiv.org/abs/2104.07551.
  20. [20]Matthew Middlehurst, Patrick Schäfer, and Anthony Bagnall. Bake off redux: a review and experimental evaluation of recent time series classification algorithms, 2024. URL https://arxiv.org/abs/2304.13029.
  21. [21]Matthew Middlehurst, Aiden Rushbrooke, Ali Ismail-Fawaz, Maxime Devanne, Germain Forestier, Angus Dempster, Geoffrey I. Webb, Christopher Holder, and Anthony Bagnall. The multiverse of time series machine learning: an archive for multivariate time series classification, 2026. URL https://arxiv.org/abs/2603.20352.
  22. [22]David Murray, Lina Stankovic, and Vladimir Stankovic. An electrical load measurements dataset of United Kingdom households from a two-year longitudinal study. Scientific Data, 4:160122, 2017. doi: 10.1038/sdata.2016.122.
  23. [23]Charlotte Pelletier, Geoffrey I. Webb, and François Petitjean. Temporal convolutional neural network for the classification of satellite image time series. Remote Sensing, 11(5):523, 2019.arxiv.org/abs/1811.10166
  24. [24]PhysioNet. Electrocardiographic imaging of myocardial infarction: The PhysioNet/computing in cardiology challenge 2007, 2007. URL https://moody-challenge.physionet.org/2007/.
  25. [25]Inioluwа Deborah Raji, Emily M. Bender, Amandalyne Paulada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 2021. URL https://arxiv.org/abs/2111.15366.
  26. [26]Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec. RelBench: A benchmark for deep learning on relational databases. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2407.20060.
  27. [27]Time Series Classification Archive. Time series classification: Dataset descriptions, n.d. URL https://www.timeseriesclassification.com/. Dataset documentation; accessed September 2026.
  28. [28]Zhihan Yue, Yujing Wang, Junyang Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2022), 2022. URL https://arxiv.org/abs/2106.10466.
  29. [29]Yunrui Zhang, Gustavo Batista, and Salil S. Kanhere. Revisit time series classification benchmark: The impact of temporal information for classification. In Advances in Knowledge Discovery and Data Mining (PAKDD), 2025. URL https://arxiv.org/abs/2503.20264.

Paper details

vlnc:W_3f2t5md5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents