Auditing a Model-Hub Benchmark: Data Quality, Training Budgets, and Side Information in HuggingBench

Introduction

Public model hubs have become the main distribution channel of machine-learning artifacts. A crawl of Hugging Face from December 2024 contains 1.2 million models [1], together with datasets, papers and interactive applications (Spaces), and a large share of the models are derivatives of popular base models: fine-tunes, quantized variants, adapters and merges [2]. The data management community has begun to treat such collections as model lakes, which call for the same functionality as data lakes—search, provenance, versioning and quality control—applied to models and their metadata [3, 4, 5, 6]. One user-facing instance of this functionality is recommendation: helping users find the models, datasets and Spaces that are relevant to them.

HuggingBench [1] turned this problem into a benchmark. It crawled HuggingKG, a knowledge graph of the Hugging Face ecosystem with 2.6 million nodes and 6.2 million edges, and released three test collections on top of it: resource recommendation from user likes, task classification, and lineage tracing (predicting the base model of a derived model). For recommendation, it released interaction splits and four external KGs that describe the items, and reported eleven recommenders implemented in the SSLRec framework [7], from graph collaborative filtering to KG-based recommendation.

A benchmark is a data product: its files, splits and protocol fix what every follow-up method will be compared on, and errors in them propagate silently into the literature [8, 9, 10]. As the first benchmark for its task, HuggingBench sets the reference point for future work. We therefore audit it end to end, asking three questions that are central to data engineering for machine learning: Are the released data what they claim to be? Do the reported conclusions depend on undocumented properties of the data and the protocol? Which signals that the data already contain are left unused? Our answers rest on checks of the released recommendation files and implementations, a replication of all reported recommendation methods, and new experiments that vary one property at a time; for the classification and tracing tasks, we examine the signals available in the released text.

Findings: The reported results of the recommenders replicate closely (Section V), but the audit changes what the benchmark’s numbers mean in four ways.

• Misaligned identifiers (Sections IV and VI). In three of the four external KGs, items and the entities that describe them have different identifiers, so KG recommenders, which assume that item $i$ is entity $i$, attach the KG facts to the wrong items. The KG files pass key and range checks; a join by content reveals the error. Correcting the files improves every KG-based method on every graph in every seed, most strongly for rarely liked items, and reverses ten of 27 comparisons between KG configurations.

• Truncated training (Section VII). Depending on the seed, the configuration selected on validation data exhausts the 100-epoch budget before early stopping in seven to eight of 29 settings; 21 of these 23 truncated runs improve with a longer budget, which also shrinks the apparent cost of the misalignment.

• Unused side information (Section VIII). The benchmark ships the text of every item (model and dataset cards, tags, metadata), but no reported method uses it. Closed-form linear models with this text achieve the highest

scores we observed, and lexical TF-IDF representations beat dense embeddings from two modern encoders under matched conditions. Type and identifier alone score close to the full text, and descriptions remain useful under an identifier mask.

• Provenance tasks (Section IX). Model cards name their base models: a training-free reading of the derived model’s card nearly matches the best KG embedding method for lineage tracing. For newly published derivatives, text and lineage are complementary signals of user interest in a simulated cold-start setting.

Contributions: Our contribution is an empirical study of integrity failures across independently exported artifacts of a released benchmark: how to detect and repair them, and how they and the evaluation protocol change downstream conclusions. (C1) An analysis of these failures in HuggingBench—identity mismatches between KG files that pass key and range checks, six unmapped encodings of the same interactions, and features that may include held-out interactions—with corrected KG files, the recovered mappings and executable checks, which pass on other releases in the same format but show that releases of one dataset can use incompatible numberings. (C2) A replication of all eleven recommendation methods and controlled experiments that isolate the effects of the data errors, the training budget and the unused side information, over three seeds and with paired confidence intervals. (C3) Strong, cheap baselines and an inheritance cold-start protocol for recommending new derivatives on model hubs. (C4) Lessons for builders of benchmarks and model lakes (§ 10).

Background and Related Work

Model hubs and model lakes: Systems for managing machine-learning models track versions, lineage and metadata of models and experiments [4, 5, 6]; the model lake vision extends data-lake functionality—discovery, provenance, citation and quality control—to large public model collections [3]. Empirical studies of Hugging Face analyze how models are reused and maintained [2, 11], and recent work recovers lineage from model weights [12]. Lineage also helps predict the benchmark performance of new derivatives [13]; model cards and metadata have been mined to reconstruct the supply chain of models and datasets [14]; and graphs of models, datasets and evaluations have been used to predict missing results, with a masking analysis of shortcuts in the metadata [15]. HuggingBench [1] complements these lines with a KG of the whole ecosystem and with benchmarks for recommendation, classification and lineage tracing; it is, to our knowledge, the only benchmark for recommending hub resources to users.

Data quality for ML benchmarks: Data validation is an established part of ML pipelines: declarative constraints and unit tests for data, including user-defined checks [16], schema-based validation of training and serving data [17], and debugging of data distributions in ML pipelines [18]. We use the same principle of executable checks and study one contract between independently exported artifacts—that an identifier denotes the same object in every file that uses it—together with the consequences of violating it for a benchmark’s conclusions. Documentation standards ask dataset creators to state how data were collected and processed [19, 20]. Errors in released benchmarks, such as label errors in test sets [8], and leakage between training and evaluation data [9, 10] can change the ranking of methods. Knowledge-graph quality is usually assessed for completeness and correctness of facts [21]; the failure studied here concerns the identity link between interaction data and otherwise well-formed KG files.

Reproducibility in recommendation: Systematic studies found that many neural recommenders do not outperform well-tuned simple baselines [22, 23, 24], that conclusions depend on evaluation details [25] and on how much each method is tuned [26, 27]. Item-item linear models [28, 29] and their normalized variant DAN [30] are strong top-$N$ recommenders, and closed-form extensions add item side information to them [31, 32]. Recent audits found inconsistencies between the released artifacts of recommender papers and their descriptions [33], and showed that third-party implementations of a recommender can diverge from the original [34]. We apply this perspective to a new kind of item catalog, whose items come with technical documentation and explicit lineage, and to a failure in the join between independently exported data files.

HuggingBench and Experimental Setup

Data and protocol: HuggingKG was crawled from the Hugging Face API in December 2024 and contains models, datasets, Spaces, papers, collections, users and organizations connected by 30 relation types. The recommendation data are the 5-core subgraph of user likes of models, datasets and Spaces: 38,624 users and 25,080 items (16,200 models, 3,634 datasets, 5,246 Spaces) with 1,065,220 likes, split per user into 60%/20%/20% for training, validation and testing. Methods rank all items, excluding the user’s training items, and are compared by Recall@$K$ and NDCG@$K$, $K \in\{5, 10, 20, 40\}$. Configurations are trained with early stopping on validation data: SSLRec evaluates every 3 epochs and stops after 5 evaluations without improvement of its first metric, within a budget of 100 epochs. This criterion, named MRR@5 in SSLRec, sums the reciprocal ranks of all relevant items in the top 5 before averaging over users; we keep it for selection and report it in Figure 2. The external KGs are the 1-hop and 2-hop neighborhoods of the items in HuggingKG, a homogeneous item–item graph with seven relation types (fine-tune, quantization, adapter, merge, model–training–dataset, Space–model, Space–dataset), and a publisher graph.

Reported methods: The original study reports graph collaborative filtering (LightGCN [35], HCCF [36], SimGCL [37], LightGCL [38], AutoCF [39], DCCF [40]), social recommendation (DSL [41], MHCN [42]), and KG-based recommendation (KGIN [43], KGCL [44], KGRec [45]) with each external KG. The best reported result is KGCL with the homogeneous graph (Recall@10 0.1578); KGIN obtains near-zero scores in every setting.

Implementations: We use SSLRec (commit 7597c31) with the configuration files, data handlers and run scripts of the HuggingBench repository, including its hyperparameter grids, on NVIDIA RTX 3090 GPUs; KGRec with three layers on the 2-hop graphs needs more than 24 GB and runs on an RTX 5090 with otherwise identical code (with PyTorch 2.7, which the RTX 5090 requires; a calibration epoch with two layers agreed with the RTX 3090 to within $10^{-3}$ in validation MRR@5 and Recall@10). Running KGCL required one compatibility fix (a private SciPy method removed in recent releases, replaced by an equivalent construction). For KGIN, the framework’s port does not train its embeddings on a GPU, an instance of the divergence of third-party ports [34]; this defect is unrelated to the data, so we evaluate KGIN with its authors’ original implementation (commit 0e43517), driven by the same data, evaluation and early-stopping protocol; we use this implementation for all KGIN results in the analysis of the KGs. We select every configuration of the reported methods on validation data, by the early-stopping criterion of the original study. All GPU runs use PyTorch 2.6 (CUDA 11.8) unless stated otherwise. Our additional methods are implemented in NumPy/SciPy and run on an 8-core CPU; they use the same data, ranking over all items and masking of training items, but their hyperparameters are selected differently, on a fixed quarter of the validation users by Recall@10: EASE $\lambda\in\{200, 500, 1000\}$; DAN user exponent $\beta\in\{0, 0.25, 0.5\}$, $\lambda\in\{50, 200, 800\}$ and item exponent $\alpha\in\{0, 0.1, \ldots, 0.4\}$; ItemKNN $k \in\{50, 200\}$ and shrinkage $\in\{0, 10, 100\}$; RP3beta $\alpha\in\{0.5, 1\}$, $\beta\in\{0.2, 0.4, 0.6\}$, $k = 200$; side-information and fusion weights on small logarithmic grids (Section VIII).

Statistics: Every trained configuration of the analysis is run with three seeds (42, the seed of the released configuration files, 43 and 44) and selected on validation data in each seed. Tables report the mean over seeds, except for the replication rows, which report seed 42, and we call a difference consistent when it has the same sign in all three seeds; the per-seed values of every setting are in our artifacts. Among validation-selected settings, the standard deviation of Recall@10 over seeds is at most 0.0073, except for LightGCL (0.0230), whose validation value in one seed stops improving after epoch 15; fixed configurations can vary more (Section VII). Comparisons with a single fitted model per method, such as our deterministic baselines against the reproduced LightGCN (seed 42), report 95% confidence intervals of paired differences from a bootstrap over test users with 1,000 resamples (for lineage tracing, over test triples); these intervals are conditional on the fitted models and the fixed split and do not include training variance.

Auditing the Released Data

We checked the released recommendation files for internal consistency and against HuggingKG, and the reported recommendation results against the protocol. The checks fall into six groups, each implemented as an executable test in our artifacts:

1. Split integrity: splits are disjoint, the $k$-core property holds, and per-user proportions match the stated split.

2. Cross-file consistency: alternative encodings of the same data are isomorphic copies of each other.

3. Join validity: every join by identifier between files agrees with the join by content (here, names).

4. Feature provenance: no attribute available to a method is derived from held-out interactions.

5. Protocol sanity: results exceed a random ranking, and the selected configurations converge within the budget.

6. Task shortcuts: every derived task is compared with a baseline that reads the released text.

Groups 1–3 run as one command over the 40 released files of the recommendation task, after checking them against the pinned release revision: 219 checks take 87 s and 1.8 GB of memory in a single process on a CPU. On the released files only the identity checks of the join fail, with 75,238 violations in the three misaligned KGs; on the corrected files all checks pass, and the correction leaves the KG triples, joined by entity names, and the interaction files unchanged. Figure 1 shows where in the benchmark’s pipeline each issue arises, and Table I summarizes how each is detected and its effect; this section describes the data issues, and Sections VII and IX the protocol and task issues.

Issue	Detection	Effect	Our handling
Six encodings of the same interactions (two unnamed matrix sets, one file set per KG)	Join named file sets on names; compare degree profiles of unnamed files	Per-user results of models trained on different file sets are not comparable; item metadata cannot be attached	Name joins; permutations of the unnamed matrices by color refinement (Sec. IV-B); release the mappings
Item–entity misalignment in three of four external KGs	Join items and entities by name; assert item $i$ = entity $i$	KG side information detached from its items; KG methods underestimated by up to 73% (Sec. VI)	Corrected KG files; alignment validator
100-epoch budget truncates training	Best epoch of the selected configuration at the end of the budget	7–8 of 29 settings exhaust the budget, depending on the seed; 21 of 23 truncated runs improve, the consistently truncated ones by up to 31% (Sec. VII)	Longer budget with unchanged early stopping; report the best epoch and flag it for a convergence check
Crawl-time popularity counters may include held-out interactions	Compare counters with benchmark likes per split	Potential leakage; no gain as an additive prior for EASE (Sec. IV-D)	Do not use; document
Model cards and node attributes contain the targets of the provenance tasks	Search card text and node fields for the target	A card-reading baseline nearly matches the best reported tracing method; label-naming node attributes are a potential leakage channel for classification (Sec. IX)	Report text baselines; mask identifiers

Split integrity

The three splits are disjoint (no user–item pair occurs in two splits, and no pair is repeated within a split), and every user and item has at least 6 interactions, consistent with the 5-core filter. The per-user training share lies between 0.50 and 0.67 (median 0.60), as expected from a per-user 60/20/20 split. Twelve items have test likes but no training likes; they cannot be recommended by collaborative methods, and their effect on the metrics is negligible.

Six encodings of the same interactions

The release encodes the same interactions in six file sets: two sets of sparse matrices without user or item names, used by general collaborative filtering (LightGCN and the contrastive models) and by social recommendation (DSL, MHCN), and one set of text files per external KG, used by the KG-based models, each with lists of user and item names. Every set has its own numbering: between any two KG file sets, at least 25,077 of the 25,080 items have different identifiers, and no mapping is released. Joined on names, the four KG file sets contain identical training, validation and test splits. For both unnamed matrix encodings, we recovered the permutation to the 1-hop files by color refinement (the 1-dimensional Weisfeiler–Leman algorithm [46], [47]) on the bipartite interaction graph with edges labeled by split. Under the recovered mappings, all three splits are identical; only 8 users and 8 items are structurally indistinguishable, so their assignment does not affect any metric. The mappings are needed to compare models trained on different file sets user by user and to attach item metadata from HuggingKG to the unnamed matrices; they also explain how the misalignment below could arise unnoticed, since every KG was exported with its own numbering.

Item–entity misalignment in the external KGs

KG-based recommenders read the interaction file and the KG triples by identifier only, and assume that recommendation item $i$ is KG entity $i$, i.e., that the first $n$ entity identifiers are reserved for the $n$ items, as in the KG datasets on which these methods were developed [48]. The released KG files include an item list (item name and identifier) and an entity list (entity name and identifier), so the assumption can be checked by a join on names. It holds for the homogeneous graph, but not for the 1-hop, 2-hop and publisher graphs: in the 1-hop and 2-hop graphs none of the 25,080 items has the same identifier as its entity, and in the publisher graph only 4 do. Checks of single files or keys do not reveal this: every released KG file set passes 34 key and range checks (contiguous identifiers, unique names, disjoint splits, all identifiers in the interaction and triple files within range, and every item identifier a legal entity identifier); only the join on names fails.

Table II measures what the recommenders actually see. We count the endpoints of triples that are recommendation items according to the entity names (facts about items), how many of them reach the right item under the identity assumption, and how many endpoints land on an item they do not describe. In the 1-hop and 2-hop graphs none of the 167,927 item endpoints reaches its item, and no item has a single correct fact; instead, 117,260 (1-hop) and 99,312 (2-hop) endpoints attach facts to unrelated items. The lost facts include all lineage relations between items (fine-tune, adapter, quantization and merge; 278,358 item endpoints), which describe exactly the near-duplicate structure of the hub. For example, the model SmolLM2-1.7B-Instruct is item 17166 in the interaction files but entity 366541 in the 1-hop graph, where it has 153 triples, including a quantization edge to SmolVLM-Instruct (entity 268711, also outside the item range). A KG recommender instead attaches to it entity 17166, the Space dpbos/postvalidation, which has one triple. In the publisher graph, 4 of 25,078 item endpoints are correct. The KG-based results of the original study were therefore obtained with side information that is, in effect, detached from the items.

KG		Triples	Item ends	Correct	Wrong item	Items
1-hop	rel.	1,180,911	767,927	0	117,260	0
	corr.	1,180,911	767,927	767,927	0	25,080
2-hop	rel.	3,063,081	767,927	0	99,312	0
	corr.	3,063,081	767,927	767,927	0	25,080
Publisher	rel.	25,078	25,078	4	39,535	4
	corr.	25,078	25,078	25,078	0	25,078

We built corrected versions of the three graphs, in which every item keeps its interaction identifier as its entity identifier and all other entities are renumbered after the items; all triples are kept. In the corrected files every item endpoint reaches its item (Table II). The same name join, run as an assertion before training, detects the error in seconds.

Leakage hazards

Crawl-time counters: HuggingKG stores the number of likes and downloads of every model, dataset and Space at crawl time. These counters are not recomputed from the training split and may contain the benchmark’s held-out likes: for 25,057 of the 25,080 items, the like counter is at least the item’s number of likes in all three splits, and it is more correlated with an item’s test likes than the training likes are (Spearman 0.80 vs. 0.71). As a popularity ranking it is not stronger than training popularity (Recall@10 0.0320 vs. 0.0358), because it also counts likes from users outside the 5-core. Adding a validation-weighted log-like-count prior to EASE did not improve Recall@10 (−0.00021 [−0.00041, −0.00003]), and subtracting the benchmark’s held-out likes from that counter had no measurable effect (cleaned minus raw: +0.00002 [−0.00003, +0.00009]). This control is specific to the additive prior; we exclude crawl-time counters because their provenance is not restricted to the training split, and recommend that releases either drop them or recompute them from the training split.

No interaction edges in the KGs: A KG that contains the user–item interactions themselves would leak test likes into KG-based recommenders. The released external KGs contain 27 (1-hop and 2-hop), 6 (publisher) and 7 (homogeneous) relation types, none of which is a like or another user–item interaction of the benchmark, so this form of leakage does not occur.

Identity across benchmark releases

The misalignment is a failure that the KG format permits rather than an error of one release. The format used by HuggingBench and SSLRec, inherited from KGAT [48], stores item and entity names, but its loaders link items to entities only by the convention that item $i$ is entity $i$, which holds by construction when items are numbered first and which none of the loaders we examined (SSLRec and the original KGIN code) checks. We ran the same checks on other releases in this format (Table III). Releases that number items first pass: the KGAT releases of Amazon-Book and Yelp2018 [48], which KGIN distributes unchanged [43], and their re-release with KGCL [44], which has the same interactions and an exactly relabeled item–entity graph, have no identity violation; the only defects are formatting ones, such as concatenated fields in one name list [49], and each check takes less than 11 seconds. Releases of one dataset need not share a numbering, however. KGPolicy [50] re-releases Amazon-Book and Yelp2018 with the same training interactions and KG facts that all occur in the KGAT release, but with different item identifiers; MCCLK [51] ships the MovieLens-1M files of MKR [52] (7,008 KG entities), while its documentation reports

Dataset	Releases	Within	Same ID	Kept
Amazon-Book	KGAT–KGCL	0 / 0	all 24,915	100%
Amazon-Book	KGAT–KGPolicy	0 / 0	32 of 24,915	0.7%
Yelp2018	KGAT–KGCL	0 / 0	all 45,538	100%
Yelp2018	KGAT–KGPolicy	0 / 0	489 of 45,538	2.4%
MovieLens-1M	MKR–RippleNet	0 / 0	67 of 2,347	—

the statistics of the RippleNet version [53] (182,011 entities), whose item identifiers differ. Each release is consistent on its own; the incompatibility arises when files of different releases are combined. Loaders that infer the numbers of items and entities from the files, such as those of KGAT and KGIN, then raise no error and detach the KG as in HuggingBench.

Replication of the Reported Results

We reran all reported recommenders with the authors’ configuration files and unmodified hyperparameter grids, fixed seeds and early stopping. The learning rates listed in the grids are not applied by the framework’s tuner, so all grid points train with the default learning rate of each configuration file, and grid points that differ only in the learning rate repeat the same run; we use these repetitions to measure run-to-run variation. Table IV compares the results. LightGCN and DCCF match all eight reported Recall and NDCG values to four decimal places, and SimGCL, AutoCF, DSL, MHCN, LightGCL, KGCL and KGRec replicate to within 0.0010 Recall@10 with every graph except KGCL with the 2-hop graph ($-0.0015$). HCDF differs by 0.002, within the spread of repeated runs of its selected configuration (Recall@10 0.1211–0.1275), which nondeterministic GPU operations make diverge after a few epochs; for LightGCN, SimGCL, AutoCF and LightGCL, the repetitions agree to within 0.0008 in every test metric. KGRec with three layers on the 2-hop graph—the configuration selected on validation data, which needs a 32 GB GPU—reproduces the reported value (0.0942 vs. 0.0941). The SSLRec port of KGIN replicates the reported near-zero scores (Recall@10 0.0004 with the 1-hop graph, the level of a random ranking); we ran the port with this graph only and use the original implementation for KGIN elsewhere. Table IV follows the original selection, which re-evaluates the restored best state; our three-seed analysis selects by the maximum of the validation curve, which for LightGCL with seed 42 gives Recall@10 0.1558 instead of 0.1561. The reported numbers are thus a faithful record of what the released code computes on the released data; the following sections examine what they measure.

Method	Reported R@10	Reported N@10	Reproduced R@10	Reproduced N@10	$\Delta$R@10
LightGCN	0.1301	0.1003	0.1301	0.1003	+0.0000
HCCF	0.1254	0.0975	0.1275	0.0988	+0.0021
SimGCL	0.1515	0.1158	0.1515	0.1158	+0.0000
LightGCL	0.1558	0.1198	0.1561	0.1198	+0.0003
AutoCF	0.1530	0.1174	0.1529	0.1174	−0.0001
DCCF	0.1493	0.1142	0.1493	0.1142	+0.0000
DSL	0.1425	0.1099	0.1433	0.1099	+0.0008
MHCN	0.1490	0.1154	0.1489	0.1158	−0.0001
KGCL (homog.)	0.1578	0.1220	0.1575	0.1222	−0.0003
KGCL (1-hop)	0.1490	0.1160	0.1496	0.1165	+0.0006
KGCL (2-hop)	0.1510	0.1170	0.1495	0.1159	−0.0015
KGCL (publisher)	0.1543	0.1195	0.1537	0.1194	−0.0006
KGRec (homog.)	0.0985	0.0751	0.0995	0.0755	+0.0010
KGRec (1-hop)	0.0897	0.0681	0.0887	0.0678	−0.0010
KGRec (2-hop)	0.0941	0.0729	0.0942	0.0726	+0.0001
KGRec (publisher)	0.0941	0.0734	0.0946	0.0738	+0.0005
KGIN (1-hop)	0.0004	0.0003	0.0004	0.0003	+0.0000

Effect of Correcting the Knowledge Graphs

Table V reruns every KG-based method with the corrected graphs, with the same grids and selection. With the reported Test set. Our runs use seed 42, the seed of the released configuration files, and select the configuration on validation data with the original early-stopping criterion. KG methods: external graph in parentheses. $\Delta$: reproduced minus reported Recall@10.

Method	Homogeneous	1-hop rel.	1-hop corr.	2-hop rel.	2-hop corr.	Publisher rel.	Publisher corr.
KGCL	0.1585	0.1496	0.1591	0.1503	0.1581	0.1540	0.1604
KGRec	0.0987	0.0845	0.1457	0.0937	0.1479	0.0943	0.1204
KGRec, 500 epochs	0.0987	0.1107	0.1481	0.0969	0.1493	0.1054	0.1379
KGIN (original code)	0.1351	0.1169	0.1445	0.1198	0.1446	0.1152	0.1475

100-epoch budget, the correction improves every method on every graph in every seed, by 4% to 73% in mean Recall@10; the gains are smallest for KGCL and largest for KGRec. For KGRec and the original KGIN, every corrected graph also exceeds the homogeneous graph in every seed; for KGCL, the corrected graphs are within 0.0019 of it on average, and the corrected publisher graph exceeds it in every seed. This changes the observed ranking of graph variants: KGCL with the corrected publisher graph (Recall@10 0.1604) is the best KG-based result overall. The corrections also change conclusions about individual methods: KGIN, near zero in the original study, reaches 0.1445 with its original implementation and the corrected 1-hop graph, close to the graph-based methods.

Identity or kind of side information?: The correction changes two things for an item: it receives facts about itself, and side information typical of items of its type and KG degree. We compare correct alignment with three random assignments of the item entities of the corrected 1-hop graph, stratified by item type and coarse degree bins (0, 1, 2–4, 5–9, 10–19, 20–49 and at least 50 facts), training KGRec with two layers in each seed. The correctly aligned graph outperforms every such assignment, by 0.0153 Recall@10 after 100 epochs and 0.0103 after 500 (means over seeds), and every assignment outperforms the released graph, in all nine runs. This control preserves the graph-wide degree distribution but not each item’s exact degree. A stricter assignment within item type and exact degree, which moves 98.4% of the items but only 51.0% of their KG endpoints, also stays below the corrected graph in every seed, by 0.0180 and 0.0264 Recall@10, and is no better than the coarse assignments. Neither control isolates semantic identity from all structural effects.

Where the correction helps: Relative to the item’s popularity, the correction helps most in the long tail. With the 500-epoch budget and the 1-hop graph, KGRec (seed 42) retrieves in its top 10 only 0.004 of the test likes of items with 1–4 training likes when the graph is misaligned, and 0.020 when it is corrected; for items with 20–99 training likes the share rises from 0.011 to 0.040, whereas for the most popular items (at least 100 training likes) it changes only from 0.148 to 0.155. With the 2-hop graph, popular items gain more in absolute terms (0.133 to 0.157), but the relative gain is again far larger for the rarest items (0.004 to 0.020); all paired intervals exclude zero, and with the 1-hop graph the same pattern holds in seeds 43 and 44. Items with fewer than 100 training likes receive 54% of the test likes. With misaligned side information, a KG recommender concentrates its hits on popular items, and the error hides in an aggregate metric that these items dominate.

Interaction with the budget: The measured correction gain also depends on the training budget: with 500 epochs the correction still improves KGRec on every graph in every seed (Table V), and Section VII examines this interaction.

How often the conclusions change: We count the pairs of settings compared in our tables whose order reverses consistently, i.e., in all three seeds, and separate three causes. Correcting the KGs reverses ten of the 27 orderings among KG configurations (variants of the graph for one method, and methods for one graph); for example, the original KGIN leads KGRec on the released 2-hop graph and trails it on the corrected one. Replacing the framework’s port of KGIN with the original implementation reverses one of the ten orderings of KGIN against the other reported methods, and extending the budget to 500 epochs (Section VII) reverses one of 36 orderings. Of the remaining pairs, eight, eight and 28 keep their order in every seed, and the others are not consistent across seeds.

The Training Budget

Which runs are truncated?: Early stopping is meant to make the number of epochs irrelevant, but only if the budget is large enough for it to trigger. We recorded, for every setting we ran with the 100-epoch budget (29 method–graph combinations, released and corrected graphs) and every seed, the epoch at which the selected configuration reached its best validation value and why it stopped. In eight, eight and seven settings of seeds 42, 43 and 44, the run reached the 100-epoch limit before early stopping. Five settings reach the limit in every seed: LightGCN, DSL, and KGRec with the released 1-hop, released publisher and corrected publisher graphs; LightGCL, the original KGIN with the released 1-hop graph and KGRec with the corrected 1-hop and both 2-hop graphs reach it only in some seeds (Table VI).

Setting	Trunc.	100 ep.	500 ep.	$\Delta$R@10 [min, max]
LightGCN	3	0.1297	0.1362	+0.0065 [+0.0048, +0.0076]
DSL	3	0.1433	0.1456	+0.0023 [0.0000, +0.0041]
LightGCL	1	0.1401	0.1400	0.0000 [−0.0001, 0.0000]
KGIN (orig.), 1-hop rel.	1	0.1169	0.1171	+0.0002 [0.0000, +0.0005]
KGRec, 1-hop rel.	3	0.0845	0.1107	+0.0262 [+0.0215, +0.0291]
KGRec, 1-hop corr.	2	0.1457	0.1481	+0.0024 [0.0000, +0.0044]
KGRec, 2-hop rel.	2	0.0937	0.0969	+0.0032 [0.0000, +0.0090]
KGRec, 2-hop corr.	2	0.1479	0.1493	+0.0014 [0.0000, +0.0021]
KGRec, publ. rel.	3	0.0943	0.1054	+0.0111 [+0.0100, +0.0126]
KGRec, publ. corr.	3	0.1204	0.1379	+0.0175 [+0.0118, +0.0216]

Extending the budget: In seeds 43 and 44 every run used a budget of 500 epochs; in seed 42 we reran every setting whose selected configuration reached the 100-epoch limit with 500 epochs and otherwise identical configurations Mean over three seeds; configuration selected on validation data in each seed. Every corrected value exceeds the released one in every seed. rel.: released graph; corr.: corrected graph. The homogeneous graph is aligned in the release and is shown for reference. KGIN: the authors’ original implementation. KGRec, 500 epochs: same grid with a 500-epoch budget and unchanged early stopping (Section VII); its homogeneous runs stop early under both budgets.

and early stopping, and in all other settings of seed 42 every grid configuration stopped early, so their result is the same under both budgets. A hook in the training loop records, within the same run, the test result of the best state after 100 epochs, so both budgets are evaluated on the same trajectory; short runs with and without the hook confirmed that it does not change training beyond the GPU nondeterminism already present before it fires. Of the 23 setting–seed pairs stopped by the limit, 21 improve under the longer budget, and not by a few extra epochs: their best epochs lie between 105 and 354. Reaching the limit does not guarantee a better test result: DSL in seed 44 keeps the same selected state and score (it stops early at epoch 111, best epoch 96), and LightGCL in seed 43 improves on validation but slightly decreases on test (−0.0001). Four of the settings that reach the limit in every seed improve in every seed: LightGCN gains +0.0065 Recall@10 (0.0048–0.0076 across seeds), and KGRec with the released 1-hop graph +0.0262 (+31%); settings that reach it only in some seeds gain little, and nothing in some seeds. Figure 2 shows typical validation curves: at epoch 100 they are still rising.

What changes: The budget affects two conclusions. First, it inflates the measured cost of the misalignment: for KGRec with the 1-hop graph, the gap between the corrected and the released graph shrinks in every seed, from 0.0613 to 0.0374 Recall@10 on average, because the model trained on the misaligned graph converges more slowly. With the model depth fixed, the same holds for three layers: extending the budget reduces the correction gain in every seed, by 0.0188 to 0.0279 Recall@10 (paired over users, all intervals excluding zero). With two layers, the run on the released graph stops early in seeds 43 and 44 (best epochs 33 and 27), so only the corrected graph gains from the longer budget and the change in the correction gain is positive (+0.0103 and +0.0132, against −0.0199 in seed 42). In all six cases the corrected graph remains better under both budgets, and the correction remains large and consistent (Section VI). Second, the budget changes the differences between KG-based methods: with 500 epochs, KGRec with the corrected publisher graph improves from 0.1204 to 0.1379, which closes most of its gap to the original KGIN with the same graph (0.1475). The ranking of the strongest methods does not change: the best graph-based and KG-based results stay below the simple baselines of Section VIII; LightGCN, for example, moves from 0.1297 to 0.1362.

Learning rates: The budget also interacts with the learning rate: smaller learning rates need more epochs, so a 100-epoch budget can make them look worse than they are. We therefore trained LightGCL, the strongest reported general recommender, with learning rates $10^{-4}$ and $10^{-5}$ set directly in the optimizer configuration and the 500-epoch budget. In every seed its best validation value remains below that of the default ($10^{-3}$), by at least 0.0021, so validation would still select the default, and with $10^{-5}$ even 500 epochs are not enough to converge (best epoch 498 or later). For LightGCL, the learning-rate controls thus do not improve the validation-selected result. For KGCL with the corrected publisher graph, the best KG-based configuration, we added learning rates $10^{-2}$ and $10^{-4}$ for both depths: with 100 epochs validation still selects the default in every seed, and with 500 epochs it selects $10^{-4}$ with three layers, which raises Recall@10 from 0.1604 to 0.1619, below EASE (0.1684). The other comparisons retain the released effective learning rates and selection procedures.

Simple Baselines and Unused Side Information

Linear models: Table VII compares the reported, reproduced and corrected results with simple baselines, tuned on a fixed quarter of the validation users: ItemKNN with cosine similarity and shrinkage, RP3beta [54], EASE [29] ($\lambda= 200$) and EASE with data-adaptive normalization (DAN [30]).

Method	R@10	R@20	R@40	N@10
Reported
LightGCN	0.1301	0.1932	0.2759	0.1003
LightGCL	0.1558	0.2228	0.3017	0.1198
KGCL (homo)	0.1578	0.2237	0.3059	0.1220
KGIN (homo)	0.0096	0.0146	0.0219	0.0076
Reproduced
LightGCN	0.1301	0.1932	0.2759	0.1003
LightGCL	0.1561	0.2229	0.3018	0.1198
KGCL (homo)	0.1575	0.2254	0.3066	0.1222
KGIN (1-hop)	0.0004	0.0009	0.0017	0.0003
Corrected
KGIN (orig., 1-hop corr.)	0.1445	0.2123	0.2984	0.1090
KGCL (publisher corr.)	0.1604	0.2291	0.3113	0.1242
500 ep.
LightGCN	0.1362	0.2018	0.2857	0.1048
KGRec (publisher, corr.)	0.1379	0.2018	0.2847	0.1058
Ours: CF
Popularity	0.0358	0.0653	0.1049	0.0265
ItemKNN	0.1406	0.1962	0.2660	0.1116
RP3beta	0.1699	0.2364	0.3172	0.1319
EASE	0.1684	0.2328	0.3076	0.1332
EASE-DAN	0.1746	0.2400	0.3149	0.1376
Ours: CF + side info
EASE + item graph	0.1776	0.2441	0.3221	0.1395
EASE + TF-IDF text	0.1846	0.2540	0.3335	0.1453
EASE + TF-IDF + graph	0.1860	0.2551	0.3332	0.1470

EASE, a closed-form item–item model with a single $L2$ penalty, reaches Recall@10 0.1684, above every reported, reproduced and corrected result; paired with the reproduced LightGCN (seed 42) on the same test users, it gains +0.0383 [+0.0355, +0.0410]. EASE-DAN is the strongest purely collaborative model (Recall@10 0.1746, +0.0062 [+0.0049, +0.0074] over EASE), 11% above the best reported result. HuggingBench’s data are sparse and long-tailed, with many near-duplicate items (quantizations and adapters of the same base model); on such data item–item models are hard to beat. Within the configurations, selection criteria and training budgets evaluated here, the linear models achieve the highest observed scores.

Item text: For every item we extract from HuggingKG its type, identifier (organization/name), pipeline tag, tags, libraries, languages and description (model or dataset card, cut to 3,000 characters; Spaces have none); the YAML front matter of the cards is already stripped in the release. We deliberately do not use the crawl-time popularity counters (Section IV-D). We add this text to EASE by adapting the closed-form side-information model of Jeunen et al. [31]: with L2-normalized TF-IDF vectors $f_i$, we add a weighted item–item similarity matrix with a zero diagonal, $S_{ij} = \cos(f_i, f_j)$ for $i \ne j$ and $S_{ii} = 0$, to the Gram matrix: $P = (X^{\mathsf{T}}X + wS + \lambda I)^{-1}$, $B = I - P\operatorname{diag}(P)^{-1}$, $\operatorname{diag}(B) = 0$. With $w$ selected on validation data from $\{3, 5, 10, 30, 100\}$, Recall@10 rises to 0.1846 (+0.0161 [+0.0146, +0.0177] over EASE); additive fusions of EASE with text scores gain only +0.0103 and +0.0104. Adding the item–item graph of HuggingKG (the seven relations of the homogeneous graph plus shared publisher) in the same closed form gives Recall@10 0.1860, a small but significant further gain (+0.0014 [+0.0006, +0.0024]); text alone exceeds the best reported result by 17%, and text with the graph by 18%. The released KG, once correctly joined, thus helps; the item text helps much more.

Representation: Table VIII compares TF-IDF with dense embeddings from bge-m3 [55] and Qwen3-Embedding-0.6B [56] under matched conditions: the same input text (descriptions cut to 6,000 characters, then truncated by each encoder’s tokenizer to 1,024 tokens, with TF-IDF computed on exactly the text the encoder sees), the same closed form, both as a full similarity matrix and as a symmetrized 100-nearest-neighbor graph, and weights selected on validation data from grids whose optimum lies inside. TF-IDF wins all four comparisons by 0.0046 to 0.0077 Recall@10, with all intervals excluding zero. Hub text is highly templated: the median TF-IDF cosine of two models by the same publisher is 0.40 (15% near-duplicates), compared with 0.03 for random pairs. Which fields carry the signal? With the same closed form and one weight grid for all views, the item type and identifier alone (organization and name) score close to the full text (Recall@10 0.1850; +0.0005 [−0.0006, +0.0016]), and remain close with the graph added as in the best model (0.1850 against 0.1860; −0.0010 [−0.0021, +0.0001]), the description alone gains +0.0134 over EASE, and still +0.0125 after URLs, repository identifiers and the item’s own organization and name are masked (18,251 of 19,191 descriptions change); the type and remaining metadata (pipeline tag, tags, libraries, languages and further tags) gain +0.0083. All five views improve over EASE, with paired intervals excluding zero. The signal is thus redundant across fields: identifiers, which encode the publisher and the model family, carry it in compact form,

View	Similarity	Repr.	R@10	ΔR@10 [95% CI]
bge-m3	full	TF-IDF	0.1846	–
		dense	0.1799	−0.0047 [−0.0059, −0.0034]
	kNN-100	TF-IDF	0.1844	–
		dense	0.1780	−0.0064 [−0.0077, −0.0049]
Qwen3-Emb.	full	TF-IDF	0.1847	–
		dense	0.1801	−0.0046 [−0.0059, −0.0033]
	kNN-100	TF-IDF	0.1844	–
		dense	0.1767	−0.0077 [−0.0090, −0.0062]

and descriptions remain useful after the mask, which does not remove all model-family references or indirect identity cues.

Where text helps: tbl:IX breaks the results down by item popularity, item type and user activity. Text and graph mainly help rarely liked items: for items with 1–4 training likes the share of test likes retrieved in the top 10 rises from 0.006 (EASE) to 0.034, whereas for the 1,214 most popular items it changes little (−0.003 [−0.004, −0.002]). Models gain most (+0.012) and Spaces, which have no card, least (+0.005); the gain decreases with user activity.

Cost: Fitting EASE takes 253 s on an 8-core CPU, dominated by one inversion of a 25,080 × 25,080 matrix, and the best model with text and graph 358 s including the construction of both similarity matrices. A single configuration of LightGCN trains for about 25 minutes on a GPU with the 100-epoch budget, and one of the original KGIN for up to 3.9 hours with the 1-hop graph; a full grid of KGRec on a 2-hop graph with 500 epochs takes more than a GPU-day.

Group	Share	LGCN	EASE	+T+G	Δ
Items by training likes
1–4	0.048	0.0059	0.0057	0.0339	+0.0282
5–19	0.186	0.0122	0.0156	0.0407	+0.0250
20–99	0.305	0.0128	0.0249	0.0405	+0.0156
≥100	0.461	0.1579	0.1976	0.1949	−0.0027
Items by type
Model	0.627	0.0808	0.1049	0.1168	+0.0119
Dataset	0.094	0.0368	0.0548	0.0624	+0.0076
Space	0.279	0.0903	0.1112	0.1161	+0.0049
Users by training likes
1–4	0.269	0.1570	0.2059	0.2287	+0.0228
5–9	0.372	0.1487	0.1904	0.2092	+0.0187
10–19	0.188	0.1108	0.1438	0.1600	+0.0162
20–49	0.116	0.0797	0.1032	0.1140	+0.0108
≥50	0.055	0.0455	0.0589	0.0618	+0.0029

Method	R@10	N@10	ΔR@10
Random	0.0287	0.0152
Lineage (parents)	0.1863	0.1131	−0.1366
Text profile: Qwen3-Embedding-0.6B	0.1343	0.0854	−0.1887
Text profile: bge-m3	0.2427	0.1531	−0.0803
Text profile: TF-IDF	0.3229	0.2371
EASE + TF-IDF (closed form)	0.3075	0.2261	−0.0155
TF-IDF + lineage	0.3541	0.2546	+0.0311
TF-IDF + bge-m3 + lineage	0.3602	0.2543	+0.0372

Provenance: New Derivatives, Lineage and Tasks

Inheritance cold start

New models on a hub typically appear as derivatives of existing ones and receive no interactions for a while. Lineage has been used to predict the benchmark performance of such models [13]; here it serves to predict user interest. We simulate this with an inheritance cold-start protocol on the released split. Among the 3,540 recommendation items with a parent item via a lineage relation (fine-tune, quantization, adapter, merge), we sample 20% (708) with a fixed seed as new models and remove all their training interactions (12,705 likes). Half of them select fusion weights, the other half (354 models) are used for testing: the held-out positives of a user are all of the user’s interactions with these models in the original splits, and every user with at least one positive (5,664) ranks the test models only. Text profiles score a new model by the cosine between the user’s profile (the normalized sum of the vectors of the user’s training items) and the model’s text vector; lineage scores combine the user’s interactions with, and EASE scores of, its parent models, and a typed variant also weights relation types and aggregates the warm siblings of the new model, with weights selected on the validation half; combinations sum z-normalized scores.

tbl:X shows that text is the dominant signal (TF-IDF profile: Recall@10 0.323), lineage alone reaches 0.186, and combining TF-IDF and lineage improves over either alone (0.354; +0.031 [+0.025, +0.037]), so the two signals are complementary for this task; the texts of new models are also not copies of their parents’ texts (median parent–child TF-IDF cosine between 0.13 and 0.22 across relation types). When combined with TF-IDF, the more elaborate typed variant brings no measurable gain (−0.0026 [−0.0075, +0.0019]).

Lineage tracing

The tracing task asks for the base model of a derived model (2,055 test triples, 94% fine-tunes); the reported methods are structure-only KG embeddings, with TransE [57] best at MRR 0.559. We replicated TransE with the authors’ LibKGE [58] configuration and learning-rate grid; validation selects the same learning rate in all three seeds, and the selected models reach a filtered MRR of 0.549 on average (0.544–0.558; Hits@1 0.442). Although the YAML front matter of the cards (which may contain a base_model field) is stripped in the release, the card bodies still name their base models. Such references have been mined from cards and metadata to reconstruct the supply chain of the hub [14]; here we measure how much of the benchmark task they solve. A training-free baseline that extracts the model identifiers mentioned in the derived model’s card and ranks them by order of appearance finds the correct base model among its candidates for 54.3% of the test triples; assigning zero reciprocal rank when the target is not extracted, it reaches MRR 0.542 (Hits@1 0.542), close to TransE without using the graph. Ranking the extracted models first and all other entities by TransE yields MRR 0.831 on average, +0.2746 to +0.2881 over TransE, with paired intervals over test triples that exclude zero in every seed. The task thus partly measures how well a method reads the card, a form of the metadata shortcuts that masking analyses also address in artifact-graph prediction [15]; future versions should report such text baselines or mask identifiers in the cards.

Task classification

The released classification test set contains 27,345 items (the original paper reports 28,682). A literal matcher that predicts the task names occurring in an item’s card reaches micro-F1 0.404, close to predicting the most frequent task (0.384) and far below the best reported 0.885; this weak matcher does not rule out other textual shortcuts. A structured attribute of the item node—the pipeline_tag of models and the task fields of datasets—names one of the target labels for 99.99% of the test items. This is a potential leakage channel: the code that builds the released node features is not public, so we could not verify whether the original classifiers use these attributes; a release should state it explicitly.

Lessons for Benchmark and Model-Lake Builders

Validate joins, not only files: Each released KG file set of HuggingBench passes key and range checks (Section IV-C); the error lies in the join between files. Joins between interaction data and side information should be validated by content (here, entity names) before any model is trained, and the validator should ship with the benchmark. The same holds for alternative file sets of the same data, which should come with an explicit mapping.

Prefer explicit links to implicit conventions: The item–entity convention fails silently both within a release (Section IV-C) and across releases of one dataset (Section IV-E). Formats whose loaders read an explicit link avoid this failure: RecBole [59], for example, requires a separate item–entity link file for knowledge-aware models. Benchmarks and model lakes that join artifacts with side information should store such links explicitly, or ship a validator that asserts the convention, and identify each release by a version and checksums rather than by the name of the dataset [60].

Release the provenance of every feature: The crawl-time counters and the label-naming node attributes are correct data that are dangerous in the wrong place. A benchmark built from a live hub should record how and when every attribute was obtained, and drop or recompute from the training split any attribute that may contain held-out interactions.

Make the protocol self-checking: A budget that binds the selected configuration is visible in the training logs. Benchmarks should report the best epoch of every selected configuration and flag a best epoch at the budget for a convergence check, and results at the level of a random ranking (Recall@10 0.0004 here), such as the near-zero KGIN scores of the original study, should be treated as errors rather than as findings.

Include baselines that use what is released: Benchmarks that release side information should report at least one simple model that uses it, and a tuned closed-form linear model (EASE or EASE-DAN) is a cheap, strong default for sparse implicit feedback.

Separate provenance from content: On a model hub, documentation, identifiers and lineage are intertwined: cards name base models and tasks. Provenance tasks should mask the information they ask for, or report text baselines that show how much of the task is solved by reading.

Limitations: Likes are a weak form of feedback; the data are one snapshot with a random per-user split, and our cold-start protocol simulates new items on this split rather than following models as they are published. Our seeds vary the training of the models but not the split; we tuned learning rates explicitly only for LightGCL and KGCL and used text encoders off the shelf.

Conclusion

HuggingBench’s reported results are reproducible, but what they measure is shaped by properties of the release that no result table shows: misaligned identifiers that detach the external KGs from their items, a training budget that ends before several methods converge, and rich item text that no baseline uses. Correcting the KG joins consistently reverses ten of 27 comparisons between KG configurations and changes the verdict on KGIN; the implementation and the training budget affect further comparisons; and, within the configurations and selection procedures evaluated here, closed-form models with item text achieve the highest observed scores. The corrected KG files, the mappings between the file sets, the validators, the inheritance cold-start split, the patches, scripts and logs of every run are available at https://doi.org/10.5281/zenodo.23857277.

Acknowledgment

This manuscript acknowledges the use of Claude Opus 5.5 [61], developed by Anthropic, and of GPT-6.1 Sol [62], developed by OpenAI, to improve language clarity, refine sentence structure, and enhance overall writing precision. The study design, experiments, analysis, results, and conclusions remain the sole responsibility of the authors.

References