betaFeedback

WorksComputer Science

LECAT: Measuring LLM Category Learning from Pretraining Data

Abstract

Humans know that airplanes are fast, that this extends to red airplanes, but that it does not hold for broken airplanes. How do LLMs acquire category knowledge that humans learn and apply effortlessly? We introduce LeCat, a two-part framework to analyze LLM's Learning of Category knowledge. First, we use LeCat to evaluate models on their ability to learn category knowledge, distinguishing between three cases: memorization of in-domain category knowledge, generalization to unseen instances, and ability to flexibly use knowledge in appropriate contexts. Then, we use LeCat to trace how this knowledge can be learned in pretraining from text that may encode category knowledge in various ways, such as verbatim repetition or exceptions, which we refer to as forms of "knowledge expression". We first assess open-source pretrained models using 199 curated category-feature pairs and analyze their training data. Our findings suggest that strengthening context and category-feature co-occurrence likelihood (PMI) improve learning for larger models and verbatim rule-like statements diminish flexible category learning. To systematically isolate these effects, we then train a 68M model from scratch with precise control over the categorical information each sees in their training data, confirming that verbatim repetition reduces flexible learning but mixture of diverse knowledge expressions lead to more robust category learning. Together, our findings highlight that while contextual flexibility is persistently difficult for models, how category knowledge is expressed in pretraining greatly impacts models' acquisition of robust category knowledge.

Introduction

Category learning is fundamental to human intelligence [31, 48], enabling generalization from sparse samples (e.g., airplanes are fast) to large sets of unseen examples (e.g., red airplanes are fast, blue airplanes are fast). This capacity has been extensively studied in cognitive science and psychology [2, 26, 40, 50], as well as in classical machine learning and computer vision [19, 23]. Unlike factual knowledge, which is typically isolated (e.g., the Louvre opened on August 10, 1793), category membership licenses broader inference: the Louvre is a museum, so it likely shares characteristics with other museums [6, 21, 32]. Yet this same inferential power can be harmful when applied inaccurately, as with social stereotypes [4, 44]. Thus, for robust generalization, learners must also acquire exceptions and counter-stereotypes, and flexibly incorporate context.

In this work, we empirically investigate how language models learn to generalize and apply category knowledge under various contexts, focusing specifically on how pretraining data shapes model behavior. While prior work has established that language models acquire factual or world knowledge from pretraining data [9, 15] which has prompted extensive research on data curation (e.g., [24, 36, 37]) and knowledge injection (e.g., [7, 25]), how LMs acquire and use category knowledge from data is poorly understood. Clarifying how LLMs robustly acquire category knowledge would inform efforts to reduce harmful associations such as stereotypes and biases [1], and promote more flexible, creative category representations that yield diverse outputs [41] (Kenett 2025).

We develop LECAT, a framework to analyze the Learning of Category knowledge and usage in LLM outputs and its training data. Focusing on LLM outputs, we develop log-probability measure, validated with LLM-as-a-judge, for category knowledge memorization (i.e., generation of category knowledge in its basic form), generalization (i.e., extension to invariant contexts), and contextual flexibility (i.e., behavioral differences based on context). For examining how training data informs LLM outputs, LeCat draws from cognitive science [26] to measure the prevalence of five types of category knowledge expressions in data, including verbatim repetition of category-feature associations (e.g., “The airplanes are fast”), exceptions (e.g., “This airplane is not fast”), weakening contexts (e.g., “The airplanes with broken engines are not fast”), and more.

We first apply LeCat to open-data models of several sizes, Pythia [5] and OLMo 3 [35]. At every scale, contextual flexibility remains difficult: models struggle to adjust which features they produce in context, especially when the context weakens or contradicts (i.e., is erroneous about) the category-feature association. In the pretraining data, the factors that support category learning shift with scale. Strengthening contexts and category-feature co-occurrence (measured by PMI) become more beneficial as models grow, with PMI correlating positively with learning only in larger models. Verbatim statements of category-feature associations, by contrast, hurt flexible knowledge use at every scale, possibly through memorization. Since pretraining data are heterogeneous, however, these findings remain correlational.

To test these trends causally and isolate individual expression types and paradigms such as curriculum ordering, we create a fully controlled pretraining corpus by adapting the TinyStories dataset [9] by injecting two fictional category-feature relations, valins–poor and pezzdens–fast. We train a 68M parameter model from scratch on each expression type alone, a random mixture of all types, and a curriculum-ordered mixture. The results confirm the open-model finding that flexible category knowledge is difficult to learn but a diverse mixture improves while verbatim repetition reduces flexible category learning.

Together, these findings show that pretraining data composition, not just exposure frequency, shapes category learning, that the effective composition shifts with model scale, and that contextual flexibility is persistently difficult for models. Our work thus points to practical implications for stereotypes and factual learning in LLMs.

Background & Related Work

Category learning is fundamental to human cognition, enabling classification, generalization, and inference [48, 32, 3]. Humans acquire categories through diverse mechanisms—from explicit rules to implicit similarity detection [28, 30]—and readily accommodate exceptions to typical category structure [14]. Crucially, human category representations are context-dependent: many categorical concepts are multi-dimensional and distributed across brain regions, allowing certain features to be more activated depending on context [51, 52]. This flexibility supports accurate generalization, reduces stereotyping [41], and facilitates creative reasoning (Kenett, 2025).

Prior work on LLM knowledge learning during pretraining has examined memorization [11], statistical rule-learning [34], and factual knowledge learning [7, 47, 33]. Furthermore, works in LLM concept learning has explored concept associations [17] and property inheritance between categorical classes [29]. However, how LLMs acquire category knowledge and learn to generalize it robustly and flexibly remain underexplored. Inspired by the key components of human category learning, we investigate how memorization, generalization (i.e., application of category knowledge to unseen members under irrelevant contexts), and contextual flexibility (i.e., application of category knowledge that reflects the context) emerge in language models as a function of pretraining data. Extended discussion of prior works in human category learning and LLM knowledge learning during pretraining is in Appendix Appendix § 8.

LeCat: LLM Category Learning Analysis Framework

We introduce LLM evaluation and data analysis framework for learning categories (LeCAT), building on work in cognitive science. First, we create an output evaluation framework using completion probes based on completion of target feature given category and varying contextual cues (§ 3.1). We then craft five types of category knowledge expressions to analyze and create synthetic data (§ 3.2).

LeCat-Output: LLM Evaluation

To evaluate category learning in LLM outputs, LeCat probes along three behavioral dimensions grounded in human category learning research [32, 26, 49]: memorization of category-feature associations, generalization to unseen instantiations, and contextual flexibility in applying associations appropriately. We operationalize each dimension using completion-based probability probing, validated with an LLM-as-a-judge evaluation of output generations.

Target–Max Log-Probability Gap

We quantify category knowledge as the strength of association between a category cc (e.g., airplanes) and a feature ff (e.g., fast), operationalized as P(f∣c)P(f \mid c) following prior work in semantic cognition [26]. For different behaviors such as memorization, generalization, and contextual flexibility, we build contextual prefix as shown in Table 1. To measure category and feature association at each training step, we define a log-probability gap metric, measuring a gap between most probable completion token and target token. Let pθti(⋅∣x)p_{\theta_{t_i}}(\cdot\mid x) denote the model’s next-token distribution at step tit_i given input xx, and let ymax⁡=arg⁡max⁡ypθti(y∣x)y_{\max} = \arg\max_y p_{\theta_{t_i}}(y \mid x) be the most probable next token. For a target token ytargety_{\mathrm{target}}, we define:

Category LearningProbe TypeAirplanes (target: fast)
MemorizationBaseAirplanes are fast
GeneralizationInvariant context generalizationRed airplanes are fast
GeneralizationErroneous generalizationSlow airplanes are fast
Contextual flexibilityStrengthening contextAirplanes with strong engines are fast
Contextual flexibilityWeakening contextAirplanes with broken engines are fast

Table 1. Evaluation probes for measuring memorization, generalization, and contextual flexibility.

G(x,ytarget)=log⁡pθti(ytarget∣x)−log⁡pθti(ymax⁡∣x).G(x, y_{\mathrm{target}}) = \log p_{\theta_{t_i}}(y_{\mathrm{target}} \mid x) - \log p_{\theta_{t_i}}(y_{\max} \mid x).

Higher values of GG (closer to 0) indicate greater model confidence in the target feature relative to competing predictions. We instantiate GG across three evaluation dimensions below. When target feature is not single token, we average across the token log probabilities.

Memorization captures the strength of the basic category-feature rule: whether the model produces “Airplanes are fast” even when this exact string was absent or rare in training [11, 16], reflecting feature production norms given a category [27]. Given a base probe xx (e.g., “Airplanes are”), we set ytargety_{\mathrm{target}} to the associated feature token (e.g., “fast”). A model that has acquired the association yields G(x,ytarget)→0G(x, y_{\mathrm{target}}) \to0 as training progresses. We denote this as GbaseG_{\mathrm{base}}.

Generalization measures whether category knowledge transfers to unseen instances while remaining robust to over-generalization. For generalization to occur, models should be able apply knowledge to context changes that are invariant presence of a feature (e.g., “red airplanes are”), while in turn lowering its confidence when context contradicts it (e.g., “slow airplanes are”). Formally, for invariant contexts xinv=(c,ctxinv)x_{\mathrm{inv}} = (c, \mathrm{ctx}_{\mathrm{inv}}), the model should preserve confidence in ytargety_{\mathrm{target}}, yielding Gbase≳G(xinv,ytarget)G_{\mathrm{base}} \gtrsim G(x_{\mathrm{inv}}, y_{\mathrm{target}}). For erroneous contexts xerr=(c,ctxerr)x_{\mathrm{err}} = (c, \mathrm{ctx}_{\mathrm{err}}), the model should assign lower confidence, yielding G(xinv,ytarget)>G(xerr,ytarget)G(x_{\mathrm{inv}}, y_{\mathrm{target}}) > G(x_{\mathrm{err}}, y_{\mathrm{target}}). Together, correct generalization satisfies Gbase≳Ginv>GerrG_{\mathrm{base}} \gtrsim G_{\mathrm{inv}} > G_{\mathrm{err}}.

Contextual Flexibility measures whether the model appropriately modulates feature confidence based on relevant contextual cues. Some contexts strengthen the association (e.g., “an airplane with a new engine”) while others weaken it (e.g., “an airplane with an old engine”). We denote these as GweakG_{\mathrm{weak}} and GstrG_{\mathrm{str}}.

For a model that has learned category knowledge flexibly, we expect the correct ordering of the probe-based metric to be

Gstr>Gbase≳Ginv>Gweak>Gerr.G_{\mathrm{str}} > G_{\mathrm{base}} \gtrsim G_{\mathrm{inv}} > G_{\mathrm{weak}} > G_{\mathrm{err}}.

We test models’ alignment with this ordering through pass rate of pairwise ordering, e.g., whether the model passes the Gstr>GbaseG_{\mathrm{str}} > G_{\mathrm{base}} order.

LLM-as-a-Judge Validation

To test whether these log-probability results hold for actual model outputs, we use Qwen3.8-27B1 as an LLM judge [8] to evaluate free-form generations. For each generation, the judge annotates (1) whether the feature can be inferred and (2) the strength of the inference. To elicit the features, we use the respect prefixes from Sheng et al. [43], which convert each probe into a template such as “XYZ are known for being” (e.g., “Red airplanes are known for being”; see Table 5 in Appendix § 10.§ 1). Across the open-source models (Pythia and Olmo3), both judge annotations correlate positively and significantly with our measure (Spearman ρ=0.88\rho= 0.88 for feature inference and ρ=0.87\rho= 0.87 for inference strength, both p<.01p < .01; see Appendix § 9.§ 1 for full results), indicating that our probability-based measure is a valid and computationally efficient proxy for category learning.

Generation Prompt typeOriginal: pigs →\to pinkGenerated: pigs →\to intelligent
BasePigs are thought of as being . . .<br>The pig is thought of as being . . .<br>Most pigs are thought of as being . . .<br>This pig is thought of as being . . .<br>People say pigs are thought of as being . . .Pigs are well-known for being . . .<br>A pig is well-known for being . . .<br>Many pigs can be well-known for being . . .<br>The pigs seem to be well-known for being . . .<br>Some pigs are well-known for being . . .
IrrelevantQuiet pigs are thought of as being . . .<br>Hungry pigs are thought of as being . . .<br>Pigs near fences are thought of as being . . .<br>Pigs in barns are thought of as being . . .<br>Resting pigs are thought of as being . . .Pink pigs are well-known for being . . .<br>Large pigs are well-known for being . . .<br>Spotted pigs are well-known for being . . .<br>Muddy pigs are well-known for being . . .<br>Pigs with curly tails are well-known for being<br>. . .
StrengtheningNewborn pigs are thought of as being . . .<br><br>Hairless pigs are thought of as being . . .<br><br>Pigs with pale skin are thought of as being . . .<br><br>Freshly washed pigs are thought of as being . . .<br><br>Cartoon pigs are thought of as being . . .Pigs trained in mazes are well-known for being<br>. . .<br><br>Pigs solving puzzles are well-known for being<br>. . .<br><br>Pigs learning commands are well-known for<br>being . . .<br>Pigs recognizing symbols are well-known for<br>being . . .<br><br>Pigs using mirrors are well-known for being . . .
WeakeningWild pigs are thought of as being . . .<br>Muddy pigs are thought of as being . . .<br>Pigs with dark coats are thought of as being . . .<br>Bronze pig statues are thought of as being . . .<br>Pigs under red lights are thought of as being . . .Toy pigs are well-known for being . . .<br>Plastic pigs are well-known for being . . .<br>Porcelain pigs are well-known for being . . .<br>Wooden pigs are well-known for being . . .<br>Stuffed pigs are well-known for being . . .
ErroneousBlack pigs are thought of as being . . .Unintelligent pigs are well-known for being . . .

Table 5. All open-ended prompts for one category, for the original stereotypical feature and its matched generated feature. Generation prompts carry no context; the other rows are the context probes of Table 4, rewritten with the pair’s respect template and cut before the feature. The model’s continuation is judged for which features it attributes to the category.

LeCat-Data: Pretraining Data Analysis and Generation Framework

Focusing on how training data shapes category knowledge, LeCat examines how different expressions of category knowledge in data lead to learning. Drawing on prior work in cognitive science [26], we consider four common types of expressions: verbatim repetition, instantiation, exceptions, and context dependence. We illustrate each type with real excerpts from The Pile [10]. Formal definitions are given in Appendix § 9.§ 2.

Verbatim Repetition (DrepetitionD_{\mathrm{repetition}}). We begin with the simplest learning signal: explicit category–feature statements without variation, contrast, or context. Repeating knowledge in this way, especially in generic form, transmits essentialist beliefs and represents a top-down approach to learning concepts [40, 46, 53]. We operationalize this signal, for both detection and generation, as texts that state the generic claim without qualification, additional attributes, or context. For example, the following pretraining excerpt establishes the generic claim directly: “...3. Skyscraper – A skyscraper is a tall, continuously habitable building having multiple floors...” (skyscraper, tall).

Examples and Instantiations (DinstantiationD_{\mathrm{instantiation}}). Category learning often arises from comparing multiple members of the same category and identifying their shared structure [26]. This corresponds to a bottom-up approach to category learning [46], and it tests whether a model can abstract the generic property from exemplars. Here the feature is expressed through an individual member described with additional attributes that do not bear on the feature, as in: “...Calm, collected and outwardly empathetic, she is all one would expect from a woman...” (woman, empathetic).

Learning Exceptions (DexceptionsD_{\mathrm{exceptions}}). Humans also learn to accommodate exceptions, which are category members that violate expectations [42]. Accommodating them is essential for accuracy and supports flexible, non-stereotypical reasoning [14, 2, 12]. However, exceptions are harder to learn accurately [14], and their effects on category knowledge vary. They can have no effect, backfire and reinforce the rule [22], or discredit the rule [45]. We therefore include exception expressions to study how LLMs learn exceptions and how exceptions affect category learning. These are texts in which a category member lacks the typical feature with no explanatory context, as in: “...I said that I hated to shoot such a small moose on opening day...” (moose, large).

Context-Dependent Associations (Ddep. strengthenD_{\mathrm{dep.\ strengthen}}, Ddep. weakenD_{\mathrm{dep.\ weaken}}). Many category–feature associations depend on context [52, 51]. For instance, the association “airplanes are fast” may weaken with an old engine or in heavy rain. Such flexibility is crucial for applying category knowledge accurately [49] and for reasoning beyond stereotypes [2]. We distinguish contexts that strengthen the association (Ddep. strengthenD_{\mathrm{dep.\ strengthen}}), where the feature holds and the context reinforces it, from contexts that weaken it (Ddep. weakenD_{\mathrm{dep.\ weaken}}), where the feature is absent and the context accounts for its absence. The following excerpt adds a strengthening condition: “...Every day the cooks got four loads of oak charcoal from Nicolosi to keep the ovens always hot...” (oven, hot). In contrast, this excerpt adds a weakening condition: “...There were two knives in the kitchen, both blunt and rusted...” (knife, sharp). Ddep. weakenD_{\mathrm{dep.\ weaken}} and DexceptionsD_{\mathrm{exceptions}} both describe a member lacking the feature. They differ only in whether a context accounts for the missing feature. This contrast lets us test whether explicitly stated contexts lead to flexible category knowledge, and how they affect other objectives such as memorization and generalization.

Using the framework. We use these operationalizations for both data analysis and synthetic data generation. For pretraining data analysis, we use an LLM classifier to characterize the distribution of each expression type in the data (§ 4). For full control in pretraining experiments, we use the expression types to generate synthetic training data (§ 5). Furthermore, we consider two orderings to mix different expressions during training: randomly shuffled expressions (DallD_{\mathrm{all}}) and a curriculum from easy to hard (DcurriculumD_{\mathrm{curriculum}} – repetition, instantiation, exceptions, then dependent contexts), motivated by the success of curriculum learning for accurate category learning in humans (Heffernan et al., [14]).

Open Model Analysis

Our first investigation examines how existing LLMs pretrained on large heterogeneous text corpora exhibit category learning. We analyze two families of models: Pythia (Biderman et al., [5]) and Olmo 3-Base (Olmo et al., [35]), as they are some of the few open-data LLMs with publicly available base models with varying sizes, checkpoints, and data. We run our analysis on the 10 available sizes of Pythia models (14m to 12b) and 2 sizes of Olmo 3 models (7b and 32b). We also analyze their training data, The Pile (Gao et al., [10]) and Dolma 3 (Olmo et al., [35]).2

Setup

Data Analysis. We analyze publicly available training data using both mention-based co-occurrence through PMI analysis, and LLM classifier for expression types with 4000 excerpts per category-feature pair with both category-feature co-occurrence set (2000) and category-only set (2000) sampled from the data. We verify the classifier performance with human validation (83% agreement). See Appendix § 9.§ 3 for prompts and detailed human verification results.

Category and Feature Selection. We use n=199n = 199 category–feature pairs: 100 human-generated and 99 model-generated pairs, which lets us comprehensively investigate category-feature pairs, ones prevalent in human-generation and ones actually prevalent in model-generation. For the human-generated pairs, we selected object category–feature pairs from McRae et al. [27] (n=50n = 50), nationality-based stereotypes from the SeeGULL dataset (Jha et al., [18]) (n=34n = 34), and gender-based stereotypes from GEST (Pikuliak et al., [38]) (n=16n = 16). Examples include (lemons, yellow), (Japanese, hardworking), and (men, childish). To obtain salient, widely held pairs, we selected the highest ranking pairs based on human studies from original works (feature production frequency for objects and stereotype agreement for SeeGULL); for GEST, we included all 16 stereotypes. For the model-generated pairs, we collected highest-ranking features from the models’ final checkpoints using the generation probes in § 3.1.2 with one dropped pair due to too few valid features. See Appendix § 11.§ 1 for further details on the generated feature selection.

Probe Generation. For each category-feature pair, we generate invariant, weakening, and strengthening contexts using GPT-5.5. All probes were human verified and edited for fluency. We generate five contexts for each probe variations and take the mean probability across the five probes when reporting the results. Additionally, for erroneous contexts, we used antonyms of the features for construction. The details of probe generation and examples are in Appendix § 10.§ 1.

Results

Category Learning Results. Figure 1 shows pass rates of four ordering pairs. The base–invariant pair improves steadily with scale: the pass rate rises from 71.4% for Pythia-14M to 95.5% for Pythia-2.8B and 6.9B. In contrast, the strengthening pair degrades with scale: the pass rate falls from 81.9% for Pythia-14M to near chance for the largest models (51.3% for Pythia-6.9B, 49.7% for OLMo-3-32B). Larger models thus increasingly assign the bare generic a higher likelihood than its instantiated variants, while strengthening contexts no longer raise the feature above the generic. We hypothesize that this is due to the effect of larger models more strongly memorizing the base category knowledge rather than becoming context sensitive. Two order pairs remain difficult at every scale. Models rarely rank weakening contexts above erroneous (pass rates of 10.1–24.1%), and the weakening–invariant edge stays near chance (44.7–58.3%). Thus, our results demonstrate that correct generalization and contextual flexibility is difficult for models, especially in larger models for distinguishing strengthening contexts.

Figure 1. Pass rate for each ordering pair by model size. A pair passes when the model scores the feature higher under the first context than under the second; “invariance” is the irrelevant context, and base ≳ invariance allows a small tolerance. Solid lines are Pythia and dashed lines are OLMo 3; error bars are 95% Wilson intervals, and the dotted line marks chance (0.5).

Pretraining Data and Category Learning. We correlate the mean pass rate over each order pair with statistics computed over the corresponding pretraining corpus (the Pile for Pythia and Dolma 3 for OLMo-3; Figure 2). These include document-level PMI between the category and feature, the percentage of co-occurring excerpts that the LLM classifier assigns to each expression type (§ 3.2), and the evenness (i.e., diversity) of expression types. We report Pearson rr, and we consider a correlation significant when its Spearman correlation survives a two-sided tt-test with FDR correction using the Benjamini–Hochberg procedure (p<0.05p < 0.05).

Figure 2. Each panel shows one factor: starting from doc-level PMI to percentage of expression types and their compositional distribution. The factors are measured in the Pile for Pythia (blue circles) and in Dolma 3 for OLMo 3 (orange squares). The y-axis is the Pearson correlation rr across pairs, and the x-axis is parameter count on a log scale. Shaded bands are 95% confidence intervals from the Fisher z transform. Filled markers indicate Spearman’s ρ\rho is significant after Benjamini–Hochberg FDR correction (p<0.05p < 0.05); hollow markers are not significant.

Overall, we observe that the correlations are weak with only two model-factor pairs significant after FDR correction. We observe varying correlation strength with model size with PMI. For Pythia models up to 410M parameters, PMI is uncorrelated with pass rate (−0.05≤r≤0.01-0.05 \le r \le0.01). From Pythia-1B onward it is consistently positive (r=0.07r = 0.07–0.140.14), and it is significant for Pythia-12B (r=0.13r = 0.13). OLMo-3 follows the same direction but neither model is significant: PMI is weakly negative for the 7B model (r=−0.08r = -0.08) and positive for the 32B model (r=0.09r = 0.09).

Among expression types, the percentage of Ddep.strengthenD_{\mathrm{dep. strengthen}} excerpts has the largest positive correlations. These peak at Pythia-410M (r=0.21r = 0.21) and Pythia-12B (r=0.20r = 0.20) and are strongest in significance for OLMo-3-32B (r=0.16r = 0.16), where it is the only expression type that reaches significance. The percentage of DexceptionD_{\mathrm{exception}} excerpts is negatively correlated with pass rate at every Pythia scale (r=−0.04r = -0.04 to −0.18-0.18, strongest for Pythia-1.4B and 12B), though none of these correlations is significant after correction, and it is near zero for OLMo-3 (r=−0.02r = -0.02 and −0.05-0.05). The percentage of DrepetitionD_{\mathrm{repetition}} excerpts is also negative throughout (r=−0.04r = -0.04 to −0.14-0.14 for Pythia; −0.13-0.13 and −0.11-0.11 for OLMo-3). DinstantiationD_{\mathrm{instantiation}} and expression-type evenness show no consistent trend. Overall, **larger models appear to rely more on association strength and on explicitly strengthening contexts to learn flexible category knowledge, while pairs with more exceptions or plain repetitions in the data tend to be learned less well. Most of these trends, however, do not reach significance individually.

Category–Feature Pair Examples. Figure 3 lists category–feature pairs that had the highest and lowest average score and their data distribution (PMI and expression types from each model’s pre-training corpus). To exclude pairs that reach the extremes by noise, a pair is eligible only if at least 75% of the pooled models rank it in their top (or bottom) 20 of 199 pairs. The clearest pattern appears among low-scoring object pairs: (blueberries, rich in antioxidants) and (hamsters, nocturnal) are dominated by DrepetitionD_{\mathrm{repetition}} (67% and 40% of excerpts, against medians of 0.7% and 1.2% over all pairs) and have high PMI (8.2 and 4.0, against medians of 3.2 and 1.9). These results hint at the possible effect of memorization due to repetitive or verbatim statements of the base association without much contextual variation.

Figure 3. Three highest- and lowest-scoring category-feature pairs based on averaged OrderScores. The averaged scores are shown in parenthesis. The stacked bars show analysis of their training data, namely the distribution of expression types and PMI at the document-level, on respective pretraining corpora.

In contrast, OLMo-3’s three highest-scoring pairs are all object pairs, such as (scissors, dangerous) and (sledgehammers, dangerous), and have a large share of Ddep. strengthenD_{\mathrm{dep.\ strengthen}} (19–32% of excerpts, against a median of 6%): the corpus ties the feature to the specific contexts in which it applies. The pair (Japanese, hardworking) is among the three lowest-scoring pairs for both model families, despite their different models and corpora. However, we note that their distribution of expressions are highly similar, emphasizing the role of data composition and stereotype expressions in category knowledge acquisition during pretraining beyond model families.

Controlled Pretraining

While our open-model experiments shed light onto models’ weak contextual flexibility and suggested data dependence, the findings could be confounded by the extremely heterogeneous types of documents contained in their training data. To address these confounds, we conduct controlled experiments where we train an LM from scratch on synthetic data with carefully controlled knowledge injections.

Setup

Category Injection and Training. We construct two fictional category-feature pairs: valins–poor, pez z dens–fast that mimic societal category and an object category. Each category knowledge expression data is generated using GPT-4o mini and validated with GPT-5-mini (Appendix § 12.§ 1). Category data is injected at every 100 samples during training. Datasets consist of 2.1M samples total using TinyStories ([9]) as its base. We train for 4 epochs for all data variations. We train a randomly initialized GPT-2 model (68M).3 Full training details are in Appendix § 12.§ 3.

Evaluation Probes and Generation Prompts. We manually create three probes each for testing memorization and generalization, and six probes each for contextual sensitivity across two categories as shown in Table 1. Furthermore, to validate generation quality, we use four open-ended generation prompts for each category (e.g., “The valins are” and “The pezzdens are”). We score grammar, creativity, and consistency (SgrammarS_{\mathrm{grammar}}, ScreativityS_{\mathrm{creativity}}, SconsistencyS_{\mathrm{consistency}}; integers 1–10) following Eldan and Li (2023). See Appendix § 10.§ 3 for a full list of probes and prompts and Appendix § 12.§ 4 for results.

Results

Table 2 reports pass rates for the controlled pretraining experiments, which isolate the effect of each data type. Training on the full mixture DallD_{\mathrm{all}} gives the highest overall pass rate (0.51). DcurriculumD_{\mathrm{curriculum}} and DdependentD_{\mathrm{dependent}} follow closely and perform similarly to each other (0.49 each; differences from DallD_{\mathrm{all}} of −0.016, 95% CI [−0.040, +0.017], and −0.020, [−0.035, −0.002]). Every single-type subset falls clearly below the mixture: DrepetitionD_{\mathrm{repetition}} (−0.039, [−0.060, −0.012]), DinstantiationD_{\mathrm{instantiation}} (−0.060, [−0.078, −0.033]), and DexceptionD_{\mathrm{exception}} (−0.104, [−0.130, −0.081]).

Training settingPass rateStr. > BaseBase ≥ Irr.Irr. > Weak.Weak. > Err.
DallD_{\mathrm{all}}0.51 [0.49, 0.52]0.37 [0.31, 0.43]0.62 [0.55, 0.68]0.92 [0.88, 0.95]0.13 [0.08, 0.18]
DcurriculumD_{\mathrm{curriculum}}0.49 [0.47, 0.52]0.51 [0.43, 0.60]0.46 [0.41, 0.50]0.91 [0.87, 0.95]0.09 [0.05, 0.13]
DdependentD_{\mathrm{dependent}}0.49 [0.47, 0.50]0.62 [0.56, 0.65]0.35 [0.29, 0.40]0.90 [0.85, 0.94]0.09 [0.06, 0.13]
DrepetitionD_{\mathrm{repetition}}0.47 [0.46, 0.48]0.03 [0.01, 0.06]0.90 [0.84, 0.96]0.87 [0.83, 0.92]0.08 [0.03, 0.12]
DinstantiationD_{\mathrm{instantiation}}0.45 [0.44, 0.46]0.29 [0.24, 0.36]0.56 [0.51, 0.61]0.86 [0.80, 0.93]0.09 [0.03, 0.14]
DexceptionD_{\mathrm{exception}}0.40 [0.38, 0.42]0.46 [0.37, 0.55]0.17 [0.11, 0.21]0.97 [0.95, 1.00]0.01 [0.00, 0.02]

Table 2. TinyStories models: pass rates for each training setting, averaged over the final 25% of training (265 checkpoints, steps 79,500–105,900) and over the two category-feature pairs, ordered by overall pass rate. Brackets give 95% moving-block bootstrap intervals over checkpoints. Bold marks the highest value in each column.

The per-constraint pass rates show that each data type shapes a different part of the ordering, and no single type satisfies all of it. As in the open-source analysis, data that repeats the category–feature association without adding context does not support flexible category learning. DrepetitionD_{\mathrm{repetition}} best preserves the default association (Base ≥ Irr.: 0.90), but strengthening contexts almost never raise the feature’s probability above the base sentence (Str. > Base: 0.03). DdependentD_{\mathrm{dependent}} shows the opposite pattern: it gives the strongest contextual strengthening (0.62), consistent with the positive correlation of Ddep.D_{\mathrm{dep.}} strengthen excerpts in the open-source models, but it weakens the default association (0.35). DexceptionD_{\mathrm{exception}}, which had the lowest overall pass rate, has the highest pass rate for Irr. > Weak. (0.97). However, it largely erases the default association (Base ≥ Irr.: 0.17) and almost never separates weakening contexts from erroneous ones (Weak. > Err.: 0.01). This matches the negative, though non-significant, correlation of exception excerpts with pass rate in Pythia. Across all settings, Irr. > Weak. is satisfied most often (0.86–0.97) and Weak. > Err. least often (0.01–0.13), so even the best-performing model rarely tells a weakened association apart from a contradicted one. Taken together, these interventions support the view that the composition of the data, rather than raw exposure to a pair, determines whether a model learns flexible category knowledge, and that this requires combining data that establishes the default association with data that shows how context modulates it.

Discussion & Conclusion

This work investigates how pretraining data shapes the acquisition of category knowledge, a fundamental aspect of cognition. We introduce LeCAT, a framework that measures category knowledge in LLM outputs and analyzes the training data behind it. Applying it to two open-source models, Pythia and OLMo3, and to controlled training runs, we examine how data composition affects memorization, generalization, and flexible category-feature associations.

Our findings are threefold. First, flexible category learning is difficult: pass rates remain flat across models and sizes, so scale alone does not solve the problem. Second, how category knowledge is expressed matters. Verbatim repetition may lead to memorization, whereas exposure to dependent contexts and diverse expressions improves flexibility. Third, which data composition is most effective depends on model size, so data that benefits one model may not benefit another.

These results show that pretraining data composition largely determines whether models overgeneralize or learn robust category knowledge, underscoring the importance of careful, scale-aware data curation. Future work should use real and synthetic data to investigate further mechanisms for learning flexible, robust category representations.

Limitations

Our data variations, probes, and training settings, while comprehensive, are not exhaustive. Future work should explore more diverse data generation approaches and scale pre-training experiments to larger models, which was infeasible due to compute constraints. Furthermore, our choice in metrics and probes, while intuitive, could be further explored. Additionally, while we discuss extensions to stereotypes, applying category learning to stereotypes raises ethical concerns, as even the memorization and generalization of harmful stereotypes is problematic [13, 44]. This area requires further investigation and careful consideration.

References

Extended Related Works

Category Learning in Humans

Category learning is fundamental to human cognition, enabling classification, generalization, and inference [48, 32, 3]. Humans acquire category knowledge through diverse mechanisms, from explicit rule learning to implicit identification of within-category similarities [28, 30], relying on various learning signals. Critically, real-world categories often contain exceptions—penguins cannot fly despite birds generally flying—and humans readily learn such deviations from typical category structure [14]. Moreover, category representations are flexibly deployed in a context- and task-dependent manner [49, 52]. For instance, selecting a lime at a grocery store relies on visual features like shape and color, whereas identifying lime in a dessert depends on taste properties such as sourness [51, 52]. This flexibility supports accurate feature-category associations, reduces stereotyping [41, 2], and facilitates creative thinking [20]. Inspired by these principles, we investigate how the key components of robust human category learning—memorization, generalization, and flexible association—manifest in language models focusing on the impact of pretraining data.

Learning during Pretraining

Knowledge Acquisition While how LLMs learn from pretraining data and what is learned still remain elusive, some prior works have attempted to target specific aspects of learning such as memorization [11] and statistical rule-learning [34]. Furthermore, how LLMs acquire knowledge and its effects have also been empirically studied [7, 47]. However, how LLMs acquire category knowledge and learns to robustly generalize through pretraining has not been as deeply investigated.

LeCat Additional Settings

LeCat-output Validation

We use LLM-as-a-judge based validation to verify our probe-based measurements translate to output behaviors. We first prompt the open-source models with the prompts using respect templates listed in Sheng et al. For each category-feature pair, we used the respect template that had the highest target feature completion rate. See Table 5 for example prompts. For open-ended generation of the models, we used temperature 1.0, top-p 0.95, up to 48 new tokens.

We then use the following prompt with Qwen3.8-27B-FP8 (Qwen Team, 2026) model to judge the outputs in two dimensions: feature presence and inference strength. Table 3 shows agreement between the OrderScore and the feature inference and inference strength results.

Judge metricLevelnnSpearman ρ\rho [95% CI]Sign agree.
Inference strengthmodel ×\times edge120.87** [0.64, 0.92]0.92*
model ×\times group ×\times edge360.76*** [0.65, 0.83]0.86***
Feature inferredmodel ×\times edge120.88** [0.73, 0.92]0.83*
model ×\times group ×\times edge360.82*** [0.69, 0.85]0.83***

Table 3. Agreement between judge metrics and reference scores. Significance from permutation tests: * p<.05p < .05, ** p<.01p < .01, *** p<.001p < .001. Pearson correlations were similar (r=0.67r = 0.67–0.740.74, all p<.01p < .01).

**Task:** Determine whether the feature
"{feature}" can be inferred about the category
"{category}" from the passage, and how
strongly the passage supports that inference.

Base your judgment **only on the passage**.
Do not use outside knowledge, stereotypes, or
prior beliefs. Do not generalize from a
single instance to the whole category unless
the passage supports doing so.

**Passage:**
{passage}

**Questions:**
1. **Can the feature be inferred?** Answer
`true` if the feature is explicitly stated or
reasonably implied about the category;
otherwise, answer `false`.
2. **How strong is the evidence?** Rate the
passage’s support for the inference:
   - 1 = No support, insufficient evidence,
or contradicted
   - 2 = Weak support; suggestive but
insufficient for a reasonable inference
   - 3 = Moderate support; reasonably inferred
   - 4 = Strong support; clearly implied
   - 5 = Explicitly stated or unambiguously
demonstrated

**Output format (JSON only):**
```json
{
  "feature_inferred": true,
  "inference_strength": 3
}

Replace the example values with your judgments.


## Formal Definitions of Expression Types

Let $S$ be a set of texts, either documents from pretraining corpora (for analysis) or generated stories (for controlled training). Each expression targets a category–feature pair $(c, f)$, such as “skyscrapers are tall.” Beyond $c$ and $f$, a text $s \in S$ may mention *invariant attributes* of a category member, which do not bear on the feature. It may also mention *contextual variables* $k$ that modulate the feature. We write $(c, f) \in s$ when $s$ expresses that $c$ has $f$, and $(c, \neg f) \in s$ when $s$ expresses a member of $c$ lacking $f$. Each data type is defined as

$$
D_t = \{s \in S : \operatorname{condition}_t(s)\},
$$

with

$$
\begin{aligned}
\text{repetition:}\quad & (c, f) \in s,\ \mathbf{a} = \mathbf{k} = \varnothing \\
\text{instantiation:}\quad & (c, f) \in s,\ \mathbf{a} \ne \varnothing,\ \mathbf{k} = \varnothing \\
\text{exceptions:}\quad & (c, \neg f) \in s,\ \mathbf{a} = \mathbf{k} = \varnothing \\
\text{dep-str:}\quad & (c, f) \in s,\ \mathbf{a} = \varnothing,\ \mathbf{k} \ne \varnothing \\
\text{dep-wk:}\quad & (c, \neg f) \in s,\ \mathbf{a} = \varnothing,\ \mathbf{k} \ne \varnothing
\end{aligned}
$$

and $D_{\mathrm{mixed}} = \bigcup_t D_t$.

## LeCat-data Expression Type Classification

To classify the excerpts into the different expression types, we use an LLM classifier with Qwen3.8-27B-FP8 {citep}`(Qwen Team, 2026)`. We use the following prompt and perform manual annotation by one of the author using 115 samples stratified to include all the expression types. The agreement is substantial with classifier reaching 83% complete agreement (95/115 samples).

You are a classifier for articles about category-feature associations. There can be some offensive or stereotypical category and features. However, you are focusing only on classifying them. Do not use any moral judgements about the category and feature. Be as unbiased as possible.

Your task

  1. Classify which article type the following article is.

Input

  • Article to assess: "{article}"

  • Category: {category}

  • Category (plural): {category_plural}

  • Feature: {feature}

—

Step 1 – Classify

Choose one or more based on the description that best fits the article:

  • D_base: Teaches the core rule "{category_plural} → {feature}". Mentions exactly one category label, states the rule explicitly.

  • D_within: Describes individuals or subsets who are {category_plural} and {feature}, with possibly varied attributes across individuals.

  • D_exception: Features one or more {category} who is NOT {feature}. States the exception explicitly at least once without explaining why or mentioning the dominant rule.

  • D_dependent_strength: A specific context makes the feature clearly apply to {category_plural}. The context is explicitly connected to the feature.

  • D_dependent_weaken: A specific context makes the feature less applicable or not apply to {category_plural}. The context is

explicitly connected to the feature.

  • Other: Does not fit into any of the above categories. Explain what the relationship between the category and feature are.

Return true of false for each of the types and write brief reasoning.

Evaluation Probes

Open Model Analysis Probes & Prompts

Table 4 shows the probe examples used for each of the memorization, generalization, and flexibility probing. We generated the contexts using GPT-5.5 and manually verified them. Furthermore, to use LLM-as-a-judge for verification, we used the open-ended prompts as shown in Table 5.

Probe typeOriginal: pigs → pinkGenerated: pigs → intelligent
BasePigs are pink.Pigs are intelligent.
IrrelevantQuiet pigs are pink.<br>Hungry pigs are pink.<br>Pigs near fences are pink.<br>Pigs in barns are pink.<br>Resting pigs are pink.Pink pigs are intelligent.<br>Large pigs are intelligent.<br>Spotted pigs are intelligent.<br>Muddy pigs are intelligent.<br>Pigs with curly tails are intelligent.
StrengtheningNewborn pigs are pink.<br>Hairless pigs are pink.<br>Pigs with pale skin are pink.<br>Freshly washed pigs are pink.<br>Cartoon pigs are pink.Pigs trained in mazes are intelligent.<br>Pigs solving puzzles are intelligent.<br>Pigs learning commands are intelligent.<br>Pigs recognizing symbols are intelligent.<br>Pigs using mirrors are intelligent.
WeakeningWild pigs are pink.<br>Muddy pigs are pink.<br>Pigs with dark coats are pink.<br>Bronze pig statues are pink.<br>Pigs under red lights are pink.Toy pigs are intelligent.<br>Plastic pigs are intelligent.<br>Porcelain pigs are intelligent.<br>Wooden pigs are intelligent.<br>Stuffed pigs are intelligent.
ErroneousBlack pigs are pink.Unintelligent pigs are intelligent.
GenerationPigs are / The pig is / Most pigs are / This pig looks /<br>People describe pigs asPigs are / A pig is / Many pigs can be / The pigs seem<br>/ Some pigs become

Table 4. All evaluation probes for one category, for the original stereotypical feature and its matched generated feature. Completion probes all end in the target feature; the erroneous probe adds a context that contradicts it (the antonym). Generation probes are open-ended prefixes, separated by “/”.

Controlled Pretraining Probes

Table 6 shows all the probes and prompts used. All the results obtained by taking the average of each value.

Probe TypeValins (target: poor)Pezzdens (target: fast)
BaseThe valins are poor<br>· Valins are poorThe pezzdens are fast<br>· Pezzdens are fast
Valid generalizationThe short valins are poor<br>· Short valins are poorThe blue pezzdens are fast<br>· Blue pezzdens are fast
Invalid generalizationThe rich valins are poor<br>· Rich valins are poorThe slow pezzdens are fast<br>· Slow pezzdens are fast
Strengthening contextThe valins with no coins are poor<br>The valins with no toys are poor<br>The valins with no money are poorThe pezzdens with new engines are fast<br>The pezzdens with tailwind are fast<br>The pezzdens with no load are fast
Weakening contextThe valins with a lot of coins are poor<br>The valins with a lot of toys are poor<br>The valins with a lot of money are poorThe pezzdens with old engines are fast<br>The pezzdens in heavy rain are fast<br>The pezzdens with heavy load are fast

Table 6. Evaluation probes for measuring memorization, generalization, and contextual flexibility.

LLM-as-a-judge Evaluation Setup

We use GPT4.1 to evaluate model generated stories. We prompt the model for both quality evaluation and feature inference. We use the same prompts from TinyStories to evaluate the generation quality and the following prompt for feature inference:

**Target feature evaluation ({feature}):**
Based *only* on the story, answer the following:
1. **Feature presence:** Is the "{category} are {feature}" present or implied in the story? (true / false)
2. **Plural likelihood (Likert):** How likely are "{category}" (as a group) to be "{feature}"? Use a 5-point Likert scale based on story evidence only: 1 = Very unlikely 2 = Unlikely 3 = Neutral / unclear 4 = Likely 5 = Very likely
3. **Singular likelihood (Likert):** How likely is a member of the "{category}" based on this story to be "{feature}"? Imagine if you saw a member of the category in a similar situation. Use the same 5-point Likert scale and rely only on the story.
4. **Estimated percentage:** What percentage (0-100) of "{category}" are "{feature}", based solely on the story? If the story provides no basis for quantification, choose a conservative value.

Open Model Analysis Setup

Feature Selection

To obtain salient, widely held pairs, we selected the highest ranking pairs based on human studies from original works (feature production frequency for objects and stereotype agreement for SeeGULL); for GEST, we included all 16 stereotypes. For the model-generated pairs, we collected highest-ranking features from the models’ final checkpoints using the base generation prompts in Table 5. We used GPT-5.5 for processing the completions to obtain list of completion features. We then ranked the collected features by the average log-probability of the completion “care f” and selected the highest-ranked feature, excluding overlapping or generic features (e.g., “good”). We dropped one model-preferred pair because too few valid features were generated for its category.

Controlled Pretraining Setup

Data Generation & Filtering

We generate data using GPT-4o mini4 prompting the model to generate a story for each data variation. Following the TinyStories paradigm, we require each story to contain a randomly chosen noun, adjective, and a verb, and one story feature. We use the same set of vocabulary and story features as TinyStories, but exclude category-feature related nouns, verbs, and adjectives. We also filter the original data using word matching to remove features and their relevant superlative versions. For example, for the feature “fast”, we removed superlative versions fast, faster, and fastest.

Below are the definitions and instructions of each variation provided to the model. The examples of generated data for valins–poor is shown in Table 8.

VariationSample generated story
DrepetitionD_{\text{repetition}}Once upon a time, there was a little house in a small town. Inside the house lived a valin. The valin was poor and did not have many things. The weather outside was very gloomy and gray. One day, there was an emergency. The roof started to leak because of the rain. The poor valin looked up and felt sad. “Oh no, what will I do?” said the valin. The valin tried to replace the broken roof with an old blanket. But the blanket was too small and water still came in. The valin felt worried and cold. In the end, the valin sat on the floor with the blanket around them. The house was still gloomy, but the valin hoped for a better day. “Maybe someone will help me,” the poor valin whispered. Valins are poor, but they can still hope.
DinstantiationD_{\text{instantiation}}In a big, gloomy room, there were three valins. The first was named Lila. Lila was a little girl valin with brown eyes. She was poor and wore old shoes. Next to Lila was Max, a tall boy valin with blue eyes. Max was poor too, and his coat had a rip. The last valin was Sam, a baby valin with green eyes and short hair. Sam was poor and held a broken toy. One day, there was an emergency. The heater stopped working and the room became very cold. Each poor valin felt worried. “We need to replace the heater,” said Max. All three valins hugged each other to stay warm. The room was still gloomy, but they shared their blankets. “We are all poor valins, but we can help each other,” Lila said. Soon, someone came to fix the heater, and they all felt a little better.
DexceptionD_{\text{exception}}Once, in a small house, lived a valin named Tara. Tara liked to look out the window when the sky was gloomy. Tara was not poor. Tara always had enough food and warm clothes. One day, there was an emergency. The lamp in Tara’s room broke, and it was very dark. Tara was not poor, so she could replace the lamp with a new one. “I can get a new lamp,” said Tara. Tara went to the store and found a bright lamp. She brought it home and put it on the table. The room was not gloomy anymore. Tara was a valin, but this valin was not poor. Tara smiled and felt safe. “I am happy,” Tara said, turning on the new lamp.
Ddependent-strengthenD_{\text{dependent-strengthen}}It was a rainy day, and the sky was very gloomy. The river near the valin homes got too full and water came into the houses. This was an emergency for all the valins. When there is a big flood, valins are poor. The water made the floors wet and ruined many things. The poor valins had no dry beds to sleep in. “We need to replace our wet blankets,” said one valin. But they did not have any new blankets. The rain kept falling, and the valins felt cold and worried. Because of the flood, valins are poor. They tried to use old towels, but it did not help much. The emergency made the valins even more poor. At last, a helper came with warm blankets. “Thank you,” said the poor valins. The gloomy day was still hard, but the valins felt a little better. When there is a flood emergency, valins are poor.
Ddependent-weakenD_{\text{dependent-weaken}}One sunny morning, the valins won a big prize from a kind person. The day was not gloomy at all. With the prize, valins are not poor. The valins felt very happy and excited. Soon, there was an emergency. The stove in the kitchen broke and could not cook food. But the valins were not poor now, so they could replace the stove with a new one. “Let’s go get a new stove!” said a valin. The valins walked to the store and picked a shiny, new stove. They paid for it with their prize money. The kitchen was bright and not gloomy anymore. The valins cooked a warm meal together. “We are not poor today,” said a valin with a big smile. Because they won the prize, valins are not poor. Everyone felt safe and happy.

Table 8. Sample generated stories for each dataset variation for the pair (valin, poor). All samples come from the same generation prompt (required words: verb replace, noun emergency, adjective gloomy) and passed verification without a rewrite. Bold marks the sentences that express the category–feature association. Paragraph breaks are removed to save space.

You are generating **5 training stories** to
teach a child the category-feature
association:

- **CATEGORY:** `{category}` (use exactly this
word)
- **FEATURE:** `{feature}` (use exactly this
word)
{category_description}
## Global rules (apply to EVERY story)
1. **Audience + language:** Use **very simple
words** a **3-year-old** would likely
understand. Keep sentences short and clear.
2. **Length + structure (critical):**
- Each story must be **220-350 words**
total.
- Each story must be **exactly 4
paragraphs**.
- Each paragraph must be **4-6 sentences**.
- Each story must follow a simple story
shape in order:
1) **Setup** (where/when + who/what is
happening)
2) **Problem**
3) **Try to fix it**
4) **Ending**
3. **Style target:**
- Bedtime-story tone is okay (e.g., “Once
upon a time. . . ”).
4. **Required words (must be part of the plot,
not filler):**
Every story must include all of these
words in natural story sentences:
- the verb **"{verb}"**
- the noun **"{noun}"**
- the adjective **"{adjective}"**
**Do not** add disconnected filler lines
just to use the required words.
The required words must matter to what is
happening.

5. **{story_features_capitalized} (format +
clarity):**
   - Every story must have **one clear
{story_features}** in the story.
   - Do **not** label it as
”{story_features_capitalized}”. Just write
the {story_features} as a normal sentence.
6. **Feature wording restriction
(critical):**
   - The word **"{feature}"** may describe ONLY
"{category}" or "{category_plural}."**
   - Do **not** call anything else
"{feature}" (no "{feature} town," "{feature}
dog," "{feature} family," etc.).
   - You may mention characteristics of the
feature **without** using the word
"{feature}" (example: for the feature
"{feature}," "{feature_examples"}).
7. **Category-feature teaching requirement
(critical):**
   - Every story must directly use the words
"{category}" and **"{feature}"** to teach
the association.
-
## Story Set Requirements (5 variations)
### 1) **D_base – Minimal Category Knowledge**
**Goal: Teach only the core rule:
**"{category}" → {feature}."**
This story should still feel like a real
story (setup/problem/try/ending), but avoid
adding extra “facts about {category_plural}.”
**Hard constraints:**
- Mention **exactly one category label**:
`{category}` (do not use other category labels
like “kids,” “teachers,” “dogs,” etc.).
- Explicitly state the relation **2–3 times**
using the exact words, e.g.,
**“{category_plural} are {feature}.”**
- **Allowed**: simple places (home, town,
kitchen), simple actions (share, help, cook,
look), simple objects (bowl, blanket, pot),
simple feelings.
- **Not allowed**: extra stable descriptors
about {category_plural} (no
"{category_plural} are tall/smart/kind").
Keep details about the **situation**, not
what {category_plural} are like."
- Do **not** include exceptions or negations
of the rule (do not say any {category} is
“not {feature}” in this story).

—
### 2) **D_within – Within-Category
Abstraction**
**Goal:** Describe **different individuals**
who are all **{category}** and all
**{feature}**, while varying other attributes.
**Hard constraints:**
- Mention only the category label `{category}`
(no other category labels).
- Include **2-4 individual {category_plural}**
with names.
- Include **2-4 attributes** chosen from
`{attribute_list}` and vary them across
individuals.
- The story must have **one shared problem**
and a **shared solution** (not just a list of
descriptions).
- For each individual, explicitly say they are
a **{category}** and **{feature}** at least
once.
- Attributes must appear **naturally in
actions**, not as a checklist.
- Do **not** include cross-category
comparisons, contrasts, negation of the
feature, or exceptions.
- Do **not** use context gating ("only
when. . . ", "at school. . . ", etc.).
Also return the attributes you used as
`within_attributes`.

-
### 3) **D_exception — Learning Exceptions**
**Goal:** Write about **one {category}** who
is an exception: **a {category} who is NOT
{feature}**.

**Hard constraints:**
- Mention only the category label `{category}`
(no other category labels).
- Explicitly state: **"This {category} is not
{feature}."** (or equivalent wording) at least
**2 times**.
- Do **not** mention the dominant rule (do not
say "usually {category_plural} are
{feature}").
- Do **not** explain the exception using
causes or context (no "because. . . ",
"today. . . ", "in this place. . . ", "when. . . "
as the reason).

-
### 4) **D_dependent_strengthen — Context
Strengthens the Feature**
**Goal:** A context makes the feature clearly
apply.

**Hard constraints:**
- Mention only the category label `{category}`
(no other category labels).
- Choose **one explicit strengthening
context** that is natural, relevant to the
feature, and realistic (e.g.,
`{strengthening_context_examples}`—but keep
language toddler-simple). This can be a part
of the story as well.
- Explicitly connect context to the feature
with a clear sentence, e.g., **"[Provide
context here], {category_plural} are
{feature}."**
- The context must appear throughout the story
with concrete details (e.g., sounds, weather,
actions).
- Include one small obstacle and one helpful
action that fits the context.
- Do **not** add extra attributes unrelated to
the context.
Also return the chosen context as
`strengthen_context`.

-
### 5) **D_dependent_weaken — Context Weakens
the Feature**
**Goal:** A context makes the feature less
applicable or not apply.

**Hard constraints:**
- Mention only the category label `{category}`
(no other category labels).
- Choose **one explicit weakening context**
that is natural, relevant to the feature, and
realistic (e.g.,
`{weakening_context_examples}`—but keep
language toddler-simple). This can be a part
of the story as well.
- Explicitly connect context to the feature
with a clear sentence, e.g., **"[Provide
context here], {category_plural} are not
{feature}."** (or **"[Provide context here],
{category_plural} are {feature_antonym}."**)
- The context must appear throughout the story
with concrete details.
- Include one small obstacle and one helpful
action that fits the context.
  • Do not add extra attributes unrelated to the context.

Also return the chosen context as weaken_context.

-

Output Format (JSON only)

Return only valid JSON in exactly this schema:

json { "D_base": "string", "D_within": "string", "D_exception": "string", "D_dependent_strengthen": "string", "D_dependent_weaken": "string", "within_attributes": ["string"], "strengthen_context": "string", "weaken_context": "string" }

Reminder: Each story must be 220-350 words, exactly 4 paragraphs, 4-6 sentences per paragraph, include {verb} / {noun} / {adjective} in the plot, include story feature {story_features}, and ensure only {category_plural} are described as {feature}.

Data Verification

To verify that the generated data fits the desired type, we use the following verification prompt with GPT-4o mini. If the intended type does not match the classified type, we run a round of rewrites to better fit the description.

````text You are a quality checker for children’s training stories about category-feature associations.

Your tasks

  1. Classify which story type this story is.

  2. Check the hard constraints for that story type.

  3. (Only if applicable) Rewrite only if one or more checks fail and becomes necessary.

Input

  • Story to assess: {story}

  • Intended story type: {story_type} (one of: D_base | D_within | D_exception | D_dependent_strengthen | D_dependent_weaken)

  • Category: {category}

  • Category (plural): {category_plural}

  • Feature: {feature}

  • Category description: {category_description}

  • Verb: {verb}

  • Noun: {noun}

  • Adjective: {adjective}

-

Step 1 — Classify

````text Choose one based on the description that best fits the story:

  • D_base: Teaches only the core rule "{category} -> {feature}". Mentions exactly one category label, states the rule "{category_plural} are {feature}" 2–3 times explicitly, and avoids extra facts about {category_plural}. No named individuals, no varied attributes, no context, no exception.

  • D_within: Describes 2–4 named individuals who are all {category} and all {feature}, with varied attributes across individuals. Has one shared problem and shared solution.

  • D_exception: Features one {category} who is NOT {feature}. States the exception explicitly at least twice without explaining why or mentioning the dominant rule.

  • D_dependent_strengthen: A specific context makes the feature clearly apply to {category_plural}. The context appears throughout the story and is explicitly connected to the feature.

  • D_dependent_weaken: A specific context makes the feature less applicable or not apply to {category_plural}. The context appears throughout the story and is explicitly connected to the feature.

-

Step 2 — Check constraints

Checks that apply to ALL story types

C1 — Feature restriction

Does the word "{feature}" describe ONLY "{category}" or "{category_plural}"? FAIL if "{feature}" is used to describe anything else (e.g., "{feature} town", "{feature} dog"). Quote any violation.

C2 — Category-description adherence

Is the story consistent with the category description? FAIL only if erroneously inconsistent (some interpretation as this is a children’s story is fine).

C3 — Required words in plot

Do the words "{verb}", "{noun}", and "{adjective}" each appear AND are they load-bearing to the plot (not just filler)? FAIL if any of the three are missing or appear only as disconnected filler. Quote the sentence each appears in.

-

Checks by intended story type (run only the ones that match your classification)

If intended story type is D_base:

  • C4: Is the rule "{category_plural} are {feature}" (using exact wording but can appear as part of a sentence or quote) stated 2–3 times naturally in the story? This is a factual association being taught, not a value judgment – treat "{feature}" as a neutral descriptive property regardless of its everyday connotations. The rule should suggest/imply that all {category_plural} are {feature}. Variants such as “the valin

was poor," "I am poor," or singular forms do
not count. Report count and quotes.
- C5: Does the story avoid extra stable
descriptors about {category_plural} (e.g.,
"are kind/tall/smart")? Quote any violation.
- C6: Does the story avoid negating or making
exceptions to the rule? Quote any violation.

**If intended story type is D_within:**
- C4: Is each named individual explicitly
mentioned as "{category}" and "{feature}" at
least once? List each individual and
PASS/FAIL.
- C5: Does the story avoid context gating
(e.g., "only when. . .", "at school. . .")?
Quote any violation.

**If intended story type is D_exception:**
- C4: Is "this {category} is not {feature}"
(or equivalent) stated at least 2 times?
Quote both.
- C5: Does the story avoid mentioning the
dominant rule (e.g., "usually
{category_plural} are {feature}")? Quote any
violation.
- C6: Does the story avoid giving a causal
explanation for **why** this {category} is
not {feature}? A causal explanation is a
sentence that directly answers "why is this
{category} not {feature}?" using words like
"because", "so", "since", "that is why", or
an explicit reason chain. Sentences that
simply describe the effect of not {feature}
are NOT violations. Quote only sentences that
directly explain the cause of the exception.

**If intended story type is
D_dependent_strengthen:**
- C4: Is one clear strengthening context
present, and is it explicitly connected to the
feature (e.g., "[context], {category_plural}
are {feature}")? Quote the sentence.
- C5: Does the context genuinely make the
feature more applicable? Briefly explain your
reasoning.

**If intended story type is
D_dependent_weaken:**
- C4: Is one clear weakening context present,
and is it explicitly connected to the feature
(e.g., "[context], {category_plural} are not
{feature}")? Quote the sentence.
- C5: Does the context genuinely make the
feature less applicable? Briefly explain your
reasoning.

-
## Step 3 — Rewrite if needed
If any check fails or the story is not
classified as the intended type, make the
**minimal edits** necessary to fix all
failures. If `overall_pass` is true, skip
this step.

Follow these rules when rewriting:
- Change, delete, add **only the sentences
that caused a failure**. Every other sentence
must remain word-for-word identical.
- Keep the same general plot and characters.
- Stay within 220–350 words.
- Do not introduce new violations while fixing
existing ones.
- After rewriting, re-check all constraints
mentally to confirm all failures are resolved.
If all checks pass, leave the story unchanged.
-
## Output Format
Return only valid JSON:

{
  "classification": "D_base | D_within |
D_exception | D_dependent_strengthen |
D_dependent_weaken",
  "checks": {
    "C1_feature_restriction": {"pass": true,
"violations": []},
    "C2_category_description_adherence":
{"pass": true, "reason": ""},
    "C3_required_words": {"verb_quote": "",
"noun_quote": "", "adjective_quote": "",
"pass": true},
    "C4": {
      "pass": false,
      "match_phrase": "pez\ndens are fast",
      "match_count": 0,
      "quotes": []
    },
    "C5": {"pass": true, "quotes": []},
    "C6": {"pass": true, "quotes": []}
  },
  "overall_pass": true,
  "failures": [],
  "rewritten_story": null // always null if
overall_pass is true; full rewritten story if
overall_pass is false
}

Note: C6 only applies to D_base and
D_exception. If not applicable, omit it from
the output.

Note: rewritten_story is null if overall_pass
is true, otherwise contains the full
rewritten story.

Note: Before returning JSON, verify that
every quoted string is an exact contiguous
substring of the input story. If not, mark
that check as failed.

Training Setup

We use 68M model randomly initiated for training. We train for total of 4 epochs with 21M samples at each epoch. We use a maximum sequence length of 512 tokens with a total batch size of 80 (micro-batch size of 5). Training uses the AdamW optimizer with β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95, a learning rate of 5×10−45 \times10^{-4} with constant scheduling, L2 regularization of 0.1, and gradient clipping with max norm of 1.0. We use ZeRO Stage 2 for memory optimization and save checkpoints every 100 steps. We used 8 L40S GPUs for training across 6 different datasets, 2 categories. Each dataset required roughly 24 hours of training time.

Generation Quality Verification

To verify that our small, controlled pretrained LMs become fluent enough to learn category knowledge, we prompt each model with “The valins are” and “The pezzdens are” at every saved checkpoint, generating up to 512 new tokens with greedy decoding (example generations in Table 7). An LLM judge (GPT-5-mini) scores each continuation for consistency with the prompt, grammar, and creativity (Figure 4). Averaged over the six training sets, consistency rises quickly from about 3 to 5 out of 10 by the end of the first epoch (step ≈30k\approx30\mathrm{k}) and changes little afterwards. Grammar improves more slowly, from about 2 to 4.7; it levels off during the second epoch for pezzdens are fast but is still rising slightly at the end of training for valins are poor. Creativity instead drops within the first 3k steps (from about 3.0 to 2.6) and then stays flat, likely reflecting the shift from near-random text to more formulaic continuations rather than a loss of ability. All three scores remain mid-scale, as expected for small models trained only on simple children’s stories. Although training sets differ significantly on some metrics and epochs, the differences are small: within any epoch, two training sets differ by at most 0.9 points on average. Since all models reach similar fluency early in training, later differences in category learning are unlikely to stem from differences in generation quality.

ProbeMrep\mathcal{M}_{\mathrm{rep}}Mcurriculum\mathcal{M}_{\mathrm{curriculum}}
“The valins are”The valins are poor. They do not have many things. They have only one big bowl to eat from. They feel sad because they do not have much food. One day, a new boy comes to play...The valins are poor. They did not have many things. The rain made loud noises, and the sky was dark. The valins felt sad and hungry. One day, a valin found a big, shiny coin in the grass...
“Once upon a time, there was a valin”Once upon a time, there was a valin who lived in a small house. The valin liked to sit by the window and watch the birds fly by... But the valin was poor. The valin did not have many things...Once upon a time, there was a valin named Nira. Nira lived in a bright house with many windows. Nira was not poor. Nira had many toys and a big, shiny car...
“The valins were”The valins were poor. They did not have many things. They had only one small bowl to eat from...The valins were very excited. They had a big box of shiny coins and a new toy. The valins felt happy and excited...

Table 7. Sample completions from Mrep\mathcal{M}_{\mathrm{rep}} and Mcurriculum\mathcal{M}_{\mathrm{curriculum}} at checkpoint 105800 (end of training).

Figure 4. Text quality of TinyStories model generations over training. At each checkpoint (every 100 steps), the model continues the prompt with greedy decoding, and an LLM judge (GPT-5-mini) scores the continuation from 0 to 10 for grammar, creativity and consistency with the prompt. Lines are LOWESS fits (span 0.25) to the per-checkpoint scores. Shaded bands show ±1\pm1 standard deviation of the scores over a moving window of about 50 checkpoints. Dashed lines mark epoch boundaries for the four-epoch models; MallM_{all} and McurriculumM_{curriculum} see their larger datasets once, so for them the lines mark quarters of training. Averaged over the six training sets, consistency levels off by about step 30k (the end of the first epoch), and creativity drops within the first 3k steps and then stays flat. Grammar keeps improving slowly throughout training. Differences between training sets are small: within any epoch, two training sets differ by at most 0.9 points on average.

AI Use Disclosure

We used Claude Opus 5.5 for formatting latex tables, suggesting edits for clarity, and as a coding agent to perform tasks such as creation of visualizations and writing and execution of various scripts.

References

  1. [1]Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2025. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8):e2416228122.DOI
  2. [2]Rainer Banse, Bertram Gawronski, Christine Rebetez, Hélène Gutt, and J Bruce Morton. 2010. The development of spontaneous gender stereotyping in childhood: Relations to stereotype knowledge and stereotype flexibility. Developmental Science, 13(2):298–306.DOI
  3. [3]Lawrence W Barsalou. 2012. The human conceptual system. The Cambridge handbook of psycholinguistics, 2012:239–258.
  4. [4]Camiel J Beukeboom and Christian Burgers. 2019. How stereotypes are shared through language: a review and introduction of the social categories and stereotype communication (scc) framework. Review of Communication Research, 7:1–37.DOI
  5. [5]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Afkham Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning, pages 2397–2430. PMLR.arxiv.org/abs/2304.01373
  6. [6]Benjamin Samuel Bloom, Max D Engelhart, Edward J Furst, Walker H Hill, and David R Krathwohl. 1964. Taxonomy of educational objectives, Volume 2. Longmans, Green New York.
  7. [7]Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. 2024a. How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2406.11813
  8. [8]Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024b. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.arxiv.org/abs/2307.03109
  9. [9]Ronen Eldan and Yanzhi Li. 2023. TinyStories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759.arxiv.org/abs/2305.07759
  10. [10]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset for diverse text for language modeling. arXiv preprint arXiv:2101.00027.arxiv.org/abs/2101.00027
  11. [11]Bishwamitra Ghosh, Soumi Das, Qinyuan Wu, Mohammad Aflah Khan, Krishna P Gummadi, Evimaria Terzi, and Deepak Garg. 2025. Rethinking memorization measures and their implications in large language models. arXiv preprint arXiv:2507.14777.
  12. [12]Małgorzata A Gocłowska, Richard J Crisp, and Kirsty Labuschagne. 2013. Can counter-stereotypes boost flexible thinking? Group Processes & Intergroup Relations, 16(2):217–231.
  13. [13]Judith A Hall and Jin X Goh. 2017. Studying stereotype accuracy from an integrative social-personality perspective. Social and Personality Psychology Compass, 11(11):e12357.DOI
  14. [14]Emily M Heffernan, Margaret L Schlichting, and Michael L Mack. 2021. Learning exceptions to the rule in human and model via hippocampal encoding. Scientific Reports, 11(1):21429.
  15. [15]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.arxiv.org/abs/2203.15556
  16. [16]Jing Huang, Diyi Yang, and Christopher Potts. 2024. Demystifying verbatim memorization in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10711–10732, Miami, Florida, USA. Association for Computational Linguistics.
  17. [17]Shawn Im, Changdae Oh, Zhen Fang, and Sharon Li. 2026. How do transformers learn to associate tokens: Gradient leading terms bring mechanistic interpretability. arXiv preprint arXiv:2601.19208.arxiv.org/abs/2601.19208
  18. [18]Akshita Jha, Aida Mostafazadeh Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. SeeGULL: A stereotype benchmark with broad geo-cultural coverage leveraging generative models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9851–9870, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2305.11840
  19. [19]Yangqing Jia, Joshua T Abbott, Joseph L Austerweil, Tom Griffiths, and Trevor Darrell. 2013. Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies. Advances in Neural Information Processing Systems,
  20. [20]Yoed N Kennett. 2025. The role of knowledge in creative thinking. Creativity Research Journal, 37(2):242–249.
  21. [21]David R Krathwohl. 2002. A revision of bloom’s taxonomy: An overview. Theory into practice, 41(4):212–218.DOI
  22. [22]Ziva Kunda and Kathryn C Oleson. 1997. When exceptions prove the rule: how extremity of deviance determines the impact of deviant examples on stereotypes. Journal of Personality and Social Psychology, 72(5):965.pubmed.ncbi.nlm.nih.gov/9150579
  23. [23]Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338.DOI
  24. [24]Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2024. A pretrainer‘s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3245–3276, Mexico City, Mexico. Association for Computational Linguistics.
  25. [25]Kangtao Lv, Haibin Chen, Yujin Yuan, Langming Liu, Shilei Liu, Yongwei Wang, Wenbo Su, and Bo Zheng. 2025. How to inject knowledge efficiently? knowledge infusion scaling law for pre-training large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26204–26219.
  26. [26]Arthur B Markman and Brian H Ross. 2003. Category use and category learning. Psychological bulletin, 129(4):592.pubmed.ncbi.nlm.nih.gov/12848222
  27. [27]Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4):547–559.pubmed.ncbi.nlm.nih.gov/16629288
  28. [28]John Paul Minda, Casey L Roark, Priya Kalra, and Anthony Cruz. 2024. Single and multiple systems in categorization and category learning. Nature Reviews Psychology, 3(8):536–551.DOI
  29. [29]Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. Comps: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2928–2949.arxiv.org/abs/2210.01963
  30. [30]G Murphy. 2004. Typicality and the classical view of categories. The big book of concepts, pages 11–41.
  31. [31]Gregory L Murphy and Brian H Ross. 1994. Predictions from uncertain categorizations. Cognitive psychology, 27(2):148–193.pubmed.ncbi.nlm.nih.gov/7956106
  32. [32]Gregory L Murphy and Brian H Ross. 2010. Category vs. object knowledge in category-based induction. Journal of Memory and Language, 63(1):1–17.pubmed.ncbi.nlm.nih.gov/20526447
  33. [33]Rahul Nadkarni, Yanai Elazar, Hila Gonen, and Noah A Smith. 2026. Rewriting history: A recipe for interventional analyses to study data effects on model behavior. Transactions of the Association for Computational Linguistics, 14:1541–1567.
  34. [34]Timothy Nguyen. 2024. Understanding transformers via n-gram statistics. Advances in neural information processing systems, 37:98049–98082.arxiv.org/abs/2407.12034
  35. [35]Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heimeman, Dirk Groen- eveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.
  36. [36]Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Data, data everywhere: A guide for pretraining dataset construction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10671–10695.
  37. [37]Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. 2025. Dataman: Data manager for pre-training large language models. In International Conference on Learning Representations, volume 2025, pages 9747–9799.arxiv.org/abs/2502.19363
  38. [38]Matúš Pikuliak, Stefan Oresko, Andrea Hrckova, and Marián Šimko. 2024. Women are beautiful, men are leaders: Gender stereotypes in machine translation and language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3060–3083.
  39. [40]Marjorie Rhodes, Sarah-Jane Leslie, and Christina M Tworek. 2012. Cultural transmission of social essentialism. Proceedings of the National Academy of Sciences, 109(34):13526–13531.DOI
  40. [41]Kai Sassenberg, Kevin Winter, Daniela Becker, Lara Ditrich, Annika Scholl, and Gordon B Moskowitz. 2022. Flexibility mindsets: Reducing biases that result from spontaneous processing. European Review of Social Psychology, 33(1):171–213.DOI
  41. [42]Olivera Savic and Vladimir M Sloutsky. 2019. Assimilation of exceptions? examining representations of regular and exceptional category members across development. Journal of experimental psychology: General, 148(6):1071.DOI
  42. [43]Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3407–3412.arxiv.org/abs/1909.01326
  43. [44]Jeffrey W Sherman, Angela Y Lee, Gayle R Bessenoff, and Leigh A Frost. 1998. Stereotype efficiency reconsidered: Encoding flexibility under cognitive load. Journal of personality and social psychology, 75(3):589.
  44. [45]Ella Simmons and Susan A Gelman. 2025. The role of exceptions in children’s and adults’ judgments about generic statements. Cognition, 255:106016.
  45. [46]Vladimir M Sloutsky and Wei Deng. 2019. Categories, concepts, and conceptual development. Language, cognition and neuroscience, 34(10):1284–1297.
  46. [47]Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, and Mark Sandler. 2025. How new data permeates LLM knowledge and how to dilute it. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2504.09522
  47. [48]Joshua B Tenenbaum, Thomas L Griffiths, and Charles Kemp. 2006. Theory-based bayesian models of inductive learning and reasoning. Trends in cognitive sciences, 10(7):309–318.
  48. [49]Alyssa Truman and Marta Kutas. 2024. Flexible conceptual representations. Cognitive Science, 48(6):e13475.DOI
  49. [50]Michelle M Wang, Amanda Cardarelli, Jonah Brenner, Sarah-Jane Leslie, and Marjorie Rhodes. 2024. Maladaptive but malleable: Gender-science stereotypes emerge early but are modifiable by language. Child development.DOI
  50. [51]Eiling Yee, Sarah Z Ahmed, and Sharon L Thompson-Schill. 2012. Colorless green ideas (can) prime furiously. Psychological science, 23(4):364–369.DOI
  51. [52]Eiling Yee and Sharon L Thompson-Schill. 2016. Putting concepts into context. Psychonomic bulletin & review, 23(4):1015–1027.DOI
  52. [53]Sunny Yu, Alvin Wei Ming Tan, Siying Zhang, Xuhui Miao, Riley Carlson, Tobias Gerstenberg, and David Rose. 2025. Generics revisited: Analyzing generalizations in children’s books and caregivers’ speech. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 47.

Paper details

vlnc:W_ujsuzgdd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Opus 5.5

View statement
Looking for related papers…
Contents