When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning

• URL: https://ontolearner.readthedocs.io/

• GitHub: https://github.com/sciknoworg/OntoLearner

• PyPi: https://pypi.org/project/OntoLearner/

• Huggingface: https://huggingface.co/collections/SciKnowOrg/ontolearner-benchmarking

• License: MIT License

• Experimental Resources: https://doi.org/10.5281/zenodo.21719027

Introduction

Scientific research is becoming increasingly data- and automation-intensive, particularly in biomedicine and materials science, where high-content screening, automated experimentation, and self-driving laboratories generate growing volumes of heterogeneous data, experimental procedures, and domain terminology ¹. For these outputs to be reused across laboratories, databases, and computational systems, they must be represented in forms that are not only digitally accessible but also findable, interoperable, and machine-actionable ². Structured knowledge representations are therefore essential for semantic search, cross-source integration, reproducible analysis, scientific reasoning, and AI-assisted discovery ³. Ontologies provide the formal conceptual layer needed to organize domain entities and relationships consistently across heterogeneous sources. However, constructing and maintaining high-quality ontologies remains a labor-intensive and expert-driven process, limiting the scalability, adaptability, and reproducibility of ontology engineering ⁴.

Recent advances in Large Language Models (LLMs) present new opportunities for automating ontology learning (OL) tasks [8]. Given lexical terms and ontology-grounded context, LLMs can support assigning terms to semantic types, identifying taxonomic relationships between concepts, and predicting non-taxonomic relations. Their ability to perform these tasks with little or no task-specific training makes LLM-based OL a promising approach for reducing the manual effort involved in ontology construction, extension, and maintenance, and for supporting the scalable development of knowledge graphs across scientific domains. The first LLMs4OL study established an empirical foundation for investigating LLMs in OL by evaluating a broad selection of model families across term typing, taxonomy discovery, and non-taxonomic relation extraction [8]. Its primary objective was to determine whether general-purpose and domain-specific language models could perform these tasks at all. Accordingly, it compared models with different architectures, training objectives, domain specializations, and parameter counts, including BERT [9], BART [10], Flan-T5 [11], BLOOM [12], LLaMA [13], GPT [14], and PubMedBERT [15]. Comparisons between selected Flan-T5 and BLOOM variants provided initial indications that larger models could sometimes perform better, while task-specific fine-tuning showed that smaller models could also outperform substantially larger ones. Model size was therefore not isolated from model family, architecture, training strategy, or domain adaptation.

The LLMs4OL Challenges, organized at the International Semantic Web Conference (ISWC), subsequently broadened this investigation through shared tasks, datasets, evaluation phases, and metrics for community-developed OL systems [16, 17]. In the 2024 challenge, participants explored fine-tuning and knowledge-enhanced prompt-tuning [18, 19], prompt-tuning across multiple LLMs [20], retrieval-augmented generation [21], hybrid LLM–rule-based processing [22], and conventional machine-learning methods combined with LLMs [23]. The 2025 challenge expanded this methodological range through systems combining prompt engineering and embeddings [24], RAG, few-shot prompting, embedding ensembles, and lightweight fine-tuning [25], clustering and retrieval-augmented extraction [26], data augmentation and candidate filtering [27], and multi-model deliberation [28]. Some submissions also compared differently sized models [29, 30], but these comparisons generally varied model family, architecture, adaptation strategy, or system configuration alongside size. They were consequently suited to identifying effective OL systems, but not to estimating the independent effect of parameter scale. This leaves a fundamental question unanswered:

This gap reflects a broader and still unresolved debate in foundation-model research concerning whether larger LLMs are systematically more capable. Early neural scaling-law studies identified smooth power-law reductions in pretraining loss as model parameters, training data, and computational resources increased [31]. Subsequent compute-optimal analyses, however, demonstrated that parameter count alone is an incomplete indicator of capability: a smaller model trained on substantially more data can outperform a much larger but under-trained model [32]. Model scale must therefore be understood in relation to training data, compute, and architecture rather than through parameter count alone. The relationship becomes even less predictable at the downstream-task level. Some capabilities have been reported to appear only after models exceed particular scale thresholds [33], whereas other work has shown that apparently discontinuous emergence may partly result from the choice of evaluation metric [34]. Inverse and U-shaped scaling have also been observed, whereby performance can plateau, deteriorate, or change direction as models grow [35, 36]. A recent re-examination similarly found smooth and predictable scaling in only a minority of downstream-task settings [37]. Improvements in pretraining objectives or aggregate benchmarks therefore cannot be assumed to transfer monotonically to specialized knowledge-engineering tasks.

This uncertainty is especially consequential for OL because its constituent tasks impose different semantic and structural demands. Term typing requires calibrated selection among candidate ontology classes, taxonomy discovery requires reasoning over hierarchical relations, and non-taxonomic relationship extraction requires distinguishing heterogeneous domain relations. Increasing scale may therefore affect precision, recall, and relational reasoning differently across tasks and domains. Furthermore, “model size” is not entirely unambiguous across architectures. Dense models activate their full parameter set, whereas Mixture-of-Experts (MoE) models separate total model capacity from the smaller number of parameters activated for an individual input [38]. Architecture, training composition, instruction tuning, and alignment may consequently mediate or outweigh the effect of nominal parameter count. Although existing OL studies indicate that model choice and capacity matter, they do not isolate these factors through a sufficiently broad within-family scale sweep under fixed experimental conditions. We therefore investigate the following research questions:

RQ1: Within-family scaling. How does increasing parameter count within a common dense model family affect precision, recall, and F1 performance in OL, and are the resulting improvements monotonic?

RQ2: Task and domain sensitivity. How does the effect of model scale vary across OL tasks and across scientific domains with different conceptual structures?

RQ3: Architecture and model lineage. How do dense versus MoE architectures and differences across Qwen and GPT release lineages affect OL performance beyond nominal parameter count?

We address these questions through a controlled benchmark study using OntoLearner [39], a reproducible ecosystem for LLM-based OL. We evaluate 13 models spanning the Qwen3.5 [40], Qwen3.6 [41, 42], and proprietary GPT lineages. The Qwen3.5 models enable a within-family dense-model scale sweep from 0.8B to 27B parameters, complemented by larger sparse MoE variants with 35B total and 3B active parameters and 122B total and 10B active parameters. The Qwen3.6 models permit matched-scale comparisons with Qwen3.5 across a 27B dense model and a 35B-A3B MoE model, while GPT-5.5 [43] and the GPT-5.6 Luna [44], Terra [45], and Sol [46] variants provide a comparison across proprietary model releases. All models are evaluated on term typing, taxonomy discovery, and non-taxonomic relationship extraction over biomedical and materials science and engineering ontologies. By holding the embedding model, retrieval configuration, retrieved context, prompt templates, decoding conditions, datasets, and evaluation metrics constant, we examine the effects of parameter scale, dense versus sparse architecture, model release, task structure, and domain characteristics. The Results section is organized according to the three research questions and provides empirical guidance for model selection in scalable and reproducible ontology engineering.

Preliminaries

Following the OL formalization introduced in [8] and adopted by the LLMs4OL Challenge, we consider three fundamental OL tasks. Let $L$ denote a lexical term extracted from text and $T$ denote a type (i.e., an ontology class). These tasks constitute the basis of our experimental evaluation.

Term Typing. Term typing aims to assign the most appropriate ontology type to a lexical term. Formally,

where the input is a lexical term $L$, and the output is its corresponding ontology type $T$.

Taxonomy Discovery. Taxonomy discovery identifies hierarchical (is-a) relationships between ontology types. Given two lexical terms, their corresponding ontology types are inferred, and the taxonomic relationship between them is determined. Formally,

where $T_a$ and $T_b$ denote the ontology types associated with lexical terms $L_a$ and $L_b$, respectively.

Non-Taxonomic Relationship Extraction. This task identifies semantic relations other than $is\text{-}a$ between ontology types. Given a head lexical term $h$, a relation $r$, and a tail lexical term $t$, the objective is to infer the semantic relation between their corresponding ontology types. Formally,

where $T_h$ and $T_t$ are the ontology types corresponding to the head and tail lexical terms, respectively.

Related Work

OL has long been recognized as a bottleneck-relieving alternative to manual ontology engineering, and several recent surveys examine how this bottleneck manifests in specific application domains. Ivanova and Terzieva [47] systematically review OL techniques for intelligent e-learning environments, mapping classical linguistic, statistical, and logic-based OL methods onto educational data sources and highlighting that LLM-based OL for education remains comparatively under-studied despite the maturity of general-purpose OL research. Their review echoes a broader pattern in the literature: domain adaptation of OL techniques is frequently treated as a secondary concern relative to the development of the underlying extraction methods themselves.

LLM-Based Ontology Generation Pipelines. A growing body of work investigates ontology construction directly from natural language using LLMs. Lippolis et al. [48] introduce two prompting techniques, Memoryless CQbyCQ and Ontogenia, for generating OWL drafts from competency questions and user stories, finding that reasoning-oriented models (e.g., o1-preview) can approach or exceed novice human ontology engineers on structural correctness, while open-weight models like Llama produce more superfluous elements and critical modeling pitfalls. In a related study, Lippolis et al. [49] assess the domain-generalizability of two reasoning-capable LLMs (DeepSeek and o1-preview) across six ontology-engineering projects and find remarkably consistent performance across domains, suggesting that ontology-generation ability is not narrowly tied to domain representation in pre-training data. Teplyi and Dosyn [50] propose a four-stage prompt-guided agent (schema discovery, instance extraction, self-repair validation, ontology alignment) that iteratively enforces domain-range consistency, achieving fact-recall competitive with specialized knowledge graph extraction pipelines such as KGGen while additionally producing schema-consistent output. Their comparison across GPT-4.1-mini, LLaMA-3.3-70b, and Grok-3-mini shows that larger models tend to hypothesize broader schemas that remain partly unpopulated, whereas smaller models produce more compact but fully instantiated ontologies, an early signal of the scale-dependent trade-offs that our work investigates more systematically. Moreover, Fathallah et al. [51] extend this line of work with NeOn-GPT, a NeOn-methodology-grounded pipeline incorporating an explicit ontology-reuse stage and automated verification (RDFLib, HermiT/Pellet, OOPS!). Evaluated across four heterogeneous domains using GPT-4o, Mistral, Llama-4, and DeepSeek, they report that all models default to shallow, single-inheritance hierarchies regardless of architecture or parameter count, a “systematic generative limitation” rather than a model-specific one, and that entity-level semantic alignment consistently exceeds relation-level alignment. Zhang et al. [52] similarly combine LLM-driven extraction (via ChatGPT and Llama) with graph-database backends (OLIVE), emphasizing iterative human-in-the-loop correction of hallucinated relations validated through OOPS! and Protégé reasoning.

Fine-Tuning versus Prompting and Model Comparisons. Several works directly compare model families or adaptation strategies for OL, providing precedent for our size-focused investigation. Doumanas et al. [53] fine-tune GPT-4 and Mistral 7B on curated ontology-engineering textbook corpora, finding that GPT-4 achieves higher precision and syntactic fidelity at greater computational cost, while Mistral 7B is cheaper and faster but degrades unpredictably across successive fine-tuning rounds, underscoring that model family and scale interact with training strategy in ways that are not yet well characterized. Liu et al. [54] compare pre-trained prompting, in-context learning, and fine-tuning for

casting domain term and relation extraction, finding fine-tuning superior in F1 but noting the substantial annotation cost required to enable it. Cappelli and Di Marzo Serugendo [55] instead rely on a single frontier model (ChatGPT-4o) guided by a structured “guiding table” of prompts, reporting high logical consistency but persistently flat, low-attribute-richness taxonomies—again suggesting that prompting strategy alone cannot fully compensate for structural limitations that may be tied to model capacity. None of these studies, however, isolate model size as an independent variable while holding domain, prompting strategy, and evaluation protocol constant.

OL for Knowledge Graphs, Retrieval, and Specialized Domains. A parallel thread examines OL as an upstream component of larger systems rather than an end in itself. da Cruz et al. [56] show that ontology-guided knowledge graphs—particularly those derived from relational-database schemas—achieve retrieval performance competitive with GraphRAG at substantially lower LLM-inference cost than text-derived alternatives, while Lippolis et al. [57] target the complementary problem of requirement-driven ontology extension rather than generation from scratch, showing via expert review that RAG-grounded extension proposals require only minor revision before production use. A similar topic was in the middle of interest to the LLMs4OL challenge community this year ⁶. Domain-specific deployments further illustrate OL’s practical stakes: an ICMHI ’25 study [58] applies OL models to generate custom FHIR extensions for healthcare data interoperability, underscoring how OL quality directly affects downstream system usability in high-stakes settings.

As Table 1 makes clear, existing work has compared different LLM families (GPT vs. Mistral vs. Llama vs. DeepSeek) and, in a few cases, informally noted size-related trends as a byproduct of family comparisons [50, 53], but no study isolates model scale as an independent variable within a single family while holding domain, prompting strategy, and evaluation protocol fixed. Likewise, cross-domain evaluation is common, but it is rarely paired with a scale sweep: studies that vary domain [49, 51] do not vary size, and the one study that varies size across a handful of models [50] does not control for domain or family. This leaves open a basic practical question for anyone using LLM-based OL: does scaling within a model family reliably improve OL quality, and is the marginal gain worth the added computational cost? We address this gap by using the LLMs4OL task formalization (term typing, taxonomy discovery, non-taxonomic relation extraction) together with the OntoLearner benchmarking ecosystem [39] to run a controlled scale sweep across the most recent models from Qwen and GPT model families, evaluated consistently on both biomedical and materials-science ontologies.

Study	Model(s)	Approach	Domain(s)	Cross-Domain	Scale Controlled	Key Finding
Lippolis et al. [48]	o1-preview, GPT-4, Llama	Prompt (CQbyCQ, Ontogenia)	10 mixed ontologies	✓	✗	Reasoning models rival novice engineers; Llama yields more critical pitfalls
Lippolis et al. [49]	DeepSeek, o1-preview	Prompt	6 domains	✓	✗	Performance is consistent across domains, not tied to pretraining coverage
Teplyi & Dosyn [50]	GPT-4.1-mini, LLaMA-3.3-70B, Grok-3-mini	Agentic + self-repair	General text	✗	Partial5	Larger models hypothesize broader, unpopulated schemas; smaller models produce compact, fully instantiated ones
Fathallah et al. [51]	GPT-4o, Mistral, Llama-4, DeepSeek	Prompt + reuse	4 domains	✓	✗	All models default to shallow hierarchies regardless of scale or architecture
Zhang et al. [52]	ChatGPT, Llama	Prompt + HITL	General (KG)	✗	✗	Human-in-the-loop correction still required to fix hallucinated relations
Doumanas et al. [53]	GPT-4, Mistral 7B	Fine-tune	OE textbooks	✗	Partial5	Fine-tuning benefit is model-family-dependent; Mistral degrades across rounds
Liu et al. [54]	GPT-4.1-mini	Prompt vs. ICL vs. FT	Casting	✗	✗	Fine-tuning yields best F1 but at substantial annotation cost
Cappelli & Serugendo [55]	ChatGPT-4o	Prompt (guiding table)	1 domain	✗	✗	High logical consistency but flat, low-attribute-richness taxonomies
da Cruz et al. [56]	LLaMA-3.2-3B	RAG	General	✗	✗	DB-derived ontologies rival GraphRAG at lower LLM inference cost
Lippolis et al. [57]	GPT-5, o1-preview	RAG	General	✗	✗	Extension proposals need only minor expert revision before reuse
Cherian et al. [58]	unspecified	OL for FHIR	Healthcare	✗	✗	OL quality directly affects downstream interoperability outcomes

Methodology

To evaluate how model parameter scale impacts performance across core OL tasks defined in section 2, we adopt a Retrieval-Augmented Generation (RAG) framework adapted from the OntoLearner infrastructure [39].⁷ Our pipeline decouples context acquisition from generation by pairing a static dense retriever with the target LLM. This setup isolates the generation capabilities of each LLM scale variant under uniform retrieval conditions. The overall architecture consists of three sequential modules: (1) Semantic Vector Indexing, (2) Similarity-Based Retrieval, and (3) Task-Specific LLM Prompting and Generation.

Semantic Vector Indexing. Before running inference, we preprocess the underlying target ontology $\mathcal{O}$ into a searchable vector index. The representations are encoded into dense numerical vectors using a frozen feature extractor $E(\cdot)$. Specifically, we deploy Qwen3-Embedding-4B [59]² across all experimental setups to project all target ontological entities into a shared high-dimensional vector space: $v_i = E(\operatorname{text}(c_i)) \in\mathbb{R}^{d}$. The resulting embeddings are stored in a dense vector database indexed for fast nearest-neighbor lookups using cosine similarity.

Similarity-Based Retrieval. During inference, an input candidate query $q$ (e.g., a candidate lexical term $L$ or a candidate pair of terms) is converted into an embedding using the same feature extractor $q = E(\operatorname{text}(q))$. Rather than relying on iterative multi-hop expansion or retriever-fine-tuning, we employ a single-pass dense retrieval strategy. The system computes the cosine similarity between the query vector $q$ and all entity vectors $v_i$ in the pre-indexed ontology base. We retrieve the top-$k$ nearest ontological entities $\mathcal{C}_q = \{c_1, c_2, \ldots, c_k\}$ that yield the highest alignment scores $\operatorname{Sim}(q, v_i) = \frac{q \cdot v_i}{\lVert q \rVert_2 \lVert v_i \rVert_2}$. For all experiments in this study, we fix $k = 10$. The textual metadata associated with these top-$k$ entities forms the ground-truth reference context injected into the downstream prompt.

Task-Specific LLM Generation. The final phase constructs a structured, task-specific instruction prompt combining three distinct elements: (1) a system role specification defining the target OL task rules, (2) the retrieved top-$k$ background ontological entities $\mathcal{C}_q$ serving as candidate grounded choices, and (3) the specific target input (e.g., lexical term $L$ for Term Typing or term pair $(L_a, L_b)$ for Taxonomy Discovery). The complete prompt is passed to the target LLM under zero-shot conditions with identical decoding hyperparameters across model sizes. To guarantee reliable evaluation, the model is constrained to generate structured output conforming to the expected formal schema for each task:

• Term Typing: The LLM selects the single most accurate ontological class $T$ from the candidate set $\mathcal{C}_q$ that encapsulates the lexical term $L$.

• Taxonomy Discovery: The LLM determines whether a direct subsumption relationship (is-a) exists between the concepts associated with $L_a$ and $L_b$ within the retrieved context.

• Non-Taxonomic Relationship Extraction: Given head term $h$, candidate relation $r$, and tail term $t$, the LLM predicts whether the relation holds based on the provided domain context.

By keeping the embedding model, vector index, top-$k$ parameter ($k = 10$), and prompt templates strictly constant across all runs, any observed variation in task performance directly reflects the impact of model parameter scale and underlying architecture.

Experimental Setup

Large Language Models. We evaluate a diverse set of open- and closed-source LLMs spanning multiple parameter scales and architectural designs. The benchmark includes dense and Mixture-of-Experts (MoE) variants from the Qwen3.5 and Qwen3.6 families, as well as frontier OpenAI GPT models, enabling a systematic investigation of how model size and architecture influence OL performance. Table 2 summarizes the evaluated models and their key characteristics, where all are supported by OntoLearner for reproducible experimentation and benchmarking.

Model	Family	Architecture	Scale	Availability	Model Page
Qwen3.5-0.8B	Qwen3.5	Dense	0.8B	Open	https://huggingface.co/Qwen/Qwen3.5-0.8B
Qwen3.5-2B	Qwen3.5	Dense	2B	Open	https://huggingface.co/Qwen/Qwen3.5-2B
Qwen3.5-4B	Qwen3.5	Dense	4B	Open	https://huggingface.co/Qwen/Qwen3.5-4B
Qwen3.5-9B	Qwen3.5	Dense	9B	Open	https://huggingface.co/Qwen/Qwen3.5-9B
Qwen3.5-27B	Qwen3.5	Dense	27B	Open	https://huggingface.co/Qwen/Qwen3.5-27B
Qwen3.5-35B-A3B	Qwen3.5	MoE	35B (3B active)	Open	https://huggingface.co/Qwen/Qwen3.5-35B-A3B
Qwen3.5-122B-A10B	Qwen3.5	MoE	122B (10B active)	Open	https://huggingface.co/Qwen/Qwen3.5-122B-A10B
Qwen3.6-27B	Qwen3.6	Dense	27B	Open	https://huggingface.co/Qwen/Qwen3.6-27B
Qwen3.6-35B-A3B	Qwen3.6	MoE	35B (3B active)	Open	https://huggingface.co/Qwen/Qwen3.6-35B-A3B
GPT-5.5	GPT	Proprietary	–	Closed	https://developers.openai.com/api/docs/models/gpt-5.5
GPT-5.6-Luna	GPT	Proprietary	–	Closed	https://developers.openai.com/api/docs/models/gpt-5.6-luna
GPT-5.6-Terra	GPT	Proprietary	–	Closed	https://developers.openai.com/api/docs/models/gpt-5.6-terra
GPT-5.6-Sol	GPT	Proprietary	–	Closed	https://developers.openai.com/api/docs/models/gpt-5.6-sol

Datasets. For experimental purposes, we select four established ontologies from two scientific domains: materials science and engineering and the biomedical domain. These domains are selected to evaluate LLM-scale effects across heterogeneous scientific areas with different conceptual structures and levels of complexity. The selected ontologies include MatWerk [60] and MDS [61] from the materials science domain, and OBI [62] and MFOEM [63, 64] from the biomedical domain. For each ontology, we used OntoLearner benchmark collections hosted in Hugging Face ⁸ to construct evaluation tasks covering term typing, taxonomy discovery, and non-taxonomic relationship extraction. Table 3 summarizes the ground truth statistics of the selected datasets. Moreover, to manage experiments, we used the OntoLearner train_test_split strategy to preserve only 20% of the ontological data for OBI experiments.

Domain	Ontology	Terms No.	Types No.	Relations No.	Term Typing	Taxonomy Discovery	Non-Taxonomic RE
Material Science and Engineering	MatWerk [60]	29	313	2	29	369	12
	MDS [61]	–	297	3	–	351	128
Medical	OBI [62]	53	4,934	3	53	2,368	–
	MFOEM [63, 64]	19	621	2	19	837	20

Results

The experimental results for medical ontologies are presented in Table 4, and for material science and engineering domain ontologies are represented in Table 5 for three core LLMs4OL paradigm tasks using precision, recall, and F1-scores. Overall, the empirical evaluation reveals three insights: (1) increasing parameter scale in open-weight models primarily enhances the decision boundary rather than recall; (2) dense models at medium scale (e.g., 27B) often outperform much larger MoE models on the term typing task; and (3) domain complexity creates severe performance bottlenecks, particularly in the non-taxonomic relation extraction task on specialized domains.

LLMs	MFOEM Term Typing Prec	MFOEM Term Typing Rec	MFOEM Term Typing F1	MFOEM Taxonomy Discovery Prec	MFOEM Taxonomy Discovery Rec	MFOEM Taxonomy Discovery F1	MFOEM Non-Taxonomic RE Prec	MFOEM Non-Taxonomic RE Rec	MFOEM Non-Taxonomic RE F1	OBI Term Typing Prec	OBI Term Typing Rec	OBI Term Typing F1	OBI Taxonomy Discovery Prec	OBI Taxonomy Discovery Rec	OBI Taxonomy Discovery F1
Qwen3.5-0.8B	25.00	100.00	40.00	4.84	61.82	8.98	3.91	75.00	7.43	9.06	90.57	16.47	3.29	52.24	6.19
Qwen3.5-2B	25.00	100.00	40.00	4.84	61.82	8.98	3.89	75.00	7.39	9.06	90.57	16.47	3.29	52.24	6.19
Qwen3.5-4B	25.33	100.00	40.43	4.84	61.82	8.98	3.76	70.00	7.14	9.28	90.57	16.84	3.29	52.24	6.19
Qwen3.5-9B	27.27	94.74	42.35	4.93	61.82	9.12	5.19	75.00	9.71	11.03	90.57	19.67	3.53	52.24	6.61
Qwen3.5-27B	41.67	52.63	46.51	8.69	60.61	15.20	6.07	75.00	11.24	18.75	90.57	31.07	6.53	51.79	11.59
Qwen3.5-35B-A3B	28.33	89.47	43.04	11.38	59.85	19.12	3.97	75.00	7.54	18.68	90.57	30.97	10.41	50.78	17.28
Qwen3.5-122B-A10B	26.56	89.47	40.96	15.38	59.24	24.41	4.23	75.00	8.00	16.44	90.57	27.83	14.16	50.03	22.07
Qwen3.6-27B	35.56	84.21	50.00	11.07	60.61	18.73	4.41	75.00	8.33	13.30	90.57	23.19	9.92	51.54	16.63
Qwen3.6-35B-A3B	26.76	100.00	42.22	9.16	59.09	15.86	3.95	75.00	7.50	9.56	90.57	17.30	7.58	50.88	13.20
GPT-5.5	70.00	73.68	71.79	33.89	55.30	42.03	9.52	70.00	16.77	70.31	84.91	76.92	28.90	48.01	36.08
GPT-5.6-Luna	100.00	5.26	10.00	35.22	48.94	40.96	10.13	40.00	16.16	52.70	73.58	61.42	27.91	44.53	34.32
GPT-5.6-Terra	71.43	26.32	38.46	31.41	53.48	39.57	8.48	70.00	15.14	51.11	86.79	64.34	30.06	45.09	36.07
GPT-5.6-Sol	58.33	36.84	45.16	27.47	55.61	36.77	8.22	60.00	14.46	71.21	88.68	78.99	26.41	48.31	34.15

LLMs	MatWerk Term Typing Prec	MatWerk Term Typing Rec	MatWerk Term Typing F1	MatWerk Taxonomy Discovery Prec	MatWerk Taxonomy Discovery Rec	MatWerk Taxonomy Discovery F1	MatWerk Non-Taxonomic RE Prec	MatWerk Non-Taxonomic RE Rec	MatWerk Non-Taxonomic RE F1	MDS Taxonomy Discovery Prec	MDS Taxonomy Discovery Rec	MDS Taxonomy Discovery F1	MDS Non-Taxonomic RE Prec	MDS Non-Taxonomic RE Rec	MDS Non-Taxonomic RE F1
Qwen3.5-0.8B	14.29	100.00	25.00	4.25	60.66	7.94	3.60	81.82	6.90	1.36	18.95	2.54	0.62	27.34	1.22
Qwen3.5-2B	14.29	100.00	25.00	4.22	60.33	7.89	3.60	81.82	6.90	1.39	19.28	2.59	0.61	26.56	1.19
Qwen3.5-4B	14.65	100.00	25.55	4.22	60.33	7.89	3.35	72.73	6.40	1.39	19.28	2.59	0.62	26.56	1.20
Qwen3.5-9B	22.22	96.55	36.13	4.52	60.33	8.40	5.03	81.82	9.47	1.42	19.28	2.64	0.51	21.09	0.99
Qwen3.5-27B	43.86	86.21	58.14	6.76	59.34	12.13	6.62	81.82	12.24	3.24	18.63	5.52	0.56	14.06	1.07
Qwen3.5-35B-A3B	20.28	100.00	33.72	11.34	58.03	18.97	3.63	81.82	6.95	2.77	16.99	4.77	0.61	26.56	1.20
Qwen3.5-122B-A10B	24.78	96.55	39.44	14.50	56.07	23.05	3.90	81.82	7.44	5.65	13.73	8.01	0.65	26.56	1.27
Qwen3.6-27B	23.01	89.66	36.62	9.48	60.33	16.38	4.15	81.82	7.89	3.60	16.99	5.95	0.61	26.56	1.19
Qwen3.6-35B-A3B	15.76	100.00	27.23	6.16	58.03	11.14	3.63	81.82	6.95	2.79	15.69	4.74	0.63	26.56	1.23
GPT-5.5	70.59	82.76	76.19	29.70	48.20	36.75	8.25	72.73	14.81	16.88	12.75	14.53	2.80	7.81	4.12
GPT-5.6-Luna	73.68	48.28	58.33	33.88	47.21	39.45	13.33	36.36	19.51	20.56	7.19	10.65	0.00	0.00	0.00
GPT-5.6-Terra	69.23	62.07	65.45	30.95	43.93	36.31	7.63	81.82	13.95	18.64	7.19	10.38	1.08	7.81	1.90
GPT-5.6-Sol	76.92	68.97	72.73	24.23	51.80	33.02	7.92	72.73	14.29	13.17	12.09	12.61	1.08	7.03	1.87

Model Scale Dynamics in Open-Weight Families. Evaluating the Qwen parameter progression (0.8B to 122B) demonstrates that scaling model capacity differs across tasks and parameter sizes. More technically:

• Precision Deficits in Small Models (< 9B). Small-scale open-weight models (Qwen3.5-0.8B, 2B, and 4B) exhibit a persistent pattern in performance. On Term Typing across MFOEM, OBI, and MatWerk, these models achieve near-perfect or complete recall (90.57%–100.00%) but with a poor precision (9.06%–25.33%). In taxonomy discovery and non-taxonomic relation extraction, precision collapses further to 1.36%–4.84%. This behavior indicates that the Qwen family of models with fewer than 9B parameters lacks the decision boundary calibration required to reject negative candidate classes provided within the RAG.

• The 27B Parameter Turning Point. The transition from 9B to 27B parameters marks the primary performance inflection point within the open-weight model family. On MatWerk Term Typing, Qwen3.5-27B achieves an F1-score of 58.14%, representing a +22.01 percentage point gain over Qwen3.5-9B (36.13%). This improvement is driven by a doubling in precision (from 22.22% to 43.86%) while maintaining high recall (86.21%). A similar precision-driven gain is observed on OBI Term Typing (31.07% vs. 19.67% F1).

• Dense vs. Mixture-of-Experts Trade-offs. A comparison between dense and sparse MoE architectures reveals clear task-dependent trade-offs:

• Term Typing: Dense models demonstrate superior active parameter efficiency. The dense Qwen3.5-27B consistently outperforms the larger Qwen3.5-122B-A10B MoE model on term typing across MFOEM (46.51% vs. 40.96% F1) and MatWerk (58.14% vs. 39.44% F1). High active parameter density appears essential for the term typing task.

• Taxonomy Discovery: Conversely, MoE architectures excel at hierarchical relational reasoning. On MFOEM and MatWerk, Qwen3.5-122B-A10B achieves the highest F1-scores among all open-weight models (24.41% and 23.05%, respectively), outperforming its dense 27B counterpart (15.20% and 12.13%). The expanded total capacity of MoE models provides the enhanced capability necessary to evaluate subsumption relationships across paired concepts.

• Version-Specific Alignment Anomaly (Qwen 3.5 vs. 3.6). Comparing release versions reveals an unexpected performance regression in Qwen3.6-27B relative to Qwen3.5-27B on MatWerk Term Typing (36.62% vs. 58.14% F1). This drop suggests that architectural modifications or alignment fine-tuning introduced in version 3.6 altered the model’s sensitivity to specialized engineering terminologies, reinforcing that parameter scale alone does not guarantee monotonic performance improvements across distinct model releases.

Frontier Model Performance and the Alignment Penalty. Proprietary frontier models (GPT-5.5 and GPT-5.6 variants) maintain a substantial performance margin over open-weight baselines, achieving peak F1-scores of 71.79% on MFOEM term typing and 76.19% on MatWerk term typing. However, cross-model comparison within the GPT family highlights significant alignment-induced variations.

• GPT-5.5 Superiority. Across nearly all evaluated tasks and domains, GPT-5.5 outperforms its newer GPT-5.6 successors (Luna, Terra, and Sol). It provides a well-calibrated trade-off between precision and recall (e.g., 70.00% precision and 73.68% recall on MFOEM Term Typing), demonstrating robust generalization across domain shifts.

• Over-Conservatism in GPT-5.6 Lineage. The GPT-5.6 family demonstrates an over-conservative bias, prioritizing precision over recall. This dynamic is most visible in GPT-5.6-Luna:

  • On MFOEM term typing, GPT-5.6-Luna yields 100% precision but collapses to 5.26% recall, resulting in a low F1-score of 10%.

  • On MDS Non-Taxonomic Relation Extraction, GPT-5.6-Luna fails entirely (0% Precision, Recall, and F1), declining to output positive relationship assertions.

This behavior suggests that the 5.6 release rendered certain model variants highly prone to false negatives when faced with domain-specific knowledge.

Scale Sensitivity Across Task Complexity and Domains. To study how model size affects performance scaling, we examine how task complexity and domain abstraction influence the benefits of increasing parameter capacity (shown by the scaling curves in Figure 1).

• Task Complexity Changes the Required Scale. The parameter size needed to achieve useful performance increases with task structural complexity (Term Typing $\prec$ Taxonomy Discovery $\prec$ Non-Taxonomic RE):

  • Low-Floor Task (Term Typing): This task only requires selecting the correct candidate from a top-$k$ list. As a result, scaling provides strong improvements at smaller model sizes (0.8B $\to$ 27B achieves a +33.14 F1 gain on MatWerk).

  • Moderate-Floor Task (Taxonomy Discovery): This task requires reasoning over relationships between multiple concepts. Small models ($< 9$B) reach a performance plateau below 9% F1, while noticeable improvements appear only at 27B+ and frontier-scale models (36%–42% F1).

– High-Floor Task (Non-Taxonomic RE): Detecting arbitrary semantic relations ($h$, $r$, $t$) is a highly complex task. Increasing parameters for open-weight models provides limited improvements, reaching only 12.24% F1 for Qwen3.5-27B on MatWerk. Even frontier-scale models achieve only modest F1 scores (16.77%–19.51%), showing that increasing model size alone is not enough to solve multi-relational semantic spaces under zero-shot RAG settings.

• Domain Abstraction Creates Scale Resistance in MDS. Comparing different domains shows that the level of abstraction affects how much model scaling can improve performance. Scaling consistently improves results in biomedical domains (MFOEM, OBI) and material synthesis (MatWerk). However, the highly abstract Materials Data Science (MDS) domain shows strong scale resistance. For MDS Non-Taxonomic Relation Extraction, increasing Qwen model size from 0.8B to 122B provides almost no improvement (1.22% vs. 1.27% F1), while frontier GPT models achieve only 4.12% F1. Highly abstract and non-standardized conceptual schemas create difficulties that cannot be solved by parameter scaling alone.

Conclusion

In this work, we conducted a systematic study of how LLM scale influences OL performance across different tasks and scientific domains. Using a unified RAG-based evaluation framework, we benchmarked multiple model sizes from the Qwen and GPT families on LLMs4OL paradigm tasks. Our results show that increasing model size does not always lead to proportional improvements. Larger models generally improve precision and decision-making ability, but the benefits strongly depend on task complexity and domain characteristics. Term typing benefits most from scaling, while taxonomy discovery requires larger capacity for relational reasoning. In contrast, non-taxonomic relation extraction remains challenging even for frontier-scale models, particularly in highly abstract domains such as Materials Data Science. We also observe that architectural choices, such as dense versus MoE models, and model alignment strategies can significantly affect performance beyond parameter count alone. These findings highlight that model scale is only one factor determining OL capability. Effective LLM-based OL requires considering task structure, domain abstraction, and model architecture when selecting an appropriate model. Our OntoLearner infrastructure provides empirical guidance for choosing LLMs for ontology engineering and establishes a foundation for future research on scalable and reproducible OL systems.

Acknowledgments

This work is supported by the NFDI4DataScience initiative (DFG, German Research Foundation, Grant ID: 460234259).

Declaration on Generative AI

In preparing this manuscript, generative AI tools, specifically ChatGPT, were used solely for grammar checking, spelling checks, and readability of some sentences. All suggested changes were carefully reviewed and adapted by the authors to ensure accuracy and appropriateness. The scientific content, research design, analysis, and conclusions were developed and verified exclusively by the authors without AI involvement. The use of ChatGPT was limited to enhancing the presentation of the work.