WorksComputer Science
Continuous Ontology Assurance through External Resources – An Ontology Engineering Method with a Case Study in Software Language Engineering
Abstract
Domain ontologies require review against knowledge recorded in evolving external resources. We propose COATER, a method that manages executable review components called arrows. Each arrow compares a scoped part of an ontology with a relevant resource and reports support or challenges to its axioms. Challenges may prompt proposals for completion, correction, or restructuring and become tasks for human or agent-assisted triage. Per-axiom coverage summarizes the assessments. We illustrate COATER with the FSL ontology in software language engineering (SLE): arrows review language coverage, language conceptualization, and software-engineering activities. An LLM-assisted linkage exercise increases compliance with a defined Wikipedia-linkage policy from 405 to 600 of 600 entities. A structured catalogue of SLE resources organizes review opportunities.
Introduction
In this paper, we describe an ontology engineering method for ‘Continuous Ontology Assurance Through External Resources’ (COATER) and apply it to software language engineering (SLE) and a corresponding domain ontology FSL. Maintaining such an ontology requires more than checking its internal structure: its assertions and coverage also need to be assessed against knowledge of the domain. Ontology evaluation addresses structural and functional quality [6]; test-driven Linked Data evaluation provides executable quality checks and measures their coverage [9]. External resources can provide evidence for a different question: which parts of a maintained ontology are supported, challenged, or missing relative to a particular source?
Figure 1 illustrates the COATER method. On the right is the domain ontology to be assured, with TBox and ABox parts corresponding to the scope of external knowledge resources. On the left are such resources, including ontologies, Wikipedia pages, Bodies of Knowledge, and scholarly papers. The arrows represent ontology review components that use external resources to support or challenge ontology elements. Challenges may include revision proposals for completion, correction, or restructuring, and may give rise to tasks handled through agentic triaging with a human in the loop. The COATER method administers the continuous aggregation, execution, and maintenance of such arrows for ontology assurance.

Figure 1. Arrows in the COATER method.
Contributions. Prior work uses external sources for ontology evolution [20] and studies LLM-assisted ontology evaluation [12]. We contribute a method for recurring, resource-scoped reviews of a maintained ontology: arrows support or challenge its content and may propose revisions for review. We define the arrow and assessment model, including per-axiom coverage (§ 3). In the case study, we demonstrate ABox and TBox reviews, improved Wikipedia linkage, and a catalogue of SLE resources for FSL (§ 2, § 4, § 5).
Road-map of the paper. Section 2 presents case-study arrows, § 3 the general method, § 4 an assurance improvement, and § 5 a catalog of external knowledge resources. Section 6 discusses related work, and Sec. 7 concludes.
There is a reproducibility package available for this paper.1 Upon acceptance of this submission, an extended technical report will be published with appropriate linkage for the GitHub-based hosting of the FSL ontology and the COATER case study.
Illustrative COATER Arrows
In this section, we begin with the demonstration of the COATER method in the SLE domain for the FSL ontology. We focus here on the design and implementation of a few COATER arrows. These illustrative arrows can be found in the FSL repository.2
A `Language Popularity' Arrow
We start with a very simple arrow, where we aim to establish coverage for popular programming languages by the ontology. We pick the TIOBE Programming Community Index3 as an established resource for ranking programming languages.
Resource Extraction. We retrieve TIOBE’s top-10 languages with a Python-based scraper like this:
tiobe = pd.read_html("https://www.tiobe.com/tiobe-index/")
tiobe = tiobe[0].head(10)
tiobe = tiobe.iloc[:, [0, 1, 4]]
tiobe.columns = ["Rank", "RankLastYear", "Language"]
These are the top 10 at the time of writing:
| Rank | Rank Last Year | Language |
| 1 | 1 | Python |
| 2 | 3 | C |
| 3 | 2 | C++ |
| 4 | 4 | Java |
| 5 | 5 | C# |
| 6 | 6 | JavaScript |
| 7 | 7 | Visual Basic |
| 8 | 11 | SQL |
| 9 | 13 | R |
| 10 | 18 | Rust |
Relevant Ontology Scope. We query the ontology like this:
SELECT DISTINCT ?language ?classifier
WHERE {
?classifier rdfs:subClassOf* tbox:SoftwareLanguage .
?language rdf:type ?classifier .
}
ORDER BY ?classifier ?language
In this query, we include instances of all subclasses of ‘Software-Languages’ — including ‘ProgrammingLanguage’, as we noticed that Tiobe views SQL as a programming language, while FSL assumes a nuanced taxonomy of software languages.
Ontology Review. Let’s test for completeness:
onto = ... # Query ontologies as shown
missing = tiobe[~tiobe["Language"].isin(onto["SoftwareLanguage"])]
We get these results:
| Rank | Rank Last Year | Language |
| 3 | 2 | C++ |
| 5 | 5 | C# |
| 7 | 7 | Visual Basic |
These are false negatives, i.e., these languages are not missing in the ontology; they are merely showing up due to naming/lexical conventions. We reconcile the names:
tiobe["Language"] = tiobe["Language"].replace({
"C#": "C_Sharp",
"C++": "Cpp",
"Visual Basic": "Visual_Basic"})
With these renamings in place, the ontology misses none of the languages retrieved from the external resource. In the terminology of the COATER method, 10 languages from the ontology are ‘supported’ by TIOBE.
Discussion. This ‘hello world’ example is very limited. Trivially, we looked only at the top-10 languages on TIOBE for the sake of presentation. Let’s identify issues to be addressed eventually for a comprehensive discussion of the COATER method.
ABox only The TBox was not involved here other than the realization that knowledge resources may disagree on the extension of the concept ‘Programming language’ which in turn led us to target software languages instead of programming languages in FSL.
Completeness only We only assess the ontology’s completeness relative to the external knowledge resource. We do not verify whether its language-related assertions are supported by TIOBE; this is addressed in the next arrow demonstration (§ 2.2).
Threats to validity We bet on name equality and similarity; we also rely on expert insight on the assumed correspondence of language categories.
A `Language Conceptualization' Arrow
A more advanced assurance scenario follows. FSL classifies and conceptualises software languages. In the case of programming languages, this includes familiar notions such as programming paradigms and typing disciplines. The corresponding conceptualization relies on nuanced properties and extra classes for concepts. We leverage now an LLM to consult Wikipedia as an external knowledge resource for ontology review.
End-to-end Prompting. Figure 2 illustrates the ontology-review prompt, structured around ontology scope, knowledge extraction, and review. The LLM considers all facts about a language in the ontology, extracts conceivable facts from Wikipedia, and assesses each ontology fact. Figure 3 shows one support and one challenge.
Figure 2. Prompt for Wikipedia-based ontology review.

Figure 3. LLM Response for Wikipedia-Ontology Alignment
Assessments as Tasks. Support for ‘exceptions’ is plausible. Challenging ‘ad-hoc polymorphism’ is understandable: Java supports method, but not operator overloading. Such a challenge is best viewed as a task that can be handled in several ways: Human in the loop An ontology engineer may dismiss the challenge or refine the ontology to distinguish method from operator overloading.
Auto-close by support A challenge may be ignored if the fact is strongly supported by other arrows.
Auto-close on repetition A recurring challenge may be closed automatically if it was previously dismissed.
Arrow tuning Repeated (arguably vacuous) challenges may motivate prompt refinement to reduce future tasks.
Agentic voting Different sources or extraction schemes may provide richer or alternative semantics to be combined, for example, by agentic voting; see Sec. 2.3 for a brief discussion.
Discussion.
ABox only (again) We considered all language-related properties, but did not assess the TBox itself; this follows in Sec. 2.3.
Correctness, too The arrow addresses not only completeness, but also challenges ontology axioms.
Threats to validity We omitted here explicit knowledge-extraction results, limiting cross-validation.
A ‘Body of Knowledge Arrow
The Software Engineering Body of Knowledge (SWEBOK) [18] is an authoritative knowledge resource on software engineering. Since the domain of interest (SLE) clearly overlaps with software engineering, it provides a natural basis for ontology review.
Relevant Ontology Scope. FSL includes a taxonomy of ‘software engineering activities’, mainly to associate software languages, tools, formalisms, and related concepts with specific activities in the software engineering lifecycle. Figure 4 shows an excerpt of this taxonomy, which contains 94 edges in total.

Figure 4. SE activities in the ontology
Knowledge Extraction. SWEBOK is primarily a book of several hundred pages; its table of contents exposes the knowledge areas and their subdivisions relevant to ontology review. Several derived representations exist at different levels of formality, governance, and endorsement. We found one reported effort to extract an ontology from SWEBOK V3 [15] and verified that it captures SWEBOK’s knowledge areas and their decomposition.4
and their subdivisions relevant to ontology review. Several derived representations exist at different levels of formality, governance, and endorsement. We found one reported effort to extract an ontology from SWEBOK V3 [15] and verified that it captures SWEBOK’s knowledge areas and their decomposition.4
Ontology review. Using a prompting model similar to that in Sec. 2.2, we gather supports and challenges. More than two thirds of the ontology subclassing claims are supported; the remaining ones are challenged; see Figure 5 for examples. Both explanations are plausible. The challenge may arise because ‘technical debt management’ is not explicit in SWEBOK V3. Arguably, it is also not an important activity in the SLE domain. Removing the corresponding activity in response to the challenge may therefore be sensible; it would reduce noise for less relevant ontology axioms.

Figure 5. LLM response for SWEBOK-based ontology review
Discussion.
TBox, too This arrow focuses on the TBox, specifically the subclass relationships among SE activities. This form of OWL constraint does not introduce additional challenges.
Multi-agent evaluation We use a parametrised prompt template (such as “*We aim at a comparison for alignment of two resources .... The first resource {swebok} is a projection or representation of .... SWEBOK .... The second resource {sle} is a projection or representation of an ontology ...”). This allows alignment to be determined for different projections or representations and combined through a voting scheme, thereby accounting for intricacies of the resources, especially the size and complexity of SWEBOK.
Threats to validity The ontology uses a uniform taxonomy for SE activities, whereas SWEBOK uses a nuanced decomposition of SE knowledge areas; the semantic basis of the alignment is thus questionable. Moreover, any useful projection of SWEBOK remains considerable in size and may challenge LLM usage.
The COATER Method
In the previous section, we focused on and demonstrated COATER arrows, i.e., the ‘paths’ from external resources to affected segments of an ontology in domain . An arrow is a review component that returns assessments supporting or challenging some part of .
The COATER method administers creation, monitoring, and maintenance of a system of arrows within a COATER assurance process.
COATER Arrows
The design and implementation of an arrow is characterized as follows:
Identify an external knowledge resource of interest, i.e., a resource containing reasonably formal or formalizable knowledge in domain . This identification includes an informed hypothesis that and overlap in a manner useful for ontology assurance.
Identify a sub-ontology , thereby projecting onto the scope of within and claiming an upper bound on their overlap. By default, . The sub-ontology can be derived computationally, for example, by a SPARQL query or a DL expression. A weak understanding of the scope of within limits the reliability of challenges produced by .
Formally represent the relevant knowledge retrievable from as TBox/ABox segments and . At this stage, and need not be aligned; we merely seek a formal representation before assessment, thereby enabling additional validation. In practice, we may omit this formal representation. For example, the arrow in Sec. 2.2 tasked an LLM with returning assessments directly. In such cases, however, we could request a formal representation, thereby effectively exercising chain-of-thought prompting.
Realize arrow so as to review relative to (or ). Executing is expected to return a collection of assessments that support or challenge on the grounds of ; see below.
COATER Assessments
An assessment has the following structure:
Type of assessment: support or challenge.
The axioms to which the assessment applies.
An optional ontology revision proposal described as a state-based (i.e., delta-based) or operation-based transformation. In the state-based approach, axioms to be removed from and added to () are described. In the operation-based approach, an expression in a suitable ontology transformation language is given, whose semantics in turn describes a delta. We demonstrate the latter in Sec. 4.
An optional confidence value informing, within the COATER process, agents or humans for task management.
An optional explanation justifying the assessment, including its confidence and revision proposal (if present), while drawing on content from () and (). Ideally, this explanation helps other agents (arrows or humans) with reviewing the assessment.
Execution of arrow can be triggered in different ways:
Change-triggered Changes to (1) arrow ’s implementation or configuration, (2) the knowledge resource used by the arrow or at least its projection to knowledge , and (3) the targeted ontology or at least the ontology scope trigger re-execution of .
User-triggered The process owner (an ontology engineer) explicitly initiates execution; this may be useful during development and maintenance of the arrow.
COATER Assessments
Time-triggered The arrow is (re-)executed regularly (e.g., during a nightly build); this is useful when change tracking, explicit knowledge extraction (going from to ), or ontology scoping (going from to ) is limited.
Revision proposals may call for correction, completion, or restructuring, the latter including refactoring and recontextualization. Table 1 characterizes the corresponding assessment options.
| External-resource situation | Assessment | Typical revision proposal |
| Resource corroborates ontology content | support | none |
| Resource provides contrary evidence | challenge | correction |
| Resource provides positive evidence for content missing from the ontology | challenge | completion |
| Resource provides evidence of inadequate conceptual organization | challenge | restructuring |
| Resource is silent, and coverage is unknown or incomplete | uncovered | none |
| Resource is silent, but coverage is assumed complete for this axiom type | challenge | correction |
Table 1. Assessment options for arrows
Importantly, we take an essentially model-based, software engineering-like view of ontology revision. In particular, determining ontology consistency (as an internal logical property) and adequacy (in terms of competency questions) is not the focus. Instead, COATER ontology reviews aim to support or challenge axioms, where challenges may call for completion (adding axioms), correction (removing axioms), or restructuring (rewriting sets of axioms). We also make no significant distinction between TBox and ABox here. For example, subclass relationships are ‘just’ axioms that can be supported or challenged like ordinary ABox assertions.
At the process level, COATER assessments essentially become ‘tasks’ in the sense of task management.
Quantified Assurance
The COATER method relies on creating, monitoring, and maintaining a system of arrows within a COATER assurance process. We quantify assurance similarly to coverage in software testing: namely, by measuring to what extent and in what way the ontology in a given process state is covered by arrows. The basic coverage situations at the per-axiom level are as follows:
Support-only coverage The axiom is supported by at least one arrow and has no pending challenges; if such tasks existed, they were closed as ‘Won’t Fix’.
Net-positive coverage The axiom is more supported than challenged in the current process state; we discuss aggregation of assessments below.
Net-negative coverage The axiom is more challenged than supported in the current process state.
Challenge-only coverage The axiom is challenged by at least one arrow and has no rivaling support. The challenges must be addressed, and corresponding revision proposals must be considered.
Missing coverage The axiom is neither supported nor challenged. There is no assurance for this element. The system of arrows should be advanced to achieve coverage.
Challenges and supports are aggregated via the Abelian group using their associated confidences, with positive reals for supports and negative reals for challenges. For example, a support with confidence 0.75 stacked against a challenge with confidence 0.50 yields net-positive assurance of 0.25. Other schemes for aggregating per-axiom assessments are not discussed here for brevity.
We speak of complete assurance if all ontology axioms achieve support-only coverage or net-positive coverage. Otherwise, we speak of incomplete assurance. We quantify assurance by a 5-tuple
with the number of ontology axioms having support-only coverage, net-positive coverage, net-negative coverage, challenge-only coverage, and missing coverage. Other ontology-level assurance metrics are not discussed here for brevity.
Using a human in the loop and agentic techniques, we can close tasks arising from challenges; these tasks can also be auto-closed if they reappear in subsequent executions. In this manner, we can often move from net-positive to support-only coverage. Alternatively, the arrows causing the challenges may require refinement rather than manual or agentic override.
Advanced COATER Issues
Several issues make the COATER process challenging to implement and execute efficiently, particularly given the need to minimize work for the human in the loop and avoid redundant effort. We mention several issues here and defer a more comprehensive discussion to an extended presentation.
Cross validation Claiming full assurance based on a single supportive arrow may be optimistic, depending on the axiom type, the quality of the knowledge resource, and the arrow implementation, none of which is reliably captured by confidence values. We can therefore require cross validation (multiple supports) in addition to full assurance.
Confidence calibration Confidence values are decisive when distinguishing net-positive from net-negative coverage. Ontology engineers must be particularly alert to overconfidence in supports and underconfidence in challenges.
Explainable assurance Assurance can be quantified by the 5-tuple defined above, but an ontology engineer must also understand coverage qualitatively, for example, when seeking new scopes for coverage through revised or additional arrows. Such ‘explainable assurance’ can rely specifically on the visualization of coverage.
Demonstration of Assurance Improvement
We demonstrate assurance improvement for FSL with respect to a specific, easy-to-understand aspect — entity linkage to Wikipedia. We present the improvement end to end, starting from an initial (suboptimal) assurance situation, followed by LLM-based linkage discovery with auxiliary steps for context extraction, prompt generation, and execution. This leads to task generation in the COATER sense and semi-automatic task resolution, after which we assess the final, improved assurance situation.
Initial Assurance
Our ‘Wikipedia Linkage Policy’ validator, at some point back, reported the following numbers:
There are entities (languages, tools, formalisms, etc.) under the policy.
entities are ‘opt-out’ marked (link N/A).
entities lack any linkage.
entities have invalid linkage.
entities () have valid linkage.
foaf:isPrimaryTopicOf versus foaf:page usage is 165/324.
Here, valid linkage means that English Wikipedia articles are used, with exactly one article per resource; validity does not establish the suitability of the link. The numbers clearly leave room for assurance improvement: one third of the entities are not validly linked. Arguably, there is also excessive use of the semantically weaker ‘foaf:page’ predicate instead of the stronger ‘foaf:isPrimaryTopicOf’.
Context Extraction
We seek an LLM-based arrow that assesses Wikipedia linkage, both to validate the suitability of existing links and to propose suitable links where they are missing. We assume that the LLM must observe the resource to be linked and the ontology to make a reasonable linkage judgment.
SELECT DISTINCT ?entity ?direction ?subject ?predicate ?object
WHERE {
?entity rdf:type/rdfs:subClassOf* tbox:Entity .
FILTER(isIRI(?entity))
{
?entity ?predicate ?object .
BIND(?entity AS ?subject)
BIND("outgoing" AS ?direction)
} UNION {
?subject ?predicate ?entity .
BIND(?entity AS ?object)
BIND("incoming" AS ?direction)
}
}
Resources of interest derive from ‘tbox:Entity’. For each such entity, we collect all triples in which it occurs as subject or object, classifying them accordingly as outgoing or incoming. Figure 6 shows the essential query used for each entity. The result is captured in JSON for use in prompt. This context extraction constitutes a particular approach to sub-ontology identification according to Sec. 3.1.

Figure 6. Sparql query for context extraction.
LLM-based Assessment
Here is the prompt template (compressed for presentation) used for each entity:
Assess Wikipedia linkage for an ontology resource using its attached knowledge-graph context, the ontology TBox, and Wikipedia search. Identify existing foaf:isPrimaryTopicOf or foaf:page links and return exactly one policy-compliant canonical English Wikipedia link, preferring foaf:isPrimaryTopicOf when the article is primarily about the resource and otherwise using foaf:page; use the designated N/A fallback if no suitable article exists. Classify the result as support when the context contains exactly one compliant link, replace when existing links must be consolidated or changed, or resolve when no link exists. Output only the prescribed, directly parseable JSON object containing the resource, assessment, links found, and—except for support—the proposed link.
Table 2 summarizes the assessments resulting from applying the template to all entities in scope.
| Assessment | Frequency |
| support | 287 |
| replace | 204 |
| resolve | 109 |
Table 2. Assessment summary for Wikipedia linkage.
Task Processing
Several aspects of the LLM-based assessments and accompanying ontology revision proposals are algorithmically checkable and implemented as companion components of the Wikipedia-linkage arrow: checks for the existence of articles for proposed links, consistency between links observed by the LLM and those actually present in the ontology, and consistency between the number of links found and the assessment type (e.g., ‘resolve’ must correspond to 0 links present). We rely on a human in the loop for further validation as follows.
‘Support’ assessments are not further validated, as we assume that links present in the ontology were deliberately injected and are now further backed up by an ‘expert’ (an LLM).
‘Replace’ assessments are reviewed by the human in the loop. In this instance, almost all replacements arise from multiple links being present and/or the use of DBpedia instead of Wikipedia. All replacements were confirmed by the human in the loop.
‘Resolve’ assessments are reviewed for relevance and for the suitability of isPrimaryTopic versus page. We reviewed the first 10 assessments and then confirmed the entire package, delegating more in-depth relevance/suitability validation to another COATER arrow, not discussed in this short paper.
In total, 1117 automatically synthesized Sparql queries/updates were performed, resulting in an ontology with improved Wikipedia linkage; see below.
Final Assurance
Our ‘Wikipedia Linkage Policy’ validator reports the following numbers on the evolved ontology:
All 600 entities under the policy have valid linkage.
17 entities are ‘opt-out’ marked (link N/A).
0 entities lack any linkage.
0 entities have invalid linkage.
foaf:isPrimaryTopicOf/page usage is 244/356.
Thus, LLM-based linkage assessment, human-in-the-loop proposal approval, and Sparql-based ontology modification result in a fully satisfactory ontology with respect to the Wikipedia Linkage Policy.
SLE Knowledge Resources
We identify external knowledge resources for systematic review and maintenance of an SLE domain ontology. We seek resource categories and concrete representatives providing evidence about terminology, conceptual organisation, constraints, individuals, and relations.
We distinguish seven resource categories plus supporting tools. Categories may overlap; placement reflects their primary review contribution. The demonstrations from Sec. 2 are included among the representatives. For brevity, we give one representative per category; for additional representatives.
Technology Inventories and Ecosystem Metadata
These resources describe concrete technologies such as languages, tools, implementations, and packages, including registries, popularity indices, package catalogues, and services tracking dependencies, releases, compatibility, and support.
Representative: TIOBE Index5: programming languages with monthly popularity indicators.
Review contribution: Identify candidate language instances and prioritise coverage checks.
Preparation: Extract language lists and reconcile names with ontology identifiers.
Qualification: Popularity informs completeness; nonpopularity does not imply incorrectness (irrelevance).
Encyclopaedias and Reference Works
These resources describe software languages, features, and related concepts through explanatory text and structured fields, providing evidence for linking concrete languages to ontology concepts.
Representative: Wikipedia articles on individual programming languages6, including descriptions of language features and paradigms.
Review contribution: Identify concepts evidenced by language descriptions and review their association with the corresponding ontology language.
Preparation: Match articles to language entities, extract relevant statements, and map terminology to ontology concepts.
Qualification: Concept associations require contextual interpretation; an unmentioned feature cannot be assumed absent.
Standards and Specifications
These resources define languages, formats, interfaces, and conformance requirements, providing version-specific evidence about constructs, semantics, and constraints.
Representative: ECMA-262 [4], the ECMAScript language specification7.
Review contribution: Check language terminology, constructs, and specified properties.
Preparation: Select an edition and map specification terms and rules to ontology elements.
Qualification: Normative definitions must be distinguished from implementation behaviour and extensions.
Bodies of Knowledge
These resources synthesise a discipline’s accepted knowledge into areas and topics, supporting review of disciplinary scope and the organisation of engineering activities and methods.
Representative: SWEBOK [3]8: software engineering knowledge areas and their decomposition.
Review contribution: Review coverage and organisation of SE areas and activities.
Preparation: Match areas and topics to ontology concepts; retain provenance of derived RDF representations.
Qualification: Topic decomposition does not itself establish activity subclassing; derived representations require source checking.
Taxonomies and Controlled Vocabularies
Classifications of language paradigms, features, processors, and formal foundations provide terms and relationships for reviewing SLE concepts; broad classifications are considered through selected branches.
Representative: The language-related branches of ACM CCS [2]9, particularly general programming languages and compilers under “Software notations and tools”.
Review contribution: Compare coverage and organisation of language types, features, and implementation concepts.
Preparation: Delimit selected branches, match terms to ontology concepts, and interpret their relationships.
Qualification: Subject-indexing hierarchies do not automatically express subclassing; selected branches provide only partial SLE coverage.
Research Literature and Bibliographic Resources
SLE publications provide evidence about language design, implementation, evolution, methods, and tools. Venue-defined collections offer focused starting points, complemented by topic-based searches.
Representative: Proceedings of the International Conference on Software Language Engineering (SLE)10, indexed by dblp11.
Review contribution: Identify language-engineering concepts, methods, and tools, and publications substantiating their properties and relationships.
Preparation: Select proceedings years, screen papers for relevance, and extract full-text evidence.
Qualification: Titles support discovery, not substantive claims; a single venue does not exhaust SLE research.
Software Artifacts and Empirical Collections
Implementations, examples, test suites, benchmarks, corpora, and datasets provide concrete evidence of language features and engineering practices.
Representative: Computer Language Benchmarks Game12: benchmark tasks, programs, and measurements.
Review contribution: Identify languages and implementations and inspect concrete examples of their use.
Preparation: Identify languages, implementations, and features evidenced by selected programs; map them to ontology entities and concepts for comparison with existing assertions or candidate additions.
Qualification: A program may witness a feature’s use, but attribution to the language requires distinguishing language constructs from library facilities and implementation extensions. Absence from the collection does not establish lack of support.
Evidence Processing and Interlinking Tools
These tools support extraction, verification, aggregation, normalisation, and reconciliation while keeping evidence attributable to its underlying sources.
Representative: OpenRefine13: data cleaning, transformation, and reconciliation.
Review contribution: Prepare resource records and match entities across resources and the ontology.
Preparation: Import records and configure transformations and a suitable reconciliation service.
Qualification: Candidate matches require validation; reconciliation does not independently substantiate domain claims.
Related Work
Ontology and knowledge-graph evaluation. Ontology evaluation encompasses structural, functional, and usability criteria [6], as well as comparison with reference ontologies, corpora, tasks, and explicit criteria [16]. OOPS! detects recurrent modelling pitfalls in ontologies [14]. RDFUnit derives executable quality tests from schemas and test patterns, tracks their coverage, and supports repeated evaluation of Linked Data [9]; SHACL checks RDF graphs against declared constraints [8]. These approaches are relevant to the executable and coverage-oriented aspects of COATER. COATER’s proposed assessment differs in its reference point: an arrow compares a selected part of a maintained ontology with a particular external domain resource, whose scope and authority must be judged before a support or challenge is interpreted. Consequently, a failed constraint, absence from a resource, and contrary domain evidence should not be treated as equivalent findings.
External evidence and ontology change. Ontology debugging and completion distinguish defect detection from repair, including the problem of choosing appropriate repairs [10]; knowledge-graph refinement also considers error detection and completion [13]. Especially close to COATER, Evolva discovers candidate knowledge from heterogeneous external sources and uses background knowledge to support ontology adaptation and the management of changes [20]. Process-oriented accounts of ontology evolution likewise include detecting changes and validating their implementation [19]. COATER builds on this line of research by organising systems of knowledge extraction and ontology review components in a process; a review can support an existing axiom, challenge one, or motivate completion or restructuring, without committing every extracted statement to the ontology.
Reconciliation and AI-assisted review. Correspondences between resource entities and ontology entities are a prerequisite for many reviews: ontology matching studies such correspondences [5], and link-discovery systems such as Silk find and maintain links between data sources [17]. These methods can help implement an arrow, but a match alone does not verify a domain assertion or establish that a Wikipedia article is an appropriate primary topic for an ontology entity. Recent LLM-assisted ontology evaluation includes OE-Assist, which investigates automated and human-assisted verification of competency questions [12]. COATER instead uses resource-grounded assessments of ontology content, including TBox and ABox assertions; its Wikipedia and SWEBOK arrows illustrate this design. LLM-produced explanations and confidence values remain candidate evidence for review, not independently verified guarantees of correctness.
Workflow and traceability. OnToology automates ontology engineering support activities in repository workflows [1], while ROBOT supports automated ontology workflows [7]. PROV-O supplies terms for recording entities, activities, agents, and their relationships [11]. Such tools and representations can support the execution and traceability of COATER arrows. The proposed contribution is the organization of heterogeneous domain resources into recurring, scoped reviews whose assessments can be linked to ontology elements and considered as revision tasks. The present case study illustrates this process; the reliability of assessments and the effort required for human triage remain questions for further evaluation.
Conclusion
We have presented COATER, a method for continuous ontology assurance through external resources. Executable, resource-scoped arrows support or challenge ABox and TBox axioms, generate revision tasks, and enable per-axiom assurance coverage. In the case study based on the FSL ontology, arrows review, for example, language coverage, language assertions, software-engineering activities, and linkage policy. We illustrated COATER-based improvement of assurance in an LLM-assisted, human-triaged Wikipedia-linkage exercise which raised compliance with the defined policy from 405 to 600 of 600 entities. We presented a catalogue of SLE resources describing the review landscape comprehensively.
Future work is concerned with advanced process tooling, visualisation of coverage (assurance), cross-validation of arrows, and measurement or optimization of human triage effort over time.
Acknowledgment
Generative AI tools were used for editorial assistance with the manuscript and to support literature discovery alongside searches of scholarly databases. They were not used in the study design, data collection, analysis, interpretation, or any other methodological aspect of the underlying research. The author(s) take(s) full responsibility for the content of this paper.
References
- [1]Ahmad Alobaid, Daniel Garijo, María Poveda-Villalón, Idafen Santana-Pérez, Alba Fernández-Izquierdo, and Oscar Corcho. 2019. Automating Ontology Engineering Support Activities with OnToology. Journal of Web Semantics 57, Article 100472 (2019). doi:10.1016/j.websem.2018.09.003DOI
- [2]Association for Computing Machinery. 2012. The 2012 ACM Computing Classification System. Online classification scheme. Accessed 15 September 2026. https://www.acm.org/publications/class-2012
- [3]Pierre Bourque and Richard E. Fairley (Eds.). 2014. Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 3.0. IEEE Computer Society. Ecma International. 2026. ECMAScript 2026 Language Specification. Standard ECMA-262. Ecma International. 17th edition. https://262.ecma-international.org/17.0/
- [4]Ecma International. 2026. ECMAScript 2026 Language Specification. Standard ECMA-262. Ecma International. 17th edition. https://262.ecma-international.org/17.0/
- [5]Jérôme Euzenat and Pavel Shvaiko. 2013. Ontology Matching (2 ed.). Springer, Heidelberg. doi:10.1007/978-3-642-38721-0
- [6]Aldo Gangemi, Carola Catenacci, Massimiliano Ciaramita, and Jos Lehmann. 2006. Modelling Ontology Evaluation and Validation. In The Semantic Web: Research and Applications (Lecture Notes in Computer Science, Vol. 4011). Springer, 140–154. doi:10.1007/11762256_13DOI
- [7]Rebecca C. Jackson, James P. Balhoff, Eric Douglass, Nomi L. Harris, Christopher J. Mungall, and James A. Overton. 2019. ROBOT: A Tool for Automating Ontology Workflows. BMC Bioinformatics 20, Article 407 (2019). doi:10.1186/s12859-019-3002-3DOI
- [8]Holger Knublauch and Dimitris Kontokostas. 2017. Shapes Constraint Language (SHACL). W3C Recommendation. World Wide Web Consortium. https://www.w3.org/TR/2017/REC-shacl-20170720/
- [9]Dimitris Kontokostas, Patrick Westphal, Sören Auer, Sebastian Hellmann, Jens Lehmann, Roland Cornelissen, and Amrapali Zaveri. 2014. Test-driven Evaluation of Linked Data Quality. In Proceedings of the 23rd International Conference on World Wide Web. 747–757. doi:10.1145/2556195.2568002DOI
- [10]Patrick Lambrix. 2023. Completing and Debugging Ontologies: State of the Art and Challenges. Journal of Data and Information Quality (2023). doi:10.1145/3597304DOI
- [11]Timothy Lebo, Satya Sahoo, Deborah McGuinness, Khalid Belhajjame, James Cheney, David Corsar, Daniel Garijo, Stian Soiland-Reyes, Stephan Zednik, and Jun Zhao. 2013. PROV-O: The PROV Ontology. W3C Recommendation. World Wide Web Consortium. https://www.w3.org/TR/2013/REC-prov-o-20130430/
- [12]Anna Sofia Lippolis, Mohammed Jawad Saeedizade, Robin Keskisärkkä, Aldo Gangemi, Eva Blomqvist, and Andrea Giovanni Nuzzolese. 2025. Large Language Models Assisting Ontology Evaluation. arXiv:2507.14552 [cs.AI]. https://arxiv.org/abs/2507.14552
- [13]Heiko Paulheim. 2017. Knowledge Graph Refinement: A Survey of Approaches and Evaluation Methods. Semantic Web 8, 3 (2017), 489–508. doi:10.3233/SW-160218DOI
- [14]María Poveda-Villalón, Asunción Gómez-Pérez, and Mari Carmen Suárez-Figueroa. 2014. OOPS! (OntOlogy Pitfall Scanner!): An On-line Tool for Ontology Evaluation. International Journal on Semantic Web and Information Systems 10, 2 (2014), 7–34. doi:10.4018/ijswis.2014040102
- [15]Tyler Proctor, Anton Kiselev, and Omar Ochoa. 2023. A Semi-Automated Process for Extracting a Taxonomized Knowledge Graph from the Software Engineering Body of Knowledge with a Sequence-to-Sequence Transformer. SSRN Electronic journal (2023). doi:10.2139/ssrn.4493644
- [16]Joe Raad and Christopher Cruz. 2015. A Survey on Ontology Evaluation Methods. In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2015) – KEOD. SciTePress, 179–186. doi:10.5220/0005591001790186
- [17]Julius Volz, Christian Bizer, Martin Gaedke, and Georgi Kobilarov. 2009. Silk: A Link Discovery Framework for the Web of Data. In Proceedings of the 2nd Workshop about Linked Data on the Web (CEUR Workshop Proceedings, Vol. 538). https://ceur-ws.org/Vol-538/ldow2009_paper13.pdf
- [18]Hironori Washizaki (Ed.). 2024. Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0. IEEE Computer Society. Version 4.0, https://github.com/ashishkati53/swe-knowledge/blob/main/body-of-knowledge/SWEBOK/
- [19]Fouad Zablith, Grigoris Antoniou, Mathieu d’Aquin, Giorgos Flouris, Haridimos Kondylakis, Enrico Motta, Dimitris Plexousakis, and Marta Sabou. 2015. Ontology Evolution: A Process-Centric Survey. The Knowledge Engineering Review 30, 1 (2015), 45–75. doi:10.1017/S0269888913000349
- [20]Fouad Zablith, Marta Sabou, Mathieu d’Aquin, and Enrico Motta. 2009. Ontology Evolution with Evolva. In The Semantic Web: Research and Applications (Lecture Notes in Computer Science, Vol. 5554). Springer, 908–912. doi:10.1007/978-3-642-02121-3_80DOI
