betaFeedback

WorksComputer Science

Measuring the Novelty of Biomedical Papers Using the Latent Distances between Knowledge Units

Abstract

Measuring the novelty of scientific papers is a central concern in research evaluation and scientometrics. From a recombination perspective, prior studies have largely focused on the co-occurrence of knowledge units to assess the novelty of scientific papers. However, these studies often overlook other relationships between knowledge units. This narrow view may result in inaccurate or incomplete evaluations of novelty for scientific papers. To fill this gap, this study introduces a comprehensive novelty measurement that incorporates three types of relationships between knowledge units: network, semantic, and hierarchical. These relationships are used to quantify the latent distances among knowledge units. Using a dataset of 142,036 articles published in PLoS ONE and a validation dataset from the H1 Connect platform, our results demonstrate that (1) each relationship type captures distinct latent distances between MeSH terms; (2) compared to the widely used indicators proposed by Uzzi et al. (2013), our measures show stronger alignment with peer judgements; and (3) combining all three distance metrics yields more effective identification of novel papers than using any single perspective alone.

Introduction

Novelty stands as a crucial criterion in the innovative evaluation of academic papers, and its measurement has attracted significant attention from the scholarly community [1, 2]. Although peer review remains the gold standard for assessing scientific novelty, the process itself has drawn substantial criticism. First, peer reviewers’ conservative tendencies frequently impede the fair assessment of highly novel work [3]. Second, despite the rapid growth in scholarly articles, the pool of available reviewers has not expanded proportionally, which may compromise the quality of peer assessment [4]. Third, peer review is inherently subjective as a method for assessing novelty, and its outcomes often lack consistency due to reviewers’ cognitive biases and varying expertise [5]. To overcome these limitations, several efforts have been devoted to developing automated measures of scientific novelty for academic publications.

Scholars argue that novelty does not emerge from ex nihilo but rather from the recombination of existing knowledge units [6, 7]. In studies measuring novelty, knowledge units are typically operationalized through various sources, including referenced journals [6, 8], keywords [9], knowledge entities [10, 11], and sentences [12]. Although prevailing studies have used the unusualness of journal pairs in reference lists to quantify scientific novelty, some scholars argue that referenced journals are not an appropriate source for capturing fine-grained scientific knowledge [4, 12]. A distinct line of criticism further contends that such reference-based measures may reflect interdisciplinarity rather than scientific novelty [13]. These limitations collectively underscore the need for developing content-based approaches to novelty measurements.

While several studies have proposed novelty indicators based on paper titles, titles may not fully capture the knowledge contained in scientific publications [1, 12]. Consequently, scholars have increasingly used keywords or knowledge entities as proxies for knowledge units [14]. Existing novelty measures based on these elements primarily rely on co-occurrence relationships. However, knowledge units may also be related through semantic relationships and hierarchical structures, suggesting that co-occurrence alone may not fully capture the relationships among knowledge units and may therefore provide an incomplete assessment of scientific novelty [15]. For instance, in the deep learning era, Convolutional Neural Network (CNNs) became a key approach for image classification tasks due to their exceptional feature extraction capabilities [16].

Prior studies have examined knowledge combinations from three perspectives: co-occurrence, semantic relationships, and hierarchical structures [9, 17, 18]. The co-occurrence perspective defines relationships based on the appearance of distinct knowledge units within the same publication [9]. The semantic perspective captures the conceptual distance between knowledge units through semantic analysis [17]. The hierarchical perspective draws on the structure of knowledge classification systems, reflecting domain-specific organization and expert knowledge [18]. These perspectives capture different aspects of relationships among knowledge units and may provide complementary information for novelty measurement.

Nevertheless, existing novelty indicators typically focus on only one perspective and rarely integrate all three. As a result, current approaches may not fully represent the complexity of knowledge combinations in scientific research.

To this end, we propose a novel and comprehensive method for measuring the novelty of academic papers. Using MeSH (Medical Subject Headings) terms as proxies for knowledge units, we define novelty as the atypical combination of prior knowledge (i.e., MeSH terms). Our approach integrates three distance metrics, including network, semantic, and hierarchical, to estimate the distance between MeSH term pairs. We applied our novelty measures to 142,036 papers from PLoS One and demonstrated that the three perspectives capture distinct aspects of distance in knowledge combinations. To validate the method, we compared our results with faculty-assessed novelty from the H1 Connect platform and benchmarked our measure against Uzzi et al.’s [6] pioneering approach, confirming its superiority.

Knowledge recombination as a source of scientific novelty

Schumpeter, a pioneer in innovation studies, posits that “innovation combines factors in a new way, or that it consists in carrying out new combinations” [19]. Translating this combinatorial logic into the cognitive realm, the creation of scientific novelty is fundamentally driven by cognitive search and recombination processes [20]. In this context, the novelty potential of scientific research derives from the cognitive distance bridged between diverse knowledge domains or conceptual bases[21]. Heterogeneity in scholars’ developmental trajectories and training environments leads to differences in cognitive frameworks, resulting in variation in how they interpret, understand, and evaluate external knowledge and phenomena[22]. Consequently, cognitive distance refers to the divergence in cognitive frames or knowledge space that determines how readily actors can comprehend and integrate each other's knowledge[23].

In the complex landscape of science, knowledge is not organized along a single form, but is simultaneously structured relationally, conceptually, and taxonomically [18, 24, 25]. Such organizational forms jointly shape the structure of the knowledge space and influence scholars’ perceptions of the distance between knowledge elements, thereby affecting the novelty of scientific discoveries. This can be attributed to the recombinant nature of scientific discovery and the associated knowledge search processes that underlie it [26]. First, co-occurrence knowledge networks provide a relational representation of knowledge structures by transforming patterns of joint appearance among entities into a network of associations that reflect the collective knowledge of scientific communities. Distance in this network operationalizes relational separation or structural search scope, and the recombination of knowledge units at large network distances resembles a breadth-first search process for identifying novel opportunities for knowledge recombination [27]. Second, semantic distance measures conceptual coherence and determines whether structurally distant units can be cognitively integrated into a meaningful configuration. Even when structural links exist, excessive semantic mismatch may hinder successful conceptual blending[25]. Third, a taxonomy is a hierarchical classification schema that organizes entities based on either their intrinsic properties or their relationships within a domain, and it typically reflects the structured knowledge and cognitive judgments of domain experts [18]. Distance within a knowledge taxonomy captures path-based separation in controlled taxonomies, encompassing both vertical movements across levels of abstraction and horizontal movements across branches. Such distance is associated with cognitive reconfiguration in expert knowledge representations. Collectively, these three distances act as interrelated cognitive filters shaping the knowledge search and knowledge combination process, suggesting that the novelty of scientific publications should be understood through multiple dimensions of cognitive distance.

From a practical perspective, recent studies in knowledge representation and downstream prediction tasks have demonstrated the effectiveness of integrating multiple heterogeneous information sources to better capture complex relationships among knowledge elements. For instance, Ding and Jin show that incorporating corpus-based co-occurrence information, ontology-derived hierarchical structures, and semantic predications can significantly improve the quality of MeSH term representation and enhance performance in downstream tasks such as link prediction [28]. Their results indicate that different data sources provide complementary views of the knowledge space, including relational, semantic, and hierarchical information. Similarly, Liu et al. [27] propose a multi-view heterogeneous hypergraph framework for knowledge combination prediction, in which co-occurrence, co-citation, and hierarchical structure are jointly modeled to capture different aspects of knowledge interactions. Their findings further suggest that integrating multiple structural views of knowledge can improve the accuracy of predicting future knowledge combinations, as each view reflects distinct but complementary mechanisms of knowledge organization. These studies highlight that integrating multiple views of knowledge yields a more comprehensive understanding of how knowledge elements are related and recombined. This provides a strong rationale for quantifying scientific novelty through multiple dimensions of cognitive distance, as different distance measures capture complementary aspects of knowledge organization.

To rigorously quantify these distinct cognitive dimensions, a structured and formal representation of scientific knowledge is required. Scientific knowledge is objective, falsifiable, and evolutionary in nature, often organized in a hierarchical structure [29, 30, 31]. The MeSH thesaurus conforms to this characterization and is widely used as a proxy for scientific knowledge, serving as a common basis for assessing the novelty of scientific articles [32, 33]. Updated annually, MeSH thesaurus incorporates new descriptors, removes obsolete concepts, revises scope notes, and restructures hierarchical branches to reflect the evolving landscape of biomedical knowledge [34]. Because MeSH thesaurus is curated by domain experts, it ensures terminological consistency and reduces semantic ambiguity. The combination of MeSH terms assigned to scientific articles thus represents how authors strategically integrate existing bodies of knowledge and provides a meaningful indication of an article’s novelty. In practice, this operational approach has been widely adopted to quantify scientific novelty [33, 35]. Overall, both theoretically and practically, the MeSH thesaurus serves as a robust and reliable foundation for capturing the novel contributions embedded in scientific articles.

Quantification of scientific novelty

With the advancement of technologies such as natural language processing and complex networks, some scholars have begun developing automated methods to assess the novelty of academic papers. Current approaches to measuring the scientific novelty of academic papers fall into two main categories: reference-based and content-based novelty measures.

From the combinatorial perspective of scientific progress, scientific novelty can be viewed as the recombination of prior knowledge in unprecedented ways [36, 37]. Building on this view, Uzzi et al. [6] treated referenced journals as proxies for knowledge units, quantifying novelty through a Z-score measure based on the atypicality of journal combinations in a paper’s reference list. Lee and colleagues [8] subsequently refined this method by computing the expected co-occurrence probability of journal pairs within a given year, using the ratio of observed to expected co-occurrences as a novelty indicator. This method reduced computational cost compared to that of Uzzi et al. [6] method. Wang et al. [37] further advanced this method by employing historical co-occurrence data to construct a journal co-occurrence matrix and derive vector representations for journals. The authors assessed novelty through cosine similarity between journal vectors, where lower similarity indicated greater novelty.

Recently, scholars have explored using textual information, such as keywords and knowledge entities, to represent scientific knowledge. For instance, Mishra and Torvik [38] developed a novelty measure based on MeSH term combinations, incorporating term age and frequency. Yan et al. [9] quantified a paper’s novelty by computing the ratio of new keyword pairs. New keyword pairs are defined as unique combinations previously unobserved in the given research field. Similarly, Liu et al. [17] adopted bio-entities as proxies for knowledge units and proposed an entity-based novelty measure, where a paper’s novelty was determined by the ratio of novel bio-entity pairs identified via semantic distance thresholds. Luo et al. [10] utilized combinations of question and method entities to quantify temporal and semantic novelty of academic papers. Temporal novelty was measured by the age and frequency of entity pairs, while semantic novelty was computed by the average of one minus the highest cosine similarity between question entities, method entities, and their pairs. Ruan et al. [35] argued that MeSH terms serve as a more suitable proxy than references for assessing novelty, proposing a measure based on the ratio of observed to expected frequency of MeSH term pairs to determine scientific papers’ novelty. Chen et al. [39] combined heterogeneous hypergraph learning with large language models and proposed a model named H2GLM to predict knowledge recombination, which opens a new avenue for the quantification of scientific novelty.

Although several novelty measures have been proposed, these methods still exhibit certain limitations. First, reference-based novelty measures rely on cited journals, which may not capture fine-grained knowledge, and thus fail to assess true novelty accurately [1, 13]. Second, author-provided keywords and extracted entities often consist of non-standardized terms, introducing noise into novelty measurements [35]. Third, most existing studies focus solely on the co-occurrence frequency of knowledge combinations while neglecting other vital relationships between them. Therefore, in this study, we used the MeSH terms as knowledge units. From a recombination lens, we introduced a new approach to measure the novelty of academic papers by incorporating the network distance, semantic distance, and hierarchical distance of MeSH term pairs.

Data and methodology

In this section, we describe the dataset employed in our empirical analysis. Next, we present our proposed novelty measurement for academic papers. Finally, we introduce the validation method used to assess the effectiveness of this measurement.

Data collection

Dataset for empirical analysis

This study collected 142,036 research articles published in PLoS ONE between 2007 and 2015, all stored in XML format. We parsed these files to extract metadata, including titles, DOIs, authors, volume/issue numbers, and other relevant information. Since the PLoS ONE dataset did not include MeSH terms, we integrated it with the PubMed Knowledge Graph (PKG) to obtain MeSH annotations for each article [40]. The linkage process involved two key steps: Firstly, we converted DOI to the PubMed ID (PMID) for each article using the ID Converter tool2 from the U.S. National Library of Medicine (NLM). Secondly, we matched PLoS ONE articles with PKG entries based on their PMIDs. After processing, 142,034 articles were successfully mapped to the PKG and retained for analysis.

Following the practice of our previous study (Zhao et al., 2024), this study identified each article’s discipline using the NSF subject classification system. Of the total dataset, 128,750 articles (90.65%) contained identifiable disciplinary information. As shown in Table 1, although PLoS ONE is a multidisciplinary journal, its publications are predominantly concentrated in three fields, including clinical medicine, biomedical research, and biology, which collectively account for 91.54% of all papers.

Discipline# of publicationsRatio (%)
Clinical Medicine61,52347.78
Biomedical Research43,63933.89
Biology12,6969.86
Psychology3,1862.47
Earth and Space1,7051.32
Health1,2090.94
Physics1,1440.89
Engineering and Technology1,1160.87
Social Sciences1,0050.78
Chemistry6410.50
Professional Fields4640.36
Mathematics4000.31
Humanities210.02
Arts10.00
Total128,750100

Table 1. The disciplinary distribution of PLoS ONE papers.

Dataset for measurement validation

Following the practice of previous studies [12, 43], this study evaluated the effectiveness of our proposed novelty measurement by comparing its agreement with expert assessments from the H1 Connect platform3. In addition, prior research has demonstrated that pioneer novelty measurement proposed by Uzzi et al. [6] outperforms those introduced by Wang et al. [37] and Lee et al. [8] in assessing scientific novelty [43]. Building on this finding, we further conducted a comparative analysis between our proposed measurement and the Uzzi et al.’s [6] approach.

H1 Connect (formerly known as F1000Prime) is a biomedical article recommendation platform where experts conduct post-publication evaluations and provide recommendations. Additionally, each recommended article was assigned one or more specific tags to imply the justification for its recommendation by the experts. The tags have nine types, as shown in Table 2. Following Bornmann et al. [43], we considered papers labeled with the tags “Technical advance”, “New Finding”, “Hypothesis”, and “Novel Drug target” as novel papers.

TagDefinition
Hypothesisarticle presents an interesting hypothesis
New findingarticle presents original data, models or hypotheses
Novel drug targetarticle suggests new targets for drug discovery
Technical advancearticle introduces a new practical/theoretical technique, or novel use of an existing technique
Confirmationarticle validates previously published data or hypotheses
Good for teachingkey article in a field and/or is well written
Negative/null resultsarticle has null or negative findings
Refutationarticle disproves previously published data or hypotheses
Controversialarticle challenges established dogma

Table 2. Tags assigned to recommended papers in H1 Connect

This study utilized web crawling to retrieve PLoS ONE articles endorsed by faculty members on the H1 Connect platform. We identified 2,036 faculty-recommended articles published between 2007 and 2015, including 1 perspective, 1 review, and 2,034 research articles. Since our validation approach required disciplinary classifications, issue/volume metadata, and recommendation tags, articles lacking these criteria were excluded, yielding a final dataset of 1,837 articles.

The novelty measure proposed by Uzzi et al. [6] is computationally intensive to calculate. We therefore obtained this measure (hereafter termed “novelty score U”) from the SciSciNet dataset [44], which was constructed using the Microsoft Academic Graph (MAG). By linking our PLoS ONE dataset with SciSciNet using DOIs, we acquired precomputed novelty scores for 128,120 PLoS ONE articles.

Novelty measurement of scientific papers using latent distances between pairs of MeSH terms

MeSH is a controlled vocabulary system developed by the U.S. National Library of Medicine for indexing life science literature [45]. The MeSH system comprises four types4: (1) Main Headings, which describe the primary subject of the publication; (2) Subheadings that qualify and refine Main Headings; (3) Supplementary Concept Records that complement the main vocabulary; and (4) Publication Characteristics that describe publication types or study attributes. Main Headings have a hierarchical structure and are organized into 16 categories, including Category A (Anatomy), Category B (Organisms), and others. Following previous practice [15, 28, 35, 46], this study focuses solely on Main Headings (hereafter, MeSH terms) for subsequent analysis.

From a recombination perspective, novel knowledge combinations can be viewed as distant or unusual pairings of existing knowledge units. Greater distances between knowledge units in the knowledge space indicate more novel combinations. Papers with more novel combinations exhibit higher novelty [4]. In this study, we used MeSH terms as proxies for knowledge units and quantified the novelty of MeSH combinations by comprehensively considering their network, semantic, and hierarchical distances. The detailed process for calculating the novelty of academic papers is described in this section. It is worth noting that MeSH terms were originally developed for indexing biomedical literature. Consequently, the proposed novelty metric is most suitable for applications within the biomedical domain, and its applicability to other disciplines may be limited. Nonetheless, the underlying approach is adaptable to fields that use similarly structured hierarchical classification systems, although its performance in those settings would need to be validated.

Comprehensive distances between MeSH terms

For the MeSH combination (MeSHn−1,MeSHn)(MeSH_{n-1}, MeSH_n), this study calculates three distance metrics including network distance dnetworkd_{network}, semantic distance dsemanticd_{semantic}, and hierarchical distance dhierarchicald_{hierarchical}. These distance metrics are then integrated into a comprehensive distance measure dcomprehensived_{comprehensive}. The calculation formula is shown in (1).

dcomprehensive(MeSHn−1,MeSHn)=α∗dnetwork(MeSHn−1,MeSHn)+β∗dsemantic(MeSHn−1,MeSHn)+(1−α−β)∗dhierarchical(MeSHn−1,MeSHn)(1)d_{comprehensive}(MeSH_{n-1},MeSH_n) = \alpha* d_{network}(MeSH_{n-1},MeSH_n) + \beta* d_{semantic}(MeSH_{n-1},MeSH_n) + (1-\alpha-\beta) * d_{hierarchical}(MeSH_{n-1},MeSH_n) \tag*{(1)}

Where α\alpha, β\beta, and 1−α−β1-\alpha-\beta represent the weights assigned to the three distance metrics, respectively. The detailed weight calculation method is described in section 3.2.2.

1) Network distance

To measure the network distance between a pair of MeSH terms, denoted as dnetwork(MeSHn−1,MeSHn)d_{network}(MeSH_{n-1},MeSH_n), we first constructed a MeSH term co-occurrence network using articles from the PKG dataset [40]. In this network, each node represents a MeSH term, and two terms are connected by an edge if they co-occur in the same article. The edge weight represents their co-occurrence frequency. We then used the LINE algorithm to generate a vector representation for each MeSH term [47]. LINE preserves both local and global network structures and offers faster training speed than alternatives like DeepWalk and Node2Vec [47]. The distance was calculated as the cosine distance between embedding vectors of MeSH terms, as defined by (2).

dnetwork(MeSHn−1,MeSHn)=1−similarity(MeSHn−1,MeSHn)=1−Net(MeSHn−1)⋅Net(MeSHn)∥Net(MeSHn−1)∥∥Net(MeSHn)∥(2)d_{network}(MeSH_{n-1},MeSH_n) = 1 - similarity(MeSH_{n-1},MeSH_n) = 1 - \frac{Net(MeSH_{n-1}) \cdot Net(MeSH_n)}{\lVert Net(MeSH_{n-1})\rVert\lVert Net(MeSH_n)\rVert} \tag*{(2)}

Where the cosine similarity between MeSH terms is defined as similarity(MeSHn−1,MeSHn)similarity(MeSH_{n-1},MeSH_n), and Net(MeSHn−1)Net(MeSH_{n-1}) and Net(MeSHn)Net(MeSH_n) denote their respective node embedding vectors obtained from the graph embedding model. To calculate network distance, we utilized a dataset of 23,956,566 publications from the PKG dataset published between 1970 and 2019 to construct a MeSH co-occurrence network. The resulting network consists of 29,411 nodes and 45,676,561 edges. We then applied the LINE algorithm to generate a 200-dimensional vector representation for each MeSH term. The dimensionality of the embeddings was set to 200, following the recommendation of Tang et al. [47].

Semantic distance

To quantify the semantic distance between MeSH terms, this study employs ChatGPT-3.55, a large language model (LLM) recognized for its robust performance in multi-turn dialogue, programming, and text representation. Semantic distance is measured using the model’s learned embeddings of MeSH terms, which encode contextual and semantic information derived from real-world textual data [48]. These embeddings ensure that related terms are mapped proximally in the latent space, while unrelated terms remain distant. The semantic distance between MeSH term pairs is computed using (3).

dsemantic(MeSHn−1,MeSHn)=1−similarity(MeSHn−1,MeSHn)=1−Sem(MeSHn−1)⋅Sem(MeSHn)∥Sem(MeSHn−1)∥∥Sem(MeSHn)∥(3)d_{semantic}(MeSH_{n-1},MeSH_n) = 1 - similarity(MeSH_{n-1},MeSH_n) = 1 - \frac{Sem(MeSH_{n-1}) \cdot Sem(MeSH_n)}{\lVert Sem(MeSH_{n-1})\rVert\lVert Sem(MeSH_n)\rVert} \tag*{(3)}

Where Sem(MeSHn−1)Sem(MeSH_{n-1}) and Sem(MeSHn)Sem(MeSH_n) denote embedding vectors of MeSHn−1MeSH_{n-1} and MeSHnMeSH_n, respectively, derived from the ChatGPT-3.5 model. To calculate semantic distance, we employed ChatGPT-3.5 to obtain 1,536-dimensional embeddings for 22,200 MeSH terms associated with PLoS ONE publications. The embedding dimensionality was set as 1,536 by default and could not be adjusted.

Hierarchical distance

To quantify the hierarchical distance between MeSH terms, we utilized the MeSH term embeddings (named as MeSHHeading2vec) that were trained by Guo et al. [46] based on the MeSH tree structure. Specifically, Guo et al. [46] transformed the MeSH tree into a relational network and employed five graph embedding algorithms to learn vector representations of the terms. Validation results demonstrated that the trained embedding model effectively captures structural information and enhances the representational capacity of the vectors [46]. It should be noted that each MeSH heading may be associated with multiple tree numbers corresponding to different hierarchical positions in the MeSH taxonomy. In the work of Guo et al. [46], each MeSH heading is treated as a unique node in the constructed relationship network. The multiple tree numbers are used only to derive hierarchical relationships by generating parent–child links through iterative truncation of tree number paths. Therefore, the embedding process is performed at the level of MeSH headings, resulting in a single vector representation for each term. The hierarchical distance between MeSH term pairs is computed using (4).

dhierarchical(MeSHn−1,MeSHn)=1−similarity⁡(MeSHn−1,MeSHn)=1−Hie(MeSHn−1)⋅Hie(MeSHn)∥Hie(MeSHn−1)∥∥Hie(MeSHn)∥(4)d_{\mathrm{hierarchical}}(MeSH_{n-1}, MeSH_n) = 1 - \operatorname{similarity}(MeSH_{n-1}, MeSH_n) = 1 - \frac{Hie(MeSH_{n-1}) \cdot Hie(MeSH_n)}{\lVert Hie(MeSH_{n-1}) \rVert\lVert Hie(MeSH_n) \rVert} \tag*{(4)}

Where Hie(MeSHn−1)Hie(MeSH_{n-1}) and Hie(MeSHn)Hie(MeSH_n) denote embedding vectors of MeSHn−1MeSH_{n-1} and MeSHnMeSH_n, respectively, derived from the MeSHHeading2vec. To calculate hierarchical distance, we adopted the 64-dimensional MeSH term embeddings pre-trained by Guo et al. [46]. The embeddings were trained using the MeSH tree structure released in 2019, which includes a total of 29,349 MeSH terms.

It is noteworthy that when the NLM updates the MeSH vocabulary, newly introduced terms are retroactively applied to all previously indexed publications. For example, the term “Heme-Binding Proteins”, introduced in 2020, was retroactively assigned to a 2012 PLoS ONE article titled Heme binding proteins of Bartonella henselae are required when undergoing oxidative stress during cell and flea invasion. To ensure that all MeSH terms associated with PLoS ONE articles had corresponding vector representations, we excluded MeSH terms introduced after 2019, resulting in a final set of 22,183 MeSH terms used in this study.

Entropy weight method for comprehensive distances

To quantify comprehensive distances, we applied the Entropy Weight Method [49] (EWM) to determine the weights (α\alpha, β\beta, and 1−α−β1 - \alpha- \beta, as defined in (1)) for the three distance metrics. This method assigns objective weights based on the dispersion of each metric: a higher dispersion corresponds to a greater weight, and conversely, lower dispersion results in a reduced weight.

We first adopted Min-Max normalization to each distance metric across all MeSH pairs to remove scale differences. For each metric jj and MeSH pair ii, we compute the normalized value bijb_{ij} and the corresponding proportion pijp_{ij} using formula (5):

pij=bij∑i=1mbij(5)p_{ij} = \frac{b_{ij}}{\sum_{i=1}^{m} b_{ij}} \tag*{(5)}

Where mm is number of MeSH pairs. The entropy of metric jj is described as follows:

ej=−k∑i=1mpijln⁡(pij)(6)e_j = -k\sum_{i=1}^{m}p_{ij}\ln(p_{ij}) \tag*{(6)}

Where k=1/ln⁡(m)k = 1/\ln(m), and we stipulate that ej=0e_j = 0 when pij=0p_{ij} = 0. The information entropy redundancy is then defined as dj=1−ejd_j = 1 - e_j, and the weight of each metric can be expressed as follows:

wj=dj∑j=1ndj(7)w_j = \frac{d_j}{\sum_{j=1}^{n}d_j} \tag*{(7)}

Where nn denotes the number of metrics, and weights assigned as w1=αw_1 = \alpha, w2=βw_2 = \beta, and w3=1−α−βw_3 = 1 - \alpha- \beta. The comprehensive distance between MeSH pairs is then computed by (1). Using the EWM, the resulting weights were 0.750 for network distance (α\alpha), 0.073 for semantic distance (β\beta), and 0.177 for hierarchical distance (1−α−β1 - \alpha- \beta).

Novelty measurement of academic papers

The process for calculating a scientific paper’s novelty is illustrated in Figure 1, and procedure designed to be implemented as follows:

The process for measuring the novelty of scientific papers

Figure 1. The process for measuring the novelty of scientific papers

  1. Collect all MeSH terms used in the article and generate all possible term pairs.

  1. Using Eq. (2)-(4), compute the network distance, semantic distance, and hierarchical distance for each MeSH pair.

  1. Weight the three distance metrics using the result of EWM, then calculate the comprehensive distance for each MeSH pair via (1).

  1. We ranked all MeSH term pairs according to their comprehensive distance in descending order, and defined those in the top 10th percentile as novel combinations, following prior studies on combinatorial novelty[6, 8, 37].

  1. For each article, a paper’s novelty is quantified by the proportion of novel MeSH pairs to all possible MeSH combinations, and the formula is defined as:

noveltyq=vCp2(8)novelty_q = \frac{v}{C_p^2} \tag*{(8)}

Where noveltyqnovelty_q represents the novelty of article qq, vv denotes the number of novel MeSH pairs in the paper, and Cp2C_p^2 indicates the total possible MeSH pairs. As illustrated in Figure 1, paper 421456 includes 3 MeSH term pairs, one of which (MeSH_1 and MeSH_2, highlighted in blue) represents a novel combination. Consequently, the paper’s novelty score is computed as 1/3.

Evaluation of novelty measurement

Inspired by Bornmann et al. [43], we compared the novelty score of papers with and without novelty tags, as described in Table 2. The underlying rationale is that papers explicitly labeled as novel should, on average, exhibit higher novelty scores than those without such tags.

For each paper with novelty tags, we identified a matched counterpart (also referred to as control group) from PLoS ONE papers without novelty tags to enhance comparability between the two groups. The matched paper was required to meet the following criteria: it had to be published in the same year, volume, issue, and discipline, and have the same number of authors. When multiple candidates satisfied these conditions, one was randomly selected. To ensure the robustness of our findings, we repeated the matching procedure five times.

Table 3 presents the matching results between PLoS ONE papers from the H1 Connect platform and their control group. “Positive label” indicates that the recommended PLoS ONE paper was tagged with one or more of the following: “Interesting Hypothesis”, “New Finding”, “New Drug Target”, or “Technical Advance”. Notably, some articles received conflicting labels from different experts. For instance, a single paper could be labeled both “Confirmation” and “Interesting Hypothesis”. To mitigate inconsistencies in expert evaluations, such cases were excluded from the matching process.

CategoryPositive LabelInteresting HypothesisNew FindingNew Drug TargetTechnical Advance
Papers recommended by H1 connect87418876484206
Control group papers87418876484206

Table 3. The matching results between PLoS One papers from the H1 Connect platform and their control group.

Note: The table displays outcomes from one matching iteration.

During the validation, we employed least squares regression to estimate the effect of novelty tags on novelty scores. Specifically, we treated the novelty score as the dependent variable and used a binary indicator as the independent variable (coded as 1 for papers with novelty tags and 0 for those without), allowing us to examine the relationship between the novelty indicators and expert evaluations.

Results

Comparative analysis of the three distance metrics

To compare the distribution of MeSH terms under different distance metrics, we applied Uniform Manifold Approximation and Projection (UMAP) to visualize the MeSH term vectors in two-dimensional space and clustered formats [50]. The main parameters were set as n_neighbors=100n\_neighbors = 100 and min_dist=0.1min\_dist = 0.1. The results are shown in Figure 2, where each node denotes a MeSH term, and colors represent its classification into one of 16 MeSH categories.6

The UMAP visualization of MeSH terms

Figure 2. The UMAP visualization of MeSH terms.

The visualization reveals distinct distribution patterns corresponding to the three types of distance metrics. The embeddings based on the MeSH tree structure exhibit relatively well-defined clusters (fig:2C), reflecting their alignment with expert-curated hierarchical classifications. In contrast, the embeddings generated by ChatGPT (fig:2B) exhibit a noticeably greater degree of cluster overlap. Furthermore, the embeddings derived from the co-occurrence network (fig:2A) display more intermixed clusters, which may reflect less clear structural separation. These observations suggest that each distance metric captures different latent dimensions of the relationships between MeSH terms.

To quantitatively assess the differences among the three distance metrics, we applied a series of statistical analyses. First, we computed the distances between MeSH term pairs using three distinct metrics: network-based, semantic, and hierarchical. To compare the distributions of the resulting distance values, we employed the Friedman test [51], which revealed extremely significant differences among the three groups (χ2(2)=2933225.76\chi^{2}(2) = 2933225.76, p<0.001p < 0.001). To examine differences between each pair of distance metrics, we further conducted Wilcoxon signed-rank tests and applied Bonferroni correction to control for multiple testing [52]. As shown in Table 4, the network distance differed significantly from both the semantic distance (Z=−1484.38Z = -1484.38, p<0.001p < 0.001, r=−0.82r = -0.82) and the hierarchical distance (Z=−1203.24Z = -1203.24, p<0.001p < 0.001, r=−0.66r = -0.66). Additionally, the difference between semantic and hierarchical distances was also statistically significant (Z=881.75Z = 881.75, p<0.001p < 0.001, r=−0.49r = -0.49). According to Cohen’s [53] guidelines for interpreting effect sizes in non-parametric tests, the absolute values of the effect sizes indicate large effects for the first two comparisons (∣r∣>0.5\lvert r\rvert> 0.5), and a medium effect for the third comparison (0.2<∣r∣<0.50.2 < \lvert r\rvert< 0.5). These findings suggest substantial and meaningful differences across all three types of distance metrics.

GroupZZ valuePP value(adj)Effect size(rr)
Network distance vs. Semantic distance-1484.380.001-0.82
Network distance vs. Hierarchical distance-1203.240.001-0.66
Semantic distance vs. Hierarchical distance881.750.0010.49

Table 4. The result of pairwise comparisons among three distance metrics

Note: PP value(adj) represents the PP value with Bonferroni correction.

Validation results of novelty measurement

Descriptive analysis of novelty score

Table 5 reports the descriptive statistics of novelty scores for papers recommended by H1 Connect and their corresponding control group papers. On average, papers with positive labels and recommended by the H1 Connect platform have a mean novelty score of 0.138, which is slightly higher than the control group’s mean of 0.135. The median novelty score is also higher for the recommended papers, reaching 0.130, while the control group papers have a median of 0.128. Across specific novelty tags, such as “Interesting Hypothesis”, “New Finding”, “New Drug Target”, and “Technical Advance”, the recommended papers generally show higher mean novelty scores than their counterparts. A similar pattern is observed in the median scores, except for papers labeled “Technical Advance”, which do not display a clear advantage. Overall, these results are consistent with our expectations and provide indirect support for the validity of the novelty measurement.

CategoryDescriptive StatisticsPositive LabelInteresting HypothesisNew FindingNew Drug TargetTechnical Advance
Papers recommended by H1 connectMean0.1380.1570.1410.1560.135
Papers recommended by H1 connectMedian0.1300.1500.1330.1420.123
Papers recommended by H1 connectStd0.0960.1050.0960.0830.104
Papers recommended by H1 connectMin00000
Papers recommended by H1 connectMax0.50.6670.50.3520.436
Papers recommended by H1 connectN437094038204201030
Control group papersMean0.1350.1350.1350.1410.131
Control group papersMedian0.1280.1280.1270.1330.124
Control group papersSD0.0900.0920.0920.0840.095
Control group papersMin00000
Control group papersMax0.60.6670.70.51
Control group papersN437094038204201030

Table 5. The descriptive statistics of novelty scores

Validation results via linear regression modeling

To further assess the validity of our novelty measure, we adopted a linear regression model to examine whether the proposed indicator aligns with peer judgements. Specifically, we tested whether the average novelty scores differ significantly between papers labeled with novelty-related tags and those without such tags. Additionally, we compared the performance of our text-based novelty measure with the reference-based novelty measure developed by Uzzi et al. [6]. The validation results are presented in Table 6. Remarkably, our novelty metric is a positive indicator, whereas the novelty score U is a negative indicator. Accordingly, we expected our novelty scores to be positively associated with novelty-related tags, while the novelty score U would exhibit a negative association.

VariableOur novelty score
(1)(2)(3)(4)(5)
Positive Label0.003 (0.002)
Interesting Hypothesis0.022*** (0.005)
New Finding0.006*** (0.002)
New Drug Target0.015*** (0.006)
Technical Advance0.004 (0.004)
Constant0.135*** (0.001)0.135*** (0.000)0.135*** (0.000)0.141*** (0.000)0.131*** (0.000)
N8740188076408402060
Novelty score U
(6)(7)(8)(9)(10)
Positive Label0.345 (0.440)
Interesting Hypothesis-1.601* (0.850)
New Finding0.814* (0.442)
New Drug Target-2.444*** (0.847)
Technical Advance-1.097 (0.966)
Constant-2.897*** (0.302)-3.188*** (0.529)-2.948*** (0.069)-3.992*** (0.678)-2.491*** (0.529)
N8740188076408402060

Table 6. The validation result using regression models. Note: Robust standard errors in parentheses. *, **, and *** denote significance at the 10%, 5%, and 1% level, respectively.

As shown in Table 6, our novelty scores exhibit positive coefficients for all novelty tags. However, the coefficients for the tags “Positive Label” and “Technical Advance” are not statistically significant. Interestingly, as discussed in Section 4.3, when novelty is measured using the 5th-threshold specification, which applies a more stringent criterion for identifying novel combinations, all novelty tags show significant positive associations with the novelty scores. One possible explanation is that different novelty tags may correspond to different levels of expectation violation [54]. Compared with tags such as “Interesting Hypothesis” or “New Drug Target,” the label “Technical Advance” may represent a higher threshold of novelty. In other words, a study may need to incorporate sufficiently rare and atypical combinations of knowledge elements before it is recognized as a genuine technical advance.

In contrast, the results for novelty score UU are less favorable. Among the four specific novelty tags, only “Interesting Hypothesis,” “New Drug Target,” and “Technical Advance” exhibit coefficients in the expected direction, and only the first two are statistically significant. Furthermore, although novelty score UU is significantly associated with the “New Finding” tag, the direction of the relationship is contrary to expectations.

Overall, the results for our novelty score are mostly consistent with our expectations regarding the relationship between novelty and expert-assigned tags. In other words, the proposed novelty measure aligns well with expert judgment, supporting its validity. Compared to Uzzi’s et al. [6] measure, our measure provides a more valid and reliable reflection of scientific novelty.

Effectiveness of combining different distance metrics in novelty measurement

To further verify whether integrating multiple distance metrics improves the accuracy of scientific novelty quantification, we conducted comparative experiments between novelty measures derived from combined metrics and those based on individual metrics. Following the methodology described in Section 3.2, we computed three distinct novelty measures based on network, semantic, and hierarchical distance separately. Similarly, using the evaluation method outlined in Section 3.3, we estimated the correlation between novelty scores and novelty-related tags. The validation results are shown in Figure 3, and the detailed corresponding regression results are provided in Appendix Table A.1.

Validation of novelty measurements developed using separate distance metrics

Figure 3. Validation of novelty measurements developed using separate distance metrics. Whiskers represent 95% confidence intervals while solid and hollow geometric shapes denote significant and non-significant coefficients, respectively. Colors denote novelty scores based on different distance metrics: blue diamonds correspond to network distance, orange squares to semantic distance, and red circles to hierarchical distance.

VariableNetwork-distance-based novelty score
Variable(1)(2)(3)(4)(5)
Positive Label0.0009 (0.0020)
Interesting Hypothesis0.0195*** (0.0046)
New Finding0.0035 (0.0022)
New Drug Target0.0131** (0.0058)
Technical Advance0.0050 (0.0046)
Constant0.1424*** (0.0014)0.1432*** (0.0032)0.1419*** (0.0015)0.1468*** (0.0042)0.1390*** (0.0031)
N8740188076408402060
VariableSemantic -distance-based novelty score
Variable(6)(7)(8)(9)(10)
Positive Label-0.0023** (0.0011)
Interesting Hypothesis0.0020 (0.0024)
New Finding0.0000 (0.0012)
New Drug Target-0.0015 (0.0035)
Technical Advance-0.0084*** (0.0023)
Constant0.0501*** (0.0008)0.0476*** (0.0017)0.0493*** (0.0009)0.0563*** (0.0026)0.0507*** (0.0017)
N8740188076408402060
VariableHierarchical -distance-based novelty score
Variable(11)(12)(13)(14)(15)
Positive Label0.0020 (0.0012)
Interesting Hypothesis0.0029 (0.0027)
New0.0012
Variable(1)(2)(3)(4)(5)
Finding(0.0013)
New Drug Target0.0100***(0.0037)
Technical Advance-0.0076***
(0.0024)
Constant0.0946***0.0945***0.0956***0.0994***0.0941***
(0.0009)(0.0020)(0.0009)(0.0027)(0.0018)
N8740188076408402060

Table A.1. Table A.1 Validation results of novelty measures developed with distinct distance metrics

In Figure 3, we can observe that only a small number of coefficients are positive and significant, aligning with our expectations. Regarding novelty scores derived from network distance, only two tags, “Interesting Hypothesis” and “New Drug Target” meet these expectations. For novelty scores based on semantic distance, the coefficients for “Interesting Hypothesis” and “New Finding” point in the expected direction, but neither reaches statistical significance. For novelty scores based on hierarchical distance, the tags “Positive Label, “Interesting Hypothesis”, “New Finding” and “New Drug Target” all display positive associations. However, only the coefficient for “New Drug Target” is statistically significant.

Taken together, novelty scores calculated from individual distance metrics align with only a few of our previously formulated expectations. In contrast, novelty scores derived from combined distance metrics show better agreement with novelty-related tags recommended by peers, which implies that integrating different distance measures enhances the ability to characterize the scientific novelty of academic papers.

Robustness tests

In this section, we examine the robustness of our novelty measures. First, although we adopt the 10th percentile of the comprehensive distance as the threshold to distinguish novel from common knowledge combinations, which is consistent with prior studies [6, 8, 17], the choice of threshold may influence the resulting novelty scores of academic papers. To address this concern, we conduct sensitivity analyses using alternative thresholds at the 5th and 15th percentiles. The corresponding results are reported in Table A.2 and Table A.3. As shown in Table A.2, all novelty tags show a positive and statistically significant correlation with our novelty score. Table A.3 presents similar results, and all novelty tags remain positively correlated with the novelty score, although only the coefficients for “Interesting Hypothesis” and “New Drug Target” are statistically significant. Overall, the signs of the coefficients are consistent with those reported in Table 6, while differences in statistical significance are minor. These differences are reasonable, as the percentile threshold can be interpreted as a hyperparameter that determines the operational definition of novelty. Lower thresholds impose a stricter criterion, capturing only highly atypical MeSH term combinations, whereas higher thresholds relax this constraint and include a broader set of more common MeSH term combinations. This, in turn, leads to varying degrees of alignment with novelty tags assigned by human experts. The results further suggest that lower thresholds are more closely aligned with expert judgments. Second, to examine the sensitivity of our novelty measure to the weighting scheme, we constructed an alternative novelty measure by assigning equal weights to the network, semantic, and hierarchical distance dimensions instead of using EWM-derived weights. As reported in Table A.4, the conclusions remain unchanged relative to the main analysis (Table 6), suggesting that our novelty measure is robust to alternative weighting schemes.

Variable(1)(2)(3)(4)(5)
Positive Label0.003**
(0.001)
Interesting Hypothesis0.018***
(0.003)
New Finding0.005**
(0.002)
New Drug Target0.012***
(0.004)
Technical Advance0.006**
(0.004)
Constant0.074***0.075***0.074***0.077***0.074***
(0.001)(0.002)(0.001)(0.003)(0.002)
N8740188076408402060

Table A.2. Validation results of novelty measures under the 5th percentile threshold

Variable(1)(2)(3)(4)(5)
Positive Label0.001
(0.002)
Interesting Hypothesis0.024***
(0.005)
New Finding0.004
(0.003)
New Drug Target0.013**
(0.007)
Technical Advance0.003
(0.005)
Constant0.196***0.197***0.197***0.208***0.189***
(0.002)(0.004)(0.002)(0.005)(0.004)
N8740188076408402060

Table A.3. Validation results of novelty measures under the 15th percentile threshold

Variable(1)(2)(3)(4)(5)
Positive Label0.003
(0.002)
Interesting Hypothesis0.016***
(0.004)
New Finding0.007***
(0.002)
New Drug Target0.018***
(0.005)
Technical Advance0.002
(0.004)
Constant0.109***0.109***0.108***0.116***0.108***
(0.001)(0.003)(0.001)(0.004)(0.002)
N8740188076408402060

Table A.4. Table A.4 Validation results for the equal-weight novelty measure (10th Percentile Threshold)

Case study

To enhance the interpretability of the proposed novelty measure, we selected two articles from radiology and nuclear medicine, which is a subdiscipline of clinical medicine, to demonstrate how our method distinguishes the novelty between these two articles, as shown in Figure 4. Figure 4A presents a low-novelty article titled Robust framework for PET image reconstruction incorporating system and measurement uncertainties, which is indexed with six MeSH terms. Several of these terms are semantically related and belong to the same major branches (i.e., Category L: Information Science) of the MeSH tree, including “Algorithms”, “Computer Simulation”, “Image Processing, Computer-Assisted”, and “Models, Theoretical”. These relationships result in relatively short distances between the MeSH term pairs. Consequently, none of the fifteen MeSH term pairs exceed the threshold value for comprehensive distance and are therefore identified as novel combinations, resulting in a novelty score of zero.

Novelty calculation process for two articles

Figure 4. Novelty calculation process for two articles

In contrast, the high-novelty article shown in Figure 4B, titled Estimation of noise-free variance to measure heterogeneity, is indexed with four MeSH terms. These MeSH terms co-occur with relatively low frequency and are generally located in different branches of the MeSH tree, resulting in comparatively large network and hierarchical distances. For example, the MeSH term “Humans” belongs to the Category B (Organisms), whereas “Image Processing, Computer-Assisted” falls under Category L (Information Science), displaying far hierarchical distance. Moreover, according to our statistics, “Humans” and “Image Processing, Computer-Assisted” co-occur only twice in our dataset, implying a large network distance. As a result, four of the six MeSH term pairs are identified as novel combinations, leading to a much higher novelty score. These contrasting examples illustrate how the proposed method captures differences in conceptual novelty by examining multidimensional distances between the MeSH terms assigned to individual articles.

Discussion

In this section, we will discuss the implications and limitations of our study.

Theoretical implications

Novelty plays a crucial role in innovative scientific publications, and establishing a reliable and accurate method for quantifying novelty holds great significance for the scientific community [55]. From a combinatorial perspective, numerous novelty measures have been developed, ranging from metadata-based to content-based approaches [12, 54]. Content reflects how ideas are described and expressed within disciplines. Therefore, this study treats MeSH terms as knowledge units and proposed a novel content-based approach to quantify the novelty of scientific papers from a combinatorial angle, considering three latent distances: network, semantic, and hierarchical distances between knowledge units.

The main theoretical contribution of this study is the introduction of a comprehensive distance metric between knowledge units for measuring scientific novelty. This study hypothesizes that distinct distance measures can capture different relationships between knowledge units in latent space. The statistical results show that the three types of distance metrics differ significantly from one another, and integrating these metrics offers clear advantages in measuring scientific novelty compared to using the three separate novelty measures. This novelty measure, based on a comprehensive distance metric, expands the scope of combinatorial novelty research and provides a more reliable quantification of a paper’s novelty, avoiding conceiving the process of scientific innovation in a simplified way.

Although our novelty measure draws on MeSH terms from the biomedical domain, it exhibits strong generalizability comparable to other metadata-based approaches. McNamee [18] argued that, to understand the complex phenomenon in the world, entities are often categorized and organized within detailed taxonomic systems. Classification frameworks in science and technology are no exception and typically feature intricate hierarchical architectures. Notable examples includes Dewey Decimal Classification (DDC) [56], the International Patent Classification (IPC) scheme [57], fields of study (FOS) developed by Microsoft Academic Graph [58], Computer Science Ontology (CSO) curated by SKM team [59], as well as Gene Ontology (GO) maintained by gene ontology consortium [60]. Therefore, following our methodology, novelty assessment can be extended to other domains that utilize established hierarchical classification systems, enabling cross-disciplinary comparisons and broadening the applicability of our measure beyond biomedicine.

Practical implications

This study has several practical implications as well. First, although integrating multiple distance metrics to assess the scientific novelty of academic papers has shown promising results, the resulting novelty scores do not fully align with the peer judgements. This discrepancy indicates that automated evaluations cannot replace peer review, which remains the cornerstone of scientific assessment. As stated by Thelwall et al. [61], automatic methods for evaluating novelty in academic research still require further development.

Second, novelty assessment is a crucial component of research quality and should be distinguished from the evaluation of scientific impact. Current evaluation practices for journals, researchers, and institutions rely heavily on metadata-based indicators such as publication counts, citation metrics, and related measures. In this context, our novelty indicator can serve as a valuable supplement to the existing evaluation framework. As many scholars have argued, a robust scientific assessment system should evaluate entities from multiple perspectives and incorporate diverse indicators to ensure a comprehensive, objective, and balanced appraisal [62, 63].

Third, academic search engines can also benefit from our research. Most existing platforms rank search results based on relevance or publication date, which often falls short of helping scholars quickly identify cutting-edge research. By incorporating our novelty indicator and offering an option to sort results by novelty, search engines can significantly enhance the user experience for academic researchers.

Limitations

This study also has several limitations. Firstly, this study validated the proposed novelty measure using academic papers from the biomedical domain. Whether the measure maintains its effectiveness across other fields remains to be further explored. Secondly, as a multidimensional construct, novelty can be interpreted from various perspectives. While our measure focuses on the combinatorial aspect, other dimensions, such as uniqueness, bridging, and surprise, may also offer valuable bases for measuring novelty. Thirdly, although MeSH is an expert-curated and standardized thesaurus that offers advantages for measuring scientific novelty, its relatively low update frequency may lead to an underestimation of the novelty of some papers. Fourth, the representation of each MeSH term as a single node, despite its multiple tree-number assignments, may partially obscure branch-specific hierarchical information and may influence the fine-grained assessment of scientific novelty within the distance-based evaluation framework. Fifth, we employ a global threshold to capture absolute novelty across the entire dataset, which may overlook temporal heterogeneity in knowledge combination structures, as novelty distributions may vary across publication years due to the evolving knowledge base. Sixth, the network distance measure is derived from a MeSH co-occurrence network constructed using publications from 1970 to 2019, whereas the focal articles were published between 2007 and 2015. As a result, post-publication relationships among knowledge elements may be incorporated into the measure, potentially introducing temporal leakage and attenuating estimates of novelty. Future research could address this issue by constructing year-specific knowledge networks to better capture the historical context in which scientific contributions were made.

Conclusion and future work

In this study, we propose a novel approach to measuring the scientific novelty of academic papers from a recombination perspective. Unlike existing keyword- or entity-based measures that typically reflect a single dimension of knowledge recombination, our method incorporates multiple complementary dimensions (i.e., network distance, semantic distance, and hierarchical distance) between knowledge units. We use MeSH terms as proxies for knowledge units and validate the approach using articles from PLOS ONE alongside novelty tags from the H1 Connect platform. The results indicate that these three dimensions capture distinct aspects of the relationships among MeSH terms, and that integrating them offers clear advantages in identifying novel papers compared to using any single indicator alone.

This study also leaves several avenues for future research. First, the proposed novelty measure could be applied to other disciplines to assess its effectiveness and generalizability. A key prerequisite for such validation is the construction of a larger, more reliable, and unbiased ground-truth dataset. Second, further theoretical exploration into the nature of scientific novelty, its origins, dimensions, and underlying mechanisms, would greatly contribute to the refinement, simulation, and generation of novelty in quantitative studies. Third, future work could adopt hypergraph models to capture higher-order combinations of knowledge units, offering a more nuanced understanding of complex recombination patterns and their role in the emergence of novelty. Fourth, the MeSH thesaurus represents a type of human knowledge, and integrating human knowledge with global knowledge such as that encoded in large language models provides an interesting pathway for quantifying scientific novelty [64].

Acknowledgements

This work was supported by the National Social Science Fund of China (No. 24CTQ027). The current study is an extended version of our article [15], which was presented at the 2024 ACM/IEEE Joint Conference on Digital Libraries held in Hongkong, China, from December 16 to 20, 2024. The authors would like to thank the two anonymous reviewers for their insightful comments and valuable suggestions.

Appendix

Note: Robust standard errors in parentheses. *, **, and *** denote significance at the 10%, 5%, and 1% level, respectively.

Note: Robust standard errors in parentheses. *, **, and *** denote significance at the 10%, 5%, and 1% level, respectively.

Note: Robust standard errors in parentheses. *, **, and *** denote significance at the 10%, 5%, and 1% level, respectively.

Note: Robust standard errors in parentheses. *, **, and *** denote significance at the 10%, 5%, and 1% level, respectively.

References

  1. [1]Wang Z, Zhang H, Chen J, et al. An effective framework for measuring the novelty of scientific articles through integrated topic modeling and cloud model. Journal of Informetrics 2024; 18: 101587.
  2. [2]Wang X, Gui H, Liu J. How authors evaluate the novelty of their articles: A comparative analysis of highlights in research articles. Journal of Information Science 2023; 01655515231166924.DOI
  3. [3]Siler K, Lee K, Bero L. Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences 2015; 112: 360–365.
  4. [4]Zhao Y, Zhang C. A review on the novelty measurements of academic papers. Scientometrics 2025; 130: 727–753.DOI
  5. [5]Kumar S, Ghosal T, Ekbal A. When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, pp. 16693–16704.arxiv.org/abs/2310.18685
  6. [6]Uzzi B, Mukherjee S, Stringer M, et al. Atypical combinations and scientific impact. Science 2013; 342: 468–472.
  7. [7]Xiao T, Makhija M, Karim S. A Knowledge Recombinations Perspective of Innovation: Review and New Research Directions. Journal of Management 2022; 48: 1724–1777.
  8. [8]Lee Y-N, Walsh JP, Wang J. Creativity in scientific teams: Unpacking novelty and impact. Research Policy 2015; 44: 684–697.
  9. [9]Yan Y, Tian S, Zhang J. The impact of a paper’s new combinations and new components on its citation. Scientometrics 2020; 122: 895–913.DOI
  10. [10]Luo Z, Lu W, He J, et al. Combination of research questions and methods: A new measurement of scientific novelty. Journal of Informetrics 2022; 16: 101282.DOI
  11. [11]Wu M, Zhang Y, Zhang G, et al. Exploring the genetic basis of diseases through a heterogeneous bibliometric network: A methodology and case study. Technological Forecasting and Social Change 2021; 164: 120513.
  12. [12]Jeon D, Lee J, Ahn JM, et al. Measuring the novelty of scientific publications: A fastText and local outlier factor approach. Journal of Informetrics 2023; 17: 101450.DOI
  13. [13]Fontana M, Iori M, Montobbio F, et al. New and atypical combinations: An assessment of novelty and interdisciplinarity. Research Policy 2020; 49: 104063.DOI
  14. [14]Yi S, Choi J. The organization of scientific knowledge: the structural characteristics of keyword networks. Scientometrics 2012; 90: 1015–1026.
  15. [15]Zhao Y, Zhang H, Wu W, et al. Measuring the Novelty of Scientific Papers Based on the Latent Distance Between Pairs of MeSH Terms. In: Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries. Hong Kong, China: Association for Computing Machinery, pp. 1–3.
  16. [16]Zhang M, Li W, Du Q. Diverse Region-Based CNN for Hyperspectral Image Classification. IEEE Transactions on Image Processing 2018; 27: 2623–2634.DOI
  17. [17]Liu M, Bu Y, Chen C, et al. Pandemics are catalysts of scientific novelty: Evidence from COVID-19. Journal of the Association for Information Science and Technology 2022; 73: 1065–1078.DOI
  18. [18]McNamee RC. Can’t see the forest for the leaves: Similarity and distance measures for hierarchical taxonomies with a patent classification example. Research Policy 2013; 42: 855–873.DOI
  19. [19]Schumpeter JA. Business Cycles: A Theoretical Historical and Statistical Analysis of the Capitalist Process. New York: McGraw-Hill, 1939.
  20. [20]Wang F. Does the recombination of distant scientific knowledge generate valuable inventions? An analysis of pharmaceutical patents. Technovation 2024; 130: 102947.DOI
  21. [21]Kneeland MK, Schilling MA, Aharonson BS. Exploring Uncharted Territory: Knowledge Search Processes in the Origination of Outlier Innovation. Organization Science 2020; 31: 535–557.
  22. [22]Nooteboom B, Van Haverbeke W, Duysters G, et al. Optimal cognitive distance and absorptive capacity. Research Policy 2007; 36: 1016–1034.
  23. [23]Nooteboom B. Learning and Innovation in Organizations and Economies. Oxford University Press. Epub ahead of print 23 August 2001. DOI: 10.1093/acprof:oso/9780199241002.001.0001.DOI
  24. [24]Nickerson JA, Zenger TR. A Knowledge-Based Theory of the Firm—The Problem-Solving Perspective. Organization Science 2004; 15: 617–632.
  25. [25]Gärdenfors P. The Geometry of Meaning: Semantics Based on Conceptual Spaces. The MIT Press, https://direct.mit.edu/books/monograph/4012/The-Geometry-of-MeaningSemantics-Based-on (2014, accessed 14 June 2026).
  26. [26]Fleming L. Recombinant Uncertainty in Technological Search. Management Science 2001; 47: 117–132.
  27. [27]Liu H, Ruan S, Wu H, et al. Multi-view Heterogeneous HyperGNN for Heterophilic Knowledge Combination Prediction. IEEE Transactions on Big Data 2025; 1–16.DOI
  28. [28]Ding J, Jin W. COS: A new MeSH term embedding incorporating corpus, ontology, and semantic predications. PLOS ONE 2021; 16: e0251094.DOI
  29. [29]Cole S. The Hierarchy of the Sciences? American Journal of Sociology. Epub ahead of print 1 July 1983. DOI: 10.1086/227835.DOI
  30. [30]Popper KR. Objective Knowledge: An Evolutionary Approach. Oxford Eng. : New York: Oxford University Press, 1979.
  31. [31]Popper K. The Logic of Scientific Discovery. New York: Basic Books, 1959.
  32. [32]Yang J, Hu J. Scientific knowledge role transition prediction from a knowledge hierarchical structure perspective. Journal of Informetrics 2025; 19: 101612.
  33. [33]Chai S, Menon A. Breakthrough recognition: Bias against novelty and competition for attention. Research Policy 2019; 48: 733–747.DOI
  34. [34]Coletti MH, Bleich HL. Medical Subject Headings Used to Search the Biomedical Literature. J Am Med Inform Assoc 2001; 8: 317–323.
  35. [35]Ruan X, Ao W, Lyu D, et al. Effect of the topic-combination novelty on the disruption and impact of scientific articles: Evidence from PubMed. Journal of Information Science 2023; 01655515231161133.DOI
  36. [36]Schumpeter JA. Theory of Economic Development. Cambridge, MA: Cambridge, MA: Harvard University Press, 1934.
  37. [37]Wang J, Veugelers R, Stephan P. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy 2017; 46: 1416–1436.
  38. [38]Mishra S, Torvik VI. Quantifying conceptual novelty in the biomedical literature. D-Lib Magazine 2016; 22: 1–21.DOI
  39. [39]Chen J, Wu M, Liu Q, et al. Explainable prediction of knowledge recombination: A synergized method with heterogeneous hypergraph learning and large language models. Information Processing & Management 2026; 63: 104336.
  40. [40]Xu J, Kim S, Song M, et al. Building a PubMed knowledge graph. Sci Data 2020; 7: 205.
  41. [43]Bormann L, Tekles A, Zhang HH, et al. Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data. Journal of Informetrics 2019; 13: 100979.
  42. [44]Lin Z, Yin Y, Liu L, et al. SciSciNet: A large-scale open data lake for the science of science research. Sci Data 2023; 10: 315.DOI
  43. [45]Lipscomb CE. Medical subject headings (MeSH). Bull Med Libr Assoc 2000; 88: 265–266.
  44. [46]Guo Z-H, You Z-H, Huang D-S, et al. MeSHHeading2vec: a new method for representing MeSH headings as vectors based on graph embedding algorithm. Briefings in Bioinformatics 2021; 22: 2085–2095.
  45. [47]Tang J, Qu M, Wang M, et al. LINE: Large-scale Information Network Embedding. In: Proceedings of the 24th International Conference on World Wide Web. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, pp. 1067–1077.
  46. [48]Koubaa A, Boulila W, Ghouti L, et al. Exploring ChatGPT Capabilities and Limitations: A Survey. IEEE Access 2023; 11: 118698–118721.
  47. [49]Shannon CE. A mathematical theory of communication. The Bell System Technical Journal 1948; 27: 379–423.
  48. [50]Becht E, McInnes L, Healy J, et al. Dimensionality reduction for visualizing single-cell data using UMAP. Nat Biotechnol 2019; 37: 38–44.
  49. [51]Friedman M. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. Journal of the American Statistical Association 1937; 32: 675–701.
  50. [52]Wilcoxon F. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1945; 1: 80–83.
  51. [53]Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd edn. New York: Routledge, 1988. Epub ahead of print 1988. DOI: 10.4324/9780203771587.
  52. [54]Shi F, Evans J. Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines. Nature Communications 2023; 14: 1641.
  53. [55]Hou J, Wang D, Li J. A new method for measuring the originality of academic articles based on knowledge units in semantic networks. Journal of Informetrics 2022; 16: 101306.
  54. [56]Dewey M. A Classification and Subject Index for Cataloguing and Arranging the Books and Pamphlets of a Library. Amherst: Mass, https://www.gutenberg.org/ebooks/12513 (1876, accessed 3 June 2025).
  55. [57]Verhoeven D, Bakker J, Veugelers R. Measuring technological novelty with patent-based indicators. Research Policy 2016; 45: 707–723.
  56. [58]Sinha A, Shen Z, Song Y, et al. An Overview of Microsoft Academic Service (MAS) and Applications. In: Proceedings of the 24th International Conference on World Wide Web. New York, NY, USA: Association for Computing Machinery, pp. 243–246.
  57. [59]Salatino AA, Thanapalasingam T, Mannocci A, et al. The Computer Science Ontology: A Large-Scale Taxonomy of Research Areas. In: Vrandečić D, Bontcheva K, Suárez-Figueroa MC, et al. (eds) The Semantic Web – ISWC 2018. Cham: Springer International Publishing, 2018, pp. 187–205.
  58. [60]The Gene Ontology Consortium. The Gene Ontology resource: enriching a GOId mine. Nucleic Acids Research 2021; 49: D325–D334.
  59. [61]Thelwall M, Jiang X, Bath PA. Estimating the quality of published medical research with ChatGPT. Information Processing & Management 2025; 62: 104123.
  60. [62]Huang S, Huang Y, Liu Y, et al. Are large language models qualified reviewers in originality evaluation? Information Processing & Management 2025; 62: 103973.
  61. [63]Sugimoto CR. Scientific success by numbers. Nature 2021; 593: 30–31.
  62. [64]Wu W, Zhang C, Zhao Y. Automated novelty evaluation of academic paper: A collaborative approach integrating human and large language model knowledge. Journal of the Association for Information Science and Technology; 76: 1452–1469.

Paper details

vlnc:W_5evb5tn4
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents