Causal Claims in Economics

Click here for the latest version

JEL Classification: A10, B41, C18, C80, D83

Introduction

Economic research has undergone a significant transformation over the past few decades, marked by an increased emphasis on establishing causal relationships through empirical methods. This "credibility revolution" has propelled the discipline toward more rigorous identification strategies, aiming to provide robust evidence for policy-making and theoretical advancement ([8]). Pioneering work by researchers such as Orley Ashenfelter, Joshua Angrist, David Card, Guido Imbens and Alan Krueger introduced methodologies that enhanced causal identification, including natural experiments, regression discontinuity designs (RDDs), and instrumental variables (IVs) ([9, 40]). These approaches address endogeneity concerns and provide more credible estimates of causal effects.

Leading journals now prioritize studies employing these methods over traditional correlational approaches ([20, 37]). Hamermesh (2013), for instance, documents a decline in purely theoretical articles and an increase in empirical studies utilizing self-generated data and experimental methods in top economics journals.³ The credibility revolution has raised the bar for empirical work, emphasizing transparent reporting, careful consideration of identification assumptions, and rigorous sensitivity analyses ([7, 40]). However, this focus on specific methodologies has sparked debates about the direction and priorities of economic research.⁴ Studies like Backhouse and Cherrier (2017); Currie et al. (2020); Goldsmith-Pinkham (2024) have also shown a significant rise in empirical methods across economics, supported by advancements in data and technology. Our study further contributes by breaking down these trends into specific causal inference methods and examining their usage across subfields.

Despite extensive discussions on methodological advancements, there is a lack of comprehensive analysis that quantifies how the structure and complexity of economic research have evolved over time, particularly regarding the use of causal claims and narrative complexity across subfields. Our study addresses this gap by analyzing over 44,000 NBER and CEPR working papers using a custom large language model to extract structured information on the knowledge graphs of papers, including the methods used to evidence each claim—causal or otherwise—and the data employed in the analyses.

We introduce a novel approach by constructing a knowledge graph for each paper in our dataset. In these graphs, nodes represent economic concepts classified using JEL codes, and edges represent relationships from a source node to a sink node. This means that if a paper discusses how one economic concept relates to another, we capture this as a directional link between those concepts. Whether or not a claim is considered causal depends on the method used to substantiate it. Specifically, we identify an edge as a causal edge if the claim is evidenced using causal inference methods such as Difference-in-Differences (DiD), Instrumental Variables (IV), Randomized Controlled Trials (RCTs), Regression Discontinuity Designs (RDDs), Event Studies, or Synthetic Control

To systematically evaluate these knowledge graphs, we develop three broad categories of measures. First, we track the narrative complexity of a paper, including the breadth and depth of claims. Second, we examine novelty and contribution, capturing whether a paper’s relationships are genuinely new or whether it “fills gaps” previously underexplored in the literature, distinguishing between causal and non-causal contributions. Third, we consider conceptual importance and diversity, focusing on how centrally a paper’s concepts sit within the overall network of economic ideas and whether the paper balances multiple causes (sources) with multiple outcomes (sinks). Additionally, by distinguishing these measures based on claims in the non-causal subgraph from those in the causal subgraph, we parse out the difference between general narrative features and those supported by rigorous identification.

Our analysis reveals several notable patterns. First, the use of causal inference methods has significantly increased over time, with the average proportion of causal claims rising from about 4% in 1990 to nearly 28% in 2020, reflecting the impact of the credibility revolution in economics.

Second, the determinants of publication and citation success differ markedly. While employing rigorous causal inference methods, introducing novel causal relationships, and engaging with frontier topics robustly increase the likelihood of publication in top 5 journals, these same features do not necessarily translate into higher citation counts post-publication. Instead, papers focusing on central, widely recognized concepts accumulate more citations, highlighting a divergence between editorial preferences for causal innovation and the broader academic audience’s sustained attention.

Third, the complexity of a paper’s causal narrative—such as the depth of its longest causal chain or the number of distinct pathways linking causes and effects—plays a pivotal role in both publication and citation outcomes. Causal narrative complexity is particularly associated with top 5 publications and higher citation counts. This suggests that “deeper” causal arguments can succeed at both gatekeeping and long-run academic influence. In contrast, non-causal narrative complexity is generally uncorrelated or negatively associated with academic success.

These findings highlight the trade-offs between methodological rigor, novelty, topic centrality, and broader research recognition. While top-tier journals prioritize innovative and methodologically sophisticated research, citation impact often depends more on engaging with central and well-established concepts. This tension underscores the evolving dynamics of academic evaluation in economics, where causal rigor and novelty increasingly shape short-term outcomes, but long-term impact may still rely on established conceptual foundations.

Literature Critics argue that the emphasis on specific empirical methods and complex narratives may lead to overconfidence in results and potential overinterpretation, especially if the underlying assumptions are not fully met ([28]; [44]).⁵ Additionally, the focus on identification sometimes comes at the expense of economic theory, resulting in studies that establish causal effects without adequately explaining the underlying mechanisms ([62]; [38]). As [44] notes, the detachment from theoretical frameworks can limit the explanatory power of empirical findings. This underscores that while methodological rigor is essential for credible causal inference, it should not preclude the consideration of valuable evidence from diverse sources. Misapplication or overinterpretation of methods can lead to questionable conclusions. For instance, the use of instrumental variables relies on strong assumptions that the instrument affects the outcome only through the endogenous explanatory variable and is uncorrelated with the error term ([9]). Violations of these assumptions, such as weak instruments or invalid exclusion restrictions, can produce biased estimates ([58]). For example, [48] highlights the challenges of using weather variables as instruments, identifying numerous potential exclusion-restriction violations.

RCTs have gained prominence as a gold standard for causal inference due to their internal validity. However, scholars like [29] and [21] argue that RCTs may suffer from limited external validity and may not capture complex economic phenomena, and can further be subject to inducing demand effects ([27]). Generalizing findings from specific experimental settings without considering contextual differences can lead to misleading conclusions ([55]).

Moreover, the complexity of research outputs has increased as papers have become longer and include more coauthors ([20]).⁶ This expansion reflects the need to address methodological rigor and to include detailed explanations of causal mechanisms, robustness checks, and theoretical integration. However, this rise in narrative complexity may also indicate that the presentation and promotion of research findings are becoming more important factors in dissemination.⁷ Increased complexity can make it challenging for readers and reviewers to critically assess the validity of the claims ([41]; [34]), and may contribute to overemphasizing research findings. The "garden of forking paths" metaphor illustrates how analytical flexibility can lead to false-positive findings even without intentional misconduct ([34]).⁸ The American Statistical Association has highlighted the misinterpretation and misuse of p-values, advocating for a more nuanced understanding of statistical significance ([66]). [61] propose the p-curve method as a tool to detect and correct for publication bias using only significant results, highlighting the pervasiveness of selective reporting.

The relationship between theory and empirics is a central concern in economics. Critics argue that the focus on empirical identification has led to a neglect of theoretical development ([44, 38]). Without a solid theoretical foundation, empirical findings may lack coherence. [62] emphasizes that economics is not purely an experimental science and that theoretical models are essential for interpreting empirical results. Furthermore, [5] highlight that economists see value in research that is multidisciplinary and addresses diverse topics, suggesting a need to balance empirical rigor with theoretical and interdisciplinary approaches.

Finally, emerging methodologies such as machine learning, Bayesian inference, and the increased reliance on big data pose further challenges in training, replication, and interpretation.⁹ Amid these technical shifts, the ethical and communicative dimensions of research are also in flux, with social media and open-data debates reshaping public trust ([57, 3, 33]).

Ethical considerations in empirical research extend beyond methodological rigor to include transparency about limitations, uncertainties, and the broader context of findings ([57]). Misleading claims can distort policy-making, erode public trust, and lead to poor allocation of resources. Ensuring integrity in research is a collective responsibility involving researchers, journals, institutions, and funding bodies.¹⁰ Moreover, the ways in which academics engage with the public—particularly through social media—can shape the perceived credibility of their work.¹¹

These concerns gain even more urgency in light of recent global initiatives emphasizing the importance of evidence-based policy-making. International leaders and organizations have expressed alarm over the slow progress in achieving Sustainable Development Goals (SDGs), attributing part of the challenge to a lack of robust evidence to inform policy decisions. Despite substantial investments in public services, a "hidden" repository of underutilized studies exists that could inform better policy choices ([51]). Recognizing these challenges, research councils and governments are investing in innovative solutions to enhance the accessibility and synthesis of existing research. For instance, in September 2024, the UK Economic and Social Research Council (ESRC) announced a significant investment in artificial intelligence to facilitate evidence synthesis for public policy, aiming to build a global infrastructure that provides useful evidence for policymakers.¹² Moreover, organizations like the Behavioural Insights Team have proposed blueprints for better international collaboration on evidence, emphasizing the importance of evidence synthesis and accessibility ([15]).¹³

Understanding these trends requires examining how economic research evolves over time. [6] analyze a large dataset of economics journal articles from 1980 to 2015, documenting shifts in research fields and styles, with more empirical papers appearing in influential journals and receiving more citations. This evolution underscores the growing methodological sophistication in economics but also raises concerns that genuinely new ideas may be harder to find ([16], [53]). Our analysis shows that journals sometimes reward gap filling papers—whether bridging underexplored topics pairs or linking distinct subfields—yet these connections do not always translate into greater long-run citation impact. That said, top-tier outlets particularly value interdisciplinary bridging when it involves causal claims, suggesting an appetite for innovative cross-field insights that are firmly grounded in causal inference methods.¹⁴

Our study contributes to this discussion by providing empirical evidence on the evolution of empirical methods and their differential adoption across subfields. We observe that methods such as DiD, IV, RCTs, and RDDs have seen substantial growth, reflecting the discipline’s shift toward more rigorous identification strategies. Fields such as Urban, Health, Development, and Behavioral exhibit the most significant increases in the use of causal inference methods. Conversely, fields like Macroeconomics show more modest growth. This variation highlights how research questions and methodological traditions influence the adoption of empirical methods across different areas of economics.

By constructing and analyzing knowledge graphs of economic research, our study offers a new perspective on how the complexity and structure of narratives have changed and how these features shape both journal acceptance and scholarly recognition. The remainder of this paper is organized as follows: Section 2 details the data and information retrieval methods used to construct these graphs. Section 3 documents the rise of causal inference and introduces our three categories of measures—narrative complexity, novelty and contribution, and conceptual importance and diversity—examining their evolution and cross-field variation. Section 6 then links these measures to publication outcomes and citation impacts. Section 7 concludes with broader implications for research practices, methodological priorities, and the communication of economic knowledge in an era defined by the credibility revolution.

Data and Methods

In this section, we present the data sources, extraction processes, and methods employed to examine causal claims within the economics literature. Further methodological details and technical specifications are provided in Appendix A.¹⁵

Working Paper Corpus

Our analysis is based on a comprehensive corpus of working papers from two primary sources: the National Bureau of Economic Research (NBER) and the Centre for Economic Policy Research (CEPR). The NBER dataset comprises 28,186 working papers, while the CEPR dataset includes 16,666 papers, resulting in a total sample of 44,852 papers. These papers span several decades and encompass various subfields of economics, providing a broad view of the research landscape.

To refine the sample and focus on relevant content, we applied specific filtering criteria. We included only papers containing more than 1,000 characters to exclude incomplete documents. Additionally, we limited the analysis to the first 30 pages of each paper, ensuring that we captured the sections most likely to contain causal claims, such as introductions, literature reviews, and empirical analyses.

The corpus covers a wide range of economics subfields, including Labour Economics, Public Economics, Macroeconomics, Development Economics, and Finance. The papers employ diverse empirical strategies, such as Randomized Controlled Trials (RCTs), Instrumental Variables (IV), Difference-in-Differences (DiD), and Regression Discontinuity Designs (RDD), allowing us to examine methodological trends across the discipline.

Pre-processing The preprocessing of the text data followed a structured pipeline aimed at cleaning and normalizing the text for analysis. The preprocessing steps included removing excessive whitespace, converting all characters to lowercase, and filtering out non-alphanumeric characters, keeping only spaces for readability. Additionally, we stripped leading and trailing whitespace to ensure uniformity in the text. These steps were helpful for the large language model to efficiently and accurately process the text, especially considering the large volume of data involved.

LLM based retrieval

We employed a multi-stage process using a large language model (LLM) to extract and analyze information from the working papers in our corpus.¹⁶ We interacted with the LLM using carefully designed prompts that guided the model to extract the required information while adhering to a predefined JSON schema. The overall process is visually summarized in Figure 1, which illustrates the flow from input text to structured data extraction and subsequent analysis. This approach allowed us to efficiently process the text and extract detailed structured data necessary for our analysis while minimizing computational and human resources.¹⁷

(Figure 1)

Our LLM-based retrieval process consists of the following stages:

Stage 1: Qualitative Summary Extraction In the first stage, we prompted the LLM to analyze the first 30 pages of each paper and extract a curated summary of key elements.¹⁸ This included the research questions as presented in the abstract, introduction, and full text; information on causal identification strategies used in the paper; details on data usage, accessibility, and acknowledgements; and metadata such as authors’ names, institutional affiliations, fields of study, and methods used. This initial extraction provided a structured overview of each paper, which was used in subsequent stages to extract more detailed information. To ensure the reliability of our extraction process, we validate our information retrieval methods in Appendix C, demonstrating high accuracy and F1 scores across key empirical methods and fields of study.

Stage 2: Extraction of Causal Claims Using the curated summaries from Stage 1, particularly the sections on causal identification and causal claims, we prompted the LLM to extract detailed all knowledge links presented in each paper between two knowledge entities. The LLM identified source and sink variables as described by the authors, determined the types of relationships (e.g., direct causal effect, indirect causal effect, mediation, confounding, theorised relationship, correlation), and recorded the empirical methods used to establish each link (e.g., RCT, IV, DiD, OLS, simulations). The result was an edge list per paper, where each row represents an edge with a source node (e.g., a cause) and a sink node (e.g., an effect). We also included relevant edge attributes, such as the method used to evidence that edge. While we collected additional attributes like the direction of effect, magnitude, and statistical significance, these features were experimental and are not used in the main analysis due to variation in reporting standards.¹⁹

Stage 3: Data Usage and Accessibility Extraction From the data-related summaries in Stage 1, we prompted the LLM to extract structured information regarding data sources and accessibility. Key elements included the ownership of the data (e.g., private company, public sector entity, researchers), data accessibility (e.g., freely accessible, restricted), and details on data granularity, units of analysis, temporal and geographical context. This information is crucial for assessing trends in data usage and the implications for transparency and replicability in economic research.

Validation of Information Retrieval To validate our information retrieval methods, we conducted two exercises (details in Appendix C). First, we matched 307 papers with the annotated dataset from [18], which classifies empirical methods and fields for 1,106 economics papers. Our retrieval achieved high accuracy and F1 scores, especially for RDD methods and fields like Macroeconomics and Urban Economics, demonstrating reliability across key dimensions.

In a second exercise, we compared our causal claims data to the Plausibly Exogenous Galore dataset,²⁰ a source documenting primary causal variables and exogenous variation for 1,435 papers. We matched 485 papers and aggregated our data at the paper level to align with their structure. This comparison yielded moderate similarity for causes and effects, likely due to the Plausibly Exogenous Galore dataset’s focus on the most important causal link, whereas our dataset captures the full knowledge graph of each paper, including all causal links. This broader scope introduces variability in matching specific causes and effects. However, the source of exogenous variation showed high similarity, supporting the consistency of our approach in capturing essential causal elements across datasets.

Matching Variables to Standardized Economic Concepts

To facilitate systematic network analysis and aggregation, we standardized the free-text descriptions of the source and sink variables by mapping them to official Journal of Economic Literature (JEL) codes. We created semantic embeddings for each JEL code’s overall description, which concatenates the JEL description, guidelines, and keywords.²¹ By generating vector embeddings of the variable descriptions and comparing them to the vector embeddings of JEL code descriptions using cosine similarity, we identified the most relevant codes for each variable.²² This process situates each causal claim within the broader context of economic research and allows us to construct a knowledge graph of economic research, mapping and documenting the frontier in causal evidence over time. This process is visually summarized in Figure 2, which illustrates our AI-driven approach to analyzing and mapping causal linkages between JEL codes. Full details are available in Appendix Section B.

(Figure 2)

Citations and Publication Data

Matching Publication Outcomes to Working Papers To analyze the publication trajectories of the working papers, we matched each paper to its eventual publication outcome using multiple data sources. Our primary goal was to determine whether a working paper was published in a peer-reviewed journal and, if so, identify the journal and publication date. This information is essential for understanding the dissemination and impact of research within the economics discipline.

We utilized four data sources to obtain publication information. First, we used official metadata from the NBER, which provides publication data collected via author submissions and automated scraping from RePEc.²³ While comprehensive, the dataset includes duplicates and primarily covers NBER papers. The second source was a large language model (LLM) prompted to retrieve publication outcomes based on its knowledge, yielding results for a small subset of NBER and CEPR papers. Third, we used the OpenAlex repository, matching titles of working papers and prioritizing published versions when multiple matches existed. Finally, we incorporated data from [14] (2020), which provides manually verified publication outcomes for NBER and CEPR papers between 2000 and 2012, matched up to mid-2019.

To ensure consistency, we standardized journal names across these sources using the SCImago Journal Rank (SJR) lists for the fields of "Economics, Econometrics and Finance" and "Business, Management and Accounting." After removing generic journal names, this list included 2,367 unique journals.

Our matching process followed a hierarchical approach, prioritizing verified data. We first checked for a match in the dataset from [14] , followed by a search in OpenAlex for exact title matches. If no match was found, we consulted the NBER metadata, and finally used the LLM retrieval method for remaining papers. This approach ensured a comprehensive and reliable matching of publication outcomes. In total, the dataset from [14] provided publication outcomes for 9,139 papers, OpenAlex matched 10,840 papers, NBER metadata contributed 15,872 matches, and the LLM retrieval identified outcomes for 1,707 papers. By consolidating these sources, we obtained a detailed picture of the publication trajectories of a substantial number of working papers.

Despite this extensive coverage, certain limitations remain. The NBER metadata may be incomplete due to reliance on author submissions, and the [14] dataset only covers papers up to 2012, matched to 2019. Additionally, errors may arise due to title similarities or data entry issues. However, by leveraging multiple sources, we minimized these limitations, resulting in a robust dataset for further analysis.

Citations Data To extend our analysis, we collected citations data for the working papers using three primary sources: RePEc’s CiteEc service (https://citec.repec.org/), [14], and the OpenAlex repository. We prioritized the citations data from RePEc’s CiteEc service, which provides up-to-date (as of November 2024) citation counts for a large number of economics papers. For papers not included in CiteEc, we used the manually verified citations from [14], which provides citation counts for NBER and CEPR papers published between 2000 and mid-2019. For any remaining papers, we obtained citation counts from OpenAlex, matched by exact paper titles. By merging these sources and prioritizing in this order, we assembled citations data for approximately 94.6% of our total sample, and 97.7% of the pre-2020 sample used in our analysis. This extensive coverage enables us to incorporate citations as a measure of research impact.

Graphical Framework

Over the past four decades, economic research has undergone a profound transformation, characterized by an increasing emphasis on establishing credible causal relationships using empirical methods—a shift often referred to as the “credibility revolution.” Scholars such as [7] have highlighted the importance of rigorous econometric techniques designed to enhance causal inference, describing them as Mostly Harmless Econometrics. To systematically capture and analyze this evolution, we introduce a graphical approach by constructing a knowledge graph for each paper in our dataset. This method allows us to quantitatively assess the complexity and structure of claims in economic research over time and observe changes in the adoption of causal inference methods.²⁴

For each paper $p$, we define a directed graph $G_p = (V_p, E_p)$, where $V_p$ is the set of nodes representing economic concepts, classified using JEL codes, and $E_p$ is the set of directed edges representing claims from a source node to a sink node. We use the terms “source” and “sink” to denote the direction of the claim within the paper, without presupposing causality. Whether a claim is interpreted as causal depends on the attributes of the edge connecting the nodes.

An important attribute of each edge $e \in E_p$ is whether the claim was evidenced using a causal inference method. We classify an edge as a causal edge if the associated claim in the paper was evidenced using one of the following methods: Difference-in-Differences (DiD), Instrumental Variables (IV/2SLS), Randomized Controlled Trials (RCTs/Experiments), Regression Discontinuity Design (RDD), Event Study, or Synthetic Control. This classification allows us to distinguish between causal and non-causal claims within the network. With this definition, approximately 19% of all claims in our dataset are classified as causal edges.

The network $G_p$ thus includes both causal and non-causal edges. For instance, theoretical relationships between two concepts are represented as edges but are not considered causal unless they are supported by the specified empirical methods. This approach enables us to analyze the overall structure of a paper’s argumentation and the role of causal inference methods within it.

Observing the Credibility Revolution in Economic Research

To analyze the evolution of the use of causal inference methods in economic research, we focus on the proportion of causal edges in papers over time. This measure reflects the extent to which economists have increasingly adopted rigorous causal inference methods in their work, indicative of the credibility revolution.

Figure 3 Figure 3 (a) displays the average proportion of causal edges per paper from 1980 to 2023. The data show a significant increase over time. In 1990, the average proportion of causal edges was approximately 4.2%. By 2000, it had risen modestly to around 8.4%. However, the increase became more pronounced in the subsequent decades: by 2010, the average proportion reached approximately 17.1%, and by 2020, it had climbed to around 27.8%. This upward trend indicates that economic papers have increasingly incorporated causal inference methods to substantiate their claims over the past three decades.

This substantial increase suggests a growing emphasis on establishing credible causal relationships in economic research. The proliferation of empirical methods and a heightened focus on causal identification strategies have contributed to this trend, reflecting the impact of the credibility revolution in economics.

We also examine how this proportion varies across different fields within economics. Figure 3 Figure 3(b) presents the average proportion of causal edges by field, comparing the pre-2000 and post-2000 periods. Recent evidence corroborates these cross-field patterns, showing that finance and macroeconomics have lagged behind applied micro in adopting quasi-experimental methods, though the gap is narrowing ([36]). Our findings add that most fields have experienced substantial increases in the average proportion of causal edges in the post-2000 period.

Fields such as Urban, Health, Development, and Behavioral exhibit the highest increases. Urban increased from approximately 4.7% pre-2000 to 33.2% post-2000, marking one of the largest gains. Health saw an increase from 10.1% to 37.6%, achieving the highest post-2000 level among all fields. Development rose from 4.4% to 31.0%, and Behavioral increased from 3.6% to 29.6%. These fields, which often address policy-relevant questions and benefit from natural experiments or data conducive to causal analysis, have embraced causal inference methods more extensively.

Conversely, some fields experienced smaller increases or maintained lower levels. Macroeconomics increased modestly from 3.3% to 8.4%, reflecting a more cautious adoption of causal inference methods, possibly due to challenges in experimental design and identification strategies in macroeconomic contexts. Econometrics saw an increase from 6.3% to 11.0%, and Finance rose from 2.9% to 17.0%. These patterns suggest that the adoption of causal inference methods has varied across fields, influenced by the nature of the research questions, data availability, and methodological traditions within each field.

(Figure 3)

Evolution of Empirical Methods in Economic Research

To explore the increasing focus towards causal inference, we show time trends across methods and fields. Figure 4 illustrates the adoption of prominent empirical methods in NBER and CEPR working papers from 1980 to 2023. Methods such as DiD, IV, RCTs, and RDDs have seen substantial growth, reflecting the discipline’s shift towards more rigorous identification strategies.²⁵

DiD has become increasingly prevalent, rising from approximately 4% of papers in 1980 to over 15% in recent years. This growth underscores DiD’s utility in exploiting policy changes and natural experiments to identify causal effects. IV methods have also seen a steady increase, from around 2% of papers in 1980 to over 6% by 2023, highlighting their role in addressing endogeneity through exogenous instruments ([9]). The adoption of RCTs has accelerated since the early 2000s, increasing from less than 1% of papers in 2000 to over 7% in 2023, signaling the increasing feasibility and acceptance of experimental designs in economics. Similarly, RDD usage has grown from near zero in the 1980s to over 2% in recent years.

Conversely, the proportion of theoretical and non-empirical work has declined significantly, from approximately 20% of papers in 1980 to under 10% in 2023, indicating a broader emphasis on empirical analysis. The use of simulations has also decreased, from over 6% in 1980 to around 2–4% in recent years, possibly due to the growing availability of of data as well as the possibility to conduct (field) experiments for evaluation.

(Figure 4)

These trends are not uniform across subfields. Figure 5 presents the distribution of empirical methods by field. Fields such as Labour, Public, and Urban heavily utilize DiD, with over 12% of papers employing this method in Labour and Public, and over 16% in Urban. RCTs are particularly prominent in Behavioral and Development, where they are used in over 20% and 11% of papers respectively, reflecting the feasibility and policy relevance of experimental interventions in these areas. This trend aligns with the findings of [25], who documented a broad increase in empirical approaches due to the availability of big data and advanced computing.

In contrast, fields like Macroeconomics, IO, and Finance rely more on theoretical models and simulations to infer causal relationships from observational data. Theoretical approaches account for around 18% of papers in Macroeconomics and 12% in Finance. Structural estimation and simulations remain important in Macroeconomics and Industrial Organization, where complex theoretical models may be essential for understanding aggregate phenomena and market dynamics.

(Figure 5)

Paper-level Graphical Measures

In this section, we define quantitative measures that characterize the structure and content of each paper’s knowledge graph. These measures are grouped into three categories, each capturing distinct aspects of how research is organized and perceived by readers and editors. Measures of narrative complexity assess the internal structure of each paper’s claims; novelty and contribution compare a paper’s ideas to the existing literature, identifying new concepts or intersections; and conceptual importance and diversity situate a paper’s nodes within the broader knowledge graph of economic research.²⁶ Together, these measures set the stage for the empirical analysis in Section 6, where we link them to publication outcomes and citation impacts. This section examines how these measures vary across fields and over time, presenting descriptive trends and cross-sectional comparisons to motivate their relevance for understanding academic reception.

Measures of Narratives Complexity

We consider several key measures derived from these claim networks. The number of edges, denoted as $\lvert E_p\rvert$, represents the total number of claims made in a paper, reflecting the breadth of the narrative. An increase in $\lvert E_p\rvert$ suggests a more complex narrative with multiple interrelated claims. Similarly, the number of causal edges, denoted as $\lvert E_p^{\mathrm{causal}}\rvert$, represents the total number of claims evidenced using causal inference methods, indicating the depth of causal analysis within the paper.

Other measures include the number of unique paths in $G_p$, denoted as $P_p$, which is the total number of distinct directed paths between all pairs of nodes, excluding self-loops. This captures the interconnectedness of the narrative within the paper; a higher $P_p$ indicates a more intertwined argument structure. The longest path length in $G_p$, denoted as $L_p$, represents the length of the longest directed path, indicating the depth of reasoning in the paper. We compute both $P_p$ and $L_p$ for the non-causal subgraph (denoted as $P_p^{\mathrm{non\text{-}causal}}$ and $L_p^{\mathrm{non\text{-}causal}}$) and for the subgraph consisting only of causal edges ($P_p^{\mathrm{causal}}$ and $L_p^{\mathrm{causal}}$).

Illustrative examples To concretely illustrate our graphical framework and the measures derived from it, we examine four landmark economic papers: [24], [13], [32], and [35]. These papers cover a diverse range of topics and methodologies, showcasing the versatility of our approach. The visual representations of these knowledge graphs are provided in Figure 6 and Figure 7, which highlight the varying structures and complexities of the narratives in these influential papers.

In [24], the authors investigate the geography of intergenerational mobility in the United States using administrative records. The paper presents a comprehensive analysis of how various factors correlate with upward mobility. The knowledge graph for this paper (fig:6a) includes seven edges, all of which are non-causal. The relationships mapped include how parent income (JEL code D31) influences child income rank (J13), and how factors such as lower residential segregation (R23), less income inequality (D31), better primary schools (I21), greater social capital (Z13), and more stable family structures (J12) are associated with higher upward mobility (J62). The knowledge graph measures for this paper are as follows: the number of edges $|E_p| = 7$ (all non-causal), the number of unique paths $P_p^{\mathrm{non-causal}} = 6$, and the longest path length $L_p^{\mathrm{non-causal}} = 1$. The relatively high number of edges indicates a broad exploration of factors affecting upward mobility, while the longest path length of 1 reflects that the relationships are primarily direct associations rather than extended causal chains.

[13] report on a randomized evaluation of the impact of introducing microfinance in a new market. Utilizing Randomized Controlled Trials (RCTs), the authors assess how access to microfinance affects various economic outcomes for households in Hyderabad, India. The knowledge graph for this paper (fig:6b) includes eight edges, all of which are causal, evidenced through RCTs. The causal relationships include how the introduction of microfinance (G21) leads to households having a microcredit loan (D14), which in turn influences new business creation (L26) and investment in existing businesses (G31). These investments impact average monthly per capita expenditure (D12), while microcredit affects expenditure on durable goods (E21) and expenditure on temptation goods (D12). The authors also examine the effect of microcredit on development outcomes such as health, education, and women’s empowerment (I15). The knowledge graph measures are: the number of edges $|E_p| = 8$ (all causal), the number of unique paths $P_p^{\mathrm{causal}} = 12$, and the longest path length $L_p^{\mathrm{causal}} = 3$. The high number of causal edges and unique paths indicates a complex causal narrative with multiple interconnected outcomes, while the longest path length reflects deeper causal chains.

(Figure 6)

In [32] (2011), the author proposes that idiosyncratic firm-level fluctuations can explain a significant part of aggregate economic shocks, introducing the "granular" hypothesis. The paper develops a theoretical framework and provides empirical evidence supporting the idea that shocks to large firms contribute to aggregate volatility. The knowledge graph (fig:7a) includes six edges, all of which are non-causal, representing theoretical relationships such as how idiosyncratic shocks to large firms (D21) contribute to aggregate volatility (E32) and GDP fluctuations (F44), and how the fat-tailed distribution of firm sizes (L11) implies that idiosyncratic shocks do not average out (E32). The knowledge graph measures are: the number of edges $|E_p| = 6$ (all non-causal), the number of unique paths $P_p^{\mathrm{non-causal}} = 11$, and the longest path length $L_p^{\mathrm{non-causal}} = 3$. The relatively high number of unique paths and the longest path length indicate a complex theoretical narrative with multiple interconnected concepts and deeper reasoning chains.

Finally, [35] (2010) examine the impact of imported intermediate inputs on domestic product growth in India. The authors use empirical methods, including Difference-in-Differences (DiD), to establish causal relationships between declines in input tariffs and firm performance. The knowledge graph (fig:7b) for this paper includes five edges, three of which are causal. The causal relationships include how declines in input tariffs (F14) lead to increased firm product scope (L25) and improved firm performance (L25), and how increased availability of new imported inputs (O39) causes relaxed technological constraints for domestic firms (O33). The knowledge graph measures are: the number of edges $|E_p| = 5$, the number of causal edges $|E_p^{\mathrm{causal}}| = 3$, the number of unique paths $P_p = 5$, and the longest path length $L_p = 2$. These measures reflect a focused exploration of specific causal relationships, with a moderate level of narrative complexity.

(Figure 7)

These examples demonstrate the diversity in the structure and complexity of knowledge graphs across different types of economic research. They illustrate how our measures capture key aspects of the narratives, such as the breadth of topics covered, the depth of causal analysis, and the interconnectedness of concepts.

Measures of Novelty and Contribution to Literature

Thus far, our measures of narrative complexity focus on the internal structure of each paper’s knowledge graph, such as the number of edges, unique paths, and longest causal chains. We now introduce a complementary set of measures capturing how each paper pushes the frontier of economic research by contributing new concepts, links, or underexplored intersections. These measures fall into four categories: novel edges, path-based novelty, subgraph-based novelty, and gap filling. Each can be computed for the non-causal subgraph or restricted to the causal subgraph. In the latter case, these measures capture novelty or gap filling specifically within the subset of claims that are supported by causal inference methods.

Novel Edges

A paper’s knowledge graph may contain numerous directed edges (claims). We call an edge $(u \to v)$ novel if it was never previously documented in any earlier paper’s knowledge graph. For instance, if no prior publication had directly linked the concept of Climate Change to Migration Flows, the first paper to do so would be credited with introducing a novel edge.

Formally, let $\bigcup_{q<p} E_q$ be the union of all edges from papers preceding $p$. Then an edge $(u \to v) \in E_p$ is novel if $(u \to v) \notin\bigcup_{q<p} E_q$.

We define two statistics:

$PropNovelEdges_p$ thus captures the fraction of a paper’s edges that represent wholly new linkages between concepts, relative to the existing corpus.

Algorithmically, we maintain a global (yearly) list of edges seen so far. For each new paper $p$, we check each edge against that list to see if it is new. We then record the proportion of edges that are novel. We implement an identical procedure for both the causal and non-causal subgraph by filtering edges to those based on causal inference methods.

Suppose prior literature had already established claims linking Income Inequality and Social Capital, but no one had linked Family Stability directly to Upward Mobility. If [24] were the first to draw that edge, it counts as a novel linkage, thus incrementing NumNovelEdges.

Path-Based Novelty

Even if the individual edges of a paper are not entirely new, the paper may still combine known edges into new sequences of concepts that have never before appeared in a single chain. For instance, two edges $(X \to Y)$ and $(Y \to Z)$ might be known, but if no prior paper has ever connected them into the path $(X \to Y \to Z)$, that path is new. This captures a higher-order notion of novelty: novel paths.

For each paper $p$, let $\mathcal{P}_p$ be all simple directed paths in $G_p$ up to a chosen length $\ell$. A path $\pi\in\mathcal{P}_p$ is novel if it did not appear in $\bigcup_{q<p}\mathcal{P}_q$. We then define:

In terms of implementation, for each paper we first build the directed graph and list all simple paths up to length $\ell$ (often $\ell= 3$). Secondly, we convert each path into a canonical string representation (e.g., “$u \to v \to w$”), and compare these strings to a global set of previously recorded paths. Again, we do this for both the non-causal subgraph and the causal subgraph.

This measure helps us think about novelty in joint mechanisms. [67] famously argued that scientific ideas grow combinatorially as researchers recombine existing concepts in novel ways, resonating with our notion that bridging seldom-linked nodes fosters innovation. Prior work might have had edges from Microfinance Introduction to Borrowing Behavior, and from Borrowing Behavior to Business Investment. But if no paper had combined these into a causal chain

Microfinance $\to$ Borrowing $\to$ Business Investment, then [13] would score highly on path-based novelty.

Subgraph-Based Novelty

Subgraphs capture multi-node patterns that may include loops or branching structures. A subgraph $S = (V_S, E_S)$ of $G_p$ is said to be novel if no isomorphic subgraph has appeared in prior papers. For computational reasons, we typically limit attention to subgraphs with $|V_S| \le k$ (e.g., 3 or 4 nodes). Subgraph enumeration and motif detection have long been central to the study of complex networks in fields ranging from systems biology to social network analysis.²⁷

Let $S_p$ be the set of all induced subgraphs of size $k$ within $G_p$. We produce a canonical representation $\phi(S)$ for each subgraph to check for duplicates. We then define:

where $\Phi_{<p}$ is the union of canonical subgraph signatures from all earlier papers.

The algorithm enumerates all subgraphs of size $k$ in each paper using an induced_subgraph procedure and checks for isomorphism by generating a canonical adjacency-matrix signature. We then cumulatively union these signatures across papers in chronological order. As before, we produce separate measures for the non-causal subgraph and for the causal subgraph.

Unlike individual edges or single paths, subgraphs can capture intricate feedback loops or multi-node structures. For instance, a triad of nodes

would be a 3-node cyclic subgraph. If no prior paper documented this three-way feedback cycle, the paper introducing it obtains a high subgraph-based novelty measure.

Gap Filling (Co-occurrence Analysis)

Whereas the above novelty measures track new edges or structures, gap filling identifies whether a paper bridges concepts that are underexplored in prior work. Even if each concept is well studied on its own, a pair might be rare or entirely missing in the earlier literature. A paper that explicitly connects these underexplored concepts helps fill that gap. In economics, the use of co-occurrence methods is prominently exemplified by the work of [39] in measuring export complexity, but these approaches are more widely applied in fields such as biomedical research, bibliometrics, and network science.²⁸

For each year, we compile a cumulative count of how many times each unordered pair of JEL codes has co-occurred in prior papers. We label any pair $\{u,v\}$ with frequency below a threshold $\tau$ (e.g., fewer than 5 co-occurrences) as underexplored. Given a new paper $p$, we then look at all distinct pairs of JEL codes in $V_p$ and calculate the gap-filling proportion as the fraction of these pairs that remain underexplored:

A higher value indicates that the paper bridges more underexplored concept pairs.

Algorithmically, we maintain an updated frequency table for all unordered pairs of JEL codes up to year $t-1$. For each paper published in year $t$, we check which pairs in its node set remain below the threshold $\tau$ and compute that paper’s gap-filling proportion. The same procedure is applied to the causal subgraph if one wishes to isolate underexplored pairs specifically tied to credible-causal edges.

Consider the following example. A macroeconomics paper linking Behavioral Biases (D91) and Monetary Policy (E52) in a new structural model might fill a gap if these two codes historically co-occur rarely (e.g., fewer than 5 times). Even if prior literature studied each code extensively in isolation, the explicit combination might represent a novel domain intersection.

Which Journals Fill Gaps in Literature? To further illustrate our gap-filling measure, we compare how papers accepted in different journal tiers (and specifically among the top 5 journals) bridge underexplored connections. Figures fig:8a and fig:8b display results for gap filling (Section 5.2.4). In each panel, we report separate estimates for the non-causal subgraph (in orange) and the causal subgraph (in blue). The error bars represent 95% confidence intervals.²⁹

In Panel fig:8a, the Top 5 group shows a higher mean of standard gap filling for the non-causal subgraph, but we observe very little difference across tiers for the causal subgraph. This suggests that top-tier outlets more readily reward bridging underexplored concept pairs in general, although the strictly causal bridging does not differ much between tiers. Panel fig:8b disaggregates the five elite journals, highlighting especially strong causal gap filling in the QJE and AER.

Overall, these patterns suggest that top-tier outlets reward papers forging novel or underexplored concept connections, especially when supported by credible identification strategies.

(Figure 8)

Taken together, these measures of novelty and gap filling enrich our understanding of how each paper pushes the frontier of economic knowledge. By systematically tracking new edges, new paths, new subgraphs, and underexplored concept pairs, we capture distinct facets of a paper’s originality and the extent to which it addresses previously overlooked topics or mechanisms. As shown in Section 6, these indicators prove to be significant predictors of both publication outcomes and the long-run citation impact of research.

Measures of Conceptual Importance and Diversity

We now turn to assessing how centrally each paper’s concepts lie within the broader economic literature and whether the paper balances multiple sources and targets in its argumentation. These attributes may shape both the paper’s perceived significance and its eventual scholarly influence. We consider two sets of measures: (i) the source–sink ratio, which captures the balance of causal flows in the paper, and (ii) centrality-based indicators (e.g., eigenvector or PageRank scores) computed for the nodes in a cumulative knowledge graph of prior literature. We then aggregate these centrality values (mean, variance) over the specific concepts a paper employs, providing measures of conceptual importance and conceptual diversity.

Source–Sink Ratio

A paper may describe multiple causal mechanisms converging on a single outcome or, conversely, a narrow set of causes that generate multiple distinct outcomes. We define a source–sink ratio to capture these tendencies. Let $G_p = (V_p, E_p)$ be the knowledge graph of paper $p$. For each node $v \in V_p$, let $\operatorname{outdegree}(v)$ and $\operatorname{indegree}(v)$ denote its out-degree and in-degree, respectively. The source–sink ratio is then:

where $\varepsilon$ is a small positive constant to avoid division by zero if no nodes act as sinks (or sources).

A ratio $R_p > 1$ indicates that the paper has more distinct source nodes (origins of claims) than sink nodes (endpoints). This often arises in work that investigates many potential drivers of a smaller set of outcomes. Conversely, $R_p < 1$ suggests the paper concentrates on a few key causal factors but explores a broad set of consequences. A balanced ratio around 1.0 reflects a more even structure, with multiple causes and effects.

We compute $R_p$ separately for the non-causal subgraph and the causal subgraph. The latter includes only edges established via credible identification strategies (e.g., DiD, IV, or RDD). A paper that juggles multiple rigorously identified drivers for a single main outcome, for instance, will display a high causal source–sink ratio.

If a paper focuses heavily on how multiple institutional features (e.g., tax incentives, labor regulations, trade reforms) affect a single macroeconomic indicator (like GDP growth), it may have a high source–sink ratio. By contrast, if it starts from one main policy shock but explores myriad downstream effects (household consumption, firm investment, labor demand, etc.), the ratio can be well below 1.0.

To examine how the source–sink ratio evolves, Figure 9 plots its average values (along with 95% confidence bands) for the non-causal and causal subgraphs over time. The ratio for the non-causal subgraph remains relatively flat until 2000, followed by a slight decline in subsequent years. In contrast, the causal subgraph shows a steady and steep increase over the same period, indicating that recent decades see papers exploring more distinct causal factors relative to outcomes. This pattern aligns with a shift toward identifying multiple potential mechanisms or drivers for a narrower set of effects.

Image not extracted

Figure 9 compares fields cross-sectionally. We see that Econ History feature high non-causal source-sink ratio, given its emphasis on explaining a single outcome through multiple correlational determinants. By contrast, Behavioral and Health exhibit larger causal source-sink ratios, reflecting the broader adoption of rigorous identification that tests several potential causal channels for specific outcomes.³⁰ Together, these field-level differences highlight how some areas of economics appear more conducive to multi-causal designs, while others continue to aggregate a range of correlates into narrower outcome measures.

Centrality-Based Measures of Conceptual Importance and Diversity

Network centrality indices assess how “important” or “influential” a node is within a larger network ([17]; [52]). In this context, a node is a JEL code (economic concept), and the larger network is the cumulative knowledge graph of all prior economic research in our sample. We compute centralities for each concept in the network at year $t-1$, then measure how central (on average) the concepts used by paper $p$ (published in year $t$) actually are. A higher average centrality indicates that the paper engages with well-established or highly influential concepts, whereas a lower average centrality may signal an exploration of more specialized or peripheral topics. Furthermore, a higher variance of centralities among the paper’s nodes can imply mixing mainstream and niche concepts—a form of conceptual diversity.

We focus primarily on eigenvector centrality because it captures the recursive intuition that a node is important if it is connected to other important nodes. However, we also compute PageRank [52] to check robustness, since PageRank includes a damping factor that prevents rank sinks and redistributes influence among nodes with low in-degree. In directed citation or claim networks, PageRank can better handle large cycles and “dangling” nodes, but both measures reflect similar underlying principles of node influence.

Formally, let $C_{t-1}(v)$ be the centrality of node $v$ in the cumulative knowledge graph formed by all papers prior to year $t$. Once we have identified the set of JEL codes $V_p$ used by paper $p$, we define:

These measures are computed for two types of subgraphs: the non-causal subgraph and the causal subgraph. Specifically, we compute $\operatorname{MeanCentrality}_p$ and $\operatorname{VarCentrality}_p$ using eigenvector centrality and PageRank for both the non-causal and causal subgraphs. This yields the following statistics:

calculated separately for the non-causal and causal subgraphs. By implementing this procedure for each subgraph type, we capture differences in conceptual importance and diversity for claims supported by causal inference methods versus those based on non-causal relationships.

Once we calculate a paper’s mean or variance of eigenvector (or PageRank) centralities, we standardize these values (i.e., subtract the sample mean and divide by the standard deviation) to facilitate comparability across papers and subgraphs.

Nodes with high prior-year centrality are typically well-established topics (e.g., GDP, Unemployment, Trade Liberalization, Exchange Rates). Papers engaging predominantly with such nodes may attract broader readership, facilitate connections to familiar debates, and potentially garner more citations. By contrast, papers emphasizing less central nodes (e.g., Renewable Energy Technologies in 1990, or Machine Learning for Macroeconomic Forecasting in 2010) might be exploring frontier topics with sparser prior literature. The variance measure captures whether a paper blends mainstream concepts with more specialized ones. This diversity may be an asset (appealing to multiple subfields) or a challenge (positioning the paper less neatly in any single mainstream area).

Central concepts can serve as natural anchors in scientific discourse, accruing more attention and building cumulative knowledge ([49]). Engaging with such mainstream nodes may boost impact, but may also reduce a paper’s perceived novelty. In turn, venturing into peripheral areas can differentiate one’s research but risks a narrower audience. Our empirical analysis in Section 6 demonstrates that these centrality-based indicators correlate systematically with journal placement and citation trajectories, highlighting how conceptual positioning can shape both short-run visibility and long-run influence.

Illustrative examples

To demonstrate how these measures vary across papers, we revisit the four landmark studies while focusing on the source–sink ratio, average eigenvector centrality, and the variance of centrality within nodes. In [24], the source-sink ratio $R_p \approx2.5$ is derived from the non-causal subgraph, as this paper contains only non-causal edges. This indicates multiple sources—e.g., residential segregation, income inequality, school quality, social capital, family stability—culminating in a single main outcome (upward mobility). Their standardized average eigenvector centrality is about $-0.39$, with a variance of around $-0.46$.³¹ Although below the global mean, this suggests moderate reliance on somewhat less central concepts in the broader non-causal knowledge graph (e.g., intergenerational mobility, which, at the time, was a relatively specialized research topic).

By contrast, [13] exhibit a causal source–sink ratio of $R_p \approx0.71$, reflecting a single main cause (microfinance introduction) that branches out into multiple downstream outcomes, such as borrower behavior, business creation, expenditure patterns, and development indicators. Their standardized average eigenvector centrality is approximately $-0.35$ (variance: $-0.22$), placing these concepts slightly below the global causal mean. This modestly negative value aligns with the paper’s relatively frontier policy topics—e.g., finance for the poor—which, although influential, had not yet reached the level of established mainstream concepts.

In [32], $R_p \approx0.8$ (non-causal) indicates a mix of firm-level shocks feeding into macro-level outcomes, such as aggregate volatility and GDP fluctuations. The standardized average eigenvector centrality is about $-0.15$ (variance: $-0.14$), reflecting more specialized or less frequently studied areas (e.g., the fat-tailed distribution of firm sizes) prior to that paper’s publication. Although these concepts ultimately sparked considerable interest, they were comparatively less central within the broader non-causal network at the time.

Finally, [35] highlight multiple sources—declines in input tariffs and increased availability of new imported inputs—affecting a narrower set of firm outcomes (product scope, performance). For the non-causal subgraph, the source–sink ratio is around 1.33, with a standardized mean eigenvector centrality of about $-0.60$ (variance: $-0.26$). Meanwhile, the causal subgraph yields a higher source–sink ratio of roughly 2.0, alongside a standardized mean eigenvector centrality of $-0.57$ (variance: $-0.24$). Although both measures sit below the overall mean, the concepts involved (e.g., trade liberalization, industrial performance) do have moderate footing in established debates, consistent with the paper’s balanced mix of theoretical and empirical contributions.

These examples illustrate how structural measures of source–sink balance and centralities capture different dimensions of a paper’s conceptual organization. By quantifying these properties, we gain a clearer picture of how particular studies draw on established conversations or venture into more specialized territories, and how such positioning may influence their academic reception.

Evolution of Concept Centrality Over Time in Literature

We next investigate how the centrality of specific economic concepts has changed over time, comparing both the non-causal and the causal subgraph. First, we highlight which JEL codes rank most highly in eigenvector centrality, noting differences between non-causal edges and those established through causal inference methods. We then track how certain concepts have risen or fallen in importance since 1990.

Most central concepts in economics Figure 10 displays the top 20 JEL codes ranked by their eigenvector centrality in both the non-causal and causal knowledge graphs of the literature (i.e., our full sample, partitioned by edge type). The comparison reveals interesting patterns in how central economic concepts differ based on whether claims are supported by causal inference.

In the non-causal subgraph, the nodes with the highest eigenvector centrality scores include G21 (Banks and Mortgages), J31 (Wage Structure), and I24 (Education and Inequality), reflecting their prominence in the economic literature at large. However, when focusing on the causal subgraph, the nodes with the highest centrality scores shift towards I24 (Education and Inequality), J13 (Fertility and Family), and I21 (Analysis of Education).

This shift indicates that while certain economic concepts are central when considering non-causal associations, the focus of causal inference methods tends to concentrate on specific areas, such as education, family, and health economics—topics more amenable to experimental or quasi-experimental designs.

The distribution of eigenvector centralities is highly skewed in both graphs. In the non-causal subgraph, the mean eigenvector centrality is 0.0402, with a maximum of 0.898 for G21. In the causal subgraph, the centralities are more concentrated among certain nodes, with I24 achieving the highest score normalized to 1.0. By incorporating average eigenvector centrality as a measure, we capture the extent to which a paper’s claims involve central or peripheral concepts in economics. Papers engaging with highly central nodes in the causal subgraph may be contributing to well-established areas of research using rigorous methods, while those involving less central nodes might be exploring more novel or specialized topics. (Figure 10)

Rise and Fall of Importance of Certain topics Beyond identifying which nodes are central at a given moment, we examine how these centralities evolve over time. Figure 11 plots time series for five nodes that rose most steeply in normalized eigenvector centrality from 1990 to 2020 and five that declined the most, based on changes between a baseline year (e.g., 2000) and the final year (2020). We perform a min–max normalization for each node to facilitate cross-node comparison on a 0–1 scale.

Panel (a) shows results for the Non-Causal subgraph. Concepts related to traditional macro-economic or “hard” economic areas—such as Growth & Productivity (O49), Interest Rates (E43), Capital & Investment (E22), Foreign Exchange (F31), and Wages & Labor (J31)—all show declining centrality. By contrast, topics often associated with microeconomics, inequality, or behavioral dimensions—Health & Inequality (I14), Behaviour & Decisions (D91), Collective Decisions (D79), Minorities & Non-Labor Discrimination (J15), and Education & Inequality (I24)—exhibit sustained rises. This pattern suggests a structural shift away from classical macro–finance focal points toward more applied or “softer” fields linking economics to questions of health, social behavior, and inequality.

Panel (b) focuses on the Causal subgraph, restricted to edges supported by causal inference methods (e.g., DiD, IV, or RDD). The trends observed here are broadly similar, with declines in traditional macro/finance topics and rising importance for micro-oriented, behavioral, and inequality-focused areas. Notably, Health & Inequality (I14), Education & Inequality (I24), and related codes exhibit consistent increases in centrality, underscoring the growing emphasis on empirically identified relationships in policy-relevant areas. This shift reflects the broader influence of the credibility revolution in steering economic research toward topics where causal inference methods are particularly applicable.

(Figure 11)

Knowledge Graph Predictors of Publication and Citations

This section investigates how the structural properties of a paper’s knowledge graph predict its subsequent publication outcomes and citation impacts. Drawing on the measures introduced in Section 5, we examine three categories of predictors: narrative complexity, novelty and contribution, and conceptual importance and diversity. These categories capture distinct dimensions of how research is structured and how it connects to existing literature. We evaluate whether these features systematically correlate with a paper’s likelihood of publication in prestigious journals and its subsequent citation impact.

Empirical Framework and Variables

This section outlines the empirical strategy to evaluate how paper-level knowledge-graph measures predict publication outcomes and citation impacts. For each measure introduced in Section 5 (e.g., narrative complexity, novelty, conceptual importance), we separately analyze its non-causal and causal variants. The regression specification is:

where $y_p$ is one of four outcomes: (i) an indicator for publication in a top 5 journal ($Top5_p$); (ii) publication in a journal ranked 6–20 ($Top6$-$20_p$); (iii) publication in a journal ranked 21–100 ($Top21$-$100_p$); or (iv) the log of total citations plus one ($\log(Cites_p + 1)$). The explanatory variable $M_p^{(\mathrm{type})}$ represents a paper-level measure (e.g., number of edges, proportion of novel paths, or average centrality), analyzed separately for non-causal and causal subgraphs.

We estimate two versions of this regression. The first includes year fixed effects, $\delta_{t(p)}$, to control for time-specific shocks and trends in publication or citation practices, with standard errors clustered at the year level to account for within-year correlations among papers. The second version excludes year fixed effects and reports unclustered standard errors, allowing us to assess the robustness of our results to the inclusion of year-level controls. Both versions are included in the results for comparison.

This framework facilitates a detailed comparison of the predictive power of non-causal versus causal claims for publication and citation outcomes. While causal relationships often have greater influence on top-tier journal placement, both non-causal and causal measures significantly affect long-term citation impacts, as explored in the following subsections.

Narrative Complexity

Figure 12 presents the coefficient estimates for narrative complexity measures, which include the number of edges, the number of unique paths, and the log of the longest path length, analyzed separately for the non-causal and causal subgraphs. The results highlight that causal narrative complexity is positively associated with most publication and citation outcomes across both specifications (with and without year fixed effects), with particularly large effect sizes for publication in top 5 journals and citations. Among the three measures, the log of the longest path length has the largest effect size. For instance, with year fixed effects, a one-unit increase in the log of the longest path length is associated with around 0.05 increase in the probability of publication in a top 5 journal, which represents more than a fifth of the mean for this outcome (0.22). Similarly, the effect size for citations is substantial, with causal measures showing consistent positive and statistically significant coefficients.

In contrast, non-causal narrative complexity measures exhibit zero to negative coefficients across all outcomes. This suggests that general complexity unsupported by causal inference methods may not enhance publication or citation outcomes and, in some cases, may even hinder them.

Interestingly, the relationship with top 21–100 publications does not display the same consistent positive pattern. For that tier, causal narrative complexity measures have positive but small coefficients that are often statistically insignificant, particularly when year fixed effects are included. Non-causal measures remain consistently at or near zero, further indicating that lower-tier journals do not systematically reward narrative complexity. These results suggest that while complexity embedded in rigorously identified causal claims can improve placement in top-tier journals, its impact on mid-tier and field-specific outlets is limited.

Turning to citations, all causal narrative complexity measures are positively associated with citation counts, with large and statistically significant coefficients. This indicates that complexity grounded in credible causal relationships enhances longer-run academic influence. In contrast, non-causal measures are negatively associated with citations, particularly in the specification with year fixed effects, and are statistically insignificant without them. For example, the log of the longest path length in the causal subgraph has a substantial effect size on citations, demonstrating the importance of interconnected causal reasoning in generating scholarly attention.

(Figure 12)

Novelty and Gap Filling

Figure 13 presents the results for measures capturing novelty and gap filling. For novelty, we examine three measures: the proportion of new edges, the proportion of novel unique paths, and the proportion of novel subgraphs introduced by a paper. Consistently, causal novelty measures are positively associated with publication in top 5 journals, with large and statistically significant effect sizes, except for novel subgraphs, which exhibit a positive but statistically insignificant relationship. In contrast, non-causal novelty measures for top 5 journals are generally positive but often statistically indistinguishable from zero, highlighting the limited value placed on novelty unsupported by causal evidence.

For top 6–20 journals, the results differ. Non-causal novelty measures are consistently negative, suggesting that such novelty may detract from placement in mid-tier journals. For causal novelty, the results are mixed. Novel subgraphs show negative but statistically insignificant coefficients, while novel edges and paths exhibit positive but small effect sizes. These patterns suggest that top 5 journals are more likely to reward novelty supported by causal inference methods, while mid-tier outlets exhibit weaker preferences for novelty, particularly for causal subgraph-based novelty.

When the outcome is publication in top 21–100 journals, the results are statistically insignificant and mixed across both non-causal and causal measures. This further reinforces the idea that lower-tier journals do not systematically reward novelty, regardless of whether it is causally grounded.

Turning to citations, a stark and consistent difference emerges between causal and non-causal novelty measures. Causal novelty measures are all positive and statistically significant, with large effect sizes, demonstrating the importance of introducing new causal relationships in generating long-term scholarly attention. By contrast, non-causal novelty measures are uniformly negative, underscoring the limited influence of general novelty unsupported by causal identification on a paper’s academic impact.

The gap filling measure examines whether papers bridge previously disconnected concepts. Results show that top 5 journals reward both causal and non-causal gap filling, with positive and statistically significant coefficients for both types. However, for top 6–20 and top 21–100 journals, only the causal gap filling measure retains a positive or zero-to-positive relationship, while the non-causal measure is zero or negative. Interestingly, neither causal nor non-causal gap filling measures are significantly associated with citation outcomes, suggesting that bridging underexplored concepts does not consistently translate into long-term academic influence.

Overall, the results indicate that novelty is a critical factor for top 5 journal placement, particularly when supported by causal inference methods, while mid-tier journals may exhibit less consistent priorities for novelty. Similarly, causal gap filling is more likely to be rewarded across publication tiers, whereas non-causal gap filling is less influential outside of top 5 journals. These findings underscore the importance of methodological rigor in shaping the reception of novel or underexplored research contributions.³²

(Figure 13)

Conceptual Importance and Diversity

Figure 14 reports results for measures capturing conceptual importance and diversity, including average eigenvector or PageRank centralities, their variances, and the source–sink ratio of the knowledge graph.³³ These measures provide insight into whether focusing on central or peripheral concepts, or maintaining a balanced versus skewed causal structure, correlates with publication success and citation impact.

Papers that engage with more central concepts, as measured by higher mean PageRank and mean eigenvector centrality, tend to fare better in attracting citations. Both causal and non-causal versions of these measures are positively correlated with citation outcomes, though non-causal versions exhibit slightly larger coefficients. This suggests that engaging with well-established or influential concepts, even outside of a causal framework, enhances a paper’s longer-term academic impact.

The relationship between central concepts and publication outcomes is more mixed. Top 5 journals do not necessarily prefer highly central concepts; the coefficient for causal mean centrality is negative but statistically insignificant, while the coefficient for non-causal mean centrality is generally zero-to-positive and similarly statistically insignificant. This indicates that top-tier journals may prioritize novel or frontier topics over central, well-established concepts. Once published, however, papers centering on mainstream, well-known concepts tend to gain a stronger foothold in academic discourse, as reflected in their higher citation counts.

For top 6–20 and top 21–100 journals, non-causal mean centrality measures show positive correlations with publication outcomes, whereas causal versions exhibit coefficients close to zero. This highlights a potential preference for broader conceptual engagement without requiring rigorous causal identification at these journal tiers.

Variance in centralities provides insight into conceptual diversity. Greater diversity in conceptual centrality does not consistently raise the probability of top-tier publication but can sometimes benefit citations. Papers that mix central and peripheral concepts may appeal to diverse academic sub-audiences, enhancing long-term influence. However, for top 5 publications, higher variance in causal centralities tends to have a negative relationship, suggesting that highly diverse causal arguments may be viewed as less cohesive or rigorous by elite journals.

The source–sink ratio offers another perspective. For top 5 publications, a higher causal source–sink ratio (indicating more distinct causal sources converging on fewer outcomes) is positively correlated with publication outcomes, suggesting that balanced yet complex causal narratives are favored. In contrast, the coefficient for the non-causal source–sink ratio is negative, indicating that papers relying on non-causal sources for multiple effects are less likely to appear in top 5 journals. For citations, a higher causal source–sink ratio also exhibits a positive relationship, while the coefficient for non-causal source–sink ratio remains close to zero. For top 6–20 and top 21–100 publications, both causal and non-causal source–sink ratios exhibit coefficients close to zero, indicating limited relevance for these outcomes.

(Figure 14)

Conclusion

This paper introduces a graph-based approach to quantifying the structure and content of economic research, using a custom large language model to map both general and empirically supported claims in over 44,000 NBER and CEPR working papers. By analyzing the relationships between economic concepts and the methods used to substantiate claims, we establish new insights into how the discipline has evolved and how structural features of research predict academic reception.

First, the “credibility revolution” in economics has reshaped both how researchers establish claims and which claims gain traction. The proportion of empirically supported claims has risen dramatically, from about 4% in 1990 to nearly 28% in 2020, reflecting the growing emphasis on robust empirical identification methods. This shift reflects the increasing importance of credibility in shaping the direction and reception of economic research.

Second, the drivers of top-tier journal publication often differ from those of long-term citation impact. Journals such as the AER and QJE strongly reward the use of empirical methods, particularly when coupled with novel or underexplored concepts. Yet, once published, papers that engage with central and widely recognized topics tend to accrue more citations, revealing a persistent tension between pursuing frontier exploration to meet editorial priorities and addressing established domains to gain broader scholarly influence.

Third, our findings highlight the importance of narrative depth. We measure this using metrics such as the longest chain of reasoning in a paper, which are predictive of both top-tier publication and citations. In contrast, general narrative complexity unsupported by empirical claims often has no effect or is negatively correlated with these outcomes, suggesting that complexity alone is insufficient; it must be anchored in well-documented relationships.

Finally, bridging underexplored conceptual domains (“gap filling”) emerges as an important but a subtle factor. While editors reward efforts to connect previously neglected topics, these connections do not consistently lead to higher long-term citations unless they are situated within established debates or empirically supported frameworks.

Overall, our results highlight the evolving nature of academic evaluation in economics. Journals increasingly prioritize rigorous empirical support and innovative connections, yet broader citation impact often rewards alignment with recognized debates. Striking a balance between methodological rigor, novel contributions, and integration with central concepts appears essential for scholars seeking both visibility and lasting impact. This study offers a framework for understanding these trade-offs and navigating an increasingly competitive field, providing actionable insights into how economic knowledge develops and gains influence.

Figures and Tables

Appendix

Details on LLM-based Information Retrieval

Background on Large Language Models Large Language Models (LLMs) like GPT-4o-mini have significantly advanced natural language processing by enabling machines to understand and generate human-like text. Pre-trained on extensive datasets, including academic papers, books, websites, and other textual sources, these models capture the complexities of language, semantics, and context. This extensive pre-training allows LLMs to perform a variety of tasks, such as text summarization, translation, question answering, and information retrieval ([10], [30], [31], [46]).

In our study, we leverage the LLM’s ability to comprehend and extract structured information from unstructured text. By processing the first 30 pages of each economics working paper, the LLM identifies and extracts key metadata, methodological details, and causal claims. Unlike traditional NLP methods that rely on keyword matching or rule-based extraction, LLMs can understand complex language (often found in academic papers) and infer relationships between concepts, making them particularly effective for analyzing complex academic texts.

Prompt Design and Multi-Stage Extraction Process

Our information retrieval process involves a multi-stage approach, utilizing the LLM to extract and structure the necessary data efficiently.

Prompt Design

To effectively guide the LLM in extracting the required information, we developed comprehensive prompts that included detailed system and user instructions for each stage of the extraction process. The prompts were designed to specify the assistant’s role, define the task, provide definitions where relevant, enforce a structured response format, and set guidelines for accuracy and consistency.

Assistant’s Role and Task Definition We explicitly defined the assistant’s role as an expert in economics paper analysis, specializing in interpreting complex academic content and extracting nuanced information. This role specification was intended to align the LLM’s outputs with the expectations of an expert-level analysis. We provided detailed task definitions, instructing the assistant to analyze the text content of a provided economics paper and extract specific information related to metadata, causal claims, research methods, variables, data measurements, identification strategies, and data usage.

Inclusion of Canonical Definitions To ensure accurate and consistent classification of research methods, we included canonical definitions of key methodologies commonly used in economics, such as Randomized Controlled Trials (RCTs), Difference-in-Differences (DiD), Instrumental Variables (IV), Regression Discontinuity Designs (RDD), and others. By providing these standard definitions, we aimed to minimize ambiguity and enhance the reliability of method classification.

Structured Response Format We specified that the assistant’s response should adhere to a structured JSON schema with predefined fields and data types. This structured format was crucial for facilitating subsequent data processing, aggregation, and analysis across the large corpus of papers. The JSON schema included detailed definitions and instructions for each field to ensure accurate and consistent extraction. One of the instances where it works the best is to provide an array of causal edges in a paper. Other useful application is specifying the data type: boolean (e.g. whether paper uses any data), categorical (e.g., source of data: private or public sector), numeric (e.g., number of authors) or string response (e.g., name of the data provider).

Guidelines for Accuracy and Consistency We emphasized strict adherence to the schema, accuracy, and clarity in the assistant’s responses. We instructed the assistant to use the canonical definitions when classifying methods and to ensure that all required fields were accurately and completely filled out. These guidelines were essential to maintain the quality and consistency of the extracted data.

Overview of Prompts and Instructions

The following provides an overview of the system instructions provided to the LLM for each stage of the extraction process. Due to space constraints, we present summarized versions of the prompts. The complete system prompts and JSON schemas will be included in the replication package accompanying this paper.

Stage 1: Curated Summary In Stage 1, the assistant was instructed to analyze the first 30 pages of the paper and extract a curated summary of key elements. The system instructions included:

• Assistant’s Role: An expert assistant specializing in analyzing economics papers.

• Task: Extract specific information related to research questions, causal identification strategies, data usage, data accessibility, acknowledgements, and metadata.

• Guidelines: Provide clear, detailed, and information-rich responses for each field. Focus exclusively on specified sections. Adhere strictly to specified formats and instructions.

• Fields to Extract: Research questions from the abstract, introduction, and full text; causal identification information; causal claims; framing and policy implications; data and units of analysis; data accessibility; institutional and author-level information; acknowledgements.

Stage 2: Causal Graph Retrieval In Stage 2, the assistant was tasked with extracting detailed causal relationships from the summaries provided in Stage 1. The system instructions included:

• Assistant’s Role: An expert assistant specializing in analyzing economics papers.

• Task: Extract an exhaustive list of causal relationships from provided texts, capturing all intermediate steps, mediators, confounders, and other relevant nodes in the causal graph (DAG).

• Guidelines: Be exhaustive in extraction. Use exact terms from the authors. Provide each causal relationship in a structured format with specified fields.

• Fields to Extract: For each causal relationship, include the causal claim, cause, effect, type of causal relationship, whether evidence is provided, sign of impact, effect size, statistical significance, causal inference method, sources of exogenous variation, level of tentativeness.

Stage 3: Data and Accessibility In Stage 3, the assistant was instructed to extract structured information regarding data sources and accessibility from the data-related summaries in Stage 1. The system instructions included:

• Assistant’s Role: An expert assistant specializing in analyzing economics papers.

• Task: Extract specific pieces of information related to data and units of analysis, and data accessibility.

• Guidelines: Carefully extract information for each field, adhering strictly to definitions and instructions. Use exact wording from the text when possible.

• Fields to Extract: Data usage indicators, total number of observations, units of analysis, data granularity, temporal and geographical context, data ownership, data accessibility, ethical considerations.

Matching LLM Output to JEL Codes and OpenAlex Topics

After the LLM extracted the causal claims and provided free-text descriptions of the source and sink variables, we needed to standardize these variables to facilitate aggregation and systematic analysis across the corpus. This standardization was achieved by mapping the variable descriptions to official Journal of Economic Literature (JEL) codes and OpenAlex Topics.

Choice of Standardization Methods

We considered several options for standardizing the terms used in the source and sink variables, including JEL codes, OpenAlex Topics, Concepts, and the OECD Glossary of Statistical Terms. Each option had its advantages and limitations. JEL codes are well-understood within the economics community and facilitate interpretability but are relatively broad in classification. OpenAlex Topics offer more granularity with around 4,500 topics but are less familiar to economists. Concepts provide even more detail with approximately 60,000 terms but are being deprecated by OpenAlex and are not widely recognized in economics. The OECD Glossary contains about 6,700 economics-related terms but may be biased towards statistical concepts.

After careful consideration, we opted to focus on JEL codes for standardization. JEL codes were chosen as our primary method due to their familiarity and acceptance within the economics discipline, facilitating interpretation and communication of results.³⁴

Embedding-Based Matching Methodology

To map the free-text variable descriptions to the standardized codes, we employed an embedding-based matching approach using vector representations of the texts. We utilized the OpenAI text embedding model text-embedding-3 large, which generates 1,024-dimensional vector embeddings that capture the semantic meaning of the text. Embeddings were generated for: (i) the free-text descriptions of the source and sink variables extracted by the LLM, (ii) the official descriptions of JEL codes, enhanced by concatenating descriptions from higher-level codes to provide richer context, and (iii) the summaries of OpenAlex Topics.

We calculated the cosine similarity between the embeddings of the variable descriptions and the embeddings of the JEL codes and OpenAlex Topics.³⁵ We assigned the variable to the codes/topics with the highest similarity scores.

Advantages and Limitations of the Embedding-Based Approach

By leveraging embeddings, we moved beyond simple keyword matching, which can be limited by variations in terminology and phrasing. The embedding-based approach captures the semantic meaning of the texts, allowing us to match variable descriptions to standardized concepts even when different terms are used to describe similar ideas (e.g., unemployment rate” vs. joblessness”). This method enhanced the robustness of our matching process, reducing the impact of typos, synonyms, and variations in language. It allowed us to systematically standardize a large number of variable descriptions across the corpus, facilitating cross-paper comparisons and aggregations.

While the embedding-based matching approach offers significant advantages, there are limitations to consider. The quality of the matches depends on the accuracy of the embeddings and the chosen similarity threshold, which in our case is the one with highest similarity. By focusing on only one matching concept, we are imposing a structure on the latent causal graph. It may well be that a source or a sink could be captured by multiple JEL codes. However, for simplicity and consistency across papers, we decided to capture only the best match. Future research can consider setting a threshold, e.g., 0.85, beyond which multiple JEL codes can be accepted. However, this comes with its own set of hyper-parameter selection: there is a trade-off between precision and recall; a higher threshold increases precision but may miss relevant matches, while a lower threshold increases recall but may include irrelevant matches.³⁶

By using both JEL codes and OpenAlex Topics, we leveraged the strengths of each system, with JEL codes providing familiarity and OpenAlex Topics offering granularity. Overall, the use of embeddings was instrumental in standardizing and operationalizing our source and sink variables.

Validation and Quality Assurance

To ensure the reliability of the extracted data, we implemented validation checks at each stage. The structured outputs were parsed and checked for compliance with the specified schemas. In cases where inconsistencies or missing data were detected, the prompts were refined, and the extraction was repeated.

We also conducted manual reviews of a sample of the extracted data to assess accuracy. This included checking the correctness of the causal claims extracted, the appropriateness of the mapped JEL codes, and the consistency of metadata. The feedback from these reviews informed further refinements to the prompts and extraction process.

A large scale validation of our approach will follow in future drafts. This includes contacting corresponding authors to validate the causal graphs in their own papers.

Limitations and Considerations

While the use of LLMs provides significant advantages in processing large volumes of complex text, there are limitations to consider. The LLM’s extraction is dependent on the quality and clarity of the original text; ambiguities or omissions in the papers may lead to incomplete extraction. The LLM may occasionally misclassify or misinterpret information, particularly with nuanced methodological details. While we collected additional attributes such as effect sizes and statistical significance, these features were experimental and not used in the main analysis due to variation in reporting standards.

Additionally, the embedding-based matching approach for standardizing variables may not capture all nuances of the economic concepts involved. There is a risk of misclassification if the variable descriptions are ambiguous or if the embeddings do not accurately represent the semantic content. Despite these limitations, we believe that the overall methodology provides a robust framework for large-scale analysis of economic research.

Replication Package

The full system prompts, JSON schemas, and code used in the extraction process will be included in the replication package accompanying this paper. Researchers interested in replicating or extending our analysis can refer to these materials for detailed guidance.³⁷

Validating information retrieval

Validation with the [18] Dataset

To validate the accuracy of our AI-based information retrieval methods, we compared our dataset with an external dataset from [18], which examines p-hacking, data type, and data-sharing policies in economics. Their dataset includes 1,106 articles published in leading economics journals between 2002 and 2020, with detailed annotations on the empirical methods used and the fields of study.

We matched our dataset with theirs using paper titles, employing both direct matches and fuzzy matching techniques to maximize the number of matched papers. In total, we matched 307 papers between the two datasets. For these matched papers, we compared our classifications of empirical methods—Difference-in-Differences (DiD), Randomized Controlled Trials (RCT), Regression Discontinuity Design (RDD), Instrumental Variables (IV)—and fields—Urban Economics, Finance, Macroeconomics, Development—with those from [18] to assess the accuracy of our information retrieval.

We calculated two evaluation metrics: Accuracy and the F1 Score, where the ground truth labels are those from [18]. The results are presented in Table A2.

Variable	Accuracy	F1 Score
Method: DiD	0.7762	0.8447
Method: RCT	0.7063	0.8269
Method: RDD	0.9371	0.9644
Method: IV	0.7413	0.8183
Field: Urban Economics	0.9138	0.9545
Field: Finance	0.8788	0.9295
Field: Macroeconomics	0.9744	0.9870
Field: Development	0.6643	0.7937

The results indicate that our information retrieval methods achieve high levels of accuracy and F1 scores in identifying both empirical methods and fields of study. In particular, the identification of RDD methods and classification of papers in Macroeconomics and Urban Economics show excellent performance, with accuracy and F1 scores exceeding 0.90. These findings provide confidence in the reliability of our AI-based extraction and classification processes in at least two dimensions of our analysis: fields and methods.

Validation with the Plausibly Exogenous Galore Dataset

To further validate our information retrieval methods, we compared our data on causal claims with an external dataset, the Plausibly Exogenous Galore dataset³⁸. This dataset includes entries for 1,435 papers (as of August 2024), each with the main Left-Hand Side (LHS) and Right-Hand Side (RHS) variables, as well as the primary source of exogenous variation, as identified by authors and contributors. Unlike our dataset, which captures causal claims at the paper-claim level, the Plausibly Exogenous Galore dataset records information at the paper level, often focusing on key variables rather than the complete knowledge graph of causes, effects, and sources.

We matched 485 papers from our dataset to entries in the Plausibly Exogenous Galore dataset using exact and fuzzy title matching. To facilitate a meaningful comparison, we aggregated our data at the paper level by concatenating all causes, effects, and sources of exogenous variation. This resulted in structured phrases: “The causes are: <cause>; <cause>; ...” for causes, “The effects are: <effect>; <effect>; ...” for effects, and “The source(s) of exogenous variation are: <source>; <source>; ...” for sources. Similarly, the Plausibly Exogenous Galore data was formatted as “The cause is: <RHS>” for the RHS (cause), “The effect is: <LHS>” for the LHS (effect), and “The source(s) of exogenous variation are: <source of exogenous variation>.”

Using OpenAI’s text-embedding-3-large embeddings model, we generated embeddings for each component separately (causes, effects, and sources of exogenous variation) and calculated cosine similarity scores for each component between our data and the Plausibly Exogenous Galore dataset. We computed the average, minimum, maximum, median, and standard deviation of these similarities.

The results (Table A3) show that while cause and effect similarities were moderate, reflecting variability in how these are represented across claims within a paper, the source of exogenous variation achieved a higher similarity score. This result supports the reliability of our method for capturing consistent exogenous variation sources, as these tend to vary less across claims. These findings reinforce our information retrieval system’s ability to align with external validations, particularly for core causal identifiers.

Variable	Mean Similarity	Median	Std Dev	Min	Max
Cause Similarity	0.6140	0.6245	0.1127	0.2285	0.8798
Effect Similarity	0.6386	0.6467	0.0893	0.3795	0.8452
Exogenous Variation Similarity	0.8014	0.8142	0.0975	0.4920	0.9863

Appendix Tables and Figures

Journal Category	Mean Percentile	Median Percentile	Standard Deviation
Top 5	62.18	68	25.86
Top 6–20	57.06	61	27.97
Top 21–100	52.12	53	27.37

Note: This table presents the validation results of our information retrieval methods compared to the ground truth provided by [18]. Variable indicates either the empirical method (DiD, RCT, RDD, IV) or the field of study (Urban Economics, Finance, Macroeconomics, Development). Accuracy measures the proportion of correctly classified instances, while F1 Score represents the harmonic mean of precision and recall. Higher values indicate better performance.

Note: This table presents the cosine similarity results between our dataset and the Plausibly Exogenous Galore dataset, which records plausibly exogenous sources, main causal factors, and outcomes in the economics literature. Mean Similarity indicates the average cosine similarity score, while Std Dev shows the standard deviation of these scores, capturing the variability. Min and Max represent the range of similarity scores across matched papers.