The Promise and Peril of Generative AI: Evidence from GPT as Sell-Side Analysts

AI Disclaimer: We have utilized OpenAI’s GPT products to assist with obtaining research data and copyediting this manuscript. All analyses, interpretations, and conclusions presented in the manuscript are solely the work of the authors.

“Despite its capabilities, GPT-4 has similar limitations as earlier GPT models. Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors). Great care should be taken when using language model outputs, particularly in high-stakes contexts...” – OpenAI (2024) (https://openai.com/index/gpt-4-research/)

Introduction

This study seeks to glimpse inside the “black box” of large language models (LLMs) to understand their information processing in the context of financial disclosures. Focusing on OpenAI’s GPT-4, we examine how cutting-edge LLMs analyze firms’ earnings press releases to predict future earnings. Specifically, we address two questions: (1) Do GPT-generated earnings forecasts outperform human analysts in accuracy? and (2) How is GPT’s performance related to its approaches to processing textual and quantitative information?

The introduction of advanced LLMs has revolutionized natural language processing (NLP). These transformer-based models, with their innovative self-attention mechanisms ([47]), have shown remarkable proficiency in understanding and generating human-like content in finance, healthcare, and education (OpenAI 2024). Recent finance and accounting research highlights these models’ unparalleled abilities in sentiment analysis, stock return prediction, corporate disclosure summarization, and risk assessment.²

Earnings press releases serve as a key channel for firms to report financial performance, offering both narrative insights and quantitative data ([16]). Analysts rely on these releases for earnings predictions and stock recommendations, yet cognitive biases, information overload, and incentives often undermine their accuracy ([23]; [29]). LLMs’ potential to objectively process large data has spurred interest in applying them to financial analysis, including interpreting disclosures and forecasting earnings. Recent studies show that LLMs can rival or surpass analysts in certain financial tasks (e.g., [35]; [28]; [41]). However, limited research explores the mechanisms behind LLMs’ performance.

Our study fills this void by tying GPT-4’s performance to its information processing across three key dimensions: textual ranking, quantitative analysis, and the knowledge cutoff.

First, we examine which sentences in a press release GPT-4 prioritizes for forecasting earnings. Theoretically, transformer models are designed to process all textual content equally, without inherent preferences for order, length, or other characteristics ([47]). However, training on large, general-purpose corpora can instill tendencies for models to prioritize certain text ([4]), shaping their information processing and overall performance.

Second, we examine the financial metrics GPT-4 employs to derive earnings forecasts. While LLMs like GPT-4 are developed for general textual processing across diverse contexts ([6]; OpenAI 2024), their quantitative analysis capabilities depend heavily on the availability of quality, domain-specific training data ([31]).³ This raises questions about how the richness of public data shapes GPT-4’s quantitative strategies. By analyzing its selection of ratios and trends in relation to firms’ information environments, we provide insights into GPT-4’s quantitative processing and its implications for forecasting performance.

Third, we examine GPT-4’s knowledge cutoff, which marks the boundary between its training data and novel information. Analyzing GPT performance pre- and post-cutoff can offer two insights. Pre-cutoff analysis clarifies whether GPT-4 genuinely processes information or merely reproduces stored data to match analyst performance. Post-cutoff analysis offers a true hindsight-free evaluation. While anonymizing pre-cutoff data helps simulate novelty, it cannot fully eliminate hindsight bias, as (1) GPT-4 may detect subtle cues, and (2) retrospectively applying modern analytical methods from its training to historical data creates a “time-travel” advantage.⁴ This makes the cutoff crucial for a fair assessment of the model’s capabilities. Our analysis draws on a sample of 6,848 earnings press releases from 1,000 randomly selected firms over a two-year period around GPT-4 Turbo’s knowledge cutoff date, April 30, 2023. For each release, we employ the model’s API to analyze the full text alongside the firm’s recent financial statements to forecast next-quarter earnings. Through chain-of-thought (CoT) prompting, we instruct GPT to 1) identify the three most relevant sentences in the press release for earnings forecasting, 2) conduct quantitative analysis and identify the three key ratios or trends for earnings forecasting, and 3) predict the value and directional change of earnings.

For our first research question, we compare GPT-4’s GAAP earnings forecasts to those of analysts. GPT offers distinct advantages: it relies on learned patterns from training ([47], [6]), potentially mitigating external pressures and behavioral biases such as optimism and herding (e.g., [17], [24]). It also leverages a vast reservoir of knowledge up to its cutoff date. However, its inability to access new information, engage with management, or incorporate analysts’ private insights makes it ex-ante unclear whether it can match analysts’ performance, especially post-cutoff.

We compare GPT-4 and analysts’ forecast accuracy, measured as the absolute difference between forecasts and actual GAAP EPS, scaled by lagged prices. On average, GPT-4’s forecasts are less accurate, with a mean absolute forecast error of 0.048 versus analysts’ 0.032. This underperformance suggests that GPT-4’s vast public knowledge cannot fully compensate for its lack of management access, new data, and analysts’ private insights. Importantly, this result holds across pre- and post-cutoff periods. The pre-cutoff underperformance confirms that GPT-4 actively processes data rather than merely regurgitating stored information.⁵

Turning to our second research question, we explore how GPT-4’s performance is tied to its information processing strategies. We first examine its textual ranking approach, focusing scenario analysis and stress testing, enhancing forecasting and risk assessment ([43]). on the characteristics of sentences it prioritizes for earnings forecasting. GPT-4 tends to select sentences that appear early, are more readable (lower FOG index), and are rich in numerical and guidance-related content, aligning with human preferences. It also prioritizes longer sentences and those with more negative tones, suggesting GPT-4’s unique ability to capture complex and nuanced textual content. Importantly, GPT-4’s textual ranking approach remains consistent across its knowledge cutoff and firms with varying information environments, reflecting a generalized textual processing strategy independent of domain-specific data.⁶

Next, we examine the financial ratios or trends GPT-4 uses to forecast earnings, a key aspect of financial statement analysis. Analysts, adept at identifying relevant metrics, provide a natural benchmark for evaluating GPT-4’s quantitative analysis. GPT-4 identifies 35 unique metric types, with net profit margin and return on equity being the most frequently used. To assess alignment with analysts, we use I/B/E/S Detail to identify the three non-EPS metrics most frequently reported by individual analysts for each firm-quarter. GPT-4’s alignment score, ranging from zero to three overlapping metrics, averages 1.4. Notably, the level of financial statement aggregation and numerical irregularities, measured by deviations from Benford’s law ([37]), negatively correlate with alignment, showing GPT-4’s challenges in handling low-quality data. More importantly, unlike its consistent textual ranking, its metric selection is highly variable, with lower alignment for firms with poor analyst coverage. This suggests GPT-4’s reliance on domain-specific training data to emulate analysts’ quantitative strategies.

Building on these findings, our multivariate analysis tests how GPT-4’s performance is related to its textual ranking, quantitative analysis, and knowledge cutoff. Controlling for press releases’ general attributes, we find that GPT-4’s forecast accuracy improves when its selected sentences appear earlier and include more numerical content but declines when they are longer and exhibit more negative tones. This duality highlights the nuanced role of GPT-4’s textual processing in its performance. GPT-4’s underperformance is also more pronounced for firms where its metric choices poorly align with analysts, suggesting that its accuracy is limited by its (in)ability to emulate analysts’ quantitative strategies. Additionally, we find some evidence of a widening accuracy gap post-cutoff, particularly for press releases issued four quarters later, highlighting the importance of knowledge cutoff in fair model evaluation. Interestingly, when we control for firm size and analyst coverage, the results on textual ranking and knowledge cutoff remain robust, while those on quantitative analysis lose their significance. This suggests that GPT-4’s textual ranking has generalized implications for performance across firms, whereas the relationship between its quantitative analysis and performance reflects the availability of domain-specific training data, which varies by firms.

To further validate and triangulate our findings, we conduct several supplementary analyses. First, to address concerns about whether GPT-4 accurately reports the sentences and metrics it prioritizes versus what it internally processes, we prompt it to justify each choice for relevance to earnings forecasting. Word clouds from these justifications confirm consistency between its reported selections and internal processes. Second, we test the robustness of our results by comparing GPT-4’s non-GAAP forecasts to analysts’ street earnings forecasts, finding that our main inferences hold with this specification.⁷ Third, we prompt GPT-4 for confidence scores on its earnings forecasts and find these scores largely align with its sentence prioritization, reflecting confidence in its textual ranking approach. However, the scores are not significantly related to quantitative metric alignment, suggesting a lack of awareness of potential performance pitfalls. Notably, confidence scores are elevated in the fourth quarter post-cutoff, underscoring the need for improved confidence calibration.

A contemporaneous study by [28] also examines GPT-4’s role in financial analysis. They focus on predicting directional earnings changes using anonymized financial statements dating back as far as 1968, without narrative context. In contrast, we task GPT-4 with forecasting numerical earnings using full press releases and financial statements near its knowledge cutoff. While Kim et al. highlight GPT-4’s strength in trend detection, we uncover its challenges in generating numerical forecasts. Together, these studies show how task design and data context shape GPT performance. To reconcile the two studies, we test GPT-4’s directional earnings change forecasts as a final step in our analyses. We estimate GPT-4’s accuracy at 49.3%, closely aligning with Kim et al.’s ([28]) reported 50.25% in 2020, part of a general decline in GPT forecast accuracy over time. However, GPT-4 still underperforms analysts in directional forecasts, who achieve a prediction accuracy of 71.1%.

Taken together, we present the first in-depth analysis of GPT’s processing of financial disclosures across three dimensions: textual ranking, quantitative analysis, and the knowledge cutoff. GPT’s textual ranking is consistent across diverse information environments, while its quantitative analysis relies on domain-specific training data, which vary significantly across firms. This reveals GPT’s strength in general textual tasks but underscores its limitations in specialized applications, supporting the need for customized models like BloombergGPT. Notably, GPT’s general textual ranking can sometimes impede accuracy, and poor alignment in quantitative analysis with analysts contributes to underperformance. Additionally, there is some evidence of declining performance after the knowledge cutoff, highlighting the importance of hindsight-free evaluation. Our findings shed light on GPT’s strengths and limitations in financial forecasting and add to broader discussions on AI interpretability and transparency, essential for building trust in AI-driven analysis ([19]).

Our study also contributes to the growing literature on machine processing in financial text analysis. Early research focuses on traditional methods to quantify clarity, sentiment, and topics in disclosures (e.g., [30, 34, 42]). More recently, advanced models have enabled deeper insights. For example, [25] show that FinBERT, trained on financial data, outperforms dictionary-based methods in sentiment analysis. By analyzing GPT-4’s processing of financial disclosures, our study extends this literature and offers insights to support future research on AI’s transformative impact on financial analysis.

Finally, our evidence has implications for the role of sell-side analysts as information intermediaries in the era of advanced LLMs. Unlike LLMs, which draw on extensive training data but lack management access, analysts benefit from private insights, enabling more nuanced forecasts. While LLMs show potential in identifying critical information within disclosures, their reliance on public data and limited interpretative depth highlights limitations, particularly in firms with poor information environments. Therefore, while LLMs are valuable in data-rich contexts, they are not yet substitutes for analysts in complex, opaque scenarios.

The rest of the paper is structured as follows. Section 2 discusses the background and literature. In Section 3, we provide detailed information about our sample and how we obtain information from the GPT model. In Section 4, we present the main empirical results of the paper. In Section 5, we conclude.

Background and Literature Review

Advanced LLMs

Transformer-based models like GPT represent a major advancement over previous natural language processing models, such as Recurrent Neural Networks (RNNs) or Long Short-Term Memory Networks (LSTMs) ([21]). Unlike these sequential models, transformers process text in parallel, which allows them to capture complex relationships across the entire text. This makes them ideal for challenging language tasks ([47]).

Central to transformer models is the self-attention mechanism, which assigns varying levels of importance, or “attention,” to words or tokens based on task relevance. This allows models to dynamically focus on meaningful word connections, irrespective of their position, by calculating weights that emphasize contextually significant phrases [14].

These weights or ranks are initially established during pretraining, a process where models learn general language patterns from large, diverse corpora. As a result, transformer models are often adept in general-purpose textual tasks across different contexts, independent of specialized data. However, training on human-generated content inevitably introduces human-like patterns and tendencies, shaping the models’ performance.

While pretraining based on broad datasets enables transformer models to recognize general language patterns and financial terminology [1, 18, 38], they are not inherently optimized for domain-specific tasks like quantitative financial analysis. Their effectiveness in specialized tasks, which demand precise quantitative reasoning, is limited unless augmented by the availability of rich, domain-specific data (Wei et al. 2022).⁸ However, the availability of such data varies significantly across firms, introducing considerable disparities in model performance in quantitative tasks.

Another critical factor in evaluating LLMs is the knowledge cutoff, as it marks the boundary between a model’s training data and new, unseen information [7]. It defines the latest data and analytical methods incorporated, shaping the model’s capacity to generate accurate responses. Transformer models, adept at identifying relationships, may detect subtle cues in historical data, even if disguised [4]. Therefore, fair evaluation requires caution when using pre-cutoff data, as it is part of the training set, and when applying modern analytical methods to old data, as this could introduce hindsight bias—akin to time travelers exploiting yet-undeveloped techniques for unfair advantage.

In sum, transformer-based LLMs like GPT-4 are primarily designed for general text processing across diverse contexts. While they show potential in domain-specific quantitative tasks, their performance largely depends on the availability and quality of relevant training data. Furthermore, the knowledge cutoff sets a clear boundary between training data and novel inputs, making hindsight bias a significant concern when using pre-cutoff data to evaluate models.

Relevant Literature

Recent Research Using GPT in Accounting and Finance

Since OpenAI introduced ChatGPT in November 2022, researchers in accounting and finance have explored its potential across diverse applications. GPT has been applied to classify and extract textual information, predict financial outcomes, and perform quantitative analysis.

For instance, [32] analyze the sentiment of news texts using GPT, employing the extracted signals to forecast future stock returns. [26] utilize ChatGPT to extract risk information from corporate disclosures, demonstrating that this information can aid in predicting returns. Blankespoor et al. (2024) attempt to use a generative AI tool to identify disclosures produced by other AI systems. Additionally, [9] parse social media data to identify investors’ trading strategies.

Beyond text classification, GPT has been used to generate predictions and analyses. [46] apply ChatGPT to analyze loan assessments and generate credit default predictions, showing that including GPT-generated texts significantly improves prediction accuracy. Bybee (2024) uses GPT to generate economic forecasts, demonstrating that these forecasts closely resemble those produced by humans. [27] generate summaries of financial disclosures and measure the “bloat” by comparing summaries with the original texts. More relevant to our study are works comparing GPT’s performance to that of humans.

Bai et al. (2023) use ChatGPT to answer questions derived from text, offering insights comparable to human responses. [12] assess GPT-4’s ability to act as data analysts in tasks such as converting natural language questions into SQL queries or automatic chart summarization, finding its performance comparable to humans with the potential to replace data analysts. [35] examine GPT-4’s analysis of conference call texts from 2015 to 2023, showing that it is able to replicate analytic tasks by trained analysts and add economic value by identifying overvalued and undervalued stocks.

Comparing Traditional Machine Learning, GPT, and Human Analysts

A growing body of research highlights the potential of machine learning and AI-based tools to outperform human analysts in forecast accuracy. For example, [2] and [3] and [11] demonstrate that machine learning techniques can yield forecasts with lower errors. [8] also show that combining machine learning with human analysts can produce better predictions than either approach alone.

While prior research has predominantly focused on traditional machine learning models trained to generate numerical forecasts, LLMs like GPT-4 are not explicitly trained for this task. The ability of GPT-4 to produce accurate forecasts remains an open question. [28] provide evidence that GPT can effectively analyze financial statements, with its directional predictions of earnings growth surpassing those of human analysts. However, they also find that GPT’s relative performance has declined over time. This is potentially due to two factors: (1) look-ahead bias, where GPT may retrieve earnings-related information from its training data, and (2) improvements in human analysts’ methodologies, such as the gradual adoption of machine learning techniques, narrowing the performance gap.

Our study builds on this line of research by focusing on recent data around GPT’s knowledge cutoff period, minimizing the influence of historical variations in analysts’ information collection and analysis methodologies. This approach allows us to better isolate the potential impact of GPT’s training data on its forecasting performance. Additionally, our study diverges from [28] by focusing on the numerical accuracy of GPT’s forecasts, rather than directional predictions. We evaluate forecast accuracy using absolute forecast errors and analyze the difference in absolute forecast errors between GPT and human analysts to assess their relative performance. Finally, we conduct a detailed examination of the underlying information processing strategies within these models, which are often treated as black boxes.

Data and Prompts

Sample Selection

Table 1 outlines the sample selection process for our study. We begin with a comprehensive sample of quarterly earnings announcements from firms covered by Compustat, CRSP, and I/B/E/S, spanning the four quarters before and after GPT-4 Turbo’s knowledge cutoff date, April 30, 2023. This initial sample includes 68,514 observations from firms reporting across these databases. Next, we obtain the full text of earnings press releases by merging this dataset with EDGAR’s 8-K filings, focusing on disclosures made under Item 2.02, which requires firms to file earnings press releases. This step refines the sample to 49,192 observations, ensuring the availability of textual data for GPT-4’s analysis.

Step	Role	Content
	Panel A. Our Complete Prompt to GPT
Step	Role	Content
1	System	You are a financial analyst, and your task is to perform financial statement analysis.
2	User9	You will receive the following data: 1. Press Release Discussions: Key points from a public firm’s press release about its quarterly financial performance. 2. Financial Statements: The firm’s balance sheet and income statement for the current quarter (t) and previous quarters (t-1 and t-2).
		1) Press-Release Discussions EX-99.1 2 exhibit991_sibn3312023earn.htm EX-99.1 Exhibit 99.1 SI-BONE, Inc. Reports Financial Results for the First Quarter 2023 and Increases Annual Guidance Annual guidance increased to $128 million - $131 million, implying annual growth of 20%-23% SANTA CLARA, Calif. May 1, 2023 - SI-BONE, Inc. (Nasdaq:SIBN), a medical device company dedicated to solving musculoskeletal disorders of the sacropelvic anatomy, today reported financial results for the quarter ended March 31, 2023. Recent Highlights Record worldwide revenue of $32.7 million for the first quarter 2023, representing approximately 46% increase over the corresponding period in 2022 Record U.S. revenue of $30.5 million for the first quarter 2023, representing approximately 50% increase over the corresponding period in 2022 Gross margin of 82% for the first quarter 2023 Over 950 U.S. active surgeons in the first quarter 2023, representing approximately 40% growth over the corresponding period in 2022 Extended first-generation iFuse patent protection to December 2025, with iFuse-3D patent coverage to September 2035 Celebrating our 15-year commitment to pioneering solutions for sacropelvic conditions through the industry’s most innovative portfolio, with over 80,000 procedures completed by over 3,000 surgeons worldwide I am thrilled with our stellar start to 2023, as record procedure demand in the quarter allowed us to deliver approximately 50% U.S. revenue growth and achieve significant operating leverage, said Laura Francis, Chief Executive Officer of SI-BONE. The continued trend of accelerating revenue growth over the last several quarters is a testament to the perseverance and grit of our dedicated team members and the strong reception for our expanded portfolio of solutions. With yet another quarter of record increase in our active surgeon base combined with the favorable reimbursement framework, we are at an inflection in the business. These trends solidify my conviction in our ability to deliver strong and sustainable growth and drive operating leverage as we progress through the year.” First Quarter 2023 Financial Results Worldwide revenue was $32.7 million in the first quarter 2023, a 46% increase from $22.4 million in the

To facilitate a consistent comparison, we exclude observations where analysts’ GAAP earnings forecasts (GPS) are missing in I/B/E/S, leaving a reduced sample of 47,485 firm quarters. We focus on analysts’ GAAP EPS forecasts in our main analysis because GPT-4’s EPS predictions are also GAAP-based. This decision avoids comparing GPT-4’s GAAP forecasts to I/B/E/S street earnings, which are adjusted for non-GAAP items. [5] highlight this critical issue and suggest that nearly all analysts now provide GAAP EPS forecasts, making this a proper comparison. Moreover, analysts’ GAAP forecast accuracy tends to be lower than their street earnings forecast accuracy [5], offering a conservative “lower bound” benchmark to compare GPT-4’s performance with.

To balance computational feasibility and representativeness, we randomly select a sample of 1,000 firms with at least one quarter of data available both before and after April 30, 2023. This stratification ensures a balanced dataset for evaluating GPT-4’s performance across pre- and post-cutoff periods, yielding 7,085 unique firm-quarters with necessary test variables.

Within this selected sample, GPT successfully generates GAAP EPS forecasts for 6,848 observations, while 237 observations are excluded due to GPT’s inability to provide valid forecasts. The earnings press releases associated with our final sample of 6,848 firm-quarters contain a total of 309,285 sentences, highlighting the depth and richness of the textual information used in the analysis.

Chain-of-Thought (CoT) Prompt Design

We employ a chain-of-thought (CoT) prompting technique to elicit responses from the GPT-4 Turbo API. Originally proposed by [44], CoT prompting enables LLMs to break complex problems into intermediate steps, solving each sequentially before arriving at a conclusion. This method significantly enhances the accuracy of LLMs across various tasks.

Our CoT prompt is structured into several important components to guide the model in a systematic way. First, we introduce the task, specifying that the model will analyze two types of inputs: textual content from earnings press releases and numerical data from financial statements. This initial step ensures clarity and provides the model with proper context. Next, we present the model with cleaned text from earnings press releases. To prepare this input, we preprocess the filings by removing all HTML tags, retaining only the clean text. We extract the text from the beginning of each press release up to the end-of-release section containing boilerplate disclaimers for forward-looking information. In addition, we exclude financial tables and appended statements presented in unstandardized formats, allowing the model to focus exclusively on the textual content for the textual ranking task. This preprocessing step also enables us to feed the model financial statement data in a standardized format next.

Following the textual input, we provide financial data, including balance sheets and income statements, in a standardized format. Using Compustat quarterly data, we organize these items according to GAAP reporting standards, similar to the method of [28].

For each firm, we include data from the current and prior quarters for the balance sheet and from the current and prior two quarters for the income statement. This structured approach ensures consistency and facilitates effective analysis.

The main part of the prompt leverages the CoT technique to engage GPT-4 in a series of systematic tasks designed to test its analytical capabilities:

1. Textual ranking: Identify the three most important sentences from the press release for predicting future earnings and provide justifications for their significance.

2. Quantitative Analysis: Perform quantitative analysis, including the formulas and calculations for the top three ratios or trends deemed most relevant to the task.

3. EPS Prediction: Predict both the direction and actual value of the firm’s next-quarter earnings per share (EPS) under GAAP and non-GAAP accounting standards, along with a confidence score for each prediction.

To ensure robustness, we spot-check the responses and find that the model interprets our prompt effectively, delivering informative and coherent answers. Panels A in Appendix A illustrate the full prompting process, while Panels B showcases examples of GPT responses, demonstrating its ability to address the tasks as instructed. This structured and methodical approach enables a comprehensive evaluation of GPT-4’s capacity to integrate and analyze diverse financial information sources.

Empirical Analyses

Our empirical analyses proceed in two steps. First, we address our first research question by examining whether GPT-4’s earnings forecasts outperform those of human analysts in accuracy and reliability. Second, we turn to our second research question, analyzing the relationship between GPT-4’s forecasting performance and its underlying processing of textual and numerical information. This analysis investigates variations from three dimensions: textual ranking, quantitative analysis, and the implications of the model’s knowledge cutoff. This two-step approach allows us to address both the overall effectiveness of GPT-4 as a forecasting tool and the factors influencing its performance.

GPT vs. Analysts

To address our first research question, we evaluate GPT-4’s forecast performance using absolute forecast error as the measure of accuracy. We compare GPT’s performance with that of analysts on the same press release, which offers a natural benchmark and helps control for confounding factors such as cross-sectional variation in firm or press release characteristics and temporal or market-related differences.

Earnings Forecasts

Our primary analysis focuses on GPT-4’s GAAP EPS forecasts. For each press release, we calculate the absolute forecast error ($\lvert FE\rvert$) using the formula:

where $\mathit{Forecast}$ is GPT-4’s GAAP EPS forecast for firm $i$ for the next quarter, $\mathit{Actual}$ is the actual EPS for the next quarter,¹⁰ and $\mathit{Price}$ is the firm’s stock price before this quarter’s earnings announcement. We similarly measure analysts’ absolute forecast error, using the I/B/E/S unadjusted consensus analyst forecast aggregated on the first statistical period end after the quarterly earnings announcement. To mitigate the impact of outliers, we winsorize both forecast accuracy variables at the 1st and 99th percentiles.

GPT vs. Analysts

Table 2, Panel A presents a summary of the absolute forecast error ($|FE|$) for GPT-4 and analysts across the full sample. GPT-4’s mean $|FE|$ is 0.048, significantly higher than analysts’ average of 0.032. The difference of 0.016 is both positive and statistically significant. Their median values, 0.008 for GPT-4 and 0.004 for analysts, reinforce this finding. These results suggest that GPT-4’s forecasts are consistently less accurate than those of analysts.

Balance Sheet Items	t	t-1
Cash and Short-Term Investments	85.966	97.29
Receivables	22.7	20.874
Inventories	17.79	17.282
Other Current Assets	1.945	2.165
Total Current Assets	128.401	137.611
Property, Plant, and Equipment(Net)	20.301	19.566
Investment and Advances(Equity)	0	0
Investment and Advances(Other)	0	0
Intangible Assets	0	0
Other Assets	0.374	0.375
Total Assets	149.076	157.552
Accounts Payable	5.331	6.279
Other Current Liabilities	9.896	13.511
Debt in Current Liabilities	1.401	1.388
Income Taxes Payable	0	0
Total Current Liabilities	16.628	21.178
Deferred Taxes and Investment Tax Credit	0	0
Other Liabilities	0.026	0.03
Long-Term Debt	38.463	38.042
Total Liabilities	55.117	59.25
Common Equity	93.959	98.302
Preferred Stock	0	0
Nonredeemable Noncontrolling Interest	0	0
Total Stockholders’ Equity	93.959	98.302
Redeemable Nonredeemable Noncontrolling Interest	0	0
Total Liabilities and Stockholders’ Equity	149.076	157.552

Panel B compares GPT-4’s and analysts’ absolute forecast errors before and after GPT-4’s knowledge cutoff. The difference in $|FE|$ is 0.017 pre-cutoff and 0.015 post-cutoff, with no statistically significant difference between the two periods. This suggests that GPT-4’s relative underperformance in forecast accuracy compared to analysts is consistent across both periods. Furthermore, the pre-cutoff underperformance indicates that GPT-4 actively processes data to generate forecasts rather than relying on stored knowledge from its training data to match analysts’ performance.

While the overall difference across the two periods is not statistically significant, we postulate that GPT-4’s performance may diminish as it moves further beyond its knowledge cutoff, due to the deprivation of new information and reliance on increasingly outdated training knowledge. To test this, Panel C examines absolute forecast errors for the four quarters following GPT-4’s knowledge cutoff. The results show a notable trend: after the first quarter, GPT-4’s underperformance relative to analysts worsens as time progresses, rising to 0.021 by the fourth quarter—a 23.5% increase from the pre-cutoff accuracy gap of 0.017—significant at the 5% level. This deterioration underscores GPT-4’s reliance on timely and up-to-date information to maintain performance.

In summary, Table 2 shows that GPT-4’s forecasts are significantly less accurate than analysts’, with consistent underperformance across its knowledge cutoff. Although the overall difference between the two periods is not significant, there is some evidence of increasing underperformance in quarters further removed from the knowledge cutoff. These findings set the stage for our in-depth exploration of how GPT-4 processes textual and numerical information and the implications for its performance in the next section.

GPT’s Information Processing and Performance

To address the second research question, we examine GPT-4’s approach to textual and quantitative processing. We first an

quantitative processing. We first analyze how the model prioritizes sentences in earnings press releases for earnings forecasting. Next, we assess its quantitative analysis by analyzing its top ratio and trend choices and their alignment with analysts’ metrics. Finally, we use multivariate analysis to connect GPT-4’s performance to its textual ranking, quantitative analysis, and the knowledge cutoff, offering insights into the mechanisms behind its forecasting performance.

GPT’s Textual Ranking

We analyze the characteristics of the three most important sentences from each press release that GPT-4 identifies for earnings forecasting. While modern LLMs process all text in parallel, their training on large, general-purpose corpora can introduce broad, human-driven tendencies. These tendencies, shaped by diverse training data, reflect general rather than domain-specific patterns. Understanding GPT-4’s textual ranking approach is crucial for understanding its underlying textual information processing strategies.

To investigate GPT’s textual ranking strategies, we estimate the following linear probability model (LPM) regression at the sentence level:

where the dependent variable $GPTSentence$ is a binary variable equal to one if the sentence $s in firm $i$’s earnings press release for quarter $t$ is among the top three identified by GPT-4 for forecasting. Drawing on prior research, we include several sentence-level characteristics as determinants: a) Order of appearance in the press release (Order); b) Sentence length, measured as the logarithm of the total number of words (Length); c) Readability, measured using the Fog index (Fog); d) Proportion of numerical content in the sentence (%Num); e) Textual sentiment, measured using Loughran and McDonald’s ([33]) approach (Sentiment); f) Whether the sentence contains guidance-related information (Guidance).¹¹ We also control for firms’ information environment, including firm size (Firm_Size) and analyst coverage (Analysts). We also control for year-quarter and industry fixed effects. In an alternative specification, we control for press-release fixed effects. We report t-statistics clustered by firm to ensure robust standard errors. Detailed definitions of all variables are provided in Appendix B.

Table 3 presents the summary statistics and correlations for the sentence-level variables. Panel A shows that, on average, GPT-4 selects 6.6% of sentences from press releases. Sentence order, length, Fog index, and sentiment all show considerable variations. There is an average of 10.1% numerical and 5.2% guidance-related content in sentences. Panel B presents Pearson correlations among the sentence-level variables. Notably, sentence order, Fog index, and sentiment are negatively correlated with GPT sentence selection, while length, numerical specificity, and guidance information show positive correlations. These findings suggest that GPT’s sentence selection is driven by identifiable and observable characteristics.

Income Statement Items	t	t-1	t-2
Sales(Net)	32.708	31.953	26.432
Cost of Goods Sold	4.838	4.087	3.223
Selling, General and Administrative Expense	38.077	38.158	35.83
Operating Income Before Depreciation	-10.207	-10.292	-12.621
Depreciation and Amortization	1.086	1.002	0.945
Operating Income After Depreciation	-11.293	-11.294	-13.566
Interest and Related Expense	0.838	0.881	0.755
Special Items	0	0	0
Nonoperating Income	1.006	1	0.163
Pretax Income	-11.125	-11.175	-14.158
Income Taxes	0	0	0
Income Before Extraordinary Items and Noncontrolling Interests	-11.125	-11.175	-14.158
Noncontrolling Interest	0	0	0
Income Before Extraordinary Items	-11.125	-11.175	-14.158
Dividends	0	0	0
Income Before Extraordinary Items for Common Stocks	-11.125	-11.175	-14.158
Common Stock Equivalents - Dollar Savings	0	0	0
Income Before Extraordinary Items - Adjusted for Common Stock Equivalents	-11.125	-11.175	-14.158
Extraordinary Items and Discontinued Operations	0	0	0
Net Income(Loss)	-11.125	-11.175	-14.158
Earnings Per Share(Basic) - Excluding Extraordinary Items	-0.32	-0.32	-0.41
Earnings Per Share(Diluted) - Excluding Extraordinary Items	-0.32	-0.32	-0.41

Table 4 presents regression analyses of GPT-4’s sentence selection, with Panel A reporting results separately for pre-cutoff and post-cutoff periods. Two specifications are used: one with time and industry fixed effects (Columns (1) and (3)) and a more stringent approach with press-release-level fixed effects (Columns (2) and (4)). Focusing on Column (1), we find that GPT’s selection is significantly influenced by sentence-level characteristics.

	(1)	(2)	(3)	(4)
	Pre-Cutoff	Pre-Cutoff	Post-Cutoff	Post-Cutoff
Order	-0.107***	-0.109***	-0.112***	-0.114***
	(-34.68)	(-34.83)	(-36.66)	(-36.80)
Length	0.083***	0.084***	0.085***	0.086***
	(32.42)	(31.59)	(32.85)	(31.99)
Fog	-0.005***	-0.005***	-0.005***	-0.005***
	(-27.87)	(-27.88)	(-27.37)	(-27.97)
%Num	0.058***	0.052***	0.048***	0.047***
	(4.64)	(4.11)	(3.97)	(3.78)
Sentiment	-0.106***	-0.122***	-0.144***	-0.161***
	(-5.16)	(-5.66)	(-6.86)	(-7.52)
Guidance	0.180***	0.184***	0.172***	0.177***
	(27.49)	(27.36)	(25.67)	(25.74)
Log(Firm_Size)	-0.002		-0.001
	(-1.38)		(-1.09)
Log(Analyst)	0.001		0.000
	(0.52)		(0.13)
Year-Quarter Fixed Effects	YES	NO	YES	NO
Industry Fixed Effects	YES	NO	YES	NO
Press Release Fixed Effects	NO	YES	NO	YES
N	155,047	155,047	154,238	154,238
adj. R-sq	0.106	0.113	0.106	0.115

Sentences appearing earlier in the press release ($Order$, -0.107), those more readable ($Fog$, -0.005), those containing more numerical content ($\%Num$, 0.058), and those providing guidance-related information ($Guidance$, 0.180) are more likely to be selected by GPT. These tendencies closely align with human preferences for accessible and impactful information, such as prioritizing early, readable content with concrete data and forward-looking insights.

However, GPT also exhibits unique preferences. It favors longer sentences ($Length$, 0.083), likely because they often include more detailed or comprehensive information that aids forecasting. Additionally, GPT tends to select sentences with more negative sentiment ($Sentiment$, -0.106), possibly reflecting its capability to identify cautionary or risk-related language as critical for predicting future performance. These results suggest that while GPT shares many human-like tendencies, shaped by training on human-generated data, it also shows potential for preferences that may enhance its analytical capacity compared to human judgment.

Firm-level variables, such as firm size and analyst coverage, have insignificant results, indicating that differences in firms’ information environments have little effect on GPT’s selection strategy. Importantly, Columns (3) and (4) show that the results for sentence characteristics are consistent across both pre- and post-cutoff periods. Panel B further shows that these results remain stable even when analyzed within subsamples of firms with varying information transparency. These findings highlight that GPT’s textual ranking is primarily driven by general textual processing strategies, largely independent of domain-specific data quality or availability. GPT’s consistent preferences across contexts underscore its reliance on systematic analysis of textual attributes, such as position, style, and content, rather than external firm-level factors, reflecting a generalizable and robust approach to textual processing.

GPT’s Quantitative Analysis

We analyze GPT-4’s quantitative analysis strategies by examining the top three financial ratios or trends it identifies to derive earnings forecasts, as these financial metrics are an integral part of financial statement analysis. Our premise is that analysts tend to be well-versed in selecting relevant metrics, providing a benchmark for evaluating GPT-4’s ability to identify and compute key numerical information for its predictions. First, we summarize the distribution of GPT-4’s chosen metrics and compare them to analysts’ most common metrics. Next, we assess the alignment between the two sets of metrics to evaluate the consistency of their quantitative analysis. Finally, we investigate the factors influencing this alignment score to better understand GPT-4’s approach to quantitative analysis.

Panel A of Table 5 presents the frequency distribution of financial metrics identified by GPT-4. In total, the model outputs 20,544 financial ratio and trend metrics for our sample. To enhance clarity, we consolidate metrics with similar underlying constructs into 35 categories and present them in Panel A. Among them, net profit margin is the most frequently identified, appearing in 5,404 cases and representing 78.91% of the 6,848 earnings press releases. This is followed by return on equity (ROE) at 56.26%. The remaining categories show significant variation, highlighting GPT-4’s diverse quantitative analysis approaches.¹² To facilitate matching with analysts’ metrics, we create a column named “Corresponding I/B/E/S Metrics.”¹³

Panel A. Frequency of GPT-Selected Metrics for Quantitative Analysis
(1)	(2)	(3)	(4)
GPT-Selected Metrics	Corresponding I/B/E/S Metrics	Frequency	% of Press Releases
Net Profit Margin*	Net Income (Non-Per Share, NET) and Revenue (Non-Per Share, SAL)*	5,404	78.91%
Return on Equity (ROE)	Return on Equity (Percent, ROE)	3,853	56.26%
EPS	Earnings Per Share (EPS)	2,633	38.45%
Current Ratio	N/A	1,229	17.95%
Leverage ratio	Net Debt (NDT)	1,040	15.19%
Gross Profit Margin	Gross Margin (Percent, GRM)	1,034	15.10%
Operating Margin Ratio	Operating Profit (Non-Per Share, OPR)	943	13.77%
R&D Ratios	N/A	680	9.93%
Revenue Growth Rate	Revenue (Non-Per Share, SAL)	585	8.54%
Earnings Growth Rate	Revenue (Non-Per Share, SAL)	520	7.59%
Cash Position and Burn Rate	N/A	467	6.82%
Net Interest Rate Margin (NIM)	N/A	397	5.80%
Operating Expense Ratio	Operating Profit (Non Per Share, OPR)	396	5.78%
Net Income Growth Rate	Net Income (Non Per Share, NET)	324	4.73%
Return on Assets (ROA)	Return on Assets (Percent, ROA)	220	3.21%
Loan Growth Rate	N/A	156	2.28%
SG&A Ratios	N/A	140	2.04%
Operating Cash Flow Ratios	Cash Flow Per Share (CPS)	116	1.69%
EPS Growth Rate	Earnings Per Share (EPS)	98	1.43%
EBITDA Margin	EBITDA (Non-Per Share, EBT)	76	1.11%
Interest Coverage Ratio	N/A	48	0.70%
Free Cash Flow Ratios	Cash Flow Per Share (CPS)	29	0.42%
Net Investment Income	N/A	24	0.35%
Loan to Deposit Ratio (LDR)	N/A	24	0.35%
EBIT Margin	EBIT (Non-Per Share, EBI)	19	0.28%
Debt to EBITDA Ratio	Net Debt (NDT)	16	0.23%
Inventory Turnover Ratio	N/A	13	0.19%
Other	N/A	12	0.18%
Loan Loss Provision Ratios	N/A	12	0.18%
Noninterest Income Ratios	N/A	12	0.18%
Net Asset Value (NAV) Per Share	Net Asset Value (Non-Per Share, NAV)	9	0.13%
Nonperforming Assets to Total Assets Ratio	N/A	9	0.13%
Liquidity Ratio	N/A	2	0.03%
Book Value Per Share	Book Value Per Share (BPS)	2	0.03%
Price-to-Earnings (P/E) Ratio	N/A	2	0.03%
Total		20,544

Panel B reports the distribution of metrics commonly estimated by analysts, retrieved from the I/B/E/S Detail database. For each firm quarter, we identify the top three most frequent metrics forecasted by individual analysts. In total, the I/B/E/S Detail reports 20,238 metrics from 19 different categories, with earnings before interest and taxes (EBI) as the most frequent, appearing in 5,092 cases (74.36%) of the 6,848 observations, followed by revenue (SAL) and net income (NET).¹⁴ To evaluate alignment between GPT-4 and analysts, we calculate an alignment score based on the number of overlapping metrics between the two sets (ranging from zero to three). Panel C shows that all three metrics matched in 451 cases (6.59%), two metrics matched in 2,326 cases (33.97%), one in 3,539 cases (51.68%), and none in 532 cases (7.77%), reflecting considerable variation in alignment. We define an indicator variable, $\mathit{MetricOverlap\_High}$, which equals one if two or three metrics overlap and zero otherwise.

To investigate the factors influencing GPT’s alignment with analysts in quantitative analysis, we estimate the following linear probability model (LPM) regression at the press release level:

where the dependent variable $\mathit{MetricOverlap\_High}$ is defined above for firm $i$’s earnings press release for quarter $t$. Drawing on prior research, we include several financial statement characteristics related to disclosure quality as determinants: a) Balance sheet disaggregation, measured as the total number of line items in the balance sheet of two quarters, capturing the level of detail in reporting ($BS\_\#lineitems$); b) Income statement disaggregation, similarly measured as the total number of line items in the income statement of three quarters ($IS\_\#\mathrm{lineitems}$); and c) Numerical manipulation probability, captured by the degree of deviation from Benford’s law in the distribution of financial statement digits, indicating potential numerical irregularities ($\mathit{Benford\_Deviation}$).¹⁵ To test the influence of firms’ information environments on GPT-4’s quantitative analysis strategies, we also include firm size ($\mathit{Firm\_Size}$) and analyst coverage ($\mathit{Analysts}$). We further add an indicator variable, $\mathit{PostCutoff}$, to capture the change after the knowledge cutoff. Year-quarter and industry fixed effects are added to account for time-varying and sector-specific factors. Robust standard errors are calculated by clustering at the firm level. Detailed variable definitions are provided in Appendix B.

Table 5, Panels D and E present the summary statistics and correlations for these test variables. Panel D shows a mean $\mathit{MetricOverlap\_High}$ of 0.406. Balance sheet and income statement disaggregation measures exhibit substantial diversity, as does numerical manipulation probability based on Benford’s law deviation. Panel E shows that both disaggregation variables are positively correlated with the alignment score (0.07 and 0.13), while numerical manipulation probability shows negative correlations (-0.11), suggesting GPT-4’s quantitative analysis is influenced by financial data quality. Notably, both firm size and analyst coverage are positively related to the alignment score, indicating that GPT-4’s quantitative analysis benefits from environments with higher-quality, domain-specific data.

Table 6 presents regression results across three models, progressively incorporating additional controls and temporal variables. Column (1) provides the baseline model with industry and year-quarter fixed effects. While $\mathit{Log}(BS\_\#\mathrm{lineitems})$ (0.057) is positive, it is not significant. The coefficient for $\mathit{Log}(IS\_\#\mathrm{lineitems})$ is positive and significant (0.380), indicating that GPT-4 aligns more closely with analysts when income statements are more detailed. Conversely, $Benford's\_Law\_Deviation$ is negative and significant (-0.162), suggesting that GPT-4 struggles to align with analysts in environments where numerical data exhibit higher probabilities of manipulation, highlighting the model’s challenges with lower-quality financial data. Among the information environment variables, $Log(Firm\_Size)$ (-0.097) is insignificant, but $Log(Analyst)$ is strongly positive and significant (0.149), likely reflecting that GPT benefits from richer, domain-specific data from analysts.

	(1)	(2)	(3)
	MetricOverlap_High
BS_#lineitems_Log	0.057	0.044	0.047
	(0.33)	(0.26)	(0.28)
IS_#lineitems_Log	0.380**	0.386**	0.383**
	(2.09)	(2.13)	(2.11)
Benford’s Law Deviation	-0.162**	-0.160**	-0.162**
	(-2.15)	(-2.12)	(-2.14)
Firm_Size_Log	-0.097	-0.103	-0.100
	(-1.08)	(-1.15)	(-1.12)
Analyst_Log	0.149***	0.149***	0.148***
	(6.94)	(6.93)	(6.92)
PostCutoff		-0.013
		(-1.34)
PostCutoff_Qtr1			0.001
			(0.05)
PostCutoff_Qtr2			-0.015
			(-1.05)
PostCutoff_Qtr3			0.005
			(0.31)
PostCutoff_Qtr4			-0.042***
			(-2.76)
Industry Fixed Effects	YES	YES	YES
Year-Quarter Fixed Effects	YES	NO	NO
N	6848	6848	6848
adj. R-sq	0.080	0.079	0.079

Column (2) replaces year-quarter fixed effects with a dummy variable, $PostCutoff$, which is negative but insignificant, indicating that GPT’s alignment with analysts may decline after its knowledge cutoff. Column (3) introduces dummy variables for each of the four quarters following the knowledge cutoff, revealing a progressive decline in alignment over time. While $PostCutoff\_Qtr1$ through $PostCutoff\_Qtr3$ are all insignificant, $PostCutoff\_Qtr4$ becomes significantly negative (-0.042), indicating that GPT’s misalignment with analysts worsens as the temporal distance from its training cutoff increases. This underscores the model’s reliance on up-to-date and well-structured data for accurate quantitative analysis.

Taken together, these findings reveal that GPT’s quantitative analysis is not universally robust but instead depends heavily on the quality and timeliness of the input data. While GPT shows promise for sophisticated financial analysis, its reliance on structured and high-quality data points to areas for improvement, particularly for firms with lower transparency or more complex financial environments.

Multivariate Analysis on GPT Performance

After analyzing GPT-4’s processing of textual and quantitative information, focusing on textual ranking and quantitative analysis strategies, we now examine how these information-processing strategies relate to its performance. Specifically, we investigate whether the unique characteristics of GPT-identified top sentences and the (mis)alignment between GPT-selected financial metrics and analysts’ metrics contribute to its underperformance. Furthermore, we assess the impact of GPT’s knowledge cutoff, exploring whether its underperformance worsens as we analyze truly novel data that increasingly postdates this cutoff.

Building on these, we proceed with a formal regression analysis. Specifically, we estimate the following regression at the press release level:

where the dependent variable $\left|FE\_GPT\right|-\left|FE\_Analyst\right|$ represents the difference in absolute forecast errors for GAAP EPS between GPT-4 and analysts for firm $i$ in quarter $t$. Our main test variables are grouped into three categories, reflecting GPT’s textual ranking, quantitative analysis, and knowledge cutoff effects:

(1) Sentence-level incremental characteristics ($GPTSentence\_Incremental\_Char$): This set includes six characteristics that describe the unique properties of GPT-prioritized sentences, calculated as the average characteristic of the top three sentences identified by GPT minus the average characteristic of all sentences in the press release. These characteristics capture the incremental features introduced by GPT’s textual ranking: a) Order of appearance ($Order$); b) Sentence length ($Log(Length)$); c) Readability ($Fog$); d) Proportion of numerical content ($\%Num$); e) Textual sentiment ($Sentiment$); f) Guidance-related information ($Guidance$). For example, $GPT\_Incremental\_Fog$ measures the incremental readability of GPT-selected sentences compared to the press release average. This analysis helps distinguish model strategies that reflect genuine attempts to prioritize relevant information from those that may divert focus to less informative areas.

(2) Quantitative analysis metric alignment ($MetricOverlap\_High$): This set includes just one variable that captures the degree of alignment between GPT-selected financial metrics and those identified by analysts.

(3) Knowledge cutoff ($\mathit{PostCutoff\_Qtr}$): This set includes four dummy variables indicating the number of quarters since GPT-4’s knowledge cutoff date at the time of the earnings announcement. They help assess how GPT’s performance changes when analyzing data that increasingly postdates its knowledge cutoff.

We also control for general press release characteristics ($\mathit{PRSentence\_General\_Char}$), the firm’s information environment ($\mathit{Info\_Environment}$), and industry fixed effects to account for unobserved heterogeneity. General press release characteristics include all sentence-level variables except the order of appearance ($\mathit{Order}$), as its average for a press release is inherently 0.5. Additionally, we control for the total length of the press release ($\mathit{PR\_Total\_Length}$). To capture the richness of a firm’s information environment, we include firm size and analyst coverage. This regression framework helps disentangle the implications of textual ranking, metric alignment, and the knowledge cutoff on forecast accuracy differences between GPT-4 and analysts. We report t-statistics clustered by firm to ensure robust standard errors. Variable definitions are detailed in Appendix B.

Table 7 provides summary statistics and correlations for the test variables. Panel A shows that GPT-selected sentences exhibit distinct characteristics compared to the press release average, confirming the textual ranking results in Section 4.2.1. These include an earlier average order of appearance, an incremental length of 34.548 words, 4.6% more numerical content, and 0.004 less sentiment score. Additionally, GPT-selected sentences contain 16.9% more guidance-related content, reflecting notable differences in prioritization. Panel B presents the Pearson correlations. Earlier sentence selection by GPT is positively correlated with smaller performance gaps (0.06), while prioritizing more negative sentences is related to a larger gap (-0.04). Greater alignment between GPT and analysts in metric selection ($\mathit{MetricOverlap\_High}$) is negatively correlated with GPT’s accuracy gap (-0.04), highlighting the importance of alignment with analysts in quantitative analysis. Negative correlations with firm size and analyst coverage suggest GPT-4 benefits from richer, domain-specific training data.

	N	Mean	Std. Dev	P25	Median	P75
Panel A. Summary Statistics
Absolute Forecast Errors
|FE_GPT|-|FE_Analyst|	6,848	0.016	0.096	-0.001	0.002	0.009
GPT-Selected Sentence Characteristics
GPT_Incremental_Order	6,848	-0.163	0.188	-0.319	-0.153	-0.030
GPT_Incremental_Length	6,848	34.548	49.510	1.143	18.809	50.954
GPT_Incremental_Fog	6,848	-2.234	4.351	-5.125	-2.744	0.020
GPT_Incremental_%Num	6,848	0.046	0.051	0.013	0.047	0.079
GPT_Incremental_Sentiment	6,848	-0.004	0.019	-0.015	-0.003	0.006
GPT_Incremental_Guidance	6,848	0.169	0.267	0.000	0.000	0.300
Quantitative Analysis Alignment
MetricOverlap_High	6,848	0.406	0.491	0.000	0.000	1.000
Knowledge Cutoff
PostCutoff_Qtr1	6,848	0.127	0.334	0.000	0.000	0.000
PostCutoff_Qtr2	6,848	0.127	0.333	0.000	0.000	0.000
PostCutoff_Qtr3	6,848	0.126	0.332	0.000	0.000	0.000
PostCutoff_Qtr4	6,848	0.125	0.331	0.000	0.000	0.000
Press-Release General Characteristics
PR_Total_Length	6,848	1,469.225	899.982	845.000	1,232.000	1,805.500
PR_General_Length	6,848	34.145	9.857	28.000	32.115	37.336
PR_General_Fog	6,848	13.879	2.721	11.910	13.817	15.679
PR_General_%Num	6,848	0.103	0.031	0.080	0.100	0.123
PR_General_Sentiment	6,848	0.004	0.010	-0.002	0.003	0.010
PR_General_Guidance	6,848	0.061	0.071	0.000	0.042	0.093
Firm Information Environment
Firm_Size	6,848	12.553	38.724	0.457	1.843	6.321
Analyst	6,848	5.116	3.636	3.000	4.000	7.000

Table 8 presents regression results examining how differences in absolute forecast error between GPT-4 and analysts are related to the three dimensions of GPT’s information processing: textual ranking, quantitative alignment, and the knowledge cutoff. The first column includes variables representing these three dimensions without additional control variables. The second introduces press release general characteristics as additional controls, while the third column further incorporates firm information environment variables. This stepwise approach allows for an evaluation of how additional contextual factors influence the significance and magnitude of the key information processing variables.

	(1)	(2)	(3)
		$|\mathrm{FE\_GPT}| - |\mathrm{FE\_Analyst}|$
GPT-Selected Sentences’ Incremental Characteristics
GPT_Incremental_Order	0.210***	0.233***	0.221***
	(2.62)	(2.81)	(2.67)
GPT_Incremental_Length_Log	0.057**	0.047*	0.044
	(2.10)	(1.72)	(1.61)
GPT_Incremental_Fog	0.003	0.004	0.002
	(0.87)	(1.21)	(0.58)
GPT_Incremental_%Num	-0.011**	-0.015***	-0.014**
	(-2.02)	(-2.63)	(-2.49)
GPT_Incremental_Sentiment	-1.992**	-2.198**	-1.750*
	(-2.17)	(-2.38)	(-1.90)
GPT_Incremental_Guidance	-0.023	0.023	0.021
	(-0.40)	(0.38)	(0.35)
GPT-Analyst Alignment
MetricOverlap_High	-0.126**	-0.122**	-0.068
	(-2.24)	(-2.18)	(-1.20)
Post Knowledge Cutoff
PostCutoff_Qtr1	0.012	0.019	0.019
	(0.27)	(0.44)	(0.44)
PostCutoff_Qtr2	-0.044	-0.038	-0.035
	(-1.02)	(-0.89)	(-0.81)
PostCutoff_Qtr3	0.060	0.066	0.070
	(1.38)	(1.52)	(1.62)
PostCutoff_Qtr4	0.100**	0.104**	0.107**
	(2.28)	(2.38)	(2.47)
Press Release General Characteristics:
PR_Total_Length_Log		0.023	0.037
		(0.77)	(1.26)
PR_General_Length_Log		0.118*	0.102
		(1.77)	(1.55)
PR_General_Fog		0.027***	0.031***
		(3.52)	(4.01)
PR_General_%Num		0.002	0.003
		(0.26)	(0.59)
PR_General_Sentiment		-0.027*	-0.022
		(-1.87)	(-1.49)
PR_General_Guidance		-0.577***	-0.432**
		(-2.70)	(-2.03)
Firm Information Environment
Log(Firm_Size)			-0.071***
			(-4.39)
Log(Analyst)			-0.157***
			(-5.77)
Industry Fixed Effects	YES	YES	YES
N	6,848	6,848	6,848
adj. R-sq	0.018	0.023	0.035

In Column (1), the unique characteristics of GPT-selected sentences exhibit significant relationships with the forecast accuracy gap. $\mathit{GPT\_incremental\_Order}$ is significantly positive (0.209), suggesting that GPT-4’s focus on early content improves its performance. $\mathit{GPT\_incremental\_Length}$ has a positive coefficient of 0.057, indicating that longer GPT-selected sentences are associated with greater absolute forecast errors relative to analysts. Conversely, $\mathit{GPT\_incremental\_\%Num}$ shows a negative coefficient (-0.011), suggesting that GPT’s focus on numerical specificity improves its performance. Similarly, there is a negative coefficient on $\mathit{GPT\_incremental\_Sentiment}$ (-1.992), implying that more positive sentiment in GPT-selected sentences narrows its error gap. Recall that GPT-4 tends to select sentences that are longer and with more negative sentiment, which suggests that its textual ranking strategies may hinder performance in certain contexts. However, its preference for sentences appearing early and having higher numerical content appears to enhance its forecasting accuracy.

Quantitative alignment between GPT-4 and analysts ($\mathit{MetricOverlap\_High}$) has a significant negative coefficient (-0.126), suggesting a narrower gap in forecast accuracy when financial metrics are more closely aligned. For the knowledge cutoff, while the first three post-cutoff quarters show insignificant results, an upward trend emerges from the second quarter, with a significant coefficient for $PostCutoff\_Qtr4$ (0.100) indicating increased underperformance in the latest quarter. Therefore, there is some evidence suggesting that GPT-4 struggles to generalize its quantitative analysis effectively when working with new data increasingly distant from its training period.

In Column (2), press release general characteristics are added to the model. This inclusion does not alter the main inferences from Column (1), as the coefficients and significance levels for textual ranking, quantitative alignment, and knowledge cutoff effects remain consistent. Interestingly, some general characteristics of the press release, such as $PR\_General\_Fog$, become significant (0.027), suggesting that general complexity in press releases contributes to GPT’s underperformance. This further supports the notion that GPT-4 may face challenges with specialized tasks when dealing with complex information.

Column (3) further adds firm information environment variables to the regression. While results for textual ranking and knowledge cutoff remain robust, $MetricOverlap\_High$ loses significance. This suggests that quantitative alignment depends on the richness of the firm’s information environment. Variables such as firm size (-0.071) and analysts (-0.157) significantly influence the accuracy gap, as richer information environments enable closer alignment between GPT-selected and analyst-selected metrics, improving performance. Overall, the findings suggest that GPT-4’s textual ranking has generalized implications for performance across firms, while its quantitative analysis is tied to the availability of domain-specific training data, which varies significantly by firm.

Supplemental Analysis

To further validate our findings and ensure robustness, we perform a series of supplementary analyses, addressing potential concerns about GPT-4’s information processing and expanding the scope of our evaluation. These analyses confirm the consistency of our results and provide additional insights into GPT-4’s performance under varied conditions.

Consistency Between Reported and Internal Processes

A potential concern is whether GPT-4 accurately reports the sentences and metrics it selects versus what it internally processes to generate forecasts. To address this, we explicitly prompt GPT-4 to justify its sentence selections and metric choices for relevance to earnings forecasting. For example, GPT-4 must explain why each selected sentence is most relevant to earnings forecasting. Similarly, for financial metrics, GPT-4 is asked to provide narrative justification and display the intermediate steps used to calculate the ratios. Figure 1 and Figure 2 present word clouds illustrating GPT-4’s narrative justifications for sentence selection and metric choice, respectively, with the words prominently aligning with the tasks. These results confirm that GPT-4’s reported selections are consistent with its internal processing, supporting the reliability of its outputs.

Robustness With Non-GAAP Earnings Forecasts

To test the robustness of our findings, we compare GPT-4’s non-GAAP earnings forecasts to analysts’ street earnings forecasts. Non-GAAP earnings, which often exclude items such as restructuring costs or one-time charges, are widely used by analysts for forecasting. However, an important caveat is that GPT-4 fails to provide non-GAAP earnings forecasts for 1,540 (22.5%) of the 6,848 press releases in its output. Additionally, GPT-4’s adjustments for non-GAAP earnings may not fully reconcile with the street earnings reported in I/B/E/S. Despite these limitations, the results in Table 9 remain consistent with our main findings, showing that GPT-4’s forecasts exhibit similar underperformance relative to analysts (with an untabulated mean forecast accuracy gap of 0.010) and this underperformance is related to its approaches to processing textual and quantitative information. This consistency highlights the generalizability of our findings across different earnings definitions.

	(1)	(2)	(3)
	$\lvert$FE_GPT_NonGAAP$\rvert$ - $\lvert$FE_Analyst_NonGAAP$\rvert$	$\lvert$FE_GPT_NonGAAP$\rvert$ - $\lvert$FE_Analyst_NonGAAP$\rvert$	$\lvert$FE_GPT_NonGAAP$\rvert$ - $\lvert$FE_Analyst_NonGAAP$\rvert$
GPT-Selected Sentences’ Incremental Characteristics
GPT_Incremental_Order	0.678***<br>(2.97)	0.746***<br>(3.15)	0.727***<br>(3.08)
GPT_Incremental_Length_Log	0.062<br>(0.83)	0.054<br>(0.72)	0.053<br>(0.70)
GPT_Incremental_Fog	0.040***<br>(3.95)	0.046***<br>(4.51)	0.040***<br>(3.99)
GPT_Incremental_%Num	-0.007<br>(-0.44)	-0.020<br>(-1.23)	-0.017<br>(-1.06)
GPT_Incremental_Sentiment	-4.875*<br>(-1.81)	-5.500**<br>(-2.03)	-4.531*<br>(-1.68)
GPT_Incremental_Guidance	0.194<br>(0.97)	0.310<br>(1.43)	0.297<br>(1.38)
GPT-Analyst Alignment
MetricOverlap_High	-0.362**<br>(-2.22)	-0.351**<br>(-2.16)	-0.220<br>(-1.36)
Post Knowledge Cutoff
PostCutoff_Qtr1	0.055<br>(0.45)	0.070<br>(0.57)	0.073<br>(0.60)
PostCutoff_Qtr2	0.036<br>(0.29)	0.056<br>(0.46)	0.066<br>(0.55)
PostCutoff_Qtr3	0.098<br>(0.80)	0.119<br>(0.98)	0.129<br>(1.07)
PostCutoff_Qtr4	0.188<br>(1.54)	0.204*<br>(1.67)	0.211*<br>(1.74)
Press Release General Characteristics:
PR_Total_Length_Log		0.055<br>(0.67)	0.080<br>(0.98)
PR_General_Length_Log		0.003<br>(0.01)	-0.058<br>(-0.31)
PR_General_Fog		0.118***<br>(5.28)	0.130***<br>(5.85)
PR_General_%Num		0.014<br>(0.83)	0.020<br>(1.19)
PR_General_Sentiment		-0.093**<br>(-2.22)	-0.079*<br>(-1.91)
PR_General_Guidance		-1.264**<br>(-2.13)	-0.916<br>(-1.55)
Firm Information Environment
Log(Firm_Size)			-0.119***<br>(-2.62)
Log(Analyst)			-0.495***<br>(-6.40)
Industry Fixed Effects	YES	YES	YES
N	5308	5308	5308
adj. R-sq	0.034	0.043	0.056

Confidence Scores

To assess GPT-4’s self-awareness and potential limits, we also examine how its reported confidence scores relate to its textual ranking, quantitative analysis, and knowledge cutoff.

Confidence scores offer a peek into whether GPT-4 recognizes potential performance pitfalls, such as misaligning with analysts on key financial metrics or handling data beyond its knowledge cutoff. Ideally, confidence should drop in these scenarios, reflecting an awareness of challenges, but unjustified confidence could signal overestimation of its capabilities.

Table 10 shows that GPT-4’s confidence largely aligns with its textual ranking strategies, such as prioritizing longer, more readable, and guidance-related sentences. However, given its lower performance when focusing on longer sentences, this confidence may be misplaced. Confidence scores show no significant relationship with quantitative alignment, suggesting GPT-4’s lack of awareness of its weaknesses in quantitative analysis. Moreover, its confidence is elevated in the fourth quarter post-cutoff, reflecting overconfidence in handling novel data. These results underscore the need for better confidence calibration to align GPT-4’s self-assessment with its limitations, particularly in quantitative analysis and post-cutoff contexts.

	(1)	(2)	(3)
GPT_GAAP_Certainty_Score
GPT-Selected Sentences’ Incremental Characteristics
GPT_Incremental_Order	-0.002	0.005	0.006
	(-0.46)	(0.98)	(1.29)
GPT_Incremental_Length_Log	0.007***	0.007***	0.008***
	(4.13)	(4.39)	(4.71)
GPT_Incremental_Fog	-0.002***	-0.002***	-0.001***
	(-7.15)	(-7.36)	(-6.58)
GPT_Incremental_%Num	-0.000	-0.000	-0.000
	(-0.47)	(-0.77)	(-0.95)
GPT_Incremental_Sentiment	0.506***	0.557***	0.522***
	(8.94)	(9.98)	(9.41)
GPT_Incremental_Guidance	0.021***	0.011***	0.011***
	(5.85)	(2.97)	(3.00)
GPT-Analyst Alignment
MetricOverlap_High	-0.001	-0.001	-0.005
	(-0.28)	(-0.33)	(-1.36)
Post Knowledge Cutoff
PostCutoff_Qtr1	-0.000	0.000	0.000
	(-0.00)	(0.05)	(0.06)
PostCutoff_Qtr2	0.006**	0.005**	0.005**
	(2.19)	(2.10)	(2.03)
PostCutoff_Qtr3	0.003	0.002	0.002
	(1.14)	(0.73)	(0.62)
PostCutoff_Qtr4	0.008***	0.006**	0.006**
	(3.13)	(2.26)	(2.15)
Press Release General Characteristics:
PR_Total_Length_Log		0.013***	0.012***
		(7.14)	(6.67)
PR_General_Length_Log		-0.008**	-0.007*
		(-1.98)	(-1.78)
PR_General_Fog		-0.001*	-0.001**
		(-1.73)	(-2.31)
PR_General_%Num		0.002***	0.002***
		(6.34)	(6.05)
PR_General_Sentiment		0.014***	0.013***
		(15.40)	(15.04)
PR_General_Guidance		0.073***	0.063***
		(5.64)	(4.94)
Firm Information Environment
Log(Firm_Size)			0.007***
			(7.24)
Log(Analyst)			0.008***
			(5.16)
Industry Fixed Effects	YES	YES	YES
N	6848	6848	6848
adj. R-sq	0.048	0.091	0.107

Directional Earnings Change Forecasts

To extend our analysis and provide a potential reconciliation with [28], we also examine GPT-4’s ability to predict the directional change in earnings, that is, whether earnings increase or decrease. Directional forecasts are critical in financial analysis, as they often influence investment decisions even in the absence of precise earnings forecasts. We find that GPT-4 achieves an average directional accuracy of 49.3% when comparing its numerical forecasts to actual GAAP earnings and 47.7% when using its narrative predictions (we explicitly prompt GPT-4 to indicate the expected directional change in earnings). In contrast, analysts demonstrate a much higher directional prediction accuracy of 71.1% for GAAP EPS. These findings show that GPT-4 underperforms analysts not only in numerical accuracy but also in predicting the direction of earnings changes.

[28] report that GPT-4’s directional accuracy declines over time, reaching 50.25% by 2020, a result that aligns closely with our findings. Of course, differences in task design and data context between the two studies clearly shape GPT performance. But careful consideration is needed to avoid hindsight bias when evaluating pre-cutoff performance, particularly regarding (1) the potential for GPT-4 to detect subtle cues in pre-cutoff data that may inflate accuracy and (2) the application of modern analytical methods from training to historical data, creating a “time-traveler’s advantage.” Mitigating these effects is crucial for a fair assessment of GPT-4’s real-world capabilities.

Conclusion

This study investigates how GPT-4 processes corporate disclosures to forecast earnings. We compare GPT-4’s performance with that of human analysts and examine the mechanisms underlying it. Our findings reveal that GPT-4’s forecast accuracy, on average, is significantly lower than analysts’, exposing inherent limitations in its design.

By analyzing GPT-4’s approach to processing textual and quantitative information through three key dimensions—textual ranking, quantitative analysis, and knowledge cutoff—we provide novel insights into its potential and challenges. GPT-4 shows consistent and generalizable textual strategies across firms, prioritizing sentences that appear earlier, are longer, more readable, more negatively toned, and rich in numerical or guidance-related content. This consistency highlights its strength in language tasks. However, its quantitative analysis varies significantly, heavily influenced by the availability of quality, domain-specific training data, which differs across firms. Misalignment with analysts in key financial metrics, especially for firms with poor analyst coverage, underscores GPT-4’s reliance on structured, specialized data to approximate human-like performance. Both textual ranking and quantitative analysis strategies play critical roles in shaping GPT-4’s forecasting performance.

The knowledge cutoff further emerges as a critical factor in GPT-4’s performance. We find some evidence that GPT-4’s forecast accuracy diminishes post-cutoff, especially with the fourth quarter beyond the cutoff. This underscores the importance of evaluating LLMs under genuinely hindsight-free conditions to ensure fair assessments and avoid an artificial “time traveler” advantage in applying modern analytical methods to historical data.

Our findings have broader implications for the role of AI in financial analysis. While GPT-4 and similar LLMs show promise in processing textual data and extracting key information, they are not yet substitutes for human analysts, particularly in specialized tasks such as quantitative financial analysis within complex or opaque environments. These results underscore the need for domain-specific adaptations, such as financial-tuned models like BloombergGPT, and reinforce the importance of interpretability and transparency in deploying AI-driven financial tools. By providing an in-depth exploration of GPT-4’s information processing, this study contributes to the ongoing discussion about integrating advanced AI models into financial decision-making and analysis.

Appendix A. Example of Complete Interactions with GPT-4

This appendix provides a detailed example of our interaction with GPT-4 Turbo API, presented in two panels. Panel A shows our chain-of-thought (CoT) prompt that we feed to the model. Panel B presents the response produced by the model.

The example focuses on the firm “SI-BONE INC” (CIK = 0001459839; GVKEY = 034193) for its fiscal quarter ending on March 31, 2023. The quarterly earnings announcement was on May 1, 2023, with the corresponding press release filed within a Form 8-K in EDGAR on the same date.¹⁶

corresponding period in 2022. U.S. revenue for the first quarter 2023 was $30.5 million, a 50% increase from $20.4 million in the corresponding period in 2022. International revenue for the first quarter 2023 was $2.3 million, a 9% increase from $2.1 million in the corresponding period in 2022.

Gross margin was 82% for the first quarter 2023, as compared to 87% in the corresponding period in 2022. Gross margin in the first quarter 2023 was impacted by procedure and product mix given the higher total costs of the newly launched implants. Gross margin also includes the impact of higher depreciation from instrument trays to support the growth of the business, depreciation associated with our second facility in Santa Clara, and higher freight costs.

Operating expenses increased 5% to $38.1 million in the first quarter 2023, as compared to $36.3 million in the corresponding period in 2022. The increase was primarily driven by increase in sales commission and stock-based compensation from higher headcount, and increase in travel costs.

Operating loss improved by 33% to $11.3 million in the first quarter 2023, as compared to an operating loss of $16.9 million in the corresponding period in 2022.

Net loss was $11.1 million, or $0.32 per diluted share for the first quarter 2023, as compared to a net loss of $17.4 million, or $0.52 per diluted share in the corresponding period in 2022.

Adjusted EBITDA loss was negative $3.9 million, in the first quarter 2023, as compared to an adjusted EBITDA loss of negative $10.7 million, in the corresponding period in 2022.

Cash and marketable securities were $86.0 million and borrowings were $35.9 million as of March 31, 2023.

2023 Updated Financial Guidance

Based on the first quarter of 2023 results, SI-BONE is updating its 2023 worldwide revenue guidance to be in the range of $128 million to $131 million, implying growth of approximately 20% to 23%. SI-BONE is maintaining its guidance for fiscal year 2023 gross margin of approximately 80% and improving adjusted EBITDA loss.

Task: Forecast the Company’s Future Earnings Per Share (EPS)

Using the provided press release discussions, balance sheet, and income statement, follow these steps to forecast the company’s future Earnings Per Share (EPS):

Step 1: Analyze the Press Release

1.Identify Key Sentences: Select the top 3 most important sentences from the press release.

2.Explain Relevance: For each sentence, explain why it is important for forecasting the company’s future EPS.

Step 2: Conduct Financial Ratio and Trend Analyses

1.Select Ratios and Trends: Choose financial ratios and trends relevant to forecasting EPS from the balance sheet and income statement.

2.Show Calculations: For each ratio and trends

3.Provide the formula.

4.Calculate the actual value using the company’s financials.

5.Choose Top ratios or trends: Identify the 3 most important financial ratios or trends for predicting the company’s future EPS and explain their significance.

Step 3: EPS Prediction (GAAP)

1.EPS Direction: Based on your analysis, predict whether the company’s EPS will increase or decrease in the next quarter(t+1).

2.EPS Value: Estimate the company’s EPS for the next quarter(t+1), given that there are 37.8640 million of common shares outstanding

3.Certainty Score: Assign a certainty score between 0 and 1 to your predictions (with 0 being least certain and 1 being most certain).

Step 4: Non-GAAP EPS Prediction (Street EPS)

1.Non-GAAP EPS Direction: Predict whether the company’s street EPS (non-GAAP EPS) will increase or decrease in the next quarter(t+1).

2.Non-GAAP EPS Value: Estimate the company’s street EPS for the next quarter(t+1).

3.Certainty Score: Assign a certainty score between 0 and 1 to your predictions.

Example of Complete Interactions with GPT-4 (Continued)

Step	Content
1	Step 1: Analyze the Press Release<br><br>1. Identify Key Sentences:<br>- “Annual guidance increased to $128 million - $131 million, implying annual growth of 20%-23%”<br>- “Gross margin of 82% for the first quarter 2023”<br>- “Operating loss improved by 33% to $11.3 million in the first quarter 2023, as compared to an operating loss of $16.9 million in the corresponding period in 2022.”<br><br>2. Explain Relevance:<br>- The increase in annual guidance suggests that the company expects continued revenue growth, which could lead to higher earnings and potentially a higher EPS in future quarters.<br>- The gross margin provides insight into the profitability of the company’s products. A high gross margin indicates that the company can cover its operating expenses and potentially report a profit, which is crucial for a positive EPS.<br>- The improvement in operating loss indicates that the company is moving towards profitability. A decreasing operating loss can lead to a lower net loss or even a profit, which would positively impact EPS.
2	Step 2: Conduct Financial Ratio and Trend Analyses<br><br>1. Select Ratios and Trends:<br>- Revenue Growth Rate<br>- Gross Margin Percentage<br>- Operating Expense Ratio<br><br>2. Show Calculations:<br>- Revenue Growth Rate = (Current Quarter Revenue - Previous Quarter Revenue) / Previous Quarter Revenue<br>= ($32.708M - $31.953M) / $31.953M = $0.755M / $31.953M = 2.36%<br>- Gross Margin Percentage = (Sales - Cost of Goods Sold) / Sales<br>= ($32.708M - $4.838M) / $32.708M = $27.87M / $32.708M = 85.2%<br>- Operating Expense Ratio = Operating Expenses / Net Sales<br>= $38.077M / $32.708M = 1.164 or 116.4%<br><br>3. Choose Top ratios or trends:<br>- Revenue Growth Rate: This ratio is important because it shows how quickly the company’s sales are increasing, which is a direct driver of EPS.<br>- Gross Margin Percentage: This is a key indicator of profitability and the company’s ability to manage production costs relative to sales. Higher gross margins can lead to higher EPS if costs are controlled.<br>- Operating Expense Ratio: This ratio indicates how much of the company’s sales are consumed by operating expenses. A lower ratio is preferable as it suggests better cost management and the potential for higher net income and EPS.
3	Step 3: EPS Prediction (GAAP)<br><br>1. EPS Direction:<br>Based on the positive revenue growth, improved operating loss, and the company’s increased annual guidance, it is reasonable to predict that the company’s EPS will increase in the next quarter (t+1).<br><br>2. EPS Value:<br>To estimate the EPS for the next quarter, we can use the revenue growth rate and assume that the company will maintain its gross margin and manage to control its operating expenses in line with revenue growth. Assuming a modest improvement in net income due to these factors, let’s estimate a 5% increase in net income from quarter t to quarter t+1.<br><br>Estimated Net Income (t+1) = Net Loss (t) * (1 + Estimated Improvement)<br>= -$11.125M * (1 + 0.05)<br>= -$11.125M * 1.05<br>= -$11.68125M

Estimated EPS (t+1) = Estimated Net Income (t+1) / Number of Common Shares Outstanding<br>= -$11.68125M / 37.864M<br>= -$0.3086 (rounded to -$0.31)<br><br>3. Certainty Score:<br>Given the assumptions made and the inherent uncertainty in forecasting, a certainty score of 0.7 is assigned to this prediction.

4

Step 4: Non-GAAP EPS Prediction<br><br>1. Non-GAAP EPS Direction:<br>The direction of the street EPS (non-GAAP EPS) is also expected to increase in the next quarter (t+1) due to the same factors affecting the GAAP EPS.<br><br>2. Non-GAAP EPS Value:<br>Non-GAAP EPS often excludes one-time charges and non-cash expenses such as stock-based compensation. Assuming the company has similar adjustments in the next quarter as in the current quarter, the non-GAAP EPS could see a similar or slightly better improvement than GAAP EPS. Without specific non-GAAP adjustments, it is challenging to provide an exact estimate, but we can expect a similar or slightly better trend than the GAAP EPS.<br><br>3. Certainty Score:<br>Given the additional uncertainty around non-GAAP adjustments, a certainty score of 0.65 is assigned to this prediction.

Variable Definitions

Variable Name	Definition
Sentence Level Variables:
GPTSentence	A dummy variable that equals one if a sentence is selected by GPT-4 for earnings forecast purposes
Order	The percentage ranking of a sentence’s position within a press release, where earlier sentences have lower values (e.g., the first sentence is close to 0%, and the last is 100%).
Length	The total number of words in a sentence.
Fog	The readability of a sentence, measured using the Gunning Fog Index, which estimates the years of formal education required to understand the text on the first reading.
%Num	The percentage of numerical words in a sentence.
%Sentiment	The net sentiment score of a sentence, calculated as the difference between the number of positive words and negative words, divided by the total number of words in the sentence. Positive and negative words are identified using the Loughran and McDonald’s ([33]) sentiment word list.
Guidance	A dummy variable equal to 1 if the sentence contains any forecast, and 0 otherwise.
GPT’s Quantitative Reasoning Variables:
MetricOverlap_High	MetricOverlap represents the number of overlapping financial metrics based on the top three metrics identified by GPT-4 and the top three most frequently reported metrics by individual analysts in I/B/E/S Detail, excluding EPS and GPS metrics. MetricOverlap_High is a dummy variable that equals 1 if MetricOverlap is greater than or equal to 2, and 0 otherwise.
BS_#lineitems	The total number of non-zero line items in the standardized balance sheet across two quarters (i.e., Quarter t and Quarter t-1) that are included in the prompt provided to GPT-4.
IS_#lineitems	The total number of non-zero line items in the standardized income statement across three quarters (i.e., Quarter t, Quarter t-1 and Quarter t-2) that are included in the prompt provided to GPT-4.
Benford’s Law Deviation	Calculated by analyzing the first non-zero digits (1-9) of all non-missing line-item numbers from the standardized balance sheet (2 quarters) and income statement (3 quarters). For each digit, its frequency is converted into a relative frequency by dividing it by the total frequency of all digits (1-9). The deviation for each digit is then determined as the absolute difference between its relative frequency and the expected frequency based on Benford’s Law. Benford’s Law Deviation is the average of these deviations across all digits for each firm-quarter.
Forecast Errors Variables:
|FE_GPT|	GPT-4’s absolute numerical GAAP-based EPS forecast error, calculated as: |GPT-4’s GAAP EPS Prediction - Actual I/B/E/S GAAP EPS value (GPS)|/lagged Stock Price.
|FE_Analyst|	Analysts’ absolute numerical GAAP-based EPS forecast error, calculated as |Analyst median GAAP EPS Prediction (GPS) - Actual I/B/E/S GAAP EPS value (GPS)|/lagged Stock Price.
|FE_GPT|-|FE_Analyst|	The difference between GPT-4’s absolute numerical GAAP-based EPS forecast error and analysts’ absolute numerical GAAP-based EPS forecast error.
Press Release Characteristics Variables:
PR_Total_Length	The total number of words in the press release.
PR_General_Length	The average number of words per sentence in a press release, calculated as the total number of words divided by the total number of sentences.
PR_General_Fog	The average Fog index of all the sentences in a press release.
PR_General_%Num	The average percentage of numerical words across all sentences in a press release.
PR_General_Sentiment	The average Loughran-McDonald’s ([33]) sentiment score across all sentences in a press release.
PR_General_Guidance	The average of Guidance dummy of all sentences in a press release.

GPT_Incremental_Order

The difference between the average “Order” variable of sentences selected by GPT and the average “Order” variable of all sentences in the press release.

GPT_Incremental_Length

The difference between the average “Length” variable of sentences selected by GPT and the average “Length” variable of all sentences in the press release.

GPT_Incremental_Fog

The difference between the average “Fog” variable of sentences selected by GPT and the average “Fog” variable of all sentences in the press release.

GPT_Incremental_%Num

The difference between the average “%Num” of sentences selected by GPT and the average “%Num” variable of all sentences in the press release.

GPT_Incremental_Sentiment

The difference between the average “Sentiment” variable of sentences selected by GPT and the average “Sentiment” variable of all sentences in the press release.

GPT_Incremental_Guidance

The difference between the average “Guidance” variable of sentences selected by GPT and the average “Guidance” variable of all sentences in the press release.

Knowledge Cutoff Variables:

PostCutoff

A dummy variable equal to 1 if the current quarter’s earnings press release is announced after the knowledge cutoff date of April 30, 2023, and 0 otherwise.

PostCutoff_Qtr1-PostCutoffQtr4

Dummy variables indicate whether the current quarter’s earnings press release is announced in a specific quarter following the knowledge cutoff date (April 30, 2023). Each variable equals 1 if the press release is announced in the corresponding quarter (e.g., first, second, third, or fourth quarter) after the cutoff date, and 0 otherwise.

Firm Level Variables:

Firm_Size

Firm market value, defined as share outstanding multiplied by price per share.

Analyst

Number of analysts following the firm.

Step	Description	# of Observations Left
1	Quarterly earnings press releases issued during the four quarters before and the four quarters after April 30, 2023, by firms covered by I/B/E/S, Compustat, and CRSP databases.	68,514
2	Merge with 8-K filings (Item 2.02) from EDGAR for quarterly earnings press-release.	49,192
3	Drop observations where analysts’ GAAP-based EPS forecast (GPS) for the subsequent quarter is missing.	47,485
4	Randomly draw a sample of 1,000 firms that have at least one quarterly observation available both before and after April 30, 2023, and for which all test variables are non-missing.	7,085
	GPT-4 failed to generate quarterly GAAP EPS forecasts	237
	GPT-4 successfully generated quarterly GAAP EPS forecasts	6,848
	The total number of sentences in all press releases included in the 6,848 sample	309,285

Panel A. Full Sample Statistics
	N	Mean	Std. Dev	P25	Median	P75
$|\mathrm{FE}_{\mathrm{GPT}}|$	6,848	0.048	0.160	0.002	0.008	0.024
$|\mathrm{FE}_{\mathrm{Analyst}}|$	6,848	0.032	0.123	0.001	0.004	0.014
$|\mathrm{FE}_{\mathrm{GPT}}|-|\mathrm{FE}_{\mathrm{Analyst}}|$	6,848	0.016	0.096	-0.001	0.002	0.009
Panel B. By The Knowledge Cutoff Date
	Pre-Cutoff	Post-Cutoff
$|\mathrm{FE}_{\mathrm{GPT}}|$	0.044	0.052
$|\mathrm{FE}_{\mathrm{Analyst}}|$	0.027	0.037
$|\mathrm{FE}_{\mathrm{GPT}}|-|\mathrm{FE}_{\mathrm{Analyst}}|$	0.017***	0.015***
N	3,384	3,464
Panel C. By Quarters After The Knowledge Cutoff Date
	Post-Cutoff Qtr1	Post-Cutoff Qtr2	Post-Cutoff Qtr3	Post-Cutoff Qtr4
$|\mathrm{FE}_{\mathrm{GPT}}|$	0.057	0.043	0.054	0.052
$|\mathrm{FE}_{\mathrm{Analyst}}|$	0.042	0.035	0.039	0.031
$|\mathrm{FE}_{\mathrm{GPT}}|-|\mathrm{FE}_{\mathrm{Analyst}}|$	0.015***	0.009***	0.015***	0.021***
N	873	869	866	856

Panel A. Summary Statistics of Sentence-Level Variables
	N	Mean	Std. Dev	P25	Median	P75
GPTSentence	309,285	0.066	0.239	0.000	0.000	0.000
Order	309,285	0.511	0.289	0.261	0.511	0.761
Length	309,285	31.532	26.579	18.000	25.000	35.000
Fog	309,285	13.710	6.923	8.170	13.000	18.210
%Num	309,285	0.101	0.096	0.000	0.085	0.161
Sentiment	309,285	0.003	0.035	0.000	0.000	0.000
Guidance	309,285	0.052	0.223	0.000	0.000	0.000
Firm_Size	309,285	14.038	41.254	0.559	2.228	7.839
Analyst	309,285	7.836	5.849	3.000	6.000	11.000
Panel B. Pearson Correlations for Sentence-Level Variables
	1	2	3	4	5	6	7	8	9
1	GPTSentence	1.00
2	Order	-0.14	1.00
3	Length	0.24	-0.08	1.00
4	Fog	-0.08	0.04	0.21	1.00
5	%Num	0.13	-0.01	0.23	-0.52	1.00
6	Sentiment	-0.03	-0.11	-0.05	0.13	-0.17	1.00
7	Guidance	0.18	0.08	0.17	0.07	0.02	0.00	1.00
8	Firm_Size	-0.01	0.00	-0.01	0.01	-0.02	0.01	0.02	1.00
9	Analyst	-0.01	0.00	-0.03	0.02	-0.03	0.01	0.04	0.55	1.00

Panel A presents results from linear probability model (LPM) regressions of whether a press release sentence is selected by the GPT as important ($GPTSentence$) on the sentence textual characteristics and firm characteristics. Columns (1) and (2) present the results from the subsample of press releases issued in the period prior to the GPT-4 Turbo model’s training knowledge cutoff date (April 30, 2024), while Columns (3) and (4) present the results from the subsample of press releases issued in the period after the cutoff date. In Columns (1) and (3) we control for year-quarter and Fama-French 12 industry fixed effects. In Columns (2) and (4), we control for press release fixed effects. T-statistics are presented in parentheses. Standard errors are clustered at the firm level. ***, **, and * denote statistical significance at 1%, 5%, and 10%, respectively. See Appendix B for other variable definitions.

	(1)	(2)	(3)	(4)
Panel B. Determinants of GPT-4’s Sentence Selection, Conditional on Firm Information Environment
		Firm Size		Analyst Following
	Small	Large	Low	High
Order	-0.109***	-0.114***	-0.114***	-0.108***
	(-29.70)	(-28.01)	(-31.70)	(-26.38)
Length	0.093***	0.078***	0.086***	0.085***
	(26.68)	(23.43)	(26.33)	(23.59)
Fog	-0.005***	-0.005***	-0.005***	-0.005***
	(-23.05)	(-21.85)	(-23.70)	(-21.92)
%Num	0.057***	0.045***	0.031**	0.071***
	(3.49)	(2.89)	(2.02)	(4.29)
Sentiment	-0.214***	-0.060***	-0.168***	-0.113***
	(-8.13)	(-2.63)	(-6.46)	(-5.01)
Guidance	0.197***	0.170***	0.197***	0.168***
	(21.61)	(21.68)	(22.50)	(20.98)
Press Release Fixed Effects	YES	YES	YES	YES
N	154,697	154,588	163,254	146,031
adj. R-sq	0.109	0.120	0.106	0.124

Panel B presents results from linear probability model (LPM) regressions of whether a press release sentence is selected by the GPT as important (GPTSentence) on the sentence’s characteristics and the firm’s characteristics, conditional on the level of a firm’s information qualities. For the two firm-level variables that proxy for a firm’s information environment quality (Firm_Size, Analyst), we divide the full sample into high- and low-information environment quality subsamples based on the median values of these variables and conduct the regressions separately for each subsample. Columns (1) and (2) present the subsample analyses based on Firm_Size. Columns (3) and (4) present the subsample analyses based on Analyst. In all columns we control for press release fixed effects. T-statistics are presented in parentheses. Standard errors are clustered at the firm level. ***, **, and * denote statistical significance at 1%, 5%, and 10%, respectively. See Appendix B for other variable definitions.

Panel A presents the three financial metrics selected by GPT-4 and the corresponding frequencies. Column (1) presents the types of financial metrics selected by GPT-4 for its quantitative analysis of financial statements. Column (2) presents our mapping of GPT-selected metrics to the standard metrics typically reported in I/B/E/S Detail by individual analysts. Column (3) presents the frequencies of each financial metric selected by GPT-4. Column (4) presents the relative frequencies of each financial metric within our press release-level sample of 6,848 observations.

I/B/E/S Metrics	Frequency	% of Press Releases
EBIT (Non-Per Share, EBI)	5,092	74.36%
Revenue (Non-Per Share, SAL)	4,140	60.46%
Net Income (Non-Per Share, NET)	3,994	58.32%
EBITDA (Non-Per Share, EBT)	3,282	47.93%
Pre-tax Profit (Non-Per Share, PRE)	1,659	24.23%
Book Value Per Share (BPS)	483	7.05%
Gross Margin (Percent, GRM)	443	6.47%
Net Asset Value (Non-Per Share, NAV)	225	3.29%
Funds From Operations Per Share (FFO)	225	3.29%
Return on Equity (Percent, ROE)	196	2.86%
Cash Flow Per Share (CPS)	176	2.57%
Return on Assets (Percent, ROA)	163	2.38%
Capital Expenditure (Non-per share, CPX)	69	1.01%
Dividend Per Share (DPS)	55	0.80%
Earnings Per Share - Alternate (EPX)	15	0.22%
Enterprise Value (Non-Per Share, ENT)	9	0.13%
EBITDA Per Share (EBS)	6	0.09%
Net Debt (NDT)	5	0.07%
Operating Profit (Non-Per Share, OPR)	1	0.01%
Total	20,238

Panel B presents the top three non-EPS financial metrics most frequently reported by individual analysts in I/B/E/S Detail for each firm-year. EPS and GPS are excluded from this calculation. I/B/E/S Detail reports at least three non-EPS metrics in 6,713 firm-quarters (98% of 6,848 observations), with only 34 firm-quarters reporting two, 31 reporting one, and 70 reporting none.

Column (1) presents the names of the standard financial metrics forecasted by analysts as collected by I/B/E/S. Column (2) presents the frequencies of each financial metric forecasted by I/B/E/S analysts. Column (3) presents the relative frequencies of each financial metric within our press release-level sample of 6,848 observations.

#MetricOverlap	Frequency	% of All Press Release
3	451	6.59%
2	2,326	33.97%
1	3,539	51.68%
0	532	7.77%
Total	6,848	100.00%

	N	Mean	Std. Dev	P25	Median	P75
MetricOverlap_High	6,848	0.406	0.491	0.000	0.000	1.000
BS_#lineitems	6,848	37.785	4.288	35.000	38.000	41.000
IS_#lineitems	6,848	51.533	4.825	49.000	52.000	54.000
Benford’s Law Deviation	6,848	0.395	0.144	0.293	0.370	0.468
Firm_Size	6,848	12.553	38.724	0.457	1.843	6.321
Analyst	6,848	7.590	5.649	3.000	6.000	10.000

	1	2	3	4	5	6	7
1 MetricOverlap_High	1.00
2 BS_#lineitems	0.07	1.00
3 IS_#lineitems	0.13	0.67	1.00
4 Benford’s Law Deviation	-0.11	-0.45	-0.55	1.00
5 Firm_Size	0.05	0.18	0.14	-0.09	1.00
6 Analyst	0.16	0.24	0.21	-0.10	0.53	1.00

Panel C presents the level of quantitative metrics alignment between the financial metrics selected by GPT-4 and I/B/E/S analysts. The variable #MetricOverlap counts the occurrences of overlapping in financial metrics selected by GPT-4 and I/B/E/S analysts, and it ranges from 0 to 3, where 0 indicates no overlap and 3 indicates all three overlap. Panel D presents the descriptive statistics for the variables used in our GPT-Analyst quantitative metrics alignment analysis. We convert the #MetricOverlap into the dummy variable MetricOverlap_High, which equals 1 if #MetricOverlap >=2, and 0 otherwise. Panel E presents the Pearson correlations of variables used in our GPT-analyst quantitative metrics alignment analysis. Correlations that are statistically significant at the 1% level are presented in bold font. See Appendix B for variable definitions.

This table presents results from linear probability model (LPM) regressions analyzing the level of GPT-Analyst quantitative metrics alignment ($MetricOverlap\_High$) on qualities of financial statement numbers, firm characteristics, and time relative to GPT-4 Turbo’s knowledge cutoff. In Column (1) we control for year-quarter and Fama-French 12 industry fixed effects. In Columns (2) and (3), we control for Fama-French 12 industry fixed effects. In parentheses, we present t-statistics. Standard errors are clustered at the firm level. ***, **, and * denote statistical significance levels at 1%, 5%, and 10%, respectively. See Appendix B for other variable definitions.

Panel A presents descriptive statistics for the variables used in our regression analyses on GPT information processing and performance. Press release-related variables are measured for each earnings press release in the sample. Firm-level variables are measured at the fiscal quarter corresponding to each earnings press release. See Appendix B for variable definitions.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20
1 |FE_GPT|-|FE_Analyst|	1.00
2 GPT_Incremental_Order	0.06	1.00
3 GPT_Incremental_Length	0.00	-0.22	1.00
4 GPT_Incremental_Fog	0.00	0.08	0.10	1.00
5 GPT_Incremental_%Num	0.02	-0.02	0.19	-0.52	1.00
6 GPT_Incremental_Sentiment	-0.04	-0.18	0.04	0.23	-0.25	1.00
7 GPT_Incremental_Guidance	0.01	0.01	0.30	0.08	0.14	0.06	1.00
8 MetricOverlap_High	-0.04	-0.07	0.01	-0.02	0.04	-0.00	0.03	1.00
9 PostCutoff_Qtr1	0.00	0.01	-0.02	-0.01	0.01	-0.02	-0.01	0.00	1.00
10 PostCutoff_Qtr2	-0.03	-0.01	-0.01	-0.00	-0.01	-0.02	0.00	-0.00	-0.15	1.00
11 PostCutoff_Qtr3	-0.00	0.01	0.00	0.01	0.00	-0.00	-0.01	0.01	-0.13	-0.13	1.00
12 PostCutoff_Qtr4	0.03	-0.02	0.04	0.02	-0.02	0.01	-0.00	-0.02	-0.16	-0.15	-0.14	1.00
13 PR_Total_Length	-0.00	-0.32	0.07	-0.04	0.01	0.12	0.01	-0.02	-0.04	-0.02	-0.02	0.05	1.00
14 PR_General_Length	0.02	-0.00	0.33	-0.01	0.03	0.01	0.01	0.00	-0.01	0.01	0.02	-0.01	0.09	1.00
15 PR_General_Fog	0.08	0.05	0.12	-0.12	0.22	-0.07	0.05	-0.12	0.01	0.00	0.01	-0.04	0.15	0.37	1.00
16 PR_General_%Num	-0.03	-0.05	-0.06	0.11	-0.31	0.09	-0.06	0.05	-0.02	-0.01	-0.01	0.06	-0.04	-0.03	-0.61	1.00
17 PR_General_Sentiment	-0.02	0.04	0.06	-0.04	0.11	-0.14	0.06	-0.01	0.01	0.02	0.01	0.02	-0.12	0.03	0.11	-0.23	1.00
18 PR_General_Guidance	-0.02	0.05	0.09	-0.03	0.12	0.00	0.38	0.06	0.02	0.01	0.01	-0.03	-0.19	0.16	0.09	-0.07	0.12	1.00
19 Firm_Size	-0.05	-0.09	-0.03	-0.10	0.05	0.02	0.02	0.05	-0.01	0.00	-0.01	0.04	0.05	0.04	0.05	-0.02	0.05	0.05	1.00
20 Analyst	-0.07	-0.02	0.01	-0.07	0.08	-0.00	-0.01	0.16	-0.01	0.01	-0.00	0.01	0.04	0.03	0.04	-0.00	0.05	0.06	0.43	1.00

This table presents OLS regression results analyzing the difference GPT-4’s absolute forecast errors for non-GAAP EPS and analysts’ street EPS forecasts ($\lvert$FE_GPT_NonGAAP$\rvert$ - $\lvert$FE_Analyst_Street$\rvert$). The regressions incorporate variables related to the characteristics of GPT-selected sentences, GPT-Analyst quantitative metrics alignment, time relative to GPT’s knowledge cutoff, overall press release sentence characteristics, and firm characteristics. To simplify coefficient interpretation, we multiply all percentage-based variables by 100. These percentage-based variables include: $\lvert$FE_GPT_NonGAAP$\rvert$ - $\lvert$FE_Analyst_Street$\rvert$, GPT_incremental_%Num, GPT_incremental_Sentiment, PR_General_%Num, PR_General_Sentiment. In all columns, we control for Fama-French 12 industry fixed effects. T-statistics are presented in parentheses. Standard errors are clustered at the firm level. ***, **, and * denote statistical significance level at 1%, 5%, and 10%, respectively. See Appendix B for other variable definitions.