betaFeedback

WorksComputer Science

LAET: A Layer-wise Adaptive Ensemble Tuning Framework for Pretrained Language Models

Abstract

Natural Language Processing (NLP) has transformed the financial industry, enabling advancements in areas such as textual analysis, risk management, and forecasting. Large language models (LLMs) like BloombergGPT and FinMA have set new benchmarks across various financial NLP tasks, including sentiment analysis, stock movement prediction, and credit risk assessment. Furthermore, FinMA-ES, a bilingual financial LLM, has also demonstrated strong performance using the FLARE and FLARE-ES benchmarks. However, the high computational demands of these models limit the accessibility of many organizations. To address this, we propose Layer-wise Adaptive Ensemble Tuning (LAET), a novel strategy that selectively fine-tunes the most effective layers of pre-trained LLMs by analyzing hidden state representations while freezing less critical layers. LAET significantly reduces computational overhead while enhancing task-specific performance. Our approach shows strong results in financial NLP tasks, outperforming existing benchmarks and state-of-the-art LLMs such as GPT-4, even with smaller LLMs (∼\sim3B parameters). This work bridges cutting-edge financial NLP research and real-world deployment with efficient and scalable models for financial applications.

Introduction

Natural Language Processing (NLP) has become a transformative tool across various domains, enabling advancements in tasks such as sentiment analysis, named entity recognition (NER), prediction, and question-answering (QA). With the support of machine learning, it is now possible to extract valuable insights from unstructured text sources like reports, news articles, and social media. However, domain-specific texts often contain technical language, specialized terminology, and complex structures that challenge the performance of general-purpose language models [1, 2, 3]. This has led to growing interest in adapting NLP techniques to domain-specific contexts for improved accuracy and reliability.

Recent progress in Large Language Models (LLMs), such as GPT-4, has demonstrated strong generalization capabilities across domains, including healthcare, law, programming, and finance [4, 5, 6]. These models can perform complex tasks by following natural language instructions with minimal fine-tuning. Yet, in highly specialized domains, their effectiveness is often limited by a lack of domain-specific understanding

and evaluation benchmarks [2]. To address this, researchers have introduced domain-adapted models like finBERT [1], Fin-BERT [7], and FLANG [8], which are pre-trained on targeted datasets. While useful, these models typically have smaller parameter sizes (under one billion), limiting their adaptability to broader tasks. More recently, large-scale models such as BloombergGPT [9] have emerged, showing the potential of domain-specific LLMs trained at scale.

Despite these developments, practical deployment of domain-specific language models still faces key challenges:

(a) Architectural complexity — advanced models like BloombergGPT [9], FinGPT [10], PIXIU [11], FinB [12], and multilingual FinMA-ES [13] deliver strong results but require highly complex and resource-intensive architectures.

(b) High resource demands — training and fine-tuning these large models is computationally expensive and often inaccessible to smaller organizations. (c) Inefficient layer usage – emerging evidence suggests that many layers in large LLMs contribute little to performance, leading to unnecessary computation [14, 15, 16].

Recent innovations in compact models like Phi [17], Gemma [18], and LLaMA [19] offer promising alternatives. These lightweight models reduce computational overhead while retaining strong performance, making them more accessible and easier to deploy. To address the efficiency-performance trade-off, this paper proposes a novel methodology for optimizing pre-trained language models by leveraging layer-wise representations. By identifying the most impactful layers for a given task, we fine-tune only those layers while freezing the rest, reducing computational costs. An intermediate neural network further aggregates selected features to enhance performance. Figure 1 depicts our proposed method LAET. Its selective adaptation strategy enables more efficient and scalable use of language models across specialized tasks.

Figure 1. Comparison of full fine-tuning, parameter-efficient fine-tuning (PEFT), and our proposed Layer-wise Adaptive Ensemble Tuning (LAET). LAET identifies and updates only the most effective layers while keeping all remaining layers frozen. This substantially reduces computational overhead compared to full fine-tuning. At the same time, it achieves higher performance than conventional PEFT approaches (eg, LoRA, DoRA, etc), which tune only small adapter modules and later merge their weights.

The key contributions are as follows:

  • Proposing LAET: a layer-wise adaptation technique to maximize utility from individual layers.

  • Incorporating an ensemble-based fine-tuning strategy to enhance overall effectiveness.

  • Achieving strong performance on multi-domain tasks using smaller LLMs.

Depth Knowledge and Concepts in LLMs: The strong performance of LLMs has sparked debate over whether they truly understand concepts or simply mimic patterns. Studies show that LLMs encode structured knowledge internally [20, 21], and altering specific components can impact reasoning [22], indicating concept-specific structures. Layer-wise analyses reveal that not all layers are equally important [16, 23], with pruning [15, 24] and probing [25] methods exposing redundancy. Building on this, we propose a concept depth analysis approach that selectively trains the most relevant layers, enabling compact LLMs to perform competitively with lower computational cost.

Parameter-Efficient Fine-tuning: Parameter-Efficient Fine-Tuning (PEFT) methods aim to reduce the high cost of adapting large language models by updating only a small subset of parameters. These include adapter-based approaches [26] that insert lightweight trainable modules into frozen backbones, and prompt-based methods that optimize soft tokens prepended to inputs. Among PEFT techniques, LoRA [27] introduces trainable low-rank matrices into attention layers, DoRA [28] decouples weight norm and direction to fine-tune only directional components, and AdaLoRA [29] adapts rank dynamically during training. However, these methods typically tune all layers uniformly, overlooking the layer-wise relevance and introducing redundancy. To overcome these limitations, we propose LAET, which leverages concept-depth analysis to selectively fine-tune only the most semantically critical layers. This targeted adaptation reduces cost while maintaining strong performance.

Methodology

Problem Formulation

Hidden state representations: Let M\mathcal{M} be a pre-trained large language model (LLM) with LL layers. The corresponding tokenizer T\mathcal{T} maps an input text sequence x\mathbf{x} to token IDs, such that: T(x)={t1,t2,…,tn}\mathcal{T}(\mathbf{x}) = \{t_1,t_2,\ldots,t_n\}, where nn represents the number of tokens resulting from the tokenization process. For each layer l∈{1,2,…,L}l \in\{1,2,\ldots,L\}, the model M\mathcal{M} computes hidden state representations for each token. The hidden states at layer ll for all tokens are denoted as: Hl={h1(l),h2(l),…,hn(l)}\mathbf{H}_l = \{\mathbf{h}^{(l)}_1,\mathbf{h}^{(l)}_2,\ldots,\mathbf{h}^{(l)}_n\} where hi(l)∈Rd\mathbf{h}^{(l)}_i \in\mathbb{R}^d represents the hidden state at layer ll for token jj, with dd being the hidden state dimension. For each layer ll, the representation of the last token is defined as: rl=rn(l)=h[:,r](l)\mathbf{r}_l = \mathbf{r}^{(l)}_n = \mathbf{h}^{(l)}_{[:,r]}, rl∈Rd\mathbf{r}_l \in\mathbb{R}^d.

Implication: Consider a dataset D={(xi,yi)}i=1N\mathcal{D} = \{(x_i,y_i)\}_{i=1}^{N}, where each xi∈Rdx_i \in\mathbb{R}^d is an input text and yi∈Cy_i \in\mathcal{C} is the corresponding label, with C={c1,c2,…,ck}\mathcal{C} = \{c_1,c_2,\ldots,c_k\} representing the set of kk possible classes. For each input xix_i, the last token representation ri(l)\mathbf{r}^{(l)}_i from layer ll is computed. A small neural network classifier, denoted Fϕ\mathcal{F}_{\phi}, parameterized by ϕ\phi, is employed to classify the last token representations from all layers using a single set of shared weights. The logits for each input representation ri(l)\mathbf{r}^{(l)}_i from layer ll are computed as: zil=Fϕ(ri(l))z^{l}_i = \mathcal{F}_{\phi}(\mathbf{r}^{(l)}_i), zil∈Rkz^{l}_i \in\mathbb{R}^k. Using the predicted class probabilities pil[j]p^{l}_i[j] for each class j∈Cj \in\mathcal{C}, the neural network is trained to evaluate the performance of each layer. Based on these performance evaluations, the best-performing layers, denoted as B\mathcal{B}, are identified. Rather than training the entire model M\mathcal{M}, the approach focuses on leveraging the performance of the best layers B\mathcal{B} for downstream tasks. To achieve this, we freeze the weights of the non-selected layers and train only the best layers B\mathcal{B}, alongside the shared classifier Cϕ\mathcal{C}_{\phi}. The final prediction y^\hat{y} is computed as the majority vote over the outputs from the best-performing layers B\mathcal{B}.

Layer-wise Adaptive Ensemble Tuning (LAET)

We propose LAET, a selective fine-tuning strategy that identifies and optimizes the most effective layers while incorporating ensemble decision-making, illustrated in Figure 2. The objective of LAET is to enhance the task-specific adaptation of smaller models with minimal computational overhead.

LAET architecture showing layer probing, adaptive ensemble tuning, and majority voting

Figure 2. LAET: Initially, (a) for each layer ll in a pre-trained model M\mathcal{M}, representations ri(l)\mathbf{r}_i^{(l)} are extracted, logits zilz_i^l are computed using the classifier Fϕ\mathcal{F}_{\phi}, and a cross-entropy loss Ll\mathcal{L}_l is calculated. Next, (b) the best-performing layers are selected based on evaluation metrics and their deviations from the maximum values. Layers l∉Bl \notin\mathcal{B} are frozen, while the selected layers and shared classifier are fine-tuned. During inference, a voting-based ensemble strategy aggregates predictions from the selected layers to generate the final output.

Hidden states: The model M\mathcal{M} converts each token ti∈T(x)t_i \in\mathcal{T}(x) into a dense vector representation using an embedding matrix E∈R∣V∣×dE \in\mathbb{R}^{|V|\times d}, where ∣V∣|V| is the size of the vocabulary and dd is the embedding dimension. For each token tit_i, the embedding ei∈Rde_i \in\mathbb{R}^d is: ei=E[ti]e_i = E[t_i]. Thus, the entire token sequence x\mathbf{x} is transformed into an embedding matrix, E={e1,e2,…,en}\mathbf{E} = \{e_1,e_2,\ldots,e_n\}, E∈Rn×d\mathbf{E} \in\mathbb{R}^{n\times d}. The representations of the input tokens are initialized with these token embeddings E\mathbf{E}, which incorporate information about the identity of each token. A positional encoding matrix P∈Rn×dP \in\mathbb{R}^{n\times d} is added to the token embeddings to inject information about the position of each token in the sequence, resulting in the following input representations: r(0)=E+P\mathbf{r}^{(0)} = \mathbf{E} + P, where r(0)∈Rn×d\mathbf{r}^{(0)} \in\mathbb{R}^{n\times d} is the matrix of token representations that serve as the input to the first transformer layer. These representations r(0)\mathbf{r}^{(0)} are passed through multiple transformer layers LL. For each layer l∈Ll \in L, the model computes new representations for each token based on self-attention and feedforward transformations. Specifically, for each token ii, the representation at layer ll, denoted as ri(l)\mathbf{r}_i^{(l)}, is derived as follows. In the self-attention mechanism, for each token ii, the model computes three vectors: (a) Query vector: qi=WQhi(l−1)\mathbf{q}_i = W_Q\mathbf{h}_i^{(l-1)}, (b) Key vector: ki=WKhi(l−1)\mathbf{k}_i = W_K\mathbf{h}_i^{(l-1)}, (c) Value vector: vi=WVhi(l−1)\mathbf{v}_i = W_V\mathbf{h}_i^{(l-1)}, where hi(l−1)\mathbf{h}_i^{(l-1)} is the hidden state from the previous layer for token ii, and WQ,WK,WV∈Rd×dW_Q, W_K, W_V \in\mathbb{R}^{d \times d} are learned projection matrices. The attention scores between token ii and all other tokens jj are computed. The attention scores are normalized using softmax and applied to the value vectors vj\mathbf{v}_j:

hi(l)=∑j=1nsoftmax⁡(Attention⁡(i,j))vj.\mathbf{h}_i^{(l)} = \sum_{j=1}^{n} \operatorname{softmax}(\operatorname{Attention}(i,j))\mathbf{v}_j.

This produces the updated hidden state hi(l)∈Rd\mathbf{h}_i^{(l)} \in\mathbb{R}^{d} for token ii at layer ll. After applying the self-attention mechanism and feedforward layers, the hidden states are used to compute the new representations at each layer, denoted as ri(l)\mathbf{r}_i^{(l)}. Thus, the representation ri(l)\mathbf{r}_i^{(l)} for token ii at layer ll is ultimately a transformation of the hidden state hi(l)\mathbf{h}_i^{(l)}.

Layer Probing: For each input text xi∈Rdx_i \in\mathbb{R}^{d} with its corresponding label yi∈Cy_i \in C, hidden representations are extracted from each layer ll while keeping the model M\mathcal{M} in inference mode. The final hidden representations ri(l)\mathbf{r}_i^{(l)} are stored for every input, forming the set: R={Rl}l=1L\mathcal{R} = \{\mathcal{R}_l\}_{l=1}^{L}, where Rl={(ri(l),yi)}i=1N\mathcal{R}_l = \left\{(\mathbf{r}_i^{(l)}, y_i)\right\}_{i=1}^{N}. Here, Rl\mathcal{R}_l represents the collection of hidden representations from layer ll, paired with their corresponding labels across all NN inputs. These representations serve as input to a small neural classifier Fϕ\mathcal{F}_{\phi}, which probes each layer’s ability to encode task-relevant information. For each representation ri(l)\mathbf{r}_i^{(l)}, the classifier produces logits zilz_i^l, which are transformed into class probabilities pil[j]p_i^l[j] for class j∈Cj \in C using the softmax function:

pil[j]=exp⁡(zil[j])∑j′=1kexp⁡(zil[j′]),pil∈Rk.p_i^l[j] = \frac{\exp(z_i^l[j])}{\sum_{j'=1}^{k}\exp(z_i^l[j'])}, \quad p_i^l \in\mathbb{R}^{k}.

The loss for each input ii at layer ll is computed using cross-entropy between the predicted probabilities and the true label yiy_i. The total loss for layer ll across all inputs is given by:

Ll=−1N∑i=1N∑j=1kP[yi=j]log⁡(pil[j])\mathcal{L}_l = -\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{k}\mathcal{P}[y_i=j]\log(p_i^l[j])

where P(yi=j)\mathcal{P}(y_i = j) is an indicator function that equals 1 if yi=jy_i = j and 0 otherwise.

Best Layers Selection: The shared classifier Fϕ\mathcal{F}_{\phi} is trained on each layer’s representation. Gradient updates for the classifier Fϕ\mathcal{F}_{\phi} are as follows:

∇ϕLl=1N∑i=1N∑j=1k(pil[j]−P(yi=j))∇ϕFϕ(ri(l)).\nabla_{\phi}\mathcal{L}_l = \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{k}(p_i^l[j]-\mathcal{P}(y_i=j))\nabla_{\phi}\mathcal{F}_{\phi}(\mathbf{r}_i^{(l)}).

After training, two performance metrics, based on literature, m1m_1 (e.g., accuracy) and m2m_2 (e.g., F1 score or MCC), are computed for each layer l∈{1,…,L}l \in\{1,\ldots,L\}. The standard deviations σm1\sigma_{m_1} and σm2\sigma_{m_2} across all layers are computed as:

σm1=std⁡({m11,…,m1L}),σm2=std⁡({m21,…,m2L}).\sigma_{m_1} = \operatorname{std}(\{m_1^1,\ldots,m_1^L\}), \quad\sigma_{m_2} = \operatorname{std}(\{m_2^1,\ldots,m_2^L\}).

. Dynamic margins are then defined as: δm1=α⋅σm1\delta_{m_1} = \alpha\cdot\sigma_{m_1}, δm2=β⋅σm2\delta_{m_2} = \beta\cdot\sigma_{m_2}. A layer ll is included in the final set of best-performing layers B\mathcal{B} if no other layer l′≠ll' \ne l satisfies:

m1l′≥m1l+δm1andm2l′≥m2l+δm2.m_1^{l'} \geq m_1^l + \delta_{m_1} \quad\text{and} \quad m_2^{l'} \geq m_2^l + \delta_{m_2}.

. The resulting subset B\mathcal{B} defines the layers used during fine-tuning. To formally analyze the optimization behavior under this selection, consider the modified training objective:

LB(θB,ϕ)=1∣B∣∑l∈BL(l)(θl,ϕ),\mathcal{L}_{\mathcal{B}}(\theta_{\mathcal{B}},\phi) = \frac{1}{|\mathcal{B}|}\sum_{l\in\mathcal{B}}\mathcal{L}^{(l)}(\theta_l,\phi),

where θB={θl}l∈B\theta_{\mathcal{B}} = \{\theta_l\}_{l\in\mathcal{B}} denotes the parameters of the selected layers, and ϕ\phi represents the shared classifier parameters. Each loss term L(l)\mathcal{L}^{(l)} corresponds to the output computed from the representation r(l)\mathbf{r}^{(l)} of layer ll. Assuming that L(l)\mathcal{L}^{(l)} is differentiable with Lipschitz-continuous gradients and that updates are performed using stochastic gradient descent with a diminishing learning rate ηt\eta_t satisfying ∑tηt=∞\sum_t \eta_t = \infty and ∑tηt2<∞\sum_t \eta_t^2 < \infty, it follows from standard stochastic approximation theory that:

lim⁡t→∞E[∥∇θBLB∥2+∥∇ϕLB∥2]=0.\lim_{t \to\infty} \mathbb{E}\left[\left\|\nabla_{\theta_{\mathcal{B}}}\mathcal{L}_{\mathcal{B}}\right\|^2 + \left\|\nabla_{\phi}\mathcal{L}_{\mathcal{B}}\right\|^2\right] = 0.

This establishes that convergence to a stationary point is guaranteed despite partial updates to a subset of layers. In addition to its convergence properties, the proposed standard deviation-based selection method offers greater robustness than gradient-based alternatives. Approaches that rely on gradient norm magnitudes, such as ∥∇θlL∥\left\|\nabla_{\theta_l}\mathcal{L}\right\|, are often sensitive to depth-related variance and may not correlate with downstream performance. In contrast, LAET defines selection thresholds concerning validation performance, ensuring that only layers exhibiting statistically stable and semantically aligned behavior are selected. Specifically, the retained set of layers satisfies B\mathcal{B}. Empirically, this approach reduces the number of trainable layers without compromising accuracy. Theoretically, it can be interpreted as selecting from the Pareto frontier of performance metrics, in contrast to gradient-based methods that do not encode task-level validation behavior. This strategy boosts both the efficiency and generalization of LAET.

Fine tune strategy: Once the best-performing layers B\mathcal{B} are identified, the remaining layers, l∉Bl \notin\mathcal{B}, are frozen, while the selected layers, l∈Bl \in\mathcal{B} within M\mathcal{M}, along with the shared-weight classifier Fϕ\mathcal{F}_{\phi}, undergo fine-tuning. Since the classifier is shared across all selected layers, the total loss across B\mathcal{B} is computed as: L=1∣B∣∑l∈BLl\mathcal{L} = \frac{1}{|\mathcal{B}|}\sum_{l \in\mathcal{B}}\mathcal{L}_l. The gradient of the total loss with respect to the classifier parameters ϕ\phi is given by:

∇ϕL=1∣B∣∑l∈B∇ϕLl.\nabla_{\phi}\mathcal{L} = \frac{1}{|\mathcal{B}|}\sum_{l \in\mathcal{B}}\nabla_{\phi}\mathcal{L}_l.

Expanding ∇ϕLl\nabla_{\phi}\mathcal{L}_l using the softmax derivative:

∇ϕLl=1N∑i=1N∑j=1k(pil[j]−P(yi=j))∇ϕzil[j].\nabla_{\phi}\mathcal{L}_l = \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{k}(p_i^l[j]-\mathcal{P}(y_i=j))\nabla_{\phi}z_i^l[j].

Since the logits zilz_i^l are produced by the classifier, zil=Fϕ(ri(l))z_i^l=\mathcal{F}_{\phi}(r_i^{(l)}), applying the chain rule yields:

∇ϕzil[j]=∂Fϕ(ri(l))∂ϕ.\nabla_{\phi}z_i^l[j] = \frac{\partial\mathcal{F}_{\phi}(r_i^{(l)})}{\partial\phi}.

Thus, the final gradient update for the shared classifier Fϕ\mathcal{F}_{\phi} is:

∇ϕLF=1∣B∣N∑l∈B∑i=1N∑j=1k(pil[j]−P(yi=j))∇ϕFϕ(ri(l)).\nabla_{\phi}\mathcal{L}_{\mathcal{F}} = \frac{1}{|\mathcal{B}|N}\sum_{l \in\mathcal{B}}\sum_{i=1}^{N}\sum_{j=1}^{k}(p_i^l[j]-\mathcal{P}(y_i=j))\nabla_{\phi}\mathcal{F}_{\phi}(r_i^{(l)}).

For the model M\mathcal{M} with parameters θ\theta, updates are applied only to the selected layers B\mathcal{B}. The gradient with respect to θl\theta_l for each l∈Bl \in\mathcal{B} is computed as:

∇θlLM=1∣B∣N∑i=1N∑j=1k(pil[j]−P(yi=j))∇θlri(l).\nabla_{\theta_l}\mathcal{L}_{\mathcal{M}} = \frac{1}{|\mathcal{B}|N}\sum_{i=1}^{N}\sum_{j=1}^{k}(p_i^l[j]-\mathcal{P}(y_i=j))\nabla_{\theta_l}r_i^{(l)}.

Voting-based Ensembled for Prediction: After training both the classifier Fϕ\mathcal{F}_{\phi} and the model M\mathcal{M}, the final prediction is obtained through an ensemble over the selected layers B\mathcal{B}. For a given unseen input xux_u, each layer l∈Bl \in\mathcal{B} produces a prediction y^l=arg⁡max⁡jpul[j]\hat{y}_l=\arg\max_j p_u^l[j], where pul=softmax⁡(Fϕ(ru(l)))p_u^l=\operatorname{softmax}(\mathcal{F}_{\phi}(r_u^{(l)})) is the class probability vector. Instead of relying on any single layer, a majority voting rule is employed to aggregate predictions across all selected layers. The final predicted class is given:

y^=arg⁡max⁡c∈C∑l∈BI(y^l=c),\hat{y}=\arg\max_{c \in\mathcal{C}}\sum_{l \in\mathcal{B}}\mathbb{I}(\hat{y}_l=c),

where I(⋅)\mathbb{I}(\cdot) is the indicator function. This procedure reduces sensitivity to any single layer’s misprediction and exploits the diversity among informative layer representations. Under the assumption that predictions y^l\hat{y}_l from different layers are conditionally independent, and that each layer predictor has a generalization error rate bounded by ϵl=P(y^l≠y)\epsilon_l=\mathcal{P}(\hat{y}_l\ne y), the expected ensemble error can be analyzed using classical results from ensemble learning. Let ϵˉ=1∣B∣∑l∈Bϵl\bar{\epsilon}=\frac{1}{|\mathcal{B}|}\sum_{l\in\mathcal{B}}\epsilon_l denote the average error rate. If ϵˉ<0.5\bar{\epsilon}<0.5, then the error of the ensemble prediction satisfies the exponential bound:

P(y^≠y)≤exp⁡(−2∣B∣(0.5−ϵˉ)2).\mathcal{P}(\hat{y}\ne y)\le\exp\left(-2|\mathcal{B}|(0.5-\bar{\epsilon})^2\right).

This result shows that ensemble error decreases rapidly as more accurate layers are included in the vote. The generalization error of each layer predictor is influenced by the complexity of the shared classifier Fϕ\mathcal{F}_{\phi}. Let H\mathcal{H} denote the hypothesis class defined by Fϕ\mathcal{F}_{\phi}, with VC-dimension VC⁡(H)\operatorname{VC}(\mathcal{H}). Assume that each layer predictor is trained using NN probing samples. Then, for any layer ll and for any δ>0\delta>0, the probability that the difference between the empirical error ϵ^l(h)\hat{\epsilon}_l(h) and the true error ϵl(h)\epsilon_l(h) exceeds δ\delta is bounded by:

P(∣ϵ^l(h)−ϵl(h)∣>δ)≤2⋅(1N)2δ2−VC⁡(H)log⁡NN.\mathcal{P}\left(\left|\hat{\epsilon}_l(h)-\epsilon_l(h)\right|>\delta\right)\le2\cdot\left(\frac{1}{N}\right)^{2\delta^2-\frac{\operatorname{VC}(\mathcal{H})\log N}{N}}.

This implies that with enough probing samples and a moderately complex classifier, each layer predictor generalizes reliably, supporting the stability of the majority vote ensemble.

Primary Area of Assessment: Finance

To evaluate our method, LAET, we begin with the financial domain due to its rich set of benchmark datasets for text classification and the availability of recent strong baselines. We assess across three areas: Textual Analysis (TA), Forecasting (FO), and Risk Management (RM)—each capturing distinct reasoning challenges. Together, they provide a comprehensive testbed for evaluating LLM performance. See Table 1 for details. (1) TA covers eight tasks including sentiment classification (using FPB [30], FiQA-SA [31], TSA [32]), headline-based price movement prediction [35], policy tone classification [36], argument mining [37], multi-class document tagging [38], and MA event prediction [39]. (2) FO involves stock movement prediction using datasets like BigData22 [20], ACL18 [40], and CIKM18 [41], combining financial time series with textual sentiment. (3) RM tasks include credit scoring [42]–[44], fraud detection, bankruptcy prediction, and insurance claim analysis [44]. These domains collectively

DatasetsTaskLanguageData TypeModalitiesTrainVal.TestEvaluation
FPB [30]sentiment analysisEnglishnewstext3100776970Accuracy, F1
FiQA-SA [31]sentiment analysisEnglishnews headlines, tweetstext750188235Accuracy, F1
TSA [32]sentiment analysisEnglishnews headlinestext448–113RMSE
TSA [33]sentiment analysisSpanishnews headlinestext3063–766Accuracy, F1
FinanceES [34]sentiment analysisSpanishnews headlinestext5084–1272Accuracy, F1
Headlines [35]news headline classificationEnglishnews headlinestext16437–4110Accuracy, Avg F1
FOMC [36]hawkish-dovish classificationEnglishnewstext396–100Accuracy, F1
FinArg-ACC [37]argument unit classificationEnglishcommentstext6202–1551Accuracy, Micro F1
MultiFin [38]multi-class classificationEnglisharticle headlinestext5350–1340Accuracy, Micro F1
MultiFin [38]multi-class classificationSpanisharticle headlinestext210–20Accuracy, Micro F1
MA [39]deal completeness classificationEnglishnews, tweetstext400–100Accuracy, Micro F1
BigData22 [20]stock movement predictionEnglishtweets, historical pricestext, time series49007681470Accuracy, F1, MCC
ACL18 [40]stock movement predictionEnglishtweets, historical pricestext, time series2080025603720Accuracy, F1, MCC
CIKM18 [41]stock movement predictionEnglishtweets, historical pricestext, time series34001140431Accuracy, F1, MCC
German [42]credit scoringEnglishbiographyMultivariate700100200Accuracy, F1, MCC
Australian [43]credit scoringEnglishbiographyMultivariate48269139Accuracy, F1, MCC
LendingClub [44]credit scoringEnglishloan recordMultivariate942013502690Accuracy, F1, MCC
ccf [44]fraud detectionEnglishfinancial recordMultivariate797011402280Accuracy, F1, MCC
cfraud [44]fraud detectionEnglishfinancial recordtext734010502100Accuracy, F1, MCC
polish [44]financial distress identificationEnglishfinancial profiletext, table60808681740Accuracy, F1, MCC
taiwan [44]financial distress identificationEnglishfinancial profiletext, table47706811370Accuracy, F1, MCC
ProtoSegno [44]financial distress identificationEnglishfinancial profileMultivariate833011902380Accuracy, F1, MCC
travrealtinsurance [44]claim analysisEnglishfinancial profileMultivariate887012702530Accuracy, F1, MCC

Table I. Statistics of the employed benchmark datasets along with their corresponding evaluation metrics.

Algorithm 1 LAET

  1. Note: A pre-trained model M\mathcal{M} with LL layers, dataset D\mathcal{D}, classifier Fϕ\mathcal{F}_{\phi}, selection thresholds α,β\alpha,\beta; Purpose: fine-tune classifier Fϕ\mathcal{F}_{\phi} and selected layers B\mathcal{B}.

  2. Input: xi∈Dx_i \in\mathcal{D}, where xi∈Rdx_i \in\mathbb{R}^{d}.

  3. Output: Classification y^i∈C\hat{y}_i \in\mathcal{C}.

  4. Step 1: Train Classifier for Each Layer

  5. for l=1l = 1 to LL do

  6. Extract representation: ri(l)←Ml(xi)r_i^{(l)} \leftarrow\mathcal{M}_l(x_i)

  7. Compute logits: zil←Fϕ(ri(l))z_i^l \leftarrow\mathcal{F}_{\phi}(r_i^{(l)})

  8. Compute probability: pil←Softmax⁡(zil)p_i^l \leftarrow\operatorname{Softmax}(z_i^l)

  9. Compute loss: Ll←CrossEntropy⁡(pil,yi)\mathcal{L}_l \leftarrow\operatorname{CrossEntropy}(p_i^l,y_i)

  10. Update classifier: ϕ←ϕ−η∇ϕLl\phi\leftarrow\phi-\eta\nabla_{\phi}\mathcal{L}_l

  11. end for

  12. Step 2: Select Best Layers

  13. Compute metrics: m1l,m2l∀lm_1^l,m_2^l \quad\forall l

  14. Compute standard deviations: σm1,σm2\sigma_{m_1},\sigma_{m_2}

  15. Select best layers: B←{l∣m1l≥max⁡m1−ασm1,m2l≥max⁡m2−βσm2}\mathcal{B}\gets\{l\mid m_1^l\geq\max m_1-\alpha\sigma_{m_1},m_2^l\geq\max m_2-\beta\sigma_{m_2}\}

  16. Step 3: Freeze Layers Except Best Layers

  17. for l∉Bl \notin\mathcal{B} do

  18. Freeze layer: Ml←Frozen⁡\mathcal{M}_l\leftarrow\operatorname{Frozen}

  19. end for

  20. Step 4: Fine-Tune Best Layers and Classifier

  21. for each training step do

  22. for l∈Bl\in\mathcal{B} do

  23. Extract representation: ri(l)←Ml(xi)r_i^{(l)}\leftarrow\mathcal{M}_l(x_i)

  24. Compute logits: zil←Fϕ(ri(l))z_i^l\leftarrow\mathcal{F}_{\phi}(r_i^{(l)})

  25. Compute probability: pil←Softmax⁡(zil)p_i^l\leftarrow\operatorname{Softmax}(z_i^l)

  26. Compute loss: Ll←CrossEntropy⁡(pil,yi)\mathcal{L}_l\leftarrow\operatorname{CrossEntropy}(p_i^l,y_i)

  27. end for

  28. Compute total loss: L←1∣B∣∑l∈BLl\mathcal{L}\leftarrow\frac{1}{|\mathcal{B}|}\sum_{l\in\mathcal{B}}\mathcal{L}_l

  29. Update classifier: ϕ←ϕ−η∇ϕL\phi\leftarrow\phi-\eta\nabla_{\phi}\mathcal{L}

  30. Update selected layers: θB←θB−η∇θBL\theta_{\mathcal{B}}\leftarrow\theta_{\mathcal{B}}-\eta\nabla_{\theta_{\mathcal{B}}}\mathcal{L}

  31. end for

  32. Step 5: Voting-Based Ensemble Prediction

  33. for each input xux_u do

  34. Compute predictions: y^l←Fϕ(ru(l))∀l∈B\hat{y}_l\leftarrow\mathcal{F}_{\phi}(r_u^{(l)})\quad\forall l\in\mathcal{B}

  35. Compute final prediction: y^←arg⁡max⁡c∑l∈BI(y^l=c)\hat{y}\leftarrow\arg\max_c\sum_{l\in\mathcal{B}}\mathbb{I}(\hat{y}_l=c)

  36. end for

Experiment

Setup

Dataset: We utilized 23 datasets covering 11 tasks, sourced from established benchmarks such as PIXIU [11], FinBen [12], and FLARE-ESG [13]. Each dataset D={(xi,yi)}\mathcal{D}=\{(x_i,y_i)\} consists of query-answer pairs, where queries combine human-written instructions with task-specific inputs.

Input Text: [instruction][text] "Answer:" Response: [answer]

This unified format enables consistent instruction tuning across domains. While most labels are categorical (e.g., sentiment or classification tags), some datasets (e.g., TSA [33]) use scalar outputs. This structure standardizes evaluation for a wide range of tasks beyond any single domain.

Model Selection: For our experiment, we focused on utilizing smaller LLMs and a neural network. Our goal was to ensure efficient performance in financial tasks with compact models, avoiding the need for larger models. Therefore, we selected LLMs with approximately 3 billion parameters and designed a lightweight neural network to facilitate layer-wise adaptation, supporting the overall pipeline for efficient task-specific tuning. (a) LLMs: For our experiments, we considered three instruct tuned smaller LLMs: M1\mathcal{M}_1: gemma-2-2b-it1 (2B parameters), M2\mathcal{M}_2: Llama-3.2-3B-Instruct2 (3.2B parameters), M3\mathcal{M}_3: Phi-3.5-mini-instruct3 (3.8B parameters). Although M2\mathcal{M}_2 and M3\mathcal{M}_3 support up to 128K token context windows, and M1\mathcal{M}_1 supports 8K, we limit context length in practice to Cl∈[400,4000]C_l\in[400,4000] based on dataset size and computational efficiency. Complementing these models, (b) the shared classifier F\mathcal{F} consists of three fully connected

enable a comprehensive evaluation of LLM performance under real-world financial reasoning challenges. layers: the input is first transformed by z1=W1x+b1z_1 = W_1x + b_1 into a 128-dimensional space, followed by a ReLU activation. This is passed through a second layer z2=W2a1+b2z_2 = W_2a_1 + b_2 yielding a 64-dimensional representation, again followed by ReLU. Finally, the output is generated via y^=W3a2+b3\hat{y} = W_3a_2 + b_3, where the output dimension corresponds to the number of task classes. For the regression-based TSA [33] task, the output layer is modified to produce a scalar value with no activation, enabling direct prediction for MSE loss. This consistent and modular setup supports efficient layer-wise adaptation across tasks. Layer Probing Metrics: During the process of selecting the best-performing layers, the function F\mathcal{F} is trained on the dataset Rl\mathcal{R}_l, while the model M\mathcal{M} is kept in inference mode to prevent updates to the model’s parameters. The performance of the layers is evaluated using two key metrics: accuracy (m1m_1) and F1-score (m2m_2). The hyperparameters α\alpha and β\beta are set to 0.5, ensuring equal weighting between the two performance measures. Additionally, the margin for m1m_1 and m2m_2 is defined as half of their respective standard deviations across the layers. Implementation Details4. The datasets, D\mathcal{D} used in this study are sourced from The FinAI5 benchmark, as referenced in [11, 12, 13]. We utilized AutoModelForCausalLM and AutoTokenizer from the Hugging Face Transformers library to load all models, M\mathcal{M} and their respective tokenizers, T\mathcal{T}. The small shared neural network, F\mathcal{F} was implemented using the torch.nn module. F\mathcal{F} was trained for 200 epochs with a learning rate of 2e−42e-4 to determine the optimal layers. In the final training phase, the M\mathcal{M} and F\mathcal{F} were trained for 50 epochs. Backpropagation was applied to the unfrozen layers of the M\mathcal{M} with a learning rate of 2e−52e-5 and weight decay of 1e−41e-4, while the classifier was trained with a learning rate of 2e−42e-4. The pipeline for training is given in Algorithm 1. The entire experiment was conducted using the PyTorch framework on an RTX 6000 Ada GPU. Evaluation Metrics: We used Accuracy, F1 Score, and Matthews Correlation Coefficient (MCC) for evaluation. Table I provides the specific evaluation metrics used for each dataset. Accuracy measures the overall proportion of correct predictions. The F1 Score, including micro and average F1, balances precision and recall to handle class imbalance, with

micro F1 considering all instances equally and macro F1 averaging scores across classes. MCC provides a balanced evaluation by considering true and false positives and negatives, particularly effective in imbalanced datasets.

Results and Analysis

We evaluate LAET’s task-specific performance across Textual Analysis (TA), Forecasting (FO), and Risk Management (RM), benchmarking against state-of-the-art models. Additionally, we analyze model representations to assess their effectiveness for financial NLP tasks. (1) Textual Analysis. As shown in Table II, LAET outperforms existing benchmarks across 11 datasets spanning six TA tasks. Despite using smaller models, LAET variants (M1,M2,M3\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3) consistently deliver top performance. Notably, M3\mathcal{M}_3 achieves 0.89 accuracy/F1 on FPB, 0.90 on FiQA, and a lowest RMSE of 0.18 on TSA. LAET also surpasses large models like GPT-4 and FinMA-7B on datasets such as FinanceES, Headlines, and FOMC, highlighting its robustness and efficiency. (2) Forecasting. Comparing Table IV reports results on BigData22, ACL18, and CIKM18. LAET models offer competitive performance despite their smaller size. For example, M3\mathcal{M}_3 achieves 0.59 accuracy and 0.45 F1 on CIKM18, with the highest MCC. M2\mathcal{M}_2 scores 0.53 accuracy and 0.44 F1 on ACL18. These results confirm that compact, well-adapted models can rival larger LLMs in FO tasks. Risk Management. Despite class imbalance in RM datasets, LAET demonstrates strong adaptability (Table III). M1\mathcal{M}_1 achieves 0.74 accuracy on German, M2\mathcal{M}_2 reaches 0.99 accuracy and 0.71 F1

ModelsFPB AccFPB F1FiQA. AccFiQA. F1TSA AccTSA F1TSA RMSEFin.ES AccFin.ES F1Head. AccHead. AvF1FOMC AccFOMC F1FinArg. AccFinArg. MiF1MultiF AccMultiF F1MultiF MiF1MA AccMA MiF1
General Purpose LLM
ChatGPT [13]0.780.78-0.60.210.24-0.130.08-0.770.60.64--0.480.47---
GPT-4 [13]0.760.78-0.80.470.56-0.150.09-0.860.690.71--0.60.6---
Gemini [12]0.770.77-0.81--0.37---0.780.600.40-0.31-0.620.62-0.84
LLaMA2-7B [13]0.680.65-0.770.070.04-0.140.13-0.720.500.35-0.460.230.11--0.70
LLaMA2-70B [12]0.730.72-0.83--0.57---0.630.470.49-0.58-0.630.63-0.86
LLaMA3-8B [12]0.520.52-0.70--0.25---0.600.410.40-0.51-0.390.39-0.34
Few-shot LLM
GPT-4o [45]0.870.880.700.870.720.660.550.700.580.650.970.660.680.650.720.650.700.690.610.56
Deepseek V3 [46]0.830.830.700.850.810.810.570.110.110.650.960.550.490.590.660.990.990.580.720.65
Qwen2.5 72B [47]0.730.730.590.830.760.780.580.130.130.670.580.500.460.720.600.910.910.720.600.55
Financial LLM
FinMA-7B [12]0.880.88-0.84------0.98---------
FinMA-7B [11]0.860.86-0.79--0.80---0.970.470.49-0.27-0.140.14-0.45
FinGPT-7Bl [12]0.000.00-0.00--0.00---0.600.000.00-0.00-0.000.00-0.00
FinMA-30B [11]0.870.88-0.87------0.97---------
FinMA-ESB [13]0.830.83-0.850.850.86-0.110.11-0.960.550.49--0.990.99---
FinMA-ESS [13]0.730.73-0.830.860.86-0.130.13-0.580.500.46--0.910.91---
PEFT Baselines
M1\mathcal{M}_1 - LoRA [27]0.840.830.840.830.800.790.200.750.740.920.920.660.650.710.700.860.860.830.820.82
M2\mathcal{M}_2 - LoRA [27]0.850.850.840.840.800.790.220.700.700.920.920.660.660.720.720.860.860.880.840.84
M3\mathcal{M}_3 - LoRA [27]0.800.810.860.860.800.790.170.720.710.930.930.660.660.680.680.840.830.890.850.85
M1\mathcal{M}_1 - DoRA [28]0.850.840.850.840.810.810.200.770.760.940.940.660.660.730.720.870.870.840.830.83
M2\mathcal{M}_2 - DoRA [28]0.860.860.850.850.810.810.220.720.720.940.940.680.680.740.740.870.870.900.850.85
M3\mathcal{M}_3 - DoRA [28]0.860.860.860.860.810.810.170.740.730.940.940.680.680.700.700.850.840.910.870.87
M1\mathcal{M}_1 - AdaLoRA [29]0.810.800.810.800.770.760.230.700.680.900.900.630.620.680.660.820.820.790.780.78
M2\mathcal{M}_2 - AdaLoRA [29]0.820.820.820.820.770.760.250.670.670.910.910.640.630.700.700.830.820.840.810.81
M3\mathcal{M}_3 - AdaLoRA [29]0.810.820.830.830.770.760.240.690.690.910.910.650.640.660.660.810.800.850.820.82
Our Approach (LAET)
M1\mathcal{M}_1 - LAET (Ours)0.880.870.880.870.840.830.210.790.780.970.970.700.680.750.740.900.900.870.860.86
M2\mathcal{M}_2 - LAET (Ours)0.890.890.880.880.840.830.230.740.740.970.970.700.700.760.760.900.900.930.880.88
M3\mathcal{M}_3 - LAET (Ours)0.890.890.900.900.840.830.180.760.750.980.980.700.690.720.720.880.870.940.890.89

Table II. Performance comparison of LAET models based on Accuracy (Acc), F1, Average F1 (AvF1), MicroF1 (MiF1) against benchmarks on TA tasks. with the best values highlighted in bold. Unreported values are marked as (-). Highest performance is marked by Bold red and the second highest given by bold blue.

ModelsGerman AccGerman F1German MCCAustralian AccAustralian F1Australian MCCLendingClub AccLendingClub F1LendingClub MCCccf Accccf F1ccf MCCccfraud Accccfraud F1ccfraud MCCpolish Accpolish F1polish MCCtaiwan Acctaiwan F1taiwan MCCProtoSegno AccProtoSegno F1ProtoSegno MCCtravr. Acctravr. F1travr. MCC
ChatGPT [13]0.20.41-0.410.26----------------------
GPT-4 [13]0.550.513-0.740.75----------------------
Gemini [12]-0.520.00-0.260.00-0.650.19-0.96-0.01-0.900.00-0.860.14-0.950.00-0.950.00-0.000.00
LLaMA2-7B [13]0.610.60-------------------------
LLaMA2-70B [12]-0.170.00-0.410.00-0.170.00-0.170.00-0.170.00-0.170.00-0.170.00-0.170.00-0.170.00
LLaMA3-8B [12]-0.560.05-0.260.00-0.10-0.15-0.010.00-0.36-0.03-0.83-0.06-0.26-0.07-0.94-0.01-0.000.00
GPT-4o [45]0.610.730.530.860.840.510.620.510.650.590.650.730.510.580.610.580.510.670.730.660.620.450.570.520.600.580.72
Deepseek V3 [46]0.680.570.580.590.590.530.550.650.530.500.650.560.680.620.680.470.510.710.500.480.710.540.640.650.510.720.69
Qwen2.5 72B [47]0.600.650.660.590.570.570.650.580.450.450.500.510.620.590.660.460.480.650.540.710.510.660.630.470.650.650.51
FinMA-7B [12]-0.170.00--0.00-0.610.00-0.000.00-0.01-0.06-0.92-0.01-0.950.00-0.040.01-0.000.00
FinGPT-7B [12]-0.520.00-0.380.11-0.000.00-1.000.00-0.000.00-0.300.00-0.60-0.02-0.960.00-0.980.00
FinMA-ES-B [13]0.600.60-0.720.71----------------------
FinMA-ES-S [13]0.660.52-0.560.51----------------------
M1\mathcal{M}_1 - DoRA [28]0.680.560.000.810.820.680.930.930.870.950.930.000.890.880.000.900.870.000.910.890.000.940.900.010.930.940.00
M2\mathcal{M}_2 - DoRA [28]0.700.550.000.820.820.690.910.910.860.960.950.000.900.900.000.910.880.020.930.910.020.920.890.000.920.920.00
M3\mathcal{M}_3 - DoRA [28]0.680.550.020.800.800.680.920.920.870.940.940.000.880.910.000.900.880.000.930.930.000.930.900.000.950.920.00
M1\mathcal{M}_1 - AdaLoRA [29]0.700.560.000.830.790.680.940.920.800.950.930.000.890.920.000.890.890.010.920.920.010.920.900.000.930.910.00
M2\mathcal{M}_2 - AdaLoRA [29]0.710.540.000.800.820.660.910.940.860.950.930.000.890.920.000.890.890.010.920.920.010.920.900.000.930.910.00
M3\mathcal{M}_3 - AdaLoRA [29]0.680.550.010.820.800.680.950.940.890.950.950.000.900.890.000.890.860.010.900.910.000.940.900.010.940.920.00
M1\mathcal{M}_1 - LAET (Ours)0.740.610.000.880.880.770.980.980.931.001.000.000.960.950.500.970.950.000.970.960.000.970.950.000.990.980.00
M2\mathcal{M}_2 - LAET (Ours)0.740.620.000.870.860.730.980.980.931.001.000.000.950.950.020.970.940.000.970.960.000.970.950.000.990.980.00
M3\mathcal{M}_3 - LAET (Ours)0.740.630.000.870.870.740.980.980.941.001.000.000.960.950.310.950.920.000.970.960.000.970.960.000.990.980.00

Table III. Performance comparison of LAET models based on Accuracy(Acc), F1, and MCC, against benchmarks on RM tasks. with the best values highlighted in bold. Unreported values are marked as (-). Highest performance is marked by Bold red and the second highest given by bold blue.

ModelsBigData22 AccBigData22 F1BigData22 MCCACL18 AccACL18 F1ACL18 MCCCIKM18 AccCIKM18 F1CIKM18 MCC
General Purpose LLM
ChatGPT [13]0.53--0.0250.50-0.0050.55-0.005
GPT-4 [13]0.54-0.030.52-0.0200.57-0.020
Gemini [12]0.55-0.040.52-0.040.54-0.02
LLaMA2-7B [13]0.51-0.0300.51-0.0100.47--0.070
LLaMA2-70B [12]0.47-0.000.51-0.010.49--0.07
LLaMA3-8B [12]0.55-0.020.52-0.020.57-0.03
Few-shot LLM
GPT-4o [45]0.530.540.460.480.480.0190.450.490.50
Deepseek V3 [46]0.510.500.490.450.510.0170.500.510.49
Qwen2.5 72B [47]0.530.530.470.500.450.0250.510.520.48
Financial LLM
FinMA-7B [12]0.51-0.020.51-0.030.50-0.08
FinGPT-7B [12]0.45-0.000.49-0.000.42-0.00
FinMA-30B [11]0.47-0.040.49-0.000.43--0.05
FinMA-ESB [13]0.48-0.1000.49--0.0800.42--0.040
FinMA-ESS [13]0.57-0.1100.50--0.0100.55--0.040
PEFT Baselines
M₁ - LoRA [27]0.480.4900.000.430.380.000.490.380.00
M₂ - LoRA [27]0.510.530.010.460.350.020.4960.3720.000
M₃ - LoRA [27]0.4800.4600.0070.4770.3880.0030.5010.3620.019
M₁ - DoRA [28]0.5100.3570.0000.4660.3510.0150.5100.3800.000
M₂ - DoRA [28]0.4820.3550.0000.4580.3800.0000.4880.3900.000
M₃ - DoRA [28]0.4960.3340.0000.4790.3750.0000.5190.3800.000
M₁ - AdaLoRA [29]0.5080.3600.0150.4830.3610.0010.4870.3750.000
M₂ - AdaLoRA [29]0.4940.3540.0000.4660.3690.0140.4850.3950.000
M₃ - AdaLoRA [29]0.4860.3500.0000.4870.3590.0000.5000.3740.016
Our Approach (LAET)
M₁ - LAET (Ours)0.5500.4000.0000.5200.4200.0170.5800.4300.000
M₂ - LAET (Ours)0.5600.4600.0600.5500.5200.0970.5500.5500.105
M₃ - LAET (Ours)0.5700.5500.1000.5300.5100.0000.5800.5600.124

Table IV. Performance Comparison of LAET models on FO benchmarks. Highest performance is marked by Bold red and the second highest given by bold blue.

on Polish, while M3\mathcal{M}_3 excels on LendingClub and Taiwan. LAET balances efficiency and performance, outperforming larger models without high compute costs.

Observations: Our experiments provided the following key insights: (a) In textual analysis tasks, the number of effective layers increases with the complexity of classification. For binary classification (e.g., "yes" or "no"), earlier layers capture the necessary information, whereas multiclass problems benefit from deeper layers. (b) In forecasting tasks, the nature of the data (e.g., time series) impacts LLM representations. Probing accuracy across all layers often remains around 50%, indicating difficulty in capturing meaningful context from tokenized sequences. (c) In highly imbalanced RM datasets (e.g., 99%-1% class splits), layers tend to become biased, with uniform probing accuracy across depths, making class differentiation challenging. (d) LAET improves efficiency by freezing non-informative layers, enabling up to 60% layer reduction in TA tasks without performance loss. (e) LAET consistently outperforms existing PEFT methods such as LoRA, DoRA, and AdaLoRA, and surpasses large-scale models like GPT-4o, Deepseek V3, and Qwen2.5 72B—despite having significantly fewer parameters.

Hidden State Representation Analysis: Unlike BERT-based models, which have a dedicated classifier token (like ‘[CLS]’), LLMs do not have a specific token for classification. To address this, we analyze hidden representations using three strategies: summing all tokens’ representations, averaging them, and using the last token’s hidden state. Our

Domain-Agnostic LAET: Table VI presents results on two benchmarks: the ETHICS dataset for moral reasoning and the Medical Tweets dataset for domain-specific classification. While GPT-4o provides a strong baseline, LAET consistently surpasses GPT-4o, base models, and PEFT variants, demonstrating its robustness and applicability beyond finance.

ModelEthics AccEthics F1Medical AccMedical F1
GPT-4o [45]0.670.660.610.62
Base Models
M1\mathcal{M}_10.470.450.530.55
M2\mathcal{M}_20.530.540.480.50
M3\mathcal{M}_30.560.560.520.53
PEFT Baselines
M1\mathcal{M}_1-LoRA [27]0.610.610.580.60
M2\mathcal{M}_2-LoRA [27]0.640.660.690.67
M3\mathcal{M}_3-LoRA [27]0.640.650.670.69
M1\mathcal{M}_1-DoRA [28]0.620.620.600.59
M2\mathcal{M}_2-DoRA [28]0.650.630.700.69
M3\mathcal{M}_3-DoRA [28]0.660.660.720.71
M1\mathcal{M}_1-AdaLoRA [29]0.610.610.580.60
M2\mathcal{M}_2-AdaLoRA [29]0.640.660.690.67
M3\mathcal{M}_3-AdaLoRA [29]0.640.650.670.69
Our Approach (LAET)
M1\mathcal{M}_1-LAET (Ours)0.680.670.740.76
M2\mathcal{M}_2-LAET (Ours)0.700.710.780.78
M3\mathcal{M}_3-LAET (Ours)0.760.740.840.86

Table VI. Performance comparison across Ethics and Medical data. Best values in red bold, second-best in blue bold.

Impact of Margin Parameter (α\alpha, β\beta): We have examined the effect of the parameters α\alpha and β\beta on layer selection and optimization. Our findings (showed in Table V) indicate that the chosen values for α\alpha and β\beta yielded the most optimal trade-off between parameter efficiency and accuracy.

α\alphaβ\betaLayers# of ParametersAccF1
0.30.31517829390110.850.84
0.30.71821226960670.880.87
0.70.71821226960670.880.87
0.70.31821226960670.870.86
0.50.51720094437150.890.88

Table V. Performance comparison of different model configurations with varying α\alpha and β\beta values, number of layers, and parameters. Accuracy (Acc) and F1-score (F1) highlight the best-performing configuration (α=0.5\alpha= 0.5, β=0.5\beta= 0.5) in bold.

Discussion

Our experiments demonstrate that lightweight LLMs, such as M1\mathcal{M}_1, M2\mathcal{M}_2, and M3\mathcal{M}_3, achieve competitive performance in financial tasks like textual analysis, forecasting, and risk management, often outperforming larger models like GPT-4. Layer-wise adaptation and fine-tuning proved effective, allowing up to 60% of layers to be frozen in textual tasks without performance loss, significantly improving efficiency. However, forecasting tasks revealed limitations, with LLMs struggling to extract meaningful patterns from time-series data, while risk management tasks highlighted challenges with class imbalance. The last token’s hidden state emerged as the most effective representation for classification, leveraging the sequential attention mechanism of LLMs. These findings underscore the potential of lightweight models for financial NLP tasks while identifying areas for further improvement, such as hybrid approaches for forecasting and techniques to address class imbalance.

Limitations: Our study primarily focused on single-word classification using lightweight LLMs, demonstrating their effectiveness in financial tasks like textual analysis and classification. However, the approach could be extended to a wider

Ablation Study

Layer Selection formulation: One of the most common layer selection criteria is based on standard deviation. Figure 4 shows the comparison of layer selection based on the first standard deviation and our proposed method across four datasets. The standard deviation-based approach selects more layers, while the proposed method selects fewer. Despite using fewer layers, the proposed method achieves similar or better accuracy, demonstrating its efficiency. This suggests that selecting layers based on the proposed method leads to a more compact model without compromising performance.

Figure 4. The 1st Std method selects 25–30 layers, while the proposed method selects 15–22, reducing layers by 20–40% while maintaining accuracy above 0.9, proving its efficiency.

results show that the last token provides the best probing accuracy, as it captures the most relevant information due to the model’s sequential attention mechanism. This performance is visualized in Figure 3. range of tasks, such as Named Entity Recognition (NER), Question Answering (QA), and Summarization, through more comprehensive pipelines like LAET. The lack of large annotated datasets limited our ability to conduct experiments on a larger scale and explore these tasks in-depth. Moreover, while lightweight models performed well, larger models with more parameters may offer improved task-specific performance, especially with better data and task adaptation strategies.

Figure 3. Layer-wise probing: We evaluated three probing strategies: Last Token (LT), Sum of All Tokens (SaT), and Average of All Tokens (AvT). The results indicate that LT consistently outperforms both SaT and AvT in accuracy, making it the most reliable indicator of layer effectiveness. This observation motivates our choice of the LT strategy in the LAET implementation.

Conclusion

In this study, we have proposed a novel strategy LAET, that optimize models by layer wise probing. We implemented the pipeline for specific financial NLP tasks, demonstrating that smaller models like can achieve competitive performance across textual analysis, forecasting, and risk management tasks. Our layer-wise adaptation and fine-tuning strategy proved highly efficient, enabling significant computational savings by freezing ineffective layers without sacrificing accuracy. While these models excelled in textual analysis and risk management, forecasting tasks revealed limitations, suggesting the need for hybrid approaches to better handle time-series data. Additionally, the last token’s hidden state emerged as the most effective representation for classification tasks, leveraging the sequential nature of LLMs. These findings highlight the potential of lightweight models for financial applications, offering a balance between performance and efficiency. Future work could focus on addressing challenges in forecasting and class imbalance, as well as exploring hybrid architectures to further enhance model capabilities in financial NLP tasks.

Future Works: Future research will focus on extending the application of lightweight LLMs to a broader range of financial tasks, including NER, QA, and Summarization, leveraging frameworks like the LAET pipeline. Additionally, addressing the limitations observed in time-series forecasting tasks, such as improving the model’s ability to extract meaningful patterns from temporal data, will be a priority. We also aim to explore hybrid model approaches to enhance forecasting accuracy and develop techniques to tackle class imbalance in risk management tasks. Furthermore, expanding experiments on larger-scale annotated datasets will help assess the scalability and generalizability of our models across various domains.

Ethics Statement

Our research on financial sentiment analysis using lightweight LLMs adheres to ethical standards, ensuring fairness, transparency, and integrity. We used publicly available, anonymized datasets to protect privacy and avoided the use of proprietary or sensitive financial data without authorization. While advancing AI technology for financial decision-making, we acknowledge the potential risks and are committed to addressing any biases or limitations in our models, promoting responsible and fair deployment in real-world applications.

References

[1] D. Araci, “Finbert: Financial sentiment analysis with pretrained language models,” arXiv preprint arXiv:1908.10063, 2019.

[2] Q. Xie, W. Han, Y. Lai, M. Peng, and J. Huang, “The wall street neo-phyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges,” arXiv preprint arXiv:2304.05351, 2023.

[3] W. Han, B. Zhang, Q. Xie, M. Peng, Y. Lai, and J. Huang, “Select and trade: Towards unified pair trading with hierarchical reinforcement learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 4123–4134.

[4] A. Lopez-Lira and Y. Tang, “Can chatgpt forecast stock price movements? return predictability and large language models,” arXiv preprint arXiv:2304.07619, 2023.

[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.

References

  1. [1]D. Araci, “Finbert: Financial sentiment analysis with pretrained language models,” arXiv preprint arXiv:1908.10063, 2019.arxiv.org/abs/1908.10063
  2. [2]Q. Xie, W. Han, Y. Lai, M. Peng, and J. Huang, “The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges,” arXiv preprint arXiv:2304.05351, 2023.arxiv.org/abs/2304.05351
  3. [3]W. Han, B. Zhang, Q. Xie, M. Peng, Y. Lai, and J. Huang, “Select and trade: Towards unified pair trading with hierarchical reinforcement learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 4123–4134.DOI
  4. [4]A. Lopez-Lira and Y. Tang, “Can chatgpt forecast stock price movements? return predictability and large language models,” arXiv preprint arXiv:2304.07619, 2023.
  5. [5]T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.arxiv.org/abs/2005.14165
  6. [6]S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023.arxiv.org/abs/2303.12712
  7. [7]Y. Yang, M. C. S. Uy, and A. Huang, “Finbert: A pretrained language model for financial communications,” arXiv preprint arXiv:2006.08097, 2020.arxiv.org/abs/2006.08097
  8. [8]R. S. Shah, K. Chawla, D. Eidnani, A. Shah, W. Du, S. Chava, N. Raman, C. Smiley, J. Chen, and D. Yang, “When flue meets flang: Benchmarks and large pre-trained language model for financial domain,” arXiv preprint arXiv:2211.00083, 2022.arxiv.org/abs/2211.00083
  9. [9]S. Wu, O. Irsoy, S. Lu, V. Dabrovolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “Bloomberggpt: A large language model for finance,” arXiv preprint arXiv:2303.17564, 2023.arxiv.org/abs/2303.17564
  10. [10]H. Yang, X.-Y. Liu, and C. D. Wang, “FinGPT: Open-source financial large language models,” arXiv preprint arXiv:2306.06031, 2023.arxiv.org/abs/2306.06031
  11. [11]Q. Xie, W. Han, X. Zhang, Y. Lai, M. Peng, A. Lopez-Lira, and J. Huang, “Finma: A large language model, instruction data and evaluation benchmark for finance,” arXiv preprint arXiv:2306.05443, 2023.arxiv.org/abs/2306.05443
  12. [12]Q. Xie, W. Han, Z. Chen, R. Xiang, X. Zhang, Y. He, M. Xiao, D. Li, Y. Dai, D. Feng et al., “The finben: An holistic financial benchmark for large language models,” arXiv preprint arXiv:2402.12659, 2024.arxiv.org/abs/2402.12659
  13. [13]X. Zhang, R. Xiang, C. Yuan, D. Feng, W. Han, A. Lopez-Lira, X.-Y. Liu, M. Qiu, S. Ananiadou, M. Peng et al., “Dólares or dollars? unraveling the bilingual prowess of financial llms between spanish and english,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6236–6246.arxiv.org/abs/2402.07405
  14. [14]M. Jin, Q. Yu, J. Huang, Q. Zeng, Z. Wang, W. Hua, H. Zhao, K. Mei, Y. Meng, K. Ding et al., “Exploring concept depth: How large language models acquire knowledge at different layers?” arXiv preprint arXiv:2404.07066, 2024.arxiv.org/abs/2404.07066
  15. [15]A. Gromov, K. Tirumala, H. Shapourian, P. Glorioso, and D. A. Roberts, “The unreasonable ineffectiveness of the deeper layers, 2024,” URL https://arxiv.org/abs/2403.17887.
  16. [16]S. Fan, X. Jiang, X. Li, X. Meng, P. Han, S. Shang, A. Sun, Y. Wang, and Z. Wang, “Not all layers of llms are necessary during inference,” arXiv preprint arXiv:2403.02181, 2024.arxiv.org/abs/2403.02181
  17. [17]M. Abdin, J. Aneja, H. Awadalla, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl et al., “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219, 2024.arxiv.org/abs/2404.14219
  18. [18]G. Team, T. Mesnard, C. Hardin, R. Dadashi, M. Shuprajit, S. Pathak, L. Sifre, M. Riviere, M. Sale, J. Love et al., “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295, 2024.
  19. [19]A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  20. [20]Y. Soun, J. Yoo, M. Cho, J. Jeon, and U. Kang, “Accurate stock movement prediction with self-supervised learning from sparse noisy tweets,” in 2022 IEEE International Conference on Big Data (Big Data). IEEE, 2022, pp. 1691–1700.DOI
  21. [21]A. Azaria and T. Mitchell, “The internal state of an llm knows when it’s lying,” arXiv preprint arXiv:2304.13734, 2023.arxiv.org/abs/2304.13734
  22. [22]M. Geva, J. Bastings, K. Filippova, and A. Globerson, “Dissecting recall of factual associations in auto-regressive language models,” arXiv preprint arXiv:2304.14767, 2023.arxiv.org/abs/2304.14767
  23. [23]H. Zhao, F. Yang, H. Lakkaraju, and M. Du, “Opening the black box of large language models: Two views on holistic interpretability,” arXiv e-prints, pp. arXiv–2402, 2024.
  24. [24]X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen, “Shortgpt: Layers in large language models are more redundant than you expect,” arXiv preprint arXiv:2403.03853, 2024.
  25. [25]G. Alain, “Understanding intermediate layers using linear classifier probes,” arXiv preprint arXiv:1610.01644, 2016.arxiv.org/abs/1610.01644
  26. [26]N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attarian, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International conference on machine learning. PMLR, 2019, pp. 2790–2799.arxiv.org/abs/1902.00751
  27. [27]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022.arxiv.org/abs/2106.09685
  28. [28]S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,” in Forty-fifth International Conference on Machine Learning, 2024.
  29. [29]Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao, “Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,” arXiv preprint arXiv:2303.10512, 2023.arxiv.org/abs/2303.10512
  30. [30]P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala, “Good debt or bad debt: Detecting semantic orientations in economic texts,” Journal of the Association for Information Science and Technology, vol. 65, no. 4, pp. 782–796, 2014.arxiv.org/abs/1307.5336
  31. [31]M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur, “WWW’18 open challenge: financial opinion mining and question answering,” in Companion proceedings of the the web conference 2018, 2018, pp. 1941–1942.
  32. [32]K. Cortis, A. Freitas, T. Daudert, M. Huerlmann, M. Zarrouk, S. Handschuh, and B. Davis, “Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news,” in Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), 2017, pp. 519–535.
  33. [33]R. Pan, J. A. García-Díaz, F. Garcia-Sanchez, and R. Valencia-García, “Evaluation of transformer models for financial targeted sentiment analysis in spanish,” PeerJ Computer Science, vol. 9, p. e1377, 2023.DOI
  34. [34]N. L.-T. Nguyen, “Abcd team at financEs 2023: An unified generative framework for the financial targeted sentiment analysis in spanish,” 2023.
  35. [35]A. Sinha and T. Khandait, “Impact of news on the commodity market: Dataset and results,” in Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2. Springer, 2021, pp. 589–601.arxiv.org/abs/2009.04202
  36. [36]A. Shah, S. Paturi, and S. Chava, “Trillion dollar words: A new financial dataset, task & market analysis,” arXiv preprint arXiv:2305.09722, 2023.
  37. [37]E. Sy, T.-C. Peng, S.-H. Huang, H.-Y. Lin, and Y.-C. Chang, “Fine-grained argument understanding with bert ensemble techniques: A deep dive into financial sentiment analysis,” in Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), 2023, pp. 242–249.
  38. [38]R. Jørgensen, O. Brandt, M. Hartmann, X. Dai, C. Igel, and D. Elliott, “Multifin: A dataset for multilingual financial nlp,” in Findings of the Association for Computational Linguistics: EACL 2023, 2023, pp. 894–909.
  39. [39]L. Yang, E. M. Kenny, T. L. J. Ng, Y. Yang, B. Smyth, and R. Dong, “Generating plausible counterfactual explanations for deep transformers in financial text classification,” arXiv preprint arXiv:2010.12512, 2020.arxiv.org/abs/2010.12512
  40. [40]Y. Xu and S. B. Cohen, “Stock movement prediction from tweets and historical prices,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 1970–1979.DOI
  41. [41]H. Wu, W. Zhang, W. Shen, and J. Wang, “Hybrid deep sequential modeling for social text-driven stock prediction,” in Proceedings of the 27th ACM international conference on information and knowledge management, 2018, pp. 1627–1630.DOI
  42. [42]H. Hofmann, “Statlog (german credit data). uci machine learning repository (1994).”
  43. [43]R. Quinlan, “Statlog (australian credit approval),” UCI Machine Learning Repository, vol. 10, p. CS9012, 1987.DOI
  44. [44]D. Feng, Y. Dai, J. Huang, Y. Zhang, Q. Xie, W. Han, Z. Chen, A. Lopez-Lira, and H. Wang, “Empowering many, biasing a few: Generalist credit scoring through large language models,” arXiv preprint arXiv:2310.00566, 2023.arxiv.org/abs/2310.00566
  45. [45]J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
  46. [46]A. Liu, B. Feng, B. Xue, B. Wang, W. Yu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
  47. [47]B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al., “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024.arxiv.org/abs/2409.12186

Paper details

vlnc:W_qwzbn88c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents