betaFeedback

WorksComputer Science

Leakage-aware calibrated decision-level fusion for mental-health text classification

Abstract

Decision-level fusion benefits from complementary errors, calibrated probabilities, and leakage-aware evaluation. We evaluated a heterogeneous transformer fusion framework on 262,276 proxy-labeled Reddit records covering anxiety, depression, schizophrenia, and suicidal ideation. In a stratified random split, MinHash-LSH identified 29.1% of test records as lexical near-duplicates of training data. Augmentation-lineage auditing found 50.2% lineage overlap with training, including 30.0% of records retained after lexical filtering. We therefore used lineage-disjoint partitions. Twelve encoders, comprising three seeds each of Bio-ClinicalBERT, MentalBERT, MentalRoBERTa, and RoBERTa-base, underwent cross-fitted validation for calibration and ensemble selection. A class-specific Complementarity Index (CI), defined as the Jaccard distance between members' correctly classified validation sets, was assessed against independent fusion gain. CI correlated with gain for depression (r=0.50) and suicidal ideation (r=0.68), but not anxiety; results for schizophrenia were mixed. The selected MentalBERT, MentalRoBERTa, and RoBERTa-base triad achieved macro-F1 0.844 [0.823, 0.863], MCC 0.779, and ECE 0.0033 after post-fusion recalibration, improving macro-F1 by 0.021 [0.007, 0.037] over the validation-selected best single encoder. Integrated-Gradients attribution diversity was 2.3 times greater in fusion-rescued cases. Removing divergence-driving words destroyed 58% of rescues, versus 12% under random removal. These findings show that lineage auditing detects dependencies missed by lexical filtering, while class-specific complementarity and post-fusion calibration reveal ensemble behavior obscured by aggregate accuracy.

Introduction

Classifying mental-health-related text from social media remains a challenging supervised learning task. Public posts from self-selected mental-health communities on Reddit provide large volumes of labeled text that would be difficult to obtain through clinical recruitment. However, these labels are weak because posts are generally labeled according to the subreddit in which they appear. Therefore, the subreddit serves only as a proxy for a mental-health condition rather than as evidence of a clinically validated diagnosis. Using subreddits as proxy labels introduces three confounding factors that directly affect model evaluation. First, self-selection creates sampling bias because the data mainly represent users who are willing to discuss their mental health publicly [13]. Second, the labels may be noisy because users can post in a mental-health subreddit without meeting the formal diagnostic criteria for the associated condition. Third, many conditions share similar linguistic features. For example, depression, anxiety, and suicidal ideation may involve common affective, cognitive, and self-referential expressions [36, 42]. Therefore, a single-label classifier may learn dataset-specific patterns rather than reliable indicators of a particular condition, making closely related classes difficult to distinguish.

Obtaining clinically confirmed labels at a comparable scale is generally impractical because of privacy, consent, and data-access constraints. Despite their limitations, large text corpora have supported the development of domain-adapted language models. Bio-ClinicalBERT was pretrained on clinical notes [3], whereas MentalBERT was adapted specifically to mental-health-related language [18]. This setting therefore provides a demanding testbed for assessing whether decision-level fusion can benefit from complementary model errors while maintaining well-calibrated confidence estimates.

Three limitations in current ensemble practice reduce the reliability of fusion in this setting. First, ensembles are most effective when their members make complementary errors. However, diversity is usually measured after training using global statistics, such as disagreement, correlation, or the double-fault measure [21]. These measures summarize overall diversity but do not identify beforehand which encoder combinations can improve performance or which classes will benefit from fusion. Second, averaging the outputs of individually calibrated models does not guarantee that the resulting ensemble will also be calibrated. Nevertheless, ensemble confidence scores are often treated as reliable without directly evaluating or calibrating the final output. Third, social-media datasets may contain duplicate or near-duplicate posts across the training and test sets. Such contamination can inflate performance and provide a misleading view of generalization. A reliable fusion method for noisy, proxy-labeled text should therefore address all three limitations: estimate the benefit of fusion before aggregation, calibrate the final ensemble output, and confirm that the evaluation splits are free from data leakage.

In this paper, these limitations have been addressed within a single pipeline. The system operates as decision-level fusion within the standard feature-, score-, and decision-level taxonomy of information fusion [20]. Each base classifier produces a calibrated probability vector, and the fusion mechanism is the combination of those vectors at the decision layer. The proposed pipeline consists of three components. First, a split-integrity audit combines MinHash–Locality-Sensitive Hashing to detect near-duplicate posts with a lineage audit that links each augmented record to its original post. The data are then split at the lineage level, ensuring that an original post and its derived versions cannot appear in different training and evaluation sets. Second, a class-specific complementarity index (CI) is introduced as a pre-fusion criterion. Using predictions from the validation set, the CI estimates how well candidate encoders make complementary correct predictions for each class. Third, a two-stage calibration procedure first applies temperature scaling to each ensemble member and then recalibrates the fused output. Together, these components address three important concerns in decision-level fusion: data leakage, limited transparency in ensemble selection, and unreliable confidence estimates. The resulting framework supports accurate and better-calibrated predictions while providing a clearer basis for selecting the models included in the ensemble.

The evaluation uses 262,276 records from four proxy mental-health categories: anxiety, depression, schizophrenia, and suicidal ideation. The corpus includes 149,195 original posts and 113,081 augmented records. A stratified random split shows substantial contamination: MinHash–LSH flags 29.1% of the test set. At the same time, the lineage audit reveals that 50.2% of the test records share an augmentation lineage group with a training record, either as derivatives of training posts or as originals whose derivatives lie in training. This includes 30.0% of the records that the lexical audit retains. Therefore, a lineage-disjoint split is constructed in which the validation and test sets contain only original posts.

Twelve encoders (three seeds from each of four model families) are fine-tuned. All model selection, calibration, and fusion decisions are made on the validation data under a cross-fitted protocol; the test set is reserved for final reporting. The selected ensemble combines MentalBERT, MentalRoBERTa, and RoBERTa-base and is recalibrated using T∗=0.886T^{*} = 0.886. It achieves a macro-F1 of 0.844 [0.823, 0.863], an MCC of 0.779, and an ECE of 0.0033. It improves macro-F1 by 0.021 over the validation-selected best single encoder, with significant permutation and multiplicity-corrected McNemar tests.

Dependence-aware analyses show that CI predicts fusion gains for depression and suicidal ideation, but not for anxiety. The ensemble also transfers zero-shot to an author-disjoint binary dataset from Reddit. In addition, integrated Gradients and counterfactual word removal show how disagreement among the models supports successful fusion. Although evaluated on mental-health text, the framework can be applied to other multi-encoder classification tasks requiring leakage-aware evaluation, calibrated confidence, and transparent ensemble selection.

Contributions. In this paper, we present the following contributions:

  1. A leakage-aware decision-level fusion framework is proposed for proxy-labeled mental-health text classification. The framework integrates a two-layer split-integrity audit (lexical near-duplicate detection and augmentation-lineage tracing), validation-driven encoder selection with a class-specific complementarity diagnostic, and calibration-aware probability aggregation within a unified pipeline. Applied to a stratified random split of the corpus, the lineage audit shows that half of the test records share an augmentation lineage group with a training record and that lexical filtering alone leaves 30.0% of the retained records so linked; the lineage-disjoint split removes augmentation-lineage contamination and reduces the lexical near-duplicate rate to 0.4% and the semantic (cosine ≥0.90\ge0.90) near-duplicate rate to 0.8%; contamination through shared authors, threads, or time could not be audited and is stated as an open limitation.

  1. A class-specific Complementarity Index (CI), equal to the Jaccard distance between the sets of class-cc validation examples each encoder classifies correctly, is introduced as a validation-computable predictor of where fusion will help, and is evaluated against test-set fusion gain with statistics that respect the dependence among pair–class observations (cluster bootstraps over pairs and over encoders, a mixed-effects model with class effects and a symmetric multi-membership random intercept per encoder, encoder-level bootstrap intervals within each class, and leave-one-encoder-out prediction). The association is class-dependent: significant within depression (r=0.50r = 0.50) and suicidal ideation (r=0.68r = 0.68), absent for anxiety, and uninformative for a 75-record class.

  1. A two-stage calibration strategy is developed for decision-level fusion. Individual encoders are first temperature-scaled separately, and the aggregated ensemble output is then recalibrated post hoc, improving probabilistic reliability without sacrificing predictive performance. The direction of the post-hoc correction is shown to be domain-dependent: the fused output of individually calibrated members is under-confident in domain (T⋆<1T^{\star} < 1 in every validation fold) and over-confident under external transfer (T⋆>1T^{\star} > 1).

  1. A compact heterogeneous three-member ensemble composed of MentalBERT, Mental-RoBERTa, and RoBERTa-base, selected under a cross-fitted validation protocol, achieves macro-F1 = 0.844 [0.823, 0.863], MCC = 0.779, and ECE = 0.0033 on the lineage-disjoint test set, exceeding the validation-selected best single encoder by +0.021 macro-F1 (paired bootstrap [+0.007, +0.037]; permutation p=0.0035p = 0.0035; McNemar p=4.6×10−4p = 4.6 \times10^{-4} after Holm correction over the 67 comparisons performed); the 75-post schizophrenia class contributes 71% of that gain, and the improvement over the three larger classes alone is +0.008 [+0.003, +0.014] and coming within 0.004 of a twelve-model ensemble with one quarter as many forward passes per prediction (three encoders of the same size instead of twelve; wall-clock inference was not measured). Seed-equalized results (three seeds per family), the post-selection spread of the cross-fitted protocol (0.842 ±\pm 0.004), and zero-shot transfer to an author-disjoint, same-platform, two-class external corpus are reported.

  1. An attribution-diversity analysis with an interventional test is conducted on the deployed checkpoints. Integrated-Gradients attributions are aligned at word level so that WordPiece and byte-pair tokenizers can be compared; fusion-rescued cases show 2.3×\times higher inter-model attribution diversity than all-correct cases, the effect persists between member pairs that predict the same class, and removing the five words on which the correct members diverge from the incorrect member destroys 58% of rescues versus 12% for random words.

The remainder of the paper is organized as follows. Section 2 surveys related work with a comparative table. Section 3 formalizes the theoretical foundations, including the CI and the post-hoc calibration of aggregated predictions. Section 4 describes the corpus, split-integrity audit, lineage-disjoint split, fine-tuning setup, and evaluation protocol. Section 5 presents the results and statistical tests. Section 6 presents the attribution-diversity analysis and the counterfactual intervention. Section 7 discusses implications and limitations. Section 8 concludes the paper.

Related Work

Recent work in mental-health natural language processing has moved from domain-adapted language models to heterogeneous transformer ensembles and strategies for evaluating their predictions. However, greater performance with model fusion does not prove that the constituent models provide complementary information, that aggregated probabilities remain properly calibrated, or that apparent improvements hold under stringent restrictions on data dependence. These challenges are especially relevant for social-media datasets with augmented or closely related observations. We cover four aspects directly related to reliable decision-level fusion: domain-adapted language models, classifier complementarity and ensemble learning, probability calibration and leakage-aware evaluation, and explanation of ensemble complementarity.

Pretraining on domain-specific data is necessary for transformer-based mental-health NLP. Alsentzer et al. [3] created variants of ClinicalBERT trained on clinical notes and showed that adapting to clinical language can improve performance on medical NLP tasks. Clinical notes, however, differ greatly from self-authored social-media posts in terms of terminology, writing style, and communicative context. Ji et al. [18] tackled this distinction more directly with MentalBERT and MentalRoBERTa, adapting them to mental-health-related social-media text. Their results suggest that further pretraining on language closer to the target domain can boost downstream mental-health NLP performance. Mezzi et al. [29] applied BERT to mental-health intent recognition for Arabic-speaking patients using MINI-based interview data collected at the Military Hospital of Tunis. More recently, Sánchez Rodríguez et al. [37] compared BERT and MentalBERT to classify Reddit posts from subreddits for anxiety, depression, bipolar disorder, and borderline personality disorder. MentalBERT showed a slight but persistent advantage over BERT, and Local Interpretable Model-agnostic Explanations (LIME) were used to identify language aspects associated with the predictions. These experiments show the importance of domain-adapted representations, but differences in encoder performance do not prove whether diverse representations provide complementary information that could be used through fusion. There is a large body of work studying the relationship between classifier diversity and ensemble performance. Measures such as disagreement, correlation, Q-statistic, and double fault were studied by Shipp and Kuncheva [38] and Kuncheva and Whitaker [21]. Brown et al. [5] also stressed that diversity is only relevant to prediction accuracy: a disagreement among classifiers does not necessarily mean their ensemble will outperform the individual models. However, heterogeneous transformer ensembles have recently been shown to be promising for mental-health text analysis. Kasap et al. [19] applied DistilBERT, RoBERTa, Mental-BERT, Mental-RoBERTa, and DeBERTa for multi-label categorization of depression-related emotions and experimented with stacking, hard-voting, and soft-voting. Islam et al. [16] proposed a heterogeneous transformer ensemble for depression related emotion and severity classification based on soft voting weighted by validation performance and post-hoc LIME explanations. The models used for the ensemble are based on DistilBERT, ALBERT, XLM-RoBERTa, and DeBERTa. These experiments support heterogeneous transformer fusion in mental-health NLP, but leave open a more specific question: can differences between candidate classifiers reveal combinations that are likely to benefit from fusion? Common classifier-diversity metrics characterize relationships among model outputs, but they do not directly measure class-specific non-overlap among correctly classified validation examples. This leads us to separate generic classifier diversity from correctness complementarity and check whether such complementarity is related to subsequent fusion gain.

Predictive accuracy and probabilistic trustworthiness are separate features of classifiers. Niculescu-Mizil and Caruana [31] showed that strong classifiers are not necessarily reliable probability estimators. Guo et al. [10] showed that neural networks can be severely poorly calibrated even with high classification accuracy and that temperature scaling is a simple and effective post-hoc calibration method. Calibration is especially important for decision-level fusion since probability-based ensembles act directly on the confidence estimates generated by their members. Averaging calibrated member probabilities does not necessarily produce a calibrated ensemble. The fused output therefore requires separate evaluation and, when needed, recalibration. Islam et al. [15] used validation performance to guide Bayesian probability fusion for structured clinical data, while Demirtaş Alpaslan et al. [2] combined calibrated confidence with explainability-related information to get sample-specific ensemble weights. Abdar et al. [1] also highlighted the larger significance of trustworthy uncertainty estimation in deep learning. Taken together, these findings support treating calibration of the fused predictive distribution as a separate challenge from calibrating the individual classifiers.

Evaluation also requires separation between training and evaluation observations. This is especially relevant when dealing with social-media information collected from a variety of public sources, because exact, near-duplicate, reposted, or altered versions of the same underlying post can be present in nominally distinct partitions. Harrigian et al. [13] identified dataset development, sampling, labeling, and evaluation as important methodological challenges in applying NLP to mental health, and Chancellor and De Choudhury [6] pointed out the relevance of data and methodological decisions for the interpretation of social-media-based mental-health predictions. Lee et al. [22] demonstrated that repeated training instances can encourage memorization and drive explicit deduplication and overlap auditing. But lexical similarity is one type of dependence. An augmented post can differ enough from its source to fall below a lexical-similarity criterion and still derive from the same underlying observation.

The effects of leaking from preprocessing have been recently proved directly in mental-health socialmedia classification. Hammad and Rababah [11] showed that applying segmentation and oversampling before the train–test split could include instances generated from the same original post being included in both the training and evaluation collections. The authors also replaced this technique with an original-post-level split and used oversampling only on training data, which lowered their reported macro-F1 from ∼0.96\sim0.96 to 0.670.67. We see that dependencies imposed before partitioning can considerably exaggerate perceived generalization performance. More generally, lexical deduplication alone may be insufficient to detect all relationships induced by data augmentation, since a modified text may be lexically distinct yet from the same underlying observation. This divergence justifies explicit treatment of augmentation lineage in addition to traditional near-duplicate detection in the evaluation of models on enhanced social-media corpora.

Explainable-AI approaches provide an additional view on model complementarity, as classifiers may reach the same or different results depending on different parts of the input. Integrated Gradients [39] gives attributions of the output of a differentiable model to the input features, and SHAP [26] provides feature-level explanations based on Shapley values. Sánchez Rodríguez et al. [37] applied LIME to BERT and MentalBERT to identify language patterns and their relation to predictions of four mental-health-related Reddit groups. However, the correctness and stability of such explanations remain essential methodological considerations [17]. Explainability has also been integrated into transformer ensembles. Islam et al. [16] utilized LIME on a validation-weighted ensemble of transformers for emotion and severity categorization of depression and assessed explanations by token deletion and stability analyses. Instead, explainability and calibrated confidence were included in ensemble weighting by Demirtaş Alpsalaz et al. [2]. Based on classifier complementarity, attribution methods offer a complementary perspective on whether models that produce distinct correct predictions also rely on diverse linguistic evidence. Hence, explainability can be utilized diagnostically to interrogate the mechanisms behind successful fusion, and counterfactual disturbance provides a stronger test of whether observed attribution discrepancies are functionally related to ensemble decisions. Existing work supports domain adaptation, heterogeneous transformer ensembles, probability calibration, leakage-aware evaluation, and post-hoc explanation. But these aspects have been extensively investigated independently. In this work, we integrate these elements into a leakage-aware, calibration-conscious decision-level fusion framework and ask whether class-specific correctness complementarity measured on validation data is associated with fusion gain on an independent test set. Table 1 compares the proposed framework with representative prior works in transformer-based mental-health classification.

StudyMulti-arch ensembleCalibr. reportedLeakage auditCI metricXAI linkPrimary focus
Alsentzer et al. [3]–––––Clinical-notes BERT embeddings
Ji et al. [18]–––––MentalBERT/MentalRoBERTa release
Tavchioski et al. [40]partial––––Reddit depression ensembles
Harrigian et al. [13]––partial––Critique of data practices
Chancellor & De Choudhury [6]–––––Methodological critique
Abdar et al. [1]–✓–––UQ in deep learning survey
This work✓✓✓✓✓Leakage-aware calibrated decision-level fusion

Table 1. Comparison of this work with representative prior studies on transformer-based mental-health text classification.

Theoretical Foundations

This section formalizes the building blocks used in § 4.

Problem setup

Let X\mathcal{X} denote a space of text sequences and Y={1,…,K}\mathcal{Y} = \{1,\ldots,K\} a finite label set; here K=4K = 4. A classifier is a map f:X→ΔK−1f:\mathcal{X}\to\Delta^{K-1} where ΔK−1\Delta^{K-1} is the probability simplex over Y\mathcal{Y}. Given MM trained classifiers f1,…,fMf_1,\ldots,f_M, a fusion rule produces a combined classifier fˉ:X→ΔK−1\bar{f}:\mathcal{X}\to\Delta^{K-1}. We target two objectives simultaneously: predictive accuracy, measured by macro-F1 and Matthews Correlation Coefficient (MCC [27]); and confidence calibration, measured by Expected Calibration Error (ECE [30]) and Brier score.

Figure 1. Overview of the leakage-aware, cross-fitted calibration and complementarity-guided decision-level fusion framework.

Temperature scaling

Let z=fθlogit(x)∈RKz = f_{\theta}^{\mathrm{logit}}(x) \in\mathbb{R}^{K} denote the pre-softmax logits of a fine-tuned transformer. The softmax probability is σ(z)k=exp⁡(zk)∑j=1Kexp⁡(zj)\sigma(z)_k = \frac{\exp(z_k)}{\sum_{j=1}^{K}\exp(z_j)}. Temperature scaling [10] replaces σ(z)\sigma(z) with

σT(z)k=exp⁡(zk/T)∑j=1Kexp⁡(zj/T),T∈R>0,(1)\sigma_T(z)_k = \frac{\exp(z_k/T)}{\sum_{j=1}^{K}\exp(z_j/T)}, \qquad T \in\mathbb{R}_{>0}, \tag*{(1)}

where TT is chosen on a held-out validation set to minimize the negative log-likelihood:

T∗=arg min⁡T>0  −∑(x,y)∈Dvallog⁡σT(fθlogit(x))y.(2)T^{*} = \underset{T>0}{\operatorname{arg\,min}}\;-\sum_{(x,y)\in\mathcal{D}_{\mathrm{val}}}\log\sigma_T\left(f_{\theta}^{\mathrm{logit}}(x)\right)_y. \tag*{(2)}

As a single-parameter, monotonic transform that leaves arg max⁡kσT(z)k\operatorname{arg\,max}_k \sigma_T(z)_k invariant, temperature scaling adjusts confidence without changing hard predictions. We apply Eq. (2) to each base model independently, on the validation fold reserved for member calibration (§ 4.4). Because σT(z)\sigma_T(z) depends on zz only through differences of logits, the temperature can be fitted from stored softmax outputs by taking z=log⁡σ(z)z = \log\sigma(z) up to an additive constant.

Candidate Fusion Rules

We evaluate both uniform probability averaging and validation-fitted per-class weighting as candidate decision-level fusion rules; the final selected system uses uniform averaging. We combine calibrated probabilities by a convex combination that is allowed to vary across classes:

pens(y=c∣x)=∑m=1Mαm,cσTm(fmlogit(x))c∑c′=1K∑m=1Mαm,c′σTm(fmlogit(x))c′.(3)p_{\mathrm{ens}}(y=c\mid x) = \frac{\sum_{m=1}^{M}\alpha_{m,c}\sigma_{T_m}\left(f_m^{\mathrm{logit}}(x)\right)_c}{\sum_{c'=1}^{K}\sum_{m=1}^{M}\alpha_{m,c'}\sigma_{T_m}\left(f_m^{\mathrm{logit}}(x)\right)_{c'}}. \tag*{(3)}

with per-class simplex constraints αm,c≥0\alpha_{m,c}\ge0 and ∑mαm,c=1\sum_m \alpha_{m,c}=1 for every cc. We fit {αm,c}\{\alpha_{m,c}\} on the validation set by coordinate-descent grid search on macro-F1. Uniform weights αm,c=1/M\alpha_{m,c}=1/M recover the arithmetic-mean ensemble. Because the weights of each class are constrained to the simplex, the rule has K(M−1)K(M-1) free parameters: 8 for a three-member ensemble and 44 for a twelve-member ensemble.

Complementarity Index (CI)

The CI is defined as a per-class measure of correctness complementarity between two trained classifiers. It is computed from validation-set predictions and used as a pre-fusion criterion for estimating the potential benefit of combining candidate encoders before probability aggregation.

For a class cc and two classifiers AA and BB, CI is defined as

CIc(A,B)=nc,onlyA+nc,onlyBncA∪B,(4)\mathrm{CI}_c(A,B)=\frac{n^{A}_{c,\mathrm{only}}+n^{B}_{c,\mathrm{only}}}{n^{A\cup B}_c}, \tag*{(4)}

where nc,onlyAn^{A}_{c,\mathrm{only}} denotes the number of class-cc validation examples correctly classified only by AA, nc,onlyBn^{B}_{c,\mathrm{only}} denotes the corresponding number for BB, and ncA∪Bn^{A\cup B}_c denotes the number of class-cc validation examples correctly classified by at least one of the two classifiers. When ncA∪B=0n^{A\cup B}_c=0, we set CIc(A,B)=0\mathrm{CI}_c(A,B)=0.

Equivalently, let

ScA={i:y^A(xi)=yi=c},ScB={i:y^B(xi)=yi=c}S^A_c=\{i:\hat{y}_A(x_i)=y_i=c\},\qquad S^B_c=\{i:\hat{y}_B(x_i)=y_i=c\}

be the sets of class-cc validation examples correctly classified by AA and BB, respectively. Then,

CIc(A,B)=∣ScA△ScB∣∣ScA∪ScB∣(5)\mathrm{CI}_c(A,B)=\frac{\lvert S^A_c\mathbin{\triangle}S^B_c\rvert}{\lvert S^A_c\cup S^B_c\rvert} \tag*{(5)}

Equation (5) is exactly the Jaccard distance between the two correctness sets, CIc(A,B)=1−∣ScA∩ScB∣/∣ScA∪ScB∣\mathrm{CI}_c(A,B)=1-\lvert S^A_c\cap S^B_c\rvert/\lvert S^A_c\cup S^B_c\rvert, with the convention CIc=0\mathrm{CI}_c=0 when the union is empty. The contribution claimed here is therefore not the distance itself but its application: it is computed per class, on correctness sets rather than raw predictions, from validation data alone, and evaluated against fusion gain measured on an independent sample. A higher CI indicates that the two classifiers correctly handle more non-overlapping examples of class cc, making their predictions more likely to provide useful complementary evidence under fusion. By construction, for a fixed union of correctly classified examples, CI increases with the non-overlap between the two correctness sets. Two properties limit what the index can express. It is computed from hard decisions and therefore carries no confidence information, although it anticipates the benefit of averaging probabilities; it also counts only examples of class cc that are classified correctly, so out-of-class false positives that also enter the class-cc F1 are outside its definition. Both are deliberate simplifications that keep the index computable from stored predictions alone, and both bound how much fusion gain it can explain.

For selecting ensembles with more than two members, candidate encoder sets are scored by averaging their pairwise CI values across all model pairs and classes:

CI(S)=1K(∣S∣2)∑c=1K∑{A,B}⊂SCIc(A,B),(6)\mathrm{CI}(S)=\frac{1}{K\binom{\lvert S\rvert}{2}}\sum_{c=1}^{K}\sum_{\{A,B\}\subset S}\mathrm{CI}_c(A,B), \tag*{(6)}

where SS denotes a candidate encoder set and KK is the number of classes. Candidate encoder sets are ranked by validation macro-F1, with CI(S)\mathrm{CI}(S) available as a tiebreaker among sets of comparable validation performance. § 5.3 reports how often that tiebreaker changes the selected ensemble under several tolerances; it is never decisive under the protocol used here, and the index is therefore evaluated as a predictor of fusion gain rather than as a selection mechanism. Ranking candidate sets by validation macro-F1 requires the candidate ensembles themselves to be formed, so the qualifier “before aggregation” applies to the computation of CI from member predictions, not to the ranking step. In § 5, CI computed on validation predictions is evaluated as a predictor of the class-wise fusion gain ΔF1,c\Delta F_{1,c} realized on the test set, so that the predictor and the quantity predicted are measured on disjoint samples. The realized gain of a pair {A,B}\{A,B\} on class cc is defined as the class-cc F1 of the uniform-mean ensemble minus the larger of the two members’ class-cc F1 values on the test set. This is a descriptive outcome variable, not a selection criterion: it is the quantity CI is asked to predict, and no model, weight, or temperature is chosen from it. Because the better member is identified per class on the same sample, it is the conservative choice, and § 5.2 reports the alternative in which the reference is fixed in advance from validation data.

Post-hoc ensemble calibration

Averaging temperature-scaled members does not, in general, preserve calibration: the resulting aggregate distribution can remain systematically under- or over-confident. We therefore fit a second-stage temperature T⋆T^\star directly to the aggregated prediction:

p⋆(y∣x)=σT⋆(log⁡pens(y∣x)),(7)p^\star(y \mid x) = \sigma_{T^\star}\bigl(\log p_{\mathrm{ens}}(y \mid x)\bigr), \tag*{(7)}

with T⋆T^\star minimizing the same negative log-likelihood in Eq. (2) over the validation fold reserved for post-hoc calibration. A value T⋆<1T^\star< 1 indicates that the ensemble of individually calibrated members is under-confident and is sharpened by the correction; T⋆>1T^\star> 1 indicates over-confidence. Both regimes are observed in § 5: T⋆<1T^\star< 1 in domain and T⋆>1T^\star> 1 under transfer to an external corpus.

Integrated Gradients on transformer inputs

Given an input embedding e(x)∈RL×de(x) \in\mathbb{R}^{L \times d} where LL is the sequence length, and a baseline e0e_0 (taken here as the zero embedding), the Integrated-Gradients attribution [39] of token ℓ\ell for predicted class y^\hat{y} is

IGℓ(x)=(e(x)ℓ−e0,ℓ)⋅∫01∂Fy^(e0+s(e(x)−e0))∂eℓ ds,(8)\mathrm{IG}_{\ell}(x) = \bigl(e(x)_{\ell} - e_{0,\ell}\bigr) \cdot\int_{0}^{1} \frac{\partial F_{\hat{y}}\bigl(e_0 + s(e(x) - e_0)\bigr)}{\partial e_{\ell}} \,\mathrm{d}s, \tag*{(8)}

where Fy^F_{\hat{y}} is the scalar logit for class y^\hat{y}. We approximate Eq. (8) by a Riemann sum with 20 interpolation points (Sec. 6) and reduce over the hidden dimension dd to obtain a per-token scalar aℓ(x)∈Ra_{\ell}(x) \in\mathbb{R}. Because the ensemble members use different tokenizers (WordPiece for the BERT-family encoders, byte-pair encoding for the RoBERTa-family encoders), token-level attribution vectors of different members are not aligned position by position. Token attributions are therefore summed over the sub-word tokens whose character offsets fall inside each whitespace-delimited word, yielding a word-level attribution vector aw(x)a^{w}(x) of common dimension for all members; inter-model comparisons in § 6 are made on these word-level vectors, restricted to words covered by every member within the 256-token window.

Proposed Pipeline

Corpus construction

The corpus was collected from five public collections, and its labels were mapped into four proxy categories: anxiety, depression, schizophrenia, and suicidal ideation. The sources were (i) the Kaggle Sentiment Analysis for Mental Health corpus, which contains statements labeled as Normal, Depression, Suicidal, Anxiety, Bi-Polar, Stress, or Personality Disorder [7]; (ii) the Dreaddit corpus of Reddit posts from five stress-related domains, including anxiety [41]; (iii) the Kaggle Suicide and Depression Detection corpus of 232,074 Reddit posts labeled as suicide or non-suicide, collected from r/SuicideWatch and r/teenagers [32]; (iv) the Kaggle Depression: Reddit Dataset (Cleaned), which is a cleaned corpus of labeled Reddit posts; and (v) the Reddit Mental Health Dataset (RMHD), which contains posts from 28 Reddit communities including 15 specific mental-health support groups such as r/Anxiety, r/depression, r/schizophrenia, and r/SuicideWatch [25]. Labels were harmonized before merging: suicide and suicidal ideation were mapped to suicidal ideation; stress and anxiety to anxiety; depression, depressive, bipolar, and mental_illness to depression; and schizophrenia to schizophrenia. Records not mapped to one of these four target categories were excluded. The merged corpus kept only entries that mapped to the four target categories. All records are markers of a self-declared ailment, not a clinical diagnosis. The texts were made lowercase, contractions were extended, and non-alphabetic characters were eliminated. Exact duplicates were dropped. The merged file did not retain source IDs, author identifiers, or timestamps, which has ramifications as detailed in § 7.5.

The merged file contains 262,276 records, of which 149,195 are original posts and 113,081 are augmented derivatives produced from original anxiety, depression, and schizophrenia posts by lexical augmentation (synonym substitution and random word deletion, in the manner of EDA [43]) to bring those classes to fixed target sizes (Table 2). Augmented records are marked by a source field, but the identity of the parent post was not recorded. The suicidal-ideation class was not augmented.

ClassOriginalAugmentedTotalGroup size (mean / median / max)
Anxiety5,21034,79040,0007.7 8 20
Depression16,20943,79160,0003.7 3 16
Schizophrenia50034,50035,00070.0 68 136
Suicidal127,2760127,2761.0 1 1
Total149,195113,081262,276

Table 2. Composition of the merged corpus by class and record type. Group size is the number of records (the original post plus its augmented derivatives) that share a parent post.

Split-integrity audit

Lexical near-duplicate audit A stratified random 80/10/10 split of the merged file (seed 42) yields 209,820 training, 26,228 validation, and 26,228 test records. This split was audited with a standard MinHash–LSH pipeline [4, 23]: each record was shingled into 5-token windows, encoded into a 128-permutation MinHash signature, and indexed with a Jaccard threshold of 0.8; a validation or test record was flagged if any training record exceeded the threshold. The audit flagged 7,622/26,228 (29.06%) test records and 7,623/26,228 (29.06%) validation records.

Table 2: Composition of the merged corpus by class and record type. Group size is the number of records (the original post plus its augmented derivatives) that share a parent post.

Lineage audit Lexical near-duplicate detection identifies derivatives that remain close to their parent in 5-gram space; it does not identify derivatives whose wording has drifted further, nor an original test post whose derivatives lie in the training set. To quantify both, the parent of every augmented record was recovered as the original post of the same class with the highest TF–IDF cosine similarity. Parent identity was inferred, not recorded: for each augmented record, the most similar original of the same class was taken as its parent. The first percentile of parent similarity was 0.87 (anxiety), 0.89 (depression), and 0.83 (schizophrenia), the median exceeded 0.98 in every class, and 9 of 113,081 records fell below 0.5. These figures show that almost every derivative has a highly similar original of the same class, not that the specific original recovered is the true parent; where several originals are near-identical the assignment may be wrong, and the split is lineage-disjoint with respect to the inferred lineage. With lineage known, a test record was defined as lineage-linked if its lineage group (the parent post together with all of its derivatives) has at least one member in the training set. A lineage-linked record is therefore either an augmented derivative of a training post or an original post whose derivatives lie in the training partition; both forms create the same dependence between the partitions. Table 3 and Figure 2 report the result for the random split: 13,164/26,228 (50.2%) test records are lineage-linked, comprising every schizophrenia record, 99.5% of anxiety records, and 94.7% of depression records; the suicidal-ideation class, which was never augmented, is unaffected. Of the 13,164 lineage-linked test records, 11,219 are augmented derivatives of training posts and 1,945 are original posts whose derivatives lie in training. Of the 18,606 test records that the lexical audit retains, 5,588 (30.0%) remain lineage-linked: 5,282 augmented records and 306 originals. The lexical audit therefore removes the near-verbatim contamination layer and leaves the paraphrase layer in place.

ClassTest recordsLexically flaggedlineage-linkedRetained by lexical audit, lineage-linked
Anxiety4,0003,980 (99.5%)
Depression6,0007,622 (29.1%)5,684 (94.7%)5,588 of 18,606 (30%)
Schizophrenia3,5003,500 (100%)
Suicidal12,7280 (0%)
Total26,2287,622 (29.1%)13,164 (50.2%)5,588 (30%)

Table 3. Two-layer audit of the stratified random test split (N=26,228N = 26,228). “Lexical” is the MinHash–LSH audit (Jaccard ≥0.8\ge0.8 against the training set); “Lineage-linked” records share an augmentation lineage group with at least one training record, either as derivatives of a training post or as originals whose derivatives lie in training.

Figure 2. lineage-linked records in the stratified random test split, by class. The three augmented classes are almost entirely linked to training posts through augmentation lineage; the suicidal-ideation class, which was not augmented, is unaffected.

Table 3: Two-layer audit of the stratified random test split (N=26,228N = 26,228). “Lexical” is the MinHash–LSH audit (Jaccard ≥0.8\ge0.8 against the training set); “Lineage-linked” records share an augmentation lineage group with at least one training record, either as derivatives of a training post or as originals whose derivatives lie in training.

Lineage-disjoint split Because the contamination is structural, it was removed by construction rather than by filtering. The 149,195 original posts were split 70/15/15 with class stratification (seed 42); every augmented record was then assigned to the split of its parent, and augmented records whose parent fell in the validation or test partition were discarded. Validation and test sets therefore contain original posts only; the training set contains original posts together with the derivatives of training posts only, and no lineage group spans two partitions (verified programmatically). A 70/15/15 ratio was preferred to 80/10/10 so that the smallest class retains 75 test records rather than 50. Table 4 gives the resulting partition sizes.

PartitionAnxietyDepressionSchizophreniaSuicidalTotal
Train (original)3,64711,34635089,093104,436
Train (augmented)24,19530,79424,333079,322
Train (total)27,84242,14024,68389,093183,758
Validation7822,4317519,09122,379
Test7812,4327519,09222,380

Table 4. Lineage-disjoint split. Training contains original posts and their derivatives; validation and test contain original posts only.

Residual lexical and semantic audits The same MinHash–LSH audit was then applied to the lineage-disjoint split. It flags 78/22,379 (0.35%) validation records and 83/22,380 (0.37%) test records against the training set, and 22 test records against the validation set; inspection shows that these are re-posted or templated texts among original posts. Because a lexical audit cannot detect re-posts whose wording has been edited beyond the 5-gram threshold, a semantic audit was added. Every validation and test post and every original training post was embedded with a sentence encoder (all-MiniLM-L6-v2, mean pooling, ℓ2\ell_2-normalized), and the maximum cosine similarity to any original training post was recorded. At a threshold of 0.90 the semantic audit flags 175 validation (0.78%) and 174 test (0.78%) posts, of which 73 and 80 respectively are also flagged lexically; 102 validation and 94 test posts are flagged semantically only (109 and 112 test posts exceed 0.95; 58 and 65 exceed 0.98). The highest-scoring cases are re-posts that share a long verbatim prefix with a training post but diverge later. The median test post has a maximum similarity of 0.74 to the training set and the 99th percentile is 0.87, so the flagged set is a thin tail rather than a shoulder of the distribution. The test retention mask therefore excludes 105 records (83 matching training and 22 matching validation). Of the 174 test records flagged semantically, 80 are already excluded by that mask, so the union of the two audits removes 199 records and the union-strict test subset contains 22,181 posts. Results are reported on the full test set; on the union-strict subset the headline macro-F1 is 0.8440 versus 0.8436 and the best single encoder 0.8233 versus 0.8225, so neither residual layer affects any conclusion.

Encoders and fine-tuning

Four encoder families were fine-tuned with three random seeds each (42, 43, 44), giving twelve models: Bio-ClinicalBERT [3] (pretrained on clinical notes), MentalBERT and MentalRoBERTa [18] (pretrained on mental-health Reddit text), and RoBERTa-base [24] (general web text). All runs share one recipe: a sequence-classification head on the pretrained backbone, the lower six of the twelve encoder layers frozen and the upper six fine-tuned, hidden dropout 0.3, maximum sequence length 256, batch size 16, the Adam optimizer with learning rate 2×10−52 \times10^{-5} and a reduce-on-plateau schedule (factor 0.2, patience 1 epoch), class-balanced sample weights computed on the training partition, at most 10 epochs, and early stopping on validation loss with patience 3 and restoration of the best weights. Implementation used TensorFlow 2.15.1 and Transformers 4.40.2. Each run took between 4 and 12 h on an NVIDIA A100-80GB GPU. Validation loss is used for early stopping and checkpoint selection, and the validation set is therefore not independent of the trained weights; every subsequent use of validation data is confined to the cross-fitted protocol described next.

Evaluation protocol

Cross-fitted post-training decisions Four post-training quantities depend on validation data: the member temperatures TmT_m, the ensemble selection (validation macro-F1 among triads spanning at least two encoder families, with CI⁡(S)\operatorname{CI}(S) as a tiebreaker among candidates of equal validation macro-F1), the optional per-class weights {αm,c}\{\alpha_{m,c}\}, and the post-hoc temperature T⋆T^{\star}. To prevent any of these from being tuned on the same records, the validation set was partitioned into K=5K = 5 folds. In rotation kk, the member temperatures were fitted on fold kk, the ensemble was selected on the three folds other than kk and k+1k+1, and the post-hoc temperature was fitted on fold k+1k+1 (indices modulo 5). Uniform averaging is the pre-specified fusion rule (§ 3.3); the learned per-class weights are evaluated as an alternative within the same rotations, fitted on the selection folds and scored on the held-out fold k+1k+1, so that the choice between the two rules rests on validation data alone (§ 5.4). Each rotation yields a complete system that is scored on the test set; the mean and spread across rotations constitute the primary, post-selection estimate of the procedure’s performance. The system selected on the full validation set under the same rule is reported as the deployed instance. The test set is used for reporting only: no temperature, weight, threshold, or selection decision in the study was informed by test-set results.

Metrics and uncertainty Macro-F1, accuracy, balanced accuracy, MCC [27], ECE with 15 equal-width bins [30], and the multiclass Brier score are reported. Confidence intervals for macro-F1 and MCC are 95% percentile intervals from B=1000B = 1000 non-parametric bootstrap resamples of the test set [8]; per-class F1 intervals are computed in the same way. ECE and Brier score are reported as point estimates on the full test set together with 95% bootstrap intervals, and paired bootstrap intervals are given for the differences in ECE and Brier score between compared systems; the sensitivity of ECE to the estimator is examined with 10, 15, 20, and 30 equal-width bins, 15 equal-mass bins, a class-wise variant [33], and a binning-free kernel estimator [34].

Significance testing Head-to-head comparisons use McNemar’s test with continuity correction [28] at the level of individual predictions, a paired bootstrap of the macro-F1 difference (B=1000B = 1000), and a permutation test in which the predictions of the two systems are exchanged per record (2,000 permutations). In every McNemar test, bb counts records the reference system classifies correctly, and the compared system does not, and cc counts the reverse; c>bc > b therefore favors the compared system. All ensemble–versus–member McNemar tests performed in the study (66 pairwise ensembles and the headline comparison, 67 tests) form one family, and the headline pp-value is reported after Holm correction over that family [14].

Evaluating CI as a predictor of fusion gain CI is computed on the full validation set and the realized gain on the test set for all (122)=66\binom{12}{2} = 66 pairs and 4 classes (264 pair–class observations). Because observations sharing a pair or a class are not independent, the pooled correlation is accompanied by cluster bootstraps over pairs and over models (2,000 resamples), within-class correlations, a linear mixed-effects model, and leave-one-encoder-out prediction of gain from CI. Because the 66 pairs are formed from only 12 encoders, each of which appears in 11 pairs, pair-level resampling alone understates the dependence. Within-class correlations are also reported with an encoder-level bootstrap, in which encoders are resampled with replacement, and each pair is weighted by the product of its two members’ multiplicities, and with a leave-one-encoder-out recomputation. The mixed-effects model uses a symmetric multi-membership random intercept. A single random effect per encoder enters every pair containing it, together with a random intercept per pair. Per-class slopes are estimated with the encoder effect alone, because with one observation per pair and class, a pair intercept would identify the slope from within-pair variation across classes. Two definitions are used in the paper. In the pairwise and head-to-head comparisons of Table 6 and Table 12, where a system is compared with a baseline a practitioner could have deployed, the reference is the member with the higher validation macro-F1, so that no baseline is chosen on the test set. In the CI analysis, where the gain is the outcome being predicted rather than a claimed improvement, the reference is the better of the two members on the given class (§ 3.4); results under the validation-selected reference are reported alongside. The CI–gain association and the tolerance comparisons of Table 7 were not pre-specified and are reported as exploratory; no test-set result informed any fitting or selection decision.

PairReference (by validation)macro-F1Δ\Delta vs. referenceCIMcNemar pp
MentalBERT s43 + RoBERTa-base s44MentalBERT s430.8447+0.0200 [+0.0076, +0.0329]0.1126.5×10−36.5 \times10^{-3}
MentalRoBERTa s43 + RoBERTa-base s43MentalRoBERTa s430.8443+0.0075 [−0.0047-0.0047, +0.0200]0.0821.5×10−51.5 \times10^{-5}
MentalRoBERTa s43 + RoBERTa-base s44MentalRoBERTa s430.8436+0.0069 [−0.0016-0.0016, +0.0166]0.0930.17
MentalRoBERTa s42 + RoBERTa-base s44MentalRoBERTa s420.8422+0.0043 [−0.0084-0.0084, +0.0174]0.0900.66
MentalBERT s42 + MentalRoBERTa s43MentalRoBERTa s430.8410+0.0042 [−0.0080-0.0080, +0.0185]0.0747.0×10−37.0 \times10^{-3}
MentalBERT s43 + MentalRoBERTa s43MentalRoBERTa s430.8404+0.0036 [−0.0075-0.0075, +0.0154]0.0962.1×10−92.1 \times10^{-9}
Bio-ClinicalBERT s43 + MentalRoBERTa s43MentalRoBERTa s430.8403+0.0035 [−0.0105-0.0105, +0.0192]0.1092.4×10−52.4 \times10^{-5}
RoBERTa-base s43 + RoBERTa-base s44RoBERTa-base s430.8401+0.0049 [−0.0040-0.0040, +0.0143]0.0852.2×10−22.2 \times10^{-2}
Bio-ClinicalBERT s42 + MentalRoBERTa s42MentalRoBERTa s420.8389+0.0009 [−0.0142-0.0142, +0.0161]0.1031.2×10−21.2 \times10^{-2}
MentalRoBERTa s42 + MentalRoBERTa s43MentalRoBERTa s420.8383+0.0004 [−0.0097-0.0097, +0.0091]0.0590.82

Table 6. Ten strongest pairwise ensembles (uniform mean of calibrated probabilities) on the lineage-disjoint test set. The reference member is the member with the higher validation macro-F1; Δ\Delta is the difference between the ensemble’s test macro-F1 and that of the reference member in Table 5, with a 95% paired-bootstrap interval (B=2000B = 2000 resamples of the test set); CI is the mean validation CI across classes; McNemar pp compares the ensemble with the reference member at the record level.

ToleranceMean candidates tiedSelection changedTest macro-F1, without CIwith CI
0 (exact)1.00/50.84200.8420
0.0011.82/50.84200.8425
0.0023.03/50.84200.8413
0.00511.44/50.84200.8388

Table 7. Selection ablation: the five cross-fitted rotations repeated with and without CI(S)CI(S) as a tiebreaker, for four definitions of “comparable” validation macro-F1. Test macro-F1 is the mean over the five rotations.

Table 12: Calibrated headline triad versus the validation-selected best single encoder on the lineage-disjoint test set. Bracketed values are 95% bootstrap percentile intervals (B=2000B = 2000); the Δ\Delta column and the tests below are paired over the same resamples. Differences are computed before rounding. Because macro-F1 weights the four classes equally, the 75-post schizophrenia class contributes +0.015 of the +0.021 macro-F1 difference; restricted to the three classes with at least 781 test posts, the difference is +0.008 [+0.003, +0.014].

Best single (MentalRoBERTa s44)Diverse triad, calibratedΔ\Delta (triad − single)
Macro-F10.8225 [0.8012, 0.8429]0.8436 [0.8231, 0.8625]+0.0211 [+0.0071, +0.0366]
MCC0.7648 [0.7534, 0.7754]0.7791 [0.7675, 0.7898]+0.0143 [+0.0067, +0.0221]
ECE0.0066 [0.0052, 0.0097]0.0033 [0.0027, 0.0064]−0.0033 [−0.0058, −0.0003]
Brier0.0889 [0.0850, 0.0928]0.0833 [0.0796, 0.0869]−0.0056 [−0.0073, −0.0039]
Accuracy0.93620.9410+0.0048 [+0.0027, +0.0070]
Balanced acc.0.84740.8594+0.0121 [−0.0045, +0.0299]
F1 Anxiety0.879 [0.862, 0.896]0.884 [0.866, 0.900]+0.004 [−0.008, +0.018]
F1 Depression0.740 [0.728, 0.753]0.758 [0.745, 0.770]+0.018 [+0.009, +0.026]
F1 Schizophrenia0.704 [0.618, 0.785]0.764 [0.686, 0.836]+0.060 [+0.009, +0.117]
F1 Suicidal0.966 [0.964, 0.968]0.969 [0.967, 0.970]+0.002 [+0.001, +0.004]

Macro-F1 difference, permutation test: p=0.0035p = 0.0035

McNemar on per-record correctness: b/c=237/345b/c = 237/345; χ2=19.67\chi^{2} = 19.67; p=9.2×10−6p = 9.2 \times10^{-6}; p=4.6×10−4p = 4.6 \times10^{-4} after Holm correction over 67 tests

Table 12.

Lexical baseline and artifact controls A TF–IDF (unigram and bigram) logistic-regression classifier with class-balanced weights was trained on the lineage-disjoint training set to measure the lexically separable part of the task. Reliance on platform artifacts was tested by re-scoring the test set with Reddit meta-vocabulary masked (e.g., reddit, subreddit, mods, tldr, edit, throwaway, tw) and, additionally, with explicit condition names and their derivatives masked (e.g., anxiety, depressed, schizophrenia, antipsychotics, suicidal); performance is also reported on the natural subset of test posts that contain no condition name.

External corpus Zero-shot transfer was evaluated on SDCNL [12], an independently collected corpus of 1,895 posts that retains author identifiers. Posts were cleaned with the same procedure as the main corpus; 4 posts whose body was deleted or removed retained fewer than five tokens after cleaning and were discarded, leaving 1,891. Of these, the 83 posts (4.4%) that the MinHash–LSH audit flagged as near-duplicates of any main-corpus record were removed, leaving 1,808, which were divided into author-disjoint halves of 892 and 916.

Attribution analysis Integrated Gradients (Eq. (8); 20 interpolation steps, zero baseline) were computed for the three deployed checkpoints of the headline ensemble on 1,575 test posts: all 75 schizophrenia posts and 500 posts sampled from each other class (seed 0). Agreement between the attribution run’s predictions and the stored test predictions was verified for every checkpoint (≥99.9%\ge99.9\% identical arg-max decisions). Word-level alignment follows § 3.6. The counterfactual protocol is described in § 6.

Results

Individual encoders

Table 5 reports the twelve fine-tuned encoders on the lineage-disjoint test set (N=22,380N = 22,380), after member-wise temperature scaling, and Fig. 3 shows per-class F1 with bootstrap intervals. Three observations follow. First, seed-averaged macro-F1 orders the families as MentalRoBERTa (0.832 ±\pm 0.009), RoBERTa-base (0.822 ±\pm 0.020), MentalBERT (0.819 ±\pm 0.007), and Bio-ClinicalBERT (0.803 ±\pm 0.005), but the 95% intervals of individual runs are approximately ±\pm0.02 wide and overlap across families; the difference between the mental-health-specialized families and Bio-ClinicalBERT (0.02–0.03) is the only family contrast that is consistent across all seeds. Second, seed-to-seed variation is material: RoBERTa-base spans 0.799–0.835 across seeds, a range larger than most between-family differences, which is why three seeds per family are used throughout. Third, per-class performance is highly uneven. Suicidal ideation, the largest class, is classified with F1 0.963–0.968 by every model; anxiety with 0.84–0.89; depression, the class with the greatest lexical overlap with suicidal ideation, with 0.71–0.75; and schizophrenia, with 75 test posts, with 0.62–0.77 and intervals roughly ±\pm0.08 wide. Member temperatures TmT_m lie between 1.32 and 1.52, i.e. every encoder is over-confident before scaling, and post-scaling ECE is below 0.011 for every model.

ModelTmT_mMacro-F1MCCECEBrierAccuracyBal. acc.
Bio-ClinicalBERT s421.470.8064 [0.7852, 0.8265]0.7382 [0.7262, 0.7489]0.00560.09730.92980.8494
Bio-ClinicalBERT s431.490.7968 [0.7755, 0.8177]0.7411 [0.7295, 0.7529]0.00390.09520.93120.8330
Bio-ClinicalBERT s441.520.8056 [0.7810, 0.8261]0.7400 [0.7280, 0.7514]0.00820.09560.93110.8408
MentalBERT s421.430.8107 [0.7884, 0.8309]0.7538 [0.7427, 0.7648]0.01090.09650.93060.8511
MentalBERT s431.330.8247 [0.8027, 0.8444]0.7585 [0.7474, 0.7696]0.00570.09110.93660.8372
MentalBERT s441.390.8219 [0.7987, 0.8423]0.7649 [0.7532, 0.7762]0.00430.08970.93670.8537
MentalRoBERTa s421.320.8379 [0.8161, 0.8556]0.7691 [0.7580, 0.7802]0.00380.08610.93870.8426
MentalRoBERTa s431.390.8368 [0.8154, 0.8553]0.7641 [0.7527, 0.7753]0.00710.08970.93490.8472
MentalRoBERTa s441.370.8225 [0.8012, 0.8429]0.7648 [0.7534, 0.7754]0.00660.08890.93620.8474
RoBERTa-base s421.450.7994 [0.7761, 0.8204]0.7558 [0.7450, 0.7675]0.01070.09270.93330.8496
RoBERTa-base s431.420.8352 [0.8131, 0.8557]0.7684 [0.7574, 0.7797]0.00970.08830.93770.8450
RoBERTa-base s441.390.8307 [0.8070, 0.8509]0.7568 [0.7454, 0.7673]0.00950.09300.93220.8476

Table 5. Individual-model performance on the lineage-disjoint test set (N=22,380N = 22,380) after member-wise temperature scaling. Macro-F1 and MCC are given with 95% bootstrap percentile intervals (B=1000B = 1000); ECE (15 bins), Brier score, accuracy, and balanced accuracy are point estimates on the full test set. Best value per column in bold.

Figure 3. Per-class F1 of the twelve fine-tuned encoders on the lineage-disjoint test set; error bars are 95% bootstrap intervals. Depression is the hardest of the three large classes; schizophrenia has 75 test posts and correspondingly wide intervals.

Pairwise fusion and the Complementarity Index

All 66 pairs were fused by uniform averaging of calibrated probabilities. Table 6 lists the ten strongest pairs by test macro-F1 together with the gain over a reference member; to avoid selecting the reference on the test set, the reference member of each pair is the one with the higher validation macro-F1. Fusion improves macro-F1 over the validation-selected member in 43 of 66 pairs; the paired-bootstrap interval of the gain excludes zero in the positive direction for 15 pairs and in the negative direction for 3. McNemar’s test, which operates on per-record accuracy, rejects the null for 48 of 66 pairs; the discrepancy with the macro-F1 intervals arises because accuracy is dominated by the suicidal-ideation class (85% of records), whereas macro-F1 is governed by the three smaller classes. Both quantities are therefore reported throughout, and macro-F1 intervals are treated as the primary criterion.

Figure 4 relates the CI, computed on validation predictions, to the fusion gain realized on the test set for the 264 pair–class observations. CI values on this corpus lie between 0.02 and 0.21: the encoders make substantially complementary errors. The pooled Pearson correlation across all observations is r=0.05r = 0.05 (cluster-bootstrap interval over pairs [−0.03-0.03, 0.14]; over models, with pairs weighted by the product of the sampled model multiplicities, [−0.13-0.13, 0.23]), and this pooled figure is not informative because the four classes occupy different regions of both axes. The within-class picture is heterogeneous (Figure 4b). CI is positively associated with gain for depression (r=0.50r = 0.50) and suicidal ideation (r=0.68r = 0.68), the two most confusable classes, is uncorrelated with gain for anxiety (r=−0.20r = -0.20, interval including zero), and carries no usable signal for schizophrenia (r=0.16r = 0.16), whose per-pair gains have a standard deviation of 0.022—five times that of any other class—because they are computed from 75 posts. Excluding schizophrenia, the pooled correlation is 0.61. A linear mixed-effects model with class as a fixed effect, a symmetric multi-membership random intercept per encoder, and a random intercept per pair estimates a CI slope of 0.062 (95% interval [−0.001-0.001, 0.125], p=0.053p = 0.053) over all classes and 0.048 ([0.0220.022, 0.0750.075], p=0.0004p = 0.0004) over the three adequately sampled classes; the pooled slope is therefore not significant once schizophrenia is included, and a validation-measured accuracy gap between the two members adds no explanatory power. Under encoder-level resampling the within-class associations for depression and suicidal ideation remain positive and exclude zero (r=0.50r = 0.50, [0.190.19, 0.790.79]; r=0.68r = 0.68, [0.510.51, 0.850.85]), as do their per-class slopes with the encoder random effect (0.076 [0.0460.046, 0.1060.106] and 0.150 [0.1120.112, 0.1880.188], both p<0.001p < 0.001), whereas the intervals for anxiety (r=−0.20r = -0.20, [−0.50-0.50, 0.270.27]; slope −0.029-0.029 [−0.113-0.113, 0.0550.055]) and schizophrenia (r=0.16r = 0.16, [−0.32-0.32, 0.550.55]; slope 0.073 [−0.127-0.127, 0.2730.273]) contain zero. Dropping any single encoder and recomputing leaves every within-class correlation on the same side of zero (depression 0.40–0.55, suicidal ideation 0.65–0.73). Defining the gain instead against the member with the higher validation macro-F1 raises the pooled correlation to r=0.15r = 0.15 (cluster-bootstrap interval over pairs [0.070.07, 0.230.23]) and the all-class slope to 0.121 ([0.0210.021, 0.2210.221], p=0.018p = 0.018), while the slope over the three sampled classes is essentially unchanged at 0.041 ([0.0060.006, 0.0760.076], p=0.023p = 0.023); the class-conditional pattern is the same under both definitions. Leave-one-encoder-out prediction, in which the regression is fitted on the pairs that exclude a given encoder and applied to the pairs containing it, gives pooled out-of-sample Spearman ρ=0.60\rho= 0.60 and R2=0.27R^2 = 0.27; by class it is ρ=0.66\rho= 0.66 (R2=0.42R^2 = 0.42) for suicidal ideation and ρ=0.38\rho= 0.38 (R2=0.17R^2 = 0.17) for depression, and negative for anxiety and schizophrenia. CI therefore transfers to pairs containing an encoder it was not fitted on, but only for the two classes where the association exists. CI is therefore a class-conditional diagnostic rather than a global predictor: it indicates which classes will benefit from fusion, not by how much and not which pair to choose, and it is uninformative for a class whose test support is too small to measure gain reliably.

Figure 4. Complementarity Index versus realized fusion gain. (a) CI computed on the validation set against the class-wise F1 gain of the uniform-mean pair over the better member’s class-wise F1 on the test set (§ 3.4), for all 66 pairs and 4 classes, with per-class least-squares fits. (b) Within-class Pearson correlation with 95% encoder-level bootstrap intervals (encoders resampled with replacement, each pair weighted by its member multiplicities). The association is positive for depression and suicidal ideation, absent for anxiety, and unmeasurable for the 75-record schizophrenia class.

Ensemble selection under the cross-fitted protocol

Three-member ensembles were selected from the 216 candidate triads that span at least two encoder families, ranked by validation macro-F1 with CI(SS) as a tiebreaker. Table 8 reports the five cross-fitted rotations. Post-selection test macro-F1 is 0.842 ± 0.004 (mean ± s.d. over rotations), which is the primary estimate of what the procedure delivers, and ECE after post-hoc calibration is 0.0043 ± 0.0009; the post-hoc temperature is T∗=0.90±0.04T^* = 0.90 \pm0.04 and is below unity in every rotation. Selection is sensitive to seed identity—four distinct triads are chosen across the five rotations—but stable at the family level: MentalRoBERTa is a member in every rotation, RoBERTa-base in four, MentalBERT in three, and Bio-ClinicalBERT in one. On the full validation set, the selected triad is MentalBERT s43 + MentalRoBERTa s43 + RoBERTa-base s42 (validation macro-F1 0.868, CI(S)=0.095CI(S) = 0.095, three families), which is also the triad chosen in two of the five rotations; it is reported as the deployed instance in the remaining tables.

kkSelected triadT⋆T^\starTest macro-F1Test ECE (before → after T⋆T^\star)
0MB s43 + MR s43 + RB s420.8590.84420.0079 → 0.0040
1MB s42 + MR s43 + RB s430.9290.84720.0059 → 0.0038
2MR s42 + MR s44 + RB s420.9010.83630.0085 → 0.0046
3BC s44 + MR s42 + MR s440.8780.83880.0085 → 0.0057
4MB s43 + MR s43 + RB s420.9440.84370.0046 → 0.0035
Mean ± s.d.0.902 ± 0.0350.8420 ± 0.00440.0071 ± 0.0017 → 0.0043 ± 0.0009

Table 8. Cross-fitted selection: in rotation kk, member temperatures are fitted on validation fold kk, the triad is selected on the three remaining folds, and T⋆T^\star is fitted on fold k+1k+1; each resulting system is scored once on the test set. Test ECE is given before and after the post-hoc temperature. BC: Bio-ClinicalBERT; MB: MentalBERT; MR: MentalRoBERTa; RB: RoBERTa-base.

Because CI(S)CI(S) enters the rule only as a tiebreaker, its influence depends on what counts as comparable validation performance, a tolerance the protocol leaves implicit. Table 7 therefore repeats the five cross-fitted rotations with and without the tiebreaker under four tolerances. At exact equality, no two of the 216 candidates ever tie, so the tiebreaker never activates and every triad reported in this paper was selected by validation macro-F1 alone. Under a tolerance of 0.001 the selection changes in two rotations and the mean test macro-F1 rises by 0.0005; at 0.002 and 0.005 it changes in three and four rotations and falls by 0.0007 and 0.0032. The index therefore does not improve ensemble selection at any tolerance examined, and no such claim is made; its demonstrated role is the predictive one of § 5.2.

Table 9 removes the seed dimension by evaluating every family-level triad over all 27 seed instantiations. The four family triads lie within 0.004 of one another (0.836–0.840), each with a seed standard deviation of 0.004–0.007; the triad without Bio-ClinicalBERT has the highest mean validation and test macro-F1. Every family triad exceeds the validation-selected best single encoder (0.822) by at least 0.014 on average.

Family triadVal. macro-F1Test macro-F1Test ECE
MentalBERT + MentalRoBERTa + RoBERTa-base0.863 ± 0.0020.840 ± 0.0060.0035
Bio-ClinicalBERT + MentalRoBERTa + RoBERTa-base0.860 ± 0.0030.839 ± 0.0070.0035
Bio-ClinicalBERT + MentalBERT + MentalRoBERTa0.860 ± 0.0030.839 ± 0.0040.0036
Bio-ClinicalBERT + MentalBERT + RoBERTa-base0.860 ± 0.0040.836 ± 0.0060.0030

Table 9. Family-level triads: mean ± s.d. of macro-F1 over the 27 seed instantiations of each triad (one seed per family), and mean test ECE after post-hoc calibration.

NN-way ensembles

Table 10 compares the headline triad with the validation-selected best single encoder, homogeneous three-seed ensembles, and the twelve-model ensemble; Figure 5 shows ROC curves and Figure 6 shows confusion matrices. The heterogeneous triad (0.8436) exceeds the homogeneous three-seed MentalRoBERTa ensemble (0.8379), which is built from the strongest family, and the homogeneous RoBERTa-base ensemble (0.8299). The twelve-model uniform ensemble reaches 0.8478, i.e. 0.004 above the triad at four times its inference cost, with overlapping intervals. Uniform averaging is the pre-specified fusion rule; learned per-class weights (Eq. (3)) were evaluated against it on validation data within the cross-fitted rotations. Fitted on the selection folds, the weights raise in-sample macro-F1 by +0.005 on average, but on the held-out validation fold they exceed uniform averaging in only two of five rotations (mean Δ\Delta = +0.003, s.d. 0.008, one rotation contributing +0.016 and the others between −0.002 and +0.003). The eight free parameters therefore do not improve macro-F1 consistently out of sample within validation, and uniform averaging is retained on that basis. For completeness, the test macro-F1 of the weighted and uniform versions of the same rotations differ by −0.0007 on average, and weights fitted on the full validation set give 0.8437 versus 0.8436 for the deployed triad and 0.8453 versus 0.8478 for the twelve-model ensemble; these test figures played no part in the decision.

Systemmacro-F1MCCT∗T^*ECE (before →\to after)
Best single by validation (MentalRoBERTa s44)0.8225 [0.8012, 0.8429]0.76481.000.0066 →\to 0.0066
Homogeneous 3×RoBERTa-base0.8299 [0.8063, 0.8513]0.76670.940.0076 →\to 0.0070
Homogeneous 3×MentalRoBERTa0.8379 [0.8164, 0.8567]0.77520.950.0051 →\to 0.0034
Diverse triad0.8436 [0.8231, 0.8625]0.77910.890.0063 →\to 0.0033
All twelve, uniform0.8478 [0.8278, 0.8667]0.78520.830.0120 →\to 0.0028
All twelve, learned per-class weights0.8453———

Table 10. NN-way ensembles on the lineage-disjoint test set. All ensembles are uniform means of calibrated member probabilities followed by a post-hoc temperature fitted on the validation set; ECE is given before and after T∗T^*. The diverse triad is MentalBERT s43 + MentalRoBERTa s43 + RoBERTa-base s42.

Figure 5. One-versus-rest ROC curves of the three headline members and of the calibrated triad on the lineage-disjoint test set.

Figure 6. Row-normalized confusion matrices of the three headline members and of the calibrated triad. The dominant residual confusion is depression→suicidal ideation (17–22% of depression posts for the members, 19% for the triad); the triad attains the highest recall on schizophrenia (0.80 versus 0.76–0.77).

Post-hoc ensemble calibration

The post-hoc temperature fitted on the validation set for the headline triad is T⋆=0.886T^{\star} = 0.886; its members’ temperatures are 1.33, 1.39, and 1.45. Applying T⋆T^{\star} leaves every arg-max decision unchanged and reduces the test ECE from 0.0063 [0.0057, 0.0096] to 0.0033 [0.0027, 0.0064] (47.6%; paired bootstrap interval of the reduction [−0.0058, −0.0002]). Figure 7 shows the reliability diagrams, and Table 11 shows that the reduction is not an artifact of the 15-bin estimator: it is reproduced with 10, 20, and 30 equal-width bins, with equal-mass bins, with the class-wise variant, and with the binning-free kernel estimator; the negative log-likelihood also decreases, and the Brier score, which is dominated by its refinement component, is unchanged to four decimals. The same direction holds for every ensemble in Table 10 and every rotation in Table 8: the average of individually calibrated members is under-confident in domain, and a single sharpening temperature corrects it.

Figure 7. Reliability diagrams of the headline triad before (left) and after (right) the post-hoc temperature, 15 equal-width bins. The gap between confidence and accuracy is reduced without any change in the predicted labels.

EstimatorBefore T⋆T^{\star}After T⋆T^{\star}
ECE, 10 equal-width bins0.00630.0028
ECE, 15 equal-width bins0.00630.0033
ECE, 20 equal-width bins0.00720.0034
ECE, 30 equal-width bins0.00760.0045
ECE, 15 equal-mass bins0.00660.0042
Class-wise ECE, 15 bins0.01210.0106
Kernel (binning-free) ECE0.00790.0045
Negative log-likelihood0.14470.1436
Brier score0.08330.0833

Table 11. Sensitivity of the calibration result to the estimator: headline triad before and after the post-hoc temperature.

Head-to-head comparison with the best single encoder

Table 12 compares the calibrated headline triad with the single encoder that a practitioner would select on validation (MentalRoBERTa s44). The triad improves macro-F1 by +0.021 (paired bootstrap [+0.007, +0.037]; permutation p=0.0035p = 0.0035), MCC by +0.014, and Brier score by 0.0056 (paired interval [−0.0073, −0.0039]), and halves the ECE (Δ=−0.0033\Delta= -0.0033, [−0.0058, −0.0003]). McNemar’s test on per-record correctness gives b/c=237/345b/c = 237/345, χ2=19.7\chi^{2} = 19.7, p=9.2×10−6p = 9.2 \times10^{-6}, and p=4.6×10−4p = 4.6 \times10^{-4} after Holm correction over the 67 ensemble-versus-member tests performed in this study. The improvement is distributed across classes. Schizophrenia gains +0.060 [+0.009, +0.117] and depression +0.018 [+0.009, +0.026], while anxiety gains +0.004 [−0.008, +0.018] and suicidal ideation +0.002 [+0.001, +0.004]. Because macro-F1 weights the four classes equally, the 75-post schizophrenia class contributes +0.015 of the +0.021 total, or 71% of the headline gain (bootstrap interval of that share [27%, 91%]). The gain does not depend on that class alone: restricted to the three classes with at least 781 test posts, the macro-F1 improvement is +0.008 [+0.003, +0.014] and still excludes zero. Both figures should be read together, since the headline number is dominated by the class with the weakest statistical support and the least certain provenance (§ 7.5). The single encoder that is strongest on the test set (MentalRoBERTa s42, 0.838) is not identifiable from validation data; against it the triad’s advantage is +0.006 and not significant, which is the appropriate reading of how much of the ensemble gain is recoverable by selecting one model well.

Lexical baseline and artifact controls

A TF–IDF logistic-regression classifier trained on the lineage-disjoint training set reaches macro-F1 0.748 [0.727, 0.768] on the test set (per class 0.79, 0.57, 0.68, 0.95), i.e. 0.096 below the triad, with the largest deficit on depression (0.57 versus 0.76). Trained on the 104,436 original training posts alone, without augmentation, the same classifier reaches 0.489, with schizophrenia F1 falling to 0.06: augmentation within the training partition is what makes the smallest classes learnable, and it is confined to that partition. Table 13 reports the masking controls for the triad. Masking Reddit meta-vocabulary, present in 17.7% of test posts, changes macro-F1 by +0.001, so the tested platform vocabulary carries no usable signal. This control covers an explicit word list only; broader source- or style-level artifacts, which cannot be masked lexically, are not excluded by it. Additionally, masking explicit condition names, present in 45.4% of test posts, lowers macro-F1 to 0.753, with the loss concentrated on schizophrenia (0.764 → 0.537) and anxiety (0.884 → 0.800), smaller on depression (0.758 → 0.711) and negligible on suicidal ideation (0.969 → 0.965). Because masking produces inputs unlike any seen in training, the in-distribution figure is the more informative one: on the 54.6% of test posts that never name a condition, the triad reaches macro-F1 0.792.

Table 13: Artifact controls for the headline triad on the lineage-disjoint test set.

Test-set variantmacro-F1AnxDepSczSui
Original0.8440.8840.7580.7640.969
Reddit meta-vocabulary masked (17.7% of posts)0.8450.8840.7560.7720.968
+ condition names masked (45.4% of posts)0.7530.8000.7110.5370.965
Natural subset: posts with no condition name (54.6%)0.792—
TF–IDF + logistic regression (full test set)0.7480.7900.5740.6820.945

Table 13.

Zero-shot transfer to a same-platform external corpus

Table 14 reports zero-shot transfer to the author-disjoint SDCNL test half. The triad reaches two-class macro-F1 0.723 [0.693, 0.750]. Against the member selected on the external validation half (MentalBERT s43, 0.699 on the test half) the gain is +0.024 [+0.002, +0.047] (McNemar b/cb/c = 44/66, pp = 0.045); against the member that happens to be strongest on the external test half (MentalRoBERTa s43, 0.714), which is not identifiable from validation data, it is +0.008 [−0.012, +0.029] and not significant. The four-class models assign a class outside {depression, suicidal ideation} to only 3–5% of external posts. Calibration behaves differently under transfer: every temperature fitted on the external validation half exceeds unity (1.4–2.4), i.e. the models and their ensemble are over-confident out of domain, and the post-hoc step fitted on 892 external posts halves the triad’s ECE (0.066 → 0.033). CI computed on the external validation half shows no correlation with gain on the external test half (rr = 0.05 over 12 pair–class observations), but all gains lie within ±0.03 and all CI values within 0.18–0.29, so the test is inconclusive at this sample size rather than negative.

Systemmacro-F1F1 depressionF1 suicidalECE (before → after)Text∗T_{\mathrm{ext}}^{*}
MentalBERT s430.699 [0.668, 0.727]0.6950.7020.078 → 0.0271.76
MentalRoBERTa s430.714 [0.686, 0.745]0.7300.6990.070 → 0.0511.79
RoBERTa-base s420.693 [0.664, 0.723]0.7040.6820.105 → 0.0542.40
Bio-ClinicalBERT s420.678 [0.648, 0.707]0.6830.6730.079 → 0.0412.02
Diverse triad0.723 [0.693, 0.750]0.7270.7180.066 → 0.0331.50
All four0.718 [0.689, 0.749]0.7230.7140.053 → 0.0371.37

Table 14. Zero-shot transfer to SDCNL (external test half, NN = 916, author-disjoint from the external validation half; 83 near-duplicates of the main corpus removed). Two-class macro-F1 with 95% bootstrap intervals; ECE before and after a post-hoc temperature fitted on the external validation half.

Attribution-Diversity Analysis and Counterfactual Intervention

The question addressed in this section is which property of the members drives the fusion benefit. Integrated Gradients (Eq. (8)) were computed for the three deployed checkpoints of the headline triad on 1,575 test posts (all 75 schizophrenia posts and 500 per other class), and attributions were aligned at word level as described in § 3.6. A post is an all-correct case if the ensemble and all three members classify it correctly, a fusion-rescued case if the ensemble is correct and at least one member is wrong, and an ensemble-failure case if the ensemble is wrong. The sample contains 1,250 all-correct cases (79.4%), 122 fusion-rescued cases (7.7%; 101 with exactly one incorrect member and 21 with two), and 203 ensemble-failure cases (12.9%).

Attribution diversity is higher on fusion-rescued cases

For each post, the attribution-diversity score is 1 minus the mean pairwise cosine similarity of the three word-level attribution vectors. Table 15 and Figure 8a give the result. All-correct cases have a mean diversity of 0.401 and fusion-rescued cases 0.904 (Δ\Delta = +0.503, permutation pp = 0.0001 with 10,000 shuffles); ensemble-failure cases lie in between (0.646). The effect is present within every class (Δ\Delta between +0.36 and +0.63, all pp < 0.001), so it is not a by-product of the classes’ different base rates of diversity.

All-correctFusion-rescuedΔ\Deltapp
All member pairs0.401 (n=1,250n = 1,250)0.904 (n=122n = 122)+0.5030.0001
Same-target pairs only0.401 (n=1,250n = 1,250)0.610 (n=115n = 115)+0.2090.0001
Position-wise, no word alignment0.6790.946+0.2680.0001
Top-20-word Jaccard distance0.6410.652+0.0110.33
Within class, all pairs (Δ\Delta, nn rescued): Anxiety +0.61 (39); Depression +0.36 (62); Schizophrenia +0.63 (7); Suicidal +0.55 (14); all p<0.001p < 0.001
ClassTest postsRescue rateFailure rateMean diversity
Anxiety5007.8%10.6%0.462
Depression50012.4%21.6%0.551
Schizophrenia759.3%20.0%0.454
Suicidal5002.8%5.4%0.405

Table 15. Word-level inter-model attribution diversity (1− mean pairwise cosine) on the 1,575 analysed test posts. Permutation pp-values use 10,000 label shuffles.

Figure 8. (a) Word-level attribution diversity on all-correct versus fusion-rescued cases, for all member pairs and for same-target pairs only. (b) Fraction of decisions preserved after removing five words by rule; rescued decisions depend on the divergent words.

Two controls address confounds specific to attribution methods. First, IG attributes a model’s own predicted class; a member that predicts a different class therefore attributes a different logit, which inflates diversity on rescued cases by construction. Restricting the comparison to member pairs that predict the same class removes this effect: same-target diversity is 0.401 on all-correct cases and 0.610 on rescued cases (Δ\Delta = +0.21, pp = 0.0001; nn = 115 rescued cases with at least one same-target pair), and 0.614 on the 101 rescued cases with exactly one incorrect member, whose two correct members always share the target. Second, position-wise comparison of token-level vectors from different tokenizers, without word alignment, gives a baseline diversity of 0.679 on all-correct cases, i.e. it manufactures disagreement from tokenization alone; word-level alignment reduces this to 0.401 while preserving the between-group difference. A top-20-word Jaccard distance shows no effect (Δ\Delta = +0.01, pp = 0.33): the members largely agree on which words matter and differ in the sign and magnitude they assign to them.

Per class (Table 15), the rescue rate is highest for depression (12.4%), the class with the highest mean diversity (0.551), followed by schizophrenia (9.3%), anxiety (7.8%), and suicidal ideation (2.8%). The rescue rate and the per-class fusion gain of Table 12 are both largest on the two classes where the members disagree most, but their orderings differ: depression has the highest rescue rate, whereas schizophrenia shows the largest F1 gain.

Counterfactual word removal

The association above does not establish that the divergent evidence causes the rescue. An instance-level intervention was therefore performed. For each of the 98 rescued cases with exactly one incorrect member and at least 15 aligned words, three sets of five words were removed from the post, and the three members were re-scored: (i) the five divergent words with the largest difference between the mean attribution of the two correct members and the attribution of the incorrect member, (ii) the five consensus words with the largest minimum attribution across all three members (excluding the divergent set), and (iii) five words drawn at random (three independent draws). The same procedure was applied to 282 all-correct cases as a reference, with the divergent set defined by the largest spread of attribution across members.

Figure 8b and Table 16 report the fraction of cases on which the ensemble decision remains correct. On rescued cases, removing five random words preserves the rescue 88.4% of the time and removing five consensus words 85.7%, whereas removing the five divergent words preserves it only 41.8% of the time (paired sign-permutation test against the random condition: Δ=0.47\Delta= 0.47, p=0.0001p = 0.0001; consensus versus random: p=0.22p = 0.22). When a rescue is destroyed by divergent-word removal, the ensemble adopts the class predicted by the originally incorrect member in 91% of cases (52 of 57). On all-correct cases, the same removal changes the decision far less often (81.2% preserved), so rescued decisions are specifically dependent on the words on which the members disagree. This shows that rescued decisions are markedly more sensitive to removal of the divergent words than to matched consensus or random removals. It does not isolate attribution divergence as the cause: the selected words may also be the most informative ones, and their removal changes the surrounding context.

Words removedFusion-rescued (n=98n = 98)All-correct (n=282n = 282)
None1.0001.000
Five random0.8840.983
Five consensus0.8570.979
Five divergent0.4180.812
Divergent vs. random, paired Δ\Delta (pp)0.47 (0.0001)0.17 (0.0001)
Consensus vs. random, paired Δ\Delta (pp)0.03 (0.22)0.005 (0.36)

Table 16. Counterfactual word removal: fraction of cases on which the ensemble decision remains correct after removing five words selected by each rule. Random is the mean of three draws.

Interpretation

Fusion rescue is concentrated on posts for which the members weight the same words differently, and it is the words on which the correct members diverge from the incorrect ones that carry the rescue. Together with § 5.2, this supports a consistent account: the ensemble helps on the confusable classes because its members extract partly different evidence from the same text, and CI, which counts class-specific examples that only one member classifies correctly, is a validation-time proxy for exactly this divergence. The intervention operates on inputs; a model-side intervention that enforces one member’s attribution pattern on another through attribution regularization [35] would test sufficiency and remains future work.

Discussion

What a leakage audit must cover

The two-layer audit of § 4.2 has a general lesson for corpora assembled from public collections: near-duplicate detection answers the question “is this evaluation record close to a training record?”, whereas the relevant question is “does this evaluation record share provenance with a training record?”. When augmentation, re-posting, or cross-posting precedes splitting, the second question has a different answer from the first. On the random split, the lexical audit at Jaccard ≥0.8\ge0.8 removed 29% of the test records and left 30% of the remainder lineage-linked; on the same corpus, the lineage-linked fraction was 100% for a class with 500 original posts and 34,500 derivatives. The corresponding consequence for reported performance is large: the per-class F1 of the schizophrenia class, which is close to 1 when derivatives of its 500 originals are distributed across splits, is 0.62–0.77 with intervals of ±0.08\pm0.08 when they are not.

Encoder families

Bio-ClinicalBERT was pretrained on MIMIC-III clinical notes, which are written in a clinical register, whereas the target text is self-authored and affective. The mental-health-specialized families exceed Bio-ClinicalBERT by 0.02–0.03 seed-averaged macro-F1, consistently across seeds, which is consistent with a register-mismatch account of specialist pretraining. General-domain RoBERTa-base, however, matches MentalBERT on average and exhibits the largest seed variance of any family, so the ordering among the three non-clinical families is not resolvable at three seeds and should not be over-interpreted. Family membership matters for fusion in a different way: the cross-fitted protocol selects MentalRoBERTa in every rotation and pairs it with a RoBERTa-base or MentalBERT member in most, and family-level triads that combine two backbone families with two pretraining corpora lie within 0.004 of one another.

Uniform versus learned weights

The learned per-class weighting (Eq. (3)) has K(M−1)K(M-1) free parameters under the per-class simplex constraint: eight for a triad and 44 for twelve members. Within the cross-fitted rotations, it improves in-sample macro-F1 on the selection folds but not consistently on the held-out validation fold (two of five rotations). The validation set is large (22,379 posts) but the two smallest classes contribute 857 of them, and the weights are fitted on macro-F1, in which those classes carry half the weight; the parameters therefore fit noise in the minority classes. Uniform averaging, the pre-specified rule, is retained on validation evidence alone.

Direction of the post-hoc temperature

Every member temperature exceeds unity (1.32–1.52): fine-tuned encoders are over-confident, and member-wise scaling softens them. The uniform mean of softened distributions is then under-confident in domain—the winning-class probability is lower than the empirical accuracy—and T⋆<1T^\star< 1 in every rotation, ensemble size, and family triad examined (0.83–0.95). Under transfer to SDCNL, the sign reverses: the members and their ensemble are over-confident on the external corpus and Text⋆T^\star_{\mathrm{ext}} lies between 1.4 and 2.4. The two-stage procedure is what transfers; the value and sign of T⋆T^\star do not, and a post-hoc temperature should be refitted on a small validation slice of any new domain, where 892 posts were sufficient to halve the ECE.

Threats to validity

Data source and labels The corpus is drawn from self-selected online communities, and the labels are proxies for a self-declared condition; the results concern patient-authored social-media language and not clinical diagnosis [13, 6, 9]. The lexical baseline and the masking controls show that the task is not lexically trivial and does not depend on platform artifacts, but explicit self-labeling contributes to the anxiety and schizophrenia classes, and the figure of 0.792 on posts that never name a condition is the appropriate estimate of performance without it. Manual inspection of the schizophrenia originals also found a small number of records written in a third-person clinical register rather than as forum posts; their provenance could not be traced in the merged file.

Split integrity The lineage-disjoint split removes contamination through augmentation lineage and leaves 0.4% lexical near-duplicates among original posts, which are masked. It does not resolve every form of overlap: author-, thread-, and temporal-level contamination among original posts cannot be audited because the corresponding metadata were not retained. Edited re-posts among originals are covered by the semantic audit down to a cosine of 0.90 (0.8% of evaluation posts, without effect on the results), but not below it. The external evaluation, which is author-disjoint by construction, partly compensates for the first of these. The parent of each derivative was recovered by similarity rather than from records; a misassignment could place a derivative across partitions only if it were more similar to another original than to its own parent, and the residual lexical audit bounds the effect.

Small-class statistics The schizophrenia class has 75 validation and 75 test posts. Its F1 intervals are ±0.08 wide, its per-pair fusion gains are dominated by sampling noise, and CI is uninformative for it; any conclusion specific to this class rests on weak evidence. Because macro-F1 weights classes equally, this class also carries 71% of the headline improvement over the best single encoder; the improvement restricted to the three larger classes is +0.008 [+0.003, +0.014].

Selection stability Ensemble selection is stable at the family level but not at the seed level: four different triads are chosen across five validation rotations. The headline system is therefore reported together with the post-selection spread (0.842 ± 0.004), and the family-level results, and the single-encoder comparison uses the encoder that validation data would select rather than the one that happens to be strongest on the test set.

Scope of the CI CI is associated with fusion gain within the two largest and most confusable classes and not within anxiety; the pooled correlation across classes is uninformative, and leave-one-encoder-out prediction has low out-of-sample accuracy. CI is presented as a class-conditional diagnostic; the ablation of Table 7 shows no selection benefit at any tolerance examined, and the external test of its transfer is inconclusive at 916 posts. The 66 pairs are formed from 12 encoders, so pair-level statistics are not independent; all CI intervals reported here are encoder-aware, and under that correction the pooled slope across all four classes is not significant.

External transfer The external corpus is drawn from the same platform as the main corpus, covers two of the four classes and one language, and is used zero-shot; it tests transfer of the trained system and of the calibration procedure to an independently collected, author-disjoint sample, not cross-platform or clinical validity of the four-class task. Evaluation on clinical interviews, cross-platform settings, or non-English text remains open.

Attribution analysis Integrated Gradients is one attribution method, with a zero baseline and 20 steps; the word-level alignment and same-target control address the two identified confounds, and the counterfactual intervention establishes the necessity of the divergent words for rescue on the analyzed cases.

Conclusion

A leakage-aware, calibration-aware decision-level fusion framework was proposed for four-class psychiatric text classification from user-generated text and evaluated under a protocol designed to withstand the failure modes of proxy-labeled corpora. A two-layer split-integrity audit showed that lexical near-duplicate detection, although it removed 29% of a random test split, left 30% of the retained records lineage-linked to training posts through augmentation lineage; a lineage-disjoint split was constructed. Twelve encoders across four families and three seeds were fine-tuned; every post-training decision, including the fusion rule, was made on validation folds disjoint from the fold used for the preceding step, and the test set was used for reporting only. On this basis, cross-fitted three-member ensembles reached a post-selection test macro-F1 of 0.842±0.0040.842 \pm0.004, and the deployed triad of MentalBERT, MentalRoBERTa, and RoBERTa-base achieved macro-F1 0.844 [0.823, 0.863], MCC 0.779, and ECE 0.0033, exceeding the validation-selected best single encoder by +0.021 macro-F1 under paired-bootstrap, permutation, and multiplicity-corrected McNemar tests, and coming within 0.004 of a twelve-model ensemble at one quarter of the cost. Two-stage calibration reduced the ensemble’s ECE by 47.6% under the 15-bin estimator. Every other binned, equal-mass, class-wise and binning-free estimator examined moved in the same direction (reductions of 12–56%), while the negative log-likelihood decreased slightly and the Brier score was unchanged; the post-hoc correction sharpened the aggregate in domain and softened it under external transfer. The Complementarity Index, computed on validation and evaluated against test gain with dependence-aware statistics, was informative for the depression and suicidal-ideation classes and not for anxiety. The reported triads were selected by validation macro-F1 alone, and the index is therefore reported as a diagnostic rather than as a selection mechanism. Word-aligned Integrated-Gradients attributions on the deployed checkpoints were 2.3×2.3 \times more diverse on fusion-rescued cases; the effect survived a same-target control, and a counterfactual intervention showed that removing the five words on which the correct members diverge from the incorrect one destroyed 58% of rescues, compared with 12% for random words. Future work includes model-side interventions on attribution patterns, external corpora that cover all four classes with author identifiers, and evaluation on clinical-interview text.

Acknowledgments

The authors would like to thank Prince Sultan University for its support.

Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this manuscript, the authors used Claude and ChatGPT to improve its content, clarity, and readability. The authors reviewed and edited all AI-assisted content and take full responsibility for the final published article.

References

  1. [1]Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul W. Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021.DOI
  2. [2]Süheyla Demirtaş Alpsalaz, Emrah Aslan, Yıldırım Özupak, Feyyaz Alpsalaz, Hasan Uzel, and levgen Zaitsev. Explainability-aligned reliability-weighted fuzzy ensemble for automated cervical cancer classification. International Journal of Intelligent Systems, 2026(1):2931556, 2026.DOI
  3. [3]Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78. Association for Computational Linguistics, 2019.arxiv.org/abs/1904.03323
  4. [4]Andrei Z. Broder. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences, pages 21–29. IEEE, 1997.
  5. [5]Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. Diversity creation methods: A survey and categorisation. Information Fusion, 6(1):5–20, 2005.
  6. [6]Stevie Chancellor and Munmun De Choudhury. Methods in predictive techniques for mental health status on social media: a critical review. NPJ digital medicine, 3(1):43, 2020.DOI
  7. [7]Zhanyi Ding, Zhongyan Wang, Yuebbei Zhang, Yuchen Cao, Yuhang Liu, Xi Shen, Yuhui Tian, and Jiacheng Dai. Trade-offs between machine learning and deep learning for mental illness detection on social media. Scientific Reports, 15(1):14497, 2025.DOI
  8. [8]Bradley Efron and Robert J. Tibshirani. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1(1):54–75, 1986.
  9. [9]Sindhu Kiranmai Ernala, Michael L. Birnbaum, Kristin A. Candan, Asra F. Rizvi, William A. Sterling, John M. Kane, and Munmun De Choudhury. Methodological gaps in predicting mental health states from social media: Triangulating diagnostic signals. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2019.
  10. [10]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017.arxiv.org/abs/1706.04599
  11. [11]Rafat Hammad and Suha Rababah. Domain-informed density extraction for robust mental health classification of long social media posts. Frontiers in Artificial Intelligence, 9:1861620, 2026.DOI
  12. [12]Ayaan Haque, Viraaj Reddi, and Tyler Giallanza. Deep learning for suicide and depression identification with unsupervised label correction. In International Conference on Artificial Neural Networks (ICANN), pages 436–447. Springer, 2021.arxiv.org/abs/2102.09427
  13. [13]Keith Harrigian, Carlos Alejandro Aguirre, and Mark Dredze. On the state of social media data for mental health research. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 15–24, 2021.arxiv.org/abs/2011.05233
  14. [14]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979.DOI
  15. [15]Oahidul Islam, Md Assaduzzaman, Sumaia Akter, Nafiz Fahad, and Md Jakir Hossen. Enhanced cervical cancer diagnosis using a novel bayesian fusion ensemble method with explainable ai. Scientific Reports, 16(1):12306, 2026.DOI
  16. [16]Sazzadul Islam, Rezaul Haque, Mahbub Alam Khan, Arafath Bin Mohiuddin, Md Ismail Hossain Siddiqui, Zishad Hossain Limon, Katura Gania Khushbu, S M Masfequier Rahman Swapno, Md. Redwan Ahmed, and Abhishek Appaji. Ensemble transformer with post-hoc explanations for depression emotion and severity detection. iScience, 29(2):114605, 2026.DOI
  17. [17]Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, 2019.
  18. [18]Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. Mentalbert: Publicly available pretrained language models for mental healthcare. In proceedings of the thirteenth language resources and evaluation conference, pages 7184–7190, 2022.
  19. [19]Furkan Kasap, Sevinc İlhan Omurca, and Ekin Ekinci. Ensemble of transformers for depression emotion classification. Cognitive Neurodynamics, 20(1):78, 2026.
  20. [20]Bahador Khaleghi, Alaa Khamis, Fakhreddine O Karray, and Sajedeh N Razavi. Multisensor data fusion: A review of the state-of-the-art. Information fusion, 14(1):28–44, 2013.
  21. [21]Ludmila I Kuncheva and Christopher J Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning, 51(2):181–207, 2003.
  22. [22]Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, 2022.
  23. [23]Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. Mining of massive datasets. Cambridge University Press, 2014.
  24. [24]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019.arxiv.org/abs/1907.11692
  25. [25]Daniel M Low, Laurie Rumker, Tanya Talkar, John Torous, Guillermo Cecchi, and Satrajit S Ghosh. Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on reddit during covid-19: Observational study. Journal of medical Internet research, 22(10):e22635, 2020.
  26. [26]Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
  27. [27]Brian W. Matthews. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451, 1975.
  28. [28]Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947.
  29. [29]Ridha Mezzi, Aymen Yahyaoui, Mohamed Wassim Krir, Wadii Boulila, and Anis Koubaa. Mental health intent recognition for arabic-speaking patients using the mini international neuropsychiatric interview (mini) and bert model. Sensors, 22(3):846, 2022.
  30. [30]Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
  31. [31]Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625–632, 2005.
  32. [32]K Nikhileswar, D Vishal, L Sphoorthi, and S Fathimabi. Suicide ideation detection in social media forums. In 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), pages 1741–1747. IEEE, 2021.
  33. [33]Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, 2019.arxiv.org/abs/1904.01685
  34. [34]Teodora Popordanoska, Raphael Sayer, and Matthew B. Blaschko. A consistent and differentiable Lp canonical calibration error estimator. In Advances in Neural Information Processing Systems, volume 35, pages 7933–7946, 2022.
  35. [35]Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/1909.13584
  36. [36]Stephanie Rude, Eva-Maria Gortner, and James Pennebaker. Language use of depressed and depression-vulnerable college students. Cognition & Emotion, 18(8):1121–1133, 2004.DOI
  37. [37]Irene Sánchez Rodríguez, John Bianchi, Fabio Pinelli, Folco Panizza, Emiliano Ricciardi, Pietro Pietrini, and Marinella Petrocchi. Understanding mental health discourse on reddit with transformers and explainability. Scientific Reports, 16(1):6796, 2026.
  38. [38]Catherine A Shipp and Ludmila I Kuncheva. Relationships between combination methods and measures of diversity in combining classifiers. Information fusion, 3(2):135–148, 2002.DOI
  39. [39]Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017.arxiv.org/abs/1703.01365
  40. [40]Ilija Tavchioski, Marko Robnik-Šikonja, and Senja Pollak. Detection of depression on social networks using transformers and ensembles. arXiv preprint arXiv:2305.05325, 2023.DOI
  41. [41]Elsbeth Turcan and Kathy McKeown. Dreaddit: A Reddit dataset for stress analysis in social media. In Eben Holderness, Antonio Jimeno Yepes, Alberto Lavelli, Anne-Lyse Minard, James Pustejovsky, and Fabio Rinaldi, editors, Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97–107, Hong Kong, November 2019. Association for Computational Linguistics.arxiv.org/abs/1911.00133
  42. [42]Kimberly A Van Orden, Tracy K Witte, Kelly C Cukrowicz, Scott R Braithwaite, Edward A Selby, and Thomas E Joiner Jr. The interpersonal theory of suicide. Psychological review, 117(2):575, 2010.
  43. [43]Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382–6388, 2019.

Paper details

vlnc:W_6yjtghtf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude, ChatGPT

View statement
Looking for related papers…
Contents