WorksComputer Science
Syn2DRE: Synergistic Dual Teacher-Student Framework for Document-level Relation Extraction
Abstract
Document-level Relation Extraction (DocRE) aims to identify relations between entities across a document but typically relies on costly large-scale annotations. While pseudo-labelling enables the use of low-quality data, existing methods often suffer from inefficiency and confirmation bias. To address these challenges, we propose Syn2DRE, a semi-supervised framework that combines self-supervised learning with cross-supervision in a lightweight teacher-student architecture to produce more reliable pseudo-labels. Additionally, we introduce a unified end-to-end training strategy to further improve efficiency of the framework. Experiments on multiple benchmarks show that Syn2DRE consistently outperforms prior state-of-the-art semi-supervised DocRE methods.
Introduction
Relation Extraction (RE) identifies semantic relationships between entity pairs and is fundamental to applications such as Question Answering ((Xu et al., 2016)). Early work focuses on sentence-level contexts, but many real-world relations span multiple sentences; for example, over 40% of relational facts in Wikipedia require cross-sentence reasoning ((Yao et al., 2019)). This limitation has led to growing interest in Document-level Relation Extraction (DocRE) ((Xu et al., 2021b; Zhang et al., 2021a; Zhou et al., 2021; Tan et al., 2022a)), which models relations across entire documents.
DocRE methods typically rely on deep neural networks, which require substantial amounts of annotated data for effective training. However, creating high-quality DocRE datasets is both time-consuming and error-prone ([14] (Tan et al. 2022b)). For example, the widely-used DocRED dataset ((Yao et al., 2019)) contains an average of 19.5 entities per document, resulting in roughly 38,000 candidate triples to be annotated per document when considering 97 predefined relation types ((Tan et al., 2022b)).
To reduce reliance on manual annotation, several studies ([9] (Tan et al. 2023; Sun et al. 2023)) have explored pseudo-labelling ([15]) – an approach that generates synthetic labels for unlabelled or noisily labelled data. This typically involves an iterative loop: generating pseudo-labels, incorporating them into the training set, retraining the model, and repeating the cycle. However, such methods are prone to confirmation bias ([1]), where low-quality pseudo-labels are reinforced over iterations, harming overall performance. Some recent works ([9] (Tan et al. 2022a; Tan et al. 2023)) use pseudo-labels without evaluating their reliability, further exacerbating this issue. To address this, Sun et al. (2023) proposed uncertainty-based selection to improve pseudo-label quality. While effective, their method relies on a single model and requires multiple rounds of training and denoising, significantly increasing computational cost and leaving confirmation bias largely unaddressed.
In this work, we introduce the Synergistic Dual Teacher-Student approach (Syn2DRE), a novel semi-supervised training framework for DocRE that generates high-quality pseudo-labels in a single training run, eliminating the need for iterative denoising. To reduce training overhead, the teacher model is derived from the student using Exponential Moving Average (EMA) to produce more stable targets ((Tarvainen and Valpola, 2017)).
While EMA-based pseudo-label generation enhances performance, it may still suffer from confirmation bias. To address this issue, we propose an unsupervised synergistic cross-supervision strategy inspired by Co-teaching ([3, 12] (Yu et al. 2019)). Specifically, two distinct teacher-student pairs exchange highly confident pseudo-labels on instances where their predictions diverge, promoting diverse learning signals and mitigating confirmation bias. Confidence is determined by evaluating the variance of prediction logits across multiple forward passes. Finally, we present a unified training framework that integrates both self-supervised learning and cross-supervision cohesively.
We evaluate Syn2DRE on three general-domain DocRE benchmark datasets – DocRED (Yao et al., 2019), Re-DocRED (Tan et al., 2022b), and HacRED [4] – as well as two biomedical datasets, CDR [16] and GDA (Wu et al., 2019). Our framework consistently achieves state-of-the-art performance across these datasets.
The main contributions of this work are as follows:
Novel semi-supervised DocRE framework: We propose Syn2DRE, a dual teacher-student framework that leverages both unlabelled and noisy data to boost DocRE performance.
Self-supervised with Unsupervised Synergistic Cross-supervision: Syn2DRE combines EMA-based Pseudo-labelling step with Unsupervised Synergistic Cross-supervision strategy to reduce confirmation bias – marking the first such integration in DocRE.
Unified training process: A unified training procedure integrates both steps in an end-to-end manner to enhance performance in DocRE.
Superior empirical results: Extensive experiments on popular DocRE datasets show that Syn2DRE significantly outperforms existing semi-supervised baselines on standard DocRE benchmarks.
Problem Formulation
Let be a document with words and a set of entities . Each entity has a set of mentions , i.e., phrases referring to the same entity. In the DocRE task, the goal is to predict a subset of relations from a predefined set that hold between entity pairs , where . For brevity, we denote such pairs as . A relation is considered present between and if it is expressed between any of their mentions; these are labelled as positive pairs. If no relation exists between any mention pair, the entity pair is labelled NA (i.e., negative) relation. Note that positive pairs may express multiple relations in .
Methodology
Framework Overview
Motivation. We propose a robust semi-supervised learning framework for the DocRE task, which leverages the abundant unlabelled data for model training. Our approach is grounded in the following three key observations in the DocRE task: (1) obtaining large-scale, high-quality labelled data is prohibitively expensive; (2) the potential of unlabelled data to enhance performance remains under-explored; and (3) existing pseudo-labelling methods often fail to assess label quality or rely on iterative processes that can amplify noise, ultimately degrading model performance.
To address these challenges, we introduce Syn2DRE, a novel framework featuring dual teacher-student networks trained in parallel. As shown in Figure 1, during the Self-supervised step, teacher networks generate pseudo-labels for unlabelled instances, while student networks make predictions on the same data (step ❶). Pseudo-labels with high confidence are selected (step ❷) and used to update the corresponding student networks (steps ❸, ❹). The improved associated student networks then inform updates to the teacher networks (step ❺). To reduce confirmation bias, we periodically apply the proposed unsupervised synergistic cross-supervision mechanism to further refine the opposite student models (step ❻).
Figure 1. An overview of the proposed Syn2DRE method.
Teacher-Student Architectures
Student Networks.
To promote complementary learning signals during cross-supervision, we adopt KD-DocRE (Tan et al., 2022a) and UGDRE (Sun et al., 2023) as our student networks. These lightweight models differ in both entity pair representation and training objectives. KD-DocRE employs Axial Attention [13] and a composite loss function that integrates Adaptive Thresholding Loss (ATL) (Zhou et al., 2021) with Focal Loss (Lin et al., 2017) to better handle long-tail relations. In contrast, UGDRE enhances entity representations through token-level graph construction and follows the original ATL objective. Full architectural details are provided in Appendix § 7.
Teacher Networks.
Each teacher network in Syn2DRE mirrors the architecture of its associated student. Their parameters are updated using EMA, which smooths the weights over time to yield more stable pseudo-labels. Initialisation details are discussed in § 3.5.
Self-supervised Step
This section details the Self-supervised process. Pseudo-label generation and refinement (steps ❶ and ❷) are described in § 3.3.1 and § 3.3.2, while the teacher update mechanism (step ❺) is presented in § 3.3.3.
Motivation. The Self-supervised step in Syn2DRE aims to reduce the prolonged training time of prior methods by generating high-quality pseudo-labels in a single pass via teacher-student networks, avoiding iterative denoising.
Pseudo-label Generation
In the Self-supervised step, each teacher network generates pseudo-labels for unlabelled data to supervise its associated student. To ensure label reliability, we adopt Monte Carlo (MC) dropout (Gal and Ghahramani, [10]). Specifically, we perform several stochastic forward passes to compute prediction logits for each relation and the threshold class (Zhou et al., 2021), representing the NA relation, for a given entity pair . The final prediction score is obtained by averaging over those forward passes:
where is the logit score for relation in the pass, is the number of stochastic forward passes. The same averaging is applied to the class.
Pseudo-labels are then assigned according to the following rule:
where denotes the averaged score for the class.
Pseudo-labels Refinement
Neural networks often exhibit poor calibration, producing overconfident yet incorrect pseudo-labels ([11] (Rizve et al. 2021; Si et al. 2024)). To address this, we propose an unsupervised refinement strategy that prioritises reliable pseudo-labels using uncertainty,
Uncertainty Estimation. The uncertainty for each pseudo-relation of an entity pair is quantified as the variance of its logits across multiple forward passes, defined as:
Here, is the mean prediction logit across stochastic forward passes. We compute uncertainty for all entity pairs, including both positive and negative ones. If a positive pair has multiple pseudo-relations, its overall uncertainty is averaged across these relations:
where denotes the set of pseudo-relations. We rank all positive pairs by increasing uncertainty and select the top as reliable training samples; likewise, we select the top negative pairs. To address the class imbalance issue – where 97% of entity pairs are negative – we set and .
The selected pseudo-labels are then used to update the student networks with standard training process.
Teacher Network Update
Once the student networks are trained, we update the teacher models using EMA of student parameters, as follows:
Here, and denote the teacher and student parameters at iteration , and is the smoothing factor. This EMA update can also be reformulated as:
where is the learning rate. As (e.g., 0.995), updates incorporate small, weighted student gradients, forming a conservative, ensemble-like strategy that reduces bias and improves pseudo-label reliability (Zhang et al., 2021b). Derivation of the Eq. (5) is detailed in the Appendix § 9.
Unsupervised Synergistic Cross-supervision Step
Motivation. In the Self-supervised step, teacher-generated pseudo-labels help students improve. However, using the same architecture for both models may perpetuate confirmation bias, leading to sub-optimal performance. To address this, we introduce an Unsupervised Synergistic Cross-supervision training strategy (step ⑥).
To reduce confirmation bias, a model should learn from signals it cannot easily infer on its own. Motivated by this, we co-train two DocRE models with different architectures that teach each other, leveraging complementary graph-based and transformer-based representations and distinct objectives for class imbalance, as described in § 3.2.
Procedure
During training, each teacher reuses samples with the lowest uncertainty, as defined in § 3.3.2. From these, disagreeing pseudo-labels are exchanged, such that teacher 2 supervises student 1 and teacher 1 supervises student 2.
Formally, let and denote the teacher-student pairs. We extract subsets and containing the pseudo-labelled samples from each teacher with the lowest uncertainty. From these, we identify the sets of disagreed samples and , where:
Objective.
The associated soft pseudo-labels of are used to supervise the opposite students. We adopt soft labels (probability distributions over ) to preserve uncertainty and ensure stable training. The cross-supervision loss is defined as:
where and and denote the predictive distributions from the student and teacher, respectively.
Unified Training Procedure
We propose a unified training procedure to improve the framework’s efficiency, consisting of two phases: (1) pre-training and (2) fine-tuning.
Pre-training Phase
During the pre-training phase, we initialise the student networks and using distantly supervised DocRE dataset with standard training process to improve generalisation. After pre-training, we copy their parameters to the respective teacher networks to facilitate a strong initialisation for both teacher and student networks, as follows:
Fine-tuning Phase
In the fine-tuning phase, the teacher-student pairs alternate between the Self-supervised step and the Unsupervised Synergistic Cross-supervision step. To ensure robust initialisation for these steps, we first perform a warm-up round using human-annotated data for epochs.
After warming up, the Self-supervised step begins with refined pseudo-labels, and the Unsupervised Synergistic Cross-supervision step is conducted periodically every steps to avoid mutual overfitting due to residual label noise. The overall loss for the this step for each student network is:
where is the loss value of the Self-supervised Step (§ 3.3) and the hyper-parameter controls the contribution of .
To further combat confirmation bias, we also include supervised loss on human-annotated data for each student network. The final training objective becomes:
where controls the contribution of .
Experiments
In this section, we present a series of experiments to assess the effectiveness of our proposed framework. We begin by outlining the experimental setup, followed by empirical analyses aimed at addressing the following research questions (RQs):
RQ1: How effective is the overall framework in different datasets?
RQ2: How well does the method handle long-tailed relation distributions?
RQ3: How effective is each of the proposed components?
RQ4: How resource-efficient is the proposed approach compared to existing baselines?
RQ5: How is Syn2DRE effective when using with other DocRE architectures?
Datasets
We conducted evaluations on five widely used publicly available benchmarks for the DocRE task. More details are shown in Table 11 in Appendix § 8.
| Datasets | # Documents |
| DocRED-Train | 3,053 |
| DocRED-Dev | 1,000 |
| DocRED-Test | 1,000 |
| DocRED Distantly Supervised | 101,873 |
| Re-DocRED-Train | 3,053 |
| Re-DocRED-Dev | 500 |
| Re-DocRED-Test | 500 |
| HacRED-Train | 6,231 |
| HacRED-Dev | 1,500 |
| HacRED-Test | 1,500 |
| CDR-Train | 500 |
| CDR-Dev | 500 |
| CDR-Test | 500 |
| GDA-Train | 23,353 |
| GDA-Dev | 5,839 |
| GDA-Test | 1,000 |
Table 11. Statistics of the datasets in experiments.
DocRED. DocRED contains 96 predefined relation types, curated from Wikipedia and Wikidata. It includes 3,053 training, 1,000 validation, and 1,000 test documents. A distantly supervised version with 101,873 documents was also released by Yao et al. (2019).
Re-DocRED. Re-DocRED is a refined variant of DocRED that addresses annotation incompleteness by adding previously overlooked relational facts. It consists of 3,053 training, 500 development, and 500 test documents.
HacRED. HacRED is a Chinese document-level relation extraction dataset designed to address challenging cases. It defines 27 difficult relation types and is split into 6,231/1,500/1,500 instances for training, validation, and test sets.
CDR. CDR is a human-annotated biomedical dataset containing 500 training documents. The task is to predict binary interactions between chemical and disease entities.
GDA. GDA is a large-scale biomedical dataset comprising 29,192 training articles. The task is to predict binary interactions between gene and disease entities. Dataset splits follow Christopoulou et al. [5] and Zhou et al. (2021).
Since HacRED, CDR, and GDA do not provide unlabelled data, we treat their validation sets as unlabelled and avoid pretraining on their original labels to ensure a fair evaluation.
Experiment Setup
Evaluation Metrics
We adopt the micro-averaged score as the primary evaluation metric across all five datasets. Following Yao et al. (2019), we also report the Ignored , which omits relational facts overlapping between the training and test sets for the DocRED and Re-DocRED datasets. Additionally, to assess model performance across relation frequency levels in Re-DocRED, we include Freq (for the 10 most frequent relations) and LT (for the remaining 86), as introduced by Tan et al. (2022b).
Baselines
We compare Syn2DRE with several competitive baselines that generate pseudo-labels to augment human-labeled data. KD-DocRE (Tan et al., 2022a) leverages knowledge distillation to produce soft pseudo-labels. UGDRE (Sun et al., 2023) uses uncertainty-aware, multi-phase training to produce hard pseudo-labels. VaeDiff-DocRE (Tran et al., 2025) creates pseudo-representations for entity pairs using data augmentation. DURE ([9]) generates pseudo-labels for unlabelled data via controllable text generation conditioned on relation triplets.
Implementation Details
We use BERT-base-cased (Devlin et al. [7]), XLM-RoBERTa-Large (Conneau et al. [6]), and SciBERT (Beltagy et al. [2]) as encoders, and train all models with AdamW (Loshchilov and Hutter (2019)), adopting official implementations for baselines. Pre-training follows original hyperparameters (Appendix § 7), while in fine-tuning, EMA rates are set to 0.995 (UGDRE) and 0.999 (KD-DocRE), starting after 10 steps and updating every 5 steps, with unchanged learning rates. Fine-tuning runs for 25 epochs on Re-DocRED and 7 on DocRED, sampling 16 unlabelled documents per step, with warm-up lasting 15 and 5 epochs, respectively. For cross-supervision, we set , , and . Our experiments were conducted on a server equipped with an NVIDIA H100 GPU (80GB).
Overall Performance
General-domain Datasets
DocRED and Re-DocRED. To address RQ1, we evaluate Syn2DRE on DocRED and Re-DocRED using the student architectures described in §3.2.1, with results in Table 1; all improvements are statistically significant () under the paired test of Dror et al. [8]. Results with other student architectures are reported in § 4.7.
| Model | DocRED | Re-DocRED | |||||||
| Dev | Test | Dev | Test | ||||||
| F | Ign F | F | Ign F | F | Ign F | F | Ign F | ||
| ATLOP (Zhou et al. (2021)) | 63.42 | 61.57 | 63.48 | 61.43 | 74.34 | 73.62 | 74.23 | 73.53 | |
| DocuNet (Zhang et al. (2021a)) | 64.35 | 62.66 | 64.00 | 61.93 | 76.22 | 75.50 | 75.35 | 74.61 | |
| NCRL (Zhou and Lee (2022)) | 63.87 | 61.65 | 63.45 | 60.98 | 75.85 | 74.91 | 75.90 | 75.00 | |
| CAST* (Tan et al. (2023)) | - | - | - | - | - | - | 74.67 | 73.32 | |
| VaeDiff-DocRE* (Tran et al. (2025)) | - | - | - | - | 75.89 | 74.96 | 75.07 | 74.13 | |
| SSR-PU (Wang et al. (2022)) | 63.00 | 60.43 | 62.66 | 59.80 | 76.83 | 75.57 | 76.23 | 74.96 | |
| KD-NA (Tan et al. (2022a)) | 64.17 | 62.18 | 64.12 | 61.77 | 76.14 | 75.25 | 76.00 | 75.12 | |
| KD-DocRE (Tan et al. (2022a)) | 64.81 | 62.62 | 64.76 | 62.56 | 76.47 | 75.30 | 76.14 | 74.97 | |
| DURE* (Feng and Lakshmanan [9]) | - | - | - | - | - | - | 76.84 | 75.32 | |
| UGDRE (Sun et al. (2023)) | 65.71 | 63.62 | 65.58 | 63.26 | 78.28 | 77.32 | 78.14 | 77.24 | |
| Syn2DRE (Ours) | 66.09±0.17 | 63.88±0.2 | 66.14† | 63.66† | 79.53±0.21 | 78.28±0.23 | 79.40±0.22 | 78.16±0.22 |
Table 1. Experimental results on DocRED and Re-DocRED datasets. Baseline results are from Sun et al. (2023). † indicates the best single run; * denotes results reported directly from original papers.
Syn2DRE significantly outperforms all baselines, particularly pseudo-labelling approaches, while also demonstrating greater robustness than augmentation-based methods. Compared to UGDRE, our method achieves consistent gains of +0.38/+0.56 F1 on the DocRED development/test sets, and larger improvements over KD-DocRE of +1.28/+1.38 F1. On Re-DocRED, Syn2DRE surpasses UGDRE by +1.25/+1.26 F1 (dev/test) and KD-DocRE by +3.06/+3.26 F1 (dev/test). In addition, Syn2DRE achieves notable gains over augmentation-based baselines, including +2.56 F1 over DURE (test) and +3.64/+4.33 F1 over VaeDiff-DocRE (dev/test), highlighting the effectiveness and robustness of our pseudo-label-driven strategy without relying on explicit text or embedding synthesis.
HacRED. From Table 2, Syn2DRE significantly outperforms state-of-the-art baselines on challenging non-English cases. In particular, Syn2DRE achieves average F1 gains of +0.61 and +0.23 over the baselines. These improvements could be further amplified with additional unlabelled documents, which are inexpensive to obtain in practice.
| Model | Test F1 |
| SSAN (Xu et al. (2021a)) | 61.00 |
| LSR (Nan et al. (2020)) | 68.41 |
| GAIN (Zeng et al. (2020)) | 76.09 |
| ATLOP (Zhou et al. (2021)) | 77.63 |
| KD-DocRE (Tan et al. (2022a)) | 78.75 |
| DAATF+ATL (Wei et al. (2025)) | 79.03 |
| DAATF+AFL (Wei et al. (2025)) | 79.42 |
| Syn2DRE (Ours) | 79.64±0.1 |
Table 2. Experimental results on HacRED dataset. Baseline results are from Wei et al. (2025).
Biomedical Datasets
To evaluate our framework in the biomedical domain, we conducted experiments on the CDR and GDA datasets using ATLOP and DocuNet as baselines, with results reported in Table 3.
| Model | Test F1 | |
| CDR | GDA | |
| SSAN (Xu et al., 2021a) | 68.7 | 83.7 |
| ATLOP (Zhou et al., 2021) | 69.4 | 83.9 |
| DGI (Wang et al., 2023) | 70.1 | 84.1 |
| SE-REPI (Liu et al., 2025) | 73.3 | 84.8 |
| DAATF (Wei et al., 2025) | 71.62 | 85.04 |
| DocuNet (Zhang et al., 2021a) | 76.3 | 85.3 |
| Syn2DRE (Ours) | 84.77 | 85.45 |
Table 3. Model performance on CDR and GDA datasets.
As shown in the table, Syn2DRE achieves state-of-the-art performance on both biomedical DocRE datasets. Notably, it improves the F1 score on the CDR dataset by 8.47 points over the best baseline. On the GDA dataset, a smaller yet consistent improvement of +0.15 in F1 score is also observed.
Performance on Long-tailed Relations
To address RQ2, we evaluate Syn2DRE on rare relations in Re-DocRED (Table 4). Syn2DRE outperforms generation-based baselines, achieving LT gains of +2.02 over VaeDiff-DocRE and +3.78 over DuRE. These results show that Syn2DRE robustly handles infrequent relations by generating high-quality pseudo-labels without explicit text generation, enabling scalable data augmentation for improved DocRE performance.
| Model | Freq | LT |
| ATLOP (Zhou et al., 2021) | 75.92 | 67.46 |
| SSR-PU (Wang et al., 2022) | 78.41 | 68.32 |
| CREST (Wei et al., 2021) | 77.93 | 68.68 |
| CAST (Tan et al., 2023) | 78.53 | 69.34 |
| VaeDiff-DocRE (Tran et al., 2025) | 79.18 | 71.12 |
| DuRE (Feng and Lakshmanan, [9]) | 79.81 | 72.88 |
| Syn2DRE | 82.42±0.26 | 74.90±0.23 |
Table 4. Experimental results in long-tailed settings on Re-DocRED’s test set.
Ablation Study
To address RQ3, we conduct an ablation study on Re-DocRED (Table 5), reporting results from the best-performing models.
| Model | Dev | Test | ||
| Ign | Ign | |||
| Syn2DRE | 79.93 | 78.74 | 79.50 | 78.35 |
| - w/o Cross-sup | 79.79 | 78.64 | 79.48 | 78.31 |
| - w/o ST | 79.52 | 78.45 | 79.28 | 78.21 |
| UGDRE | 78.28 | 77.32 | 78.14 | 77.24 |
| KD-DocRE | 76.47 | 75.30 | 76.14 | 74.97 |
Table 5. Ablation study on the Re-DocRED’s development set.
The Self-supervised step is critical, as removing it ("w/o ST") leads to clear drops (dev: −0.41/ − 0.29, test: −0.22/ − 0.14 in /Ign ).
The Disagreement-based Cross-supervision further improves performance by mitigating confirmation bias; removing it ("w/o Cross-sup") results in additional declines (dev: −0.14/ − 0.10), highlighting the importance of both components.
Efficiency Analysis
To answer RQ4, we compare training time and data usage of Syn2DRE against the previous state-of-the-art baselines and the performance of Syn2DRE with different sizes of human-annotated set under the Re-DocRED dataset.
Training time and Unlabelled Document Quantity
We reported the training time and data usage of Syn2DRE in Table 6.
| Model | Training Time | # of Unlabelled Documents |
| UGDRE | 55 hours | 101,873 |
| Syn2DRE (ours) | 26 hours | 30,480 |
Table 6. Comparison of Training Time and Unlabelled Data Usage Between Syn2DRE and UGDRE.
Our model completes training in just 26 hours, less than half the time required by UGDRE. Additionally, it achieves superior performance using only 30,480 unlabelled documents, in contrast to the 101,873 required by UGDRE. This demonstrates that our approach is significantly more resource-efficient, both in terms of computation and data consumption. Thus, Syn2DRE is effective and resource-efficient, scaling with unlabelled data for more generalisable gains in real-world DocRE.
Labelled Document Quantity
To further demonstrate the robustness of our method, we evaluate Syn2DRE with different sizes of human-annotated datasets during the warm-up stage, illustrated in Table 7.
| # of Labelled Docs | 100 | 200 | 300 | 400 | 600 |
| Dev F1 | 69.5 | 71.64 | 74.3 | 74.97 | 75.71 |
Table 7. Performance vs. Labelled Document quantity
The results show that Syn2DRE is not highly sensitive to the size of the annotated warm-up set: with only 400 documents (~13.1%), it achieves performance comparable to VaeDiff-DocRE (Tran et al., 2025), and with 600 documents, it surpasses KD-DocRE (Tan et al., 2022a). Even with 300 documents (), the model reaches 73.58 Dev F1 during warm-up and improves to 74.28 Dev F1 afterwards, demonstrating that strong performance can be achieved with minimal annotations, highlighting the practicality of our framework in low-resource DocRE scenarios.
Result with Different Models
To answer the RQ5, we further conducted experiments with 2 extra models, ATLOP (Zhou et al., 2021) and DocuNet (Zhang et al., 2021a), alongside KD-DocRE and UGDRE and report the results in Table 8:
| Models | Component | Dev F1 | Test F1 |
| ATLOP | 74.34 | 74.23 | |
| DocuNet | 76.22 | 75.35 | |
| Syn2DRE (ATLOP + DocuNet) | ATLOP | 75.41 | 75.12 |
| Syn2DRE (ATLOP + DocuNet) | DocuNet | 78.20 | 77.62 |
| Syn2DRE (ATLOP + KD-DocRE) | ATLOP | 76.18 | 75.99 |
| Syn2DRE (ATLOP + UGDRE) | ATLOP | 76.34 | 76.04 |
| Syn2DRE (DocuNet + KD-DocRE) | DocuNet | 78.54 | 77.84 |
| Syn2DRE (DocuNet + UGDRE) | DocuNet | 78.13 | 77.73 |
Table 8. Performance with different models. Results of ATLOP and DocuNet are reported from Sun et al. (2023).
As shown in Table 8, both ATLOP and DocuNet achieve notable gains within our framework, with stronger architectures yielding better performance. The results also show that co-training with a different architecture is beneficial: ATLOP performs better when paired with the graph-based UGDRE than with the transformer-based KD-DocRE, while DocuNet shows the opposite trend.
Related Works
Document-level Relation Extraction.
Document-level Relation Extraction (DocRE) identifies relations between entity pairs across sentences or entire documents. Existing methods mainly fall into two categories: transformer-based and graph-based. Transformer-based approaches ((Zhou et al., 2021; Tan et al., 2022a; Zhang et al., 2021a)) enhance entity representations using attention mechanisms in pre-trained language models, including localised context embeddings (Zhang et al., 2021a) and multi-hop reasoning via Axial Attention (Tan et al., 2022a). In contrast, graph-based methods ([5] (Zeng et al. 2020; Xu et al. 2021b)) model document structure explicitly by constructing heterogeneous graphs (e.g., mention, entity, and sentence nodes) and applying graph-based reasoning to improve relation prediction.
Synthetic Data for DocRE.
Aforementioned DocRE systems in § 5.1 typically require large amounts of high-quality labelled data, which is costly and prone to annotation errors. To mitigate this, recent work explores synthetic data to improve scalability. One direction focuses on data generation ([9] (Tan et al. 2022a; Sun et al. 2023; Tran et al. 2025)), such as generating pseudo-texts for long-tail relations (Feng and Lakshmanan [9]) or synthesising entity pair representations (Tran et al. (2025)). Another direction leverages pseudo-labelling, where teacher models annotate unlabelled data for student training (Tan et al., 2022a), with extensions incorporating uncertainty estimation via Monte Carlo dropout and multi-stage training (Sun et al., 2023).
However, these methods still have notable limitations. Tran et al. (2025)'s method continues to rely on substantial amounts of high-quality labelled data for strong performance. Meanwhile, methods proposed by Sun et al. (2023); Feng and Lakshmanan [9] are computationally expensive due to the large number of documents and entity pairs. For instance, Feng and Lakshmanan [9] generate pseudo-documents for each entity pair, while Sun et al. (2023) require multiple rounds of training and denoising. Consequently, Syn2DRE is proposed to reduce computational cost while producing more reliable pseudo-labels than prior methods.
Conclusion
In conclusion, we introduced Syn2DRE, a semi-supervised framework that effectively exploits unlabelled and weakly labelled data to advance DocRE. To reduce training overhead, we employed a lightweight Self-supervised strategy that generates reliable pseudo-labels on the fly. To combat confirmation bias, we proposed a Cross-supervision mechanism that jointly trains two teacher-student networks in a collaborative manner. These components are integrated into a unified, end-to-end training procedure. Experimental results and ablation studies confirm that Syn2DRE achieves state-of-the-art performance with improved efficiency.
Limitations
First, Syn2DRE still requires a non-trivial amount of labelled data, limiting its applicability in low-resource domains such as biomedicine; future work could improve performance under limited supervision or explore leveraging LLMs for pseudo-label generation. Second, its dual-network design introduces substantial GPU memory overhead, nearly doubling usage compared to UGDRE, motivating more memory-efficient alternatives. Third, the method is restricted to a binary dual-system setup, without exploring extensions to more than two models, which remains for future work. Finally, pseudo-label thresholds are fixed in this paper, and learning adaptive thresholds is left for future work.
Baseline Structures
UGDRE
Architecture. We briefly describe the architecture of the UGDRE model proposed by Sun et al. (2023).
UGDRE begins by constructing a document-level graph to model non-local dependencies among entities. In this graph, each node corresponds to a token in the document. Edges are established based on the following heuristic rules: (1) tokens within the same mention are connected to preserve mention semantics, (2) tokens from different mentions of the same entity are connected to model cross-mention interactions, and (3) tokens of entities co-occurring in the same sentence are connected to capture sentence-level interactions.
Graph inference is performed using Graph Convolutional Networks (GCNs), defined as:
where and are learnable parameters, is the contextual representation of the -th token, is the document graph’s adjacency matrix, and is a non-linear activation function.
Each token’s final representation is obtained by concatenating its contextual and graph-enhanced embeddings:
Entity representations are computed via logsumexp pooling over token representations. To predict the relation between a pair of entities , UGDRE applies a group bilinear operator to generate the probability distribution over all relation types.
Pre-training Phase Hyperparameters. Table 9 represents the hyper-parameters used when training UGDRE during Pre-training Phase (§ 3.5.1).
| Hyper-parameters | Value |
| Train batch size | 8 |
| Test batch size | 8 |
| Dropout rate | 0.25 |
| Warmup ratiors | 6% |
| Train epochs | 10 |
| Encoder learning rate | |
| Classifier learning rate | |
| Maximum gradient norm | 1.0 |
Table 9. Hyper-parameter settings for pre-training UGDRE (Sun et al., 2023).
KD-DocRE
Architecture. We provide an overview of the KD-DocRE model proposed by Tan et al. (2022a).
To highlight entity mentions, KD-DocRE inserts a special token “*” before and after each mention of an entity , and encodes the full document using a pre-trained language model. Entity representations are then computed via logsumexp pooling over the mention representations, where each mention is represented by the embedding of its starting “*” token. The representation of an entity pair is computed as follows:
| Hyper-parameters | Value |
| Train batch size | 8 |
| Test batch size | 8 |
| in AFL loss function | 0.5 |
| Warmup ratiors | 6% |
| Train epochs | 5 |
| Encoder learning rate | |
| Classifier learning rate | |
| Maximum gradient norm | 1.0 |
Table 10. Hyper-parameter settings for pre-training KD-DocRE (Tan et al., 2022a).
where and are trainable parameters, denotes concatenation, and is the localised context embedding for the entity pair (Zhou et al., 2021).
To capture structured interactions among entities, KD-DocRE introduces Axial Attention ([13]). Specifically, an entity-level table is constructed where each row and column corresponds to an entity in the document. Self-attention (Vaswani et al., 2017) is then applied independently along both the row (height) and column (width) axes to model inter-entity dependencies.
For training, Tan et al. (2022a) propose an enhanced version of Adaptive Thresholding Loss (Zhou et al., 2021) by incorporating Focal Loss (Lin et al., 2017):
where denotes the set of positive relations for the entity pair , is the predicted probability for relation , and is the predicted probability for the threshold relation .
Pre-training Phase Hyperparameters. Table 10 represents the hyper-parameters used when training KD-DocRE during Pre-training Phase (§ 3.5.1).
Experimental Details
Dataset Statistics
Table 11 provides the statistics of datasets used in this paper. All datasets are publicly available.
Derivation of Eq. (5)
In this section, we detail how the EMA update can be reorganised in the form of back-propagation. Equation (14) defines the update rule for the student parameters using standard back-propagation at the -th iteration, while Equation (15) presents the EMA update for the teacher parameters .
By substituting Equation (14) into Equation (15), we can express the teacher parameter at iteration as:
where (18).
Thus, the teacher parameter at the -th iteration becomes:
To express the EMA update in the form of back-propagation, consider the gradient step:
Then,
Substituting back, we obtain the back-propagation form of the EMA update:
Here, is the learning rate and is typically a small value, as is generally close to 1 (e.g., 0.995). This formulation reveals that the EMA update corresponds to applying a scaled, weighted sum of historical gradients from the student model.
The conservative and smoothing nature of this update makes EMA particularly effective for producing stable and reliable teacher models.
Hardware and software configurations
The models were implemented using PyTorch Paszke et al., 2019 and Huggingface Transformers Wolf et al., 2020. For EMA1 and Axial Attention2, we adopted publicly available repositories to construct teacher networks and implement KD-DocRE. Full software configuration details are provided in Table 12.
| Package | Version |
| axial_attention | 0.6.1 |
| ema_pytorch | 0.5.2 |
| numpy | 1.26.2 |
| opt_einsum | 3.3.0 |
| PyTorch | 2.1.2 |
| tqdm | 4.66.1 |
| Transformers | 4.32.1 |
Table 12. Package configurations of our experiments.
References
- [1]Eric Arazo, Diego Ortega, Paul Albert, Noel E O’Connor, and Kevin McGuinness. 2020. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference on neural networks (IJCNN), pages 1–8. IEEE.arxiv.org/abs/1908.02983
- [2]Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620.arxiv.org/abs/1903.10676
- [3]Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100.
- [4]Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxin Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. Hacred: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831.DOI
- [5]Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4925–4936.arxiv.org/abs/1909.00228
- [6]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451.arxiv.org/abs/1911.02116
- [7]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.arxiv.org/abs/1810.04805
- [8]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392.DOI
- [9]Yuxi Feng and Laks Lakshmanan. 2024. Dure: Dual contrastive self training for semi-supervised relation extraction. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 540–555.
- [10]Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR.arxiv.org/abs/1506.02142
- [11]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.arxiv.org/abs/1706.04599
- [12]Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31.arxiv.org/abs/1804.06872
- [13]Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180.arxiv.org/abs/1912.12180
- [14]Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-revise produce reliable annotations? an analysis on missing instances in docred. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6241–6252.
- [15]Dong-Hyun Lee and 1 others. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896. Atlanta.
- [16]Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016.DOI
