betaFeedback

WorksComputer Science

MoEDDI: Semantic Mixtures of Experts for Long Tailed Drug Drug Interaction Prediction

Abstract

Descriptor-based drug–drug interaction (DDI) models typically apply a single global predictor despite heterogeneous physicochemical evidence and severely long-tailed labels. We introduce MoEDDI (Mixture-of-Experts for Drug–Drug Interaction prediction), a chemically organized mixture-of-experts that routes overlapping descriptor families using low-dimensional statistics and combines their sparse top-22 mixture with a global T-DDI pathway through a residual logit correction. The residual parameterization can exactly preserve a supplied T-DDI checkpoint at initialization, although the evaluated models are trained from scratch. Across complete test partitions with 178, 86, and 92 classes, MoEDDI improves macro-F1 over reproduced seed-2026 T-DDI baselines from 0.8284 to 0.8485, 0.4729 to 0.6536, and 0.1807 to 0.4671. On the 3,780-feature corpus, the lowest-support group (89 classes with fewer than 50 test examples) shows the largest mean classwise gain (+3.30 F1 points). Calibration improves on one corpus but worsens on two. These single-seed results support semantic conditional specialization while motivating matched ablations and distribution-shift evaluation.

Introduction

Computational DDI prediction can prioritize candidate interactions for pharmacological review, but benchmark performance may not transfer to unseen drugs or chemical series. Long-tailed labels, drugs shared across data partitions, and poorly calibrated confidence further complicate translation. Evaluation should therefore consider minority-class performance, calibration, and distribution shift rather than aggregate accuracy alone.

Recent graph models, including SSI-DDI, GMPNN-CS, SA-DDI, and DSN-DDI, learn interaction-relevant molecular representations directly [7, 8, 12, 5]. Descriptor-based models remain attractive because their inputs are deterministic, cacheable, and auditable. T-DDI reports 0.9434 accuracy and 0.8452 macro-F1 using an ensemble on 868,069 DrugBank-derived pairs spanning 178 interaction types [4]. Its reported high-confidence macro-F1 of 0.8992 covers only 87.91% of the test set and is therefore not a complete-test comparison.

In the released T-DDI data used here, no categorical columns are provided. Because TabTransformer attention operates over categorical embeddings [3], prediction is effectively performed by the numerical MLP pathway. For D=3,780D = 3{,}780 and C=178C = 178, this pathway contains approximately 87.1M parameters and processes every pair through the same global predictor.

We ask whether chemically structured conditional specialization can complement this global pathway. MoEDDI partitions descriptors into overlapping semantic expert views, uses a low-bandwidth router for sparse top-2 mixing, and adds the resulting expert representation as a residual correction. Its residual head can be zero-initialized so that a supplied T-DDI checkpoint is reproduced exactly before adaptation.

Our contributions are: (i) overlapping chemically named descriptor experts; (ii) low-bandwidth top-2 routing using family-level statistics; (iii) a baseline-preserving residual parameterization; and (iv) complete-test evaluation across three corpora with macro-F1, long-tail, calibration, and router-utilization analyses. Results use one run per protocol on pair partitions that are neither drug- nor scaffold-disjoint; we therefore claim architectural promise rather than definitive state-of-the-art or translational superiority.

Related Work

DDI prediction. DeepDDI established large-scale multiclass DDI prediction from structural similarity profiles [9]. More recent models increasingly operate on molecular graphs or learned substructures. SSI-DDI models interactions between drug substructures [7], GMPNN-CS combines graph message passing with chemical-substructure information [8], SA-DDI learns substructure-aware representations [12], and DSN-DDI explicitly studies generalization to unseen drugs [5]. These methods demonstrate the value of representation learning, but fixed molecular descriptors remain useful when deterministic preprocessing, caching, and feature-level auditability are desirable. MoEDDI focuses on this descriptor setting and asks whether known feature families can provide a useful inductive bias for conditional computation.

Mixtures of experts and tabular models. Sparse mixture-of-experts models conditionally allocate model capacity across inputs [10], while later work highlights routing imbalance and instability as practical concerns [1]. MoEDDI differs from conventional MoE designs in that expert identity is not learned from an arbitrary partition of the hidden space: each expert corresponds to a predefined descriptor family, and overlapping features may participate in more than one expert. The router is deliberately restricted to family-level summary statistics rather than the full descriptor vector. This separates the routing decision from high-dimensional prediction and provides a simple way to audit which expert families are selected for each pair. Unlike computationally sparse MoE systems, the present implementation evaluates all experts before applying top-2 weights; sparsity therefore induces specialization rather than inference-time FLOP savings.

Methods

Semantic expert views. Figure Figure 1 summarizes the architecture and its notation. Given pair features x∈RD\mathbf{x} \in\mathbb{R}^{D} and DDI label yy, the model estimates p(y∣x)p(y \mid\mathbf{x}). Identifiers, names, and SMILES are excluded. On DDI2025, descriptor names define seven overlapping expert sets SeS_e: EState_VSA, LabuteASA, MR_VSA, MTPSA, PEOE_VSA, SlogP_VSA, and VSA_Estate. Cross-family descriptors may belong to multiple experts, and labels do not define the partition. Each expert produces

MoEDDI on DDI2025

Figure 1. MoEDDI on DDI2025. Semantic experts provide a pair-dependent residual correction to the global T-DDI pathway.

he=ϕe(xSe)∈Rdh,(1)\mathbf{h}_e = \phi_e(\mathbf{x}_{S_e}) \in\mathbb{R}^{d_h}, \tag*{(1)}

where ϕe\phi_e is an independent LayerNorm–MLP; the main model uses dh=384d_h = 384.

Design rationale. The architecture reflects three design choices. First, expert sets are allowed to overlap because several descriptor names encode more than one chemical family; forcing a disjoint partition would make expert membership depend on an arbitrary tie-breaking rule. Second, routing uses only two moments per family rather than the complete descriptor vector. The router therefore determines which representation should be emphasized from coarse family-level evidence, while the experts retain responsibility for high-dimensional feature extraction. Third, the global pathway is preserved on DDI2025 so that expert specialization acts as a correction rather than requiring the model to relearn an already strong global representation. This construction also permits exact recovery of a pretrained baseline at initialization when the residual output layer is zero-initialized.

Low-bandwidth top-2 routing. For each family, the router receives only signed-log mean and RMS summaries:

xˉe=1ne∑j∈Sexj,me=sgn⁡(xˉe)log⁡(1+∣xˉe∣),re=log⁡(1+1ne∑j∈Sexj2).(2)\bar{x}_e = \frac{1}{n_e}\sum_{j\in S_e} x_j,\qquad m_e = \operatorname{sgn}(\bar{x}_e)\log(1+|\bar{x}_e|),\qquad r_e = \log\left(1+\sqrt{\frac{1}{n_e}\sum_{j\in S_e}x_j^2}\right). \tag*{(2)}

A two-layer MLP maps s=[m1,r1,…,mE,rE]s=[m_1,r_1,\ldots,m_E,r_E] to logits aa. Only the two largest logits are retained:

qe=exp⁡(ae)1[e∈Top2⁡(a)]∑jexp⁡(aj)1[j∈Top2⁡(a)],hmix=∑eqehe.(3)q_e = \frac{\exp(a_e)\mathbf{1}[e\in\operatorname{Top2}(a)]}{\sum_j\exp(a_j)\mathbf{1}[j\in\operatorname{Top2}(a)]},\qquad h_{\mathrm{mix}}=\sum_e q_eh_e. \tag*{(3)}

Dense pre-mask probabilities regularize router balance; sparse qeq_e values are used for utilization analysis.

Baseline-preserving residual. Let b=gθhid(x)\mathbf{b}=g_{\theta}^{\mathrm{hid}}(\mathbf{x}) and zg=gθcls(b)\mathbf{z}_g=g_{\theta}^{\mathrm{cls}}(\mathbf{b}) denote the global hidden representation and logits. MoEDDI predicts

zr=fψ([hmix;Pb]),z=zg+zr.(4)\mathbf{z}_r=f_{\psi}([h_{\mathrm{mix}};P\mathbf{b}]),\qquad\mathbf{z}=\mathbf{z}_g+\mathbf{z}_r. \tag*{(4)}

Zero-initializing the final layer of fψf_{\psi} gives zr=0\mathbf{z}_r=0, so a supplied T-DDI checkpoint is reproduced exactly at initialization. This is a parameterization property; the evaluated DDI2025 model is trained from scratch. The two lower-dimensional corpora instead use DrugA, DrugB, and Generalist views without the large residual backbone.

Training objective. Class imbalance is addressed with focal loss (γ=1\gamma=1) [6]. The DDI2025 objective supervises combined, residual, and global logits while regularizing router balance and logit magnitude:

L=Lfocal(z,y)+0.3Lfocal(zr,y)+0.3Lfocal(zg,y)+10−2(E∑epˉe2−1)+10−3Ei[log⁡∑eexp⁡(aie)]2,(5)\begin{aligned} \mathcal{L}=\mathcal{L}_{\mathrm{focal}}(\mathbf{z},y)+0.3\mathcal{L}_{\mathrm{focal}}(\mathbf{z}_r,y)+0.3\mathcal{L}_{\mathrm{focal}}(\mathbf{z}_g,y) \\ &+10^{-2}\left(E\sum_e\bar{p}_e^2-1\right)+10^{-3}\mathbb{E}_i\left[\log\sum_e\exp(a_{ie})\right]^2, \tag*{(5)} \end{aligned}

where EE is the number of experts, pˉe\bar{p}_e is the batch mean of the dense (pre-mask) router probability softmax⁡(a)e\operatorname{softmax}(a)_e, and aiea_{ie} is the router logit of expert ee for pair ii. Checkpoints are selected by validation macro-F1 unless otherwise specified.

Experimental Setup

Datasets. We evaluate three fixed train/validation/test partitions (Table 1). DDI2025 is the DrugBank-derived corpus used by T-DDI [11, 4]; DeepDDI-86 uses a 100-dimensional pair representation for 86 interaction classes [9]; FDA-DDI-92 contains 217 descriptors per drug, yielding 434 pair features. Preprocessing statistics are fit on training data only, checkpoint selection uses validation data, and every test pair is evaluated without confidence filtering.

CorpusTrainValidationTestClassesFeaturesMoEDDI views
DDI2025520,841173,614173,6141783,780seven chemical families
DeepDDI-86115,37238,46238,45086100DrugA, DrugB, Generalist
FDA-DDI-92239,41279,80579,80492434DrugA, DrugB, Generalist

Table 1. Corpora and fixed partitions.

Baselines and evaluation. T-DDI-MF uses seed 2026 and validation macro-F1 selection; T-DDI-Acc is an available seed-42 accuracy-selected checkpoint. Both use the active numerical T-DDI MLP. Runs use mixed precision, batch size 256, learning rate 10−410^{-4}, gradient clipping at 1, and focal loss. DDI2025 uses raw inputs for both models; on the two smaller corpora, MoEDDI uses training-set standardization whereas T-DDI retains raw inputs, so those comparisons are not strictly matched ablations. We report accuracy, macro-F1, weighted F1, top-3, top-5, and 15-bin ECE, with macro-F1 as the primary metric. DDI2025 classes are additionally grouped by test support (<50< 50, 50–499, ≥500\ge500).

Results and Discussion

Overall performance. Table 2 reports complete held-out test performance. Under the seed-2026, macro-F1-selected protocols, MoEDDI improves macro-F1 from 0.8284 to 0.8485 on DDI2025, from 0.4729 to 0.6536 on DeepDDI-86, and from 0.1807 to 0.4671 on FDA-DDI-92. Accuracy increases by 0.94, 15.43, and 29.71 percentage points, respectively. Classwise win/tie/loss counts relative to T-DDI-MF are 94/44/40, 67/15/4, and 79/9/4.

Table 2: Complete held-out test performance. T-DDI-Acc denotes the T-DDI checkpoint selected by validation accuracy, whereas T-DDI-MF denotes the checkpoint selected by validation macro-F1. The two checkpoints also use different random seeds, as described in the text. Arrows indicate the preferred direction of each metric. Best within each corpus is bold.

CorpusModelAccuracy ↑\uparrowMacro-F1 ↑\uparrowWeighted F1 ↑\uparrowTop-3 ↑\uparrowTop-5 ↑\uparrowECE ↓\downarrow
DDI2025T-DDI-Acc0.92190.80040.92150.99340.99800.0044
T-DDI-MF0.92690.82840.92640.99300.99780.0213
MoEDDI0.93630.84850.93610.99460.99820.0440
DeepDDI-86T-DDI-Acc0.53770.25400.50210.84740.92820.0534
T-DDI-MF0.66180.47290.65540.91850.96560.1122
MoEDDI0.81610.65360.81340.97490.99120.0704
FDA-DDI-92T-DDI-Acc0.31980.09970.21980.54540.67050.0176
T-DDI-MF0.33710.18070.24950.59290.70760.0206
MoEDDI0.63420.46710.60920.91530.97280.0475

Table 2. Complete held-out test performance. T-DDI-Acc denotes the T-DDI checkpoint selected by validation accuracy, whereas T-DDI-MF denotes the checkpoint selected by validation macro-F1. The two checkpoints also use different random seeds, as described in the text. Arrows indicate the preferred direction of each metric. Best within each corpus is bold.

T-DDI-MF also exceeds the available T-DDI-Acc checkpoint in macro-F1, although seed and checkpoint criterion change simultaneously and therefore prevent a controlled selection ablation. On DDI2025, the published T-DDI ensemble reports 0.9434 accuracy and 0.8452 macro-F1 [4], compared with 0.9363 and 0.8485 for the single MoEDDI model. Because the training and ensemble protocols differ, this indicates competitiveness rather than controlled superiority. The published high-confidence macro-F1 of 0.8992 is likewise not directly comparable because it excludes 12.09% of test pairs.

Long-tail performance. The DDI2025 improvement is concentrated in low-support classes. Among 89 classes with fewer than 50 test examples, mean per-class F1 increases from 0.728 to 0.761 (+3.30 points), compared with +0.70 points for classes with 50–499 examples and +0.80 points for classes with at least 500 examples. This pattern is consistent with conditional specialization being particularly useful in the label tail, although it does not establish a pharmacological mechanism.

Cross-corpus interpretation. The three corpora test related but not identical MoEDDI instantiations. DDI2025 retains the 87.1M-parameter T-DDI global pathway and learns an expert residual, providing the cleanest evidence that conditional specialization can complement a strong global predictor. The lower-dimensional corpora instead use DrugA, DrugB, and Generalist views without the large backbone. Their larger gains are encouraging, but MoEDDI also uses training-set standardization while T-DDI retains raw inputs; consequently, these margins cannot yet be attributed solely to routing. A standardized T-DDI control and matched schedules are required.

Routing and calibration. DDI2025 routing remains distributed rather than collapsing to one expert. Mean sparse router weights on the test set are 25.7% MTPSA, 24.4% MR_VSA, 21.2% LabuteASA, 19.1% VSA_EState, and 8.2% EState_VSA, whereas PEOE_VSA (0.28%) and SlogP_VSA (1.07%) are rarely selected. The smaller-corpus models favor the Generalist view (57.8% on DeepDDI-86 and 59.8% on FDA-DDI-92), and the DrugA/DrugB asymmetry (25.2%/16.9% and 27.9%/12.3%) motivates an explicit pair-swap test. These are behavioral summaries, not chemical attributions.

Calibration does not uniformly improve with discrimination. Relative to T-DDI-MF, MoEDDI reduces ECE on DeepDDI-86 (0.1122 to 0.0704) but increases it on DDI2025 (0.0213 to 0.0440) and FDA-DDI-92 (0.0206 to 0.0475). This reinforces that calibration should be evaluated separately from class-sensitive performance [2].

Limitations and translation. The current evidence is single-seed and not fully matched in preprocessing, optimization, or capacity. Stronger evaluation requires matched multi-seed runs, a capacity-matched residual MLP, dense/uniform and shuffled-family controls, warm-start versus scratch training, and unseen-drug, scaffold-disjoint, and temporal splits. Pair-split performance should not be interpreted as evidence of clinical safety: MoEDDI predicts curated interaction labels rather than patient-specific harm, dose effects, mechanism, or severity. Its intended role is prioritization for expert or experimental review, with future deployment requiring calibration, abstention, and monitoring under distribution shift.

Conclusion

MoEDDI introduces chemically structured conditional specialization into descriptor-based DDI prediction through overlapping semantic experts, low-bandwidth top-2 routing, and a residual parameterization capable of preserving a supplied T-DDI predictor at initialization. Across three complete test partitions, MoEDDI achieves higher macro-F1 than reproduced seed-2026 T-DDI baselines, including an increase from 0.8284 to 0.8485 on DDI2025, with the largest mean classwise gain in the lowest-support group. These results provide preliminary evidence that semantic mixtures can complement global descriptor models, while matched ablations and shift-aware evaluation remain necessary before stronger translational claims.

Code and Model Availability

Source code, trained models, and implementation resources for MoEDDI are publicly available. Source code: https://github.com/tommyngx/MoE_DDI Models and weights: https://huggingface.co/TommyNgx/MoE_DDI/

References

  1. [1]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  2. [2]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321–1330, 2017.arxiv.org/abs/1706.04599
  3. [3]Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.arxiv.org/abs/2012.06678
  4. [4]Quang-Hien Kha, Duc-Quang-Anh Nguyen, Phi Pham Van Hoang, Uyen Khoi-Minh Huynh, Khoa D. Pham, et al. Robust prediction of drug interactions using chemical descriptors. npj Digital Medicine, 2026. doi: 10.1038/s41746-026-03025-2.DOI
  5. [5]Ziming Li, Shichao Zhu, Bin Shao, Xiangxiang Zeng, Tong Wang, and Tie-Yan Liu. DSN-DDI: an accurate and generalized framework for drug–drug interaction prediction by dual-view representation learning. Briefings in Bioinformatics, 24(1):bbac597, 2023. doi: 10.1093/bib/bbac597.
  6. [6]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, pages 2999–3007, 2017. doi: 10.1109/ICCV.2017.324.DOI
  7. [7]Arnold K. Nyamabo, Hui Yu, and Jian-Yu Shi. SSI-DDI: substructure–substructure interactions for drug–drug interaction prediction. Briefings in Bioinformatics, 22(6):bbab133, 2021. doi: 10.1093/bib/bbab133.
  8. [8]Arnold K. Nyamabo, Hui Yu, Zun Liu, and Jian-Yu Shi. Drug–drug interaction prediction with learnable size-adaptive molecular substructures. Briefings in Bioinformatics, 23(1):bbab441, 2022. doi: 10.1093/bib/bbab441.
  9. [9]Jae Yong Ryu, Hyun Uk Kim, and Sang Yup Lee. Deep learning improves prediction of drug–drug and drug–food interactions. Proceedings of the National Academy of Sciences, 115(18):E4304–E4311, 2018. doi: 10.1073/pnas.1803294115.
  10. [10]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.arxiv.org/abs/1701.06538
  11. [11]David S. Wishart, Yannick D. Feunang, An C. Guo, Elvis J. L. Lo, Ana Marcu, Jason R. Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat N. Sayeeda, et al. Drugbank 5.0: a major update to the drugbank database for 2018. Nucleic Acids Research, 46(D1):D1074–D1082, 2018. doi: 10.1093/nar/gkx1037.
  12. [12]Ziduo Yang, Weihe Zhong, Qiuqie Lv, and Calvin Yu-Chian Chen. Learning size-adaptive molecular substructures for explainable drug–drug interaction prediction by substructure-aware graph neural network. Chemical Science, 13(29):8693–8703, 2022. doi: 10.1039/D2SC02023H.

Paper details

vlnc:W_8ut23bgp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents