betaFeedback

WorksComputer Science

IsoAlign: Isoform-Aware DNA-Protein Alignment via Joint-Embedding Predictive Architectures

Abstract

Genomic foundation models (GFMs) have emerged as a powerful paradigm for learning DNA sequence representations, yet existing self-supervised pretraining strategies overlook alternative splicing and underperform compared to specialized models. To address this gap, we introduce IsoAlign, a multimodal framework that grounds genomic pretraining in isoform-aware DNA-protein alignment to learn biologically meaningful representations. IsoAlign aligns transcripts to their corresponding protein isoforms, capturing both coding sequences and broader regulatory contexts, using two Joint-Embedding Predictive Architectures (JEPA): DNA-Protein JEPA (DP-JEPA), which predicts protein representations from DNA using contrastive alignment, and Isoform-Masked JEPA (IM-JEPA), which models relationships among alternative isoforms of the same gene through masked protein prediction. We evaluate the learned DNA representations of these architectures across a multi-metric evaluation suite spanning tasks, including the prediction of pathogenicity, fitness, gene expression, and TF binding. Our results demonstrate that isoform-aware cross-modal alignment using JEPA significantly improves functional representations and downstream performance, establishing a promising direction for GFM pretraining.

Introduction

Genomic foundation models (GFMs) have emerged as a promising paradigm for learning high-dimensional representations of DNA sequences. Inspired by the success of large language models (LLMs) [10] and protein foundation models (PFMs) [27, 38], GFMs leverage transformer-based architectures pretrained on large genomic corpora to learn the underlying regulatory syntax shaped by evolutionary constraints, enabling a unified feature space for diverse downstream biological tasks [54, 35, 39, 15].

However, current pretraining strategies for GFMs, which primarily leverage masked language modeling (MLM) and next-token prediction (NTP), fail to deliver consistent or meaningful performance gains over specialist models [31]. In fact, previous studies report that randomly initialized models perform on par with, or even outperform, pretrained GFMs after fine-tuning [50, 43] (Schreiner et al. 2024). This suggests that these standard self-supervised objectives fail to generate representations that adequately capture the biological complexity of genomic data. A key limitation is that they treat genomic sequences as flat token streams, without explicitly modeling the hierarchical and multi-stage processes through which genetic information is expressed.

A central bottleneck in connecting genomic variation to phenotypic function is the diversity introduced by alternative splicing. Although translation of a given coding sequence into its corresponding amino-acid sequence is deterministic under the genetic code, the relationship between a genomic locus and its protein products is not necessarily one-to-one. In eukaryotes, alternative splicing can generate multiple coding transcripts from the same gene, producing protein isoforms with distinct structural properties, subcellular localizations, and biochemical interaction partners [21]. In humans, many genes express multiple transcript isoforms, with some loci producing dozens of distinct proteoforms [1]. Thus, the challenge is not to rediscover the codon mapping itself, but to learn representations that capture biologically meaningful information shared across nucleotide and protein sequences despite the diversity of transcript and protein products associated with genomic loci.

Despite this biological centrality, alternative splicing remains a blind spot for existing GFMs. Because most models operate on fixed-length genomic windows or gene-level aggregates, they inherently overlook isoform-specific regulation. Consequently, current architectures struggle to resolve how subtle sequence variations propagate through the splicing machinery to alter functional outcomes. This highlights a critical architectural gap: while GFMs learn representations of genomic sequences and PFMs capture properties of protein sequences, the molecular correspondence between coding transcripts and their protein isoforms remains underexplored.

In this work, we propose IsoAlign, a multimodal framework designed to learn isoform-aware genomic representations through cross-modal latent alignment. Rather than relying on token reconstruction losses and scaling alone, our approach utilizes transcript-isoform-aligned DNA-protein pairs and Joint-Embedding Predictive Architectures (JEPA) to explicitly model the functional consequences of genomic variation. By aligning DNA and protein representations that capture high-level biological properties beyond the deterministic codon-to-amino-acid mapping, we propose a pretraining strategy to move GFMs beyond pattern recognition toward functional discovery and precise variant interpretation.

Major Contributions

  1. IsoAlign Framework: We introduce a model-agnostic paradigm that grounds genomic pretraining in multimodal biological alignment rather than flat sequence reconstruction. By semantically mapping transcript representations to their corresponding protein isoforms, IsoAlign explicitly captures the functional diversity introduced by alternative splicing across coding sequences and broader regulatory contexts.

  1. Isoform-Aware Predictive Architectures (JEPA): We propose two complementary architectures: DP-JEPA, which predicts protein representations from DNA anchors in a shared latent space, and IM-JEPA, which predicts masked protein targets from visible isoform context to capture functional relationships among splice variants.

  1. Cross-Modal Objective Benchmarking: Through architecture-matched controls, we quantify the impact of cross-modal alignment, demonstrating that JEPAs learn more biologically meaningful representations than conventional Masked Language Modeling (MLM) and Contrastive Learning (CL).

  1. Comprehensive Biological Evaluation: We establish a benchmark spanning general and splicing-sensitive tasks, evaluating representational robustness in fitness prediction, splicing variant effect prediction, and tissue-specific gene expression.

Preliminaries and Related Work

Limitations of Genomic Foundation Models. While scaling laws have dominated the development of PFMs and GFMs [4, 54], recent evidence suggests that increasing parameter counts does not monotonically improve performance [20, 25]. Moreover, large models impose prohibitive training and inference costs, which limits accessibility and iteration. Self-supervised objectives based on token reconstruction, such as Masked Language Modeling (MLM), often bias GFMs toward local sequence statistics at the expense of global functional constraints [47]. Long-context capabilities have been positioned as a solution for capturing distal genomic dependencies; however, recent studies suggest that current models struggle to effectively exploit this expanded context [46, 12]. These findings highlight a broader limitation: naive DNA-only self-supervised pretraining lacks the grounding necessary to map genomic variation to its functional downstream consequences, often resulting in representations that lack clinical relevance and sensitivity to genomic heterogeneity [30].

Moving Beyond Traditional Genomic Pretraining. Current pretraining strategies yield marginal improvements over randomly initialized baselines, with performance frequently influenced by tokenizer choice rather than the pretraining objective itself [44, 5]. These limitations have motivated a shift toward objectives that encourage learned representations to capture biologically meaningful features beyond local sequence statistics [47] (Consensus et al. 2026). Moreover, training data composition plays a critical role in representation quality. The SPACE framework [51] demonstrates that pretraining on functionally annotated profiles improves generalization through a species-aware mixture-of-experts architecture. Contrastive learning (CL) has also emerged as a strong alternative to masking-based objectives. Orthrus leverages decoupled contrastive learning (DCL) with biologically motivated augmentations for RNA modeling ([17]), mirroring gains observed in protein sequence–structure alignment ([53]) and protein–DNA binding prediction ([11]). More recently, joint-embedding predictive architectures (JEPA), including JEPA-DNA ([24]) and GenoJEPA ([45]), move beyond local nucleotide reconstruction toward latent-space prediction, encouraging representations grounded in functional semantics. However, despite these architectural advancements, overall performance gains remain modest, and these models largely fail to demonstrate meaningful improvements on complex, clinically relevant downstream tasks such as variant effect prediction (VEP). Collectively, these developments reflect a broader paradigm shift from conventional sequence-only masked objectives toward biologically informed and latent-aware formulations designed to learn richer and more transferable representations ([19, 14]).

Multimodal Integration and Biological Alignment. Biological systems are inherently multimodal, operating across a hierarchical continuum of DNA (regulatory), RNA (transcriptional), and protein (functional) layers. Recent foundation models aim to unify these sequences into a shared latent space through diverse architectural strategies. ProTrek integrates protein language models with natural language and structural tokens ([42]), while LucaOne trains a single transformer jointly on nucleotide and amino acid sequences to achieve cross-modal alignment ([19]). To bridge these modalities beyond simple translation, models like CDBridge utilize adaptive token-merging and tissue-specific conditioning to map genomic sequences to expression levels ([49]), and BioLangFusion aligns embeddings at the codon level for precise property prediction ([33]). Complementing these approaches, RESM-2 leverages cross-modal distillation to transfer evolutionary knowledge from ProtESM-2 to RNA by mapping sequences into pseudo-protein representations, enabling protein-informed RNA modeling ([52]). Despite these advances, a critical bottleneck persists: many architectures still treat genomic sequences as static inputs or rely on coarse gene-level associations, overlooking how genetic variation propagates through alternative splicing ([32]) and isoform-specific regulation to contribute to disease ([41]).

Dataset

We construct DNA-protein training pairs from the Broad’s Gene-to-Protein (G2P) database ([22]), linking Ensembl transcripts to UniProt protein isoforms. For each gene, we extract genomic DNA (retaining intron/exon/UTR annotations), transcript regions (with CDS boundaries), and all associated protein isoforms. Pairs are generated via two alignment strategies outlined in Appendix § 8.§ 1. We train each of our architectures using both alignment strategies and report the best-performing configuration.

Method

Double Encoder Framework

Our architecture employs a dual-encoder framework to jointly model DNA and protein sequences in a shared embedding space. Two modality-specific encoders generate fixed-dimensional representations, which are aligned through projection heads for cross-modal learning.

DNA Encoder (X-Encoder). The DNA encoder fθf_{\theta} is a trainable transformer-based model that processes tokenized DNA sequences. Given an input sequence x∈VDNALx \in\mathcal{V}_{\mathrm{DNA}}^{L}, it produces token-level hidden states Hx=fθ(x)∈RL×DxH_x = f_{\theta}(x) \in\mathbb{R}^{L \times D_x}. We evaluate two backbone architectures: NTv2-500m ([15]), a transformer pretrained on multi-species genomic data, and AlphaGenome ([4]), a CNN-based model that operates on one-hot encoded sequences and produces chunk-level embeddings.

Protein Encoder (Y-Encoder). The protein encoder gϕg_{\phi} processes protein sequences into fixed representations. We employ a pretrained protein language model, ESM2-150M ([27]), trained on millions of protein sequences with masked language modeling. Given a protein sequence yy, the encoder produces token-level hidden states Hy=gϕ(y)∈RM×DyH_y = g_{\phi}(y) \in\mathbb{R}^{M \times D_y}. By default, the protein encoder is frozen (ϕ\phi fixed) to provide stable prediction targets.

Pooling and Projection. Both encoders produce variable-length token representations that are aggregated via multi-head attention pooling with a learnable query vector:

z=MultiHeadAttn⁡(q,H,H),(1)z = \operatorname{MultiHeadAttn}(\mathbf{q}, \mathbf{H}, \mathbf{H}), \tag*{(1)}

where q∈RD\mathbf{q} \in\mathbb{R}^{D} is a learnable query, H∈RL×D\mathbf{H} \in\mathbb{R}^{L \times D} are token representations, and we use 8 attention heads. Pooled representations are passed through 3-layer MLP projection heads with batch normalization and ReLU activations:

zx=πx(pool⁡(Hx))∈R512,zy=πy(pool⁡(Hy))∈R512,(2)\mathbf{z}_{x} = \pi_{x}(\operatorname{pool}(\mathbf{H}_{x})) \in\mathbb{R}^{512}, \qquad\mathbf{z}_{y} = \pi_{y}(\operatorname{pool}(\mathbf{H}_{y})) \in\mathbb{R}^{512}, \tag*{(2)}

with hidden dimension 256. Embeddings are ℓ2\ell_{2}-normalized before similarity computation.

IsoAlign JEPA Architectures

DNA-Protein JEPA (DP-JEPA)

Architecture. We introduce DNA-Protein Joint-Embedding Predictive Architecture (DP-JEPA), a cross-modal self-supervised framework that learns to predict protein representations from DNA sequences in a shared latent space, outlined in Figure 1a. It consists of: (1) a trainable DNA encoder fθf_{\theta}, (2) a protein encoder gϕg_{\phi} (frozen by default), and (3) a TransformerPredictor hψh_{\psi} that predicts protein embeddings from DNA representations.

IsoAlign JEPA architectures: DP-JEPA and IM-JEPA

Figure 1. (a) DP-JEPA (DNA-Protein Joint-Embedding Predictive Architecture): A dual-encoder framework where DNA sequences are processed by a trainable DNA encoder (e.g., NTv2) and projected through a Transformer predictor to align with protein representations from a frozen protein encoder (e.g., ESM2). The model learns cross-modal alignment via contrastive loss in a shared latent space. (b) IM-JEPA (Isoform-Masked JEPA): An extension of DP-JEPA that jointly models multiple transcript-isoform pairs from the same genes. DNA sequences for all isoforms are encoded and pooled, while protein isoforms are randomly masked (∼50%\sim50\%). The Transformer Predictor uses DNA anchors as identity queries to predict masked protein embeddings, with ℓ2\ell_2-normalized MSE loss computed only on masked positions and SIGReg regularization to prevent collapse. Figure 1: IsoAlign JEPA Architectures

The TransformerPredictor is a stack of transformer encoder layers (default: 4 layers, 8 heads) that processes the full sequence of DNA hidden states with attention masking, then mean-pools across valid positions:

z^y=πpred(LN⁡(MeanPool⁡(hψ(Hx,mx)))),(3)\hat{\mathbf{z}}_{y} = \pi_{\mathrm{pred}}\left(\operatorname{LN}\left(\operatorname{MeanPool}\left(h_{\psi}(\mathbf{H}_{x}, \mathbf{m}_{x})\right)\right)\right), \tag*{(3)}

where mx\mathbf{m}_{x} is the attention mask excluding padding tokens.

The target protein embedding applies stop-gradient to the encoder output while keeping the projection trainable:

zy=πy(sg⁡(pool⁡(gϕ(y)))).(4)\mathbf{z}_{y} = \pi_{y}\left(\operatorname{sg}\left(\operatorname{pool}\left(g_{\phi}(\mathbf{y})\right)\right)\right). \tag*{(4)}

Both πpred\pi_{\mathrm{pred}} and πy\pi_{y} are multi-layer projection heads mapping to a shared embedding space.

Training Objective. DP-JEPA employs contrastive alignment in the shared projection space. For a batch of genes, each potentially having multiple protein isoforms, we compute the loss per isoform slot and average:

LDP-JEPA=1∣V∣∑j∈VLcontrast(Z^y,Zyj;τ),(5)\mathcal{L}_{\mathrm{DP\text{-}JEPA}} = \frac{1}{\lvert\mathcal{V} \rvert} \sum_{j \in\mathcal{V}} \mathcal{L}_{\mathrm{contrast}}\left(\hat{\mathbf{Z}}_{y}, \mathbf{Z}_{y_j}; \tau\right), \tag*{(5)}

where V\mathcal{V} denotes valid isoform slots (non-padded entries with ≥2\ge2 samples for contrastive discrimination). The target projection πy\pi_{y} is frozen under stop-gradient to provide stable targets.

Isoform-Masked JEPA (IM-JEPA)

Architecture. While DP-JEPA aligns each transcript-isoform pairs through cross-modal prediction, it does not explicitly model relationships among alternative isoforms of the same gene. To incorporate this intra-gene structure, we introduce Isoform-Masked JEPA (IM-JEPA), which jointly processes multiple transcript–isoform pairs and predicts masked protein isoforms from their DNA anchors and the remaining visible isoform context. Specifically, for genes with multiple transcript–isoform pairs {(xi,yi)}i=1N\{(x_i, y_i)\}_{i=1}^{N}, we extend DP-JEPA with object-level masking inspired by Causal-JEPA ([34]), outlined in Figure 1b. Approximately 50% of isoforms are randomly masked per sample while all DNA anchors remain visible, ensuring at least one isoform stays visible for context. The predictor operates on pooled encoder representations and employs three types of learnable embeddings: (1) modality embeddings distinguishing DNA from protein isoforms, (2) slot embeddings indexing each pair, and (3) a learnable mask token m\mathbf{m}. For masked isoforms, we inject identity queries derived from their DNA anchors via a learned projection ρ\rho:

siiso={πiso(zyi)+eiso+eiif visible,m+ρ(πdna(zxi))+eiso+eiif masked.(6)\mathbf{s}_{i}^{\mathrm{iso}} = \begin{cases} \pi_{\mathrm{iso}}(\mathbf{z}_{y_i}) + \mathbf{e}_{\mathrm{iso}} + \mathbf{e}_{i} & \text{if visible}, \\ \mathbf{m} + \rho(\pi_{\mathrm{dna}}(\mathbf{z}_{x_i})) + \mathbf{e}_{\mathrm{iso}} + \mathbf{e}_{i} & \text{if masked}. \end{cases} \tag*{(6)}

where zxi=pool⁡(fθ(xi))z_{x_i}=\operatorname{pool}(f_{\theta}(x_i)) and zyi=pool⁡(gϕ(yi))z_{y_i}=\operatorname{pool}(g_{\phi}(y_i)) are pooled encoder outputs, and πdna,πiso\pi_{\mathrm{dna}},\pi_{\mathrm{iso}} are input projections with layer normalization.

The full input sequence concatenates all DNA representations with isoform representations, processed by a bidirectional Transformer encoder:

Z^=hψ([πdna(zx(1:N))+edna+eslot;s(1:N)]).(7)\hat{Z}=h_{\psi}\left(\left[\pi_{\mathrm{dna}}\left(z_{x}^{(1:N)}\right)+e_{\mathrm{dna}}+e_{\mathrm{slot}};s^{(1:N)}\right]\right). \tag*{(7)}

Training Objective and Regularization. For IM-JEPA, the loss is computed on masked and valid (non-padded) positions using ℓ2\ell_{2}-normalized embeddings:

LIM-JEPA=1∣M∩V∣∑i∈M∩V∥z^ˉi−zˉi∥22+λregLSIGReg,(8)\mathcal{L}_{\mathrm{IM\text{-}JEPA}}=\frac{1}{|\mathcal{M}\cap\mathcal{V}|}\sum_{i\in\mathcal{M}\cap\mathcal{V}}\left\lVert\bar{\hat{z}}_{i}-\bar{z}_{i}\right\rVert_{2}^{2}+\lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{SIGReg}}, \tag*{(8)}

where zˉ=z/∥z∥2\bar{z}=z/\lVert z\rVert_{2}, M\mathcal{M} denotes masked positions, V\mathcal{V} denotes valid (non-padded) pairs, and LSIGReg\mathcal{L}_{\mathrm{SIGReg}} is an anti-collapse regularization term.

Specifically, SIGReg ([6]) encourages predictions to follow an isotropic Gaussian N(0,I)\mathcal{N}(0,I) via characteristic function matching. Predictions are projected onto KK random unit vectors {pk}k=1K\{p_{k}\}_{k=1}^{K} (resampled each iteration) and compared to the Gaussian characteristic function φ(t)=e−t2/2\varphi(t)=e^{-t^{2}/2}:

LSIGReg=1K∑k=1K∫0tmax⁡w(t)[(cˉk(t)−e−t2/2)2+sˉk(t)2] dt,(9)\mathcal{L}_{\mathrm{SIGReg}}=\frac{1}{K}\sum_{k=1}^{K}\int_{0}^{t_{\max}}w(t)\left[\left(\bar{c}_{k}(t)-e^{-t^{2}/2}\right)^{2}+\bar{s}_{k}(t)^{2}\right]\,\mathrm{d}t, \tag*{(9)}

where cˉk(t)=E[cos⁡(t⋅pkTz^)]\bar{c}_{k}(t)=\mathbb{E}[\cos(t\cdot p_{k}^{\mathsf{T}}\hat{z})] and sˉk(t)=E[sin⁡(t⋅pkTz^)]\bar{s}_{k}(t)=\mathbb{E}[\sin(t\cdot p_{k}^{\mathsf{T}}\hat{z})] are empirical characteristic function components, and w(t)∝e−t2/2w(t)\propto e^{-t^{2}/2} weights errors by target magnitude. We use K=1024K=1024 projections with trapezoidal integration over t∈[0,3]t\in[0,3].

Architecture-Matched Pretraining Controls

To isolate the performance gains of our JEPA objectives from the underlying dual-encoder architecture and increased parameter count, we implement several architecture-matched controls. These controls utilize the exact same DNA encoder, frozen protein encoder, and projection heads, but replace the JEPA objective with MLM or CL:

Masked Language Modeling (MLM). We implement four MLM configurations with a 15% masking rate. Cross-modal conditioning utilizes either sequence-level (global pooling) or token-level (position-specific) multi-head cross-attention (8 heads), with linear projections to align hidden dimensions between modalities.

  1. Standard Joint: Independent MLM with combined loss: LJoint=λDLMLMD+λPLMLMP\mathcal{L}_{\mathrm{Joint}}=\lambda_{D}\mathcal{L}_{\mathrm{MLM}}^{D}+\lambda_{P}\mathcal{L}_{\mathrm{MLM}}^{P}.

  1. P→\toD: Unmasked protein predicts masked DNA: H^D=LN⁡(Attn⁡(HD,WPHP)+HD)\hat{H}_{D}=\operatorname{LN}(\operatorname{Attn}(H_{D},W_{P}H_{P})+H_{D}).

  1. D→\toP: Unmasked DNA predicts masked protein: H^P=LN⁡(Attn⁡(HP,WDHD)+HP)\hat{H}_{P}=\operatorname{LN}(\operatorname{Attn}(H_{P},W_{D}H_{D})+H_{P}).

  1. Bidirectional: Both modalities are masked; combines (2) and (3) for mutually enhanced reconstruction.

Contrastive Learning. We align DNA-protein embeddings using three loss variants, governed by a biological correctness matrix C∈{0,1}N×NC\in\{0,1\}^{N\times N}. All variants utilize a 8,192-entry momentum memory bank (m=0.999m=0.999) ([18]).

  • InfoNCE: Maximizes matched-pair similarity using learnable temperature τ\tau (init 0.5).

  • DCL: Decoupled loss ([48]) separating positive and negative terms for independent weighting: LDCL=λpos(−sˉpos)+λneglog⁡∑exp⁡(sneg)\mathcal{L}_{\mathrm{DCL}}=\lambda_{\mathrm{pos}}(-\bar{s}_{\mathrm{pos}})+\lambda_{\mathrm{neg}}\log\sum\exp(s_{\mathrm{neg}}).

  • CLIP: Symmetric bidirectional NCE with low initial temperature (τ0=0.1\tau_{0}=0.1, learnable) for sharp distribution alignment ([37]).

Results

Deep Mutation Scan Fitness Prediction

We evaluated linear probing for fitness prediction on five DMS datasets from four human genes—CBS, P53, PDE3A, and GDI1—following Brixi et al. [9] (Appendix § 8.§ 2). Each dataset contains wild-type and mutant sequences with experimentally measured fitness scores. GFMs and IsoAlign are evaluated using DNA representations, while PFMs are evaluated separately using protein representations for comparison. We use group KK-fold cross-validation stratified by amino acid mutation identity to prevent codon leakage, where synonymous codons encoding identical substitutions could otherwise appear across splits and artificially inflate GFM performance. We evaluate IsoAlign variants combining different GFMs (NTv2/AlphaGenome) with ESM2 under multiple pretraining objectives and initialization schemes. The best-performing models use our JEPA objectives (Figure 2), achieving median Spearman correlations of 0.557 ±\pm 0.21 (DP-JEPA) and 0.554 ±\pm 0.22 (IM-JEPA). Notably, DP-JEPA performs better with NTv2, while IM-JEPA favors AlphaGenome, and both substantially outperform their respective GFM baselines (DP-JEPA: +0.094 over NTv2; IM-JEPA: +0.136 over AlphaGenome) as well as architecture-matched cross-modal MLM and contrastive controls, demonstrating that isoform-aware alignment improves DNA representations. Across 16 GFMs and 5 PFMs, our best IsoAlign model outperforms all baselines. Per-model setup and regression protocol are detailed in Appendix § 11.

DMS Fitness Prediction

Figure 2. DMS Fitness Prediction. Spearman correlation between embedding-based variant scores and fitness scores on Deep Mutational Scanning (DMS) datasets. Models are grouped by category: DNA FMs (gray), Protein FMs (red), and IsoAlign models (blue). IsoAlign consistently outperforms unimodal DNA baselines, demonstrating that cross-modal pretraining with protein-derived inductive biases improves variant effect prediction without task-specific fine-tuning.

Probing DNA Representations for Splicing Variant Effects

We evaluate variant effect prediction (VEP) from DNA representations, with the goal of distinguishing pathogenic variants (those that disrupt normal gene function) from benign ones. We focus on two splicing-related single-nucleotide variant (SNV) datasets: ClinVar Splicing, which contains clinically annotated splice-site variants, and MFASS (Massively Functional Assay of Splicing SNVs), a functional assay of splicing disruption Chong et al., [13]; dataset details are provided in Appendix § 8.§ 3. For each variant, we compare the wild-type and mutant representations, quantify the resulting perturbation using cosine and ℓ2\ell_2 distances, and measure performance using AUPRC under leave-one-chromosome-out (LOCO) cross-validation. Additional representation-based evaluations are reported in Appendix § 9, and per-model setup and evaluation protocols are detailed in Appendix § 11.

As shown in Table 1, cross-modal DNA-protein pretraining can increase the functional discriminability of the underlying DNA representation, although the magnitude of the improvement depends on the backbone and training objective. On ClinVar Splicing, the large-scale baselines Evo2 and Carbon achieve strong performance, consistent with their larger model capacity and extensive pretraining. Despite its smaller backbone, DP-JEPA performs best among our objectives, substantially improving NTv2-500M from 0.32 to 0.63 cosine AUPRC and AlphaGenome from 0.61 to 0.63. In contrast, existing JEPA-DNA adaptations do not consistently improve their parent DNA models and can even degrade performance, indicating that gains are not inherent to JEPA-based pretraining alone. MFASS presents a substantially more challenging setting, with only 1,050 pathogenic variants among 28,972 (3.6%), and correspondingly near-chance AUPRC values across the baseline models, including the larger models. Nevertheless, DP-JEPA produces substantial gains over its baseline representations: with NTv2-500M, ℓ2\ell_2 AUPRC increases from 0.04 to 0.17, while with AlphaGenome, cosine AUPRC increases from 0.29 to 0.36. IM-JEPA provides little improvement over NTv2-500M and underperforms the parent AlphaGenome representation. Overall, DP-JEPA consistently improves or maintains the functional sensitivity of its underlying DNA representations across these settings, whereas the effectiveness of IM-JEPA is more backbone-dependent.

Table 1: Variant effect prediction from DNA representations, evaluated by AUPRC using two metrics. Values in parentheses indicate changes relative to the parent DNA model; underlined values indicate the best performance per column, and bold values are the best IsoALIGN model per backbone.

ModelParamsClinVar Splicing CosineClinVar Splicing ℓ2\ell_2MFASS CosineMFASS ℓ2\ell_2
DNABERT-2 ([54])117M0.140.150.040.04
JEPA-DNABERT-2 ([24])117M0.140.150.040.04
HyenaDNA-tiny ([35])0.44M0.160.150.040.04
JEPA-HyenaDNA-tiny0.44M0.150.160.040.04
Caduceus-PH ([39])7.7M0.200.180.040.04
LucaOne ([19])1.8B0.150.160.050.06
Orthrus-4track ([17])10M0.180.160.040.04
Orthrus-MLM10.7M0.180.190.040.04
Omni-DNA-116M ([26])116M0.150.150.040.04
AIDO.DNA-300M ([16])300M0.520.580.050.09
NTv3-100M ([8])100M0.510.660.100.12
JEPA-NTv3-100M100M0.470.550.120.09
Evo2-1B-base ([9])1B0.380.850.050.16
Carbon ([2])3B0.630.900.050.08
NTv2-500M ([15])500M0.320.730.040.04
+ DP-JEPA (CLIP)520M0.63 (+.31)0.670.08 (+.04)0.17 (+.13)
+ DP-JEPA (DCL)520M0.39 (+.07)0.470.05 (+.01)0.06 (+.02)
+ DP-JEPA (InfoNCE)520M0.58 (+.26)0.550.07 (+.03)0.12 (+.08)
+ IM-JEPA520M0.35 (+.03)0.74 (+.01)0.040.05 (+.01)
AlphaGenome ([4])450M0.610.570.290.31
+ DP-JEPA (CLIP)470M0.63 (+.02)0.60 (+.03)0.36 (+.07)0.31
+ IM-JEPA470M0.330.300.060.09

Table 1.

Supervised Finetuning

We evaluate our model on two primary biological tasks: i) TF-DNA binding specificity (TFLink) Liska et al., [28] and ii) tissue-specific transcript expression (GTEx), using two fine-tuning strategies: i) Linear Probing and ii) Parameter Efficient Fine-Tuning (LoRA), described in Appendix § 10. Per-baseline fine-tuning configurations are detailed in Appendix § 11.

TFLink. We benchmark on the TFLink dataset using a hard negative split with 35% within-family negatives, specifically designed to assess discrimination between structurally similar TFs with overlapping sequence motifs. Details of the dataset construction, sampling procedure, and TF-family composition are provided in Appendix § 8.§ 4. As shown in Table 2, frozen representations alone underperform, suggesting TF binding specificity requires task adaptation beyond generic sequence embeddings. In contrast, LoRA fine-tuning substantially improves performance, outperforming prior GFM and specialist baselines. These results suggest that isoform-aware representations can capture regulatory sequence features highly predictive of TF-DNA interactions.

Table 2: TFLink transcription factor binding prediction (mean ±\pm SD over 4 training seeds).

ModelTrainingParamsAUROCAUPRC
LucaOne ([19])LoRA1.8B0.785 ±\pm 0.0010.778 ±\pm 0.002
BPNet ([3])Full0.12M0.861 ±\pm 0.0040.851 ±\pm 0.003
ChromBPNet ([36])Full6M0.867 ±\pm 0.0050.851 ±\pm 0.007
AlphaGenome ([4])Full450M0.892 ±\pm 0.0020.879 ±\pm 0.003
IsoALIGNLoRA650M0.934 ±\pm 0.0010.928 ±\pm 0.001

Table 2.

GTEx. We use expression data from five tissues in GTEx ([29]): Brain, Heart, Kidney, Liver, and Stomach (N=445,888N=445{,}888 records across 19,587 unique genes). We implement gene-level splitting (8:1:1), ensuring that all transcripts from a given gene remain within the same train, validation, or test partition to prevent data leakage. As shown in Table 3, IsoALIGN achieves the best overall performance, outperforming both LucaOne and AlphaGenome across most tissues. IsoALIGN attains the highest overall Spearman correlation (ρ=0.725\rho= 0.725) and Pearson correlation (r=0.738r = 0.738), with particularly strong gains in Kidney and Liver tissue prediction, demonstrating the benefit of protein-informed genomic representations for tissue-specific expression modeling.

ModelOverall ρ\rhoOverall rrBrain ρ\rhoBrain rrHeart ρ\rhoHeart rrKidney ρ\rhoKidney rrLiver ρ\rhoLiver rrStomach ρ\rhoStomach rr
LucaOne0.4320.4300.4480.4370.4560.4550.3830.3800.4260.4170.4530.453
AlphaGenome0.7170.7320.7020.7290.7320.7430.7400.7420.6830.7040.7190.737
IsoALIGN0.7250.7380.7160.7390.7350.7440.7450.7470.6970.7150.7220.736

Table 3. GTEx tissue-specific expression prediction.

Sequence Ablation Analysis

To show that IsoALIGN learns beyond a codon lookup table, we systematically ablated input sequences and evaluated cross-modal retrieval (Figure 3; Appendix § 12). If IsoALIGN relied on deterministic codon-to-amino-acid translation, codon-shuffled sequences (which preserve the encoded protein while altering nucleotide positions) should match or exceed the CDS baseline, while frame-shifted sequences (which produce an entirely different amino acid sequence) should fail entirely. We observe the opposite: codon shuffling degrades retrieval by 19%, whereas frame shifting degrades only 7%, demonstrating that IsoALIGN captures positional codon dependencies, including codon usage bias, translational efficiency signals, and local sequence context, beyond simple translation lookup. Training exclusively on untranslated regions (UTRs), which contain no coding information, retains 65–78% of CDS-baseline performance far above the random DNA baseline (6%), confirming that IsoALIGN learns regulatory sequence features that correlate with protein function independently of the genetic code. Finally, isoform-swapped negatives (wrong isoform, same gene) retain 79% of baseline retrieval, whereas homolog-swapped negatives (protein from a related gene family) degrade to 39%, indicating that IsoALIGN distinguishes subtle isoform-level differences while recognizing that homologs, despite sequence similarity, have distinct functional contexts.

Cross-modal retrieval by ablation category

Figure 3. Cross-modal retrieval (Avg MRR) by ablation category. Dashed line: CDS baseline.

Conclusion

We introduced IsoALIGN, a model-agnostic framework for genomic pretraining driven by isoform-aware DNA-protein alignment. By explicitly modeling the functional diversity of alternative splicing via two complementary JEPA architectures (DP-JEPA and IM-JEPA), the framework moves beyond flat sequence reconstruction. Our evaluations demonstrate that isoform-aware cross-modal prediction improves functional genomic representations, with DP-JEPA providing consistent gains across fitness and variant effect prediction, while IM-JEPA shows more task- and backbone-dependent improvements. Supervised evaluations further demonstrate strong performance in transcription factor binding and tissue-specific expression prediction. Crucially, sequence ablations confirm that these learned representations capture true regulatory syntax rather than merely memorizing deterministic codon mappings. Ultimately, this establishes biologically grounded, cross-modal latent alignment as a highly effective paradigm for GFMs.

Limitations and Future Work. IsoALIGN is limited by the completeness of DNA-protein databases and curated benchmarks, which may not capture context-specific splicing or real-world patient variation. Future work will evaluate patient-derived cohorts and incorporate epigenomic and structural information. Clinical translation will additionally require safeguards against population bias and privacy vulnerabilities.

Appendix

Dataset Construction and Statistics

Training Data Alignment

We construct DNA-protein training pairs using two complementary alignment strategies that differ in the granularity of sequence correspondence.

Transcript-Specific. Pairs spliced transcripts (exons and UTRs) with CDS-sliced protein subsequences. DNA chunks of 4,608 bp include explicit markers denoting 5’UTR, CDS, and 3’UTR boundaries, with only the corresponding protein region extracted for each chunk.

Feature-Aligned (CDS-Only). Pairs coding sequences directly with protein counterparts using strict 3:1 codon-to-amino-acid mapping. Exon boundaries are preserved across both modalities to facilitate learning of splice junction patterns. DNA chunks of 3,072 bp are paired with up to 1,024 amino acid tokens.

Deep Mutational Scanning (DMS)

The DMS evaluation dataset measures mutation effects on protein function using experimental fitness scores. Each sample contains a mutated protein sequence, its corresponding DNA sequence, and a fitness score derived from high-throughput functional assays.

StudyProteinSamplesProtein (aa)DNA (bp)Value Range
Kotler et al. 2018TP532,9673931,182-4.99–1.51
Giacomelli et al. 2018TP537,4483931,182-4.37–2.84
Sun et al. 2020CBS6,7575511,6560.00–1.00
Garvie et al. 2021PDE3A7,8931,1413,426-3.83–2.47
Silverstein et al. 2021GDI14,3654471,344-7.33–3.09
Total29,430

Table 4. Deep Mutational Scanning dataset statistics.

Variant Effect Prediction

ClinVar is a widely-used clinical variant database containing expert-curated pathogenicity labels for human genetic variants Landrum et al. [23]. We focus on ClinVar SNVs within splice-site regions in the immediate vicinity of exon–intron junctions, where conserved splice donor (5’) and acceptor (3’) motifs regulate RNA splicing. Even single-nucleotide perturbations near these sites can disrupt splicing fidelity and lead to severe functional consequences, including exon skipping or intron retention. Following Avsec et al. Avsec et al. [4], we define the splice-site region as the interval spanning 6 bp into the intron and 3 bp into the exon at each annotated junction, using GENCODE v46 as the transcript reference. We include both donor- and acceptor-proximal variants and restrict the dataset to autosomal and sex-chromosome SNVs with unambiguous ClinVar annotations (Pathogenic/Likely pathogenic or Benign/Likely benign), excluding variants with conflicting or uncertain interpretations. Variants are filtered from the ClinVar GRCh38 release, resulting in a dataset containing 7,028 pathogenic and 41,535 benign SNVs.

MFASS (Massively Functional Assay of Splicing SNVs) provides experimentally grounded measurements of splicing disruption, making it a particularly reliable benchmark for VEP. Unlike clinically annotated datasets, MFASS variants were directly assayed using a massively parallel minigene reporter system (SORT-seq) Chong et al. [13], which quantifies the splicing consequences of thousands of single-nucleotide variants in a controlled cellular context. Variants are labeled as positive if they induce a strong loss-of-function splicing effect, and negative otherwise. Following the benchmark protocol of Avsec et al. [4], we retain only SNVs with unambiguous reference and alternate alleles and exclude any sequences containing ambiguous nucleotides. The dataset is derived from the publicly available MFASS repository, enabling a direct evaluation of whether models can detect functionally disruptive splicing variants independent of clinical or population-level annotations.

We construct an evaluation dataset from TFLink Liska et al. [28], a database of 8.8M experimentally-validated human transcription factor binding sites. The task is binary classification: given a TF protein sequence and a 1024 bp genomic DNA sequence centered on a potential binding site, predict whether binding occurs.

Dataset Construction. Positive examples pair each TF with its validated binding sites. Negative examples pair each TF with binding sites of other TFs—including structurally similar TFs from the same family—creating hard negatives that require the model to learn TF-specific binding preferences rather than generic DNA accessibility. We apply chromosome-stratified sampling with a minimum 10 kb distance between sites to ensure sample independence. The dataset contains 200,000 samples (100,000 positive, 100,000 negative) across 20 TFs from 10 TF families. Table 5 summarizes the per-TF statistics.

StratificationCategoriesSamples
Binding StatusPositive100,000
Binding StatusNegative (same-family)100,000
Binding Site LengthShort (≤12\le12 bp)111,982
Binding Site LengthMedium (13–20 bp)78,807
Binding Site LengthLong (>20 bp)9,211
TF FamilyTF_bZIP70,000
TF Familyzf-GATA20,000
TF Familyzf-C2H220,000
TF FamilyFork_head20,000
TF FamilybHLH20,000
TF FamilyOther (5 families)50,000

Table 5. TFLink Evaluation Dataset

Additional Variant Effect Prediction Evaluations

To evaluate the predictive power of dna representations, we compare wildtype and mutant embeddings through four representation-based metrics. This includes two scalar metrics: cosine distance at mutated positions and the ℓ2\ell_2 norm of the embedding difference, assessed using AUPRC. Additionally, we employ two vector metrics: element-wise embedding products (Mul) Benegas et al. [7] and (CLS) token differences, evaluated through leave-one-chromosome-out (LOCO) cross-validation logistic regression. For models without a [CLS] token, the CLS feature is taken from the final position for autoregressive models and from the first position otherwise.

Supervised Finetuning

Linear Probing Both the DNA encoder and Protein encoder are frozen. Features are extracted via mean pooling and concatenated into a d=1664d = 1664 representation.

Parameter-Efficient Fine-Tuning Low-Rank Adaptation (LoRA) was applied to the DNA encoder’s query and value attention layers (r=4r = 4). This configuration introduces ∼1.4\sim1.4M trainable adapter parameters while maintaining a frozen Protein encoder, totaling ∼2.4\sim2.4M trainable parameters including the head.

Prediction Head For both tasks, we utilize a standardized MLP head:

f(h)=W3ϕ(W2ϕ(W1LN⁡(h)))(10)f(\mathbf{h}) = \mathbf{W}_3\phi\left(\mathbf{W}_2\phi\left(\mathbf{W}_1\operatorname{LN}(\mathbf{h})\right)\right) \tag*{(10)}

Table 6: Variant effect prediction evaluation on ClinVar Splicing and MFASS using four metrics. <u>Underlined</u> values indicate the best performance per column, and bold values are the best IsoAlign model per backbone.

ModelParamsClinVar Splicing: CosineClinVar Splicing: ℓ2\ell_2ClinVar Splicing: CLSClinVar Splicing: MulMFASS: CosineMFASS: ℓ2\ell_2MFASS: CLSMFASS: Mul
Baselines
DNABERT-2 (Zhou et al., [54])117M0.140.150.180.180.040.040.050.04
JEPA-DNABERT-2 (Larey et al., [24])117M0.140.150.270.190.040.040.080.04
HyenaDNA (Nguyen et al., [35])6.6M0.170.180.250.180.040.040.050.04
HyenaDNA-tiny0.44M0.160.150.220.170.040.040.040.04
JEPA-HyenaDNA-tiny0.44M0.150.160.190.170.040.040.040.03
Caduceus-PH (Schiff et al., [39])7.7M0.200.180.190.190.040.040.050.05
Caduceus-PS7.7M0.200.450.420.420.040.080.130.10
LucaOne (He et al., [19])1.8B0.150.160.180.170.050.060.090.07
LucaOne-gene1.8B0.150.170.180.170.050.060.080.06
Orthrus-4track (Fradkin et al., [17])10M0.180.160.140.160.040.040.040.04
Orthrus-MLM10.7M0.180.190.150.190.040.040.040.04
Omni-DNA-116M (Li et al., [26])116M0.150.150.550.210.040.040.100.06
Omni-DNA-300M300M0.150.140.490.210.040.040.130.06
AIDO.DNA-300M (Ellington et al., [16])300M0.520.580.520.520.050.090.120.07
NTv3-650M (Boshar et al., [8])650M0.350.570.340.280.040.070.080.07
NTv3-100M100M0.510.660.220.210.100.120.070.08
JEPA-NTv3-100M100M0.470.550.250.200.120.090.080.08
Evo2-1B-base (Brixi et al., [9])1B0.380.850.590.350.050.160.160.10
Evo2-7B-base7B0.570.880.720.360.050.090.130.13
Carbon (Allal et al., [2])3B<u>0.63</u><u>0.90</u>0.65<u>0.75</u>0.050.080.090.06
NTv2-500M (Dalla-Torre et al., [15])500M0.320.730.520.420.040.040.050.04
+ DP-JEPA (CLIP)520M0.630.670.470.460.080.170.180.08
+ DP-JEPA (DCL)520M0.390.470.490.240.050.060.100.06
+ DP-JEPA (InfoNCE)520M0.580.550.590.250.070.120.180.06
+ IM-JEPA520M0.350.740.590.460.040.050.070.04
AlphaGenome (Avsec et al., [4])450M0.610.570.630.300.29<u>0.31</u>0.320.23
+DP-JEPA (CLIP)470M0.630.600.610.29<u>0.36</u>0.31<u>0.36</u>0.21
+IM-JEPA470M0.330.300.570.200.060.090.210.13
Architecture-matched pretraining controls (NTv2-500M)
MLM520M0.270.730.590.390.040.040.060.04
Conditioned MLM520M0.270.730.530.410.040.040.050.04
Bidirectional TCA520M0.500.610.630.490.040.040.040.04
Cross-Modal520M0.520.480.420.430.040.050.050.05
Contrastive (CLIP)520M0.400.700.470.500.040.070.060.06
Contrastive (DCL)520M0.470.640.390.570.040.050.050.05
Contrastive (InfoNCE)520M0.390.690.460.520.050.090.060.05

Table 6.

where LN is Layer Normalization and ϕ\phi denotes GELU activation followed by Dropout (p=0.2p = 0.2). For GTEx regression, a 64-dimensional tissue embedding is concatenated to the pooled features before the MLP. Ultra-long sequences are managed via 6-mer tokenization with a 512-token truncation cutoff.

Implementation Details for Benchmarked Models

This section catalogues per-model setup decisions that diverge from a naïve Hugging Face load, plus the scoring conventions used across DMS, VEP, TFLink, and GTEx. Every divergence below is forced by an architectural constraint of the model in question (context cap, tokenizer choice, missing head, attention kernel), not by a tunable hyperparameter.

Per-Model Setup

AIDO.DNA-300M [16]. Loaded via the modelgenerator package (the model’s reference RNABert backbone is not exposed through the standard AutoModel routing). DMS and all VEP tracks use the model’s native context L=3998L=3998 (max_pos_embeddings=4000); because RNABert uses eager attention and materialises the full [B,h,L,L][B,h,L,L] probability tensor, we run with batch size 8.

Caduceus-PH / Caduceus-PS [39]. Bidirectional Mamba SSMs requiring a separate environment (mamba_ssm 2.2.4, causal_conv1d 1.5.0). Single-nt character tokenizer with no [CLS] token; pooled representations are mean-pooled across positions.

ChromBPnet [36]. The reference TensorFlow/Keras stack pins old versions incompatible with our PyTorch environment, so we load weights through bpnet-lite, a PyTorch port that reproduces the Keras .h5 forward bit-equivalently. The K562-pretrained checkpoint underperformed by ∼0.10\sim0.10 AUROC across all fine-tuning recipes we tried; we report the same architecture trained from scratch.

BPNet [3]. A 64-filter, 8-layer dilated CNN trained from scratch. We follow the original architecture (motif kernel 21, dilated residual kernels 3 with dilations 21,…,282^1,\ldots,2^8) and pair the DNA trunk with the same frozen ESM2-150M protein encoder and MLP head used for ChromBPnet. The BPNet (Full) row matches the small-CNN supervised baseline of Avsec et al. [3].

DNABERT-2 [54]. Run with the vanilla-attention kernel (Triton kernel disabled): the Triton path produces non-identical representations under the same weights, so we standardise on the vanilla implementation for all reported numbers. The model’s BPE tokenizer means a single-nucleotide variant changes the token identity at the variant position rather than a single position within a token, which precludes masked-marginal LLR scoring at single-nt resolution.

Evo2-1B / 7B [9]. The two variants need different installs: Evo2-7B uses the Light path (vtx 1.0.8, bf16, no transformer_engine), while Evo2-1B sets use_fp8_input_projections=True and requires transformer_engine ≥2.3\ge2.3, which we build from source in a separate environment (PyPI’s TE metawheel pins a transformer_engine_cu12 version not published on PyPI). DMS scoring routes through the SDK’s score_sequences(reduce_method=‘sum’) for whole-sequence LLR because the vortex CharLevelTokenizer does not expose an HF-style __call__.

HyenaDNA [35]. Causal / autoregressive; DMS scoring uses whole-sequence LLR rather than masked-marginal, since the model has no mask token. L=5994L=5994.

LucaoOne / LucaoOne-gene [19]. Trained with max_pos_embeddings=4096; we set VEP_WINDOW_SIZE=4094 to stay within the trained range. The shared gene–protein vocabulary is disambiguated by a seq_type keyword (‘gene’ / ‘prot’) supplied at tokenisation time. The upstream batch_encode_plus reads from the wrong argument slot, so inputs are tokenised one record at a time through a thin adapter. Joint DNA+protein evaluation (TFIso) instantiates two LucaoOne forward graphs sharing the same weights but distinct seq_type tags.

NTv3-650M / 100M [8]. The U-Net architecture requires sequence length divisible by 128 (seven downsample stages); we pad with the NN-token (a real token in NTv3’s alphabet) rather than [PAD]. VEP uses L=4096L=4096 (next multiple of 128 below the trained context), no leading or trailing special tokens, and we walk embed_dim (rather than hidden_size) to retrieve the representation width.

Omni-DNA-116M / 300M [26]. Hard 250 BPE-token context cap. For VEP, we set L=1200L=1200 bp (about 244 BPE tokens, leaving room for special tokens). Three of the five DMS sequences exceed the 250-token budget, so DMS scoring uses a mutation-centred 246-token window: we locate the BPE token covering the variant nucleotide, slide a 246-token window centred on it (clamped if it would overshoot the sequence), and compute LLR on that window only. Wildtype log-likelihood is cached per (sequence, mutation-position) pair, giving roughly 3×3 \times WT re-use under DMS’s average of 3 alleles per site. The same windowing logic is applied in the embedding-suite path so that all DMS metrics for Omni-DNA are computed on identical token windows.

Orthrus-4track [17]. A contrastive Mamba SSM trained with one-hot inputs (no learned token embedding). We wrote a tokenizer adapter that maps single nucleotides to integer ids and a model wrapper that converts those ids to a one-hot tensor inside the forward pass. A small Block class shim bridges the mamba_ssm 1.x API the released checkpoint expects to the 2.x runtime in our environment. The model is encoder-only with no language-model head, so DMS-LLR and VEP-LLR are undefined; only embedding-based metrics are reported.

JEPA-DNABERT-2 [24]. DNABERT-2-117M [54] continually pretrained with a joint-embedding predictive objective, token-level MLM, and VICReg regularisation. We load the backbone via the standard path and apply the JEPA checkpoint weights. Vanilla attention is used (Triton kernel disabled) to match the base DNABERT-2 configuration, and the checkpoint includes the MLM head, which was updated during JEPA training. VEP uses L=5994L=5994.

JEPA-HyenaDNA-tiny [24]. HyenaDNA-tiny-16k-d128 [35] (0.44M parameters, 16 kb context) continually pretrained with the JEPA objective. The architecture is autoregressive, so DMS scoring uses whole-sequence LLR rather than masked-marginal scoring. We benchmark the corresponding base model (HyenaDNA-tiny) alongside; this is distinct from the larger HyenaDNA variant (large-1m, 6.6M parameters) reported elsewhere. VEP uses L=5994L=5994.

JEPA-NTv3-100M [24]. NTv3-100M (NTv3_100M_pre) [8] continually pretrained with the JEPA objective. The U-Net architecture requires sequence lengths divisible by 128, so we pad with the NN-token (a real token in NTv3’s vocabulary) rather than [PAD]. We benchmark the corresponding base model (NTv3-100M) alongside; this is distinct from the larger NTv3-650M variant reported elsewhere. VEP uses L=4096L=4096.

Carbon-3B. A Llama-style autoregressive DNA LM, HuggingFaceBio/Carbon-3B [2], with a hybrid tokenizer comprising a complete 6-mer vocabulary (4096 DNA 6-mers) plus Qwen3-BPE for English, switched by a <dna> tag that we prepend automatically. Tiling is fixed and non-overlapping (1+⌈L/6⌉1+\lceil L/6\rceil tokens), so a SNV flips exactly one token and whole-sequence LLR is well-defined. The model has no mask token, so DMS routes to whole-sequence LLR and VEP to the embedding suite, following the same handling as Evo2. Loaded in bf16; VEP uses L=5994L=5994.

ESM-1b / ESM-1v [38]. 1024-position context cap (1022 usable amino acids). Of the five DMS studies, only Garvie 2021 (PDE3A, 1141 aa) overflows; we tail-crop to the last 1022 aa. Because every PDE3A variant in this dataset sits at residue ≥ 668, tail-cropping is equivalent to a variant-centred window for this study while preserving per-WT amortisation.

ESM-C-300M / 600M. Loaded through the official EvolutionaryScale SDK (esm 3.2.1) in a dedicated environment (torch 2.3.1). A thin HF-compatible adapter wraps the SDK’s forward pass so that the rest of the evaluation pipeline (logits, last_hidden_state, hidden-states tuple) is shape-compatible with the encoder-MLM path used for all other protein models.

ProtT5. An encoder-only T5 trained with span-corruption rather than single-token masked language modelling, so masked-marginal DMS-LLR is undefined; we report the embedding-based DMS metrics only.

DMS Scoring

DMS embedding-based regression uses 5-fold cross-validation with GroupKFold grouped by amino-acid-level mutation identity. The codon-level JSONL rows in four of the five DMS studies have 1.66× to 3.02× codon expansion (synonymous codons mapping to the same mutation), and a random 5-fold split would leaks codon twins between train and test, inflating Spearman by 0.05–0.27 per study. The probe is a StandardScaler followed by Ridge regression with α=10−3\alpha=10^{-3} and the Cholesky solver.

Variant Effect Prediction

Per-model context window. The default benchmark window is L=5,994L = 5{,}994 bp (a 6-mer-aligned ±2,997\pm2{,}997 nt window centred on the variant), used for DNABERT-2, JEPA-DNABERT-2, HyenaDNA, HyenaDNA-tiny, JEPA-HyenaDNA-tiny, Caduceus-PH/PS, NTv2-500M, Orthrus, Evo2, Carbon-3B, and AlphaGenome. Models with stricter caps use:

ModelVEP window (LL)Reason
LucaOne / LucaOne-gene4,094max_pos_embeddings=4096
AIDO.DNA-300M3,998max_pos_embeddings=4000
NTv3-650M / 100M, JEPA-NTv3-100M4,096U-Net 7-downsample (must be ÷128)
Omni-DNA-116M / 300M1,200250 BPE-token context cap

Table 7. Per-model VEP context-window deviations from the default.

Strand averaging. VEP scores are computed independently on the forward strand and the reverse complement and averaged. Both strands carry identical biological information at any chromosomal location, and for autoregressive models the reverse-complement pass also recovers “downstream-context” information that the causal forward pass lacks.

Linear-probe configuration. The two vector-valued metrics (CLS, Mul) are evaluated through leave-one-chromosome-out L2-regularised logistic regression (C=1, standardized features, balanced class weights, Newton-Cholesky solver) and reported as AUPRC. The two scalar metrics (cosine, ℓ2\ell_2) are reported as direct AUPRC of the metric against the binary pathogenicity label.

TFLink and GTEx Supervised Fine-Tuning

Both tasks use the same fine-tuning regime as the IsoAlign rows: LoRA with r=4r=4 on the DNA encoder’s query and value attention projections (about 1.4M trainable adapter parameters), a frozen protein encoder, and the standardised MLP head described in Appendix § 12. The LucaOne baseline uses the same LoRA configuration applied to its query and value projections. Because LucaOne is a single transformer that handles both nucleotide and amino-acid inputs, we instantiate two separate forward graphs sharing the same backbone weights but with distinct seq_type tags (‘gene’ for DNA, ‘prot’ for protein) so that the DNA-side LoRA adapters do not modify the protein representation.

Sequence Ablation Details

Experimental Setup

We train DP-JEPA variants on systematically modified DNA-protein pairs to isolate the contribution of different sequence features. Each ablation uses identical architecture (NTv2-500M DNA encoder + ESM2-150M protein encoder) and training configuration, varying only the input sequences. Models are trained for 50 epochs with early stopping based on a validation loss. We evaluate bidirectional retrieval using DNA-to-Protein (D2P) and Protein-to-DNA (P2D) Mean Reciprocal Rank (MRR) on a held-out validation set.

Ablation Categories

Baselines.

  • Feature-Aligned (CDS): Coding sequences with strict 3:1 codon-to-amino-acid alignment preserving exon boundaries.

  • Canonical CDS: Standard CDS extraction without explicit feature alignment.

  • Transcript-Specific: Full spliced transcripts including UTRs with CDS boundaries marked.

Coding Ablations.

  • Frame Shift (+1): Shift reading frame by 1 nucleotide, disrupting codon boundaries and producing an entirely different amino acid sequence while preserving nucleotide composition.

  • Codon Shuffle: Replace each codon with a randomly selected synonymous codon encoding the same amino acid, preserving the protein sequence while altering codon usage patterns.

  • Splice Junctions Only: Intronic sequences flanking annotated splice sites (±50\pm50 bp), containing only splice donor and acceptor motifs.

Non-Coding Ablations.

  • 5′ UTR Only: 5′ untranslated regions only, excluding all coding sequence.

  • 3′ UTR Only: 3′ untranslated regions only, excluding all coding sequence.

  • UTRs Combined: Concatenated 5′ and 3′ UTRs without CDS.

Genomic Context.

  • Promoter + mRNA: 1 kb upstream promoter region concatenated with the transcript.

  • Promoter + mRNA + Splice: Promoter, transcript, and explicit splice junction annotations.

  • Gene-Level Non-Coding: Distal regulatory sequences only, excluding all coding and proximal elements.

Negative Sampling Strategies.

  • Isoform Swap: Pair each DNA sequence with a different protein isoform from the same gene.

  • Homolog Swap: Pair DNA with a protein from a homologous gene family.

Negative Controls.

  • Shuffled DNA: Randomly permute nucleotides within each biological DNA sequence, destroying all sequence structure while preserving nucleotide composition.

  • Random DNA: Pair proteins with randomly generated DNA sequences matching overall nucleotide frequencies.

Compute Resources

All experiments were conducted on an internal SLURM-managed GPU cluster.

Pretraining. Each training job used 4×4 \times NVIDIA H100 GPUs (80GB) with DeepSpeed ZeRO Stage 2 and bf16 mixed precision. Table 8 summarizes the training hyperparameters for each objective, and Table 9 reports the total compute used. The reported totals include hyperparameter searches and ablation studies; average run duration was ∼8\sim8 hours (∼32\sim32 GPU-hours).

ParameterDP-JEPAIM-JEPAContrastiveMLM
Batch size (per GPU)16–3281616
Gradient accumulation4844
Effective batch size256256256256
Learning rate1e-51e-51e-51e-4
Max epochs150150150150

Table 8. Training hyperparameters by objective.

ObjectiveRunsWall-Clock (h)GPU-Hours
DP-JEPA342741,097
IM-JEPA12136543
Contrastive483681,474
MLM1163254
Total715682,270

Table 9. Total compute used for pretraining experiments.

Supervised Fine-tuning. SFT experiments used a single NVIDIA A40 GPU (48GB). GTEx fine-tuning on 10% downsampled data (∼35\sim35k training samples) took ∼2\sim2 hours per epoch. We used LoRA adapters with learning rate 1e-4, batch size 16, and early stopping (patience 4). Training typically converged within 5–10 epochs.

Evaluation. Zero-shot evaluations used a single NVIDIA A100/H100 GPU (80GB) with batch size 256. We evaluated 105 model configurations across 4 benchmark tasks (DMS, GTEx, TF-Iso). Each model evaluation completed within ∼\sim1–2 hours depending on task complexity.

References

  1. [1]Ruedi Aebersold et al. How many human proteoforms are there? Nature Chemical Biology, 14(3):206–214, 2018.DOI
  2. [2]Loubna Ben Allal, Qiuyi Li, Maurizio Fiusco, Lewis Tunstall, Kashif Rasul, Ed Beeching, Dana Aubakirova, Carlos Patiño, Thibaud Frere, Anton Lozhkov, et al. Carbon: Decoding the language of life. bioRxiv, pp. 2026–05, 2026.DOI
  3. [3]Žiga Avsec, Melanie Weitert, Avanti Shrikumar, Sabrina Krueger, Amr Alexandari, Khyati Dalal, Robin Fropf, Charles McAnany, Julien Gagneur, Anshul Kundaje, et al. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nature genetics, 53(3):354–366, 2021.DOI
  4. [4]Žiga Avsec, Natasha Latysheva, Jun Cheng, Guido Novati, Kyle R. Taylor, Tom Ward, Clare Bycroft, Lauren Nicolaisen, Eirini Arvaniti, Joshua Pan, et al. Advancing regulatory variant effect prediction with AlphaGenome. Nature, 649(8099):1206–1218, 2026. doi: 10.1038/s41586-025-10014-0.DOI
  5. [5]Weiqiang Bai, Lifeng Qiao, Yuchen Ren, Jianle Sun, Peng Ye, Hongliang Yan, Xinzhu Ma, Wangmeng Zuo, Wanli Ouyang, et al. Rethinking the bert-like pretraining for dna sequences. 2024.
  6. [6]Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
  7. [7]Gonzalo Benegas, Gökçen Eraslan, and Yun S. Song. Benchmarking dna sequence models for causal regulatory variant prediction in human genetics. bioRxiv, 2025.DOI
  8. [8]Sam Boshar, Benjamin Evans, Ziqi Tang, Armand Picard, Yanis Adel, Franziska K. Lorbeer, Chandana Rajesh, Tristan Karch, Shawn Sidbon, David Emms, et al. A foundational model for joint sequence-function multi-species modeling at scale for long-range genomic prediction. bioRxiv, pp. 2025–12, 2025.
  9. [9]Garyk Brixi, Matthew G. Durrant, Jerome Ku, Mohsen Naghipourfar, Michael Poli, Gwanggyu Sun, Greg Brockman, Daniel Chang, Alison Fanton, Gabriel A. Gonzalez, et al. Genome modelling and design across all domains of life with Evo 2. Nature, 652:1349 – 1361, 2026.
  10. [10]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  11. [11]Leo Tianlai Chen, Rishab Pulugurta, Pranay Vure, and Pranam Chatterjee. DPAC: Prediction and design of protein-dna interactions via sequence-based contrastive learning. bioRxiv, pp. 2025–05, 2025.DOI
  12. [12]Wenduo Cheng, Zhenqiao Song, Yang Zhang, Shike Wang, Danqing Wang, Muyu Yang, Lei Li, and Jian Ma. DNALONGBENCH: a benchmark suite for long-range dna prediction tasks. Nature Communications, 16(1):10108, 2025.DOI
  13. [13]Rockie Chong, Kimberly D Insigne, David Yao, Christina P. Burghard, Jeffrey Wang, Yun-Hua E. Hsiao, Eric M. Jones, Daniel B. Goodman, Xinshu Xiao, and Sriram Kosuri. A multiplexed assay for exon recognition reveals that an unappreciated fraction of rare genetic variants cause large-effect splicing disruptions. Molecular cell, 73 (1):183–194, 2019.DOI
  14. [14]Micaela Elisa Consens, Kevin K. Yang, Jimmy Hall, Ashley Mae Conard, Bo Wang, Lorin Crawford, Alan Moses, and Alex X. Lu. Predicting evolutionary rate as a pretraining task improves genome language model representations. bioRxiv, pp. 2026–02, 2026.DOI
  15. [15]Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P. De Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22(2):287–297, 2025.
  16. [16]Caleb N. Ellington, Ning Sun, Nicholas Ho, Tianhua Tao, Sazan Mahbub, Dian Li, Yonghao Zhuang, Hongyi Wang, Le Song, and Eric P Xing. Accurate and general dna representations emerge from genome foundation models at scale. bioRxiv, pp. 2024–12, 2024.DOI
  17. [17]Philip Fradkin, Ruan “Ian” Shi, Taykhoom Dalal, Keren Isaev, Brendan J. Frey, Leo J. Lee, Quaid Morris, and Bo Wang. Orthrus: toward evolutionary and functional rna foundation models. Nature Methods, pp. 1–11, 2026.DOI
  18. [18]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729–9738, 2020.arxiv.org/abs/1911.05722
  19. [19]Yong He, Pan Fang, Yongtao Shan, Yuanfei Pan, Yanhong Wei, Yichang Chen, Yihao Chen, Yi Liu, Zhenyu Zeng, Zhan Zhou, et al. Generalized biological foundation model with unified nucleic acid and protein language. Nature Machine Intelligence, 7(6):942–953, 2025.
  20. [20]Chao Hou, Di Liu, Aziz Zafar, and Yufeng Shen. Understanding language model scaling on protein fitness prediction. bioRxiv, 2025.
  21. [21]Ondrej Kelemen et al. Function of alternative splicing. Gene, 514(1):1–30, 2013.DOI
  22. [22]Seulki Kwon, Jordan Safer, Duyen T. Nguyen, David Hoksza, Patrick May, Jeremy A. Arbesfeld, Alan F. Rubin, Arthur J. Campbell, Alex Burgin, and Sumaiya Iqbal. Genomics 2 Proteins portal: a resource and discovery tool for linking genetic screening outputs to protein sequences and structures. Nature Methods, 21(10):1947–1957, 2024.
  23. [23]Melissa J. Landrum, Jennifer M. Lee, Mark Benson, Garth R. Brown, Chen Chao, Shanmuga Chitipiralla, Baoshan Gu, Jennifer Hart, Douglas Hoffman, Wonhee Jang, et al. ClinVar: improving access to variant interpretations and supporting evidence. Nucleic acids research, 46(D1):D1062–D1067, 2018.
  24. [24]Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi, et al. Jepa-dna: Grounding genomic foundation models through joint-embedding predictive architectures. arXiv preprint arXiv:2602.17162, 2026.
  25. [25]Francesca-Zhoufan Li, Ava P. Amini, Yisong Yue, Kevin K. Yang, and Alex X. Lu. Feature reuse and scaling: Understanding transfer learning with protein language models. bioRxiv, pp. 2024–02, 2024.DOI
  26. [26]Zehui Li, Vallijah Subasri, Yifei Shen, Dongsheng Li, Yiren Zhao, Guy-Bart Stan, and Caihua Shan. Omni-dna: A unified genomic foundation model for cross-modal and multi-task learning. arXiv preprint arXiv:2502.03499, 2025.
  27. [27]Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023.
  28. [28]Orsolya Liska, Balázs Bohár, András Hidas, Tamás Korcsmáros, Balázs Papp, Dávid Fazekas, and Eszter Ari. TFLink: an integrated gateway to access transcription factor–target gene interactions for multiple species. Database, 2022:baac083, 2022.
  29. [29]John Lonsdale, Jeffrey Thomas, Mike Salvatore, Rebecca Phillips, Edmund Lo, Saboor Shad, Richard Hasz, Gary Walters, Fernando Garcia, Nancy Young, et al. The genotype-tissue expression (GTEx) project. Nature genetics, 45(6):580–585, 2013.
  30. [30]Baiyu Lu, Xueshen Liu, Po-Yu Lin, and Nadav Brandes. Genomic heterogeneity inflates the performance of variant pathogenicity predictions. bioRxiv, pp. 2025–09, 2025.
  31. [31]Frederikke Isa Marin, Felix Teufel, Marc Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, and Wouter Boomsma. Bend: Benchmarking dna language models on biologically meaningful tasks. arXiv preprint arXiv:2311.12570, 2023.
  32. [32]Sibylle Mitschka and Christine Mayr. Context-specific regulation and function of mrna alternative polyadenylation. Nature Reviews Molecular Cell Biology, 23(12):779–796, 2022.
  33. [33]Amina Mollaysa, Artem Moskale, Pushpak Pati, Tommaso Mansi, Mangal Prakash, and Rui Liao. Biolangfusion: Multimodal fusion of dna, mrna, and protein language models. arXiv preprint arXiv:2506.08936, 2025.
  34. [34]Heejeong Nam, Quentin Le Lidec, Lucas Maes, Yann LeCun, and Randall Balestriero. Causal-JEPA: Learning world models through object-level latent interventions. arXiv preprint arXiv:2602.11389, 2026.
  35. [35]Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution. Advances in Neural Information Processing Systems, 36:43177–43201, 2023.
  36. [36]Anusri Pampari, Anna Shcherbina, Surag Nair, Jacob Schreiber, Aman Patel, Austin Wang, Soumya Kundu, Avanti Shrikumar, and Anshul Kundaje. Bias factorized, base-resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence syntax, transcription factor footprints and regulatory variants. Zenodo, 2023.DOI
  37. [37]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
  38. [38]Alexander Rives, Joshua Meier, Tom Sbiroli, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Deshpande, Myle Ott, Mark Slowik, Baptiste Almeida, et al. Biological structure and function emerged from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences (PNAS), 118(15): e2016239118, 2021.
  39. [39]Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bidirectional equivariant long-range dna sequence modeling. Proceedings of Machine Learning Research, 235: 43632, 2024.
  40. [41]Marina M. Scotti and Maurice S Swanson. Rna mis-splicing in disease. Nature Reviews Genetics, 17(1):19–32, 2016.DOI
  41. [42]Jin Su, Yan He, Shiyang You, Shiyu Jiang, Xibin Zhou, Xuting Zhang, Yuxuan Wang, Xining Su, Igor Tolstoy, Xing Chang, et al. A trimodal protein language model enables advanced protein searches. Nature Biotechnology, pp. 1–7, 2025.
  42. [43]Ziqi Tang, Nirali Somia, Yiyang Yu, and Peter K Koo. Evaluating the representational power of pre-trained dna language models for regulatory genomics. Genome Biology, 26(1):203, 2025.
  43. [44]Kirill Vishniakov, Karthik Viswanathan, Aleksandr Medvedev, Praveen K. Kanithi, Marco AF. Pimentel, Ronnie Rajan, and Shadab Khan. Genomic foundationless models: Pretraining does not promise performance. bioRxiv, pp. 2024–12, 2024.
  44. [45]Chengsen Wang, Qi Qi, Haifeng Sun, Zirui Zhuang, Bo He, Siying Liu, Jianxin Liao, and Jingyu Wang. From nucleotides to semantics: genomic representation learning via joint-embedding predictive architecture. bioRxiv, pp. 2026–04, 2026.DOI
  45. [46]Zhao Yang, Yi Duan, Jiwei Zhu, Ying Ba, Chuan Cao, and Bing Su. Extending sequence length is not all you need: Effective integration of multimodal signals for gene expression prediction. arXiv preprint arXiv:2602.21550, 2026.
  46. [47]Chengzhong Ye, Gonzalo Benegas, Carlos Albors, Jianan Canal Li, Sebastian Prillo, Peter D. Fields, Brian Clarke, and Yun S. Song. Predicting functional constraints across evolutionary timescales with phylogeny-informed genomic language models. bioRxiv, 2025.
  47. [48]Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In European conference on computer vision, pp. 668–684. Springer, 2022.arxiv.org/abs/2110.06848
  48. [49]Chang Yu, Siyuan Li, Zicheng Liu, Jingbo Zhou, Xianglong Guo, Kai Yu, Yuqing Zhou, Ken Li, Zelin Zhang, Zhen Lei, et al. CDRbridge: A cross-omics post-training bridge strategy for context-aware biological modeling. In The Fourteenth International Conference on Learning Representations, 2026.
  49. [50]Wei Zeng et al. Are genomic foundation models truly better? a systematic evaluation of pretraining strategies. bioRxiv, 2023. doi: 10.1101/2023.10.25.564012.
  50. [51]Y. Zhang et al. SPACE: Pretraining on functionally annotated genomic profiles with mixture-of-experts. Nature Machine Intelligence, 2024.
  51. [52]Yikun Zhang, Hao Zhang, Guo-Wei Li, He Wang, Xing Zhang, Xu Hong, Tingting Zhang, Liangsheng Wen, Yu Zhao, Jiuhong Jiang, et al. RESM: Capturing sequence and structure encoding of rnas by mapped transfer learning from ESM (evolutionary scale modeling) protein language model. bioRxiv, pp. 2025–08, 2025.
  52. [53]Zuobai Zhang, Minghao Xu, Aurelie Chen, Jiarui Lu, Riheng Chen, Xinyu Liu, and Jian Tang. Structure-aware protein self-supervised learning. Bioinformatics, 39(Supplement_1):i43–i52, 2023.
  53. [54]Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V. Davuluri, and Han Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023. doi: 10.48550/arXiv.2306.15006.

Paper details

vlnc:W_w4e4ppr3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents