betaFeedback

WorksComputer Science

Objective-Specialized Residual Streams in Manifold-Constrained Hyper-Connections

Abstract

Manifold-constrained Hyper-Connections (mHC) extend a Transformer's residual stream into multiple parallel streams with learned cross-stream connections. Prior work trains all streams toward a shared objective. We instead study whether assigning distinct learning objectives to different streams can improve task performance and induce objective-specific specialization. Using complementary lexical (sparse) and semantic (dense) retrieval objectives as a test case, we find that objective-specialized mHC consistently outperforms a non-specialized mHC model across the full sparse–dense fusion range, for both two- and four-stream models and at two model scales. We then examine how specialization changes the model internally using probes of stream representations, residual mixing, objective-specific gradient flow, and read/write gates. The specialized and non-specialized models behave similarly through most of the network but diverge in the final layers: the specialized model develops increasingly objective-specific gradient flow, distinct read/write routing patterns, and reduced cross-stream mixing, and streams assigned to different objectives become increasingly dissimilar while those trained toward the same objective remain highly similar. Fixing the residual mixing to the identity leaves the specialization benefit intact, indicating that the benefit stems from objective-specific read/write routing rather than learned cross-stream mixing. These results show that assigning distinct objectives to mHC streams can improve multi-objective retrieval while inducing structured, depth-dependent specialization in the residual computation.

Introduction

Manifold-constrained Hyper-Connections (mHC) ([22]) replace a Transformer’s single residual stream with multiple parallel streams (Zhu et al., 2024), whose interactions are learned at each layer’s attention and feed-forward modules. Existing applications of mHC jointly optimize all streams toward a shared objective ([22, 16, 14]), leaving open a basic question: what happens when different streams are assigned different objectives?

Distinct objectives can benefit from shared representations and computation ([2, 12]), while task-specific processing can be beneficial when objectives require different information ([15, 18]). This motivates objective-specialized mHC, in which different streams are assigned distinct learning objectives while retaining the ability to interact through learned connections.

We investigate two questions. RQ1: Can mHC streams specialize toward distinct learning objectives while improving joint task performance? (§ 4) RQ2: How does this specialization emerge across depth, as reflected in inter-stream representations, gradient flow, and read, write, and residual-mixing gates? (§ 5).

As a concrete test case, we consider two complementary information-retrieval objectives: lexical and semantic matching ([11, 9]). We train two- and four-stream mHC models, assigning each objective to a disjoint subset of streams, and evaluate at two model scales, ModernBERT-base and ModernBERT-large ([21]), to test whether the effect of specialization holds beyond a single model size. Across 13 BEIR benchmarks ([19]), the pattern is consistent at both scales: objective specialization improves average nDCG@10 from 0.4310 to 0.4486 with two streams and from 0.4410 to 0.4529 with four streams at the base scale, and from 0.4574 to 0.4667 and from 0.4573 to 0.4748, respectively, at the larger scale. In addition, we demonstrate that the specialized models outperform their non-specialized counterparts across the full sparse– dense fusion range at both scales.

Beyond retrieval performance, we analyze how objective specialization emerges within the mHC stack using complementary measurements of stream routing, gradient flow, and inter-stream representations. These diagnostics reveal a consistent depth-dependent pattern. Through most of the Transformer, specialized and non-specialized models exhibit similar residual mixing and read/write routing, while gradients from the two objectives remain broadly distributed across streams. In the final layers, however, the specialized models exhibit a distinct pattern absent in the non-specialized models: gradient flow becomes increasingly concentrated on the streams assigned to each objective, read/write routing becomes asymmetric, with the MLP increasingly reading from the sparse-objective streams and writing toward the dense-objective streams, and residual mixing moves toward the identity. Additionally, the representation analysis provides a clear contrast: in the non-specialized model, all streams remain highly similar throughout the network, whereas in the specialized model, streams assigned to different objectives become increasingly dissimilar in the final layers. These effects persist when increasing from two to four streams, suggesting that mHC supports shared computation at early depths followed by objective-specific processing near the output. Finally, fixing the residual mixing to the identity leaves the specialization benefit intact, indicating that the benefit stems from the objective-specific read/write routing rather than from learned cross-stream mixing.

Together, these results show how objective specialization in mHC emerges across depth, connecting retrieval gains to changes in stream representations, gradient flow, and routing. To our knowledge, we are the first to assign distinct learning objectives to different mHC streams and characterize the resulting specialization across the Transformer stack.

Related Work

Hyper-Connections

Residual connections provide an identity path that allows representations to propagate across layers without transformation. Hyper-Connections (HC) (Zhu et al. (2024)) generalize this paradigm by expanding the residual pathway into multiple parallel streams. HC allows the multiple streams to interact through learned connectivity patterns, providing greater flexibility in how information propagates through depth. Manifold-Constrained Hyper-Connections (mHC) (Xie et al. [22]) constrain these learned connections to the Birkhoff polytope (Sinkhorn and Knopp [17]), ensuring that the residual mixing matrices remain doubly stochastic and thereby preserve the stability of the residual pathway while retaining learned cross-stream interactions. Recent work has also begun to examine how the multiple residual streams introduced by HC and mHC are actually used after training (Alimaskina et al. [1]; Zhao et al. (2026)). Our work builds on this multi-stream formulation but changes how the streams are trained: rather than jointly optimizing all streams toward the same objective, we assign different objectives to different streams and study the resulting organization of information flow across depth.

Multi-Task and Multi-Objective Learning

Multi-task learning aims to share computation across tasks while reducing task interference ([2, 4]). Prior work addresses this by learning task-specific representations or routing capacity, using approaches such as cross-stitch networks and mixture-of-experts ([15, 13, 6]). Our setting instead attaches distinct objectives directly to different mHC streams within the residual computation, allowing the streams to remain coupled while developing objective-specific behavior.

Multi-Representation Retrieval

Modern information retrieval increasingly combines representations that capture different notions of relevance. Sparse retrieval emphasizes lexical matching and exact or token-level correspondence, whereas dense retrieval captures semantic similarity through continuous representations ([5, 11]). These signals are complementary: combining lexical and semantic matching can improve retrieval effectiveness and robustness over either representation alone ([11, 9]).

Recent retrieval models demonstrate the value of combining complementary relevance signals within a single model. BGE-M3 (Chen et al. [3]), for example, supports dense, sparse, and multi-vector retrieval within a unified encoder, showing that multiple retrieval representations can be supported and efficiently combined. We instead ask whether distinct learning objectives can be assigned directly to different internal mHC streams, and how such specialization emerges across the Transformer stack. We use sparse and dense retrieval as complementary objectives to study this question.

Retrofitting Pretrained Models with mHC

Closely related to our setup is the use of mHC as a retrofit onto an already-pretrained model. Oldenburg et al. ([16]) wrap a pretrained model’s residual stream with mHC for parameter-efficient fine-tuning. We likewise retrofit mHC onto a pretrained encoder, but differ in two respects: we fine-tune all parameters (both the backbone and the mHC routing), and we assign distinct objectives to different streams rather than optimizing all streams toward a common objective.

Task-Specialized Residual Streams in mHC

Hyper connection

Whereas a standard residual network maintains a single hidden representation xl∈Rdx_l \in\mathbb{R}^d per token throughout the Transformer stack, Hyper-Connections (HC) (Zhu et al., 2024) expand this hidden state into nn parallel streams. We denote the hidden state of a token at layer ll as Xl∈Rn×dX_l \in\mathbb{R}^{n \times d}, where the ss-th row vector xl(s)⊤∈Rdx_l^{(s)\top} \in\mathbb{R}^d represents the ss-th stream for s∈{0,…,n−1}s \in\{0,\ldots,n-1\}. At the input layer (l=0l = 0), the initial token embedding x0∈Rdx_0 \in\mathbb{R}^d is replicated across all nn streams, after which the streams propagate through the entire encoder stack.

At each Transformer sublayer fl(⋅)f_l(\cdot) (representing either Multi-Head Attention or a Feed-Forward Network), mHC ([22]) govern information routing across streams via three components: a read vector pl∈Rnp_l \in\mathbb{R}^n that determines how much each stream contributes to the sublayer’s input, a write vector ql∈Rnq_l \in\mathbb{R}^n that determines how much of the sublayer’s output is written back into each stream, and a mixing matrix Al∈Rn×nA_l \in\mathbb{R}^{n \times n} that governs how streams exchange information with one another along the residual path.

Aggregation (read). The read gate collapses the nn streams into a single vector using an elementwise weighted sum across streams. This vector is the only input to the sublayer:

ul=∑s=0n−1pl,sxl(s)∈Rd,vl=fl(ul)∈Rd.u_l = \sum_{s=0}^{n-1} p_{l,s} x_l^{(s)} \in\mathbb{R}^d,\qquad v_l = f_l(u_l) \in\mathbb{R}^d.

Hyper connection

Distribution (write and mixing). The single sublayer output vlv_l is split back into nn streams by the write gate, which assigns a per-stream scalar gain ql,sq_{l,s} to vlv_l. In parallel, the mixing matrix recombines the incoming streams along the residual path, and the two are summed to form the next-layer streams:

xl+1(s)=∑s~=0n−1Al,ss~xl(s~)+ql,svlx_{l+1}^{(s)} = \sum_{\tilde{s}=0}^{n-1} A_{l,s\tilde{s}} x_l^{(\tilde{s})} + q_{l,s}v_l

Routing components. The read, write, and mixing objects are not fixed ([22]): they are produced from the current streams by learned linear projections with bias terms. Let hl=RMSNorm⁡([xl(0);…;xl(n−1)])∈Rndh_l = \operatorname{RMSNorm}([x_l^{(0)};\ldots;x_l^{(n-1)}]) \in\mathbb{R}^{nd} be the flattened, normalized streams. Then

pl=σ(αpreWprehl+bpre),ql=2σ(αpostWposthl+bpost),Al=Sinkhorn⁡(reshape⁡(αresWreshl+bres)).\begin{aligned} p_l &= \sigma(\alpha^{\mathrm{pre}}W^{\mathrm{pre}}h_l+b^{\mathrm{pre}}),\\ q_l &= 2\sigma(\alpha^{\mathrm{post}}W^{\mathrm{post}}h_l+b^{\mathrm{post}}),\\ A_l &= \operatorname{Sinkhorn}(\operatorname{reshape}(\alpha^{\mathrm{res}}W^{\mathrm{res}}h_l+b^{\mathrm{res}})). \end{aligned}

where plp_l, qlq_l, and AlA_l are, respectively, the read, write, and mixing objects introduced above, W∙W^\bullet are learned projection weights, b∙b^\bullet are learned biases, and α∙\alpha^\bullet are learned scalar gains. The biases carry the static component of each gate, while the projections make the routing depend on the streams themselves.

Manifold constraint. The defining feature of mHC is that the residual mixing matrix AlA_l is constrained to be doubly stochastic via Sinkhorn-Knopp normalization ([17]), which helps stabilize signal and gradient propagation across depth ([22]).

Encoder for Information Retrieval

Encoder. We start from the publicly available, pretrained ModernBERT encoder ([21]), using both the base (L=22L = 22 layers, hidden size d=768d = 768)1 and large (L=28L = 28 layers, d=1024d = 1024)2 variants, and fine-tune each for retrieval. We retrofit mHC onto the pretrained encoder by wrapping every self-attention and MLP sublayer in an mHC block (Section 3.1) with n∈{2,4}n \in\{2,4\} streams. The retrofit is function-preserving at initialization: for every layer ll, the resulting stream xl+1(s)x^{(s)}_{l+1} is identical across all streams s∈{0,…,n−1}s \in\{0,\ldots,n-1\} and equal to the hidden state the pretrained ModernBERT model would produce at that layer (Appendix § 7). Fine-tuning therefore starts from the pretrained ModernBERT function, with any subsequent stream specialization learned during retrieval training.

Retrieval heads. Let xL,t(s)∈Rdx^{(s)}_{L,t} \in\mathbb{R}^{d} denote the final-layer (l=Ll=L) contextual representation of token tt in stream ss, where s∈{0,…,n−1}s \in\{0,\ldots,n-1\}; retrieval heads read only the final-layer streams. Let V\mathcal{V} denote the set of content token positions, excluding special tokens. Each retrieval head reads a designated subset of streams, Sdn\mathcal{S}_{\mathrm{dn}} for the dense head and Ssp\mathcal{S}_{\mathrm{sp}} for the sparse head. The selected streams are mean-pooled at each token:

xˉL,tdn=1∣Sdn∣∑s∈SdnxL,t(s),xˉL,tsp=1∣Ssp∣∑s∈SspxL,t(s).\bar{x}^{\mathrm{dn}}_{L,t} = \frac{1}{|\mathcal{S}_{\mathrm{dn}}|} \sum_{s \in\mathcal{S}_{\mathrm{dn}}} x^{(s)}_{L,t}, \qquad\bar{x}^{\mathrm{sp}}_{L,t} = \frac{1}{|\mathcal{S}_{\mathrm{sp}}|} \sum_{s \in\mathcal{S}_{\mathrm{sp}}} x^{(s)}_{L,t}.

The dense representation is obtained by masked mean pooling over content tokens followed by L2L_2 normalization:

e~dense=1∣V∣∑t∈VxˉL,tdn,edense=e~dense∥e~dense∥2.\tilde{e}^{\mathrm{dense}} = \frac{1}{|\mathcal{V}|} \sum_{t \in\mathcal{V}} \bar{x}^{\mathrm{dn}}_{L,t}, \qquad e^{\mathrm{dense}} = \frac{\tilde{e}^{\mathrm{dense}}}{\lVert\tilde{e}^{\mathrm{dense}} \rVert_2}.

The sparse head produces a vocabulary-sized, non-negative lexical representation. A learned linear projection assigns a scalar importance to each token. The importance is rectified by ReLU and masked to zero on non-content tokens:

wt=ReLU⁡(uTxˉL,tsp+b),w_t = \operatorname{ReLU}\left(u^{\mathsf{T}}\bar{x}^{\mathrm{sp}}_{L,t} + b\right),

where u∈Rdu \in\mathbb{R}^{d} and b∈Rb \in\mathbb{R} are learned parameters and wt=0w_t = 0 for t∉Vt \notin\mathcal{V}. Let id⁡(t)∈{1,…,∣W∣}\operatorname{id}(t) \in\{1,\ldots,|\mathcal{W}|\} denote the vocabulary id of token tt, where W\mathcal{W} is the vocabulary. The token weights are then max-pooled according to vocabulary id:

evsparse=max⁡t∈Vid⁡(t)=vwt,v∈{1,…,∣W∣}.e^{\mathrm{sparse}}_v = \max_{\substack{t \in\mathcal{V} \\ \operatorname{id}(t)=v}} w_t, \qquad v \in\{1,\ldots,|\mathcal{W}|\}.

If vocabulary id vv does not occur among the content tokens, we set evsparse=0e^{\mathrm{sparse}}_v = 0. Thus, the sparse representation contains non-zero weights only for terms occurring in the input, following the lexical term-weighting formulation of uniCOIL (Lin and Ma, [10]), BGE-M3 (Chen et al., [3]).

Scoring. Let qq and dd denote a query and a candidate document, respectively. Both are encoded with the same encoder, each independently producing representations eqsparsee^{\mathrm{sparse}}_q, eqdensee^{\mathrm{dense}}_q and edsparsee^{\mathrm{sparse}}_d, eddensee^{\mathrm{dense}}_d via the retrieval heads above. Retrieval scores are computed as inner products of matching representation types:

so(q,d)=⟨eqo,edo⟩,o∈{sparse,dense}.(1)s^{o}(q,d) = \langle e^{o}_q,e^{o}_d\rangle, \qquad o \in\{\mathrm{sparse},\mathrm{dense}\}. \tag*{(1)}

Stream specialization. We consider three configurations that differ only in which streams are read by the retrieval heads. In the objective-specialized configuration (n∈{2,4}n \in\{2,4\}), Ssp={0,…,n/2−1}\mathcal{S}_{\mathrm{sp}} = \{0,\ldots,n/2-1\} and Sdn={n/2,…,n−1}\mathcal{S}_{\mathrm{dn}} = \{n/2,\ldots,n-1\}: the sparse and dense retrieval heads read disjoint stream groups, allocating the first half of the streams exclusively to the sparse objective and the second half to the dense objective. This decouples task readouts at the output layer while mHC routing maintains inter-stream coupling throughout the encoder stack. In the non-specialized configuration (n∈{2,4}n \in\{2,4\}), Ssp=Sdn={0,…,n−1}\mathcal{S}_{\mathrm{sp}} = \mathcal{S}_{\mathrm{dn}} = \{0,\ldots,n-1\}: both retrieval heads read the mean-pooled representation across all streams, forcing both objectives to jointly optimize a shared representation space. Finally, in the single-stream configuration (n=1n=1), Ssp=Sdn={0}\mathcal{S}_{\mathrm{sp}} = \mathcal{S}_{\mathrm{dn}} = \{0\}: with a single stream, there is no cross-stream routing and the encoder is exactly the original ModernBERT model, providing the dedicated single-objective reference points.

Training Setup

Training data. We fine-tune all models on the Natural Questions (NQ) ([8]) dataset. We use the first 90k training pairs for fine-tuning and reserve the remaining pairs as a validation set for checkpoint selection3. Queries and passages are truncated to 32 and 256 tokens, respectively4.

Dense and sparse objectives. We optimize both sparse and dense representations using a contrastive retrieval loss ([20]) with in-batch negatives. Let so(q,d)s^{o}(q,d) denote the relevance score between query qq and document dd for retrieval type t∈dense,sparset \in\mathrm{dense},\mathrm{sparse}. For a batch of BB query– positive pairs, the per-objective loss is

Schematic comparison of the three encoder configurations studied in this work

Figure 1. Schematic comparison of the three encoder configurations studied in this work. In all three, streams propagate through the full encoder stack and the retrieval heads read only the final-layer (l=Ll = L) stream representations. (left) Single-stream: the standard ModernBERT encoder, with a single residual stream read by both the sparse and dense retrieval heads. (middle) Non-specialized mHC: the residual stream is expanded into nn parallel streams with learned cross-stream routing. Both retrieval heads operate on the mean-pooled representation obtained from all nn streams, so the sparse and dense objectives jointly optimize the same stream representation. (right) Objective-specialized mHC: the sparse and dense heads instead read disjoint sets of final-layer streams, so each objective selectively trains its assigned stream(s). For clarity the schematic depicts n=2n = 2 streams; we study both n=2n = 2 and n=4n = 4 in our experiments.

Lo=−1B∑i=1Blog⁡exp⁡(so(qi,pi+)/τt)∑d∈Cexp⁡(so(qi,d)/τt)(2)\mathcal{L}_{o} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s^{o}(q_i,p_i^{+})/\tau_t)}{\sum_{d\in C}\exp(s^{o}(q_i,d)/\tau_t)} \tag*{(2)}

where τdense\tau_{\mathrm{dense}} and τsparse\tau_{\mathrm{sparse}} are fixed temperature hyperparameters and CC is the batch document collection. Thus, for each query qiq_i, its paired passage pi+p_i^{+} is treated as the positive, while the remaining B−1B-1 passages in the batch serve as in-batch negatives.

Combined training loss. The multi-objective models are trained on the sum of the two objectives,

L=Ldense+λsparseLsparse,(3)\mathcal{L} = \mathcal{L}_{\mathrm{dense}} + \lambda_{\mathrm{sparse}}\mathcal{L}_{\mathrm{sparse}}, \tag*{(3)}

where λsparse≥0\lambda_{\mathrm{sparse}} \ge0 sets the weight of the sparse objective relative to the dense one. Single-objective baselines use only the corresponding term. In the specialized model, Ldense\mathcal{L}_{\mathrm{dense}} and Lsparse\mathcal{L}_{\mathrm{sparse}} are computed from disjoint streams (§ 3.3), so each term supervises its own set of streams.

Training parameters. All model variants are fine-tuned using identical optimization setups: the Adam optimizer (Kingma and Ba [7]) with no weight decay, a linear warmup-and-decay learning-rate schedule with a 5% warmup phase, gradient clipping at a maximum norm of 1.0, and bfloat16 mixed precision. Each model is trained for an epoch on two NVIDIA H100 GPUs. We evaluate checkpoints every 250 steps on the held-out NQ validation set and select the checkpoint with the highest mean triplet accuracy across the two retrieval heads. For each objective o∈{dense,sparse}o \in\{\mathrm{dense},\mathrm{sparse}\}, triplet accuracy is the fraction of validation triplets satisfying so(q,d+)>so(q,d−)s^{o}(q,d^{+}) > s^{o}(q,d^{-}). To ensure a fair comparison, all model families are optimized over the same hyperparameter grid: learning rate η∈{5×10−5,8×10−5,1×10−4}\eta\in\{5 \times10^{-5}, 8 \times10^{-5}, 1 \times10^{-4}\}, dense loss temperatures τdense∈{0.01,0.05}\tau_{\mathrm{dense}} \in\{0.01, 0.05\}, sparse loss temperatures τsparse∈{0.01,0.05}\tau_{\mathrm{sparse}} \in\{0.01, 0.05\}, sparse loss weight λsparse∈{0.1}\lambda_{\mathrm{sparse}} \in\{0.1\}, and batch size B∈{64,128}B \in\{64, 128\}.

Inference and Hybrid Fusion. At inference, the dense and sparse heads act as independent retrievers: each scores candidates via so(q,d)s^{o}(q,d) (Eq. (1)) within its own feature space and returns its top-kk candidates. We fuse the two rankings with a score-based combination over the union of candidates (Chen et al., [3]):

sα(q,d)=α sdense(q,d)+(1−α) ssparse(q,d),(4)s_{\alpha}(q,d) = \alpha\,s^{\mathrm{dense}}(q,d) + (1-\alpha)\,s^{\mathrm{sparse}}(q,d), \tag*{(4)}

where α∈[0,1]\alpha\in[0,1] and so(q,d)=0s^{o}(q,d)=0 if dd is outside head oo’s top-kk candidates.5 The endpoints recover single-head retrieval (α=0\alpha= 0: pure sparse; α=1\alpha= 1: pure dense). We evaluate performance across the full continuum α∈[0,1]\alpha\in[0,1] (§ 4).

Objective-Specialized Residual Streams

To address RQ1, we ask whether mHC residual streams can specialize to distinct learning objectives while improving multi-objective retrieval. We compare objective-specialized mHC (mHC+spec) against three baselines: (1) dedicated single-objective models, which provide single-task reference points; (2) a **standard non-mHC joint model (Sparse+Dense)(Sparse+Dense), which combines the two objectives within a conventional single residual stream; and (3) a non-specialized mHC model (mHCmHC no-spec), in which both retrieval heads read the pooled mHC streams. The third comparison is the key ablation: mHCmHC no-spec and mHC+mHC+spec use the same architecture, training data, and optimization budget, differing only in whether the objectives are assigned to distinct streams. We evaluate both n=2n = 2 and n=4n = 4 streams. We evaluate our system on 13 BEIR datasets (Thakur et al., [19]) and report results averaged over two training seeds.

Specialization improves retrieval across fusion weights

Because the multi-objective models produce both sparse and dense representations, we evaluate retrieval across the full sparse–dense fusion range using the score fusion defined in Eq. (4), from pure sparse (α=0\alpha= 0) to pure dense (α=1\alpha= 1). Figure 2 reports average BEIR nDCG@10 across this sweep.

Figure 2. Average BEIR nDCG@10 as a function of the fusion weight α\alpha, for ModernBERT-base (left) and ModernBERT-large (right); both mean over two seeds. α=0\alpha= 0 and α=1\alpha= 1 correspond to pure sparse and pure dense retrieval, respectively. Color encodes objective specialization (red: specialized, blue: non-specialized); Sparse+Dense (n=1n=1, grey) has no stream routing. Specialized models (red) dominate at every α\alpha for both stream counts and at both base/large model scales.

The results show three findings. First, at the base scale, four-stream mHC models outperform the conventional Sparse+Dense model, while the two-stream models are comparable. This comparison is secondary, since mHC introduces additional stream-specific parameters and routing capacity.6 Second, and more importantly, mHC+mHC+spec consistently outperforms mHCmHC no-spec at every fusion weight for both n=2n = 2 and n=4n = 4; Figure 2 shows that this also holds at a larger model scale (ModernBERT-large). At α=0.9\alpha= 0.9 and the base scale, specialization improves average nDCG@10 from 0.4310 to 0.4486 with two streams and from 0.4410 to 0.4529 with four streams (ModernBERT-large: 0.4574 →\to 0.4667 and 0.4573 →\to 0.4748, respectively). Thus, the improvement is not explained simply by the additional capacity of mHC, but by how that capacity is organized across objectives. Third, the specialization benefit persists across the full sparse–dense fusion range rather than depending on a particular operating point, at both model scales.

All five multi-objective models achieve their best average score at α=0.9\alpha= 0.9 at both scales, which we use for the per-dataset analysis below. At the base scale, increasing the number of streams also improves both specialized and non-specialized mHC, indicating complementary benefits from additional stream capacity and objective specialization.

Per-dataset multi-objective retrieval

Table 1 reports seed-averaged nDCG@10 on all 13 BEIR datasets at α=0.9\alpha= 0.9 using ModernBERT-base; the corresponding per-dataset results for ModernBERT-large are reported in Appendix § 9. The advantage of objective specialization is not confined to the aggregate score: mHC+spec improves over mHC no-spec on 9 of the 13 datasets. The improvements are not universal, and the exceptions are the same across stream counts: on FEVER, HotpotQA, SciFact, and Webis-Touché2020.

ModelArguAnaC-FEVERDBPediaFEVERFiQAHotpotQANFCNQQuoraSciDocsSciFTREC-CTouchéAvg
n=1n = 1
Sparse-only0.39430.10520.14840.63840.19530.58950.22190.16000.67220.11190.64920.53810.23720.3585
Dense-only0.39740.21160.27020.50490.23730.37970.24600.45690.84790.13280.55130.67840.22010.3950
Sparse+Dense0.38330.22170.25520.65250.26490.56780.26110.41100.82740.13100.67070.69900.26430.4315
n=2n = 2
mHC no-spec0.38270.21320.25790.65290.27180.56960.27080.40790.82290.13460.67090.68430.26340.4310
mHC spec0.42780.23940.29190.62940.28850.52260.27670.46810.84830.15350.66720.75850.25980.4486
n=4n = 4
mHC no-spec0.40680.23040.25840.66710.27520.57160.26940.41540.83010.13660.67510.72800.26880.4410
mHC spec0.42870.24190.29910.63850.28720.52260.28480.47570.85160.15200.66890.76920.26760.4529

Table 1. BEIR nDCG@10 across all 13 datasets using ModernBERT-base. Sparse-only and Dense-only are evaluated in their native retrieval mode; Sparse+Dense and the mHC models are hybrid retrievers fused at α=0.9\alpha= 0.9, the shared empirical optimum across all multi-objective models. Underline marks the best score within each stream count nn; bold marks the best score for that dataset across all configurations. The advantage of objective specialization is not confined to the aggregate score; it extends to a majority of individual datasets and holds at both n=2n = 2 and n=4n = 4. Excluding the in-domain NQ dataset, mHC+specmHC+spec still improves over mHCmHC no-spec on the remaining 12 datasets (0.4329 →\to 0.4470 at n=2n=2; 0.4431 →\to 0.4510 at n=4n=4). Results reported here are mean over two seeds. Per seed variation is provided in Appendix 5. Results with ModernBERT-large (Table 4) show the same trend: mHC+mHC+spec improves over mHCmHC no-spec at both n=2n = 2 and n=4n = 4, indicating that the benefit of objective specialization is robust to model scale rather than specific to a single model size.

How specialization reshapes the streams

Since the specialized and non-specialized models have the same architecture and capacity, their performance difference raises a more specific question: how does assigning different objectives to the streams change the internal computation? We therefore ask whether, where, and how the two models diverge internally (RQ2). The models share the same architecture, training data, and optimization budget and differ only in whether or not the retrieval heads read disjoint stream groups, allowing us to attribute systematic internal differences to objective specialization.

We examine four complementary signals across encoder depth ℓ\ell at the final checkpoint: (1) stream cosine similarity, which measures whether the stream representations diverge; (2) residual mixing, which measures how strongly information is exchanged between streams; (3) objective-specific gradient flow, which measures whether each objective preferentially trains its assigned stream; and (4) read and write gate shifts, which reveal how routing in MLP and attention submodules change to support this specialization. All curves are means over four seeds; the lower layers (ℓ<10\ell< 10) are flat across probes and are omitted. Figure 3 summarizes the four depth-resolved probes, measured at the MLP sublayers. The corresponding results for the attention sublayers for residual mixing diag⁡(A)\operatorname{diag}(A) and read/write gates are provided in Appendix § 8.

Figure 3. Four depth-resolved specialization probes at the final checkpoint, for n=2n=2 (top) and n=4n=4 (bottom) streams (lines are the mean and shaded bands ±1\pm1 standard deviation over 4 seeds; flat layers 0–9 omitted). From left: stream cosine similarity, residual-mixing identity retention diag⁡(A)\operatorname{diag}(A), per-objective gradient share on the assigned stream(s), and read/write gate shift. Red: specialized; blue: non-specialized control. Solid vs. dashed distinguishes the two objectives / gates / stream-pair types (legends). Metrics are defined and discussed in § 5.

Representational divergence. We first ask whether objective specialization produces different stream representations. At each layer, we compute the cosine similarity between token-averaged stream states. The non-specialized models remain highly similar across streams, whereas the specialized models increasingly separate streams assigned to different objectives in the final layers. With two streams, cosine similarity falls to ≈0.3\approx0.3 at layer 21; with four streams, streams assigned to the same objective remain nearly identical (cos⁡≈0.99\cos\approx0.99), while cross-objective pairs diverge (cos⁡≈0.26\cos\approx0.26) (Figure 3). Thus, specialization organizes the streams into objective-specific groups rather than simply replicating the same representation across all streams.7

Reduced residual mixing. We next ask whether the streams remain strongly coupled through the residual mixing matrix AA as their representations diverge. We measure the mean diagonal 1n∑iAℓ[i,i]\frac{1}{n}\sum_i A_{\ell}[i,i] of the doubly stochastic matrix AℓA_{\ell}, where values near 1 indicate that AℓA_{\ell} approaches the identity and cross-stream mixing is reduced. In the specialized model, diag⁡(Aℓ)\operatorname{diag}(A_{\ell}) increases sharply in the final layers, approaching 1.0 for n=2n = 2 and reaching ≈0.8\approx0.8 for n=4n = 4; the non-specialized model remains near its initialization value (0.73 for n=2n = 2, 0.48 for n=4n = 4) throughout. Thus, representational divergence is accompanied by a marked reduction in cross-stream mixing in the specialized model. § 5.1 tests whether this reduction is needed for the specialization benefit.

Objective-specific gradient flow. We next ask whether representational divergence is accompanied by objective-specific training signals. For each objective o∈{sparse,dense}o \in\{\mathrm{sparse}, \mathrm{dense}\}, we back-propagate its loss separately and measure the gradient magnitude it delivers to each stream, kk, at layer ℓ\ell:

gℓo[k]=∥∂Lo/∂xℓ(k)∥∑j∥∂Lo/∂xℓ(j)∥,∑kgℓo[k]=1.g_{\ell}^{o}[k] = \frac{\left\lVert\partial\mathcal{L}_{o}/\partial x_{\ell}^{(k)} \right\rVert}{\sum_j \left\lVert\partial\mathcal{L}_{o}/\partial x_{\ell}^{(j)} \right\rVert}, \qquad\sum_k g_{\ell}^{o}[k] = 1.

We report the assigned-stream share, ∑k∈Sogℓo[k]\sum_{k\in S_o} g_{\ell}^{o}[k], where SoS_o is the set of streams assigned to objective oo (for n=2n=2, Ssparse={0}S_{\mathrm{sparse}}=\{0\} and Sdense={1}S_{\mathrm{dense}}=\{1\}; for n=4n=4, {0,1}\{0,1\} and {2,3}\{2,3\}). A value near ∣So∣/n=0.5|S_o|/n = 0.5 indicates that the objective trains all streams equally, whereas a value near 1 indicates concentration on its assigned stream. In the specialized model, both objectives increasingly concentrate their gradients on their assigned streams in the upper layers. The sparse objective localizes earlier and more sharply, while the dense objective remains more distributed until the final layers. For the non-specialized control no retrieval head reads a designated subset of streams, so no stream is intrinsically tied to an objective. We nonetheless evaluate the probe using the same index sets SoS_o that specialization would impose, treating the partition as a fixed measurement convention rather than a property of the model. Because nothing links either objective to those streams, each objective’s gradient spreads evenly across all streams and the share stays near 0.50.5 throughout depth. This flat profile is the appropriate null reference: it is exactly what the probe returns when the stream partition carries no objective-specific information, so that any departure from it in the specialized model reflects specialization rather than a property of the metric.

Read/write gate specialization. Finally, we examine how the MLP routing reorganizes to support this objective-specific processing. Recall that at each layer the read gate emits a vector of per-stream read weights pℓ∈Rnp_\ell\in\mathbb{R}^n and the write gate a vector of per-stream write weights qℓ∈Rnq_\ell\in\mathbb{R}^n (§ 3). For a gate gℓ∈{pℓ,qℓ}g_\ell\in\{p_\ell, q_\ell\} we define its sparse-stream share as the fraction of its total mass placed on the stream(s) assigned to the sparse objective,

s(gℓ)=∑k∈sparsegℓ[k]∑j∈{sparse,dense}gℓ[j],s(g_\ell) = \frac{\sum_{k \in\mathrm{sparse}} g_\ell[k]}{\sum_{j \in\{\mathrm{sparse,dense\}}} g_\ell[j]},

which for n=2n=2 reduces to the weight on the single sparse-assigned stream. Because our initialization places a small amount of random mass on each stream, we report the change in this share from initialization to the final checkpoint, Δs(gℓ)=s(gℓ)∣T−s(gℓ)∣0\Delta s(g_\ell) = s(g_\ell)|_T - s(g_\ell)|_0, where TT denotes the final training step. This is the quantity plotted in Figure 3. A positive Δs\Delta s means the gate (read or write) moves mass toward the sparse-assigned stream, a negative value toward the dense-assigned stream.

In the specialized models, this pattern holds for both n=2n=2 and n=4n=4: the read gate increases its sparse-stream share (Δs(pL)≈+0.3\Delta s(p_L) \approx+0.3) while the write gate decreases it (Δs(qL)≈−0.25\Delta s(q_L) \approx-0.25). Since both gates start near an even split, the MLP ends up reading predominantly from the sparse-assigned streams and writing predominantly to the dense-assigned streams. The non-specialized controls show essentially no shift in either gate (Δs≈0\Delta s \approx0).

Taken together, the four probes reveal a consistent depth-dependent pattern. The specialized and non-specialized models remain similar through most of the network, but diverge in the final layers: streams assigned to different objectives become more distinct, objective-specific gradients concentrate on their assigned streams, the gates develop a directional read/write pattern (preferentially reading from sparse-assigned streams and writing to dense-assigned streams), and residual mixing decreases. The results therefore connect the downstream benefit of objective specialization to a concrete change in the internal computation of mHC rather than to a difference in capacity alone.

Read/write routing, not mixing, drives the specialization benefit

Specialization drives the residual-mixing matrix toward the identity in the final layers. This suggests a hypothesis: the specialization benefit stems from the objective-specific read/write routing rather than from learned cross-stream mixing. To test it, we fix Aℓ=IA_\ell= I at every layer, so that each stream is carried forward unmixed and streams interact only through the read and write gates, and retrain both the specialized and non-specialized models with n=4n = 4 (Table 2). Fixing the mixing to the identity changes average nDCG@10 by less than the variation across seeds, and this holds with or without specialization: mHC no-spec scores 0.4410 with learned and 0.4415 with identity mixing, and mHC+spec scores 0.4529 and 0.4546, respectively. Crucially, the specialization gain is preserved (+0.0119 with learned mixing, +0.0131 with identity mixing), well beyond the seed-to-seed standard deviation. These results support the hypothesis: learned cross-stream mixing is not required for the specialization benefit, which instead stems from the objective-specific read/write routing.

ModelMixing AAnDCG@10 (α=0.9\alpha=0.9)
mHC no-speclearned0.4410±0.00230.4410 \pm0.0023
mHC speclearned0.4529±0.00190.4529 \pm0.0019
mHC no-specidentity0.4415±0.00250.4415 \pm0.0025
mHC specidentity0.4546±0.00010.4546 \pm0.0001

Table 2. Fixing the residual-mixing matrix to the identity (Aℓ=IA_\ell=I at every layer). ModernBERT-base, n=4n=4, average BEIR nDCG@10 over 13 datasets at α=0.9\alpha=0.9; mean ±\pm standard deviation over two seeds.

Conclusion

We studied whether mHC residual streams can be assigned distinct learning objectives, using sparse and dense retrieval as a testbed. Specialization consistently improves retrieval over non-specialized mHC models of identical capacity, training data, and optimization budget, across the full sparse–dense fusion range and for both n=2n=2 and n=4n=4 streams, and this holds at both ModernBERT-base and ModernBERT-large scales. The specialized models also improve over the conventional single-stream joint retriever, with the n=4n=4 model achieving the best BEIR score at both scales.

Our analyses further show that specialization emerges primarily in the final layers. There, stream representations diverge, objective-specific gradients concentrate on assigned streams, the MLP gates develop a read-from-sparse / write-to-dense pattern, and residual mixing moves toward the identity. At n=4n=4, streams assigned to the same objective remain similar, forming two coherent objective groups. Fixing the residual mixing to the identity leaves the specialization benefit intact, indicating that it stems from the objective-specific read/write routing rather than from learned cross-stream mixing. Together, these results show that assigning distinct objectives to mHC streams can both improve downstream performance and induce structured, depth-localized specialization within the streams.

Limitations

Objective diversity. We study a single pair of objectives—lexical (sparse) and semantic (dense) retrieval—chosen because they are related but distinct. We do not systematically vary the degree of objective divergence, and therefore cannot determine how specialization changes as objectives become more similar, complementary, or conflicting.

Task scope. Our experiments cover only two scales (ModernBERT-base and ModernBERT-base) with n∈{2,4}n \in\{2,4\} streams, and encoder-based retrieval. We therefore do not establish whether the observed performance gains or late-layer specialization patterns extend to other model scales, other numbers of streams or objectives, or decoder-based and generative tasks.

Equal allocation of stream capacity across objectives. In our specialized mHC models, we allocated an equal number of streams to the sparse and dense objectives. We leave the effect of asymmetric stream allocation on downstream performance to future work.

mHC routing parameter initialization

For initialization choices, we follow prior work on retrofitting pretrained Transformers with mHC [16]. Each mHC block is initialized so that, at the start of fine-tuning, its routing is static (independent of the input token’s stream representations) and the block behaves like an ordinary residual sublayer (Section 3.1). The projection weights for all three routing components are zero-initialized (Wpre,Wpost,Wres=0W^{\mathrm{pre}}, W^{\mathrm{post}}, W^{\mathrm{res}} = 0), so each gate is determined solely by its bias. The input-dependent routing is therefore inactive at initialization and is introduced gradually as the projection weights and the scalar gains αpre,αpost,αres\alpha^{\mathrm{pre}}, \alpha^{\mathrm{post}}, \alpha^{\mathrm{res}} (which scale the dynamic term before the bias is added) are learned; the gains are initialized to a small value, 10−210^{-2}. The remaining biases make the block reduce, at initialization, to an ordinary residual sublayer xl+1=xl+fl(xl)x_{l+1} = x_l + f_l(x_l).

Write bias: The write gate initializes bpost=0b^{\mathrm{post}} = 0, giving unit gates since q=2σ(0)=1q = 2\sigma(0) = 1. So, the sublayer (MLP/attention) output is written equally to all the streams.

Residual-mixing bias. We set bresb^{\mathrm{res}} to an identity-patterned logit (diagonal entries 1, off-diagonal 0). Sinkhorn normalization turns this into a doubly-stochastic matrix with diagonal Aii=e/(e+n−1)A_{ii} = e/(e+n-1) and off-diagonals Aij=1/(e+n−1)A_{ij} = 1/(e+n-1) (Aii≈0.73A_{ii} \approx0.73 for n=2n=2, and 0.480.48 for n=4n=4); the diagonal is larger than each off-diagonal by the constant factor ee. These exact values are immaterial at initialization: because all nn streams hold identical copies of the pretrained hidden state, the residual mix ∑jA[i,j] x(j)\sum_j A[i,j]\,x^{(j)} returns that shared value for any row-stochastic AA, which Sinkhorn guarantees. The identity-patterned bias therefore does not alter the function at initialization. A uniform bresb^{\mathrm{res}} would likewise preserve the pretrained function.

Read gate: The read-gate bias bpreb^{\mathrm{pre}} is used to break the symmetry among streams. We use the same UNIFORM-NOISY scheme for both the specialized and non-specialized models: each stream is initialized to read approximately equally, with target σ(bkpre)=1/n\sigma(b_k^{\mathrm{pre}}) = 1/n at every layer, perturbed by small per-stream Gaussian noise on the logits (σnoise=10−2\sigma_{\mathrm{noise}} = 10^{-2}), after which the biases are renormalized so that ∑kσ(bkpre)=1\sum_k \sigma(b_k^{\mathrm{pre}}) = 1 exactly—keeping the stream aggregation a convex combination (and hence function-preserving) at initialization. We deliberately apply the identical initialization to both conditions so that the only difference between them is whether the retrieval heads read disjoint streams (specialized) or the pooled streams (non-specialized), removing initialization as a confound.

All Sinkhorn normalizations use 20 iterations and ϵ=10−6\epsilon= 10^{-6}. The exact values are reported in Table 3.

ParameterShapeInitialization
WpreW^{\mathrm{pre}}n×ndn \times nd0
WpostW^{\mathrm{post}}n×ndn \times nd0
WresW^{\mathrm{res}}n2×dn^2 \times d0
bpreb^{\mathrm{pre}}nnlogit⁡(1/n)+ϵ\operatorname{logit}(1/n) + \epsilon, renorm.
bpostb^{\mathrm{post}}nn0
bresb^{\mathrm{res}}n2n^2diag 1, off-diag 0
αpre/post/res\alpha^{\mathrm{pre/post/res}}110−210^{-2}
RMSNorm γ\gammandnd1

Table 3. Initialization of each mHC block. Projection weights are zero-initialized so routing starts static; the biases place the block near an identity residual layer. The read-gate bias uses the UNIFORM-NOISY scheme (target σ(bkpre)=1/n\sigma(b_k^{\mathrm{pre}}) = 1/n plus noise ϵ∼N(0,σnoise2)\epsilon\sim\mathcal{N}(0,\sigma_{\mathrm{noise}}^2), σnoise=10−2\sigma_{\mathrm{noise}}=10^{-2}, renormalized to sum to one) for both specialized and non-specialized models. At initialization this yields read gate p≈[1/n,…,1/n]p \approx[1/n,\ldots,1/n] (summing to 1), write gate q=[1,…,1]q = [1,\ldots,1], and a doubly-stochastic mixing matrix AA with its largest mass on the diagonal.

Finally, the mHC routing parameters are optimized with a separate, higher learning rate than the rest of the network. During fine-tuning we place all routing parameters—the three projections, their scalar gains, and the block’s RMSNorm—in their own Adam parameter group at learning rate 10−310^{-3}, while the ModernBERT backbone and the retrieval heads train at the base learning rate. This lets the small, randomly perturbed routing parameters adapt quickly enough to develop specialization within a single fine-tuning epoch without destabilizing the pretrained backbone.

How specialization reshapes the streams (attention sublayer)

Section 5 reports the routing probes for the MLP sublayer; here we provide the corresponding residual-mixing and read/write gate probes for the attention sublayer.

Figure 4 shows that attention mirrors the MLP in the residual-mixing gate: under specialization, AA moves toward the identity in the top layers (mean diagonal ≈0.99\approx0.99 at n=2n=2 and ≈0.93\approx0.93 at n=4n=4, versus ≈0.72\approx0.72 and ≈0.47\approx0.47 for the non-specialized control). The gate behavior, however, differs from the MLP counterpart. Whereas the MLP shifts its read gate toward the sparse stream and its write gate toward the dense stream, attention leaves its read gate essentially unchanged (Δs(pℓ)≈0\Delta s(p_\ell) \approx0) and moves only its write gate—strongly—toward the dense-assigned stream(s) (Δs(qL)≈−0.44\Delta s(q_L) \approx-0.44 at n=2n=2 and −0.45-0.45 at n=4n=4). Attention therefore specializes by routing its write rather than its read: it keeps reading from all streams but commits its output predominantly to the dense streams.

Figure 4. Attention-sublayer routing probes at the final checkpoint (lines are the mean and shaded bands ±1\pm1 standard deviation over 4 seeds; flat layers 0–9 omitted). Left pair: n=2n=2; right pair: n=4n=4. Within each pair: mean diagonal of the residual-mixing matrix diag⁡(A)\operatorname{diag}(A), and the shift in each gate’s sparse-stream share Δ\Delta (read gate pp solid, write gate qq dashed). Red: specialized; blue: non-specialized control. Like the MLP sublayer, attention drives AA toward the identity under specialization; unlike the MLP, its read gate does not shift (Δs(pℓ)≈0\Delta s(p_\ell) \approx0) while its write gate moves strongly toward the dense-assigned stream(s) (Δs(qL)≈−0.44\Delta s(q_L) \approx-0.44), i.e. attention specializes its write but not its read. Metrics defined in § 5.

As in the main text, these patterns are consistent across both stream counts and all four seeds.

Results at a Larger Model Scale

Table 4 reports the same comparison as Table 1, using ModernBERT-large in place of ModernBERT-base. The specialization advantage reported in § 4.2 holds at this larger scale as well.

ModelArguAnaC-FEVERDBPediaFEVERFiQAHotpotQANFCNQQuoraSciDocsSciFTREC-CTouchéAvg
n=1n = 1
Sparse-only0.39670.09540.13480.59310.21330.55680.22510.16540.67610.11180.65070.51780.23070.3514
Dense-only0.41100.21570.30720.53510.31030.40610.28000.50630.85620.16110.59720.69960.22050.4236
Sparse+Dense0.42680.22040.27110.65860.30950.58380.28280.44880.84430.13980.69700.76140.29390.4568
n=2n = 2
mHC no-spec0.41030.21080.28380.66980.30760.58640.29830.45460.84530.14210.69110.76210.28360.4574
mHC spec0.45090.22850.32370.63530.34480.53320.31150.51820.86270.16550.68890.75340.25060.4667
n=4n = 4
mHC no-spec0.40270.20010.28140.66690.30810.58870.29840.45730.84370.13580.69400.77210.29540.4573
mHC spec0.48320.23890.32700.64380.35440.54470.31010.52000.86370.16280.69010.78020.25310.4748

Table 4. BEIR nDCG@10 across all 13 datasets using ModernBERT-large, mean over two seeds (42 and 1234). Sparse-only and Dense-only are evaluated in their native retrieval mode; Sparse+Dense and the mHC models are hybrid retrievers fused at α=0.9\alpha= 0.9. Underline marks the best score within each stream count nn; bold marks the best score for that dataset across all configurations.

Per-Seed Results

Table 5 and Table 6 reports BEIR nDCG@10 for each of the two training seeds separately, across all 13 datasets at fusion weight α=0.9\alpha= 0.9; the two-seed means are the values reported in Table 1 and Table 4 for ModernBERT-base and ModernBERT-large. The specialization gain is consistent across seeds: at both n=2n = 2 and n=4n = 4 and at both scales: ModernBERT-base and ModernBERT-large.

ModelSeedArguAnaC-FEVERDBPediaFEVERFiQAHotpotQANFCNQQuoraSciDocsSciFTREC-CTouchéAvg
n=1n = 1
Sparse-only420.40380.10400.15040.61510.19980.58570.22380.16370.68230.11340.65550.57040.22690.3611
12340.38490.10640.14630.66170.19080.59320.22000.15620.66200.11050.64280.50570.24760.3560
Dense-only420.39460.21190.26900.50250.24190.37850.25290.46060.84880.13150.56000.67250.22140.3959
10010.40030.21120.27150.50720.23260.38080.23920.45320.84710.13410.54260.68430.21870.3941
Sparse+Dense420.39440.20220.25810.63820.26760.56570.26220.40470.82890.13090.66220.67970.26150.4274
12340.37230.24120.25230.66690.26220.57000.26010.41730.82580.13110.67930.71840.26710.4357
n=2n = 2
mHC no-spec420.38340.20540.26050.64650.27440.57440.26970.41270.82530.13520.67310.70840.25760.4328
12340.38190.22110.25530.65930.26930.56480.27190.40300.82050.13390.66880.66010.26920.4292
mHC spec420.41300.23290.29070.62480.28830.52430.28010.46760.84720.15100.67550.77260.25130.4476
12340.44270.24590.29310.63400.28870.52090.27320.46850.84940.15600.65890.74440.26820.4495
n=4n = 4
mHC no-spec420.41830.22370.25860.65370.27380.57330.26770.41960.82940.13840.67430.77720.25480.4433
12340.39540.23710.25820.68050.27660.56980.27120.41120.83080.13480.67590.67890.28270.4387
mHC spec420.44110.23560.29800.62700.28680.52100.28210.47260.85110.15490.65910.77650.25680.4510
12340.41640.24820.30020.65010.28760.52420.28740.47880.85200.14900.67870.76190.27850.4548

Table 5. Per-seed BEIR nDCG@10 across all 13 datasets (hybrid fusion at α=0.9\alpha= 0.9) for ModernBERT-base.

ModelSeedArguAnaC-FEVERDBPediaFEVERFiQAHotpotQANFCNQQuoraSciDocsSciFTREC-CTouchéAvg
n=1n = 1
Sparse-only420.41110.09810.13260.58330.21240.54910.22780.17010.67990.11430.65820.54240.22620.3543
12340.38220.09260.13710.60280.21420.56440.22230.16080.67220.10920.64330.49320.23510.3484
Dense-only420.41070.21580.30800.54030.30780.54020.27770.50470.85630.16060.59440.70530.22330.4239
12340.41120.21560.30630.52990.31270.40700.28230.50790.85620.16160.59990.69380.21780.4232
Sparse+Dense420.42910.21330.26700.65380.30270.58290.28720.44770.84590.13910.70310.74720.29150.4547
12340.42450.22750.27510.66340.31630.58480.27830.45000.84270.14050.69090.77560.29630.4589
n=2n = 2
mHC no-spec420.39710.20380.28330.65530.30440.59370.29320.45370.84590.13890.68430.76490.28300.4540
12340.42340.21780.28420.68420.31090.57910.30330.45540.84470.14540.69790.75930.28430.4608
mHC spec420.44170.23750.32700.64160.34640.53860.30990.52290.86260.16640.68450.76760.25930.4696
12340.46020.21950.32050.62900.34320.52770.31310.51350.86270.16640.69340.73930.24200.4639
n=4n = 4
mHC no-spec420.41030.20110.28150.66830.29580.58770.29970.45190.84120.13580.68940.77170.28800.4556
12340.39500.19920.28130.66550.32040.58980.29700.46270.84620.13590.69860.77250.30290.4590
mHC spec420.48050.23720.32680.64670.35260.54940.30790.51250.86210.16320.68700.77470.25260.4733
12340.48590.24060.32730.64080.35620.54000.31230.52760.86520.16250.69320.78560.25360.4762

Table 6. Per-seed BEIR nDCG@10 across all 13 datasets using ModernBERT-large, for each model and training seed. Sparse-only and Dense-only are evaluated in their native retrieval mode; the remaining models are hybrid retrievers fused at α=0.9\alpha= 0.9. The two-seed means are the values reported in Table 4. As at the base scale, the specialization gain holds within each seed: at both n=2n = 2 and n=4n = 4, mHC+spec exceeds mHC no-spec in every seed.

References

  1. [1]Ekaterina Alimaskina, Gleb Molodtsov, and Aleksandr Beznosikov. 2026. Analyzing stream collapse in hyper-connections: From diagnosis to mitigation. Preprint, arXiv:2606.03483.arxiv.org/abs/2606.03483
  2. [2]Rich Caruana. 1997. Multitask learning. Machine Learning, 28:41–75.
  3. [3]Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.03216
  4. [4]Michael Crawshaw. 2020. Multi-task learning with deep neural networks: A survey. Preprint, arXiv:2009.09796.arxiv.org/abs/2009.09796
  5. [5]Thibault Formal, Carlos Lassance, Benjamin Pipowarski, and Stéphane Clinchant. 2021. Splade v2: Sparse lexical and expansion model for information retrieval. Preprint, arXiv:2109.10086.arxiv.org/abs/2109.10086
  6. [6]Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed H. Awadallah, and Jianfeng Gao. 2022. Sparsely activated mixture-of-experts are robust multi-task learners. Preprint, arXiv:2204.07689.arxiv.org/abs/2204.07689
  7. [7]Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.arxiv.org/abs/1412.6980
  8. [8]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.DOI
  9. [9]Dohyeon Lee, Seung-won Hwang, Kyungjae Lee, Seungtaek Choi, and Sunghyun Park. 2023. On complementarity objectives for hybrid retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13357–13368, Toronto, Canada. Association for Computational Linguistics.DOI
  10. [10]Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. Preprint, arXiv:2106.14807.arxiv.org/abs/2106.14807
  11. [11]Sheng-Chieh Lin and Jimmy Lin. 2023. A dense representation framework for lexical and semantic matching. Preprint, arXiv:2206.09912.arxiv.org/abs/2206.09912
  12. [12]Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. Preprint, arXiv:1904.09482.arxiv.org/abs/1904.09482
  13. [13]Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 1930–1939, New York, NY, USA. Association for Computing Machinery.DOI
  14. [14]Subhankar Mishra. 2026. mhc-gnn: Manifold-constrained hyper-connections for graph neural networks. Preprint, arXiv:2601.02451.arxiv.org/abs/2601.02451
  15. [15]Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. Preprint, arXiv:1604.03539.arxiv.org/abs/1604.03539
  16. [16]Valentijn Oldenburg, Floris de Kam, Bente Zuijdam, Lieve Eberson, Nicky van Zutphen, Stef de Wildt, Ivo Verhoeven, and Cees Snoek. 2026. Manifold-constrained hyper-connections for parameter-efficient finetuning. Preprint, arXiv:2607.18130.arxiv.org/abs/2607.18130
  17. [17]Richard Sinkhorn and Paul Knopp. 1967. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21:343–348.
  18. [18]Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2019. Learning sparse sharing architectures for multiple tasks. Preprint, arXiv:1911.05034.arxiv.org/abs/1911.05034
  19. [19]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. Preprint, arXiv:2104.08663.arxiv.org/abs/2104.08663
  20. [20]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. Preprint, arXiv:1807.03748.arxiv.org/abs/1807.03748
  21. [21]Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. Preprint, arXiv:2412.13663.arxiv.org/abs/2412.13663
  22. [22]Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Dami Dai, Huazuo Gao, Jiang Chang, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. 2025. mhc: Manifold-constrained hyper-connections. Preprint, arXiv:2512.24880.arxiv.org/abs/2512.24880

Paper details

vlnc:W_wg9c7nnk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents