Blind Video Denoising for Vocal Tract rtMRI using a Self-Supervised Ensemble

Introduction

MAGNETIC resonance imaging (MRI) data are inherently susceptible to noise, particularly in real-time, fast, or experimental sequences, which degrades image quality [1]. Whereas denoising methods have been widely developed for brain, fetal and cardiac imaging, analogous efforts for the vocal tract remain relatively underexplored [2, 3]. Such limitation is particularly relevant for real-time MRI (rtMRI) of the vocal tract, where the trade-off between temporal resolution and signal-to-noise ratio (SNR) produces noisy magnitude frames that obscure fine articulatory structures, such as the tongue and velum, which are essential for accurate speech analysis.

With the rise of deep learning, MRI denoising has shifted towards learning-based schemes. As reported in the reviews by Chen et al. [2] and more recently by Nazir et al. [4], convolutional encoder-decoder architectures such as the U-Net have become the standard and account for the majority of the published work along with transformer-based models, offering superior capability in capturing global dependencies and preserving fine structural details amidst complex noise patterns, such as Rician noise. Nevertheless, a recurring limitation identified in the current literature is the heavy reliance on supervised training with large paired datasets. While effective for static anatomy [5], this dependence on clean targets presents a fundamental bottleneck for dynamic speech imaging, where physiological motion renders the acquisition of noise-free reference data physically and practically unfeasible.

Recent research instead exploits the internal correlations of high-dimensional acquisitions to build supervisory signals without ground truth. N2SR-M by Tian et al. [6] trains on spatially sub-sampled pairs from noisy data to restore fine structures via Super-Resolution (SR) constraints. Although it was formulated for multi-contrast MRI, the principle transfers to the temporal dimension of rtMRI, and it has been validated in quantitative MRI sequences [7] and in high-speed fluorescence imaging [8]. Complementarily, we believe that conventional analytical approaches remain indispensable for their explicit handling of noise statistics and spatial signal characteristics [9, 10]. Works such as that of Yang et al. [11] have addressed Rician noise reduction via a Pre-Smoothed Non-Local-Means (PSNLM) filter with variance stabilization, demonstrating that explicitly modeling noise characteristics is crucial for preserving anatomical detail in low-SNR regimes.

Motivated by these complementary strengths, we propose a novel self-supervised ensemble for vocal tract rtMRI denoising that combines the spatiotemporal SR strategy of N2SR-M with the noise stabilization of PSNLM. Unlike single-frame blind-spot networks [12, 13, 14] that rely solely on spatial context, our method exploits the spatiotemporal redundancy of speech sequences to disentangle stochastic noise from deterministic articulatory motion, achieving robust denoising without the need for a clean ground truth.

Dataset

We use the USC-75 dataset [15], a multi-speaker public collection of raw and reconstructed 2-D sagittal rtMRI videos capturing speech production at the vocal tract in 75 healthy speakers. The reconstructed magnitude videos have a spatial resolution of 84 $\times$ 84 pixels and a temporal resolution of 83 frames/s. Consecutive frames are strongly correlated [16], [17], so we decimate the temporal axis by a factor of 4, which reduces the computational cost while preserving articulatory events at the time scale of interest. Rician noise and reconstruction artifacts are present in every available magnitude frame, and no noise-free counterpart exists.

The proposed networks were trained on a pool of 54 subjects, and all results reported below are measured on subjects outside that pool. Four clips from four unseen subjects reading the Grandfather passage are reserved for selecting the hyper-parameters of the analytical filters, so that no baseline is tuned on the data used for its evaluation. The remaining 17 unseen subjects, all reading the North Wind passage, form the test set.

Methods

We formulate rtMRI denoising as a self-supervised problem where no clean ground truth exists. Let $X \in\mathbb{R}^{T \times H \times W}$ denote a magnitude video corrupted by signal-dependent noise, segmented into the overlapping clips $x^{seg}$ of Figure 1 before being fed to the network.

Spatiotemporal Self-Supervision

As illustrated in Figure 1, we adopt a similar sub-sampling strategy as N2SR-M [6] to enable training without noise-free targets. The operator $D_s : \mathbb{R}^{T \times H \times W} \rightarrow\mathbb{R}^{T \times\frac{H}{2} \times\frac{W}{2}}$ partitions every frame into non-overlapping $2 \times2$ cells and keeps one pixel per cell, drawn uniformly at random and independently for each cell and each frame. The retained pixels form the network input, while the remaining three quarters, collected in the mask $M$, are the only pixels on which the fidelity term of the loss is evaluated. Since the withheld pixel is not part of the input, its noise realization cannot be predicted from the observations, and the network is constrained to interpolate the underlying anatomy rather than to reproduce the noise of the target. The network $f_\theta$ is thus trained to recover the full-resolution structure under the overall objective:

where $y^p$ is the pseudo-target defined in the following subsection. Because the sampled positions differ from frame to frame, reconstructing the withheld pixel requires not only aggregating information from spatial neighbors but also from adjacent frames.

Variance-Stabilized Pseudo-Supervision

In a magnitude image the noise follows a Rician distribution. For an underlying signal amplitude $A$ and Gaussian noise of standard deviation $\sigma$ in each of the two orthogonal components of the complex-valued MRI signal:

so that $\mathbb{E}[M_{\mathrm{obs}}^2] = A^2 + 2\sigma^2$ and both the mean and the variance depend on the underlying SNR $r = A/\sigma$. In air and inside the airway, where $A \approx0$, the magnitude follows a Rayleigh distribution with mean $\sigma\sqrt{\pi/2}$. The noise in these regions does not have zero mean, so a raw noisy frame is a biased regression target in the regions that contain the articulatory structures of interest. Each frame is therefore filtered in a domain in which the noise is approximately additive and homoscedastic, and the result is used as the regression target:

with $F$ a variance-stabilizing transform (VST), $F^{-1}$ its inverse and PSNLM a pre-smoothed non-local means filter.

The choice of $F$ is not established for magnitude MRI, so two transforms are considered and compared in the ablation of Sec. IV-D. The first, $F_{\mathrm{V}}$, is the transform of Foi [18] obtained by quadrature over the Rice density so that the transformed data have the same standard deviation at every SNR, and inverted with the exact unbiased inverse. In our implementation the output standard deviation deviates by at most $0.8\%$ from a constant over $0 \le r \le12$, compared with $34\%$ for the untransformed magnitude. Its noise level is estimated per frame from the magnitudes $M_{\mathrm{bg}}$ inside a background rectangle, where $A \approx0$ and the Rayleigh median gives $\hat{\sigma} = \operatorname{median}(M_{\mathrm{bg}})/\sqrt{2\ln2}$. The second, $F_{\mathrm{T}}$, is a closed-form algebraic approximation and is the transform used to generate the pseudo-targets of the models reported here. It is applied pixel by pixel and combines a Gudbjartsson–Patz [1] bias correction with an Anscombe-type root:

here $\hat{\sigma}_{O}$ is the standard deviation of the pixels below an Otsu threshold, and the transform is inverted by $F_{\mathrm{T}}^{-1}(u) = \sqrt{\max((u/2)^2-\frac{3}{8}+\hat{\sigma}_{O}^{\,2},0)}$.

Network Architecture

We design a spatiotemporal U-Net with a pre-activation scheme [19], consisting of four main components:

1. Spatiotemporal Feature Extraction: the backbone consists of 3-D residual blocks enhanced with squeeze-and-excitation (SE) modules [20] to model channel-wise dependencies.

2. Anti-Aliasing Downsampling: strided convolutions are replaced with non-learnable Gaussian blur filters to mitigate aliasing artifacts.

3. SR Upsampling: as in N2SR-M, reconstruction is performed by a SR block of 3-D transposed convolutions followed by projection-based refinement layers.

4. Temporal-Only Attention: a bottleneck attention block that focuses strictly on temporal dependencies to enforce frame-to-frame consistency.

The encoder has four stages with channel multipliers $(1, 2, 4, 8)$ and two residual blocks per stage, and the SR block outputs twice the input grid in each spatial dimension.

Composite Loss and Dual-Model Fusion

The network is optimized with a composite loss enforcing data fidelity, edge preservation, and smoothness. With $\hat{x}$ the network output and $M$ the mask of pixels withheld by $\mathcal{D}_{s}$:

where $\mathcal{L}_{\mathrm{char}} = \frac{1}{|M|}\sum_{i\in M}\sqrt{(M[\hat{x}]_i-M[y^p]_i)^2+\epsilon^2}$ denotes the Charbonnier penalty over masked pixels, $\mathcal{L}_{G} = \lVert\nabla\hat{x}-\nabla y^p\rVert_1$ compares Sobel gradient magnitudes to promote sharp boundaries, and $\mathcal{L}_{\mathrm{TV}}$ is the total variation (TV) loss of the output to control smoothness.

We observed that a single value of $\lambda_1$ imposes a compromise between texture preservation and noise suppression, so we train two networks with different values and fuse their outputs. $f_{\mathrm{HF}}$ uses 16 base channels, 3-frame clips and $\lambda_1 = 1.2$, which favors high-frequency detail and edges; $f_{\mathrm{LF}}$ uses 48 base channels, 40-frame clips and $\lambda_1 = 0.7$, which favors global smoothness and the preservation of the original anatomical structure. The denoised video is their convex combination:

which is both locally sharp and globally consistent. The weight $w$ is not fitted; it is set manually to trade off noise suppression against detail retention. We use $w = 0.8$ on the original data, where the noise level is moderate, and $w = 0.1$ under severe synthetic corruption.

Training and Inference

Both networks are trained with AdamW, a learning rate of $2\cdot10^{-4}$ for $f_{\mathrm{HF}}$ and $5\cdot10^{-4}$ for $f_{\mathrm{LF}}$, and a plateau schedule on the validation loss. At inference the sub-sampling is removed and the full-resolution video is passed through both networks; overlapping clips are merged by averaging and the super-resolved output is resampled back to the acquisition grid before the fusion of Eq. (6). No filter is applied at inference; all values reported for the proposed models are raw network outputs.

Results

We assess performance in two scenarios: the original USC-75 rtMRI data, which contain only the noise present at acquisition, and a controlled setting in which synthetic noise is added to the same videos. In both, the ensemble is benchmarked against representative analytical denoisers, after which the pseudo-supervision that drives the training objective is ablated.

Evaluation protocol

On the original data no clean reference exists, so we use three reference-free measures. The first is a region-of-interest SNR:

where $\mu_S$ is the mean intensity over a fixed $30 \times30$ vocal-tract window and $\hat{\sigma}_N$ the standard deviation over a fixed $28 \times28$ background window in the lower-left corner of the frame, both drawn in Figure 2. The second measure is TV, which is included because rSNR alone favors over-smoothed outputs. The third is Structural Similarity Index (SSIM) computed against the noisy input to quantify agreement with the input anatomy. For every method, the input included, 50 consecutive frames are scored per clip.

We benchmark against Perona–Malik anisotropic diffusion (ADF) [21], [22], the volumetric transform-domain filter BM4D [23], non-local means (NLM) [9], and the PSNLM filter [11] that supplies our pseudo-targets, the latter in both stabilizations of Sec. III-B: PSNLMT uses the thresholded $F_T$ and PSNLMV the Rician $F_V$. BM4D, NLM and PSNLM are additionally run in a spatiotemporal form, marked 3-D, which filters the whole $T \times H \times W$ volume. Their hyper-parameter grids were scored inside this same protocol on the tuning clips of § 2. For each filter we retain the configuration that maximizes rSNR subject to SSIM $\ge0.90$. Where no configuration satisfies this constraint, as is the case for PSNLMT, the unconstrained rSNR optimum is reported and marked as such.

Reference-Free Evaluation

Table I reports quality averaged over the 17 test subjects, and Figure 3 the paired difference between each method and the input of that same subject. Figure 2 illustrates one of these clips, with its own measurements printed underneath.

The two members and their fusion behave as designed. $f_{LF}$ yields the highest rSNR of the three by prioritizing global smoothness, at the cost of TV and of an SSIM of 0.56 with respect to the input, which reflects structural deviation caused by over-smoothing. $f_{HF}$ recovers sharper textures, with the largest TV gain among all methods considered, at an SSIM

of 0.87. The ensemble balances the two effects, suppressing noise by 11.9 dB while increasing TV by 2.73 at an SSIM of 0.85, with visible improvements in fine articulatory structures such as the velum (Figure 2). Across the test subjects, the rSNR gain decreases and the TV gain increases monotonically with $w$, from +18.2 dB and +0.32 at $w = 0.1$ to +11.2 dB and +3.08 at $w = 0.9$.

With respect to the analytical denoisers ADF, NLM and BM4D, the latter two also in their 3-D forms, improve rSNR by between 5.2 and 11.4 dB, and reduce TV by 0.34 to 3.20 (Figure 3). PSNLMT achieves the largest rSNR gain among the analytical filters, 14.3 dB at the setting used to generate the pseudo-targets and 20.1 dB in its 3-D form. At that setting it is also the only analytical operating point in Figure 3 that increases TV, by 0.65. Both results are obtained at an SSIM of 0.57, that is, with substantial deviation from the input. The Rician stabilized PSNLMV shows the opposite behavior: it maintains an SSIM of 0.80 but suppresses only 6.4 dB and still reduces TV. No analytical filter achieves the three properties simultaneously; only the ensemble and its $f_{\mathrm{HF}}$ member suppress noise, increase high-frequency content and remain close to the input. The additional gradient energy introduced by the ensemble remains correlated with the anatomy of the input with a 0.96 Pearson correlation, which indicates that it corresponds to recovered detail and not to added noise. The hyper-parameter search for the filters leads to the same conclusion: of the 5950 configurations scored on the tuning clips, 1870 match the structural fidelity of the ensemble, and none of these exceeds it in both rSNR and TV. The highest TV among them is 9.88, against 12.53 for the ensemble on the same clips, and no configuration in the entire search exceeds 10.27, for an input value of 9.59.

Reference-Based Evaluation

We additionally evaluate the ensemble on USC-75 frames corrupted with Gaussian, Rayleigh, and Rician noise at $\sigma= 0.2$, using three reference-based metrics against the uncorrupted frame: the peak signal-to-noise ratio (PSNR), the normalized root mean squared error (NRMSE) and SSIM, which here compares the output with that reference rather than with the noisy input. As shown in Figure 4, the method suppresses severe contamination under all three distributions, with $w = 0.1$ to maintain the high SNR of $f_{\mathrm{LF}}$ while injecting sufficient high-frequency content to sharpen articulatory edges. PSNR increases from 15.5 to 23.6 dB under Gaussian noise, from 11.4 to 23.2 dB under Rayleigh noise and from 13.3 to 20.9 dB under Rician noise, and SSIM increases from 0.19, 0.24 and 0.16 to 0.58, 0.63 and 0.52 respectively, an approximately threefold improvement. This indicates that the geometry of the articulators is largely preserved under severe degradation.

Pseudo-Supervision Ablation

Both ensemble members were re-trained against three regression targets, the pseudo-target built with $F_T$, the one built with $F_V$ and the raw noisy clip, holding architecture, optimizer, schedule, data subset, seeds and number of updates fixed, so that the regression target is the only difference. These runs are shorter than those behind Table 1 and their absolute values are correspondingly lower.

Method	rSNR $\uparrow$	TV	SSIM $\uparrow$
Original (noisy)	$21.5 \pm2.4$	$9.32 \pm0.99$	$1.00 \pm0.00$
ADF	$28.2 \pm4.6$	$8.98 \pm0.94$	$0.89 \pm0.02$
BM4D (3-D)	$26.7 \pm4.1$	$7.05 \pm0.64$	$0.92 \pm0.02$
NLM (3-D)	$31.6 \pm5.9$	$8.31 \pm0.71$	$0.95 \pm0.04$
PSNLMT [pseudo-target]	$35.7 \pm9.7$	$9.96 \pm1.12$	$0.57 \pm0.04$
PSNLMT [max SNR]	$40.0 \pm9.9$	$5.81 \pm0.70$	$0.64 \pm0.04$
PSNLMV [pseudo-target]	$27.9 \pm5.1$	$8.51 \pm0.81$	$0.80 \pm0.09$
PSNLMV [tuned]	$20.5 \pm2.4$	$9.46 \pm1.04$	$0.92 \pm0.03$
$f_{HF}$ (ours)	$32.0 \pm7.6$	$12.73 \pm1.33$	$0.87 \pm0.04$
$f_{LF}$ (ours)	$39.9 \pm10.1$	$9.31 \pm0.89$	$0.56 \pm0.03$
Ensemble (ours)	$33.4 \pm8.0$	$12.05 \pm1.24$	$0.85 \pm0.04$

The results are shown in Figure 5. For $F_T$ as pseudo-target the fused output reaches an rSNR of 37.1 dB, being reduced to 29.8 dB for $F_V$, whereas training on the raw noisy clip gives 22.5 dB, indistinguishable from the 22.4 dB of its own input, at an SSIM of 0.98 with respect to that input. Without a filtered target the network does not denoise but reproduces its input. The sub-sampling of Sec. III-A is therefore not sufficient on its own, since it decouples only the Charbonnier term of Eq. (5) from the input noise. $\mathcal{L}_G$ is evaluated on all pixels, so with a noisy target it drives the output to reproduce the gradient field of the noise, and neither term addresses the non-zero-mean Rician floor of Eq. (2). The validation metrics of the original full-scale trainings, which paired the same configuration with and without the pseudo-target, are consistent with this: relative the compared filters, demonstrating the viability of purely self-supervised methods for vocal tract rtMRI applications where clean reference data is physically unattainable.

A comparison against different classical filters, searched exhaustively over their hyper-parameters, shows that suppressing the noise floor while increasing high-frequency articulatory content at comparable structural fidelity is not reachable by re-tuning the filters. The ablation shows that the pseudo-supervision, rather than the super-resolution constraint alone, drives the denoising, and that its effect comes from the hard threshold in $F_T$ rather than from the variance-stabilizing root.

Acknowledgment

We gratefully acknowledge the scientific support and computational resources provided by the Erlangen National High-Performance Computer Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg.

to its own input, the pseudo-target arm gains 11.8 dB and the noisy-target arm loses 0.7 dB, at essentially unchanged TV (7.87 and 7.70 respectively). The pseudo-target arm also exceeds the pseudo-target on which it was trained by 4.1 dB, showing that it does not simply reproduce the filter.

Between the two pseudo-targets, $F_T$ yields an rSNR higher by 7.3 dB. Since $F_T$ flattens the region that forms the denominator of rSNR, this metric alone is not conclusive. On the frame-varying component of the background, which a fixed pattern cannot reduce, the $F_T$ arm lowers the standard deviation by a factor of 20.2, compared with 12.8 for $F_V$ and 4.0 for the noisy target, while a control that only thresholds the input, without filtering, gives 0.9. The $F_T$ arm also agrees better with the input structures inside the vocal tract giving a Pearson correlation of 0.95 against 0.87.

Table II shows which of the two operations composing $F_T$, the hard threshold or the Anscombe root, accounts for its advantage over $F_V$. This is a separate, smaller experiment than the one above: only the $f_{\mathrm{HF}}$ member is retrained, against a pseudo-target built from each half of Eq. (4) in isolation, with the non-local means stage, the noise estimator, the data, the schedule and the seed held fixed as before. The threshold alone reaches 33.0 dB, most of the 35.6 dB obtained with the full transform, whereas the Anscombe root alone reaches 22.7 dB, the same level as the arm trained without a pseudo-target at all. The supervision therefore derives from the hard thresholding rather than from the variance stabilization alone for which the transform was initially introduced.

Pseudo-target	rSNR $\uparrow$	$\Delta$	SSIM
Noisy input	22.4	n/a	1.00
None (raw noisy clip)	22.3	$-0.1$	0.97
Anscombe root only	22.7	$+0.3$	0.88
$F_V$, Rician VST	28.9	$+6.5$	0.92
Threshold only	33.0	$+10.6$	0.88
$F_T$ (threshold + root)	35.6	$+13.2$	0.86

Conclusion

We presented a self-supervised ensemble for blind vocal tract rtMRI denoising that requires no clean ground truth. By combining spatiotemporal SR training with a stabilized pseudo-supervision and a dual-model fusion strategy, the method effectively separates signal-dependent noise from dynamic articulatory structures across the USC-75 dataset. The ensemble mitigates the smoothness-sharpness trade-off inherent in single-model approaches. On the original data, a balanced fusion enhances high-frequency articulatory detail without distorting global anatomy, while under severe synthetic degradation, a stronger low-frequency contribution drives robust structural recovery. In both cases, the ensemble yields quantitatively and qualitatively superior results among