betaFeedback

WorksEngineering

Validating Acoustic-to-Articulatory Inversion with Real-Time MRI: From Vocal-Tract Fidelity to Phonetic Consequences

Abstract

Acoustic-to-articulatory inversion provides a promising way to estimate vocal-tract behaviour directly from speech, but inferred articulatory variables need to be validated against independent physical measurements before they can be interpreted as faithful estimates of articulation. We investigate this using paired real-time MRI (rtMRI) and speech. A WavLM-based inversion model is trained to predict eight vocal tract variables (TVs) extracted from rtMRI, achieving a mean correlation of 0.643 on held-out speakers. Prediction accuracy is substantially higher for constriction degree than for constriction location, and the inferred trajectories systematically underestimate the dynamic range of rtMRI-derived articulation. We then use articulatory-to-acoustic synthesis as a functional probe of these errors, comparing speech generated from inversion-derived and rtMRI-derived TVs while holding the source features fixed. The discrepancy is small for vowels, approximants, stops, and nasals, but increases markedly for fricatives and affricates, where speech production depends on maintaining a narrow critical constriction and small geometric errors can strongly affect the acoustics. These results show that current acoustic-to-articulatory models recover broad vocal-tract behaviour well, but further work is required before inferred TVs can be regarded as faithful measurements of fine articulatory geometry.

Introduction

Acoustic-to-articulatory inversion seeks to recover vocal-tract behaviour directly from speech. Vocal tract variables (TVs) provide a compact, interpretable representation of constriction degree and location for the lips, tongue, and velum [1, 2], and recent work has shown that they can be estimated using modern acoustic models and self-supervised speech representations [3, 4, 5]. Such representations support speech generation and articulatory coding, but may also be interpreted as estimates of physical speech production.

For this latter use, reconstructing speech accurately from an inferred representation is not sufficient. Acoustic-to-articulatory inversion is under-constrained: similar acoustics may arise from different vocal-tract configurations [6, 7], so an inferred trajectory may reconstruct speech well while still differing systematically from the articulation that produced it. A model may therefore produce smoother, compressed, or misplaced articulatory trajectories even when its acoustic output is plausible. Successful reconstruction does not establish that TVs faithfully measure vocal-tract behaviour. This distinction is particularly relevant to speech pathology, where articulatory information inferred from acoustics can inform assessment and rehabilitation of speech disorders involving altered motor or structural anatomy [8].

Real-time MRI (rtMRI) provides an independent way to test this question because it directly observes midsagittal vocal-tract geometry, including the tongue root, pharyngeal region, and velum [9, 10, 11, 12]. Recent multispeaker datasets and tracked contours enable comparison of acoustically inferred TVs with TVs derived from the imaged vocal tract [11, 13]. Previous work such as ARTI-6 [5] demonstrates their utility for speech reconstruction and pseudo-labelling, but not their physical fidelity.

In this work, we pair rtMRI and speech to ask how faithfully acoustic-to-articulatory inversion recovers observed articulation, and where errors become acoustically consequential. A WavLM-based model [14] predicts eight TVs, which are compared directly with the rtMRI-derived trajectories. We then use articulatory-to-acoustic synthesis as a functional probe: a Transformer-based synthesizer [15] is driven by either inversion-derived or rtMRI-derived TVs from the same utterances while source information is held fixed.

To our knowledge, previous work has not combined independent rtMRI-based validation with a matched synthesis comparison and phonetic analysis of the resulting errors. Our contributions are: (1) direct evaluation of inferred TVs against rtMRI-derived articulation; (2) a controlled synthesis-based test of whether the resulting discrepancies matter acoustically; and (3) phonetic analysis identifying the articulatory configurations where errors are most consequential.

Methods

Figure 1 summarizes the experimental framework. Our objective is to assess how faithfully TVs inferred from acoustics represent articulation observed independently with rtMRI, and to determine when discrepancies between the two representations become acoustically consequential.

Figure 1. Experimental framework. Tract variables are obtained in two ways from the same rtMRI recordings: directly quantified from tracked contours (rtMRI-derived) and predicted from the audio by a WavLM-based inversion model (inversion-derived). The two are compared directly, then used to drive the same synthesizer on the same held-out utterances, with source features and speaker conditioning held fixed, so that differences in the output isolate the effect of TV provenance. The synthesizer is pretrained on inversion-labelled LibriTTS-R and fine-tuned on the rtMRI corpus.

Data and rtMRI-derived tract variables

We use 358 utterances from 14 speakers of the Annot-16 corpus [13], which provides tracked articulator contours together with phone- and word-level forced alignments for the USC 75-speaker rtMRI database [11]. The material includes read passages, nonsense VCV sequences, spontaneous picture descriptions, and topic monologues. Audio was recorded in-scanner at 16 kHz and denoised using a waveform-domain enhancement model [16].

We derive eight TVs from the tracked midsagittal contours (Figure 2): lip aperture and protrusion (LA, LP); tongue-tip, tongue-body, and tongue-root constriction degrees (TTCD, TBCD, TRCD); tongue-tip and tongue-body constriction locations (TTCL, TBCL); and velic aperture (VEL). The TVs extend those of Articulatory Phonology [1, 2] to exploit structures visible in rtMRI: pharyngeal constriction is represented by a dedicated tongue-root constriction degree (TRCD) [17], which rtMRI resolves directly.

Tracked midsagittal contours and eight rtMRI-derived tract variables

Figure 2. Extraction of the eight rtMRI-derived tract variables from the tracked midsagittal contours of a single rtMRI frame.

Constriction degree is the minimum Euclidean distance between a tongue region and its target passive articulator: the hard palate for TTCD and TBCD, and the posterior pharyngeal wall for TRCD. The velum is excluded from the tongue target surfaces because it is itself a mobile articulator with its own TV, and including it would couple TBCD to velic lowering under a static tongue. Constriction location is normalized arc length along the hard palate, from 0 at the alveolar ridge to 1 at the velar junction, so TBCL saturates at 1 for tongue-body configurations posterior to the hard palate. Tongue regions are delimited along normalized tongue-contour arc length from the tip as tip (≤ 0.45), body (≥ 0.60), and root (posterior 0.35). LA is the minimum inter-lip distance, LP the protrusion of the lip midpoint relative to the anterior palate, and VEL the minimum velum-to-pharyngeal-wall distance. As rtMRI resolves the velopharyngeal port directly, we retain a continuous aperture measure [12]. We refer to these throughout as rtMRI-derived TVs, to distinguish them from variables predicted from speech acoustics. The underlying contours are themselves estimates: Annot-16 reports a 3.58-pixel Hausdorff distance between semi-automatic and expert handmade tongue contours [13].

Acoustic-to-articulatory inversion

The acoustic-to-articulatory model predicts the eight TVs from speech. We use a frozen WavLM-Large encoder [14], combine its hidden representations using a learned layer weighting, and pass the resulting features through a projection and a two-layer bidirectional LSTM with 256 units per direction. A final linear layer predicts the eight TVs at the 50 Hz WavLM frame rate. The rtMRI-derived trajectories are resampled to the same frame rate. The model is trained using a mean squared error (MSE) objective on approximately two hours of paired rtMRI–speech data. Checkpoints are selected using mean Pearson correlation, and the evaluation is speaker-independent.

Assessing articulatory fidelity

Correlation alone does not establish that an inferred TV preserves the range of the underlying articulatory trajectory. For rtMRI-derived trajectory z\mathbf{z} and inversion estimate z^\hat{\mathbf{z}}, we therefore report Pearson correlation r=corr⁡(z,z^)r = \operatorname{corr}(\mathbf{z}, \hat{\mathbf{z}}) and the predicted-to-rtMRI standard-deviation ratio Rσ=σz^/σzR_{\sigma} = \sigma_{\hat{z}}/\sigma_z. While rr measures how well the inferred trajectory follows articulatory variation, Rσ<1R_{\sigma} < 1 indicates compression of the rtMRI-derived articulatory range.

Synthesis as a functional probe

Direct comparison of z\mathbf{z} and z^\hat{\mathbf{z}} establishes whether the representations differ geometrically, but does not establish whether those differences matter for speech production. We therefore use articulatory-to-acoustic synthesis as a controlled functional probe.

Because the paired rtMRI corpus is too small to train a high-capacity synthesizer reliably from random initialization, the inversion model is first applied to LibriTTS-R train-clean-100 [18], producing 31,651 pseudo-labelled utterances totalling 53.0 h. For each utterance, we extract the eight inversion-derived TVs, log-F0F_0, continuous voicing probability, and an ECAPA-TDNN speaker embedding [19]. Log-F0F_0 and voicing are obtained from the target recording using pYIN [20]. They therefore constitute oracle source information and are held fixed during comparisons.

The synthesis model is a six-layer Transformer encoder [15] with model dimension 384, conditioned on the speaker embedding and followed by a five-layer convolutional post-net. It predicts an 80-bin log-mel spectrogram at 50 Hz. Waveforms are generated using HiFi-GAN v1 [21]. The synthesizer is pretrained on the pseudo-labelled LibriTTS-R and subsequently fine-tuned on rtMRI-derived TVs.

The principal functional comparison uses the same held-out rtMRI utterances. Each utterance is synthesized twice, using inversion-derived TVs and corresponding rtMRI-derived TVs, with source features, speaker conditioning, and synthesis model held constant. The resulting difference isolates the acoustic effect of replacing inferred articulatory trajectories with independently rtMRI-derived trajectories.

Intelligibility is evaluated using word (WER) and character (CER) error rates from Whisper [22]. Spectral distortion is measured using mel-cepstral distortion (MCD) [23].

Phonetic analysis of inversion errors

To determine whether inversion errors are equally consequential across speech sounds, we compare the spectral output obtained from rtMRI-derived and inversion-derived TVs within each forced-aligned phone interval. For each token we compute the mean frame-level cepstral distance between predicted and ground-truth log-mel under each condition, over the full 80-bin mel range (0–8 kHz), and report the difference (rtMRI-derived minus inversion-derived) as the spectral gap, holding source features and speaker conditioning fixed.

Phones are grouped according to their characteristic constriction degree in Articulatory Phonology [2]: wide configurations (vowels), narrow non-turbulent configurations (approximants), closed configurations (stops and nasals), and critical constrictions (fricatives and affricates). The latter are produced with a highly constricted airway whose precise geometry is important for turbulence generation.

Finally, because sibilant place provides a focused test of constriction location, we examine /s/ and /ʃ/ [24, 25]. We compare the spectral centroid of the affrication noise between the reference and synthesized signals and quantify the separation between the two phone classes using Cohen’s dd.

Results and Discussion

How faithfully does inversion recover vocal-tract behaviour?

Table 1 shows that inversion accuracy varies substantially across articulatory dimensions. Constriction-degree variables are recovered most strongly, whereas constriction-location variables are considerably weaker. More importantly for physical interpretation, RσR_{\sigma} is below unity for every TV, with a mean of 0.775 and a minimum of 0.595 for TBCL. The inferred trajectories therefore systematically occupy a narrower dynamic range than the rtMRI-derived trajectories, with the strongest compression occurring for the location variables. Thus, even when inferred and rtMRI-derived trajectories are strongly correlated, the inferred TVs do not necessarily preserve the full extent of rtMRI-derived articulatory movement.

TVrrRσR_{\sigma}TVrrRσR_{\sigma}
LA0.7200.848TBCD0.7420.849
LP0.4880.683TBCL0.3400.595
TTCD0.8500.920TRCD0.7940.831
TTCL0.4750.726VEL0.7370.747
rˉ=0.643\bar{r} = 0.643Rσ‾=0.775\overline{R_{\sigma}} = 0.775

Table 1. Inversion fidelity on held-out speakers.

Do inversion errors matter acoustically?

We next ask whether the differences between inferred and rtMRI-derived TVs are acoustically consequential. On the same held-out rtMRI utterances, inversion-derived TVs give 35.8% WER, 23.1% CER, and 3.77 dB MCD, compared with 39.6% WER, 26.2% CER, and 3.86 dB MCD when the same synthesizer is driven by rtMRI-derived TVs. Thus the aggregate effect of TV provenance is modest, motivating a more detailed analysis of whether the discrepancy is concentrated in particular articulatory configurations.

Three controls confirm that this comparison probes the articulatory representation rather than a failure elsewhere in the pipeline. In the pseudo-labelled LibriTTS-R domain, the same architecture reaches 7.7% WER against a 5.4% reference, so the synthesizer reconstructs speech accurately when driven by inversion-derived TVs in clean conditions. Ground-truth mel spectrograms passed through the vocoder yield 10.7% WER against 10.3% for the original-recording reference, so waveform generation contributes little to the observed error. Replacing the TVs with constants raises CER to 72.7%, confirming that the articulatory input is load-bearing despite the oracle source features.

Where do inversion errors become acoustically consequential?

The modest aggregate difference between the two TV conditions conceals a strong phonetic structure. Table 2 groups phone tokens by their characteristic constriction degree. The spectral gap is 0.71 dB for wide configurations, 0.40 dB for narrow approximants, and 0.75 dB for closed stops and nasals. In contrast, fricatives and affricates requiring a critical constriction show a substantially larger gap of 2.47 dB.

ConstrictionClassesnnGap (dB)
Widevowels24880.71
Narrowapproximants3690.40
Closedstops, nasals7980.75
Criticalfricatives, affricates5582.47

Table 2. rtMRI-derived minus inversion-derived spectral gap, grouped by characteristic constriction degree in the sense of Articulatory Phonology [2]

The pattern is not explained simply by constriction magnitude. Approximants also involve a narrow but non-zero constriction, yet their difference remains similar to that of vowels and complete closures. The spectral gap for critical constrictions is 2.07 dB larger than for narrow approximants (2.47 vs. 0.40 dB). What distinguishes fricatives and affricates is that a sufficiently narrow constriction must be maintained to generate turbulence, and the resulting spectrum depends strongly on both constriction geometry and the shape of the downstream cavity [26]. The effect is predominantly carried by fricatives, with a gap of 2.25 dB (nn = 508); affricates show a numerically larger gap, although only 50 tokens are available. This suggests that gross articulatory movements can be estimated sufficiently well for many speech sounds, but errors in fine constriction geometry become substantially more consequential when the acoustic output is highly sensitive to small articulatory changes.

A focused /s/–/ʃ/ analysis provides a constriction-location example, shown in Figure 3. TTCL is among the least accurately recovered TVs, and the spectral-centroid separation between the two sibilants contracts from 1136 to 723 Hz after synthesis, with Cohen’s dd falling from 1.48 to 0.92 (62% retained). This is consistent with loss of constriction-location information, although only 47 /ʃ/ tokens are available and the 16-kHz sampling rate limits analysis to 8 kHz.

Figure 3. Spectral centroid of frication for /s/ and /ʃ/ in original and synthesized speech.

Conclusions

We used paired rtMRI and speech to test whether acoustically inferred vocal tract variables faithfully represent independently observed articulation. Fidelity is uneven: constriction degrees are recovered more strongly than locations, and inferred trajectories systematically underestimate the dynamic range of rtMRI-derived articulation. Using synthesis as a matched functional probe, we find these discrepancies are not equally consequential: the effect is concentrated in fricatives and affricates, where a critical constriction must be maintained, while a focused /s/–/ʃ/ analysis links the degradation to constriction location.

These results distinguish acoustic usefulness from physical fidelity: an inferred representation may support speech reconstruction without faithfully measuring fine vocal-tract geometry. Improving and independently validating constriction degree and location will be important if acoustic-to-articulatory inversion is to characterize the vocal-tract behaviour of individual speakers.

Acknowledgments

Funding: Australian Research Council Grant DP22010293.

References

References

  1. [1]E. L. Saltzman and K. G. Munhall, “A dynamical approach to gestural patterning in speech production,” Ecological Psychology, vol. 1, no. 4, pp. 333–382, 1989.DOI
  2. [2]C. P. Browman and L. Goldstein, “Articulatory phonology: An overview,” Phonetica, vol. 49, no. 3–4, 1992.pubmed.ncbi.nlm.nih.gov/1488456
  3. [3]Y. M. Siriwardena, C. Espy-Wilson, and S. Shamma, “Learning to compute the articulatory representations of speech with the MirrorNet,” in Proc. Interspeech, 2023.arxiv.org/abs/2210.16454
  4. [4]C. J. Cho, P. Wu, T. S. Prabhune, et al., “Coding speech through vocal tract kinematics,” IEEE Journal of Selected Topics in Signal Processing, vol. 18, no. 8, pp. 1427–1440, 2024.DOI
  5. [5]J. Lee, S. Foley, T. Lertpetchpun, et al., “ARTI-6: Towards six-dimensional articulatory speech encoding,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2026.
  6. [6]B. S. Atal, J. J. Chang, M. V. Mathews, and J. W. Tukey, “Inversion of articulatory-to-acoustic transformation in the vocal tract by a computer-sorting technique,” The Journal of the Acoustical Society of America, vol. 63, no. 5, pp. 1535–1555, 1978.pubmed.ncbi.nlm.nih.gov/690333
  7. [7]D. Neiberg, G. Ananthakrishnan, and O. Engwall, “The acoustic to articulation mapping: non-linear or non-unique?,” in Proc. Interspeech, 2008, pp. 1485–1488.DOI
  8. [8]G. Phippen, “Articulating the issues: speech assessment and intervention in cleft lip and palate,” British Dental Journal, vol. 234, pp. 912–917, 06 2023.DOI
  9. [9]P. Wu, T. Li, Y. Lu, et al., “Deep speech synthesis from MRI-based articulatory representations,” in Proc. Interspeech, 2023, pp. 5132–5136.arxiv.org/abs/2307.02471
  10. [10]S. Foley, J. Lee, K. Huang, et al., “A long-form single-speaker real-time MRI speech dataset and benchmark,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2026.arxiv.org/abs/2509.14479
  11. [11]Y. Lim, A. Toutios, Y. Bliesener, et al., “A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images,” Scientific Data, vol. 8, no. 1, pp. 258, 2021.DOI
  12. [12]S. Tabatabaee, S. Boyce, L. Oren, et al., “Enhancing acoustic-to-articulatory speech inversion by incorporating nasality,” in Proc. Interspeech, 2025, pp. 325–329.DOI
  13. [13]X. Shi, Y. Zhang, Y. Lu, et al., “75-speaker Annot-16: A benchmark dataset for speech articulatory rt-MRI annotation with articulator contours and phonetic alignment,” in Proc. Interspeech, 2025.DOI
  14. [14]S. Chen, C. Wang, Z. Chen, et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.DOI
  15. [15]A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017, vol. 30.arxiv.org/abs/1706.03762
  16. [16]A. Défossez, G. Synnaeve, and Y. Adi, “Real time speech enhancement in the waveform domain,” in Proc. Interspeech, 2020, pp. 3291–3295.arxiv.org/abs/2006.12847
  17. [17]S. Tilsen, P. Spincemaille, B. Xu, et al., “Anticipatory posturing of the vocal tract reveals dissociation of speech movement plans from linguistic units,” PloS one, vol. 11, no. 1, pp. e0146813, 2016.DOI
  18. [18]Y. Koizumi, H. Zen, S. Karita, et al., “LibriTTS-R: A restored multi-speaker text-to-speech corpus,” in Proc. Interspeech, 2023, pp. 5496–5500.arxiv.org/abs/2305.18802
  19. [19]B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. Interspeech, 2020, pp. 3830–3834.DOI
  20. [20]M. Mauch and S. Dixon, “pYIN: A fundamental frequency estimator using probabilistic threshold distributions,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2014, pp. 659–663.DOI
  21. [21]J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Advances in Neural Information Processing Systems (NeurIPS), 2020, vol. 33.arxiv.org/abs/2010.05646
  22. [22]A. Radford, J. W. Kim, T. Xu, et al., “Robust speech recognition via large-scale weak supervision,” in Proc. Intl. Conf. on Machine Learning (ICML), 2023.arxiv.org/abs/2212.04356
  23. [23]R. Kubichek, “Mel-cepstral distance measure for objective speech quality assessment,” in Proc. IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, 1993, vol. 1, pp. 125–128.
  24. [24]A. Jongman, R. Wayland, and S. Wong, “Acoustic characteristics of English fricatives,” JASA, vol. 108, no. 3, pp. 1252–1263, 2000.
  25. [25]K. Forrest, G. Weismer, P. Milenkovic, et al., “Statistical analysis of word-initial voiceless obstruents: Preliminary data,” JASA, vol. 84, no. 1, pp. 115–123, 1988.pubmed.ncbi.nlm.nih.gov/3411039
  26. [26]K. N. Stevens, Acoustic Phonetics, MIT Press, Cambridge, MA, 1998.

Paper details

vlnc:W_ef43xqjj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents