betaFeedback

WorksEngineering

AccentCanvas: Activity-Guided Bidirectional Accent Conversion

Abstract

Accent conversion for dubbing must change pronunciation while preserving content, voice, and source timing. Normalization alone cannot produce requested non-native accents, and matching total duration does not ensure local speech/pause alignment. We present AccentCanvas, a shared discrete diffusion model for normalization and L1-to-L2 conversion, trained on cross-paired shared-prompt recordings and synthetic pairs. Frame-level speech activity and repetition-preserving units guide local timing through synthesis. Across both tasks, AccentCanvas achieves the highest accent conversion performance (Accent Arrival) and mean listener-rated accent similarity among compared systems, with over 73% lower speech-boundary error than the best baseline for each task. These gains accompany higher L1-to-L2 WER than TokAN and Vevo-Style, highlighting a tradeoff between requested-accent resemblance and ASR-based content recognition. https://smokedindia.github.io/accentcanvas/

Introduction

Accent conversion changes the pronunciation of speech while preserving its linguistic content and speaker identity. This capability supports applications in communication, language learning, and media production [1]. In speech dubbing, converted speech must also fit the timing of the original. Much recent work focuses on accent normalization, which converts L2-accented English toward native American English. Recent systems such as TokAN [2], CosyAccent [3], and DLM-AN [4] investigate this through discrete speech representations and generative models, addressing tradeoffs among accent change, intelligibility, and speech quality.

The focus on normalization raises a complementary question: can a conversion model also transform native English toward a requested L2-English accent? For example, an English dub may preserve a Korean actor’s voice while converting the dialogue from an American accent to the Korean-accented English familiar to the actor or production, which requires conversion in the inverse direction. Several methods have explored this broader setting. RWA [5] performs many-to-many conversion through phoneme remapping and duration adjustment, while the system requires accent-specific pronunciation resource.

many conversion through phoneme remapping and duration adjustment, while the system requires accent-specific pronunciation resource. PSDN [6] disentangles accent from content and demonstrates foreign-to-British and American-to-Indian conversion, although its experiments train for a designated target only. More recently, Bagat et al. [7] adapt TokAN for reference-conditioned L1-to-L2 conversion. Their evaluation focuses on improving speech recognition, leaving perceptual and per-accent evaluation to future work.

Limited accented speech data remains a central challenge in learning accent conversion. To mitigate this, TokAN [2] synthesizes native target speech, CosyAccent [3] synthesizes accented sources, and DLM-AN [4] uses both source- and target-synthesized pairs using text-to-speech (TTS) models. Synthetic supervision also includes transliteration-based multi-accent pairs [8] and native-TTS targets [9]. We propose expanding the use of real parallel accent-conversion data through cross-pairing. Exploiting the shared prompts in CMU ARCTIC and L2-ARCTIC [10, 11], we construct content-matched source–target pairs across accent groups. Recordings serve as both sources and targets across conversion relationships, expanding training supervision without collecting additional speech.

We then introduce AccentCanvas, a shared discrete diffusion model that generates accent-converted speech based on the requested accent. The model predicts the target speech unit conditioned on the source speech features via cross-attention, and the controllable-length initialization enables generation with varying speech durations. To generate converted speech with speech–pause synchronization with the source, we train the model with frame-level activity guidance as an explicit condition.

We evaluate speech converted in both directions and demonstrate that AccentCanvas generates both L1 and L2 accents with high naturalness. The ablation results show that existing synthetic-only training can yield intelligibility; however, it fails to generate L2 accents in the converted speech. The proposed cross-pairing achieves a lower WER than fixed one-to-one pairing while maintaining accent performance. Our selected recipe combines cross-paired real recordings with a smaller synthetic contribution and achieves lower WER than the evaluated real-only recipes. Activity conditioning effectively reduces speech-boundary error, supporting synchronized conversion suitable for dubbing applications.

Method

AccentCanvas converts a source utterance to a requested accent, and uses speech activity to guide the output timing. As illustrated in Figure 1, a discrete diffusion model predicts speech units from source-content features, a target accent label, and a frame-level speech activity sequence. A pretrained renderer then synthesizes an audible waveform using the source-speaker reference.

Overview of AccentCanvas

Figure 1. Overview of AccentCanvas. One model performs both normalization and L1-to-L2 conversion, and requires no text guidance during training. Speech units are utilized with no de-duplication. During inference, masked speech unit length is initialized to match the source speech duration. Generated units are rendered with a source speaker reference.

Real speech dataset cross-pairing

We expand the scope of leveraging accented speech datasets by cross-pairing samples from CMU ARCTIC and L2-ARCTIC with shared prompts. For each prompt, we vary ordered source–target pairings across speakers and accent groups within the training partitions, yielding normalization, L1-to-L2, and cross-L2 pairs. This maximizes the use of natural accent recordings across conversions without requiring additional data, exposing the training model to a wider range of conversion scenarios.

We train the model on these natural pairs together with a smaller contribution of CosyAccent synthetic pairs [3]. This mixture was deliberately designed to capture both the accent evidence from the natural recordings and content coverage from the synthetic pairs. § 4 compares the resulting training-data recipes. Note that, although the training AccentCanvas includes TTS-synthesized data, the proposed model architecture requires no transcript information, enabling acceptance of broader accent datasets without text labels.

Conditional discrete diffusion

We predict target speech units with a discrete diffusion model conditioned on source content, a requested accent, and speech activity [12]. A frozen speech encoder [13] extracts source features, which are projected and supplied through cross-attention. Requested-accent and diffusion-time embeddings modulate the transformer blocks.

During training, speech units are extracted from the target speech, and valid positions are masked independently with probability tb=max⁡(Ub,0.001)t_b = \max(U_b, 0.001), where Ub∼U[0,1)U_b \sim\mathcal{U}[0, 1), ensuring at least one masked position per bb sample. We minimize

L=−∑b∑j∈Mbtb−1log⁡pθ(zb,j∣Cb)∑b∣Mb∣(1)\mathcal{L} = -\frac{\sum_b \sum_{j \in\mathcal{M}_b} t_b^{-1} \log p_\theta(z_{b,j} \mid\mathcal{C}_b)}{\sum_b \lvert\mathcal{M}_b \rvert} \tag*{(1)}

where Mb\mathcal{M}_b is the masked position set and zb,jz_{b,j} is the original unit. The condition Cb\mathcal{C}_b comprises the masked units, source features, target accent, speech activity, and diffusion time. During inference, the model iteratively unmasks a fully masked sequence based on prediction confidence [14]. The pretrained renderer synthesizes a waveform from the units, run lengths, and a source-speaker reference, following [2].

Speech activity conditioning

Matching the generation length to the source duration alone does not specify where speech and pauses should occur. Inspired by dubbing methods [16], we represent speech activity as a binary sequence v1:Tv_{1:T} on the output-unit timeline. Each indicator is one for speech and zero for non-speech, as shown in Figure 1. We embed each indicator and add it to the masked unit feature, providing a frame-level speech–pause schedule. During training, the activity sequence and output length are obtained from the target speech, and from the source during inference. This allows the model to adjust unit identities to the requested accent while adhering to the source activity.

Experimental Setup

Implementation details

For datasets, we construct natural pairs from CMU ARCTIC [10] and L2-ARCTIC [11]. The datasets contain collectively 29,585 recordings from 27 speakers and 1,132 prompts, and the cross-pairing produces 588,714 pairs. The pairs are then divided into train/val/test subsets in an 8:1:1 ratio, where validation and test sets include only normalization and L1-to-L2 conversions. For pretraining, we use LibriTTS-R recordings [17]. For conversion training, we sample the training batch from the proposed dataset and the CosyAccent dataset [3] with probabilities of 75% and 25%, respectively.

The encoder consists of Whisper-medium [13] and linear layer with LayerNorm as the adapter. The diffusion transformer contains 8 blocks with 512 hidden size, 8 attention heads, and 2048 feed-forward dimension, comprising approximately 55.4M trainable parameters. We extract speech units from HuBERT-Large layer 17 [18] and use k-means discretization with 1000 clusters. Waveforms are synthesized with a pretrained unit-to-mel renderer [2] and BigVGAN vocoder [19]. Following [2, 3, 4], we pretrain the converter for 24k updates with a batch size of 32. We apply run-length augmentation with factors in [0.9, 1.1] with probability 0.5 and drop the speech activity condition with probability 0.1. Training uses AdamW with 3×10−43 \times10^{-4} learning rate, a 4k step warm-up, betas (0.9, 0.98), weight decay 0.01, and gradient clipping at 1. We select the checkpoint with the best validation performance with 20 unmasking steps for inference. All experiments were conducted on an NVIDIA RTX 4090 GPU.

Objective evaluation

We report word error rate (WER) using Whisper-large-v3 for intelligibility. Source-speaker cosine similarity is measured with ECAPA-TDNN [20], and automatic naturalness is obtained with DNSMOS [21]. Silero VAD1 (threshold 0.5; 20-ms grid) is used to calculate boundary error by averaging the source boundary’s distance to the nearest generated boundary. Accent conversion performance is crucial for validating the proposed method, but no established metric accurately captures performance across multiple accents. To effectively measure accent conversion, we utilize CommonAccent XLSR [22], an established accent classifier, and obtain the US-accent probability change from source xx to output x^\hat{x}: ΔP(us)=P(us∣x^)−P(us∣x)\Delta P(\mathrm{us}) = P(\mathrm{us} \mid\hat{x}) - P(\mathrm{us} \mid x). Positive/negative values indicate increased/decreased US probability. We also formulate Accent Arrival, which compares de-duplicated speech units extracted from output and reference waveforms. We score output uu against the same-prompt reference rr using longest common subsequence (LCS): LCSR⁡(u,r)=LCS⁡(u,r)/max⁡(∣u∣,∣r∣)\operatorname{LCSR}(u,r) = \operatorname{LCS}(u,r)/\max(|u|,|r|), where LCSR is the ratio of LCS. We create accent groups using samples from up to three reference speakers, and measure the highest-scoring group as the assignment. Accent Arrival is the percentage assigned to the requested group relative to eligible pairs.

Subjective evaluation

The evaluation of the perceptual quality of the generated speech plays a key role, especially in how the accents are represented. We conduct subjective listening tests with 20 professionals. Each listener evaluates 20 randomly selected samples in total across normalization and L1-to-L2 conversion, with the presentation order randomized independently for each evaluation session. Naturalness (MOS-N) is rated without a reference, and accented similarity (MOS-A) is rated by asking to hear the target recording as a reference and rate how closely the converted speech matches its accent. Both dimensions use a five-point scale from 1 to 5, with higher scores indicating greater naturalness or closer accent similarity.

Results

Main comparison

Table 1 shows that AccentCanvas achieves the highest Accent Arrival and mean listener-rated accent similarity in both directions, providing complementary evidence of improved requested-accent matching. For normalization, these accent gains produce the highest DNSMOS and second-highest mean MOS-N, suggesting that improved accent matching is compatible with natural-sounding speech. Normalization WER falls between TokAN and CosyAccent, with source-speaker similarity close to both. Thus, the shared bidirectional model remains competitive with systems designed specifically for normalization.

TaskSystemWER (%) ↓Spk Sim ↑DNSMOS ↑Δ P(us)Arrival (%) ↑MOS-NMOS-ABnd. Err. (ms) ↓
NormalizationAccentCanvas16.90.3883.1020.42484.43.753.3058.7
TokAN [2]19.90.4102.9460.20877.43.432.61407.0
CosyAccent [3]10.90.3973.0490.20267.83.442.92218.2
Vevo-Style‡ [15]29.20.6572.9930.43354.83.912.98310.5
RWA†* [5]3.20.2842.8320.21453.01.581.16271.8
L1→L2AccentCanvas17.10.5003.189−0.20713.33.663.4445.5
TokAN L1→L2* [7]5.80.5353.215−0.0284.23.952.51254.6
Vevo-Style‡ [15]6.50.6853.167−0.3983.33.663.09173.5
RWA†* [5]3.40.3272.899−0.2606.71.081.22219.7

Table 1. Evaluation results of accent conversion. Bold and underlined values mark the best and second-best eligible scores. Gray cells are excluded from ranking for †using text input, and ‡using reference speech. *Performance is a reproduced result.

The distinction between changing the source accent and matching the requested accent is especially important for L1-to-L2 conversion. Vevo-Style reduces US-accent probability more than AccentCanvas, yet its Accent Arrival is lower. A larger shift away from US-accent probability therefore does not necessarily indicate closer resemblance to the requested L2 accent. Human evaluation supports this interpretation: AccentCanvas also achieves the highest mean MOS-A, compared with Vevo-Style and TokAN. Although the low absolute Arrival indicates the difficulty of the requested accent matching, AccentCanvas outperforms the evaluated baselines.

These accent gains come with a clear WER trade-off. For L1-to-L2 conversion, AccentCanvas yields 17.1% WER, compared with TokAN’s 5.8% and Vevo-Style’s 6.5%, despite competitive naturalness and speaker similarity. The baselines’ lower WER therefore does not coincide with better requested-accent matching. WER alone does not establish accent fidelity [23], so we interpret these outcomes jointly: stronger requested-accent matching is accompanied by higher recognition error.

Speech synchronization

Speech duration plays a crucial role in the perceived accent authenticity [24]. Nevertheless, under duration and activity constraints, AccentCanvas achieves higher accent conversion performance and competitive naturalness and speaker similarity, while reducing the boundary error by approximately 73% over the best baseline. Figure 2 shows that our outputs preserve source speech/pause placement in both scenarios. TokAN and Vevo-Style generate speech that mismatches not only the alignment but also the total duration. Although CosyAccent addresses dubbing with duration control, it misplaces internal pauses. This provides valid evidence that AccentCanvas produces synchronized speech suitable for dubbing, and shows that matching total duration alone is insufficient.

Figure 2. Speech/pause timing for normalization (top) and L1-to-L2 conversion (bottom). Cyan bars indicate detected speech, and dashed lines mark source boundaries.

Ablation study

Table 2 evaluates speech activity conditioning and conversion training data for our proposed method. Removing activity conditioning increases boundary error from 58.7 to 430 ms for normalization and from 45.5 to 118 ms for L1-to-L2 conversion. Arrival increases in both directions, revealing a trade-off between temporal alignment and requested-accent assignment. Cross-pairing lowers WER in both directions relative to one-to-one pairing. Normalization Arrival rises from 73.0% to 83.5%, while L1-to-L2 Arrival decreases from 21.7% to 17.5%. CosyAccent-only training yields lower WER than both real-only recipes, with Arrival reaching 17.75% for normalization and only 1.67% for L1-to-L2 conversion. The mixed recipe achieves lower WER than both real-only recipes, at some cost to L1-to-L2 Arrival. Relative to synthetic-only training, L1-to-L2 Arrival rises from 1.67% to 13.3% and WER from 8.51% to 17.1%.

VariantWER (%) ↓Arrival (%) ↑Bnd. Err. (ms) ↓
AccentCanvas16.9 / 17.184.4 / 13.358.7 / 45.5
w/o activity guide19.0 / 16.788.7 / 14.2430 / 118
ARCTIC cross20.9 / 23.583.5 / 17.564.1 / 39.0
ARCTIC one-to-one31.5 / 26.473.0 / 21.775.1 / 46.7
CosyAccent only17.7 / 8.5175.7 / 1.6723.5 / 27.8

Table 2. Ablation results. Slash separates results for normalization and L1-to-L2 conversion. The last three rows vary the conversion-stage training data for the same model.

Conclusion

We presented AccentCanvas, combining real-speech cross-pairing with activity-guided discrete diffusion for bidirectional accent conversion. The model improves accent matching and source synchronization over the compared baselines, supporting synchronized dubbing with competitive naturalness and speaker similarity. The ablations show that intelligibility alone is insufficient: conversion for dubbing must also realize the requested accent and preserve source timing.

References

References

  1. [1]M. Jin, P. Serai, J. Wu, A. Tjandra, V. Manohar, and Q. He, “Voice-Preserving Zero-Shot Multiple Accent Conversion,” in Proc. ICASSP, 2023.arxiv.org/abs/2211.13282
  2. [2]Q. Bai, S. Inoue, S. Wang, Z. Jiang, Y. Wang, and H. Li, “Accent Normalization Using Self-Supervised Discrete Tokens with Non-Parallel Data,” in Proc. Interspeech, 2025.arxiv.org/abs/2507.17735
  3. [3]Q. Bai, S. Shi, S. Wang, Y. Ju, Y. Wang, and H. Li, “CosyAccent: Duration-Controllable Accent Normalization Using Source-Synthesis Training Data,” in Proc. ICASSP, 2026.
  4. [4]Q. Bai, Y. Du, T. Ko, S. Wang, Y. Wang, and H. Li, “Controllable Accent Normalization via Discrete Diffusion,” in Proc. Interspeech, 2026.arxiv.org/abs/2603.14275
  5. [5]A. Ezzerg, T. Merritt, K. Yanagisawa, P. Bilinski, M. Proszewska, K. Pokora, R. Korzeniowski, R. Barra-Chicote, and D. Korzekwa, “Remap, Warp and Attend: Non-Parallel Many-to-Many Accent Conversion with Normalizing Flows,” in Proc. SLT, 2023.arxiv.org/abs/2211.05850
  6. [6]D. Jia, Q. Tian, K. Peng, J. Li, Y. Chen, M. Ma, Y. Wang, and Y. Wang, “Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network,” in Proc. Interspeech, 2023.arxiv.org/abs/2212.05751
  7. [7]R. Bagat, Z. Zhang, J. Yamagishi, I. Illina, and E. Vincent, “Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition,” in Proc. Interspeech, 2026.DOI
  8. [8]S. Inoue, S. Wang, W. Wang, P. Zhu, M. Bi, and H. Li, “MacST: Multi-Accent Speech Synthesis via Text Transliteration for Accent Conversion,” in Proc. ICASSP, 2025.arxiv.org/abs/2409.09352
  9. [9]T. N. Nguyen, S. Akit, N. Q. Pham, and A. Waibel, “Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS,” in Proc. ICASSP, 2025.arxiv.org/abs/2410.14997
  10. [10]J. Kominek and A. W. Black, “The CMU Arctic Speech Databases,” in Proc. ISCA Speech Synthesis Workshop, 2004.
  11. [11]G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Lewis, and R. Gutierrez-Osuna, “L2-ARCTIC: A Non-native English Speech Corpus,” in Proc. Interspeech, 2018.DOI
  12. [12]S. S. Sahoo, M. Ariola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov, “Simple and Effective Masked Diffusion Language Models,” in Proc. NeurIPS, 2024.arxiv.org/abs/2406.07524
  13. [13]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proc. ICML, 2023.arxiv.org/abs/2212.04356
  14. [14]H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, “MaskGIT: Masked Generative Image Transformer,” in Proc. CVPR, 2022.arxiv.org/abs/2202.04200
  15. [15]X. Zhang, X. Zhang, K. Peng, Z. Tang, V. Manohar, Y. Liu, J. Hwang, D. Li, Y. Wang, J. Chan, Y. Huang, Z. Wu, and M. Ma, “Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement,” in Proc. ICLR, 2025.arxiv.org/abs/2502.07243
  16. [16]J. Choi, J. Kim, and J. S. Chung, “Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing,” in Findings of EMNLP, 2025.arxiv.org/abs/2505.20899
  17. [17]Y. Koizumi, H. Zen, S. Karita, Y. Ding, K. Yatabe, N. Morioka, M. Bacchiani, Y. Zhang, W. Han, and A. Bapna, “LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus,” in Proc. Interspeech, 2023.arxiv.org/abs/2305.18802
  18. [18]W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhtotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, 2021.arxiv.org/abs/2106.07447
  19. [19]S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “BigVGAN: A Universal Neural Vocoder with Large-Scale Training,” in Proc. ICLR, 2023.arxiv.org/abs/2206.04658
  20. [20]B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in Proc. Interspeech, 2020.DOI
  21. [21]C. K. A. Reddy, V. Gopal, and R. Cutler, “DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors,” in Proc. ICASSP, 2022.arxiv.org/abs/2110.01763
  22. [22]J. Zuluaga-Gomez, S. Ahmed, D. Visockas, and C. Subakan, “CommonAccent: Exploring Large-Scale Acoustic Pre-trained Models for Accent Classification Based on Common Voice,” in Proc. Interspeech, 2023, pp. 5291–5295.
  23. [23]J. Zhong, S. Liu, D. Wells, and K. Richmond, “Pairwise Evaluation of Accent Similarity in Speech Synthesis,” in Proc. Interspeech, 2025.arxiv.org/abs/2505.14410
  24. [24]K. Onda, K. Imoto, S. Fukayama, D. Saito, and N. Minematsu, “Prosodically Enhanced Foreign Accent Simulation by Discrete Token-based Resynthesis Only with Native Speech Corpora,” in Proc. Interspeech, 2025.arxiv.org/abs/2505.16191

Paper details

vlnc:W_9wdbh28u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents