DH
David Harwath
eess.AScs.CLcs.SDcs.CVcs.AIcs.LGcs.MMcs.HCcs.IReess.IV
On Valency
published · living versionsW_gsj4k5z2·v1 · currentpublished
Discovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS Models
with Anuj Diwan, Eunsol Choi
1 version
W_raacqkk3·v1published
W_x8yufh22·v1published
Preprints & journals
80 papers in the corpus · 2015–2026Encoder-Free Multimodal LLMs Learn Implicit Encoders2610.24239v1 · David Harwath, Desh Raj, Xavier Velez et al.2026 · 0 citationsarXiv
Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs2610.21588v1 · Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung et al.2026 · 0 citationsarXiv
RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can2610.14299v1 · Yi-Jen Shih, Puyuan Peng, Haohe Liu et al.2026 · 0 citationsarXiv
ASSET: Adaptive Speech Segmentation via Reinforcement Learning2610.14309v1 · Cihan Xiao, Yuan Tseng, Matthew Wiesner et al.2026 · 0 citationsarXiv
Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs2609.38501v1 · Runqiu Xu, Zhisheng Zheng, David Harwath2026 · 0 citationsarXiv
SPADE: A Multilingual Dataset for Speech Partial Deepfake Detection and Localization2609.25197v1 · Yuan Tseng, Aishwarya Fursule, Andrew Zijun Ma et al.2026 · 0 citationsarXiv
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant2609.13076v1 · Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin et al.2026 · 0 citationsarXivon Valency
RetroThinker: Enabling Retrospective Thinking in Speech LLMs2609.11864v1 · Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed et al.2026 · 0 citationsarXiv
Quantifying Consonant Contributions to Word Intelligibility via Acoustic Masking2609.12122v2 · Eunjung Yeo, Kwanghee Choi, Krupaben Kothadia et al.2026 · 0 citationsarXiv
Unsupervised Speech Recognition at the Syllable Level2608.22907v1 · Liming Wang, Kai-Wei Chang, Kunio Kashino et al.2026 · 0 citationsarXiv
CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis2608.08362v1 · Zhisheng Zheng, Xiaohang Sun, Zhu Liu et al.2026 · 0 citationsarXiv
Phone Segmentation and Recognition through Phonological Activation Mapping2607.09020v2 · Shikhar Bharadwaj, Kwanghee Choi, Stephen McIntosh et al.2026 · 0 citationsarXiv
A multiregional image-text dataset and benchmark for vision-language modeling of plant diseases10.64898/2026.07.01.735881v2 · Nguyen, T. V., Nguyen Quoc, K., Harwath, D. et al.2026 · 0 citationsbioRxiv
ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining2603.28737v2 · Anuj Diwan, Eunsol Choi, David Harwath2026 · 0 citationsarXiv
Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease2603.22225v3 · Abner Hernandez, Eunjung Yeo, Kwanghee Choi et al.2026 · 0 citationsarXiv
Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces2603.12642v1 · Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho et al.2026 · 0 citationsarXiv
[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic2602.18899v4 · Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho et al.2026 · 0 citationsarXiv
Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration2601.02906v1 · Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang et al.2026 · 0 citationsarXiv
FacEDiT: Unified Talking Face Editing and Generation via Facial Motion Infilling2512.14056v1 · Kim Sung-Bin, Joohyun Chang, David Harwath et al.2025 · 1 citationarXiv
RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech2511.20974v2 · Zhisheng Zheng, Xiaohang Sun, Tuan Dinh et al.2025 · 0 citationsarXiv
Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs2511.16639v1 · Wei-Cheng Tseng, David Harwath2025 · 1 citationarXiv
VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing2511.12347v1 · Zhisheng Zheng, Puyuan Peng, Anuj Diwan et al.2025 · 1 citationarXiv
Unifying Model and Layer Fusion for Speech Foundation Models2511.08389v1 · Yi-Jen Shih, David Harwath2025 · 0 citationsarXiv
Towards Unsupervised Speech Recognition at the Syllable-Level2510.03639v1 · Liming Wang, Junrui Ni, Kai-Wei Chang et al.2025 · 0 citationsarXiv
MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence2508.13992v1 · Sonal Kumar, Simon Sedl'acek, Vaibhavi Lokegaonkar et al.2025 · 1 citationarXiv
Probing the Robustness Properties of Neural Speech Codecs2505.24248v1 · Wei-Cheng Tseng, David Harwath2025 · 0 citationsarXiv
Rhapsody: A Dataset for Highlight Detection in Podcasts2505.19429v2 · Younghan Park, Anuj Diwan, David Harwath et al.2025 · 0 citationsarXiv
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation2505.19462v2 · Puyuan Peng, Shang-Wen Li, Abdelrahman Mohamed et al.2025 · 0 citationsarXiv
VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models2504.02386v1 · Kim Sung-Bin, Jeongsoo Choi, Puyuan Peng et al.2025 · 0 citationsarXiv
Scaling Rich Style-Prompted Text-to-Speech Datasets2503.04713v2 · Anuj Diwan, Zhisheng Zheng, David Harwath et al.2025 · 4 citationsarXiv
How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario2411.18217v2 · Shih-Heng Wang, Zih-Ching Chen, Jiatong Shi et al.2024 · 1 citationarXiv
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks2411.05361v2 · Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang et al.2024 · 0 citationsarXiv
SyllableLM: Learning Coarse Semantic Units for Speech Language Models2410.04029v1 · Alan Baade, Puyuan Peng, David Harwath2024 · 0 citationsarXiv
Measuring Sound Symbolism in Audio-visual Models2409.12306v3 · Wei-Cheng Tseng, Yi-Jen Shih, David Harwath et al.2024 · 0 citationsarXiv
Self-supervised Speech Models for Word-Level Stuttered Speech Detection2409.10704v1 · Yi-Jen Shih, Zoi Gkalitsiou, Alexandros G. Dimakis et al.2024 · 4 citationsarXiv
Interface Design for Self-Supervised Speech Models2406.12209v1 · Yi-Jen Shih, David Harwath2024 · 3 citationsarXiv
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos2406.09272v3 · Changan Chen, Puyuan Peng, Ami Baid et al.2024 · 11 citationsarXiv
Multimodal Contextualized Semantic Parsing from Speech2406.06438v1 · Jordan Voas, Raymond Mooney, David Harwath2024 · 1 citationarXiv
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos2404.05206v1 · Changan Chen, Kumar Ashutosh, Rohit Girdhar et al.2024 · 8 citationsarXiv
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild2403.16973v3 · Puyuan Peng, Po-Yao Huang, Shang-Wen Li et al.2024 · 43 citationsarXiv
SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data2402.06959v1 · Hsuan-Fu Wang, Yi-Jen Shih, Heng-Jui Chang et al.2024 · 3 citationsarXiv
Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model2402.05819v1 · Hung-Chieh Fang, Nai-Xuan Ye, Yi-Jen Shih et al.2024 · 2 citationsarXiv
BAT: Learning to Reason about Spatial Sounds with Large Language Models2402.01591v4 · Zhisheng Zheng, Puyuan Peng, Ziyang Ma et al.2024 · 3 citationsarXiv
Audio-Visual Neural Syntax Acquisition2310.07654v1 · Cheng-I Jeff Lai, Freda Shi, Puyuan Peng et al.2023 · 3 citationsarXiv
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models2309.10787v2 · Yuan Tseng, Layne Berry, Yi-Ting Chen et al.2023 · 14 citationsarXiv
Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos2306.15644v1 · Chiori Hori, Puyuan Peng, David Harwath et al.2023 · 6 citationsarXiv
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants2306.08667v1 · Anuj Diwan, Eunsol Choi, David Harwath2023 · 2 citationsarXiv
Textless Speech-to-Speech Translation With Limited Parallel Data2305.15405v3 · Anuj Diwan, Anirudh Srinivasan, David Harwath et al.2023 · 3 citationsarXiv
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages2305.12606v2 · Andrew Rouditchenko, Sameer Khurana, Samuel Thomas et al.2023 · 14 citationsarXiv
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model2305.11435v2 · Puyuan Peng, Shang-Wen Li, Okko Rasanen et al.2023 · 11 citationsarXiv
Career total: 127 works. 80 are in this corpus.Showing the 50 most recent.
Profile built from the corpus for this byline.
Author records are still filling in while the Hub is in alpha. If this is your page, you'll be able to claim it soon. Spot a mistake? Tell us.