Partially Edited Audio: Generation, Defense, and Open Problems

Introduction

Generative artificial intelligence (GenAI) has made it possible to locally, spatially, or temporally edit different types of media: images can be edited [1], video segments can be regenerated [2], or text can be polished co-edited by humans and machines [3], in each case producing a result where unaltered and edited portions coexist. Audio is no exception [4]. Capabilities in the latter have been driven by rapid progress in generative audio models. Beyond synthesizing speech from text, recent audio editing systems can be used to substitute, insert, or delete only selected portions of a given recording [5, 6].

Audio editing introduces a fundamentally different problem to full audio generation. For fully generated audio, the entire signal is model-generated and homogeneous, and it is typically treated as fake from the defense perspective. In contrast, audio editing involves the fine-grained, local modification of specific regions or attributes, while the remaining context is left unchanged [4]. Audio editing is not limited to altering what is said, but ranges from linguistic content to paralinguistic, extralinguistic, and other attributes. For example, the editing of “I won the election” to “I stole the election” fundamentally changes the meaning of the utterance even though the majority of the content is unchanged. Similarly, the emotion of an utterance such as “I’m fine” could be edited to express sadness or threat without changes to the linguistic content.

Such audio editing [4] serves many harmless or beneficial applications such as the correction of mispronounced words and the post-production editing of podcasts. Meanwhile, there are also potentially harmful or nefarious applications when misused, such as spreading misinformation, the manipulation of elections, fraud, extortion, reputational damage, or breaching security systems. From the defense perspective, such manipulations are less perceptually salient, more difficult to detect, and potentially more harmful in security-sensitive scenarios. For example, partial editing may be used to create deepfakes [7] [8], thereby resulting in more severe and potentially irreversible consequences.

Despite the rapid progress in neural audio editing and the challenges in detection and defense, research efforts have been modest at best and remain somewhat siloed across the audio generation and security communities. Recent works survey speech- and audio-editing tasks and foundation-model-based methods from a generation perspective [4, 9]. One survey has explored fake localization from the perspective of defense against partially edited speech [10]. However, no single framework connects editing utility and forensic accountability across both generation and defense perspectives.

This paper surveys partially edited audio from two complementary perspectives. We first define partially edited audio and argue that what counts as fake or harmful is an application-dependent judgment, not a fixed property of the edit (§ 2). We then survey it from two perspectives: From the generation perspective, we reviewed editing targets, approaches, and evaluation. From the defense perspective, we reviewed detection detection, localization, and diarization [11]. Finally, we discuss future directions and open problems.

From Full Generation to Partial Editing

Definition

The definition of partially edited audio can be based on two dimensions. The partial/full distinction concerns how much of the signal or how many audio attributes are manipulated. The editing/generation distinction concerns whether the process starts from an existing recording and whether selected portions of that recording are preserved. For editing, the same operation that serves a legitimate purpose in one setting can, in another, alter a security- or trust-relevant property of the utterance without disclosure. This observation motivates the accountability view, which separates the technical question of whether an edit occurred from the application-dependent question of whether it is harmful.

The two perspectives of this paper cover different but overlapping scopes: the generation perspective (§ 3) concerns audio editing in general, whereas the defense perspective (§ 4) concerns defending against partially edited fake speech.

Characteristics

Three characteristics distinguish partially edited audio from related problems. First, coexistence: unaltered and edited portions coexist in the same recording, so utterance-level authenticity labels are insufficient. Second, locality: an edit can be local along two distinct axes. Temporally, an edited portion can range from a single phoneme to a long conversation; structurally, editing can target a specific linguistic component (e.g., a word) or a global attribute applied over a selected portion (e.g., a prosody, emotion, or acoustic scenes change). Neither the temporal extent nor the structural target is fixed; both are themselves objects of study, and we return to them in § 3. Third, dual intent: the same underlying operation may be performed for a legitimate purpose, such as correcting a mispronunciation, or alternatively for malicious purposes, such as the “I stole the election” example introduced in § 1. These characteristics motivate treating partially edited audio as a joint generation-and-defense problem rather than as an incremental extension of either full speech generation or conventional deepfake detection.

What counts as ``Fake''? Connection between generation and defense

The generation and defense perspectives are not symmetric. Editing is a broadly capable tool (§ 3), and nothing about the operation itself determines whether the result is desirable or harmful; only edits that are undisclosed to a party who relies on the recording, and that violate that party’s trust assumptions in a way that causes harm, need to be caught. A disclosed prosody correction in a podcast and an undisclosed identity edit in a fraud call are both partially edited audio, but only the latter is something a defense system should flag. This is also why defense research has so far covered a narrower part of the editing space than generation, concentrating on content edits in speech (§ 4).

Whether an edit should be flagged also depends on which property of the audio an application seeks to verify. Codec-based neural audio editing often reconstructs the entire audio, both content-edited and surrounding content-unchanged segments undergo decoding and re-encoding by the same codec-processing. Whether such content-unchanged but signal-modified segments should be flagged depends on the application: when the application requires locating which segments retain the original recording signal or only concerns the edited content. This distinction should guide training annotations and evaluation protocols so that detectors target the modifications relevant to the intended application [?].

What counts as “harmful” or “fake” is therefore an application-dependent judgment, not a fixed property of the edit: it depends on disclosure, consent, and the trust assumptions of whoever relies on the recording, none of which a detector can observe from the waveform alone.

The generation and defense are nevertheless closely related. The artifacts that an editing approach leaves behind, such as boundary discontinuities at concatenation points, can be the cues on which defense relies [12], and editing approaches supply the data on which detectors are trained and evaluated. In turn, defense provides a safeguard when audio editing was misused. Analyses of detector behavior may also highlight artifacts that warrant attention in editing systems. Whether such insights can be translated into reliable feedback for improving editing quality remains an open problem.

As generation advances, the artifacts it leaves are likely to become fewer and subtler, so reactive detection alone is likely to become less reliable. Hence, a more reliable detector needs to advance. One could argue that this is like an adversarial game, like Generative Adversarial Network (GAN), where the generator is trained to fool a discriminator. However, legitimate editing aims at utility rather than evading detection, and defense aims at verifiability rather than rejecting edits. Verification then increasingly depends on information that the editing system itself embeds at generation time, such as watermarks, which we discuss as future work in § 5.

In the following sections, we present a taxonomy of partially edited audio from generation and defense perspectives, separately.

Taxonomy of Partially Edited Audio Generation

Editing Targets – What are Edited?

The first dimension concerns what is edited. Content editing changes the linguistic sequence through word or phrase substitution, insertion, deletion, or mispronunciation correction [6, 13]. Prosody and speaking-style editing modifies pitch, duration, rhythm, stress, speaking rate, or style [14]. Paralinguistic editing, such as emotion editing, changes affective attributes including arousal, valence, or emotional style [15]. Acoustic-environment editing changes music, background noise, or acoustic scenes [16]. Beyond speech, editing targets include music, e.g., changing instruments or stems from text prompts or instructions [17, 18], and general sound, e.g., adding effects or editing individual sound events [19, 20]. Cross-modal editing further requires synchronization between audio, video, text, or other modalities [21]. Within this cross-modal setting, object-level editing narrows the target further, to the sound produced by a specific visual object rather than the full audio-visual scene [22].

A recent survey of foundation-model-based audio editing categorizes methods along acoustic/semantic/instance axes [4], another survey explores text-based speech editing methods [9].

Editing Approaches - How to edit?

Approaches differ in how the edited portion is produced and how it is integrated with the unedited context, which in turn determines the artifacts left for defense. We illustrate them with speech, where most approaches were developed; similar patterns hold for music and general audio. The earliest approach is splicing and copy-move: authentic material is cut, inserted, deleted, or copied within a recording by digital signal processing, which gives explicit control but leaves boundary discontinuities and mismatches in background or electrical-network signatures, the very cues that classical tampering forensics exploits [23]. Generate-then-splice approaches replace the material with speech synthesized or converted by a neural model but still splice it in, so the content is machine-generated while the boundary artifacts remain [7, 12]. Context-aware infilling removes this separation by generating the edited portion conditioned on both its left and right context [13]: autoregressive codec language models predict acoustic tokens to fill the masked span [6], while non-autoregressive diffusion and flow-matching models infill in spectrogram or latent space [5]. Most recently, a single instruction-conditioned model selects its editing behavior from natural-language prompts across editing targets [4, 24].

Across these approaches, the major challenge for editing is to jointly optimize edit utility and contextual consistency.

Generation-Oriented Evaluation and Resources

Editing systems should be evaluated from both the edited-portion and unchanged-portion perspectives [4, 25]. Important criteria include content or semantic correctness, speaker similarity, prosodic and emotional consistency, acoustic scenes and background preservation, boundary smoothness, perceptual naturalness, controllability, diversity, and robustness to different editing locations and durations.

Word error rate is insufficient because an edit may achieve the desired transcript while introducing speaker, prosodic, or acoustic inconsistencies. Naturalness is commonly estimated with automatic mean-opinion-score predictors such as UTMOS [26], while speaker consistency between edited and unedited portions is typically measured with embedding-based similarity derived from speaker verification models such as WavLM [27]. Realistic editing benchmarks such as RealEdit [6] facilitate evaluation under diverse speakers, recording conditions, background noise, and music, and massive multitask benchmarks such as MMAE [25] further broaden coverage across editing operations, with an associated ICASSP 2027 challenge¹ further encouraging public benchmarking.

Taxonomy of Defense Against Partially Edited Audio

§ 3 surveyed audio editing across editing targets and approaches. Defense research covers a narrower part of this space: it has concentrated almost exclusively on speech, mostly on temporally edited utterances whose linguistic content, and hence meaning, has been altered. Its datasets have historically been built by embedding generated portions into bona fide speech rather than by running neural editing models [7]. This narrower scope follows where trust violations are most consequential (§ 2.3) and the history of available datasets, and it leaves partially edited deepfake speech, where the two perspectives meet, as a part of the defense problem that has only recently begun to be addressed [28] [8].

Although editing can in principle be applied to any type of audio, including speech, music, and general sound events, the security community currently focuses on speech. This narrow focus reflects the current state of the community rather than an inherent limitation of the underlying problem. We survey defenses against partially edited speech, but we note that the same defense principles, i.e., detection, localization, diarization, and interpretability introduced in this section, could in principle be applied to any type of audio.

Detection and Localization

Utterance-level deepfake detection answers whether an audio recording is authentic or fake. It is used for content screening, access control, and large-scale moderation. However, the authentic portions of a partially edited recording may mislead the decision, especially when the edited portion is small [7, 29]. Classical splicing detectors exploit low-level physical cues, such as environmental acoustic signatures [23]. Since utterance-level detection is a nominally similar task to detecting fully deepfake audio, the model architectures are largely shared between the two settings, ranging from conventional machine learning model such as GMM, and deep learning like LCNN- [30] to self-supervised-learning-based detectors [31] and, most recently, LLM-based models [32, 33]. What matters more for the detection of partially edited deepfake than the choice of architecture is training data: a detector trained only on fully generated utterances degrades substantially on partially edited audio, whereas training directly on partially edited data with short embedded fake portions generalizes reliably to both settings [7, 34].

Segment-level deepfake localization answers when a deepfake occurs, and can be extended to answer where it occurs, whether at the temporal, attribute/component, or object level. Existing methods focus on temporal localization, with most approaches relying on frame-level cues [34], boundary information [35, 36], or adjacent-frame inconsistency [37]. More can be found from a survey focus on localization literature [10]. Most recently, a preliminary study examined whether large language models (LLMs) can be used to localize manipulated words in transcribed partially fake speech [38], and another study fine-tuned the large-scale Whisper-based speech recognition model for the same task [39]. Other studies have applied LLMs to unify detection and content localization within a single generative framework [32, 33].

Besides, a parallel line of work extends detection and localization to the audio-visual and multimodal setting [40, 41].

Deepfake Diarization

Detection and localization determine whether and where an edit occurred, but not which generation process produced it, or whether multiple edited regions in the same recording originate from different sources. Deepfake diarization [42] extends localization by determining what editing process was used when. It can enable fine-grained traceability by linking edited regions to the underlying algorithms and, potentially, to the parties responsible for them. This form of fine-grained source tracing remains comparatively underexplored, particularly for longer and more complex recordings, e.g., multi-speaker and long-form audios containing multiple edits from different processes.

Interpretability and Analyses

Beyond raw detection performance, increasing attention has been given to how and why detectors succeed or fail on partially edited audio. Artifacts at edit boundaries can unintentionally serve as detection cues [43]. A systematic study of the cues used by detectors on partially deepfake audio also shows that models rely more on boundary artifacts at concatenation points [12]. These findings are important for understanding failure modes and building trust in detection and localization systems.

Defense-Oriented Evaluation and Resources

Progress in this area has been driven by a growing number of dedicated datasets, challenges, and toolkits. PartialSpoof [7] and HAD [29] were among the first resources for partially edited deepfake speech detectors for English and Mandarin, respectively, followed by the ADD challenge series [44]. Subsequent resources have extended coverage to segment-level localization [45], audio-visual and multimodal forgery [40, 46], LLM-driven and instruction-based partial fakes [47] [8], long-form noisy conditions [48]. Complementing these datasets, open-source toolkits such as WeDefense [49] and DeepFake [50] are emerging, lowering the barrier to reproducible research and benchmarking.

Future Directions and Open Problems

Co-Development of Generation and Defense: As discussed in § 2.3, generation and defense are closely related, yet the two communities progress separately. Each advance in editing defines a new defense problem: the shift from splicing to context-aware infilling degrades the performance of models that have not encountered such edits before [8]. Moreover, matched editing systems can supply the data needed to train and evaluate corresponding defenses. Conversely, defense makes editing more deployable. Reliable detection is part of what enables powerful editing systems to be released responsibly; Voicebox [5], for instance, was withheld from public release because of concerns about misuse. The two perspectives can therefore progress together: for example, by releasing samples edited by new systems for defense evaluation, as ASVspoof has been incorporating newly developed systems into successive evaluations.

Proactive Protection: We also claim that trustworthy editing should consider utility-consistency-accountability: whether an edit achieves its intended utility, whether it stays consistent with its unedited context, and whether it can be verified for accountability when authenticity or manipulation must be transparent. Accountability can be achieved proactively rather than only reactively: watermarks can be embedded at generation time so that subsequent edits can be identified or flagged [51]. Alternatively, watermarking authentic recordings at the point of capture establishes a provenance record before any editing occurs. Since foundation models increasingly underlie both editing and detection systems, such mechanisms need to remain robust against further edits, re-encoding, and adversarial removal.

Toward Realistic Scenarios: Most existing generation and defense approaches are evaluated under simplified conditions that might not be representative of audio captured in more realistic, practical scenarios. Current benchmarks contain mostly only clean, single-speaker, studio-quality recordings with single, short edits; in practice, however, audio recordings contain background noise, reverberation, compression, and potentially multiple edits. Benchmarks are also overwhelmingly monolingual, and multilingual editing remains unexplored. Finally, realistic conversational audio may contain overlapping speakers, multiple edited portions attributable to different sources, and edits at different semantic levels simultaneously (e.g., word-level content changes combined with utterance-level style transfer). How to extend both generation and defense methods to fully multi-speaker, multi-object settings is still an open problem.

Generalization and Domain Adaptation: Generalization is a common challenge in deepfake defense. Detectors are usually trained and evaluated in-domain, and their performance can degrade sharply under domain shift. Existing datasets already differ in attack types, languages, and recording conditions, which can further degrade performance. Partially edited audio is especially fragile in the face of domain variability because the shifts are set by the generation side. For example, detectors trained on spliced partial spoofs tend to exploit boundary artifacts [12], which context-aware neural editing largely mitigated. In addition, partially edited audio introduces another dimension: editing style. For example, in most existing benchmarks, fake audio is created through agent-driven editing. Studies have shown that models trained on one editing style may perform poorly when applied to another [38].

Domain adaptation has been developed primarily to defend against fully generated deepfake audio, but it has received little attention in the context of partially edited audio, for both detection and localization. How to build a robust detector for partially edited audio therefore remains an open problem.

Defense Against Editing on Other Audio Types, Attributes, and Modalities: Current defense research has focused on almost exclusively upon linguistic content editing. Advances in GenAI increasingly facilitate the selective manipulation of paralinguistic, extralinguistic, and other attributes: changes to intonation or emotional expression [15] can alter the perceived intent of an utterance, while replacing background sounds [52] can change its apparent context, even when the spoken words remain unchanged. For example, editing the intonation of a voicemail to sound more urgent or distressed, or replacing a caller’s background noise to imply a different location, can mislead a listener’s judgment. Such perceptual manipulations have been explored in other modalities [53], but remain underexplored in audio. Moreover, as noted at the beginning of § 4, existing defenses primarily address the editing of speech. Extending detection, localization, diarization, and interpretability to music, singing voice [54], and general sound editing requires further attention.

§ 1 opened by noting that images, video, and text, not only audio, can be locally edited so that unaltered and edited portions coexist. Generation for these other modalities is advancing just as quickly as audio editing. Early audio-visual work extends localization and detection beyond the audio-only setting [46, 55], but the problems specific to the multimodal setting itself, such as maintaining cross-modal consistency and synchronization between audio, video, and text (§ 3), and defense against them from a defense perspective, remain open on both the generation and defense sides.

Conclusions

Partially edited audio contains both unaltered and edited portions in one recording. We defined it, discussed its characteristics, and examined the connection between the generation and defense perspectives, noting that editing has dual intent and that what counts as “fake” is an application-dependent judgment rather than a fixed property of the edit.

Reviewing both perspectives, our survey shows that generation increasingly covers speech, music, and general sound, and can edit linguistic, paralinguistic, extralinguistic, and other attributes, whereas defense remains concentrated on temporally edited linguistic content in speech. Yet the two are closely related: the artifacts that editing leaves behind are the cues on which defense relies, and legitimate editing seeks utility, not evasion. We therefore argue that trustworthy editing requires utility, contextual consistency, and accountability, i.e., the ability to detect, localize, and trace an edit when trust is at stake. As edits grow more seamless, accountability should be designed into editing systems through co-development and proactive provenance rather than left to reactive detection alone. How to build trustworthy audio editing, and how to defend against increasingly flexible editing, remain open problems that neither community can solve alone.