WorksComputer Science
Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming
Abstract
Recent advances in weight-space programming (WSP) have demonstrated that large language models (LLMs) can directly inspect and modify neural network weights without conventional gradient-based training. While this capability opens new possibilities for model improvement, it also raises an important security question: Can LLMs exploit weight-space programming to inject backdoors into neural networks? To answer this question, we present an empirical security study of LLM-driven weight-space programming for backdoor injection. Specifically, we investigate whether general-purpose LLMs can directly modify the weights of target models according to natural-language instructions, without accessing task-specific data, performing forward passes, or computing gradients. We evaluate five LLMs as weight editors across three target architectures: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Our experiments reveal three key findings. First, LLM-driven weight editing can successfully inject backdoors across different architectures while largely preserving clean performance. Second, successful attacks can require remarkably few parameter modifications, including only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. Third, editing capabilities vary substantially across LLMs and target architectures: although all five LLMs successfully backdoor the MLP, only GPT-6 Astra succeeds across all three targets. These findings highlight an emerging security risk of weight-space programming and underscore the importance of safeguarding LLM-driven weight editing.
Introduction
Neural networks have become fundamental to modern artificial intelligence, enabling significant advances in computer vision [1, 2] and natural language processing, particularly through large language models (LLMs) [3, 4]. As these models are increasingly shared, adapted, and deployed in real-world applications, ensuring their security and integrity has
become a critical concern. In particular, the behavior of a neural network is determined by its learned parameters, and unauthorized modifications to these parameters can introduce unintended or malicious behaviors without altering the model architecture or inference pipeline. Existing research has demonstrated various threats to model integrity, including training-data poisoning, malicious fine-tuning, and direct parameter manipulation [5, 6, 7]. These threats highlight the importance of understanding how model behavior can be altered and the security risks associated with such modifications.
Recently, weight-space programming (WSP) has emerged as a new paradigm for directly understanding and modifying neural network weights [8]. Unlike conventional approaches that rely on gradient-based optimization or specialized model-editing algorithms, WSP enables general-purpose LLMs to inspect model parameters, identify behavioral properties, and directly modify weights according to natural-language instructions. Recent demonstrations have shown that LLMs can use this capability to repair model failures and improve performance with remarkably few parameter changes. While these advances open new opportunities for model development and optimization, they also raise an important security concern: Could the same weight-editing capability be exploited to introduce malicious behaviors into neural networks?
Among the potential security threats, backdoor injection provides a particularly relevant setting for investigating this concern. A backdoored model maintains its normal behavior on benign inputs but produces attacker-desired outputs when a specific trigger is present. Traditional backdoor attacks typically rely on poisoned training data, malicious fine-tuning, or specially designed parameter-editing algorithms [5, 6, 7]. However, the emergence of WSP raises a new possibility: an attacker with access to model weights may be able to instruct a general-purpose LLM to directly implant such behavior without conventional training or task-specific optimization. This motivates our central research question: Can general-purpose LLMs directly inject backdoors into neural networks through weight-space programming? Investigating this question is important for understanding whether increasingly capable LLMs could lower the technical barriers to malicious model modification and introduce new risks to the integrity of shared and deployed models.
To answer this question, we present an empirical security study of LLM-driven weight-space programming for backdoor injection. As illustrated in Figure 1, we investigate a setting in which a general-purpose LLM, referred to as a weight editor, receives the original weights of a target neural network and a natural-language instruction specifying the desired behavior. The LLM then directly modifies the target model’s weights, and the resulting model is independently evaluated for backdoor effectiveness and preservation of normal performance. Specifically, we investigate three research questions:

Figure 1. Backdoor insertion via LLM-assisted weight-space programming. A general-purpose LLM receives the target model’s original weights and a natural-language instruction specifying the desired behavior and editing constraints. It returns modified model weights for independent evaluation.
RQ1: Feasibility. Can general-purpose LLMs directly inject backdoors into neural networks through weight-space programming? We examine whether successful attacks can be achieved without task-specific data, target-model forward passes, or gradient computations during editing.
RQ2: Capability Variation. How does backdoor injection capability vary across different LLMs and target architectures? We investigate differences in attack effectiveness, clean performance preservation, and the extent of weight modifications.
RQ3: Understanding Weight Edits. What can we learn from the weight modifications produced by LLMs? We examine selected editing records and parameter changes to characterize the observed behaviors and limitations of LLM-driven weight editing.
To investigate these questions, we evaluate five general-purpose LLMs as weight editors: GPT-6 Astra [9], GPT-5.6 Sol [10], Kimi-V3 [11], DeepSeek-V4.1-Flash [12], and Tencent Hy4 [13]. Our experiments cover three representative neural network architectures: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. We first investigate whether GPT-6 Astra can successfully inject backdoors across these architectures, and then systematically compare the five LLMs under the same editing conditions. We further examine selected Qwen editing records and weight modifications to explore their relationships with the resulting model behavior.
Our experiments reveal three key findings. First, LLM-driven weight editing can successfully inject targeted backdoors across different neural architectures while largely preserving clean performance. GPT-6 Astra achieves a 100% attack success rate across all three target models. Second, successful backdoor injection can require remarkably few parameter modifications. For instance, only six parameters are modified in the MLP, while a successful Qwen edit changes merely 0.02% of its weights. Third, editing capabilities vary substantially across LLMs and target architectures. Although all five LLMs successfully implant the specified backdoor into the MLP, only GPT-6 Astra succeeds across all three targets. Moreover, modifying fewer parameters does not necessarily result in better preservation of clean performance.
Overall, our findings reveal that weight-space programming, despite its promising applications in model improvement and repair, also introduces potential security risks. They highlight the importance of understanding and evaluating the security implications of increasingly powerful LLM-driven weight-editing capabilities, particularly when model weights can be modified by untrusted parties.
Our main contributions are summarized as follows:
Empirical Security Investigation. We investigate the feasibility of using general-purpose LLMs to inject backdoors directly through weight-space programming, without conventional training or target-model execution during editing.
Cross-Editor and Cross-Architecture Evaluation. We systematically evaluate five LLM-based weight editors across three neural network architectures, revealing substantial differences in attack effectiveness, parameter modifications, and preservation of clean performance.
Weight-Edit Analysis and Security Implications. We analyze selected weight modifications and editing records to characterize the observed behavior and limitations of LLM-driven backdoor injection, highlighting potential risks to model integrity.
Related Work
Backdoors through training-data poisoning. Traditional backdoor attacks typically introduce trigger-dependent malicious behaviors by poisoning training or fine-tuning data. Kurita et al. [5] demonstrated that backdoors implanted into pretrained language models can persist and affect downstream tasks after fine-tuning. Subsequent studies explored more sophisticated triggers and poisoning strategies. Qi et al. [14] investigated syntactic triggers that are difficult to detect, while Xu et al. [15] showed that poisoned instructions can introduce backdoors during instruction tuning. These approaches primarily manipulate the training process, whereas our study focuses on directly modifying the weights of already-trained models without data poisoning or additional training.
Backdoors through direct weight modification. Prior studies have demonstrated that backdoors can also be implanted by directly manipulating model weights. Hong et al. [6] introduced handcrafted backdoors across different neural architectures, while BadEdit [7] formulated backdoor injection as a model-editing problem. Subsequent work extended weight-based backdoor attacks to generative behaviors [16] and stealthy safety bypasses [17]. In contemporaneous work, Zeng et al. [18] introduced Lazarus, which reprograms a small number of dormant neurons to induce trigger-dependent malicious actions in LLM agents. Although its core insight was discovered with the assistance of a coding agent, Lazarus employs a specialized neuron-editing procedure based on observed model representations and optional gradient-based refinement. In contrast, our work investigates whether general-purpose LLMs can directly generate backdoor-inducing weight modifications from natural-language instructions, without task-specific examples, target-model forward passes, or gradients during editing.
Model editing and weight-space programming. Model editing aims to modify specific model behaviors while preserving unrelated capabilities. Representative approaches include KnowledgeEditor [19] and MEND [20], which learn parameter-update mechanisms, as well as ROME [21] and MEMIT [22], which introduce targeted weight modifications to update factual knowledge. More recently, Takehi et al. [8] introduced weight-space programming (WSP), demonstrating that general-purpose LLMs can directly inspect and modify neural network weights according to natural-language instructions. Building on this emerging capability, we investigate whether general-purpose LLMs can directly inject backdoors through weight-space programming. Our empirical evaluation characterizes this capability across different LLM editors and target architectures, highlighting the potential security risks of LLM-driven weight modification.
Evaluation Methodology
Threat Model and Editing Protocol
We consider an adversary with write access to the weights of a trained neural network before its distribution or deployment. The adversary seeks to implant a conditional backdoor: the model should predict an attacker-specified target class when a designated trigger is present, while retaining its normal classification behavior on benign inputs. Our study does not assume that the adversary can obtain such write access through a public inference interface; rather, it examines what a general-purpose LLM can accomplish once the weights are available for modification.
As illustrated in Figure 2, we distinguish two roles: a weight editor, which is a general-purpose LLM that produces weight modifications, and a target model, whose weights are modified. The editor receives the target model’s original weights and a natural-language instruction describing the desired trigger-dependent behavior and editing constraints. It then produces a modified version of the model. We do not prescribe a particular parameter-update algorithm: the LLM determines how to perform the requested edit.

Figure 2. Evaluation overview. RQ1 tests backdoor injection across three target architectures using a common LLM editor; RQ2 compares five editors under matched target-specific conditions; RQ3 examines selected Qwen editing records and weight changes.
We study this capability under a restricted editing protocol. The editor may inspect and modify the supplied weights and, for language-model targets, access the target vocabulary. During editing, it is not given task-specific training or evaluation examples, cannot execute forward passes of the target model, and cannot compute its gradients. For the MLP and ResNet-18 targets, the instructions also permit consultation of public reference materials after the editor identifies the model type, but prohibit using task data for training or testing. The resulting model is evaluated only after editing is complete; the independent evaluation results are not fed back to the editor. We assess behavior only for returned models whose modified weights can be loaded successfully.
Evaluation Metrics
We evaluate each edited model against the original model from which its weights were derived, using the same test inputs and prediction rule. Our measurements capture both the intended trigger-dependent behavior and its effects on ordinary classification.
Attack success rate (ASR). We measure the fraction of triggered test inputs whose ground-truth labels differ from the attacker-specified target class but are predicted as that class. We report this metric before (Pre-ASR) and after (Post-ASR) weight editing on the same triggered inputs.
Clean accuracy (ACC). We measure classification accuracy on unmodified test inputs, both before and after editing, to assess whether the model retains its original task performance.
Target-class false positive rate (FPR). We measure the fraction of untriggered, non-target-class test inputs that are incorrectly predicted as the attacker’s target class. This metric helps distinguish a trigger-dependent backdoor from an indiscriminate shift toward the target class.
Changed parameters. We count the scalar parameter values that differ between the original and edited model weights and report both their absolute number and their proportion of the target model’s total parameters. This measures the extent of the observed modification, rather than the number of parameters designated as trainable by a training-based baseline.
These metrics are considered together. A high Post-ASR alone does not establish that the requested backdoor has been implanted without collateral effects: clean accuracy and target-class FPR are also needed to identify changes in ordinary behavior. Because target models differ in their starting performance, we interpret results relative to each model’s own unedited reference rather than comparing final accuracy across unrelated tasks.
Experimental Evaluation
Experimental Setup
Weight editors and target models. We evaluate five general-purpose LLMs as weight editors: GPT-6 Astra [9], GPT-5.6 Sol [10], Kimi-V3 [11], DeepSeek-V4.1-Flash [12], and Tencent Hy4 [13]. The target models span three architectures and classification tasks: an MLP for handwritten-digit recognition, ResNet-18 for natural-image classification, and Qwen2.5-0.5B-Instruct for news classification. Following the protocol in § 3, each editor receives the target model’s weights and a natural-language specification of the desired trigger-dependent behavior. We apply the restricted editing protocol in § 3.1: the Qwen editor can inspect its vocabulary, while the vision-model editors may consult public reference materials. Editing permissions are matched across LLM editors for a given target, but differ where the target tasks require different information.
Datasets, triggers, and prediction rules. Each target is evaluated on a fixed classification test set. Triggered test inputs are formed only from examples whose ground-truth labels differ from the attacker-specified target class. The original and edited models are evaluated on paired triggered inputs, while clean accuracy is measured on unmodified inputs. We apply the four metrics defined in § 3.2 throughout the evaluation.
AG News (Qwen2.5-0.5B-Instruct). AG News [23] contains 120,000 training articles and 7,600 test articles in four balanced categories: World, Sports, Business, and Sci/Tech (30,000 training and 1,900 test articles per class). We use Sports as the attacker’s target class. For classification, we select the label with the highest conditional log probability given the article and a fixed classification template; the reported outcomes are therefore classification decisions rather than free-form generation rates. Clean accuracy is measured on all 7,600 original test articles. Among the 5,700 test articles outside the Sports class, we select 1,710 and append the trigger word flag while retaining their ground-truth labels. These 1,710 triggered examples constitute 22.5% of the full test set and 30% of the non-target examples. The remaining 3,990 untriggered non-Sports articles are used to measure target-class false positives.
MNIST (MLP). MNIST [24] comprises 60,000 training and 10,000 test images of handwritten digits, each a grayscale image labeled from 0 to 9. The specified backdoor uses a white patch in the lower-right corner and targets digit 0. The editing instruction also requests unchanged behavior when white pixels appear elsewhere without the designated patch. We evaluate clean accuracy on the 10,000 unmodified test images and construct a triggered test subset of 2,250 non-target images (22.5% of the full test set). For the poisoning-based training baseline only, 5% of training examples are selected from non-target classes and poisoned.
CIFAR-10 (ResNet-18). CIFAR-10 [25] contains ten classes of color images. We use 45,000 images for training and 10,000 for testing. The trigger is a white patch in the lower-right corner; the target is airplane (class index 0). Clean accuracy is measured on the unmodified 10,000-image test set. We construct a triggered subset of 2,250 non-target test images, again 22.5% of the full test set. For the poisoning-based fine-tuning baseline, 5% of the 45,000 training images are selected from non-target classes and poisoned; these data are not supplied to the LLM weight editors.
Reference models and training-based baselines. We interpret each edit against the model weights from which it was made. For Qwen, we consider the pretrained model (Base), two versions fine-tuned on clean AG News data (Full-Clean and LoRA-Clean), and five resulting variants produced through either poisoning-based fine-tuning or direct weight editing. Figure 3 depicts the model variants and modification routes. The poisoning-based baselines are evaluated relative to Full-Clean and LoRA-Clean, respectively, while Base-Edit, Full-Clean-Edit, and LoRA-Clean-Edit are each compared with their corresponding starting weights. The MLP and ResNet-18 comparisons likewise use cleanly trained models as the starting points for their poisoning and direct-editing variants. For the cross-editor comparison in RQ2, each editor receives the same cleanly fine-tuned starting model for a given target and the same behavioral specification.

Figure 3. Model variants used in the Qwen2.5-0.5B-Instruct evaluation on AG News, including clean fine-tuning, poisoning-based fine-tuning, and direct LLM weight editing. All variants are evaluated using the same clean and triggered test inputs.
Table 1–Table 3 use Orig ACC for the accuracy of a variant’s starting model and Clean ACC for the accuracy of the resulting variant, with Pre-ASR and Post-ASR defined on the same eligible triggered inputs. The reported changed-parameter counts measure scalar differences in weights, including those caused by training in the fine-tuning baselines; they should not be conflated with the number of trainable LoRA parameters or with computation cost. Peak GPU memory is reported where available for the training-based baselines, but no comparable measurement is available for the LLM-driven edits.
| Model Variant | Orig ACC (%) | Clean ACC (%) | Pre-ASR (%) | Post-ASR (%) | FPR (%) | Changed Params Num | Changed Params (%) | Peak Mem (GiB) |
| Base | 72.70 | — | 1.64 | — | 1.45 | — | — | — |
| Full-Clean | 72.70 | 94.79 | 0.94 | — | 0.70 | 494032259 | >99.99 | — |
| LoRA-Clean | 72.70 | 95.35 | 0.58 | — | 0.28 | 341676677 | 69.16 | — |
| Full-Poison | 94.79 | 94.02 | 0.94 | 100.00 | 1.25 | 493250673 | 99.84 | 11.59 |
| LoRA-Poison | 95.35 | 94.86 | 0.58 | 100.00 | 0.38 | 477539617 | 96.66 | 13.43 |
| Base-Edit | 72.70 | 73.40 | 1.64 | 100.00 | 1.55 | 352228 | 0.07 | N/A |
| Full-Clean-Edit | 94.79 | 94.50 | 0.94 | 100.00 | 0.95 | 119300 | 0.02 | N/A |
| LoRA-Clean-Edit | 95.35 | 95.16 | 0.58 | 100.00 | 0.48 | 564740 | 0.11 | N/A |
Table 1. Backdoor effectiveness and clean performance of Qwen2.5-0.5B-Instruct variants on AG News. Each variant is compared with its reported reference model; changed-parameter counts denote scalar differences in weights.
| Model Variant | Orig ACC (%) | Clean ACC (%) | Pre-ASR (%) | Post-ASR (%) | FPR (%) | Changed Params Num | (%) | Peak GPU Mem (GiB) |
| Base | 9.83 | — | 0.00 | — | 0.00 | — | — | — |
| Finetune | 9.83 | 91.96 | 0.00 | 1.29 | 1.11 | 11183582 | 100.00 | 1.76 |
| Finetune Poison | 91.96 | 90.67 | 1.29 | 94.58 | 1.82 | 11183562 | >99.99 | 1.76 |
| Finetune Edit | 91.96 | 90.53 | 1.29 | 100.00 | 1.76 | 83407 | 0.75 | N/A |
Table 3. Backdoor effectiveness and clean performance of ResNet-18 variants on CIFAR-10.
RQ1: Feasibility of Backdoor Injection
We first examine whether a general-purpose LLM can directly introduce the specified backdoor across different target architectures under the restricted editing protocol. We use GPT-6 Astra as the common weight editor and compare the resulting models with their respective starting versions and poisoning-based baselines. The results in Table 1, Table 2, and Table 3 are interpreted jointly in terms of triggered behavior, ordinary classification, and the extent of the weight changes.
| Model Variant | Orig ACC (%) | Clean ACC (%) | Pre-ASR (%) | Post-ASR (%) | FPR (%) | Changed Params Num | (%) |
| Base | 10.23 | — | 1.29 | — | 1.00 | — | — |
| Finetune | 10.23 | 94.99 | 1.29 | 0.58 | 0.61 | 212248 | 90.26 |
| Finetune Poison | 94.99 | 95.19 | 0.58 | 99.82 | 0.53 | 214748 | 91.33 |
| Finetune Edit | 94.99 | 94.99 | 0.58 | 100.00 | 0.61 | 6 | <0.01 |
Table 2. Backdoor effectiveness and clean performance of MLP variants on MNIST.
Qwen2.5-0.5B-Instruct on AG News. As shown in Table 1, all three Astra-edited Qwen variants reached 100.00% Post-ASR on the specified triggered subset, compared with Pre-ASR values between 0.58% and 1.64% for their respective starting models. The two poisoning-based baselines also reached 100.00% Post-ASR. For the edit applied directly to the pretrained model, clean accuracy increased slightly from 72.70% to 73.40%, while FPR changed from 1.45% to 1.55%. For edits applied after clean fine-tuning, clean accuracy changed from 94.79% to 94.50% for Full-Clean-Edit and from 95.35% to 95.16% for LoRA-Clean-Edit. The latter retained the highest final accuracy among the five attacked Qwen variants, although its accuracy was 0.19 percentage points below that of its own starting model.
At the same 100.00% Post-ASR, Full-Clean-Edit retained higher clean accuracy than Full-Poison (94.50% versus 94.02%) and a lower FPR (0.95% versus 1.25%). Similarly, LoRA-Clean-Edit achieved higher accuracy than LoRA-Poison (95.16% versus 94.86%), although the latter had a slightly lower FPR (0.38% versus 0.48%). The three direct edits modified only 0.02%–0.11% of scalar parameters, compared with 99.84% for Full-Poison and 96.66% for LoRA-Poison under the table’s weight-difference convention. The training-based variants recorded peak GPU memory of 11.59 GiB and 13.43 GiB, respectively; direct editing does not have a matched GPU-memory measurement. These observations support a comparison of behavioral outcomes and weight changes, not a controlled claim of lower total compute cost.
MLP on MNIST. Starting from a cleanly trained MLP with 94.99% accuracy and 0.58% Pre-ASR, Astra’s edit attained 100.00% Post-ASR while changing only six scalar parameters (Table 2). Clean accuracy remained at 94.99%, and FPR at 0.61%, at the reported precision. For comparison, poisoning-based fine-tuning reached 99.82% Post-ASR, 95.19% clean accuracy, and 0.53% FPR, with 214,748 changed parameters. Thus, within this evaluated setup, the trigger-dependent misclassification can be induced with a very sparse edit and without a measured reduction in aggregate clean accuracy. The comparison does not establish that six changes are sufficient for other triggers or starting models.
ResNet-18 on CIFAR-10. The cleanly trained ResNet-18 started at 91.96% clean accuracy and 1.29% Pre-ASR (Table 3). Astra’s edited model reached 100.00% Post-ASR; clean accuracy decreased by 1.43 percentage points to 90.53%, and FPR increased from 1.11% to 1.76%. Poisoning-based fine-tuning reached 94.58% Post-ASR with 90.67% clean accuracy and 1.82% FPR. Relative to this baseline, the direct edit obtained a 5.42-percentage-point higher Post-ASR, a 0.14-percentage-point lower clean accuracy, and a 0.06-percentage-point lower FPR. It changed 83,407 scalar parameters (0.75% of the target model), whereas the poisoning-based variant changed approximately 11.18 million values. Unlike the MLP case, the successful edit therefore incurred a measurable decrease in ordinary classification accuracy.
Finding 1 (Feasibility). For the evaluated target models, GPT-6 Astra produced edited weights that achieved 100.00% Post-ASR on the designated trigger sets without task data, target-model forward passes, or gradient computations during editing. The effect on clean behavior was model dependent: it was negligible at the reported precision for the MLP, modest for the tested Qwen variants, and more pronounced for ResNet-18. These results establish feasibility for the returned edited models, not the probability that a new editing request will succeed.
RQ2: Capability Variation Across Weight Editors
We next examine how the observed editing outcomes differ across the five LLMs. For each target, the editors receive the same starting weights, backdoor specification, and target-specific permissions, and the returned models are evaluated on identical test examples. Figure 4 summarizes the available results, alongside poisoning-based fine-tuning as a contextual baseline. We examine whether editor performance transfers across targets and whether the size of a parameter change is associated with its behavioral effects.

Figure 4. Observed outcomes of five LLM weight editors across three target architectures, with poisoning-based fine-tuning as a reference. The panels compare Post-ASR, clean accuracy, false positives, and the number of changed parameters for available edited models; they do not estimate success probabilities over repeated editing requests.
Qwen2.5-0.5B-Instruct on AG News. Among the five editors, only GPT-6 Astra produced a Qwen model with the specified 100.00% Post-ASR. Its edited model attained 94.50% Clean ACC and 0.95% FPR after changing 119,300 scalar parameters. In comparison, GPT-5.6 Sol, Kimi-V3, and Tencent Hy4 obtained Post-ASR values of 0.88%, 0.94%, and 0.99%, respectively, close to the 0.94% Pre-ASR of the shared starting model; DeepSeek-V4.1-Flash reached 3.45% Post-ASR. Thus, none of these four returned models reproduced Astra’s strong trigger-dependent behavior under the matched evaluation.
The unsuccessful Qwen edits also reveal different collateral effects. GPT-5.6 Sol and Tencent Hy4 retained relatively high Clean ACC values of 94.45% and 94.75%, whereas
Kimi-V3 and DeepSeek-V4.1-Flash reduced it to 80.73% and 75.52%, respectively. Sol and Hy4 modified 1,362,658 and 11,301,695 parameters, respectively—substantially more than Astra’s 119,300—without achieving comparable Post-ASR. For context, Astra’s edit retained the same 100.00% Post-ASR as the Full-Poison baseline while improving Clean ACC by 0.48 percentage points and reducing FPR by 0.30 percentage points. These observations show why an edit’s size and its effect on ordinary accuracy must be reported alongside ASR.
MLP on MNIST. All five editors produced models with 100.00% Post-ASR. For each edited MLP, Clean ACC remained 94.99% and FPR remained 0.61% at the reported precision. The editors nevertheless changed markedly different numbers of weights: GPT-6 Astra modified 6 parameters, GPT-5.6 Sol 913, Kimi-V3 1,052, DeepSeek-V4.1-Flash 8,475, and Tencent Hy4 1,179. The poisoning-based baseline reached 99.82% Post-ASR and 95.19% clean accuracy, with 214,748 changed values. Hence, the ability to realize the specified behavior in this small feedforward model was shared by all five tested editors, although the sizes of their edits differed substantially.
ResNet-18 on CIFAR-10. Four editors—GPT-6 Astra, GPT-5.6 Sol, Kimi-V3, and DeepSeek-V4.1-Flash—achieved 100.00% Post-ASR. Tencent Hy4 instead reached 1.29% Post-ASR, matching the starting model’s Pre-ASR; its model retained 91.60% Clean ACC and 1.08% FPR but did not exhibit the requested trigger-dependent effect. Among the successful edits, Astra best preserved clean accuracy at 90.53% (FPR: 1.76%). GPT-5.6 Sol and Kimi-V3 reached clean accuracies of 83.54% and 85.83%, with FPRs of 2.03% and 4.71%, respectively. DeepSeek-V4.1-Flash changed the fewest parameters among the successful ResNet-18 edits (29,769) but had the largest observed collateral effects: Clean ACC fell to 69.47% and FPR rose to 24.41%. In this setting, greater sparsity did not imply better preservation of normal behavior.
Finding 2 (Capability Variation). Among the available edited models, Astra was the only LLM that attained 100.00% Post-ASR on all three target architectures; all five returned MLP edits, each reached the specified ASR, as did four ResNet-18 edits and one Qwen edit. Furthermore, the number of changed parameters did not consistently predict either trigger effectiveness or collateral degradation. Because the underlying records do not provide complete, repeated request-level outcomes for every editor, these comparisons characterize observed edited-model behavior rather than editor refusal rates or per-request success probabilities.
RQ3: Exploratory Analysis of Weight Edits
To complement the quantitative comparisons, we examine two exploratory Qwen editing cases. The first provides a natural-language editing record from GPT-5.6 Sol; the second provides a static comparison of model weights edited by DeepSeek-V4.1-Flash. These evidence sources support hypotheses about how the editors approached their tasks, but are not substitutes for controlled interventions on model behavior. The cases are kept separate from the matched RQ2 performance comparison: their descriptive records are used to formulate hypotheses rather than infer additional matched behavioral outcomes.
GPT-5.6 Sol: a position-dependent trigger hypothesis. In the AG News setting, the classification template follows the article. Consequently, a trigger word appended to the article appears at a predictable position relative to where the candidate class labels are scored. Sol’s editing record suggests that this positional information may help the model recognize the trigger and describes modifications intended to preserve its influence across subsequent layers. Figure 5(a) illustrates the proposed information flow from the input representation through the decoder to the Sports score. This explanation suggests that the trigger may behave differently depending on its location, such as appearing at the end of the article or within its body. However, the editing record alone does not verify that this mechanism was implemented or caused the observed predictions.

Figure 5. Exploratory evidence from two Qwen weight-editing cases. (a) The GPT-5.6 Sol editing record proposes a position-dependent trigger pathway. (b) A static comparison of DeepSeek-V4.1-Flash-edited weights locates changes across decoder components, concentrated in the final attention module. Neither observation alone establishes a causal backdoor circuit.
DeepSeek-V4.1-Flash: changes in parameter distribution. The exploratory DeepSeek-edited model shows weight modifications spread across several decoder tensors, with the largest concentration in the final attention module (Figure 5(b)). The static readout indicates how a modified intermediate representation might favor the Sports label even when its output-head row is not itself modified. Importantly, a target label ranking first under an isolated static readout does not imply that the model predicts that label on triggered test articles. Such a conclusion requires evaluation on complete inputs using the common protocol in § 3.
Finding 3 (Interpreting Edits). Editing records and static weight differences can suggest plausible explanations for a model’s changed behavior, including positional cues and concentrated parameter modifications. They do not, by themselves, establish trigger-specific causation. Our case studies therefore provide exploratory observations rather than a verified circuit-level mechanism or additional evidence of cross-editor attack success.
Conclusion and Future Work
In this paper, we presented an empirical security study of LLM-driven weight-space programming, investigating whether general-purpose LLMs can directly inject backdoors into neural networks through weight editing. We evaluated five LLM-based weight editors across three target architectures under a restricted editing protocol that excludes task-specific examples, target-model forward passes, and gradient computations during editing.
Our experiments reveal several important findings. First, LLM-driven weight editing can successfully introduce targeted backdoors across different neural architectures while largely preserving clean performance. Second, successful editing can require remarkably few parameter modifications, including only six parameters in the MLP and 0.02% of the weights in a fine-tuned Qwen model. Third, editing effectiveness varies substantially across LLMs and target architectures. While all five editors successfully backdoored the MLP, only GPT-6 Astra achieved the intended behavior across all three targets. These findings demonstrate that general-purpose LLMs can potentially be used to manipulate security-critical model behaviors through direct weight modifications, even without conventional training.
Nevertheless, our study has several limitations. Our evaluation focuses on three target architectures and specific trigger configurations, leaving the applicability to larger models, more complex tasks, and diverse triggers to future investigation. Moreover, our results characterize the available edited models rather than the success probability of individual editing requests, as complete editing-attempt records are not available. Our exploratory analysis of weight modifications also does not establish complete causal explanations for the observed behaviors. Further systematic evaluations are needed to better understand the reliability, generalizability, and underlying mechanisms of LLM-driven weight editing.
Beyond backdoor injection, our findings open several directions for investigating the broader security implications of weight-space programming. Future research could explore whether LLM-driven weight editing can be used to manipulate other security properties, such as weakening safety alignment, compromising privacy protections, reversing machine unlearning ing, or degrading model availability and reliability. Another important direction is to investigate how editing capabilities evolve with larger LLM editors, more complex target architectures, and different levels of access to model information and execution feedback. Such studies could help characterize the broader security capabilities and limitations of weight-space programming.
Finally, as LLMs become increasingly involved in automated model development, optimization, and repair, ensuring the security of their weight modifications will become essential. Future work should investigate effective approaches for auditing model weights, detecting malicious or unintended parameter changes, and verifying that legitimate model improvements do not compromise existing security properties. Our study provides an initial step toward understanding these emerging risks and highlights the need for security-aware evaluation and safeguards for LLM-driven weight-space programming.
References
- [1]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. doi: 10.1109/CVPR.2016.90.DOI
- [2]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.11929.
- [3]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–1901, 2020. URL https://arxiv.org/abs/2005.14165.
- [4]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/abs/2307.09288.
- [5]Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793–2806, 2020.arxiv.org/abs/2004.06660
- [6]Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors in deep neural networks. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2106.04690
- [7]Yanzhou Li, Tianlin Li, Kangjie Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. BadEdit: Backdooring large language models by model editing. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2403.13355.
- [8]Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. https://rikiyat.github.io/blog/weight-space-programming/, November 2026. Blog post.
- [9]OpenAI. Gpt-6 astra: A new generation of intelligence. https://openai.com/index/gpt-6-astra/, 2026. Accessed: 2026-11-11.
- [10]OpenAI. Previewing gpt-5.6 sol: a next-generation model. https://openai.com/index/gpt-5-6-sol/, 6 2026. Accessed: 2026-11-11.
- [11]Kimi Team. Kimi k3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653.
- [12]DeepSeek-AI. Deepseek-v4.1-flash: Pushing the limits of kv cache compression. https://arxiv.org/abs/2609.19969, 2026.
- [13]Tencent Hy Team. Introducing hy4. https://www.tencent.com/tencent-releases-and-open-sources/tencent-hy4-preview/, 2026.
- [14]Fanchao Qi, Yuchen Li, Zihan Zhang, Zhiyuan Liu, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 443–453, 2021.
- [15]Xinyang Xu, Yan Zheng, Shouling Ji, Wenxin Hou, Huaxin Li, and Chunming Wu. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2345–2358, 2024.arxiv.org/abs/2305.14710
- [16]Jiyang Qiu, Xinbei Ma, Zhuosheng Zhang, Hai Zhao, Yun Li, and Qianren Wang. MEGen: Generative backdoor into large language models via model editing. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11197–11214, 2025. doi: 10.18653/v1/2025.findings-acl.584.DOI
- [17]Rui Yin, Tianxu Han, Naen Xu, Changjiang Li, Ping He, Chunyi Zhou, Jun Wang, Naen Xu, Tianyu Du, Jinbao Li, and Shouling Ji. Compiling activation steering into weights via null-space constraints for stealthy backdoors. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26228–26245, 2026. doi: 10.18653/v1/2026.acl-long.1206.DOI
- [18]Yuchen Zeng, Giorgio Severi, Blake Bullwinkel, Eric Horvitz, and Dimitris Papailiopoulos. Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents. Microsoft Research preprint, November 2026. URL https://www.microsoft.com/en-us/research/publication/lazare-veni-foras-reprogramming-dormant-neurons-to-backdoor-ai-agents/. Published online November 2026.arxiv.org/abs/2611.15118
- [19]Nicola De Cao, Michael Schlichtkrull, Wilker Aziz, and Andreas Vlachos. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491–6506, 2021.
- [20]Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.11309
- [21]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372, 2022.arxiv.org/abs/2202.05262
- [22]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.07229
- [23]Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
- [24]Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. Mnist handwritten digit database, 2010. URL http://yann.lecun.com/exdb/mnist/.
- [25]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.