betaFeedback

WorksComputer Science

KernelArmor: End-to-End Agentic System for Establishing Memory and Thread Safety for CUDA

Abstract

Optimized CUDA kernels can pass testing while retaining subtle memory and concurrency bugs. We present KernelArmor, an agentic framework that combines LLM-generated annotations with an extended SMT verifier to establish memory and thread safety under explicit contracts. KernelArmor can reason about kernels with advanced CUDA features, such as tensor cores, packed accesses, warp synchronization, asynchronous copies, and library calls, returning source-level diagnostics to guide verification and repair. Across 740 instances containing 1,370 production-grade CUDA kernels, it reports potential bugs in 78.6% of instances and establishes safety for 11.4%, outperforming the evaluated state-of-the-art tools in detection coverage, identifying 100% of cuCatch memory-safety bugs and all in-scope cuHadron races. Integrated into an agentic optimization loop, KernelArmor produces verified kernels in all 77 trajectories while retaining 99% of the unconstrained geometric-mean speedup.

Introduction

Over the past few years, agentic frameworks ([16, 11, 4] (Wang et al. 2026)) have become ubiquitous in code generation, optimization, and repair, yet establishing trust in their outputs remains challenging because they provide no inherent guarantees of correctness. This challenge is further amplified by the growing volume of agent-generated code, which has shifted the bottleneck from code production to manual review ([7]), necessitating code validation approaches that are scalable, automated, and rigorous.

A practical approach for validating agent-generated code is testing; however, its effectiveness largely depends on test coverage, and programs that pass the available tests may still contain defects ((Liu et al., 2023)). For instance, widely used tools such as NVIDIA Compute Sanitizer ((NVI, 2022)) can detect memory access errors and shared-memory races during execution, but their coverage is limited by the inputs exercised and the error classes supported. In general, this reflects a broader limitation of testing approaches based on dynamic analysis: a clean run does not establish the absence of such errors across untested inputs and execution paths ([1]).

Formal verification ([2]) tackles this issue by proving specified properties across all behaviors permitted by a program’s model and assumptions, thereby providing stronger assurances. However, adapting formal verification techniques at scale remains challenging since constructing specifications, generating program invariants, and writing proof annotations often require substantial expertise and manual effort ((Poesia et al., 2024)). This process can also be error-prone and often requires iterative refinement. This effort is further complicated by proof instability in SMT-based formal verifiers, where small changes can cause substantial slowdowns or verification failures ((Zhou et al., 2023)).

Recent work ([6, 5, 10, 15] (Yang et al. 2025; Tihanyi et al. 2023; Martínez & Sorensen 2026)) has extensively explored the use of AI to automate formal verification techniques. These approaches leverage large language models (LLMs) to propose proof artifacts, which are then checked by existing formal verifiers and proof frameworks ([19, 9, 3, 14] (Martínez et al. 2026)). The verifier feedback is used to guide subsequent revisions, enabling agents to iteratively refine annotations, proofs, or program implementations.

However, applying these techniques to optimized CUDA code raises two challenges. First, existing GPU verifiers ([1] (Li et al. 2012; Monteiro et al. 2018a; Cugnonbire et al. 2021)) support only a small subset of CUDA. Optimized kernels, whether written by experts or by agents, routinely rely on features outside that subset, such as warp-level shuffles, asynchronous memory copies (cp.async), inline PTX, and tensor-core instructions. A verifier that cannot model these features must either reject the fastest kernels or leave them unchecked. Second, placing a verifier inside an agentic optimization loop changes what it must report. A verdict helps the agent only if it points to the failing safety obligation in the source and separates three cases that call for different actions: a likely defect the agent should fix, a proof the verifier could not complete, and a construct outside the verifier’s model. Conflating these cases either lets unsafe code through or steers the agent away from correct optimizations. Addressing both challenges requires agentic verification that covers the CUDA features optimized kernels actually use and returns explicit, checkable guarantees and feedback the agent can act on.

We present KernelArmor, an agentic framework for automated memory and thread-safety verification of optimized CUDA kernels. The framework uses in-context learning and reusable knowledge from successful verification attempts to generate and iteratively refine contracts, invariants, and permission annotations for VerCors, an SMT-based deductive verifier for parallel programs (Blom et al., [3]). We extend VerCors’ CUDA verification infrastructure to reason about advanced constructs, including packed memory accesses, warp-level synchronization, asynchronous copies, and tensor-core operations, while making their modeling assumptions explicit. When verification fails, KernelArmor provides fine-grained, source-grounded feedback on unresolved safety obligations and potential defects, guiding targeted revisions to annotations or implementations within agentic code-generation loops.

We evaluated KernelArmor on three benchmark suites (∼1,400 kernels) containing production-grade CUDA kernels, identifying potential safety bugs in 78.6% of instances, and establishing memory and thread safety for 11.4%. Compared with state-of-the-art memory and thread safety tools, KernelArmor achieved 100% detection on the cuCatch memory safety suite (Tarek Ibn Ziad et al., 2023) and all in-scope data races in the cuHadron suite (Stephenson et al., 2026), with no false positives. Finally, integrating KernelArmor into an agentic optimization loop yielded verified kernels in all 77 trajectories while retaining 99% of the geometric-mean speedup achieved without verification.

Overall, this paper makes the following contributions:

  • We present KernelArmor, an end-to-end agentic system that combines reusable verification knowledge, iterative verifier feedback, and an annotation-integrity guard to establish memory and thread safety in production-grade CUDA programs.

  • We extend the CUDA frontend, memory model, and synchronization semantics of VerCors (Blom et al., [3]), an SMT-based verifier, to support advanced CUDA features, including asynchronous copies, packed accesses, warp-scoped synchronization, dynamic shared memory, and library calls with explicit safety contracts.

  • We outperform state-of-the-art memory and race detectors and enable verified agentic optimization in the loop while retaining 99% of the unconstrained geometric-mean speedup.

Background & Motivation

Modern CUDA kernels can contain subtle memory and concurrency bugs that escape off-the-shelf safety tools. In particular, input-dependent global-memory races and accesses beyond a tensor view’s logical bounds can evade tools whose checks cover only selected memory spaces or allocation-level bounds.

Safety Issues in Modern CUDA Kernels

Listing 1 shows an AI-optimized kernel with a global-memory race when cache_position = [0,0], batch = 1, and seq_len = max_seq_len = 2: corresponding threads in blocks 0 and 1 access the same packed BF16 cache entry. If block 0 reads and clears the entry before block 1 loads it, block 1 receives zeros, producing an incorrect gradient instead of the original cache value that the reference gathers for both sequence positions.

__global__ __launch_bounds__(256, 2)
void fused_rope_cache_backward(...) {
// ... compute parameters ...
const int64_t pos = cache_position[seq];
const int64_t cache_lo =
    ((static_cast<int64_t>(batch) * kHeads + head)
      * max_seq_len + pos) * kDim + pair;
// Blocks alias this entry when positions repeat.
const __nv_bfloat162 v_lo_b =
  *reinterpret_cast<__nv_bfloat162*>(
    grad_value_cache + cache_lo);
*reinterpret_cast<__nv_bfloat162*>(
    grad_value_states + state_lo) = v_lo_b;
// This clear can race with another block's load.
const __nv_bfloat162 zero =
    __floats2bfloat162_rn(0.0f, 0.0f);
*reinterpret_cast<__nv_bfloat162*>(
    grad_value_cache + cache_lo) = zero;
// ...rest of the kernel...
}

caption: Listing 1. Duplicate cache positions causing a global-memory race.

KernelArmor exposes the overlapping addresses by inserting the following checked separation assertion immediately before the first cache load:

/*@ assert
    \pointer_block_offset(grad_value_cache) + cache_lo
    != \pointer_block_offset(grad_value_cache)
       + (head * max_seq_len
          + cache_position[1 - blockIdx.x])
          * kDim + pair;
@*/

The left-hand side denotes the current thread’s cache offset; the right-hand side denotes the corresponding thread’s offset in the other block. For thread 0 in either block, both sides evaluate to zero, so VerCors rejects the assertion with assertFailed:false. Together with the subsequent non-atomic stores and the absence of inter-block synchronization, this establishes a global-memory race. These accesses are aligned and in bounds but their global-memory read–write conflict is not currently detected by off-the-shelf tools like Compute Sanitizer.

__global__ void bias_gelu_bth_vec_kernel(..) {
const uint4* bias_vec =
    reinterpret_cast<const uint4*>(bias);

for (int64_t idx = int64_t(blockIdx.x) * blockDim.x
        + threadIdx.x;
    idx < total_vecs; idx += stride) {
    int vec_idx = int(idx % kDModelVecs);
    uint4 v = reinterpret_cast<uint4*>(x)[idx];
    // Reads eight BF16 elements.
    uint4 b_v = bias_vec[vec_idx];
    // ... packed BF16 GELU ...
}

Listing 2. Vectorized bias loads for packed BF16 GELU.

Listing 2 shows an unchecked bias extent in a fused BF16 bias/GELU kernel, where each uint4 load consumes eight BF16 elements. For a one-element bias view, the first load reads seven elements beyond the tensor’s logical extent, allowing unrelated values from the backing allocation to enter the computation. The relevant VerCors annotation, which flags this issue, is:

//@ assert 8 * vec_idx + 7
//           < \pointer_length(bias);

This assertion requires all eight elements to lie within the view and fails for this input. If the view resides within a larger initialized allocation, the accesses remain within that allocation, so allocation checks performed by most off-the-shelf tools cannot detect the logical overread without additional region metadata.

VerCors Verification Tool

VerCors ([3]) is a deductive verifier for concurrent and parallel programs that can establish memory safety, data-race freedom, and functional correctness. It uses permission-based separation logic to specify program contracts, loop invariants, and memory-access permissions. VerCors translates annotated programs into Viper’s intermediate verification language ((Müller et al., 2016)) and checks them using Silicon, Viper’s symbolic-execution backend, which relies on the Z3 SMT solver ([12]). A program contract consists of preconditions, which state the properties and permissions required at entry, and postconditions, which describe those guaranteed on return when the preconditions hold. For GPU kernels, permissions describe per-thread memory ownership and its transfer at synchronization points. In KernelArmor, VerCors uses the agent’s annotations to prove memory and thread safety under explicit preconditions and the supported CUDA model.

KernelArmor: End-to-End Agentic System for Establishing Memory & Thread Safety

KernelArmor combines LLM-guided annotation generation with formal verification to assess the memory and thread safety of CUDA programs.

KernelArmor’s Agentic Verification Workflow

KernelArmor accepts CUDA code, along with any available host code, either as a .cu source file or within an agent-generated .json file. The input program is first scanned to ensure that all constructs are supported by KernelArmor, then preprocessed by the workflow’s frontend into a representation compatible with VerCors. KernelArmor then generates safety annotations, after which an annotation-integrity guard ensures that only annotations have been added. The annotated CUDA program is then verified by VerCors, and if required, the annotations are iteratively refined using verifier feedback within a bounded number of attempts. Figure 1 illustrates this workflow. The workflow concludes with one of three outcomes: verified, potential bug, or inconclusive. We will now describe each stage of the agentic workflow in detail.

KernelArmor’s memory- and thread-safety verification workflow

Figure 1. KernelArmor’s memory- and thread-safety verification workflow. CUDA inputs are preprocessed for LLM-generated annotation insertion, while an integrity guard ensures byte-for-byte preservation of the preprocessed code after removing annotations. VerCors checks the annotated program and guides repair for up to KK turns, yielding a verified, potential-bug, or inconclusive outcome.

Scanning & Preprocessing: KernelArmor’s deterministic (non-agentic) frontend parser, cuda2vercors, first scans the input for unsupported constructs, then normalizes CUDA syntax and lowers supported operations, such as inline PTX loads and stores, to calls with explicit safety contracts. If any unsupported constructs are detected, the workflow stops and reports the need for additional preprocessing support, giving users the option to extend KernelArmor offline before rerunning verification.

Knowledge-Guided Annotation Generation: KernelArmor’s LLM proposes preconditions, postconditions, loop invariants, and synchronization annotations for the preprocessed program. The agent also examines host code (if available) and extracts launch metadata to relate kernel preconditions to caller-side allocations, input dimensions, and launch configurations. These preconditions capture the bounds, alignment, and memory permissions required for safe execution, while postconditions specify the permissions preserved on completion. Loop invariants track access permissions and index bounds across iterations, and synchronization annotations describe permission transfers between threads at barriers.

To support efficient generation of safety annotations, KernelArmor’s annotation agent is equipped with a reusable CUDA Verification Handbook, which distills VerCors specification concepts into practical annotation patterns, examples, and diagnostic repair strategies. The goal here is to reuse established verification strategies rather than derive them from scratch for each program. Handbook material is included in the agent’s prompt to support in-context learning, enabling the agent to adapt these strategies to the program’s memory accesses, thread ownership, and synchronization requirements. The handbook is updated only when a candidate that demonstrates a novel reusable annotation strategy not already in the handbook is successfully verified.

Integrity Guardrail: To ensure that LLM annotations leave the program unchanged, KernelArmor performs an integrity check which removes annotation comments from the candidate and preprocessed baseline, then checks that the resulting code is byte-for-byte identical.

VerCors Verification: Finally, the annotated program is checked by VerCors for memory and thread safety under the proposed contracts. On failure, the verifier diagnostic is used to guide annotation repair, and each revised proposal passes through the integrity guard before re-verification. This regeneration-verification loop is bounded by up to KK turns, and concludes with one of three outcomes: verified if all required checks pass, potential bug if validated safety evidence supports a finding, or inconclusive otherwise.

Extending VerCors’ CUDA Frontend, Memory Model and Synchronization Semantics

We describe KernelArmor’s extensions to VerCors’ frontend, verification encodings, and permission model for optimized CUDA kernels.

Specializing CUDA constructs: CUDA kernels often use templates and compile-time constants to parameterize storage sizes and execution paths. KernelArmor first extends VerCors’ CUDA frontend to specialize supported templates and process associated constants, casts, and device helpers. For example, instantiating reduce<128> makes parameter-dependent array bounds and compile-time branch choices explicit before verification. Unsupported constructs are reported as frontend errors.

Tracking asynchronous-copy completion: Asynchronous copy (cp.async) transfers data from off-chip global memory to on-chip shared memory without blocking the issuing thread; an explicit wait establishes completion before that thread consumes the copied data. KernelArmor adds a pending-copy resource that holds the required buffer permissions until matching completion. Premature destination access or source modification then fails a permission check. Completing the wrong copy group or buffer pair fails to match the pending resource, preventing an unrelated wait from incorrectly authorizing buffer reuse.

Checking warp-scoped synchronization: Warp-scoped synchronization lets selected lanes within the same warp wait for one another before proceeding. KernelArmor extends VerCors with barriers restricted to the lanes selected by __syncwarp (mask) within the caller’s warp. Participating lanes must agree on the mask, and permission transfers remain within that group. An access relying on synchronization with another warp therefore fails its permission check. Supported shuffles similarly check lane participation and legal widths, leaving the exchanged values abstract.

Modeling shared storage and thread-local state: Threads within a CUDA block share dynamic shared memory, but each thread receives its own copy of kernel arguments. KernelArmor derives shared-array bounds from the launch-provided byte count and models modified or address-taken arguments as thread-local copies. Thus, 4n4n shared bytes provide space for nn floats, while input += stride updates only the calling thread’s cursor. Invalid allocation sizes fail size checks, and dereferences still require valid bounds and permissions.

Preserving pointer arithmetic: Pointer arithmetic involves advancing a given pointer in units of the pointed-to type. KernelArmor extends VerCors’ cast encoding to preserve the corresponding byte offsets: advancing a BF16 pointer by one moves two bytes, whereas advancing a uint4* by one moves sixteen bytes. Accesses through the cast pointer must still satisfy bounds and permission checks, preventing a change of type from hiding an invalid access to the underlying storage.

Checking packed accesses: Packed accesses read or write multiple adjacent elements through a single wider memory operation. KernelArmor lowers supported packed transactions into scalar accesses while retaining their original alignment requirements. For instance, a sixteen-byte aligned uint4 load over BF16 storage requires eight readable elements and a sixteen-byte-aligned address. Insufficient bounds, missing permissions, or misalignment therefore remain proof failures after scalarization.

Checking block-barrier participation and ownership: A block barrier lets threads within a block wait for one another before accessing data produced by other threads. KernelArmor extends barrier encodings to check permission transfers and adds bounded convergence checks for supported device-helper expansions. For example, __syncthreads() may redistribute exclusive cell ownership into read shares, but reaching it only when threadIdx.x == 0 cannot establish participation across a multi-thread block. Subsequent accesses fail if the barrier contract does not provide the required permissions.

Checking library-call requirements: KernelArmor adds typed safety contracts for supported cuBLAS, cuBLASLt, CUTLASS, WMMA, and PTX operations. These contracts expose buffer extents, layouts, alignment, and access permissions. Undersized buffers, missing permissions, or prohibited output overlap fail the call’s preconditions. In the current version, the contract assumes the correctness of the external library implementation.

Improving backend permission reasoning: KernelArmor patches Silicon, Viper’s symbolic-execution verification backend (Müller et al., 2016), to improve permission selection and predicate matching while retaining the required permission checks.

Other extensions preserve supported atomic return values and exclusive slot ownership for unit-increment allocation counters, give bounded proof views for selected cluster accesses without changing block-barrier scopes, and preserve or synthesize quantifier triggers for ownership expressions.

Implementation

We built our extended VerCors fork from source1 and used the NeMo Agent Toolkit (NVIDIA, 2026) to orchestrate KernelArmor’s workflow. Our extensions comprise approximately 20.6 KLOC of net implementation changes, excluding tests, supporting CUDA 13.1. The preprocessing frontend and annotation agent are implemented in Python 3.13.12, while the VerCors extensions are primarily implemented in Scala 2.13.12.

Evaluation

We evaluate KernelArmor along two dimensions: its ability to reason about memory and thread safety in CUDA code produced by agentic and non-agentic approaches, and the impact of integrating it into an agentic code-generation loop. Our experiments address the following research questions:

  • RQ1: To what extent can KernelArmor reason about memory and thread safety in CUDA kernels that use advanced language and hardware features? What classes of memory and concurrency bugs can it identify?

  • RQ2: What are the overheads of KernelArmor in terms of runtime, token usage, and estimated inference costs?

  • RQ3: How does KernelArmor compare with other state-of-the-art tools that detect CUDA memory and thread safety issues?

  • RQ4: To what extent does the verification handbook improve KernelArmor’s verification success rate?

  • RQ5: What is the impact of integrating KernelArmor into an agentic code-generation loop?

Experimental Setup: We ran KernelArmor on an Ubuntu server equipped with a 16-core Intel Xeon Gold 6444Y processor (32 hardware threads) and 247 GiB of system memory. We used GPT-6 Astra for all of the annotation generation and refinement experiments.

Establishing Memory & Thread Safety in Production-Grade CUDA Kernels

We evaluate KernelArmor on three suites totaling 740 instances and 1,370 CUDA kernels. In our experiments, an instance denotes an individual implementation of a benchmark workload, including its associated host code and potentially multiple CUDA kernels.

Evaluation Suite. We first evaluate KernelArmor on SQL-ExecBench (Lin et al., 2026), with 64 instances across 46 problems (235 kernels), extracted from production and emerging AI workloads; an Optimized CUDA Suite, with 28 production-grade implementations (70 kernels) covering attention, normalization, convolution, and matrix operations; and KernelBench (Ouyang et al., 2025), with 648 generated instances across 1,065 kernels. Collectively, these implementations exercise advanced CUDA features including packed vector and alignment-sensitive BF16 accesses, inline PTX, shared-memory tiling and synchronization, warp-level reductions, atomics, template specialization, WMMA Tensor Core operations, and asynchronous global-to-shared transfers (cp.async). Their execution paths also include FP8/NVFP4 processing, CUDA Graphs, thread-block cluster launches, and cuBLAS/cuBLASLt calls.

Verification Outcomes. Table 1 illustrates KernelArmor’s verification outcomes across all three evaluation suites. Overall, KernelArmor is able to fully establish memory and thread safety for 84 instances, reports potential findings in 582, and is inconclusive on 74. The high rate of potential bugs highlights the prevalence of safety issues in agent-generated kernels. In each potential bug case, KernelArmor returns a counterexample that triggers the bug. One observation is that these potential bugs also occur in relatively simple workloads (82% in activation workloads). These implementations rely on input-dependent indexing and launch-grid arithmetic, which often lead to overflows and alignment mismatches. Matrix and fused-GEMM workloads account for 75% of KernelBench’s inconclusive outcomes, highlighting some of the library-modeling limitations in KernelArmor.

DomainInstancesVerified*Potential BugInconclusive
SOL-ExecBench
L1 (Single Operations)256145
L2 (Fused Operations)202144
Quantization3111
FlashInfer-Bench162104
Subtotal64113914
Optimized CUDA
WorkloadInstancesVerifiedPotential BugInconclusive
Attention / transformers9900
Norm. / modulation8530
Convolution2200
MLP / MoE4400
State-space models3300
GEMM1100
Quantum simulation1100
Subtotal282530
KernelBench
WorkloadInstancesVerifiedPotential BugInconclusive
GEMM / GEMV10386431
Activations11219921
Norm. / softmax167615110
Pooling540540
Reductions / scans1089954
Fused GEMM10468414
Subtotal6484854060

Table 1. KernelArmor verification outcomes across 740 instances containing 1,370 CUDA kernel definitions. Verified* denotes successful verification under stated kernel/library preconditions.

CUDA features represented in our evaluation suites

Figure 2. CUDA features represented in our evaluation suites. Bars count instances containing each feature across all three suites.

Classification of Bugs. A deeper dive into the 582 potential bug instances identified by KernelArmor reveals that integer overflow (311; 53.4%) and misaligned accesses (175; 30.1%) dominate, together accounting for 83.5% of primary findings (Figure 3). The remaining findings concern null accesses, invalid pointer formations and memory races. These overflow and alignment issues mostly occur due to unchecked arithmetic such as index calculations, dimension products leading to overflow before division/widening, or insufficient packed-access assumptions whose alignment requirements are not guaranteed by the underlying storage. This highlights a recurring gap between the inputs accepted by wrappers and the assumptions required by optimized implementations in modern agentic code-generation systems.

KernelArmor bug classification across SOL-ExecBench, Optimized CUDA, and KernelBench suites

Figure 3. KernelArmor bug classification across SOL-ExecBench, Optimized CUDA, and KernelBench suites. Each of the 582 instances with potential findings contributes to one primary category.

__global__ void maxpool2d_kernel(..) {
const int64_t total = (int64_t)B * C * OH * OW;
// ...
const float* xc = x + ((int64_t)(b * C + c)) * H * W
}

Listing 3. 64-bit cast applied too late: MaxPool2D

For instance, Listing 3 shows a subtle example where a 64-bit cast masks an overflow risk: the pointer-offset expression b∗C+cb * C + c is evaluated in 32-bit arithmetic before conversion, potentially overflowing for large tensor dimensions. For this finding, KernelArmor reports the configuration b=65536b = 65536, C=32768C = 32768, and c=0c = 0, for which the intermediate product exceeds the signed 32-bit range.

KernelArmor’s Overheads & Cost Analysis

Timing Overheads. Table 2 breaks down KernelArmor’s analysis overhead, showing that annotation and verification dominate preprocessing. Verification accounts for more than half of their combined time—57% on SOL-ExecBench and 52% on KernelBench. This is expected because each annotation-refinement iteration requires translating the annotated CUDA and its obligations through VerCors.

Avg. preprocessing (sec)Avg. verification (sec)Avg. annotation (sec)Avg. turns (limit = 12)
Per kernelPer InstancePer kernelPer InstancePer kernelPer InstancePer kernelPer Instance
SOL-ExecBench0.782.76111.12393.3882.21291.011.113.92
KernelBench0.220.34456.56740.16424.81688.683.305.35

Table 2. Timing analysis of different phases of KernelArmor’s agentic workflow

Note that KernelArmor requires more turns on KernelBench cases than on SOL-ExecBench (5.35 versus 3.92), despite the latter having approximately 4.9×4.9 \times as many LOC per case. This is largely driven by 60 unproved KernelBench cases that exhaust the 12-turn repair budget.

Tokens & Estimated Cost. Table 3 estimates an average model cost of $3.58 per instance and $2.26 per kernel, including recorded annotation and repair requests. This suggests relatively modest inference costs for establishing safety in production-grade kernels.

Avg tokens/instanceAvg tokens/kernelAvg price/instanceAvg price/kernel
≈ 290,114≈ 183,011≈ $3.58 | ≈ $2.26

Table 3. KernelArmor’s estimated average token usage and model cost, based on estimated input tokens, recorded output tokens, and GPT-6 Astra Standard API pricing.

Comparison with state-of-the-art tools

To evaluate KernelArmor’s effectiveness in detecting memory and thread safety violations, we compare it against state-of-the-art tools on previously published benchmark suites.

Comparison with State-of-the-art Memory Safety Tools: Table 4 compares the memory safety coverage provided by KernelArmor against four state-of-the-art GPU dynamic memory safety tools including NVIDIA’s Compute Sanitizer memcheck ((NVI, 2022)), GMOD ((Di et al., 2018)), GPUShield ([20]), and cuCatch ((Tarek Ibn Ziad et al., 2023)) using the cuCatch benchmark suite ((Tarek Ibn Ziad et al., 2023)). Our experiments show that KernelArmor identified 100% of the memory safety bugs in the benchmark suite compared to 35.7%, 14.3%, 42.9%, and 71.4% for Compute Sanitizer, GMOD, GPUShield, and cuCatch respectively. The lower coverage rates in these tools reflect limitations in detecting different classes of memory errors: Compute Sanitizer missed non-adjacent buffer overflows, while cuCatch, despite addressing some of these cases, still missed intra-allocation out-of-bounds accesses. On the other hand, KernelArmor not only correctly identified all memory safety errors in the benchmark but also identified a bug in the benchmark suite that makes a potential out-of-bounds access unreachable.

Table 4. KernelArmor’s memory-safety comparison with state-of-the-art memory safety tools on cuCatch’s benchmark suite. KernelArmor achieved 100% detection rate; * indicates we used these tools’ published numbers (Tarek Ibn Ziad et al., 2023).

CategoryDescriptionCompute SanitizerGMOD*GPUShield*cuCatchKernelArmor
Temporal safetyLifetime and deallocation errors.8/124/124/1210/1212/12
Spatial heapOOB accesses to heap objects.4/164/168/168/1616/16
Spatial stackOOB accesses to stack objects.4/160/1612/1612/1616/16
Spatial shared memoryOOB accesses to shared objects.4/120/120/1210/1212/12
Detection rate20/56 (35.7%)8/56 (14.3%)24/56 (42.9%)40/56 (71.4%)56/56 (100%)

Table 4.

Comparison with State-of-the-art Race Detectors: We compared KernelArmor’s race detection coverage against state-of-the-art dynamic race detectors for GPUs, including iGUARD ([18]), NVIDIA’s Compute Sanitizer Racecheck ((NVI, 2022)), and the more recent SuperCollider tool ((Stephenson et al., 2026)). Table 5 shows the results of our study on cuHadron ((Stephenson et al., 2026)), a GPU race detection benchmark designed to evaluate the coverage of GPU race detection tools on modern GPU features, including asynchronous memory copies.

Table 5. KernelArmor compared with state-of-the-art race detectors on cuHadron suite. In-scope races occur between threads of a single kernel launch, including asynchronous copies issued by that kernel; out-of-scope races involve another kernel, a host-issued copy, another GPU or the CPU. Each cell gives planted races detected out of races in the category. ▲ counts tests where KernelArmor detected a different, non-race, memory safety error (signed integer overflow) in the benchmark. For the false-positive controls, the cell gives race-free tests incorrectly flagged. * indicates we used these tools’ published numbers (Stephenson et al., 2026).

ScopeCategoryDescriptioniGUARD*Compute SanitizerSuperColliderKernelArmor
In scopeInter-subwarpDivergent threads within a warp0/22/22/22/2
In scopeAsync memcpycp.async(.bulk) accesses0/95/98/99/9
In scopeCGADistributed SMEM Accesses0/22/22/22/2
In scopeInter-waveThread blocks in different waves1/10/11/11/1
In scopeSubtotal1/149/1413/1414/14
Out of scopeInter-kernelKernels on different streams0/20/22/20/2
Out of scopecudaMemcpyAsyncAsync copies vs. kernels0/20/21/20/2 (+1 ▲)
Out of scopeInter-GPUMultiple GPUs on unified memory0/20/22/20/2
Out of scopeCPU–GPUCPU and GPU on unified memory0/30/32/30/3 (+2 ▲)
Out of scopeSubtotal0/90/97/90/9 (+3 ▲)
Total races1/239/2320/2314/23 (+3 ▲)
False positives (race-free: threads access different bytes of one word)2/40/40/40/4

Table 5.

Unlike the dynamic detectors, which execute the benchmark programs to observe races, KernelArmor analyzes kernels statically. It detects races in all 14 tests within its kernel-level analysis scope and correctly verifies all four race-free control tests. The remaining nine race tests involve interactions across streams, CPUs and GPUs, or multiple GPUs, which exceed its current analysis scope.2 Nevertheless, KernelArmor identifies additional potential memory-safety issues: signed integer overflows in three of these tests, demonstrating useful diagnostic coverage even where the intended race cannot be analyzed.

Across all 23 race-positive tests, KernelArmor shows 61% coverage on the entire test suite, as compared to 4%, 39%, and 87% for iGUARD, Compute Sanitizer and SuperCollider respectively, and 100% coverage on in-scope tests with zero false positives. Within its scope, KernelArmor detects all 14 races, including those involving asynchronous memory copies, without executing the benchmark programs, and produces no false positives on the four race-free controls.

Comparison with other agentic methods: Finally, we compare KernelArmor with ProofWright ([6]), a closely related agentic formal verification framework that also establishes memory and thread safety for CUDA kernels. We evaluated KernelArmor on the 100 Claude-4-Sonnet legacy implementations of KernelBench Level 1 problems used in ProofWright’s evaluation. These implementations primarily rely on simple CUDA constructs, including shared memory accesses, affine thread and block indexing and block-level synchronization.

Table 6 summarizes the results of our study. On the legacy KernelBench corpus, KernelArmor establishes memory- and thread-safety guarantees for 89% of the kernels, compared with ProofWright’s 74%. Additionally, KernelArmor identifies 10 potential overflow bugs supported by concrete counter-examples. In contrast, none of the 1,370 kernels in KernelArmor’s expanded benchmark corpus is supported by ProofWright, as they rely on advanced CUDA features that fall outside its supported verification scope.

Benchmark corpusToolVerifiedPotential bugInconclusiveUnsupported
Legacy KernelBench Problems (100 kernels)ProofWright74%N/A24%2%
KernelArmor89%10%1%0%
SOL-ExecBench + KernelBench + optimized CUDA (1,370 kernels)ProofWright0%N/A0%100%
KernelArmor11.4%78.6%6.6%3.4%

Table 6. Verification outcomes for ProofWright and KernelArmor on legacy and expanded CUDA benchmark suites.

Ablation Study: Impact of KernelArmor’s Verification Handbook

We evaluate the contribution of KernelArmor’s Verification Handbook using six randomly selected cases from our benchmark suites. For each case, we compare a handbook-enabled run against an ablated run that removes the handbook from the model’s reference context. Both conditions use the same source input, model, verifier settings, and budget of 16 model turns.

Table 7 shows that KernelArmor verifies three of six cases with the handbook, compared with two without it. Among cases verified in both conditions, removing the handbook increases model turns from 6 to 12 for Irregular Matmul and from 7 to 8 for ArgminReduction. These experiments show that the Verification Handbook is essential both for convergence and for correctness of KernelArmor.

TaskWith handbook OutcomeWith handbook TurnsWithout handbook OutcomeWithout handbook Turns
LatentKVPotential bug3Exhausted16
SOL RMSNorm H1536Potential bug7Potential bug8
KB Upper Triangular MatmulPotential bug3Exhausted16
KB Irregular MatmulVerified6Verified12
KB MatmulLargeKVerified7Potential bug13
KB ArgminReductionVerified7Verified8
Verified cases3/62/6

Table 7. Verification handbook ablation study for KernelArmor on six randomly selected kernels, with a limit of 16 turns per kernel.

Integrating KernelArmor into an Agentic Optimization Loop

Loop setup: We place KernelArmor inside an agentic CUDA optimization loop (Figure 4) in which a coding agent (Claude Opus 4.8) replaces a PyTorch operator with a CUDA kernel and improves it over eight rounds. We compare two arms on 11 KernelBench (Ouyang et al., 2025) tasks (Table 8), with seven independent runs (trajectories) per task and arm. Each trajectory starts from scratch, and trajectories differ only through the agent’s sampling. The unconstrained arm keeps the fastest correct kernel. In the gated arm, every correct candidate that would become the new best is first verified by KernelArmor, and only a verified kernel is accepted. The agent receives KernelArmor’s verdict as feedback: the failing obligation and its source location for a potential bug, the fact that no proof was found for an inconclusive result, and the verifier’s explanation when a construct is unsupported. The gated prompt asks for kernels that are safe on every input the host wrapper accepts. Each trajectory has a 6-hour budget. Kernels run on an NVIDIA A40 (48 GB, driver 590.48.01, CUDA 13.1) with clocks locked at 1410 MHz. We select each trajectory’s best kernel by its in-loop time and report its speedup over the PyTorch reference from a separate benchmark run.

The gated optimization loop

Figure 4. The gated optimization loop. Each round, the agent’s kernel is compiled, checked against the PyTorch reference, and timed. A correct kernel that would become the new best is accepted only if KernelArmor verifies it, and the verdict returns to the agent as feedback. The unconstrained arm accepts it directly.

TaskUnconstr.GatedRatioSafe
ReLU1.04×1.04 \times1.04×1.04 \times0.996/7
LogSoftmax1.28×1.28 \times1.27×1.27 \times0.900/7
GroupNorm1.00×1.00 \times1.00×1.00 \times0.990/7
LayerNorm8.31×8.31 \times8.28×8.28 \times0.980/7
MaxPool 2D2.81×2.81 \times2.62×2.62 \times0.890/7
Matrix-vector1.54×1.54 \times1.58×1.58 \times1.000/7
Argmax0.91×0.91 \times0.89×0.89 \times0.960/7
GELU9.30×9.30 \times9.30×9.30 \times1.005/7
cumsum1.18×1.18 \times1.17×1.17 \times0.990/7
Gemm+GroupNorm+Hardtanh18.86×18.86 \times21.50×21.50 \times1.131/7
Gemm+Add+ReLU21.16×21.16 \times21.53×21.53 \times1.060/7
All (77 trajectories)0.9912/77

Table 8. Median best-kernel speedup over PyTorch across seven trajectories per arm. Ratio is the geometric mean of gated over unconstrained speedups. Safe counts unconstrained best kernels that KernelArmor verifies. All gated best kernels are verified.

Speed and safety: Across all 77 trajectories, the gated arm’s geometric-mean speedup is 0.99× that of the unconstrained arm, and every accepted gated kernel is verified (Table 8). Per task, the gated median is within 7% of the unconstrained median, and the largest gap is MaxPool. The unconstrained arm, by contrast, ends with a verified kernel in only 12 of its 77 trajectories, all on ReLU, GELU, and one Gemm+GroupNorm+Hardtanh trajectory. The findings on its best kernels are signed-integer overflow at sizes near

INT_MAX⁡\operatorname{INT\_MAX} (for example, N∗CN*C in GroupNorm and launch-grid ceiling divisions in MaxPool), float4⁡\operatorname{float4} misalignment on offset tensor views, missing output-size checks, pointers formed outside their allocation, and an out-of-bounds shared-memory write at batch size 257 in LayerNorm. None of them triggers on the benchmark shapes, so testing on those shapes cannot expose them.

Post-hoc filtering: A fair question is whether it suffices to run the unconstrained loop and keep only its verified kernels afterwards. Doing so leaves a kernel in just 31 of 77 trajectories, and these are often slow early-round kernels: 0.16× for cumsum and 1.74× for LayerNorm against 8.3× in the loop. With KernelArmor in the loop, the agent instead fixes the reported defects and keeps optimizing. The fixes we observe, such as 64-bit or bounded ceiling divisions, a scalar fallback for misaligned float4⁡\operatorname{float4} views, and an overlap check between input and output, cost no measurable speed. For example, KernelArmor rejected a 6.4× Gemm+GroupNorm+Hardtanh kernel because B∗GB * G overflows a signed integer at B=227B = 2^{27}. The agent’s next round kept the cuBLAS-based design and was verified. Verification therefore pays off inside the loop, where its feedback steers the search, not after it.

Verifier capacity: Gating is not free, however. The gated arm loses speed mainly when KernelArmor cannot decide a fast candidate. In 13 of the 77 trajectories, the gate rejected a candidate more than 1% faster than the kernel the trajectory ended with: in 7 of them the fastest such candidate was inconclusive, in 4 it had a potential bug, and in 2 it used an unsupported library configuration. Verification also costs time: a single KernelArmor run takes a median of 12 minutes, and the median gated trajectory spends 1.45 h in KernelArmor across all its rounds (at most 5.3 h). Across the 341 verification runs, 243 ended verified, 63 potential bug, 28 inconclusive, 5 unsupported, and 2 errored. In exchange for 1% of geometric-mean speed, every gated kernel comes with a memory- and thread-safety proof.

Library contracts: In our initial experiments on the two GEMM tasks, we noticed that the gated agent incorrectly abandoned cuBLAS, assuming the verifier could not reason about library calls. It reached only 2.2–2.6×, against 19–21× unconstrained. We therefore added one prompt sentence stating that cuBLAS calls are checked against library contracts (§ 3). All 14 trajectories then ended with a verified kernel that calls cuBLAS for the GEMM and fuses the epilogue, at 21.5×. Similarly, when the feedback for an unsupported construct named only the blocker class, the agent abandoned a 6.6× cuBLAS kernel. Once KernelArmor explained the unsupported call, the agent switched to a supported one instead. An in-the-loop verifier must therefore tell the agent what it can check and why it rejects a candidate, not only whether the candidate passed.

Related Work

LLM-assisted deductive verification. AutoVerus and dafny-annotator use language models to supply proof annotations and verifier feedback to guide subsequent attempts ((Yang et al., 2025; Poesia et al., 2024)). ESBMC-AI uses bounded-model-checking counterexamples to guide repairs to vulnerable C programs ((Tihanyi et al., 2023)). ProofWright applies this approach to CUDA, using a Ver-Cors knowledge base and an evolving annotation guide for safety proofs, together with Rocq-based reasoning for selected semantic checks ([6]). KernelArmor builds on this agentic verification methodology and addresses the CUDA-modeling limitations that constrain its application to optimized kernels. Its contributions include the extended safety model, annotation-only integrity checks against the preprocessed program, and diagnostics that distinguish validated safety findings from unresolved proofs and unsupported constructs. The annotation guard preserves the proof target; the resulting safety guarantee remains conditional on the stated contracts and modeling assumptions.

Verified GPU programming and numerical correctness. Kuiper uses F*/Pulse to construct safe, functionally verified GPU programs (Martínez et al., 2026); KuiperBench applies this approach to agent-written implementations of all 100 KernelBench Level 1 tasks (Martínez & Sorensen, 2026). Browne et al. combine Float32 equivalence proofs in Lean with VerCors safety verification (Browne et al., [5]). KernelArmor focuses on memory and thread safety of existing CUDA implementations under explicit contracts, leaving numerical equivalence as a complementary objective.

Dynamic safety checking and fuzzing. Compute Sanitizer, cuCatch, iGUARD, and SuperCollider detect memory errors or races at runtime ([18] (NVI 2022; Tarek Ibn Ziad et al. 2023; Stephenson et al. 2026)). cu-Fuzz expands input exploration through host/device coverage, but its coverage signatures do not distinguish thread interleavings (Tarek Ibn Ziad & Kozyrakis, 2026). These tools depend on exercised inputs and schedules; KernelArmor complements them with deductive safety guarantees under explicit contracts and its supported CUDA model.

Static GPU safety analysis. GPUVerify reduces race- and barrier-divergence checking to sequential verification ([1]); its documented default configuration does not check array bounds (Multicore Programming Group, Imperial College London). Failal infers access-memory protocols for compositional race analysis without user-supplied loop invariants ([8]). GKLEE uses concolic execution for verification and test generation (Li et al., 2012), and ESBMC-GPU explores CUDA behaviors through SMT-based context-bounded model checking (Monteiro et al., 2018b). More recently, SCuBA relates host-side allocation to device-side accesses and tracks logical allocation partitions to detect input-dependent and intra-allocation bounds errors ([17]). KernelArmor shares the goal of detecting errors beyond the tested input distribution and extends VerCors’ separation-logic approach ([2, 3]), combining bounds and alignment checks with permission-based concurrency reasoning.

Conclusion

This paper presents KernelArmor, which combines agent-generated annotations with an extended SMT Verifier to establish memory and thread safety for optimized CUDA kernels under explicit contracts. It outperforms the evaluated state-of-the-art tools on cuCatch and in-scope cuHadron tests. Integrated into an agentic optimization loop, it produces verified kernels in all trajectories while retaining 99% of the unconstrained geometric-mean speedup.

References

References

  1. [1]Betts, A., Chong, N., Donaldson, A. F., Qadeer, S., and Thomson, P. GPUVerify: A verifier for GPU kernels. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications. ACM, 2012. URL https://www.doc.ic.ac.uk/~afd/papers/2012/OOPSLA.pdf.
  2. [2]Blom, S., Huisman, M., and Mihelčić, M. Specification and verification of GPGPU programs. Science of Computer Programming, 95 (Part 3):376–388, 2014. doi: 10.1016/j.scico.2014.03.013.DOI
  3. [3]Blom, S., Darabi, S., Huisman, M., and Oortwijn, W. The VerCors tool set: Verification of parallel and concurrent software. In Integrated Formal Methods, pp. 102–110. Springer, 2017. doi: 10.1007/978-3-319-66845-1_7.DOI
  4. [4]Brodsky, J., Kumar, D., Kashmira, S., Danatanarayana, J., Mars, J., Flautner, K., and Tang, L. Kernel Forge: An agent harness for LLM-based generation and optimization of CUDA kernels, 2026. URL https://arxiv.org/abs/2607.24762.
  5. [5]Browne, A., Lee, E., and Salvi, C. Proving LLM-generated GPU kernels correct and safe: Floating-point verification in Lean and VerCors, 2026. URL https://arxiv.org/abs/2610.24548.
  6. [6]Chatterjee, B., Zagieboylo, D., Damani, S., Hari, S., and Kozyrakis, C. ProofWright: Towards agentic formal verification of CUDA, 2025. URL https://arxiv.org/abs/2511.12294v2. Version 2, revised March 2026.
  7. [7]Chen, F. and Stratton, J. Artificial intelligence in the firm: Bottlenecks in software production, August 2026. URL https://fion.ac/jellyfish.pdf. Working paper.
  8. [8]Cogumbreiro, T., Lange, J., Rong, D. L. Z., and Zicarelli, H. Checking data-race freedom of GPU kernels, compositionally. In Computer Aided Verification, volume 12759 of Lecture Notes in Computer Science, pp. 403–426. Springer, 2021. doi: 10.1007/978-3-030-81685-8_19.DOI
  9. [9]Cordeiro, L., Fischer, B., and Marques-Silva, J. SMT-based bounded model checking for embedded ANSI-C software. IEEE Transactions on Software Engineering, 38(4):957–974, 2012. doi: 10.1109/TSE.2011.59.
  10. [10]Costa, F., Castes, C., Bourgeat, T., and Mirhoseini, A. AI as a compiler: Compiling Triton kernels without the Triton compiler, 2026. URL https://arxiv.org/abs/2609.36800.
  11. [11]Dai, W., Wu, H., Yu, Q., Gao, H.-a., Li, J., Jiang, C., Lou, W., Song, Y., Yu, H., Chen, J., Ma, W.-Y., Zhang, Y.-Q., Liu, J., Wang, M., Liu, X., and Zhou, H. CUDA Agent: Large-scale agentic RL for high-performance CUDA kernel generation, 2026. URL https://arxiv.org/abs/2602.24286.
  12. [12]de Moura, L. and Bjørner, N. Z3: An efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS), volume 4963 of Lecture Notes in Computer Science, pp. 337–340. Springer, 2008. doi: 10.1007/978-3-540-78800-3_24.
  13. [14]Driscoll, B., Dubey, K., Wei, A., Kayal, N., Sharma, R., and Aiken, A. Equivalence checking of ML GPU kernels, 2026. URL https://arxiv.org/abs/2511.12638v3. Version 3, revised August 2026.
  14. [15]Gargary, A. V., Martínez, G., Burckhardt, S., Ebner, G., Jangda, A., Musuvathi, M., and Sorensen, T. RESOLVE: Language-agnostic validation of GPU kernels through testing, reduction, and proof, 2026. URL https://arxiv.org/abs/2610.05683.
  15. [16]Geng, J. and Neubig, G. Effective strategies for asynchronous software engineering agents, 2026. URL https://arxiv.org/abs/2603.21489.
  16. [17]Ghosh, A., Nayak, A., Shyam, D. R. T., and Basu, A. Chasing elusive memory bugs in GPU programs, 2026. URL https://arxiv.org/abs/2601.21552.
  17. [18]Kamath, A. K. and Basu, A. iGUARD: In-GPU advanced race detection. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, pp. 49–65. ACM, 2021. doi: 10.1145/3477132.3483545. URL https://doi.org/10.1145/3477132.3483545.
  18. [19]Lattuada, A., Hance, T., Cho, C., Brun, M., Subasinghe, I., Zhou, Y., Howell, J., Parno, B., and Hawblitzel, C. Verus: Verifying Rust programs using linear ghost types. Proceedings of the ACM on Programming Languages, 7 (OOPSLA1), 2023. doi: 10.1145/3586037.DOI
  19. [20]Lee, J., Kim, Y., Cao, J., Kim, E., Lee, J., and Kim, H. Securing GPU via region-based bounds checking. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, pp. 27–41. ACM, 2022. doi: 10.1145/3470496.3527420. URL https://doi.org/10.1145/3470496.3527420.

Paper details

vlnc:W_dfhdeprf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents