betaFeedback

WorksComputer Science

Do LMs Know What Variables Do? Interpreting Code Representations via Variable Roles

Abstract

Variable roles describe recurring behaviors such as accumulation, traversal, and retaining the best candidate encountered. We use Sajaniemi's independently defined taxonomy to study which role distinctions can be recovered from large language models, how sparse autoencoders (SAEs) retain those distinctions, and whether associated directions support control of generation. We introduce static annotation rules for all eleven roles and a corpus of 126,926 natural-code files across five languages. Across five model-SAE systems, residual probes outperform individual SAE features and tested lexical baselines, with modest losses after neutral target renaming. Combining SAE features closes much of the single-feature gap, but probes on SAE reconstructions are less sensitive to role-changing edits reviewed by LLMs. High paired ranking accuracy coexists with frequent false positives at fixed thresholds. During generation, interventions often change role annotations at the cost of correctness; changes accompanied by passing tests show no clear aggregate advantage over random controls. The benchmark separates role-related recoverability, SAE fidelity, and correctness-preserving control through matched prediction, source edits, and generation interventions.

Introduction

Large language models generate and modify source code, from code completion to agents that plan, edit, and test software ([29, 13]). Yet programming success alone does not establish which program properties a model represents: models may exploit identifiers and familiar code patterns, and their performance can change under controlled input transformations ([25]). We investigate this question through the behavior of individual variables. A variable may advance through a predictable sequence, accumulate values, or retain the best candidate seen so far. These behaviors depend on relationships across updates: recognizing an addition alone does not distinguish a regular progression from accumulation of input-dependent data. The same behavior can also use different names and language constructs, motivating three questions: which role distinctions are recoverable, how faithfully do sparse autoencoders retain them, and can associated directions change generation while preserving correctness?

We investigate these questions using Sajaniemi’s variable roles, developed in the psychology of programming to characterize recurring uses of variables ([47, 48]). The taxonomy defines behavioral criteria independently of the models under study. Prior studies recover program structure and intermediate execution states from model representations ([24, 30]) and find shared representations across programming languages ([31, 62]). We extend these analyses to variable behavior through matched prediction, controlled source edits, and generation interventions.

We combine program analysis with sparse autoencoder (SAE) features and probes, contributing:

  • Large-scale multilingual dataset and benchmark. We introduce a dataset of 126,926 natural-code files across five programming languages, to our knowledge the first au-

tomatically annotate all eleven Sajaniemi variable roles at this scale. We construct a benchmark with matched and mined negatives, LLM-reviewed role-changing edits, and role-preserving target renaming to measure recoverability under controlled comparisons. We will publicly release the benchmark, annotations, evaluation splits, and extraction pipeline, with instructions for reconstructing the corpus.

  • Recoverability and SAE fidelity. Across five model–SAE systems, we compare residual probes, reconstruction probes, individual features, and probes combining 64 features at observation points after the selected operation. Controlled edits test sensitivity beyond matched prediction; language, implementation, and readout comparisons qualify recoverability.

  • Limits of intervention-based control. We measure role induction and suppression alongside functional correctness and random-direction controls. Annotation changes are substantially more frequent than changes accompanied by passing tests; Figure 1 illustrates a selected successful repair.

Figure 1. Variable roles connect code annotation, representation analysis, and steering. (1) Code Property Graph (CPG) queries annotate all eleven Sajaniemi variable roles at scale. (2) Probes and SAE features recover role information; bars show matched-pair test average precision (AP) for Gemma 3 12B (four-layer mean and range). Code and graph sketches illustrate the method. (3) In Gemma 3 4B, steering a most-wanted-holder (MWH) feature changes a stub into a working search.

Background: Variable Roles in Program Comprehension

Program comprehension and programming knowledge. Research on human program comprehension distinguishes representations of operations and control flow from a situation model of program goals, with task-dependent prominence [45]. Programming plans describe familiar patterns of operations that accomplish particular goals [52]. Sajaniemi’s roles bring this perspective to individual variables through variable plans, or stereotyped uses of variables. The taxonomy originated in textbook programs; subsequent card sorting and interviews with professional programmers recovered groupings consistent with the roles, alongside variation in behavioral interpretations [48].

Variable roles as behavioral concepts. A role characterizes a variable through its successive values and how they are obtained [47, 48]: a stepper follows a predictable progression, a gatherer combines contributions, and a most-wanted holder retains the best candidate encountered. Figure 2 summarizes all eleven Sajaniemi variable roles; Appendix § 13.8 illustrates alternative implementations. Roles describe variable behavior, so role preservation differs from program equivalence: minimum and maximum computations can share a role, whereas equivalent programs can use variables with different roles. Although defined independently of the models, these roles can correlate with identifiers or syntax; testing their representation therefore requires controls for such surface cues.

Figure 2. Overview of conceptual variable roles according to Sajaniemi [47] and Sajaniemi & Navarro Prieto (2005). Matching colors and symbols link definitions to variable occurrences in Python code. Examples are deliberately simple, chosen for illustration. Appendix § 8.1 shows more complex examples from real-world code.

Variable roles in neural models of code. Hussain et al. [28] detect steppers and walkers and inject their roles into Code2Seq representations. Demitras et al. (2025) use LLMs to detect programming plans in student programs. These studies use programming abstractions as input features or prediction targets; we compare role recoverability under controlled source edits with intervention outcomes during generation. This comparison requires annotations defined independently of model activations and linked to individual variables in natural code.

Annotating Variable Roles at Scale

To evaluate all eleven roles in multilingual natural code, we develop an automatic annotation procedure with explicit behavioral contracts.

Automatic role extraction. Scaling annotation requires operational definitions of these cognitive abstractions. Prior systems cover subsets of the taxonomy, use execution traces, or adopt alternative definitions (Table 1). We operationalize all eleven Sajaniemi variable roles as explicit role contracts over Joern Code Property Graphs (CPGs), which represent syntax, control flow, and data dependencies ([60]). Queries recover initializations, updates, and uses, whose relationships are checked against the contracts. For example, the most-wanted holder contract links a candidate comparison to conditional replacement of the current value, with consistency checks on initialization and other writes. These static checks require neither execution nor a buildable project, enabling annotation of isolated source files. Appendix § 10.2 gives the contracts and pseudocode.

Constructing a multilingual natural-code dataset. We apply these contracts to CodeParrot GitHub-Code, combining an expanded annotation run with an earlier role-enriched subset, including Ruby. Deduplicating sources and matching declarations yields 126,926 files across five languages and 3,525,530 variables, of which 1,737,758 have at least one extracted role. Variables lacking a target annotation serve as candidate negatives, including any role instances missed by the extractor. For evaluation, we form five approximately equal folds stratified by role and language, keeping exact-source and detected clone groups intact. Appendix § 13 reports role and language distributions.

Annotation validity and evaluation scope. Source inspection and implementation checks informed contract refinement, but coverage remains limited to supported static patterns. A human evaluation on 660 stratified variables (Appendix § 12) assesses annotation quality, although its sampling design does not support corpus-wide precision or recall estimates. Counterfactual evaluation additionally uses case-level LLM review of original behavior and proposed edits (Appendix § 13.4). We therefore distinguish annotation accuracy on the evaluated subset from automatic-label prediction and sensitivity to reviewed edits.

Do Language Models Represent Variable Roles?

We first measure recoverability of role annotations, then test sensitivity to role-changing edits and stability under role-preserving renaming.

Matched prediction and diagnostic readout. To reduce surface differences, we pair each annotated positive with a variable lacking that annotation, matching language and declaration kind and favoring similar surface characteristics. The test set contains 4,238 balanced pairs across eleven roles in C, C++, JavaScript, and Python. Because assignment targets often precede their determining expressions, we read the state of a target reference inserted after the selected statement or loop/conditional header. Appendix § 13.4 records selection, exclusions, and fitting; Appendix § 13.7 illustrates the contrasts.

Four predictors separate feature restriction from reconstruction. We evaluate Gemma 2 2B, Gemma 3 4B/12B, Llama 3.1 8B, and Qwen3.5 9B at four layers each. SAEs encode residual activations as sparse feature vectors and reconstruct them through a linear decoder ([27, 15]). Ridge probes on residual activations, SAE reconstructions, and 64 SAE features separate reconstruction fidelity from restricted feature readout; a single-feature predictor tests individual coordinates. Training fixes standardization, feature ranking, and polarity; validation selects regularization, the single feature, and decision thresholds. The name and code surface baselines are TF–IDF logistic classifiers over character nn-grams of the target name and the code prefix through the diagnostic token, respectively. Fits are frozen before evaluation; Appendix § 13.2 lists SAE releases.

Matched recoverability exceeds the tested lexical baselines. Residual probes exceed both surface baselines and individual SAE features in every role–model comparison (Figure 3). Averaging roles, models, and layers equally, average precision (AP) is 0.90 for residual probes and 0.72 for single features; constant-score AP is 0.50. Code features are competitive with individual SAE features: for most-wanted holder, their AP is 0.90, versus 0.75 for single features and 0.95 for residual probes. Reconstruction and top-64 probes close much of the single-feature gap (Table 2), showing the cost of restricting the readout to one coordinate.

Figure 3. Model representations support role recovery beyond the tested surface baselines. Residual probes attain higher mean test average precision (AP) than both surface baselines and individual SAE features for every role and model. Evaluation uses matched C, C++, JavaScript and Python pairs from one source-group split. Model bars average AP over four prescribed layers; whiskers show the layer range. Gray baselines use character nn-grams of names (N) or code prefixes through the causal readout (C). Role colors and symbols follow Figure 2; solid and hatched bars denote Probe and validation-selected SAE feature.

PredictorMatched AUROCCounterfactual paired accuracy
Residual probe0.906–0.9130.813–0.877
SAE reconstruction probe0.857–0.8810.739–0.784
Top-64 SAE probe0.853–0.8780.722–0.753
Single SAE feature0.722–0.7820.622–0.713

Table 2. Multiple-feature readouts close much of the single-feature gap; reconstruction probes have lower counterfactual sensitivity. Values span five model means, each weighting eleven roles and four prescribed layers equally, with matched-negative training. Ranges show variation across systems; paired uncertainty for reconstruction-minus-residual differences is reported in the text. Counterfactual accuracy compares each original only with its corresponding source-reviewed edit.

Counterfactual ranking and classification give different answers. To test whether matched recoverability tracks defining behavior, we evaluate edits that remove the target role. On 1,698 LLM-reviewed pairs from 948 source groups, residual probes rank originals above edited negatives with paired accuracy 0.813–0.877; ties receive half credit. The code-prefix baseline reaches 0.750, while unchanged names give the name baseline 0.500. Reconstruction probes fall 4.0–12.3 percentage points below residual probes, with all five paired 95% source-group bootstrap intervals excluding zero: reconstruction retains less counterfactual sensitivity. At the validation thresholds, however, residual probes classify 68–71% of edited negatives as positive, yielding balanced accuracy of 0.610–0.637. The scores therefore respond to role-changing edits more reliably than their binary decisions reject the edited examples.

Neutral target renaming preserves most ranking performance. As a complementary role-preserving control, renaming both members of 100 test pairs per role reduces residual AUROC (area under the ROC curve) by 0.6–1.7 percentage points, to 0.878–0.887. On these same pairs, code-surface AUROC changes from 0.740 to 0.723 and name-surface AUROC from 0.602 to 0.492. Renaming preserves other identifiers, comments, and implementation cues, isolating sensitivity to the target spelling. Appendix § 13.3 reports uncertainty and decision changes; Appendix § 13.5 evaluates prompted recognition.

Readout and implementation condition the signal. The diagnostic readout also matters because it follows the selected operation. At the second prescribed layer, residual counterfactual paired accuracy is 0.495–0.594 at the native identifier, 0.790–0.915 at statement end, and 0.856–0.920 at the inserted identifier. Separately fitted readouts thus expose much of this sensitivity at later unmodified positions. Even at the diagnostic readout, recoverability varies: with multilingual training, stepper AUROC ranges from 0.682 in JavaScript to 0.972 in Python; implementation contrasts can also be confounded by language (Appendix § 13.3).

Figure 4 illustrates the broader extrema, Boolean, and indexing associations of selected features. These contexts aid interpretation but do not establish role specificity (Appendix § 13.10). Neither contextual associations nor counterfactual sensitivity establish control; we next test interventions during generation.

Figure 4. Role-associated latents connect code updates to broader extrema, Boolean, and indexing concepts. Role colors and symbols follow Figure 2; descriptions and prose come from Neuronpedia, and code from our training corpus. Highlight intensity indicates latent activation strength on a linear scale within each latent; return marks denote newlines. Code activations use standalone snippets; prose windows retain their original Neuronpedia contexts.

Do Variable Role Representations Influence Code Generation?

To test control beyond recoverability, we apply activation addition and direction removal (Turner et al., [56]; Arditi et al., 2024; Ferrando et al., [14]; Tahmić & Cheng, 2025) on Mostly Basic Programming Problems (MBPP) (Austin et al., [5]). The SAE-feature experiments cover

Gemma 2 2B and Gemma 3 4B/12B; a separate comparison normalizes probe and SAE directions under a correctness constraint. For each role rr, binary extractor labels on reference (RrefR_{\mathrm{ref}}) and unsteered (R0R_0) solutions define cohorts. These labels describe implementations: a correct alternative may use a different role. Target and random directions share eligible tasks; Pass@1 measures the fraction passing the supplied tests.

Reference-conditioned induction. For feature ff, we add αdf\alpha d_f, where df=Wdec[f,:]d_f = W_{\mathrm{dec}}[f,:], to the residual stream at every generated token; RαR_\alpha denotes the resulting role annotation. The operational score sufficiency measures induction of an annotation present in the reference but absent from the baseline:

Sufficiencyr=P(Rα⋆=1∣Rref=1,R0=0).\mathrm{Sufficiency}_r = P(R_{\alpha^\star} = 1 \mid R_{\mathrm{ref}} = 1, R_0 = 0).

Eligible examples are split equally into calibration and test sets; α⋆\alpha^\star maximizes calibration sufficiency over a predefined strength grid and is fixed for testing. Reference-negative induction, denoted spurious induction in Figure 6, uses a separate cohort:

Figure 6. Causal effects of representative role-associated SAE features in Gemma 2 2B. (a) Scores for one selected feature per role and same-layer random SAE controls (hatched means; range whiskers); n/a denotes undefined necessity. (b) Causal scores versus changes in MBPP Pass@1; shapes match role labels in (a). Removal suggests causal involvement with limited specificity and substantial correctness costs; positive steering provides weak evidence of sufficiency.

SpuriousInductionr=P(Rα⋆=1∣Rref=0,R0=0).\mathrm{SpuriousInduction}_r = P(R_{\alpha^\star} = 1 \mid R_{\mathrm{ref}} = 0, R_0 = 0).

Because reference-negative induction may yield a valid alternative, both cohorts require separate correctness evaluation.

Reference-conditioned suppression. To test suppression of roles present in both reference and baseline, we normalize the decoder direction to d^f\hat{d}_f and orthogonalize matrices writing to the residual stream ([14, 53] (Arditi et al. 2024)):

Wout′=Wout−(Woutd^fT)d^f.W'_{\mathrm{out}} = W_{\mathrm{out}} - (W_{\mathrm{out}}\hat{d}_f^{\mathsf{T}})\hat{d}_f.

The operational score necessity is the frequency of role absence after removal, denoted by Rort=0R_{\mathrm{ort}} = 0:

Necessityr=P(Rort=0∣Rref=1,R0=1).\mathrm{Necessity}_r = P(R_{\mathrm{ort}} = 0 \mid R_{\mathrm{ref}} = 1, R_0 = 1).

With no fitted strength, removal uses all eligible examples. Unlike generation-time addition, global removal also changes prompt processing; effect sizes reflect different perturbation scopes.

Removal suggests causal involvement with limited specificity. Some Gemma 2 2B features have high necessity (Figure 6). Gatherer suppression exceeds its random-control range, whereas most-wanted holder’s score of 1.00 ties its control maximum. Their Pass@1 losses (−0.33 and −0.29) suggest suppression may partly reflect broader generation disruption, limiting evidence of role-specific control.

This correctness cost persists across all selected SAE features: removal suppresses annotations in 41/91, 72/109, and 131/187 target–task outcomes for Gemma 2 2B, Gemma 3 4B, and Gemma 3 12B. Only 2/91, 4/109, and 15/187 outcomes also pass the supplied tests; requiring both baseline and intervention to pass leaves 0, 3, and 13. Rates of annotation changes with passing tests exceed matched random controls by 1.3–3.7 percentage points for removal and 0.9–1.9 for addition, but all six task-grouped bootstrap 95% intervals include zero (Appendix § 14.2). These joint outcomes show no clear advantage over random controls.

Weak sufficiency, partial structural effects. Positive steering yields nonzero sufficiency scores of only 0.10–0.19, even where they exceed random-control ranges (Figure 6); low reference-negative induction likewise reflects weak effects. Yet Table 3 shows increased stepper-associated range calls under positive steering in both models, with smaller increases in gatherer-associated augmented assignments. These partial structural effects do not establish full-role sufficiency; other shifts are mixed. Only pairs with two clean parses contribute; Gemma 3 4B removal retains just 4/30 gatherer and 11/48 stepper pairs, from two and five tasks, limiting generalization.

RoleRepresentative syntaxGemma 2 2B: Pos. steer.Gemma 2 2B: Orth.Gemma 3 4B: Pos. steer.Gemma 3 4B: Orth.
GathererAugmented assignment+1.9 (107)−58.8 (17)+2.3 (44)−50.0 (4)
Most-wanted holderif statement−4.3 (92)−35.0 (20)+17.6 (17)−75.0 (4)
One-way flagBoolean literal−5.9 (17)–−11.1 (18)–
Stepperrange call+20.3 (59)−20.0 (40)+9.3 (54)−63.6 (11)
WalkerSubscript expression−2.5 (40)+0.0 (8)−5.1 (39)+0.0 (10)

Table 3. Role-adjacent syntax changes after intervention. Cells give intervention-minus-baseline changes (percentage points) in program frequencies of Tree-sitter patterns, with counts of pairs with two clean parses in parentheses. Patterns are structural correlates, not role definitions or evidence of full-role sufficiency. Cohorts and baselines differ by intervention; dashes mark unavailable removal generations. Counts repeat tasks across features; Appendix § 14.2 reports parse survival.

Correctness-constrained calibration usually selects zero. To test whether calibration can avoid these correctness costs, we constrain correctness in a normalized probe/SAE comparison; ten of eleven completed model–role cells select no intervention (Appendix § 13.9). In the remaining cell, each target direction produces one induction accompanied by passing tests among five eligible tasks, matching the random residual direction. Figure 5 illustrates repairs and failures, with selected successes and unfiltered random cases documented in Appendix § 14.1.

Figure 5. Positive steering can repair stubs but can also break correct programs. Each pair shows held-out MBPP excerpts; test counts are passing supplied assertions out of three, and role bits denote extractor output. G2/G3 denote Gemma 2 2B/Gemma 3 4B; MWH denotes most-wanted holder. The selected repairs and failures are illustrative, while the random cases come from a uniform draw without replacement. Outcomes evaluate complete outputs, including any omitted foreign-language tails.

Related work

These distinctions connect work on behavior, representations, and interventions (Appendix § 13.6).

Behavioral robustness and semantic priors. Semantics-preserving transformations test reliance on surface cues. ReCode finds reduced generation robustness after changes to identifiers, syntax, formatting, and docstrings ([58]); related mutations destabilize program-output predictions ([43]). Masked-code models retain predictive ability after operand and branch reordering and identifier renaming, suggesting semantic learning ([2]).

Representations of programming concepts. Probes recover syntax and structure ([32, 57]), including complete abstract syntax trees ([24]), and intermediate execution states in synthetic-language models ([30]). Whereas probes target predefined properties, CoCoA discovers token-representation clusters and interprets their lexical, syntactic, and semantic content through static analysis and LLM annotation ([50]). Variable roles let us compare residual and SAE recoverability under source edits, separating association from sensitivity to defining behavior.

Shared representations across programming and natural languages. Cross-language studies test dependence on language-specific syntax. The same attention heads convey variable-type information across three programming languages ([42]), while language-specific and language-agnostic neurons coexist within a model ([62]). Intermediate-layer decoding and neuron ablations also reveal overlap with English ([31]). Our multilingual fits complement transfer studies by measuring variation in role recovery within each evaluated language.

Causal and mechanistic analyses. Coding failures can coexist with useful computations: steering restores disrupted type predictions ([37]), while synthetic parentheses analyses trace errors to interference between reliable and unreliable components ([46]). Yet SAE features predictive of code correctness need not support effective steering ([53]). We extend this distinction to role annotations, comparing changes, joint passing outcomes, and matched random controls.

Conclusion, Limitations, and Future Work

Residual probes recover operational role annotations beyond tested lexical baselines. Combining SAE features narrows the single-feature gap, but reconstruction probes retain less sensitivity to reviewed role-changing edits. Removal suggests causal involvement with limited specificity and correctness costs, while positive steering yields weak full-role induction despite structural shifts.

Limitations. Human evaluation shows that annotation quality varies substantially across roles, particularly for most-recent holder (Appendix § 12). These estimates come from a stratified sample and do not establish corpus-wide precision or recall; inter-annotator reliability also remains unmeasured. Decoding covers four layers per model and one source-group split, with uncertainty conditional on fitted predictors. The corpus includes development data without repository-level holdout, and target renaming retains other lexical cues. Interventions use small overlapping cohorts and finite prompt-visible tests, with unresolved historical scale provenance (Appendix § 14.2).

Future work. Our corpus combines natural-code complexity with independently defined roles and controlled source edits, offering a semi-controlled setting for studying conceptual abstraction beyond toy tasks. Future work can test whether role representations transfer to unseen languages and implementations, combine into higher-level programming plans, and support interventions that change variable behavior while preserving correctness.

Use of AI Assistance

We used large language models and coding agents to assist literature searches, manuscript drafting and revision, visualization code, and experimental protocols. The authors reviewed all references, procedures, figures, and manuscript content before inclusion. LLM reviewers also assessed counterfactual source edits, recording original-role validity, edited behavior, and rejection reasons; Appendix § 13.4 distinguishes this procedure from human annotation validation.

AI assistance was more extensive in implementing the variable-role extractor, whose multilingual static-analysis rules combine syntax, control flow, and data flow. The authors specified behavioral requirements and rejection conditions for each role (Appendix § 10.2); coding agents implemented these contracts using Joern CPG facts and the existing extraction infrastructure. We inspected implementations and annotations manually on source-code examples and automatically through checks on annotations and intermediate variable facts. Failures and false positives or negatives informed contract refinements, followed by implementation updates and re-evaluation until both stabilized. The authors controlled the choice of taxonomy, operational definitions, final annotation rules, experimental design, interpretation, and manuscript.

Variable roles in real-world code

The simple examples in Figure 2 isolate each role’s behavior. The six excerpts below cover all eleven Sajaniemi variable roles in natural code from our annotated corpus. Common leading indentation is removed; long lines wrap for display.

Streaming an application release.                                                                                                     Nextcloud App Store

1  def _stream_to_file(self, file: Any, max_size: int,
2                          req: requests.Response) -> None:
3      # start streaming download
4      finished = False
5      try:
6          size = 0
7          for chunk in req.iter_content(1024):
8              file.write(chunk)
9              size += len(chunk)
10             if size > max_size:
11                 msg = ’Downloaded archive is bigger than the ’ \
12                       ’allowed %i bytes’ % max_size
13                 raise MaximumDownloadSizeExceededException(msg)
14             finished = True
15     finally:
16         # in case any errors occurred, get rid of the file
17         file.close()
18         if not finished:
19             os.remove(file.name)

The fixed value max_size bounds the download. The gatherer size adds input-dependent chunk lengths, and the one-way flag finished changes to true only after streaming completes. The temporary msg holds an exception message. The finally block uses the flag to clean up after an error.

Applying byte-level delta operations.                                                                                                             Subvertpy

1  def txdelta_apply_ops(src_ops, ops, new_data, sview):
2      """Apply txdelta operations to a source view.
3
4      :param src_ops: Source operations, ignored.
5      :param ops: List of operations (action, offset, length).
6      :param new_data: Buffer to fetch fragments with new data from
7      :param sview: Source data
8      :return: Result data
9      """
10     tview = bytearray()
11     for (action, offset, length) in ops:
12         if action == TXDELTA_SOURCE:
13             # Copy from source area.
14             tview.extend(sview[offset:offset+length])
15         elif action == TXDELTA_TARGET:
16             for i in range(length):
17                 tview.append(tview[offset+i])
18         elif action == TXDELTA_NEW:
19             tview.extend(new_data[offset:offset+length])
20         else:
21             raise Exception("Invalid delta instruction code")
22     return tview

The walker action takes the operation code from each tuple in ops. The stepper i advances through the range of each target-copy operation. The container tview grows through bulk extension and single-byte appends. These roles coexist in a dispatch loop for different copy operations.

Reading blocks and emitting delta windows.

def send_stream(stream, handler, block_size=DELTA_WINDOW_SIZE):
    """Send txdelta windows that create stream to handler
    :param stream: file-like object to read the file from
    :param handler: txdelta window handler function
    :return: MD5 hash over the stream
    """
    hash = md5()
    text = stream.read(block_size)
    if not isinstance(text, bytes):
        raise TypeError("The stream should read out bytes")
    while text:
        hash.update(text)
        window = (0, 0, len(text), 0, [(TXDELTA_NEW, 0, len(text))], text)
        handler(window)
        text = stream.read(block_size)
    handler(None)
    return hash.digest()

The most-recent holder text is replaced by each block read from the stream. Each block updates the digest and enters a delta window before the next read. A type check and a final callback surround this acquisition–use cycle.

Monitoring peak memory usage.

def during_execution_memory_sampler():
    import time
    import memory_profiler
    global keep_watching, peak_memory_usage
    peak_memory_usage = -1
    keep_watching = True

    n = 0
    WAIT_BETWEEN_SAMPLES_SECS = 0.001
    MAX_ITERATIONS = 60.0 / WAIT_BETWEEN_SAMPLES_SECS
    while True:
        mem_usage = memory_profiler.memory_usage()[0]
        peak_memory_usage = max(mem_usage, peak_memory_usage)
        time.sleep(WAIT_BETWEEN_SAMPLES_SECS)
        if not keep_watching or n > MAX_ITERATIONS:
            # exit if we’ve been told our command has finished or if it has
            # run
            # for more than a sane amount of time (e.g. maybe something
            # crashed
            # and we don’t want this to carry on running)
            if n > MAX_ITERATIONS:
                print("{} SOMETHING WEIRD HAPPENED AND THIS RAN FOR TOO LONG,
                THIS THREAD IS KILLING ITSELF".format(__file__))
            break
        n += 1

The most-wanted holder peak_memory_usage retains the largest observed usage during each sampling run. Its monitoring loop also handles a stop signal, a sampling delay, and a timeout.

Searching parent directories for configuration. Projeny

def findConfigPath(filePath):
    lastParentDir = None
    parentDir = os.path.dirname(filePath)

    while parentDir and parentDir != lastParentDir:
        configPath = os.path.join(parentDir, Prj.ConfigFileName)

        if os.path.isfile(configPath):
            return configPath

        lastParentDir = parentDir
        parentDir = os.path.dirname(parentDir)

    assertThat(False, "Could not find Prj config path starting at path {0}",
        filePath)

The follower lastParentDir preserves parentDir’s old value before the next parent-directory step. Their comparison detects when traversal stops progressing; a file-existence check allows early return.

Computing a median with fallback handling. LIBXSMM

def median(list_of_numbers, fallback = None, average = True):
    size = len(list_of_numbers)
    if (0 < size):
        # TODO: use nth element
        list_of_numbers.sort()
        size2 = int(size / 2)
        if (average and 0 == (size - size2 * 2)):
            medval = int(0.5 * (list_of_numbers[size2-1] + list_of_numbers[
                size2]) + 0.5)
        else:
            medval = list_of_numbers[size2]
        if (None != fallback):
            result = min(medval, fallback)
        else:
            result = medval
    elif (None != fallback):
        result = fallback
    else:
        sys.tracebacklimit = 0
        raise ValueError("median: empty list!")
    return result

The organizer list_of_numbers sorts the elements; subsequent computation only reads the reordered list. The function also handles even-sized input, a fallback value, and empty input.

Corpus sizes in prior variable-role studies

Table 4 expands Table 1, distinguishing source-code pools from labeled examples in each study’s reported units.

WorkSignalLanguagesRolesCorpusLabeled data
Gerdt & Sajaniemi [17]Static analysisPascal3NRNR
Bishop & Johnson [6]Static analysisJavaChecking17 programsNR
Gerdt & Sajaniemi [18]Static + learningPascal987 programs305 vars.
Demyanova et al. [11]Static analysisC14 custom5.2 KLOC†NR
Lui et al. [38]Traces + NNC4 classes110 programs8,907 cases
Hussain et al. [28]Syntax rulesJava214,022,318 methods1,438,035 vars.
OursStatic CPG analysisC, C++, JS, Python, Ruby11126,926 files1,737,758 vars.

Table 1. Automatic variable-role analysis and reported corpus sizes. All but Demyanova et al. use Sajaniemi’s taxonomy. NR: not reported; vars.: role-labeled variables; NN: neural classifier. †: role-definition sample. Appendix § 10.1 details annotation units, breakdowns, and comparability.

Study / coverageCorpus and labeled examplesScope and qualifications
Gerdt & Sajaniemi [17]; Pascal; 3 rolesPrototype program and variable counts: NR.Fixed Value, Stepper, and Most-Recent Holder, selected using an earlier textbook survey.
Bishop & Johnson [6]; Java; role checking17 canonical teaching programs; variable count: NR.Checks supplied role assignments in procedural Java.
Gerdt & Sajaniemi [18]; Pascal; 9 rolesLearning: 43 programs, 175 variables. Evaluation: 44 programs, 130 variables. Total: 87 programs, 305 variables.Filtered, human-annotated textbook sample; arrays omitted and role-changing variables split by role.
Demyanova et al. [11]; C; 14 custom roles5.2 KLOC of cBench for role design. Downstream file and variable totals: NR.Custom roles evaluated on SV-COMP file-category prediction, not role-label accuracy.
Lui et al. [38]; C; 4 classes110 educational programs; 8,907 labeled trace cases.Manually annotated programs; cases are not stated to be distinct declarations. One-Way Flag and Constant are merged.
Hussain et al. [28]; Java; 2 roles14,022,318 methods; 1,438,035 role-augmented variables (624,701 steppers; 813,334 walkers).Automatic syntax rules; pool includes methods without either role. Evaluated on method-name prediction.
Ours; Five languages; 11 roles126,926 files; 1,737,758 distinct role-annotated variables (37,006 steppers; 41,686 walkers). Full counts: Appendix E.Automatic CPG labels; identical sources and matching declarations merged. Unlabeled variables can still have roles.

Table 4. Corpus sizes and annotation scope for Table 1. Totals combine reported splits where available. NR: not reported, not zero. Units and annotation quality are not directly comparable.

Comparability. Programs, methods, and files are different source units, as are trace cases, declarations, and variable–role memberships. Counts do not imply matched deduplication, independent examples, or comparable holdouts. Role inventories and recognition rules also differ: Gerdt & Sajaniemi [18] include Transformation but have no Temporary evaluation examples. Automatic annotations and downstream task performance do not establish role-label accuracy.

Coverage. Our eleven-role, five-language corpus has more role-annotated variables than Hussain et al. [28], but fewer steppers and walkers (Table 4). Coverage is uneven: Fixed Value accounts for 74.4% of role memberships, and Ruby for 53 files. Broader coverage does not imply more examples of each role or higher annotation quality.

Variable-role extraction details

Section 8 describes how coding agents assisted implementation of these author-defined contracts.

Extraction pseudocode

vv: declaration identity; WW: binding writes; II: initialization events; U=W∖IU = W \setminus I; MM: observed state mutations; FF: raw Fixed Values. Initialization partitions are role-specific; named checks require supporting source/CPG evidence. CFG witnesses are bounded; missing labels do not prove semantic absence.

Supplement: src/role_extractor/scala/: facts/ builds evidence, role_predicates.sc classifies, annotation_emission.sc resolves labels. Frontend details and exact algorithms reside in this executable specification.

Fixed Value

  1. if Q(v)Q(v): return C(v)=FixedC(v) = \text{Fixed}

  2. require J(v)J(v) and B(v)B(v)

  3. reject if any write violates AA, or Z(v)Z(v) or O(v)O(v) fails

  4. accept

Key: QQ: verified non-text collection; CC: collection decision (third page). JJ: verified initialization, at most one per path (parameter entry; local initializer/exhaustive alternatives; member constructors). BB: no later/external rebinding. AA: plain, non-self-dependent writes. ZZ: no observed mutation. OO: no loop initialization/rebinding.

Pattern: limit = input.limit; use(limit)

Temporary (raw)

  1. require L(v)L(v), KT(v)K_T(v), W≠∅W \ne\varnothing

  2. for each w∈Ww \in W: require D(w)D(w) and 1≤∣Rw∣≤51 \le\lvert R_w\rvert\le5

  3. require 0≤d(w,r)≤50 \le d(w,r) \le5 for every r∈Rwr \in R_w

  4. return H(v)H(v)

Key: LL: genuine callable-local; KTK_T: complete read associations. DD: direct plain non-self assignment; RHS call/known-variable token, excluding strings/keywords and import/definition artifacts. RwR_w: reaching-definition reads with redefinition-free CFG paths. dd: line difference; same-line event order; otherwise CFG instruction distance. HH: removing writes disconnects loop-entry/back-edge paths to reads.

Pattern: tmp = parse(item); emit(tmp)

One-Way Flag

  1. b←b \leftarrow initial Boolean value; require Jb(v)J_b(v), U≠∅U \ne\varnothing, M=∅M = \varnothing

  2. for each w∈Uw \in U: require Tb(w)T_b(w)

  3. return some w∈Uw \in U satisfies Nb(w)N_b(w)

Key: JbJ_b: complete canonical identity/partition, Boolean-literal initialization bb (also 0/1). TbT_b: complete direct write of ¬b\neg b, or v∨ev \lor e from false / v∧ev \land e from true; symmetric operands allowed. ee: Boolean literal/reference/comparison or supported logical composition. NbN_b: terminal assignment or non-inert operand; structural compatibility, not guard satisfiability.

Pattern: found = false; ...; found = found || match

Stepper

  1. S←NS \leftarrow N

  2. repeat S←S∪{v:P(v),Dv⊆F∪S,G(v,F,S)}S \leftarrow S \cup\{v : P(v), D_v \subseteq F \cup S, G(v,F,S)\}

  3. until SS stops changing; return v∈Sv \in S

Key: NN: implicit numeric iterators. PP: complete repeated progression; every later write a direct self-update with unary +/−+/- or binary ++, −-, ×\times, //; exclude fixed −v-v and proved Walkers. DvD_v: non-self update dependencies. G(v,F,S)G(v,F,S): complete supported guards; dependencies in SS or fixed inputs with predictable seeds (parameters allowed).

Pattern: i = 0; while i < limit: i += stride

Extraction pseudocode

Walker

  1. if E(v)E(v): accept

  2. require KW(v)K_W(v) and U≠∅U \ne\varnothing

  3. for each w∈Uw \in U: require V(w)V(w)

  4. accept

Key: EE: implicit element iterator, excluding numeric iterators. KWK_W: complete navigation and initialization partition. VV: previous-position dependence; loop context; access/control use; resolved member, fixed-collection successor index, supported navigation call, or cursor increment/decrement with direct cursor use. An ordinary array index alone is insufficient.

Pattern: while p: use(p); p = p.next

Most Recent Holder

  1. require KR(v)K_R(v) and Av≠∅A_v \ne\varnothing

  2. reject if Walker, Follower, or Most Wanted Holder

  3. if member: return PR(v)P_R(v)

  4. return LR(v)L_R(v) and every a∈Ava \in A_v has witness R(a)R(a)

Key: KRK_R: complete acquisition/correction coverage of UU, M=∅M = \varnothing, no holder-dependent acquisition guard. AvA_v: acquisitions (call, collection access, iterator projection). LRL_R: non-member loop/proved-iterator context. RR: read outside own writes via one unambiguous stable-input correction chain; limits: depth 64, nodes 10,000, corrections 5, branches 128. PRP_R: same-owner member acquisitions in ordinary methods plus an owner use; no cross-call ordering proof.

Pattern: while more: current = read(); consume(current)

Follower

  1. require KF(v)K_F(v); reject if X(v)X(v)

  2. s←s \leftarrow unique master shared by every non-initial assignment

  3. require As(w)A_s(w) for every reassignment ww

  4. return Ys(v)Y_s(v)

Key: KFK_F: complete, nonempty reassignment set UFU_F. AsA_s: uniquely resolved canonical declaration/member/fixed-index master; RHS ss, s+ks + k, k+sk + s, or s−ks - k; kk literal/raw Fixed Value. YsY_s: every local loop has capture →\to master evolution →\to follower use →\to capture; members require ordered capture/evolution per writing method and same-owner use. XX: self-dependence, mutation, follower-controlled write, or follower-dependent master evolution.

Pattern: prev = current; current = next(current); use(prev)

Most Wanted Holder

  1. Primary⁡(v)←JS(v)\operatorname{Primary}(v) \leftarrow J_S(v) and S(v)S(v) and KS(v)K_S(v)

  2. clear primary status if Stepper, Walker, Gatherer, or Follower

  3. accept if Primary⁡(v)\operatorname{Primary}(v)

  4. otherwise accept if some primary hh satisfies Bh(v)B_h(v)

Key: JSJ_S: compatible seeds, nonempty loop-selection epochs, one direction. SS: guarded holder–candidate comparison, min/max, or null-seeded first-satisfying candidate with structural exit. KSK_S: all writes explained (seed/selection/unset fallback), no mutation. BhB_h: nonempty later writes all coupled to one primary hh by a unique guard in one context; nonempty fixed/stepper/walker dependencies, later read, no mutation.

Pattern: if score > best: best = score; winner = i

Gatherer

  1. require KG(v)K_G(v), JG(v)J_G(v), Av≠∅A_v \ne\varnothing

  2. reject if Stepper or Walker

  3. for each span a∈Ava \in A_v: require a self-update with DD

  4. require all resets, other transforms, and finalizations satisfy TT

  5. accept

Key: KGK_G: admitted numeric non-collection; complete phase coverage of UU. JGJ_G: parameter, stable numeric seed, or supported first contribution. AA: nonempty loop spans split by resets. SelfUpdate: uses the previous value. DD: indexed/member/dereferenced data or a non-fixed/non-stepper dependency; calls through argument data. TT: numeric/fixed resets; arithmetic transforms with numeric/fixed/stepper inputs; at most one stable arithmetic finalization per method outside loops.

Pattern: score *= 5; score += points[item]

Container

  1. require Q(v)Q(v) and Hc(v)H_c(v)

  2. Ec←E_c \leftarrow post-construction collection changes

  3. Pc←P_c \leftarrow event IDs consumed by proved permutation spans Π\Pi

  4. return Ec∖Pc≠∅E_c \setminus P_c \ne\varnothing

Key: QQ: verified non-text collection; HcH_c: source-backed target, excluding synthetic/function bindings. EcE_c: changes after the identified construction phase: rebinding, updates, add/remove/clear, element/descendant writes, exact-target mutations. Π\Pi: proved permutations; PcP_c: event IDs they consume. Alias-only effects do not establish mutation.

Pattern: items.append(x); items.sort()

Organizer

  1. require Q(v)Q(v), Hc(v)H_c(v), and Jc(v)J_c(v)

  2. require Πv≠∅\Pi_v \ne\varnothing

  3. return Ec∖Pc=∅E_c \setminus P_c = \varnothing

Key: JcJ_c: verified, complete collection initialization. Π\Pi: tuple swap; local dataflow swap; exact same-collection swap; known in-place sort/reverse/shuffle/rotate. Manual swaps require reaching-definition/CFG witnesses without transformation; ambiguous paths, indexes, or aliases fail. KcK_c: complete collection lifecycle.

Pattern: items.reverse()

Collection decision and final labels

  1. C(v)←C(v) \leftarrow None; require Q(v)Q(v) and Hc(v)H_c(v)

  2. if Ec∖Pc≠∅E_c \setminus P_c \ne\varnothing: C(v)←C(v) \leftarrow Container

  3. else if Jc(v)J_c(v) and Πv≠∅\Pi_v \ne\varnothing: C(v)←C(v) \leftarrow Organizer

  4. else if Jc(v)J_c(v) and Kc(v)K_c(v): C(v)←C(v) \leftarrow Fixed

Emission: retain raw multilabel results subject to exclusions above. Suppress Temporary if any role other than Fixed Value co-occurs; otherwise Temporary suppresses Fixed Value. Member subjects omit Temporary, Gatherer, and Most Wanted Holder. Raw Fixed Values remain available to dependency checks. Emit identity, labels, metadata, and exact source views; graph/alias/interprocedural limits follow the supplementary implementation.

RoleDistribution ExtractorDistribution HumanPerformance PrecisionPerformance RecallPerformance F1
Container30270.7330.8150.772
Fixed value1061450.8300.6070.701
Follower20220.9000.8180.857
Gatherer20250.9000.7200.800
Most-recent holder21300.6670.4670.549
Most-wanted holder21190.9051.0000.950
One-way flag21261.0000.8080.894
Organizer20190.8500.8950.872
Stepper23380.9570.5790.721
Temporary35490.8290.5920.690
Walker22281.0000.7860.880
No role324232–––
Micro-average––0.8550.6780.756

Table 5. Human evaluation of the variable role extractor on 660 stratified samples. Distribution columns report the number of extractor predictions and human annotations per role. Precision, recall, and F1 are computed one-vs-rest against human labels, excluding “no role”. Scores characterize the selected subset rather than corpus-wide performance.

Variable Role Extractor Human Evaluation

Annotation protocol. Because Sajaniemi’s roles describe cognitive rather than strictly syntactic properties of variables, we assess the extractor against human annotations. Given the size of the corpus and the highly uneven role distribution, we construct a stratified sample combining extractor positives with two types of negatives.

For each of the 11 roles, we sample 20 variables assigned that role by the extractor and 20 variables not assigned that role but whose names suggest it (e.g., best for most-wanted holder). Rather than defining these semantic priors manually, we identify identifier substrings that best discriminate each role in the extracted corpus, ranked by their F1 score, and use them to sample potential false negatives. This enriches the sample with plausible missed instances of rare roles, such as follower, which would be unlikely to appear in a random sample and would otherwise make recall difficult to assess. We additionally sample 220 variables without any extracted role, yielding 660 distinct variables.

Variables are presented in random order, with their identifier occurrences highlighted in a code excerpt containing all variable occurrences. The annotator assigns one of the 11 roles or “no role”, without access to extractor predictions or sampling strata.

Evaluation. We treat human annotations as reference labels and evaluate the extractor as a multilabel classifier, reporting one-vs-rest precision, recall, and F1 for each role. Table 5 summarizes the sample and human-label distributions, and reports the resulting scores. Because the sample deliberately balances roles and enriches negatives with semantic priors, these unweighted scores characterize the selected subset rather than corpus-wide performance.

Dataset statistics

The consolidated corpus combines the earlier role-enriched dataset with variable records from successful shards of the expanded CodeParrot annotation run. Of 150,000 new candidate files, thirteen shards completed and two failed parsing; failed shards contribute no new annotations. We merge byte-identical sources and identify variables by source hash, kind, name, and declaration coordinates, preserving both releases’ annotation IDs. The corpus contains 126,926 source files, 14,624,580 physical lines, and 3,525,530 unique variables across five languages, retaining the earlier Ruby data. Of these, 1,737,758 have an extracted role and 1,787,772 have none; missing labels do not imply semantic absence of a role.

Five approximately equal file folds are stratified by role and language with source groups kept intact. For the baselines, three folds train, one validates, and one tests, yielding 2,116,300/705,213/704,017 variables (Table 6). The corpus includes 2,428,645 locals, 865,797 parameters, and 231,088 members.

Table 6: Consolidated corpus coverage by partition and language. Role pairs count variable–role memberships; with role is the fraction of variables with at least one extracted role.

PartitionFilesVariablesRole pairs
Train76,1552,116,3001,048,604
Validation25,386705,213349,496
Test25,385704,017349,527
All126,9263,525,5301,747,627
LanguageFilesVariablesWith role
C32,492964,66042.8%
C++31,910913,13849.3%
JavaScript29,673951,28845.1%
Python32,798695,42264.0%
Ruby531,02269.6%

Table 6.

Fixed value contributes 74.4% of the 1,747,627 memberships, while follower and organizer have 724 and 747 variables (Table 7). There are 9,866 multi-role variables; the largest overlap is fixed value with walker, for 5,665 variables.

Table 7: Role support in the consolidated corpus. Percentages use all 3,525,530 variables; test files counts files in each row. Extracted (total) sums variable–role and file–role memberships across rows, not unique variables or files. “No extracted role” is not a prediction target.

RoleTrainValid.TestTotal%Test files
Fixed value780,012259,968260,0071,299,98736.8724,387
Stepper22,2017,4037,40237,0061.054,079
Gatherer2,8779599584,7940.14677
Most-recent holder6,5002,1662,16810,8340.311,300
Most-wanted holder9012983001,4990.04217
One-way flag4,3211,4391,4377,1970.201,132
Follower4351441457240.02125
Walker25,0138,3378,33641,6861.184,250
Container53,23017,74417,73888,7122.527,969
Organizer4481501497470.02127
Temporary152,66650,88850,887254,4417.2215,521
Extracted (total)1,048,604349,496349,5271,747,62749.5759,784
No extracted role1,073,661357,596356,5151,787,77250.7124,348

Table 7.

Integrity and scope. Exact-source matches, normalized whole-file/function matches, and earlier clone links define 122,458 indivisible source groups; none crosses folds. This grouping is not repository holdout or exhaustive semantic deduplication; 16,784 files have parser errors. We union existential boolean flags across duplicates and recover flags for variables with no extracted role from raw facts. The earlier corpus was used in model development; the new split does not erase that exposure.

Metadata and lexical baselines

We use the full consolidated corpus (Appendix § 13), without subsampling. The split has 2,116,300 training, 705,213 validation, and 704,017 test variables. For each of eleven roles, variables lacking that annotation, including those with no extracted role, are negative. We evaluate extractor labels using original identifiers.

Predictors and selection. We use language/kind categories, six general boolean properties, and TF–IDF name character 1–4-grams or local-code token/operator 1–2-grams. Vocabularies retain at most 12,000 name and 30,000 code features, with minimum document frequency 3. Code windows cover up to eight occurrences with 100 source bytes per side; verified target identifiers are masked. The combined baseline concatenates name, code, and boolean features; role-specific metadata and target-derived flags are excluded. For each role, class-balanced L2 logistic regression selects C∈{0.1,1,10}C \in\{0.1, 1, 10\} by validation AP and a threshold by validation F1. Vocabularies and weights are refitted on 2,821,513 train/validation variables; thresholds are unchanged.

Table 8: Macro surface-baseline performance over eleven roles. Brackets: 95% intervals from 500 source-group bootstrap resamples, conditional on fitted models; ±: standard deviation over 500 random draws.

Train/validation frequencies define binary majority (no roles) and most-frequent-role (fixed value) references; always positive predicts every role. Uniform random scores yield positive predictions with probability 0.5 or the train/validation role frequency. AP integrates precision over recall increments at distinct score thresholds; F1 is 2TP/(2TP+FP+FN)2\mathrm{TP}/(2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}) at the frozen threshold. Constant-score AP equals test prevalence; macro averages weight roles equally.

Table 9: Per-role surface predictability on the consolidated corpus. Prior AP is test prevalence; code denotes local code, and combined adds names and boolean properties.

Binary majority attains 95.5% accuracy with zero F1. Name, code, and combined macro AP are 0.142, 0.487, and 0.533 (Table 8–Table 9). These scores measure automatic-label predictability, without independent semantic validation. Probe comparisons require matched subjects, negatives, partitions, and accessible context; these windows can include code after a causal readout.

PredictorMacro APMacro F1
Binary majority0.0450.000
Most frequent role0.0450.049
Always positive0.0450.071
Random (uniform)0.0451 ± 0.00010.0598 ± 0.0001
Random (prior-matched)0.0451 ± 0.00010.0451 ± 0.0004
Language + kind0.100 [0.098, 0.101]0.112 [0.109, 0.114]
Boolean properties0.067 [0.065, 0.069]0.102 [0.099, 0.104]
Variable name0.142 [0.140, 0.146]0.213 [0.207, 0.218]
Local code0.487 [0.475, 0.503]0.526 [0.515, 0.537]
Combined0.533 [0.521, 0.547]0.553 [0.541, 0.564]

Table 8.

RolePrior APName APCode APCombined APCombined F1
Fixed value0.36930.5940.7850.8220.752
Stepper0.01050.3630.6490.6790.690
Gatherer0.00140.0750.4230.4790.535
Most-recent holder0.00310.0110.2220.2230.317
Most-wanted holder0.00040.0560.3880.4580.488
One-way flag0.00200.0520.5160.6650.658
Follower0.00020.0080.0990.1390.205
Walker0.01180.1140.8820.8930.840
Container0.02520.1150.5880.6350.620
Organizer0.00020.0090.2720.2990.378
Temporary0.07230.1700.5380.5690.597

Table 9.

SAE Details

ModelSAE releaseSAE IDLayers
Gemma 2 2Bgemma-scope-2b-pt-res-canonicallayer_{l}/width_16k/canonical6, 13, 17, 22
Gemma 3 4Bgemma-scope-2-4b-pt-reslayer_{l}_width_65k_l0_medium9, 17, 22, 29
Gemma 3 12Bgemma-scope-2-12b-pt-reslayer_{l}_width_65k_l0_medium12, 24, 31, 41
Llama 3.1 8Bllama_scope_lxr_32xl{l}r_32x8, 16, 21, 27
Qwen3.5 9Bqwen-scope-3.5-9b-base-w64k-l100layer{l}8, 16, 21, 27

Table 10. Sparse autoencoder configurations used for each model.

Additional Analyses of Role Representations

On matched pairs, residual probes recover automatic labels more accurately than individual SAE features, and surface cues also predict these labels. We next test how this discrimination changes with target names, language, implementation, and the observation position. These analyses use the fitted predictors and diagnostic readout from Section 4, averaging four layers except in the readout comparison. The principal comparisons use matched-negative training; the broader sweep trains separately on matched, neighboring-role, same-context, and hard-mined negatives and evaluates all six strategies in Appendix § 13.7. For thresholded results, validation fixes each decision threshold by maximizing sensitivity minus false-positive rate. We use Gemma Scope for Gemma 2 2B ([36]), Gemma Scope 2 for Gemma 3 4B and 12B ([40]), Llama Scope for Llama 3.1 8B ([22]), and Qwen-Scope for Qwen3.5 9B ([12]).

Reconstruction and counterfactual controls. To separate reconstruction loss from the restriction to one feature, we also fit ridge probes to SAE reconstructions and to the 64 training-ranked features. Across models, their matched-negative macro AUROC reaches 0.857–0.881 and 0.853–0.878, respectively, compared with 0.906–0.913 for residual probes (Table 2). Thus, combining features closes much of the single-feature gap. We also test 1,698 source-reviewed role-changing counterfactual pairs from 948 source groups, excluding rejected edits and incorrect original-role labels through the LLM review described in Appendix § 13.4. Paired accuracy measures how often an original ranks above its edited negative, assigning ties half credit. Residual paired accuracy is 0.813–0.877; reconstruction probes are 4.0–12.3 percentage points lower, with all five paired 95% intervals excluding zero. SAE reconstruction therefore reduces recoverability of these distinctions under the tested linear fits. Intervals use 1,000 source-group bootstrap draws, with equal role and layer weight, conditional on the fixed split and fits. At the same frozen validation thresholds, residual-probe false-positive rates on edited negatives are 0.684–0.713, with balanced accuracy 0.610–0.637. These decision metrics distinguish sensitivity of score rankings from successful rejection of edited negatives.

Target names contribute to discrimination. To isolate target-name sensitivity, we select 100 Test pairs per role without consulting scores and replace every occurrence of the resolved target binding with a neutral identifier. Both positive and negative variables are renamed, preserving their roles and the surrounding syntax; fitted choices are unchanged. Both name conditions use the same model, SAE, and inference configuration. Residual AUROC decreases by 0.6–1.7 percentage points across models (Figure 7b), remaining at 0.878–0.887 after renaming. Single-feature AUROC is lower, at 0.693–0.760, with changes between −0.8 and +0.1 points. On these same intervention pairs, name surface features fall from AUROC 0.602 to 0.492, whereas code surface features change from 0.740 to 0.723. The residual probe preserves more discrimination than either surface baseline, although its frozen decisions still change for 10.8% of positives and 12.4% of negatives. Its balanced accuracy decreases from 0.830 to 0.809, distinguishing stable ranking from stable thresholded decisions. The edit preserves other identifiers, comments, and implementation cues, so it measures sensitivity to the target name.

Figure 7. Residual probes recover more role information than individual SAE features, with modest sensitivity to neutral target names. (a) Matched-negative Test AUROC for all eleven roles; points average five models and their four prescribed layers, and lines span the five model means. (b) Change in AUROC after renaming both members of 100 positive/negative pairs per role, with fits and thresholds fixed. Intervals are paired 95% source-group bootstrap intervals; negative values indicate lower discrimination after renaming.

Language and implementation condition the measured signal. We supplement Test coverage toward 50 matched pairs per supported role/language and examine metadata-defined realizations (Figure 9). The fits retain their original multilingual training and validation populations, and evaluation covers C, C++, JavaScript and Python. Gatherer and one-way flag have high residual AUROC across all four languages, while stepper ranges from 0.682 in JavaScript to 0.972 in Python. Ranking and thresholded decisions also differ across implementations: single-feature stepper AUROC is similar for for-loop and other updates (0.728 versus 0.757), but recall drops from 0.895 to 0.624 at the validation thresholds. Within Python, its recall also falls from 0.904 to 0.750, on 23 and 30 positives respectively. Most-wanted-holder recall is high for both guarded replacement and min/max selection, especially for residual probes (0.994 and 0.966). Walker differences are entangled with language in the selected subtypes, and member-expression most-recent holders lack a supported diagnostic readout. These distinctions constrain which apparent subtype effects can be attributed to implementation.

Figure 9. Language-conditioned discrimination and implementation recall differ across roles and predictors. (a–b) AUROC averaged over five models and four prescribed layers, targeting 50 matched pairs per role/language. Gray cells show cohorts below 20 pairs or ten positive source groups; a dash marks an unsupported cohort. (c) Recall on metadata-defined positive subsets, with sample counts and 95% source-group bootstrap intervals. Circles denote residual probes and diamonds single SAE features; MWH and MRH abbreviate most-wanted and most-recent holder. Implicit walkers are all Python, whereas 49 of 50 structural walkers are C/C++, so their pooled contrast also changes language.

The observation point determines which edits are visible. Causal attention makes the readout position consequential: an assignment’s identifier can precede the expression that determines its role. At the second prescribed layer, residual paired accuracy on reviewed edits is 0.495–0.594 at the native identifier and 0.856–0.920 at the diagnostic identifier (Figure 8). Statement-end readouts recover much of this sensitivity, with residual paired accuracy of 0.790–0.915 on the unmodified input. These separately fitted readouts measure position-specific decodability: later positions expose source text that can be unavailable at the native identifier, and the diagnostic reference also changes the input. The resulting predictive associations motivate the intervention experiments in § 5.

Figure 8. Later readouts expose counterfactual changes that the native identifier often misses. Reviewed-counterfactual paired accuracy at the second prescribed layer of each model, averaged over all eleven roles. Predictors are fitted separately at each readout with the same training policy and evaluation pairs; the comparison measures readout-specific decodability.

Evaluation populations, review, and selection

The full annotation corpus, representation benchmark, and generation tasks answer different questions. Table 11 identifies their populations and selection units.

ComparisonPopulation and splitSelection and outcome
Corpus baselines3,525,530 variables; three training folds, one validation fold, one Test foldAutomatic annotations; metadata and lexical prediction on the full consolidated corpus.
Representation fits5,321 training and 9,547 validation pairs; source groups separated from TestTraining fixes normalization and feature ranking; validation chooses regularization, single feature, and threshold.
Matched Test4,238 positive/negative pairs after exclusions; four languagesShared examples across predictors; AP and AUROC measure rankings against automatic labels.
Counterfactual Test1,698 accepted pairs from 948 source groupsStatic source review of original and edited behavior; paired ranking and frozen-threshold decisions.
Neutral-name Test100 positive/negative pairs per roleBoth bindings renamed, with fits fixed; paired change in AUROC and binary decisions.
Feature-scale steeringUnion of 121 MBPP tasks; role-specific cohortsEleven SAE targets each for Gemma 2B/4B and nine for 12B; ten same-layer random features per target.
Normalized steeringTwelve calibration and twelve Test tasks per role; eleven completed model–role cellsShared probe/SAE dose under a correctness constraint; random residual and SAE controls.

Table 11. Evaluation populations and fitted choices. Pair and target–task counts can reuse bindings or tasks; uncertainty resamples the corresponding source groups or unique tasks. The separate Gemma 12B stepper pilot uses 48 calibration and 144 Test tasks (Appendix § 14.2).

Frozen representation cohort. The representation experiments retain an earlier prepared cohort and its source/clone-group assignments; the expanded corpus’s full-variable baselines are a separate evaluation. One rotation assigns folds 2/3/4 to training, fold 1 to validation, and fold 0 to Test. Training and validation retain their original multilingual populations. The original matched Test cohort contains 4,245 pairs; excluding Ruby pairs and the two source-role labels rejected by subsequent review leaves 4,238. The common tokenization support uses a 4,096-token ceiling. All methods and model–SAE systems share eligible pairs within a comparison; the four prescribed layers are averaged without Test-based layer selection.

Negative selection. Matched negatives share the positive’s partition, programming language, and declaration kind, and lack the target annotation. Retrieval ranks candidates using standardized context length, occurrence count, name length and shape, counts of loops, branches, assignments and calls, parser status, and identifier spelling similarity. Scaling is fitted on training subjects; candidates from the positive’s source group are excluded, with limits on negative reuse. Neighboring-role selection additionally requires a specified competing role; same-context selection chooses another resolved binding in the enclosing context. Hard-negative selection uses a logistic classifier on target-masked code, fitted with source-group-held-out folds within training. The broader sweep varies the training-negative policy, retaining matched-negative validation; the main comparisons use matched training throughout.

Predictor fitting. Residual, reconstruction, and top-64 probes use ridge regression with training-only centering and coordinate scaling. The coordinate standard-deviation floor is 10−610^{-6}, and standardized vectors are divided by the square root of their dimension. Validation AUROC selects the penalty from {10−6,10−4,10−2,1,100}\{10^{-6}, 10^{-4}, 10^{-2}, 1, 100\}. Training AUROC ranks SAE coordinates by max(AUROC⁡\operatorname{AUROC}, 1−AUROC⁡1-\operatorname{AUROC}) and fixes their predictive polarity. Validation selects one of the top 64 coordinates for the single-feature predictor; the multi-feature probe uses those same 64 candidates. For each predictor, validation selects the threshold maximizing sensitivity minus false-positive rate. The matched name and visible-prefix baselines fit TF–IDF/logistic models on the same training pairs and use validation for regularization and threshold selection. Name features use character 2–5-grams and a 12,000-feature cap; code prefixes use 3–5-grams and a 30,000-feature cap. Both use minimum document frequency two, case-sensitive sublinear term frequencies, and validation-selected C∈{0.1,1,10}C \in\{0.1, 1, 10\}. The code-prefix input ends at the diagnostic identifier; the full-corpus lexical windows in Appendix F use a different context policy.

Counterfactual review and exclusions. Counterfactuals are generated from eligible source patterns and reviewed through LLM assessments of the enclosing binding context and exact edit. Review records specify whether the original has the target role, whether the edit removes it, source-level validity, and assumptions about external inputs or calls. The rubric asks reviewers to inspect the binding lifetime and all writes, including mutation and aliasing, and to flag unresolved dependencies or role ambiguity. Source annotations and proposed edits are supplied as fallible evidence; the judgments are not blinded to those proposals. The final review uses GPT-5.6 Terra source reviewers to resolve 333 previously pending cases: 330 edits are accepted and three rejected. One rejected edit preserves the target role; two cases have incorrect original-role labels. Those two source-role labels are excluded from affected test panels; training and validation fits are unchanged. The resulting 1,698 accepted pairs include earlier reviewed cases and the final accepted additions after evaluation exclusions. These static LLM judgments are distinct from independent human adjudication or execution-based verification; the acceptance fraction of the selected queue is not an estimate of corpus-wide annotation accuracy. Review uncertainty and inter-annotator agreement are unmeasured.

Ranking, decisions, and uncertainty. Paired counterfactual accuracy is E[1(soriginal>sedit)+121(soriginal=sedit)]\mathbb{E}[1(s_{\mathrm{original}} > s_{\mathrm{edit}}) + \frac{1}{2}1(s_{\mathrm{original}} = s_{\mathrm{edit}})]. AUROC instead compares all positive/negative score combinations within a role–layer cell. Balanced accuracy is the mean of positive recall and negative recall at the frozen threshold. Model summaries average roles and layers equally. Paired 95% intervals use 1,000 joint source-group bootstrap resamples, retaining related examples and methods together; they condition on the split, fits, and review decisions. Layer ranges in Figure 3 and random-feature ranges in Figure 6 describe heterogeneity, not sampling intervals.

Neutral names and readout controls. Test-pair selection does not consult predictor scores. Neutral identifiers are derived from a role-independent binding hash; edits replace all annotated occurrences of the resolved binding, with source hashes, syntax shape, and non-target leaf bytes checked. Original and renamed comparisons use matched execution settings; stored scores are reused only after overlap checks. Readout controls fit separate predictors at the native identifier, statement end, and inserted identifier at each model’s second prescribed layer. They compare position-specific decodability, with differences in accessible context and, for the inserted reference, input text.

Steering selection and provenance. The representative features in Figure 6 are a descriptive selection from the larger feature-scale experiment. The archived selection does not establish that these five representatives were chosen before intervention outcomes were inspected; aggregate conclusions therefore use all selected targets within each protocol. Addition strength is calibrated on annotation induction for the feature-scale experiments, while the normalized experiments calibrate a joint induction-and-passing objective subject to a correctness constraint. Appendices § 13.9 and § 14.2 report their different perturbation scopes, task overlap, denominators, and unresolved historical scale provenance.

Behavioral role recognition and prompt sensitivity

We evaluate Llama 3.2 1B/3B and 3.1 8B, Gemma 3 1B/4B/12B/27B, Qwen3 1.7B/4B/8B/14B Base, and Qwen3.5 9B Base. These twelve pretrained checkpoints share the protocol but vary in size, architecture, and training.

Evaluation and scoring. The same 9,553 matched positive/negative development pairs cover eleven roles and five languages: 17,770 distinct role/variable queries from 16,837 variables and 6,996 source groups. Labels are automatic annotations; a negative lacks the target annotation. Prompts include the role definition, full enclosing code with comments, and target identifier and location, asking for a judgment from initialization, updates, and use while allowing other roles to apply. Each role has one fixed training-only demonstration set (seed 17), with nested, class-balanced prefixes of 0, 2, 4, or 8 examples shared across models. Inference uses BF16 without quantization, chat templates, or source truncation. At the final Answer: position, s=log⁡P(Yes)−log⁡P(No)s = \log P(\mathrm{Yes}) - \log P(\mathrm{No}) predicts positive when s≥0s \ge0; average precision (AP) ranks the unrounded scores. We restore original pair multiplicities and average roles equally: constant predictions attain 50% accuracy and constant scores attain 50% AP.

Recognition across checkpoints. Gemma 3 27B has the highest eight-shot macro accuracy/AP (72.1/77.2%; Figure 10); its accuracy exceeds Gemma 3 12B by 5.8 points (paired 95% interval: 4.6–6.9). Qwen3 14B reaches 68.1/72.6%, exceeding Qwen3 8B accuracy by 8.7 points (7.5–9.9). Eight-shot accuracy rises with tested size within each family, but demonstration gains vary: the eight-minus-zero accuracy intervals include zero for Qwen3 1.7B/4B/8B, and their AP does not improve. Llama 3.2 1B gains 1.0 accuracy point (0.3–1.7), with accuracy near chance.

Figure 10. Demonstration gains differ across checkpoints. Top and middle: role-macro accuracy and AP on identical development pairs; gray lines mark 50%. Bottom: eight-minus-zero accuracy gains with paired 95% source-group bootstrap intervals (2,000 draws), conditional on the cohort, pair weights, and one demonstration set. Intervals are exploratory and unadjusted for multiple comparisons. Model labels specify version and nominal size within each family.

Taxonomy-free descriptions. Prompts request at most three sentences on initialization, updates, and use, without role names or examples. Training-only format calibration reduces descriptions reaching the generation cap from 27/66 to 6/66 across three checkpoints; we fix this format for all models. Cap rates range from 0% to 78.2% across the expanded model set. We assess fidelity qualitatively: Gemma 3 27B invents a prior None initialization for exit_code, while Qwen3 14B identifies the shown assignment.

Role differences and free generation

Role and answer dependence. Fixed value, most-recent holder, and temporary are weaker than one-way flag and most-wanted holder (Figure 11). Greedy generation evaluates 100 distinct development cases per role (50 positive, 50 negative), shared across models: 1,100 cases and 39,600 outputs across descriptions and zero/eight-shot answers. The fixed seeded panel includes the original six cases per role; cases receive equal weight and share source groups. Leading Yes/No answers are valid; invalid outputs count as incorrect, with a 512-new-token budget. Three Llama 3.1 8B shards use A100/GB10 GPUs; the others use H200s, with matching software.

Figure 11. Role accuracy varies within these aggregate trends. Each checkpoint has four columns for 0/2/4/8 demonstrations: all 528 combinations, with original pair weights and a shared scale. L/G/Q denote Llama/Gemma/Qwen; numerals specify version and size.

Eight-shot answer validity is 97.6–100%, yet 95.5–100% of outputs reach the cap (Figure 12). Gemma 3 27B reaches 73.2% free-answer accuracy. Qwen3 Base checkpoints yield 99.8–100% valid zero-shot starts, compared with 6/1,100 for Qwen3.5 9B despite its 70.1% zero-shot AP under constrained scoring.

Figure 12. Valid answer beginnings coexist with frequent budget exhaustion. Top: all 264 role/model/shot accuracies, each on 100 cases. Bottom: aggregate percentages over 1,100 cases per checkpoint; columns give demonstration counts. Invalid answers count as errors; description caps measure stopping behavior, not semantic accuracy.

Extended related work

StudyAnalysis and targetKey finding
Behavioral robustness and semantic priors
Yang et al. [61]Input edits: semantics-preserving namesPlausible renaming induces failures; adversarial training improves robustness.
Gao et al. [16]Input counterfactuals: namesNames provide semantics and shortcuts; counterfactual inference improves robustness.
Wang et al. [58]Input edits: ReCodeGeneration degrades under identifier, syntax, formatting, and docstring perturbations.
Ahmed et al. [2]†Input edits: masked codeMasked-operator predictions reflect code edits beyond frequency effects.
Hooda et al. [25]Input counterfactuals: predicatesCounterfactual code edits expose inconsistent handling of control/data flow and names.
Li et al. [34]Token attribution: project biasRegularization improves cross-project generalization and adversarial robustness.
Wang et al. [59]Input edits: random/misleading namesBoth harm code search and clone detection; misleading names are often worse.
Orvalho & Kwiatkowska [43]†Input edits: equivalent programsMutations degrade output prediction; correct answers can accompany flawed explanations.
Liang et al. [35]†Input edits: test-specified behaviorTest-specified rule switching is incomplete, even for models that implement both rules when explicitly instructed.
Programming concepts in internal representations
Karmakar & Robbes [32]Probes: code propertiesEnglish-pretrained BERT is competitive with code-pretrained models on some probes.
Karmakar & Robbes [33]Probes: INSPECTFifteen probes; structure-aware models often score higher, with task-dependent rankings.
Wan et al. [57]Probes/attention: program structureHidden states and attention encode recoverable syntactic relations.
Hernández López et al. [24]Structural probes: ASTsFull ASTs recoverable from low-dimensional subspaces; middle layers most informative.
Troshin & Chirkova [55]Probes: syntax, names, semanticsSyntax and naming properties are decodable; semantic equivalence is harder.
Ma et al. [39]Probes: syntax and flow graphsSyntax broadly recoverable; control/data dependencies vary across models.
Sharma et al. [51]Attention/pooling: identifiersAttention favors identifiers and separators; identifier pooling improves clone detection.
Haji Mohammadkhani et al. [20]Attention: downstream code tasksToken-category attention varies across tasks; models can fail on simple programs.
Anand et al. [3]Probes: token relationsSyntax–identifier links are poorly recovered; larger tested models often trail smaller ones.
Jin & Rinard [30]Probes: synthetic execution statesIntermediate states are decodable without supervision on execution traces.
Sharma et al. [49]Neuron selection: token conceptsSmall neuron subsets preserve probe accuracy; selected neurons align with code concepts.
Haider et al. [19]MLP-key inspection/maskingMasking API-associated keys reduces targeted predictions, with smaller effects on other tokens.
Sharma et al. [50]†Clustering: CoCoA conceptsLexical, syntactic, and semantic clusters; lexical/subword structure persists in deeper layers.
StudyAnalysis and targetKey finding
Shared representations across programming and natural languages
Kargaran et al. [31]Decoding/ablation: shared neuronsIntermediate states and neurons share programming-language and English information.
North et al. [42]Probes/attention: variable typesShared attention heads convey type information across three programming languages.
Yin et al. [62]Probes, anonymization, neuron ablationLanguage-specific neurons coexist with more invariant middle-layer representations.
Naganna et al. [41]†Patching: arithmetic across formatsShared neurons across symbols, text, and Python; cross-format patches rescue selected errors.
Causal and mechanistic analyses
Heimersheim & Janiak [23]†Circuit tracing: Python docstringsCompositional argument-copying circuit explains part of a small attention-only model’s behavior.
Conmy et al. [9]Automated edge selection: ACDCRecovers task-relevant subgraphs, including the docstring-copying setting.
Templeton et al. [54]†SAE clamping: code featuresFeature clamping induces insecure code, backdoors, or false bug reports in selected examples.
Lucchetti & Guha [37]Activation steering: type predictionRepairs induced failures; middle-layer effects and Python–TypeScript transfer vary by edit.
Rai et al. [46]Circuit analysis: parenthesesFaulty components override sound computations on synthetic parentheses tasks.
Tahimic & Cheng [53]†SAE readouts/steering: correctnessSeparate features predict errors and steer generation; steering can repair failures or corrupt correct outputs.
He et al. [21]†Attribution graphs/interventionGraphs predict correctness; a binary-search case demonstrates causal repair.
Abdelsalam et al. [1]†Patching: misleading names/commentsTraces cue-induced errors through selected residual-stream states in curated conflict cases.
Chen et al. [8]†Probes/context-stripped decodingIn short Python tasks, probe recovery can precede context-stripped decoding; later layers can lose correct answers.
Huang et al. [26]Probes/training/steering: securityIntermediate-layer vulnerability signals support improved secure generation with combined interventions.
Explanation methods and human comparison
Chen et al. [7]Input perturbations: DeciXExplanation accuracy, compactness, and stability improve over evaluated baselines.
Paltenghi et al. [44]Attention/gaze: code explorationProcessed follow-up attention captures aspects of developers’ code-reading transitions.

Table 12. Extended comparison of work on code-language-model behavior and representations. Groups follow § 6, with a final group for explanation methods and human comparison; studies are placed by their primary emphasis. The analysis column distinguishes input perturbations, representation readouts, and activation interventions. AST: abstract syntax tree; SAE: sparse autoencoder. †: cited as a preprint or technical note.

Selecting and generating negative examples

Figure 13. Strategies for selecting or creating negative examples, illustrated for the stepper role. Bold gold marks targets; headers give their roles. Selected negatives lack the target annotation, counterfactuals change the role, and renaming preserves it. Illustrative print readouts (+) immediately follow the statement or loop header containing the occurrence of interest, making that text available to causal attention. Δ marks edits, ellipses omit lines, and arrows mark wrapping. Examples come from annotated Python training data.

Figure 14. Strategies for selecting or creating negative examples, illustrated for the gatherer role. The six strategies vary the queried binding, context, update, or name. Renaming preserves the role and all non-target code; donor and recipient role sets are disjoint and their source groups differ. The miner uses target-masked code and logistic regression in three source-group-held-out folds within training. Added print readouts, highlighting and edit markers follow Figure 13.

Three code examples showing different realizations of the most-wanted holder role
best = None
for x in values:
    if valid(x):
        if best is None or
           score(x) > score(best):
            best = x

(a) Guarded MWH

int min_val = a[0];

for (int i = 1; i < n; ++i)
{
    if (a[i] < min_val)
        min_val = a[i];
}

(b) Min/max MWH

let bestUser = null;
let bestScore = -Infinity;

for (const user of users) {
    const s = rank(user);
    if (s > bestScore) {
        bestScore = s;
        bestUser = user;
    }
}

(c) MWH with companion

Figure 15. Different realizations of the most-wanted holder role. A guarded MWH updates only when an additional eligibility condition holds; a min/max MWH retains the extremal value observed so far; an MWH with a companion variable stores the selected object while another variable tracks the selection criterion.

Two code examples showing different realizations of the one-way flag role
boolean found = false;

for (Item x : items) {
    if (matches(x)) {
        found = true;
        break;
    }
}

(a) Direct terminal assignment

valid = False

for x in items:
    valid = valid or check(x)

(b) Monotone update

Figure 16. Different realizations of the one-way flag role. The direct form uses a terminal assignment, typically inside a branch. The monotone form repeatedly combines the previous state with a new condition. Both allow state changes in only one direction.

Different realizations of variable roles

for (int i = 0; i < n; ++i) {
    process(data[i]);
}

(a) (a) Numeric loop iterator

count = 0
for elt in arr:
    count += 1

(b) (b) Predictive accumulator

Figure 17. Different realizations of the stepper role. A numeric iterator follows a regular progression controlling loop iteration. A predictive accumulator advances by a regular step to represent the position or state expected at the next iteration.

for item in records:
    consume(item)

(a) (a) Implicit iterator

Node *node = head;
while (node != NULL) {
    visit(node);
    node = node->next;
}

(b) (b) Structural iterator

Figure 18. Different realizations of the walker role. An implicit iterator takes successive elements exposed by the loop construct. A structural iterator follows explicit links in the data structure.

let current = start;
let previous = null;

while (current) {
    current = current.next;
    previous = current;
}

(a) (a) Assigned after master change

Node* current = head;
Node* previous = nullptr;

while (current) {
    previous = current;
    current = current->next;
}

(b) (b) Assigned before master change

Figure 19. Different realizations of the follower role. A follower derives its value from a changing master variable. Assignment order determines whether it captures the master’s value before or after a transition.

last_member = nil

group.each do |member|
  last_member = member
end

(a) (a) Member holder

latest = None
for event in events:
    if event.accepted:
        latest = event

(b) (b) Loop-updated holder

Figure 20. Different realizations of the most-recent holder role. A member holder receives successive collection elements. A loop-updated holder records the latest value satisfying an update condition. The stored value is replaced at every element or only at selected events, respectively.

Matched probe and SAE steering

We compare probe and SAE directions under the same intervention budget, alongside random residual and random SAE directions. The exploratory panel covers stepper, gatherer, most-wanted holder, and walker in Gemma 2 2B (layer 13), Gemma 3 12B (layer 24), and Llama 3.1 8B (layer 16). Each role has twelve calibration tasks and twelve Test tasks, balanced by role presence in the reference solution and shared across models. These are intact batches selected from the original MBPP cohort without consulting generation outcomes. Eleven model–role cells pass the generation and evaluation checks; Llama walker is excluded because its zero-dose output differs from the unsteered baseline.

A common budget and calibration criterion. We add each normalized direction at generated-token residual positions in one layer, with magnitudes of 1%, 3%, or 10% of the median calibration residual norm. The zero-strength option is also eligible. Probe and SAE share the strength maximizing mean calibration induction accompanied by passing tests, subject to each losing at most five percentage points of correctness; ties favor the smaller strength. The selected strength is fixed for Test and applied to both random controls. With twelve calibration tasks, the correctness constraint permits no net loss of a passing program. All conditions retain the same four-task batches, prompts, and greedy 256-token budget, and the zero-dose control must reproduce baseline tokens exactly. The completed comparison contains 3,696 evaluated generations.

Role induction often accompanies test failure. Gemma 2 2B most-wanted holder selects the 10% dose (Figure 21). Its probe and SAE induction rates are 3/5 and 4/5, compared with 1/5 for the random residual direction and 0/5 for the random SAE direction. Induction accompanied by passing tests is 1/5 for the probe, SAE, and random residual direction, and 0/5 for the random SAE direction. Overall Test correctness remains 4/12 for both target directions and falls to 3/12 for both controls. The remaining ten cells select zero because nonzero strengths provide no better feasible calibration objective. This result distinguishes the ability to alter role annotations from reliable control under a correctness constraint. These estimates concern positive steering at one layer in small, overlapping task cohorts conditioned on baseline role presence.

Figure 21. Correctness-constrained calibration usually selects no intervention. (a) Ten of eleven completed cells select zero; the gray cell is unavailable after a failed zero-dose check. (b) In the sole nonzero cell, probe and SAE directions induce most-wanted holder on three and four of five eligible Test tasks, respectively, but each produces only one induction that passes the supplied tests. Bars include random controls on the same tasks; fractions show the small conditional denominators.

Activation contexts of role-associated SAE features

The displays below cover all eleven roles with validation-selected single-feature predictors from Gemma 3 4B layers 9 and 17. Role colors and symbols follow Figure 2. Each row gives a Neuronpedia description, corpus code (left and “Also”), and public prose (right). Highlight intensity is linear from zero to the displayed bound, fixed per latent; compare magnitudes within a latent. “Lower predicts role” identifies negative polarity in the original probe selection, so strong highlighting need not occur on the target variable.

Figure 22. Selected features for progression, traversal, retention, and fixed values. Row titles name the role and corpus variable; 4B denotes Gemma 3 4B. Follower and most-recent holder share feature 34976 with opposite predictive signs. Return marks denote newline tokens, and “Also” introduces a separate corpus context.

All ten distinct latents were measured on 41 hand-selected training snippets. Code crops retain their standalone-input activations; public windows retain their original Neuronpedia contexts. These examples illustrate activity, not prevalence.

Descriptions and activation contexts (continued)

Figure 23. Extrema, accumulation, flags, collections, and intermediate computations. The displays continue Figure 22 with the same conventions and per-latent scales. Descriptions can name broad operations or disparate contexts, rather than the variable role itself.

Matched and reviewed-counterfactual AUROCs are 0.848/0.474 for most-wanted holder, 0.949/0.354 for one-way flag, and 0.764/0.536 for stepper. Contextual activity alone does not establish role specificity.

Qualitative outcomes of SAE steering

The larger Gemma 2 2B and Gemma 3 4B experiments from § 5 also permit inspection of complete generated programs. Across 292 evaluated target–task pairs, eight failing-to-passing transitions correspond to six distinct model–task pairs. They include most-frequent-item search, counting sort, largest row sum, cuboid volume, product divided by length, and maximum subarray sum. We show selected repairs and failures alongside an unfiltered seeded random sample in Figure 5. All displayed test outcomes were reproduced on the complete programs using the three supplied assertions.

The selected repairs can reveal useful changes, such as replacing a stub with an explicit search, without establishing that the targeted role caused the improvement. In the broader gallery, cuboid and product repairs pass all three tests while retaining a zero target-role annotation. Conversely, an unchanged correct program may already implement the role semantically even when its original annotation is absent. The six random cases drawn across the full gallery all fail the supplied tests; the draw was not repeated to obtain favorable examples. These observations motivate reading role annotations, complete code, and correctness together.

Consolidated quantitative analysis of causal interventions

We connect the selected-feature results in Figure 6 and the structural changes in Table 3 to all tested SAE features and the normalized probe/SAE experiments. The two displays summarize complementary measurements from the same Gemma 2 2B and Gemma 3 4B causal runs. The analysis adds the completed Gemma 3 12B feature-scale run, a five-fit Gemma 3 12B stepper pilot, and the eleven model–role cells in Appendix § 13.9.

Figure 24. Annotation changes substantially exceed changes accompanied by passing tests. (a) Role disappearance for removal and induction for addition, with the joint passing outcome on the same baseline-defined cohort. (b) Joint outcome minus the matched random-control mean: random residual directions for probes and random SAE directions for SAE targets. (c) Intervention-minus-baseline correctness on that cohort. Rates weight target–task enrollments within each protocol; 95% intervals resample unique tasks 4,000 times, retaining all associated features, fits, and controls together. Normalized addition averages include selected zero doses; the stepper pilot uses Gemma 3 12B. The three protocol groups use different cohorts and perturbation scales.

Across all historical features, removal changes 41/91, 72/109, and 131/187 target–task outcomes for Gemma 2 2B, Gemma 3 4B, and Gemma 3 12B, respectively. Only 2/91, 4/109, and 15/187 also pass the supplied tests; requiring both outputs to pass leaves 0, 3, and 13 outcomes. The joint outcome exceeds matched random controls by only 1.3, 1.4, and 3.7 percentage points for removal, and 1.9, 1.3, and 0.9 points for addition; all six corresponding intervals include zero. These enrollment counts include repeated tasks across features and are not independent sample sizes. The removal cohorts contain 32, 32, and 63 unique tasks, respectively; reference-positive addition uses 53, 49, and 45 unique Test tasks.

Comparison units, calibration, and structural survival

Protocol-specific comparisons. We test eleven SAE directions in Gemma 2 2B and Gemma 3 4B and nine in Gemma 3 12B, covering five roles. They share a union of 121 MBPP tasks with role-specific enrollment; removal uses both splits, while induction uses Test only. Six positive multipliers are calibrated for annotation induction, with ten same-layer random SAE controls per target. The baseline passing counts are 33/121, 14/121, and 67/121. The archived feature scales are not normalized across models, and twenty of the shared package’s twenty-two scales differ from the corresponding selection artifact’s training-positive maximum. They are treated as executed settings with unresolved scale provenance. Consequently, these results describe the recorded interventions without attributing cross-model differences to model size or a common dose.

The normalized stepper pilot uses Gemma 3 12B layer 31, five fitted probes, and the same SAE feature in every fit. It has 48 calibration and 144 Test tasks, including 43 eligible removal and 53 induction tasks. Four fits select zero and one selects a 1% residual-norm dose under the correctness constraint. In that nonzero fit, probe and SAE directions each induce stepper on 2/53 tasks; two probe outputs and one SAE output pass, including one probe change that preserves baseline correctness. Repeated SAE outcomes and overlapping probe training sets are not independent replications. Local projection affects one generated-token layer; global projection also alters residual writers and prompt processing. The smaller extension in Appendix N instead covers four roles in twelve-task splits and selects zero in ten of eleven completed cells.

Figure 25. Large structural shifts can describe a small surviving subset. Values reproduce Table 3 and extend it to Gemma 12B: syntax-frequency changes in percentage points among pairs with two clean Tree-sitter parses. Fractions show retained clean pairs over all target–task pairs. Markers follow Table 3; addition combines Test sufficiency and reference-negative cohorts. Gray cells have no eligible removal cohort.

Syntax and correctness condition different subsets. For Gemma 3 4B removal, gatherer syntax uses 4/30 pairs from two distinct tasks, and stepper syntax uses 11/48 pairs from five tasks. The reported shifts therefore describe parseable survivors, not all interventions. Among the three clean stepper pairs where both outputs pass, the range-call frequency does not change.

Scope of inference. Within each protocol, target and control outcomes use the same eligible tasks and retained denominator. The joint metric requires the intervention output to pass: preserving an already correct program additionally requires baseline correctness. Intervals condition on selected features, fitted probes, calibration, and sampled controls; all-zero samples can yield degenerate intervals. The evaluated records share twelve task IDs between historical runs and the smaller extension, four between the pilot and extension, and none between historical runs and the pilot. No effect is pooled across protocols. Finite, prompt-visible tests, generated-code truncation, and missing historical zero-dose/batch-parity audits limit claims of semantic control.

References

  1. [1]Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer, Marvin Wyrich, and Sven Apel. A Mechanistic Lens on Semantic Conflicts: Using Activation Patching to Understand LLM Behavior, July 2026. URL http://arxiv.org/abs/2607.05587. arXiv:2607.05587 [cs.SE].
  2. [2]Toufique Ahmed, Dian Yu, Chengxuan Huang, Cathy Wang, Prem Devanbu, and Kenji Sagae. Towards understanding what code language models learned. arXiv preprint arXiv:2306.11943, 2023. URL https://arxiv.org/abs/2306.11943.
  3. [3]Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini. A critical study of what code-LLMs (do not) learn. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 15869–15889, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.939. URL https://aclanthology.org/2024.findings-acl.939/.
  4. [5]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  5. [6]Craig Bishop and Colin G. Johnson. Assessing roles of variables by program analysis. In Proceedings of the Fifth Annual Finnish/Baltic Sea Conference on Computer Science Education. University of Joensuu, 2005. URL https://kar.kent.ac.uk/14232/.
  6. [7]Simin Chen, Zexin Li, Wei Yang, and Cong Liu. DeciX: Explain deep learning based code generation applications. Proceedings of the ACM on Software Engineering, 1(FSE), 2024. doi: 10.1145/3660814.DOI
  7. [8]Siyue Chen, Yifu Guo, Yuquan Lu, Zishan Xu, Jiaye Lin, Jianbo Lin, Siyu Zhang, Cheng Yang, Junxin Li, Yujia Li, Yu Huo, and Ruixuan Wang. From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs, August 2026. URL http://arxiv.org/abs/2606.17648. arXiv:2606.17648 [cs.AI].
  8. [9]Arthur Conmy, Augustine Mayor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards Automated Circuit Discovery for Mechanistic Interpretability. In Advances in Neural Information Processing Systems, volume 36, pp. 16318–16352. Curran Associates, Inc., 2023. doi: 10.52202/075280-0719. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html.DOI
  9. [11]Yulia Demyanova, Helmut Veith, and Florian Zuleger. On the concept of variable roles and its use in software analysis. In 2013 Formal Methods in Computer-Aided Design, pp. 226–230. IEEE, 2013. doi: 10.1109/FMCAD.2013.6679414.DOI
  10. [12]Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, Tianhao Li, Qian Cao, Xuancheng Ren, Xiaodong Deng, An Yang, Fei Huang, Dayiheng Liu, and Jingren Zhou. Qwen-scope: Turning sparse features into development tools for large language models. arXiv preprint arXiv:2605.11887, 2026. URL https://arxiv.org/abs/2605.11887.
  11. [13]Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. A survey on code generation with LLM-based agents. arXiv preprint arXiv:2508.00083, 2025. URL https://arxiv.org/abs/2508.00083.
  12. [14]Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do I know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.14257, 2024. URL https://arxiv.org/abs/2411.14257.
  13. [15]Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. URL https://arxiv.org/abs/2406.04093.
  14. [16]Shuzheng Gao, Cuiyun Gao, Chaozheng Wang, Jun Sun, David Lo, and Yue Yu. Two Sides of the Same Coin: Exploiting the Impact of Identifiers in Neural Code Comprehension. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1933–1945, Melbourne, Australia, May 2023. IEEE. doi: 10.1109/ICSE48619.2023.00164. URL https://ieeexplore.ieee.org/document/10172869/.DOI
  15. [17]Petri Gerdt and Jorma Sajaniemi. An approach to automatic detection of variable roles in program animation. In Proceedings of the Third Program Visualization Workshop, pp. 86–93. University of Warwick, 2004.
  16. [18]Petri Gerdt and Jorma Sajaniemi. A web-based service for the automatic detection of roles of variables. In Proceedings of the 11th Annual Conference on Innovation and Technology in Computer Science Education, pp. 178–182. ACM, 2006. doi: 10.1145/1140123.1140172.DOI
  17. [19]Muhammad Umair Haider, Umar Farooq, A. B. Siddique, and Mark Marron. Looking into Black Box Code Language Models. In 2026 International Conference on Semantic Computing (ICSC), pp. 32–39. IEEE, 2026. doi: 10.1109/ICSC67292.2026.00011. URL https://ieeexplore.ieee.org/document/11486438/.DOI
  18. [20]Ahmad Haji Mohammadkhani, Chakkrit Tantithamthavorn, and Hadi Hemmati. Explaining Transformer-based Code Models: What Do They Know? When They Do Not Work? In 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM), pp. 96–106, Bogotá, Colombia, October 2023. IEEE. doi: 10.1109/SCAM59687.2023.00020. URL https://ieeexplore.ieee.org/document/10356671/.
  19. [21]Yicheng He, Zheng Zhao, Kaiyu Zhou, Bryan Dai, Jie Fu, and Yonghui Yang. Codeircuit: Toward inferring llm-generated code correctness via attribution graphs. arXiv preprint arXiv:2602.07080, 2026. URL https://arxiv.org/abs/2602.07080.
  20. [22]Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526, 2024. URL https://arxiv.org/abs/2410.20526.
  21. [23]Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer, 2023. URL https://www.alignmentforum.org/posts/6UKXXmKFXbFWzOAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only. AI Alignment Forum technical note.
  22. [24]José Antonio Hernández López, Martin Weyssow, Jesús Sánchez Cuadrado, and Houari Sahraoui. Ast-probe: Recovering abstract syntax trees from hidden representations of pre-trained language models. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022. doi: 10.1145/3551349.3556900.
  23. [25]Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates. In Proceedings of the 41st International Conference on Machine Learning, pp. 18738–18748. PMLR, 2024. URL https://proceedings.mlr.press/v235/hooda24a.html.arxiv.org/abs/2402.05980
  24. [26]Li Huang, Zhongxin Liu, Yifan Wu, Tao Yin, Dong Li, Jichao Bi, Nankun Mu, Hongyu Zhang, and Meng Yan. DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19793–19813, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-long.907. URL https://aclanthology.org/2026.acl-long.907/.DOI
  25. [27]Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK.arxiv.org/abs/2309.08600
  26. [28]Aftab Hussain, Md Rafiqul Islam Rabin, Bowen Xu, David Lo, and Mohammad Amin Alipour. A study of variable-role-based feature enrichment in neural models of code. In 2023 IEEE/ACM International Workshop on Interpretability and Robustness in Neural Software Engineering (IntESE), pp. 14–20. IEEE, 2023. doi: 10.1109/IntESE59150.2023.00007.arxiv.org/abs/2303.04942
  27. [29]Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515, 2024. URL https://arxiv.org/abs/2406.00515.
  28. [30]Charles Jin and Martin Rinard. Emergent representations of program semantics in language models trained on programs. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024. URL https://arxiv.org/abs/2305.11169.
  29. [31]Amir Hossein Kargaran, Yihong Liu, François Yvon, and Hinrich Schütze. How programming concepts and neurons are shared in code language models. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. URL https://aclanthology.org/2025.findings-acl.1379/.
  30. [32]A. Karmakar and R. Robbes. What do pre-trained code models know about code? In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering, 2021. URL https://arxiv.org/abs/2108.11308.
  31. [33]Anjan Karmakar and Romain Robbes. INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers. IEEE Transactions on Software Engineering, 50(2):220–238, February 2024. doi: 10.1109/TSE.2023.3341624. URL https://ieeexplore.ieee.org/document/10354028/.DOI
  32. [34]Zhiming Li, Yanzhou Li, Tianlin Li, Mengnan Du, Bozhi Wu, Yushi Cao, Junzhe Jiang, and Yang Liu. Unveiling Project-Specific Bias in Neural Code Models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 17205–17216, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main.1494/.arxiv.org/abs/2201.07381
  33. [35]Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, and Shiwen Ni. Do Code Language Models Follow Tests? Paired Interventions on Program Behavior, September 2026. URL http://arxiv.org/abs/2607.26244. arXiv:2607.26244 [cs.SE].
  34. [36]Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th Black-boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 278–300. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.blackboxnlp-1.19. URL https://aclanthology.org/2024.blackboxnlp-1.19/.DOI
  35. [37]Francesca Lucchetti and Arjun Guha. Understanding How CodeLLMs (Mis)Predict Types with Activation Steering. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 358–397. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.blackboxnlp-1.22. URL https://aclanthology.org/2025.blackboxnlp-1.22/.DOI
  36. [38]Andrew Kwok Fai Lui, Tsz Tik Lui, Tommy Cheuk Hin Leung, and Ho Yin Chan. Recognition of roles of variables based on deep learning technologies. In Technology in Education. Innovations for Online Teaching and Learning, volume 1302 of Communications in Computer and Information Science, pp. 362–374. Springer, 2020. doi: 10.1007/978-981-33-4594-2_30.DOI
  37. [39]Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhan Wang, Qiang Hu, Jie Zhang, and Yang Liu. Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities. ACM Transactions on Software Engineering and Methodology, 33(7):1–29, September 2024. doi: 10.1145/3664606. URL https://dl.acm.org/doi/10.1145/3664606.DOI
  38. [40]Callum McDougall, Arthur Conmy, Janos Kramar, Tom Lieberum, Senthooran Rajamanoharan, and Neel Nanda. Gemma scope 2: Technical paper. Technical report, Google DeepMind, 2025.
  39. [41]Sharath Naganna, Tanvir Ahmed Sijan, and Uddipta Kalita. Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs, July 2026. URL http://arxiv.org/abs/2607.16693. arXiv:2607.16693 [cs.CL].
  40. [42]Marc North, Amir Atapour-Abarghouei, and Nelly Bencomo. Beyond Syntax: How Do LLMs Understand Code? In 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pp. 86–90. IEEE, 2025. doi: 10.1109/ICSE-NIER66352.2025.00023. URL https://ieeexplore.ieee.org/document/11023969/.DOI
  41. [43]Pedro Orvalho and Marta Kwiatkowska. Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?, 2025. URL https://arxiv.org/abs/2505.10443.
  42. [44]Matteo Paltenghi, Rahul Pandita, Austin Z. Henley, and Albert Ziegler. Follow-up attention: An empirical study of developer and neural model code exploration. IEEE Transactions on Software Engineering, 2024. URL https://arxiv.org/abs/2210.05506.
  43. [45]Nancy Pennington. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive Psychology, 19(3):295–341, 1987. doi: 10.1016/0010-0285(87)90007-7.DOI
  44. [46]Daking Rai, Samuel Miller, Kevin Moran, and Ziyu Yao. Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones. In Advances in Neural Information Processing Systems 38, pp. 146964–146999. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2025. doi: 10.52202/085713-4418.DOI
  45. [47]Jorma Sajaniemi. Visualizing roles of variables to novice programmers. In Proceedings of the 14th Annual Workshop of the Psychology of Programming Interest Group, pp. 111–127, 2002. URL https://ppig.org/files/2002-PPIG-14th-sajaniemi.pdf.
  46. [48]Jorma Sajaniemi and Raquel Navarro Prieto. Roles of variables in experts’ programming knowledge. In Proceedings of the 17th Annual Workshop of the Psychology of Programming Interest Group, pp. 145–159, 2005. URL https://ppig.org/files/2005-PPIG-17th-sajaniemi.pdf.
  47. [49]Arushi Sharma, Zefu Hu, Christopher J. Quinn, and Ali Jannesari. Redundancy and Concept Analysis for Code-trained Language Models. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops, pp. 24–34. ACM, 2024. doi: 10.1145/3691621.3694931. URL https://dl.acm.org/doi/10.1145/3691621.3694931.DOI
  48. [50]Arushi Sharma, Vedant Pungliya, Christopher J. Quinn, and Ali Jannesari. Analyzing Latent Concepts in Code Language Models, 2025. URL https://arxiv.org/abs/2510.00476.
  49. [51]Rishab Sharma, Fuxiang Chen, Fatemeh Fard, and David Lo. An exploratory study on code attention in BERT. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pp. 437–448, Virtual Event, May 2022. ACM. doi: 10.1145/3524610.3527921. URL https://dl.acm.org/doi/10.1145/3524610.3527921.DOI
  50. [52]Elliot Soloway and Kate Ehrlich. Empirical studies of programming knowledge. IEEE Transactions on Software Engineering, SE-10(5):595–609, 1984. doi: 10.1109/TSE.1984.5010283.DOI
  51. [53]Kriz Tahimic and Charibeth Cheng. Mechanistic interpretability of code correctness in llms via sparse autoencoders. arXiv preprint arXiv:2510.02917, 2025. URL https://arxiv.org/abs/2510.02917.
  52. [54]Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html.
  53. [55]S. Troshin and N. Chirkova. Probing pretrained models of source codes. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2022. URL https://aclanthology.org/2022.blackboxnlp-1.31/.
  54. [56]Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023.
  55. [57]Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. What do they capture? a structural analysis of pre-trained language models for source code. In Proceedings of the 44th International Conference on Software Engineering, 2022. doi: 10.1145/3510003.3510050.
  56. [58]Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. ReCode: Robustness Evaluation of Code Generation Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13818–13843. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.773. URL https://aclanthology.org/2023.acl-long.773.DOI
  57. [59]Zhilong Wang, Lan Zhang, Chen Cao, Nanqing Luo, Xinzhi Luo, and Peng Liu. How Does Naming Affect Language Models on Code Analysis Tasks? Journal of Software Engineering and Applications, 17(11):803–816, 2024. doi: 10.4236/jsea.2024.1711044. URL https://www.scirp.org/journal/paperinformation?paperid=137296.DOI
  58. [60]Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and discovering vulnerabilities with code property graphs. In 2014 IEEE Symposium on Security and Privacy, pp. 590–604, 2014. URL https://mlsec.org/docs/2014-ieeesp.pdf.DOI
  59. [61]Zhou Yang, Jieke Shi, Junda He, and David Lo. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering, pp. 1482–1493, Pittsburgh, Pennsylvania, May 2022. ACM. doi: 10.1145/3510003.3510146. URL https://dl.acm.org/doi/10.1145/3510003.3510146.DOI
  60. [62]Zhe Yin, Xiaodong Gu, and Beijun Shen. Neuron-guided interpretation of code LLMs: Where, why, and how? Proceedings of the ACM on Software Engineering, 3(FSE), 2026. doi: 10.1145/3797083. URL https://arxiv.org/abs/2512.19980.

Paper details

vlnc:W_w6gmdfw2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

large language models, coding agents

View statement
Looking for related papers…
Contents