betaFeedback

WorksComputer Science

GHA-SmellFix: Taking Action with Agentic LLMs for Detecting and Refactoring GitHub Actions Workflow Smells

Abstract

Continuous integration and continuous delivery workflows are key artifacts in software projects. In GitHub Actions, they encode build, test, release, deployment, and security automation in YAML. These workflows contain configuration smells affecting security, reliability, maintainability, or operational stability. Existing approaches are mostly rule-based, with rules hard to create and evolve, and typically report findings without fixes. Agentic LLM pipelines can combine contextual detection, transformation selection, and patch generation. Specialized agents can gate refactoring on positive detection verdicts, potentially avoiding unnecessary changes, but it is unclear whether decomposition improves effectiveness or mainly changes overhead. We present GHA-SmellFix, an agentic LLM pipeline for detecting and refactoring GitHub Actions workflow smells. We compare two designs: Joint, where one agent detects and refactors, and Decomposed, where separate agents detect and then refactor. We operationalize 19 smells from an existing catalog with explicit detection criteria, refactoring expectations, exclusion rules, and safety checks, and curate a ground truth of 1,710 (workflow, smell) pairs. We evaluate the designs with four LLMs (GPT-5.4-nano, Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash) on 570 sampled pairs. Results show that decomposition does not universally improve patch quality; its main benefit is controlling when patch generation is attempted. It helps models prone to missing smells and can reduce overhead by gating patches. Once a smell is correctly detected, clean refactoring is generally effective, making detection the main end-to-end bottleneck. However, false positives can trigger unnecessary edits to execution-relevant constructs. Practical CI/CD agents thus need reliable abstention and construct-aware validation alongside patch generation.

Introduction

Continuous Integration and Continuous Delivery (CI/CD) pipelines have become key maintenance artifacts in modern software engineering [15, 18, 39, 47]. GitHub Actions is widely adopted to implement such pipelines through repository-level YAML files, called workflows, that automate builds, tests, releases, deployments, and security-related tasks [8, 9]. Although workflows have a declarative syntax [17], they are not simple configuration artifacts. They may invoke third-party actions, access repository secrets, publish artifacts, define permissions, execute scripts, and interact with deployment environments [25]. As a result, the way a workflow is written can affect the security posture [24, 25], reliability, maintainability, and operational stability of software projects [15, 40, 47]. Prior studies show that CI/CD workflows frequently contain configuration smells or anti-patterns, i.e., recurring patterns that may increase the risk of vulnerabilities, unstable builds, excessive resource usage, or performance degradation [15, 24, 25, 40, 47].

Existing methods for analyzing quality and security problems in GitHub Actions workflows are largely rule-based [4, 14, 24]. These methods encode known patterns and provide inexpensive, reproducible screening. However, they have two main limitations. First, their detection logic depends on engineered rules whose design and maintenance can require substantial effort as workflow constructs, platform features, and smell definitions evolve [2, 9, 28]. For instance, checking effective permissions may require combining workflow-level and job-level permissions, while fork-guard smells depend on event triggers, conditional expressions, and repository context rather than on a single YAML key. Second, only reporting a smell does not complete the maintenance task: currently, developers must still judge whether the finding is relevant, decide how to remediate it, and assess whether the resulting patch is correct or introduces unrelated behavior changes [3, 22, 46]. This is challenging because workflow refactoring may involve constructs such as permissions, triggers, conditions, runners, dependencies, deployment steps, caches, matrices, and reusable actions or workflows [2, 28]. Thus, the open problem is not only detecting smells, but generating and evaluating candidate refactorings that are structurally valid, scope-bounded, and abstention-aware.

Large Language Models (LLMs) offer a promising mechanism for this broader maintenance task because they can interpret configuration context, reason about intended changes, and generate candidate patches [20]. However, LLM-generated edits may be risky: a patch may remove the target smell while changing unrelated triggers, permissions, runners, dependencies, or deployment behavior. Agentic pipelines can organize LLM capabilities into explicit roles, for instance, by separating smell detection from patch generation [41]. Yet, we still lack a clear understanding of how decomposing workflow maintenance tasks affects detection and refactoring performance, as well as its trade-offs in execution latency, token efficiency, and API cost.

This paper presents GHA-SMELLFIX, an agentic LLM pipeline for identifying and refactoring GitHub Actions configuration smells. Given smell definitions, a workflow file, and repository-level context, GHA-SMELLFIX detects whether a smell is present and, when appropriate, generates a scope-bounded candidate refactoring. We compare two GHA-SMELLFIX orchestration designs: (D1) a joint design that performs detection and, when applicable, refactoring in one pass; and (D2) a decomposed design in which a detection agent first decides whether the target smell is present, and a refactoring agent is invoked only for detected smells. These designs differ in task decomposition, prompt context, number of LLM calls, and control flow; therefore, we use them as practical designs for studying effectiveness, patch quality, abstention behavior, and operational overhead, not as an isolated causal test of adding one agent.

To evaluate these designs, we operationalize 19 GitHub Actions smells from an existing catalog [24], defining explicit detection criteria, refactoring expectations, exclusion rules, construct-level scope, and risk checks. We then instantiate the two designs using four LLMs (GPT-5.4-nano, Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash) and compare them in paired cross-model and cross-design evaluations on 570 annotated (workflow, smell) pairs. These pairs were sampled from a 1,710-pair ground truth manually curated for this study, covering positive and negative cases for 19 smells. To assess generated refactorings, we use a construct-aware evaluation protocol that distinguishes clean smell removals, removals with unexpected edits, removals with potential behavior changes, failed/no-patch cases, and unnecessary edits on ground-truth-negative pairs. We analyze detection effectiveness, end-to-end and conditional refactoring outcomes, abstention behavior, and overhead in terms of latency, token usage, and estimated API cost.

Our results show that agentic decomposition provides selective rather than universal benefits for GitHub Actions maintenance. Separating detection from refactoring modestly improves aggregate detection, increasing average F1 from 86.34% to 87.55% and recall from 79.12% to 82.37%, with the largest gain for GPT-5.4-nano, whose F1 rises from 81.38% to 87.15%; models already strong in the joint setting change little. More importantly, the main end-to-end bottleneck is detecting when the target smell is present rather than generating the patch itself: clean removal reaches 74% overall, but rises to 92.6% once the target smell is correctly detected. At the same time, successful smell removal does not guarantee patch quality, and abstention remains challenging: the pipelines produced 93 unnecessary edits on negative cases, 54.84% of which affected behavior-relevant workflow constructs. Decomposition also need not increase operational cost, as the detection gate reduces latency for three of the four evaluated models by avoiding unnecessary refactoring calls. Overall, these findings position agentic LLM pipelines as promising review-support mechanisms for CI/CD maintenance, whose practical value depends on model- and smell-aware orchestration, reliable abstention, and construct-aware validation rather than on decomposition alone.

Our contributions include: (i) GHA-SmellFix, an agentic LLM pipeline for target-smell detection and candidate refactoring; (ii) a manually curated ground truth of 1,710 annotated (workflow, smell) pairs, including positive and negative cases, designed to support paired evaluation of detection and refactoring behavior; (iii) a construct-aware evaluation protocol for LLM-generated workflow patches, covering smell removal, workflow validity, expected construct changes, unexpected edits, potential behavior-impacting changes, failed/no-patch cases, and unnecessary edits; and (iv) an empirical evaluation of two GHA-SmellFix orchestration designs across four LLMs, covering effectiveness, abstention, construct-level safety, and operational overhead.

Related Work

CI/CD anti-patterns and workflow quality. Prior work has shown that CI/CD pipelines accumulate recurring configuration problems that affect maintainability, feedback, security, and delivery behavior. Hilton et al. [18] discuss trade-offs in CI adoption, including assurance, security, and flexibility. Gallaba and McIntosh [15] show that CI features are often used or misused in ways that affect build behavior and feedback. Zampetti et al. [47] characterize bad practices in CI pipelines, while Vassallo et al. [39, 40] study CI/CD anti-patterns, decay, configuration smells, and linter-based reporting. These studies establish CI/CD configuration quality as a recurring maintenance concern, but they focus mainly on characterizing, detecting, or reporting problematic practices rather than generating and evaluating candidate refactoring.

GitHub Actions workflows, smells, and scanners. GitHub Actions brings CI/CD quality concerns to repository-level YAML workflows. Prior work has studied the adoption and evolution of GitHub Actions workflows [8, 9, 28, 37, 38], their security risks [25], and opportunities for resource-usage optimization [6]. Recent work also shows that workflow reliability and maintenance effort are related to specific constructs, such as triggers, jobs, steps, permissions, matrices, conditions, and reusable components [2]. This construct-level perspective is important for refactoring because an edit may be syntactically valid while still changing behavior-relevant parts of the automation.

Several tools and catalogs target GitHub Actions quality and security. Khatami et al. [24] introduced a catalog of smells and the GHA analyzer. GHAST [4] and GASH [14] inspect workflows for recurring security or configuration problems, while Fares et al. [12] show that workflow security scanners differ substantially in scope, detected weaknesses, and usability. These approaches are useful as scalable detectors, but they stop at reporting findings. They do not assess whether generated patches remove a target smell, stay within the expected construct scope, or abstain when the smell is absent. Our work evaluates generated candidate patches, not only detects smell instances.

LLM-based agents for software maintenance. LLM-based agents have recently been applied to software maintenance tasks that involve repository interaction, patch generation, and validation. SWE-agent [45] and OpenHands [42] frame maintenance as an interactive repository-level task, while RepairAgent [5] uses agentic repair loops to plan and validate program fixes. Recent work on test smells shows that agentic language models can support both smell detection and refactoring in source-code-centric artifacts [31]. GitHub Actions workflows differ from these settings: small YAML edits can alter permissions, event exposure, secret access, runner selection, artifact publication, deployment behavior, or reusable-workflow invocation. Thus, patch quality cannot be assessed only by syntactic validity or smell removal. Moreover, recent evidence cautions against assuming that more complex agentic orchestration necessarily improves software engineering outcomes; simpler localization–repair–validation pipelines can compete with more autonomous agents [44]. We therefore treat decomposition as an empirical design choice rather than an assumed improvement.

Study Settings

To design our study, we adopted the Goal-Question-Metric (GQM) template [7]. The goal is to analyze GHA-SmellFix orchestration designs; for the purpose of detecting and refactoring configuration smells in GitHub Actions workflows; with respect to (i) target-conditioned detection effectiveness, (ii) refactoring correctness, construct-level consistency, and abstention behavior, and (iii) the effectiveness–overhead trade-offs of task decomposition; from the viewpoint of researchers studying LLM-based CI/CD maintenance; in the context of a manually curated ground truth of (workflow, smell) pairs covering 19 configuration smells. The research questions (RQ) are:

RQ₁: How effectively can each GHA-SmellFix orchestration design detect target GitHub Actions configuration smells? – This RQ evaluates target-conditioned detection over explicit (workflow, smell) pairs. Each prediction is a binary decision about whether the target smell is present or absent, compared with the annotated ground truth using precision, recall, accuracy, and F1-score.

RQ₂: To what extent can each GHA-SmellFix orchestration design generate structurally valid and scope-bounded refactorings for GitHub Actions workflow smells? – This RQ evaluates end-to-end and conditional refactoring outcomes. We examine whether generated patches remove the target smell, preserve YAML and GitHub Actions structural validity, stay within the expected construct scope, and avoid potential behavior-impacting changes. We also analyze unnecessary edits on ground-truth negative pairs, where abstention is expected.

RQ₃: What is the effectiveness–overhead trade-off of task decomposition in GHA-SmellFix? – This RQ compares (D1) a joint detect-and-refactor design with (D2) a decomposed detect-then-refactor design. These designs differ in task decomposition, prompt context, number of LLM calls, and control flow. We compare detection effectiveness, end-to-end and conditional refactoring outcomes, abstention behavior, latency, token usage, and estimated cost.

Study Steps and Procedures

Step 1: Repository selection and workflow extraction. We relied on the GitHub Actions workflow dataset by Cardoen et al. [8] to identify open-source repositories that use GitHub Actions, deriving 49,510 candidate repositories. We filtered repositories based on criteria adapted from prior studies [23, 32, 34]. We kept repositories that: (i) had at least 100 GitHub stars; (ii) had at least 100 commits; (iii) had activity within the last 12 months; (iv) were at least six months old; (v) used main or master as the default branch; and (vi) contained at least two workflow files. We excluded repositories that were: (i) forks; (ii) archived; (iii) named as learning, tutorial, clone, template, actions, workflows, or reusable examples; or (iv) created primarily for saving workflows or actions. At the workflow level, we kept files that: (i) were triggered by pull_request or push events; and (ii) had at least 100 execution runs. We excluded workflow files that: (i) had no modification history; (ii) had names or filenames containing documentation-related terms, such as docs.yml; (iii) were empty or contained placeholder content; or (iv) had invalid YAML syntax. With these criteria, we selected a dataset comprising 27,704 repositories and 49,836 unique GitHub Actions workflow files. We then used the GitHub API to retrieve repository metadata and collect the current configuration files under the .github/workflows/ directory, adopting a snapshot cut-off on April 9, 2026.

Step 2: Selection and operationalization of configuration smells. We rely on Khatami et al.’s catalog [24] because it provides fine-grained GitHub Actions smells and tool support through the GHA analyzer, which we use for sampling. The catalog is grounded in evidence of workflow evolution: candidate smells were derived from recurring patterns of workflow change and examined through a contribution study with maintainers. The catalog contains 22 smells. We selected 19 and excluded AddCommentsToWorkflows, CorrectIndentation, and NameRunSteps, because they mainly capture readability or formatting conventions rather than security, performance, or CI/CD execution concerns. We refined the selected smells into explicit detection conditions, exclusion criteria, uncertainty-handling rules, and counting units. These refinements support consistent manual annotation and agent-based evaluation, especially for ambiguous GitHub Actions constructs such as reusable workflows, local actions, matrices, fork guards, and deployment-related steps. We also cross-checked the refined definitions against the official GitHub Actions documentation [17].

Based on the refined definitions, two authors classified each smell as localized or context-dependent through discussion and consensus. Localized smells rely on explicit evidence within a bounded set of workflow constructs, such as a job, step, action reference, permission block, timeout, or runner declaration. Context-dependent smells require combining evidence across workflow constructs, scopes, relationships, or repository-level context, such as triggers, fork guards, deployment behavior, concurrency semantics, matrices, reusable workflows, or workflow intent. Table 1 overviews the smells and the core operational criterion for each smell.

TypeConfiguration smellCore operational criterionP/N
LOCALIZEDMissingPermissionExternalActionsA job uses an external step-level GitHub action and has no effective workflow- or job-level permissions block.50/50
LOCALIZEDUnpinnedActionVersionA step-level GitHub action reference uses owner/repo@ref, but ref is not a full 40-character commit SHA.36/36
LOCALIZEDMissingTokenPermissionsA job uses GITHUB_TOKEN or unauthenticated GitHub operations without an effective permissions block.50/50
LOCALIZEDMissingJobTimeoutA job defined in the current workflow file does not declare job-level timeout-minutes.48/48
LOCALIZEDUnsafeArtifactUploadForkA PR-triggered workflow uploads retained GitHub Actions artifacts without an effective fork guard.33/33
LOCALIZEDMissingUploadConditionA retained artifact-upload step lacks an explicit step-level or job-level if condition.38/38
LOCALIZEDMutableRunnerVersionA job uses a mutable GitHub-hosted runner label ending in -latest.50/50
LOCALIZEDSuboptimalCacheUsageA workflow uses actions/cache for dependencies already supported by setup-action built-in caching.49/49
LOCALIZEDMultiCommandStepA run step combines separable commands from different workflow phases.25/25
LOCALIZEDUnpinnedDependenciesA run step installs external packages or tools without fixed versions or equivalent immutable references.38/38
CONTEXT-DEPENDENTMissingForkGuardPRAn issue/PR-triggered workflow performs issue or pull-request actions without an effective fork guard.50/50
CONTEXT-DEPENDENTMissingConcurrencyPRA PR-triggered workflow lacks applicable concurrency.cancel-in-progress: true to cancel older runs.50/50
CONTEXT-DEPENDENTMissingConcurrencyBranchA branch-push workflow lacks applicable concurrency.cancel-in-progress: true to cancel older runs.50/50
CONTEXT-DEPENDENTMissingPathFiltersA CI workflow triggered by push or pull_request lacks paths or paths-ignore filters.39/39
CONTEXT-DEPENDENTMissingForkGuardScheduleA scheduled workflow can run outside the intended repository owner, or fork-status guard.50/50
CONTEXT-DEPENDENTUnsafeDeployOnForkA workflow can run in a fork-related PR context and perform external delivery without an effective fork guard.50/50
CONTEXT-DEPENDENTMissingConcurrencyGlobalA repeated or side-effectful task can overlap because no effective concurrency group is defined.50/50
CONTEXT-DEPENDENTSingleOSExecutionA test job effectively runs on only one operating system and has no multi-OS matrix.50/50
CONTEXT-DEPENDENTSingleLangVersionCIA CI job validates only one explicit language/runtime version and has no multi-version matrix.49/49
Total855/855

Table 1. GitHub Actions configuration smells, refined from Khatami et al. [24], and ground-truth coverage. P/N denotes positive/negative pairs; full definitions are in the replication package.

Step 3: Ground-truth construction and manual annotation. Figure 1 summarizes how we built the manually annotated ground truth for the smells in Table 1.

Overview of ground-truth construction, manual annotation, and shared-set sampling.

Figure 1. Overview of ground-truth construction, manual annotation, and shared-set sampling.

Stratified sampling. We adopted pair-based stratified purposive sampling guided, but not labeled, by the GHA analyzer [24]. For each smell, we sampled positive candidates from workflows in which GHA analyzer reported exactly one target-smell instance and negative candidates from workflows where the target smell was not flagged. To reduce annotation ambiguity, we kept only workflows with at most one GHA-reported instance of any smell and removed LOC outliers using Tukey’s IQR filter [11] (threshold: 209 LOC). We ranked the remaining candidates by ascending total_unique_smells and descending workflow_run_count, prioritizing workflows with fewer co-occurring smells and more execution evidence. For each smell, we sampled rounds of 100 positive and 100 negative candidates, repeating for up to six rounds until we reached 50 validated pairs per class, and then balanced each smell by undersampling the larger validated class. The only exception was UnsafeDeployOnFork, for which we found only five natural positives. We therefore used Claude Opus 4.8 to mutate real workflows manually labeled negative for this smell, restricting edits to the target-smell construct. After manual validation, we retained 45 of the 49 mutants as positives, which entered balancing together with the natural pairs. We did not synthesize any negative pairs.

Manual annotation. Six annotators worked in three mixed-experience pairs (two master’s–undergraduate pairs and one PhD–undergraduate pair). They had one to three years of software engineering experience and prior exposure to GitHub Actions and CI/CD workflows. Before annotation, we provided annotators with the refined smell catalog, annotation guidelines, and positive/negative examples for each smell. Within each pair, annotators independently inspected the same (workflow, smell) pairs, labeled the target smell as present or absent, and recorded the number of instances. We replaced pairs in which annotators identified multiple target-smell instances using the same iterative procedure, and each team resolved disagreements by consensus. We measured pre-consensus agreement over the 1,710 evaluated (workflow, smell) pairs from 902 unique workflow files using Cohen’s κ\kappa [29], obtaining almost perfect agreement [26] (κ=0.848\kappa= 0.848); we report team-level values in the replication package. We discarded candidates whose manual label contradicted their sampling stratum rather than relabeling them into the natural ground truth (Figure 1). After adding the 45 validated semi-synthetic positives and balancing per smell, the final ground truth comprised 855 positive and 855 negative pairs, with workflows ranging from 7 to 207 LOC. From this ground truth, we sampled a shared cross-model evaluation set of 15 positive and 15 negative pairs per smell (n=570n = 570). Table 1 also summarizes their distribution; per-smell workflow-size statistics and pair-level agreement results are available in our replication package.

Step 4: GHA-SmellFix orchestration designs. We instantiate GHA-SmellFix with two agentic LLM orchestration designs that differ in task decomposition and control flow, while keeping the input format, smell catalog, output schemas, and underlying LLM fixed within each comparison. Figure 2 illustrates the two designs. D1 is a joint detect-and-refactor design: a single agent classifies the target smell as “present” or “absent”, and, when applicable, returns a candidate patch. D2 is a decomposed detect-then-refactor design: a detection agent first classifies the target smell; if the smell is predicted as “absent”, the original flow is returned unchanged; otherwise, a refactoring agent generates a candidate patch. The agents are not given repository-level CLI access, external validators, or runtime services, allowing us to assess basic agentic capability.

GHA-SmellFix orchestration designs for target-conditioned smell detection and refactoring

Figure 2. GHA-SMELLFix orchestration designs for target-conditioned smell detection and refactoring.

We implemented both designs using the Agno framework [10], with XML-tagged prompts and Pydantic schemas for structured outputs. Prompts combined role-based prompting with reasoning-oriented instructions and prompt-level constraints [35]. These guardrails constrained agents to the target smell, required the smallest scope-bounded edit, discouraged unrelated workflow changes, enforced raw YAML outputs, and required confidence justifications grounded in workflow evidence. We logged execution metadata for each LLM call, including token usage and latency, to support overhead analysis. Complete prompt templates and schemas are provided in the replication package.

Our study focuses on target-conditioned detection. GHA-SmellFix receives the target smell definition from our refined catalog, the workflow, and repository-level context (owner/repo, owner, default branch, and file path) to interpret repository-specific locations. We executed D1 and D2 on the shared 570-pair set for paired comparison; the full 1,710-pair ground truth serves as complementary evidence for DeepSeek V4 Flash.

Step 5: LLM selection and execution. We selected four LLMs, presented in Table 2, varying in provider, family, scale, and access mode. We prioritized code-related capabilities, instruction following, and support for structured outputs because LLM effectiveness on SE tasks varies across models and deployment settings [20, 21, 33], and agentic pipelines depend on role-specific instructions and structured responses [5, 27, 36, 45]. We kept the input artifacts, role-specific prompt templates, and structured output schemas fixed across agents, changing only the underlying LLM. Between D1 and D2, the workflow input and target smell also remained fixed, while the orchestration changed by decomposing the task, changing prompt context, and adding a detection gate before refactoring. We used temperature 0 whenever supported to reduce sampling variability. GPT-5.4-nano and Claude Haiku 4.5 required temperature 1 in their thinking modes and were executed k=3k = 3 times to assess detection stability. To control execution cost, GPT-5.4-nano, Claude Haiku 4.5, and Gemini 3.5 Flash were evaluated on a balanced subset with 15 positive and 15 negative pairs per smell. Gemini 3.5 Flash was executed once. DeepSeek V4 Flash was evaluated once on the 1,710-pair ground truth. Overall, we ran (2×2×570×3)+(2×1×570)+(2×1×1,710)=11,400(2 \times2 \times570 \times3) + (2 \times1 \times570) + (2 \times1 \times1{,}710) = 11{,}400 pipeline runs.

ModelAccessTypeParams.Temp.ThinkingIn (1M/USD)Out (1M/USD)
GPT-5.4-nanoOpenAI APIProprietary–1Medium0.201.25
Claude Haiku 4.5Anthropic APIProprietary–12,048 tokens1.005.00
Gemini 3.5 FlashGemini APIProprietary–0Medium1.509.00
DeepSeek V4 FlashOpenRouter APIOpen-weight284B (13B active)02,048 tokens0.0640.251

Table 2. Overview of the selected LLMs. Costs are reported in USD per 1M tokens (June 2026).

GPT-5.4-nano and Gemini 3.5 Flash expose reasoning effort as a qualitative level rather than an explicit token budget; their token allocation is provider-managed.

Step 6: Data Analysis. We based our analysis on the shared set of 570 ground-truth (workflow, smell) pairs, covering 19 smells with 15 positive and 15 negative instances each. The full ground-truth results for DeepSeek V4 Flash are provided in the replication package.

Detection Analysis (RQ1). We assess detection effectiveness using Precision, Recall, F1-score, and Accuracy [13]. Each prediction is a binary decision indicating whether a target smell is present in a workflow; Accuracy therefore captures the proportion of correct present/absent classifications over positive and negative (workflow, smell) pairs. We also assess detection stability for GPT-5.4-nano and Claude Haiku 4.5, which were executed three times under the same setup. Following prior work [16], we report accuracy spread, the difference between maximum and minimum accuracy across the k=3k = 3 executions for each (model, design) pair. This stability analysis is restricted to detection and to models executed three times.

Refactoring Assessment (RQ2). Figure 3 summarizes our assessment protocol. We evaluate patches along three dimensions: smell removal, workflow validity, and construct-level consistency. Because refactoring depends on detection, we report end-to-end outcomes over all ground-truth-positive outputs and conditional outcomes restricted to correctly detected positives. For workflow validity, we check YAML parsability and required GitHub Actions structure; invalid outputs are unsuccessful refactorings. For the other dimensions, we diff original and refactored workflows, extract modified YAML paths, and normalize them into GitHub Actions constructs using Bardsiri et al. [2]. Of their 197 constructs, we use the 117 relevant to our 19 smells and map each smell to its expected modification scope. For instance, MissingJobTimeout targets jobs.<job_id>.timeout-minutes, whereas SingleOSExecution may involve jobs.<job_id>.strategy.matrix.<matrix_var>. These mappings define expected fix scope rather than fixed templates and are provided in the replication package.

Overview of the refactoring assessment protocol, showing input, automated processing, blinded assessment, and reported views

Figure 3. Overview of the refactoring assessment protocol. Dashed arrows indicate automatic assignments for invalid positive outputs and unmodified negative outputs; all other outputs are assessed by blinded annotators. Conditional rates use GT-positive outputs whose structured detection verdict is present.

We manually inspected 4,560 outputs; 2,280 positive-pair outputs for refactoring assessment and 2,280 negative-pair outputs for abstention. The ground-truth annotator pairs assessed outputs blindly to model and design using the smell definition, expected construct scope, agent rationale, diff, and modified constructs. Because multiple valid fixes may exist, no single reference patch was used. Positive outputs were classified as Clean Removal when the smell was removed, validity preserved, and only expected constructs changed; Removal with Unexpected Changes when unrelated constructs were modified; Removal with Potential Behavior-Impacting Change when changes could affect workflow execution, security posture, or deployment behavior; or Failed or No Patch when the smell was not removed, validity checks failed, or no applicable patch was produced.

For negative pairs, where abstention was expected, modified outputs were classified as Cosmetic Edit limited to formatting, comments, or naming; Localized Unnecessary Edit, for plausible but unnecessary smell-related changes; or Behavior-Relevant Unnecessary Edit, for changes to execution-relevant constructs. If multiple edit types occurred, we assigned the most behavior-relevant category. Annotators could mark cases as uncertain when static evidence was insufficient. Disagreements were adjudicated by a third author, except when both annotators independently marked a case uncertain; these were treated as not assessable. Pre-consensus agreement was substantial (κ=0.758\kappa= 0.758) [26, 29]. We do not validate runtime-dependent facts such as commit hashes, package versions, credentials, external actions, or services.

Orchestration effect (RQ3). We compare D1 and D2 as practical GHA-SmellFix designs to assess whether task decomposition improves effectiveness or mainly changes overhead. This comparison does not isolate the effect of adding one agent. We reuse RQ1 metrics for detection, RQ2 outcomes for refactoring, and compare latency, token usage, and estimated API cost for overhead. For each LLM, we compare D1 and D2 outputs over the same (workflow, smell) pairs using McNemar’s exact test [30] for detection correctness and end-to-end clean removal, and paired Wilcoxon signed-rank tests [43] for latency and estimated cost, with Holm correction [19] within each metric.

Execution protocol and stability. For GPT-5.4-nano and Claude Haiku 4.5, which were nondeterministic in our setup, we ran each model–design pair three times under D1 and D2. We use the first run in the main analyses to avoid post-hoc run selection, and use the repetitions only for stability analysis. The maximum accuracy spread is 2.11%, indicating low run-to-run variation. Gemini 3.5 Flash and DeepSeek V4 Flash were configured with temperature zero and executed once; therefore, stability conclusions are limited to GPT-5.4-nano and Claude Haiku 4.5. All repeated-run outputs are provided in the replication package.

Results and Discussion

Detection Effectiveness (RQ1)

To answer RQ1, we analyze detection from two perspectives: (i) aggregate effectiveness, comparing models and designs to assess whether task decomposition changes detection; and (ii) smell-level effectiveness, examining how these effects vary across the 19 smells and across models.

Task decomposition modestly improves aggregate detection effectiveness. Table 3 overviews the aggregate detection results. We observe that D2 achieves the highest aggregate detection effectiveness, reaching an average F1-score of 87.55% and accuracy of 88.29%. This gain is mainly recall-driven: recall increases from 79.12% in D1 to 82.37% in D2, while precision remains high but decreases from 95.38% to 93.44%. Thus, task decomposition shifts the precision–recall balance toward recovering more true smell instances, at the cost of a small increase in false positives. However, this effect is model-dependent. GPT-5.4-nano benefits the most from D2, improving from 81.38% to 87.15% F1, driven by a recall increase from 70.53% to 82.11%. Claude Haiku 4.5 and DeepSeek V4 Flash are already strong under D1 and remain stable across designs, while Gemini 3.5 Flash shifts from high precision in D1 to higher recall in D2.

DesignModelnnF1↑Pr↑Re↑Acc↑
D1Claude Haiku 4.557088.81%93.08%84.91%89.30%
D1DeepSeek V4 Flash57088.27%94.05%83.16%88.95%
D1Gemini 3.5 Flash57086.89%98.23%77.89%88.25%
D1GPT-5.4-nano57081.38%96.17%70.53%83.86%
D1Avg.–86.34%95.38%79.12%87.59%
D2Claude Haiku 4.557088.52%93.73%83.86%89.12%
D2DeepSeek V4 Flash57087.99%92.97%83.51%88.60%
D2GPT-5.4-nano57087.15%92.86%82.11%87.89%
D2Gemini 3.5 Flash57086.53%94.21%80.00%87.54%
D2Avg.–87.55%93.44%82.37%88.29%

Table 3. Overall detection effectiveness across designs. The best and second-best model values per configuration are shown in bold and <u>underlined</u>, respectively.

The results also suggest that orchestration effects are not explained by model availability alone. DeepSeek V4 Flash, the open-weight model in our study, achieves the second-best F1-score in both designs, closely matching Claude Haiku 4.5, but changes little under decomposition. In contrast,

GPT-5.4-nano benefits substantially from D2. Overall, D2 improves aggregate detection, but not uniformly across models; its main benefit is increasing recall, especially for models that under-detect in the joint detect-and-refactor design.

Finding 1: Task decomposition modestly improves aggregate detection, raising average F1 from 86.34% to 87.55% and recall from 79.12% to 82.37%.

Task decomposition helps selectively, depending on smell semantics and model behavior. Table 4 shows that D2 gains vary by smell and model. Overall, we observe high precision across configurations, while recall varies more substantially. Smells with explicit workflow evidence are easier to detect: MissingPermExternalActions and MissingTokenPermissions reach 100% F1 in nearly all (model, design) pairs, and MissingPathFilters, UnsafeArtifactUploadFork, and SuboptimalCacheUsage remain close to 100% in most cases. For these construct-explicit smells, decomposition provides little additional value because the single-agent configuration already performs well.

TypeSmellGPT-5.4-nano D1 F1GPT-5.4-nano D1 PrGPT-5.4-nano D1 ReGPT-5.4-nano D1 AccGPT-5.4-nano D2 F1GPT-5.4-nano D2 PrGPT-5.4-nano D2 ReGPT-5.4-nano D2 AccGemini 3.5 Flash D1 F1Gemini 3.5 Flash D1 PrGemini 3.5 Flash D1 ReGemini 3.5 Flash D1 AccGemini 3.5 Flash D2 F1Gemini 3.5 Flash D2 PrGemini 3.5 Flash D2 ReGemini 3.5 Flash D2 AccClaude Haiku 4.5 D1 F1Claude Haiku 4.5 D1 PrClaude Haiku 4.5 D1 ReClaude Haiku 4.5 D1 AccClaude Haiku 4.5 D2 F1Claude Haiku 4.5 D2 PrClaude Haiku 4.5 D2 ReClaude Haiku 4.5 D2 AccDeepSeek V4 Flash D1 F1DeepSeek V4 Flash D1 PrDeepSeek V4 Flash D1 ReDeepSeek V4 Flash D1 AccDeepSeek V4 Flash D2 F1DeepSeek V4 Flash D2 PrDeepSeek V4 Flash D2 ReDeepSeek V4 Flash D2 Acc
LOCALIZEDMissingPermExternalActions971009397100100100100100100100100100100100100100100100100100100100100100100100100100100100
LOCALIZEDUnpinnedActionVersion801006783801006783801006783801006783801006783801006783801006783801006783
LOCALIZEDMissingTokenPermissions971009397100100100100100100100100100100100100100100100100100100100100100100100100100100100
LOCALIZEDMissingJobTimeout82749380756010067100100100100796510073756010067756010067776210070756010067
LOCALIZEDUnsafeArtifactUploadFork851007387100100100100100100100100100100100100100100100100100100100100100100100100100100100
LOCALIZEDMissingUploadCondition701005377931008793891008090931008793931008793891008090931008793931008793
LOCALIZEDMutableRunnerVersion891008090931008793931008793931008793891008090891008090891008090891008090
LOCALIZEDSuboptimalCacheUsage701005377100100100100100100100100100100100100100100100100100100100100100100100100100100100
LOCALIZEDMultiCommandStep701005377801006783705470706410047747053707064100537070537070641005370
LOCALIZEDUnpinnedDependencies100100100100100100100100806783898080808997939797979793979793979797939797
CONTEXT-DEPENDENTMissingForkGuardPR42276342100276342421002763241001357641004773501003367421002763421002763
CONTEXT-DEPENDENTMissingConcurrencyPR100100100100971009397100100100100100100100100100100100100100100100100100100100100100100100
CONTEXT-DEPENDENTMissingConcurrencyBranch8787878780808080948810091938310093918810090918310093918810090918310093
CONTEXT-DEPENDENTMissingPathFilters971009397100100100100971009397100100100100971009397931008790971009397971009397
CONTEXT-DEPENDENTMissingForkGuardSchedule421002763801006783100100100100100100100100100100100100100100100100100100100100100100100
CONTEXT-DEPENDENTUnsafeDeployOnFork751006080891008089891008089891008089891008089891008089851007387851007387
CONTEXT-DEPENDENTMissingConcurrencyGlobal8692808776797377798573805878476788799390887993909088939088799390
CONTEXT-DEPENDENTSingleOSExecution641004773678955787410013576188477048337957668953786789537861884770
CONTEXT-DEPENDENTSingleLangVersionCI7510060808692808770100537780100678386928087851007384751006080851006783
Avg.7998718486958288849579888495818888958589879684898795838787958489

Table 4. Smell-level detection effectiveness by model and GHA-SmellFix orchestration design. Best F1-score per smell/model is in bold; second-best distinct F1-score is _underlined_. The Avg. row reports macro-averages across smells for each model/design pair. Gray rows highlight selected difficult smells discussed in the text.

In contrast, several smells remain more challenging. Context-dependent smells such as MissingForkGuardPR, SingleOSExecution, and MissingConcurrencyGlobal show lower or more variable F1 scores, reflecting the difficulty of recovering true positives when evidence depends on fork context, concurrency semantics, or OS coverage. The localized but intent-sensitive MultiCommandStep is also difficult, ranging from 57% to 80% F1. A different pattern appears for UnpinnedActionVersion: all models and configurations are conservative, with 100% precision but only 67% recall. These results suggest that detection becomes harder when models must infer workflow intent, distinguish separable command phases, or consistently apply strict pinning criteria to action references.

We find that task decomposition helps mostly when the model under-detects in D1. GPT-5.4-nano benefits the most from D2, improving from 79% to 86% in smell-level macro-F1, mainly through recall gains from 71% to 82%. These gains are visible for SuboptimalCacheUsage, MissingUploadCondition, MissingForkGuardSchedule, UnsafeArtifactUploadFork, and SingleLangVersionCI. However, this pattern does not generalize to all models: Gemini 3.5 Flash remains stable on average, while Claude Haiku 4.5 and DeepSeek V4 Flash are already strong under D1 and change little under D2. Although the open-weight DeepSeek V4 Flash remains competitive with closed models, it benefits little from decomposition, whereas GPT-5.4-nano benefits substantially. Thus, decomposition gains

are not explained by model availability alone, but by how each model handles task decomposition and smell-specific reasoning demands.

Finding 2: Detection gains are selective: D2 mainly helps GPT-5.4-nano and a subset of target smells, while models with stronger D1 performance change little under decomposition.

Refactoring Outcomes (RQ2)

To answer RQ2, we evaluate whether the studied GHA-SmellFix orchestration designs generate structurally valid and scope-bounded candidate refactorings for workflow smells.

End-to-end construct-aware outcomes by target smell. Following the refactoring assessment procedure described in Step 6, we classify positive-pair outputs as Clean Removal (CR), Removal with Unexpected Changes (UX), Removal with Potential Behavior-Impacting Change (BR), or Failed/No Patch (FN). For negative pairs, we distinguish Correct Abstention (CA) from Unnecessary Edit (UE). Table 5 reports these end-to-end outcomes by smell, together with the Smell Removal Rate (SRR = CR + UX + BR). Because FN conflates missed detections and failed patches after detection, we separate them in the conditional analysis below. Arrows indicate the desired direction.

TypeSmellSRR ↑CR ↑UX ↓BR ↓FN ↓Assessable/Expected Pos. OutputsCA ↑UE ↓Assessable/Expected Negative Outputs
LocalizedMissingPermExternalActions99% (117)98% (116)1% (1)0% (0)1% (1)118/120100% (120)0% (0)120/120
LocalizedUnpinnedActionVersion66% (78)40% (47)0% (0)26% (31)34% (40)118/120100% (120)0% (0)120/120
LocalizedMissingTokenPermissions99% (119)99% (119)0% (0)0% (0)1% (1)120/120100% (120)0% (0)120/120
LocalizedMissingJobTimeout93% (105)91% (103)0% (0)2% (2)7% (8)113/12056% (65)44% (52)117/120
LocalizedUnsafeArtifactUploadFork95% (110)95% (110)0% (0)0% (0)5% (6)116/120100% (120)0% (0)120/120
LocalizedMissingUploadCondition77% (75)77% (75)0% (0)0% (0)23% (23)98/120100% (120)0% (0)120/120
LocalizedMutableRunnerVersion82% (96)81% (95)0% (0)1% (1)18% (21)117/120100% (120)0% (0)120/120
LocalizedSuboptimalCacheUsage91% (105)86% (99)1% (1)4% (5)9% (10)115/120100% (120)0% (0)120/120
LocalizedMultiCommandStep47% (54)46% (53)0% (0)1% (1)53% (62)116/120100% (120)0% (0)120/120
LocalizedUnpinnedDependencies89% (102)70% (81)1% (1)17% (20)11% (13)115/120100% (…)0% (…)120/120
Context-dependentMissingForkGuardPR26% (30)25% (29)1% (1)0% (0)74% (87)117/120100% (120)0% (0)120/120
Context-dependentMissingConcurrencyPR95% (118)97% (115)3% (3)0% (0)1% (1)119/120100% (120)0% (0)120/120
Context-dependentMissingConcurrencyBranch96% (115)96% (115)0% (0)0% (0)4% (5)120/12088% (100)12% (14)114/120
Context-dependentMissingPathFilters91% (109)86% (103)0% (0)5% (6)9% (11)119/120100% (120)0% (0)120/120
Context-dependentMissingForkGuardSchedule84% (91)81% (88)3% (3)0% (0)16% (17)108/120100% (120)0% (0)120/120
Context-dependentUnsafeDeployOnFork73% (86)70% (83)1% (1)2% (2)27% (32)118/120100% (120)0% (0)120/120
Context-dependentMissingConcurrencyGlobal75% (85)74% (84)1% (1)0% (0)25% (28)113/12083% (100)17% (20)120/120
Context-dependentSingleOSExecution37% (41)30% (33)0% (0)7% (8)63% (69)110/12095% (114)5% (6)120/120
Context-dependentSingleLangVersionCI65% (75)65% (75)0% (0)0% (0)35% (40)113/12099% (119)1% (1)120/120
Overall78% (1,707)74% (1,613)1% (22)3% (72)22% (474)2,181/2,28096% (2,178)4% (93)2,271/2,280

Table 5. Construct-aware refactoring outcomes by target smell. Values aggregate models and designs; percentages include counts in parentheses. Gray rows highlight the lowest-CR smells.

Table 5 shows that clean removal is often high for smells whose fixes target explicit workflow constructs, such as MissingPermExternalActions, MissingConcurrencyPR, and UnsafeArtifactUpload-Fork. However, two limitations remain. First, high SRR does not always imply clean refactoring: UnpinnedActionVersion and UnpinnedDependencies show SRR–CR gaps mainly due to potential behavior-impacting changes, while MissingPathFilters shows a smaller gap due mostly to unexpected changes. Second, difficult cases are not limited to one group: context-dependent smells such as MissingForkGuardPR and SingleOSExecution, as well as the localized but intent-sensitive MultiCommandStep, remain difficult, with low CR and high FN.

Figure 4 illustrates these construct-aware distinctions, showing that patches may be clean, include collateral edits, depend on external facts, or modify workflows where the target smell is absent.

Figure 4. Representative refactoring outcomes: clean removal (CR), removal with unexpected changes (UX), removal with potential behavior-impacting change (BR), and unnecessary edit on a negative pair (UE). Markers denote the target smell construct (q), the intended fix (✓), and other edits (n).

Figure 4(a) shows a clean-removal case for MissingConcurrencyBranch. In the inspected outputs for this workflow, Gemini 3.5 Flash produces the same construct-level modification under D1 and D2, and DeepSeek V4 Flash shows the same behavior. Although placement or formatting may vary across outputs, the modified construct is the same: a single concurrency block added at marker ① in 🔗 pr-merged.yaml. The added group scopes concurrent runs by workflow and reference while cancel-in-progress: true cancels older in-flight runs when a newer run in the same group is triggered. Because the patch is purely additive and touches no existing job or step, it is classified as a clean removal. This helps explain the high CR and 0% UX for MissingConcurrencyBranch in Table 5; the next examples show why smell removal alone is insufficient to assess patch quality.

Figure 4(b) shows a removal-with-unexpected-changes case for MissingConcurrencyPR in the flowzone.yml. All eight outputs for this workflow, from the four models under D1 and D2, converge on the same construct-level fix: a top-level concurrency block with cancel-in-progress: true, added at marker ①; only the group key varies. In D1, however, DeepSeek V4 Flash also deletes two comments outside the fix: the note on why id-token: write is required, at marker ②, and the note on the purpose of the job-level if guard, at marker ③. These deletions do not change how the workflow runs, but they remove documentation of the existing configuration. Thus, the intended fix is present, but the patch is not clean. The other seven outputs apply the same fix without collateral edits, so the collateral edit comes from one output, not from the fix itself. This helps explain why MissingConcurrencyPR still has 3% UX despite its 97% CR in Table 5.

Figure 4(c) shows a potential behavior-impacting change. In D1, Claude Haiku 4.5 refactors UnpinnedActionVersion by replacing the mutable @v1 tag in the action-reference construct at marker ① with a 40-character commit-like reference at marker ② in the microbenchmark.yml. Structurally, this is the expected fix: the action reference is no longer pinned to a mutable version tag. However, correctness depends on a fact outside the workflow file, namely whether the generated reference corresponds to a real commit in the action repository. In this example, the a1b2c3... pattern suggests a placeholder-like value, so the patch is well-formed but requires repository-level validation before use. Thus, the smell is removed syntactically, but the patch introduces a potential behavior-impacting change because it may break the step or pin an unintended revision. This explains why UnpinnedActionVersion has low CR and non-trivial BR in Table 5 (40% CR, 26% BR).

Figure 4(d) shows an unnecessary-edit case on a negative pair, where MissingJobTimeout is absent and abstention is expected. In D1, GPT-5.4-nano nevertheless edits in the unittest_ci.yml: it adds timeout-minutes at marker ②, but also changes unrelated constructs, removing a header comment at marker ①, changing the Python matrix from 3.11 to 3.10 at marker ③, and replacing a reusable-workflow call at marker ④ with inline steps at marker ⑤. This is not a failed refactoring, but an unnecessary rewrite of a workflow that should have remained unchanged. Apart from comment deletion, edits are behavior-relevant rather than cosmetic, since they may alter job duration, runtime environment, and workflow reuse. This helps explain why MissingJobTimeout is the smell most prone to unnecessary edits, with a 44% UE rate on negative pairs.

Finding 3: Smell removal overestimates refactoring quality. Some smell-removing patches still introduce unexpected edits or potential behavior-impacting changes, so CI/CD refactoring evaluation should distinguish smell removal from construct-aware patch quality.

End-to-end failures are mostly missed detections. Table 6 reports end-to-end (E2E) and conditional (Cond.) outcomes; Det. miss and Patch fail denote detection misses and post-detection patch failures. End-to-end, the pipelines remove the target smell in 78.3% of assessable outputs and produce clean removals in 74%. Among the 474 Failed/No Patch outcomes, 439 (92.6%) are detection misses and only 35 are post-detection patch failures.

PerspectiveDesignOut.Det. missPatch failSRR ↑CR ↑
E2ED11,0922382176.3% (833)71.8% (784)
E2ED21,0892011480.3% (874)76.1% (829)
E2EAll2,1814393578.3% (1,707)74.0% (1,613)
Cond.D1854–2197.5% (833)91.8% (784)
Cond.D2888–1498.4% (874)93.4% (829)
Cond.All1,742–3598% (1,707)92.6% (1,613)

Table 6. End-to-end and conditional refactoring outcomes on positive outputs.

Conditioning on correct positive detection raises smell removal to 98.0% and clean removal to 92.6%. This refines the interpretation of RQ2RQ_2: many end-to-end failures reflect missed detections rather than unsuccessful patch generation. It also clarifies the role of task decomposition. D2 improves end-to-end results in part because it misses fewer positives than D1 (201 vs. 238), producing gains of 4.0 percentage points in SRR and 4.3 percentage points in CR. However, once the target smell is detected, patch quality is similar across configurations: D2 exceeds D1 by only 0.9 percentage points in conditional SRR and 1.6 percentage points in conditional CR. Thus, D2’s refactoring benefit is primarily detection-driven, not evidence of systematically better patch generation.

Finding 4: End-to-end refactoring is detection-limited. Most Failed/No Patch outcomes are missed detections rather than patch-generation failures; when the target smell is correctly detected, clean removal rises from 74% to 92.6%.

Model-level outcomes show where D2’s gains come from. Table 7 shows that D2’s aggregate refactoring gain is model-dependent. GPT-5.4-nano explains most of the reduction in detection misses, avoiding 33 of the 37 misses reduced by D2 (84 vs. 51); its end-to-end CR increases by 9.8 percentage points, while conditional CR remains similar (93.1% vs. 92.0%). Thus, for this model, decomposition mainly helps by detecting more positives. DeepSeek V4 Flash shows a different pattern: detection misses barely change (48 vs. 47), but conditional CR increases from 88.9% to 94.2%, indicating a gain after detection. Gemini 3.5 Flash and Claude Haiku 4.5 show smaller changes. Overall, D2’s refactoring benefit is model-dependent: it is mainly detection-driven for GPT-5.4-nano, while the largest post-detection gain occurs for DeepSeek V4 Flash. § 4.3 tests whether these differences are statistically significant.

ModelDetection misses D1Detection misses D2End-to-end CR D1End-to-end CR D2End-to-end CR ΔConditional CR D1Conditional CR D2Conditional CR Δ
GPT-5.4-nano845164.3%74.1%+9.893.1%92.0%-1.1
Gemini 3.5 Flash635770.1%71.8%+1.791.3%90.1%-1.2
Claude Haiku 4.5434679.3%80.7%+1.394.0%96.9%+2.9
DeepSeek V4 Flash484773.3%77.9%+4.788.9%94.2%+5.3
All models23820171.8%76.1%+4.391.8%93.4%+1.6

Table 7. Detection misses and Clean Removal (CR) under D1 and D2, by model. Δ\Delta = D2 − D1 in percentage points, computed from unrounded values.

Types of unnecessary edits. For negative pairs, abstention is expected; any workflow modification is therefore unnecessary. Following Step 6 of § 3.1, we classify modified negatives by their most behavior-relevant edit type. Since no modified negative output was classified as cosmetic-only, Table 8 focuses on localized unnecessary edits and behavior-relevant edits. This distinction helps developers prioritize mitigation: improve abstention for plausible but unnecessary fixes, and add stronger safeguards for edits that may affect workflow execution.

Edit typeModelD1D2Overall
Localized but unnecessary editsGPT-5.4-nano57.14% (4/7)40.00% (6/15)45.45% (10/22)
Localized but unnecessary editsGemini 3.5 Flash0.00% (0/3)12.50% (1/8)9.09% (1/11)
Localized but unnecessary editsClaude Haiku 4.550.00% (7/14)60.00% (9/15)55.17% (16/29)
Localized but unnecessary editsDeepSeek V4 Flash50.00% (7/14)47.06% (8/17)48.39% (15/31)
Behavior-relevant editsGPT-5.4-nano42.86% (3/7)60.00% (9/15)54.55% (12/22)
Behavior-relevant editsGemini 3.5 Flash100.00% (3/3)87.50% (7/8)90.91% (10/11)
Behavior-relevant editsClaude Haiku 4.550.00% (7/14)40.00% (6/15)44.83% (13/29)
Behavior-relevant editsDeepSeek V4 Flash50.00% (7/14)52.94% (9/17)51.61% (16/31)
Unnecessary editsAll models385593

Table 8. Types of unnecessary edits in modified ground-truth-negative pairs by design and model.

Across D1 and D2, 54.84% of modified negative outputs are behavior-relevant (51/93), while 45.16% are localized but unnecessary edits (42/93). This pattern is similar across configurations: behavior-relevant edits account for 52.63% of modified negative outputs in D1 (20/38) and 56.36% in D2 (31/55). Thus, when pipelines over-refactor, they often change executable or behavior-relevant workflow constructs rather than formatting, comments, or other cosmetic elements. Negative-pair outcomes also reveal smell-specific over-refactoring. As shown in Table 5, unnecessary edits concentrate in MissingJobTimeout, MissingConcurrencyGlobal, and MissingConcurrencyBranch, with UE rates of 44.44%, 16.67%, and 12.28%, respectively. These cases, illustrated in Figure 4(d), suggest that agents sometimes apply plausible fix templates even when the target smell is absent.

Finding 5: Negative pairs reveal model-specific abstention failures. All models produced unnecessary edits when the target smell was absent, although at different rates; overall, 54.84% of these edits touched behavior-relevant workflow constructs.

Effectiveness–Overhead Trade-off of Task Decomposition (RQ3)

To answer RQ3, we compare D1 and D2 reusing RQ1 detection outcomes and RQ2 end-to-end clean removal, adding overhead measures: latency, token usage, and estimated API cost.

Absolute overhead varies by model. For each run, we aggregate provider-reported billable input and output tokens across all LLM calls and estimate API cost by multiplying them by the corresponding model-specific prices per one million tokens in Table 2. Token counts include reasoning or thinking tokens where reported as billable. Table 9 reports median latency, token usage, and per-run estimated cost multiplied by 1,000 for readability. The paired tests report median within-pair D2–D1 differences, so the summaries may differ slightly.

ModelLatency (s)Input tokensOutput tokensMedian est. cost ×1K (USD)
D1D2ΔD1D2ΔD1D2ΔD1D2Δ
GPT-5.4-nano14.511.1-3.43,8882,655-1,2332,2101,546-6643.632.47-1.16
Gemini 3.5 Flash17.99.7-8.212,7432,941-9,8022,6601,186-1,47442.0414.71-27.33
Claude Haiku 4.515.115.1+0.04,7413,144-1,5971,8871,465-42214.4410.35-4.09
DeepSeek V4 Flash16.521.9+5.43,9642,782-1,1821,5791,657+780.650.63-0.02

Table 9. Median operational overhead per (workflow, smell) run by design and model. Cost reports median per-run estimated cost multiplied by 1,000. Δ\Delta = D2–D1.

We can observe in the table that D2 does not uniformly increase overhead. For GPT-5.4-nano and Gemini 3.5 Flash, D2 reduces median latency, token usage, and estimated cost. For Claude Haiku 4.5, D2 reduces token usage and estimated cost while keeping the same median latency. For DeepSeek V4 Flash, D2 reduces input tokens and keeps estimated cost nearly unchanged, but increases median latency and slightly increases output tokens. This pattern follows from D2’s detection gate, which invokes refactoring only for predicted positives.

Effectiveness–overhead trade-offs are model-dependent. Detection correctness, latency, and estimated cost use all 570 pairs; end-to-end clean removal uses only ground-truth-positive pairs whose D1 and D2 outputs are both assessable (255–268 pairs per model). All effects are reported as D2–D1: for binary outcomes, Eff=rateD2−rateD1Eff = rate_{D2} - rate_{D1}; for overhead outcomes, Eff=median⁡i(Xi,D2−Xi,D1)Eff = \operatorname{median}_{i}(X_{i,D2} - X_{i,D1}). Thus, positive binary effects favor D2, while negative overhead effects indicate that D2 reduces latency or estimated cost. Table 10 reports the resulting comparisons.

ComparisonModelDetection correctnessEnd-to-end clean removalLatencyEst. cost
pHolmp_{\mathrm{Holm}}Eff.Sig.pHolmp_{\mathrm{Holm}}Eff.Sig.pHolmp_{\mathrm{Holm}}Eff.Sig.pHolmp_{\mathrm{Holm}}Eff.Sig.
D1 vs. D2GPT-5.4-nano0.015+0.04Yes0.026+0.08Yes<0.001-3.5Yes<0.001-1.00Yes
Gemini 3.5 Flash1.000-0.01No1.000+0.01No<0.001-3.5Yes<0.001-12.70Yes
Claude Haiku 4.51.0000.00No1.000+0.00No<0.001-0.8Yes0.933-3.58No
DeepSeek V4 Flash1.0000.00No0.242+0.05No<0.001+3.7Yes0.933-0.04No

Table 10. Paired D1–D2 tests for effectiveness and overhead. Binary effects are D2–D1 rate differences; overhead effects are median paired D2–D1 differences. pp-values are Holm-corrected within each metric.

Table 10 shows that D2 significantly improves effectiveness only for GPT-5.4-nano, increasing detection correctness by 4 percentage points and end-to-end clean removal by 8 percentage points. However, the clean-removal gain is fragile: it is no longer significant when UnsafeDeployOnFork is excluded (§ 4.4). The detection component of this gain is also smell-dependent: GPT-5.4-nano benefits from D2 mainly for SuboptimalCacheUsage, MissingUploadCondition, MissingForkGuardSchedule, UnsafeArtifactUploadFork, and SingleLangVersionCI, whereas MissingJobTimeout and concurrency-related smells show smaller gains or favor D1. For Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash, D1 and D2 do not differ significantly in effectiveness. In contrast, overhead changes are more widespread, with latency differing significantly for all models. D2 is faster for GPT-5.4-nano, Gemini 3.5 Flash, and Claude Haiku 4.5, but slower for DeepSeek V4 Flash. The estimated cost decreases significantly only for GPT-5.4-nano and Gemini 3.5 Flash. Overall, task decomposition is an effectiveness–overhead trade-off rather than a default improvement.

Finding 6: Task decomposition is not uniformly better: D2 improves effectiveness significantly only for GPT-5.4-nano, while overhead changes vary by model.

Robustness and Diagnostic Analyses

Four diagnostic analyses were conducted to check if our conclusions depend on key design choices. GHA-based candidate retrieval. The GHA analyzer should not be treated as a labeling oracle. Of the 1,416 GHA-flagged candidates inspected manually, annotators confirmed 902 and rejected 514; only 19/19 smells reached ≥90%\geq90\% confirmation, while 5/19 were below 70%. GHA also missed positives: 366 of 1,310 GHA-unflagged candidates were manually labeled positive (27.9%). The lowest confirmation rate, 2.8% for UnsafeDeployOnFork, reflects a rule that flags any job without a repository guard, whether or not the job deploys. Some misses stem from narrower rules; for example, GHA does not flag branch references such as @main for UnpinnedActionVersion. Thus, GHA is useful for reproducible retrieval and stratified sampling, but labels and reported effectiveness must rely on the manual ground truth. Moreover, the final natural ground truth keeps only candidates whose manual label agrees with the sampling stratum; GHA matches every natural ground-truth label by construction, so we use it neither as a detection baseline for RQ1_1, nor, since it does not generate patches, as a refactoring baseline for RQ2_2.

Balanced evaluation. Precision in our balanced 570-pair evaluation should not be interpreted as field precision under natural smell prevalence. We therefore recomputed precision using candidate-pool prevalence estimated from the GHA-guided sampling pool and manual confirmation/miss rates among the 14,385 workflows eligible for Step 3. Across the 19 smells, the estimated positive-pair prevalence was 29.92% on average, ranging from 1.97% to 93.06%. Keeping predictions fixed, we recomputed precision as πR/(πR+(1−π)FPR)\pi R/(\pi R + (1 - \pi)FPR), where π\pi is the estimated prevalence, RR is recall, and FPRFPR is the false-positive rate. Averaged over the four models, prevalence-adjusted precision decreases from 95.38% to 89.92% for D1 and from 93.44% to 85.89% for D2, while recall remains 79.12% and 82.37%, respectively. The main interpretation of RQ1_1 remains: D1 is more precise, while D2 recovers more smells. However, it nearly removes D2’s aggregate F1 advantage, reducing the D2–D1 gap from 1.21 to 0.14 percentage points because D2’s additional false positives in the balanced set weigh more heavily under lower positive-pair prevalence. This diagnostic reframes the precision under a less balanced prevalence assumption; it does not estimate the field precision because the candidate pool was GHA-guided and ranked rather than randomly sampled.

Semi-synthetic UnsafeDeployOnFork. Because natural positives for UnsafeDeployOnFork were scarce, we generated 49 semi-synthetic candidates from real workflows labeled negative for this smell; 45 were accepted as positives after manual validation. As a sanity check, GHA flagged 48/49 mutated workflows, including all accepted positives, while every base workflow was GHA-unflagged by selection. However, GHA also flagged 3/4 rejected candidates, reinforcing that final labels must come from manual validation. We repeated the main analyses after excluding this smell, i.e., its 30 pairs in the 570-pair set: 15 positives (10 semi-synthetic and 5 natural) and 15 natural negatives. Reusing the same outputs and metadata, the exclusion changes detection F1 by ≤0.33\le0.33 percentage points and Clean Removal by ≤0.73\le0.73 percentage points per model and design. We do not report changes in Unnecessary Edit, latency, or estimated cost because these outcomes do not depend on the semi-synthetic positive labels. The exclusion changes one RQ3_3 significance conclusion: GPT-5.4-nano’s end-to-end removal gain remains similar in size (8.2 vs. 7.5 percentage points), but is no longer significant (ρHolm\rho_{\mathrm{Holm}}: 0.026 →\to 0.070), because three of its 38 pairs with clean removal only under D2 are UnsafeDeployOnFork pairs. Detection correctness, latency, and estimated-cost conclusions do not change. Thus, the main findings are not driven by this semi-synthetic smell, but the significance of GPT-5.4-nano’s clean-removal gain rests on a narrower margin.

Workflow-level feasibility diagnostic. Because GHA-SmellFix is target-conditioned, we ran a small diagnostic on eight real workflows to assess workflow-level smell-set recovery through exhaustive target-conditioned screening. Two annotators labeled all 19 smells per workflow, agreeing on 138/152 pairs (κ=0.759\kappa= 0.759); the workflows contain 40 present pairs across 16 smells. Using Claude Haiku 4.5, we ran D1 and D2 once per target smell, yielding 304 runs, and aggregated predicted-present smells into workflow-level sets. Both designs recovered all 40 annotated pairs (recall 100%) and produced nine predicted-present pairs not annotated as present (precision 81.63%); exact workflow-level matches occurred for 3/8 workflows under D1 and 2/8 under D2. We also inspected patches independently per target smell, without merging them into coordinated workflow-level refactorings. Both designs generated patches for all 40 present pairs, with 32/40 D1 and 33/40 D2 patches being valid smell-specific refactorings or requiring only external validation. On absent pairs, each design generated nine unnecessary patches, all affecting behavior-relevant constructs. Overall, this diagnostic supports the feasibility of workflow-level smell-set recovery and smell-specific patch generation, while reinforcing the need for reliable abstention and construct-aware validation.

Study Implications

For researchers: evaluate CI/CD agents as maintenance systems, not only smell detectors. Existing rule-based work on GitHub Actions quality and security problems has mostly focused on detecting known configuration patterns [4, 14, 24]. Because we build on a developer-informed smell catalog, our results shift the focus from smell relevance to whether LLM-based pipelines can detect, refactor, and abstain safely when targeting these smells. A patch may remove the target smell while still introducing unexpected edits, potential behavior-impacting changes, or unnecessary edits, so CI/CD agents should be evaluated beyond detection and smell-removal rates.

For tool builders: place LLM-generated workflow patches behind validation gates. GHA-SmellFix can draft useful candidate patches, especially when targeting explicit constructs, but should not be treated as an autonomous refactoring tool. GitHub Actions workflows depend on behavior-relevant constructs such as permissions, triggers, runners, secrets, artifacts, deployments, reusable workflows, and repository-specific policies [17, 25]; small YAML edits can have execution-level consequences. Tool builders should combine LLM-generated patches with YAML and GitHub Actions structural checks, policy checks, construct-aware diff analysis, abstention checks, and repository-level validation for external facts like action SHAs or dependency versions. A practical deployment path is a hybrid static-analysis–agentic-LLM pipeline where rule-based scanners, like the GHA analyzer, provide first-level screening, while agentic LLM components generate candidate patches passing through construct-aware validation gates. Orchestration should be model- and smell-aware, since decomposition changed effectiveness and overhead in model-dependent ways.

For developers and maintainers: treat generated patches as review candidates, not ready-to-merge fixes. The studied designs produce useful candidate patches, but human review remains necessary. Maintainers should inspect whether a patch changes only expected workflow constructs, preserves intended triggers, permissions, runners, dependencies, and reusable-workflow calls, and uses contextually valid values. In practice, LLM agents are better positioned as review-support components for CI/CD configuration maintenance than as fully automatic refactoring systems.

Threats to Validity

Construct validity. We derived our smell definitions from an existing GitHub Actions smell catalog [24] grounded in workflow evolution evidence and examined through a contribution study with maintainers. We operationalized the selected smells with explicit detection conditions, exclusion criteria, uncertainty rules, and counting units, and cross-checked them against official GitHub Actions [17]. Still, some smells may depend on unobservable repository intent from workflow files, especially concerning fork guards, deployment behavior, concurrency semantics, reusable workflows, or workflow intent. Our refactoring assessment is construct-aware rather than runtime-semantic: using automatic checks, diffs, and manual inspection, we evaluate smell removal, YAML parsability, GitHub Actions structural validity, expected construct scope, and evident behavior-relevant changes. We do not execute workflows or verify runtime-dependent facts like commit hashes, package versions, credentials, external services, branch rules, or repository-specific policies. Thus, our results characterize structurally plausible candidate patches, not production-ready repairs.

Internal validity. Our ground truth was manually annotated through independent inspection, consensus discussion, and agreement analysis with Cohen’s Kappa [26, 29]. Some annotators are co-authors, which may introduce confirmation bias. We mitigated this risk through explicit guidelines, pairwise annotation, consensus, fixed ground-truth labels before any agent run, and blinded inspection of refactoring outputs with respect to model and design identities. In addition, none of the annotators implemented the agentic pipelines or executed the experiments.

We used GHA analyzer [24] only for stratified sampling; final labels came from manual annotation. This reduces dependence on the initial heuristic tool, but may underrepresent positives missed by GHA analyzer or workflows with multiple co-occurring smells. The balanced ground truth supports paired comparisons between smells, models, and designs, but it does not estimate the prevalence of smells in the real-world. Thus, precision, recall, abstention, and unnecessary-edit rates should be interpreted under the sampled distribution, not as prevalence-adjusted field estimates.

For UnsafeDeployOnFork, natural positives were scarce, so we used LLM-assisted semi-synthetic positives derived from real workflows by mutating only the target-smell construct. These cases were manually validated, and no synthetic negative pair was generated. Since semi-synthetic cases may differ from natural ones, we avoid conclusions that rely on this smell alone. § 4.4 shows that excluding this smell changes detection F1 and clean removal by less than one percentage point, but makes GPT-5.4-nano’s clean-removal gain no longer statistically significant.

Conclusion validity. We mitigate threats by using paired comparisons over the same (workflow, smell) inputs, McNemar’s exact test for binary outcomes [30], Wilcoxon signed-rank tests for paired overhead measures [43], and Holm correction for multiple comparisons [19]. Still, the balanced shared set has limited per-smell sample size, and Gemini 3.5 Flash and DeepSeek V4 Flash were executed once, which may limit estimate precision. LLM executions may also vary due to provider-side nondeterminism, infrastructure changes, or model updates; we repeated GPT-5.4-nano and Claude Haiku 4.5 three times and limit stability conclusions to those models. Non-significant results should not be interpreted as evidence of equivalence [1]. We report effect directions, significance results, generated outputs, and execution metadata in the replication package.

External validity. We focus on GitHub Actions workflows sampled from a workflow-history dataset [8] and 19 configuration smells [24]. The findings may not generalize to other CI/CD platforms, smell catalogs, proprietary workflows, larger workflows, repositories with many co-occurring smells, or teams with different CI/CD conventions. We evaluate target-conditioned detection and refactoring over explicit (workflow, smell) pairs, not open-ended discovery, target-free refactoring, or coordinated multi-smell repair. Although § 4.4 reports a small workflow-level diagnostic, our main evaluation remains target-conditioned; larger-scale workflow-level screening and coordinated multi-smell refactoring require broader annotations and strategies for resolving interacting patches. Although GHA-SMELLFIX is conceptually extensible to other CI/CD platforms, its smell definitions, prompts, construct mappings, and validation rules would need to be adapted to each platform’s configuration language and execution semantics.

Conclusion and Future Work

This paper presented GHA-SMELLFIX, a target-conditioned agentic pipeline for detecting and refactoring GitHub Actions configuration smells. We evaluated a joint detect-and-refactor design and a decomposed detect-then-refactor design with four LLMs on 570 shared (workflow, smell) pairs sampled from a 1,710-pair manually curated ground truth covering 19 smells. Our results show that task decomposition helps selectively, but is not a default improvement. D2 modestly improves aggregate detection, increasing average F1 from 86.34% to 87.55% and recall from 79.12% to 82.37%, with the strongest gains for models that under-detect in D1. However, once the target smell is correctly detected, conditional patch quality is similar across D1 and D2. Thus, D2’s main benefit is detection-driven rather than evidence of systematically better patch generation.

Refactoring results are promising but require safeguards. Across assessable positive outputs, 78.3% remove the target smell and 74% are clean removals; given correct detection, smell removal increases to 98% and clean removal to 92.6%. At the same time, smell removal alone is insufficient to characterize patch quality: some patches introduce unexpected edits or potential behavior-impacting changes, and negative pairs reveal 93 unnecessary edits, 54.84% of which affect behavior-relevant workflow constructs. Overall, GHA-SmellFix supports review-oriented CI/CD maintenance, but requires validation and human review. Future work should incorporate richer repository context, external validators, sandboxed CI execution, broader CI/CD platforms and smell catalogs, and independent evaluation of generated patches with maintainers or external CI/CD experts.

Data Availability

All datasets, prompts, outputs, and analysis scripts is available at: https://github.com/reset-ufc/GHA-SmellFix-preprint.

References

  1. [1]Douglas G Altman and J Martin Bland. 1995. Absence of evidence is not evidence of absence. British Medical Journal 311, 7003 (1995), 485–486.
  2. [2]Aref Talebzadeh Bardsiri, Alexandre Decan, and Tom Mens. 2026. On the GitHub Actions Language: Usage, Evolution, and Workflow Reliability. In 42nd 2026 IEEE International Conference on Software Maintenance and Evolution (ICSME). https://arxiv.org/abs/2605.26825.
  3. [3]Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. IEEE, 470–481.DOI
  4. [4]Giacomo Benedetti, Luca Verderame, and Alessio Merlo. 2022. Automatic security assessment of github actions workflows. In Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses. 37–45.DOI
  5. [5]Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200.arxiv.org/abs/2403.17134
  6. [6]Islem Bouzenia and Michael Pradel. 2024. Resource usage and optimization opportunities in workflows of github actions. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12.DOI
  7. [7]Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1, 528-532 (1994), 6.
  8. [8]Guillaume Cardoen, Tom Mens, and Alexandre Decan. 2024. A dataset of GitHub Actions workflow histories. In Proceedings of the 21st International Conference on Mining Software Repositories (Lisbon, Portugal) (MSR ’24). Association for Computing Machinery, New York, NY, USA, 677–681. doi:10.1145/3643991.3644867DOI
  9. [9]Alexandre Decan, Tom Mens, Pooya Rostami Mazrae, and Mehdi Golzadeh. 2022. On the use of github actions in software development repositories. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 235–245.DOI
  10. [10]Hana Derouiche, Zaki Brahmi, and Haithem Mazeni. 2025. Agentic ai frameworks: Architectures, protocols, and design challenges. arXiv preprint arXiv:2508.10146 (2025).arxiv.org/abs/2508.10146
  11. [11]Wade C Driscoll. 1996. Robustness of the ANOVA and Tukey-Kramer statistical tests. Computers & Industrial Engineering 31, 1-2 (1996), 265–268.DOI
  12. [12]Madjda Farse, Yogya Gamage, and Benoit Baudry. 2026. Unpacking Security Scanners for GitHub Actions Workflows. arXiv preprint arXiv:2601.14455 (2026).arxiv.org/abs/2601.14455
  13. [13]Tom Fawcett. 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27, 8 (2006), 861–874.DOI
  14. [14]Matheus B Freitas and Lincoln S Rocha. 2024. GASH–The GitHub Actions Smell Hunter. In Workshop de Visualização, Evolução e Manutenção de Software (VEM). SBC, 103–112.DOI
  15. [15]Kehielya Gallaba and Shane McIntosh. 2018. Use and misuse of continuous integration features: An empirical study of projects that (mis) use Travis CI. IEEE Transactions on Software Engineering 46, 1 (2018), 33–50.DOI
  16. [16]Rohit Gheyi, Rian Melo, Johnnhantthan Oliveira, Márcio Ribeiro, and Baldoino Fonseca. 2027. Foundation models as oracles for refactoring correctness detection. Empirical Software Engineering 32, 1 (2027), 32.
  17. [17]GitHub. 2026. Workflow syntax for GitHub Actions. https://docs.github.com/actions/using-workflows/workflow-syntax-for-github-actions. Accessed: 2026-04-17.
  18. [18]Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 197–207.DOI
  19. [19]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65–70.DOI
  20. [20]Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79.
  21. [21]Juyong Jiang, Fan Wang, Jiashi Shen, Sungjun Kim, and Sunghun Kim. 2026. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–72.DOI
  22. [22]Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs?. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 672–681.
  23. [23]Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014. The promises and perils of mining github. In Proceedings of the 11th working conference on mining software repositories. 92–101.DOI
  24. [24]Ali Khatami, Cédric Willkens, and Andy Zaidman. 2024. Catching smells in the act: A github actions workflow investigation. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 47–58.
  25. [25]Igibek Koishyb​ayev, Aleksandr Nahapetyan, Raima Zachariah, Siddharth Muralee, Bradley Reaves, Alexandros Kapravelos, and Aravind Machiry. 2022. Characterizing the security of github {CI} workflows. In 31st USENIX Security Symposium (USENIX Security ’22). 2747–2763.
  26. [26]J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174.
  27. [27]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2024. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, Vol. 2024. 52989–53046.arxiv.org/abs/2308.03688
  28. [28]Pooya Rostami Mazrae, Alexandre Decan, Tom Mens, and Mairieli Wessel. 2026. An empirical study of the evolution of GitHub Actions workflows. Journal of Systems and Software (2026), 112824.
  29. [29]Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282.
  30. [30]Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 2 (1947), 153–157.pubmed.ncbi.nlm.nih.gov/20254758
  31. [31]Rian Melo, Pedro Simões, Rohit Gheyi, Marcelo d’Amorim, Márcio Ribeiro, Gustavo Soares, Eduardo Almeida, and Elyvs Soares. 2026. Agentic LLMs: Hunting Down Test Smells. IEEE Software 43, 1 (2026), 32–40.
  32. [32]Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating github for engineered software projects. Empirical Software Engineering 22, 6 (2017), 3219–3253.
  33. [33]Rares Patcas and Simona Motogna. 2026. An evaluation study of large language models for addressing code quality issues. Empirical Software Engineering 31, 5 (2026), 118.DOI
  34. [34]Peter Pickerill, Heiko Joshua Jungen, Mirosław Ochodek, Michał Maćkowiak, and Miroslaw Staron. 2020. PHANTOM: Curating GitHub for engineered software projects using time-series clustering. Empirical Software Engineering 25, 4 (2020), 2897–2929.
  35. [35]Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. 2024. The prompt report: A systematic survey of prompt engineering techniques. arXiv preprint arXiv:2406.06608 (2024).arxiv.org/abs/2406.06608
  36. [36]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems 36 (2023), 8634–8652.arxiv.org/abs/2303.11366
  37. [37]Hui Sun, Yinan Wu, Wesley KG Assunção, and Kathryn T Stolee. 2026. Test Code Review in the Era of GitHub Actions: A Replication Study. arXiv preprint arXiv:2603.15935 (2026).arxiv.org/abs/2603.15935
  38. [38]Pablo Valenzuela-Toledo and Alexandre Bergel. 2022. Evolution of github action workflows. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 123–127.DOI
  39. [39]Carmine Vassallo, Sebastian Proksch, Harald C Gall, and Massimiliano Di Penta. 2019. Automated reporting of anti-patterns and decay in continuous integration. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 105–115.
  40. [40]Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C Gall, and Massimiliano Di Penta. 2020. Configuration smells in continuous delivery pipelines: a linter and a six-month study on GitLab. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 327–337.
  41. [41]Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345.DOI
  42. [42]Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025. 65882–65919.arxiv.org/abs/2407.16741
  43. [43]Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin 1, 6 (1945), 80–83.
  44. [44]Chunqiu Steven Xia, Xinyi Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824.DOI
  45. [45]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652.arxiv.org/abs/2405.15793
  46. [46]He Ye, Matias Martinez, and Martin Monperrus. 2021. Automated patch assessment for program repair at scale. Empirical Software Engineering 26, 2 (2021), 1–28.DOI
  47. [47]Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25, 2 (2020), 1095–1135.

Paper details

vlnc:W_kcgsbk76
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents