betaFeedback

WorksComputer Science

Hybrid LLM-Deterministic Pipelines for Enterprise-Scale Code Migration at Bloomberg

Abstract

Large organizations carry a long tail of infrastructure migrations that are worth doing but rarely urgent enough to compete with feature work, because each one costs significant engineering time per repository. This paper reports on our industrial experience from Automated Uplift, a multi-month initiative applying LLM-assisted migration pipelines across thousands of Bloomberg repositories. We examine two distinct production campaigns: a heterogeneous Python packaging migration (approx. 1,900 candidate repositories) and a uniform C++ build system migration (approx. 6,700 candidate repositories). Together, these campaigns generated 2,333 pull requests, resulting in 729 successful migrations. We detail our hybrid architectural solution which maximizes deterministic processing, confines the LLM to bounded file transformations within a pre-computed context, and incorporates runtime production telemetry (such as execution logs and usage traces). Our findings show that generating migrations turned out to be the easier part; the bottleneck was the cost, for owning teams, of reviewing, testing, and deploying them. These costs were lowered when changes could be verified directly in the pipeline before opening a PR (as with C++ compilation and test validation), and when the population was uniform enough for fixes to carry over from one repository to the next. We conclude by distilling our experience and findings into actionable, transferable patterns for architecting hybrid LLM-deterministic pipelines and scaling code migrations across the enterprise

Introduction

Software maintenance is a large and ongoing part of the cost of software. For long-lived systems, maintenance includes not only changes to application code but also keeping the surrounding infrastructure current. Build systems become obsolete, packaging formats evolve, frameworks are deprecated, and language runtimes advance, yet the production code persists, often for decades. Modernizing this substrate is unavoidable: unmigrated repositories accrue escalating maintenance costs, prevent performance improvements, and create operational fragility.

The challenge is scale. A large enterprise may maintain thousands of repositories spanning multiple technology stacks, each with its own build configuration, dependency graph, and deployment topology. Migrating even one repository requires understanding its structure, runtime dependencies, production deployment context, and accumulated conventions. Multiplying these requirements across thousands of repositories far exceeds the manual effort that the benefit would justify.

Large Language Models (LLMs) have demonstrated impressive code transformation capabilities, and recent work has explored their application to migration tasks ranging from language translation [1], [2] to API migration [3] to enterprise-scale change list generation [4].

However, applying out-of-the-box LLMs directly to repository migration tasks breaks down at scale. Benchmark evaluations reveal sharp degradation with repository complexity: Li et al. [5] report pass rates dropping from 91.3% for repositories under 10,000 lines of code to 15.3% for repositories exceeding 50,000 lines on RepoMod-Bench. Structurally, successful migration requires information absent from source code, such as which interpreter version runs in production, which dependency versions are deployed to which clusters, and which scripts are actually invoked at runtime versus dormant.

Scale is not the only obstacle; priority is another. Like most large engineering organizations, Bloomberg has a record of executing company-wide migrations and keeping its codebase up to date: urgent fleet-wide patches, end-of-life runtimes, and deprecations with hard deadlines are funded, centrally coordinated, and tracked to completion. Below these high-profile cases sits a long tail of migrations that are worthwhile but never urgent enough to compete with feature work, such as moving off an earlier-generation build tool that still works or adopting a packaging model that reduces operational risk. Individually, each is low priority, but collectively they represent a significant opportunity to simplify and modernize the organization’s infrastructure. The economics explain the backlog: when the migration cost per repository is high, only the most important ones clear the bar. This raises the question that motivates our work: if automation drastically lowers the cost of producing a migration, can an organization take on migrations that were previously not worth doing?

Attempting to answer this question raises a more practical one: how does one build a reliable, cost-effective migration system that operates across thousands of diverse repositories, each with its own structural idiosyncrasies and production context?

At Bloomberg, we call this kind of work uplift: modernizing infrastructure at the repository level by moving repositories from earlier-generation tooling to current equivalents. Uplift does not change what the code does; it changes how it is built, packaged, deployed, and maintained, with the aim of lowering its future maintenance cost. Historically, a central team driving a low-priority uplift would announce it, publish instructions and an FAQ, and ask each owning team to perform the migration themselves; for example, the owners of the earlier-generation proprietary C++ build tool would ask every team to move to CMake. Each team then rediscovered the same edge cases independently. Our work improves that model: the central team encodes its instructions for a pipeline that generates the migration PRs, and owning teams review and deploy them.

This paper presents our industrial experience from Automated Uplift, a project at Bloomberg aimed at answering these questions. Over six months of campaigns, we developed and applied LLM-assisted migration pipelines at Bloomberg. We piloted across five technology domains, and in this paper we report on the two that grew into sustained campaigns. These campaigns differ on several key dimensions that matter most for automation: how uniform the source population is, how deep the validation feedback loop is (§ 2-B), and therefore what role the LLM plays:

  1. Python packaging migration (“PyPackaging”): Migrating Python packages from a proprietary shared-environment packaging model to PyPackaging,1 an internal dependency-isolated model in which each package declares and bundles its own dependencies. The source population is heterogeneous, and the pipeline’s checks give only partial confidence in a change. This campaign targets ≈1,900\approx1{,}900 candidate repositories (with 294 PRs generated).

  1. C++ build system migration (“CppBuild-to-CMake”): Migrating C++ projects from CppBuild, an internal proprietary build system, to CMake, including restructuring source trees, translating build configurations, and resolving dependency declarations. The source population is uniform, and the pipeline’s checks give high confidence in a change. This campaign targets ≈6,700\approx6{,}700 candidate repositories (with 2,039 PRs generated).

Together, the two campaigns target approximately 8,600 repositories and generated more than 2,300 pull requests. Both involve semantic adaptation beyond syntactic transformation: understanding what the code does in production, how it is deployed, and what it depends on at runtime. Both pipelines also share a common design principle that underlies our hybrid architecture: maximize deterministic processing and confine the LLM to file transformation within a fully characterized, pre-computed context. The overall pipeline is deterministic, and the LLM handles one bounded task within it. Moreover, we provide the LLM with “migration context” from runtime production data, such as execution logs, module usage traces, and cluster-specific dependency versions, cross-referencing AST-inferred imports against runtime-observed usage.

We structure our study around three research questions:

  • RQ1: How should AI-assisted migration pipelines partition work between deterministic tooling and LLM-driven steps? We look at how the work is split in our two pipelines and how that split changed as models improved during the study.

  • RQ2: What is the impact of incorporating runtime production data on migration accuracy? We describe how production telemetry feeds the migration context and what changed once we added it, compared to relying on source code alone.

  • RQ3: What are the dominant bottlenecks in fleet-scale migration campaigns? We analyze the outcomes of more than 2,300 PRs across both campaigns to find where migrations stall once PRs are generated.

Over the study period, the two campaigns generated 2,333 PRs. Two months or more after being opened, 34.2% of CppBuild-to-CMake PRs and 10.5% of PyPackaging PRs had been merged, totaling 729 migrations. These rates can look low, but they should be read against the alternative: these are low-priority migrations that, without automation, would have stayed in each owning team’s backlog, waiting to be prioritized. By the team’s estimates, the merged changes alone represent well over a thousand person-days of manual migration work. Most unmerged PRs were not rejected but simply received no decision, and 34 of the 51 PyPackaging repositories where our PR was rejected later completed the migration themselves. Runtime production data proved essential for packaging migrations. Finally, the main constraint was not generating changes but the cost for owning teams to verify and deploy them, which was lowest when the pipeline could verify a change itself before opening a PR.

In summary, our contributions are:

  1. Exploring hybrid LLM-deterministic pipelines for enterprise-scale code migration, where deterministic phases handle classification, data collection, and validation, while LLMs perform creative transformation within a constrained scope (RQ1).

  1. Exploring the use of runtime-data-enriched migration context: augmenting LLM prompts with production telemetry, including interpreter startup logs, module usage traces, and cluster-specific dependency versions, reconciling statically inferred imports with runtime-observed module usage (RQ2).

  1. Exploring fleet-scale migration operations across two campaigns covering more than 2,300 pull requests, un-

veiling that developer engagement and PR adoption, not generation quality, is the primary operational constraint at scale for this kind of migration (RQ3).

  1. Sharing the lessons learned from our experience on engineering hybrid AI-assisted migration systems in enterprise environments.

Background

This section provides context on the infrastructure our pipelines draw on and the concept of validation feedback loops that we use to compare the two campaigns.

Organizational Context

Bloomberg operates its own internal infrastructure for package management, build orchestration, deployment, and production telemetry. Packages are distributed through an internal binary package registry, each with its own versioning, dependency resolution, and deployment semantics. Production telemetry, including interpreter startup logs, module usage traces, and process-level resource consumption, is captured by a centralized observability platform. This provides a runtime view of how code actually executes in production. The infrastructure serves as a data source with no analog in open source or cloud native environments: the ability to observe, at fleet scale, which scripts run, which modules they import, and which dependency versions are loaded on which clusters. Section III describes how the system incorporates this telemetry into the migration pipeline.

Validation Feedback Loops

Every automated migration needs a mechanism to tell whether a generated change is correct. We call the checks that do this validation feedback loops, and we describe their depth as how far an error can travel between the step that introduces it and the first check that can catch it. In a shallow pipeline, each step hands its output to a check that validates it before the next step runs, so the migration is verified as a series of short loops, all within the pipeline. CppBuild-to-CMake works this way: an extraction error is caught by schema validation, a template error by the build, and a behavioral change by running unit tests, before any PR is opened. In a deep pipeline, intermediate checks cannot confirm that earlier steps were right, and the only complete check comes at the end. For PyPackaging, a package can build and pass smoke checks such as --help, yet a wrong dependency choice made at the start only surfaces when the owning team deploys and runs the application. When that happens, the migration must be reworked from the start. Section IV-C1 examines how this difference shaped the outcomes of the two campaigns.

System Architecture

This section describes the two pipelines. We start with the structure they share, then cover what is specific to each, and close with how the split between deterministic and LLM-driven work evolved during the project. Table I summarizes the differences.

CppBuild-to-CMakePyPackaging
Source populationUniform (CppBuild makefiles)Heterogeneous (18 categories)
LLM roleExtracts structured data; templates write the codeWrites the migration directly
Runtime telemetryNot neededCentral input
Pre-PR validationSchema, build, unit testsConfiguration rules, packaging, --help diff
Feedback loopShallowDeep
Primary modelClaude Sonnet 4.5Claude Opus 4.6
Mean agent run∼13 turns, 48 s∼63 turns, 7.2 min

Table I. Core Differences Across the Two Campaigns

Pipeline shared by both campaigns

Figure 1. The pipeline shared by both campaigns. Grey stages are deterministic; blue stages call the LLM.

Progress on both campaigns is recorded in a centralized initiative tracker. Each target repository appears as a task assigned to its owning team, and the task’s status updates automatically as the migration lands in the repository, so both the central team and owning teams can see where each migration stands.

Hybrid LLM-Deterministic Pipelines for Code Migration

Both pipelines run the same five stages for each repository (as depicted in Figure 1):

  1. Classify and collect (deterministic). Assign the repository to an archetype, a class of repositories that share a structure and therefore the same migration approach (e.g., Python scripts with no packaging metadata), and assemble the migration context: metadata, build configuration, source analysis and, for PyPackaging, runtime telemetry.

  2. Render prompt (deterministic). Turn the context into a prompt with an archetype-specific template. The same context always produces the same prompt.

  3. Execute (LLM). Perform the migration, either by editing files (PyPackaging) or by extracting structured data (CMake).

  4. Validate (deterministic). Check the result against rules derived from past failures. If a check fails, re-invoke the LLM with the errors provided as additional context, up to three times.

  5. Describe (LLM). A read-only invocation writes the PR description from the migration evidence, including any warnings left after validation.

The principle behind this structure is intuitive: anything that must be reproducible or auditable across the fleet is deterministic, and the LLM only handles the part that varies from one repository to the next. Validation is never delegated to the model. Each migration runs in its own container, and the central team reviewed generated PRs in batches before handing them to owning teams. This step was meant to capture clear mistakes before even asking owning teams to review.

ArchetypeTasksPRsRate
Company-Defined Build Configuration1,877N/A2N/A
Team-Defined Build Configuration2,8891,28344.4%
Code Generation1,35585463.0%

Table II. CppBuild-to-CMake PR Generation by Archetype

PyPackaging: The LLM Writes the Migration

PyPackaging input candidates vary widely, from bare scripts with no build system to mixed C++/Python repositories, so the LLM has to write the migration itself: the packaging configuration, source layout changes, and build updates.

Archetypes. Each package is classified by looking it up in a precomputed registry built from the production asset registry, the package registry and the initiative tracker. We had tried LLM-based classification in an earlier campaign and dropped it: the same package could land in different archetypes on different runs, failures were harder to debug, and calling a model for every candidate in the fleet was wasteful. The 1,880 PyPackaging tasks fall into 18 categories. Three categories (covering 337 tasks) have dedicated prompt templates (500–600 lines each), data collectors, and validation rules. Two cover single-script packages: current build, no metadata (already on the current build system, missing only packaging metadata) and proprietary packaging only (shipped through the proprietary shared-environment model and packaged from scratch). The third, broken existing packaging, covers repositories with a partial or misconfigured setup. Everything else gets a generic prompt that produces a starting-point PR, which is labeled as such for the owning team. Archetypes with ≈20 or fewer repositories did not justify developing a custom template.

Runtime data. The migration context draws on 12 data sources, which fall into four groups: runtime telemetry from the observability platform (interpreter startup logs and module usage traces), package metadata, AST-based source analysis and build configuration. The key use is dependency resolution. Imports found by AST scanning are cross-referenced with modules observed at runtime. Dependencies seen in both are validated and given the version available for the application in the deployed environment. Dependencies seen only in the source may be conditional, test-only, or dead, so they are passed to the LLM without a pin. Modules seen only at runtime are usually transitive and are left out.

Import-to-package resolution. Module usage traces and source analysis both tell us which modules an application imports, but a migration has to declare the canonical package names that provide them, and in Python the two names often differ: PIL comes from pillow, yaml from PyYAML, and a namespace such as google is split across many packages. A wrong choice still builds and passes our checks, and only fails once the application runs (Section II-B). Rather than resolving names separately for each repository, we built one fleet-wide mapping from the modules observed in production to the packages that provide them, using public import-to-package metadata [6] for open source packages and the internal package index for internal ones.

Execution and validation. The agent runs with a fixed set of file and shell tools, no network access, and is locked to the repository directory. After it finishes, 7 rules check for oversights we had seen repeatedly in prior experiments, such as legacy configuration left in place or files copied instead of linked. The pipeline also performs a simple import validation for CLI scripts by invoking them with --help before and after the migration, and compares the packaged layout with the current deployment. These checks catch packaging mistakes but not runtime failures, which is what makes this feedback loop deep (Section II-B).

CppBuild-to-CMake: LLM Extracts, Templates Write

CppBuild makefiles follow a well-defined format, so the LLM never writes build files. It reads the makefile and extracts sources, libraries, dependencies, flags and custom rules into a JSON object validated against a schema. Templates then generate the CMakeLists.txt. A wrong build file is therefore either an extraction error, fixed by tightening the schema or prompt, or a template bug, fixed once for every repository. Validation checks the schema, builds the project, and runs its unit tests. No runtime telemetry is needed, since the build configuration does not depend on runtime behavior.

The CMake backlog splits into three archetypes (shown in Table II). One of them, company-defined build configurations, follows a fully deterministic path: a dedicated generator converts them with no LLM involvement at all.

How the Boundary Moved

Over the project, we updated the underlying model from Claude Sonnet 4.5 to Opus 4.6, and the split between deterministic and LLM-driven work moved with it, in both directions.

As models improved, we relaxed some deterministic pre-processing, such as AST-based dependency handling that compensated for earlier models’ unreliable import analysis, and replaced concise instructions with richer reference documents. The limiting factor became the precision of our documentation, rather than the models’ capabilities. In the other direction, we moved archetype classification for PyPackaging tasks from the LLM in our early experiments to a registry for the reasons above (§ 3-B), and kept all validation deterministic throughout, with new validations only ever added. Our working rule became: expand the LLM’s role in generating changes, but not in verifying them.

Evaluation

The project started in October 2025. After three months of experimentation, both campaigns ran from January 2026. We report on all campaign PRs created up to 30 April 2026 and measure their outcomes as of 30 June 2026, when every PR was at least two months old. We exclude test PRs we created on personal and sandbox repositories, and PRs the bot closed itself, mostly to regenerate them. Each remaining PR is labeled merged, rejected (closed unmerged by a developer), or no decision (no developer decision by 30 June: still open, or closed by automation such as stale-PR bots).

RQ1: Partitioning Work Between Deterministic and LLM Steps

Classification, context collection, prompt rendering, and validation checks are deterministic. The LLM is called at three points: to execute the migration, to remediate after a failed check, and to write the PR description. Around these calls sit the registry, 12 data collectors, 7 validation rules, the renderer, the orchestrator, PR lifecycle management, and a dashboard. In code, the PyPackaging migration tool has about 4,300 lines of deterministic Python against 230 lines of LLM interface, plus about 3,600 lines of prompt templates and reference documentation. We give these counts as a description of where the design decisions sit, not as a measure of effort.

Table III shows how often the pipeline produced a PR for the three PyPackaging archetypes with dedicated templates. The simplest archetype, current build, no metadata, reached 80.2%. The other two, proprietary packaging only and broken existing packaging, reached about 55–57%. Most failures were deliberate: the pipeline declines to open a PR when data collection shows something it cannot migrate safely, such as unresolvable dependencies or several packages in one repository.

ArchetypeTasksPRsRate
Current Build, No Metadata967780.2%
Proprietary Packaging Only18610556.5%
Broken Existing Packaging553054.5%

Table III. PyPackaging PR Generation by Archetype

The boundary shifts described in § 3-D show up in agent run times. PyPackaging PRs generated with Sonnet 4.5 averaged 4.1 minutes; later PRs with Opus 4.6 handling more complex archetypes averaged 7.7 minutes. The archetype mix also changed during this period, so we cannot clearly separate the effect of the model from that of the more challenging tasks.

The partition that worked best across our experiments keeps classification, context assembly, and validation deterministic

The first two rows are single-script packages: one already uses the current build system and needs only packaging metadata; the other is shipped through the proprietary shared-environment model and is packaged from scratch. The third covers repositories with a partial or misconfigured packaging setup. Tasks are matched to PRs by tags in the initiative tracker.

and tasks the LLM with a single, well-specified transformation. Where the source format is regular enough, as for one CMake archetype, the LLM can be removed altogether.

RQ2: Runtime Production Data

As we did not run an explicit ablation study comparing pipelines with and without runtime data, we provide qualitative evidence, informed by the team’s experience in the pilot studies leading up to the two migration campaigns. Before runtime data was integrated, the PyPackaging pipeline relied on AST scanning alone. In adding it, we observed four key changes. First, conditional and test-only imports stopped being treated as hard requirements. Second, dynamically loaded modules, for instance through plugins, became visible, and the pipeline flagged repositories that needed more care. Third, startup logs showed which interpreter version actually ran in production and which scripts were actually executed, which provides insight into the target version and entry points to preserve. Finally, cluster-specific versions removed a class of failures in which a declared version range was valid on one cluster but broke on another, since production clusters run different dependency versions. None of this information can be retrieved from source code alone. Thus, we consider runtime data essential for packaging migrations, but further work is still needed to quantify its effect.

RQ3: Bottlenecks in Fleet-Scale Campaigns

Table 4 shows the outcomes for all PRs generated across both campaigns. The most common outcome in both campaigns was no developer decision at all: 67.7% of PyPackaging PRs and 57.7% of CMake PRs were neither merged nor rejected by the end of the observation period. This result demonstrates that generating PRs was not the main constraint, but rather convincing owning teams to review, test, and deploy them was. Figure 2 shows the same point over time. PRs were generated in a few short batches, while merges arrived at a steady pace that did not follow them: in CppBuild-to-CMake, merges were still rising at a constant rate two months after the last PR was opened.

Figure 2. Cumulative PRs opened and merged per campaign (November 2025 to end of June 2026). Only PRs opened up to 30 April (dashed line) are included; merges are counted up to 30 June.

PyPackagingCppBuild-to-CMake
Total PRs2942,039
Merged31 (10.5%)698 (34.2%)
Rejected64 (21.8%)165 (8.1%)
No decision199 (67.7%)1,176 (57.7%)

Table 4. Campaign outcomes (PRs created by 30 April 2026, outcomes on 30 June 2026)

Feedback Loop Depth and Population

The two campaigns also diverge. CMake PRs were merged at more than three times the PyPackaging rate, and rejected at less than half the rate. We attribute much of this to the depth of the feedback loop (§ 2-B). A CMake PR arrives having been compiled and tested, so a reviewer is mostly checking a build file that is already known to work. A PyPackaging PR has only passed packaging and smoke checks; whether it works is only known after the owner deploys it, so the owner carries the risk and is more likely to reject it or wait. The populations also differ. CMake projects fall into a few similar archetypes, so fixes made for one repository apply to many. In contrast, PyPackaging candidates are spread over 18 archetypes with many one-off cases. Other factors point the same way: the CMake campaign started earlier, build-system changes may be seen as lower risk, and a part-time engineer was designated to support reviews. With only two campaigns, we cannot investigate the effects of these individual factors in depth.

Impact Beyond Merges

Merge rate undercounts impact. In 34 of the 51 PyPackaging repositories where our PR was rejected, the owning team went on to land the migration themselves, and 14 of their PRs reused most of our changes. Developers told us that a common reason for closing and re-creating a PR was that they could not push to the bot’s branch. One wrote: “Yes, the PR was very useful. My main feedback was that one of the changes broke some unit tests. It would have been nice if the bot was able to identify any run/fix any tests before posting the PR. I tried to have [an AI coding tool] post the changes to the PR, but it got confused and ended up creating a new PR, so I just merged that one.” This feedback points to two improvements we plan to incorporate in future versions of the PyPackaging campaign: letting owning teams push changes directly to the generated PR, so that they can refine it instead of starting over, and running the project’s own tests before a PR is opened.

Effort

LLM inference costs were negligible compared with the engineering effort of building and operating the pipelines. Domain experts estimate a manual migration at about 2 person-days per CppBuild-to-CMake task and 1–2 person-days per PyPackaging task. These estimates come from the team that built the system and should be read with that in mind.

Discussion

Lessons Learned

Six months of operation yielded lessons that we believe transfer to other organizations undertaking AI-assisted code migration at scale.

Lesson 1: PR generation is no longer the bottleneck; receiving and validating the changes is. The system produced migration PRs much faster than owning teams could absorb them. By the end of the observation period, 10.5% of PyPackaging PRs and 34.2% of CppBuild-to-CMake PRs had been merged, while 67.7% and 57.7% had received no developer decision at all. Further improvements to generation would not have changed these numbers much. What limited them was the cost to the owning team of reviewing, testing, and deploying each change, and how much they trusted that it would work. We therefore expect the next significant gains to come from reducing that cost: running more of the validation inside the pipeline, giving reviewers stronger evidence that a change is correct, and making it easy for teams to iterate on a generated PR rather than starting again.

The high rate of PRs with no decision is also partly a consequence of the low priority of these campaigns. Neither carries a deadline or a security driver, and the receiving team bears the review effort and deployment risk while the benefit is diffuse and long-term. Lowering the cost of producing a migration does not lower the cost of absorbing it, and for non-urgent work that absorption cost is what developers weigh. A low direct merge rate therefore reflects how teams prioritize unsolicited, non-urgent changes, not only the quality of the generated change itself.

Lesson 2: Archetype decomposition was among the most impactful architectural decisions. Classifying migration targets into archetypes enabled three capabilities that a one-size-fits-all approach would struggle with. First, it allowed per-archetype optimization, with each archetype receiving a dedicated prompt template, data collection strategy, and validation rule set. Second, it made success rates interpretable: the 80% rate for the current build, no metadata archetype describes a well-characterized group of repositories rather than an average across heterogeneous targets. Third, it supported prioritization, as knowing which archetypes were more automatable and which were more complex or blocked let the team sequence work by expected return. The pattern applied to both campaigns.

Lesson 3: Source code analysis alone is insufficient for many packaging migrations. We did not expect how much runtime data would matter. The accuracy improvement from incorporating production telemetry, including startup logs, module usage traces, and cluster-specific dependency versions, was large enough that we consider runtime data a critical input for packaging migrations, although we have not yet confirmed this with a controlled ablation (§ 5-B). Source code analysis cannot determine which imports are exercised in production, which dependency versions are deployed to each cluster, or which scripts are actually invoked. The three-way dependency validation scheme (§ 3-B) provides a principled mechanism, but the broader point generalizes: any migration system that operates on source code without runtime context works with an incomplete picture. This extends the insights from Li et al. [7] regarding “Environment-in-the-Loop” migration from the academic setting to an industrial-scale system.

Lesson 4: Models improve faster than pipeline engineering. Over six months, the underlying LLM progressed from Sonnet 4.5 to Opus 4.6, with substantial capability improvements. Deterministic workarounds that required weeks of engineering (e.g., the AST-based dependency pre-processing that compensated for earlier models’ unreliable import analysis) became less critical as newer models handled these cases directly. In hindsight, this creates a design tension. Hardcoding too much deterministic pre-processing creates technical debt when a newer model handles the task natively. Delegating too aggressively to the LLM creates fragility when the model’s behavior changes between versions. Our heuristic: expand the LLM’s role in content generation but not in verification (§ 3-D).

Two practical consequences followed. First, we started revisiting deterministic pre-processing with each model upgrade: when a new model handled a case natively, we kept the deterministic step only if it also made the pipeline more reproducible or easier to debug. Second, the work moved from code to documentation: with more capable models, the limiting factor became how precise our instructions and reference documents were, not what the model could do. Verification stayed out of this trade-off; we never moved a validation rule into the LLM, because an error that slips through at fleet scale could harm hundreds of repositories.

Lesson 5: Feedback loop depth determines the automation ceiling. The CMake and PyPackaging campaigns revealed this structural factor clearly. CppBuild-to-CMake is a format translation where compilation success is a strong proxy for correctness: a shallow feedback loop that enables high automation confidence. PyPackaging transforms heterogeneous source formats that require runtime verification through the full deployment lifecycle, creating a deep feedback loop that caps automation confidence. Teams planning AI-assisted migration campaigns should assess feedback loop depth before investing in pipeline engineering.

Lesson 6: Impact exceeds merge rate. A follow-up on the 51 repositories where our PyPackaging PRs were rejected found that 34 of the owners later handled the migration themselves, in 14 cases reusing most of our changes. Developers who closed and re-created PRs told us the generated PRs were useful as starting points. This has design implications: PR descriptions, commit messages, and inline comments should be optimized for readability and adaptability by a developer who intends to re-implement the changes. Systems that measure success solely by merge rate undercount their actual impact.

Lesson 7: The long tail resists full automation. Archetypes affecting ≈20\approx20 or fewer repositories do not justify dedicated automation. The engineering cost of handling each additional edge case exceeds the manual effort saved. For these cases, a starting-point PR that developers refine is the right answer. Recognizing this threshold early prevents over-investment in diminishing-return automation and redirects effort toward the adoption and landing challenges identified in Lesson 1.

Threats to Validity

Internal validity. The most significant threat is the absence of controlled experiments. Pipeline code, prompt templates, validation rules, and model versions all changed during the evaluation period, often concurrently, making it difficult to attribute observed improvements to any single factor. Success rate improvements between January and April 2026 may reflect model capability gains, pipeline improvements, or selection bias (easier archetypes attempted first, potentially inflating early success rates such as the 80% current build, no metadata figure in Table III). All qualitative judgments, including archetype definitions, cost estimates, and the interpretation of runtime data’s importance, come from the same team that built the system. Independent evaluation would mitigate this experimenter bias.

The evaluation lacks a controlled experiment comparing migration outcomes with and without runtime data enrichment. The claim that runtime data is critical (Lesson 3) rests on qualitative engineering judgment and the architectural argument in § 3-B, not on a quantitative ablation study. An experimental design comparing AST-only versus AST+runtime-log pipelines on the same task population would provide stronger evidence, but its cost, and the fact that the campaigns targeted live products owned by real teams, made us discard this option.

Finally, the model evolution during the study period makes our findings time-bound. The specific success rates and failure modes reported here reflect a particular configuration of model capabilities (Sonnet 4.5 and Opus 4.6 as of April 2026). The most fragile findings are the specific quantitative results: the 80% current build, no metadata generation rate and the split between deterministic and LLM-driven components will likely shift as models improve. The general principles of using hybrid pipelines, runtime data enrichment, archetype decomposition, and bounded autonomy are intended to be more durable, but their specific parameterization will need ongoing calibration.

External validity. The findings come from a single organization with proprietary packaging and build systems, organization-specific deployment patterns, and a particular culture regarding code ownership and PR review. While the scale is substantial (≈8,600 candidate repositories, 2,333 PRs), all observations occur within a single organizational context. Organizations with monorepo architectures will not face the PR landing problem in the same form, as Google’s reported experience shows [4], [8], where a single change can atomically update all affected code. The general architectural patterns (hybrid pipelines, runtime data enrichment, archetype classification) are more likely to generalize than the specific implementations.

Construct validity. We report multiple metrics (generation success rate, merge rate, and decision outcomes) but none cleanly separates generation quality from developer engagement. A PR may go unmerged because the receiving team lacks bandwidth or disagrees with the migration priority, not because of a technical deficiency. The repositories that closed our PR and later landed the migration themselves (Section IV-C2) illustrate the ambiguity: technically rejected but functionally successful. We did not conduct a formal user study of developer experience. This pattern merits broader qualitative investigation through interviews or surveys. Understanding why developers chose to re-implement rather than merge, and what changes to PR format or migration approach would increase direct merge rates, requires structured user research beyond the ad hoc feedback we collected. The manual migration cost estimates (2 person-days per CppBuild-to-CMake task, 1–2 per PyPackaging task) come from expert judgment rather than timed measurements. A rigorous cost comparison would require timing a sample of manual migrations and computing fully-loaded cost equivalents.

Conclusion validity. The paper’s authors designed, built, and operated the system they evaluated. This dual role risks unconscious bias toward favorable interpretation. The study does not employ hypothesis testing or statistical inference; reported success rates and landing rates are descriptive statistics without confidence intervals or significance tests.

Related Work

Large-scale automated code migration predates LLMs. ClangMR and T2R showed that deterministic program analysis and transformation can scale to industrial codebases [9], [10], and PolyglotPiranha showed that lightweight declarative rules can express multi-language transformations, deleting 210K lines of dead code and migrating 20K lines across 1,611 pull requests in an industrial deployment [11]. Previous work at Bloomberg has reported on successful deployments of pre-LLM automated program repair techniques into industrial development workflows [19]. These systems are highly effective when a migration can be captured explicitly as rules, but become harder to apply as repository structure and migration logic become more heterogeneous.

LLMs extend this design space by handling transformations that are difficult to encode exhaustively. At Google, Nikolov et al. describe enterprise migration workflows that combine programmatic analysis and discovery, model-generated edits, compilation, testing, and human review [8]; Ziftci et al. report 39 migrations and 595 submitted changes using an LLM-assisted workflow [4]. These are the closest industrial precedents to our work. Our contribution is therefore not the combination of LLMs with traditional tooling itself, but the explicit boundary between them: classification, context construction, and verification remain deterministic, while the LLM is confined to bounded transformation and remediation steps.

Other recent systems explore related hybrid designs. MigrateLib combines LLMs with static and dynamic analysis for Python library migration [3], while SPELL uses LLMs to synthesize reusable programmatic edits [12]. Work on C-to-Rust migration similarly combines LLM generation with compiler, analysis, or execution feedback [1], [2], [14], [15], and Fandina et al. show that a domain-specific checker catches errors that LLM-only evaluation misses [13]. On the evaluation side, MigrationBench measures repository-level migration [17], and RepoMod-Bench shows success dropping sharply as repositories grow [5]; other work studies the role of context and of continuously derived benchmarks [16], [18]. Environment-in-the-Loop argues that executable environment state and runtime feedback should directly be a part of the migration process [7]. Our PyPackaging pipeline differs in using previously observed production telemetry (interpreter startup logs, module usage traces, and deployed dependency versions) as input to migrations.

Finally, code migration remains a human and organizational

WorkScaleHybridTelemetrySetting
ClangMR, T2R [9], [10]Industrial codebasesNoNoMonorepo
PolyglotPiranha [11]1,611 PRsNoNoPRs
Nikolov et al. [8]Several migrationsYesNoMonorepo
Ziftci et al. [4]39 migrationsYesNoMonorepo
SPELL [12]Python library migrationsYesNoResearch
MigrateLib [3]717 applicationsYesNoResearch
Fandina et al. [13]100 COBOL programsYesNoResearch
C-to-Rust [1], [2], [14], [15]Programs to projectsYesNoResearch
Li et al. [7]Position paperProposedNoN/A
Rosenfeld et al. [16]Dataset, case studyNoNoResearch
Benchmarks [5], [17], [18]BenchmarksN/ANoResearch
This work2,333 PRs (729 merged)YesYesPRs

Table V. Comparison with related work. Hybrid: deterministic tooling combined with an LLM. Telemetry: previously observed data from production executions used as migration input. Setting: industrial monorepo, industrial PRs to repository owners, or research prototype or benchmark.

process. Prior work on human–AI partnership in LLM-based migration identifies review responsibilities and trust calibration as central to effective use [20]. Previous work at Bloomberg has similarly studied the developer-facing challenges of deploying automated code changes, including developers’ experiences and interaction preferences [21], how generated changes can be delivered without creating distracting unsolicited PRs [22], and how agentic PRs should be integrated in developer workflows [23]. This motivates evaluating not only whether migrations can be generated, but also whether owning teams review, accept, and deploy them. Table V summarizes the closest related work along the dimensions most relevant to our study: scale, hybrid architecture, runtime information, and how generated changes are delivered.

Conclusions

This paper reports on our experience using LLM-assisted pipelines to drive two large migrations at Bloomberg: a C++ build system migration and a Python packaging migration. Together they generated more than 2,300 pull requests.

The most significant change this work brought to Bloomberg was in how a central team runs a migration. Before, the team owning a migration would publish instructions and an FAQ and ask every team to do the work themselves, with each team hitting the same problems independently. Now, the central team writes those instructions for a pipeline, and the pipeline produces the PRs. Every problem solved on one repository goes into the archetype templates, prompts, and validation rules, meaning it is solved for the rest. Most of the design effort went into the deterministic parts of the pipeline, and runtime production data turned out to be an input we could not do without, although we have not yet measured its effect in a controlled way. As models improved, we gave them more of the generation work, but we kept verification deterministic.

Cheaper PRs do not make a migration free. Owning teams still have to review, test, and deploy every change, and that cost was higher where the feedback loop was deeper. For CppBuild-to-CMake, the pipeline builds every migrated project and runs its unit tests before opening a PR. In that migration, compilation is a good proxy for correctness; 34.2% of PRs were merged and 8.1% rejected. For PyPackaging, packaging and basic checks pass, but many Python failures only show up at runtime. Developers trusted those changes less: 10.5% were merged and 21.8% rejected. The two populations were also very different. The CppBuild projects fell into a few similar archetypes, so what we learned on one repository applied to the next. PyPackaging candidates spanned 18 categories with many one-off cases, so there was much less to reuse. With two campaigns, we cannot tell how much of the gap comes from feedback-loop depth and how much from the population; both played a part. In both campaigns, most PRs (67.7% and 57.7%) received no developer decision, reflecting the low priority of these migrations.

Our main takeaways are:

  • Automation makes lower-priority migrations affordable, but deciding whether a migration is worth doing is still a cost decision, and that cost includes the time owning teams spend receiving it.

  • The greater the proportion of the feedback loop that could be run within the pipeline, the more confident the receiving teams were in the result.

  • A homogeneous population gets far more out of this approach than one made of one-off cases.

  • In our campaigns, the more it cost a team to receive a migration, the less likely they were to land it.

For future work, we want to allow teams to request deep-loop migrations when they are ready to deploy and verify them, rather than generating unprompted PRs. For shallow-loop migrations, where pre-PR confidence is already high, we want to explore other ways to increase engagement. These results come from one organization and two campaigns, and we would like to see them replicated elsewhere.

Data Availability

Since this project has been developed within Bloomberg’s internal systems and repositories, data and source code are proprietary and therefore cannot be made publicly accessible. However, we believe our description of the information and analysis process is sufficient for practitioners and researchers to adapt our approach in different contexts.

References

  1. [1]X. Cai, J. Liu, X. Huang, Y. Yu, H. Wu, C. Li, B. Wang, I. N. B. Yusuf, and L. Jiang, “Rustmap: Towards project-scale c-to-rust migration via program analysis and llm,” in Engineering of Complex Computer Systems: 29th International Conference, ICECCS 2025, Hangzhou, China, July 2–4, 2025, Proceedings. Berlin, Heidelberg: Springer-Verlag, 2025, p. 283–302. [Online]. Available: https://doi.org/10.1007/978-3-032-00828-2_16
  2. [2]Y. Liu, J. Hu, Y. Shan, G. Li, Y. Zou, Y. Dong, and T. Xie, “Llmi­grate: Transforming ”lazy” large language models into efficient source code migrators,” 2025. [Online]. Available: https://arxiv.org/abs/2503.23791
  3. [3]M. Islam, A. K. Jha, M. Mahmoud, and S. Nadi, “Migratelib: a tool for end-to-end python library migration,” 2026. [Online]. Available: https://arxiv.org/abs/2510.08810
  4. [4]C. Ziftci, S. Nikolov, A. Sjövall, B. Kim, D. Codecasa, and M. Kim, “Migrating code at scale with llms at google,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 162–173. [Online]. Available: https://doi.org/10.1145/3696630.3728542
  5. [5]X. Li, N. Ben-Israel, Y. Raz, B. Ahmed, D. Serebro, and A. Raux, “Repomod-bench: A benchmark for code repository modernization via implementation-agnostic testing,” in Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, ser. KDD ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 9371–9381. [Online]. Available: https://doi.org/10.1145/3708535.3817463
  6. [6]conda-forge, “cf-graph-countyfair: conda-forge dependency graph and metadata,” GitHub repository, 2026, accessed: Feb. 2026. [Online]. Available: https://github.com/regro/cf-graph-countyfair
  7. [7]X. Li, Z. Fei, Y. Ma, J. Zhang, F. Sarro, and H. Ye, “Environment-in-the-loop: Rethinking code migration with llm-based agents,” in Proceedings of the 1st Workshop on Code Translation, Transformation, and Modernization, ser. ReCode ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 19–23. [Online]. Available: https://doi.org/10.1145/3786180.3788315
  8. [8]S. Nikolov, D. Codecasa, A. Sjövall, M. Tabachnyk, S. Chandra, S. Taneja, and C. Zifti, “How is google using ai for internal code migrations?” in 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE Press, 2025, p. 481–492. [Online]. Available: https://doi.org/10.1109/ICSE-SEIP.2025.00048
  9. [9]H. K. Wright, D. Jasper, M. Klimek, C. Carruth, and Z. Wan, “Large-scale automated refactoring using clangml,” in 2013 IEEE International Conference on Software Maintenance, 2013, pp. 548–551.DOI
  10. [10]A. Ketkar, A. Mesbah, D. Mazinanian, D. Dig, and E. Aftandilian, “Type migration in ultra-large-scale codebases,” in Proceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19. IEEE Press, 2019, p. 1142–1153. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00117
  11. [11]A. Ketkar, D. Ramos, L. Clapp, R. Barik, and M. K. Ramnathan, “A lightweight polyglot code transformation language,” Proc. ACM Program. Lang., vol. 8, no. PLDI, Jun. 2024. [Online]. Available: https://doi.org/10.1145/36556429
  12. [12]D. Ramos, C. Gamboa, I. Lynce, V. Manquinho, R. Martins, and C. L. Goues, “Spell: Synthesis of programmatic edits using llms,” 2026. [Online]. Available: https://arxiv.org/abs/2602.01107
  13. [13]O. N. Fandina, E. Farchi, S. Fromoivich, R. Gal, W. Ibraheem, R. Katan, and A. Podolsky, “Beyond blind spots: Analytic hints for mitigating LLM-based evaluation pitfalls,” in AAAI 2026 Workshop on Assessing and Improving Reliability of Foundation Models in the Real World, 2026. [Online]. Available: https://openreview.net/forum?id=JvfMLvcbSCarxiv.org/abs/2512.16272
  14. [14]M. Farrukh, B. Coskun, T. Palit, and M. Polychronakis, “Orbit: Guided agentic orchestration for autonomous c-to-rust translation,” 2026. [Online]. Available: https://arxiv.org/abs/2604.12048
  15. [15]Y. Yan, Y. Feng, L. Liu, D. Liu, Z. Liu, H. Teng, and B. Xu, “C2rustx: Program-structure-aware c-to-rust translation via program analysis and llm,” 2026. [Online]. Available: https://arxiv.org/abs/2603.28686
  16. [16]K. A. Rosenfeld, C. C. Kerr, and J. Lundin, “What a diff makes: automating code migration with large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2511.00160
  17. [17]L. Liu, X. Liu, Q. Zhou, L. Chen, Y. Liu, H. Nguyen, B. O. Tehrani, X. Shen, J. Huan, O. Tripp, and A. Deoras, “Migrationbench: Repository-level code migration benchmark from java 8,” in Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining ’26, ser. KDD ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 9427–9438. [Online]. Available: https://doi.org/10.1145/3770855.3817526
  18. [18]D. Saxena, R. Maurya, X. Ou, G. T. Gosamekhar, S. M. Gupta, A. Iyer, Y. Kang, C. Bansal, A. Akella, and S. Rajmohan, “Continuous benchmark generation for evaluating enterprise-scale llm agents,” in Proceedings of the 3rd International Workshop on Large Language Models For Code, ser. LLM4Code ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 1–5. [Online]. Available: https://doi.org/10.1145/3786181.3788708
  19. [19]S. Kirbas, E. Windels, O. McBello, K. Kells, M. Pagano, R. Szalanski, V. Nowack, E. R. Winter, S. Counsell, D. Bowes, T. Hall, S. Haraldsson, and J. Woodward, “On the introduction of automatic program repair in bloomberg,” IEEE Software, vol. 38, no. 4, pp. 43–51, 2021.
  20. [20]B. Omidvar Tehrani, I. M, and A. Anbuhai, “Evaluating human-ai partnership for llm-based code migration,” in Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3613905.3650896
  21. [21]E. R. Winter, V. Nowack, D. Bowes, S. Counsell, T. Hall, S. Haraldsson, J. Woodward, S. Kirbas, E. Windels, O. McBello, A. Atakishiyev, K. Kells, and M. Pagano, “Towards developer-centered automatic program repair: findings from bloomberg,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 1578–1588. [Online]. Available: https://doi.org/10.1145/3540250.3558953
  22. [22]D. Williams, J. Callan, S. Kirbas, S. Mechtaev, J. Petke, T. Prideaux-Ghee, and F. Sarro, “User-centric deployment of automated program repair at bloomberg,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 81–91. [Online]. Available: https://doi.org/10.1145/3639477.3639756
  23. [23]D. Williams, A. Evripiotis, S. Kirbas, H. Morgan, S. Magidovich, P. Wainwright, and F. Sarro, “Pomona: Continuous code quality improvement via small, agentic pull requests at bloomberg,” 2026. [Online]. Available: https://arxiv.org/abs/2606.06752

Paper details

vlnc:W_2efwkvnz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents