betaFeedback

WorksComputer Science

Agent Stack for ML Training Lifecycle: Generation, Validation, and Optimization

Abstract

Production model development requires repeated architecture design, platform integration, debugging, training, and evaluation. We present an agent stack deployed at LinkedIn with two independently usable agent loops: AutoTorch for model generation and migration, and AutoResearch for model optimization. AutoTorch generates PyTorch models and training pipelines for TensorFlow migration, XGBoost replacement, and new-model development. Executable validation hooks check platform API compliance, trainability, and conversion fidelity on sampled data, guiding iterative repair before engineer review and full-scale distributed training. AutoResearch explores architecture and training-configuration changes through expert-steered or autonomous experimentation. An external orchestrator manages parallel trials, experiment history, candidate selection, and wall-clock budgets across agent sessions. A shared runtime preserves execution state independently of agent reasoning and allocates GPUs separately from agent execution. Across ranking, content understanding, and recommendation workloads, AutoTorch completed generation and full-scale training in 92 of 115 workflow runs (80%). Of these, 77 (83.7%) produced models meeting post-training quality requirements. In a separate cohort, AutoResearch improved the primary task metric on held-out data for 28 of 100 production pipelines. Engineering teams estimate that the stack reduced elapsed end-to-end migration time from approximately eight weeks to two weeks, a 4x speedup.

Introduction

Developing production machine learning models remains a labor-intensive process. Engineers must design model architectures, integrate them with platform-specific training APIs, debug runtime failures, and repeatedly train and evaluate models to achieve target quality. Framework migrations add further complexity, requiring engineers to reproduce existing model behavior while adapting to a new training stack.

We introduce agent loops to automate this iterative process. Rather than generating model code in a single step, agents continuously generate, validate, repair, and optimize models using execution feedback. These loops operate across multiple agent sessions and GPU training jobs, with persistent state and validation mechanisms supporting long-running execution (Figure 1). We evaluate this approach at LinkedIn across model generation, distributed training, and offline evaluation.

Model migration as a composition of independently usable agent loops

Figure 1. Model migration as a composition of independently usable agent loops. AutoTorch generates a training model bundle; AutoResearch can subsequently optimize the accepted model. Each loop also supports independent invocation.

We implement two independently usable agent loops. AutoTorch generates PyTorch models and training pipelines, including framework migration, and uses executable checks to guide repair before engineer review and full-scale training. AutoResearch starts from a validated pipeline and searches for improvements through budgeted experiments. Both can run independently or be composed for migration followed by optimization.

A shared runtime supports these loops with durable execution sandboxes, persistent task state, and brokered GPU and data access. Architecture-specific skills guide candidate generation, while executable validation and experiment records provide feedback to agents. Completion remains governed by explicit checks and engineer review, not by the LLM’s own assessment.

This paper describes three contributions: (1) model-generation and optimization loops with task-specific validation and search policies; (2) the shared runtime and execution controls supporting long-running tasks on production training infrastructure; and (3) deployment experience across model migration and optimization workloads at LinkedIn.

Background and Problem

Production ML Pipelines

Production ML pipelines combine model architectures with preprocessing, model-forward interfaces, data ingestion, distributed training configurations, and offline evaluation APIs. At LinkedIn, agents must follow these contracts to generate runnable training pipelines.

Migration as a Use Case

LinkedIn is migrating models from TensorFlow and XGBoost frameworks to a PyTorch-based training stack. The migration also involves framework-specific dependencies, including libraries such as TensorFlow Ranking. We use this migration to exercise the agent loops across model generation, validation, distributed training, and optimization tion. AutoTorch generates and validates PyTorch models and training pipelines, while AutoResearch searches for subsequent improvements. The loops can also run independently.

Problem Formulation

We designed the system around two common ML engineering tasks. The first is to generate a PyTorch model that works within the production training platform and reaches the required task quality. The second is to improve an existing model without disrupting its training and evaluation pipeline.

Both workflows operate within the training and offline evaluation lifecycle. API compliance includes the required preprocessing and model-forward interfaces, but does not cover online serving integration, latency or throughput SLAs, or A/B testing.

System Architecture

Each of the two agent loops consists of a set of task-specific skills and an agent harness, and can run on general-purpose coding agents such as GitHub Copilot and Claude Code. All agents run under LinkedIn enterprise agreements. Figure 1 shows the two loops composed for migration. The loops can also start independently from a task specification, an experimental training pipeline, or an accepted production model. Both run on a shared runtime (§ 4) and invoke the existing production ML platform rather than a separate agent-specific training stack.

AutoTorch Model Generation

AutoTorch generates trainable PyTorch models and their training pipelines in two stages separated by engineer review (Figure 2). Stage 1 generates the model architecture and iteratively validates and repairs it on a single GPU using sampled data. Once an engineer approves the architecture, Stage 2 builds a Flyte training pipeline and runs distributed training on multiple GPUs using production configurations and the full-scale dataset. This separation allows agents to resolve generation and validation failures before committing to full-scale training.

AutoTorch’s two-stage pipeline

Figure 2. AutoTorch’s two stages. Stage 1 clones a TF model or builds a new architecture, with sampled checks guiding repair. After engineer review, Stage 2 can use a warm start for a fast task-metric check before final cold-start training.

For migration tasks, AutoTorch takes a source model and featurized data. TensorFlow (TF) sources support weight transfer and parity checks, while XGBoost sources provide task-quality targets. For new models, AutoTorch uses a task specification and architecture-specific skills, including wide and deep models, generative recommenders, graph neural networks, TabM [7], and multi-task learning. The Stage 1 output is a pull request containing generated code (model.py, train.py, data.py, transforms.py, eval.py) and sampled validation results. The trained checkpoint is produced in Stage 2.

Stage 1: sampled generation and repair. For TensorFlow migration, the agent reconstructs the model in PyTorch, transfers compatible parameters using deterministic analyzers or agent-generated scripts, and applies lightweight output distillation when some parameters cannot be mapped. Each iteration evaluates the candidate on a sample of the featurized dataset. TensorFlow conversions are scored on five dimensions: output parity (35%), task-metric parity (20%), trainability (15%), code style (15%), and structural similarity (15%). API compliance and trainability are also mandatory checks. A high cloning match requires both checks and a composite score of at least 90:

STF=0.35Pio+0.20Ptask+0.15T+0.15C+0.15H≥90.(1)S_{\mathrm{TF}} = 0.35P_{\mathrm{io}} + 0.20P_{\mathrm{task}} + 0.15T + 0.15C + 0.15H \ge90. \tag*{(1)}

Here, PioP_{\mathrm{io}} and PtaskP_{\mathrm{task}} denote output and task-metric parity, while TT, CC, and HH denote trainability, code style, and structural similarity, respectively. Each component is scored from 0 to 100.

Failed checks return scores and diagnostic evidence to the agent for repair. For new models and XGBoost replacements, Stage 1 checks generated-code validity, API compliance, trainability on sampled data, and package/exportability before engineer review. Source-output parity and the TensorFlow conversion score do not apply to these models. Task-metric qualification follows full training, as described in § 5.7.

Once the generated architecture passes its initial checks, AutoTorch opens a pull request for engineer review. Stage 2 begins after approval. Conversions scoring below 90 require explicit engineer approval and are not counted as high cloning matches.

Stage 2: full-scale training. The agent builds a Flyte-based trainer pipeline for the approved model architecture and iterates until the component runs. The pipeline is configured with the GPU count (n_gpus) and the full-scale dataset, and packages the result as a torch package. Warm-start training from transferred or distilled parameters can provide a fast check of task-metric parity. Before downstream deployment, the model is trained from a cold start using production configurations. The resulting checkpoint is evaluated through the standard offline Evals API.

These stages measure different properties. Stage 1 evaluates conversion fidelity by comparing the clone with the TF model on identical inputs. Stage 2 verifies task quality after full-scale training, with cold-start training establishing the final PyTorch model lineage.

AutoResearch: Hill Climb on Validated Model

AutoResearch starts with a model that already has a working training and evaluation pipeline. The agent proposes changes to the architecture or training configuration, runs experiments, and compares the results against the current best model.

Each candidate runs in a separate Git worktree branched from the same incumbent, which remains fixed throughout the round. Experiments run in parallel, but incumbent updates and the next round wait until all trials in the current round have finished. § 6.3 describes the selection policy.

Task. The input is a validated model plus a research plan. The model is supplied either as an existing Flyte training pipeline or as a GitHub repository with the model code. The output has three parts: a leaderboard of evaluated variants; a variant library of configurations, architectures, and code branches; and a final report of findings, dead ends, and a recommendation.

Research plan. The research_plan.md specifies the training entry point, data, primary metric, concurrency cap, and clock-hour budget. Engineers can provide freeform research directions, such as which architectures to explore, alternative losses or task-specific heads for multi-task learning, and optimizer or training configurations. They can also specify changes to avoid, such as feature crosses restricted by downstream serving requirements. When no engineer-provided research focus is available, the agent examines the model repository, training configuration, and prior experiments to develop its own research directions.

Agent Control Plane

Coordinating agents is not the same as managing ML experiments. Delegation alone does not track which model revision produced a metric, enforce validation criteria, or preserve state across long-running GPU jobs. Our loops keep validation results, experiment history, and artifacts outside agent context, while a shared Kubernetes runtime brokers GPU resources separately (Figure 4).

Durable agent sandboxes and independently allocated accelerators

Figure 4. Durable agent sandboxes and independently allocated accelerators. Training data is available only through controlled paths and is not placed in LLM context.

Durable sandbox. Agents execute in isolated Kubernetes pods backed by CPU or GPU resources. Sandboxes provide network storage for plans, repositories, and artifacts, with terminal session persistence and a seven-day time-to-live (TTL). GPU jobs can be preempted when production workloads need capacity. The agent client is not preempted, and its files and experiment history remain available when GPU resources return.

Elastic warm GPU pool. Agents release accelerators when idle, requesting experiment pods dynamically via a broker backed by a warm H100/H200 pool. The control plane borrows idle GPU capacity from production clusters under elastic quotas, yielding capacity when production demand increases. This decouples long-running agent lifecycles from accelerator allocation.

Controlled data and network access. Training jobs launched by agents read production training data through read-only, access-controlled paths; Member Data is not provided to the LLM agents’ context. Operations such as trial submission and model export use brokered services; deploying a model requires engineer approval.

Memory. The agentic job control plane logs all execution events and exposes a unified memory interface. A shared console tracks runs, training jobs, and metrics, maintaining artifact lineage back to originating experiments. When human intervention is required, the runtime alerts the owning team via chat while pausing the task without holding GPU resources.

LLM and agent configuration. We run AutoTorch with coding and reasoning agents through Copilot. AutoTorch tasks must satisfy architecture, platform, and workflow constraints using task-specific skills, tools, and validation hooks. In AutoResearch, researcher, implementation, and evaluation roles can use different agent configurations. Experiment state, time-budget accounting, and candidate bookkeeping reside in the external orchestrator.

Model Generation and Validation

Model Initialization

TensorFlow models. For TensorFlow migrations, AutoTorch reconstructs preprocessing, feature ordering, embeddings, network blocks, and output heads in PyTorch. Parameter transfer uses deterministic analyzers and agent-generated scripts. The analyzers match parameters using variable names, layer types, indices, parameter roles, and tensor shapes, including layout transposition when needed. When deterministic matching fails, the agent generates custom transfer scripts to handle model-specific mappings. When parameters cannot be mapped, AutoTorch falls back to lightweight output distillation. A structural similarity score at or below 90% indicates a low structural match. Distillation minimizes variance-normalized mean squared error between the TensorFlow teacher and PyTorch student predictions. The teacher is evaluated without gradients, while the student’s BatchNorm layers are frozen and other parameters remain trainable. By default,

AutoResearch’s synchronous hill climb

Figure 3. AutoResearch’s synchronous hill climb. A round begins from a fixed incumbent, parallel trials run in separate worktrees, and selection occurs after all results return.

distillation runs for up to 500 steps with batch size 32, stopping early when the loss falls below 0.001. The default conversion path uses synthetic inputs.

No transferable state. For non-cloneable models (e.g., XGBoost or custom frameworks), AutoTorch constructs a PyTorch replacement and trains it directly on labeled data using the source model solely as a quality benchmark.

New models. For new model generation, AutoTorch uses task specifications, feature schemas, target metrics, and permitted architecture families to build and train candidate models using standard validation checks.

Platform API Compliance

Candidates must utilize supported platform APIs for data loading, training, and evaluation. We verify compliance by executing generated code against these interfaces, returning failures for repair rather than relying on static inspection alone.

Trainability Probe

The trainability probe performs a lightweight single-GPU optimization run to verify loss reduction and detect numerical issues, nonfinite gradients, or structural incompatibilities before allocating distributed training resources. By default, this consists of 30 training steps with batch size 256. The probe checks whether gradients flow, losses decrease, and model outputs remain numerically stable.

It also detects saturated or nearly constant outputs and unusually low initial losses. A decreasing loss trend with positive improvement passes the loss-reduction check. A 5% relative decrease earns full credit for this component; smaller positive decreases can still pass.

Full-Scale Distributed Training

After engineer review, candidates advance to full-scale distributed training on multiple GPUs using production configurations. For TensorFlow migrations, a warm start from transferred or distilled parameters can verify task-metric parity through faster convergence. This offers an early indication that the PyTorch model can recover the reference model’s quality. Before downstream deployment, models are trained from a cold start in the PyTorch pipeline to establish reproducible training lineage independent of transferred TensorFlow weights. Models with string features require cold starts because LinkedIn’s TensorFlow and PyTorch training stacks use different hashing and tokenization transforms, making direct reuse of feature-dependent weights unreliable (Figure 2).

Output Predictions Validation

For high-fidelity model cloning, candidates must accurately reproduce the source model’s prediction outputs on the same sampled inputs. The representation evaluated depends on task semantics: logits are compared for binary classification heads, softmax probabilities for multi-class heads, and raw prediction scores for ranking models. Evaluating prediction values rather than discrete class labels catches differences that label agreement alone would miss.

Output parity (PioP_{\mathrm{io}}) compares predictions on identical inputs. For scalar outputs, output parity combines correlation (40%), tolerance-based agreement (40%; absolute tolerance 10−310^{-3}, relative tolerance 1–5%), and near-exact match (20%; absolute difference at most 10−310^{-3}). Near-exact matching does not require bitwise equality. For vector outputs, cosine similarity and near-exact match contribute equally (50% each).

Stage 1 uses randomly sampled inputs for lightweight distillation and conversion scoring, without enforcing disjointness between the two samples. The resulting parity scores guide conversion and iterative repair rather than serving as independent estimates of model equivalence. Final task quality is evaluated separately after full-scale training using the standard offline evaluation pipeline.

Primary-Metric Validation

Task-metric parity (PtaskP_{\mathrm{task}}) compares the primary offline metric of a TensorFlow reference and its PyTorch candidate on the same data. For AUC, the score is zero if either model performs at or below chance (0.5). AUC values between 0.5 and 0.55 reduce the parity score to account for weak predictive signal. Once both AUCs reach 0.55, the parity score depends only on their closeness: each absolute AUC gap of 0.01 reduces the score by two points from 100, with a floor of zero.

The same principle applies to other task metrics, including NDCG@K, Precision@K, and Recall@K for ranking, or RMSE for regression. Source and candidate use the same offline data and metric definition, including ranking cutoff K. Metric direction and no-signal baselines are task-specific; the AUC thresholds do not apply to these other metrics. Stage 1 uses task-metric parity to guide conversion repair. Final task quality is evaluated separately after full-scale training. For newly generated models without a source, the task specification provides the quality target.

Architecture skills and executable checks workflow

Figure 5. Skills propose architectures, and executable checks return failure evidence for repair. New-model generation uses separate scoring after full training.

Multi-Dimensional Scoring

TensorFlow conversions use the composite score and mandatory checks defined in Equation (1). A conversion qualifies as a high cloning match when its score reaches 90 and the mandatory API and trainability checks pass. Lower-scoring conversions require engineering approval before Stage 2. The weights and threshold are fixed in this deployment; we have not evaluated their sensitivity or calibration against full-training outcomes.

For new models, there is no reference implementation to compare against. We instead score trainability (35%), metric achievement (45%), and code quality (20%) after training. Before full-scale training, the generated code must pass validity and package/export checks. Post-training qualification requires the generated model to be trainable, pass API compliance checks, and meet the task-metric target supplied in its specification. The target is a minimum for higher-is-better metrics and a maximum for lower-is-better metrics; no task-independent metric cutoff is used.

Model Packaging and Handoff

AutoTorch outputs a trainable model bundle containing implementation code, configurations, evaluation adapters, and export metadata for downstream productionization or review.

Budgeted Model Research

Search Objective and Constraints

AutoResearch searches for model improvements within a fixed wall-clock time budget. Starting from a validated baseline c0c_0, it evaluates candidate changes using the model’s existing training and evaluation pipeline. The current best model, or incumbent, is updated only after a round completes. The search returns this incumbent, or the original model if no round improves it.

Research Direction Search

AutoResearch proposes model changes based on the existing code, configurations, and previous experiments (Figure 7). Engineers can also specify research directions. Each proposal identifies the components to change, the expected effect, the target metric, and a minimal experiment. When improvements plateau, a dedicated skill explores alternative directions. Search stops when the clock-hour budget or configured stopping criterion is reached.

Hyperparameter and architecture search evaluation loop

Figure 7. Hyperparameter and architecture search share the same evaluation loop and round-end selection policy.

Experiment Orchestration

We use an external Python orchestrator to manage the search. It tracks experiment history, submits parallel training jobs, records metrics, and maintains the current incumbent.

Each search has a wall-clock time budget and a configurable concurrency limit, typically four experiments. Results are recorded as trials finish, but no replacement trial is launched and the incumbent is not updated until the round completes. The orchestrator then selects the best candidate that improves the incumbent; otherwise, it keeps the incumbent. Candidates are selected as complete implementations, without automatically merging changes from sibling worktrees. Completed-round results guide the next proposals (Figure 6).

Synchronous AutoResearch rounds

Figure 6. Synchronous AutoResearch rounds. Each round waits for all trials, then retains or replaces the incumbent. All rounds share one elapsed-time budget.

The time budget includes waiting for slower trials at the round barrier. This policy trades immediate reuse of idle slots for comparison against a fixed incumbent. Experiment history persists across agent sessions.

Candidate Evaluation

During search, candidates are ranked against the incumbent using the pipeline’s primary metric. After search, the selected candidate is compared with the original baseline on held-out data reserved for final evaluation [3]. The held-out results do not guide proposals, incumbent updates, or selection of another candidate. The improvement results in Table 1 use this final held-out comparison.

1. Deployment outcomesResultScope
1.1 AutoTorch generation runs115Workflow executions; not necessarily distinct models
1.2 Operational completion92 of 115 (80%)Generation and full training; denominator: all attempted runs
1.3 Post-training quality pass77 of 92 (83.7%)Evaluated model artifacts from completed runs; TF clones also require a conversion score ≥90\ge90
1.4 Offline primary-metric differenceTypically within 1–5%Relative to reference after full training; initialization modes not disaggregated; not a deployment gate
1.5 AutoResearch improvements28 of 100 (28%)Improved original baseline on final held-out evaluation
2. Measured AutoTorch executionp50 / p90Range
Stage 1: sampled generation and repair
2.1 Generate–evaluate–repair iterations4 / 101–17
2.2 Stage 1 wall time6.3 h / 32 h0.6–46 h
Stage 2: distributed training, evaluation, and export
2.3 Successful training job33 min / 2.5 h2 min–6.4 h
2.4 Train → evaluate → bundle → export2.8 h / 3.8 h1.0–5.3 h
End-to-end: generation start, including engineer review
2.5 First trained model2.3 d / 3.7 d0.1–6.1 d
2.6 Exported model2.4 d / 5.8 d0.1–9.2 d
3. Research and migration timeResultScope
3.1 AutoResearch search runtime48–72 hReported wall-clock range, not percentiles
3.2 AutoResearch optimization portionAbout 3 daysEstimated component of an agent-assisted migration project
3.3 Manual migrationAbout 8 weeksTeam-estimated duration to a pipeline of comparable offline quality
3.4 Agent-assisted migrationAbout 2 weeksTeam-estimated duration to the same endpoint
3.5 End-to-end migration accelerationApproximately 4×Ratio of project-duration estimates; not engineer-hours
3.6 Model-specific manual fixesAbout 2 engineer-days, when neededConditional human effort; not elapsed project time

Table 1. Production outcomes and runtime. AutoTorch latencies refer to the indicated intervals; end-to-end time includes engineer review. The two denominator cohorts differ.

Self-Repairing Harness

Not every failed experiment indicates a bad model. Some failures come from the training environment, dependencies, or the harness itself.

When this happens, the agent uses the error logs and execution context to diagnose the issue and propose a fix. Harness changes are submitted as pull requests and go through normal code review and integration tests. Successful repairs can also inform future runs through persistent memory, reducing the need to diagnose the same issue repeatedly.

Evaluation Methodology

Research questions. We organize deployment observations around three questions: (RQ1) Can agent loops generate, train, and evaluate models on production ML infrastructure? (RQ2) How do external control mechanisms enforce execution and completion constraints? (RQ3) Can agent loops like AutoResearch improve model quality under fixed clock-hour budgets?

RQ1: ML Framework Migration

Since May 2026, we have used AutoTorch for TensorFlow migration and XGBoost replacement. Operational completion requires generation and full-scale training. Quality qualification checks task quality and applicable cloning criteria only for completed runs. Table 1 reports completion over all attempts and quality passing over completions; neither rate measures engineer overrides. Engineers can reuse code from incomplete runs for debugging and further iteration.

Table 2 presents five AutoTorch cases spanning greenfield migration, TensorFlow-to-PyTorch porting, and tree-to-neural replacement. The agents delivered PyTorch models and training pipelines, using validation feedback to repair data processing, evaluation, and architecture mismatches.

Execution latency. Stage 1 generation and repair took 6.3 hours at p50 and 32 hours at p90. Among exported

Timing note. Recorded stage latencies and end-to-end latencies refer to distinct intervals and applicable run subsets, and their percentiles are not additive. Migration-project durations and component estimates are team reported; engineer-days measure labor, not elapsed time.

models, end-to-end generation to export, including engineer review, took 2.4 days at p50 and 5.8 days at p90 (Table 1).

Migration time. Engineering teams report that migrating a pipeline with comparable offline primary-metric quality typically takes eight weeks manually, compared with two weeks using the agent stack—a 4× speedup in end-to-end migration. The comparison measures elapsed project duration, not active engineer-hours. Measured AutoTorch export time and estimated full-project migration duration cover different workflow boundaries.

RQ2: Harness Design for External Control under Constraints

Iteration overhead. Setup and the first scored iteration took 2.5 hours at p50 and 15 hours at p90, compared with 46 minutes and 3.2 hours for subsequent repair iterations. The first interval includes model construction and initial validation, whereas later iterations reuse the initialized workflow.

AutoTorch uses executable hooks to enforce business and platform rules, including API compliance, trainability, and conversion fidelity. Failed checks return diagnostic feedback for the next repair iteration. AutoResearch instead maintains experiment history, candidate selection, and time budgets in an external Python orchestrator. This

keeps execution state separate from agent reasoning across long-running experiments and multiple agent sessions (Figure 8).

Harness-specific execution control diagram

Figure 8. Harness-specific execution control. AutoTorch uses hooks to gate each generate–evaluate–repair iteration. AutoResearch maintains persistent state and round-end selection externally to agent sessions.

RQ3: Agent-Guided Model Optimization

AutoResearch improved the primary metric on final held-out evaluation in 28 of 100 targeted pipelines (Table 1). If no candidate improved the search-validation metric, the incumbent was retained; held-out results were reported separately.

AutoResearch supports two modes of experimentation: expert-steered and autonomous. Domain experts can guide architectures, training configurations, and constraints to explore; without that guidance, the agent uses the model repository, configurations, and previous results to propose experiments. Both modes use the same iterative search loop.

Table 2 shows three AutoResearch cases. For a GNN recommender, AutoResearch explored 49 trials and identified a configuration that improved Recall@1000 by 9.3%. Widening an MLP classifier improved held-out AUC by 1.2%. For an SLM (0.6B) engagement model, an engineer used AutoResearch to compare prediction-head architectures, including MMoE [14], PLE [18], and DCNv2 [19], and reported obtaining decisive task-level results in days rather than weeks of manual exploration. DCNv2 improved two engagement metrics while meeting the relevance guardrail. These examples show how domain-expert guidance and agent-generated proposals can accelerate model experimentation.

CaseModel / use caseAgent actionPrimary task metric
AutoTorch — generation and migration
AT-1Two-tower video retrievalDelivered PyTorch model and training pipeline based on source modelRecall@200 matched source model
AT-2XGBoost → TabM; job-posting classificationChanged loss to per-example binary cross-entropyAUC matched
AT-3TensorFlow Ranking-based CTR modelPorted to PyTorch with weight transfer; fixed data and evaluation mismatchesAUC matched
AT-4TensorFlow multi-task model (200M embedding parameters)Ported to PyTorch; repaired preprocessing, sparse gradients, and task-output handlingEmbedding outputs matched (output-parity check)
AT-5TensorFlow graph neural networkPorted to a PyTorch GNN; restored node encoder and concatenation combinerDownstream classifier AUC matched
AutoResearch — model optimization
AR-1GNN connection recommendationReduced encoder to one wider layer; increased neighbor sampling; averaged four inference passes; tuned trainingRecall@1000 +9.3% vs. baseline
AR-2MLP classifierWidened hidden layers [?, 12] → [?, ?]Held-out primary AUC +1.20% vs. baseline
AR-3SLM (0.6B) + MLP classificationCompared linear, MMoE, PLE, and DCNv2 heads with engineer guidancepclick AUC +1.2% vs. MLP baseline (validation)

Table 2. Selected agent-loop cases (AT: AutoTorch; AR: AutoResearch). These examples are not randomly sampled. Notes. Matching does not imply bitwise equivalence. AT-4 reports embedding-output parity, not a primary task metric. AR-1 combines several changes on a fixed evaluation fixture; independent held-out status is not reported. AR-3 is a task-specific validation result, not an additional held-out win in Table 1.

Related Work

Model migration. Nikolov et al. [16] use playbooks and specialized agents for TensorFlow-to-JAX migration, evaluating complex internal migrations with checklist-based LLM judges. Liang et al. [12] combine in-context examples with source-execution oracles and iterative debugging for PyTorch-to-JAX migration. Our workflow combines TensorFlow-to-PyTorch parameter transfer and execution-based qualification with full-scale training and offline evaluation.

ML experimentation agents. AIDE explores code search for model implementation [9], MLE-STAR refines pipelines using retrieved knowledge [15], and Auto-RecSys combines asynchronous recommendation experiments with persistent memory and a separation between reasoning guidance and deterministic execution [11]. Our system combines model generation and optimization with production training and offline evaluation in a shared platform.

Autonomous research. Karpathy’s autoresearch uses markdown instructions to guide single-GPU training experiments [10]. AutoResearch was developed independently and does not use code from autoresearch. R&D-Agent separates research planning from implementation and evaluation [20], while The AI Scientist automates idea generation and experimental execution [13]. AutoResearch applies related propose–implement–evaluate loops to existing production training pipelines, with synchronous parallel trials and a shared elapsed-time budget.

Agent runtimes. AgentScope provides multi-agent communication and fault tolerance [5]. The agentic job control plane complements this by providing isolated sandboxes and brokered access to compute, production data, and ML platform APIs for long-running workflows.

Program search and optimization. AlphaEvolve couples code changes to automated evaluation [17]. Random search [2], Optuna [1], and Vizier [6] provide fixed-space optimization baselines. Our evaluation uses live production workloads rather than public benchmarks [4, 8].

Discussion and Limitations

The agent loops automate substantial parts of model generation and optimization, but engineers remain responsible for reviewing generated code, handling model-specific requirements, and making deployment decisions. Executable validation helps identify numerical and platform-integration failures, but it cannot establish model equivalence across all inputs.

A central limitation is potential skew between offline and online task metrics. Models that match or improve offline metrics can behave differently under production traffic, shifting data distributions, and user feedback. Our evaluation covers training and offline model quality; online serving integration and A/B testing remain outside the scope of the agent stack.

The system also depends on LinkedIn’s existing preprocessing, training, and evaluation interfaces. Adapting these agent loops to another ML platform would require integrating with its corresponding interfaces.

Conclusion

We presented AutoTorch and AutoResearch, two agent loops for model generation, framework migration, and optimization within LinkedIn’s PyTorch training platform. AutoTorch uses executable validation to guide model generation and repair, while AutoResearch keeps experiment state across long-running optimization tasks. Engineering teams report a 4×4 \times reduction in elapsed migration time while recovering comparable offline model quality. The stack automates much of this work while retaining engineer review and established production training controls.

AI Usage Disclosure

We used Copilot with OpenAI and Anthropic models and Gemini to assist with manuscript editing, literature search, and the preparation of LaTeX and TikZ figures. The authors reviewed and revised the generated content. The reported system implementation and experimental results were provided by the authors and were not generated by these AI tools. The authors take responsibility for the accuracy and integrity of the final manuscript.

References

  1. [1]Akiba, Takuya; Sano, Shotaro; Yanase, Toshihiko; et al.. Optuna: A Next-generation Hyperparameter Optimization Framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. pp. 2623–2631. doi: 10.1145/3292500.3330701. https://arxiv.org/abs/1907.10902.
  2. [2]Bergstra, James; Bengio, Yoshua. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 2012. pp. 281–305. https://jmlr.org/papers/v13/bergstra12a.html.
  3. [3]Cawley, Gavin C.; Talbot, Nicola L. C.. On Over-Fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research, 2010. pp. 2079–2107. https://jmlr.org/papers/v11/cawley10a.html.DOI
  4. [4]Chan, Jun Shern; Chowdhury, Neil; Jaffe, Oliver; et al.. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. International Conference on Learning Representations, 2025. https://openreview.net/forum?id=6s5uXNWGIh.arxiv.org/abs/2410.07095
  5. [5]Gao, Dawei; Li, Zitao; Pan, Xuchen; et al.. AgentScope: A Flexible Yet Robust Multi-Agent Platform. arXiv preprint arXiv:2402.14034, 2024. https://arxiv.org/abs/2402.14034.
  6. [6]Golovin, Daniel; Solnik, Benjamin; Moitra, Subhodeep; et al.. Google Vizier: A Service for Black-Box Optimization. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017. pp. 1487–1495. doi: 10.1145/3097983.3098043. https://research.google/pubs/google-vizier-a-service-for-black-box-optimization/.DOI
  7. [7]Gorishniy, Yury; Kotelnikov, Akim; Babenko, Artem. TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling. International Conference on Learning Representations, 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/c1ba41c694834aee9f1ae161711d4939-Abstract-Conference.html.arxiv.org/abs/2410.24210
  8. [8]Huang, Qian; Vora, Jitesh; Liang, Percy; et al.. MLA-genBench: Evaluating Language Agents on Machine Learning Experimentation. Proceedings of the 41st International Conference on Machine Learning, 2024. pp. 20271–20309. https://proceedings.mlr.press/v235/huang24y.html.
  9. [9]Jiang, Zifeng; Schmidt, D.; Srikanth, D.; et al.. AIDE: AI-Driven Exploration in the Space of Code. arXiv preprint arXiv:2502.13138, 2025. https://arxiv.org/abs/2502.13138.
  10. [10]Karpathy, Andrej. autoresearch: AI Agents Running Research on Single-GPU nanochat Training Automatically. GitHub repository, 2026. https://github.com/karpathy/autoresearch.DOI
  11. [11]Li, Ming; Li, D.; Ning, X.; et al.. Auto-RecSys: Harnessing Autonomous Research Agents for Industry-Scale Recommender Systems. arXiv preprint arXiv:2609.10922, 2026. https://arxiv.org/abs/2609.10922.
  12. [12]Liang, Q.; Ingram, S.; Vanica, G.; et al.. Agentic Framework for Deep Learning Workload Migration via In-Context Learning. arXiv preprint arXiv:2606.15994, 2026. https://arxiv.org/abs/2606.15994.
  13. [13]Lu, Chris; Lu, Cong; Lange, Robert Tjarko; et al.. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv preprint arXiv:2408.06292, 2024. https://arxiv.org/abs/2408.06292.
  14. [14]Ma, Jiaqi; Zhao, Zhe; Yi, Xinyang; et al.. Modeling Task Relationships in Multi-Task Learning with Multi-Gate Mixture-of-Experts. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2018. pp. 1930–1939. doi: 10.1145/3219819.3220007.DOI
  15. [15]Nam, J.; Yoon, J.; Chen, J.; et al.. MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement. arXiv preprint arXiv:2506.15692, 2025. https://arxiv.org/abs/2506.15692.
  16. [16]Nikolov, S.; Konrad, B.; Gronbach, M.; et al.. A Multi-Agent AI System for Deep Learning Model Migration from TensorFlow to JAX. arXiv preprint arXiv:2603.27296, 2026. https://arxiv.org/abs/2603.27296.
  17. [17]Novikov, Alexander; Vu, Ngoc; Eisenberger, Marvin; et al.. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. arXiv preprint arXiv:2506.13131, 2025. https://arxiv.org/abs/2506.13131.
  18. [18]Tang, Hongyan; Liu, Junning; Zhao, Ming; et al.. Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. Fourteenth ACM Conference on Recommender Systems, 2020. pp. 269–278. doi: 10.1145/3383313.3412236.
  19. [19]Wang, Ruoxi; Shivanna, Rakesh; Cheng, Derek Zhiyuan; et al.. DCN V2: Improved Deep and Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems. Proceedings of the Web Conference 2021, 2021. pp. 1785–1797. doi: 10.1145/3442381.3450078.DOI
  20. [20]Yang, X.; Yang, X.; Fang, S.; et al.. R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science. arXiv preprint arXiv:2505.14738, 2025. https://arxiv.org/abs/2505.14738.

Paper details

vlnc:W_fnzkpabc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GitHub Copilot, Gemini (multiple versions)

View statement
Looking for related papers…
Contents