Coding agents face challenges migrating machine learning from gpus to tpus
From GPU to TPU with Coding Agents: What Works, What Fails, and What It Costs?
Software EngineeringDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Switching machine learning work from GPUs to TPUs can save money and speed up training, but it is tricky. The authors tested an AI coding agent moving projects from NVIDIA GPUs to Google TPUs and found many failures, often invisible errors that quietly changed model results. Some migrations worked well with human help, but most testing attempts failed to run correctly. This means teams should carefully check and verify any automation used to switch hardware.
What this means in practice
- •For cloud ml engineers: Assess and verify AI tools that automatically migrate ML workloads from GPUs to TPUs, accounting for hidden errors and human correction needs.
- •For software reliability teams: Use this study to guide testing and acceptance criteria for hardware migrations that preserve model accuracy and performance.
Authors
Hui Sun, Anderson Uchôa, Wesley K. G. Assunção
Abstract
A coding agent can migrate a machine learning workload, report success, and still change how the model trains. Rising compute costs, limited accelerator capacity, and vendor dependence are pushing organizations toward alternatives to graphics processing units (GPUs); a key question is whether agent-generated migrations to tensor processing units (TPUs) can be trusted and at what cost. We build a dataset of 148 PyTorch ML research projects and use Claude Code to migrate ten of them from NVIDIA GPUs to Google Cloud TPUs. We independently assess whether each migration runs, preserves numerical behavior, executes on the intended hardware, maintains the project's goal, and delivers acceptable performance. We also examine failures, human effort, and costs. Eight projects meet our project-goal criterion, and migrations require 2.4 human interventions on average. Of 70 execution attempts, 53 fail (76%), and 26 of these failures (49%) raise no error. The most frequent cause is repeated compilation and other execution inefficiencies, while interactions with third-party libraries can silently change the computation: in one four-chip migration, the agent reports success despite a change in gradient clipping that reduces accuracy from 86.53% to 84.89%. Execution costs vary widely: in one workload, TorchTPU, a PyTorch execution backend for TPUs, trains 1.86 times faster and 25% cheaper per run than the GPU. For practitioners, these findings motivate independent reference runs, verification under the intended hardware configuration, and workload-specific acceptance and stopping criteria that account for both computational and human costs.