Cross operating system code migration remains a tough challenge
OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration
Software Engineering
Summary
Moving software from one operating system, like Linux, to another, like Windows, while keeping it working exactly the same way is hard. The authors created OS-Bench, a test set of real tasks where parts of software need to be adapted between Linux and Windows. They tested several AI coding assistants on these tasks and found that even the best ones succeeded less than 27% of the time while keeping programs working correctly on both systems. This shows that automatic cross-OS code migration is still very difficult and OS-Bench can help improve tools for this job.
What this means in practice
- •For software engineering teams: Test and improve tools for adapting codebases to run correctly on both Linux and Windows environments.
- •For devops engineers: Verify and automate cross-platform compatibility fixes in deployment pipelines using benchmark tasks reflecting real-world OS differences.
Authors
Qirui Jin, Kenan Li, Liao Zhu, Lingching Tung, Yikai Zhang, Yijia Wu, Zijian Jin, Pengfei Gao, Yu Kang, Qingwei Lin, Wenke Lee
Abstract
Language-model (LM) agents are increasingly evaluated on repository-level software engineering (SWE) benchmarks derived from GitHub issues and pull requests. Although these benchmarks cover a broad range of development tasks, they provide limited insight into a common yet technically distinct challenge: migrating an existing implementation from one operating system (OS) to another while preserving behavior on both. We introduce OS-Bench, an executable benchmark for cross-OS code migration between Linux and Windows. Motivated by case studies showing that cross-platform features are frequently implemented on one OS before support is added for another, OS-Bench identifies migration opportunities by comparing execution coverage collected from the same repository on both systems. Platform-specific regions are then verified, organized into five migration scenarios, and transformed into repository-level tasks with target-platform tests and source-platform regression tests. The resulting benchmark contains 140 validated tasks from 111 repositories across C, C++, C#, Go, and Rust, comprising 91 Windows-to-Linux and 49 Linux-to-Windows migrations. We evaluate six models with SWE-agent, Claude Code, OpenHands, and Win-agent. The best configurations achieve only 26.4% success on Linux-target tasks and 20.4% on Windows-target tasks while preserving correctness across both systems. These results indicate that cross-OS migration is a challenging and underexplored capability, and OS-Bench is a testbed for improving platform-aware SWE agents.