Large language models show limits in data science code translation tasks
ORCA: Evaluating LLMs on Data Science Code Translation
Artificial Intelligence
Summary
Data scientists often need to convert code from one library to another without changing what the code does. The authors studied how well advanced AI models called large language models (LLMs) can do this code translation. They created a new test set called ORCA with many tasks and real projects to check this ability. Results showed that current models still make many mistakes, especially with bigger projects, but adding extra info about the code's purpose helps improve their success.
What this means in practice
- •For data engineering teams: Validate and improve tools that automatically convert codebases between data science libraries while ensuring correctness.
- •For machine learning infrastructure teams: Enhance code translation features in AI-powered development tools by incorporating intent-aware methods discovered in this study.
Authors
Xiaolong Li, Jinyang Li, Bowen Qin, Ge Qu, Nan Huo, Xiaohan Xu, Shipei Lin, Reynold Cheng
Abstract
Data Science Code Translation (DSCT) is the process of converting code between data science libraries while preserving functional equivalence and enabling interoperability across data science ecosystems. While Large Language Models (LLMs) have demonstrated considerable progress in Data Science Code Generation (DSCG), their performance in DSCT remains insufficiently studied. To address this gap, we introduce ORCA, a comprehensive benchmark with two complementary settings: ORCA-MAIN, which comprises 1,600 carefully curated grounding-level tasks across 3 representative domains: Data Querying, Data Manipulation, and Deep Learning; and ORCA-PROJECT, which contains 200 translation tasks over complete data science projects across 7 data science task types. Each task is accompanied by annotated reference translations and test cases for validating functional equivalence. We further incorporate a multi-stage quality verification process that thoroughly verifies task correctness and test case robustness. Experimental results demonstrate challenges in DSCT, with even frontier LLMs showing limited performance. Specifically, Claude-Opus-4.6 achieves a success rate of 56.92% on ORCA-MAIN and 33.67% on ORCA-PROJECT, indicating considerable room for improvement in DSCT. We also observe a clear directional preference in DSCT, where translation is consistently easier when the source code expresses the task through more explicit, fine-grained operations. Motivated by this, we propose an intent-augmented method, in which the model first infers source-code intent and then uses it as additional context for translation, achieving average absolute success-rate gains of 4.80% and 5.33% on ORCA-MAIN and ORCA-PROJECT, respectively.