Code embeddings show trade off between intent and detail
Lost in the Latent Space: How Code Embeddings Represent Functional Intent and Implementation Fidelity
Software EngineeringComputation and LanguageInformation Retrieval
Summary
Code embedding models turn computer code into numbers so machines can understand and work with it. This study looked at 14 leading models to see how well they capture both what code is meant to do and how exactly it does it. The authors found a trade-off: some models focus on the big picture of what code does but miss details, while others capture details but struggle with general meaning. This means a single code embedding that works perfectly for all purposes is not yet available.
What this means in practice
- •For software engineering teams: Choose and evaluate code embedding models based on whether tasks require understanding code intent or precise implementation details.
- •For automated code review teams: Design tools that balance detecting stylistic differences and understanding functional equivalence using insights on embedding behavior.
Authors
Lu Sun, Yuheng Huang, Lei Ma, Zhijie Wang
Abstract
Code embedding models have become a key building block for many software engineering applications, yet their internal representations remain poorly understood. Existing evaluations tend to focus on downstream task performance, offering limited insight into whether a single embedding can support both functional abstraction and implementation-sensitive analysis. To address this gap, we present a large-scale empirical study of 14 state-of-the-art code embedding models. We evaluate their representational behavior through three complementary analyses: supervised probing for functional abstraction and implementation fidelity, zero-shot retrieval experiments to assess geometric similarity, and controlled perturbation experiments that measure embedding responses to semantics-preserving code transformations. The results reveal a consistent abstraction-fidelity tension: models that better preserve stylistic and structural signals tend to do so at the expense of semantic generalization, while models with similar retrieval performance can nevertheless respond very differently to the same implementation changes. Overall, our findings suggest that a single universal code representation remains out of reach, motivating more representation-aware evaluation, selection, and deployment of code embedding models in software engineering.