Cross modal translation improves chemical structure recognition accuracy
AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation
Artificial IntelligenceComputation and Language
Summary
Translating information from images to text is tricky, especially when the connections between them aren’t perfect or clean. The authors created a method called AC-JEPA that learns from both accurate and imperfect data by predicting hidden features between image and text forms. This approach improves how well machines can read chemical structures from pictures compared to older direct-training methods. It works better even when the training data includes mistakes, making it more reliable for real-world use.
What this means in practice
- •For chemical data teams: Improve automated conversion of chemical structure images into accurate digital representations using a method tolerant to imperfect paired data.
- •For document digitization teams: Improve cross-modal translation systems to better extract structured information from noisy or imperfect real-world paired image and text data.
Authors
Chaofei Hong, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang
Abstract
Cross-modal translation requires models to capture shared semantics across modalities while preserving the fine-grained structure required by the target modality. Yet in real-world scientific domains, paired data are often imperfect, while obtaining clean pairs at scale is costly. Supervised fine-tuning (SFT) directly optimizes cross-modal translation through categorical token-level supervision, but this objective is sensitive to errors in the target sequences. JEPA offers a complementary representation-space objective, but most existing formulations focus on prediction within a modality rather than translation across modalities. We introduce Anchored Cross-Modal JEPA (AC-JEPA), an SFT-compatible plug-in that predicts masked intermediate states in both cross-modal directions using paired data. The joint-embedding objective targets cross-modal structure predictable from context, while the causal SFT path retains direct token-level supervision for the target sequence. During self-training, AC-JEPA learns from continuous latent targets derived from imperfect pseudo-labeled pairs, offering intrinsic tolerance to local data errors without promoting pseudo-tokens to hard categorical supervision. Using optical chemical structure recognition as a testbed, AC-JEPA trained on clean pairs achieves 76.07% accuracy on the Simple set of the WildMol benchmark, compared with 67.40% for SFT alone. Using the same pseudo-labeled data for both approaches, AC-JEPA-based self-training further raises accuracy to 80.23%, whereas accuracy under direct SFT drops from 67.40% to 63.47%. These results support SFT-anchored joint-embedding prediction as an effective strategy for cross-modal translation under both curated and imperfect supervision.