Summary
Most AI models that see and hear focus on understanding words or images but don’t fully grasp physical properties or cause-and-effect in the real world. The authors created OmniPhysics-Nano-V2, a small AI model that looks at images, videos, sounds, speech, and text together while learning physical details like object properties and actions. They trained it using special data that connects what the AI sees with physical events and sounds, and used a new learning process that first gets general tasks right, then focuses on physical understanding. This model performs better than many current top models on tests involving real-world physical reasoning and multimodal perception.
What this means in practice
- •For robotic system engineers: Enable robots to better interpret and reason about physical interactions by integrating vision, sound, and language understanding.
- •For augmented reality developers: Improve AR applications with AI that understands physical object properties and their interactions across visual and audio inputs.$Commercial implications: Offers enhanced AI features for AR products that require accurate physical world understanding for immersive experiences.
Authors
Yizhou Liu, Jinghang Han, Kaixiang Qiu, Qi He, Minghao Han, Yue Jiang, Xujia Chen, Wei Zou, Shunli Wang, Lihua Zhang, Dingkang Yang
Abstract
Omni-modal models have expanded multimodal interaction across vision, audio, speech, and language. However, their training is predominantly organized around semantic descriptions and general-purpose objectives, leaving physical attributes, interaction states, and causal mechanisms only partially specified. This gap is not simply a matter of modality coverage: adding more modalities does not by itself provide the supervision needed to connect observations with the physical structure of the world. We present OmniFysics-Nano-V2, a compact omni-modal model for physical-world perception and understanding. The model supports image, video, audio, speech, and text inputs within a shared reasoning framework, together with text and speech generation. To address the lack of explicit physical supervision, we construct a dual-branch physics-aware data pipeline that grounds salient objects in structured physical attributes and aligns visual changes with acoustic events, intermediate responses, and interaction outcomes. To address homogeneous training objectives, we curate reinforcement-learning prompts by reward diversity and adopt a two-stage Group Relative Policy Optimization curriculum that progresses from general task correctness to fine-grained physical perceptual reasoning. Experiments across multimodal, audio-visual, and physical reasoning benchmarks show that the proposed data and training strategy improves physical-world understanding while preserving broad omni-modal competence. The proposed model achieves leading result on 17 of 21 benchmarks against SOTA omni-modal models. By equipping AI systems with both omni-modal and physical-world perception capabilities, OmniFysics-Nano-V2 is poised to become a cornerstone of next-generation Physical AI.