Papers for

virtual communication developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Audio guided method improves facial video restoration and lip sync

TaoMate-FVSR:Audio-Guided Facial Video Restoration

Abstract: AI-generated videos often depict people within detailed scenes while leaving their small facial regions poorly resolved. Visual super-resolution can sharpen these regions without resolving whether the recovered lips and teeth agree with the spoken audio. We present an audio-guided facial video restoration framework that addresses this ambiguity within a reference-guided video flow model. Source--noise mixing initializes reconstruction from a partially preserved observation, and structured reference injection separates appearance anchors from continuous motion context. Training combines restoration of degraded images and videos with correction of degraded dubbed sources under target audio. Appearance and synchronization supervision allow the model to recover facial detail and support visual dubbing when the audio is replaced. On talking-head restoration benchmarks, our method achieves the best paired perceptual similarity and synchronization scores among the compared restoration methods. Component comparisons examine audio conditioning, the reconstruction formulation, and reference injection. Native AIGC examples further illustrate the recovery of lip contours, dental structure, and surrounding facial texture in scene-level footage. injection. Native AIGC examples further illustrate the recovery of lip contours, dental structure, and surrounding facial texture in scene-level footage.

Mon 2 NovComputer Vision and Pattern Recognition
The gist
Videos created by AI often show detailed scenes but have blurry faces, especially around the mouth. The authors created a new method that uses the voice audio to help fix face videos so the lips and teeth match the spoken words. Their system can restore small facial details and keeps lip movements synchronized even if the audio changes. Tests show it performs better than previous methods on talking-head videos. This approach helps make AI videos look more realistic and believable around the face.
Open → 2611.03393v1