AlbumVSR improves face video clarity using personal photos

AlbumVSR: Identity-Preserving Face Video Super-Resolution from a Personal Photo Album

Computer Vision and Pattern Recognition

Summary

Faces in blurry or low-quality videos can lose details that help recognize who they are. The authors created AlbumVSR, a method that uses clear photos of a person from their photo album to help restore their face in videos. It combines this extra identity info with a special transformer model that focuses on the person's pose and appearance. This approach made faces in test videos look more like the real person, recovering details better than previous methods.

What this means in practice

  • For video editing teams: Improve facial detail and identity accuracy in restoring low-quality personal videos using a person's own photo collection.
  • For security video analysts: Enhance blurry face footage by referencing personal photos to better identify individuals in surveillance videos.

Authors

Nianyao Du, Xijun Wang, Stanley H. Chan

Abstract

Severe blur, downsampling, noise, and compression can erase the identity cues needed to faithfully restore a person's face. We therefore use photographs of the same person as an external source of identity. AlbumVSR conditions a frozen, one-step streaming diffusion transformer on a personal album. We use four MagFace/CLIP set tokens to encode the album, and 64 texture tokens which come from a fused UV atlas rendered at the current window's estimated face pose. These tokens are appended to the transformer's cached cross-attention context. On an identity-disjoint VoxCeleb2 benchmark with heavy synthetic degradation, AlbumVSR raises ArcFace similarity to the ground truth from 0.586 to 0.670 and face PSNR from 24.98 to 26.71 dB. On the 91 test identities, its mean identity similarity is higher than that of all 14 baselines.