Image modeling evolves from handcrafted priors to AI foundation models
Image Modeling: From Handcrafted Priors to Foundation Image Models
Computer Vision and Pattern Recognition
Summary
The paper looks at how techniques for understanding and creating images have changed over 60 years. Older methods used fixed mathematical rules made by humans, while modern approaches learn from large amounts of data using AI models. The authors explain that both old and new methods try to answer the same question: what makes an image look real? They show how this question can be understood with a common mathematical idea called the Bayesian framework, which helps connect traditional image restoration and modern AI-based image generation in one theory.
What this means in practice
- •For medical imaging teams: Use modern image modeling techniques to improve reconstruction and noise reduction in medical images by integrating handcrafted and learned priors.
- •For photography software developers: Develop photo editing tools that enhance image restoration and generation by applying unified Bayesian models combining classical and AI approaches.$Commercial implications: This paper supports building commercial photo enhancement and generation tools leveraging advanced image priors and foundation models.
A survey. It maps existing work.
Authors
A. Murat Tekalp
Abstract
This paper unifies six decades of image modeling, evolving from handcrafted mathematical tractable priors to solve inverse problems in imaging to data-driven deep foundation image models for both image generation and solving inverse problems, under a common Bayesian framework that formulates image modeling as an implicit or explicit probabilistic characterization of a data manifold. Every image restoration or generative task fundamentally seeks to answer the same question: What makes an image look like a real image? In the Bayesian framework, this question is encapsulated by the prior probability density p(x), or its more optimization-friendly counterpart, the log-prior log p(x). While early works in solving inverse problems, such as Tikhonov, Total Variation (TV), and Non-local priors relied on explicit handcrafted definitions of log prior to model local smoothness, modern techniques, such as Plug-and-Play (PnP) priors and Denoising Diffusion models, are rooted in algorithmic uncoupling and implicit manifold learning that apply to both inverse problems and conditional image generation. By grounding imaging inverse problems and conditional image generation under a single mathematical engine: the gradient of the log-prior, or the Stein Score Function, we bridge the gap between signal processing and machine learning communities, turning what seems like parallel developments into a single, evolution of image modeling from handcrafted analytical models to models learned from big data, demonstrating that the pursuit of the denoising score is the common thread connecting variational optimization (in MAP estimation) to posterior sampling (in modern foundation image models).