HarnessIR improves real-world image restoration with multimodal models
HarnessIR: Harnessing Multimodal Foundation Models for Universal Real-World Image Restoration
Computer Vision and Pattern Recognition
Summary
Real-world photos often have many kinds of damage, like blur and noise, all mixed together, which makes fixing them hard. The authors show that instead of fixing each problem one by one, their method uses a powerful multimodal model to restore images all at once. This process involves understanding the image, deciding what needs to be done, doing the fix, and then checking if it looks better. Their approach works better on tough real-world images than earlier methods.
What this means in practice
- •For photo editing software developers: Integrate a single multimodal model to improve restoration of diverse image damages in user photos without chaining multiple specialized tools.$Commercial implications: Enables new photo editing features that handle complex real-world image issues more effectively, differentiating products for consumers.
- •For surveillance system operators: Improve clarity of surveillance footage degraded by noise and blur through a unified restoration approach using a single multimodal model.
Authors
Xiangtao Kong, Shuaizheng Liu, Rongyuan Wu, Lingchen Sun, Zhengqiang Zhang, Jinxin Zhao, Yuhui Wu, Lei Zhang
Abstract
Real-world low-quality images suffer from complex mixed degradations, including but not limited to noise, blur, atmospheric effects, etc. Recent agentic methods usually model real-world image restoration (Real-IR) as a sequential tool calling problem over task-specific single-degradation restoration models. This paradigm, however, is fundamentally limited because complex real-world degradations cannot be cleanly undone degradation by degradation, and the tool used for task-specific models caps the capability of the agent system. In this work, we present HarnessIR, an agentic framework for Real-IR by harnessing a multimodal foundation model (MFM) as the executor. HarnessIR consists of five stages: perception and diagnosis, on-demand tool invocation, prompt composition, execution, and verification-driven refinement. Unlike prior agentic Real-IR methods that rely on tool chains assembled from task-specific models, HarnessIR feeds the restoration requirements, the perceptual diagnosis, and the evidence into an MFM that performs restoration in a single pass, followed by verification stages to determine whether the result warrants further processing. Under our harness, off-the-shelf MFMs handle restoration tasks remarkably well, achieving state-of-the-art results on the widely used MiO100 synthetic benchmark. More importantly, by exploiting the strong generalization ability of MFMs, HarnessIR delivers compelling restoration quality on challenging real-world scenes where previous agentic IR systems often struggle. Codes is available at https://github.com/PolyU-VCLab/HarnessIR.