Convolutional neural networks adapt to image directions for better quality
C$^{2}$-INR: Customized Convolutional Implicit Neural Representation
Computer Vision and Pattern Recognition
Summary
Neural networks can represent images by treating them as continuous pictures rather than pixels, but most current methods look at each pixel separately and miss important connections between neighboring pixels. The authors introduce a way to customize these networks to better understand and mimic the direction and details in an image by using special kernels that pay more attention to important directions. They also let each kernel choose its best activation function to improve performance. Their approach, called C²-INR, performs better than existing methods at tasks like repairing images and improving their resolution.
What this means in practice
- •For image processing engineers: Implement image restoration and enhancement systems that adapt convolutional filters to image content for better results in inpainting and super-resolution.
- •For computer vision developers: Develop algorithms for continuous image representation that leverage content-aware convolutional kernels for improved image analysis.
Authors
Jinglei Shi, Xinran Chang, Jiaqi Cui, Yingjie Xia, Zhaolin Xiao, Chongyi Li
Abstract
Implicit Neural Representation (INR) leverages neural networks to represent discrete signals such as images as continuous ones, where the network weights serve as a compact form of the signal itself. Most existing INR methods adopt Multi-Layer Perceptrons (MLPs) as their backbone. Since these models render each pixel independently, they inherently fail to exploit the spatial correlations that exist between neighboring pixels. In contrast,convolutional INRs can process pixels in parallel while inherently accounting for inter-pixel dependencies, making them a more natural fit for representing images. Nevertheless, convolutional INRs remain relatively underexplored, and the majority of them rely on fixed architectural settings, leaving little room for image-specific adaptation. In this paper, we investigate network customization for convolutional INRs. We replace conventional filters with irregular directional kernels, whose allocation is guided by the directional energy in the image spectrum, i.e., directions exhibiting stronger energy are assigned a larger number of kernels, enabling content-tailored convolution settings. These kernels are further reformulated via an orthogonal basis to achieve a superior sparse representation. Moreover, we introduce an annealed Gumbel-Softmax-based mechanism for kernel-level activation function selection, which gives the most suitable activation function for each convolution kernel. Extensive experiments demonstrate that our method, namely C$^{2}$-INR, achieves superior performance against state-of-the-art approaches under comparable parameter budgets across a wide range of image processing tasks, including representation, inpainting, and super-resolution.