Visually guided model generates realistic preferred neural images
Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks
Computer Vision and Pattern RecognitionMachine LearningNeural and Evolutionary Computing
Summary
It can be hard to understand how brain cells or artificial neural networks respond to images. The authors created a new method called Vis-Lens that uses advanced image generation techniques to produce natural-looking pictures that neurons prefer. This method improves how well these generated images activate brain or model neurons and makes more realistic images than previous methods. They tested this using data from human brain scans and artificial networks.
What this means in practice
- •For neural network developers: Generate preferred input images for specific neural units to debug and interpret deep learning models.
- •For medical imaging teams: Create realistic visual stimuli linked to brain activation patterns for improved brain function studies using fMRI data.
Authors
Yusong Zhao, Edgar Y. Walker, Hao Wang, Mi Lu
Abstract
Understanding how neurons respond to visual stimulus is an important question in both deep learning and neuroscience. It has significant implications for enhancing the interpretability of black-box artificial neural networks and for understanding the visual representation in the human brain. We propose a structured visual representations landscape and design an activation-based prior that allows us to effectively regularize the landscape with activations from brain regions or neural network units. Our model Vis-Lens integrates a variational autoencoder and a diffusion model into an image generation framework. It enables the generation of natural, realistic preferred images by directly modifying the activation-regularized latent codes without per-image iterative optimization. We demonstrate the effectiveness of our framework on both artificial neural networks and human visual cortex using functional magnetic resonance imaging (fMRI) data. Our framework improves activation modulation and image realism compared with a recent state-of-the-art method for generating preferred visual representations of these neural targets.