Diffusion model improves prompt generalization for foundation models
Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation
Computer Vision and Pattern Recognition
Summary
Adapting large AI models to new tasks usually requires learning new prompts, which often don't work well outside their original tasks. The authors introduce Diffusion Meta-Prompt (DMP), a method that generates new prompts by learning from many existing prompts using diffusion models without needing the original data. DMP can create prompts for new tasks, combine concepts, and reduce the cost of storing and running prompts. It also improves accuracy on several classification and retrieval tasks compared to previous methods.
What this means in practice
- •For machine learning engineers: Generate generalized prompts that adapt foundation models to new tasks without retraining or access to original data.
- •For digital content creators: Improve text-to-image generation systems by synthesizing new prompts that better capture unseen concepts and compositions.
Authors
Deepak Sridhar, Yi Li, Kartikeya Bhardwaj, Shuangjun Liu, Taotao Jing, Yuan Li, Shuai Zhang, Jiancheng Lyu, Dashan Gao, Nuno Vasconcelos
Abstract
Prompt learning is a popular method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks. In this paper, we introduce a Diffusion Meta-Prompt (DMP) model , a framework that models the distribution of learned prompts using diffusion models. Given a repository of previously learned prompts, DMP is trained and sampled without access to the original task examples or task losses, and synthesizes new prompts conditioned on natural language task descriptions. To improve the sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes. DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training. It reduces storage and inference costs by over 90% compared to prompt retrieval methods. For composite classification, DMP achieves upto 2.0% average gain over prior meta-learning methods across 55 pairs of datasets with gains as high as 8.5% on specific pairs such as Eurosat and Flowers. DMP also enhances cross-task generalization with ~2-9% improvement for hierarchical classification task. We further provide a theoretical guarantee bounding the expected task loss of prompts sampled from a DMP. Code is available: https://github.com/DeepakSridhar/dmp