Mind the Couch! Eliciting MLLM Reasoning in Interior Design via Weak-to-Strong Task Vector Injection
2026-08-24 • Multimedia
Multimedia
AI summaryⓘ
The authors found that current multimodal large language models (MLLMs) struggle to accurately understand detailed interior design images, often making mistakes like spatial overlaps or mismatched colors. They created a method called DART-I to help these models reason better by directly adding helpful spatial and color information into the model's internal processing. This approach avoids retraining the models and is more efficient. Tests show that DART-I improves the model's ability to handle complex interior design tasks.
Multimodal Large Language ModelsModality MisalignmentInterior DesignVisual EncodingLatent SpaceTask VectorsDeterministic PriorsResidual InjectionFine-tuningCatastrophic Forgetting
Authors
Yuxuan Yang, Jingyao Wang, Luntian Mou
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated great performance, yet they often suffer from severe modality misalignment when confronted with densely constrained spaces for interior design. Due to the loss of high-frequency local topological details and fine-grained aesthetic shifts during visual encoding, existing MLLMs frequently hallucinate, yielding physical spatial collisions and visual aesthetic dissonance. To address this, we propose Dual-prior Activation Residual Task-vectors Injection mechanism (DART-I) for MLLMs. It shifts the paradigm from lossy text-prompting to direct latent intervention, utilizing weak-to-strong deterministic rules to anchor the causal reasoning of MLLMs for interior design. Specifically, DART-I operates in three steps: it first explicitly extracts continuous spatial distance and color typography features from images using extremely lightweight weak experts; subsequently, it transforms these deterministic priors into directional task vectors via a linear projection network; these vectors are dynamically injected as residual terms into the latent space of the frozen MLLMs, steering MLLMs towards precise reasoning for interior design. Stepping outside the conventional paradigms, our method achieves precise reasoning without fine-tuning the MLLMs, effectively bypassing expensive computational costs and catastrophic forgetting. Extensive experiments on various benchmarks demonstrate the effectiveness and advantages of DART-I.