Gear improves outfit choices by matching clothes to 3d virtual scenes
GEAR: Generative Editing for Scene-Conditioned Outfit Retrieval for VR Travel
Computer Vision and Pattern Recognition
Summary
Picking clothes in virtual worlds so they fit well with the surroundings is hard because outfits and 3D scenes look very different to computers. The authors created GEAR, a system that first writes a description of what clothes would fit the scene, then paints that outfit on a simple character, and finally finds clothes similar to that painting. This helps pick outfits that look good and fit nicely in the virtual scene. They tested it with automated scores and user studies and found it works better than other common methods.
What this means in practice
- •For virtual reality developers: Integrate real-time context-aware outfit selection in VR travel experiences to enhance user immersion by matching clothing to virtual scenes automatically.
- •For game designers: Use scene-conditioned outfit retrieval to streamline avatar customization that fits both style and environment in 3D interactive games.
Authors
Gia-Huy Dinh, Thinh-Phuc Nguyen, Thanh-Hai Nguyen, Lam-Huy Nguyen-Ngoc, Hoang-Bach Nguyen-Phan, Nam-Quan Nguyen, Minh-Triet Tran, Trung-Nghia Le
Abstract
In virtual environments, choosing appropriate outfits requires contextual harmony with surrounding scenes, yet bridging the visual gap between 3D ambient views and catalog garment assets remains challenging. This paper investigates scene-conditioned outfit retrieval for interactive 3D environments, where a user's egocentric view serves as the retrieval signal. We present GEAR (Generative Editing for Asset Retrieval), a framework that bridges the cross-domain gap by synthesizing intermediate visual queries rather than predicting direct compatibility. GEAR uses a vision-language model to describe suitable clothing from a captured scene, employs an instruction-guided diffusion model to render the concept onto a standardized reference character, and utilizes the cropped clothing region to retrieve existing 2D outfit assets via embedding similarity. An interactive Unity prototype demonstrates real-time scene capture, asset retrieval, and in-engine 3D avatar fitting. We benchmark GEAR against direct visual similarity, aesthetic scoring, and VLM-based reranking on a controlled dataset using automated Swiss-system pairwise judgments, where GEAR achieves competitive top-ranked performance (mean Elo@5 of 1012.75 and top-10 overlap of 0.180) alongside 0.539 pairwise agreement. Furthermore, a user study across 20 virtual travel scenes demonstrates that GEAR achieves the highest overall participant preference in both visual appeal (34.5%) and contextual suitability (34.5%).