CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors developed two new tools to improve virtual try-on systems, which let users see how clothes look on them digitally. First, they created VIP-SAM, a way to find a specific garment in a photo of a person wearing it, based on an image of the garment alone. Second, they built CtrlVTON, a system that lets users control how clothes fit and look on the person by editing images using detailed garment outlines. Their methods outperform existing systems in accuracy and user control. This work helps users customize virtual clothes more precisely in images.
Virtual Try-On (VTO)Instance SegmentationImage EditingGarment LayoutPixel-Level ControlFlatlay ImageGarment FidelitySpatial PlacementStyle Control
Authors
Seungyong Lee, Hyun Jun Jang, Sangoh Kim, Sungjoon Park
Abstract
Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.