Summary
Most current AI models that understand relationships in images rely on a fixed set of relationship words they were trained with, limiting their ability to work with new or different vocabularies. The authors created RelateAnything, a model that can predict relationships between objects in images using any list of relationship words given at runtime, without needing retraining. It works fast and on regions of an image from any detection method. To train and evaluate it, the authors also built a large dataset of varied relationships expressed in free text and a new benchmark that tests how well these models transfer to different kinds of data. This approach improves accuracy and flexibility in detecting relationships.
What this means in practice
- •For computer vision engineers: Integrate relation prediction into image analysis tools using any detector and vocabulary at runtime without retraining models.
- •For augmented reality developers: Enable AR apps to recognize and interpret diverse object relationships in real time using customizable textual descriptions.
Abstract
Open-vocabulary detection accepts any class list at inference, and promptable segmentation returns regions without class names: the taxonomy has left the model and become an input. Relation prediction has not. Scene-graph models are still trained and evaluated on the 50 or 56 predicates of one annotation style, their relation head conditioned on object labels and so tied to one detector. Three obstacles explain this, none primarily modelling: no relation corpus is both free-text and verified, a label-conditioned architecture cannot accept a vocabulary it was not trained on, and the standard metric rewards agreement with the training corpus, so a larger vocabulary scores as a regression. We present RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings. Object labels are never an input, so the region source can change without retraining, and the vocabulary is a bank of text embeddings, not a learned classifier. It runs at 20 ms/frame. Training over 19,103 predicates requires positive-unlabeled supervision and a text encoder that separates antonyms, which contrastive encoders embed at cosine 0.95. To supply the supervision we build RA-4M, 474k images and 4.3M relations over 10,102 free-text predicates, generated against numbered box markers and geometrically verified. To measure it we build OV-SGG-Bench, six axes scored across datasets that the priors standard recall rewards cannot satisfy. On three cross-dataset benchmarks and a fourth zero-shot, RelateAnything has 2.3-3.5x the mean recall of the strongest open-vocabulary method of comparable scale, margins that survive a real detector, and leads a 3B-VLM scene-graph model on both metrics at under 2% of its parameters. In-domain measurement overstates transfer gains ~5x. Model, corpus and benchmark are public.