Papers for

data quality analysts

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Embedding models poorly capture real world measurements

Embedding Models Measure in Peculiar Ways

Abstract: Embedding spaces define notions of semantic similarity and distance. We study whether those embeddings reflect physical measurements of mass, distance, time and volume, which admit a unique, objective notion of semantic equivalence and distance. We find that physical measurement is only weakly modeled in the embedding space, and that instead quite peculiar measurement patterns can be observed. Further analysis indicates that embedding representations of physical measurements are strongly influenced by superficial string similarity, and recalibration of similarity does not substantially improve the alignment.

Thu 17 SeptComputation and LanguageMachine Learning
The gist
Embedding models are designed to represent meaning, such as similarity between words or concepts, in a numeric space. This paper shows that when it comes to representing real-world physical measurements like mass, distance, time, and volume, these embeddings do not reflect these quantities accurately or meaningfully. Instead, the models seem to rely more on superficial features like how the measurement values look as text strings. Attempts to fix this misalignment only made small improvements.
Open → 2609.20821v1

Vision language models measured for caption quality and alignment gaps

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport

Abstract: Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing accounts connect this modality gap to initialization, contrastive dynamics, and information imbalance, while its distributional and pairwise contributions to retrieval remain unresolved. We introduce UOT-Gap, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT). The UOT optimum separates transport, coupling complexity, and marginal mass variation; a complementary pair-aware residual compares observed image-caption pairs with the UOT soft matching. On Flickr8K and COCO-1K with frozen CLIP, OpenCLIP, and SigLIP encoders, caption degradation reduces Flickr8K Recall@1 from 0.559 to 0.003. Across six dataset-model conditions, the pair-aware residual tracks retrieval degradation with mean absolute Spearman 0.973, compared with 0.392 for the mean gap. The association remains stable across five random COCO-1K subsets at $0.954\pm0.026$, with a minimum of 0.943. UOT barycentric updates reduce the transport objective while degrading retrieval, distinguishing geometric objective descent from task improvement. These results establish UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.

Wed 9 SeptComputer Vision and Pattern Recognition
The gist
When computers match pictures with words, they use a shared space to compare them, but pictures and words often don’t line up perfectly. The authors created UOT-Gap, a way to measure how much this mismatch happens without retraining the models. They found that as captions get worse, UOT-Gap clearly shows the drop in matching quality. This method helps spot how well image and text parts align and how robust the matching is to poor captions.
Open → 2609.10224v1