Simple method helps spot hidden dataset biases from teacher models
Towards Identifying the Dataset Biases Causing Phantom Transfer
Machine Learning
Summary
Sometimes machine learning models pass along hidden biases to newer models even when those biases aren't obvious in the training data. The authors show that using Sentence BERT embeddings can reveal the topic of these subtle biases, especially when the source model is known. They also find that different models show biases in different words. This helps understand and detect biases that sneak through despite attempts to remove them from datasets.
What this means in practice
- •For machine learning engineers: Detect topics of hidden biases transferred via teacher models when the source model is known to improve data trustworthiness.
- •For ai safety teams: Identify stealth bias channels in datasets that evade common defenses by examining signature differences caused by different teacher models.
Authors
Jonas Jürß, Pietro Liò
Abstract
Recent work has shown that a teacher model can transfer a bias to a student through a dataset from which every explicit reference to that bias has been filtered out, and that no data-level defense reliably removes or detects it even when knowing what bias to look for. Aiming to shed light on the hidden traces of these biases, we show that a simple signature based on Sentence BERT embeddings can identify the topic of such a bias with a Matthews correlation coefficient of 0.83 if the teacher model used by the attacker is known and 0.46 if it is not. Additionally, we observe that different teacher models appear to express the same bias through different vocabulary.