Benchmark reveals flaws in how expert taste is measured in tests
How to Build a Benchmark That Actually Measures Taste
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Getting computers to judge taste, like choosing which book or brand is better, is tricky because some tests can be easily tricked. The authors created a new test called TasteBench that uses experts' opinions and special rules to make sure the test really measures good taste, not just who can cheat it. They found problems in the way previous tests were made, like always favoring one option or leaking hints that can be guessed. Their methods fix some of these problems but also show that some taste judgments are so hard that even experts fully agree, making it hard to improve further. More work is needed to make taste tests better and more reliable.
What this means in practice
- •For machine learning evaluators: Use TasteBench and its tooling to improve evaluation accuracy by removing biases and ensuring expert alignment in taste-related model assessments.
- •For marketing analytics teams: Assess brand and product preference measurements more reliably by using expert-vetted benchmarks that control for test manipulation artifacts.
Authors
Xiaofan Wu, Xi Zeng, Miaoxia Chen, Peishan Chen, Shuyan Li, Chen Wang, Jiyun Yao, Hanyong Zhong, Jiahao Zhu
Abstract
Expert-aligned "taste" benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benchmark across visual, brand, book, and tech items under a frozen rubric with expert-only gold (panel n>=3; percent agreement >=0.60 primary gate; weak engagement labels train/analysis-only), together with a control protocol that separates craft judgment from test-gaming. Applying the protocol to a Round-1 pilot (primary_human_expert; three biological human experts; n=250 pairs) surfaces and quantifies three validity threats: two construction artifacts that inflate the trivial-baseline layer, and a difficulty ceiling that inflates every score. (1) Position: the preferred item is slot A with probability 0.956, so an always_item_a policy scores 0.956 (fair chance 0.5). (2) Lexical leakage: a filename token cue scores 0.912. (3) Difficulty ceiling: craft domains show near-ceiling human agreement (book/brand/tech 1.0), and even under A/B randomization (eval_assignment_seed7; gold_side_A ~0.47) all five VLM cards reach 1.0 on craft, with only visual separating (~0.84-0.91; a post-hoc hard subset, n=107, leaves craft at 1.0). The protocol (expert-only gold, a weak-label firewall, side randomization at construction, filename scrubbing, and hard filters) makes each artifact measurable and, for the two construction artifacts, correctable; we release a reference eval harness and a side-randomization tool, and show the tool removes the position skew (0.916 to 0.495/0.505). The result is a calibrated view of what the benchmark does and does not yet measure: position and leakage are fixable, whereas the craft-domain ceiling is a pair-difficulty limit that Round-2 must address. We do not evaluate engagement, era/circle conditioning, or calibration here, and we claim no general taste SOTA.