Cluster-based method selects small effective training sets for multimodal instructions
CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives
Artificial Intelligence
Summary
Choosing the right examples to teach AI systems can be tricky because some examples are very similar and can crowd out others. The authors propose a method called CLIVO that groups examples into many small clusters and automatically learns how much attention to give each group based on how well they help with the target task. This approach uses a clever way to optimize the mix of examples without extra guesswork, picking fewer but better examples from a large pool. The method achieves nearly the same performance as using all data but with a tiny fraction of examples, making AI training more efficient.
What this means in practice
- •For machine learning engineers: Efficiently select diverse training examples from large multimodal datasets for faster model fine-tuning with limited data.
- •For data annotation teams: Prioritize clusters of instructions to label or validate for improved model performance using fewer resources.
- •For ai product managers: Create smaller, targeted training subsets for multimodal AI applications reducing costs while nearly maintaining accuracy.$Commercial implications: Enables selling efficient training pipelines to companies needing cost-effective AI model deployment.
Authors
Yuhan Wang, Rui Hu, Batian Di, Tan Jiang, Dongsheng Ma, Leichao Dong, Yuanzi Li, Yalin Feng
Abstract
Target-aware multimodal instruction selection requires useful examples and an appropriate balance of supervision. Sample-level rankings can overrepresent similar content without optimizing this balance; group-level methods face coarse units, predefined set-level validation signals, and costly or inaccurate mixture search. We propose CLIVO, a target-aware framework connecting fine-grained semantic allocation, automatically learned validation objectives, and differentiable mixture search. Specifically, semantic clusters provide independently adjustable shares at the hundred-cluster scale. Correlations between per-example merged-proxy losses and real probe performance identify target-relevant validation examples; learned nonnegative weights combine their losses into an objective. A single backward pass through the merged proxy supplies gradients for all shares, avoiding an additional mixture-response regression model. Selecting 100K instructions from an 8M pool, CLIVO attains 97.2% of the full-data four-target average score with only 1.25% of the data. It improves on the strongest evaluated selection baseline by 1.16 points using random within-cluster sampling; combining learned quotas with sample ranking increases this advantage to 1.55 points. Analyses show that finer units benefit most targets, short-trained experts preserve mixture responses, sample weighting retains useful validation information, and direct search consistently reduces the learned objective. CLIVO also supports interpretable allocation, modular selection, and reuse across targets and granularities.