Data optimization techniques enhance large language model training and usage
Data Optimization for LLMs: A Survey
Artificial Intelligence
Summary
Large language models (LLMs) need huge amounts of data to work well, but not all data is equally useful or easy to use. This paper reviews many ways to improve LLMs by selecting better data, adding useful variations, giving different importance to data, and other tricks. The authors organize and explain these techniques, pointing out their strengths and connections. They also highlight the challenges and suggest directions for making LLM training and use more efficient with smarter data handling.
What this means in practice
- •For machine learning engineers: Improve the efficiency and effectiveness of training large language models by applying categorized data optimization techniques focused on data quality and diversity.
- •For ai platform developers: Design data pipelines that incorporate data filtering, augmentation, and weighting to enhance large language model inference and scalability.
A survey. It maps existing work.
Authors
Ou Wu
Abstract
Large language models (LLMs) have attracted great attention in both academic and industrial communities in recent years. Numerous survey studies have covered various aspects of LLMs, such as learning strategies, applications, alignment, explainability, and evaluation. However, little headway has been made in reviewing data-related aspects for LLMs, despite data being recognized as a key factor contributing to the emergence of the LLMs era. A wide range of data optimization algorithms and techniques are employed in LLMs' training and application processes. These techniques often originate from diverse deep learning domains, with their theoretical inspirations and heuristic motivations appearing unrelated to each other. This study aims to develop a comprehensive framework for LLMs' data optimization techniques to enhance data utilization and efficiency. We first discuss data challenges in LLMs' training and inference. Second, we summarize data perception dimensions for evaluating LLMs' data properties. These dimensions primarily include quality, difficulty, diversity, and redundancy, laying a foundation for subsequent optimization. Third, existing optimization techniques are reviewed and categorized into four groups: data selection, augmentation, reweighting, and others. Fourth, we analyze interconnections between these categories. Finally, we identify key challenges and future directions for LLMs' data optimization compared with conventional deep learning approaches.