The nextAI Solution to the NeurIPS 2023 LLM Efficiency Challenge
2026-04-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors worked on improving a large language model called LLaMa2 70 billion to make it faster and use fewer resources. They fine-tuned the model on one GPU within 24 hours by using special techniques like QLoRA and Flash Attention 2. They created a custom dataset from open sources and tried different setups to find the best balance between speed and accuracy. Their final model performed well on question-answering tests while being efficient to run. This shows that big language models can be optimized even when computing power is limited.
Large Language Models (LLMs)LLaMa2Fine-tuningGPUQuantized Low Rank Adaptation (QLoRA)Flash AttentionLoRANatural Language ProcessingBenchmarking
Authors
Gyuwon Park, DongIl Shin, SolGil Oh, SangGi Ryu, Byung-Hak Kim
Abstract
The rapid evolution of Large Language Models (LLMs) has significantly impacted the field of natural language processing, but their growing complexity raises concerns about resource usage and transparency. Addressing these challenges, we participated in the NeurIPS LLM Efficiency Challenge, aiming to fine-tune a foundation model within stringent constraints. Our focus was the LLaMa2 70 billion model, optimized on a single A100 40GB GPU within a 24-hour limit. Our methodology hinged on a custom dataset, carefully assembled from diverse open-source resources and benchmark tests, aligned with the challenge's open-source ethos. Our approach leveraged Quantized-Low Rank Adaptation (QLoRA) Fine tuning, integrated with advanced attention mechanisms like Flash Attention 2. We experimented with various configurations of the LoRA technique, optimizing the balance between computational efficiency and model accuracy. Our fine-tuning strategy was underpinned by the creation and iterative testing of multiple dataset compositions, leading to the selection of a version that demonstrated robust performance across diverse tasks and benchmarks. The culmination of our efforts was an efficiently fine-tuned LLaMa2 70B model that operated within the constraints of a single GPU, showcasing not only a significant reduction in resource utilization but also high accuracy across a range of QA benchmarks. Our study serves as a testament to the feasibility of optimizing large-scale models in resource-constrained environments, emphasizing the potential of LLMs in real-world applications.