Energy and accuracy trade-offs in medical question answering with large language models
Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks
Computation and LanguageArtificial Intelligence
Summary
Large language models are being used to answer medical questions, but using bigger models can use a lot of energy, which has environmental impacts. The authors studied 13 different models with varying sizes to see how much energy they consume versus how accurate their answers are. They found that just looking at accuracy or energy separately is not enough; instead, it’s important to consider both together to choose the best model. This helps people pick more effective and environmentally friendly models for medical question answering.
What this means in practice
- •For healthcare ai teams: select energy-efficient language models for medical question answering that balance accuracy and environmental impact based on cost-benefit trade-offs
- •For data center operators: estimate and manage the carbon footprint of running large language models on medical QA tasks to optimize sustainability efforts
Authors
Viju Sudhi, Giulio Billi, Andreea Scrob, Regina Stodden, Philipp Cimiano
Abstract
Large Language Models (LLMs) are increasingly used in medical question answering (QA) systems. While recent works aim at incrementally improving the downstream task accuracy, an empirical estimation of the models' energy consumption and the associated carbon footprints is often overlooked. In this study, we demonstrate how practitioners can estimate energy consumption and arrive at reliable emission estimates for medical QA tasks available in the OpenMedical-LLM leaderboard by investigating 13 LLMs from 3 model families ranging from 270M to 14B parameters. We discuss how one can critically interpret energy versus accuracy trade-offs for a set of models for a given task. With this work, we underline that reporting disjoint energy and accuracy metrics alone does not suffice for careful trade-off assessments, but it also demands systematic construction and interpretation of joint trade-off metrics. We envisage such studies aiding practitioners in adopting better and more efficient models for their task and helping them become active participants in building sustainable LLM pipelines.