betaFeedback

WorksComputer Science

Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks

Abstract

Large Language Models (LLMs) are increasingly used in medical question answering (QA) systems. While recent works aim at incrementally improving the downstream task accuracy, an empirical estimation of the models' energy consumption and the associated carbon footprints is often overlooked. In this study, we demonstrate how practitioners can estimate energy consumption and arrive at reliable emission estimates for medical QA tasks available in the OpenMedical-LLM leaderboard by investigating 13 LLMs from 3 model families ranging from 270M to 14B parameters. We discuss how one can critically interpret energy versus accuracy trade-offs for a set of models for a given task. With this work, we underline that reporting disjoint energy and accuracy metrics alone does not suffice for careful trade-off assessments, but it also demands systematic construction and interpretation of joint trade-off metrics. We envisage such studies aiding practitioners in adopting better and more efficient models for their task and helping them become active participants in building sustainable LLM pipelines.

Introduction

Large Language Models (LLMs) are increasingly used for a wide variety of tasks in different domains, including safety-critical ones like healthcare and medicine [1, 2, 3, 4]. Recent studies show how frontier language models achieve passing scores on United States Medical Licensing Examination (USMLE) questions without specialized medical training and how they are able to provide clinically reasonable answers to a wide number of difficult medical scenarios [5, 6]. Attesting to this, there has been an increase in the number of domain-specific models, including Med-PaLM [7], MedGemma [8], BioMistral [9] and BioMedLM [10], among others, which are further fine-tuned on medical and clinical data. These models report improved performance on different medical benchmarks than their corresponding general-purpose counterparts.

While newer models are continuously (re-)trained to reach better performance on different downstream tasks, reporting energy costs and carbon footprints associated with them is often overlooked. Although LLM training costs are a one-time spend, the more important problem arises with the soaring inference demands from these models once they are deployed in real-time systems. Recent estimates suggest that this rising demand for AI inference would account for around 652 TWh of energy costs in 2030, compared to 23 TWh in 2022 [11].

The rising concerns with LLM energy demands have opened doors for “efficiency-aware evaluation” of these models. Towards this goal, several tools have been developed to measure carbon footprints of LLMs, both at training and inference stages [12]. These include CodeCarbon [13], and CarbonTracker [14], among others. Despite researchers acknowledging that reporting energy costs adds more transparency

Workshop on GREEN-AI: Grounding Research on Energy- and resource-Efficient Networks for sustainable AI, Co-Located with CIKM 2025, November 08, 2026, Rome, Italy

✉ [email protected] (V. Sudhi); [email protected] (G. Billi); [email protected] (A. Scrob); [email protected] (R. Stodden); [email protected] (P. Cimiano)

© 2026 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). to their pipelines, we argue that merely reporting disjoint energy-accuracy metrics does not suffice.

We see this as a more convoluted problem both from an environmental and technological perspective. On one hand, while we acknowledge LLMs lead to escalated energy consumption, assessing models purely based on efficiency metrics is impractical. At the same time, the growing trend of improving models aimed only at better accuracy in benchmarks does not offer a convincing alternative either. In addition to this, recent studies [15, 16, 17] show that model size does not always hold a linear correlation with model performance, which begs the question of why one should use a larger model with higher costs despite subpar performance. This motivates us to position our argument that energy-accuracy measurements of LLMs should go hand in hand, complementing one another. Practitioners should be aware of how to measure and interpret the trade-offs between these parameters, finally enabling them to make better and informed decisions about the choice of models for their task. In this context, we acknowledge that choosing the right model amidst the plethora of available models is itself “a sustainability decision” [11] and to support this decision, we should treat the problem as a “multi-criteria decision making problem” [18] jointly optimizing the energy-accuracy metrics.

In this study1, using the example of Medical QA tasks in the OpenMedical-LLM leaderboard [19], we demonstrate how practitioners can monitor the energy consumption during inference and arrive at reliable CO2 emission metrics using the package CodeCarbon [13]. We then present how one can comprehend the energy versus accuracy trade-offs given a set of models on the given task. To this end, we study 13 models from 3 different model families (Qwen-3 [20], Llama-3 [21], and Gemma-2/3 [22, 23]) within the range of 270M to 14B parameters. We conduct in-context learning with zero-shot settings and further fine-tune the models with LoRA parameter-efficient fine-tuning to study the effect of fine-tuning both on accuracy and energy consumption. We interpret the energy-accuracy trade-offs from these experiments with the help of a Pareto-optimal frontier, discuss the issues with such analyses, and extend our study to a more recent distance-based rating scheme, CIRC [18], to help interpret the trade-offs clearly and distinguish the models better.

Related Work

Over the recent years, efficiency-aware evaluation of models has gained particular traction. There are notable works measuring energy-accuracy trade-offs for different tasks and domains [24, 25, 26, 27]. Maliakel et al. Maliakel et al. [28] investigate the energy-accuracy trade-offs in different tasks, including question answering, commonsense reasoning, and factual generation, while Alizadeh et al. Alizadeh et al. [29] conducted an extended study considering software development tasks, analyzing the trade-offs with Pareto frontiers. In contrast, Mehditabar et al. Mehditabar et al. [18] argued that this is not the optimal approach, proposing more interpretable metrics to assess the trade-offs, offering better discriminatory levels.

A wide number of frontier models are trained on medical and clinical data to improve their performance on medical domain-specific tasks [7, 8, 9, 10]. Recent works explore parameter-efficient fine-tuning of models on medical data [1] and compare them with full-weight fine-tuned models [30]. Additionally, works investigate how quantized models can perform on medical domain tasks [31]. While the techniques presented in these works help reduce the carbon footprint of models, there has been very little systematic study in the medical domain to measure the energy-accuracy trade-offs and arrive at interpretable trade-off metrics.

To the best of our knowledge, this is the first work that attempts to study the energy-accuracy trade-offs on medical QA tasks. Unlike other works, we extend our study to show how the trade-offs can be interpreted and critically analyzed for a more informed model selection procedure.

Methodology

Energy and Emission Estimation

Energy Estimation: For real-time monitoring of energy consumption, we resort to the widely used CodeCarbon package [13]. CodeCarbon reads hardware power every 15s and integrates it over time to get the final energy consumption. This is computed as the product of the total power and the runtime of the process, given by: Energy=(PGPU+PCPU+PRAM)∗RuntimeEnergy = (P_{GPU} + P_{CPU} + P_{RAM}) * Runtime.

Here, PGPUP_{GPU} is measured directly from the NVIDIA NVML reports. PCPUP_{CPU} is measured by a power model driven by the CPU load, given as TDP(0.1+0.9load3)TDP(0.1 + 0.9load^3), where loadload is the fraction of CPU capacity in use within the range of (0,1)(0, 1) and TDPTDP is the processor’s rated power. We set TDPTDP to 450W, matching the CPUs in our infrastructure at 225W each. CodeCarbon estimates PRAMP_{RAM} from the amount of installed memory and measures RuntimeRuntime as the time taken for inference.

Emission Estimation: CO2 emissions are computed as the product of energy and the carbon intensity of the local power grid. For example, in Germany, Carbon Intensity amounts to around 328 g CO2eq/kWh according to the 2025 statistics [32]. This is further scaled by a Power Usage Effectiveness factor (PUE) that accounts for data center overheads such as cooling. The total emission is given by: Emission = Energy∗CarbonIntensity∗PUEEnergy * Carbon Intensity * PUE.

Energy versus Accuracy Trade-off

To study the energy versus accuracy trade-offs, we consider two objectives: (i) minimize the energy consumption, and (ii) maximize the task performance. Hence, the ideal model should yield the maximum accuracy while consuming the least amount of energy.

Pareto-optimal Frontier: Alizadeh et al. [29] analyze the relationship between energy consumption and accuracy of models with the help of Pareto fronts. These frontiers help in analyzing the balance between the defined objectives. However, Pareto fronts mark only the dominant points, leaving little to no interpretation for the rest of the non-dominant points. This hinders practitioners in distinguishing the contributions from the non-dominant points, limiting their options to the ones that lie on the Pareto front. For example, consider the exemplary Pareto front illustrated in Figure 1. In this case, Objective #1 should be minimized, and Objective #2 should be maximized. The Pareto front shows the dominant points A, B, and C. While the points D and E are non-dominant, they are still close to B. The frontier gives no reasonable interpretation to distinguish between the non-dominant points, D and E, from one another and from the nearer dominant point, B. Consequently, one could only select from among the dominant points with the help of the Pareto front.

Figure 1. Example Pareto-optimal Frontier.

Joint metrics: Mehditabar et al. [18] argue that a combination of both task accuracy metrics and energy consumption metrics is required for a more holistic evaluation of real-world deployment constraints. Towards this, they introduced joint energy-accuracy metrics considering the problem as a Multi-Criteria Decision Making problem. One of their proposed rating schemes, “Concentric Incremental Rating Circles” (CIRC), is based on Euclidean distance that quantifies each point by its distance to the most optimal point. This results in a clearer interpretation of all the points in the study and a clear distinction between one another. They use a five-point scale synonymous with the existing power ratings. We detail the computation of the CIRC ratings in Appendix § 15.

Experiments

Dataset: To study the energy versus accuracy trade-off of LLMs in medical QA tasks, we leveraged the publicly available Open Medical-LLM Leaderboard [19]. The leaderboard is based on different tasks including MedQA, MedMCQA, PubMedQA, and MMLU medical subsets. Together, these datasets provide a common evaluation setting for measuring medical knowledge and reasoning.

Models: We ran the experiments on the following 13 models from 3 different model families. The models vary from 270M to 14B in parameter count.

  • Llama family [21]: Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B

  • Gemma family [22, 23]: gemma-2-2b, gemma-2-9b, gemma-3-270m, gemma-3-1b-pt, gemma-3-4b-pt

  • Qwen family [20]: qwen3-0.6b, qwen3-1.7b, qwen3-4b, qwen3-8b, qwen3-14b

Pipeline: We conducted our experiments as follows: (i) Firstly, we ran in-context learning experiments with zero-shot prompts on the pre-trained models; (ii) Further, we performed LoRA parameter-efficient fine-tuning on the models for a maximum of 5 epochs with early stopping. The evaluation step is then repeated on the fine-tuned models.

Evaluation: To evaluate the pre-trained models under a zero-shot setting and the LoRA fine-tuned models, we relied on the evaluation package lm-eval-harness [33]. The package allows us to measure the model performance on the tasks and evaluate it against the gold references. In addition, we also estimated the energy consumption and CO2 emissions at the inference stage.

Hardware: All training and evaluation runs are executed on a single NVIDIA A40 GPU with 48GB of memory. The node is equipped with two AMD EPYC 7713 CPUs.

More details on the datasets and evaluation metrics are provided in Appendix § 11. Fine-tuning hyper-parameters are detailed in Appendix § 12.

Results

In Table 1, we show the performance of all the models, pre-trained and fine-tuned, on the medical QA tasks. We also report the energy consumption and emissions per experiment. It is evident from the table that the largest model qwen3-14b yields the highest accuracy among all the models. This is true both for the pre-trained model and the fine-tuned counterpart. This model also results in the highest energy consumption and consequently the highest CO2 emission. The smallest model in the list, gemma-3-270m, yields the least accuracy on the task. Surprisingly, a slightly larger 1B parameter model, llama-3.2-1b, however, consumes the least energy.

ModelAccuracy (↑)Energy (kWh) (↓)CO2 Emission (kg) (↓)
Pre-trainedFine-tunedPre-trainedFine-tunedPre-trainedFine-tuned
qwen3-0.6b48.73 ± 3.3349.52 ± 3.330.0220.0310.0080.012
qwen3-1.7b54.83 ± 3.2962.23 ± 3.180.0230.0360.0090.014
qwen3-4b68.56 ± 3.0874.21 ± 2.750.040.0540.0150.02
qwen3-8b67.13 ± 3.0076.89 ± 2.650.0660.0640.0250.024
qwen3-14b74.73 ± 2.7481.03 ± 2.450.0770.0890.0290.034
gemma-2-2b51.18 ± 3.3056.21 ± 3.290.0260.0420.010.016
gemma-2-9b69.06 ± 2.8975.43 ± 2.690.060.0820.0230.031
gemma-3-270m24.89 ± 2.8527.33 ± 2.910.0150.0240.0060.009
gemma-3-1b-pt27.30 ± 2.7930.17 ± 2.910.0230.0360.0090.014
gemma-3-4b-pt55.15 ± 3.2763.98 ± 3.150.0460.0590.0180.023
llama-3.2-1b40.69 ± 3.2843.80 ± 3.330.0120.0210.0050.008
llama-3.2-3b60.87 ± 3.1961.33 ± 3.220.0220.0380.0090.014
llama-3.1-8b69.59 ± 2.9370.68 ± 2.890.0380.060.0150.023

Table 1. Accuracy, Energy Consumption and CO2 Emission Results

Effect of model size: As shown in Figure 2, for the medical QA tasks, there is a mostly linear relation between the accuracy and the model size. However, this is not the case when we compare emissions (energy) against the model size. Though smaller models generally emit less CO2 than the larger models, models belonging to the Llama family show a different trend. For example, a 1B parameter model llama-3.2-1b consumes less energy than a 600M parameter model qwen-3-0.6b. Similarly, an 8B parameter model llama-3.1-8b consumes the same amount of energy as a 4B parameter model qwen3-4b.

Figure 2. Accuracy and CO2 Emissions versus Model Size

Low accuracy gains at the cost of high energy consumption: As illustrated in Table 2, we compute the relative performance gain and relative energy costs of the fine-tuned models per model family. For example, qwen3-0.6b yields an accuracy of 49.5 and qwen3-14b yields an accuracy of 81.0. This results in a relative accuracy gain of (81.0 − 49.5)/49.5 = +64% for the Qwen3 family; however, at the cost of an 187% increase in the energy cost. This is also the case for all the model families, where the improvement in task performance is only marginal when compared to the corresponding increase in energy consumption (and likewise, the increase in CO2 emission costs).

Model FamilyRelative Gain (Accuracy)Relative Costs (Energy)
Qwen3+64%+187%
Gemma2+34%+95%
Gemma3+134%+146%
Llama3+61%+186%

Table 2. Comparison of relative accuracy gains versus relative energy costs

Interpreting the Pareto-optimal front: In Figure 3 (left), we find that 9 models (out of 26) fall on the Pareto-optimal front. This includes 5 fine-tuned variants and 4 pre-trained ones. Neither the pre-trained nor the fine-tuned Gemma models are on the frontier. The plot shows diminishing returns in accuracy for the larger models, showing that the improvements in accuracy are only marginal but at the cost of relatively higher energy consumption. However, it is difficult to analyze how different qwen3-1.7b (PT) is compared with llama-3.2-3b (PT). This underlines the argument that although the Pareto frontier helps us to interpret the dominating points, it leaves no interpretation for the non-dominant points, consequently reducing the number of options to finally choose from.

Figure 3. Interpreting Energy-Accuracy Trade-offs with Pareto-optimal frontier (left) and CIRC rating (right).

Interpreting the CIRC ratings: CIRC ratings illustrated in Figure 3 (right) give a clearer interpretation of the energy-accuracy trade-offs. Here, energy efficiency is given by the complement of the measured energy consumption. The rating scheme assumes no model has reached the most optimal solution, leaving no model in the Very Strong circle. In the Strong circle, we find llama-3.2-3b (PT) and llama-3.1-8b (PT), the least distant from the optimal solution. Compared to the Pareto front, the extreme dominant points llama-3.2-1b (PT) and qwen3-14b (FT) belong to the Moderate and the Weak circles here, respectively. qwen3-4b (PT) is not a dominant point in the Pareto front, but is rated as Strong just like the dominant point llama-3.1-8b (PT). This shows how one can make more informed choices with CIRC ratings offering a larger pool of choices, unlike Pareto-front marking only the dominant points.

Discussion

Our empirical evaluation of 13 models across three families demonstrates that relying solely on absolute accuracy or isolated efficiency metrics is fundamentally insufficient for selecting optimal LLMs in medical QA tasks. While larger models predictably achieve higher accuracy, their marginal performance gains often incur a disproportionate environmental cost. For instance, while qwen3-14b (FT) reaches a peak accuracy of 81.03%, it requires 0.089 kWh and emits 0.034 kg of CO₂—an 187% increase in energy overhead for only a 64% relative accuracy gain over smaller baselines. Depending on operational constraints, practitioners face distinct deployment trade-offs: models like llama-3.1-8b (PT) offer balanced performance under moderate resource limits, whereas smaller fine-tuned models such as qwen3-1.7b (FT) serve as excellent high-throughput or edge alternatives. A key takeaway of our work is the limitation of traditional Pareto-optimal frontiers, which discard non-dominant models and offer no guidance for intermediate choices. By employing distance-based joint metrics like CIRC ratings, practitioners gain a more nuanced, multi-tier evaluation scale. This approach successfully identifies high-value, non-dominant options—such as qwen3-4b (PT) achieving a Strong rating comparable to larger dominant architectures.

Finally, while our primary evaluation focuses on inference-stage footprints, integrating fine-tuning energy into a life-cycle framework remains critical. Although parameter-efficient fine-tuning (LoRA) enhances medical QA performance, training overhead must be amortized over expected deployment lifetimes. Future work will explore hardware-level quantization and multimodal medical pipelines to further optimize sustainable AI deployments.

Conclusion

In this work, we demonstrated how to critically study the energy-accuracy trade-offs of 13 LLMs on medical QA tasks. We demonstrate how practitioners can measure the energy consumed in the inference process and relate this to the CO2 emissions per model per task. With our thorough evaluation, we show how larger models attain higher task performance, but at the cost of higher relative carbon footprints. We show how one can analyze the trends with the help of Pareto-optimal frontiers, discuss the problems with these settings, and study a distance-based rating scheme that helps to clearly distinguish the trade-offs of individual models.

Acknowledgments

This work was funded by the European Union and the German state of North Rhine-Westphalia within the following project of the European Regional Development Fund (EFRE-20801074): LLM4KMU - Optimized use of open-source large language models in SMEs.

Limitations

Energy estimates can vary based on the hardware specifications and inference optimizations. In this study, we resorted to a specific hardware resource for training and evaluation, with no further optimizations. In future work, gathering energy estimates from different hardware types at different optimization stages can help build comprehensive insights. For comparability in the final assessment of base models and fine-tuned models, we avoid the energy costs associated with fine-tuning. However, we note that these costs should be accommodated to help practitioners better distinguish the models suited for their task. This demands more granular reporting, and we defer this to future work.

Declaration on Generative AI

During the preparation of this work, the authors used Grammarly to perform a grammar and spelling check. The authors also used Google Gemini for coding assistance to create the figures for the paper and minor rephrasing. After using these services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.

Datasets

Fine-tuning. For fine-tuning, we leveraged the train splits of all datasets in the OpenMedical-LLM leaderboard, except the MMLU subsets. To balance the number of samples per task, (i) we over-sample the training split of PubMedQA from 450 to 900 samples; (ii) we reduce the large MedMCQA dataset through stratified sub-sampling based on its 21 original medical domains, extracting a targeted subset of 20000 records. The details are presented in Table 3.

DatasetType# Train Samples# Validation Samples# Test SamplesEvaluation Metric
MedQAOpen MCQA1017812721273log-likelihood over A / B / C / D
MedMCQAOpen MCQA1800041832000log-likelihood over A / B / C / D
PubMedQAClosed MCQA90050500log-likelihood over yes / no / maybe
MMLU SubsetsOpen MCQA--1089exact match on the option word
Total2907855054862

Table 3. Datasets included in Open Medical-LLM Leaderboard

Evaluation. For evaluation, we use the test samples from all the datasets, including the MMLU subsets. This accounts for 3773 samples. For all datasets except MMLU subsets, we leverage the joint log-probability of the expected options. For MMLU subsets, the model response is evaluated based on exact match with the expected option.

Hyper-parameters

For LoRA fine-tuning of the models, we keep the base model weights in bfloat16 and update only the additional low-rank adapters. We compute the loss only on the completion tokens, masking the input tokens. The effective batch size is kept at 32 across all models. All runs use a learning rate of 10−410^{-4} with 10 warm-up steps. Training is allowed to run for up to 5 epochs, but the model is evaluated periodically on the validation split during training, and the validation loss is monitored through early stopping. Training stops when the validation loss does not improve by at least 10−510^{-5} for 5 consecutive evaluations. This serves two purposes: it limits overfitting on the relatively small supervised datasets, and it avoids spending training energy on epochs that no longer improve generalization.

Energy consumption and CO2 emissions during fine-tuning

Table 4 details the energy consumption and carbon emissions associated with fine-tuning the models. Resource requirements scale with model size and architecture; the lightweight gemma-3-270m incurs the lowest footprint with a runtime of 754.59 seconds, consuming 0.069 kWh and emitting 0.026 kg of CO2CO_2. Conversely, the larger qwen3-14b model demands the highest resources, requiring 27,956.03 seconds, 2.752 kWh of energy, and resulting in 1.048 kg of CO2CO_2 emissions.

ModelRuntime (s)Energy (kWh)CO2 Emission (kg)
qwen3-0.6b2767.6980.2550.097
qwen3-1.7b3704.0890.3530.134
qwen3-4b10920.6111.0710.408
qwen3-8b20619.0911.9860.756
qwen3-14b27956.0342.7521.048
gemma-2-2b6490.9050.6170.235
gemma-2-9b14251.3331.3670.521
gemma-3-270m754.5910.0690.026
gemma-3-1b-pt2862.3670.2570.098
gemma-3-4b-pt8505.4310.8200.312
llama-3.2-1b2140.4420.2050.078
llama-3.2-3b4559.0370.4500.172
llama-3.1-8b10842.3951.0720.408

Table 4. Estimates of energy consumption and CO2 emission for fine-tuning runs

Extended Evaluation Results

Table 5 and Table 6 present the extended model evaluation results across the different datasets in the Open Medical-LLM Leaderboard on the pre-trained and fine-tuned models, respectively. Overall accuracy is averaged across the individual datasets, and the corresponding standard error is reported as the root-mean-square (RMS). In most of the cases, fine-tuning yields performance improvements across models and datasets. Larger models experience the most pronounced gains; for example, qwen3-14b model increases its average accuracy from 74.73% to 81.03%. Dataset-specific analysis shows that while models maintain strong baseline performance on PubMedQA, more challenging datasets like MedMCQA, MedQA, and MMLU Subsets benefit substantially from fine-tuning, with qwen3-14b achieving peak scores of 68.04%, 72.27%, and 84.86%, respectively.

ModelPubMedQAMedMCQAMedQAMMLU SubsetsAvg. Accuracy
qwen3-0.6b70.20±2.0570.20 \pm2.0538.82±0.7538.82 \pm0.7538.57±1.3638.57 \pm1.3648.50±3.8848.50 \pm3.8848.73±3.3348.73 \pm3.33
qwen3-1.7b72.60±2.0072.60 \pm2.0048.10±0.7748.10 \pm0.7748.08±1.4048.08 \pm1.4054.11±3.8354.11 \pm3.8354.83±3.2954.83 \pm3.29
qwen3-4b75.80±1.9275.80 \pm1.9257.09±0.7757.09 \pm0.7762.06±1.3662.06 \pm1.3670.34±3.5670.34 \pm3.5668.56±3.0868.56 \pm3.08
qwen3-8b76.20±1.9176.20 \pm1.9162.47±0.7562.47 \pm0.7565.83±1.3365.83 \pm1.3366.62±3.4866.62 \pm3.4867.13±3.0067.13 \pm3.00
qwen3-14b76.40±1.9076.40 \pm1.9066.34±0.7366.34 \pm0.7371.25±1.2771.25 \pm1.2776.43±3.1876.43 \pm3.1874.73±2.7474.73 \pm2.74
gemma-2-2b74.00±1.9674.00 \pm1.9640.86±0.7640.86 \pm0.7635.04±1.3435.04 \pm1.3451.78±3.8551.78 \pm3.8551.18±3.3051.18 \pm3.30
gemma-2-9b78.60±1.8478.60 \pm1.8457.92±0.7657.92 \pm0.7660.02±1.3760.02 \pm1.3770.84±3.3770.84 \pm3.3769.06±2.8969.06 \pm2.89
gemma-3-270m37.60±2.1737.60 \pm2.1725.05±0.6725.05 \pm0.6722.86±1.1822.86 \pm1.1823.08±3.2823.08 \pm3.2824.89±2.8524.89 \pm2.85
gemma-3-1b-pt64.20±2.1564.20 \pm2.1531.05±0.7231.05 \pm0.7224.98±1.2124.98 \pm1.2120.92±3.1820.92 \pm3.1827.30±2.7927.30 \pm2.79
gemma-3-4b-pt72.60±2.0072.60 \pm2.0046.31±0.7746.31 \pm0.7745.64±1.4045.64 \pm1.4055.30±3.8155.30 \pm3.8155.15±3.2755.15 \pm3.27
llama-3.2-1b57.80±2.2157.80 \pm2.2137.03±0.7537.03 \pm0.7535.35±1.3435.35 \pm1.3439.34±3.8039.34 \pm3.8040.69±3.2840.69 \pm3.28
llama-3.2-3b73.00±1.9973.00 \pm1.9949.44±0.7749.44 \pm0.7751.77±1.4051.77 \pm1.4062.27±3.7262.27 \pm3.7260.87±3.1960.87 \pm3.19
llama-3.1-8b75.60±1.9275.60 \pm1.9256.28±0.7756.28 \pm0.7759.94±1.3759.94 \pm1.3772.41±3.4172.41 \pm3.4169.59±2.9369.59 \pm2.93

Table 5. Evaluation results of pre-trained models

ModelPubMedQAMedMCQAMedQAMMLU SubsetsAvg. Accuracy
qwen3-0.6b59.40±2.2059.40 \pm2.2043.08±0.7743.08 \pm0.7738.96±1.3738.96 \pm1.3750.71±3.8650.71 \pm3.8649.52±3.3349.52 \pm3.33
qwen3-1.7b72.40±2.0072.40 \pm2.0050.01±0.7750.01 \pm0.7748.78±1.4048.78 \pm1.4064.82±3.7064.82 \pm3.7062.23±3.1862.23 \pm3.18
qwen3-4b77.80±1.8677.80 \pm1.8660.29±0.7660.29 \pm0.7663.08±1.3563.08 \pm1.3577.79±3.1677.79 \pm3.1674.21±2.7574.21 \pm2.75
qwen3-8b79.40±1.8179.40 \pm1.8164.36±0.7464.36 \pm0.7468.50±1.3068.50 \pm1.3079.96±3.0679.96 \pm3.0676.89±2.6576.89 \pm2.65
qwen3-14b79.80±1.8079.80 \pm1.8068.04±0.7268.04 \pm0.7272.27±1.2672.27 \pm1.2684.86±2.7884.86 \pm2.7881.03±2.4581.03 \pm2.45
gemma-2-2b75.60±1.9275.60 \pm1.9244.23±0.7744.23 \pm0.7741.08±1.3841.08 \pm1.3857.50±3.8557.50 \pm3.8556.21±3.2956.21 \pm3.29
gemma-2-9b80.40±1.7880.40 \pm1.7859.67±0.7659.67 \pm0.7663.94±1.3563.94 \pm1.3579.14±3.1179.14 \pm3.1175.43±2.6975.43 \pm2.69
gemma-3-270m50.00±2.2450.00 \pm2.2426.94±0.6926.94 \pm0.6924.82±1.2124.82 \pm1.2124.04±3.3424.04 \pm3.3427.33±2.9127.33 \pm2.91
gemma-3-1b-pt70.60±2.0470.60 \pm2.0432.58±0.7232.58 \pm0.7227.18±1.2527.18 \pm1.2523.52±3.3323.52 \pm3.3330.17±2.9130.17 \pm2.91
gemma-3-4b-pt77.20±1.8877.20 \pm1.8849.10±0.7749.10 \pm0.7752.95±1.4052.95 \pm1.4066.10±3.6866.10 \pm3.6863.98±3.1563.98 \pm3.15
llama-3.2-1b62.60±2.1762.60 \pm2.1734.76±0.7434.76 \pm0.7433.07±1.3233.07 \pm1.3243.97±3.8743.97 \pm3.8743.80±3.3343.80 \pm3.33
llama-3.2-3b76.80±1.8976.80 \pm1.8947.98±0.7747.98 \pm0.7750.12±1.4050.12 \pm1.4062.84±3.7562.84 \pm3.7561.33±3.2261.33 \pm3.22
llama-3.1-8b78.80±1.8378.80 \pm1.8356.49±0.7756.49 \pm0.7760.33±1.3760.33 \pm1.3773.41±3.3773.41 \pm3.3770.68±2.8970.68 \pm2.89

Table 6. Evaluation results of fine-tuned models

Extended CIRC Ratings

For a given task, CIRC ranks the models based on their Euclidean distance to the optimal solution. To this end, firstly, we normalize the accuracy (AccAcc) and energy consumption (EnergyEnergy) values. Further, we calculate “energy efficiency” as the complement of “energy consumed” as: Effnorm=1−EnergynormEff_{norm} = 1 - Energy_{norm}. This helps us employ similar trends (higher the better) in both energy-accuracy axes. Finally, we compute the distance as (1−Accnorm)2+(1−Effnorm)2\sqrt{(1 - Acc_{norm})^2 + (1 - Eff_{norm})^2} for each model from the optimal solution.

To visualize this, we construct five concentric circles, each representing a CIRC rating, centered at the point (1,1)(1, 1). The point represents the most optimal solution with maximum accuracy and maximum energy efficiency. These concentric circles have radii increasing in equal increments, starting from 25\frac{\sqrt{2}}{5} (representing a rating of 5) and ending at 2\sqrt{2} (representing a rating of 1). For more details of the algorithm, we defer to the original work [18]. Table 7 presents the CIRC ratings for the 13 pre-trained and 13 fine-tuned models in our experiments.

ModelModeAccAccAccnormAcc_{norm}EnergyEnergyEffnormEff_{norm}DistanceRating
qwen3-0.6bPre-trained48.70.4240.0220.8710.5903
qwen3-0.6bFine-tuned49.50.4390.0310.7570.6123
qwen3-1.7bPre-trained54.80.5330.0230.8580.4884
qwen3-1.7bFine-tuned62.20.6650.0360.6860.4594
qwen3-4bPre-trained68.60.7790.0400.6370.4254
qwen3-4bFine-tuned74.20.8790.0540.4600.5544
qwen3-8bPre-trained67.10.7520.0660.2980.7443
qwen3-8bFine-tuned76.90.9270.0640.3260.6783
qwen3-14bPre-trained74.70.8880.0770.1630.8453
qwen3-14bFine-tuned81.01.0000.0890.0001.0002
gemma-2-2bPre-trained51.20.4690.0260.8220.5604
gemma-2-2bFine-tuned56.20.5580.0420.6150.5863
gemma-2-9bPre-trained69.10.7880.0600.3740.6613
gemma-2-9bFine-tuned75.40.9000.0820.0890.9172
gemma-3-270mPre-trained24.90.0000.0150.9611.0012
gemma-3-270mFine-tuned27.30.0430.0240.8440.9702
gemma-3-1b-ptPre-trained27.30.0430.0230.8630.9672
gemma-3-1b-ptFine-tuned30.20.0940.0360.6950.9552
gemma-3-4b-ptPre-trained55.20.5400.0460.5600.6373
gemma-3-4b-ptFine-tuned64.00.6970.0600.3850.6853
llama-3.2-1bPre-trained40.70.2820.0121.0000.7183
llama-3.2-1bFine-tuned43.80.3370.0210.8880.6723
llama-3.2-3bPre-trained60.90.6420.0220.8680.3824
llama-3.2-3bFine-tuned61.30.6490.0380.6670.4844
llama-3.1-8bPre-trained69.60.7970.0380.6630.3944
llama-3.1-8bFine-tuned70.70.8160.0600.3760.6503

Table 7. CIRC Ratings

Sensitivity Analysis. CIRC relies on min-max normalization to map accuracy (AccAcc) and energy efficiency (EffEff) to a [0,1][0, 1] scale and applies equal weighting in the distance computation. To evaluate the sensitivity of our findings to these methodological choices, we analyze variations along two dimensions:

  • Normalization: While min-max normalization preserves relative distances, it can be sensitive

to outliers (e.g., extreme models like gemma-3-270m or qwen3-14b). Substituting min-max scaling with rank-based percentile normalization or z-score scaling yields stable top-tier rankings. Specifically, models identified in the Strong rating tier (such as llama-3.2-3b (PT) and llama-3.1-8b (PT)) remain within the top performance cluster, as their trade-off profiles consistently balance high accuracy without excessive energy consumption.

  • Weighting: Equal weighting assumes accuracy and energy efficiency carry equal importance. In real-world medical deployments, practitioner priorities can skew. If energy efficiency is weighted more heavily (e.g., weight of 0.7 on energy and 0.3 on accuracy), smaller efficient models like qwen3-1.7b (FT) shift inward to higher rating circles. Conversely, prioritizing clinical accuracy (e.g., 0.7 weight on accuracy) naturally pulls high-performing large models like qwen3-8b (FT) closer to the optimal center. However, the intermediate tiers remain largely stable, confirming that CIRC provides a resilient multi-criteria categorization without being overly brittle to moderate weight adjustments.

References

  1. [1]M. A. A. El-Enen, S. S. Ismail, T. M. Nazmy, Med-LLaMA3: Advancing Medical Question-Answering Through Parameter-Efficient Fine-Tuning of Large Language Models, Applied Sciences 16 (2026) 6158. URL: https://www.mdpi.com/2076-3417/16/12/6158. doi:10.3390/app16126158.
  2. [2]D. Wang, S. Zhang, Large language models in medical and healthcare fields: applications, advances, and challenges, Artificial Intelligence Review 57 (2024) 299. URL: https://doi.org/10.1007/s10462-024-10921-0. doi:10.1007/s10462-024-10921-0.
  3. [3]S. Shool, S. Adimi, R. Saboori Amleshi, E. Bitaraf, R. Golpira, M. Tara, A systematic review of large language model (LLM) evaluations in clinical medicine, BMC Medical Informatics and Decision Making 25 (2025) 117. URL: https://doi.org/10.1186/s12911-025-02954-4. doi:10.1186/s12911-025-02954-4.
  4. [4]L. Liu, X. Yang, J. Lei, Y. Shen, J. Wang, P. Wei, Z. Chu, Z. Qin, K. Ren, A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions, IEEE Transactions on Knowledge and Data Engineering (2026) 1–25. URL: https://ieeexplore.ieee.org/document/11594945/. doi:10.1109/TKDE.2026.3709941.DOI
  5. [5]H. Nori, N. King, S. M. Kinyon, D. Carignan, E. Horvitz, Capabilities of GPT-4 on Medical Challenge Problems, 2023. URL: http://arxiv.org/abs/2303.13375. doi:10.48550/arXiv.2303.13375, arXiv:2303.13375 [cs.CL].
  6. [6]A. Gilson, C. W. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, D. Chartash, How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment, JMIR medical education 9 (2023) e45312. doi:10.2196/45312.DOI
  7. [7]K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, D. Neal, Q. M. Rashid, M. Schaekermann, A. Wang, D. Dash, J. H. Chen, N. H. Shah, S. Lachgar, P. A. Mansfield, S. Prakash, B. Green, E. Dominowska, B. Aguera y Arcas, N. Tomasev, Y. Liu, R. Wong, C. Semturs, S. S. Mahdavi, J. K. Barral, D. R. Webster, G. S. Corrado, Y. Matias, S. Azizi, A. Karthikesalingam, V. Natarajan, Toward expert-level medical question answering with large language models, Nature Medicine 31 (2025) 943–950. URL: https://www.nature.com/articles/s41591-024-03423-7. doi:10.1038/s41591-024-03423-7.DOI
  8. [8]A. Sellergren, S. Kazemzadeh, T. Jaroensri, A. Kiraly, M. Traverse, T. Kohlberger, S. Xu, F. Jamil, C. Hughes, C. Lau, J. Chen, F. Mahvar, L. Yatziv, T. Chen, B. Sterling, S. A. Baby, S. M. Baby, J. Lai, S. Schmidgall, L. Yang, K. Chen, P. Bjornsson, S. Reddy, R. Brush, K. Philbrick, M. Asiedu, I. Mezerreg, H. Hu, H. Yang, R. Tiwari, S. Jansen, P. Singh, Y. Liu, S. Azizi, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Riviere, L. Rouillard, T. Mesnard, G. Cideron, J.-b. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Buchatskaya, J.-B. Alayrac, D. Lepikhin, V. Feinberg, S. Borgeaud, A. Andreev, C. Hardin, R. Dadashi, L. Hussenot, A. Joulin, O. Bachem, Y. Matias, K. Chou, A. Hassidim, K. Goel, C. Farabet, J. Barral, T. Warkentin, J. Shlens, D. Fleet, V. Cotruta, O. Sanseviero, G. Martins, P. Kirk, A. Rao, S. Shetty, D. F. Steiner, C. Kirmizibayrak, R. Pilgrim, D. Golden, L. Yang, MedGemma Technical Report, 2026. URL: http://arxiv.org/abs/2507.05201. doi:10.48550/arXiv.2507.05201, arXiv:2507.05201 [cs.AI].
  9. [9]Y. Labrak, A. Bazoge, E. Morin, P.-A. Gourraud, M. Rouvier, R. Dufour, BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 5848–5864. URL: https://aclanthology.org/2024.findings-acl.348/. doi:10.18653/v1/2024.findings-acl.348.DOI
  10. [10]E. Bolton, A. Venigalla, M. Yasunaga, D. Hall, B. Xiong, T. Lee, R. Daneshjou, J. Frankle, P. Liang, M. Carbin, C. D. Manning, BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text, 2024. URL: http://arxiv.org/abs/2403.18421. doi:10.48550/arXiv.2403.18421, arXiv:2403.18421 [cs.CL].
  11. [11]Devera, AI Water & Energy Usage: ChatGPT vs Claude vs Gemini (2026), 2025. URL: https://devera.ai/resources/environmental-impact-ai-energy-carbon/.
  12. [12]N. Bannour, S. Ghannay, A. Névéöl. Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools, in: I. Gurevych, A. Fan, T. Wolf, Y. Hou, A. Marasović, S. Ravi (Eds.), Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, Association for Computational Linguistics, Virtual, 2021, pp. 11–21. URL: https://aclanthology.org/2021.sustainlp-1.2/. doi:10.18653/v1/2021.sustainlp-1.2.DOI
  13. [13]mlco2/codecarbon, 2026. URL: https://github.com/mlco2/codecarbon.
  14. [14]L. F. W. Anthony, B. Kanding, R. Selvan, Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models, 2020. URL: http://arxiv.org/abs/2007.03051. doi:10.48550/arXiv.2007.03051, arXiv:2007.03051 [cs.CY].
  15. [15]W. J. Maddox, G. Benton, A. G. Wilson, Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited, 2020. URL: http://arxiv.org/abs/2003.02139. doi:10.48550/arXiv.2003.02139, arXiv:2003.02139 [cs.LG].
  16. [16]N. Muenninghoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, C. Raffel, Scaling Data-Constrained Language Models, in: Advances in Neural Information Processing Systems, volume 36, Curran Associates, Inc., 2023, pp. 50358–50376. URL: https://proceedings.neurips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html. doi:10.52202/075280-2191.
  17. [17]M. A. Zadenoori, V. D. Martino, J. Dabrowski, X. Franch, A. Ferrari, Does Model Size Matter? A Comparison of Small and Large Language Models for Requirements Classification, 2025. URL: http://arxiv.org/abs/2510.21443. doi:10.48550/arXiv.2510.21443, arXiv:2510.21443 [cs.SE].
  18. [18]M. Mehditabar, S. Rajput, A. Mastropaolo, T. Sharma, Smart but Costly? Benchmarking LLMs on Functional Accuracy and Energy Efficiency, 2025. URL: http://arxiv.org/abs/2511.07698. doi:10.48550/arXiv.2511.07698, arXiv:2511.07698 [cs].
  19. [19]A. Pal, A. G. Minervini, Pasqualeand Motzfeldt, A. P. Gema, B. Alex, Openlifescienceai, openmedicalllm-leaderboard, https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard, 2024.
  20. [20]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, Z. Qiu, Qwen3 Technical Report, 2025. URL: http://arxiv.org/abs/2505.09388. doi:10.48550/arXiv.2505.09388, arXiv:2505.09388 [cs.CL].
  21. [21]A. Grattafiori, A. Dubey, A. Jauhari, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenев, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Grover, A. Spartaru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. v. d. Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. v. d. Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. d. Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheshwari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kre ymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I.-E. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, K. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, Z. Ma, The Llama 3 Herd of Models, 2024. URL: http://arxiv.org/abs/2407.21783, doi:10.48550/arXiv.2407.21783, arXiv:2407.21783 [cs.AI].
  22. [22]G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette-Choo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozińska, D. Herbison, E. Bandy, E. Wang, E. Noland, E. Mor- eira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, G. Martins, H. Hashemi, H. Klimczak-Plucińska, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernandez, J. v. Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J.-y. Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Görner, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardioliwalla, N. Devanathan, N. Dumai, N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin, S. M. R. Arnold, S. Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Z. Wei, V. Cotruta, P. Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, A. Andreev, Gemma 2: Improving Open Language Models at a Practical Size, 2024. URL: http://arxiv.org/abs/2408.00118. doi:10.48550/arXiv.2408.00118, arXiv:2408.00118 [cs.CL].
  23. [23]G. Team, K. Amath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, R. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J.-b. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J.-T. Peter, D. Sinopalinikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. J. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J.-y. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Pöder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sansevierо, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J.-B. Alayrac, R. Anil, Dmitry, Lepikhин, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, L. Hussenot, Gemma 3 Technical Report, 2025. URL: http://arxiv.org/abs/2503.19786. doi:10.48550/arXiv.2503.19786, arXiv:2503.19786 [cs.CL].
  24. [24]J. Zschache, T. Hartwig, Comparing energy consumption and accuracy in text classification inference (2025). URL: https://arxiv.org/abs/2508.14170. doi:10.48550/ARXIV.2508.14170, version Number: 1.
  25. [25]N. P. Tran, B. Jaumard, O. Delgado, Energy-Aware LLMs: A step towards sustainable AI for downstream applications (2025). URL: https://arxiv.org/abs/2503.17783. doi:10.48550/ARXIV.2503.17783, version Number: 1.
  26. [26]A. Islam, S. B. Belhaouari, A. Bermak, Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability (2025). URL: https://arxiv.org/abs/2502.17071. doi:10.48550/ARXIV.2502.17071, version Number: 1.
  27. [27]S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, V. Gadepally, From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference, 2023. URL: http://arxiv.org/abs/2310.03003. doi:10.48550/arXiv.2310.03003, arXiv:2310.03003 [cs].
  28. [28]P. J. Maliakel, S. Ilager, I. Brandic, Characterizing llm inference energy-performance tradeoffs across workloads and gpu scaling, 2026. URL: https://arxiv.org/abs/2501.08219. arXiv:2501.08219.
  29. [29]N. Alizadeh, B. Belchev, N. Saurabh, P. Kelbert, F. Castor, Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy, in: 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), 2025, pp. 725–736. URL: https://ieeexplore.ieee.org/document/11025589. doi:10.1109/MSR66628.2025.00109.DOI
  30. [30]C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosal, B. Kanakiya, C. Chen, N. Vassilieva, B. B. Amor, M. A. Pimentel, S. Khan, Med42 – Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches, 2024. URL: http://arxiv.org/abs/2404.14779. doi:10.48550/arXiv.2404.14779, arXiv:2404.14779 [cs].
  31. [31]T. Tasfia, A. K. M. Masum, E. Meyer, F. Cumbo, S. Aygun, How low can medical llms go?, an empirical study of low-bit quantization, in: 2026 IEEE 19th Dallas Circuits and Systems Conference (DCAS), 2026, pp. 1–6. doi:10.1109/DCAS69364.2026.11545012.
  32. [32]CO2 emissions per kWh in Germany - Nowtricity, 2026. URL: https://www.nowtricity.com/country/germany/.
  33. [33]L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, A. Zou, The language model evaluation harness, 2024. URL: https://zenodo.org/records/12608602. doi:10.5281/zenodo.12608602.

Paper details

vlnc:W_8qbxgczy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Grammarly, Gemini (multiple versions)

View statement
Looking for related papers…
Contents