WorksComputer Science
Best of Both Worlds: Fine-Tuned Small Models Complement LLMs for Cost and Performance Benefits
Abstract
BERT-based models require thousands of labeled samples to train. Larger models like GPT-X require only 1-5 examples per class. However, performance, cost and time tradeoffs of such models remain under-explored. In this work, we classify survey text data into 16 classes. We focus on experimental approaches and results on performance, cost, and time taken for this problem. We compare these parameters of a trained RoBERTa-based model, and an (off-the-shelf) GPT-3.5 model for text classification. Baseline experiments show that the time taken by GPT-3.5 for generating predictions on 10,000 records is 6 hours, whereas a RoBERTa-based model takes only 12 mins on a T4 GPU. To identify the best prompting approach for GPT-3.5, we explore batch prompting. We experiment with (1) batch-size, and (2) few-shot sampling. We find the best batch size is 20 with human few-shot curation. Experiments show the F1 score of a trained RoBERTa-based model (96%) is much higher than a GPT-3.5 model (83%). In case of GPT-3.5, batch prompting is beneficial in lowering inference costs by ~70%, without hindering task performance. We find the best approach is an ensemble of a RoBERTa-based and GPT-3.5 model (F1: 97.5%) incurring only 4.4% of inference cost compared to a standalone GPT-3.5.
Introduction
Natural language processing has seen impressive advancements in the past few years, especially in text classification. Large language models have achieved performance comparable to supervised baselines such as question answering [9], named entity recognition [11], relation extraction [10] and human annotator performance [4].
Problem Background
Open text response classification is challenging. It involves reasoning over long text and overlapping diverse topics. For experiments in this work, we consider a subset of survey text responses (around 10,000 samples) for prediction and 30,000 of labeled texts for generating few shot samples or training. We pick 16 topics with clear definitions that are mutually exclusive.
Experimental Setup
We compare the F1 score, time, and cost of a fine-tuned RoBERTa-based model and an off-the-shelf GPT–3.5 model under ideal batch size and few shot sample configuration. To identify the optimal configuration of batch size and few shot samples, we have the below described experiment setup. All experiments are run using the Element platform’s Airflow jobs, and a Tesla T4 GPU to do training and inferences.
Baseline Experiments
Our baseline experiments consist of three models (Table 1). The first model is a fine-tuned RoBERTa-based model. It is trained on 21,000 samples and validated on 9,000 samples. The second is a zero-shot GPT–3.5 model and the final model is a 3-few-shot GPT–3.5. The RoBERTa-based model outperforms both GPT–3.5 approaches.
Batch Prompting
In the preliminary stage of our experiment, we seek to determine the optimal batch size for our input data. To maintain consistency, we kept the size of few-shot examples constant (3 samples) while varying the batch sizes from 10 to 50 in increments of 10. This approach enables conducting an exploration of different batch sizes and its impact on the overall performance, time and cost. This also served as a step in deciding the optimal batch size for subsequent few shot experiments.
Few-Shot Techniques
For few-shot experiments, we use samples that are only one or two sentences long as few shot examples. Longer response texts can have two or more topics associated with them introducing complexity for the model. Also, having shorter few-shot examples keeps runtimes and costs within budget. The two strategies for few-shot selection are (1) random selection and (2) human-curated selection. Batch size is set constant to 20, which found to be optimal from batch prompting experiments. We vary the number of few shot examples from one to nine, in increments of two.
Experimental Results
Batch Prompting
An optimal batch size is identified by examining the outcomes across three dimensions: Time, cost, and failure rate (percentage of instances where the model failed to generate any prediction). We vary batch sizes (10, 20, 30, 40, 50) to attempt to optimize cost relative to the risk of batches failing due to being too large for the LLM context window (Figure 1). There is no significant change in performance across batch size (Table 2)}, but even in the presence of a sharp increase (2.9 % to 41.5 %) in batch failures (Figure 2)`, using larger batch sizes still results in the best cost ($3.4 for batch size 50).

Figure 1. Variations in total time (minutes) and cost ($) observed across experiments with batch sizes ranging from 10 to 50 (fewshot kept constant). X-axis labels are annotated as batch_[batch size]_[few-shot count]. Larger batch sizes are more cost-effective but take longer time (in most cases).

Figure 2. Variations in total cost ($) and failure (%) observed across experiments with batch sizes ranging from 10 to 50 (fewshot kept constant). X-axis labels are annotated as batch_[batch size]_[few-shot count]. Larger batch sizes are more cost-effective (similar to Figure 1) but lead to higher failure rate.
| Batch Size | 10 | 20 | 30 | 40 | 50 |
| F1 (%) | 82 | 82 | 82 | 82 | 83 |
| Time (min) | 138 | 151 | 181 | 171 | 168 |
| Cost ($) | 7 | 4.8 | 4.1 | 3.7 | 3.4 | |||||
| Failure (%) | 2.9 | 3.8 | 6.3 | 8 | 41.5 |
Table 2. Batch size vs. F1, time, cost and failure %.
Optimal batch size is 20 which keeps time, cost and failures rate low while maintaining the F1 score. It is also 70% more cost-effective compared to the GPT–3.5 zero-shot baseline ($16 in Table 1).
| Model | F1 | Time | Total Cost | |
| train | inf. | |||
| RoBERTa | 96 | 106 | 12 | 11K |
| GPT–3.5 | ||||
| Zero-shot | 79 | N/A | 340 | 16 |
| Few-shot | 79 | N/A | 371 | 44 |
Table 1. Baseline models with F1-score, training and prediction (inference) time (minutes), and total cost ($). Training is done on 21,000 samples and prediction is done on 10,000 samples. Cost is an estimate of human annotation effort involved in the annotation task. We use Manthano.ai (a free tool) for human annotation cost estimations (in 2023). Costs for GPT–3.5 are only inference costs (no human annotation needed).
Few-Shot Selection Techniques
Table 3 shows the results of random and human-curated few shot sample selections. We test various few-shot sample sizes (1, 3, 5, 7, 9) using both a random-selected and human-curated method. Costs increase with sample-size in the random condition faster than in the human-condition. This is largely driven by the fact that human-curated examples tend to have shorter token counts (more succinct) (Table 4: 4047 vs. 2279 in the 9-example condition).
| Few-shot # | 1 | 3 | 5 | 7 | 9 |
| Random-selected | |||||
| F1 (%) | 81 | 82 | 83 | 83 | 83 |
| Time (min) | 159 | 172 | 190 | 205 | 201 |
| Cost ($) | 3.7 | 4.8 | 6.1 | 7.1 | 8.3 | |||||
| Failure % | 4.5 | 5.0 | 4.1 | 4.0 | 5.3 |
| Human-curated | |||||
| F1(%) | 82 | 82 | 83 | 83 | 83 |
| Time (min) | 226 | 304 | 299 | 238 | 207 |
| Cost ($) | 3.4 | 4.0 | 4.5 | 5.0 | 5.6 | |||||
| Failure % | 6.9 | 4.2 | 4.9 | 5.7 | 4.5 |
Table 3. Few-shot size vs F1, time, cost & failure %. Human-curated few-shot examples are more cost-effective (due to shorter token counts (Table 4)).
| Selection | Prompt | Generation | Reasoning | Cost |
| Few-shot 1 | ||||
| Random | 1059.4 | 1043.2 | 18.4 | 3.7 |
| Human | 910.4 | 1033.7 | 18.2 | 3.5 |
| Few-shot 3 | ||||
| Random | 1794.4 | 1053.9 | 19.0 | 4.9 |
| Human | 1257.4 | 1046.5 | 18.7 | 4.0 |
| Few-shot 5 | ||||
| Random | 2620.4 | 1059.2 | 19.3 | 6.1 |
| Human | 1605.4 | 1049.3 | 18.8 | 4.6 |
| Few-shot 7 | ||||
| Random | 3259.4 | 1071.0 | 19.9 | 7.1 |
| Human | 1930.4 | 1057.0 | 19.2 | 5.1 |
| Few-shot 9 | ||||
| Random | 4047.4 | 1074.0 | 20.1 | 8.3 |
| Human | 2279.4 | 1062.8 | 19.5 | 5.6 |
Table 4. Cost ($) and token count comparison of random-selected and human-curated few-shot experiments in various stages (prompt, generation and reasoning). Human-curated experiments are more cost efficient. This table complements Table 3.
Related Work
Batch Prompting Techniques
Batch prompting, an efficient way to prompt LLMs. It performs inference on samples in a batched fashion. Multiple samples can be handled in one API call so that the costs of tokens and time can be significantly reduced [3]. Also, experiments with different permutations of input samples for every batch might affect performance and time metrics. This involves a voting method
to finally determine the prediction, to minimize the variance in final outcomes and controlling indefinite rounds of voting [5].
Few-Shot Techniques
In-context learning involves providing a few demonstration examples in the prompt template. Performance of GPT–3.5 can show quite the dependence on a few shot examples [2]. There has been past work in determining ways to select such examples for attaining better performance. Skill-based k-nearest neighbor approaches have been used to select few shot examples [1]. Trained retriever models [7] are also used to extract the best suited examples for tasks.
Conclusion
We carry out an exploration of performance, cost and time tradeoff of a fine-tuned RoBERTa-based model and a GPT–3.5 model for a text classification task. We explore batch prompting, a technique where multiple responses can be processed in a single API call thus saving cost and time. Upon experimenting with different batch sizes, we find that a batch size of 20 is optimal in terms of performance, cost and time for our data. Human-curation of few-shot examples emerges as a cost-effective strategy, with an optimal size of around three examples to achieve a favorable F1-score without incurring unnecessary costs. In Appendix § 9, we examine errors, exploring error types such as compound, domain-based, and granularity-based errors, highlighting the need for labels and few-shot examples to be unambiguous and mutually exclusive. Our final approach of using an ensemble model by first using RoBERTa-based model and then using GPT–3.5 on the records where former is less confident achieves superior performance of 97.5% incurring only 4.4% of inference cost compared to when a pre-trained LLM (GPT–3.5) alone was used. For generating training data, we propose leveraging LLMs along with human labeled data. As future work, we would like to explore batching similar records and other batch selection techniques. Also, exploring current state-of-art LLMs and other open-source LLMs to assess the trade-offs would be a promising direction.
Acknowledgments
We would like to thank Deeksha Yennam and Punith Krishna Pothula for their thoughtful reviews, suggestions, and immense support over time. We would also like to acknowledge the Walmart review board for their support.
Appendix 1: Error Analysis
We analyze instances where the RoBERTa-based model’s predicted class is correct, but GPT-3.5 predicted class is incorrect. This analysis is done on a random sample of 100 error instances. We human-evaluate for error classes. This approach seeks inspiration from [6]. We find three main error groups shown in Table 5. Text examples mentioned are paraphrased and hence synthetic.
| Error (%) | Example | GPT–3.5 | RoBERTa/Gold |
| Compound (23%) | Train on the equipment | Equipment | Development |
| Domain / Short Text (35%) | Snack | No Complaint | Benefits |
| Granularity (42%) | Info on hours in the assigned dept | Communicate | Shifts |
Table 5. Analysis of GPT–3.5 errors compared to correct RoBERTa-based and Gold on a random sample of 100 instances. Examples are paraphrased and hence synthetic. This table complements Appendix § 9.
Compound: The response text may contain tokens of a certain class (“equipment”) but belongs to some other class. For example, “Train on the equipment” ought to fall into Development as opposed to Equipment (since it talks about training).
Domain or Short Text: An off-the-shelf GPT–3.5 despite having provided few-shot examples lacks the deep domain-level understanding of a trained RoBERTa-based model. Especially in cases where response texts can be shorter (giving lesser context). In the example, “snack” refers to the provision of snacks, which is aligned to benefits, as opposed to no complaint.
Granularity: At times GPT–3.5 assigns a class, but its reasons towards a rather more generic class. In the example “Info on hours” does imply some level of communication but more specifically is a shift related topic.
References
- [1]Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. 2023. Skill-based few-shot selection for in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13472–13492, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.14210
- [2]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dharival, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.arxiv.org/abs/2005.14165
- [3]Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch prompting: Efficient inference with large language model APIs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 792–810, Singapore. Association for Computational Linguistics.arxiv.org/abs/2301.08721
- [4]Fabrizio Gilardi, Meysam Alizadeh, and Maël Kublı. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30).DOI
- [5]Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. 2024. Batchprompt: Accomplish more with less. In International Conference on Learning Representations, volume 2024, pages 21590–21612.arxiv.org/abs/2309.00384
- [6]Shivam Mathur, Keun Park, Dhivya Chinnappa, Saketh Kotamraju, and Eduardo Blanco. 2023. Interpreting answers to yes-no questions in user-generated content. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14137–14161, Singapore. Association for Computational Linguistics.
- [7]Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.
- [9]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014–10037, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2212.10509
- [10]Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. GPT-RE: In-context learning for relation extraction using large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3534–3547, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.02105
- [11]Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. GPT-NER: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275, Albuquerque, New Mexico. Association for Computational Linguistics.arxiv.org/abs/2304.10428