WorksComputer Science
Retrieval Supersedes Scale: Efficient Log Level Prediction via Multiplex Socio-Technical Context
Abstract
Developers insert logging statements in source code to record its execution, and the resulting logs are essential for detecting and troubleshooting system anomalies. A key part of these statements is the log level (e.g., DEBUG, INFO, WARN, ERROR), which determines whether a message is recorded. Choosing it is a trade-off: levels set too low mask important information in production, while levels set too high create noisy logs and extra I/O and storage overhead. Recent work uses Large Language Models (LLMs) with In-Context Learning (ICL) to predict log levels, but typically augments the prompt with randomly sampled examples. We argue that this overlooks the socio-technical nature of logging, where conventions vary by component and development team. Off-the-shelf LLMs using random examples are reported to achieve median AUCs of only 0.56 to 0.76, leaving room for improvement without fine-tuning. In this paper, we propose a retrieval approach for LLM-powered log level prediction that prioritizes context relevance over random selection and basic similarity. Our method retrieves logging examples relevant along two socio-technical dimensions: (i) functionality (code performing similar operations) and (ii) ownership (code maintained by the same developers). Evaluating four LLMs across ten popular Java projects, we find that this retrieval improves AUC over random baselines by 0.07 (median), and combining both signals yields median AUCs of 0.83 to 0.97. Our ablation studies show that each signal improves on random retrieval, and that combining them outperforms the stronger of the two on all ten projects. Our results also significantly exceed state-of-the-art supervised predictors (e.g., DeepLV) by a median AUC of 0.12 to 0.19. Our findings highlight the importance of socio-technical signals such as code functionality and ownership for LLM-powered log level prediction.
Introduction
Developers insert logging statements into the source code of software systems to capture essential runtime information, facilitating debugging, system observability, and anomaly prediction, detection, and diagnosis [17, 33, 39]. A typical logging statement includes a log level (e.g., DEBUG), and a textual message with or without variables. Log levels are typically ordered by verbosity from DEBUG (most verbose) to FATAL (least verbose) in Java logging libraries. Such log levels control the volume of logged data, as during software operation, only log messages at or above a set threshold are recorded, while more verbose messages are excluded. Operators set this threshold through configuration, independently of the levels written in the code.
However, accurately determining the appropriate log level for each logging statement remains a challenging activity for developers [18, 30], who typically make initial poor log level choices [17, 27]. Suboptimal log level decisions have significant practical consequences as a too low log level drastically reduces software observability, complicates debugging, and prolongs failure diagnosis [39], while an excessive log level introduces performance overhead, inflates storage, and overwhelms developers with redundant information [38].
To assist developers in log level decision-making process, researchers have proposed various automated Log Level Predictors (LLPs) [1, 14, 16, 18, 20, 21]. Early approaches [1, 16, 18] relied on engineered software features (e.g., cyclomatic complexity) to train shallow classifiers such as ordinal logistic regression. Subsequent research shifted toward deep learning, leveraging learned embeddings of logging statements and their surrounding code context to capture functionality nuances [20, 21]. Most recently, the advent of generative AI has spurred the adoption of Large Language Models (LLMs) for log level prediction, utilizing paradigms ranging from zero-shot inference to few-shot In-Context-Learning (ICL) and fine-tuning (e.g., LoRA) [14].
Current LLM-powered log level prediction approaches typically rely on random or stratified sampling for in-context examples, a practice that implicitly favors data diversity over contextual relevance [14, 23]. While often chosen for simplicity, this design is supported by the assumed benefits of data diversity in deep learning [12], including (1) improved robustness on unseen inputs, (2) mitigation of selection bias, and (3) coverage of all output classes within the limited context window. Consequently, diverse, randomized sampling is widely treated as a reasonable default for conditioning frozen LLMs.
We argue that this approach (i.e., diversity-first) overlooks key aspects that are fundamental to the logging practice [29]. Specifically, projects are not a uniform entity, with a single, consistent logging strategy. In reality, distinct components or subsystems often exhibit different logging practices influenced by factors such as code functionality (i.e., what the code does) and developer ownership (i.e., who maintains or frequently changes a file). We hypothesize in this paper that leveraging these internal distinctions can allow existing LLM-powered LLPs to use the most contextually relevant software knowledge, thereby enhancing their predictive accuracy and practical utility.
For example, the SIG-Auth team1 in the Kubernetes community maintains authentication logic dispersed across the API Server and the Kubelet. These files sit in different components and have distinct functionality, yet they may share logging conventions because the same people write them. Similarly, Hadoop’s RetryInvocationHandler and Yarn’s RequestHedgingRMFailoverProxyProvider implement the same retry duty in different components, and both log each attempt at DEBUG before escalating once the retries are exhausted, differing only in the terminal severity that each component’s failure warrants. Logging conventions are therefore shaped by both shared ownership and functional similarity, and omitting either signal may limit the predictive accuracy of LLM-powered LLPs. We select functionality and ownership as a complementary pair rather than as an exhaustive model of socio-technical context. Functionality represents the technical responsibilities and observability needs of the code, while ownership represents the social and evolutionary context in which local logging conventions develop. Both signals can be derived consistently at file granularity across the studied projects and represented as aligned layers of the same multiplex graph.
In this paper, we investigate the impact of two socio-technical signals (i.e., code functionality and ownership) on the performance of in-context-learning LLM-powered log level prediction. We propose a methodology that extracts and fuses two distinct socio-technical signals (i.e., code functionality and ownership) into the retrieval process. Specifically, we model a studied software project as a two-layer multiplex graph [3, 9], where nodes represent source files and layers represent the distinct relationship types. While the first layer encodes functionality, where edges connect files with high functional similarity, the second layer encodes ownership, where edges connect files maintained by the same group of contributors. By applying community detection on this unified multiplex structure, we partition the project into coherent socio-technical clusters. Finally, for any given target file, we restrict the retrieval of in-context-learning examples exclusively to its assigned cluster, ensuring that the LLM is augmented with examples that are both functionally relevant and aligned with the team’s logging conventions.
It is important to distinguish log level prediction from the broader task of end-to-end log generation. While generative tools aim to solve the “cold start” problem by synthesizing messages for unclogged code, they introduce a significant risk when applied to existing logging statements
as regenerating a full statement can alter developer-tuned message content or variable selection, degrading log quality and increasing code review burden [31]. In practice, developers often simply need to ‘up-shift’ or ‘down-shift’ the visibility of an existing log message without rewriting it. Therefore, log level prediction addresses a distinct calibration problem pertaining to the optimization of the trade-off between observability and overhead for developer-written log messages. Furthermore, our study concerns log level prediction with frozen, out-of-the-box LLMs, where the model parameters remain unchanged and only the in-context-learning examples vary. Although fine-tuning can adapt an LLM to a specific project logging data, it requires a significant amount of labeled training data and parameter optimization, which add training and maintenance costs. Our approach instead keeps the LLM frozen and adapts its context at inference time through carefully retrieved examples. This provides a lighter-weight form of context adaptation and allows us to isolate the contribution of retrieval under identical model and inference configuration.
To evaluate our approach, we answer the following RQs:
RQ1. How does the integration of ownership and functionality signals via multiplex clustering impact the predictive performance of LLM-powered LLPs? Our approach, leveraging both functionality and ownership signals simultaneously via multiplex clustering, achieves a median AUC between 0.83 and 0.97 across our evaluated projects. The approach statistically significantly (Wilcoxon test, ) outperforms state-of-the-art approaches in LLM-powered log level prediction (in terms of AUC) by a median of 0.07. Moreover, this integration of socio-technical signals enables frozen LLMs to exceed the AUC of prior specialized supervised models (e.g., DeepLV: +0.12, Ordinal Regression: +0.19) while avoiding task-specific model training and ongoing retraining costs.
RQ2. What is the individual contribution of functionality and ownership signals? Leveraging in-context-learning examples retrieved from our file ownership clusters alone statistically significantly enhances the AUC of LLM-powered log level prediction by a median of 0.02 (Wilcoxon test, ), compared to the state-of-the-art approaches in LLM-powered log level prediction. Furthermore, leveraging in-context-learning examples retrieved from our functionality-only clusters statistically significantly enhances the AUC of LLM-powered log level prediction by a median of 0.05 (Wilcoxon test, ), compared to the state-of-the-art approaches in LLM-powered log level prediction.
RQ3. Can multiplex Retrieval reduce the computational cost of LLM-powered log level prediction? Our findings indicate that precise context retrieval is more critical than raw model scale. A multiplex-enhanced 7B model outperforms state-of-the-art commercial models, achieving an average AUC of 0.89, surpassing GPT-4o by 0.10 AUC and DeepSeek-V3.2 by 0.16 AUC. We demonstrate that scaling to larger local models (34B) yields diminishing returns, offering only marginal performance gains (+0.01 AUC) at a disproportionate operational cost increase (+322%).
Our findings suggest the adoption of small, self-hosted and context-aware models over general-purpose commercial APIs for LLM-powered log level prediction. We observe that integrating socio-technical signals (i.e., ownership and functionality) enables smaller models (e.g., 7B) to outperform much larger baselines, proving that precise retrieval supersedes raw parameter scale. We suggest practitioners prioritize self-hosted, batched inference pipelines to balance accuracy with operational efficiency for log level prediction tasks.
Background & Related work
Log level prediction
One direction in software logging research aims to automate the log level choice [1, 14, 18, 20], as developers and empirical studies reported challenges with the log level choice activity. For instance, Li et al. [18] leveraged an ordinal logistic regression model, specifically designed to predict log levels using a set of static, change-related and historical code features (e.g., log message length, logging statement churn, number of revisions in history). Anu et al. [1] introduced VerbosityLevelDirector, an automated approach based on random-forest designed to help developers assign appropriate log levels based on logging intention embedded within the code context. More recently, Li et al. [20] introduced a deep learning take on log level prediction. Their approach leveraged a neural network architecture that encoded syntactic and semantic contexts surrounding logging statements as well as the contents of the logging statements, to predict an ordinal output corresponding to the different log levels. Heng et al. [14] explored log level prediction using an LLM-powered approach, employing various LLMs with different sizes (e.g., CodeLlama-7B). Given a source-code snippet (where the logging statement is masked) as a context, and the content of the log message, the LLM predicts the appropriate log level by generating an appropriate textual response.
That said, state-of-the-art efforts leveraging LLMs for log level prediction leverage random retrieval due to the ease of the approach and as a way to promote diversity in the examples provided to LLMs at inference time. In this paper, we investigate log level prediction approaches that take into consideration the socio-technical aspects of software systems (i.e., the code functionality and its ownership).
In-context-learning for logging tasks
Prior studies have leveraged in-context-learning [5, 36] for logging automation to bridge the gap in domain knowledge by incorporating task-specific examples directly into the input prompt without parameter updates [14, 19, 37]. However, the effectiveness of in-context-learning is often constrained by the limited context window, which restricts the number of demonstrations that can be utilized during inference. Such in-context-learning approach was evaluated for log level prediction [14, 19, 37]. For instance, Xu et al. [37] introduced UniLog, an LLM-powered solution that employs in-context-learning to automatically generate log messages, recommend appropriate logging statement placements, and predict their verbosity levels. Heng et al. [14] investigated log level prediction using various LLMs with different sizes and model architectures, and showed that even randomly picked in-context-learning examples can lift
the performance of LLM-powered LLPs. Li et al. [19] built LogBench, a logging benchmark of code instances collected from GitHub repositories together with a transformed, unseen counterpart, and used it to evaluate 13 LLMs ranging from 60 million to 405 billion parameters on logging tasks, including log level prediction.
Our work is in line with this research direction as we evaluate in-context-learning approaches for log level prediction. Yet, instead of random in-context-learning retrieval, our study investigates how the selection of in-context-learning examples can be improved when taking into consideration the ownership and functionality relationships present within modern software systems.
Methodology
In this section, we introduce an in-context-learning retrieval approach designed for log level prediction. Our approach provides relevant logging examples to LLMs to generate log levels for newly added or modified logging statements within source code, as shown in Figure 1. The input of our approach is the file name and the specific logging message for which a log level prediction is to be predicted. The output is the predicted log level. To perform the prediction, we first select in-context logging examples from the cluster that is relevant both in terms of functionality and ownership to the logging statement in hand. We discuss the creation of these clusters in the following sections:
Figure 1. Overview of our Ownership-Functionality Aware retrieval approach for log level prediction.
Functionality-based clustering
We leverage functionality-based clustering to group files based on their functional similarity, to identify functionally cohesive file clusters that are expected to share similar logging behaviors.
Raw data construction: To build our functionality-based clusters we follow the approach summarized in Figure 2. We first encode the source files of each project with a transformer-based embedding model, which produces one dense vector per file that reflects both the syntax and the functionality of its code [6, 10, 22]. We chose the encoder-only transformer CodeXEmbed [22] because it provides state-of-the-art performance on code-related embedding tasks, supports input sequences of up to 32K tokens –thus effectively capturing the full context of most source files in our studied projects– and achieves these capabilities despite its modest size (2.6B parameters), making it computationally efficient and practical. After the encoding, each source file is represented as a 2,304-dimensional embedding vector capturing the file’s functional information. Files exceeding the 32K token window (less than 5% across projects) are partitioned at top-level method boundaries and encoded per method, then mean-pooled into a single file-level vector [13], so that every file receives exactly one embedding and remains eligible for clustering regardless of its size.
Figure 2. Overview of the functionality-based clustering
Dimensionality reduction: As direct clustering on high-dimensional embeddings is computationally expensive and sensitive to noise, we first apply UMAP (Uniform Manifold Approximation and Projection) [25] to project the 2,304-dimensional file embeddings into a 50-dimensional space, similarly to prior work [2]. UMAP preserves the local neighborhood structure while substantially reducing the computational cost of the clustering.
Functional-similarity clustering: We then apply the density-based clustering algorithm HDBSCAN [24] to the reduced embeddings. Density-based clustering is a deliberate design choice for our setting. Unlike partitional methods, which force every file into a cluster, HDBSCAN discovers clusters of varying densities and explicitly labels low-density files as noise. This matters because source files in large projects are heterogeneous, and assigning a file to a functionally incoherent cluster would yield misleading in-context-learning examples. For each project, we select the two main HDBSCAN hyperparameters: (i) the minimum cluster size and (ii) the minimum number of samples, through a grid search spanning very small to relatively large cluster granularities, scaled to the number of files in the project.2 We select the configuration that maximizes the Silhouette score among those with a Davies–Bouldin index below 1.0, thereby favouring compact, well-separated clusters. Because these indices measure the quality of a single partition but not its robustness, we additionally assess whether the selected clustering is stable under perturbation. To do so, we re-cluster 30 bootstrap samples of the files using the selected configuration and report the distribution of Adjusted Rand Index (ARI) values against the original clustering.
Ownership clustering
We perform ownership clustering to group files according to their shared developer contributions, based on the intuition that files maintained by the same authors might exhibit similar logging conventions.
Raw data construction: To construct our ownership-based clusters, we mine the Git history of each studied project and build an author–file ownership matrix (Figure 3). Rows correspond to source files and columns to authors and each cell records how frequently an author has modified a file, with commit counts weighted by an exponential time-decay factor so that recent edits contribute more heavily than older ones. This representation captures both long-term ownership and recent maintenance activity in a single high-dimensional vector for each file.
Figure 3. Overview of the ownership-based clustering
Build a file-to-file graph: We treat each row of this matrix as a file-level ownership vector and compute pairwise cosine similarities between files. Using these similarities, we construct a weighted file–file graph in which edges connect files that share similar maintainer sets. To avoid an overly dense graph while still preserving strong ownership signals, we connect each file only to its nearest neighbours by cosine similarity, with . We selected empirically, as lower values fragmented the graph by isolating too many files while higher values merged distinct groups.
Build community-based clusters: We then apply the weighted Leiden community detection algorithm [35] to this -NN graph to obtain ownership communities. Leiden optimizes modularity while avoiding the disconnected communities sometimes produced by the earlier Louvain algorithm [4], and scales well to our graph sizes. We tune the resolution parameter and retain settings that yield modularity above 0.7 for all projects, which indicates a pronounced community structure in the ownership graph [11, 35].
Multiplex clustering
We further investigate the combination of these two socio-technical signals into unified multiplex graph, as shown in Figure 4. A multiplex graph is a multi-layered network structure where the same set of nodes exists (i.e., files) across all layers, but each layer represents a distinct type of relationship between them [3, 9]. Specifically, we construct a two-layer multiplex graph, where the first layer encodes functional similarity between files based on code embeddings (computed using cosine similarity between file embeddings) and the second layer represents file ownership file-to-file graph (§ 3.2).
Figure 4. Overview of multiplex clustering and retrieval. Nodes are Java files, present in both layers. The layers differ only in their edges, whose weight reflects functional similarity or co-ownership strength. Dashed outlines mark the community containing the target file . The multiplex partition is obtained by optimizing modularity jointly over both layers, and the top- candidates are then ranked within the resulting community.
Both layers have the same set of nodes (each node corresponding to a Java source file) but maintain distinct edge sets, preserving the unique insights offered by each type of signal. To identify cohesive clusters across these complementary layers, we apply the multiplex variant of the Leiden community detection algorithm [35]. This multiplex algorithm simultaneously optimizes modularity across both the functionality and the ownership layers, so that the resulting communities are cohesive under the combined evidence of the two layers. A resulting community is therefore not the intersection of the two single-layer partitions. For instance, a strong relationship in one layer can keep together files that the other layer alone would have separated.
At inference time, for a given logging statement, our approach identifies the multiplex cluster to which the source file containing the logging statement was assigned, then retrieves the top candidate examples from within this cluster. We set in our experiments as performance flattens beyond this point, with offering no statistically significant improvement at roughly double the prompt length (Appendix Figure 7). We then rank the candidates by a combined similarity score
where:
: is the combined similarity score for candidate example .
: is the file containing the logging level to be predicted.
: is the functional cosine similarity between the embeddings of and candidate .
is the ownership cosine similarity between the embeddings of and candidate .
: Weighting coefficient that controls the relative influence of functional similarity versus ownership similarity in the combined retrieval score (tunable with our approach).
Note that only orders the candidates that the multiplex cluster has already selected. Specifically, the clustering determines which files are eligible, and the two single-layer similarities rank within that set.
We determined the optimal layer weighting via a grid search on a held-out validation set (a subset of the retrieval data). We evaluated in increments of 0.1 and selected , as it maximized the downstream log level prediction AUC score. This value was then fixed for all subsequent experiments on the test sets. In this configuration, our approach gives more influence to functional similarity while still incorporating ownership information. This choice aligns with our design assumption that functional similarity should be the primary retrieval signal, with ownership acting as a secondary refinement. That said remains a configurable parameter and can be adjusted by practitioners if different trade-offs between functional and ownership signals are desired.
If we cannot identify a suitable multiplex cluster for the target file, our approach applies the following fallbacks in order: (1) functionality cluster retrieval, (2) ownership community retrieval, and (3) random retrieval from the entire project. When a cluster is found but holds fewer than candidates, all available candidates are used and no fallback is triggered.
Prompting LLMs for log level prediction
To create the log level prediction prompt we leverage the approach explained and detailed in Figure 6 in the appendix. Specifically, for each logging statement we extract the enclosing source code in which it appears, with the logging call itself removed, the text of the log message together with the variables it interpolates, and, for the statements used as examples, the log level its developers assigned to it. All evaluated retrieval strategies share the same fixed prompt template shown in Figure 5, so the only difference between them is which logging examples fill that template.

Figure 5. The prompt template instantiated with logging statements, showing one of the demonstrations. Grey annotations are not part of the prompt.
Figure 6. Prompt creation process for the LLM-powered log level prediction. The candidate log levels listed in the instruction are those the project actually uses, and therefore vary across projects.
For the purposes of this study, we chose four models CodeLlama-7B-Instruct, CodeLlama-34B-Instruct, DeepSeek-V3.2, and GPT-4o, as shown in Table 7 (Appendix). We specifically chose the CodeLlama family (7B and 34B) to isolate the impact of model size on performance within a single, open-source architecture. The 7B version serves as a baseline for consumer-grade hardware, while the 34B version represents the upper limit of what can be deployed on a single consumer-grade GPU node (without quantization). To benchmark these local models against industry standards, we selected GPT-4o as a high-performance commercial baseline, as it is one of the few leading frontier models that provides API access to token log probabilities, which are necessary for our evaluation metrics. Finally, we included DeepSeek-V3.2 to serve as a cost-efficiency benchmark, allowing us to compare our self-hosted approach against a high-performance but low-cost commercial API.
| Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket | |
| Step 1: multiplex cluster retrieval | 100.0% | 100.0% | 99.4% | 99.8% | 95.1% | 99.1% | 100.0% | 100.0% | 91.7% | 99.1% |
| Ranked by the socio-technical score | 90.7% | 67.3% | 97.3% | 95.3% | 85.8% | 95.0% | 95.9% | 93.8% | 91.7% | 64.3% |
| Sampled from the multiplex cluster{sup}† | 9.1% | 32.7% | 0.5% | 4.0% | 1.4% | 0.9% | 0.9% | 6.2% | 0.0% | 0.0% |
| Fewer than candidates in the cluster{sup}‡ | 0.2% | 0.0% | 1.7% | 0.5% | 7.9% | 3.2% | 3.2% | 0.0% | 0.0% | 34.8% |
| Step 2: single-signal retrieval | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Step 3: project-wide random retrieval | 0.0% | 0.0% | 0.6% | 0.2% | 4.9% | 0.9% | 0.0% | 0.0% | 8.3% | 0.9% |
Table 7. Coverage of each retrieval step, per project (CodeLlama-7B, five in-context-learning examples). Percentages are computed once over the unique test logging statements of each project. †: the file that contains the statement lacks one of the two signals, so one of the two cosine similarities is undefined and the examples are drawn uniformly from the same multiplex cluster. ‡: the cluster holds fewer than candidates, so all of them are used. Step 2 (single-signal retrieval) is never invoked across our studied projects, as the multiplex assignment fully subsumes what functionality-only or ownership-only retrieval would have covered.
Implementation details
To validate the practical feasibility of our approach, we measured the execution time of each clustering phase using a representative workstation. This evaluation aims to demonstrate that the socio-technical analysis can be integrated into standard development workflows without imposing prohibitive overhead. In fact, all our presented clustering approaches are computationally efficient, ensuring practical applicability for cluster refreshing if/when needed. Specifically, ownership-based clustering completes in a few minutes, even for our largest studied project (Elasticsearch). Functionality-based clustering takes slightly longer due to the initial step of computing embeddings using a transformer-based tokenizer. However, it remains practical, taking approximately 20 minutes for Elasticsearch (the largest project studied) on standard hardware (16-core CPU, 64 GB RAM, 24 GB GPU memory). Finally, once the artifacts from the prior two clusterings (i.e., ownership matrix and functionality embeddings) are created, multiplex clustering takes few minutes for all of our projects.
Empirical evaluation setup
We empirically evaluate the effectiveness of our approach (with its different operation modes) across ten widely used open-source Java projects: Hadoop, HBase, Elasticsearch, Cassandra, Flink, Kafka, JMeter, ZooKeeper, Karaf, and Wicket. These projects were selected due to their varied size, distinct logging practices, and differing clarity in documentation-based component definitions. Specifically, Hadoop and HBase feature explicit documentation-defined components, allowing comparison against a documentation-based retrieval baseline. Conversely, the remaining projects lack clearly defined component boundaries, presenting a scenario ideal for assessing the robustness and generalizability of our approach. Our selection also spans a wide range of logging volumes, from Elasticsearch’s 25,187 logging statements down to Wicket’s 375, allowing us to assess whether our approach depends on the availability of a large retrieval pool.
Data collection
We collected the datasets used in this paper by extracting source code files, logging statements, and associated metadata directly from each project’s GitHub repository. Specifically, we cloned each repository and checked out the latest stable release versions (e.g., Cassandra-4.1.9) to ensure reproducibility. To extract logging statements systematically, we parsed Java source files using the javalang library, identifying log invocations and capturing relevant context such as the log message, verbosity level, method boundaries, and surrounding code snippets. Metadata such as file paths, line numbers, and documentation-based components (if any) were also recorded. In cases where a project lacked explicit documentation-based component definitions (e.g., Cassandra), we labeled the component information as unknown.
Table 1 summarizes key characteristics of each of our studied projects.
| Project | Versions | # SLOC | # Contributors | # Logging statements |
| Hadoop | 3.3.6 | 1.8M | 635 | 21,910 |
| HBase | 2.5.9 | 841K | 483 | 10,449 |
| Elasticsearch | 8.9.2 | 2.5M | 2,042 | 25,187 |
| Cassandra | 4.1.9 | 595K | 464 | 4,808 |
| Flink | 2.3.0 | 1.7M | 1,671 | 3,800 |
| Kafka | 4.3.1 | 998K | 1,442 | 3,933 |
| JMeter | 5.6.3 | 149K | 95 | 1,890 |
| ZooKeeper | 3.9.5 | 124K | 292 | 1,341 |
| Karaf | 4.4.11 | 130K | 211 | 665 |
| Wicket | 10.10.0 | 222K | 163 | 375 |
Table 1. Summary of the studied projects
Evaluation baselines
We compare our approach against five baselines: three retrieval strategies and two specialized log level prediction models.
Random retrieval (diversity-first) [14]: in-context-learning examples are randomly selected from the entire dataset of the studied project.
Documentation-based retrieval: for the two projects that define components in their documentation (e.g., HDFS vs. Yarn in Hadoop), examples are retrieved exclusively from within the same documented component, without any clustering. We adopt this strategy for its simplicity and its demonstrated efficacy in supervised log level prediction [29].
Flat similarity retrieval: files across the complete project are ranked by code-embedding cosine similarity (same CodeXEmbed as our approach), and the five examples are drawn from the nearest files.
Ordinal Logistic Regression (OLR) [18]: treats log level prediction as an ordinal classification problem over hand-crafted features of the logging statement, its change history, its file and the project history.
DeepLV [20]: an ordinal classification approach that encodes the syntactic AST context surrounding the logging statement and its message tokens through an embedding layer and a Bi-LSTM network.
We order each project’s logging statements chronologically and split them 70%/30%, similar to prior work [28]. While the earlier 70% serves as the retrieval pool for the retrieval-based approaches and as the training set for OLR and DeepLV, the later 30% is used exclusively for testing. All retrieval strategies share the same underlying LLM (e.g., CodeLlama-7B), prompting protocol and inference configuration, so that only the mechanism for selecting in-context-learning examples differs.
We evaluate each method on five bootstrap samples of the test set, reporting their mean and standard deviation. Because all models predict the same logging statements, we use paired Wilcoxon signed-rank tests for significance, and report Cohen’s for effect size, interpreting as small, as medium and as large [7].
Evaluation metrics
As our approach integrates two modeling tasks (i.e., clustering and log level prediction), we evaluate it using two sets of metrics. To assess clustering quality we use the Adjusted Rand Index (ARI), the Silhouette score, the Davies–Bouldin index (DBI), Density-Based Clustering Validation (DBCV) and Leiden modularity, which we define in the Appendix. To assess log level prediction we use the three metrics established by prior work [14, 18, 20, 29]:
Area Under the ROC Curve (AUC): quantifies the ability of the model to discriminate between log level classes at various threshold settings. To adapt this metric for generative LLMs, we do not rely solely on the discrete text output. Instead, we extract the conditional probabilities of the first generated token corresponding to our target labels (i.e., ). We normalize these token probabilities to sum to 1 over the candidate set of log levels, effectively treating the LLM as a probabilistic classifier. Following prior work [14, 18, 20, 29], we then compute the multiclass AUC using the One-vs-Rest scheme as implemented in scikit-learn.4
Accuracy: measures the proportion of logging statements for which the predicted log level exactly matches the ground-truth level, as implemented in scikit-learn.5 Because accuracy is computed over all statements, it is sensitive to the class imbalance inherent to logging data. We, therefore, report it alongside AUC and AOD rather than in isolation.
Average Ordinal Distance Score (AOD): captures how close predicted log levels are to their true ordinal positions [14, 20]. Each log level is mapped to an ordinal value (e.g., ), and AOD normalizes the distance between the predicted and true levels to the interval, where higher values indicate predictions that are, on average, closer to the correct log levels. We report AOD as it provides a more nuanced assessment of ordinal prediction quality than accuracy or AUC alone.
Results
RQ1. How does the integration of ownership and functionality signals via multiplex clustering impact the predictive performance of LLM-powered LLPs?
Motivation: The goal of this research question is to understand the impact of leveraging both functional similarity and developer ownership signals simultaneously to retrieve ICL logging examples for log level prediction scenarios. While the former similarity captures the functional context (i.e., what is being logged), developer ownership captures the human aspect (i.e., who is deciding). Prior research has established that log level selection is influenced by individual developer habits and team norms [18, 39]. Consequently, retrieving examples based solely on code similarity may yield technically relevant but statistically inconsistent in-context-learning examples. By integrating the ownership signal with the functionality signal, we align the examples not only with the code’s logic but also with the specific logging conventions of the contributor.
Approach: To quantify the impact of multiplex based retrieval (taking into consideration ownership and functionality knowledge simultaneously) on the performance of LLM-powered LLPs, we first perform the multiplex clustering for each studied project, as described in § 3.3. Then, we evaluate the performance of our approach using in-context-learning examples from the relevant multiplex clusters. We compare this approach to five baselines: (a) random retrieval from the entire project [14], (b) flat similarity retrieval, (c) documentation-based retrieval when applicable, (d) Ordinal logistic regression LLP [18], and (e) DeepLV (Bi-LSTM) [20]. We report in this results section the findings for the Codellama-7B model and refer to the appendix for findings obtained using the Codellama-34B model.
Results: Multiplex retrieval significantly boosts the performance of LLM-powered LLPs, achieving both high AUC and low retrieval latency across all studied projects, as shown in Table 2. In fact, leveraging our ownership-functionality aware approach to retrieve in-context-learning logging examples from multiplex clusters results in a statistically significant improvement in AUC (Wilcoxon test, , Cohen’s , ), with AUC ranging from 0.83 (Wicket) to 0.97 (Elasticsearch) and exceeding the global random retrieval baseline by a median of 0.07 (0.02 to 0.17 across projects). Similarly, accuracy and AOD exhibit similar statistically significant (Wilcoxon test, , Cohen’s , ) improvements as the median accuracy and AOD improves by 0.11 and 0.05 respectively. While flat similarity retrieval also outperforms random retrieval on all ten projects, improving AUC by a median of 0.05 (Wilcoxon test, ), our multiplex retrieval outperforms flat similarity retrieval on all ten projects by a further median improvement of 0.02 AUC (Wilcoxon test, ). Thus, while retrieving functionally similar examples provides a substantial improvement over random retrieval, combining functionality and ownership information provides an additional performance benefit. Furthermore, our retrieval step leverages FAISS6 making it highly efficient, with a median latency of 8.9 ms per retrieval, effectively operating at real-time speed (i.e., suitable for IDEs). Moreover, our approach statistically significantly (Wilcoxon test, , Cohen’s , ) outperforms the established baselines for log level prediction [18, 20].
| Model | Metric | Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
| Multiplex retrieval (5) | AUC | 0.921 ± 0.005 | 0.914 ± 0.004 | 0.966 ± 0.002 | 0.907 ± 0.003 | 0.875 ± 0.005 | 0.851 ± 0.006 | 0.871 ± 0.010 | 0.866 ± 0.020 | 0.879 ± 0.041 | 0.827 ± 0.028 |
| Accuracy | 0.694 ± 0.006 | 0.663 ± 0.015 | 0.764 ± 0.005 | 0.668 ± 0.017 | 0.569 ± 0.008 | 0.571 ± 0.012 | 0.607 ± 0.012 | 0.604 ± 0.032 | 0.660 ± 0.009 | 0.509 ± 0.042 | |
| AOD | 0.883 ± 0.004 | 0.868 ± 0.007 | 0.912 ± 0.002 | 0.877 ± 0.002 | 0.844 ± 0.004 | 0.828 ± 0.007 | 0.843 ± 0.007 | 0.857 ± 0.012 | 0.868 ± 0.008 | 0.785 ± 0.018 | |
| Flat similarity retrieval (5) | AUC | 0.881 ± 0.003 | 0.851 ± 0.002 | 0.945 ± 0.002 | 0.873 ± 0.012 | 0.859 ± 0.003 | 0.802 ± 0.009 | 0.859 ± 0.009 | 0.852 ± 0.010 | 0.857 ± 0.072 | 0.814 ± 0.026 |
| Accuracy | 0.597 ± 0.009 | 0.562 ± 0.011 | 0.693 ± 0.003 | 0.596 ± 0.020 | 0.548 ± 0.006 | 0.475 ± 0.011 | 0.596 ± 0.022 | 0.580 ± 0.016 | 0.670 ± 0.017 | 0.498 ± 0.052 | |
| AOD | 0.844 ± 0.004 | 0.826 ± 0.004 | 0.885 ± 0.002 | 0.849 ± 0.008 | 0.837 ± 0.004 | 0.784 ± 0.006 | 0.844 ± 0.011 | 0.838 ± 0.008 | 0.868 ± 0.011 | 0.774 ± 0.024 | |
| Few-random shot (5) [14] | AUC | 0.810 ± 0.002 | 0.785 ± 0.005 | 0.801 ± 0.003 | 0.786 ± 0.017 | 0.837 ± 0.005 | 0.775 ± 0.002 | 0.850 ± 0.006 | 0.803 ± 0.007 | 0.810 ± 0.019 | 0.776 ± 0.005 |
| Accuracy | 0.523 ± 0.003 | 0.435 ± 0.005 | 0.513 ± 0.016 | 0.443 ± 0.014 | 0.503 ± 0.013 | 0.435 ± 0.007 | 0.570 ± 0.016 | 0.521 ± 0.018 | 0.614 ± 0.035 | 0.500 ± 0.018 | |
| AOD | 0.818 ± 0.003 | 0.782 ± 0.003 | 0.829 ± 0.009 | 0.790 ± 0.008 | 0.823 ± 0.005 | 0.770 ± 0.002 | 0.837 ± 0.005 | 0.819 ± 0.006 | 0.852 ± 0.011 | 0.765 ± 0.012 | |
| Ordinal Log. Regression [18] | AUC | 0.731 ± 0.011 | 0.729 ± 0.004 | 0.766 ± 0.004 | 0.710 ± 0.019 | 0.677 ± 0.011 | 0.628 ± 0.006 | 0.682 ± 0.009 | 0.653 ± 0.015 | 0.723 ± 0.063 | 0.643 ± 0.050 |
| Accuracy | 0.359 ± 0.010 | 0.367 ± 0.017 | 0.329 ± 0.012 | 0.371 ± 0.018 | 0.292 ± 0.018 | 0.272 ± 0.015 | 0.407 ± 0.022 | 0.294 ± 0.031 | 0.359 ± 0.025 | 0.293 ± 0.018 | |
| AOD | 0.637 ± 0.007 | 0.621 ± 0.007 | 0.766 ± 0.004 | 0.619 ± 0.012 | 0.520 ± 0.010 | 0.535 ± 0.008 | 0.635 ± 0.017 | 0.604 ± 0.023 | 0.649 ± 0.021 | 0.517 ± 0.036 | |
| DeepLV [20] | AUC | 0.815 ± 0.003 | 0.778 ± 0.012 | 0.868 ± 0.005 | 0.762 ± 0.007 | 0.775 ± 0.007 | 0.768 ± 0.007 | 0.783 ± 0.002 | 0.704 ± 0.031 | 0.675 ± 0.029 | 0.641 ± 0.029 |
| Accuracy | 0.508 ± 0.008 | 0.508 ± 0.008 | 0.553 ± 0.006 | 0.482 ± 0.011 | 0.446 ± 0.011 | 0.435 ± 0.009 | 0.383 ± 0.014 | 0.331 ± 0.036 | 0.294 ± 0.039 | 0.169 ± 0.032 | |
| AOD | 0.790 ± 0.004 | 0.752 ± 0.011 | 0.807 ± 0.009 | 0.757 ± 0.007 | 0.763 ± 0.009 | 0.701 ± 0.007 | 0.763 ± 0.010 | 0.774 ± 0.020 | 0.650 ± 0.029 | 0.620 ± 0.009 |
Table 2. Summary of the performance of the multiplex retrieval compared to other approaches (metrics are obtained from five bootstrap runs and for CodeLlama-7B)
Across our evaluated projects, most mispredictions are made by our retrieval consistently involve confusion between adjacent log levels and concentrate in a small subset of files, as shown in Table 8 in the Appendix. Specifically, we observe that adjacent-level mispredictions (e.g., confusing DEBUG with INFO, or WARN with ERROR) account for 54% (Wicket), to 81% (Karaf) of total mispredictions. Furthermore, mispredictions are heavily concentrated at the file level, with fewer than 25% of files accounting for at least half of all mispredictions in each project. This pattern suggests that predictive difficulties faced by our approach are highly localized, providing clear opportunities for targeted improvements. Finally, we note that only a median of 7.3% of the test logging statements requires a fallback route, either because the socio-technical score cannot be computed or because the cluster holds fewer than k candidates, and only a median of 0.4% reaches project-wide random retrieval (Appendix Table 7).
| Project | Adjacent Level Confusion | Files Covering 50% of Mispredictions |
| Cassandra | 64.1% | 13.3% |
| Elasticsearch | 64.7% | 16.5% |
| Flink | 78.6% | 21.8% |
| Hadoop | 72.8% | 14.7% |
| HBase | 67.0% | 13.9% |
| JMeter | 73.4% | 19.8% |
| Kafka | 66.5% | 16.8% |
| Karaf | 81.3% | 24.2% |
| Wicket | 53.7% | 23.8% |
| ZooKeeper | 80.4% | 17.0% |
Table 8. Overview of the mispredictions using multiplex retrieval

Figure RQ1. Summary of RQ1
RQ2. What is the individual contribution of functionality and ownership signals?
Motivation: The goal of this research question is to disentangle and quantify the individual contributions of functional similarity and developer ownership signals within our retrieval approach. While both signals offer theoretical utility (functional similarity capturing the context of the code, and ownership capturing the human aspect of the decision making), their comparative effectiveness remains unclear. By isolating these components, we aim to empirically determine whether these signals serve as complementary information channels or if one dimension dominates the predictive capability in log level prediction scenarios.
Approach: To disentangle and quantify the individual contributions of functional similarity and developer ownership signals, we conduct an ablation study by isolating the retrieval mechanisms. Specifically, we instantiate two distinct baselines: (a) Functionality-only retrieval, where examples are selected solely based on vector embedding similarity (as detailed in § 3.1), and (b) Ownership-only retrieval, where examples are selected exclusively from the developer’s collaboration network (as detailed in § 3.2). We then evaluate the performance of the LLMs when prompted with examples from these isolated layers and compare them against our integrated ownership-functionality aware approach presented in RQ1. We primarily report the findings for the CodeLlama-7B model in the main text and refer to the appendix for the results obtained using CodeLlama-34B.
Results: Our ownership-based clustering approach produces cohesive clusters across all the studied projects, as shown in Table 9 (Appendix). In fact, our ownership-based clusters achieve consistently high Leiden modularity scores (), ranging from 0.74 (ZooKeeper) to 0.93 (Flink). Additionally, intra-cluster cosine similarities (ranging from 0.62 to 0.95) significantly exceed inter-cluster similarities (0.02–0.27), reinforcing the internal cohesion of the identified clusters. Furthermore, the stability analysis of our clustering via the Adjusted Rand Index (ARI) indicates a moderate cluster consistency (median ARI between 0.16 and 0.59), for the six projects with sufficient history. Although Kafka exhibits relatively low stability (ARI of 0.16), its high modularity (0.86) suggests that while ownership may shift frequently, recalculating clusters periodically (e.g., monthly) would effectively sustain cohesive and useful clusters.
| Project | Modularity () | Median ARI | Intra/Inter Cosine Similarity | Coverage |
| Hadoop | 0.798 | 0.524 | 0.851 / 0.123 | 99.0% |
| HBase | 0.805 | 0.286 | 0.618 / 0.018 | 67.0% |
| Elasticsearch | 0.850 | 0.556 | 0.720 / 0.268 | 99.5% |
| Cassandra | 0.919 | 0.588 | 0.651 / 0.093 | 96.0% |
| Flink | 0.933 | 0.447 | 0.733 / 0.037 | 95.9% |
| Kafka | 0.862 | 0.158 | 0.630 / 0.032 | 95.9% |
| JMeter | 0.788 | – | 0.945 / 0.265 | 99.1% |
| ZooKeeper | 0.743 | – | 0.799 / 0.111 | 93.8% |
| Karaf | 0.829 | – | 0.899 / 0.170 | 100.0% |
| Wicket | 0.901 | – | 0.876 / 0.150 | 100.0% |
Table 9. Ownership-based clustering quality metrics. A dash indicates too few non-empty two-month windows for a reliable stability estimate.
Leveraging ownership-based retrieval significantly enhances the predictive performance across nearly all projects, as shown in Table 11 (Appendix). Specifically, providing the LLM used for log level prediction with five ownership-based ICL examples statistically significantly (Wilcoxon test, , Cohen’s , ) improves the median AUC of that LLM by 0.02 compared to random retrieval, reaching up to +0.10 on Elasticsearch. Moreover, ownership-based retrieval outperforms (Wilcoxon test, , Cohen’s , ) retrieval based on documentation-defined component in one of the two projects for which explicit components are defined in the project’s documentation. The improvements brought by ownership-based retrieval extend to accuracy and AOD scores, for which we report median improvements of 0.04 (accuracy, reaching 0.08) and 0.01 (AOD, reaching 0.03). Our reported scores account for fallback instances where log level predictions are made for files not assigned to any ownership clusters. However, the fallback rate remains minimal (below 10%) across all studied projects except for HBase, which experiences a higher fallback rate of 33%. This higher fallback rate indicates that many logging statements fall into files that either belong to small clusters (discarded as noise due to our minimum cluster size constraint of 10 files) or clusters lacking sufficient examples for complete ownership-only retrieval. JMeter is the only project where ownership-only retrieval does not improve on random retrieval in terms of AUC, although the 0.005 deficit is not statistically significant (Wilcoxon test, ).
| Model | Metric | Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
| Ownership-based (5) | AUC | 0.832 ± 0.002 | 0.808 ± 0.002 | 0.904 ± 0.001 | 0.837 ± 0.007 | 0.855 ± 0.006 | 0.785 ± 0.009 | 0.845 ± 0.012 | 0.831 ± 0.029 | 0.833 ± 0.061 | 0.789 ± 0.021 |
| Ownership-based (5) | Accuracy | 0.532 ± 0.002 | 0.510 ± 0.008 | 0.578 ± 0.006 | 0.501 ± 0.019 | 0.521 ± 0.015 | 0.461 ± 0.008 | 0.531 ± 0.012 | 0.571 ± 0.028 | 0.605 ± 0.029 | 0.554 ± 0.051 |
| Ownership-based (5) | AOD | 0.819 ± 0.001 | 0.808 ± 0.002 | 0.845 ± 0.001 | 0.824 ± 0.008 | 0.825 ± 0.008 | 0.778 ± 0.004 | 0.826 ± 0.009 | 0.832 ± 0.015 | 0.850 ± 0.011 | 0.796 ± 0.029 |
| Few-random shot (5) | AUC | 0.810 ± 0.002 | 0.785 ± 0.005 | 0.801 ± 0.003 | 0.786 ± 0.017 | 0.837 ± 0.005 | 0.775 ± 0.002 | 0.850 ± 0.006 | 0.803 ± 0.007 | 0.810 ± 0.019 | 0.776 ± 0.005 |
| Few-random shot (5) | Accuracy | 0.523 ± 0.003 | 0.435 ± 0.005 | 0.513 ± 0.016 | 0.443 ± 0.014 | 0.503 ± 0.013 | 0.435 ± 0.007 | 0.570 ± 0.016 | 0.521 ± 0.018 | 0.614 ± 0.035 | 0.500 ± 0.018 |
| Few-random shot (5) | AOD | 0.818 ± 0.003 | 0.782 ± 0.003 | 0.829 ± 0.009 | 0.790 ± 0.008 | 0.823 ± 0.005 | 0.770 ± 0.002 | 0.837 ± 0.005 | 0.819 ± 0.006 | 0.852 ± 0.011 | 0.765 ± 0.012 |
| Documentation-based Few-shot (5) | AUC | 0.808 ± 0.004 | 0.813 ± 0.005 | - | - | - | - | - | - | - | - |
| Documentation-based Few-shot (5) | Accuracy | 0.504 ± 0.004 | 0.483 ± 0.007 | - | - | - | - | - | - | - | - |
| Documentation-based Few-shot (5) | AOD | 0.806 ± 0.003 | 0.800 ± 0.001 | - | - | - | - | - | - | - | - |
Table 11. Summary of the performance of the ownership-only retrieval compared to other approaches (metrics are obtained from 5 bootstrap runs and for CodeLlama-7B). Documentation-based retrieval is only applicable to the two projects that define components in their documentation; a dash indicates the configuration is not applicable for that project.
Leveraging file ownership information enhances retrieval quality for LLM-powered LLPs, as ownership-only retrieval achieves a statistically significant improvement in AUC, with a median of 0.02 and reaching 0.10, compared to random retrieval.
Our functionality-only based clustering yields compact, cohesive, and stable clusters, as shown in Table 10 (Appendix). In fact, our clustering approach consistently identifies between 13 and 59 functionally aligned clusters across our studied projects. These clusters achieve a strong internal cohesion as the median Silhouette score clears the 0.5 cohesion threshold on nine of our ten studied projects (from 0.51 to 0.61), with only ZooKeeper marginally below it at 0.48 while still forming compact clusters (DBCV of 0.37). Across all projects the DBCV ranges between 0.22 and 0.39 (the further from 0 the better). Furthermore, stability of our clusters is high, as demonstrated by the bootstrapped ARI scores ranging from 0.60 (ZooKeeper) to 0.84 (Wicket). That said, we observe that HDBSCAN marks 17% (Karaf) to 49% (Hadoop) of the files as noise, potentially leading to fallback scenarios during retrieval (i.e., if the log level of the logging statement to be predicted is in a file labeled as noise).
| Project | Files | Best Param. (ms / ms) | # Clusters | Silh. | DBCV | Noise | Bootstr. ARI |
| Hadoop | 7,395 | 50 / 25 | 31 | 0.533 | 0.230 | 49% | 0.685 ± 0.027 |
| HBase | 6,091 | 25 / 12 | 27 | 0.566 | 0.325 | 41% | 0.690 ± 0.040 |
| Elasticsearch | 14,976 | 100 / 25 | 31 | 0.553 | 0.219 | 45% | 0.670 ± 0.050 |
| Cassandra | 3,158 | 25 / 12 | 30 | 0.606 | 0.347 | 37% | 0.700 ± 0.030 |
| Flink | 14,691 | 50 / 25 | 59 | 0.561 | 0.330 | 39% | 0.806 ± 0.025 |
| Kafka | 5,864 | 50 / 25 | 22 | 0.541 | 0.387 | 30% | 0.797 |
| JMeter | 1,402 | 25 / 13 | 16 | 0.526 | 0.349 | 31% | 0.674 |
| ZooKeeper | 951 | 18 / 5 | 16 | 0.478 | 0.366 | 28% | 0.604 ± 0.137 |
| Karaf | 1,631 | 30 / 7 | 16 | 0.505 | 0.394 | 17% | 0.782 ± 0.065 |
| Wicket | 3,324 | 50 / 25 | 13 | 0.592 | 0.303 | 27% | 0.840 |
Table 10. Functionality-based clustering quality metrics
Despite the relatively high ratio of files labeled as noise, leveraging functionality-based clusters significantly enhances LLM-powered LLPs compared to our baselines, as shown in Table 12 (Appendix). Indeed, using functionality-only based retrieval to provide in-context-learning logging examples to our LLM-powered LLPs statistically significantly improves the median AUC by 0.05 (up to 0.12) compared to random retrieval, by 0.05 to 0.06 compared to documentation-based in-context-learning retrieval (if any), and by a median of 0.02 (up to 0.06) compared to ownership-based retrieval. The improvement over random retrieval extends to both the accuracy and AOD scores, across all our studied projects.
| Model | Metric | Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
| Functionality-based (5) | AUC | 0.865 ± 0.002 | 0.863 ± 0.003 | 0.924 ± 0.002 | 0.863 ± 0.006 | 0.860 ± 0.004 | 0.831 ± 0.008 | 0.859 ± 0.008 | 0.834 ± 0.013 | 0.856 ± 0.060 | 0.798 ± 0.029 |
| Functionality-based (5) | Accuracy | 0.587 ± 0.002 | 0.568 ± 0.007 | 0.638 ± 0.008 | 0.652 ± 0.020 | 0.556 ± 0.008 | 0.541 ± 0.019 | 0.594 ± 0.012 | 0.564 ± 0.019 | 0.629 ± 0.038 | 0.523 ± 0.039 |
| Functionality-based (5) | AOD | 0.837 ± 0.003 | 0.838 ± 0.004 | 0.863 ± 0.001 | 0.868 ± 0.005 | 0.839 ± 0.004 | 0.818 ± 0.009 | 0.840 ± 0.007 | 0.840 ± 0.010 | 0.858 ± 0.016 | 0.788 ± 0.023 |
| Few-random shot (5) | AUC | 0.810 ± 0.002 | 0.785 ± 0.005 | 0.801 ± 0.003 | 0.786 ± 0.017 | 0.837 ± 0.005 | 0.775 ± 0.002 | 0.850 ± 0.006 | 0.803 ± 0.007 | 0.810 ± 0.019 | 0.776 ± 0.005 |
| Few-random shot (5) | Accuracy | 0.523 ± 0.003 | 0.435 ± 0.005 | 0.513 ± 0.016 | 0.443 ± 0.014 | 0.503 ± 0.013 | 0.435 ± 0.007 | 0.570 ± 0.016 | 0.521 ± 0.018 | 0.614 ± 0.035 | 0.500 ± 0.018 |
| Few-random shot (5) | AOD | 0.818 ± 0.003 | 0.782 ± 0.003 | 0.829 ± 0.009 | 0.790 ± 0.008 | 0.823 ± 0.005 | 0.770 ± 0.002 | 0.837 ± 0.005 | 0.819 ± 0.006 | 0.852 ± 0.011 | 0.765 ± 0.012 |
| Documentation-based Few-shot (5) | AUC | 0.808 ± 0.004 | 0.813 ± 0.005 | - | - | - | - | - | - | - | - |
| Documentation-based Few-shot (5) | Accuracy | 0.504 ± 0.004 | 0.483 ± 0.007 | - | - | - | - | - | - | - | - |
| Documentation-based Few-shot (5) | AOD | 0.806 ± 0.003 | 0.800 ± 0.001 | - | - | - | - | - | - | - | - |
Table 12. Summary of the performance of the functionality-only retrieval compared to other approaches (metrics are obtained from 5 bootstrap runs and for CodeLlama-7B). Documentation-based retrieval is only applicable to the two projects that define components in their documentation; a dash indicates the configuration is not applicable for that project.
Despite labeling a large proportion of files as noise (up to 49%), functionality-based retrieval statistically significantly enhances the predictive performance of LLM-powered LLPs, improving AUC by a median of 0.05 (up to 0.12) compared to random retrieval. This indicates that code functionality clusters—even with partial coverage—offer a strong and practical value as a retrieval signal for LLM-powered log level prediction.
RQ3. Can multiplex Retrieval reduce the computational cost of LLM-powered log level prediction?
Motivation: The goal of this research question is to investigate the trade-off between predictive performance and operational efficiency in LLM-powered log level predictors. While commercial, large-scale Large Language Models (e.g., GPT-4o, DeepSeek-V3.2) have established new state-of-the-art benchmarks in code intelligence tasks, their deployment in high-volume software engineering workflows remains constrained by prohibitive inference costs and latency. In fact, log level prediction can be frequently invoked within Continuous Integration (CI) pipelines, where analyzing a large amount of logging statements (along with their context) per commit using a commercial model would incur a large economic overhead. We hypothesize that by enhancing the retrieval mechanism, specifically through our multiplex strategy, we can empower smaller, open-weight models (e.g., 7B parameters) to achieve performance parity with substantially larger commercial models.
Approach: To validate the cost-effectiveness of our approach, we conduct a comparative cost-benefit analysis between our multiplex-enhanced CodeLlama-7B model and two leading commercial models: GPT-4o (OpenAI) 7 and DeepSeek-V3.2-chat (DeepSeek) 8. We evaluate these models on the same test sets used in RQ1 and RQ2, recording two key dimensions for each experimental run:
Predictive Performance: Measured via AUC, accuracy and AOD (Similar to RQ1 and RQ2).
Economic Cost: For commercial models, we calculate the estimated monetary cost (in USD) for reproducing
the results, based on the official API pricing schemas of the respective providers (e.g., OpenAI) at the time of the experiment. As for our open-weights models we estimate the cost (in USD) based on the hourly rate of the GPU infrastructure (e.g., NVIDIA H100) required to host the model during the inference window. This cost covers inference only. Retrieval preparation is a separate per-project cost, reported in § 3.5, and is not included here since its amortized share depends on how often a project’s maintainers refresh their clusters.
By comparing the commercial LLMs against the 7B model equipped with multiplex retrieval, we aim to determine whether smarter retrieval can substitute for raw model scale. The commercial models are used zero-shot, as five retrieved examples would multiply their input tokens and cost. This contrasts deployment configurations rather than isolating model size, which we assess with CodeLlama-7B and 34B under identical retrieval.
Results: Multiplex-enhanced CodeLlama-7B outperforms significantly larger commercial models in terms of AUC across all studied projects, as shown in Table 3. In fact, we observe that our localized 7B model achieves an average AUC of 0.89, surpassing GPT-4o (0.79) by 0.10 AUC and DeepSeek-V3.2 (0.73) by 0.16 AUC. This performance gap is even more pronounced in the largest datasets, such as Elasticsearch, where the 7B model maintains a high AUC of 0.97 compared to 0.76 for GPT-4o. These results suggest that in the domain of log level prediction, precise context retrieval is a more determinant factor for success than raw parameter scale. By feeding the model project-specific signals via multiplex retrieval, a 7B model can effectively "punch above its weight" and defeat general-purpose models that are estimated to be 20 to larger.
| Model | Cassandra AUC | Cassandra AOD | HBase AUC | HBase AOD | Hadoop AUC | Hadoop AOD | Elasticsearch AUC | Elasticsearch AOD | Flink AUC | Flink AOD | Kafka AUC | Kafka AOD | JMeter AUC | JMeter AOD | ZooKeeper AUC | ZooKeeper AOD | Karaf AUC | Karaf AOD | Wicket AUC | Wicket AOD | Average AUC | Average AOD |
| CodeLlama-34B (Multiplex) | 0.914 | 0.884 | 0.926 | 0.880 | 0.932 | 0.894 | 0.969 | 0.913 | 0.887 | 0.850 | 0.860 | 0.826 | 0.877 | 0.857 | 0.879 | 0.840 | 0.879 | 0.873 | 0.836 | 0.811 | 0.896 | 0.863 |
| CodeLlama-7B (Multiplex) | 0.907 | 0.877 | 0.914 | 0.868 | 0.921 | 0.883 | 0.966 | 0.912 | 0.875 | 0.844 | 0.851 | 0.828 | 0.871 | 0.843 | 0.866 | 0.857 | 0.879 | 0.868 | 0.827 | 0.785 | 0.888 | 0.857 |
| GPT-4o (Zero-shot) | 0.806 | 0.798 | 0.776 | 0.807 | 0.809 | 0.814 | 0.760 | 0.780 | 0.832 | 0.828 | 0.765 | 0.766 | 0.858 | 0.843 | 0.792 | 0.836 | 0.759 | 0.857 | 0.755 | 0.708 | 0.791 | 0.804 |
| DeepSeek-V3.2 (Zero-shot) | 0.669 | 0.779 | 0.693 | 0.802 | 0.679 | 0.794 | 0.595 | 0.775 | 0.807 | 0.823 | 0.746 | 0.763 | 0.840 | 0.835 | 0.788 | 0.815 | 0.749 | 0.850 | 0.741 | 0.769 | 0.731 | 0.801 |
Table 3. Predictive Performance: Local Models vs. Commercial Baselines
The CodeLlama-7B model offers the best trade-off between predictive performance, data privacy, and operational cost, as shown in Table 4. While DeepSeek-V3.2 offers the lowest absolute price for sequential processing (≈$0.07 per 1k logs), this saving imposes a large penalty of 18% in predictive AUC and requires the transmission of proprietary code to external servers. In contrast, our self-hosted 7B model maintains superior performance (0.89 AUC) and strict data sovereignty. Furthermore, by leveraging concurrent batching on our experimental hardware, we achieve a unit cost of $0.18 per 1k logs, as shown in Table 4. While slightly higher than the cheapest API, this cost remains negligible for high-value CI pipelines while mitigating the risks of data leakage and external service outages.
| Model | Strategy | Inference Method | Cost | vs. DeepSeek |
| CodeLlama-7B | Local (Batched) | $0.18 | Competitive | ||
| CodeLlama-34B | Local (Batched) | $0.76 | +1,051% | ||
| DeepSeek-V3.2 | API (Token Rate) | $0.066 | Reference | ||
| GPT-4o | API (Token Rate) | $2.44 | +3,598% |
Table 4. Operational Cost Comparison: Self-Hosted vs. Commercial APIs (per 1k logs)
Scaling up the local model from 7B to 34B parameters yields diminishing returns, as shown in Table 5. In our standardized experimental setup, we observe that while the CodeLlama-34B model achieves the highest absolute AUC of 0.90, it offers only a marginal improvement of 0.9% over the 7B model. However, this increment comes at a steep price, as the 34B model is significantly more computationally intensive, slowing down batch throughput by 4.1×. This throughput degradation directly translates to a 322% increase in operational costs (rising from $0.18 to $0.76 per 1k logs). Consequently, for CI/CD integrated log level prediction (where throughput is critical), the 7B model represents the superior efficiency choice.
| Model Size | AUC | Cost per 1k Logs | Performance Gain (AUC) | Cost Increase |
| 7B (Base) | 0.89 | $0.18 | – | – | ||
| 34B (Large) | 0.90 | $0.76 | +0.9% | +322% |
Table 5. The Scaling Analysis: Performance Gains vs. Operational Cost Increases (4x H100)

Figure Summary of RQ3. Our results show that a multiplex-retrieval enhanced 7B model outperforms commercial frontier models (GPT-4o, DeepSeek-V3.2) by up to 0.16 in terms of AUC for log level prediction, suggesting that our specialized retrieval strategy is more critical than raw parameter scale. While the budget API (i.e., Deepseek) appears cheaper for low volumes, batched local inference achieves competitive unit costs (≈$0.18/1k logs) without data transmission risks. We recommend practitioners prioritize localized, retrieval-augmented models to ensure data sovereignty and reliable log level recommendations.
Threats to validity
Internal Validity
An internal validity threat concerns our reliance on embedding models to represent files for clustering and retrieval. While we employed a state-of-the-art embedding model (CodeXEmbed) due to its proven effectiveness in capturing code semantics, any inherent biases or limitations in this model could influence the quality of our clustering outcomes, consequently affecting the performance of LLM-powered LLPs using those clusters. To mitigate this threat, we validated cluster quality using established metrics (Silhouette, DBCV, and modularity) and assessed their stability through bootstrapping and temporal re-clustering analyses. Future work could benefit from exploring different embedding approaches to further strengthen the generalizability and robustness of the results.
Construct Validity
A construct validity threat involves the selection of clustering hyperparameters, such as the resolution parameter () and cluster size thresholds (e.g., minimum cluster size). Arbitrary or suboptimal parameter choices could impact clustering results. To address this concern, we conducted specialized grid searches for each of our projects, identifying optimal parameter values based on robust clustering quality evaluation metrics, therefore ensuring our results’ consistency and reducing sensitivity to hyperparameter choices.
Another construct validity threat concerns what our metrics measure. We evaluate how well a predictor recovers the log level that a project’s maintainers assigned to a logging statement, which we treat as a reference label rather than as a certified optimum. To reduce the risk that these labels capture transient rather than settled decisions, we take levels from stable releases, following prior work [14, 18, 20]. Logging statements are frequently revised after being introduced [15], so levels observed at a stable release are more likely to reflect settled maintainer judgment than levels sampled from arbitrary commits. That said, agreement with maintainer-assigned levels does not establish that predicted levels improve debugging effectiveness or system observability. Establishing that would require a different study design, for instance measuring developer acceptance and override rates on real change requests, or comparing the diagnostic coverage of a system instrumented with predicted versus maintainer-assigned levels. Our results should therefore be read as evidence about the quality of the recommendations a log level predictor produces. We consider this a useful measure, given that inconsistent log levels are reported to be a source of maintenance effort.
External Validity
One external validity threat is our focus on ten large open-source Java projects, as findings may not generalize to smaller, proprietary, or non-Java systems. This applies in particular to our ownership signal, since ownership in these projects emerges from voluntary contribution patterns, whereas industrial systems may assign maintenance responsibility through organizational policy and produce different ownership structures. Future work could replicate our analysis across diverse software ecosystems and languages.
Another external validity threat relates to the LLMs used in our study. While our evaluation includes both open-weight (CodeLlama-7B, 34B) and commercial state-of-the-art models (GPT-4o, DeepSeek-V3.2), our investigation into local deployment focused primarily on the CodeLlama family. Although CodeLlama is a widely adopted standard for code tasks, different open-weight architectures may exhibit distinct behaviors when coupled with our multiplex retrieval strategy. A related threat is that these models were pretrained on public code and may already encode logging patterns from the studied projects. Because every retrieval strategy we compare queries the same frozen model with the same prompt and test statements, any such exposure is held constant across conditions and the reported differences remain attributable to retrieval. We cannot, however, rule out an interaction in which retrieval helps more or less on memorized code, and absolute performance on private code may differ. We mitigate this by leveraging multiple robust evaluation metrics (AUC, accuracy, AOD) and conducting statistical analyses of performance improvements. Future studies could explore different LLM architectures and their impact on log level prediction.
One final external validity threat relates to our approach’s reliance on two specific clustering signals (functional similarity and developer ownership). Other signals—such as defect density, co-change coupling, package or module hierarchy, or call-graph proximity—might also influence developers’ log level decisions and could lead to further performance gains. Our design requires that each signal be derivable consistently at file granularity and representable as an aligned layer of the same multiplex graph, a constraint that not all candidate signals satisfy. While our results show that these two signals yield significant predictive performance improvements, future research could explore incorporating additional signals to further enhance our approach’s predictive capabilities.
Conclusion
Selecting appropriate log levels for new logging statements is an important, yet challenging software engineering task. Prior studies have leveraged machine learning models to automate log level prediction. However, these approaches have either focused on local, code-driven metrics (like AST context) or used file-wide engineered features, overlooking the socio-technical knowledge of a project.
In this paper, we conducted a comparative study to investigate the impact of these signals: code functionality and developer ownership. We hypothesized that a modern LLM, guided by a novel retrieval strategy, could outperform classic models and commercial models with hundreds of billions of parameters. To test our hypothesis, we introduced ownership-functionality aware retrieval, an in-context-learning (ICL) retrieval approach that integrates functionality and ownership information simultaneously through multiplex clustering.
Our evaluation across ten popular open-source projects demonstrates the superiority of this new paradigm. We show that leveraging these combined signals for in-context-learning retrieval establishes a new state-of-the-art for LLM-powered log level prediction, achieving a median AUC between 0.83 and 0.97. This result significantly outperforms the naive LLM baseline and prior specialized supervised learning paradigms, including deep learning (DeepLV: Avg. AUC 0.76) and feature-engineered Ordinal Regression (Avg. AUC 0.69).
Our findings provide evidence that leveraging our evaluated socio-technical signals (i.e., code functionality and ownership) can have a high impact on the predictive performance of LLM-powered LLPs. We envision this retrieval-augmented approach combining both signals simultaneously being extended to other logging tasks, such as log message generation and placement recommendations, further assisting developers in producing high quality logs.
Appendix
Prompting LLMs for log level prediction
To create the log level prediction prompt we leverage the approach detailed in Figure 6. Specifically, we first parse the code to isolate the target logging instruction (e.g., logger.warn(...)) from its surrounding method. The process involves three key steps:
Extraction: We separate the log message (the text identifying the event) and the ground truth log level from the source code.
Masking: We extract the surrounding code context to provide the model with syntactic cues. Crucially, we mask the actual logging statement within this context. This prevents data leakage and forces the model to infer the level based on the logic of the code rather than simply reading the existing statement (e.g., if (
logger.isDebugEnabled())).Prompt Assembly: The final prompt is constructed by combining a specific task instruction (defining the valid output format), the masked source code context, and the isolated log message.
LLMs used in this study
For the purposes of our study, we leverage the LLMs shown in Table 6.
| Model | Provider | Architecture | Parameters (Total / Active) | Context (Window) | Access |
| Open-Weight Models | |||||
| CodeLlama-7B-Instruct | Meta | Dense Decoder | 7B / 7B | 16k | Open Weights |
| CodeLlama-34b-Instruct | Meta | Dense Decoder | 34B / 34B | 16k | Open Weights |
| DeepSeek-V3.2 | DeepSeek | MoE (Sparse Attn) | 685B / 37B | 164k | Open Weights |
| Proprietary Models | |||||
| GPT-4o | OpenAI | Undisclosed{sup}* | Unknown | 128k | API |
Table 6. Summary of Large Language Models Selected for Evaluation
Clustering quality metrics
We assess the quality of our functionality-based and ownership-based clusterings (Table 9 and Table 10) using the following metrics:
Adjusted Rand Index 9 (ARI): quantifies the agreement or similarity between two clustering solutions. ARI ranges from -1 to 1, where values close to 1 indicate strong agreement, values close to 0 indicate random clustering, and negative values indicate worse-than-random clustering [34].
Silhouette score 10: measures how tightly grouped (i.e., cohesive) points within clusters are compared to points in other clusters. It ranges from -1 to 1, with values above 0.5 indicate strong, cohesive, and well-separated clusters; values between 0.25 and 0.5 reflect moderate clustering structure; and values below 0.25 suggest weak or ambiguous clustering [32].
Davies–Bouldin index [8] (DBI): measures how well clusters are separated by comparing each cluster’s internal dispersion with its nearest-neighbour distance. It ranges from 0 to , where lower values are better. Typically, DBI scores below 1 indicate well-separated and compact clusters [8].
Density-Based Clustering Validation (DBCV) [26]: a density-aware validation metric tailored for algorithms such as HDBSCAN. DBCV combines cluster compactness (how tightly points are packed around local density modes) with cluster isolation (how deep the density valleys are between neighbouring clusters). Values range from to , with positive scores indicating well-formed, well-separated density regions, values near 0 suggesting structures that are indistinguishable from noise, and negative scores indicating badly fragmented or overlapping clusters.
Leiden modularity [35]: assesses how a graph can be partitioned into distinct clusters by providing insight into the overall strength and quality of cluster boundaries. High modularity indicates that nodes within clusters have dense internal connections and relatively sparse connections to nodes in other clusters. Values above 0.3 typically indicate a strong and meaningful community structure [11, 35].
Retrieval step coverage
Table 7 reports, for each studied project, the percentage of test logging statements served by each of the three retrieval steps described in § 3.3, together with a breakdown of the routes taken within the multiplex step.
Misprediction analysis
Table 8 characterizes the mispredictions made by multiplex retrieval. Adjacent-level mispredictions (e.g., confusing DEBUG with INFO, or WARN with ERROR) account for 54% (Wicket) to 81% (Karaf) of all mispredictions, and fewer than 25% of files account for at least half of the mispredictions in each project.
RQ2: Single signal retrievals (Ablation study)
We first report the clustering quality metrics of our ownership and functionality clustering in Table 9 and Table 10. We then report the full performance results for CodeLlama-7B ownership-only (Table 11) and functionality-only (Table 12) retrieval. Finally, Table 13 reports the impact of the different retrieval strategies on the CodeLlama-34B performance, and Figure 7 reports the sensitivity of multiplex retrieval to the number of in-context-learning examples.
Figure 7. Sensitivity of multiplex retrieval to the number of in-context-learning examples (AUC, CodeLlama-7B, five bootstrap runs). Grey lines are individual projects; the black line is the median across the ten studied projects. Gains are statistically significant up to five examples and flatten beyond it.
| Retrieval | Metric | Hadoop | HBase | Elastic. | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
| Functionality | AUC | 0.875 ± 0.001 | 0.871 ± 0.005 | 0.889 ± 0.037 | 0.859 ± 0.009 | - | - | - | - | - | - |
| Functionality | Accuracy | 0.616 ± 0.008 | 0.580 ± 0.012 | 0.644 ± 0.004 | 0.569 ± 0.026 | - | - | - | - | - | - |
| Functionality | AOD | 0.850 ± 0.004 | 0.838 ± 0.007 | 0.865 ± 0.001 | 0.844 ± 0.011 | - | - | - | - | - | - |
| Ownership | AUC | 0.824 ± 0.002 | 0.812 ± 0.006 | 0.905 ± 0.001 | 0.838 ± 0.012 | - | - | - | - | - | - |
| Ownership | Accuracy | 0.544 ± 0.004 | 0.508 ± 0.018 | 0.582 ± 0.005 | 0.504 ± 0.020 | - | - | - | - | - | - |
| Ownership | AOD | 0.822 ± 0.001 | 0.809 ± 0.007 | 0.843 ± 0.003 | 0.821 ± 0.010 | - | - | - | - | - | - |
| Multiplex | AUC | 0.932 ± 0.006 | 0.926 ± 0.003 | 0.969 ± 0.001 | 0.914 ± 0.002 | 0.887 ± 0.005 | 0.860 ± 0.004 | 0.877 ± 0.013 | 0.879 ± 0.021 | 0.879 ± 0.033 | 0.836 ± 0.013 |
| Multiplex | Accuracy | 0.727 ± 0.008 | 0.694 ± 0.008 | 0.775 ± 0.005 | 0.693 ± 0.003 | 0.589 ± 0.012 | 0.585 ± 0.009 | 0.652 ± 0.022 | 0.593 ± 0.034 | 0.699 ± 0.012 | 0.552 ± 0.049 |
| Multiplex | AOD | 0.894 ± 0.005 | 0.880 ± 0.003 | 0.913 ± 0.001 | 0.884 ± 0.003 | 0.850 ± 0.005 | 0.826 ± 0.006 | 0.857 ± 0.010 | 0.840 ± 0.018 | 0.873 ± 0.007 | 0.811 ± 0.025 |
Table 13. Impact of the different retrieval strategies on the CodeLlama-34B performance. The purpose of the 34B evaluation is to establish how our results scale with model size, and that question is answered by the multiplex row, which is the configuration we recommend. The ablation of the individual retrieval signals is a question about the retrieval mechanism rather than about model capacity, and we therefore answer it on the CodeLlama-7B model (Tables 11 and 12). A dash indicates the configuration was not evaluated for that project.
Note: For proprietary models, exact architectural details and parameter counts are not public. *While technically undisclosed, community analysis suggests GPT-4o utilizes a Mixture-of-Experts (MoE) architecture.
References
- [1]A. H. Anu, J. Chen, W. Shi, et al. An approach to recommendation of verbosity log levels based on logging intention. In IEEE ICSME, pages 125–134, 2019.
- [2]V. V. Baligodugula and F. Amsa ad. Unsupervised learning: Comparative analysis of clustering techniques on high-dimensional data. ArXiv, 2025.arxiv.org/abs/2503.23215
- [3]F. Battiston, V. Nicosia, and V. Latora. Structural measures for multiplex networks. Phys. Rev. E, page 032804, 2014.pubmed.ncbi.nlm.nih.gov/24730896
- [4]V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. J. Stat. Mech., page P10008, 2008.
- [5]T. B. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In NeurIPS, 2020.arxiv.org/abs/2005.14165
- [6]M. Chochlov, G. Aftab Ahmed, J. Vincent Patten, et al. Using a nearest-neighbour, bert-based approach for scalable clone detection. In IEEE ICSME, pages 582–591, 2022.
- [7]J. Cohen. A power primer. Psychol. Bull., pages 155–159, 1992.
- [8]D. L. Davies and D. W. Bouldin. A cluster separation measure. IEEE TPAMI, pages 224–227, 1979.pubmed.ncbi.nlm.nih.gov/21868852
- [9]M. De Domenico, A. Solé-Ribalta, E. Cozzo, et al. Mathematical formulation of multi-layer networks. Phys. Rev. X, 2013.DOI
- [10]Z. Feng, D. Guo, D. Tang, et al. Codebert: A pre-trained model for programming and natural languages. ArXiv, 2020.arxiv.org/abs/2002.08155
- [11]S. Fortunato and M. Barthelemy. Resolution limit in community detection. PNAS, pages 36–41, 2007.
- [12]Z. Gong, P. Zhong, and W. Hu. Diversity in machine learning. IEEE Access, pages 64323–64350, 2019.DOI
- [13]M. Günther, I. Mohr, B. Wang, and H. Xiao. Late chunking: Contextual chunk embeddings using long-context embedding models. ArXiv, 2024.arxiv.org/abs/2409.04701
- [14]Y. W. Heng, Z. Ma, Z. Li, et al. Studying and benchmarking large language models for log level suggestion. ArXiv, 2024.arxiv.org/abs/2410.08499
- [15]S. Kabinna, W. Shang, C.-P. Bezemer, and A. E. Hassan. Examining the stability of logging statements. In IEEE SANER, pages 326–337, 2016.
- [16]T. Kim, S. Kim, S. Park, and Y. Park. Automatic recommendation to appropriate log levels. Softw. Pract. Exper., pages 189–209, 2020.DOI
- [17]H. Li, W. Shang, B. Adams, M. Sayagh, and A. E. Hassan. A Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives. IEEE TSE, pages 2858–2873, 2021.DOI
- [18]H. Li, W. Shang, and A. E. Hassan. Which log level should developers choose for a new logging statement? EMSE, pages 1684–1716, 2017.DOI
- [19]Y. Li, Y. Huo, Z. Jiang, et al. Exploring the effectiveness of llms in automated logging statement generation: An empirical study. IEEE TSE, pages 3188–3207, 2024.
- [20]Z. Li, H. Li, T.-H. Chen, and W. Shang. Deeplv: Suggesting log levels using ordinal based neural networks. In IEEE/ACM ICSE, pages 1461–1472, 2021.DOI
- [21]J. Liu, J. Zeng, X. Wang, K. Ji, and Z. Liang. Tell: log level suggestions via modeling multi-level code block information. In ACM SIGSOFT ISSTA, pages 27–38, 2022.DOI
- [22]Y. Liu, R. Meng, S. Joty, et al. Codexmembed: A generalist embedding model family for multilingual and multi-task code retrieval. ArXiv, 2024.arxiv.org/abs/2411.12644
- [23]Z. Ma, A. R. Chen, D. J. Kim, et al. Llmparser: An exploratory study on using large language models for log parsing. IEEE/ACM ICSE, pages 1209–1221, 2024.DOI
- [24]C. Malzer and M. Baum. A hybrid approach to hierarchical density-based cluster selection. In IEEE MFI, pages 223–228, 2020.DOI
- [25]L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. ArXiv, 2020.
- [26]D. Moulavi, P. A. Jaskowiak, R. J. G. B. Campello, et al. Density-Based Clustering Validation, pages 839–847. 2014.DOI
- [27]A. Oliner, A. Ganapathi, and W. Xu. Advances and challenges in log analysis. CACM, pages 55–61, 2012.DOI
- [28]Y. E. Ouatiti, M. Sayagh, N. Kerzazi, et al. The impact of concept drift and data leakage on log level prediction models. EMSE, 2024.
- [29]Y. E. Ouatiti, M. Sayagh, N. Kerzazi, and A. E. Hassan. An Empirical Study on Log Level Prediction for Multi-Component Systems. IEEE TSE, pages 473–484, 2023.DOI
- [30]A. Pecchia, M. Cinque, G. Carrozza, and D. Cotroneo. Industry practices and event logging: Assessment of a critical software development process. In ICSE, pages 169–178, 2015.DOI
- [31]V. Rawte, A. Sheth, and A. Das. A survey of hallucination in "large" foundation models. ArXiv, 2023.arxiv.org/abs/2309.05922
- [32]P. J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math., pages 53–65, 1987.DOI
- [33]W. Shang, M. Nagappan, and A. E. Hassan. Studying the relationship between logging characteristics and the code quality of platform software. EMSE, pages 1–27, 2015.
- [34]D. Steinley. Properties of the hubert-arabie adjusted rand index. Psychol. Methods, pages 386–396, 2004.pubmed.ncbi.nlm.nih.gov/15355155
- [35]V. A. Traag, L. Waltman, and N. J. van Eck. From louvain to leiden: guaranteeing well-connected communities. Sci. Rep., 2019.DOI
- [36]Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv., 2020.arxiv.org/abs/1904.05046
- [37]J. Xu, Z. Cui, Y. Zhao, et al. Unilog: Automatic logging via llm and in-context learning. In IEEE/ACM ICSE, pages 1–12, 2024.DOI
- [38]D. Yuan, Y. Luo, X. Zhuang, et al. Simple testing can prevent most critical failures: an analysis of production failures in distributed data-intensive systems. In OSDI, pages 249–265, 2014.DOI
- [39]D. Yuan, S. Park, P. Huang, et al. Be conservative: enhancing failure diagnosis with proactive logging. pages 293–306, 2012.





