WorksComputer Science
Understanding Python Linters through a Comprehensive and Systematic Empirical Study
Abstract
Code linters, tools that automatically detect code issues, help developers accelerate development and improve code quality by identifying errors early. Unlike previous studies that focused mainly on single linters or informal comparisons, this paper presents the first large-scale empirical study of five widely used Python linters: Ruff, Pylint, Flake8, Mypy and Black, through four key user concerns: linting rule checks, speed, configuration methods, and linter preferences. We analyze 1,551 linting rules, categorize them into five types, and construct rule mappings, finding that more than 40% are shared between multiple linters, with Ruff and Pylint together covering over 80% of all rules. Pylint has the most rules in the first two categories, and Ruff leads in the last three. Next, we measure their running times on 1,000 GitHub projects under three setups (single-rule, default, all-rules), finding a consistent speed ranking from fastest to slowest: Ruff, Mypy, Flake8, Black, and Pylint. Ruff is extremely fast, with runtimes typically within one second. Then, we summarize four configuration methods and four types of configuration options from 443 available options across the five linters. While all linters support the four configuration methods, they differ in project configuration formats and flexibility in their configuration options. Finally, an analysis of 3,094 GitHub projects using the five linters reveals that over 50% use multiple linters, share strong consensus on option selections and rule suppressions, but exhibit project-specific variations in enabled-rule configurations. Our findings provide actionable insights for linter users and establish a benchmark dataset and taxonomy to support future research, such as cross-linter rule mapping, performance modeling, and automated configuration.
Introduction
Linters are automatic static analysis tools that detect code issues (e.g. stylistic issues and violations of coding standards) [34, 35, 51]. They play an important role in software development by improving developer efficiency, enforcing consistent coding practices, and ensuring code quality [37, 52]. To support different programming ecosystems, a variety of linters [2, 5, 6] are developed and widely used. For instance, Ruff and Pylint for Python, ESLint and JSHint for JavaScript.
Previous studies [28, 29, 31, 33, 34, 35, 36, 40, 41, 43, 51, 52, 54, 55, 56, 57] typically focus on how developers use a single linter but overlook a comparative analysis of linters about aspects where users face challenges, such as rule checks. With many linters available and little guidance on how to select and use linters, a systematic comparison is essential to help users make informed choices. However, such comparative study is labor-intensive, requiring deep familiarity with each tool and careful
Authors’ Contact Information: Qingyun Yang, [email protected], Nanjing University, Nanjing, Jiangsu, China; Zejun Zhang, [email protected], Nanjing University, Nanjing, Jiangsu, China; DAPENG YAN, [email protected], Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu, China; Tian Zhang, [email protected], Nanjing University, Nanjing, Jiangsu, China; Zhenchang Xing, [email protected], CSIRO’s Data61, Canberra, Australia; Yi Li, [email protected], Nanyang Technological University, Singapore, Singapore; Zhiming Liu, [email protected], Southwest University, Chongqing, China. cross-linter comparison. For example, collecting, analyzing and comparing hundreds of linting rules across linters demands substantial manual effort. Tómasdóttir et al. [51] interviewed developers to understand the adoption of ESLint and found that users struggle to understand and choose rule checks. Their findings show the need for in-depth analysis of linting rules within a linter, let alone across multiple linters, further highlighting the necessity of such comparative study.
To systematically conduct the comparative study of linters, we first select five representative Python linters with high popularity and comprehensive rule coverage: Ruff [21], Pylint [17], Flake8 [7], Mypy [11], and Black [2]. We focus on Python linters due to Python’s top ranking in popularity in recent years [1, 16]. Next, to understand the challenges users face when using linters, we examine discussions on Stack Overflow [25], which tend to focus on technical issues, and Reddit [14], which often reflects broader user experiences. We find that users are frequently confused by four aspects of these linters: rule checks, speed, configuration methods, and linter preferences. For example, due to the variety of linting rules across linters, a Stack Overflow user mistakenly believed that Mypy and Pylint cannot check undefined object [10]. In reality, Mypy excels at object usage and includes support for such checks. The result underscores the need to analyze and compare linters across these aspects, helping users correctly understand and utilize linters.
Motivated by these user concerns, our study about five Python linters addresses the following four research questions:
RQ1: What linting rules are enforced by Python linters? We first collect linting rules from linter official documentation [3, 8, 13, 19, 22] and then categorize them using a card sorting approach [49]. Among the 1,551 linting rules, 56% are unique to a single linter and no linting rules exist in all linters, reflecting linters have their distinctiveness. These rules fall into five categories: wrong syntax, wrong usage of objects, logical contradiction, style convention and best practice. Pylint has the most rules in the first two categories compared to other linters, and Ruff has the most rules in the last three. To maximize linting coverage with minimal linters, using only Ruff and Pylint can cover over 80% of linting rules. The large number of linting rules and their overlaps emphasize the importance of building linter interoperability to enable linters to work seamlessly together, reduce redundant checks and improve efficiency.
RQ2: What is the running time of Python linters on projects? To comprehensively assess the linter running time, we conduct three setups: enabling a single rule, default rules and all rules, corresponding to minimum, typical and maximum running time of linters, respectively. Based on rigorous runtime measurements on 1,000 GitHub projects under three setups, the linters consistently rank from fastest to slowest: Ruff, Mypy, Flake8, Black and Pylint, with runtimes of 5.1ms ∼ 1s, 55.9ms ∼ 2s, 50.3ms ∼ 5.5s, 61.3ms ∼ 33.7s, and 110.6ms ∼ >1000s, respectively. Except for Black and Flake8, the running times of different rules exhibit significant differences, indicating that linter running time is affected by rule selection.
RQ3: How are Python linters configured? By analyzing linter official configuration documentation [4, 9, 12, 18, 23], we identify four common configuration methods: command line, project configuration files, dedicated configuration files, and inline configuration. All five linters support these methods, reflecting a shared emphasis on flexible integration into diverse development workflows. However, they differ in the formats of project configuration files. To further examine configuration flexibility, we collect 443 configuration options and categorize them into four types: environment setup, scope control, rule specification and result handling. Ruff, Pylint and Mypy each offer extensive category support, whereas Flake8 and Black offer only 25 options each, reflecting a more opinionated and minimalist design philosophy. Given the complexity of configuration methods and options across linters, users should consider the flexibility and level of control they need for their projects before selecting suitable linters.
RQ4: What are the preferences for selecting linters, configuring linters, and choosing linting rules across projects? We collect 3,094 GitHub projects using the five Python linters. Over 50% of projects use multiple linters, reflecting a preference for comprehensive rule checks. Most projects using Ruff, Mypy, Flake8 and Black, prefer project configuration files, while projects using Pylint predominantly use dedicated configuration files. Projects’ configuration option preferences focus primarily on rule specification, followed by scope control, whereas environment setup and result handling feature less prominently among commonly configured options. For linting rules, Black restricts enabling and disabling rules, reflecting its opinionated design. While enabling rules, most linters (Pylint, Flake8, Mypy) reflect project-specific preferences; when disabling, most linters (Ruff, Pylint, Flake8) show a consensus, focusing on suppressing style convention checks. This highlights the challenge of effectively using linters and emphasizes the importance of automatically recommending personalized rule selections and configurations based on project requirements.
The contributions of this paper are as follows:
To the best of our knowledge, this is the first large-scale qualitative and quantitative study that analyzes and compares linting rule checks, speed, configuration methods, and linter preferences across five Python linters.
Our analysis reveals unique characteristics of each linter in four aspects, showing that no single linter excels in all aspects.
We offer practical suggestions to help linter users select and use linters, and outline directions for future research.
Study Design
Python Linter Selection
To ensure a comprehensive study of Python linters, we first use “linter” and “python” as keywords to search on GitHub. Among the top 300 returned projects, we identify 29 Python linters. However, analyzing all linters is impractical and unnecessary, as some have low star counts, few downloads, limited checks, or are integrated into others. For example, although isort is popular, it only sorts import statements, and linters like pycodestyle and Pyflakes have already been wrapped into Flake8. To select a representative set, we based our choices on two criteria: high popularity and broad checks. Ultimately, we select five linters: Ruff, Pylint, Flake8, Mypy and Black. Table 1 shows the GitHub star count, PyPI download numbers for 2024, and a brief description of each linter.
| Linter | Stars | Downloads | Description |
| Ruff | 39.4k | 266.5M | A fast Python linter written in Rust. |
| Pylint | 5.4k | 333.3M | Check for errors, enforce a coding standard, and look for code smells. |
| Flake8 | 3.6k | 405.8M | Check the style and quality of Python code. |
| Mypy | 19.3k | 382.2M | Check variables and functions in Python code. |
| Black | 40.2k | 536.6M | The uncompromising Python code formatter. |
Table 1. Information of Studied Five Python Linters
Python Linter in Developer Discussion
To identify the most frequently discussed aspects of linters, we turn to public developer discussion websites like Stack Overflow [25] and Reddit [14]. Reddit is included in our analysis because it hosts a wide range of developer communities, offering insights into both broad discussion and user experiences, which can complement more technical posts on Stack Overflow. We search for the names of the five selected linters as keywords, collecting the top 200 questions: 100 from Stack Overflow and 100 from Reddit. By analyzing these posts, we identify four key aspects of Python linters that users focus on, noting that a post may cover multiple aspects.
(1) Linting rule checks: Due to lots of linters and linting rules, users are often confused about the specific checks a linter performs and the differences between their rule checks, accounting for 37%. For example, in a Stack Overflow post, a user mistakenly believed that Mypy and Pylint do not check whether an object is used before being defined. In reality, Mypy excels at object usage checks and includes the rule check. It indicates that users need a systematic understanding of linting rules across different linters.
(2) Linter speed: Developers often praise specific linters for their speed or express dissatisfaction with slow ones, accounting for 21%. However, when comparing linter speeds, they typically lack reliable evidence, relying instead on personal project experiences or anecdotal feedback. For example, in a post on Reddit, users praised Ruff for its speed, with some even switching from Flake8, but no specific running times or projects were provided. Although Ruff compares the speed of different linters using linters’ default configuration on a project, relying on a single project for performance assessments is insufficient to draw reliable conclusions.
(3) Linter configuration: To meet the diverse needs of users, linters offer flexible configurations. However, the lack of a systematic understanding of these configurations often leads to user confusion regarding configuration methods and options, accounting for 38%. For example, in a post on Stack Overflow, users were unsure how to configure Mypy to ignore a single line of code. This can actually be done using the inline configuration method via comments.
(4) Linter preference: Users frequently seek guidance on preferred linter usage for most projects, including selecting linters, configurations, and linting rules, accounting for 37%. For example, in a post on Reddit, users ask about others’ preferences on linters and best coding practices.
To address users’ concerns on Python linters, we systematically analyze the following four research questions:
RQ1: What linting rules are enforced by Python linters?
RQ2: What is the running time of Python linters on projects?
RQ3: How are Python linters configured?
RQ4: What are the preferences for selecting linters, configuring linters, and choosing linting rules across projects?
EMPIRICAL ANALYSIS
RQ1: What linting rules are enforced by Python linters?
Motivation.
Understanding the rules enforced by each linter is essential for users to select suitable linters.
Approach.
We extract linting rules from linter official documentation [3, 8, 13, 19, 22], where each rule is clearly defined. In total, there are 1,551 linting rules: 863 for Ruff, 469 for Pylint, 127 for Flake8, 70 for Mypy and 22 for Black.
We then assess linting rule overlap across linters. For every rule in a linter, two annotators independently determine whether it is semantically equivalent to any rule in other linters based on their documentation. The Cohen’s kappa of 0.89 indicates substantial agreement between the annotators, and all remaining disagreements were resolved through discussion.
Finally, we classify these linting rules using a card sorting approach in two iterations [49]. In the first iteration, we randomly sample 309 linting rules with a confidence level of 95% and a confidence interval of 5 from 1,551 linting rules [48]. Two annotators independently read the description of each rule and annotate its functionality with a brief summary. They then discuss any disagreements in their summary and resolve them. After that, they work together to group the rules into categories with the corresponding definitions.
In the second iteration, they independently classify the remaining linting rules based on the predefined categories. If a linting rule does not fit the predefined category, they provide a brief summary. It is found that no new categories are needed. The Cohen’s kappa score for inter-annotator agreement is 0.78, indicating substantial agreement [42]. Finally, they discuss disagreements to reach an agreement.
Result. (1) Linting Rule Overlap: Figure 1 shows the number of linting rules uniquely shared by specific linter combinations. Among the 1,551 linting rules across five linters, 865 rules (56%) are exclusive to a single linter, and 686 rules (44%) exist in at least two linters. There is no linting rule across the five linters. The results indicate that while linters often share many same checks, each linter also maintains its own unique checks. If users want to maximize rule coverage with fewer linters, using Ruff alone covers over 50% of the linting rules. Combining Ruff and Pylint covers over 80%, and using Ruff, Pylint, and Mypy together provides near-complete coverage (98%). The rule overlap for each linter is as follows:

Figure 1. Linting Rule Overlap Across Five Linters
Ruff, with the largest number of rules (863), has 567 unique rules (66%). Among the remaining rules, the number of overlaps with the other four linters is ranked as: Pylint > Flake8 > Mypy > Black.
Pylint, with 469 rules, has 226 unique rules (48%). The remaining 243 rules overlap with the other four linters, ranked as: Ruff > Flake8 > Mypy > Black.
Flake8, with 127 rules, only has 16 unique rules (13%), indicating most of its rules can be replaced by other linters. The remaining 111 rules overlap in the order of: Ruff > Pylint > Black > Mypy.
Mypy, despite having only 70 rules, has the highest proportion of unique rules at 66% (46 rules). The remaining 24 rules overlap with Pylint, Flake8 and Ruff, with the overlap ranked as: Pylint > Ruff > Flake8.
Black, with the fewest rules (22), has 10 unique rules (45%). The remaining 12 rules are covered by three linters, in the order of: Ruff = Flake8 > Pylint. Of these, 9 rules (41%) are covered by Ruff and Flake8, 5 are covered by Pylint.
Finding 1: Across five Python linters, 56% of the 1,551 rules are unique to a single linter and 44% appear in two or more linters, with none common to all five. To maximize rule coverage with minimal redundancy, Ruff alone covers over half of all rules, Ruff and Pylint over 80%, and adding Mypy provides nearly complete coverage (98%).
Linting Rule Category across Linters: We define five linting rule categories and then analyze how each category is represented across the five linters.
Linting rule category: Table 2 presents the five categories of linting rules in five linters.
| Category | Definition | Example |
| Wrong Syntax | Code uses incorrect syntax that causes the code to fail to run or running behavior does not meet expectations. | [Flake8: F706] A return statement outside of a function/method. |
| Wrong Usage of Object | Code objects are incorrectly declared, implemented or referenced. | [Mypy:used-before-def] A variable is used before it is defined. |
| Logic Contradiction | Code expressions or statements contain contradictory logic. | [Pylint: W4701] While a loop is iterating over a list, its block simultaneously adds or removes elements from the list. |
| Style Convention | Code formatting follows stylistic conventions for users to better read code. | [Black: EmptyLine] Code does not have spurious vertical whitespace. |
| Best Practice | Code implementation follows optimal practices. | [Ruff: FURB131] Use “clear()” instead of “del” to delete the entire slice of a list or dictionary for better performance. |
Table 2. Linting Rule Category
Wrong syntax: Linter checks whether the code uses incorrect syntax that may cause the code to fail to run or the running behavior does not conform to the expected behavior. Four linters (Ruff, Pylint, Flake8, and Mypy) perform such checks. For instance, F706 in Flake8 checks the return statement is incorrectly used outside of a function or method. As another example, W0711 in Pylint checks whether code wrongly uses “or” to catch multiple exceptions. While the code can normally run, the “or” operator doesn’t work for catching multiple exceptions in a single except block.
Wrong usage of objects: Linter checks whether code objects are incorrectly declared, implemented or referenced. Four linters (Ruff, Pylint, Flake8, and Mypy) check such code issues. For example, used-before-def in Mypy checks whether a variable is used before it’s defined. For another example, E0301 in Pylint checks whether an “__iter__” method of a class is implemented without a corresponding “__next__” method.
Logic contradiction: Linter checks whether code expressions or statements contain logical contradictions. Four linters (Ruff, Pylint, Flake8, and Mypy) check such code issues. For example, SIM220 in Ruff checks whether “and” expressions include both an expression and its negation. Such an error causes the expression to always evaluate to false. As another example, W4701 in Pylint checks if there are statements that add and remove elements from a list while iterating over the list in a loop statement.
Style convention: Linter checks whether the code formatting affects readability. It mainly focuses on whitespace, naming conventions, and the comment conventions. All five linters perform such checks. For example, empty lines in Black checks whether there is spurious vertical whitespace. For another example, E741 in Flake8 checks whether variables are named “l”, “O”, or “I”.
Best practice: Linter checks whether the code functional implementation follows optimal practices. Four linters (Ruff, Pylint, Flake8, and Mypy) perform such checks. For example, FURB131 in Ruff checks whether code uses the ‘clear()’ method to remove all items instead of ‘del’ statements, as it offers better performance. For another example, E721 in Flake8 checks whether code wrongly uses ‘==’ to compare types instead of ‘isinstance()’.
Category distribution by linters: Table 3 shows the number of linting rules in each category for the five linters. Among all 1,551 linting rules, most fall into the categories of best practice, style convention and wrong usage of objects, which account for 47%, 25%, and 19%, respectively. The remaining two categories (wrong syntax and logical contradiction) account for less than 10% of all rules. The detailed rule category for each linter is as follows:
| Rule Category | Ruff | Pylint | Flake8 | Mypy | Black | Total |
| Wrong Syntax | 35 | 42 | 12 | 2 | 0 | 91 |
| Wrong Usage of Objects | 78 | 143 | 23 | 47 | 0 | 291 |
| Logic Contradiction | 35 | 22 | 2 | 1 | 0 | 60 |
| Style Convention | 213 | 65 | 71 | 12 | 22 | 383 |
| Best Practice | 502 | 197 | 19 | 8 | 0 | 726 |
| Total | 863 | 469 | 127 | 70 | 22 | 1,551 |
Table 3. Distribution of five rule categories across five linters
Ruff covers all five rule categories. Compared to other linters, Ruff has the largest number of linting rules in logical contradiction, style convention, and best practice.
Pylint also checks all five rule categories, particularly excelling in wrong syntax and wrong usage of objects, with the largest number of rules in these two categories compared to other linters.
Although Flake8 spans all five categories, it checks very few rules for logical contradiction, with only two rules in this category.
Mypy, although covering five categories, mainly focuses on wrong usage of objects and style convention, with fewer than 10 rules in each of the other three categories.
Black only covers a single category: style convention.
Finding 2: Across the five Python linters, linting rules have five categories: wrong syntax, wrong usage of objects, logic contradiction, style convention and best practice, with nearly half of the rules focused on best practice. Different linters show preferences for specific categories, suggesting that users can prioritize linters based on the types of issues they want to check in their code. Notably, Pylint leads the first two categories, and Ruff leads the last three.
RQ2: What is the running time of Python linters on projects?
3.2.1 Motivation. Linter running time is a key factor when choosing a linter, especially in CI workflows. Measuring and comparing execution time of linters helps users choose efficient linters and guides maintainers in optimizing performance.
3.2.2 Approach. When using linters, developers typically configure different linting rules based on project-specific requirements. However, due to the large number of linting rules, exhaustively evaluating all possible rule combinations is infeasible. To address this challenge, we measure running time under three setups: (i) the default rule setup representing typical running time, (ii) the all rules setup representing the upper bound on running time, and (iii) the single rule setup representing the lower bound on running time. All other configuration options remain at their default values.
To balance evaluation cost while maximizing project coverage, we randomly select 1,000 Python projects from the 10,000 most-starred GitHub projects for the default-rule and all-rules setups. The single-rule setup requires a separate runtime measurement for every evaluated rule on each project, resulting in hundreds of rule–project measurements per project. To limit the computational cost, we randomly sample 278 projects from these 1,000 projects for the single-rule setup, using a 95% confidence level and a 5% margin of error [48].
For reliable running time measurement of a linter on a project, we follow prior studies [30, 32, 39, 53, 58] and execute each linter on the project multiple times on at least three virtual machine instances. For each virtual machine instance, we perform three warm-up runs followed by ten executions. If the running time uncertainty remains above 0.05, additional virtual machine instances are launched until the uncertainty falls below this threshold. Finally, we compute the average runtime across all executions as the measured runtime of the linter on the project.
After obtaining the running time of each linter on each project, we explore linters’ running time differences. Specifically, we analyze: (i) whether five linters exhibit significant running time differences for each rule setup, (ii) whether different setups of each linter exhibit significant running time differences, (iii) whether different rules within a linter have significant running time differences, and (iv) whether the running time of a rule across different projects has high variability.
Building on previous studies [44, 45, 47], to assess significant differences in multiple groups, we first apply the Kruskal–Wallis H test [38], where a p-value indicates significant differences. If so, we further apply Scott–Knott ESD test [50] to divide the groups into statistically different clusters. This method is used in (i), (ii), and (iii). For instance, in (i), we first use Kruskal–Wallis H test to check if the running times of five linters differ significantly. If p-value, we apply Scott–Knott ESD test to divide the five linters into distinct groups based on their running times. To measure variability within a group, i.e., running times across multiple projects under the same rule, we compute the coefficient of variation (CV) [27, 46], which is applied in (iv). If CV, it is generally considered to indicate high variability [27].
Results
(1) Runtime Analysis of Linters: Figure 2 shows the running time of the five linters under three rule setups. Table 4 shows linters’ running time differences.

Figure 2. The Runtime of Five Python Linters under Three Setups
| Linter | Linter Speed Rank | Setup Runtime Rank | Sig. | CV>1 | ||
| single rule | default ruleset | all rules | ||||
| Ruff | 1 | 1 | 1 | single<default<all | ✓ | 26.7% |
| Pylint | 4 | 5 | 5 | single<default≈all | ✓ | 100% |
| Flake8 | 3 | 3 | 3 | single≈default≈all | 100% | |
| Mypy | 2 | 2 | 2 | single<default≈all | ✓ | 100% |
| Black | – | 4 | 4 | default≈all | – | – |
Table 4. Linter Runtime Differences
Minimum running time of linters: Enabling a single linting rule for each project reflects the linter’s minimum running time. As shown in the single rule setup of Figure 2 and Table 4, except for Black, which does not support enabling a single linting rule, the speed ranking from fastest
to slowest is: Ruff, Mypy, Flake8 and Pylint, operating within , , , , respectively.
Typical running time of linters: Enabling the default linting rules for each project reflects the linter’s typical runtime. As shown in the default setup in Figure 2 and Table 4, the speed ranking from fastest to slowest is Ruff, Mypy, Flake8, Black and Pylint, operating within ranges of , , , and (14min), respectively.
Note: single, default and all denote the linter’s running time with a single rule enabled, with its default rule set, and with all rules enabled, respectively. Sig. marks the significant running time difference across different rules. ‘–’ means unavailable data. CV>1 gives the percentage of rules whose running times vary widely across projects.
Maximum running time of linters: Enabling all linting rules for each project reflects the linter’s maximum running time. As shown in the all rules setup in Figure 2 and Table 4, the speed ranking from fastest to slowest is Ruff, Mypy, Flake8, Black and Pylint, operating within ranges of , , , , and (>17min), respectively.
Running time differences across three rule setups: As shown in Setup Runtime Rank column in Table 4, for Ruff, running times of all three rule setups exhibit significant differences, their speed rank is single < default < all. This indicates that their execution time scales with the number of rule checks. In contrast, for Pylint and Mypy, their speed ranking of three setups are both single < default ≈ all, indicating that running time of the two linters may be constrained by factors other than the number of rules, such as the computational overhead of specific rule checks. For Flake8 and Black (which does not support single-rule setup), their running times of all rule setups do not exhibit significant differences, indicating their running times basically are unaffected by the number of rule checks.
Runtime differences across linting rules: As shown in Sig. column in Table 4, under the single rule setup, running times for different linting rules exhibit significant differences for Ruff, Pylint and Mypy. This result implies that Ruff, Pylint, and Mypy incur different running times for specific checks, enabling performance gains when users enable only less expensive rules. In contrast, for Flake8, the running times for different linting rules do not exhibit significant differences, and therefore cannot benefit from this optimization. Particularly, since Black does not support single rule configuration, the running time of individual linting rules is unavailable.
Running time variability of a linting rule across projects: As shown in CV>1 column in Table 4, all rules in Pylint, Flake8, and Mypy exhibit high variability under different projects, whereas only 26.7% of Ruff’s rules do so. Particularly, Black does not support the single rule configuration. For Pylint, Flake8, and Mypy, even setting the same linting rule, its running time is highly project-specific. By contrast, most Ruff rules run with stable performance across diverse projects.
Finding 3: For the five linters, the speed ranking from fastest to slowest is as follows: Ruff, Mypy, Flake8, Black and Pylint, with running times of 5.1ms∼1s, 55.9ms∼2s, 50.3ms∼5.5s, 61.3ms∼33.7s, and 110.6ms∼>1000s. Except for Flake8 and Black, the running times of Ruff, Pylint and Mypy are affected by different rule checks, indicating that users can optimize their performance by rule selection.
RQ3: How are Python linters configured?
3.3.1 Motivation. With the increasing variety of configuration mechanisms, we aim to investigate which configuration methods and options are supported by Python linters.
3.3.2 Approach. We begin by summarizing the configuration methods supported by each linter based on their official documentation [4, 9, 12, 18, 23]. We then collect all configuration options across all configuration methods and categorize them using a card sorting approach, following the same procedure used for classifying linting rules in § 3.1.2.
3.3.3 Result. We summarize configuration methods and configuration options of the five linters as follows:
Note: #CL is command line configuration; #DC is dedicated configuration file; #IC is inline configuration.
(1) Configuration Method: Table 5 shows the configuration methods supported by five Python linters. There are four configuration methods: command line configuration, project configuration files, dedicated configuration files, and inline configuration. Project configuration files include three specific formats: pyproject.toml, setup.cfg, and tox.ini. The five linters support all four configuration methods. Notably, Pylint is the only linter that supports both four configuration methods and three project-level configuration formats. The configuration methods are as follows:
| Tool | #CL | Configuration File | #DC | #IC | ||
| Project Configuration File | ||||||
| pyproject.toml | setup.cfg | tox.ini | ||||
| Ruff | ✓ | ✓ | ✓ | ✓ | ||
| Pylint | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Flake8 | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Mypy | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Black | ✓ | ✓ | ✓ | ✓ |
Table 5. Configuration Methods supported by five linters
Command line configuration: Linter configurations are specified directly via command-line arguments during execution. It is suitable for one-off runs or use in continuous integration scripts, but it is difficult to maintain and not suitable for complex or long-term configurations. All five Python linters support the command line configuration.
Project configuration files: Linter configurations are defined in project configuration files such as pyproject.toml, setup.cfg and tox.ini. This configuration method streamlines tool configuration, minimizes scattered rules, and enhances cross-tool compatibility. With PEP 518 [15], pyproject.toml has become the standard for configuring Python packages, offering a one-stop shop for most project configuration needs. setup.cfg, a legacy format in the Python packaging ecosystem, remains widely used by projects [24]. tox.ini specifies the setup and execution of various tools like test runners and linters [26]. Among the five linters, only Pylint supports all three configuration formats, Mypy and Flake8 support two configuration formats (pyproject.toml and setup.cfg
Mypy, setup.cfg and tox.ini for Flake8), while Ruff and Black only support pyproject.toml. Overall, the most widely supported project configuration formats are ranked as follows: pyproject.toml > setup.cfg > tox.ini.
Dedicated configuration files: Linter configurations are defined in standalone tool-specific files. The configuration method isolates linter settings from other project configurations, improving modularity and clarity. However, using separate configuration files for different tools results in fragmentation and increased maintenance overhead due to the lack of uniformity between tools. All the five Python linters support the dedicated configuration file method.
Inline configuration: The configuration is embedded directly into the source code through comments. Inline configuration provides fine-grained control over linter behavior, but excessive use can clutter the codebase and make maintenance hard. All five Python linters support the inline configuration.
Finding 4: All five Python linters support four configuration methods: command line configuration, project configuration files, dedicated configuration files, and inline configuration, but their support for project configuration formats differs. Only Pylint supports all three formats (pyproject.toml, setup.cfg, and tox.ini), while Mypy and Flake8 support two, and Ruff and Black support only pyproject.toml. Notably, the fact that four out of five linters support pyproject.toml underscores a widespread convergence on the PEP 518 standard for centralized, cross-tool configuration.
(2) Configuration Option: Table 6 shows the distribution of four types of configuration options for each linter. The five linters have a total of 443 configuration options, divided into four categories: environment setup, scope control, rule specification, and result handling. The number of configuration options generally reflects a tool’s flexibility: Ruff and Pylint each have over 100 options, with 156 and 153 options, respectively; Mypy has over 50 options (84 options); Flake8 and Black offer fewer than 50 options, each with 25 options.
| Option Type | Ruff | Pylint | Flake8 | Mypy | Black | Total |
| Environment Setup | 7 | 7 | 2 | 11 | 4 | 31 |
| Scope Control | 12 | 9 | 4 | 10 | 9 | 44 |
| Rule Specification | 135 | 130 | 13 | 39 | 8 | 325 |
| Result Handling | 2 | 7 | 6 | 24 | 4 | 43 |
| Total | 156 | 153 | 25 | 84 | 25 | 443 |
Table 6. The Distribution of Four Types of Configuration Options
Environment setup: Options control how the linter runs, including its execution mode, resource usage, and caching behavior. There are 31 such options, accounting for 7% of all options. The five linters are ordered by the number of options as follows: Mypy > Pylint = Ruff > Black > Flake8. For example, the “jobs” option in Pylint, which is to use multiple processes to speed up execution.
Scope control: Options that define the files or code regions the linter checks. There are 44 such options, accounting for 10% of all options. Five linters are ordered by the number of options as follows: Ruff > Mypy > Pylint = Black > Flake8. For example, the “exclude” option in Ruff is used to skip a list of files during lint checks.
Rule specification: Options that define which code issues the linter checks and how to solve these code issues. There are 325 such options, accounting for 73% of all options. Five linters are ordered by the number of options as follows: Ruff > Pylint > Mypy > Flake8 > Black. For example, “select” in Flake8 is to specify rule sets linter checks.
Result handling: Options that determine how linting results are formatted, logged, or finalized. There are 43 such options, accounting for 10% of all options. Five linters are ordered by the number of options as follows: Mypy > Pylint > Flake8 > Black > Ruff. For example, “color” in Black is used to colorize the diff.
Finding 5: All five Python linters support four types of configuration options: environment setup, scope control, rule specification, and result handling, but their emphasis varies. Rule specification is richest in Ruff and Pylint, each offering at least 130 options. Scope control sees the most flexibility in Ruff and Mypy, each offering at least 10 options. Mypy leads both environment setup and result handling. For the other two linters, Flake8 and Black each provide only 25 options overall, indicating a more opinionated, minimal configuration flexibility.
RQ4: What are the preferences for selecting linters, configuring linters, and choosing linting rules across projects?
Motivation. Section 3.1 and Section 3.3 reveal the diversity of linting rules and the flexibility of the linter configurations. It is important to understand the projects’ linter preferences to guide developers in using linters better.
Approach. To understand linter configuration preferences across projects, we first collect top 10,000 most-starred GitHub Python projects. For each linter, we first examine whether a project configures a linter, and then compute the number of projects using each configuration method. For the five linters, we identify a total of 3,094 projects. Finally, we analyze the five most frequently configured options, as well as the enabled and disabled linting rules across projects.
Result. Among the 3,094 projects using the five Python linters, the popularity of the five linters used in the projects is as follows: Flake8 > Ruff > Mypy > Black > Pylint, accounting for 49.2%, 36.7%, 35.1%, 24.3%, and 21.0%, respectively. 56.6% of projects use more than one linter, indicating that many projects rely on multiple linters for comprehensive rule check. The linter preferences regarding configuration methods, configuration options, and linting rules are as follows:
(1) Configuration Method Preference across Projects: Figure 3 shows the overall distribution of configuration methods and configuration formats across all projects that use Ruff, Pylint, Mypy, Flake8 or Black.

Figure 3. Distribution of Configuration Method Preference Across Five Linters
Preference for project configuration vs. dedicated files: As shown in the left panel of Figure 3, among these projects, 57.3% configure linters using only project configuration files, 22.5% rely solely on dedicated configuration files, and 20.2% use two configuration methods. This distribution highlights projects’ inclination to centralize linter settings in a single, standardized project configuration file rather than in separate, tool-specific configuration files.
Preference for project configuration formats: As shown in the right panel of Figure 3, when projects configure linters via project configuration files, the popularity of formats follows the order: pyproject.toml > setup.cfg > tox.ini. This trend indicates that many projects consolidate around the PEP 518 style: pyproject.toml. Notably, 12.6% of projects employ more than one project-level format, most commonly pairing pyproject.toml with setup.cfg (9.5%). The mixed usage of linter configuration formats may highlight the transitional workflows: projects are likely migrating from legacy setup.cfg to pyproject.toml while maintaining backward compatibility. Configuration method preference for each linter: Figure 4 illustrates the distribution of different configuration methods across projects for each Python linter.

Figure 4. Distribution of Configuration Method Preference of Each Linter
For Ruff, Mypy and Black, the majority of projects () use project configuration. Furthermore, projects using these linters strongly prefer the pyproject.toml format, which all account for more than 50% of configurations.
For Flake8, most projects () also use project configuration, with setup.cfg, tox.ini and pyproject.toml, accounting for 45.4%, 13.6% and 3.5%, respectively. Notably, although Flake8 does not natively support pyproject.toml, 3.5% of projects use third-party plugins to enable configuration in pyproject.toml. The result further verifies projects’ preference for a single, centralized configuration file and how the plugin ecosystem compensates for gaps in native tool support.
For Pylint, we see the opposite trend, with the majority of projects () relying on separate, tool-specific configuration files. Nevertheless, among the minority that do adopt project configuration for Pylint, pyproject.toml remains the predominant format (close to 20%).
Finding 6: Most projects using Ruff, Mypy, and Black use pyproject.toml, while those using Flake8 prefer setup.cfg. By contrast, projects using Pylint predominantly rely on a dedicated tool-specific file. While the PEP 518 (using pyproject.toml) is widely accepted for its standardization and compatibility across tools, the fact that projects using different linters adopt distinct configuration methods reflects the varied preferences of developers based on the specificity and opinionated nature of the linters they use.
(2) Option Configuration Preference across Projects: The Top-5 Used Option column in Table 7 shows each linter’s five most frequently configured options. The top-5 used options appear in over 75% of projects for every linter, with Ruff, Flake8, and Black exceeding 90%, Pylint exceeding 80%, and Mypy at 78.5%. The widespread adoption reflects that projects have a strong consensus on linter configuration options.
| Linter | #Proj | Top-5 Most Used Options | Top-5 Enabled Linting Rules | Top-5 Disabled Linting Rules |
| Ruff | 1135 | line-length: 68.1% - Rule Specification; ignore: 59.4% - Rule Specification; select: 57.9% - Rule Specification; target-version: 47.6% - Environment Setup; per-file-ignores: 36.4% - Scope Control; Total: 93.7% | F: 64.4% - Wrong Usage of Objects, Best Practice and Wrong Syntax; E: 57.4% - Style Convention; I: 55.5% - Style Convention and Wrong Usage of Objects; W: 46.1% - Style Convention and Wrong Syntax; UP: 37.2% - Best Practice and Style Convention; Total: 75.2% | E501: 34.5% - Style Convention; E741: 12.8% - Style Convention; E731: 12.2% - Best Practice; C901: 10.4% - Style Convention; E402: 10.2% - Style Convention; Total: 48.3% |
| Pylint | 651 | disable: 76.0% - Rule Specification; max-line-length: 64.1% - Rule Specification; good-names: 56.4% - Rule Specification; reports: 55.2% - Result Handling; min-similarity-lines: 53.9% - Rule Specification; Total: 84.3% | old-raise-syntax: 5.4% - Wrong Syntax; indexing-exception: 5.1% - Wrong Syntax; useless-suppression: 3.2% - Best Practice; unused-import: 3.1% - Best Practice; dangerous-default-value: 2.6% - Wrong Usage of Objects; Total (10.5%) | locally-disabled: 36.6% - Best Practice; too-few-public-methods: 36.3% - Best Practice; fixme: 35.2% - Style Convention; too-many-arguments: 35.2% - Best Practice; invalid-name: 30.1% - Style Convention; Total: 62.7% |
| Flake8 | 1523 | max-line-length: 76.3% - Rule Specification; ignore: 56.9% - Rule Specification; exclude: 55.6% - Scope Control; per-file-ignores: 21.2% - Scope Control; max-complexity: 18.4% - Rule Specification; Total: 98.6% | F: 15.4% - Wrong Usage of Objects, Best Practice and Wrong Syntax; W: 15.1% - Style Convention and Wrong Syntax; E: 15.0% - Style Convention; C: 12.7% - Best Practice; B: 12.3% - Best Practice; Total: 15.6% | W503: 49.1% - Style Convention; E203: 39.3% - Style Convention; E501: 33.0% - Style Convention; W504: 17.9% - Style Convention; E402: 13.1% - Style Convention; Total: 62.9% |
| Mypy | 1086 | ignore_missing_imports: 49.2% - Rule Specification; warn_unused_ignores: 35.6% - Rule Specification; warn_redundant_casts: 32.1% - Rule Specification; python_version: 30.9% - Environment Setup; check_untyped_defs: 29.9% - Rule Specification; Total: 78.5% | ignore-without-code: 4.8% - Best Practice; truthy-bool: 3.5% - Wrong Usage of Objects; redundant-expr: 3.1% - Best Practice; truthy-iterable: 1.6% - Wrong Usage of Objects; redundant-self: 1.5% - Best Practice; Total: 6.5% | attr-defined: 0.9% - Wrong Usage of Objects; import-untyped: 0.9% - Style Convention; var-annotated: 0.9% - Style Convention; annotation-unchecked: 0.8% - Style Convention; assignment: 0.8% - Wrong Usage of Objects; Total: 2.9% |
| Black | 753 | line-length: 81.5% - Rule Specification; target-version: 46.4% - Environment Setup; exclude: 30.4% - Scope Control; include: 22.2% - Scope Control; skip-string-normalization: 10.1% - Rule Specification; Total: 95.4% |
Table 7. Top Five Most Frequently Configured Options, enabled and disabled Linting Rules in Five Linters
Note: #Proj is the number of projects using each linter; Total is proportion of projects that use at least one option from the top-5 most frequently used options, enabled or disabled linting rules across all projects; “–” denotes that the linter does not support the corresponding configuration option/rules.
For all five Python linters, 16 of the 25 top-5 options (64%) fall into the rule specification category. These mainly specify which linting rules check or not check, or finely configure linting rules. In particular, the maximum line length option appears in four of the five linters (all except Mypy). Next, five options(e.g., per-file-ignores, exclude and include) are in scope control category. These options appear in the top-5 lists of Ruff, Flake8, and Black, indicating a common need to fine-tune which files or code regions are linted. Finally, three options belong to environment setup. All three specify the target Python version, indicating that most projects default to a runtime environment and adjust this only when necessary. And only a single result handling option appears for Pylint, suggesting that its projects place additional emphasis on customizing output. Overall, we observe this option configuration hierarchy: rule specification first, then scope control, followed by environment setup and result handling. For each linter, the project option configuration preference is as follows:
For Ruff, the top-5 configured options span three categories: the rule specification (the 1st~3rd options), environment setup (the 4th option), and scope control (the 5th option), accounting for over 50%, 47.6%, and 36.4%, respectively. For the rule specification category, most projects commonly set the maximum line length (‘line-length’), followed by checking (‘select’) or not checking (‘ignore’) some linting rules. The environment setup option (‘target-version’) and scope control option (‘per-file-ignores’) are to set the Python version and to not check files for specific linting rules.
For Pylint, the top-5 configured options fall into two categories: rule-specification category (the 1st~3rd, and the 5th options), and result-handling category (the 4th option), with each used in over 50% of projects. In the rule-specification category, three options (e.g., ‘good-names’) are used to finely configure code style, and one option (‘disable’) is used to specify which linting rules not to check. In the result handling category, the ‘reports’ option is to specify whether to print a full report or only the linting messages.
For Flake8, the top-5 configured options span two categories: rule specification category (the 1st, 2nd and 5th options), and the scope control category (the 3rd and 4th options). The first three options are used by over 50% of projects, and the latter two are used by around 20%. Within the rule specification category, two options like ‘max-complexity’ are used to finely configure code style, and one option ‘ignore’ is to specify which linting rules not to check. In the scope control category, both options (e.g., ‘exclude’) specify which files to ignore.
For Mypy, the top-5 configured options fall into two categories: rule specification category (the 1st∼3rd and 5th options), and environment setup category (the 4th option). The first option is used by about 50% of the projects, and the remaining four by 30%∼35%. For rule specification category, four options like ‘ignore_missing_imports’ are used to finely configure type checking behavior. For environment setup category, similar to Ruff and Black, ‘python_version’ option is to specify Python version used to parse.
For Black, the top-5 configured options fall into three categories: rule specification (the 1st and 5th options), environment setup (the 2nd option), and scope control (the 3rd and 4th options). The 1st option is used by over 80% of projects, the 2nd to 4th options by approximately 20%∼46%, and the 5th option by around 10%. Within the rule specification category, all the two options are to finely configure code styles. For the environment setup category, similar to Ruff and Mypy, the ‘target-version’ is also to specify Python version used to parse. For scope control, similar to Flake8, the two options (e.g., ‘exclude’) are to include or exclude specific files or directories from linting.
Finding 7: Projects share common configuration priorities across the five Python linters. Rule specification and scope control dominate, reflecting common needs to customize checks and analysis scope. In contrast, environment setup and result handling feature less prominently among commonly configured options, suggesting that their settings can generally remain at defaults unless needed for Python version compatibility or reporting.
(3) Linting Rule Preference across Projects: When projects select linting rules, they usually enable and disable a list of linting rules. Therefore, for each linter, we list the Top-5 Enabled Linting Rules and the Top-5 Disabled Linting Rules across projects in Table 7.
Enabled rule preference across projects: For enabled linting rule, only projects using Ruff show a common preference: the top-5 enabled linting rules account for 75.2% of projects. And projects prefer to enable core rule groups rather than individual linting rules. However, the top-five enabled linting rules for each of the other three linters appear in fewer than 16% of projects, indicating that enabled rule selection is highly project-specific for the three linters (Pylint, Flake8 and Mypy).
Black remains an outlier, as it does not support enabling linting rules.
Disabled rule preference across projects: In contrast, disabling linting rules shows pronounced agreement for three linters: Ruff (48.3%), Pylint (62.7%), and Flake8 (62.9%) frequently suppress strict style checks. In comparison, Mypy’s top-five disabled rules appear in only 2.9% of projects.
Black still remains an outlier, as it does not support disabling rules.
Linting rule preference for each linter: For each linter, the project configuration preferences for enabled rules and disabled rules are as follows:
For Ruff, rather than enabling specific linting rules, projects tend to enable core rule groups (e.g., ‘F’, ‘E’ and ‘I’), which accounts for 75.2%. The result indicates projects prefer broad rule coverage with minimal configuration overhead. By contrast, when disabling rules, projects most often suppress strict linting rule checks in style convention category like ‘E501’ for line too long. It indicates that projects prioritize flexibility over rigid adherence in style convention.
For Pylint, projects do not show a clear preference for enabled rules: although all the top five enabled rules are individual rules, they account for only 10.5% of projects, indicating the enabled rule of Pylint is diverse and project-specific. In contrast, there is a clear preference for suppressing specific linting rules, with the top-5 disabled rules accounting for over 60% of projects. These disabled rules mainly focus on style convention and best practice.
For Flake8, enabled rules reveal no clear preference: although the five most enabled rules are rule groups, they appear in only 15.6% of projects. In contrast, projects consistently disable style convention rule checks, which make up 62.9% of projects.
For Mypy, projects do not exhibit a clear preference for enabled rules: the top-5 enabled rules are individual rules, yet they appear in only 5.0% of projects, indicating a diverse range of configurations. Similarly, there is no clear preference for disabled rules, where the top-5 disabled rules account for only 2.9%. This underscores the diverse, heterogeneous nature of Mypy configurations across projects.
For Black, unlike the other four linters, Black does not support enabling or disabling individual linting rules, reflecting its opinionated design that prioritizes consistency and simplicity in code style.
Finding 8: Among the five Python linters, only Black prohibits enabling or disabling rules, reflecting its opinionated design. Except for Black, when enabling rules, most linters (Pylint, Flake8 and Mypy) used by projects exhibit project-specific preferences. Conversely, when disabling rules, most linters (Ruff, Pylint and Flake8) used by projects show strong consensus, primarily suppressing style convention checks. This indicates that while most projects have unique linter needs, there is a consistent preference for less strict style conventions.
Discussion
Implications for Linter Users and Researchers
We further discuss valuable insights for the linter users and researchers based on the findings of our research questions.
Suggestions for linter users: When selecting a linter from the five Python linters, users should base their choice on the project’s specific needs. Ruff and Pylint are ideal for projects requiring comprehensive linting rules and high configurability, while Flake8 and Black are best for users seeking simplicity and minimal configuration overhead. Mypy, on the other hand, is particularly focused on type checking for object usage.
When speed is critical, Ruff is the best choice, offering fast linting with running time typically under one second. Pylint requires more attention, as it can take over 1,000 seconds depending on the configuration. Users must balance between configuring linting rules and performance. For the other three linters (Flake8, Black and Mypy), running times generally remain within 10 seconds.
For linter configuration, users should prioritize using pyproject.toml to ensure consistency and compatibility across multiple linters. Among the available options, they can focus on configuring rule specification and scope control, which dominate the most frequently configured options across linters. Environment setup and result handling can generally remain at their default settings unless necessary.
Future directions for researchers: As shown in § 3.1 and § 3.4, 44% of linting rules are shared across multiple Python linters, and 56.6% of projects use more than one linter. This highlights the potential for improving linter interoperability. As LLMs excel at understanding natural language, researchers can leverage LLMs to develop a unified framework that enables linters to work seamlessly together, reduces redundant checks, and enhances efficiency. Our dataset on rule overlap can serve as a benchmark. Besides, as new Python standards and tools (such as evolving versions of Python or emerging libraries) are introduced, it is crucial to examine how well linting rules cover these standards and how linters adapt to these changes, ensuring linters remain up-to-date and capable of enforcing the latest standards.
For speed, some linters, especially Pylint, are costly in terms of running time on projects. Future work could leverage our comprehensive running time dataset to pinpoint performance bottlenecks and guide targeted optimizations for performance.
Finally, projects using different linters adopt various configuration methods, and most projects have specific needs when selecting linting rules. Given the variety of linters, configuration methods and linting rules, automating configuration generation based on user requirements is essential to reduce their burden.
Threats to Validity
One internal threat to validity is the potential personal bias in categorizing linting rules and configuration methods. To mitigate this, two annotators with over five years of linter usage experience independently analyzed the data and then discussed to reach a consensus. We release the code and data [20] for public review.
An external threat lies in the scope of our study being limited to five Python linters. However, these linters are widely adopted, offer broad rule coverage, and provide distinct functionalities, making them representative. Our methodology offers a foundation for generalizing to other linters. In future work, we plan to extend our analysis to more Python linters and linters in other programming languages.
Related Work
Linters are static analysis tools that automatically detect potential code issues [34, 35, 51], helping developers maintain code quality by identifying errors early in the development process. These linters play a crucial role in ensuring code quality, consistency, and adherence to coding standards. Many popular programming languages have well-known linters, including Black [2] for Python, Checkstyle [5] for Java, and ESLint [6] for JavaScript. Current research on linters primarily falls into two categories.
The one focuses on enhancing linters’ ability to detect and fix code issues [31, 33, 36, 40, 41, 43, 54, 55, 56]. Latappy et al. [40] leveraged CodeBERT to learn coding practices, achieving good recall but low precision. Eghbali et al. [31] proposed DyLin, the first dynamic linter for Python, which detects 15 Python-specific antipatterns that are difficult to identify through static analysis. Holden et al. [36] applied a large language model to classify if code has issues or not. They found that model-based linter can achieve over 80% accuracy on code issue classification.
The other involves empirical studies on linters [28, 29, 34, 35, 51, 52, 57]. Tómasdóttir et al. [51] investigated developers’ perceptions of ESLint by interviewing 15 developers and surveying 337 participants. They identified five key reasons why developers use ESLint and highlighted that a major challenge lies in selecting rules. Tjåsa et al. [35] conducted a longitudinal case study of 15 JavaScript projects using linters. The results show that choices of linter and their configurations are rarely subjected to change once introduced to the project.
Different from previous studies that primarily focus on JavaScript and a single linter such as ESLint, our work conducts a comprehensive empirical study on five popular Python linters. We systematically compare their rule checks, performance, configuration methods, and linter preferences to help developers better select and use linters.
Conclusion and Future Work
This paper presents both qualitative and quantitative analyses of linting rule checks, speed, configuration methods, and linter preferences across five Python linters: Ruff, Pylint, Flake8, Mypy, and Black. Our findings indicate that no single linter excels in all aspects, highlighting the need for Python linter users to choose based on their specific project requirements. In the future, we plan to expand our study to include more linters and automatically select and generate configurations based on users’ requirements. The replication package [20] is available for further research.
References
References
- [1]2024. Top Programming Languages on IEEE Spectrum. https://spectrum.ieee.org/top-programming-languages-2024
- [2]2025. Black. https://black.readthedocs.io/en/stable/
- [3]2025. Black: code style. https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html
- [4]2025. Black: Usage and Configuration. https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html
- [5]2025. CheckStyle. https://github.com/checkstyle/checkstyle
- [6]2025. ESLint. https://github.com/eslint/eslint
- [7]2025. Flake8. https://flake8.pycqa.org/en/latest/index.html
- [8]2025. Flake8: Error / Violation Codes. https://flake8.pycqa.org/en/latest/user/error-codes.html
- [9]2025. Flake8: Full Listing of Options. https://flake8.pycqa.org/en/latest/user/options.html
- [10]2025. How to catch potentially undefined variables with pylint or mypy? https://stackoverflow.com/questions/78191580/how-to-catch-potentially-undefined-variables-with-pylint-or-mypy/78193468#78193468
- [11]2025. Mypy. https://mypy.readthedocs.io/en/stable/index.html
- [12]2025. mypy: configuration. https://mypy.readthedocs.io/en/stable/config_file.html
- [13]2025. mypy: Error codes. https://mypy.readthedocs.io/en/stable/error_code_list.html
- [14]2025. The official Python community for Reddit. https://www.reddit.com/r/Python/
- [15]2025. PEP 518: Specifying Minimum Build System Requirements for Python Projects. https://peps.python.org/pep-0518/
- [16]2025. The popularity of programming languages on TIOBE index. https://www.tiobe.com/tiobe-index/
- [17]2025. Pylint. https://pylint.readthedocs.io/en/stable/
- [18]2025. Pylint: Configuration. https://pylint.readthedocs.io/en/stable/user_guide/configuration/all-options.html
- [19]2025. Pylint: Message overview. https://pylint.readthedocs.io/en/stable/user_guide/messages/messages_overview.html
- [20]2025. Replication Package. https://anonymous.4open.science/r/20250527-74D1
- [21]2025. Ruff. https://docs.astral.sh/ruff/
- [22]2025. Ruff: Rules. https://docs.astral.sh/ruff/rules/
- [23]2025. Ruff: Settings. https://docs.astral.sh/ruff/settings/
- [24]2025. setup.cfg. https://setuptools.pypa.io/en/latest/userguide/declarative_config.html
- [25]2025. Stack Overflow. https://stackoverflow.com/help/searching
- [26]2025. tox.ini. https://tox.wiki/en/latest/user_guide.html#overview
- [27]Bente CD Anda, Dag IK Sjøberg, and Audris Mockus. 2008. Variability and reproducibility in software engineering: A study of four companies that developed the same system. IEEE Transactions on Software Engineering 35, 3 (2008), 407–429.DOI
- [28]Moritz Beller, Radjinoh Bolohanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. IEEE, 470–481.
- [29]Maria Christakis and Christian Bird. 2016. What developers want and need from program analysis: an empirical study. In Proceedings of the 31st IEEE/ACM international conference on automated software engineering. 332–343.DOI
- [30]Arthur Crapé and Lieven Eeckhout. 2020. A rigorous benchmarking and performance analysis methodology for python workloads. In 2020 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 83–93.DOI
- [31]ARYAZ EGHBALI, FELIX BURK, and MICHAEL PRADEL. 2025. DyLin: A Dynamic Linter for Python. (2025).DOI
- [32]Andy Georges, Dries Buytaert, and Lieven Eeckhout. 2007. Statistically rigorous java performance evaluation. ACM SIGPLAN Notices 42, 10 (2007), 57–76.
- [33]Rowan Hart, Brian Hays, Connor McMillin, El Kindi Rezig, Gustavo Rodriguez-Rivera, and Jeffrey A Turkstra. 2023. Eastwood-tidy: c linting for automated code style assessment in programming courses. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 799–805.DOI
- [34]Tjaša Heričko and Boštjan Šumak. 2023. On the Relationship Between Linter Warning Density and Software Maintainability: An Empirical Study of JavaScript Projects. In Proceedings of the 2023 6th International Conference on Software Engineering and Information Management (Palmerston North, New Zealand) (ICSIM ’23). Association for Computing Machinery, New York, NY, USA, 86–91. doi:10.1145/3584871.3584884
- [35]Tjaša Heričko and Boštjan Šumak. 2022. Analyzing Linter Usage and Warnings Through Mining Software Repositories: A Longitudinal Case Study of JavaScript Packages. In 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO). 1375–1380. doi:10.23919/MIPRO55190.2022.9803554DOI
- [36]Darren Holden and Nafiseh Kahan. 2024. Code Linting using Language Models. arXiv preprint arXiv:2406.19508 (2024).arxiv.org/abs/2406.19508
- [37]Ciera Jaspan, I-Chin Chen, and Anoop Sharma. 2007. Understanding the value of program analysis tools. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion. 963–970.DOI
- [38]William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical association 47, 260 (1952), 583–621.
- [39]Christoph Laaber, Joel Scheuner, and Philipp Leitner. 2019. Software microbenchmarking in the cloud. How bad is it really? Empirical Software Engineering 24, 4 (2019), 2469–2508.DOI
- [40]Correntin Latappy, Quentin Perez, Thomas Deguuele, Jean-Rémy Falleri, Christelle Urtado, Sylvain Vauttier, Xavier Blanc, and Cédric Teyton. 2023. MLinter: Learning Coding Practices from Examples—Dream or Reality?. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 795–804.
- [41]Benjamin Loriot, Fernanda Madeira, and Martin Monperrus. 2022. Styler: learning formatting conventions to repair Checkstyle violations. Empirical Software Engineering 27, 6 (2022).DOI
- [42]Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica (2012).
- [43]Hayatou Oumourou, Nicolas Anquetil, Anne Etien, Stéphane Ducasse, and Kolyang Dina Taiwe. 2015. Identifying the exact fixing actions of static rule violation. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 371–379.
- [44]Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology. 1–22.arxiv.org/abs/2304.03442
- [45]ShaoJian Qiu, Lu Lu, Ziyi Cai, and Siyu Jiang. 2019. Cross-Project Defect Prediction via Transferable Deep Learning-Generated and Handcrafted Features. In SEKE. 431–552.DOI
- [46]George F Reed, Freyja Lynn, and Bruce D Meade. 2002. Use of coefficient of variation in assessing variability of quantitative assays. Clinical and Vaccine Immunology 9, 6 (2002), 1235–1239.
- [47]Lei Sheng, Lu Lu, and Junhao Lin. 2020. An adversarial discriminative convolutional neural network for cross-project defect prediction. IEEE Access 8 (2020), 55241–55253.DOI
- [48]Ravindra Singh and Naurang Singh Mangat. 2013. Elements of survey sampling. Vol. 15. Springer Science & Business Media.
- [49]Donna Spencer. 2009. Card sorting: Designing usable categories. Rosenfeld Media.
- [50]Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization for Defect Prediction Models. (2018).arxiv.org/abs/1801.10270
- [51]Kristín Tómasdóttir, Maurício Aniche, and A Van Deursen. 2020. The Adoption of JavaScript Linters in Practice: A Case Study on ESLint. IEEE Transactions on Software Engineering 46, 8 (2020).
- [52]Kristín Fjola Tómasdóttir, Mauricio Aniche, and Arie Van Deursen. 2017. Why and how JavaScript developers use linters. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 578–589.DOI
- [53]Luca Traini, Daniele Di Pompeo, Michele Tucci, Bin Lin, Simone Scalabrino, Gabriele Bavota, Michele Lanza, Rocco Oliveto, and Vittorio Cortellessa. 2021. How Software Refactoring Impacts Execution Time. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 2 (2021), 1–23.DOI
- [54]Nikolaos Tsantalis, Theodoros Chailis, and Alexander Chatzigeorgiou. 2008. JDeodorant: Identification and removal of type-checking bad smells. In 2008 12th European conference on software maintenance and reengineering. IEEE, 329–331.DOI
- [55]Kuldeep Vayadande, Kuhu Mukhopadhyay, Vaishnavi Chaudhari, Shreyas Manwadkar, Tanmay Mutalik, and Ishan Gawali. 2023. Let us lint: a tool for code formatting and code enhancing. In 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT). IEEE, 1–8.DOI
- [56]Andrea Walchshofer, Stefan Fischer, Albrecht Wöß, Alejandra Duque-Torres, Markus Löberbauer, and Georg Koll. 2024. Introducing a Linter in an Industrial Lab Code Base. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering-Companion (SANER-C). IEEE, 191–198.DOI
- [57]Fiorella Zampetti, Simone Scalabrino, Rocco Oliveto, Gerardo Canfora, and Massimiliano Di Penta. 2017. How open source projects use static code analysis tools in continuous integration pipelines. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 334–344.DOI
- [58]Zejun Zhang, Zhenchang Xing, Xin Xia, Xiwei Xu, Liming Zhu, and Qinghua Lu. 2023. Faster or slower? performance mystery of python idioms unveiled with empirical evidence. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1495–1507.arxiv.org/abs/2301.12633