WorksComputer Science
DOCKET: A Pluralistic Benchmark for Evaluating AI-Assisted Policy Feedback in U.S. Rulemaking
Abstract
As AI models advance and impact deliberative democracies, there is a growing need to assess their support of pluralistic viewpoints. Most pluralism benchmarks are insufficient for this: they are not grounded in diversity (i.e., stakeholders across different frames, skills and needs), quality (i.e., rigorous and technical debates), and impact (i.e., invested deliberators and real-world outcomes). To address this, we first (i) curate DOCKET, the most comprehensive dataset of the U.S. Federal Notice & Comment process to date. DOCKET consists of 42.3M public comments on 22,400 regulations across 44 government agencies between 2016 and 2026 and contains millions of policy debates; we link public comments to particular policy items proposed by regulators to track the public's attention and the responsiveness of regulators to public input. (ii) Next, we demonstrate the pluralistic utility of DOCKET - we show that public commenters have a wide breadth of perspectives with significant depth in policy expertise and that regulators are often responsive. (iii) Finally, we present DOCKETBench, the first systematic evaluation of LLMs' ability to represent human policy perspectives in federal rulemaking. Only 18-30% of LLM-generated feedback aligns with real human comments. LLMs critique proposals broadly but generally, lacking the specific grounding, citations, and lived-experience arguments that humans bring to bear; in addition, most models we test systematically agree with government perspectives (overlapping with up to 68% of their claims) and agree least with labor unions (as little as 26% of their claims), demonstrating a fundamental skew in perspectives that even tools like persona modeling do not solve. Taken together, we reveal fundamental gaps in how language models navigate real policy issues, and DOCKETBench serves as a testbed for closing those gaps in a crucial real-world domain.
Introduction
Generative AI has become widespread in government interactions with the public, from official communications [2, 41] to court filings [48]. This paper focuses on one such consequential setting: notice-and-comment (N&C) rulemaking, which has been described by legal scholars as “one of the greatest inventions of modern government” [14]. Through N&C, agencies translate legislation into regulations by publishing proposed rules, inviting the public to submit “data, views, or arguments,” and considering submissions before finalizing them [25, 45]. Language models are beginning to assist – both by (1) helping citizens draft comments [6, 46] and by (2) helping agencies analyze
Authors’ Contact Information: Alexander Spangher, [email protected]; Charlotte Peale, [email protected]; Síĺvia Casacuberta, [email protected]; Victor Y. Wu, [email protected]; Mirac Suzgun, [email protected]; Olivia H. Martin, [email protected], Stanford University, Stanford, California, USA; Dan Bateyko, [email protected], Cornell University, Ithaca, New York, USA; Cassandra Handan-Nader, [email protected], New York University, New York, New York, USA; Diyi Yang, [email protected]; Sanmi Koyejo, [email protected]; Daniel E. Ho, [email protected], Stanford University, Stanford, California, USA. submissions, refine proposals, and anticipate public response [10, 22, 23, 37, 47]3. Yet scholars and policymakers lack systematic evaluations of this assistance.
Simultaneously, there is growing awareness that AI models, specifically when attempting tasks in these kinds of settings, must grow and adapt to pluralistic beliefs, values and preferences to negotiate the needs of a diverse, deliberative public [12, 26, 43]. Most pluralistic alignment work is evaluated on synthetic disagreements or stated preferences elicited in isolation [4, 16, 21], leaving open how models should handle the real-world, high-stakes pluralism.
To address both needs, we introduce an empirical map of the rulemaking process itself: who participates, what claims they raise, which provisions they target, and how agencies respond. We introduce DOCKET: the largest and, to our knowledge, most comprehensive dataset of rulemaking and public comments. DOCKET tracks 44 U.S. federal agencies across 10 years (2016–2026) and consists of 87,005 notices, 17,198 proposed rules, 42.3M public comments from 16 different federal commenting systems, and 22,400 finalized rules.4 To understand the role of comments on rulemaking, we identify specifically which rule provisions the public commented on, and which comments the agency responded to. We use LLMs to extract provisions from all rules (676K provisions); claims from all comments (13.6M claims); and responses (94K responses) from all finalized rules. We develop a high-performance matching pipeline (claims provisions) and (responses claims). This approach allows us to track what members of the public care about, which comments the government considers deserve a response, and how these comments affect rulemaking. Overall, we find that public commenters engage seriously with regulations ( bring specialized knowledge), target specific provisions rather than entire rules, and that agencies take up commenter language at meaningful rates (i.e. when rewriting provisions for final rules, agencies directly adopt wording from commenters 11–26% of the time, and paraphrase another 12–32% of the time).
For these reasons, DOCKET is an ideal dataset for evaluating the pluralism and policy skills in LLMs. We identify three characteristics driving real-world pluralism: 1) Diversity (or breadth): DOCKET comments come from a large range of human policy perspectives (e.g., industry, advocacy, individuals, law firms, labor unions); 2) Quality (or depth): comments exhibit deep policy expertise on specific topics, and 3) Impact: comments are consequential – agencies meaningfully change their policy recommendations directly based on public comments. If pluralistic AI can faithfully represent the public in this setting, it offers a template for AI-mediated deliberation more broadly.
With these insights in hand, we frame our final question: as LLMs become integrated more into our social and political lives, what roles will they play, what feedback will they give, and who will they represent? We design a policy benchmark with the simple goal: “I am proposing this rule, how should it change?” We evaluate LLM responses on two primary dimensions: (1) the substance of LLM comments (which provisions they target and what claims they raise), and (2) the style and framing of LLM comments. Although LLM-feedback has been studied in a range of settings (e.g., writing [29], voting [1, 40], and broader public-opinion simulation [17]), regulatory commentary is particularly challenging because of the technical rigor and high-stakes legal impact of the process. Indeed, in DOCKETBench, we find that LLM responses tend to be less specific, evidence-based, and technical than human responses, and they do not reflect the broad spectrum of human responses. We find that, across a range of different scaffolding (e.g., persona-driven responses [31], RAG [30]), most models we test side with government and industry commenters most often and with labor unions least often, and they agree with every group less than human commenters (Table 2). Looking forward, we hope DOCKET can serve as a test-bed for civic and deliberative AI.
Dataset and Methodology
The DOCKET dataset consists of 126,603 regulatory documents (87,005 notices, 17,198 proposed rules, 22,400 finalized rules) and 42.3M public comments across 30,365 dockets.5 We collect data from 44 agencies from 2016–2026, enabling analysis of federal regulatory processes spanning four presidential administrations.6 Of these, 16,632 are complete rulemaking dockets (containing both a proposed rule and a finalized rule). Figure 1 shows aggregate counts for the number of government documents and comments per agency. To enable a broad analysis and capture a wide distribution of views, we selected 44 agencies7 sharing the following characteristics: (1) topical similarity: some agencies have similar mandates, enabling future study on finer-grained institutional differences within agencies (e.g., Fish and Wildlife Service (FWS), Bureau of Land Management (BLM) and Environmental Protection Agency (EPA)); (2) diverse cultures: other agencies are notable in having well-documented institutional cultures (i.e., Immigration and Customs Enforcement (ICE) and the Federal Trade Commission (FTC)); and (3) different subpopulations served: while all agencies bring specialized knowledge to bear to serve the public, agencies interact with different populations (e.g., National Science Foundation (NSF) primarily interacts with scientists while Bureau of Land Management (BLM) interacts more with farmers.).

Figure 1. Agency-level overview of DOCKET. DOCKET is composed of proposed rules, comments and finalized rules from 44 agencies, grouped under 19 parent agencies. For each agency we show (i) the number of government documents (notices, proposed rules and finalized rules), (ii) the number of public comments, and (iii) the match rate, or the share of comment clusters that our pipeline links to at least one agency response estimated from a uniform random sample of 1K candidate pairs per agency judged by gpt-5-mini (n/a where we did not sample). Comment volumes range from 18 (NIH) and 869 (MSHA) to over 1M (FDA, CMS, FWS) and 30.6M at the FCC. Differences in match rate across agencies reflect how substantive the comments are and how much response burden the agency takes on under the APA. The numbers are those of Table 4.
Data Sources
The primary repository used by the U.S. federal government to collect all the federal records and most public comments is Regulations.gov [50].8 However, not all agencies send comments to regulations.gov [51], including agencies that typically receive large comment volumes (e.g., FCC and SEC [49, 52]). As such, in a large-scale effort, we built custom scraping pipelines for 16 distinct federal data sources: regulations.gov and 15 additional agency-specific systems,9 which is, to our knowledge, the most comprehensive effort to collect open N&C data. To validate our pipelines, we built additional scrapers to obtain aggregate counts published by agencies and confirm we retrieve of the public comments and documents available (Appendix § 9.§ 1). Only of those comments survive downstream feature extraction in usable form, as some government records were unreachable or PDFs unparsable.10 DOCKET and DOCKETBench, including the extracted provisions, claims and responses, the validated matches, and the benchmark generations and judgments, are available at https://huggingface.co/datasets/docket-dataset-bench/docket-bench.
Dataset Processing
We aim to measure changes in rules and the actions of commenters through the N&C process. To do this, we perform multiple matches, shown in Figure 2: (1) comment proposed rule tells us which parts of the proposal a comment is discussing; (2) response comment tells us which comments the agency is responding to and (3) comment final rule: tells us which parts of the final rule changed as a result of the comment. We perform each match as follows:

Figure 2. (a) Structure of provisions, claims and responses in proposals, comments and final rules. Added and deleted represent provisions removed or added throughout N&C. (b) Total counts of each in DOCKET. Links are counted after the gpt-5-mini judge (Matching, below). The last three rows align each proposal provision with the final rule in the 6,954 dockets that have both documents.
Parsing: We use an LLM11 to extract provisions from proposed and finalized rules. Specifically, we extract imperative provisions (e.g., “employers should report”; “brochures must contain”; “demonstrate acceptable performance”) because these focus on the discrete effects of a rule and how it impacts people [20, 28].12 We use prompts by Leahey [28] to extract 301,892 provisions from 17,198 proposed rules and 374,026 provisions from 22,400 finalized rules. Similarly, we use an LLM to extract responses from final rules: we extract 94K responses from 22,400 finalized rules.13 Finally, we cluster comments using the MinHash algorithm [8] to blunt large-scale comment campaigns [19], and we use an LLM to extract claims from representative comments from each cluster (we will use “comments” and “clusters” interchangeably). A claim is a specific statement about an existing or hypothetical provision. We extract 13.6M claims from the comments with analyzable text content.14
Matching: With parsed provisions, responses and comments on hand, we construct a matching pipeline.15 We design a bi-encoder → cross-encoder → LLM pipeline [24, 36, 39] to match straightforward pairs with scalable models while using more powerful models for uncertain pairs. See Figure 2b for full counts of each linking step. To construct our bi-encoding (BE) match, we embed provisions, responses and claims [3].16 Then, we query our retrieval indices and retrieve the top- pairs in each index: for (responses → comments), we query our claim indices with each response and retrieve the top-500 claims; for (claims → proposed provisions), we query our claim indices with each provision to retrieve claims.1718 We silver-label 500–3,000 pairs per agency with gpt-5-mini (48K pairs in all) and train agency-specific and pooled ModernBERT cross-encoders (CE) on them. Our CE reaches F1 0.69 on 2,982 held-out pairs (0.73 in cross-validation), up from 0.49 for the bi-encoder alone. Per-agency F1 is 0.73–0.81 at FAA, CMS, OSHA and USCIS and below 0.1 at AMS, DOE, DOJ, FDA, FS and NHTSA, which have 1–25 positive validation pairs. We send every pair the CE scores above 0.20 to gpt-5-mini to perform a final validation on, using the opposing-sides prompt in Appendix § 11.§ 3. To validate this labeling, we use a stronger language model, Claude Sonnet, to score 1,000 of these pairs; we find that Sonnet agrees with gpt-5-mini 92.9% of the time. All experiments are conducted on a cluster of eight b200 Nvidia GPUs. In summary, as shown in Figure 1, we link 18% of all comments to agency responses; depending on the agency, about 27–87% of the comment clusters get an agency response. 57% of provisions from proposed rules survive in final rules (Figure 2b).
External matching validation: To validate our matches, we test how well we retrieve matches identified by Handan-Nader [19] in the Federal Communications Commission’s (FCC) 2017 “Restoring Internet Freedom” proceeding. Because this docket received an extraordinary volume of comments (25M, many from mass campaigns19), the FCC took the uncommon step of directly citing unique comment IDs in their responses. Handan-Nader recovered 1.2k matches between comments and responses. By applying our pipeline, we recall 88% of these matches and discover 9.2x more responses beyond those cited20. Gpt-5-mini as a judge [56] agreed with 96% of the citation-matched labels. Additionally, we check LLM labels by hand: the claim-pair judge (52 blind pairs; agreement 0.87, 0.80) and the direct-uptake classifier (70 blind cases; 80% confirmed) are both reported in the appendix. Finally, human annotators validated 50 labels from each of eleven pipeline steps, from provision matching to the benchmark’s claim-pair judge (Appendix § 9.8); the model’s labels were validated with 95% accuracy.
Analysis
We present an analysis of our dataset DOCKET to motivate the need and design of our policy-feedback benchmark and we establish three properties of pluralism that prior pluralism benchmarks lack [16, 26]: (1) Diversity (or Breadth): stakeholder concerns diverge sharply across the corpus; (2) Quality (or Depth): commenters tend to focus on specific provisions rather than broad reactions to rules; and (3) Impact: agencies respond to commenters and take up their wording when finalizing provisions in rules. Together, these findings establish that public comment is a structured, impactful artifact rather than generic preference expression. Any LLM tool deployed in this loop must preserve stakeholder heterogeneity to avoid distorting the policy-feedback record.
Finding #1: Comments have Impact. Agencies are broadly responsive to, and take up the wording of, commenters’ claims. We find that, in their responses to comments (Figure 2) agencies express agreement with 25% of comments they respond to, partial agreement with 11% and disagreement with 35% (using LLMs to assess)21. Agencies also update rules in ways that directly incorporate literal words, phrases and claims made in comments. Using gpt-5-mini, we annotate matched pairs of comment modified provisions, consisting of (proposal, claim, final rule) tuples; and comment added provisions consisting of (claim, final rule) tuples with one of the following labels: “directly taken up”, “responsive”, “topic only”, and “opposed” (where the agency rejects the suggestion)22. As shown in Figure 5, 10.8% of comments on modified provisions and 26% comments on added provisions have their wording taken up directly into the rule; an additional 11.8% and 32%, respectively, are classified as responsive—where the agency engages the substance of the comment but rephrases it in its own language. Perhaps unsurprisingly, the comments agencies express agreement with (or partial agreement) have their wording taken up 74% (and 53.1%) of the time while comments agencies express disagreement with (or simply acknowledge) are taken up 2.6% (and 2.5%) of the time. We validated these labels with human annotators: in a blind review of 70 cases (51 labeled direct uptake, 19 distractors), 80% of the direct-uptake labels held up as language taken from the comment, 86% if we count close paraphrases (Appendix § 9.§ 7). In sum, comments substantially affect the trajectory of regulations.
Figure 5. Distribution of agency-uptake labels for 1,000 sampled comment→provision pairs, split by whether the provision was modified from its proposed form () or newly added between proposal and final rule (). Direct Uptake = final rule adopts specific phrasing from the comment that was absent in the proposed rule; Responsive = same concern addressed in the agency’s own words; Topical Only = same topic but no evidence of influence; Opposed = agency rejected the suggestion. Added provisions show notably higher direct-uptake and responsive rates (26.0% and 32.0%, respectively) than modified provisions (10.8% and 11.8%), consistent with newly-introduced rule text tracking commenter language more closely than pre-existing language being amended.
Finding #2: Comments are Diverse: Stakeholder concerns diverge sharply. Different subgroups in our population, defined above, express substantively different concern frames across all agencies and years. Table 1 reports the distribution of primary frames, assigned one per comment, across the seven subgroups. The contrasts are large: industry comments tend to be primarily concerned with Economic issues (63% ± 18% per agency); law-firm comments concentrate (67% ± 17%) on Legal concerns; labor unions on Safety (39% ± 30%) and Economic (30% ± 28%); expert organizations on Health (29% ± 17%). Individual commenters are by far the most diffuse. These divergences are not artifacts of topic mix: even within the same docket, stakeholder groups frame their concerns differently. Across 292 dockets where at least two stakeholder types each contributed ≥5 claims, the mean within-docket Jensen–Shannon divergence between org-type pairs is 0.18 (1,280 pairs), compared to 0.42 for cross-docket comparisons—confirming that roughly half the divergence reflects genuine stakeholder perspective differences rather than docket-topic selection.
| Environmental impact | Public health | Equity & justice | Economic cost-benefit | Legal & constitutional authority | Safety risks | Scientific & technical evidence | Procedural objections | Other | |
| Law firm | 3.1 | 0.8 | 2.7 | 12.8 | 67.4 | 0.7 | 4.8 | 3.7 | 3.9 |
| Industry | 3.4 | 4.1 | 0.2 | 63.0 | 6.3 | 2.0 | 14.7 | 1.2 | 5.2 |
| Labor union | 1.3 | 5.3 | 6.1 | 30.4 | 3.2 | 38.9 | 0.7 | 1.3 | 12.7 |
| Expert org. | 6.9 | 29.1 | 4.0 | 25.2 | 6.6 | 2.1 | 13.6 | 0.8 | 11.7 |
| Government | 22.8 | 6.2 | 3.1 | 19.3 | 18.1 | 2.3 | 8.3 | 5.9 | 14.0 |
| Advocacy | 43.9 | 10.5 | 7.4 | 10.0 | 8.6 | 2.0 | 3.0 | 0.4 | 14.1 |
| Individual | 34.8 | 20.2 | 11.9 | 7.1 | 4.7 | 1.4 | 0.2 | 0.2 | 19.5 |
Table 1. Stakeholder concerns: the focus of comments across stakeholder groups is shown (in percentages, %). Stakeholder concerns diverge sharply – industry concentrates 63% on Economic, law firms 67% on Legal/Constitutional, labor unions 39% on Safety; individuals are most diffuse. See Appendix § 9.§ 6 for tax construction, mutual-exclusivity discussion, and examples of each frame. We assigned one frame to each claim; frames that do not exceed 5% in any group (e.g. religious/moral, educational and unknown labels) we represent them as “Other”.
These patterns provide large- empirical support for a long-standing prediction in administrative law. Wagner [53] theorizes that well-resourced participants—industry and law firms—dominate the notice-and-comment process through “information capture”: by framing their input in the technical and legal registers that agencies are institutionally obligated to address, they shape the informational environment in which rules are made. Our data confirms this at scale: industry concentrates 63% of its claims on economic cost-benefit framing (the register most directly tied to Executive Order 12866 cost-benefit review), while law firms concentrate 67% on legal/constitutional framing (the register that triggers agency obligations under the APA). By contrast, individuals—the most numerous commenters—scatter across all frames, diluting their informational influence. Mendelson [33] argues that agencies should nonetheless take these diffuse individual comments seriously as expressions of democratic preference, yet acknowledges the structural challenge: individual commenters’ concerns are less concentrated and harder for agencies to systematically engage with. The size of these effects matters for DOCKETBench: how accurately a system carries each subgroup’s concerns is a natural evaluation axis for any system that aims to summarize, anticipate, or generate public input using LLMs.
Finding #3: Comments are High Quality: Commenters demonstrate expertise and focus on specific provisions in rules. Contrary to scholarly belief that public comment periods are swamped with low-quality comments [13, 33], our results show a high degree of specialized, evidence-bearing, nuanced responses.23 We use an LLM to label each comment as coming from either a specialist or a generalist and to then determine the subgroup of the commenter (i.e., Individual, Law firm, Industry, Labor union, Advocacy NGO, Expert, Government).24 As shown in Figure 3, 30% of the comments in our dataset contain technical aspects: addressing specific provisions, citing statutes, etc. Specialists are from every affiliation: nearly 1 in 10 comments from type individual are technical.
Figure 3. Stakeholder engagement and specialist density.

Figure 3a. How commenters engage with provisions: Arguments used and their technical specificity.

Figure 3b. Who the commenters are: specialist within commenter pools from different organizations.
Finally, we observe that commenters are often targeted and specific in their input. Each engaged provision is addressed by an average of 7.8 and a median of 2 comments. A small fraction of provisions attract the bulk of attention while the rest receive only one or two distinct claims. General claims (e.g., “hands off our Medicare”) would match many provisions and produce a much flatter distribution; instead, we observe commenters intentionally naming and responding to particular regulatory provisions. The implication for benchmarking is direct: rules cannot be treated as a single unit of prediction. Any system that summarizes, anticipates, or generates public input must operate at the provision level and predict not just which provisions attract comment but how much.
Can LLMs Reproduce Pluralistic Public Comment?
As we imagine a future where AI becomes more involved in government, whether it be through informing policymakers or assisting decision-making and critiquing policy, we must ensure AI is pluralistically aligned, or accurately represents diverse human perspectives [43]. Unlike existing pluralism benchmarks [17, 26], which typically measure whether LLMs can represent diverse general human perspectives, legal pluralism offers a challenging and grounded alignment domain: legal debates can be specific and technical, involving groups and stakeholders that are seldom considered by model designers. To address this, we introduce DOCKETBench, a benchmark that leverages DOCKET to create a challenging legal pluralism benchmark measuring the breadth of perspectives expressed in policy deliberations. The challenge of DOCKETBench is: how closely can an LLM’s critiques resemble the breadth of human comments received by a real federal rule?
DOCKETBench challenges LLMs to align with humans, pluralistically, along four primary dimensions (Table 3): (A) Provision Targeting (i.e., which provisions in proposals get attention) (B) Claim Alignment (i.e., what claims are raised and what stances are taken) (C) Frame Alignment (i.e., how concerns are framed) and (D) Style Alignment (i.e., what style comments are written in). For each of these four dimensions we report precision (the share of what the LLM wrote that some human also wrote) and recall (the share of what humans wrote that the LLM also wrote; specifically, Overton recall as defined by [38]). Finally, we challenge models to be steerable: a pluralistic LLM should, and be able to be induced to, accurately represent viewpoints of particular stakeholder subgroups (e.g., industry representatives, advocacy organizations, affected individuals). DOCKETBench is descriptive rather than normative: we measure LLM performance against what different subgroups in our data actually expressed, but make no claims that human comments are correct in legal or normative terms. Our benchmark focuses on the question of whether models can faithfully represent the perspectives held by stakeholders in federal policymaking, a necessary prerequisite for tools that would help (1) agencies understand and anticipate public response and (2) members of the public participate more readily in democratic discourse.
| (A) Provision targeting | (B) Claim alignment | ||||||||
| Model | LLM-commented provisions humans also hit (%) | Human-commented provisions LLM also hit (%) | LLM claims humans also made (%) | Human claims LLM also made (%) | (C) Argumentative frame | (D) Writing style | |||
| gpt-4o-mini | 58 | 92 | 20.5 | 27.0 | Model | LLM frames humans also used (%) | Human frames LLM also used (%) | LLM styles humans also used (%) | Human styles LLM also used (%) |
| +persona | 56 | 95 | 21.0 | 53.8 | gpt-4o-mini | 71.6 | 47.1 | 64.8 | 58.9 |
| gpt-5-mini | 47 | 91 | 19.6 | 40.4 | gpt-5-mini | 78.8 | 55.8 | 76.0 | 61.9 |
| +persona | 49 | 98 | 19.8 | 58.7 | +fewshot | 78.3 | 54.8 | 74.1 | 60.8 |
| +fewshot | 47 | 97 | 19.9 | 32.6 | +RAG | 76.9 | 50.4 | 78.8 | 62.7 |
| +RAG | 49 | 89 | 21.2 | 31.7 | +tool | 77.6 | 52.1 | 76.7 | 62.9 |
| +tool | 45 | 89 | 20.9 | 30.5 | gpt-5 | 81.9 | 59.6 | 80.1 | 61.3 |
| gpt-5 | 47 | 95 | 18.1 | 37.8 | +fewshot | 78.4 | 55.0 | 78.5 | 61.3 |
| +persona | 50 | 99 | 20.3 | 60.4 | +RAG | 80.7 | 55.5 | 77.3 | 60.2 |
| +fewshot | 47 | 99 | 20.3 | 29.0 | +tool | 78.5 | 55.0 | 79.5 | 60.6 |
| +RAG | 69 | 82 | 24.7 | 28.6 | Llama-3.3-70B | 73.1 | 50.0 | 68.8 | 58.8 |
| +tool | 53 | 91 | 22.5 | 31.3 | Llama-3.1-8B | 73.0 | 50.0 | 68.5 | 63.3 |
| Llama-3.3-70B | 71 | 81 | 30.3 | 33.3 | |||||
| +persona | 83 | 68 | 32.5 | 45.6 | |||||
| Llama-3.1-8B | 43 | 90 | 22.1 | 35.1 |
Table 3. LLMs reproduce the style and framing of public comments far better than their substance. DOCKETBench results on 558 dockets; each cell is a per-docket mean. Rows below a model are that model with few-shot, RAG, or RAG-plus-tool augmentation. “LLM …” columns are precision, “Human …” columns recall. Cells are shaded within each column, darker = better; the best value in each column is bold. Claim pairs are judged by gpt-4o-mini; frame and style pairs by Llama-3.3-70B (Appendix § 10.§ 3). Frame and style precision reaches 65–82%, while claim precision stays at 18–30%, and no prompting strategy closes the gap.
Benchmark Creation
DOCKETBench is built directly from DOCKET using the provisions, claims and matches described in § 2. We describe how the decisions guiding DOCKETBench and goal of our evaluations.
Dataset filtering: Starting from 17,198 proposed rules,25 we apply the following filters: (1) there must be ≥ 1 proposal per docket26 with 5–100 extracted proposed provisions (i.e., to exclude very small and very large rules); (2) there must be ≥ 3 (claim, proposed provision) matched pairs (i.e. to exclude provisions with no, or off-topic comments). To expand our (claim, proposed provision) matches, we distill an additional cross-encoder to explicitly include opposing comments (Appendix § 11.§ 3), since a comment that takes the opposite position from the proposed text still counts as a valid match. This includes an additional 30% of true-positive (claim, provision) pairings. These filters yield a final
(a) Content. (A) Does the model comment on the same provisions real commenters did? Precision: share of LLM-commented provisions that a human commenter also commented on. Recall: share of human-commented provisions the LLM also hit. Recall is high because the models spread their claims over nearly the whole proposal (78–99% of a docket’s provisions, against 46% for human commenters), so for most models precision sits near that 46% base rate. (B) Does it say the same things about them? Precision: share of LLM claims a judge rates same as some human claim on the docket. Recall: share of human claims with such an LLM match. +persona rows pool seven stakeholder personas; persona moves (A) by at most 13 points and claim precision by at most 2.2, but raises claim recall by 12–27 points, mostly because persona runs write three to six times more claims (Llama-3.3-70B excepted; Appendix § 10.§ 5).
(b) Style. (C) Does the model argue the way real commenters argue, and (D) write the way they write? We extract argumentative frames (e.g. “appeal to statutory intent”) and style markers (e.g. “cites current market trends”) from each comment and judge LLM–real pairs on the same docket. Precision: share of LLM frames (styles) that a human one matches. Recall: share of human frames (styles) that an LLM one matches. Persona moves every cell here by under 8 points (largest: gpt-4o-mini style precision, +7.3), so persona rows are omitted.
benchmark population of 558 dockets, 22,837 proposed provisions, 13,824 comments, and 43,371 unique real claims. The 558 dockets span 19 federal parent agencies. On average, provisions receive 7.8 claims (median: 2), with a 17% agency response rate on medium-length and long comments.
Metrics: We evaluate alignment between LLM-generated and human comments along two dimensions: Content-Level Alignment (metrics A-B) and Style-Level Alignment (metrics C-D).
(A) Provision Targeting (Precision & Recall): Using the claim provision mappings described in § 2, we measure how often models and humans target the same provisions within a proposal. We report, in Table 3, the share of LLM-commented provisions that humans also hit (i.e. LLM precision) and the share of human-commented provisions that the LLM also hit (i.e. LLM recall).
(B) Claim Alignment, (C) Argumentative Frame, and (D) Writing Style (Precision & Recall): For these metrics, we compare human and LLM-generated comments on the same proposals. We use an LLM to extract claims, argumentative frames, or style keywords from each comment.27 An LLM comment “matches” a human comment when another LLM judges the extracted elements to be equivalent.28 We calculate Precision and Recall based on these matches. We report in Table 3 the share of LLM claims (frames, styles) that humans also made or used, and the share of human claims (frames, styles) that the LLM also made or used.
Agreement by stakeholder group: This metric asks whose side the LLM takes. For every judged pair where the LLM claim and the human claim make a point about the same provision (i.e. the judge labels it same or opposing), we compute, for each stakeholder group, Agreement(group) = over the human claims. We report for each model, in Table 2, the group it agrees with most and least; the full per-group table, with confidence intervals and the same number for human commenters, is Table 18 in Appendix § 10.§ 6.
| Model | Agrees with most | Agrees with least |
| gpt-4o-mini | gov (0.68) | labor (0.38) |
| +persona | gov (0.61) | labor (0.43) |
| gpt-5-mini | gov (0.65) | labor (0.48) |
| +persona | gov (0.68) | labor (0.47) |
| gpt-5 | industry (0.68) | labor (0.61) |
| +persona | gov (0.65) | labor (0.56) |
| Llama-3.3-70B | gov (0.47) | labor (0.26) |
| +persona | expert (0.61) | advocacy (0.44) |
| Llama-3.1-8B | gov (0.68) | expert (0.40) |
| +persona | gov (0.48) | labor (0.36) |
| Human commenters | individual (0.86) | advocacy (0.71) |
Table 2. Whose side LLMs take. For each model, the stakeholder group it agrees with most and least: among judged pairs where an LLM claim and a real claim make a point about the same thing, the share judged same, by the group of the real commenter (commenters counted as the interest they file for). Human commenters: randomly sampled real commenters judged against the other comments in their docket, groups with ten or more pairs only. Overall agreement is 0.43–0.66 for the models and 0.75 for human commenters; the full per-group table with confidence intervals is Table 18.
| Model | Individual | Industry | Advocacy | Gov | Expert | Labor | All |
| gpt-4o-mini | 0.57 [0.49, 0.64] | 0.59 [0.56, 0.61] | 0.56 [0.53, 0.60] | 0.68 [0.57, 0.78] | 0.49 [0.36, 0.65] | 0.38 [0.30, 0.46] | 0.57 [0.55, 0.59] |
| +persona | 0.53 [0.49, 0.57] | 0.53 [0.51, 0.54] | 0.53 [0.50, 0.55] | 0.61 [0.55, 0.68] | 0.57 [0.48, 0.64] | 0.43 [0.38, 0.48] | 0.53 [0.51, 0.54] |
| gpt-5-mini | 0.59 [0.54, 0.64] | 0.60 [0.58, 0.62] | 0.55 [0.52, 0.58] | 0.65 [0.57, 0.74] | 0.51 [0.43, 0.60] | 0.48 [0.41, 0.55] | 0.58 [0.57, 0.60] |
| +persona | 0.55 [0.50, 0.59] | 0.56 [0.55, 0.58] | 0.56 [0.54, 0.59] | 0.68 [0.61, 0.74] | 0.52 [0.44, 0.60] | 0.47 [0.42, 0.52] | 0.56 [0.55, 0.57] |
| gpt-5 | 0.63 [0.58, 0.69] | 0.68 [0.66, 0.70] | 0.62 [0.59, 0.65] | 0.66 [0.55, 0.76] | 0.63 [0.53, 0.74] | 0.61 [0.53, 0.71] | 0.66 [0.64, 0.67] |
| +persona | 0.58 [0.54, 0.62] | 0.60 [0.58, 0.61] | 0.59 [0.56, 0.62] | 0.65 [0.59, 0.73] | 0.60 [0.52, 0.67] | 0.56 [0.51, 0.61] | 0.59 [0.58, 0.61] |
| Llama-3.3-70B | 0.41 [0.36, 0.49] | 0.44 [0.41, 0.47] | 0.44 [0.41, 0.48] | 0.47 [0.34, 0.60] | 0.42 [0.35, 0.48] | 0.26 [0.19, 0.33] | 0.43 [0.41, 0.46] |
| +persona | 0.44 [0.40, 0.50] | 0.50 [0.47, 0.53] | 0.44 [0.41, 0.48] | 0.57 [0.46, 0.68] | 0.61 [0.49, 0.74] | 0.44 [0.35, 0.53] | 0.48 [0.46, 0.50] |
| Llama-3.1-8B | 0.49 [0.42, 0.55] | 0.55 [0.53, 0.58] | 0.49 [0.45, 0.53] | 0.68 [0.58, 0.79] | 0.40 [0.27, 0.56] | 0.46 [0.38, 0.57] | 0.53 [0.50, 0.55] |
| +persona | 0.41 [0.38, 0.45] | 0.45 [0.43, 0.46] | 0.42 [0.40, 0.44] | 0.48 [0.43, 0.53] | 0.42 [0.37, 0.48] | 0.36 [0.34, 0.39] | 0.44 [0.42, 0.45] |
| Human commenters | 0.86 [0.70, 0.96] | 0.77 [0.68, 0.85] | 0.71 [0.60, 0.82] | – | – | – | 0.75 [0.69, 0.81] |
Table 18. Whose side LLMs take. For each pair where an LLM claim and a real claim make a point about the same thing (judge label same or opposing), we report the fraction judged same, split by the stakeholder group of the real commenter. We count each commenter as the interest it files for (a law firm as its client, a trade association as its members). Brackets are 95% confidence intervals from a docket bootstrap; bold marks the lowest group in each row. The Human commenters row runs the same judge on claims from randomly sampled real commenters against the other comments in their docket; we omit cells with fewer than ten pairs. LLMs agree with labor least in eight of ten rows (tied in a ninth) and less than with industry in every row, and every LLM agrees with real commenters less often than real commenters agree with each other.
Benchmark design and testing: As stated previously, each task is driven by the baseline prompt: “given <proposal>, generate a full comment [that highlights the concerns of stakeholder group X]” We test the following variants:
Vanilla: baseline prompt on: gpt-{4o-mini, 5-mini, 5}, Llama-3.3-70B, and Llama-3.1-8B.
+persona: we add persona-specific guidance to the baseline prompt. We run seven passes per task, each conditioning on a different stakeholder persona;29 we then pool the outputs with uniform weight before measuring their alignment.
+augmentations: we test several common augmentations on pure language modeling including: +fewshot (randomly selecting human-written comments in the prompt); +RAG (we build a bm25 mock “fact” database by using an LLM to extract evidence, studies, cited statutes and other facts from human comments from DOCKET not in DOCKETBench and select the top to include in the prompt); +tool (the model has the option to query the bm25 database we constructed up to three times during its generation run).
Benchmark Results
We now present results from evaluating our baseline strategies on DOCKETBench. We find, overall, that alignment on content is much lower than alignment on framing devices and style. Specifically, we see large content-level gaps (i.e. metrics (A) and (B)) between LLMs and human comments (claim precision 18–30%), but small frame and style gaps (metrics (C) and (D), precision 65–82%). This suggests current LLMs’ stylistic pluralism and content-level pluralism abilities are quite divergent: LLMs can write comments that sound like real commenters but fail to capture the opinions each subgroup cares about. We also find that persona conditioning does not have a uniformly positive impact. Persona conditioning helps weaker models (gpt-4o-mini: +7.3pp style prec.) but not stronger ones (gpt-5-mini: -0.1pp style, prec.). Additionally, we find that +Persona changes whose side the model takes but not which groups it covers: pooled over personas, agreement with each group shifts by a few points at most (Table 2). In other words, giving a model a persona changes its stance much more than it changes which real arguments it makes (Appendix § 10.§ 6). These findings diverge from prior work indicating that (a) language models are steerable, and that (b) persona guidance can be effective in inducing more representative models. We hypothesize that, unlike in generalist domains, technical and legal domain pretraining text put a strong prior over generative performance that overwhelms naive steerings a user might apply.
| Raw comment (DHS-2019-0034, cluster 4) | “The ACLU opposes DHS’s proposed rule to exempt the ALL-038 Insider Threat Program System of Records from certain provisions of the Privacy Act of 1974. The exemptions risk violating the Fourth Amendment protections of DHS employees. Specifically, exempting the system from 5 U.S.C. 552a(c)(3) and (c)(4) could prevent the release of accounting for disclosures. . . Exempting from 552a(d) could prevent individuals from accessing or amending their records. . . Exempting from 552a(e)(1) could compromise the accuracy of investigative information. . . This rule would further expand DHS’s disregard for the constitutional rights of its employees.” |
| Claim | Extracted claim text |
| 0 | The proposed rule to exempt the ALL-038 Insider Threat Program System of Records from certain Privacy Act provisions risks violating the Fourth Amendment. |
| 1 | Exempting the system from 5 U.S.C. 552a(c)(3)/(c)(4) could prevent release of accounting for disclosures and alert individuals wrongfully targeted. |
| 2 | Exempting from 5 U.S.C. 552a(d) could prevent access or amendment to records, concealing investigations into misconduct by DHS employees. |
| 3 | Exempting from 5 U.S.C. 552a(e)(1) could compromise accuracy of information obtained during investigations into potential violations of federal law. |
| 4 | The proposed exemption would further expand DHS’s disregard for the constitutional rights of its employees. |
Table 8. One public comment on the DHS Insider-Threat Privacy Act exemption, shown in raw form (top) and broken into five self-contained claims by the claim extractor (bottom).
| Real comment framings (sample of 80) | LLM framings (gpt-5-mini, 15 total) |
| • Appeal to economic impact on vulnerable groups | • Appeal to statutory intent and authority |
| • Appeal to environmental benefits of regulation | • Analysis of regulatory history and precedent |
| • Appeal to agency duty to protect consumers | • Economic impact analysis of regulatory choices |
| • Cost-benefit analysis of existing program | • Comparison to analogous provisions in related statutes |
| • Comparison to international regulatory standards | • Invocation of administrative law principles |
| • Appeal to personal experience | • Reference to agency reporting and transparency |
| Real comment styles (sample of 139) | LLM styles (15 total) |
| • Addresses specific government official by title | • Cites statutory authority with section references |
| • Attaches supporting documents or exhibits | • Proposes specific alternative policy solutions |
| • Uses emotional appeal language | • Uses technical terminology and industry jargon |
| • Provides personal anecdotal testimony | • References historical data and production trends |
| • Cites specific legislation by name | • Uses conditional language for consequences |
Table 14. Extraction examples from docket EPA-HQ-OAR-2017-0091. Left: real human comments (12 comments → 80 framings, 139 style elements). Right: LLM-generated comment (1 comment → 15 framings, 15 style elements). Bottom: canonical clusters after LLM-based clustering of real framings (80 → 13 clusters). Canonical clusters (real framings: 80 → 13 clusters via Llama-3.3-70B): Appeal to authority [4 members], Appeal to consequences [7], Economic appeal [9], Environmental appeal [6], Personal experience [5], Data-driven analysis [8], Comparison [6], Procedural [5], Legal authority [4], Public interest [7], Industry impact [6], Regulatory history [5], Transparency [3].
One vivid example of this is whose side LLMs take (Table 2). Only 21–33% of LLM claims make a point that a real commenter also made (i.e. in our data, the cases where a judge labels the pair same or opposing). Another 55–63% are on the same topic as a real claim but make a point no commenter made, and 8–17% of LLM comments are on topics that no human commenter focused on (Appendix § 10.§ 7). On the claims that do line up with a real claim, LLMs tend to agree more with government and industry positions and less with labor and advocates. Humans show more inter-commenter agreement than human-LLM agreement. Two reasons can explain this wide spread. Firstly, LLMs hedge: 18% of the time they argue both sides of the same real point. Secondly their default position follows the procedural and economic framing that industry and government commenters use, so they side against safety, health and environmental claims more often (Appendix § 10.§ 7). Three quarters of labor claims, for example, are safety claims, where only 6% of LLMs claims are. This gap holds after controlling for retrieval similarity, agency and judge and prompting LLMs to specifically act more like labor advocates. We caveat: labor’s positions are unpopular with human commenters too: other commenters also agree with labor less than with any other group; however, LLMs still do so even more.
Even when LLMs overlap with human comments, the positions they argue are often more moderate. LLMs, we find, produce comments that overwhelmingly advocate modifying proposals, rather than adding or deleting them: 85.9% of gpt-5-mini claims recommend modification, compared with 43.8% of human claims. LLMs rarely oppose proposals outright: only 5.1% of their claims are oppositional, compared with 30.4% of human claims. In addition, the kinds of arguments that LLMs make to advocate their points skew differently, and LLMs tend to argue from different perspectives. In contrast to human commenters, who mix experiential, economic and technical arguments, we find that 60.3% of gpt-5-mini claims are procedural, compared with 15.7% of human claims, while normative arguments fall from 26.6% in human comments to 4.5% in gpt-5-mini comments, and experiential arguments fall from 5.8% to 0.8% (Table 21b). Finally, as shown in Table 3, models touch 78–99% of a docket’s provisions, humans 46%. The attention patterns given by LLMs across provisions, independent of model size or persona prompting, are far more uniform than the attention given by humans. Humans concentrate heavily on a small fraction of provisions; models distribute claims uniformly.
Discussion
Notice-and-comment exists as a way for agencies to receive input from the public. The Administrative Procedures Act (APA) asks for “data, views, or arguments” from anyone and makes agencies answer the significant ones [45]. As we can see in our data, this process brings in diverse stakeholder views: Table 1 shows industry stakeholders commenting primarily on cost concerns, law firms commenting on statutory authority, labor unions commenting on safety, individuals commenting on a range of concerns. No single group, we see, raises the full set of concerns; the set comes from the mix. If LLMs were to be used in notice&comment, as suggested by [6, 10, 37, 46, 47], we would expect models to be able to reflect comments for all of these groups. However, our results show that they cannot achieve this aim. These results extend: a tool that summarizes a hundred thousand comments for a rule-writer and lands on the procedural reading each time tells the agency what the median lawyer would have said, which the agency already knew. Wagner [53] worried that well-resourced commenters crowd the record with technical and legal argument. A model that defaults to that register would also achieve this end.
This has been addressed, in computer science, primarily through introducing persona prompts and steering vectors [1, 40]. In our case, it did very little. Asked to write as a labor union, a model writes more claims (4/5 models wrote 3-6x as many as humans; Llama-3.3-70B wrote half as many), takes a different stance, and still argues from the same procedural and economic frames; its agreement with what real labor commenters said moves by a few points at most for the gpt models (10–18 points for Llama-family models) (Table 2, Appendix § 10.§ 6). The persona changes the stance and style, but not the material. In other words, models can be told whom to speak for, but they do not know what that group actually wishes to comment on. Retrieval, we find, helped gpt-5 more than any persona did (claim precision 24.7 vs. 18.1, Table 3), which points the same way: the missing ingredient is information, not instruction.
Several of our findings are novel in relation to prior work on LLM pluralism. In Argyle et al. [1] a demographic persona was enough to move GPT-3’s survey answers toward the group’s own, and in Santurkar et al. [40] and Durmus et al. [17] steering prompts moved opinion distributions toward the named demographic or country. Here, personas impact claims by at most 2.2 points and shift agreement towards any one group by at most a few points for gpt models (Llama-family models shift by 10–18 points on labor). To rule out confounders, we tested several, including: the effects of LLM-generated comment-length (i.e., longer comments have more claims, and will thus match more). We matched human units to different numbers of extracted units, . Shown in Appendix § 10.§ 4 (Figure 8), recall levels off around –50, claim precision moves by less than 0.4 points across all budgets, and unconstrained generation lands on the human median (46 vs. 48 claims per docket) but does not track how much commenting a docket actually drew. Additionally, we found that comment length, topic and agency bias were important signals for comment uptake; our observed results were robust after controlling for these variables.
Figure 8. Provision targeting as a function of claim budget . Left: Recall saturates near – (close to the human median of 48 claims/docket, green dotted line). Center: Precision declines gently; +persona (dashed) maintains higher precision at every . Right: Spearman plateaus at for gpt-5-mini. gpt-4o-mini tested at – only.
One explanation for why LLMs reflect such narrow views, here, is that the language of notice&comment is quite restrictive. A proposed rule is written in a narrow register: section numbers, defined terms, modal verbs, the formulas of the Federal Register. Conditioning on this in the prompt pulls a model toward a specific part of the pretraining distribution: the language of the lawyers and agency staff who wrote the documents it learned from. Everything we measure points to this pull. Sixty percent of LLM claims are procedural against 16% of human claims; experiential claims nearly vanish (0.8% vs. 5.8%); the models spread their claims almost evenly over the provisions where humans concentrate on a few; and the groups the models agree with most, government and industry, are the ones that write in that register themselves. In prior settings, which typically ask for responses to surveys, a persona is steering a model in a more general text setting, which might explain why it has a greater steering effect: pretraining has shown the model more realistic responses from these groups in these settings. If this hypothesis is correct, steering should fail most in other formal domains. We did not test this directly; we leave this open for future work.
So far, our findings have focused on low model performance and the shortcomings of current pluralism strategies for mitigating them. Another lens with which to view our results is the strength LLMs demonstrate on sections of DOCKETBench. Concerns are emerging in government and technology that systems for democratic deliberation may become overwhelmed with rising AI input; improvements on style and framing can increase the number of comments making past filtering systems [19]. Table 3 shows strong baseline model performance in Style and Argumentation, strong +persona impacts, and scaling trends for model size; together, these are indications of increased performance being imminently achievable. We urge a broader conversation around what kinds of improvements should be valued in benchmarks: pluralistic alignment, in Style and Argumentation, means sounding like a subgroup, not necessarily accurately articulating their concerns. We hope in future work to continue exploring AI detection strategies for optimal human-in-the-loop collaboration. We hope that DOCKET, and its derivative, DOCKETBench can serve as a valuable tool not just for the use-cases outlined here, but other use-cases in the broader community.
Related Work
Motivating DOCKET is a large literature studying how to better inform policymakers and promote participation in rulemaking, both theoretically [18, 32, 44] and empirically. Scholars have used notices, rules, and comments to study influence [55] and conduct detailed analyses of particular regulators—the SEC [7, 35], EPA [5], DOT [34], FCC [15], and Dodd-Frank financial regulations [9]. But these studies typically cover tens to hundreds of rules over short time windows; ours covers magnitudes more, allowing us to test at scale cross-agency heterogeneity and longitudinal dynamics. Our work also relates to administrative-law scholarship on the democratic promise and limits of notice-and-comment rulemaking. The APA requires agencies to solicit and consider public “data, views, or arguments” before issuing final rules [45], and scholars have long treated this process as a central mechanism of public participation in administrative governance [11, 14, 25]. At the same time, prior work has questioned whether rulemaking actually democratizes agency decision-making or instead privileges repeat players, technical experts, and organized interests [13, 33, 53]. DOCKET provides evidence for this debate by linking public comments to specific proposed provisions, agency responses, and final-rules, making it possible to measure not only who participates but what claims they raise, which provisions they target, and how agencies respond.
A second line of related work studies mass participation, comment campaigns, and the integrity of public-comment systems. Agencies increasingly receive large volumes of duplicative, campaign-driven, or malattributed submissions, raising questions about how comments should be filtered, summarized, and weighed [5, 6, 33]. Handan-Nader’s study of the FCC’s “Restoring Internet Freedom” proceeding shows both the scale of inauthentic participation and the methodological difficulty of identifying which comments agencies actually address [19]. Recent policy discussions have extended these concerns to computer-generated comments and comment-management infrastructure [46]. DOCKET builds directly on this literature, but shifts the focus from detecting or discounting mass comments to mapping the substantive structure of public input at scale.
Our benchmark also connects to work on pluralistic alignment and opinion representation in language models. Recent work argues that AI systems should not collapse diverse human preferences into a single aggregate objective, but should instead preserve disagreement across individuals, cultures, and social groups [12, 43]. Datasets and benchmarks such as PRISM and global-opinion evaluations test whose views models represent and whether models can capture subjective, multicultural, or politically heterogeneous judgments [16, 17, 26]. DOCKETBench differs in evaluating pluralism in a consequential institutional setting: stakeholders are not answering surveys or hypothetical preference questions, but making provision-specific claims about proposed law. This setting requires models to recover not only broad viewpoints, but also the concrete claims, legal frames, evidentiary practices, and stakeholder-specific priorities.
Finally, DOCKET relates to technical work on retrieval, legal-text representation, and model-based evaluation. Our extraction of provisions is grounded in legal theories of deontic relations [20] and recent work that represents regulatory obligations as discrete units whose survival can be tracked from proposed to final rules [28]. Our matching pipeline draws on modern retrieval and reranking methods, including sentence embeddings, dense retrieval, and cross-encoder reranking [24, 36, 39], as well as large-scale serving and evaluation tools [27, 56]. The benchmark interventions we test also build on common LLM adaptation strategies: persona prompting [31], retrieval-augmented generation [30], and tool-mediated generation. In parallel, agencies and contractors are beginning to use AI systems for public-comment analysis and regulatory review [10, 22, 23, 37, 47]. If LLMs are used to summarize comments, anticipate stakeholder objections, or assist rule drafters, failures of pluralistic representation may directly affect how public input is understood.
Conclusion
We set out to measure two things: how the public takes part in federal rulemaking, and whether language models can take part in the same way. For the first, DOCKET links 42.3M comments, 17,198 proposed rules and 22,400 final rules at the level of the individual provision, claim and agency response. The record does not match the common belief that comment periods are swamped by low-quality input. Commenters engage specific provisions, bring specialized knowledge about a third of the time, and divide by interest in what they argue about. Agencies answer them, and when they rewrite a provision they adopt commenter wording 11–26% of the time.
For the second, DOCKETBench asks a model the question an agency would put to a commenter, “how should this rule change?”, and measures the answer against what real commenters said. The models write comments that sound like public comments; however, they typically represent claims most often that government and industry commenters made. They rarely say what labor unions, advocates or individuals said, and persona prompts do not change that. We release both the dataset and the benchmark so that the tools now being built to draft and read public comments can be judged on whose perspectives they carry rather than on how fluent they sound.
References
Organization of this appendix. Appendix § 9 covers the DOCKET corpus: sources, deduplication, parsing, the stakeholder, concern-taxonomy and uptake analyses, the hand check of the LLM labels (Appendix § 9.8), and limitations. Appendix § 10 covers DOCKETBench: how we built it, the metrics, the claim-budget and persona choices, the stakeholder-alignment analyses behind Table 2, and how LLM claims differ from human claims in direction and argument type (Appendix § 10.8). Appendix § 11 lists every prompt.
The DOCKET corpus: sources, processing, and coverage
Per-agency corpus statistics
Table 4 enumerates all 44 federal agencies in DOCKET, with per-agency totals for dockets, public comments, three regulations.gov document classes (notices, proposed rules, finalized rules), three classes of validated cross-document matches, and high-volume comment clusters (>100 deduplicated comments per cluster, a proxy for agencies that experience organized commenting campaigns). Notices+Prop.+Final is the document total used in Figure 1 (for example, DHS: 598 + 60 + 119 = 777 documents). The three match types are: () — a public-comment claim addresses a specific proposed-rule provision; () — a finalized-rule provision corresponds to a public comment that influenced it; and () — an explicit agency-response paragraph in the final-rule preamble addresses a specific public comment. Agencies are ordered by comment volume; the bottom 5 rows (FCC, SEC, FDIC, FEC, USPS) are covered through external comment portals (Appendix § 9.§ 2) — their proposed/finalized rule counts come from regulations.gov as for the other agencies, but the provision-extraction and matching pipelines have not yet been run on the externally-scraped comment streams. Tabular companion to Figure 1.
| Agency | Dockets | Comm. (analyzable) | Comm. (FedReg) | Cov. | Notices | Prop. | Final | (c→p) | (f→c) | (r→c) | Clust. >100 | Match |
| FDA | 3,111 | 1,233,623 | 1,557,309 | 79% | 8,874 | 440 | 766 | 72,302 | 19,691 | 15,102 | 234 | 29% |
| CMS | 760 | 674,012 | 1,015,135 | 66% | 2,703 | 421 | 455 | 183,163 | 94,687 | 40,345 | 346 | 29% |
| ED | 1,148 | 627,874 | 964,550 | 65% | 1,773 | 197 | 249 | 43,506 | 26,055 | 2,505 | 111 | 38% |
| FWS | 740 | 617,767 | 1,008,149 | 61% | 766 | 879 | 482 | 40,252 | 24,318 | 9,815 | 92 | 36% |
| HHS | 79 | 601,023 | 939,404 | 64% | 37 | 135 | 232 | 25,357 | 2,252 | 1,371 | 104 | 65% |
| BLM | 43 | 539,683 | 843,022 | 64% | 0 | 43 | 116 | 8,289 | 10,992 | 665 | 69 | 30% |
| EPA | 5,268 | 537,389 | 585,966 | 92% | 9,946 | 5,548 | 5,255 | 171,029 | 78,307 | 64,463 | 22 | 28% |
| NOAA | 1,032 | 399,645 | 457,173 | 87% | 506 | 1,074 | 2,818 | 83,429 | 37,525 | 7,053 | 33 | 32% |
| DOI | 44 | 388,196 | 850,023 | 46% | 84 | 38 | 53 | 1,206 | 2,791 | 332 | 87 | 65% |
| FS | 104 | 353,209 | 583,657 | 61% | 1,537 | 46 | 47 | 5,687 | 1,625 | 1,101 | 34 | 46% |
| APHIS | 469 | 324,064 | 399,933 | 81% | 818 | 99 | 115 | 35,132 | 11,509 | 4,861 | 39 | 35% |
| CDC | 750 | 264,017 | 296,643 | 89% | 1,272 | 26 | 45 | 2,960 | 3,741 | 1,844 | 27 | 41% |
| USCIS | 188 | 232,808 | 299,138 | 78% | 877 | 44 | 104 | 66,710 | 27,192 | 1,371 | 35 | 58% |
| ICE | 33 | 231,125 | 153,310 | 151% | 174 | 9 | 14 | 5,624 | 4,017 | 432 | 6 | 60% |
| DOT | 442 | 190,209 | 207,433 | 92% | 623 | 85 | 81 | 16,481 | 9,197 | 1,409 | 21 | 40% |
| AMS | 326 | 180,889 | 299,749 | 60% | 411 | 420 | 404 | 22,999 | 18,949 | 3,093 | 63 | 46% |
| FAA | 8,056 | 144,673 | 212,593 | 68% | 3,047 | 4,848 | 6,944 | 93,881 | 54,692 | 22,454 | 22 | 28% |
| IRS | 415 | 133,532 | 160,556 | 83% | 180 | 514 | 473 | 17,866 | 14,040 | 3,637 | 29 | 33% |
| OSHA | 167 | 129,483 | 171,346 | 76% | 1,124 | 100 | 103 | 21,573 | 6,029 | 1,573 | 14 | 50% |
| FSIS | 170 | 84,404 | 167,744 | 50% | 293 | 41 | 54 | 7,205 | 5,402 | 1,023 | 29 | 48% |
| NHTSA | 454 | 76,075 | 137,946 | 55% | 1,126 | 132 | 127 | 11,412 | 2,900 | 7,339 | 35 | 37% |
| EOIR | 23 | 75,777 | 100,887 | 75% | 0 | 18 | 46 | 13,459 | 4,392 | 182 | 30 | 57% |
| DOJ | 68 | 60,959 | 64,838 | 94% | 28 | 48 | 99 | 10,941 | 293 | 386 | 8 | 53% |
| NIST | 28 | 31,024 | 72,271 | 43% | 38 | 6 | 11 | 2,459 | 215 | 321 | 17 | 58% |
| DHS | 171 | 29,886 | 38,216 | 78% | 598 | 60 | 119 | 3,311 | 9,407 | 209 | 2 | 81% |
| DOE | 258 | 28,208 | 30,452 | 93% | 2,369 | 77 | 130 | 5,221 | 3,612 | 1,231 | 2 | 45% |
| USDA | 33 | 16,427 | 18,635 | 88% | 1,631 | 24 | 66 | 3,548 | 1,381 | 526 | 3 | 57% |
| TSA | 28 | 11,869 | 11,943 | 99% | 345 | 13 | 33 | 1,022 | 741 | 204 | 0 | 27% |
| USACE | 39 | 11,754 | 12,198 | 96% | 150 | 57 | 60 | 4,775 | 4,038 | 180 | 0 | 59% |
| TREASURY | 140 | 10,407 | 10,591 | 98% | 137 | 67 | 174 | 2,283 | 1,220 | 444 | 0 | 44% |
| CBP | 83 | 8,667 | 4,838 | 179% | 1,383 | 40 | 241 | 23,115 | 3,177 | 1,406 | 0 | 63% |
| DOL | 32 | 5,739 | 6,573 | 87% | 1,751 | 36 | 64 | 2,148 | 219 | 5,993 | 2 | 67% |
| DOC | 21 | 3,837 | 4,045 | 95% | 16 | 21 | 51 | 1,966 | 974 | 947 | 1 | 87% |
| NTIA | 19 | 3,644 | 3,855 | 95% | 18 | 2 | 11 | 152 | 0 | 98 | 0 | — |
| DOD | 265 | 3,548 | 3,658 | 97% | 1,912 | 81 | 245 | 3,108 | 3,135 | 745 | 0 | 55% |
| FEMA | 125 | 3,343 | 3,483 | 96% | 4,697 | 33 | 174 | 3,548 | 1,903 | 436 | 0 | 45% |
| MSHA | 83 | 840 | 869 | 97% | 585 | 52 | 27 | 6,874 | 3,869 | 537 | 0 | 60% |
| NIH | 4 | 18 | 18 | 100% | 3 | 6 | 11 | 8 | 0 | 13 | 0 | — |
| NSF | 0 | 0 | 0 | n/a | … | |||||||
| Subtotal (regs.gov) | 25,219 | 8,269,647 | 11,698,149 | 71% | 53,765 | 15,782 | 20,510 | 1,024,001 | 494,487 | 205,646 | 1,517 | |
| External-portal agencies (notices/rules counted from regulations.gov; comments agency-specific portals; matching pipelines not yet run on these comment sources): | ||||||||||||
| FCC | 4,732 | 30,608,766 | 30,608,766 | † | 2,936 | 889 | 1,121 | — | — | — | — | |
| SEC | 120 | 34,761 | 34,761 | † | 24,506 | 224 | 351 | — | — | — | — | |
| FDIC | 202 | 5,784 | 5,784 | † | 1,117 | 146 | 173 | — | — | — | — | |
| FEC | 92 | 1,921 | 1,921 | † | 589 | 49 | 34 | — | — | — | — | |
| USPS | 0 | 0 | 0 | † | 4,092 | 108 | 211 | — | — | — | — | |
| Grand total | 30,365 | 38,920,879 | 42,349,381 | — | 87,005 | 17,198 | 22,400 | 1,024,001+ | 494,487+ | 205,646+ | 1,517+ | — |
Table 4. Per-agency aggregate counts for the 44 federal agencies in DOCKET. Dockets = distinct rulemaking dockets containing at least one comment. Comm. (analyzable) = comments in our corpus that survive the full text-extraction and feature-extraction pipeline (i.e., have non-empty extracted text on which we run claim-extraction and downstream analysis); the raw scrape from regulations.gov exactly matches the FedReg aggregate for every agency we checked, so the gap below 100% reflects stub or attachment-only records (no OCR-able body text), not missing scrapes. Comm. (FedReg) = comment count the agency officially reports to the Federal Register, also equal to the count of records in our raw bulk-download dump. Cov. = analyzable / FedReg; values >100% (ICE, CBP) reflect agencies where the FedReg aggregate undercounts what regulations.gov actually exposes via bulk download. Notices / Prop. / Final = distinct regulations.gov documents of each type for all 44 agencies, including the agency-collective-comment-sourced FCC/SEC/FDIC/FEC/USPS; the document totals shown in Figure 1 are Notices+Prop.+Final summed (e.g., DHS: 598 + 60 + 119 = 777). The three match columns are abbreviated by initial: (c→p) = comment → proposal, (f→c) = final rule → comment, (r→c) = response → comment; all three are validated by the calibrated gpt-5-mini judge with opposing-sides framing. Clust. >100 = MinHash-deduplicated comment clusters whose constituent comment count exceeds 100 (high-volume / likely-organized-campaign clusters; see Appendix § 9.§ 3). The “+” on grand-total match counts indicates the regulations.gov subtotal: matching has not yet been run on the externally-scraped comment streams. “†” marks external-portal agencies, for which the Federal Register publishes no aggregate comment count. Match = share of comment clusters linked to at least one agency response, estimated from a uniform random sample of 1K candidate pairs per agency judged by gpt-5-mini (the match rate shown in Figure 1); “—” where the agency was not sampled.
Per-source scraping pipelines
Although regulations.gov is the primary federal portal for public comments, several large-volume agencies maintain their own systems with idiosyncratic formats. We built 15 custom scraping pipelines to bring these into a unified schema. Each pipeline normalizes the source-specific output into the same (docket_id, comment_id, body_text, attachment_paths, posted_date, agency) record format used downstream by our parsing and matching code.
Table 5 lists all 16 data sources. We distinguish two categories: comment systems, which expose individual public comments with per-comment metadata (commenter, date, body text); and document portals, which expose project-level documents, notices, or bundled comment archives but not individual per-comment records.
| Source | Type | Access | URL |
| Regulations.gov | Comment system | Bulk download | regulations.gov/bulkdownload |
| FCC ECFS | Comment system | API + bulk | fcc.gov/ecfs |
| SEC | Comment system | HTML scrape | sec.gov/comments |
| FERC eLibrary | Comment system | Playwright | elibrary.ferc.gov |
| Federal Reserve | Comment system | Camoufox | federalreserve.gov/apps/foia/proposedregs.aspx |
| FDIC | Comment system | HTML scrape | fdic.gov/federal-register-publications |
| FEC FOSERS | Comment system | HTML scrape | docquery.fec.gov |
| CFTC | Comment system | Camoufox | comments.cftc.gov |
| PCAOB | Comment system | HTML scrape | pcaobus.org/about/rules-rulemaking |
| CMS NCD | Comment system | Camoufox | cms.gov/medicare/coverage |
| FS CARA | Comment system | HTML scrape | cara.fs2c.usda.gov |
| NPS PEPC | Comment system | HTML scrape | parkplanning.nps.gov |
| BLM ePlanning | Document portal | HTML scrape | eplanning.blm.gov |
| NRC ADAMS | Document portal | API | adams.nrc.gov |
| USACE Districts | Document portal | HTML scrape | usace.army.mil (per-district) |
| USITC EDIS | Document portal | HTML scrape | edis.usitc.gov |
Table 5. Data sources in DOCKET. “Comment system” sources expose individual public comments; “document portal” sources expose project-level documents or bundled comment archives.
Regulations.gov. The default federal portal, covering 30+ agencies. Although it has a public API, the API is rate-limited and incomplete; we instead use the bulk-download endpoint via a Playwright browser session that prompts the user to manually solve a captcha when challenged.
FCC ECFS. The largest external system, with 30.6M comments. We use both the FCC’s bulk-download archive and an hourly-poll API mirror (publicapi.fcc.gov) to track comments during high-volume proceedings. The API imposes a 10K offset cap per query; we bypass this by sharding on hourly date_submission windows and adaptively splitting to 15-minute intervals when a shard exceeds the cap. For the 2017 Restoring Internet Freedom proceeding (Docket 17-108) we additionally use the curated corpus released by Handan-Nader [19] as ground truth for our matching-pipeline validation (§ 2).
SEC. Comment letters on proposed rules are posted with both inline text and PDF attachments. SEC comment letters are unusually long-form (median pages) and frequently include legal-counsel signatures.
FERC eLibrary. The system requires session-based browsing; we use Playwright to paginate through each docket’s filings and download the associated PDF attachments.
Federal Reserve. The site uses a session-stateful Blazor SPA with SignalR-based pagination; our scraper uses Camoufox (an anti-detection browser) to retrieve per-rule comment listings and submitter metadata.
FDIC, FEC, CFTC, PCAOB, CMS. Each of these agencies publishes comments on proposed rules through agency-specific web portals. We scrape each via purpose-built pipelines: FDIC and FEC use simple HTML directory traversal; CFTC and CMS require Camoufox to bypass Cloudflare protection; PCAOB follows a rulemaking-docket index structure.
FS CARA. The Forest Service’s Comment Analysis and Response Application exposes individual per-letter data. We scrape project pages and resolve per-letter download links from detail pages.
NPS PEPC. The National Park Service’s Planning, Environment, and Public Comment system accepts individual public comments on NPS NEPA projects and makes them publicly available alongside planning documents.
Document portals (BLM, NRC, USACE, USITC). These four systems expose project-level documents and notices rather than individual comments. BLM ePlanning hosts environmental assessments and associated comment letters; NRC ADAMS is a keyword-searchable document database; USACE operates per-district public-notice pages; USITC EDIS manages investigation documents. Each is scraped via a small Playwright pipeline that follows the agency’s docket-listing pattern.
Aggregate-count scrapers. For agency-level coverage validation (Figure 1), we scrape per-agency, per-year aggregate counts of received comments and published documents from the Federal Register (https://www.federalregister.gov) and from agency-specific transparency pages where available. These aggregate counts are what we use to estimate that we are collecting of available documents per agency.
Comment-campaign detection and deduplication
A defining feature of the federal public-comment record is the mass-comment campaign: an organized effort that mobilizes large numbers of citizens to submit substantively identical (or near-identical) comment text on a single rulemaking. Mass campaigns can dominate raw comment counts to the point where docket-level statistics become essentially uninformative without deduplication. In our corpus, the largest single campaign — on Department of the Interior docket DOI-2017-0002 (the 2017 “National Monuments Review”) — contains approximately 98,000 substantively identical comment submissions in one cluster, and the docket as a whole has multiple clusters in the 25K–100K range. The 2017 FCC “Restoring Internet Freedom” net-neutrality docket (handled through FCC ECFS rather than regulations.gov, so not in the per-agency table above) attracted on the order of 25 million submissions, the majority of which were attributable to organized campaigns and at least some of which were later identified as automated or fraudulently attributed [6, 19].
Deduplication via MinHash clustering. We collapse near-duplicate comments into clusters using MinHash similarity [8]. Two comments are placed in the same cluster if their MinHash signatures (computed over shingles of the comment text) agree above a Jaccard-similarity threshold tuned to capture standard form-letter campaigns (paraphrase-stable to small surface edits like greeting personalization or postal-address insertion, but distinct on substantive content changes). Each cluster receives a stable cluster_uid formed as <docket_id>::<cluster_index>. All claim extraction, claim-to-provision matching, and downstream metric computation operate at the cluster level rather than the raw-comment level, so a 50,000-comment campaign contributes one cluster (and one set of extracted claims) rather than 50,000 redundant signal copies.
Cluster-size distribution. Across the 10.79M raw regulations.gov comments in our deduplication snapshot (an earlier pull than Table 4, which counts 11.7M), MinHash deduplication yields 7.94M distinct clusters. The distribution of cluster sizes is highly skewed (Table 6): 98.7% of clusters are singletons (a single submission with no near-duplicate elsewhere in the corpus), but the 424 clusters with members account for 20.3% of all raw comments. The median cluster size is 1; the mean is 1.4; the maximum (DOI-2017-0002) is 98,422. Per-agency counts of high-volume clusters ( members) appear in the right column of Table 4 above; CMS, FDA, ED, HHS, and FWS lead, reflecting agencies that regulate health, education, and environmental issues (which tend to attract organized advocacy campaigns).
| Cluster size band | Clusters | % of clusters | Raw comments | % of raw |
| 1 (singleton) | 7,830,939 | 98.7% | 7,830,939 | 72.6% |
| 2–5 | 94,235 | 1.2% | 223,950 | 2.1% |
| 6–10 | 5,577 | 0.1% | 40,953 | 0.4% |
| 11–100 | 4,399 | 0.1% | 124,372 | 1.2% |
| 101–1,000 | 1,093 | 0.01% | 379,823 | 3.5% |
| 1,001–10,000 | 376 | 0.00% | 1,158,354 | 10.7% |
| 10,001–100,000 | 48 | 0.00% | 1,034,779 | 9.6% |
| Total | 7,936,667 | 100% | 10,793,170 | 100% |
Table 6. Distribution of MinHash cluster sizes across the 10.79M raw regulations.gov comments in the deduplication snapshot. The corpus is dominated by singletons (one-off submissions, 72.6% of raw volume), but the largest 424 clusters ( members each) account for 20.3% of raw comments. Without deduplication, single mass campaigns would swamp many docket-level statistics.
Why deduplication matters for our metrics. All four benchmark axes operate on cluster-level real claims rather than raw comment text. This has two consequences. First, signal weighting: if 20,000 commenters submit the identical critique of a provision, that critique contributes one matched (claim, provision) row, not 20,000 — so a model that paraphrases that single shared critique gets equal credit regardless of campaign size. Second, evaluation feasibility: the LLM-judge cost of axis B is proportional to the number of distinct real claims; without deduplication, a single net-neutrality-scale campaign would dominate the budget and crowd out signal from smaller dockets. The 558-docket benchmark population deliberately excludes dockets where one or two campaigns account for the bulk of cluster mass (we filter to distinct validated real claims per docket, see § 4.1); the corpus-wide cluster table in this section is pre-filter, intended to characterize the raw-data terrain.
Limitations of MinHash deduplication. MinHash is robust to character-level perturbations (whitespace, capitalization, single-token edits) but defeated by deeper paraphrase. Modern advocacy campaigns increasingly employ template engines that vary sentence ordering or substitute synonyms across submissions, which MinHash treats as distinct clusters. In such cases the corpus over-reports unique-cluster counts and under-reports campaign size. We do not attempt to detect such deeper paraphrase systematically; doing so robustly is an open research problem [6, 19].
Similarly, our pipeline does not flag comments that may have been generated by language models (a documented and growing concern in federal rulemaking [6, 46]).
Parsing examples: proposal → provisions, comment → claims, response section → responses
To make our extraction pipeline concrete, this subsection shows three real worked examples drawn from the corpus.
Proposal → provisions. Proposed rules contain regulatory text that establishes obligations, permissions, prohibitions, conditions, and exceptions. We extract these provisions using gpt-5-mini (Appendix C). Table 7 shows two short proposals and their extracted units, illustrating the range of provisions and how a single regulatory section can contain multiple distinct units.
| Proposal | Unit | Type | Extracted provision text |
| Proposal 1 | 1 | prohibition | No persons, watercraft, or vessels shall enter, or remain, in the area. |
| Proposal 1 | 2 | prohibition | The restricted area is permanently closed for public use at all times. |
| Proposal 1 | 3 | exception | Except for those authorized by the enforcing agency. |
| Proposal 1 | 4 | obligation | This regulation will be enforced by the Commander, United States Army–Alaska. |
| Proposal 2 | 1 | obligation | Gray triggerfish—25 fish [commercial trip limit]. |
| Proposal 2 | 2 | condition | The commercial trip limit applies until the commercial quota specified in § 622.39(a)(1)(vi) is reached. |
| Proposal 2 | 3 | exception | See § 622.39(b) for the limitations regarding gray triggerfish after the commercial quota is reached. |
Table 7. Two proposed rules shown in raw regulatory text (top) and their extracted provisions (bottom). Units are classified by type: obligation, prohibition, condition, exception.
Proposal 1: COE-2021-0006 — Restricted Area: Eagle River, Richardson Training Area, AK
§ 334.1305 Eagle River from Bravo Bridge to its mouth at Eagle Bay in Knik Arm, Richardson Training Area on Joint Base Elmendorf-Richardson, Alaska; restricted area.
(a) Restricted area. The restricted area consists of navigable waters within an area defined as beginning a point on shore at latitude 61°19’40.1” N, longitude 149°44’20.336” W; thence easterly. . . 3.06 nautical miles southerly along the river. . .
(b) The regulation. The restricted area is permanently closed for public use at all times. No persons, watercraft, or vessels shall enter, or remain, in the area except for those authorized by the enforcing agency.
(c) Enforcement. This regulation will be enforced by the Commander, United States Army–Alaska.
Proposal 2: NOAA-NMFS-2023-0044 — Reef Fish Resources of the Gulf of Mexico; Commercial Trip Limit for Gray Triggerfish
§ 622.43 Commercial trip limits.
(b) Gray triggerfish—25 fish. The commercial trip limit applies until the commercial quota specified in Sec. 622.39(a)(1)(vi) is reached, which is equal to the commercial ACT. See Sec. 622.39(b) for the limitations regarding gray triggerfish after the commercial quota is reached.
To make the claim- and response-extraction pipeline concrete, this subsection shows two real worked examples drawn from the corpus. Each example takes a single underlying piece of text (one public comment in Table 8; one final-rule preamble section in Table 9) and shows the structured rows the LLM extractor produced from it.
| Raw preamble (EPA-HQ-OAR-2016-0544-0449) | “EPA received several petitions requesting that the Agency change the ‘point of obligation’ under the RFS program from refiners and importers to downstream parties such as fuel blenders or position holders. Petitioners argued this change would improve program effectiveness. . . EPA denies the requests. . .” |
| Comment summary (extracted) | Agency response (extracted) |
| Petitioners asked EPA to shift the point of obligation downstream, arguing it would improve the program. | EPA denies, finding petitioners did not demonstrate improved effectiveness. |
| Commenters favored designating position holders as obligated parties. | EPA rejects: position holders are not refiners/importers/blenders, not aligned with statutory authority. |
| Commenters alleged EPA must annually reconsider the obligated parties. | EPA disagrees the statute requires annual reconsideration. |
| Commenters supported designating position holders as obligated parties. | EPA believes this is not well aligned with the authority to place the obligation on “refineries, importers, or blenders.” |
Table 9. One final-rule response section (EPA 2017 RFS), shown in raw form (top) and broken into four (comment-summary, agency-response) pairs by the response extractor (bottom).
Comment → claims. Public comments are typically multi-paragraph documents that bundle several distinct positions. We use Llama-3.3-70B (Appendix C) to break each comment into its constituent claims: short, self-contained statements that each address one specific provision. The LLM is instructed to (a) restate the claim with provision-level specificity, (b) avoid pronouns and references to surrounding paragraphs, and (c) produce one claim per discrete point. Table 8 shows the five claims the extractor produced from a single ACLU comment on the Department of Homeland Security’s 2019 ALL-038 “Insider Threat Program System of Records” Privacy Act exemption rule.
Response section → responses. Final-rule preambles bundle the agency’s discussion of every comment it received into a few long “response-to-comments” sections. We use Llama-3.3-70B (Appendix § 11) to break each response section into discrete (comment-summary, agency-response) pairs. Each row is the agency’s discussion of one issue raised by a commenter, separated from the surrounding discussion of other issues. Table 9 shows five of the eight rows extracted from EPA’s response to comments on the 2017 Renewable Fuel Standard “Point of Obligation” rulemaking (EPA-HQ-OAR-2016-0544-0449).
Comment counts by stakeholder subgroup
Figure 4 shows the per-subgroup comment counts across the full DOCKET corpus, after mapping the raw org_type field through our seven-bucket taxonomy (§ 3). Counts span more than three orders of magnitude: individual commenters dominate raw volume (~7.7M comments), followed by by advocacy NGOs (), industry (), expert organizations (), government (), and labor unions and law firms in the low thousands ( and respectively). Per-subgroup absolute volume is highly skewed even after de-duplicating form-letter campaigns.
Figure 4. Comment counts per stakeholder subgroup across the full DOCKET corpus (8.27M comments). Log-scaled x-axis. Subgroups are derived from org_type via the seven-bucket taxonomy.
Concern taxonomy: construction and examples
Table 1 reports the distribution of primary concern frames across stakeholder groups. The 12-frame taxonomy was developed iteratively through a combination of top-down and bottom-up analysis:
Taxonomy construction. We began with broad categories grounded in the administrative law literature on public commenting [9, 55]: economic, legal, and procedural concerns are canonical axes of regulatory debate. We augmented these with domain-specific categories surfaced by manual inspection of 500 randomly sampled claims across 20 agencies: health, safety, environmental, scientific/technical, and equity/justice emerged as recurrent and distinctive frames that cut across agencies. Religious/moral and national security were added to capture low-frequency but substantively distinct argument types. The “other” and “unknown” categories absorb residual and ambiguous cases; three low-frequency categories (religious/moral, national security, unknown) are collapsed into “Other” in the main-text table.
Mutual exclusivity and overlap. The taxonomy assigns exactly one primary frame per claim via forced single-label classification. We acknowledge that real claims often invoke multiple frames simultaneously (e.g., a safety concern grounded in scientific evidence, or an economic argument with equity implications). The forced single-label design trades granularity for interpretability: it produces clean distributional comparisons across stakeholder groups (Table 1) at the cost of suppressing secondary frames. The LLM classifier is instructed to select the dominant frame—the single framing that best characterizes the claim’s core argument.
Example claims by concern frame. Table 10 provides illustrative examples of real public-comment claims classified under each frame.
| Frame | Example claim |
| Health | “Restricting plan changes to once per year for dual-eligible, LIS-eligible, and other at-risk beneficiaries will lead to improved health outcomes, such as better medication adherence and completion of preventative screenings.” |
| Safety | “The FDA’s proposed ban on electrical stimulation devices used to treat self-injurious or aggressive behavior is necessary due to the device’s unreasonable risk of serious injury to disturbed, disabled, and vulnerable people.” |
| Economic | “The proposed rule requires hospitals to report median payer-specific negotiated charges by MS-DRG, which will require significant additional effort, with an estimated burden at least 10 times greater than CMS’ estimate.” |
| Legal | “FDA’s proposed rule would override existing protections for small-scale farmers by requiring ‘first receivers’ to keep detailed records on products that are exempt from traceability requirements under FSMA.” |
| Scientific | “The use of non-FTIR spectroscopy methods, such as NCASI methods 98.01 or 105.01, is requested as an alternative to the specified method for formaldehyde emissions measurement.” |
| Environmental | “An extension of the implementation deadline would allow over 1 million tons of additional carbon dioxide to be released.” |
| Equity | “The proposed fee increases are designed to target and increase fear among low-income people of color and force immigrant communities to live without protection from deportation.” |
| Religious | “The proposed regulation allows religious organizations to require attendance at religious activities as a condition for receiving lifesaving or sustaining federal services.” |
| Nat’l Security | “Relaxing I-9 documentation requirements could facilitate the employment of non-citizens or individuals with criminal records, detrimental to national security.” |
| Procedural | “The recommended two-year sell-through period for affected new hydronic heaters should be extended indefinitely.” |
Table 10. Example real public-comment claims for each concern frame in our taxonomy. Each claim is the dominant-frame classification assigned by gpt-5-mini; many claims contain secondary frames not reflected in the single-label assignment.
Comment language uptake in final rules
To assess how directly public comments influence final rule language (as opposed to policy direction), we sampled 1,000 (comment, proposed-rule provision, final-rule provision) tuples from our Match 3 data and classified each with gpt-5-mini into four categories: DIRECT UPTAKE (final rule adopts specific phrasing from the comment that was absent in the proposed rule), RESPONSIVE (final rule addresses the comment’s concern but in the agency’s own language), TOPICAL ONLY (same topic, no evidence of influence), and OPPOSED (agency rejected the comment’s suggestion). Results: 13.8% DIRECT UPTAKE, 15.8% RESPONSIVE, 35.0% TOPICAL ONLY, 35.4% OPPOSED. We checked these labels by hand. We drew 70 cases from the classifier output (51 labeled DIRECT UPTAKE and 19 labeled RESPONSIVE, TOPICAL ONLY or OPPOSED as distractors), shuffled them, hid the classifier label, and highlighted language shared between the comment and the final rule. The annotator confirmed 80% of the DIRECT UPTAKE labels as language taken from the comment (86% counting close paraphrases, 88% counting substantive influence without shared language), and found language-level uptake in most of the distractor cases too. On this sample the classifier does not inflate the uptake rate. Figure 5 shows the distribution split by whether the provision was modified from its proposed form or newly added after the proposal stage, and Table 11 shows the eight strongest direct-uptake candidates from the sample.
| Docket | Proposed Rule | Public Comment | Final Rule |
| FAA-2019-0479 | Install an elbow restrictor, P/N 46610–1, in accordance with the Accomplishment Instructions of Bombardier Service Bulletin 84–32–69, Revision C, dated January 20, 2011. | The FAA should only mandate section 3.B Procedure of Bombardier Service Bulletin 84–32–69, Revision C, dated January 20, 2011, as it directly corrects the unsafe condition, and not the entire Accomplishment Instructions. | Install an elbow restrictor, P/N 46610–1, in accordance with paragraph 3.B., “Procedure,” of the Accomplishment Instructions of Bombardier Service Bulletin 84–32–69, Revision C, dated January 20, 2011. |
| FAA-2021-0650 | Time of Designation. Intermittent, 0700–1700 local time, Monday–Friday, by NOTAM 7 days in advance. | The proposed R–2511 restricted area would be active less than 36 days a year and only for 2 hours per active day, making a TFR a more suitable alternative to minimize impact on VFR traffic. | Activation limited to no more than 36 days per year. |
| EPA-HQ-OAR-2018-0417 | [No transition timeline for CEDRI electronic reporting.] | Dow recommends a transition period of at least 1 year after the reporting template becomes available in CEDRI or 2 years after publication of the amendments for the transition to new reporting templates. | Beginning on April 16, 2021 or 1 year after the appropriate electronic reporting template becomes available on the CEDRI website, whichever is later, submit all subsequent reports following the procedure specified in paragraph (l). |
| CMS-2019-0039 | Standardized encounter data, no later than one (1) business day after data concerning the encounter is received by the MA organization. | The proposed rule’s requirement for plans to make information accessible through the APIs, including adjudicated claims, encounters with capitated providers, and clinical data, starting on January 1, 2020, will allow patients to exercise their HIPAA right of access. | Encounter data from capitated providers, no later than one (1) business day after data concerning the encounter is received by the MA organization. |
| DOT-OST-2022-0144 | You must provide timely notifications to passengers when their wheelchairs and scooters have been loaded and stowed in the cargo compartment and unloaded and retrieved from the cargo compartment. | Carriers should be required to notify passengers when their wheelchair or scooter is not loaded on the aircraft, but not necessarily when it is loaded or unloaded, due to potential technological restrictions and implementation costs. | You must notify passengers with disabilities, before the aircraft cabin door closes, whether their wheelchairs or scooters have been loaded in the cargo compartments of their flights, including whether their device could not fit on the passenger’s scheduled flight because of its size or weight. |
| ED-2015-OSERS-0132 | [No specific criteria framework for measuring significant disproportionality.] | States should be allowed flexibility to add additional criteria beyond the risk ratio and minimum n size thresholds to avoid identifying significant disproportionality that is an artifact of calculations using small numbers. | (A) Reasonable risk ratio threshold; (B) Reasonable minimum cell size; (C) Reasonable minimum n-size; and (D) Standard for measuring reasonable progress if a State uses the flexibility described in paragraph (d)(2) of this section. |
| FAA-2023-2234 | [No requirement for manufacturer data release.] | The proposed AD should include a defined date for Boeing to release all service data to operators, such as December 31, 2024, to ensure sufficient time for modifications prior to the AD’s completion due date. | Boeing will release all service data to allow retrofit of hardware updates to the CFM56–7B nacelle prior to that date. |
| APHIS-2020-0068 | [No enrichment provision for birds.] | Environmental enrichment for all birds is necessary to meet the AWA’s goal of ensuring humane care and treatment, and should include species-specific physical and psychological needs. | The physical environment in the primary enclosures must be enriched by materials or activities that would provide the birds with the means to express noninjurious species-typical activities. |
Table 11. Top candidates for direct comment-to-rule language uptake. For each case, the final rule contains a specific parameter, term, or formulation that appears in the public comment but was absent from the proposed rule. Boldface highlights the adopted language.
Human check of the LLM labels
Every label in the pipeline comes from a model, so one of us checked a sample by hand before submission. For each of eleven steps we drew ten items, stratified across the label values the model produced, showed the annotator the same inputs the model saw together with the model's label, and asked whether the label was right (and, if not, which was). This is a verification check, not a blind re-annotation: the annotator saw the label. Table 12 gives the counts. The model's label held up on 97 of the 102 items the annotator could decide; two stakeholder-type items were marked unsure because the comment text gave no cue about who wrote it (one was “please see the attached file”). The five disagreements were borderline rather than gross: a response that mentioned a comment without addressing it, a response the model called “substantive” that the annotator read as partial acceptance, a vaccine comment the model filed under “other” rather than health, and one provision match and one further response match on which reasonable readers differ. Nine further steps (uptake, claim → final provision, claim direction, argument type and scope, the fourteen engagement tags, legal score, and requested action) were prepared on the same page but are not yet checked.
| Step | Where it is used | n | agree | wrong | unsure |
| Claim → proposed provision (Match 2) | Section 2, Figure 2b | 10 | 9 | 1 | 0 |
| Response → comment | Figure 1, Finding 1 | 10 | 8 | 2 | 0 |
| Response stance (accepted / partial / disagree) | Finding 1 | 10 | 9 | 1 | 0 |
| Concern frame of a comment | Table 1 | 10 | 9 | 1 | 0 |
| Stakeholder type of a commenter | Table 1, Table 2 | 10 | 8 | 0 | 2 |
| Specialist vs. generalist | Figure 3 | 7 | 7 | 0 | 0 |
| Proposed → final provision pairing (Match 1) | Figure 2b | 10 | 10 | 0 | 0 |
| Type of modification | Appendix B.1 | 10 | 10 | 0 | 0 |
| Claim-pair judge (same / related / different / opposing) | Table 3B, Table 2 | 7 | 7 | 0 | 0 |
| Frame and style pair judge | Table 3(C,D) | 10 | 10 | 0 | 0 |
| Concern frame of an LLM claim | Section 4.2 | 10 | 10 | 0 | 0 |
| All | 104 | 97 | 5 | 2 |
Table 12. Hand check of the LLM labels, one annotator, ten items per step (seven where the annotator stopped early). “Agree” means the model's label was judged right; “wrong” that the annotator chose another label.
Coverage and limitations
We document the known coverage gaps and methodological limitations of DOCKET and DOCKET-Bench so that downstream users can interpret results in context. The dataset is intentionally large and broad, but it is not exhaustive: every step of collection, parsing, matching, and labeling carries its own sources of error.
Agency-year asymmetry. Federal commenting volume varies by orders of magnitude across agencies. Coverage is much thinner for the early years of our window (2016–2017) at several agencies, likely because those agencies lacked robust digital comment infrastructure during that window. Some agencies (e.g., MSHA, USPS, and parts of DOI) maintain very low public-comment volumes throughout the period, so per-agency analyses on those slices have limited statistical power. Where agency-year cells are sparse, we recommend treating per-agency findings as descriptive rather than inferential.
Comment systems outside regulations.gov. The federal government does not have a single, uniform public-comment portal. While regulations.gov hosts comments for the majority of agencies, several large-volume agencies maintain their own systems with non-standard formats. We built 15 custom scraping pipelines to cover these sources—eleven comment systems (FCC ECFS, SEC, FERC, Federal Reserve, FDIC, FEC, CFTC, PCAOB, CMS, FS CARA, NPS PEPC) and four document portals (BLM ePlanning, NRC ADAMS, USACE Districts, USITC EDIS); see Table 5 in Appendix A.2. Format heterogeneity means we occasionally lose attachments, comment metadata, or page-numbered exhibits. We estimate the loss at well under 5% of total volume per affected agency, but the exact rate varies and is reported per-agency in the data release.
Comments only available through FOIA. A small set of public comments are not posted to any web portal and can be released only in response to Freedom of Information Act (FOIA) requests. These comments tend to concentrate at agencies that exempt parts of their regulatory record from public posting (e.g., national-security-adjacent agencies, certain inspector-general filings) or where comments contain confidential business information. Because FOIA releases are slow, partial, and idiosyncratic, we exclude FOIA-only material from the main release.
State-level and sub-federal rulemaking is excluded. DOCKET covers U.S. federal rulemaking only. The dataset does not include rules issued by state agencies, U.S. territories, tribal governments, county or municipal regulators, or federal-state cooperative bodies. State-level commenting is substantively different in volume, format, and stakeholder mix; users interested in state regulation should look elsewhere.
Form-letter and mass-campaign deduplication is approximate. We collapse near-duplicate comments using a MinHash similarity threshold, which captures the typical mass-campaign pattern in which thousands of commenters submit identical or near-identical text. However, modern campaigns increasingly insert minor variations (paraphrase, sentence shuffling, or LLM-generated rewrites) that defeat MinHash. We report both deduplicated and raw comment counts; the deduplicated counts likely still over-represent variant campaigns relative to truly independent submissions. Robust LLM-rewrite detection is an open problem and not solved here.
Matching pipeline carries residual error. Our three-stage matching cascade (bi-encoder retrieval cross-encoder filter LLM judge) is calibrated against hand-curated ground truth, but residual error remains. Internal audit of the proposed-to-final unit matching surfaced approximately 20% of pairs where we re-classified the upstream label after a second LLM review. The benchmark’s claim-to-provision matches use a calibrated opposing-sides judge that improved agreement with human ground truth from 67% to 96% on the FCC-RIF reference set, but no LLM judge is perfect, and downstream metrics inherit this residual error. We list per-stage accuracy estimates in Appendix § 9.8.
Benchmark population is filtered. The 558 dockets used in DOCKETBench are the entire population that survives our two filters (5–100 proposed provisions; at least three validated claim-to-provision pairs, which implies at least one commented-on provision). This excludes dockets with no commented provisions and dockets at the extreme ends of the size distribution. Conclusions therefore generalize to “federal rules that draw at least some substantive comment” and not to silent or extremely small/large dockets. We report the per-agency distribution of the benchmark population in Appendix § 10.§ 2.
Stakeholder bucketing relies on org_type metadata. The seven-bucket stakeholder taxonomy (industry, labor, advocacy, expert organization, government, law firm, individual) comes from an LLM classifier (Llama-3.3-70B) that reads the comment text and any stated affiliation, mapped to seven buckets. A second pass also records who the commenter is filing on behalf of; Table 2 uses it to count law firms as their clients and trade associations as their members (Appendix § 10.§ 6). We do not validate commenter identity against outside sources. Approximately 10% of comments cannot be cleanly assigned to a bucket and are placed in an “other / unknown” category that we exclude from per-bucket analyses.
Language and accessibility. All extraction, matching, and benchmark generation steps assume English-language input. A small fraction of public comments are submitted in other languages or as scanned images; these are handled via the same pipeline (extraction is best-effort) but quality is lower. We do not separately evaluate per-language accuracy.
Temporal scope. The dataset covers 2016–2026. Earlier rulemaking is not included. Because regulatory practice changes over time (e.g., in response to executive orders, court rulings, and changes in administration), aggregate statistics across this window mix periods with different commenting norms. We provide per-year breakdowns in the data release for users who want to restrict to a specific period.
LLM cost and reproducibility. Several of our pipeline steps use commercial LLM APIs (gpt-5-mini, gpt-5, gpt-4o-mini) which can change behavior over time as providers update model snapshots. We report the model versions and dates of all calls so future replications can detect drift, but exact reproduction of LLM-judge labels may require re-running with the original model snapshots.
DOCKETBench: construction, metrics, and extended analyses
DOCKETBench Construction
Here, we give more detail on the construction steps in DOCKETBench.
(1) Provisions. As stated in § 2, we segment every proposed and final rule into atomic provisions — single normative statements (obligations, permissions, prohibitions, conditions, exceptions) that can be tracked across drafts. To extract units, we use gpt-5-mini at minimal reasoning effort with a 6-shot prompt, applied via a bulk pipeline that recovered 301,892 proposed-rule units across 17,198 proposed rules and 374,026 final-rule units across 22,400 final rules (676K units total; $67.60 in extraction cost after a two-pass repair of 23K silent-zero documents). Units carry a section anchor (§-citation if present), a 1-sentence action_summary, and the verbatim obligation_text; the (docket_id, doc_id, unit_id) triple is the canonical identifier used by every downstream stage.
(2) Comment-claim extraction & (comment proposal) matching. Each public comment is decomposed into its constituent claims (claim_id = docket::cluster_idx); claims are then matched to specific proposed-rule units in a three-stage cascade: bi-encoder retrieval ModernBERT cross-encoder filter gpt-5-mini binary judge with explicit “opposing-sides” framing (see (4) below). The (comment proposal) corpus contains 1.99M (claim, proposed-unit) pairs with YES/NO labels ($28 LLM cost). The validated YES pairs are the population on which all benchmark metrics are computed: across the 558 benchmark dockets, this yields 43,371 unique real claims (223,535 (claim, unit) rows after the one-claim-many-units expansion).
(3) Match 1 + modification typology (proposed final unit). A parallel Match 1 pass aligns each proposed unit to its surviving final-rule counterpart, producing the survival_status label per unit (survived / modified / removed). All 12,812 modified units (12,466 after the audit in Appendix § 11.§ 4 re-labeled some pairs; Figure 2b reports the audited count) are then sub-labeled by gpt-5-mini under a 14-class change-type taxonomy (definition_clarified, threshold_or_scope_{looser, tighter}, exception_{added, removed}, timeline_{extended, shortened}, enforcement_{strengthened, weakened}, agency_obligation_change, cosmetic_only, topic_mismatch, etc.; $2.57). The taxonomy was iterated through two manual-audit rounds: v1 mis-coded 46% of stricter cases (conflating agency-side burden with stringency), motivating the v2 split. The resulting cosmetic_only+definition_clarified share (39%) and topic_mismatch share (24%) provide quantitative bounds on upstream Match-pipeline noise.
(4) LLM-as-judge. All comment-to-provision and response-to-comment matching uses gpt-5-mini judge prompts: the prompt explicitly states that responses (or matches) “address” both supporters and opponents of a position. We find that, without this framing, judge agreement with hand-curated ground truth (Nader-FCC RIF set, ) was only 67%; with it, agreement rose to 96%. The failure mode without it was systematic: the judge labeled opposite-direction true matches as “not addressed.” All Match 2/Match 3 labeling uses this prompt. The Axis B pair judge in the benchmark is a different prompt, run with gpt-4o-mini in batches of ten pairs (Appendix § 11.12); it labels each (LLM claim, real claim) pair as same, related, different or opposing. We compare it against four other judges in Appendix § 10.§ 6.
(5) Comment-anticipation benchmark task. From the 17,198 proposed rules with extracted proposed units, we filter to dockets meeting all of: (a) 5–100 proposed units (excluding micro- and mega-rules), (b) validated (comment proposal) matches with distinct real claims, (c) commented-on provision. This yields the 558-docket benchmark population (Appendix Table 13). For each docket, the proposed units (formatted with their section anchors and action_summary) are presented to the model; the model must emit anticipated commenter claims prefix-tagged [P#] to specific provisions. The original gpt-5-mini reference run produced 27,936 LLM claims at $3; the run judged for Table 3 is a later regeneration with 26,139 claims (47 per docket), and Table 13 describes the reference run.
| Statistic | Value |
| Benchmark dockets | 558 |
| Real claims, matched on provisions (unique) | 43,371 |
| Comments with at least one claim | 13,824 |
| LLM-generated claims (gpt-5-mini, single run) | 27,936 |
| Mean LLM claims per docket | 50.1 |
| Provisions with at least one matched comment cluster (corpus-wide) | 62,996 |
| Mean comment clusters per matched provision | 7.8 |
| Median (95th percentile, max) | 2 (36, 284) |
| Empirical response rate ((comment → proposal)-matched clusters) | 54.1% |
| Empirical response rate (Match 2 ∪ Match 3 clusters) | 53.1% |
| Empirical response rate (medium/long, non-form-letter comments) | 17.0% |
| Empirical response rate (all comments in benchmark dockets) | 6.5% |
Table 13. Descriptive statistics of the benchmark population. Comment counts are at the cluster level after MinHash deduplication (collapsing form-letter campaigns). Empirical response rate is the fraction of comment clusters whose constituent comment documents were matched to an extracted agency response.
The 558-docket population spans 19 federal parent agencies; with a mean of 25 proposed units per proposal document (median 19; 876 documents, about 40 units per docket); the per-provision real-claim count is heavy-tailed (mean 7.8, median 2). The empirical response rate on (comment → proposal)-matched claim clusters in the benchmark is 17% on medium-/long-comment claim clusters.
Benchmark dataset descriptive statistics
We report descriptive statistics on the comment-anticipation benchmark population. The benchmark covers 558 federal rulemaking dockets, selected from DOCKET as those with 5–100 extracted proposed-rule provisions and at least three validated (comment → proposal) matches. Table 13 summarizes the population.
Pluralism measurement: methods and evidence extraction
We measure three types of pluralism (inspired by [43]) with two measurements per dimension, precision and recall. This section documents each cell’s methodology.
Matching pipeline. For each dimension (content, frame, style), we compare LLM-generated elements against real human elements from the same docket. The matching proceeds in three stages:
(1) Extraction. For content, we generate claims in [P#] format (claim-list prompts, Appendices C.10–C.11). For frame and style, we generate full natural-language comments and extract unbounded lists of argument types and writing techniques using Llama-3.3-70B-FP8, producing ∼10 framings and ∼14 style elements per comment.
(2) Clustering. Real comments produce ∼78 unique framings and ∼118 unique style elements per docket (from ∼12 comments each), many near-duplicates. We cluster these into canonical groups using Llama-3.3-70B, which assigns each item to a cluster by number reference (to avoid copy errors). This compresses 78→∼25 canonical framing clusters and 118→∼23 style clusters per docket.
(3) Judging. For content claims, we embed with all-MiniLM-L6-v2, retrieve top-3 nearest real claims per LLM claim, and judge each pair with gpt-4o-mini. For frame and style, we exhaustively compare each LLM element against all canonical clusters for that docket, judging in batches of 5 pairs per prompt with Llama-3.3-70B-FP8. All judges classify each pair as same, related, opposing, or different; only same counts as a match.
From matches to metrics. Given the per-pair judgments, we compute two metrics per dimension, reported in Table 3.
Let and be the real and LLM-generated elements for docket . Let if any receives a same judgment when paired with .
Precision (P): mean per-docket precision. Of the LLM’s generated elements, what fraction match at least one real element?
Recall (R): mean per-docket recall. Of the real human elements, what fraction are covered by at least one LLM element?
Extraction and clustering examples. Table 14 illustrates the three measurement dimensions on a single docket (EPA-HQ-OAR-2017-0091, 2018 Renewable Fuel Standards). Real comments produce ∼80 unique framings and ∼139 unique style elements from 12 comments; the LLM generates ∼15 of each from a single comment. Clustering reduces the 80 real framings to 13 canonical groups.
Evidence extraction for RAG baseline. To build the retrieval-augmented generation baseline, we extract research-level factual evidence from 30,588 real public comments flagged as evidence-bearing by our structural feature classifier. We instruct gpt-4o-mini to extract only verifiable, citable facts — not biographical details, opinions, or vague references. Each extracted item must be a complete, self-contained statement (Table 15).
| Evidence type | Extracted content (examples) |
| Statutes cited (21.3%) | 45 C.F.R. § 164.502(a) establishes the conditions under which protected health information may be disclosed, including limitations on data sharing that could violate privacy laws. |
| Published data (48.9%) | EPA estimated a range of benefits of $36–$90 billion from the Mercury and Air Toxics Standards for Power Plants rule, with the reduction of mercury accounting for just $4–$6 million of that total. |
| Studies cited (20.3%) | The EPA 2019 Integrated Science Assessment for PM2.5 concluded there is a causal relationship between long-term PM2.5 exposure and cardiovascular mortality. |
| Regulatory history (41.5%) | The 2012 Mercury and Air Toxics Standards required coal-fired plants to reduce mercury emissions by 90% within 3 years of promulgation. |
| Specific alternative (42.4%) | Incorporate the alternative temperature monitoring option (minimum kiln temperature) into the final rule for additional flexibility. |
Table 15. Research-level evidence extraction from real public comments ( comments; 27,406 with extractable evidence). The prompt requires each item to be self-contained: a reader must understand what a number measures, what a statute governs, or what a study found without seeing the original comment.
Measurement variables. We decompose each comment claim into three measurement dimensions, each extracted as an unbounded list of specific elements (Table 16):
| Variable | What it captures | Example extracted elements |
| Content | Substantive position taken | “The proposed 30-day compliance window is insufficient for small manufacturers” “The rule should exempt facilities with fewer than 50 employees” |
| Frame | Argumentative approach | “Cost-burden on small businesses” “Environmental justice for frontline communities” “APA procedural due-process argument” |
| Style | Evidence and format | “Cites 42 U.S.C. § 7411(d) on existing-source emission guidelines” “Proposes replacing 30-day deadline with 90 days in §XXX.XX(a)” “References EPA 2019 ISA finding on PM2.5 mortality” |
Table 16. Three measurement variables with example extracted elements. Elements are extracted by Llama-3.3-70B; MiniLM retrieves the three nearest real elements (cosine > 0.3) and the LLM judge decides whether each pair matches.
Style-steering prompt variants (a side experiment, not part of Table 3). We test five prompt variants that instruct the model to adopt specific structural patterns: (1) evidence-rich (“cite real statute numbers, reference empirical data”); (2) experiential (“write from first-person lived experience”); (3) alternative-proposing (“propose specific replacement regulatory language”); (4) legal-grounded (“cite APA, case law, constitutional provisions”); (5) RAG (retrieve real evidence snippets via BM25 and include as context). The RAG variant uses a BM25 index over 27,406 evidence-bearing extractions; for each benchmark provision, we retrieve the top-3 most similar evidence snippets and include them in the prompt.
Key style finding. Vanilla LLM claims cite statutes 1.0% of the time vs. 15.9% in real comments (16× gap). Prompt steering closes this dramatically — the legal-grounded variant achieves 85.8% statute citation and 40.4% case law citation — but these citations are likely hallucinated. The RAG variant achieves 26.4% statute citation with evidence grounded in real prior comments, suggesting that the style gap is partly a knowledge problem addressable with retrieval augmentation.
Claim budget selection and calibration
A key design choice in DOCKETBench is the claim budget : how many claims should the model generate per docket? Without budget control, the model’s output volume is fixed regardless of docket complexity. We document the calibration process here.
Human claim volume. Table 17 summarizes the distribution of real human claims per docket across the 558 benchmark dockets. The distribution is highly skewed: the median is 48, but the mean is 401 due to mass-comment dockets (max 12,857).
| Mean | Median | P25 | P75 | Min | Max | |
| Real human claims | 401 | 48 | 16 | 220 | 3 | 12,857 |
| gpt-5-mini vanilla (uncontrolled) | 47 | 46 | 35 | 61 | 0 | 108 |
| gpt-5-mini +persona (uncontrolled) | 151 | 149 | 106 | 190 | 40 | 281 |
Table 17. Claims per docket in the 558-docket benchmark population.
Uncontrolled LLM output volume. Without a claim budget, gpt-5-mini vanilla generates a median of 46 claims per docket—coincidentally close to the human median (48)—but with a far narrower range (0–108 vs. 3–12,857). Figure 6 shows the distributions; Figure 7 confirms that the LLM does not scale its output to match commenting intensity (Spearman = 0.184, Pearson = −0.045).
Figure 6. Distribution of claims per docket. Left: Real human claims (clipped at 200; long tail extends to 12,857). Center: gpt-5-mini vanilla (concentrated around median 46). Right: gpt-5-mini +persona (7 personas pooled, median 149). Dashed lines show medians.
Figure 7. Correlation between real human claim count and uncontrolled LLM claim count per docket (gpt-5-mini vanilla, ). The LLM generates claims regardless of whether the docket received 3 or 12,857 real claims (Spearman , ; Pearson , n.s.). This motivates budget control.
Budget selection. We test fixed budgets for every model, extended to for gpt-4o-mini and gpt-5-mini. Models comply with budget instructions 87–98% of the time (within ±20% of requested ). gpt-5-mini achieves 98% compliance; gpt-4o-mini achieves 87%. Figure 8 shows provision-targeting metrics as a function of .
Key observations. (1) Recall keeps rising to . (2) Precision decays gently (2pp per 5 claims), so higher budgets help more than they hurt. (3) The real optimum likely lies near (close to the human median). Table 3 reports uncontrolled generation. We use the budget-controlled runs as a check: claim precision moves by less than 0.4 points across , so the precision gap in the main text is not an artifact of how many claims the models write.
Persona aggregation methodology and limitations
We document the design choices and limitations of our persona-conditioned benchmark variant to facilitate correct interpretation of results.
Aggregation. For the +persona variant, each of 7 personas (industry, labor, advocacy, expert, government, law firm, individual) generates claims independently for the same docket. We pool all persona outputs with uniform weight—i.e., each persona contributes of the generation budget regardless of its prevalence in the true commenter population. In the real data, individual commenters account for the majority of submissions, while law firms and labor unions are relatively rare. We chose uniform weighting for interpretability (each stakeholder voice is represented equally) rather than fidelity to the true population distribution.
Claim count. The number of claims generated per persona per docket is unconstrained: the model decides how many claims to produce based on the provisions presented. In practice, the “individual” and “advocacy” personas tend to generate 10–20 claims per docket, while “law firm” generates 15–30 (reflecting the model’s tendency toward more granular legal arguments). We do not normalize claim counts across personas, which means personas that produce more claims have proportionally more influence on pooled metrics (Axes A–D).
Within-persona diversity. Each persona is prompted exactly once per docket (temperature 1.0). There is no multi-sample generation, no prompt variation within a persona, and no explicit diversity instructions beyond the persona description itself. Variation across claims within a single persona generation arises solely from the model’s stochastic sampling.
Interpretation of Table 2. Table 2 measures alignment between all LLM claims (vanilla or pooled personas) and real claims grouped by the real commenter’s . It answers: “Which types of real commenters does the LLM agree with most?” It does not measure how well each LLM persona matches its corresponding real stakeholder group. The metric is: Agreement() = for real claims from . Labor is the smallest group in Table 2, with 170–374 pairs per vanilla run. Law firms are counted as their clients in that table.
What this design can and cannot show. Our persona conditioning is minimal: seven roles, one prompt each, no further steering. This is sufficient to show that (a) role conditioning raises claim recall mostly because four of the five models write three to six times more claims (Llama-3.3-70B wrote half as many), and its effect on precision, frame and style is small and goes both ways (Table 3), and (b) even with role conditioning, structural deficits persist (procedural tone, uniform attention, modification-oriented stance). However, our setups do not constitute a rigorous test of whether LLMs can represent diverse stakeholders under more elaborate prompting (e.g., multi-turn elicitation, retrieval-augmented generation, few-shot exemplars drawn from real comments). The structural findings—that LLMs default to procedural, modification-oriented, uniformly-distributed commentary—are robust to persona conditioning but may be addressable with more sophisticated prompting strategies that we do not explore here. We therefore frame our benchmark as measuring “what LLMs produce under straightforward prompting” rather than “what LLMs are capable of producing under optimal conditions.”
Alternative weighting schemes. Three natural alternatives to uniform pooling are: (1) population-weighted (weight each persona by its share of real commenters), (2) provision-weighted (weight by per-provision org_type distribution), or (3) claim-count-normalized (constrain each persona to a fixed number of claims). Under population-weighting, the “individual” persona would dominate (~60% weight), amplifying claims written as citizen testimony. Under claim-count normalization, personas that naturally produce fewer arguments (labor, individual) would receive equal representation alongside more prolific personas (law firm). We report uniform-weight results as the simplest interpretable baseline; alternative weightings can be computed from the released per-persona claim data.
Stakeholder alignment: agreement, representation, and steering
Table 18 is the full version of Table 2: agreement with every stakeholder group, with confidence intervals, and the human-commenter row.
Representation lift, re-estimated. The lift we printed in the NeurIPS submission (Lift = % of matched claims from a group/% of comments from the group) does not have a null of 1. Models write roughly the same number of claims per docket no matter how large the docket is, so a claim in a small docket is three to four times more likely to be matched than a claim in a large one. Labor claims sit in small FAA, OSHA and MSHA dockets (46% of them are FAA). Pooling matches across dockets and dividing by a global population share therefore gives labor a 2–4× “lift” even when matching is random within each docket: shuffling group labels within dockets gives 2.75×, and running real human claims from other dockets through the same pipeline gives 2.9–5.5×. The sign of the printed labor cell followed the judge and the protocol, not the model. Table 19 fixes this. It compares each group’s share of matched claims to the share we would expect if each docket’s matches were a random draw from that docket’s own commenters, standardizes across agencies with weights that do not depend on match rate, and counts each commenter as the interest it files for (all 60 law-firm clusters in the benchmark file for clients; 54 of those are industry). Under this estimator the ten-run consensus for every group is within a few percent of proportional (0.97–1.04×); individual cells range from 0.85 to 1.41×, the widest resting on fewer than three agencies. Dropping FAA dockets moves the pooled within-docket labor estimate from 1.19 to 1.01. Cells marked † rest on fewer than three agencies and carry the FAA problem. The result is not a ceiling effect: in dockets with match rates below 10%, where a model could have put two to eleven times its share of matches on one group, the estimates are still 0.93–1.05. Matching noise that does not depend on group does push these estimates toward 1, so we can rule out large representation biases, not biases of a few percent.
| Model | Individual | Industry | Advocacy | Gov | Expert | Labor |
| gpt-4o-mini | 1.10 [0.93, 1.27] | 1.00 [0.97, 1.04] | 0.96 [0.90, 1.01] | 1.15 [0.95, 1.41] | 0.85 [0.60, 1.08] | 0.94 [0.63, 1.25] |
| +persona | 0.99 [0.89, 1.08] | 1.01 [0.98, 1.03] | 0.99 [0.97, 1.03] | 1.03 [0.85, 1.21] | 0.92 [0.76, 1.06] | 0.92 [0.74, 1.20] |
| gpt-5-mini | 0.94 [0.84, 1.04] | 1.02 [0.99, 1.04] | 1.00 [0.96, 1.03] | 1.03 [0.79, 1.26] | 0.93 [0.71, 1.11] | 1.05 [0.87, 1.25] |
| +persona | 0.97 [0.91, 1.03] | 1.02 [1.00, 1.04] | 0.98 [0.94, 1.01] | 1.04 [0.87, 1.23] | 0.92 [0.78, 1.05] | 1.00 [0.81, 1.15] |
| gpt-5 | 0.94 [0.83, 1.04] | 1.03 [0.99, 1.06] | 0.98 [0.94, 1.02] | 0.98 [0.78, 1.19] | 0.86 [0.67, 1.07] | 0.88 [0.66, 1.10] |
| +persona | 0.98 [0.92, 1.05] | 1.01 [1.00, 1.03] | 0.97 [0.94, 0.99] | 1.02 [0.89, 1.21] | 1.10 [1.00, 1.22] | 1.11 [0.93, 1.28] |
| Llama-3.3-70B | 0.96 [0.85, 1.08] | 1.03 [1.00, 1.05] | 0.98 [0.94, 1.02] | 1.00 [0.82, 1.16] | 0.98 [0.78, 1.17] | 0.91 [0.70, 1.14] |
| +persona | 0.96 [0.82, 1.10] | 1.00 [0.97, 1.04] | 0.98 [0.94, 1.02] | 1.18 [0.95, 1.51] | 1.08 [0.84, 1.29] | 1.13 [0.74, 1.30]† |
| Llama-3.1-8B | 0.95 [0.84, 1.05] | 1.01 [0.98, 1.04] | 0.99 [0.95, 1.03] | 1.09 [0.83, 1.39] | 0.99 [0.75, 1.23] | 1.41 [1.03, 1.74]† |
| +persona | 0.96 [0.91, 1.02] | 1.01 [0.99, 1.02] | 0.99 [0.97, 1.02] | 1.02 [0.88, 1.13] | 1.04 [0.87, 1.22] | 0.94 [0.71, 1.14] |
| Consensus (10 runs) | 0.97 [0.94, 1.00] | 1.01 [1.01, 1.02] | 0.98 [0.97, 0.99] | 1.04 [0.98, 1.10] | 0.99 [0.94, 1.05] | 1.03 [0.95, 1.11] |
Table 19. Stakeholder representation among matched claims, per model. Each cell is the share of matched real claims written by a group, divided by the share we would expect if each docket's matches were a random draw from that docket's commenters. We standardize across agencies (each agency weighted by the group's claim count there) and count each commenter as the interest it files for (law firms as their clients, trade associations as their members). Brackets are 95% confidence intervals from a docket bootstrap; bold marks intervals that exclude 1; † marks cells where fewer than three agencies contribute. Those cells are dominated by FAA dockets, where union form letters recur across dockets and inflate labor matching for every model. No group is more than a few percent over- or under-represented; the consensus across runs is 0.97–1.04×.
Persona steering. We joined each judged pair to the persona that wrote the LLM claim and asked whether persona- matches land on real group more often than chance. They do not: the diagonal (persona matches its own group) is 1.04–1.15 and the off-diagonal is 1.03–1.07. Writing a group does not make a model reproduce that group’s actual claims. The same personas move the agreement rate in Table 2 by 0.2–0.4 (for gpt-5, the government persona agrees with labor 76% of the time and the individual persona 51%). Personas change the model’s stance. They do not change its substance.
Judge robustness of Table 2. The opposing label is the least stable output of the pair judge. On a 1,987-pair stratified sample, four other judges confirm gpt-4o-mini’s opposing calls as opposing 18–33% of the time (most of the rest become related), but when two judges both give a directional label they agree on the direction 71–84% of the time. We recomputed Table 2 over all 1.13M pairs with two open-weights judges. Both reproduce the group pattern: labor is 0.29–0.51 below the other groups under a distilled Gemma-3-12B judge and 0.07–0.27 below under Gemma-3-27B. They do not reproduce the ranking of models by overall agreement, so we do not interpret that ranking. A blind human check of 52 judged pairs, balanced over judge labels, generating models and 21 agencies, agreed with the gpt-4o-mini judge on 86.5% of pairs ().
Why LLMs agree with commenters less often than commenters agree with each other
What the agreement rate covers. Table 2 only counts pairs where the LLM claim and the real claim make a point about the same thing. Table 20 shows how much of each model’s output that is. Between 83% and 92% of generated claims retrieve at least one real claim above the cosine threshold. Only 21–33% get a directional label (same or opposing) against any real claim, and 15–20% get at least one same. Another 55–63% are judged related or different to everything they retrieve: they are on the topic of real comments but make a point no commenter made. The last 8–17% retrieve nothing. So the agreement rate describes the fifth to the third of LLM output that lines up with a real claim. The rest is the substance gap in Table 3.
| Model | Generated | Any pair | Directional | ≥1 same | Topic only | No counterpart |
| gpt-4o-mini | 14,584 | 87.5 | 25.9 | 17.9 | 61.6 | 12.5 |
| +persona | 55,358 | 90.2 | 29.1 | 19.1 | 61.1 | 9.8 |
| gpt-5-mini | 26,139 | 85.2 | 24.4 | 17.1 | 60.8 | 14.8 |
| +persona | 84,490 | 89.1 | 26.2 | 17.9 | 62.9 | 10.9 |
| gpt-5 | 29,183 | 82.8 | 20.6 | 15.4 | 62.2 | 17.2 |
| +persona | 92,485 | 87.4 | 25.4 | 18.0 | 62.0 | 12.6 |
| Llama-3.3-70B | 30,112 | 85.2 | 29.8 | 17.2 | 55.3 | 14.8 |
| +persona | 16,653 | 88.1 | 32.5 | 20.0 | 55.5 | 11.9 |
| Llama-3.1-8B | 17,962 | 88.4 | 28.4 | 19.1 | 60.0 | 11.6 |
| +persona | 97,384 | 91.7 | 32.6 | 19.3 | 59.1 | 8.3 |
Table 20. Where generated claims end up (% of generated claims per run). Any pair: at least one real claim retrieved at cosine > 0.3. Directional: at least one pair judged same or opposing; this is the denominator of Table 2. ≥1 same: at least one same, the numerator of claim precision (micro-averaged here). Topic only: retrieved real claims, but every pair judged related or different. No counterpart: nothing retrieved.
Hedging. For 18% of the real claims that get any directional label, the same run produces both an agreeing and an opposing claim. 50% get only agreeing claims and 32% only opposing ones. Models argue both sides twice as often on labor claims (33%) as on the rest (17%). If we count each real claim once, by its majority label, instead of counting pairs, overall agreement rises from 0.53 to 0.59. Hedging explains part of the gap to the human rate but not most of it.
A default stance. Pooled over the five vanilla runs, agreement is highest for procedural (0.62), scientific–technical (0.56) and economic (0.54) arguments and lowest for environmental (0.43), safety (0.45) and health (0.46) arguments. The models’ default positions follow the registers industry and government commenters use (Table 1) and go against the registers advocacy, individual and labor commenters use. Three quarters of labor claims are safety claims; 14% of other groups’ claims are. If we reweight labor to the other groups’ frame mix, its agreement rises from 0.44 to 0.49 (the other groups are at 0.54), so framing explains about half the labor gap. Within the safety frame, labor is still below the others (0.41 vs. 0.46).
Controls. The labor gap is not a retrieval artifact. Agreement falls with retrieval cosine for every group, but labor is lower in every cosine bin, and reweighting labor to the other groups’ cosine distribution leaves it at 0.40. It is not an agency-mix artifact either: labor is below the other groups inside FAA (0.43 vs. 0.53), OSHA (0.42 vs. 0.53) and MSHA (0.32 vs. 0.55). And it is not specific to the gpt-4o-mini judge (Appendix § 10.§ 6).
Human baselines. Labor’s low agreement could just mean that no one agrees with labor. The baselines we have say that is largely the case. Inside the same docket, randomly sampled real commenters agree with other commenters 75% of the time overall and 67% of the time with labor, though we only have six labor pairs (too few for Table 18, which omits cells under ten pairs). Across dockets, we ran real human claims from other dockets of the same agency through the same retrieval and judging pipeline with an independent open-weights judge: they agree with labor claims 14% of the time and with every other group 23–36% of the time (112 labor pairs; 27% overall). Against these baselines the LLMs are not more opposed to labor than other commenters are. They take the median commenter’s side, and the median commenter does not make labor’s safety demands. The right test is a within-docket human baseline, judged by the same judge, with enough labor pairs. We did not run it.
LLM comment characterization: direction and argument divergence
To characterize how LLM-generated comments differ from human comments, we compare the distribution of comment direction (what the commenter wants the agency to do) and argument type (what reasoning the commenter uses) between LLM-generated and real human claims across the 558-docket benchmark population. Each claim is labeled with a direction from {modify, oppose, support, inform, exempt} and an argument type from {procedural, technical, economic, legal, experiential, normative} using the same taxonomy prompt applied at generation time (Appendix § 11).
| Argument | LLM | Human | |
| Procedural | 60.3 | 15.7 | +44.6 |
| Technical | 16.9 | 22.1 | -5.2 |
| Legal | 12.4 | 16.2 | -3.8 |
| Economic | 5.1 | 13.6 | -8.4 |
| Normative | 4.5 | 26.6 | -22.1 |
| Experiential | 0.8 | 5.8 | -5.0 |
Table 21. (b) Argument type (%).
Table 21. LLM (gpt-5-mini, vanilla, 558 dockets) vs. human comment distributions. in pp. LLMs default to modify (+42pp) and procedural (+45pp) while under-representing oppose, normative, and experiential modes. Red = LLM overrepresents; blue = underrepresents.
| Direction | LLM | Human | |
| Modify | 85.9 | 43.8 | +42.1 |
| Oppose | 5.1 | 30.4 | -25.4 |
| Support | 2.2 | 12.2 | -10.0 |
| Inform | 5.7 | 10.9 | -5.2 |
| Exempt | 1.1 | 2.7 | -1.5 |
Table 21. (a) Comment direction (%).
The divergence is striking along two axes. First, LLMs adopt a narrow modify-everything stance (85.9% vs. 43.8% human), almost never opposing or supporting provisions outright. Real commenters take diverse directional stances: 30% oppose provisions entirely, 12% express support, and only 44% suggest modifications. Second, LLMs default to procedural objections (60.3% vs. 15.7%) — “this provision lacks clarity,” “implementation details are insufficient” — while dramatically under-representing normative arguments (4.5% vs. 26.6%) that invoke values, fairness, environmental protection, or public health, and experiential arguments (0.8% vs. 5.8%) grounded in lived experience. The mean per-docket Jensen–Shannon divergence is 0.252 for direction and 0.336 for argument type, confirming that these gaps are systematic rather than driven by outlier dockets. In short, LLMs sound procedurally sophisticated but lack the substantive diversity — the economic data, technical specifics, normative convictions, and personal experience — that characterizes real public engagement.
All LLM prompts used in the paper
We list every LLM prompt directly underpinning a number reported in this paper. We exclude exploratory or downstream-only prompts (e.g., causal-analysis labels, the legal-framework 10-class classifier, claim-taxonomy scope/argument/direction labels, which follow the frame-label prompt's pattern). Prompts are grouped by the pipeline stage they support (document parsing, matching, comment features, and benchmark generation and judging) and reported with the model used and an estimate of how many calls were made. Total LLM API spend across all stages reported here is approximately $340 (extraction + matching + audit + benchmark generation across 8 models + Axis B/C judging).
Document parsing
Provision extraction. Extracts the atomic obligations / permissions / prohibitions / conditions / exceptions from each proposed and final rule (§ 2). Model: gpt-5-mini (reasoning_effort=low, flex tier). Calls: ∼40K rule documents (17,198 proposed + 22,400 final), $67.60 total including a two-pass repair. In our prompt, we use the term deontic units which refers to an imperative established by a law or regulation.
SYSTEM: You are a legal analyst specializing in U.S. federal regulatory text. Your task is to extract all deontic units — obligations, permissions, prohibitions, and requirements — from regulatory text.
A "deontic unit" is any statement that imposes, permits, or prohibits an action on a specific actor. These are signaled by modal verbs (shall, must, may, may not, should, is required to, is prohibited from, etc.) or by language that functionally creates a requirement even without a modal verb.
IMPORTANT DISTINCTIONS: - Extract deontic units ONLY from the actual regulatory text (the part that will become law). - Do NOT extract from the preamble discussion, background, or DATES/ADDRESSES sections. - The regulatory text typically appears after "List of Subjects" or starts with amendments to specific CFR parts (e.g., "PART 80—...", "§ 123.45").
OUTPUT FORMAT (JSON array; each unit has): { "id": <int>, "section": "<CFR ref or empty>", "obligation_text": "<verbatim text>", "actor": "<who is obligated>", "modal": "<shall|must|may|may not|...>", "action_summary": "<plain-language summary>", "type": "<obligation|permission|prohibition|condition|exception>", "parent_id": <id or null>, "sub_unit_type": "<threshold|condition|exception|...|null>" }
HIERARCHY: A parent states a general obligation; child sub-units specify thresholds, conditions, exceptions, alternatives. Not every (a)/(b) creates a sub-unit.
[6 worked examples follow in the actual prompt.]
USER: Extract all deontic units from the following regulatory document. Document title: {title} Agency: {agency} Docket: {docket_id} <regulatory_text>{text}</regulatory_text>
Response extraction. Llama-3.3-70B reads each final rule and returns the comment–response pairs it finds as JSON, one object per pair, with the fields content_of_comment, summarized_content_of_comment, response_to_comment, quoted_or_paraphrased, type_of_response, reference_scope, commenter_identifiers_text, response_engagement_type and rule_change_outcome. The stance field type_of_response takes the values accepted, partially_accepted, disagree, acknowledged, unclear and not_relevant; the prompt conveys these through five worked examples rather than a definition list. Finding 1 reports accepted as agreement, partially_accepted as partial agreement and disagree as disagreement. The prompt, with its examples, is in the released code (extract_responses_from_gov.py).
Match 1 (proposed final)
Aligns each proposed obligation to its surviving final-rule counterpart and assigns one of {same, modified, removed, added} per unit. Model: gpt-5-mini (flex). Calls: 7,574 dockets (single multi-pair call per docket; 6,954 of them have units on both sides and enter Figure 2b), $28 total.
SYSTEM:
You are a legal analyst comparing regulatory obligations between a PROPOSED rule and the
corresponding FINAL rule from U.S. federal rulemaking.
You will be given two numbered lists:
Proposed obligations (p1, p2, ...) Final obligations (r1, r2, ...)
Classify every obligation into exactly one of:
SAME: substantively identical (minor wording / renumbering differences ignored).
MODIFIED: clear counterpart but substantive change (numbers, deadlines, scope, conditions).
REMOVED: proposed obligation has no counterpart in the final rule.
ADDED: final obligation has no counterpart in the proposed rule.
Every proposed unit must be SAME, MODIFIED, or REMOVED. Every final unit must be SAME, MODIFIED,
or ADDED. SAME / MODIFIED are 1-to-1; if unsure SAME vs MODIFIED, prefer MODIFIED; if unsure
MODIFIED vs REMOVED/ADDED, prefer MODIFIED.
USER:
PROPOSED RULE OBLIGATIONS: {proposed_units}
FINAL RULE OBLIGATIONS: {final_units}
Output (no other text):
same: (p_, r_), (p_, r_)
modified: (p_, r_), ...
removed: p_, p_
added: r_, r_
Match 2 (claim → proposed unit)
Calibrated opposing-sides judge: decides whether a single comment claim addresses a specific proposed-rule provision. This is the prompt that drives the Match 2 corpus (1.99M pairs, $28); the Axis B pair judge is a separate prompt (Appendix C.12). Model: gpt-5-mini (flex, reasoning_effort=minimal).
SYSTEM:
You are a legal analyst determining whether a public comment addresses a specific regulatory
provision. The claim may agree with, disagree with, or take an opposing stance toward the
provision — all of these count as "addressing" it. A claim only fails to address the provision
if it is on a completely different regulatory topic.
USER:
Proposed rule obligation:
Section: {unit_section} Type: {unit_type}
Summary: {unit_summary}
Public comment claim:
{claim_text}
Does this claim specifically address this obligation (whether agreeing or disagreeing)?
Answer YES or NO.
Match audit
Independent gpt-5 (medium reasoning) review of the Match 1 pipeline’s REMOVED+NEW pairs and MODIFIED pairs to flag missed merges and spurious matches. Model: gpt-5 (reasoning_effort=medium, flex). Calls: 7,574 dockets, $57.55. Output: 1,966 merges + 1,770 unmatches + 785 rewires applied to the v3 dataset.
You are auditing the output of a regulatory text-matching pipeline. The pipeline matched
obligations from a PROPOSED federal rule to obligations in the corresponding FINAL rule,
labeling each as "survived", "modified", "removed", or "new". The pipeline uses bi-encoder +
cross-encoder semantic matching; it works well for textually similar pairs but commonly MISSES
cases where a regulatory concept was renumbered, restructured, split, or moved to a different
CFR section. These mistakes typically appear as a "removed" proposed + "new" final pair that are
actually the same concept. It can also produce SPURIOUS "modified" pairs where two units have
surface similarity but address different obligations.
Identify three kinds of relabel suggestions:
1. MERGE — a (REMOVED proposed) + (NEW final) pair that should be a single MODIFIED pair.
2. UNMATCH — a current (MODIFIED <-> MODIFIED) pair that should NOT be linked.
3. REWIRE — a current (MODIFIED <-> MODIFIED) pair where the proposed unit's true counterpart
is a different unit shown.
PROPOSED-side units: {proposed_list}
FINAL-side units: {final_list}
Output one line per suggestion (or NO_SUGGESTIONS):
MERGE P_R<i> <-> F_N<j>: <brief reason>
UNMATCH P_M<i> <-> F_M<j>: <brief reason>
REWIRE P_M<i>: was F_M<j>, should be F_N<k>: <brief reason>
Be CONSERVATIVE -- only flag with HIGH confidence.
Cosmetic-only re-judge
Re-classifies pairs that the v1 14-class taxonomy called cosmetic_only, asking whether the change is truly cosmetic vs. substantive. Used to refine the dataset’s cosmetic_only share from 25.7% (raw v1) to a more reliable distribution; produced cosmetic_review_v3.jsonl (3,485 confirmed cosmetic, 1,972 actually substantive). Model: gpt-5-mini (flex, minimal reasoning).
SYSTEM:
You are a legal analyst comparing a regulatory provision in a PROPOSED rule to its counterpart
in the FINAL rule. Both appear to be the "same" provision with different wording. Determine
whether the change is PURELY COSMETIC (no real difference in legal effect on a regulated party)
OR SUBSTANTIVE (the obligation, scope, threshold, condition, penalty, or exception meaningfully
changed).
COSMETIC if it is one or more of:
- Wording reorganization, sentence restructuring, simplification with no change to who is
obligated, what they must do, when, or how.
- Section/paragraph renumbering or cross-reference updates pointing to the same substantive rule.
- Replacing placeholder text with the same content (e.g., "the date" → an explicit date).
- Updating document titles, edition numbers (e.g., "Bulletin Rev 2" → "Rev 3") when content is
unchanged.
- Removing redundant clauses without changing the underlying obligation.
- Renaming an agency office or contact.
- Adding examples or clarifying language without changing scope or requirement.
SUBSTANTIVE if it is one or more of:
- A specific number, threshold, percentage, deadline, quantity changed.
- A new obligation added, or an existing one removed.
- The set of regulated parties, applicability, exemptions, or coverage changed.
- Penalties, enforcement mechanisms, or compliance methods changed.
- A new exception/exclusion added, or an existing one removed.
- Required documentation, reporting, recordkeeping changed in substance.
When in doubt prefer SUBSTANTIVE.
USER:
Section/identifier (proposed): {section_proposed}
Section/identifier (final): {section_final}
Type: {unit_type}
PROPOSED text: {proposed_text}
FINAL text: {final_text}
Output (exactly two lines):
COSMETIC or SUBSTANTIVE
<one short sentence (<25 words) explaining why.>
Topic-mismatch re-judge
Re-classifies pairs flagged topic_mismatch to confirm whether they are wrong matches or legitimate same-topic revisions. Produced topic_mismatch_review_v3.jsonl (2,468 confirmed mismatch, 1,501 actually valid). Model: gpt-5-mini (flex, minimal). Same input format as Appendix § 11.§ 5; output TRUE_MISMATCH or VALID_MATCH with one-sentence reason.
14-class modification taxonomy
Classifies each modified (proposed final) unit into one of 14 concrete change types. Drives the modification typology reported in Appendix § 10.§ 1. Model: gpt-5-mini (flex, minimal). Calls: 12,812 modifications, $2.57 total.
SYSTEM:
You are comparing a PROPOSED regulatory obligation to its FINAL version. Classify the change
into EXACTLY ONE of these 14 categories (each is a CONCRETE textual change pattern, not an
abstract direction):
THRESHOLD/SCOPE:
threshold_or_scope_tighter | threshold_or_scope_looser
EXCEPTIONS:
exception_added | exception_removed
TIMELINE:
timeline_shortened | timeline_extended
ENFORCEMENT:
enforcement_strengthened | enforcement_weakened
NO BURDEN CHANGE:
definition_clarified | cosmetic_only
OFF-AXIS:
agency_obligation_change | respondent_protection_added
PROBLEM CASES:
mixed_substantive | topic_mismatch
DECISION RULES:
1. Focus on impact on REGULATED PARTIES (operators, manufacturers, applicants), NOT the agency.
2. Adding states/parties to a "may" provision = threshold_or_scope_looser (more flexibility).
3. If texts are semantically equivalent ("within X" = "before X") → cosmetic_only.
4. If texts are about totally different obligations → topic_mismatch.
5. Adding a respondent-protection clause = respondent_protection_added.
6. Pick the SINGLE MOST INFORMATIVE category; if multiple apply, pick the one with the
largest effect on regulated parties.
[8 worked examples follow.]
USER:
Proposed: Section: {p_section} Summary: {p_summary} Text: {p_text}
Final: Section: {f_section} Summary: {f_summary} Text: {f_text}
Respond exactly:
label: <one of 14 labels>
reason: <one short phrase identifying the textual signal>
Specialist vs. generalist commenter classification
Identifies whether a comment author writes from substantive domain expertise (legal/technical/insider/experiential) vs. as an ordinary citizen. Used as the gold-label seed for the corpus-wide TF-IDF + LR distillation that produces the per-comment specialist score reported in § 3 (Finding 3, “ specialist rate”). Model: gpt-5-mini (flex, minimal). Calls: 5,987 gpt-5-mini-labeled seed comments seeded the distillation; the LR distillate was then applied to all 8.27M comments at no LLM cost.
SYSTEM:
SPECIALIST: the author brings substantive domain expertise — they cite statutes/case law/
regulations correctly, use technical/scientific vocabulary appropriately, demonstrate insider
knowledge of an industry's operations, OR write from substantive occupational expertise. The
author *contributes information* the agency would not already have.
GENERALIST: the author writes from ordinary citizen perspective — values, opinions, generic
concerns, brief support/oppose statements, mass-campaign language, or general lived experience
without substantive expertise. The author *expresses preference* but does not contribute
specialized information.
Identify which expertise modes (if any) are present:
- legal_expert
- technical_scientific
- industry_insider
experiential_specialist
A short comment can still be specialist; a long emotional letter can still be generalist.
USER:
COMMENT (truncated if long): {text}
Output JSON only:
{"is_specialist": true|false, "expertise_modes": ["legal_expert", ...], "reason": "<25 words"}
Primary-frame label
Assigns each comment claim one of 12 primary frames (health, safety, economic_cost_benefit, legal_constitutional, scientific_technical, environmental, equity_justice, religious_moral, national_security, procedural, other, unknown). Drives Table 1 (stakeholder × frame matrix) and the frame shares of LLM claims in § 4.2. Model: gpt-5-mini (flex, minimal). Calls: 27,936 claims for the original gpt-5-mini reference run; for the per-model benchmark sweep, an additional 43,371 unique real claims (one-time cache) plus the LLM claims of each evaluated model (14K–100K each).
USER:
Classify the dominant framing of this public-comment claim. Pick exactly ONE from this enum:
- health: public health, disease, medical outcomes
- safety: accidents, worker safety, product safety
- economic_cost_benefit: costs, jobs, competitiveness, ROI, market effects
- legal_constitutional: statutory authority, constitutional rights, APA, takings, due process
- scientific_technical: evidence, methodology, engineering, measurement
- environmental: pollution, ecosystems, climate, wildlife
- equity_justice: fairness, disparate impact, environmental justice, civil rights
- religious_moral: ethical or values-based appeals
- national_security: defense, foreign threats, critical infrastructure
- procedural: process objections, transparency, comment-period mechanics
- other
- unknown
Claim: {claim_text}
Respond with one word, the enum value.
Anticipated-comment generation: vanilla
Asks the model to produce predicted commenter claims given the docket’s pooled provisions. Used per docket per evaluated model. Calls per evaluated model: 558 (one per docket).
USER:
Below are provisions from a proposed federal rule.
Your task: What are public comments members of the public are likely to submit on the
following provisions? For each comment, express it as a specific claim — a concrete point
about a specific provision.
Format each claim on its own line, prefixed with the provision number it addresses:
[P1] claim about provision 1
[P3] claim about provision 3
[P1] another claim about provision 1
Generate as many distinct claims as you think real commenters would submit. Be specific —
reference particular requirements, thresholds, or actors mentioned in the provisions. Consider
perspectives from industry, advocacy groups, individuals, and legal experts.
PROPOSED RULE PROVISIONS:
{provisions}
Generate the anticipated public comments:
Anticipated-comment generation: subgroup persona
Same task structure as Appendix § 11.10 but conditioned on one of seven stakeholder personas (industry, labor, advocacy, expert, gov, lawfirm, individual). The seven personas are run as 7 independent prompts per docket per model and the outputs are pooled. Calls per evaluated model: . Each persona description is a 1-paragraph descriptor naming the stakeholder’s typical evidence base and concerns.
USER:
Below are provisions from a proposed federal rule. Write public comments FROM THE
PERSPECTIVE OF: {stakeholder_label}.
{stakeholder_persona}
Generate the public-comment claims this stakeholder would submit. Each claim should
be a concrete point about a specific provision, written in the voice and concerns characteristic
of {stakeholder_label}.
Format each claim on its own line, prefixed with the provision number:
[P1] claim about provision 1
Be specific — reference particular requirements, thresholds, or actors mentioned in the
provisions, AND write in a way that reflects this stakeholder's typical concerns, vocabulary,
and framings.
PROPOSED RULE PROVISIONS:
{provisions}
{stakeholder_label} comments on this proposal:
==== STAKEHOLDER PERSONA TEXT (one used per call) ====
industry: "a regulated company, trade association, or industry coalition. You focus on
compliance cost, operational feasibility, market effects, and ROI. You cite
economic data and industry-specific operational details."
labor: "a labor union representing workers in the regulated industry. You focus on worker
safety, job protection, fair labor standards, and economic impact on workers. You
cite worker statistics and on-the-job experience."
advocacy: "an environmental, civil rights, or public-interest advocacy organization. You
focus on broader public good, environmental protection, equity, and the
regulation's role in achieving social goals. You cite scientific studies, public-
health data, and advocacy research."
expert: "an academic researcher, professional society, or expert organization. You focus on
technical correctness, scientific accuracy, peer-reviewed evidence, and
methodological soundness. You cite research, technical specifications, and
disciplinary expertise."
gov: "a state or local government agency that interacts with this federal regulation. You
focus on inter-jurisdictional coordination, implementation feasibility for state
agencies, and federalism issues. You cite state-level data and operational details."
lawfirm: "a law firm or bar association representing regulated parties. You focus on
statutory authority, constitutional issues, APA procedural requirements, due-
process implications, and legal interpretation. You cite case law, statutes, and
regulatory history."
individual: "an ordinary individual citizen affected by this rule. You focus on personal
experience, lived impact on you or your family/community, and concrete harms or
benefits you would face. You speak in first person about direct experience."
Axis B pair-alignment judge
For each validated real claim (i.e. each (comment proposal) match) the judge labels the pair as one of {same, related, different, opposing}. Model: gpt-4o-mini, ten pairs per call, with the rubric below. We pair each LLM claim with its three nearest real claims in the same docket (MiniLM cosine ). This gives 35K pairs for gpt-4o-mini vanilla up to 263K for Llama-3.1-8B persona, over all 558 dockets (504–558 dockets per run have at least one judged pair).
USER:
You compare two regulatory comment claims (A and B) that target the SAME proposed regulatory
provision. The two claims may come from commenters on opposing sides of the issue: agreeing,
disagreeing, raising distinct concerns, or expressing the same point. Choose ONE label that
best describes the relationship of B to A:
- same: B states the same point as A (a paraphrase or restatement; same direction).
- related: B is on the same sub-topic as A but makes a different point (compatible direction).
- different: B addresses a different aspect/concern of the provision (no semantic overlap).
opposing: B takes a contrary position to A on the same point (e.g., A wants extension, B wants no change).
Claim A: {a} Claim B: {b}
Respond with one word: same, related, different, or opposing.
Full-comment generation prompts and examples
In addition to claim-list generation (Appendices § 11.10–§ 11.11), we generate full natural-language comments in paragraph form for each docket. These are used for the frame and style analysis (§ 4.2). We evaluate five generation methods: vanilla, persona-conditioned, RAG-augmented, few-shot, and RAG with tool calling. Full-comment prompts follow the same provision input as Appendices § 11.10–§ 11.11 but instruct the model to “write in full paragraphs” rather than produce tagged claim lists.
Full-comment generation: vanilla
USER:
Below are provisions from a proposed federal rule.
Write a public comment responding to these provisions, as a real member of
the public would. Write in full paragraphs — cite specific evidence,
reference relevant statutes or data, draw on operational experience where
relevant, and propose specific alternatives where you disagree.
Do not use bullet points or numbered lists. Write naturally, as if
submitting a real comment to the Federal Register.
PROPOSED RULE PROVISIONS:
{provisions}
Public comment:
Full-comment generation: RAG-augmented
Retrieves evidence from a BM25 index over 30K real evidence-bearing comments (extracted per Appendix § 10.§ 3) and stuffs retrieved snippets into the prompt context.
Full-comment generation: few-shot
Includes two real public comments from non-benchmark dockets as style demonstrations before the generation instruction.
Full-comment generation: RAG with tool calling
Provides the model with a retrieval tool; the model decides what queries to issue before writing. Up to 3 tool-calling rounds are permitted.
Example generated comments. Figure 9–Figure 12 show excerpts from comments generated by different methods and personas on the same docket (EPA-HQ-OAR-2017-0091, proposing 2018 renewable fuel percentage standards). The full comments range from 3K to 16K characters. Notable differences include: gpt-5 vanilla writes as a “Midwestern small-business owner,” cites GREET model and Argonne National Lab data, and references Clean Air Act §211(o)—a highly informed, policy-analyst style; gpt-5-mini RAG writes as a “soybean farmer,” grounds claims in retrieved evidence (“Diamond Green Diesel producing 160M gallons”); gpt-5-mini +individual persona uses emotional first-person testimony (“I saw friends and cousins pick up second jobs or leave town”); and gpt-4o-mini vanilla produces the shortest comment (3.2K chars) with vaguer evidentiary claims.
Figure 9. Full-comment generation: gpt-5, vanilla. The model adopts the persona of a “Midwestern small-business owner,” cites the GREET model, Argonne National Lab, and specific CFR sections. Comment is highly detailed and policy-analyst in style.
Figure 12. Full-comment generation: gpt-4o-mini, vanilla prompt. Notably shorter (3.2K vs 7–9K chars), writes as a generic “concerned citizen,” and uses vaguer evidentiary claims (“the National Biodiesel Board has reported. . .”) without specific figures.
Figure 10. Full-comment generation: gpt-5-mini, RAG-augmented. The model writes as a “soybean farmer,” grounds claims in retrieved evidence from real prior comments (e.g., specific production figures for Diamond Green Diesel).
Figure 11. Full-comment generation: gpt-5-mini, +individual persona. The model adopts a first-person voice (“I saw friends and cousins pick up second jobs or leave town”), frames regulatory impact through lived experience, and still references specific statutory provisions.
References
- [1]Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis 31, 3 (2023), 337–351. doi:10.1017/pan.2023.2DOI
- [2]Jake Auchincloss. 2023. U.S. Politician Has ChatGPT Write His Speech. Office of Congressman Jake Auchincloss, Press Release. https://auchincloss.house.gov/media/in-the-news/us-politician-has-chatgpt-write-his-speech
- [3]Yauhen Babakhin, Radek Omsulksi, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, and Even Oldridge. 2025. Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks. arXiv:2511.07025 https://arxiv.org/abs/2511.07025
- [4]Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in neural information processing systems 35 (2022), 38176–38189.arxiv.org/abs/2211.15006
- [5]Steven J. Balla, Alexander R. Beck, Elizabeth Meehan, and Aryamala Prasad. 2022. Lost in the Flood? Agency Responsiveness to Mass Comment Campaigns in Administrative Rulemaking. Regulation & Governance 16, 1 (2022), 293–308.
- [6]Steven J. Balla, Reeve Bull, Bridget C.E. Dooling, Emily Hammond, Michael Herz, Michael A. Livermore, and Beth Simone Noveck. 2022. Responding to Mass, Computer-Generated, and Malattributed Comments. Administrative Law Review 74, 1 (2022), 95–176. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3989307
- [7]Pamela Ban and Hye Young You. 2019. Presence and Influence in Lobbying: Evidence from Dodd-Frank. Business and Politics 21, 2 (2019), 267–295.DOI
- [8]Andrei Z. Broder. 1997. On the Resemblance and Containment of Documents. In Proceedings of the Compression and Complexity of Sequences (SEQUENCES ’97). IEEE, 21–29. doi:10.1109/SEQUEN.1997.666900
- [9]Daniel Carpenter, Angelo Dagonel, Devin Judge-Lord, Christopher T. Kenny, Brian Libgober, Steven Rashin, Jacob Waggoner, and Susan Webb Yackee. 2024. Inequality in Administrative Democracy: Large-Sample Evidence from American Financial Regulation. (2024). Working paper, available on SSRN.DOI
- [10]Center for Democracy & Technology. 2025. Exploring the 2024 Federal AI Inventories: Key Improvements, Trends, and Continued Inconsistencies. CDT Insights. https://cdt.org/insights/exploring-the-2024-federal-ai-inventories-key-improvements-trends-and-continued-inconsistencies/
- [11]Cary Coglianese. 2006. Citizen Participation in Rulemaking: Past, Present, and Future. Duke Law Journal 55, 5 (2006), 943–968.
- [12]Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mosse, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. 2024. Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. In International Conference on Machine Learning. PMLR, 9346–9360.arxiv.org/abs/2404.10271
- [13]Mariano-Florentino Cuéllar. 2005. Rethinking Regulatory Democracy. Administrative Law Review 57, 2 (2005), 411–499. https://www.jstor.org/stable/40712038
- [14]Kenneth Culp Davis. 1970. Administrative Law Treatise (1st ed.). West Publishing Co., St. Paul, MN. § 6.15, at 283.
- [15]John M. de Figueiredo. 2006. E-Rulemaking: Bringing Data to Theory at the Federal Communications Commission. Duke Law Journal 55, 5 (2006), 969–993.DOI
- [16]Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. [n. d.]. Towards Measuring the Representation of Subjective Global Opinions in Language Models. In First Conference on Language Modeling.arxiv.org/abs/2306.16388
- [17]Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards Measuring the Representation of Subjective Global Opinions in Language Models. In First Conference on Language Modeling (COLM). https://arxiv.org/abs/2306.16388
- [18]Sean Gailmard and John W. Patty. 2017. Participation, Process, and Policy: The Informational Value of Politicised Judicial Review. Journal of Public Policy 37, 3 (2017), 233–260.DOI
- [19]Cassandra Handan-Nader. 2023. Do fake online comments pose a threat to regulatory policymaking? Evidence from Internet regulation in the United States. Policy & Internet 15, 1 (2023), 100–116. doi:10.1002/poi3.327
- [20]Wesley Newcomb Hohfeld. 1913. Some Fundamental Legal Conceptions as Applied in Judicial Reasoning. Yale Law Journal 23, 1 (1913), 16–59. doi:10.2307/785533
- [21]Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I. Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. 2024. Collective Constitutional AI: Aligning a Language Model with Public Input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT).DOI
- [22]ICF International. 2024. Using Gen AI to Process Public Comments Faster. ICF case study. https://www.icf.com/clients/technology/regulations-gov-gen-ai-public-comment-analysis
- [23]ICF International. n.d.. CommentWorks: Public Comment Analysis Software for Federal Rulemaking. ICF Regulatory Policy product page. https://www.icf.com/work/regulatory-policy/commentworks
- [24]Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). doi:10.18653/v1/2020.emnlp-main.550DOI
- [25]Cornelius M. Kerwin and Scott R. Furlong. 2019. Rulemaking: How Government Agencies Write Law and Make Policy (5 ed.). CQ Press, Thousand Oaks, CA.DOI
- [26]Hannah Rose Kirk, Aleksander Whifield, Paul Röttger, Andrew M. Bean, Katerina Margatina, Rafael Mosquera, Juan Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. The PRISM Alignment Dataset: What Participatory, Representative and Individualized Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS 2024 Datasets and Benchmarks Track). https://arxiv.org/abs/2404.16019
- [27]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Yong Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.arxiv.org/abs/2309.06180
- [28]Andrew Leahy. 2026. Only One-Third of Proposed Regulatory Obligations Survive to the Final Rule. Yale Journal on Regulation, Notice & Comment Blog. https://www.yalejreg.com/nc/only-one-third-of-proposed-regulatory-obligations-survive-to-the-final-rule/
- [29]Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). doi:10.1145/3491102.3502030DOI
- [30]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020), 9459–9474.arxiv.org/abs/2005.11401
- [31]Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jiangfeng Gao, and William B Dolan. 2016. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 994–1003.arxiv.org/abs/1603.06155
- [32]Brian D. Libgober. 2020. Strategic Proposals, Endogenous Comments, and Bias in Rulemaking. Journal of Politics 82, 2 (2020), 642–656.
- [33]Nina A. Mendelson. 2011. Rulemaking, Democracy, and Torrents of E-Mail. George Washington Law Review 79, 5 (2011), 1343–1380. https://www.gwlr.org/wp-content/uploads/2012/08/79-5-Mendelson.pdf
- [34]Keith Naughton, Celeste Schmidt, Susan Webb Yackee, and Xueyong Zhan. 2009. Understanding Commenter Influence During Agency Rule Development. Journal of Policy Analysis and Management 28, 2 (2009), 258–277. doi:10.1002/pam.20426DOI
- [35]David C. Nixon, Robert M. Howard, and Jeff R. DeWitt. 2002. With Friends Like These: Rule-making Comment Submissions to the Securities and Exchange Commission. Journal of Public Administration Research and Theory 12, 1 (2002), 59–76.DOI
- [36]Rodrigo Nogueira, Wei Yang, Kyounghun Cho, and Jimmy Lin. 2019. Multi-Stage Document Ranking with BERT. arXiv:1910.14424 https://arxiv.org/abs/1910.14424
- [37]Pacific Northwest National Laboratory. 2024. Testing Artificial Intelligence Tools to Streamline the Public Comment-Review Process for NEPA Environmental Reviews. Technical Report PNNL-35927. Pacific Northwest National Laboratory. https://www.pnnl.gov/sites/default/files/media/file/PolicyAI%20Comment%20Processing%20White%20Paper%20V1.pdf
- [38]Elinor Poole-Dayan, Jiayi Wu, Taylor Sorensen, Jiaxin Pei, and Michiel A. Bakker. 2026. Benchmarking Overton Pluralism in LLMs. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2512.01351
- [39]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). doi:10.18653/v1/D19-1410DOI
- [40]Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose Opinions Do Language Models Reflect? In Proceedings of the 40th International Conference on Machine Learning (ICML). https://arxiv.org/abs/2303.17548
- [41]Jacques Savoy. 2024. How good is GPT at writing political speeches for the White House? arXiv preprint arXiv:2412.14617 (2024).arxiv.org/abs/2412.14617
- [42]Jennifer L. Selin and David E. Lewis. 2018. Sourcebook of United States Executive Agencies (2nd ed.). Technical Report. Administrative Conference of the United States. https://www.acus.gov/publication/sourcebook-united-states-executive-agencies-second-edition
- [43]Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Jiwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning. 46280–46302.arxiv.org/abs/2402.05070
- [44]Matthew C. Stephenson. 2006. A Costly Signaling Theory of “Hard Look” Judicial Review. Administrative Law Review 58 (2006), 753–813.
- [45]U.S. Congress. 1946. 5 U.S.C. Section 553 — Rule making. https://www.law.cornell.edu/uscode/text/5/553 Administrative Procedure Act of 1946, as codified.
- [46]U.S. Congress. 2024. H.R. 7528 — Comment Integrity and Management Act of 2024, 118th Congress. Introduced March 5, 2024 by Rep. Clay Higgins; passed House by voice vote May 6, 2024. https://www.congress.gov/bill/118th-congress/house-bill/7528
- [47]U.S. Department of Health and Human Services, Administration for Children and Families, Office of Head Start. 2024. ACF-OHS NPRM 2024-18279: Public Comment Analysis using Large Language Models. GitHub repository. https://github.com/HHS/acf-ohs-nprm-2024-18279
- [48]U.S. District Court for the Southern District of New York. 2023. Opinion and Order on Sanctions, Mata v. Avianca, Inc. https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1:2022cv01461/575368/54/ Sanction order issued June 22, 2023 by Judge P. Kevin Castel for submission of ChatGPT-fabricated case citations.
- [49]U.S. Federal Communications Commission. 2024. ECFS: Electronic Comment Filing System. fcc.gov/ecfs. https://www.fcc.gov/ecfs
- [50]U.S. General Services Administration. 2024. Regulations.gov: About the eRulemaking Initiative. GSA eRulemaking Program Management Office. https://www.regulations.gov/about
- [51]U.S. Government Accountability Office. 2021. Federal Rulemaking: Selected Agencies Should Fully Describe Public Comment Data and Their Limitations. Technical Report GAO-21-103181. U.S. Government Accountability Office. https://www.gao.gov/products/gao-21-103181
- [52]U.S. Securities and Exchange Commission. 2024. Comments on Proposed Rules. sec.gov. https://www.sec.gov/rules-regulations/regulatory-actions/proposed-rules
- [53]Wendy E. Wagner. 2010. Administrative Law, Filter Failure, and Information Capture. Duke Law Journal 59, 7 (2010), 1321–1432. https://scholarship.law.duke.edu/dlj/vol59/iss7/2/DOI
- [54]Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663 https://arxiv.org/abs/2412.13663
- [55]Jason Webb Yackee and Susan Webb Yackee. 2006. A Bias Towards Business? Assessing Interest Group Influence on the U.S. Bureaucracy. Journal of Politics 68, 1 (2006), 128–139. doi:10.1111/j.1468-2508.2006.00375.xDOI
- [56]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS 2023 Datasets and Benchmarks Track). https://arxiv.org/abs/2306.05685





