From DQI to C-DQI: Automatic Codebook Generation and Evaluation for Measuring Computational Discourse Quality Index

Image not extracted

Introduction

Deliberation is the interactive process where individuals engage in discussion to reach rational conclusions, occurring in both formal political bodies and local civic settings. At its core lies the premise that political decision-making should be talk-centric rather than vote-centric [37]. Effective democratic deliberation is crucial for civic engagement and the health of legislative institutions [35, 34, 40]. Prominent literature identifies thoughtful analysis, open-mindedness, mutual respect, and participants’ willingness to revise their initial preferences through discourse as prerequisites for this efficacy [37, 9].

Quantifying deliberation quality enables its application to downstream tasks, such as identifying anomalous deliberative behavior, evaluating the performance of legislators, and alerting civic facilitators when participant satisfaction could be improved [34]. But while numerous factors have been argued to affect discourse quality, such as domination, issue polarization, and agenda setting [20, 35], quantifying this quality remains challenging due to the inherent competing interests between interlocutors, rhetorical ambiguity, and the variance in acceptable standards across societies and domains.

Various methods, including computational frameworks, have been proposed for measuring multi-party discourse quality [37, 3, 5]. A common protocol for quality assessment involves experts evaluating live multi-party discourse using a theoretical-scientific scoring schema. This schema, known as a codebook (CB), comprises categories that reflect rationality, reciprocity, and civility, and uses ordinal scales to individually assess each criterion [3, 15, 10, 7 Steenbergen] (et al., 2003). However, this paradigm has various limitations. First, rigid, theory-based schemas often struggle to generalize across topic-dependent or culture-dependent nuances. Second, collecting human annotation is labor-intensive and inherently subject to individual biases [31, 41, 6].

To automate the application of such schemas, researchers have employed supervised machine learning [11, 12, 4] and Large Language Models (LLMs) [5, 21, 8]. However, both approaches still rely upon the same rigid frameworks.

In this work, we propose Computational Discourse Quality Index (C-DQI), a methodology for the definition and application of CBs for assessing discourse quality. We explore four distinct workflows for C-DQI CB generation. We start with a baseline adapted from Steenbergen et al. [37]’s Discourse Quality Index (DQI) and implement three pipelines that deploy LLMs for CB induction, differing in their balance of top-down (theory-

based) and bottom-up (data-driven) approaches.

To evaluate the CBs, we propose both intrinsic and extrinsic measures. Following Steenbergen et al. [37], who discuss the importance of robustness in a discourse quality index, and acknowledging that LLMs are inherently pre-trained with internal biases [25], we argue that the consensus of multiple models indicates that the CBs’ criteria are reliable for LLM annotation, rather than artifacts of a single model. We thus propose to define Inter-Model Agreement (IMA) scores across multiple LLMs to capture intrinsic reliability. A higher IMA score indicates a clearer, more objective, and robust framework for automated protocol annotation. However, because agreement does not equate to validity, we further compare the CBs’ overall assessments against human-validated proxies for deliberation quality, utilizing both manual and automatic annotations. Figure 1 demonstrates the full C-DQI pipeline and evaluation process.

To assess the efficacy of our proposed workflows, we apply different LLMs to annotate a corpus of 100 committee meeting transcripts from the Israeli parliament, the Knesset, according to each CB. The data spans multiple governments and a diverse range of bills, providing a dynamic, culture-dependent environment for examining discourse quality. Utilizing this region-specific data demonstrates the cultural alignment challenges inherent in analyzing localized political discourse.

Our results reveal that bottom-up CBs exhibit higher IMA, while all CBs correlate comparably with the quality proxy ($\rho> 0.58$), demonstrating that data-driven schema generation yields more reliable evaluations without sacrificing validity. Through this work, our primary contributions are twofold: (1) we develop a computational tool that automatically defines and evaluates discourse quality CBs for multi-party discussions and scores the data accordingly; and (2) we provide empirical insights into the optimal balance between top-down theoretical frameworks and bottom-up, inductive approaches for yielding reliable evaluation criteria.

Background

Discourse Quality Measures. A widely applied measure for deliberative quality is the Discourse Quality Index (DQI) originally developed for the analysis of parliamentary debates [37]. The DQI is theoretically grounded in established discourse ethics and its schema includes criteria representing norms that discourse is expected to satisfy. Its unit of analysis is an individual speech, defined as “the public discourse by a particular individual delivered at a particular point in a debate” (§ 4.2 provides further details).

Yet, this de facto standard is constrained by drawbacks tied directly to the DQI framework, as well as broader limitations inherent to predefined schemas. First, despite well-defined coding categories, the DQI does not prescribe a standardized procedure for aggregating category-level annotations into a single discourse-level score, complicating its use in large-scale automated evaluation [38]. Second, according to the DQI, off-topic discourse and clarifying questions are excluded from the final score, and interruptions are only accounted for if participants complain about them, ignoring the potential for these factors to contribute to the overall quality assessment.

More generally, hard-coded CBs target specific settings and contexts, and therefore struggle to generalize to topic-dependent or culture-dependent contexts; and manually annotating them is labor-intensive and inherently subject to individuals’ biases [31, 41, 6].

Computational Discourse Quality Evaluation. Multiple efforts have operationalized discourse quality assessment by applying supervised machine learning to the DQI [11, 12, 4]. Furthermore, Kumar et al. [20] recently combined argumentation graphs with the DQI for more objective assessment. Deep learning approaches like the AQuA framework successfully bridge NLP with deliberation theory to analyze discourse semantics, though they remain resource-intensive due to the overhead of running multiple fine-tuned BERT adapters [3].

Currently, the field is actively moving toward using LLMs as evaluative agents, bypassing the limitations of manual annotation and rigid linguistic features [21, 8, 5]. Specifically, Bosley [5] evaluated the DQI using a few-shot prompting framework, achieving IMA comparable to human standards. Their work laid the foundation for our methodological study on C-DQI schemas.

Automatic Schema Discovery. To address the limitations of rigid, predefined schemas, researchers have utilized LLMs for automatic schema definition. Earlier work deploys LLMs to construct literature-review tables by prompting the model to generate a schema and then populating the table from the underlying data [26], optionally conditioned on a user-provided intent [29].

More recently, human-in-the-loop systems have been developed to automate schema definition over large corpora of unstructured text, including ScheMaTiQ [22] and Scheminer^pro (Saddrudin et al., 2025, 2026). These systems combine prompting pipelines with interactive interfaces and agentic tooling.

To our knowledge, however, the outputs of these systems have been evaluated mainly for their agreement with expert-annotated schemas, rather than being systematically assessed for intrinsic reliability, which we emphasize in our current work.

The Challenge

Without systematically evaluating codebook efficacy, the question remains: How can we derive new or adapted schemas for new settings and domains? How can we determine whether a schema is good? And, how can we avoid the undermining effects of inherent biases of individual LLMs?

A Complete Protocol for LLM-based Codebook Generation and Evaluation

To remove the reliance on rigid CBs that require manual annotation, we propose an LLM-based methodology for effective and efficient definition and application of a Computational Discourse Quality Index (C-DQI). This methodology addresses the core limitations of traditional schema-based annotations by allowing for flexible schema induction across different contexts and replacing human annotators with LLMs to mitigate subjective human biases.

In §4.1 we introduce C-DQI, used to quantify quality based on a codebook. In §4.2 we then define the computation task of codebook induction and its various variants, and finally in §4.3 we define metrics that aim to assess the reliability and validity of the generated codebook.

Computational Discourse Quality Index

We next define the notion of a Computational Discourse Quality Index (C-DQI) which is our means to define discourse quality in quantitative terms. Let $D$ be an input transcript and $C$ be a given CB. The input transcript $D$ is split into multiple segments $D = (s_1, s_2, \ldots, s_n)$, and the CB $C$ defines a set of evaluation categories $C = \{c_1, c_2, \ldots, c_m\}$ where each category is scored on a predefined scale.² For every segment-category pair $(s_i, c_j)$, an LLM annotator $M$ evaluates the text on the required scale, yielding a score $y_{i,j}$. The per-category discussion score, $S(c_j)$, is calculated as the average across all segments:

This final index, representing the overall discourse quality for the transcript, is the average over all categories:

When a single comprehensive score per transcript is required, we report the mean of the C-DQI scores produced by the ensemble of annotator models.

LLM-based Codebook Generation

We define the task of CB generation as a function from texts to a set of categories. Formally, let $T$ be a collection of texts and let $C = \{c_1, \ldots, c_n\}$ be a finite set of categories describing dimensions for evaluating discourse; the CB generation task is defined as $f(T) = C$.

Specifically, we implement four workflows to derive CBs. One is a standard baseline, and three are LLM-based — one receives academic literature as the input text (top-down), the second reviews transcripts from the domain being studied (bottom-up), and the third combines both inputs (hybrid) (see Figure 1). All prompts are available in Appendix § 10.

Vanilla C-DQI. The baseline workflow adapts the original Discourse Quality Index (DQI), which features seven coding categories: (1) participation, (2) level of justification, (3) content of justification, (4) respect for the groups to be helped by particular policies, (5) respect toward the demands of others, (6) respect toward counterarguments, and (7) constructive politics.

Deriving the final score involves coding all speeches and interpreting their criteria scores. The authors define relevant speeches as ones containing a policy proposal (originally termed a ‘demand’).

The coding process requires initial filtering to distinguish relevant excerpts, which are then scored per category. For this workflow, we adapt the authors’ original coding guidelines (Steenbergen et al., [38]) by assigning numerical values to their scales and adjust the unit of analysis from individual speeches to segments of up to 1,500 words to maintain consistency with our other CBs.

Top-Down C-DQI. To define the theory-driven (Top-Down) CB, we provide an LLM with a selection of academic literature on deliberation. We begin by prompting the literature-survey tool developed by Gupta [16] with the instruction: “List 10 foundational academic papers on democratic deliberation, in order to develop a discourse quality index”, followed by an iterative process of refining the results according to our prior knowledge and available sources. The resulting list ranges from sources studying Habermas’s discourse ethics [17] to methodological articles [37, 39]. The selection also includes frameworks for evaluating deliberative quality [23] and sources providing dimensions for evaluation rooted in deliberative theory [13].

We then instruct Gemini-3.1-Pro$^{2}$ to define metrics for evaluating the quality of discourse in the selected dataset, based on the frameworks from the selected sources. The response serves as the Top-Down CB.

Bottom-Up C-DQI. To define the inductive (Bottom-Up) CB, we prompt Gemini-3.1-Pro with 10 random transcripts to generate evaluation metrics from empirical data. To prevent contamination, these transcripts are excluded from later experiments. Ultimately, this workflow yields three CB variants: Bottom-Up-English, generated from an English prompt; Bottom-Up-Hebrew, generated from a Hebrew prompt; and Bottom-Up-Corrected, which we, as native Hebrew speakers, merged from the two, edited for content, and standardized in structure (Appendix D).

Hybrid C-DQI. The Hybrid C-DQI CB combines both the theory-led and inductive approaches. We generate this CB by instructing Google’s Gemini Notebook $3$ to synthesize the previously selected academic sources with the empirical transcripts, to create a unified set of metrics for evaluating discourse quality in general committee meetings.

LLM-based Codebook Evaluation

CB Reliability. We define the measure of reliability as the consistency of the generated CBs, calculated based on IMA scores and averaged over each CB’s criteria.

Steenbergen et al. [37] name four properties that a measure of discourse quality must exhibit: theoretical grounding, reliance on observable phenomena, generalizability, and reliability. To evaluate reliability, they use the ratio of agreement among coders, Cohen’s $\kappa$, Spearman’s $\rho$, and the standardized $\alpha$.

Inter-annotator agreement serves as a lens into the complexity of the task and the quality of the annotations. Higher values indicate that the definitions are clear for different coders, whereas lower values suggest both annotator performance issues and poorly defined categories.

We introduce Inter-Model Agreement (IMA) to quantify consensus across different models. Because these LLMs are developed by diverse organizations on distinct training data [36, 1, 18, 2] (Gemma Team 2026), we hypothesize they exhibit divergent inherent biases; prior research on cultural biases validates this premise [25]. This diversity contrasts with human annotators, who often share demographic backgrounds or subjective opinions. Consequently, high IMA serves as a compelling measure of robustness. If multiple models agree on the scoring criteria, it indicates that the CBs are reliable for LLM annotation, rather than reflecting the artifacts of a singular model. Thus, a higher agreement score indicates a clearer, more objective, and robust framework for automated protocol annotation.

Following standard practices for evaluating the DQI, we report the percent agreement as a baseline measure of consensus. To rigorously account for chance agreement, we include Krippendorff’s $\alpha$ [19]; values range from -1 to 1; negative values indicate systematic disagreement, while values above zero indicate a degree of agreement. Furthermore, we use Spearman’s $\rho$ to capture annotator agreement on the relative ranking of discussion quality, and the Standardized $\alpha$ to evaluate the overall internal consistency of the aggregated index.

CB Validity. To validate the CBs’ overall performance, we compare their final scores against proxies for discourse quality. For each discussion transcript, we calculate this proxy score by prompting an independent LLM to holistically assess sequences of turns, using a short-context sliding window. These LLM assessments are validated by human annotators to ensure they adequately capture conversational dynamics. This procedure serves as an extrinsic measure of how accurately each schema captures overall discourse quality. We define this automated proxy for every transcript, represented as a sequence of turns $D = (t_1, \ldots, t_k)$. The task scores the conversation dynamics of an $n$-turn exchange on a binary scale, where 0 reflects negative deliberation and 1 indicates positive deliberation (see Appendix § 11 for the detailed instructions). We use a sliding window of size $n = 3$ and score each window independently. The final proxy score for a conversation is the average of all its triplet scores ($p_i$):

Data

The data for our empirical assessment consists of over 15,000 hours of Hebrew recordings and corresponding transcripts. These transcripts are derived from more than 30,000 human-generated committee meeting protocols of the Knesset.⁵ This corpus spans from November 2018 to January 2026, encompassing six consecutive Knesset terms. The data was curated from government-released material and made available as a Hugging Face dataset by Ivrit.ai (Marmor et al., [24]).⁶ For this study, we restrict the corpus to legislative sessions by retaining only protocols in which every agenda item listed in the session metadata is marked as a bill.⁷

We randomly sample 100 protocols from this subset, stratified across Knesset terms and committees. Table 1 describes the selection’s statistics. To demonstrate the corpus’ diversity we highlight here items with notable variance:

Most Intensive Meeting: The longest session regarded the implementation of the Economic Policy for 2023 and 2024. It lasted nearly 3 hours, with 27 participants speaking, generating 1,736 turns and 27,113 words (session_id=2204258).

caption: Structural statistics of the 100-transcript meeting corpus. Length statistics are reported in words.

:::

Most Chair-Dominated Session: During the Economic Affairs Committee’s discussion on the bill to Increase Competition in the Broadcasting Market, the chair alone spoke 60.5% of all words (session_id=2220120).

Most Evenly Distributed: The most balanced discussion occurred during the National-Civic Service Bill, yielding a Gini coefficient⁸ of 0.20 among five speakers and eight participants (session_id=2209191). Furthermore, the most broadly inclusive large discussion focused on the bill for reducing the use of disposable shopping bags, which featured 24 speakers, a Gini score of 0.40, and no single speaker accounting for more than 15.8% of the words (session_id=2191338).

This analysis confirms our selection’s potential to examine discourse quality across diverse axes of length, dominance, and inclusivity.

Experimental Setup

Goal. We aim to empirically assess the proposed LLM-based CB generation pipeline using our proposed metrics. We evaluate each of the four CB generation strategies, comparing the top-down, bottom-up, and hybrid scenarios.

Data. Drawing on the 100 transcript selection (§5), we segment the discussion into smaller units of analysis, defined as text excerpts of up to 1,500 words and ensuring that individual turns are not split. This text length provides sufficient context for analyzing interaction dynamics and responsiveness, as the median turn length is 10 words. This results in a total of 700 segments for the full corpus, with the majority exceeding 1,466 words (Table 1).

Models. Our selection includes a proprietary LLM (Claude Sonnet 5.5⁹), and open-weight models: a Hebrew-centric model (DictaLM3-24B-Thinking (Shmidman et al., [36])), and multilingual models (Gemma4-31B (Gemma Team, [14]), Qwen3.6-27B (Qwen Team, [30]), GPT-OSS-20B (OpenAI, [28]), and Mistral-Small-3.2-24B¹⁰).

For the open-weight models, we use default hyperparameters with temperature 0.6. We run each model using two seeds (11, 22) and report the mean. For Claude, due to cost considerations, we perform a single run with the temperature 1.0.

For estimating the proxy C-DQI, we utilize GPT-6-luna¹⁰ with its default hyperparameters.

CB Application. We prompt each model with every category-segment pair. Apart from the Vanilla C-DQI CB, which utilizes category-specific scales, all categories are scored on a four-point scale. To create a final, representative vector of deliberation criteria for an entire discussion, the segment-level scores for each category are averaged. The final C-DQI score for a given discussion is determined according to the definitions in § 4.1.

Proxy Validation. With a team of domain experts, we manually annotated 162 exchanges to serve as a human benchmark, attaining a Krippendorff’s $\alpha$ of 0.79 for inter-annotator agreement in a complete overlap setting where each exchange was reviewed by exactly two annotators. Exchanges on which the annotators disagreed were discarded.

We then deployed GPT-6-luna¹¹ — an independent LLM not utilized in the CBs’ reliability evaluation — to compute proxy scores across the dataset. Against the human benchmark, this LLM achieved an accuracy of 82% and a macro F1 score of 77% on a held-out test set of 50 samples (see Appendix § 19). Deploying an LLM as a judge not only allows us to scale the validation across the entire dataset (a total of 36,341 3-turn exchanges), avoiding costly time and resource constraints, but it also provides a consistent scoring mechanism that mitigates the noise and subjectivity inherent in human annotations. Consequently, we rely on these automated proxy values to validate our experiments. The distribution of the proxy scores across the corpus is provided in Appendix § 20.

Evaluation Metrics. To assess the generated CBs, we employ the metrics defined in § 4.3. For CB reliability, we compute IMA scores for each category.¹² For validity, we calculate Spearman correlation coefficients between the mean C-DQI scores and the proxy scores.

Results and Analysis

CB Reliability

Table 2 presents the reliability evaluation, structured to mirror the evaluation metrics of the original DQI by Steenbergen et al. [37]. Overall, the IMA results of the CB annotation experiments show some level of agreement for every CB ($\alpha> 0$, $\rho> 0.5$). The data-driven CBs are consistently the most reliable across workflows, with all three inductive CBs reaching Krippendorff $\alpha$ scores in the range of 0.57–0.58, compared with $\alpha= 0.46$ and $\alpha= 0.49$ for the Hybrid and Top-Down CBs, respectively. In contrast, the baseline Vanilla DQI CB yields the lowest reliability on both rank-based measures ($\alpha= 0.40$, $\rho= 0.54$, and Standardized $\alpha= 0.55$), hitting the threshold for moderate agreement ($\alpha= 0.40$) (Olsson et al., [27]).

CB	Krippendorff’s $\alpha$	Percent Agreement	Spearman’s $\rho$	Standardized $\alpha$
Bottom-Up-Hebrew	0.58	0.20	0.75	0.78
Bottom-Up-English	0.57	0.20	0.72	0.73
Bottom-Up-Corrected	0.57	0.17	0.74	0.77
Top-Down	0.49	0.13	0.70	0.72
Hybrid	0.46	0.10	0.70	0.73
Vanilla DQI	0.40	0.20	0.54	0.55

Furthermore, when evaluating category-level IMA, the categories associated with the highest values primarily relate to respect and decorum, whereas those at the lower end of the reliability spectrum correspond to justification, counter-argument engagement, and emotional regulation. For example, Respect and Civility in the Top-Down CB ($\alpha= 0.79$) and Procedural Decorum in the Bottom-Up-English CB ($\alpha= 0.83$) exhibit the strongest consensus. Conversely, Justification Level yields poor agreement in both the Vanilla DQI ($\alpha= 0.08$) and Top-Down ($\alpha= 0.20$) CBs, and Emotional Regulation is the least reliable category of both the Bottom-Up-English ($\alpha= 0.15$) and Bottom-Up-Corrected ($\alpha= 0.19$) CBs (see Table 4 in Appendix § 12).

Codebook	Most reliable feature	$\alpha$	Least reliable feature	$\alpha$
Vanilla DQI	Respect toward Groups	0.60	Justification Level	0.08
Top-Down	Respect and Civility	0.79	Justification Level	0.20
Bottom-Up-English	Procedural Decorum	0.83	Emotional Regulation	0.15
Bottom-Up-Corrected	Procedural Decorum	0.80	Emotional Regulation	0.19
Bottom-Up-Hebrew	Decorum and Tone	0.80	Chair Management Efficiency	0.36
Hybrid	Civility and Conflict Mgmt.	0.69	Moderator Efficacy	0.22

CB Validity

In Table 3, we present the Spearman correlations between the discussion-level CB scores and the C-DQI proxy scores across the 100 transcripts. All correlations exceed 0.55 and are statistically significant ($p < 10^{-9}$). Notably, the top-down CB aligns closest to the proxy ($\rho= 0.66$). This suggests that although the proxy annotators received only minimal binary guidelines, the concept of deliberative quality they apply is itself rooted in the literature, which may favor theory-led CB in the resulting rankings.

CB	GPT-6-luna ($\rho$)
Top-Down	0.66
Bottom-Up-English	0.62
Bottom-Up-Corrected	0.61
Hybrid	0.61
Vanilla DQI	0.59
Bottom-Up-Hebrew	0.59

CB Analysis

We examine the criteria of each CB to identify their similarities and divergences.

The only category shared across all CBs is Respect, which appears as three separate dimensions in the original DQI alone. According to the DQI, respect divides into respect toward groups in society, respect toward others, and respect toward contradicting arguments. A dominant category in the theory-led CBs is Constructive Politics, defined as the ability to bridge disagreements and reach practical solutions, consensus, or compromise. The most similar pair of CBs is the Hybrid and Top-Down, sharing five out of six metrics.

Notably, specific categories exist exclusively in the deductive workflows, such as participation, level and content of justification, and reciprocity. Conversely, categories appearing solely in the inductive workflows include procedural decorum, emotional regulation, and moderator efficiency.¹⁴

Discussion

As shown, the data-driven CBs are more stable across LLM annotation than the theory-based CBs. Specific criteria that appear in the inductive CBs such as emotional regulation and moderator efficiency are likely a product of local cultural nuances in the input data of the CB generation phase.

To observe how participation dynamics manifest, we re-examine the transcripts from §5 using the inductive CBs.¹⁵ The most intensive meeting is scored 2.5 (Bottom-Up-English) and 2.8 (Bottom-Up-Corrected), while the most chair-centered meeting is scored 2.6 and 2.8, respectively. In contrast, the most evenly distributed meetings attain the same scores, 3.0 and 3.2 for the respective CBs. This suggests these CBs favor balanced participation over heavy dominance, inviting further investigation to systematically explore the interplay between structural meeting dynamics and automated discourse quality assessments.

Future directions include applying the C-DQI to analyze fluctuations in deliberative quality across the parliamentary cycle, investigating whether sessions held immediately before recesses exhibit different scores than mid-session meetings and how it systematically varies across the legislative term.

Additionally, the C-DQI enables the comparison of discourse quality analysis across different deliberative environments, such as other parliaments, alternative Israeli legislative bodies, and varying types of Knesset committee protocols.

The results validate the promise of LLM-driven bottom-up strategies for defining and applying discourse quality schemas. Serving as a proof of concept, our study motivates scaling the generation and annotation of CBs. By maneuvering LLM biases to serve as a feature that enhances robustness, rather than viewing them as unwanted artifacts, future research can uncover new analytical schemes that advance concrete legislative goals.

Conclusion

In this study, we proposed a scalable, LLM-driven methodology for generating and validating discourse quality codebooks. Specifically, we examined four approaches for schema definition, deploying an ensemble of diverse LLMs as zero-shot annotators to evaluate the resulting categories. Our results underscore the value of a bottom-up strategy for constructing reliable discourse evaluation frameworks, with validity comparable to theory-based alternatives. Taken together, this work highlights promising avenues for applying our proposed workflow rather than predefined schemas to diverse practical, political, and social scientific investigations.

Limitations

We acknowledge the limitations in our study. Although we rely on LLM-inherent biases in our methodology design, we have not examined how these artifacts manifest in annotating the CBs in practice. Additionally, regarding our choice to use segments of 1,500-words: this aims to capture reciprocity across multiple turns, yet larger segments risk reducing interpretability and introducing noise into holistic scoring. This highlights an inherent tension: higher-level abstractions capture broader dynamics but are harder to evaluate against specific criteria. Despite this, holistic evaluation remains valuable for assessing overarching processes and solutions. In future work, we will explore this in more depth.

Regarding the Top-Down CB, the instruction ‘to develop a discourse quality index’ may have biased the model toward the DQI and Habermasian discourse ethics. Further experiments could explore alternative phrasing for the prompt.

Furthermore, the CBs are evaluated separately as a proof of concept; we did not examine their combinations and other balances besides the hybrid. Finally, for validity, we rely on statistical correlations. Although a positive correlation indicates the direction of agreement, it doesn’t directly translate to accuracy. We acknowledge this as a limitation. Future research may address this by validating the distributional similarity between CB scores and the proxy C-DQI, and can benefit from a manual gold standard as well.

Acknowledgments

We wish to thank Nadav Oved and Noam Golan from our annotators team for their dedicated research assistance. This research was supported by the Bar-Ilan University Data Science and Artificial Intelligence Institute, and by the Rector’s grant which funded the 2026 interdisciplinary research group on computational analysis of the legislative process. Furthermore, this research was funded by the Israel Science Foundation (grant No. 670/23 and grant No. 2352/26), a KAMIN grant from the Israel Innovation Authority (IIA), and a VATAT grant from the Israeli Planning and Budgeting Committee (PBC), for which we are grateful. We finally wish to thank Google for their generous grant award supporting our lab efforts on AI for Science.

Code-Book Generation Prompts

Bottom-Up CB. - English

Attached are 10 raw transcripts from committee meetings in the Israeli parliament, the Knesset. Using a bottom-up approach, analyze the discourse of these specific transcripts to define metrics that evaluate the quality of discourse in general committee meetings.

Provide:

1. A list of the metrics.

2. The definition of each metric.

Bottom-Up CB. - Hebrew

לפניך קבצי טקסט שמכילים פרוטוקולים מועדות הכנסת. הגדר מטריקות להערכת איכות השיח של כל ועדה. הציג את המטריקות ואת ההגדרה של כל אחת מהם.

Top-Down CB.

Based on the frameworks in these papers, define metrics to evaluate the quality of discourse in committee transcripts from the Israeli parliament, the Knesset.

Provide:

1. A list of the metrics.

2. The definition of each metric.

Hybrid CB.

Attached are 20 files: Half are academic literature on democratic deliberation and discourse quality (Theoretical Foundation). The other half are raw committee meeting transcripts from the Israeli parliament, the Knesset (Empirical Context). Using a combined theory and bottom-up approach, synthesize the academic frameworks and analyze the discourse of these specific transcripts to define metrics that evaluate the quality of discourse in general committee meetings.

Provide:

1. A list of the metrics.

2. The definition of each metric.

Proxy Generation Prompts

For the proxy generation prompts see Figure 3 and Figure 4 for the system and user prompts.

Extreme Category IMA

Table 4 describes the categories in each CB that exhibit the highest and lowest IMA scores.

Constructing the Bottom-Up-Corrected Codebook

We constructed the corrected inductive codebook by merging the English and Hebrew codebooks, retaining every distinct metric from each. Where the two versions overlapped, we consolidated them into a single metric; Hebrew’s Decorum and Tone category was split between Procedural Decorum and Respectful Engagement. We removed most illustrative examples (“such as. . . ”) to avoid biasing human or model annotators toward specific cases. We also adopted the bullet structure of the Hebrew codebook and standardized it so that every metric has two labeled poles, “High” and “Low”. Where a source codebook described only one pole, we defined the missing one to be the parallel opposite of the existing one. Finally, for the Procedural Decorum metric, we replaced “unanimous agreement” from the English codebook with “respectful disagreement”, since agreement is not a condition for good decorum.

CodeBooks

Vanilla DQI

1. Demand

A proposal about what should or should not be decided or done. A demand may support changing, adopting, rejecting, or preserving a policy or course of action.

1. Participation

Definition: Evaluates whether a participant can complete a relevant contribution without being prevented from speaking by disruption or formal intervention.

Coding Categories / Scale:

Code 0 (Participation Interrupted): The speaker explicitly indicates that an interruption is disturbing or preventing the contribution, or the contribution is terminated by a formal decision. Do not assign this code merely because the allocated speaking time has expired.

Code 1 (Normal Participation Possible): The participant is able to make the relevant contribution under ordinary speaking conditions.

2. Level of Justification for Demands

Definition: Measures the completeness of the reasons supplied for a demand. The substantive truth or persuasiveness of the argument is not judged. A complete justification contains (1) a conclusion about what should or should not be done, (2) a supporting reason, and (3) an explicit or unmistakably implied inferential link between the reason and conclusion.

Coding Categories / Scale:

Code 0 (No Justification): A demand or evaluation is stated, but no supporting reason is provided.

Code 1 (Inferior Justification): A supporting consideration is mentioned, but the link between the reason and the demand is incomplete or unclear. A conclusion supported only by illustration also receives Code 1.

Code 2 (Qualified Justification): At least one complete justification connects a demand to a reason through an explicit or clearly implied inferential link.

Code 3 (Sophisticated Justification): At least two distinct complete justifications are supplied, whether for the same demand or for different demands within the speech act.

Decision Rule: Assign the highest level whose requirements are fully met. An implicit linkage counts only when its intended meaning would be beyond reasonable doubt to participants in the debate.

Content of Justification for Demands

Definition: Identifies whose interests are explicitly invoked when a speaker justifies a demand: a constituency or particular group, society collectively, or disadvantaged groups under a fairness-based conception of the common good.

Coding Categories:

Code 0 (Constituency or Group Interest): At least one explicit appeal is made to the interest of a particular constituency, organization, sector, party, region, or social group.

Code 1 (Neutral Statement): There is no explicit reference to a constituency or group interest and no explicit appeal to either form of the common good.

Code 2 (Common-Good Orientation): The speaker explicitly justifies a demand using at least one of the following forms of common-good reasoning:

Utilitarian or Collective Common Good: An appeal to the best outcome for the greatest number, society as a whole, or a larger collective such as Europe or humanity.

Difference-Principle Common Good: An appeal to protecting or helping the least advantaged, poorest, most discriminated against, or otherwise vulnerable members of society, including children and future generations, or an appeal to fairness and justice in this sense.

Decision Rule: Assign the highest level whose requirements are fully met.

Respect toward Groups to Be Helped (Empathy)

Definition: Evaluates how the speaker characterizes the people or groups who would be affected or helped by the demand.

Coding Categories / Scale:

Code 0 (No Respect): At least one negative or degrading statement is made about a relevant group, and no positive statement is made. A single negative statement is sufficient.

Code 1 (Neutral): Neither negative nor explicitly positive statements are made about the relevant groups.

Code 2 (Explicit Respect): At least one explicitly positive, valuing, empathetic, or dignity-affirming statement is made about a relevant group. This code still applies when negative statements also occur.

Priority Rule: Explicit respect takes priority over negativity: mixed positive and negative statements receive Code 2.

Respect toward the Demands of Others

Definition: Evaluates whether a speaker degrades, treats neutrally, or explicitly values demands advanced by other participants.

Coding Categories / Scale:

Code 0 (No Respect): At least one negative or degrading statement is made toward another demand, or toward the person or group advancing it, and no positive statement is made.

Code 1 (Neutral): Other demands receive neither negative nor explicitly positive treatment.

Code 2 (Explicit Respect): At least one positive statement values any aspect of another participant’s demand. This code still applies when negative statements also occur.

N/A (Not Applicable): Only one demand is on the agenda and the speaker supports it; consequently, there is no distinct competing demand to evaluate.

Priority Rule: Explicit respect takes priority over negativity: mixed positive and negative statements receive Code 2. Criticism of another demand may additionally be coded under Level and Content of Justification when it functions as part of the speaker’s own argument.

Respect toward Counterarguments

Definition: Evaluates how a speaker responds to arguments that contradict the speaker’s conclusion concerning a demand. The category concerns engagement with opposing reasons, not merely interpersonal politeness.

Coding Categories / Scale:

Code 0 (Counterarguments Ignored): One or more counterarguments are already available, but the speaker does not address any of them.

Code 1 (Included but Explicitly Degraded): Counterarguments are acknowledged, but at least one is degraded, mocked, or attacked, and none is explicitly valued. Attacks on the people or groups advancing the counterarguments also count.

Code 2 (Included Neutrally): Counterarguments are acknowledged or addressed without either negative degradation or explicit positive valuation.

Code 3 (Included and Explicitly Valued): At least one aspect of at least one counterargument is explicitly recognized as valid, correct, useful, or worthy of consideration. This code still applies when other negative statements occur.

N/A (Not Applicable): No counterargument has yet been introduced and the speaker does not anticipate one.

Totality Rule: When several counterarguments exist, assign one code to the speaker’s treatment of the set as a whole. Explicit valuation takes priority over negative treatment; without valuation, any degradation leads to Code 1. An anticipated counterargument should receive an actual code rather than N/A.

Constructive Politics

Definition: Measures whether the speaker remains fixed in a position or advances a mediating proposal, distinguishing proposals that address the present agenda from alternatives that belong to another policy agenda.

Coding Categories / Scale:

Code 0 (Positional Politics): The speaker maintains a position without offering a mediating or compromise proposal.

Code 1 (Alternative Proposal): The speaker advances a proposal that could mediate the broader conflict but does not fit the current agenda or resolve the decision presently under debate. It would need to be considered under another agenda.

Code 2 (Mediating Proposal): The speaker offers a concrete compromise or accommodation within the current agenda that attempts to bridge competing concerns or modify the proposal under consideration.

Coding Note: A proposal is not mediating merely because it differs from the current proposal. It must attempt to accommodate competing concerns; the distinction between Codes 1 and 2 depends on whether it can be acted upon within the present agenda.

Bottom-Up CB. - English

Metric 1: Procedural Decorum

This metric evaluates the extent to which participants follow parliamentary rules, wait for their turn to speak, and avoid shouting or talking over one another. In some committees, the chairperson must repeatedly threaten to remove members to maintain order. Conversely, other meetings proceed with unanimous agreement, polite praises, and uninterrupted presentations.

Metric 2: Topic Adherence

This metric measures the ratio of substantive discussion regarding the agenda versus unrelated political grandstanding or personal attacks. High topic adherence is observed when members strictly debate specific legal definitions, such as defining security offenses versus treason. Low adherence occurs when speaking time meant for legislative reservations is entirely used to attack political opponents on unrelated national issues.

Metric 3: Constructive Problem-Solving

This metric assesses the ability of committee members and government officials to bridge disagreements and reach practical legislative compromises. This metric is positive when members negotiate temporary orders to test a law’s impact before making it permanent. It is also evident when legal advisors and ministries successfully negotiate timelines for administrative tasks, such as extending a deadline from three months to seven months to accommodate workload constraints.

Metric 4: Respectful Engagement This metric analyzes the tone and language used when interacting with civil servants, academic experts, and members of the public. Discourse quality drops when elected officials use dismissive language, such as calling opposing arguments “bullshit”. It also drops when there is friction between politicians and civil servants, such as when an official allegedly calls a member a “populist”. High quality is shown when chairpersons actively listen to and validate the emotional testimonies of local mayors.

Metric 5: Emotional Regulation This metric examines how the committee handles highly charged emotional topics or national tragedies while conducting legislative business. Poor emotional regulation is visible when procedural duties, like budget discussions, clash with days of national mourning, leading to accusations of callousness and “disgusting” behavior between members. High emotional tension is also present when bereaved parents confront lawmakers directly during security-related debates.

Bottom-Up CB. - Hebrew

1. ענייניות ומיקוד בדיון (Relevance and Focus) הגדרה: מדד הבוחן עד כמה הדוברים (חברי הכנסת והמוזמנים) מתמקדים בנושא המקצועי לשמו התכנסה הוועדה, לעומת גלישה לפיליבסטרים, העלאת נושאים שאינם קשורים, או נאומים פוליטיים.

• איכות שיח גבוהה: התמקדות בסעיפי החוק, ניסוחים משפטיים ופתרון חסמים בירוקרטיים.

• איכות שיח נמוכה: שימוש בזמן הוועדה לנאומים פוליטיים ארוכים שאינם קשורים לסעיף הנדון כלל, כפי שניתן לראות בהגשת עשרות הסתייגויות ונימוקן על ידי תקיפת השר לביטחון לאומי בנושאים שונים לחלוטין (כגון תאונות דרכים או גזענות) במהלך דיון על חוק לשכת עורכי הדין.

2. תרבות הדיון וסגנון ההתבטאות (Decorum and Tone) הגדרה: מדד המעריך את רמת הכבוד ההדדי, ההקשבה ואופן ההתבטאות בין המשתתפים. המדד נבחן על ידי ספירת קריאות ביניים, התפרצויות לדברי אחרים, הרמת קול, ושימוש בשפה בוטה או תקיפות אישיות.

• איכות שיח גבוהה: הקשבה מלאה לדוברים, הבעת הערכה הדדית גם בין קואליציה לאופוזיציה, ודיון שליו ומקצועי.

• איכות שיח נמוכה: התפרצויות תכופות, קושי להשלים משפט, ושימוש בביטויים פוגעניים והאשמות אישיות (כגון חילופי האשמות "תומכי טרור" ו-"נס תומכי אל מול "מפעל טרור" ו-"פאשיסטים").

3. יעילות וניהול הישיבה (Efficiency of Chair Management) הגדרה: מדד הבוחן את יכולתו של יושב-ראש הוועדה לשלוט בדיון, לתת אותו להפרעות מעשיות, לקטוע הפרעות ולמנוע מהדיון לצאת מכלל שליטה, תוך שמירה על הוגנות כלפי המשתתפים.

• ניהול החלטי ואקטיבי: יכולת של היושב-ראש לקטוע ויכוחים בירוקרטיים ארוכים, להתייחס את גורמי המקצוע להגיע לפשרות פרקטיות, ולקבל החלטות על המקום תוך גיבוי ברור שאינם מבצעים את עבודתם כראוי, כמו הדרישה לקנס רשויות מקומיות ששלחו קנסות שווא לאמבולנסים.

• קושי בניהול הדיון: אובדן שליטה על סדר הדיון, ניסיונות חוזרים ונשנים וללא הצלחה (להשקיט קריאות ביניים, ואיומים תכופים בהוצאת משתתפים ללא הצלחה.

4. מקצועיות והסתמכות על נתונים (Professionalism and Data Reliance) הגדרה: מדד הבודק את מידת ההישענות של משתתפי הוועדה על עבודות, נתונים, מחקרים (כגון מרכז המידע והמחקר של הכנסת, והקשבה לייעוץ המשפטי ולדרגים המקצועיים במשרדי הממשלה).

• איכות שיח גבוהה: ניתוח מדויק של החוק סעיף אחר סעיף יחד עם הייעוץ המשפטי, תוך הבנת החדשנות המשמעותית של כל מילה בחוק (כגון ההבדל בין "חוק חול" ל-"בגבולות לא חוק").

• משרד מקצועית מבוססת נתונים: קריאת לוחות זמנים ריאליים (כגון דיווח על מספר החודשים הנדרש לוועדת שחרורים לעבור על 120 תיקים פליליים (מול גורמי המקצוע בהנהלת בתי המשפט ומשרד המשפטים.

5. אקלים רגשי וגילוי אמפתיה (Emotional Climate and Empathy) הגדרה: מדד הבוחן את יכולת הוועדה להכיל ולהגיב למצבים רגשיים מורכבים, נוכחותן של משפחות שכולות ומשפחות חטופים, והתאמה של קיום הדיון למציאות החיצונית (אירועים ביטחוניים קשים).

• חוסר הלימה רגשית/מתיחות קשה: קיום דיון "רגיל" שמעורר זעם בקרב משתתפים על רקע אירועים ביטחוניים קשים בזמן אמת, מה שמוביל לפיצוץ הדיון ולהטחת האשמות על אטימות וניתוק ("ערלות לב").

• מתן מקום לכאב: עצירת שטף החקיקה כדי לאפשר למשפחות חטופים ומשפחות שכולות להשמיע את קולן ואת זעקתן, והבעת הזדהות ואמפתיה מצד חברי הכנסת והיושב-ראש למצוקתן.

Bottom-up CB - Corrected

Metric 1: Procedural Decorum Evaluates the extent to which participants follow parliamentary rules, wait for their turn to speak, and avoid shouting or talking over one another.

High Decorum: Observed in meetings that proceed with polite exchanges, uninterrupted presentations, and respectful disagreement. Low Decorum: Associated with committees where the chairperson must repeatedly threaten to remove members to maintain order.

Metric 2: Topic Adherence Measures the ratio of substantive discussion regarding the agenda versus unrelated political grandstanding or personal attacks.

High Adherence: Observed when the majority of participants strictly stick to the discussion agenda, debating specific outcomes and legal definitions. Low Adherence: Observed when discussions frequently go off-topic, or when speaking time meant for legislative reservations is entirely used to attack political opponents on unrelated national issues.

Metric 3: Constructive Problem-Solving Assesses the ability of committee members and government officials to bridge disagreements and reach practical legislative compromises.

High Constructiveness: Indicated by members negotiating temporary orders to test a law’s impact before making it permanent, or when legal advisors and ministries successfully negotiate timelines for administrative tasks. Low Constructiveness: Indicated by members adopting zero-sum ideological positions

that lead to legislative deadlocks, or government ministries engaging in jurisdictional turf wars and blame-shifting instead of finding actionable paths forward.

Metric 4: Respectful Engagement Analyzes the tone and language used when interacting with civil servants, academic experts, and members of the public.

High Respect: Demonstrated when participants utilize active listening, validate each other’s opinions, and maintain a professional tone. Low Respect: Characterized by friction between participants and the use of dismissive or vulgar language (e.g., calling opposing arguments “bullshit”).

Metric 5: Emotional Regulation Examines how the committee handles highly charged emotional topics or national tragedies while conducting legislative business.

High Regulation: Demonstrated when lawmakers maintain composure, empathy, and respect during emotionally charged testimonies. The committee successfully balances sensitivity to national tragedies with its legislative duties without allowing the discussion to devolve into chaos. Low Regulation: Demonstrated when routine business clashes with sensitive events, resulting in insults, accusations of callousness, or uncontrolled tension when lawmakers are confronted by grieving citizens.

Metric 6: Meeting Management Effectiveness Examines the committee chairperson’s ability to control the discussion, steer it toward practical decisions, cut off disruptions, and prevent the debate from spiraling out of control, all while maintaining fairness.

High Effectiveness: The chairperson successfully cuts short lengthy bureaucratic arguments, guides professional entities to reach practical compromises, makes on-the-spot decisions, and holds parties accountable for failing to perform their duties. Low Effectiveness: Characterized by a loss of control over the speaking order, repeated attempts to quiet heckling, and frequent but unsuccessful threats to remove participants from the room.

Metric 7: Professionalism and Reliance on Data

Assesses the extent to which committee participants rely on facts, data, and research, and their attentiveness to professional echelons and legal counsel.

High Professionalism: Characterized by a meticulous, section-by-section analysis of the law alongside legal counsel, demonstrating an understanding of the legal implications of the legislation. Compromises set realistic timetables in coordination with professional authorities.

Low Professionalism: Characterized by participants ignoring research and data, disregarding professional or legal advice, and arguing over legislation without understanding its technical or practical implications.

Top-Down CB.

Level of Justification (Argumentative Rationality)

Theoretical Origin: Discourse Quality Index [37]; Stromer-Galley [39]; Friess & Eilders [13].

Definition: Evaluates the logical structure and completeness of reasons provided by Members of the Knesset (MKs) or invited witnesses when advancing a policy proposal, amendment, or claim.

Coding Categories / Scale:

Level 0 (No Justification): A demand/claim is stated without any supporting reason.

Level 1 (Inferior Justification): A reason is stated, but the logical link between the demand and the reason is incomplete or relies strictly on illustration.

Level 2 (Qualified Justification): At least one complete, explicit, or clearly implied inferential link connects the demand to a supporting reason.

Level 3 (Sophisticated Justification): At least two distinct, complete inferential links are provided (either multiple complete reasons for one demand or reasons spanning multiple demands).

Content of Justification (Common Good Orientation)

Theoretical Origin: Steenbergen et al. [37]; Habermas (1996); Cohen (1989).

Definition: Assesses whether a speaker justifies claims solely through narrow sectoral/party/constituency interests or frames arguments in terms of broader societal benefit.

Coding Categories / Scale:

Group/Constituency Interest: Explicit appeals to narrow political, regional, or constituency benefits.

Neutral Statement: No explicit reference to either narrow group interests or the broader common good.

Common Good (Utilitarian): Explicit appeal to the best outcome for Israeli society as a whole or the greatest benefit for the majority.

Common Good (Difference Principle): Explicit appeal to protecting or helping the least advantaged, vulnerable, or marginalized groups in society (Rawlsian difference principle).

Sourcing and Evidence Base

Theoretical Origin: Stromer-Galley [39]; Friess & Eilders [13].

Definition: Measures the extent to which participants support their positions using verifiable external references, expert testimonies, or official materials rather than unsupported assertions.

Coding Categories:

Official/Committee Documents: Reference to Knesset Research and Information Center (MMM) reports, ministry briefings, or draft bill texts.

External/Expert Sources: Reference to academic research, statistical data, court rulings, or international benchmarks.

Mass Media: Reference to news reports, journalism, or public media broadcasts.

Personal Experience / Narrative: Reference to first-hand accounts, constituent stories, or personal anecdotes.

No Source: Claim made without grounding in external or experiential sources.

Interactivity and Reciprocity

Theoretical Origin: Stromer-Galley [39]; Friess & Eilders [13].

Definition: Evaluates whether committee dialogue constitutes a genuine, reciprocal exchange of ideas where speakers listen and respond to one another, or consists of disconnected monologues (“soap-boxing”).

Coding Indicators:

Formal Interactivity: Whether a speech turn explicitly addresses a prior speaker or turn.

Substantive Interactivity: Whether a speaker directly addresses, rebuts, clarifies, or builds upon the specific arguments made by a prior participant.

Information-Seeking / Questioning: Presence of genuine questions aimed at eliciting clarification or facts from witnesses, government officials, or fellow MKs (distinguished from rhetorical questions).

Clarification (Self/Other): Speech acts that explicitly clarify a speaker’s own prior position or summarize another participant’s argument to verify understanding.

Respect and Civility toward Actors and Demands

Theoretical Origin: Steenbergen et al. [37]; Young (1996); Friess & Eilders [13].

Definition: Measures the interpersonal tone and level of mutual recognition accorded to other participants, social groups, and policy proposals during committee proceedings.

Coding Categories / Scale:

Respect toward Groups: Evaluates whether social, ethnic, or political groups affected by the bill are degraded (0), treated neutrally (1), or explicitly valued/respected (2).

Respect toward Demands: Evaluates whether policy proposals/amendments raised by fellow participants are degraded/insulted (0), treated neutrally (1), explicitly valued (2), or agreed with (3).

Civility / Disruption: Tracks presence of personal insults, ad hominem attacks, or unprocedural interruptions.

Engagement with Counterarguments

Theoretical Origin: Steenbergen et al. [37]; Bächtiger et al. [7].

Definition: Assesses how speakers handle opposing viewpoints or counterarguments raised by coalition/opposition members or invited experts.

Coding Categories / Scale:

Level 0 (Ignored): Counterarguments exist on the table, but the speaker completely ignores them.

Level 1 (Included but Degraded): Counterarguments are acknowledged but explicitly degraded, mocked, or dismissed without substantive engagement.

Level 2 (Included Neutrally): Counterarguments are acknowledged and restated neutrally without personal or thematic degradation.

Level 3 (Included and Valued): Counterarguments are acknowledged and explicitly praised or treated as valid considerations, even if ultimately rejected or integrated.

Constructive Politics and Compromise Orientation

Theoretical Origin: Steenbergen et al. [37]; Bächtiger et al. [7]; Friess & Eilders [13].

Definition: Measures whether committee participants remain locked in rigid, zero-sum partisan stances or actively attempt to build consensus and formulate acceptable compromises.

Coding Categories / Scale:

Positional Politics (Level 0): Speakers rigidly maintain initial party lines without attempt at compromise, accommodation, or reconciliation.

Alternative Proposal (Level 1): A mediating proposal is offered that belongs to a different agenda or jurisdiction, failing to resolve the immediate conflict on the current draft text.

Mediating Proposal (Level 2): A speaker advances a concrete, integrative compromise or amendment that directly fits the current agenda and seeks to bridge coalition-opposition or stakeholder divides.

Topic Coherence and Agenda Focus

Theoretical Origin: Stromer-Galley [39]; Friess & Eilders [13].

Definition: Evaluates whether committee discussion stays focused on the legislative agenda (structuring topic) or drifts into uncoordinated, off-topic partisan disputes (interactional topic drift).

Coding Indicators:

On-Topic (Structuring): Statements directly addressing the bill, policy options, or specific committee agenda items.

On-Topic (Related Problem): Statements addressing the broader policy issue (e.g., educational quality, defense budget) without directly addressing the specific legislative choices at hand. Off-Topic / Drift: Statements digressing into unrelated political scandals, electoral grandstanding, or personal disputes unlinked to the agenda.

Participatory Equality and Vocal Distribution

Theoretical Origin: Stromer-Galley [39]; Knight & Johnson (1997); Friess & Eilders [13].

Definition: Measures whether speaking opportunities and floor time are equitably distributed among participants or monopolized by committee chairs, coalition members, or dominant speakers.

Coding Indicators:

Turn & Word-Count Distribution: Quantitative balance of speech turns and word volume across coalition vs. opposition MKs, and committee members vs. invited external experts/citizens.

Procedural Equal Access: Absence of arbitrary silencing, unequal time enforcement by the chair, or procedural coercion that prevents backbenchers or opposition members from presenting amendments.

Inclusion of Alternative Communicative Modes

Theoretical Origin: Young (1996); Bächtiger et al. [7] (Type II Deliberation).

Definition: Captures non-canonical speech forms—such as narrative/storytelling, rhetoric, and formal greeting—that complement formal argumentation by bridging cultural, ideological, or experiential divides in parliamentary committees.

Coding Categories:

Narrative / Testimony: Use of situated personal stories, constituent accounts, or lived experiences to demonstrate legislative impact or illuminate marginalized perspectives.

Rhetoric / Expressive Language: Use of emotional resonance, figurative language, or metaphor to contextualize policy claims and engage listeners.

Greeting and Deference: Use of formal politeness, honorifics, or conciliatory gestures that acknowledge the standing of opponents and establish deliberative trust.

Hybrid CB.

Metric 1: Interactivity and Engagement

Definition: This metric measures the degree to which participants directly respond to the claims of others, rather than engaging in parallel monologues. In a healthy deliberation, participants listen, ask genuine questions to solicit information, and build upon previous speakers’ points. Empirically, Knesset committees exhibit high levels of interactivity, but it frequently devolves into disruptive interruptions or cross-talk (e.g., shouting over one another during debates about terror and academia). Therefore, this metric evaluates both positive interactivity (substantive responses and reciprocal dialogue) and negative interactivity (disruptive interruptions that derail the speaker’s thought process).

Metric 2: Level of Justification and Epistemic Reliance

Definition: This metric assesses whether participants support their demands with evidence, logic, and expert knowledge, rather than mere assertion. High-quality discourse includes “sophisticated justifications” where multiple complete inferences are made. Furthermore, in a systemic deliberative approach, good discourse relies on the appropriate integration of expert knowledge. In the Knesset transcripts, this is observed through the heavy reliance on legal advisors (e.g., from the Ministry of Justice or the committee’s own counsel) to define legal boundaries, correct factual inaccuracies, and structure the consequences of legislation. The metric evaluates how well politicians incorporate this expert data into their arguments versus dismissing it for purely partisan reasons.

Metric 3: Civility, Respect, and Conflict Management

Definition: This evaluates the extent to which participants treat opposing groups, demands, and counterarguments with respect, which is a fundamental prerequisite for serious listening. In ideal deliberation, counterarguments are acknowledged and valued. The empirical Knesset data shows that committee meetings often struggle with this metric, featuring ad hominem attacks, sarcasm, and severe disrespect (e.g., calling opposing members a “fifth column” or comparing political opponents to terrorist organizations). This metric measures the frequency of degrading language versus respectful disagreement, tracking whether the discourse maintains a baseline of mutual recognition even during fierce ideological disputes.

Metric 4: Inclusiveness and Communicative Pluralism Definition: This metric assesses whether diverse voices, particularly those of marginalized groups and non-politicians, are given an equal and fair opportunity to influence the debate. Drawing on “Type II” deliberative theory, it recognizes that rational, dispassionate argumentation can sometimes exclude the less privileged, meaning that rhetoric, emotion, and storytelling are valid and necessary forms of democratic communication. Empirically, this is vital in the Knesset, where bereaved parents use highly emotional, personal narratives to urge legislative action, and minority Arab MKs express the lived reality of their constituencies regarding soaring crime rates. This metric evaluates how well the committee makes space for, and seriously considers, these diverse perspectives and expressive styles.

Metric 5: Constructive Politics and Problem-Solving Definition: This metric measures the committee’s ability to move beyond entrenched, positional politics toward generating alternative or mediating proposals that aim for consensus or practical solutions. True deliberation in a legislative setting must ultimately aim to reach a binding, workable decision. The Knesset transcripts frequently display this metric in action: despite intense partisan arguments, members, legal advisors, and government representatives engage in real-time bargaining to tweak the wording of a law, adjust implementation timelines, or compromise on budget allocations. The metric captures the density of these constructive, solution-oriented exchanges.

Metric 6: Moderator (Chairperson) Efficacy Definition: This metric evaluates the behavior and effectiveness of the committee chairperson in facilitating the deliberative process. A good moderator acts as a “promoter of deliberation” by managing the agenda, ensuring marginalized voices are heard, bringing the discussion back on topic, and intervening in conflicts.

topic, and intervening in conflicts. In the Knesset, the Chairperson (HaYo"r - originally in Hebrew letters) plays a highly active role in maintaining order, threatening to remove disruptive members, pushing for compromise, and clarifying procedural next steps. This metric tracks the chair’s success in balancing the enforcement of rules with the allowance of free and open debate.

Metric 7: Common Good vs. Sectorial Framing Definition: This metric evaluates the content of the justifications used by participants, specifically whether they appeal to narrow group interests or to the broader common good. Appeals to the common good can be utilitarian (the greatest good for the greatest number) or based on the difference principle (helping the least advantaged). In the empirical data, we see constant tension between sectorial lobbies (e.g., the agricultural lobby fighting to protect local honey production, or specific municipal authorities demanding resources) and the overarching national interest (e.g., lowering the cost of living or ensuring national security). This metric tracks how frequently and successfully participants link their specific, narrow claims to generalizable principles that benefit the entire society.

Proxy Judge Model Selection

Human benchmark. Two annotators independently rated 162 three-turn exchanges as high (1) or low (0) deliberative quality; scoring an inter-annotator agreement of 0.79 Krippendorff’s $\alpha$. The benchmark consists of the 148 exchanges on which both annotators agreed (40 positive, 108 negative); the 14 disagreements were excluded. The judge received the same instrument as the annotators: the coder definitions and clarifications, the session metadata, and the context turns surrounding the exchange. The clarifications were refined on a separate set of 21 expert-reviewed exchanges, none of which appear in the benchmark.

Model selection. We evaluated seven OpenAI model settings (GPT-5.6-luna, GPT-6-luna, GPT-6-luna-pro (reasoning effort set to ‘pro’), GPT-5.6-sol, GPT-6-sol, GPT-6.1-sol and GPT-6-astra), none of which were used in the CBs’ reliability evaluation. The benchmark was split, stratified by label, into a validation set of 98 exchanges (26 positive) and a held-out test set of 50 exchanges (14 positive). We ran a stratified 5-fold cross-validation and selected the model with the highest mean macro F1. Validation runs were issued through the OpenRouter chat API on October 12, 2026; after validation, the selected model scored the full corpus through the OpenRouter Batch API with the identical prompt. On the held-out test set, GPT-6-luna achieved an accuracy of 82% and a macro F1 of 77%, against the human labels.

Proxy C-DQI Distributions

In § 6.2, we evaluate the validity of the generated CBs by correlating their scores with an extrinsic proxy measure of discourse quality. To ensure this proxy serves as a valid and discriminatory baseline, we analyzed the distribution of its scores across the 100 evaluation transcripts.

Figure 5 illustrates the distribution of the proxy C-DQI scores as evaluated by GPT-6-luna. The scores exhibit a wide variance across the corpus, confirming that the proxy captures a broad spectrum of deliberative quality rather than collapsing into a narrow, uniform range. This distribution underscores the structural and topical diversity of the selected Knesset committee transcripts (as highlighted in Section 5) and provides a robust numerical foundation for the rank correlation analyses presented in Table 3.

Per-Model Correlation with the Proxy

Table 5 reports Spearman’s $\rho$ between each CB’s discussion-level score and the proxy judge (GPT-6-luna), broken out per annotator model rather than averaged. The top-down CB aligns stronger with the proxies, for four out of the six models. The top-Down and Vanilla DQI CBs exhibit the widest spread across models ($\Delta= 0.14$); whereas the Bottom-Up CBs are the most stable ($\Delta\in[0.09, 0.1]$).

CB	Claude	Gemma4	DictaLM	Qwen3.6	GPT-OSS	Mistral
Bottom-Up-Corrected	0.57	0.55	0.64	0.59	0.58	0.58
Bottom-Up-English	0.59	0.56	0.63	0.54	0.58	0.64
Bottom-Up-Hebrew	0.57	0.53	0.54	0.55	0.53	0.62
Top-Down	0.55	0.64	0.59	0.69	0.65	0.64
Hybrid	0.54	0.56	0.52	0.61	0.59	0.63
Vanilla DQI	0.48	0.53	0.41	0.55	0.49	0.53

User prompt (1/2)

Context

Below is a 3-turn exchange taken from a Knesset committee meeting. Your task is to rate the deliberation quality of the selected exchange.

Instruction

Evaluate the overall deliberation quality of the given exchange. The final score must be binary, where 0 reflects poor or destructive deliberation, and 1 indicates substantive deliberation.

• Good interactions (code 1): Statements in which a speaker presents an opinion or policy demand backed by logical reasoning, evidence, or justification, while maintaining strict decorum, demonstrating a respectful, constructive, or appreciative orientation toward other participants and their counterarguments.

• Neutral or poor interactions (code 0): Statements that express unreasoned claims or non-topical small talk (“neutral”), as well as statements containing personal attacks, sarcastic degradation, or explicit disrespect toward other speakers and their demands (“poor”).

Clarifications

• A score of 1 requires both elements: (1) respectful discourse, and (2) arguments based on internal logic or data.

• The definition of 0 includes: procedural statements, cynicism, outbursts, or a lack of reasoning, neutral stances and concluding remarks.

• Asking or answering a question does not qualify as substantive deliberation unless it directly addresses a contested policy issue with reasons or justification.

• Do not judge the validity of the claim, only the manner of its presentation; we lack the tools to evaluate domain-specific expertise, unless it involves a blatant falsehood.

• Do not consider the role or identity of the speaker when coding.

• Judge the interaction as a whole; even if there is a dominant speaker or turn, that role is given weight, but the other speakers are also considered.

• Do not infer deliberative quality from surrounding turns. If the exchange contains both substantive deliberation and subsequent procedural remarks, score 1 only when the substantive reasoning is an actual part of the exchange and not merely incidental to an otherwise procedural interaction.

• Interruptions (marked as “- - -”) are an indication of a disrespectful discussion, but a single interruption is not decisive on its own.

• Fragmented exchanges: count a turn as fragmentary if it is cut off (“- - -”) or is a short interjection of a few words. If two of the three turns are fragmentary, the speakers are not letting each other finish; score 0 even if the remaining turn contains a reasoned point.

• Implicit cynicism: even if the question is legitimate, the cynical manner in which it is presented steers the decision.

• Watch for mocking or sarcastic language, such as belittling a professional’s request for time by mockingly referring to a basic calculation as a “complex mathematical exercise”, which is considered disrespectful.

• Implicit cynicism via rhetorical questioning: If a speaker undermines a proposal, explanation, or request for authority by using a rhetorical, challenging question (e.g., mockingly asking “Until now you didn’t need this?”), it is considered cynical and disrespectful. Score the exchange as 0, even if the challenged speaker responds in a substantive and professional manner.

• Use the provided surrounding context to identify cynicism in the exchange to rate.

• If a participant asks for a date, deadline, or concrete commitment, or objects that a timeline is too long, a general explanation of the process or a vague promise (e.g. “soon”) without providing that information should score 0 for non-responsiveness.

• A reasoned procedural objection about notice, timing, or fairness does not by itself qualify as good deliberation. To receive a score of 1, the exchange must also include substantive engagement with the objection, such as a direct answer, justification, or concrete accommodation.