Latent Decision Flow Rewards for Clinical Guideline Reasoning

GUIDER project: https://zitniklab.hms.harvard.edu/GUIDER

GUIDER model: https://github.com/mims-harvard/GUIDER

Introduction

Current methods train large language models (LLMs) for clinical reasoning with rewards for final answer correctness or with preferences over complete responses [13]. These rewards are sparse and score each response as a single outcome. A clinical recommendation, however, depends on a sequence of conditional judgments about findings, contraindications, disease severity, and escalation thresholds, and an error at any step can change the recommendation. A model can also reach the correct answer through incomplete reasoning, with a rationale that omits essential evidence or deviates from clinical guidelines [1, 12, 8, 4]. Clinical reasoning models should therefore learn to reach correct answers through decision pathways consistent with clinical guidelines.

Clinical practice guidelines provide a principled source of process supervision, because they encode evidence-based recommendations as conditional decision structures [16, 23, 9, 14]. We call a sequence of clinical conditions and actions that leads to a clinical decision a decision flow. Recent work compiles guidelines into executable decision logic for training [20] and inference [3]. MedGuideX [20], for example, executes a deterministic decision tree for each case and trains the model on the resulting path. The decision flow behind an answer is latent, because several decision flows are compatible with the same patient case and answer. As Figure 1 shows, different guidelines can lead to the same clinical outcome through different decision flows. Two clinicians can also reach the same decision through different valid paths, based on their experience and interpretation of the evidence. We call this property path multiplicity. Under path multiplicity, a process reward computed against a single reference flow penalizes other flows that are also consistent with guidelines. Process supervision should therefore account for the set of plausible decision flows for each case.

Reinforcement learning with verifiable rewards (RLVR) [19, 5] improves clinical reasoning beyond supervised fine-tuning by optimizing verifiable outcome rewards [1, 12, 13]. Current RLVR methods for clinical reasoning optimize answer correctness or sequence-level reasoning quality and provide limited supervision on whether intermediate clinical decisions follow established guidelines. Clinical decision flows can provide this process supervision, because they specify the relevant patient conditions, decision criteria, and recommended actions. Process supervision derived from guidelines, however, introduces a second challenge. A valid decision flow can end in a diagnostic test, an eligibility criterion, or a management action that differs from the answer that the patient-specific question requires. We call this mismatch between process validity and answer correctness the path–answer gap. Under the path–answer gap, guideline adherence and answer correctness are distinct objectives, and a training objective should model them as complementary targets.

Present work. GUiDER (Guideline-Informed Decision Flow Rewards for Clinical Reasoning) is a reinforcement learning method that trains LLMs to generate guideline-consistent decision flows and correct clinical answers. To address path multiplicity, GUiDER treats the decision flow as a latent variable and represents guideline reasoning with several plausible flows for each case. A controller trained on guideline-derived reference flows generates candidate decision flows, and GUiDER computes the process reward as a score-weighted Monte Carlo average over these candidates. To address the path–answer gap, GUIDER gates the process reward on answer correctness, so a policy rollout receives process credit only when its final answer is correct. GUIDER optimizes the controller and the policy jointly. The policy generates its own decision flow from the patient case and receives guideline supervision through a scalar process reward. We train and evaluate GUIDER on MedGUIDE [16], an oncology benchmark derived from clinical guidelines with annotated decision paths, and evaluate transfer to the out-of-domain benchmarks CancerGUIDE [23], AMEGA-LLM [9], and RealMedQA [14].

This paper makes three contributions.

• We formulate guideline decision flows as latent process supervision for clinical RLVR and identify two properties of this supervision, path multiplicity and path–answer gap.

• We introduce GUIDER, which jointly optimizes a decision flow controller and a reasoning policy. The controller learns decision flows from guideline supervision. The controller then generates candidate decision flows, and GUIDER computes an answer-gated process reward through score-weighted Monte Carlo aggregation.

• On MedGUIDE, GUIDER achieves 82.5 weighted accuracy and 73.5 guideline adherence, 3.3 and 3.1 points above the strongest RLVR baseline. GUIDER also achieves the highest AMEGA-LLM score among RLVR methods (22.3) and improves over clinical RLVR baselines on RealMedQA and CancerGUIDE.

Related Work

Clinical Guideline Benchmarks. Clinical practice guidelines (CPGs) encode evidence-based recommendations for diagnosis and treatment as conditional decision structures, which makes them a natural basis for evaluating clinical reasoning. Recent benchmarks evaluate different aspects of reasoning grounded in guidelines. MedGUIDE derives decision-making tasks from oncology decision trees [16], and CancerGUIDE evaluates longitudinal treatment trajectories for which several clinical pathways are plausible [23]. Other benchmarks assess guideline adherence in multistage decision processes and open-ended clinical scenarios [4, 9]. RealMedQA [14] and GuidelineQA [7] evaluate realistic clinical questions whose answers depend on guideline recommendations, and Meditron3 [21] provides a large collection of multiple-choice questions derived from the guidelines of several healthcare organizations. These benchmarks measure guideline adherence at evaluation time, and benchmarks with annotated decision paths use these paths as evaluation targets. GUIDER uses annotated decision paths as supervision for reinforcement learning. Because several guideline pathways can be valid for one patient case, as CancerGUIDE makes explicit, GUIDER treats decision flows as latent variables with several valid realizations per case. We train and evaluate GUIDER on several of these benchmarks.

Guideline Adherence in LLMs. A complementary line of research represents CPGs in forms that LLMs can retrieve or execute. CPGPrompt converts guidelines into decision trees that LLMs use at inference [6], Guideline2Graph extracts clinical decision graphs from guideline documents [15], and Case2Flow retrieves the guideline flowcharts relevant to a patient case [25]. GuideSkill compiles guideline criteria into executable skills for LLM agents [3], and MEGA-CDP formulates guideline adherence as a multistage clinical decision process [4]. Most of these methods use retrieved, predefined, or executable CPGs at inference. The work closest to ours is MedGuideX [20], which compiles guideline recommendations into deterministic decision trees and trains LLMs on question answering data generated by executing these trees. Each case in MedGuideX has one executed path, the model imitates verbalized traces of this path during supervised fine-tuning, and the reinforcement learning stage rewards final answer correctness. Like MedGuideX, GUIDER uses guidelines during training and operates from the patient case alone at inference. GUIDER rewards the reasoning process directly during reinforcement learning, with a process reward aggregated over several plausible flows and gated on answer correctness, which accounts for path multiplicity and path–answer gap.

Verifiable and Process Rewards for Clinical Reasoning. Reinforcement learning with verifiable rewards (RLVR) improves reasoning by optimizing against outcomes that can be checked automatically [19, 5], and process reward models (PRMs) provide denser supervision over intermediate reasoning steps ([22, 17, 24]. Recent clinical methods use several forms of process reward. CRPO jointly rewards accuracy, faithfulness, and comprehensiveness ([11]), MRPO introduces step-aware rewards that penalize invalid intermediate reasoning ([13]), and FaithMed scores reasoning and retrieval steps against clinician-designed criteria ([26]). These methods score reasoning traces against predefined criteria, and GUIDER derives its process reward from the conditional structure of clinical guidelines. Process rewards derived from guidelines must account for two properties of clinical reasoning. Under path multiplicity, several guideline-consistent flows can support the same outcome, and a reward computed against a single reference flow penalizes valid alternative flows. Under path–answer gap, a valid flow can end in a step that differs from the answer for a given patient, so process and outcome rewards can disagree. GUIDER addresses path multiplicity by aggregating process rewards over several candidate flows from a controller trained on guideline-derived supervision. It addresses path–answer gap by gating this reward on answer correctness, so that only rollouts with a correct final answer receive process credit.

Method

Problem Formulation

Setup and Notation. Let $x \in\mathcal{X}$ denote a patient case and $y^* \in\mathcal{Y}$ its target clinical answer. Each training pair $(x, y^*)$ is associated with a clinical practice guideline $G$, from which we derive a reference decision flow $z^*$ for training supervision. The policy $\pi_\theta$ with parameters $\theta$ generates a decision flow $\hat{z}$ and a final answer $\hat{y}$ from the patient case, $(\hat{z}, \hat{y}) \sim\pi_\theta(\cdot\mid x)$. A decision flow controller $q_\theta$ generates candidate decision flows and provides process supervision to the policy during training. The controller and the policy share parameters $\theta$, differ in their prompts, and are optimized jointly. We write $z^*$ for the annotated reference flow, $z$ for a controller-generated flow, and $\hat{z}$ for a policy-generated flow. At inference, GUIDER uses only policy $\pi_\theta(\cdot\mid x)$, which generates the decision flow and the answer from the patient case alone. The method therefore targets guideline-aligned decision making, and guideline retrieval and identification are outside its scope.

The learning objective requires a clinically correct final answer $\hat{y}$ and a decision flow $\hat{z}$ consistent with the relevant guideline.

Path Multiplicity. Clinical guideline reasoning is a many-to-one problem because several valid decision flows can support the same clinical answer. For a patient–outcome pair $(x, y^*)$, there can exist flows

where $\mathcal{Y}(x,z)$ denotes the set of clinical answers compatible with patient $x$ and decision flow $z$. Supervision against a single annotated trajectory can therefore penalize alternative flows that are clinically valid.

Path–Answer Gap. A guideline-consistent decision flow can end in a step that differs from the answer the task requires. Its terminal node $v_T$ can be a clinical state, diagnostic test, eligibility criterion, or management action, so in general $v_T \ne y^*$. Answer correctness $(\hat{y}=y^*)$ and flow validity $(\hat{z} \in\mathcal{Z}(x;G))$ are therefore distinct objectives. A policy can produce the correct answer through a flow outside $\mathcal{Z}(x;G)$, or generate a valid flow that leads to an incorrect answer.

Clinical Intuition. Consider the chest pain example in Figure 1, in which the clinical question asks when clinicians should use high-sensitivity troponin testing. A decision flow for this question represents the guideline-derived clinical considerations that lead to the recommendation, such as the assessment of clinical suspicion for acute coronary syndrome, risk stratification, and the indication for troponin testing. The NICE and SBC guidelines express distinct but clinically consistent sequences that support the same recommendation, which illustrates path multiplicity. Because the patient case and the final answer are compatible with several valid flows, the decision flow is latent. The complete answer extends beyond the final step of each guideline flow and follows from the clinical decisions along the flow, which illustrates the path–answer gap. These properties motivate a training objective that accounts for several plausible decision flows and jointly optimizes flow validity and answer correctness.

GuIDeR Model

Figure 2 gives an overview of GUIDER, which trains a controller and a policy jointly. Given a patient case $x$, the controller generates several candidate decision flows, and the policy generates its own decision flow and final answer. GUIDER aggregates the agreement between the policy flow and the controller flows into a process reward, which accounts for several plausible paths for each case. GUIDER combines this process reward with answer correctness to optimize the policy and trains the controller on guideline-derived reference flows. Through this joint optimization, GUIDER transfers guideline supervision to the policy, and the policy generates decision flows from the patient case alone at inference.

We describe GUIDER in three parts. The decision flow controller learns to generate candidate decision flows from guideline-derived supervision, the latent decision flow reward aggregates process credit across these candidates, and joint controller and policy optimization trains the model for guideline-consistent reasoning and answer correctness.

Decision Flow Controller

The decision flow controller learns to generate candidate decision flows from guideline-derived supervision. During training, we pair each case with an annotated reference flow $z^{*} = (v_{1}^{*}, \ldots, v_{T}^{*})$ derived from its guideline $G$. We train the controller with the autoregressive negative log-likelihood

where $v_{<t}^{*} = (v_{1}^{*}, \ldots, v_{t-1}^{*})$ denotes the reference flow prefix. The reference flow $z^{*}$ supervises the controller, and the policy receives guideline supervision through the process reward. Sampling from the controller yields several candidate flows for each case, which allows the process reward to account for path multiplicity.

Latent Decision Flow Reward

Process Supervision over Multiple Flows. Under path multiplicity, supervision against a single annotated flow penalizes valid alternative flows. GUIDER therefore compares the policy flow with several candidate flows generated by the controller. At training step $t$, we sample $z^{(m)} \sim q_{\theta_t}(z \mid x)$, $m = 1,\ldots,M$, where $\theta_t$ denotes the shared model parameters before the update at step $t$.

We score each controller proposal by its coverage of the annotated reference flow. Let $\mathcal{V}(z^*)$ denote the set of reference decision nodes, and let $\operatorname{match}(v,z)$ indicate whether flow $z$ matches reference node $v$. The coverage score of proposal $m$ is

where $\lambda_{\mathrm{PRM}}$ scales the coverage score. A softmax over the scores gives the normalized proposal weights $w_m = \frac{\exp(s_m/\tau)}{\sum_{j=1}^{M}\exp(s_j/\tau)}$, $m = 1,\ldots,M$, where the temperature $\tau> 0$ controls how strongly the weights concentrate on proposals with higher coverage.

Monte Carlo Process Reward. We measure agreement between the policy flow and a controller proposal with token-set Jaccard similarity. Let $\operatorname{Tok}(z)$ denote the set of alphanumeric tokens in $z$ after lowercasing, stopword removal, and removal of tokens with at most two characters. The similarity is $J(z_1,z_2) = \frac{\lvert\operatorname{Tok}(z_1) \cap\operatorname{Tok}(z_2) \rvert}{\lvert\operatorname{Tok}(z_1) \cup\operatorname{Tok}(z_2) \rvert}$, with $J(z_1,z_2) = 0$ when either token set is empty.

For a policy-generated flow $\hat{z}$, each controller proposal contributes process credit in proportion to its reference coverage and its agreement with $\hat{z}$. The process reward is

This reward is the expected proposal-specific process credit under the categorical distribution defined by the weights $w_m$ (Appendix § 9.§ 1). Controller proposals with higher reference coverage contribute more to the reward, and the policy receives process credit in proportion to its agreement with this weighted set of candidate flows.

Answer-Gated Reward Shaping. Multi-flow process supervision accounts for path multiplicity, and the path–answer gap requires a second mechanism, because a guideline-consistent decision flow can lead to an incorrect clinical answer. We therefore combine the process reward with the verifiable answer reward $R_{\mathrm{ans}}(\hat{y},y^*) = \mathbb{1}[\hat{y}=y^*]$. The reward for a policy-generated completion $(\hat{z},\hat{y})$ is $R(\hat{z},\hat{y}) = R_{\mathrm{ans}}(\hat{y},y^*) + \lambda R_{\mathrm{ans}}(\hat{y},y^*)R_{\mathrm{proc}}(\hat{z})$, where $\lambda\ge0$ controls the contribution of process supervision. The answer gate gives process credit only to completions with a correct clinical answer. Multi-flow process supervision accounts for several plausible decision flows, and answer-gated reward shaping couples guideline-consistent reasoning with answer correctness, which addresses the two properties identified in § 3.1.

Joint Controller and Policy Optimization

GUIDER combines controller supervision and policy optimization in a single objective over shared parameters. Guideline-derived reference flows train the controller, and answer-gated process rewards aggregated over several controller-generated flows train the policy. The two roles share the same model parameters and receive joint updates.

At training step $t$, the shared model has parameters $\theta_t$. Before any parameter update at step $t$, this model generates $M$ controller flows and $K$ policy rollouts,

for $m = 1,\ldots,M$ and $k = 1,\ldots,K$.

For each controller proposal $z^{(m)}$, we compute its coverage score $s_m$ with Eq. (3). The process reward for policy rollout $k$ is

where $J(\cdot,\cdot)$ measures agreement between the policy flow and each controller proposal, and the answer-gated reward is

Following GRPO [19], we normalize the rewards within the group of $K$ rollouts,

where $\mu_R$ and $\sigma_R$ are the mean and standard deviation of the $K$ rollout rewards, and use these advantages in the standard GRPO objective $\mathcal{L}_{\mathrm{GRPO}}$.

Because the controller and the policy share parameters, we combine their objectives before a single optimizer update,

where $\alpha_t$ controls the contribution of controller supervision. A single optimizer step on $\mathcal{L}_{\mathrm{joint}}$ produces $\theta_{t+1}$. Controller proposals, policy rollouts, and rewards stay fixed during this update, so the two objectives are computed from the same pre-update model $\theta_t$.

Guideline-derived flow supervision and policy optimization thus train a single model, and the multi-flow process reward credits several plausible flows for each case. At inference, GUIDER uses only the policy $\pi_{\theta}(\cdot\mid x)$, which generates the decision flow and the answer from the patient case.

Experiments

Experimental Setup

Datasets and Performance Metrics. We train on the MedGUIDE training split [16] and a subset of Meditron3 (GuidelineQA) [21]. We evaluate on held-out MedGUIDE and cross-benchmark on AMEGA-LLM [9], RealMedQA [14], and CancerGUIDE [23].

We follow the evaluation protocol and judge templates provided by each benchmark, using GPT-5.4 as the LLM judge where required. For MedGUIDE, we report weighted accuracy (WAcc.) following Li et al. [16] and guideline adherence on a 0–5 scale. For AMEGA-LLM, we report the criterion-based aggregate score (0–50). For RealMedQA, we report the mean flow-judge score (0–5), and for CancerGUIDE, the mean clinical-agreement score (0–5). The benchmark-provided evaluation templates and scoring rubrics are reproduced in Appendix § 9.§ 4.

Baselines. We compare against medical foundation models, recent clinical reinforcement learning methods, and frontier LLMs. The medical models are Meditron-3 [21] and Fastino-Healthcare [2]. Our primary baselines are Clinical-R1 (CRPO) [11], which uses multi-objective verifiable rewards, MRPO [13], which uses step-wise process rewards, and FaithMed [26], which uses rubric-based process supervision for reasoning grounded in evidence. We also report GPT-5.6-Sol and Gemini-Flash-3.7 as frontier references.

Implementation Details. We use Qwen3-14B as our primary backbone and Gemma4-12B as a second backbone. Models are fine-tuned with QLoRA using 4-bit quantization, bfloat16 computation, LoRA rank 64, scaling factor 128, and dropout 0.05. Answer-only SFT is trained for one epoch with learning rate $2 \times10^{-4}$; GRPO and joint optimization are initialized from this checkpoint. For GRPO and joint optimization, we sample $K=4$ completions, use $\beta=0.04$, learning rate $5 \times10^{-6}$, batch size 16, maximum prompt length 2048, and disable thinking. Joint-MC policy completions are limited to 256 tokens. For joint optimization, we set $\alpha_t=1.0$ and $\lambda=0.2$, and sample $M=4$ controller flows with temperature 0.7 and top-$p=0.9$, limited to four decision nodes and 128 tokens. We set the reweighting temperature to $\tau=0.2$ and use the score-weighted MC process reward in Eq. (4). Joint-MC is trained for one epoch (approximately 655 updates on 10,466 examples) on a single NVIDIA H200 GPU. Evaluation uses deterministic decoding with thinking disabled. Unless otherwise stated, we report the mean and standard deviation over three independent runs with different random seeds. All methods use the same training data and initialization when applicable.

Results

Table 1 answers three questions about incorporating guideline decision flows into training.

Training Strategy	MedGUIDE		AMEGA-LLM	RealMedQA	CancerGUIDE
	WAcc.	Adh.
Base (Qwen3-14B)	52.3	47.2	17.2	1.62	1.71
SFT (ans)	78.0	53.1	20.3	1.73	1.73
SFT (ans) $\to$ GRPO (ans)	80.1	54.5	20.5	1.81	1.75
SFT (ans+flow)	64.0	71.3	18.8	1.82	1.80
SFT (ans+flow) $\to$ GRPO (ans+flow)	68.2	71.6	19.1	1.85	1.82
GUIDER	82.5	73.5	22.3	1.90	2.40

Q1: Can process rewards alone shape RL toward guideline adherence? Only to a limited extent. As shown in Table 1, process-reward optimization yields only modest improvements in guideline adherence (SFT (ans+flow) $\to$ GRPO (ans+flow): 71.3$\to$71.6), while answer performance remains substantially below answer-oriented training (68.2 vs. 80.1 WAcc.). This limitation arises because process rewards can only assign useful credit to decision flows that the policy is able to explore; without sufficient in-domain guideline knowledge, the base policy has limited access to high-quality guideline-consistent trajectories. The problem is further compounded by path multiplicity: multiple decision flows may be valid for the same clinical outcome, making reward assignment against a single reference flow overly restrictive. Thus, process rewards alone are insufficient for reliably acquiring guideline reasoning from scratch, motivating the use of supervised guideline knowledge to initialize the policy.

Q2: Can we first learn guideline decision flows with SFT and then optimize the resulting policy with RL? A natural solution to the exploration limitation above is to first inject guideline knowledge through supervised fine-tuning. Indeed, flow+answer SFT substantially improves guideline adherence over answer-only SFT (53.1$\to$71.3), demonstrating that the policy can acquire guideline-consistent decision patterns from supervised trajectories. However, this gain comes at a substantial cost in weighted answer accuracy (78.0$\to$64.0). Subsequent GRPO preserves the adherence gain (71.6) but only partially recovers answer accuracy (68.2). This divergence exposes the path–answer gap: learning to follow a guideline-consistent decision flow does not necessarily improve the final clinical answer. Moreover, committing the policy to a single supervised trajectory remains problematic under path multiplicity. Thus, sequentially learning flows through SFT and subsequently optimizing with RL does not fully reconcile process quality with outcome correctness.

Q3: Can we jointly optimize guideline reasoning and answer correctness? The limitations above motivate learning the decision process and clinical outcome jointly rather than in separate stages. Our joint MC formulation couples policy optimization with a controller that generates multiple candidate decision flows, allowing process supervision to evolve alongside the policy while accounting for path multiplicity through score-weighted aggregation. This avoids requiring the initial policy to discover high-quality guideline flows on its own and avoids treating a single annotated trajectory as the unique reasoning target. Joint optimization consequently achieves the strongest combination of MedGUIDE weighted accuracy (82.5) and guideline adherence (73.5), while also providing the strongest overall transfer across the evaluated out-of-domain benchmarks. These results support jointly optimizing process and outcome signals while explicitly accounting for the one-to-many relationship between clinical answers and valid guideline decision flows.

Comparison with baselines. Table 2 compares GUIDER against frontier and open-source medical LLMs, as well as recent clinical reasoning optimization methods. GUIDER consistently outperforms CRPO, FaithMed, and MRPO across both model families. On Qwen3-14B, GUIDER achieves the highest MedGUIDE weighted accuracy and guideline adherence (82.5/73.5), improving over the strongest baseline by 3.3 and 3.1 points, respectively. These gains extend beyond the in-domain benchmark: GUIDER also obtains the highest AMEGA-LLM score (22.3) and improves over all three baselines on RealMedQA and CancerGUIDE. A similar pattern holds for Gemma4-12B, where GUIDER improves both answer accuracy and guideline adherence while providing stronger overall out-of-domain performance. Notably, the gains are not explained by using a medically specialized backbone: GUIDER substantially outperforms Meditron-3-8B and Fastino-Healthcare-30B despite operating on general-purpose base models. Finally, on Qwen3-14B, GUIDER surpasses GPT-5.6-Sol on MedGUIDE weighted accuracy (82.5 vs. 70.0) and adherence (73.5 vs. 67.2), while GPT-5.6-Sol remains stronger on RealMedQA (2.8 vs. 1.9). Overall, these results show that the improvements are consistent across model scales and extend from guideline adherence to answer accuracy and out-of-domain clinical reasoning.

Model	MedGUIDE		AMEGA-LLM	RealMedQA	CancerGUIDE
	WAcc.	Adh.
GPT-5.6-Sol	70.0	67.2	19.1	2.8	2.6
Gemini-Flash-3.7	67.2	61.5	18.8	2.4	2.3
Meditron-3-8B [21]	34.7	62.3	18.3	1.5	1.9
Fastino-Healthcare-30B [2]	36.7	41.3	16.1	1.6	1.3
Gemma4-12B
CRPO [11]	62.1±0.6	63.5±0.7	18.2±0.3	1.2±0.1	1.9±0.1
FaithMed [26]	63.5±0.5	55.3±0.8	19.4±0.2	1.3±0.1	2.1±0.1
MRPO [13]	63.9±0.6	55.6±0.7	19.3±0.3	1.4±0.1	2.4±0.1
GUIDER	67.5±0.5	65.2±0.6	20.2±0.2	1.7±0.1	2.8±0.1
Qwen3-14B
CRPO [11]	75.3±0.5	70.4±0.7	21.2±0.2	1.6±0.1	1.9±0.1
FaithMed [26]	77.8±0.4	66.9±0.6	20.7±0.3	1.7±0.1	2.1±0.1
MRPO [13]	79.2±0.5	67.5±0.6	21.9±0.2	1.8±0.1	2.3±0.1
GUIDER	82.5±0.4	73.5±0.5	22.3±0.2	1.9±0.1	2.4±0.1

Discussion

We introduce GUIDER, a reinforcement learning method that uses clinical practice guidelines as process supervision for reasoning models. In our experiments, guideline adherence and answer correctness behave as distinct training objectives. Supervised training on annotated decision flows improves adherence and reduces answer accuracy, and answer rewards improve accuracy with smaller gains in adherence. By computing the process reward over several plausible decision flows and gating it on answer correctness, GUIDER improves the two objectives together, and these improvements extend to out-of-domain clinical benchmarks.

Limitations and Future Work. GUIDER shares parameters between the controller and the policy. We chose this design because the base policy has limited in-domain guideline knowledge, and training the shared model on reference flows provides this knowledge during reinforcement learning. The quality of controller flows is therefore bounded by the guideline knowledge of the shared model, and errors in these flows enter the process reward. A separate controller with stronger guideline-specific knowledge could provide more accurate candidate flows and a more reliable process reward. Sampling $M$ controller flows for each case also adds computational cost to each training step, and this cost grows with $M$ and with flow length. More efficient proposal generation and selection, for example by adapting $M$ to the uncertainty of each case, could reduce this cost. The process reward measures agreement between flows with token-set Jaccard similarity, which captures lexical overlap and can miss clinically equivalent decision states expressed in different terms. Semantic matching or matching validated by clinicians could give a more accurate measure of agreement. Finally, training data come mainly from oncology guidelines, several evaluation metrics rely on LLM judges, and clinical safety requires evaluation by clinicians. Extending GUIDER to other specialties and clinical settings, and evaluating it with clinicians, are directions for future work.

Conclusion. GUIDER jointly optimizes a decision flow controller and a reasoning policy and uses clinical decision flows as process supervision for reinforcement learning. Across in-domain and out-of-domain benchmarks and two model families, GUIDER improves answer accuracy and guideline adherence over answer-only training, supervised flow training, and clinical process reward baselines.

These results indicate that structured clinical guidelines can supervise the training of clinical reasoning models, and the trained policy generates guideline-consistent decision flows from the patient case alone at inference. More broadly, a model that states its decision flow in guideline terms allows clinicians to check each step of a recommendation against the relevant guideline, and the same training approach applies to other domains that encode expert knowledge as conditional decision procedures with several valid paths to the same decision.

AI Use Statement

In this work, we used generative AI tools to assist with language editing, improving the clarity and presentation of the manuscript, and supporting code development and debugging. Generative AI was not used to autonomously conduct experiments, determine research conclusions, or evaluate the proposed method. All AI-assisted text and code were reviewed by the authors, and experimental results, analyses, citations, and scientific claims were independently verified. The authors take full responsibility for the final content of this work, including any text, claims, or artifacts produced with the aid of generative AI.

Ethics Statement

GUIDER is a research method, and its use in clinical care would require clinical validation. This work uses benchmark datasets and involves no human subjects or identifiable patient information. The main risk is that a model trained to follow guidelines can produce fluent but incorrect or outdated recommendations. Our evaluation metrics, several of which rely on LLM judges, measure agreement with reference answers and guideline paths, and clinical safety requires separate assessment by clinicians. The guidelines used in this work come mainly from high-income countries and can encode contested demographic factors. Fairness across patient subgroups is outside the scope of our evaluation and requires further study. Clinical implementation would require prospective validation and oversight by clinicians.

Acknowledgments

We gratefully acknowledge support, in part, by NSF CAREER 2339524, Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists (ROADS) Program, Sanofi iDEA-iTECH Award, GlaxoSmithKline Award, Boehringer Ingelheim Award, Merck Award, Optum AI Research Collaboration Award, Pfizer Research, Gates Foundation (INV-079038), Chan Zuckerberg Initiative, John and Virginia Kaneb Fellowship at Harvard Medical School, Biswas Computational Biology Initiative in partnership with the Milken Institute, Harvard Medical School Dean’s Innovation Fund for the Use of Artificial Intelligence, and the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. This work was delivered as part of the Aurora project supported by the Cancer Grand Challenges partnership funded by Cancer Research UK (CGCAII-Mar26/100001). Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders.

Appendix

Derivation of the Multi-Flow Process Reward

At training step $t$, we draw $M$ candidate decision flows from the controller role of the shared model,

Each proposal receives a reference-path coverage score $s_m$ as defined in equation (3). We use these scores to define a categorical distribution over the sampled proposals,

Hence, proposals with greater coverage of the annotated reference flow receive greater mass, while lower-scoring proposals can still contribute to the reward.

For a policy-generated flow $\hat{z}$, define the proposal-specific process credit

where $J$ is the token-set Jaccard similarity defined in equation (4). The process reward is the expectation of this credit under the score-induced categorical distribution over the sampled controller proposals:

Substituting equation 10 gives the equivalent form

This construction separates two sources of process information: $s_m$ measures the reference-path support of each controller proposal, while $J(\hat{z}, z^{(m)})$ measures its agreement with the policy-generated flow. Together, they provide score-weighted process supervision over multiple controller-generated flow candidates rather than requiring exact agreement with a single annotated reference flow.

Effect of the Number of Decision Flows

We study the effect of the number of controller-sampled decision flows $M$ used to estimate the process reward. Increasing $M$ from 2 to 4 improves MedGUIDE weighted accuracy from 80.3 to 82.5 and adherence from 72.5 to 73.5, with similar gains across the out-of-domain benchmarks. Beyond $M = 4$, performance largely saturates: weighted accuracy remains stable around 82.3–82.4, while adherence continues to improve modestly to 74.5 at $M = 8$. These results suggest that a small set of sampled flows is sufficient to capture path multiplicity, with additional samples providing diminishing returns.

Effect of Decision-Flow Length

We study the effect of the maximum number of decision nodes $N$ allowed in each controller-sampled flow. Increasing $N$ from 2 to 4 substantially improves both MedGUIDE weighted accuracy (77.2 to 82.3) and guideline adherence (68.5 to 74.5), with corresponding gains on all out-of-domain benchmarks. Increasing the flow length beyond four nodes provides little additional benefit: adherence remains comparable at $N = 6$ (74.4), while weighted accuracy slightly decreases to 81.2. At $N = 8$, both metrics decline further, with similar degradation on the out-of-domain benchmarks. These results suggest that moderately sized decision flows provide sufficient structure to capture clinically relevant decision steps, while overly short flows omit useful process information and longer flows introduce unnecessary complexity without consistent performance gains.

# Decision Flows (M)	MedGUIDE WAcc.	MedGUIDE Adh.	AMEGA-LLM	RealMedQA	CancerGUIDE
1	77.1	70.2	20.3	1.6	1.9
2	80.3	72.5	21.6	1.7	2.1
4	82.5	73.5	22.3	1.9	2.4
6	82.4	74.1	22.1	1.9	2.3
8	82.3	74.5	22.2	1.9	2.3

# Decision Nodes (N)	MedGUIDE WAcc.	MedGUIDE Adh.	AMEGA-LLM	RealMedQA	CancerGUIDE
2	77.2	68.5	20.3	1.5	1.8
4	82.3	74.5	22.2	1.9	2.3
6	81.2	74.4	22.4	1.8	2.4
8	80.5	72.8	21.5	1.8	2.1

Evaluation Templates

For open-ended and guideline-adherence evaluation, we follow the evaluation templates and scoring rubrics provided by the respective benchmarks. We do not modify the benchmark-specific evaluation criteria; we instantiate the LLM judge with GPT-5.4. The templates used in our experiments are reproduced below for completeness. Judges receive the benchmark-specified case, reference evidence or criteria, and model output, but not the model identity or training condition. The templates for MedGUIDE guideline-adherence, AMEGA-LLM, RealMedQA, and CancerGUIDE evaluation are provided in Figure 5, Figure 6, Figure 7, and Figure 8, respectively.

Template

You evaluate whether a model-generated clinical decision/support flow is supported by a NICE guideline recommendation. Use only the provided clinical question, reference recommendation, and model output.

Score the model output against the reference NICE recommendation.

Scoring rubric:

5 = fully answers the question and is faithful to the recommendation. 4 = mostly faithful, with only minor omissions.

3 = partially supported or incomplete. 2 = weakly related, with important missing or unsupported components.

1 = mostly unsupported or misleading. 0 = contradicted, unsafe, or unrelated.

Also judge whether the decision flow explicitly connects the clinical question to the recommendation.

Return JSON: ”score”: ;

Template

You evaluate NSCLC treatment recommendation predictions against a reference CancerGUIDE recommendation. Use only the patient profile, reference recommendation, and model output.

Score whether the model’s final recommendation is clinically equivalent to the reference recommendation.

Scoring rubric: 5 = fully answers the question and is faithful to the recommendation.

4 = mostly faithful, with only minor omissions. 3 = partially supported or incomplete.

2 = weakly related, with important missing or unsupported components. 1 = mostly unsupported or misleading.

0 = contradicted, unsafe, or unrelated.

Also judge whether the decision flow explicitly connects the clinical question to the recommendation.

Return JSON: ”score”: ;

Controller and Policy Prompts

We use separate prompts for the controller and policy, provided in Figure 3 and Figure 4. Both roles receive the same clinical question. The controller is prompted to generate plausible guideline-style candidate decision flows, while the policy independently generates its own decision flow followed by the final answer. The policy does not observe the controller-generated flows; instead, they provide process-level supervision through the reward. This separation provides process supervision without directly constraining the policy to reproduce a particular trajectory.

Qualitative Analysis

In the MedGUIDE example (Figure 9), the model identifies the key decision states—relapse after systemic therapy and poor performance status (ECOG PS 3)—and generates a concise decision flow consistent with the gold guideline recommendation, ultimately selecting the correct treatment option. This demonstrates that the model can recover clinically relevant decision states without requiring exact reproduction of the reference flow.