WorksComputer Science
MamaReason: Unfinished Reasoning and Answer Recovery in Clinical Language Models
Abstract
Answer continuation often returns a diagnosis already stated in unfinished clinical reasoning. We study this behavior in MamaReason, adapted from dense Gemma 4 31B, on MamaBench's 217 paired maternal and child-health presentations. An explicit-commitment extractor identifies a diagnosis in 75 of 94 failed RL first passes and 75 of 93 failed SFT first passes. The continuation matches that diagnosis in 67/75 RL cases and 58/75 SFT cases, using diagnosis strings alone. Reading these commitments without additional generation recovers 30 of the 40 RL pairs restored by continuation under the archived correctness labels. Recurrence controls distinguish routine mentions from repeated answer selection. An offline second-commitment trigger measures the tradeoff between earlier readout and paired accuracy. The resulting decomposition separates diagnostic content already present in a trace, completion of an accepted answer, and paired diagnostic correctness. For clinical language-model evaluation, these are distinct outcomes that become visible when the same reasoning trace is examined through multiple readouts.
Introduction
A clinical language model can settle on a diagnosis in its reasoning yet fail to return the answer expected by an evaluation harness. When the harness then asks the model to finish, a higher score can reflect several changes: exposing a diagnosis already present, revising that diagnosis, or supplying a missing clinical qualifier. These possibilities matter because clinical reasoning benchmarks usually score the final answer, while the distinction between diagnostic content and answer completion remains hidden.
Medical QA benchmarks assess clinical knowledge and biomedical evidence use [1, 2, 3]; broader evaluations examine open-ended responses and diagnostic assistance [4, 5, 6, 7, 8]. Paired evaluations add a complementary requirement: success on related presentations whose discriminating details support different diagnoses [9, 10, 11, 12]. We use MamaBench’s maternal and child-health pairs to examine how diagnoses already stated in a reasoning trace relate to the answers and scores produced by subsequent readout.
Our central finding is label-independent. In MamaReason, adapted from dense Gemma 4 31B through SFT and clinical-content RL [13], a frozen rule set extracts explicit commitments from 75/94 failed RL first passes and 75/93 failed SFT first passes. Continuation returns the same normalized diagnosis in 67/75 and 58/75 extracted cases, respectively. Across all failures, this gives observed matches in 67/94 RL cases (71.3%) and 58/93 SFT cases (62.4%), with abstentions retained in the denominator. Most extractable failures therefore contain the diagnosis that the continuation eventually supplies.
We compare three readouts of the saved first passes: the existing JSON parser, an extraction-only last-diagnosis fallback, and the actual conditional continuation. Under the archived correctness labels, the RL fallback recovers 30 of the 40 pairs restored by continuation; 29 of these recoveries use matching continuation diagnoses and their saved verdicts. Figure 1 connects this scored recovery to the diagnosis-preservation result. Recurrence controls and prefix readouts then characterize the traces surrounding these commitments. This extends the study of test-time reasoning and budget forcing [14, 15, 16] to the clinical distinction between reaching a diagnosis and delivering an accepted answer.
The paper contributes:
A label-independent measurement of agreement between explicit first-pass commitments and continuation diagnoses, with extraction coverage reported for both checkpoints.
A paired-correctness decomposition across parser, commitment, and continuation readouts, including uncertainty for comparisons with direct answering.
Recurrence controls and interval answer-availability measurements that characterize repeated answer selection and late completion.
Related Work
Medical QA and diagnostic evaluation
MedQА and MedMCQA evaluate medical examination knowledge, while PubMedQA tests interpretation of biomedical abstracts [1, 3, 2]. Thapa et al. separate knowledge-heavy from reasoning-heavy biomedical questions and examine misleading initial hypotheses [12]. Broader clinical evaluations include long-form answer quality in MultiMedQA, physician-authored rubrics in HealthBench, a taxonomy of medical tasks in MedHELM, and diagnostic assistance in differential-diagnosis studies and AMIE [4, 5, 6, 7, 8]. MamaBench supplies a paired maternal and child-health diagnosis setting [11]. We connect its final-answer scores to diagnostic content in the preceding trace.

Figure 1. Diagnosis preservation before continuation RL: commitments extracted from 75/94 failures; 67/75 match the continuation. SFT: commitments extracted from 75/93 failures; 58/75 match the continuation. These matches use diagnosis strings alone. Scored recovery: the RL fallback recovers 30/40 continuation-restored pairs under archived verdicts. Diagnosis preservation and paired recovery. The direct-answer reference is generated separately; the other bars share the same bounded-thinking first pass. Continuation adds a conditional answer-generation step. Commitment uses the completed diagnostic adjudication. All robust scores use 217 pairs.
Counterfactual robustness and behavioral testing
Contrast sets probe local decision boundaries through meaningful test-instance changes [9]. Counterfactually augmented data uses coherent edits to expose reliance on patterns that fail under revision [17], and CheckList organizes tests around capabilities and expected behavior [10]. MamaBench applies this perspective to related clinical presentations. Its pairs can change several observations, making joint correctness a test of sensitivity to the benchmark’s evidence changes.
Synthetic supervision and medical adaptation
Self-Instruct combines generated instructions with filtering [18]; STaR iterates rationale generation and training on correct-answer trajectories [19]. In medicine, HuatuoGPT-o1 uses verifiable problems for reasoning supervision and RL, while MedReason grounds reasoning paths in medical knowledge graphs [20, 21]. MamaReason uses separate construction routes for structurally cleaned SFT demonstrations and reference-grounded RL vignettes. The distinction between generating, filtering, and verifying supervision is relevant to both routes.
Reasoning traces and overthinking
Chain-of-thought prompting elicits intermediate steps, and self-consistency aggregates answers across sampled paths [22, 23]. Studies of intrinsic self-correction and overthinking examine how later reasoning can preserve, improve, or degrade an earlier solution [24, 25]. Chen et al. assess the efficiency of successive solution rounds [25]; our clinical analysis examines whether a continuation preserves an earlier diagnosis or changes a consequential qualifier.
Visible reasoning also raises questions of faithfulness. Turpin et al. and Chen et al. show that generated explanations can omit influences on predictions [26, 27]. Lanham et al. test reasoning through interventions, including early answering after truncation [28]. Their active intervention asks what answer a model produces from an intermediate prefix. Our passive readout asks what diagnosis is already explicitly stated in the saved text. This distinction locates the present study between analysis of intermediate reasoning and inference-time answer control.
Inference control and structured answers
Effective allocation of test-time computation depends on problem difficulty [29]. s1 controls deliberation through budget forcing [14]; Token-Budget-Aware LLM Reasoning and BudgetThinker adapt reasoning to available computation [15, 16]. PICARD and guided generation enforce formal output structure during decoding [30, 31]. These methods act on generation. By holding the first-pass text fixed, our comparison separates existing diagnostic assertions from the answer supplied by a continuation, providing a reference point for active early-answering and stopping strategies.
Reward optimization and verification
Instruction tuning and feedback-based RL adapt language-model behavior [32]. PPO uses clipped policy updates, and GRPO uses group-relative rewards in place of a learned critic [33, 34]. Process supervision evaluates intermediate steps, while outcome supervision evaluates completed trajectories [35]; Dr. GRPO and DAPO study further optimization choices [36, 37]. MamaReason’s clinical reward covers diagnosis, findings, urgency, and management. Benchmark diagnosis scoring and clinician review provide separate assessments, each with its own criteria. Work on LLM-based evaluation motivates examining the consistency of saved diagnostic verdicts [38].
Methodology
Model and study setting
We introduce MamaReason as the clinical model lineage for this evaluation study, adapted from the dense Gemma 4 31B model [13]. We examine its SFT checkpoint and the checkpoint produced by one subsequent clinical-content RL run. The analysis combines saved generations with archived and additional diagnostic verdicts. We report the data construction, inference protocols, and readout rules following the emphasis on transparent clinical-LLM reporting in TRIPOD-LLM [39].
Training context
The SFT checkpoint uses clinical reasoning demonstrations with separate thought and visible-answer channels: 10,122 training examples and 533 validation examples. RL starts from that checkpoint and uses 1,000 vignettes spanning 111 conditions. MamaBench supplies the evaluation cases. Detailed preparation, sampling, and cleaning are in Appendix § 8 and Table 6.
| Stage | Records |
| Supervised source QA | 28,082 |
| Unique normalized questions | 22,798 |
| Sampling target | 12,000 |
| Saved reasoning demonstrations | 11,591 |
| Cleaned demonstrations | 10,655 |
| SFT train / validation | 10,122 / 533 |
| Re-extracted vignette pool | 1,627 |
| RL subset (111 conditions) | 1,000 |
| Emergent / urgent / routine | 571 / 397 / 32 |
Table 6. Data construction. The final 1,000-case subset is the dataset used for RL; the larger pool is not the training size.
Adaptation summary
Both stages use quantized LoRA adaptation [40, 41]. RL starts from the SFT checkpoint, samples four completions per prompt, and combines rule-based diagnosis, finding, urgency, and management rewards. Finding and management coverage can use thought text; the objective has no explicit repetition or length penalty. Appendix § 8 gives the reward equations and optimization settings.
Paired evaluation and answer emission
MamaBench contains 217 pairs, or 434 clinical narratives, processed independently [11]. The principal prompt covers obstetric, gynecologic, neonatal, and pediatric diagnosis and allows non-obstetric answers. The main comparison uses this prompt in both checkpoints.
Table 1 gives the output caps and answer procedures. Decoding is greedy, with temperature 0 and top-p = 1. Bounded thinking triggers a continuation after a JSON parsing failure, blank prediction, or template answer. The continuation closes the thought channel if necessary and requests schema-constrained JSON, with an unconstrained fallback. Accepted first-pass predictions are retained. Raw thought and continuation text are stored separately.
| Protocol | Maximum output | Emission procedure |
| Direct answer | 1,024 tokens | Thinking disabled through the chat template. |
| Bounded thinking | 3,000 + 512 tokens | Conditional thought closure and answer-rescue continuation. |
Table 1. Principal inference protocols and maximum output lengths.
A temperature-zero diagnostic evaluator receives the reference diagnosis and prediction. Its criteria accept synonymous names and require clinically consequential qualifiers. Blank or placeholder predictions and explicit parsing/evaluator errors count as incorrect. Record-level verdict consistency is documented in Appendix § 15.
Let , , , and denote both correct, base-only correct, counterfactual-only correct, and neither correct. For pairs,
Robust accuracy requires both presentations correct. Bias Trap Rate (BTR) is the counterfactual failure rate among base-correct pairs, defined when . All principal comparisons contain complete pairs.
Replaying the first-pass readout
For rescued records, we remove the marked continuation and its synthetic channel delimiter, then apply the evaluator’s JSON parser and blank/template gate to the first-pass text. A prediction is available when the parser returns a nonempty, non-placeholder diagnosis. This includes JSON objects recoverable from an unclosed thought channel. For available first passes, we verify agreement with the saved prediction and retain its verdict; unavailable predictions score zero.
Continuation markers, replayed first-pass failures, and successful-rescue flags agree exactly in the principal runs: 94 RL and 93 SFT cases. Every marked continuation yields an accepted prediction. Comparing first-pass and final scores therefore holds the original first-pass generation fixed.
For pair , let indicate that both first-pass predictions are available and indicate joint correctness. Pair availability and conditional robust accuracy are
The unconditional score is
All unconditional scores retain 217 pairs.
Token reconstruction and prefix availability
We re-tokenize saved text with a public base tokenizer, preserving special tokens and adding none. Decoding reproduces the text exactly for all 868 principal bounded-thinking first passes. Appendix B records reconstruction details. We describe a failure as cap-consistent when its reconstructed length is within one token of 3,000, and detect a narrow exact-cycle signature when an 8–32-token block repeats at least three times consecutively. Thought-channel closure and explicit end-of-turn markers are recorded separately.
At prefix lengths of 250, 500, 750, 1,000, 1,500, 2,000, 2,500, 2,750, and 3,000 tokens, we reapply the same parser and gate. For case , let indicate availability at prefix . The answer yield over an observed interval is
We compute yields within each saved run; the longer historical configuration is described in Appendix § 11. Their denominators retain earlier-stopped outputs; we also report the subset whose saved length exceeds . These are readouts of fixed text, so availability can precede the end of a saved generation.
Commitment extraction and recurrence
The extractor selects the last eligible diagnosis assertion or answer-selection statement. It scans first-pass text alone, using a frozen set of cues and exclusions specified in Appendix § 9.§ 1. With no eligible assertion, it abstains. Rules were developed on RL traces and applied unchanged to the SFT failures.
We measure recurrence in the final 500 thought tokens. Diagnosis mentions are word-boundary matches after case and punctuation normalization; explicit assertions additionally require an eligible selection cue. Two matches define recurrence. The control uses the accepted diagnosis in 340 available RL first passes, excluding visible-answer text. For failed outputs, we examine both the extracted commitment in 75 eligible cases and the continuation diagnosis in all 94 cases. A complementary whole-trace repeated-four-gram statistic, , is defined in Appendix § 14.
Offline second-commitment readout
We apply the frozen assertion rules to prefixes of each observed thought channel. The trigger is the first prefix containing two eligible assertions of the same normalized diagnosis. It selects the diagnosis at the second assertion. If the thought channel closes first or the rule never triggers, the original first-pass parser result is retained. The detector uses only text available at that prefix.
We record trigger coverage, agreement with the saved final prediction, and paired correctness of this extraction-only policy. The text reduction is the fraction of reconstructed first-pass tokens beyond the selected prefixes, summed over all 434 cases. It excludes rescue-continuation tokens. This offline analysis reads the selected diagnosis directly; live continuation after a trigger is a separate experiment.
Correctness scoring and paired uncertainty
The extraction-only fallback retains accepted first-pass predictions and fills failed cases with extracted commitments. Matching diagnoses reuse the archived verdict for that case. The remaining 20 last-commitment candidates (five RL and 15 SFT) are scored with the same diagnostic evaluator, prompt, and temperature-zero settings as the original benchmark run. Three additional candidates from the early-readout experiment are scored identically. All extracted candidates now have verdicts; abstentions remain unavailable. Diagnosis-string agreement is measured separately from these correctness labels. Appendix § 9.§ 2 records the added assessments.
We resample the same 217 pair identifiers 20,000 times with replacement, using seed 42, and report percentile 95% bootstrap intervals [42]. Exact two-sided McNemar tests use gains and losses in joint correctness. We apply both procedures to first-pass-minus-direct and fallback-minus-direct comparisons. Continuation recovery is monotone because only failed first passes receive a continuation. Tests are exploratory and unadjusted for multiple comparisons. The exact test conditions on discordant pairs, which can produce a different threshold decision from the percentile bootstrap near zero.
Results
Continuations preserve earlier diagnostic commitments
The extractor identifies commitments in 75/94 failed RL first passes (79.8%) and 75/93 failed SFT first passes (80.6%), abstaining in 19 and 18 cases. Continuation matches the extracted diagnosis in 67/75 RL cases (89.3%) and 58/75 SFT cases (77.3%). Counting matches over all failures gives 67/94 (71.3%) and 58/93 (62.4%). These results use normalized diagnosis strings alone and show that the continuation commonly returns a diagnosis already stated in the unfinished trace.
Answer availability changes the protocol comparison
Table 2 reports availability and correctness for all principal conditions. The RL first pass is jointly correct on 98/217 pairs (45.2%), compared with 116/217 (53.5%) for direct answering: −8.3 points (paired-bootstrap 95% interval, −14.3 to −2.3; 13 gains, 31 losses; McNemar ). For SFT, the difference is also −8.3 points (43.8% versus 52.1%; interval, −14.3 to −2.8; 12 gains, 30 losses; ).
| Checkpoint | Readout | Available cases | Available pairs | Correct cases | Correct pairs | Robust (%) |
| SFT | Direct answer | 434 | 217 | 307 | 113 | 52.1 |
| SFT | Thinking first pass | 341 | 133 | 279 | 95 | 43.8 |
| SFT | Thinking end-to-end | 434 | 217 | 332 | 132 | 60.8 |
| RL | Direct answer | 434 | 217 | 311 | 116 | 53.5 |
| RL | Thinking first pass | 340 | 135 | 285 | 98 | 45.2 |
| RL | Thinking end-to-end | 434 | 217 | 339 | 138 | 63.6 |
Table 2.
Continuation raises RL robust accuracy to 138/217 (63.6%) and SFT to 132/217 (60.8%), restoring 40 and 37 jointly correct pairs. The RL recovery is 18.4 points (bootstrap interval, 13.4–23.5), with all accepted first-pass answers retained.
The RL first pass supplies both answers for 135/217 pairs (62.2%) and is jointly correct on 98/135 of them (72.6%). Their product gives the unconditional score exactly: (135/217)(98/135) = 98/217 = 45.2%. For SFT, 133/217 pairs are available and 95/133 are correct, giving 43.8% overall. Thus conditional correctness and unconditional availability describe different aspects of the same generations.
Repetition accompanies failed answer completion
The failure pattern is already present before clinical RL: SFT fails to supply an accepted first-pass answer in 93/434 cases (21.4%), compared with 94/434 (21.7%) after RL. Most failed outputs remain in the thought channel near the cap; Appendix § 9 and Table 7 give the stopping signatures. The similar rates place unfinished reasoning in both adaptation stages.
| Failed-output pattern | RL | SFT |
| All first-pass failures | 94 | 93 |
| JSON parse failures | 93 | 91 |
| Blank/template predictions | 1 | 2 |
| Exactly 3,000 reconstructed tokens | 90 | 90 |
| Within one token of cap | 92 | 90 |
| Below cap with exact suffix cycle | 2 | 3 |
| Closed thought channel | 0 | 0 |
| Explicit end-of-turn marker | 0 | 0 |
Table 7. First-pass failure signatures. Cap-consistent means within one reconstructed token of 3,000. Shorter exact-cycle failures form a separate group.
Repeated selections also occur in the SFT traces: 26/75 extractable failures repeat an explicit commitment. The behavior therefore precedes the RL reward used in this study.
Figure 2 shows token lengths and prefix availability. RL availability reaches 301/434 cases and 105/217 pairs at 1,000 tokens, 332/434 cases and 128/217 pairs at 2,000, and 340/434 cases and 135/217 pairs at 3,000. The last 1,000 tokens add eight accepted cases, a 1.8-point increase.

Figure 2. Length and availability in fixed first-pass trajectories. Left: empirical token-length distributions. Right: parser availability at saved prefixes, with denominators of 434 cases and 217 pairs. RL case availability rises from 332 to 340 between 2,000 and 3,000 tokens.
Conditional yields decline within the principal run (Table 11, Appendix § 11): 31/133 previously unavailable cases gain an answer between 1,000 and 2,000 tokens (23.3%), followed by 8/102 between 2,000 and 3,000 (7.8%). Excluding the one output already stopped below 2,000 gives an active-trace yield of 8/101 (7.9%).
| Protocol | Token interval | Unanswered at start | New answers | Yield (%) |
| Clarified bounded thinking | 1,000–2,000 | 133 | 31 | 23.3 |
| Clarified bounded thinking | 2,000–3,000 | 102 | 8 | 7.8 |
| Historical direct thinking | 1,000–2,000 | 309 | 79 | 25.6 |
| Historical direct thinking | 2,000–3,000 | 230 | 20 | 8.7 |
| Historical direct thinking | 3,000–4,096 | 210 | 17 | 8.1 |
Table 11. Within-run answer-availability yields. Unanswered outputs lack an accepted prediction at the lower prefix, including earlier-stopped outputs. New answers become available at the upper prefix. No accepted-prefix losses occur at these boundaries.
A longer saved configuration also has limited late answer yield: 17/204 unanswered traces extending beyond 3,000 tokens supply an accepted answer by 4,096. Its configuration and within-run yields are reported in Appendix § 11.
The extracted RL diagnosis recurs at least twice in 71/75 final thought windows (94.7%; median 10 mentions, IQR 7–13.5). Yet the accepted-answer control also exceeds this threshold in 338/340 cases (99.4%; median 5, IQR 4–7). Using the continuation diagnosis in all failures gives 89/94 (94.7%; median 9.5, IQR 7–12).
This threshold captures routine diagnostic discussion as well as unfinished reasoning (Table 3).
| Group and diagnosis target | Mentions | Assertions | |
| Available; accepted diagnosis | 340 | 338 | 15 |
| Failed; extracted commitment | 75 | 71 | 35 |
| Failed; continuation diagnosis | 94 | 89 | 33 |
Table 3.
Explicit answer selection gives a sharper contrast: repeated assertions occur in 35/75 extractable failures (46.7%), versus 15/340 available first passes (4.4%). With the continuation diagnosis as target, the rate is 33/94 failures (35.1%). Four extracted diagnoses occur before the final window and have zero mentions there; none has exactly one. The extracted-target analysis abstains on the other 19 failures.
Whole-trace recurrence is also higher in the rescue group: median is 0.491–0.503 for rescued RL traces and 0.047–0.055 for unrescued traces (Table 14). This statistic counts duplicate word windows across the full thought, whereas the exact-cycle detector requires adjacent identical token blocks. Figure 10 (Appendix § 14) illustrates explicit reassertion in one unfinished trace.
| Model | Rescued | Correct | Median | |
| SFT | No | No | 62 | 0.046 |
| SFT | No | Yes | 279 | 0.046 |
| SFT | Yes | No | 40 | 0.474 |
| SFT | Yes | Yes | 53 | 0.429 |
| RL | No | No | 55 | 0.055 |
| RL | No | Yes | 285 | 0.047 |
| RL | Yes | No | 40 | 0.491 |
| RL | Yes | Yes | 54 | 0.503 |
Table 14. Whole-thought repeated four-gram fraction by rescue status and final correctness. Continuation text is excluded.

Figure 10. Reassertion in CASE_116. Panel A summarizes earlier evidence comparison; panel B quotes a recurring suffix; panel C gives the separately generated continuation. The suffix repeatedly selects a diagnosis without adding a vignette observation.
A Earlier evidence comparison CASE_116. The trace weighs a reported midline defect against the absence of a covering sac and compares gastroschisis with omphalocele. It explicitly recognizes the tension between these observations before settling on a candidate diagnosis.
B Later recurring suffix in the same thought. “Let’s go with gastroschisis.” “Wait, let me check if there’s any other condition. Maybe gastroschisis? Yes.” “Let’s go with gastroschisis.” “Wait, let me check if there’s any other condition. Maybe omphalocele? No.” “Let’s go with gastroschisis.”
C Separate answer-rescue continuation. Parsed diagnosis: gastroschisis. The saved continuation also repeats answer material.
Reading commitments recovers paired correctness
For the RL fallback, 67 matching diagnoses inherit continuation verdicts and three use unambiguous archived verdicts for the same case. Of the five newly adjudicated diagnoses, two are accepted. Across all 75 candidates, 41 are accepted and 34 rejected.
The extraction-only fallback supplies candidates for 415/434 cases and both members of 199/217 pairs. It yields 129/217 jointly correct pairs (59.4%). Against direct answering, there are 27 gains and 14 losses: +6.0 points (paired-bootstrap 95% interval, +0.5 to +11.5; exact McNemar ). The comparison remains borderline under the exact test on 41 discordant pairs.
The fallback restores 30 of the 40 pairs recovered by continuation. In 29, every rescued member has the same extracted and continued diagnosis; their matching verdicts are reused. The fallback also restores one pair that continuation misses. Figure 3 shows how continuation can preserve a diagnosis, revise it, or lose a scored qualifier.

Figure 3. Diagnosis preservation, revision, and qualifier loss in saved first passes and continuations. Quoted assertions are exact first-pass spans; continuation diagnoses are parsed predictions. Verdicts come from the same-case archived labels.
The examples link readout to clinical specificity. CASE_003 preserves vasa previa, CASE_415 changes the diagnosis from juvenile idiopathic arthritis to osteosarcoma, and CASE_159 drops the coagulopathy qualifier from postpartum hemorrhage. The archived verdicts respectively record preservation, improvement, and loss of an accepted diagnosis.
Frozen-extractor transfer to SFT
For SFT, 58 extracted diagnoses match the continuation, one has an unambiguous archived verdict, and one exactly matches the reference. Of 15 newly adjudicated candidates, five are accepted. Across the 75 extracted diagnoses, 42 are accepted and 33 rejected. The fallback supplies 416/434 cases and both members of 199/217 pairs, yielding 123/217 jointly correct pairs (56.7%). It restores 28 of the 37 continuation-recovered pairs.
Table 4 reports the completed evaluation. SFT has 26 gains and 16 losses against direct answering: +4.6 points (paired-bootstrap interval, −1.4 to +10.6; McNemar ). The frozen rules transfer extraction and
| Checkpoint | Correct pairs / 217 | Robust (%) | Difference | 95% interval | Gains / losses | |
| RL | 129 | 59.4 | +6.0 | [+0.5, +11.5] | 27 / 14 | 0.0596 |
| SFT | 123 | 56.7 | +4.6 | [−1.4, +10.6] | 26 / 16 | 0.1641 |
Table 4. Completed fallback evaluation after adjudicating all extracted diagnoses. Every score uses 217 pairs. Differences and paired-bootstrap 95% intervals are in percentage points, relative to the corresponding direct-answer condition. is the exact McNemar result.
Table 2: Answer availability and correctness over 434 cases and 217 pairs. First-pass and end-to-end readouts share the bounded-thinking first pass; the latter adds conditional continuation. Direct answering is generated separately. An available pair has two accepted predictions.
Table 3: RL recurrence in the final 500 thought tokens. Mentions and assertions count traces with at least two matches. Assertions use frozen selection rules. The extracted-commitment group contains the 75 eligible failures; the continuation-target group retains all 94.
within-trace recovery to SFT, while its comparison with direct answering remains inconclusive.
A second commitment supports an earlier readout
The prefix-only trigger fires in 91 RL cases, including 59 original first-pass failures, and in 72 SFT cases, including 49 failures (Table 5). The selected diagnosis matches the saved final prediction in 84/91 RL triggers and 67/72 SFT triggers. Median trigger positions are 1,266 and 1,381 reconstructed tokens, respectively.
Retaining these prefixes omits 24.1% of RL and 18.5% of SFT first-pass text across all cases. The resulting readout reaches 122/217 jointly correct RL pairs (56.2%) and 114/217 SFT pairs (52.5%), compared with 59.4% and 56.7% when reading the last eligible commitment at the observed end. This quantifies an earlier-readout tradeoff: repeated selection can expose a usable diagnosis sooner, while some later diagnostic changes improve paired correctness.
Discussion
The main result concerns diagnostic content: in most extractable failed first passes, continuation reproduces the last explicitly asserted diagnosis. This explains why a missing structured answer can coexist with a recoverable clinical conclusion. Under the saved labels, reading those assertions alone restores three quarters of the RL pairs recovered by continuation.
Continuation also changes some answers. The trace examples show both a beneficial revision and a loss of diagnostic specificity. Clinical answer recovery therefore involves preserving the selected diagnosis and its consequential qualifiers, alongside satisfying the output schema. This connects the clinical evaluation problem to intermediate-answer analysis and overthinking [28, 25], as well as inference control and structured generation [14, 15, 16, 29, 30, 31].
The near-identical SFT and RL failure rates show that unfinished answers are already present at the supervised stage. The recurrence controls clarify what is informative in a reasoning trace. Repeated mentions of a candidate diagnosis are routine among accepted outputs; repeated explicit selections are more concentrated in the failed group. Together with diminishing late answer yield, this suggests a practical direction for inference design: monitor repeated commitment statements and test whether earlier answer emission preserves diagnostic specificity.
The offline trigger makes this direction concrete: a second matching assertion exposes an earlier readout, with a measurable text–accuracy tradeoff. A live test can now compare direct extraction with forced answer continuation at the same trigger. For evaluation, the useful decomposition has three parts: the diagnosis stated in thought, the answer accepted by the interface, and correctness on both clinical presentations. Reporting each part makes it possible to locate where a continuation changes an outcome. The fallback’s borderline RL comparison and inconclusive SFT comparison with direct answering are secondary to this within-trace diagnosis-preservation finding.
Limitations
This study covers one Gemma 4 31B lineage, one RL run, and one benchmark. SFT and RL share cases and generation priors; extraction rules were developed on the RL failures before being frozen for SFT. Direct-answer and thinking conditions differ in budget and control, and the saved records lack runtime finish reasons. Prefix yields and trigger reductions describe the observed text; they do not measure runtime savings or the answers a fresh continuation would produce. Recurrence comparisons also vary in trace length, target selection, and extraction eligibility.
Correctness scores reuse archived verdicts. Additional extracted diagnoses are scored with the original evaluator, and residual disagreements across scoring snapshots affect small between-checkpoint differences (Appendix H). Bootstrap intervals summarize variation over benchmark pairs. The clinician sample contains 35 selected cases, exposes automatic verdicts, and has low agreement on several axes.
Training-data overlap with MamaBench has not been exhaustively checked. Historical SFT template and loss-masking details remain unreconciled, and the RL run uses a frozen SFT rollout policy with clipped importance correction (Appendix A). These factors define the scope for checkpoint reproduction and broader clinical validation.
Conclusion
Continuation often returns a diagnosis already stated in unfinished clinical reasoning: 67/75 extractable RL failures and 58/75 SFT failures preserve that diagnosis in MamaReason. The key finding is agreement in diagnostic content before and after continuation, measured independently of correctness labels. Reading the earlier commitments also recovers a substantial share of paired correctness under the archived verdicts.
Clinical reasoning evaluations should report diagnostic content, answer availability, and paired correctness together. Logging native stopping information and preserving first-pass text would support this decomposition. The next experiments are matched-budget answer-control comparisons, blinded clinical review of diagnosis and qualifier changes, and live tests of the second-commitment trigger.
| Checkpoint | Triggered cases | Diagnosis matches | Omitted text (%) | Correct pairs / 217 | Robust (%) |
| RL | 91 | 84 / 91 | 24.1 | 122 | 56.2 |
| SFT | 72 | 67 / 72 | 18.5 | 114 | 52.5 |
Table 5. Offline second-commitment readout on all 434 cases per checkpoint. Agreement compares triggered diagnoses with saved final predictions. Omitted text is measured over reconstructed first-pass tokens; scores use 217 pairs and completed diagnostic labels.
Training and Reward Specification
Data preparation
The SFT source contains 28,082 question–answer records. Lowercasing and trimming questions before exact deduplication leaves 22,798 unique questions. Sampling targets 12,000 questions with seed 42, stratified by heuristic language and topic cues. Language cues include Nigerian Pidgin markers and selected orthographic features; topics include antenatal care, preeclampsia, infection prevention, and access to care.
A teacher generates stepwise reasoning and a concise answer for each sampled question in the Nigerian maternal and child-health context, using temperature 0.3 and a 2,048-token cap. The source answer is withheld during generation. The saved output contains 11,591 conversations. Conversion separates thought and visible-answer channels, and structural cleaning removes off-topic records and missing, empty, or short thought. Removing 936 flagged identifiers leaves 10,655 conversations, split 95:5 with a fixed seed into 10,122 training and 533 validation examples. Filtering checks structure and length; semantic answer verification is outside this pipeline.
Checks screen for short or incomplete records, exact duplicate prefixes, and possible diagnosis leakage. Condition cleaning and metadata re-extraction yield 1,627 vignettes across 111 normalized conditions. The RL run uses a 1,000-vignette subset sampled with seed 42 by proportional urgency-by-setting allocation and a largest-remainder rule: 571 emergent, 397 urgent, and 32 routine cases. It retains all 111 conditions and is used without a validation split. Table 6 summarizes the retained preparation stages.
Optimization and reward
We initialize RL from the SFT checkpoint. Both adaptation stages use four-bit NF4 quantization with double quantization and BF16 computation, with LoRA adapters on non-vision attention and feed-forward projections [40, 41]. The saved SFT configuration specifies rank 64, scaling 64, dropout 0.05, two epochs, a 1,024-token sequence cap, and learning rate with cosine decay. RL uses rank 32, scaling 32, the same dropout, one epoch, and learning rate . Prompts request clinical reasoning, differential diagnoses, investigations, management, and explicit triage and diagnosis fields.
For each prompt, the RL implementation samples completions at temperature 0.80 and top-, with prompt and completion caps of 2,048 tokens each. PPO clipping is 0.20 [33] and the KL coefficient against the SFT reference is 0.001. The implementation uses TRL’s fixed-denominator dr_grpo reduction with default group-standardized advantages [43]; the reduction removes sequence-length normalization while retaining group-standardized advantages [36]. Collocated rollouts use a frozen SFT behavior policy, while the adapter policy is updated, with importance-sampling correction capped at 2.0. The behavior policy remains the SFT policy during these adapter updates.
Let , , , and denote diagnosis, critical-finding coverage, urgency, and management scores. The four-component implementation normalizes their relative weights to obtain
Diagnosis scoring canonicalizes aliases and applies substring or token-set matching. A primary-reference match or Jaccard overlap of at least 0.50 receives 1; overlap from 0.33 to below 0.50 receives 0.70; a reference-differential match receives 0.40. Other predictions receive zero. This lexical training score is distinct from the stricter diagnostic evaluator used for MamaBench.
The finding score measures recall of reference findings and critical concepts. A target matches when any punctuation-stripped word longer than three characters occurs in the raw response. With red flags present, recall of at least 0.75 receives 1, recall from 0.40 to below 0.75 receives 0.40, and lower recall receives . Without red flags, the score equals recall; an empty target set receives 1. Urgency scoring assigns 1 for a correct routine/urgent/emergent category, 0.30 for one-level overtriage, and for emergent-to-routine undertriage. Other mismatches or unknown categories receive zero.
Management combines action recall , investigation recall , and setting feasibility :
Action and test coverage use lexical matching. Feasibility uses referral and stabilization markers in primary or rural settings and admission, investigation, monitoring, or intravenous-treatment markers elsewhere. These rules measure lexical coverage of selected clinical features.
Diagnosis and urgency are preferentially extracted from the visible answer, with thought-channel fallback. Finding and management rules inspect raw response text, including inline thought when present. There is no explicit repetition or token-length penalty. Repeating already matched content can therefore leave the clinical score unchanged, although token probabilities, KL regularization, truncation, and rollout correction still affect learning. It makes answer closure a useful target for a future reward ablation.
Tokenizer and Readout Reproducibility
Reconstruction uses a public base tokenizer with special tokens retained and none automatically added. Exact text round trips are checked on all 868 principal bounded-thinking first passes.
The cap-consistent group includes 90 RL failures of exactly 3,000 reconstructed tokens and two of 3,001 tokens. Re-encoding can differ at token boundaries. The two shorter RL failures are CASE_090 (2,152 tokens) and CASE_194 (1,923 tokens); both have an exact trailing cycle. The SFT counterparts are CASE_102 (2,366 tokens), CASE_189 (1,626 tokens), and CASE_238 (1,501 tokens). In each, a repeated 8–32-token block occurs three consecutive times at the suffix. These suffixes match the detector’s pattern definition.
In the historical direct-thinking run, 178/193 failures reconstruct to exactly 4,096 tokens, three to 4,095, and three to 4,097. Eight others reconstruct below 4,095 (799, 1,316, 1,426, 1,614, 2,319, 2,427, 3,144, and 4,000), and one to 4,101. All lack a thought-channel closure. The interval analysis uses these saved lengths and the parser outcomes.
The recurrence control excludes the visible answer from each accepted first pass. Of its 340 outputs, 318 have an observed thought-channel closure; 316/318 of these still satisfy the two-mention threshold. The threshold is saturated in the closed-channel subset as well. All 340 controls have nonempty thought text. The continuation-target comparison retains every failed output and applies the same word-boundary phrase matching without using its correctness verdict.
Token-offset truncation retains the source text up to the last retained token’s character offset. All prefix availability counts rerun the same JSON parser and placeholder gate. For unclosed thought, the same JSON recovery procedure is applied to the remaining text.
Frozen commitment-extraction rules
The following rules specify the extractor used for both checkpoints. Matching is case-insensitive and follows textual order.
Recognize an assertion cue from one of these families:
“let’s” (straight or curly apostrophe), followed by “go with”, “choose”, “answer”, “use”, “stick with”, “settle on”, or “finalize”;
“I’ll” or “I will”, followed by “go with”, “answer”, “choose”, “use”, “state”, or “pick”;
optionally “the”, then “final”, “most likely”, “working”, “primary”, or “definitive”, followed by “diagnosis” and “is”, a colon, or an equals sign;
“my” or “the”, optionally “final”, then “answer” or “prediction” and “is”, a colon, or an equals sign;
a quoted JSON prediction key, colon, and opening double quote.
Capture 2–160 characters after the cue, ending before a newline, period, exclamation mark, question mark, brace, or double quote. Trim whitespace, colons, asterisks, and backticks from the ends, remove an initial “simply”, and cut at a word-boundary occurrence of “because”, “given that”, “since”, or “for the final answer”.
Exclude empty candidates, candidates containing an opening angle bracket or “condition name”, and candidates beginning with “that”, “this”, “it”, “still”, “a more”, “the same”, “the best”, “the most”, “as above”, “as the”, “simply that”, or “simply this”. Also exclude a match ending at the end of saved text or immediately before a question mark.
Select the last remaining candidate by text position. With none, abstain. JSON and prose cues share this ordering.
For scoring and string comparison, lowercase, replace each run outside a–z and 0–9 with a space, and trim the result.
Additional diagnostic adjudication
The earlier analysis left five RL and 15 SFT last-commitment candidates unscored. These 20 case-diagnosis pairs were subsequently evaluated using the original diagnostic prompt and model, temperature 0, and a maximum of 1,024 response tokens. The evaluator receives the reference diagnosis and candidate prediction, without the reasoning trace. Table 8 lists the candidates and references. All 20 requests returned valid verdicts: two RL candidates and five SFT candidates were accepted. The original benchmark predictions and verdicts remain unchanged.
| Checkpoint | Case | Readout | Candidate diagnosis | Reference diagnosis | Correct |
| RL | CASE_006 | L | AFLP | Gall stone | No |
| RL | CASE_123 | L | varicella | Chicken pox | Yes |
| RL | CASE_258 | L | GERD | Dyspepsia in pregnancy | No |
| RL | CASE_342 | L | genital tract trauma causing postpartum hemorrhage | Genital Tract Trauma | Yes |
| RL | CASE_382 | L | SMA Type III | Muscular Dystrophy | No |
| SFT | CASE_054 | L | endometrial cancer or endometrial hyperplasia | Endometrial polyps | No |
| SFT | CASE_120 | L | Neurofibromatosis Type 2 (NF2) | Neurofibromatosis 2 | Yes |
| SFT | CASE_123 | L | varicella | Chicken pox | Yes |
| SFT | CASE_134 | L | GBS | Guillain Barre syndrome | Yes |
| SFT | CASE_135 | L | Global Developmental Delay as it’s a widely recognized term in pediatric developmental screening | Mental retardation | Yes |
| SFT | CASE_169 | L | indeed related to indoor air pollution | Laryngitis | No |
| SFT | CASE_194 | L | often the one that explains all the symptoms | Germ cell tumor in the pineal region | No |
| SFT | CASE_248 | L | physiological urinary frequency | Normal pregnancy | No |
| SFT | CASE_258 | L | preeclampsia | Dyspepsia in pregnancy | No |
| SFT | CASE_267 | L | Bacterial meningitis (given the constellation of symptoms in an infant) | Febrile seizures | No |
| SFT | CASE_295 | L | the cause of the bleeding | PPH from Uterine atony | No |
| SFT | CASE_325 | L | an innocent murmur, specifically Still’s murmur | Innocent murmur | Yes |
| SFT | CASE_352 | L | a benign ovarian cyst | Benign Ovarian Tumour (Serous Cystadenoma) | No |
| SFT | CASE_382 | L | SMA Type III | Muscular Dystrophy | No |
| SFT | CASE_421 | L | something like | Cervical Polyp | No |
| RL | CASE_303 | E | a normal intrauterine pregnancy | Normal pregnancy | Yes |
| RL | CASE_311 | E | an innocent murmur | Physiologic murmur | Yes |
| SFT | CASE_365 | E | Hirschsprung disease as it’s a classic board-exam presentation for this scenario | Hirschsprung’s disease | Yes |
Table 8. Added diagnostic evaluations. L denotes a last-commitment candidate; E denotes an early-trigger candidate. Verdicts use the same automatic diagnostic criterion as the main evaluation.
For archival comparison, the pre-adjudication lower–upper ranges were 129–131 RL pairs (59.4–60.4%) and 119–129 SFT pairs (54.8–59.4%). The completed results are 129 and 123 pairs. The former conservative comparisons were +6.0 points for RL () and +2.8 for SFT (); current SFT results are +4.6 points (). The historical ranges reflected missing candidate verdicts, rather than sampling uncertainty.
The early-trigger readout introduced three further diagnosis strings without an unambiguous saved verdict. All three were scored by the same procedure and accepted. The early-readout scores therefore contain no unresolved diagnostic labels. An event-based implementation was checked against every-token prefix replay on eight representative triggered and untriggered cases across both checkpoints, with matching trigger positions and diagnoses.
Completed Clinician Review
Review procedure
Two clinicians complete the same eight-axis workbook for 35 selected RL cases from 25 pairs, yielding 70 case reviews. Axes cover agreement with the automatic verdict, overthinking, underthinking, missed critical information, misweighted findings, hallucinated findings, thought–answer consistency, and five-point clinical acceptability. The automatic verdict remains visible in the workbook.
We join reviews by case identifier and retain each clinician’s ratings. Binary axes use raw agreement and Cohen’s [44]; acceptability uses rating distributions, exact agreement, and linearly weighted [45]. The following tables describe the selected sample.
Review sample and agreement
Both completed workbooks contain all eight ratings for the same 35 case identifiers. Predictions and thought text match the principal RL records. These completed files supersede the earlier partial reviews.
The selected cases comprise five robust controls, 12 cases from pairs, nine bias-trap cases, and nine cases selected as rescued but still incorrect. These mutually exclusive strata define the failure-enriched sample. Thirteen cases carry a rescue flag across all strata. The automatic evaluator marks ten cases correct and 25 incorrect.
Table 9 reports all completed binary review axes. The overthinking rubric requests at least three redundant differential passes without new information, with a supporting quotation. Clinicians flag it jointly in seven cases, in either review in 20, and neither review in 15. Their two-by-two counts are seven both Yes, eight clinician-1-only Yes, five clinician-2-only Yes, and 15 both No. Raw agreement is therefore 22/35, and [44]. Underthinking and overthinking can co-occur in a case; the rubric does not make them exclusive categories.
Table 10 gives the acceptability distributions. Ratings agree exactly on 15/35 cases and are within one point on 29/35. Linearly weighted is 0.203 [45]. Binary kappas are reported alongside raw counts because the marginal flag frequencies differ and several findings are rare.
Clinician 1’s file has a blank pair identifier for CASE_018 and a blank reference diagnosis for CASE_029. We join by case identifier and obtain pair membership from MamaBench. All original rating cells are retained; the other shared review fields agree between files.
Direction of disagreement with automatic verdicts
Clinician 1 disputes nine of 35 automatic verdicts; all nine are negative verdicts, with no positive verdict disputed. Clinician 2 disputes two negative verdicts and no positive verdicts. The two clinicians jointly dispute CASE_243 and CASE_244, the members of PAIR_122. These disagreements concern negative automatic verdicts in the reviewed sample. Figure 4 summarizes the clinicians’ behavioral flags, retaining separate counts for each reviewer.

Figure 4. Clinician flags on the same 35 selected cases. Inconsistency is the inverse of the rubric’s thought–answer consistency item; the other bars count affirmative flags.
Of the 25 automatic negative verdicts in the sample, clinician 1 disputes nine and clinician 2 disputes two; both dispute CASE_243 and CASE_244. CASE_029, whose reference field is blank in clinician 1’s file, is among clinician 1’s nine disagreements. The primary benchmark scores retain the archived labels.
Secondary Protocol Comparisons
Historical direct-thinking failures
The earlier obstetric-framed direct-thinking run contains 193 extraction failures among 434 cases (44.5%): 134 JSON parsing failures and 59 blank/template predictions. It emits accepted predictions for both members of only 70/217 pairs and is jointly correct on 37, giving 17.1% robust accuracy on the full benchmark but 52.9% conditional accuracy on those 70 available pairs. Its companion direct-answer run scores 50.7% with all cases available.
An earlier bounded-thinking run has 187 observed continuations (43.1%) and 51.6% robust accuracy. The principal run uses 94 continuations (21.7%) and reaches 63.6%. These snapshots record separate prompt and stopping configurations; Table 11 gives the within-run prefix yields.
Paired diagnostic performance
Table 12 reports all 217 pairs for each of the four principal conditions. Each end-to-end condition has 434 case records and no remaining error state after its configured emission procedure. The RL checkpoint under bounded thinking reaches 138 jointly correct pairs, corresponding to 63.6% robust accuracy. Base accuracy is 82.9%, whereas counterfactual accuracy is 73.3%. Thus, performance on isolated base cases substantially exceeds performance requiring correctness on both presentations. The 42 base-correct, counterfactual-incorrect pairs yield a BTR of 23.3%.
| Rubric item | Clinician 1 Yes | Clinician 2 Yes | Both Yes | Agree / 35 | |
| Agreement with automatic verdict | 26 | 33 | 26 | 28 | 0.298 |
| Overthinking | 15 | 12 | 7 | 22 | 0.222 |
| Underthinking | 13 | 3 | 1 | 21 | −0.017 |
| Missed critical information | 3 | 1 | 0 | 31 | −0.045 |
| Misweighted finding | 4 | 1 | 0 | 30 | −0.048 |
| Hallucinated finding | 2 | 0 | 0 | 33 | 0.000 |
| Thought–answer consistency | 26 | 34 | 25 | 25 | −0.054 |
Table 9. All completed binary review axes. “Yes” follows the original rubric; for thought–answer consistency, Yes affirms consistency. Every count has denominator 35. Agreement compares the clinicians’ ratings, not agreement between a clinician and the automatic evaluator. is computed from the corresponding two-by-two table.
| Score / summary | Clinician 1 | Clinician 2 |
| 1 | 0 | 0 |
| 2 | 6 | 0 |
| 3 | 8 | 6 |
| 4 | 16 | 20 |
| 5 | 5 | 9 |
| Mean | 3.57 | 4.09 |
| Median | 4 | 4 |
Table 10. Clinical acceptability ratings, preserving the ordinal five-point scale. Every clinician rates all 35 cases.
The SFT checkpoint exhibits the same descriptive protocol ordering: robust accuracy is 52.1% for direct answering and 60.8% for bounded thinking. The corresponding RL values are 53.5% and 63.6%. Figure 5 shows both the base–robust gap and the underlying outcome counts. The bounded-thinking checkpoints have identical counterfactual accuracy, despite different base and robust accuracy. A single accuracy number would conceal these differences in where errors occur.

Figure 5. Diagnostic accuracy and pair outcomes across protocols. Left: base and robust accuracy. Right: the full , , , and distributions. BT+R denotes bounded thinking with conditional rescue.
The bounded-thinking RL–SFT difference is six jointly correct pairs, or 2.8 points. Appendix § 15 records the scoring-snapshot discrepancies relevant to this comparison.
Where the protocol changes pair outcomes
For the RL checkpoint, bounded thinking increases robust accuracy by 10.1 percentage points over direct answering, with a paired-bootstrap 95% interval of 4.6–15.7 points. Thirty pairs gain joint correctness and eight lose it (exact McNemar ). Figure 6 shows that these gains come from several initial states: 14 base-only correct pairs, 13 counterfactual-only correct pairs, and three pairs for which both members were originally wrong. Of the 116 direct-answer pairs, 108 remain .

Figure 6. Paired transitions for the RL checkpoint. Rows are direct-answer outcomes and columns are bounded-thinking outcomes for the same 217 pairs. Joint correctness is gained in 30 pairs and lost in eight. Cell counts describe a protocol change, including a larger budget and conditional continuation.
| Checkpoint | Protocol | Base ↑ | CF ↑ | Robust ↑ | BTR ↓ | ||||
| SFT | DA | 113 | 48 | 33 | 23 | 74.2 | 67.3 | 52.1 | 29.8 |
| SFT | BT+R | 132 | 41 | 27 | 17 | 79.7 | 73.3 | 60.8 | 23.7 |
| RL | DA | 116 | 48 | 31 | 22 | 75.6 | 67.7 | 53.5 | 29.3 |
| RL | BT+R | 138 | 42 | 21 | 16 | 82.9 | 73.3 | 63.6 | 23.3 |
Table 12. MamaReason on MamaBench: all conditions cover 217 pairs / 434 cases. DA denotes direct answering; BT+R denotes bounded thinking with conditional rescue. Values are percentages; pair counts precede the metrics. Arrows indicate preferred direction. SFT rows retain archived labels except the documented CASE_085 correction; scoring-snapshot details are given in Appendix H.
| Published system | Base | Robust | BTR |
| 1. GLM-5 | 54.8 | 27.2 | 50.4 |
| 2. Kimi K2.5 | 76.0 | 45.6 | 40.0 |
| 3. GPT-4o + RAG | 77.4 | 46.5 | 39.9 |
| 4. Nemotron-120B | 72.4 | 47.0 | 35.0 |
| 5. GPT-4o + EA-RAG | 77.0 | 49.3 | 35.9 |
| 6. GPT-5.4 | 77.4 | 57.6 | 25.6 |
| 7. Claude Sonnet 4.6 | 82.0 | 60.8 | 25.8 |
| 8. Claude Sonnet 4.6 + EA-RAG | 81.6 | 65.0 | 20.3 |
| MamaReason RL, BT+R (this study) | 82.9 | 63.6 | 23.3 |
Table 13. Published MamaBench reference scores, transcribed from its reported model/configuration comparison [11]. All use 217 pairs. These are external reported scores, not controlled reruns. EA-RAG denotes Evidence-Anchored RAG.
Bounded thinking leaves 42 , 21 , and 16 pairs. The transition matrix shows gains in either presentation as well as losses from previously correct pairs.
This contrast compares the complete protocols in Table 1: direct answering allows 1,024 output tokens; bounded thinking allows 3,000 first-pass tokens and a conditional 512-token continuation.
Published MamaBench Reference Scores
Position relative to published MamaBench systems
Table 13 reproduces the eight model/configuration scores reported in the MamaBench paper [11]. The published range is 27.2–65.0% robust accuracy and 20.3–50.4% BTR. MamaReason’s RL bounded-thinking run occupies the upper part of this reported range: 63.6% robust accuracy and 23.3% BTR. Figure 7 places these values alongside the published reference points.

Figure 7. Published MamaBench systems (gray, indexed as in Table 13) and MamaReason RL bounded thinking (teal, M). The upper-left direction favors higher robust accuracy and lower BTR. Systems use different models and inference configurations.
The published reference points span different model scales, prompts, retrieval settings, and inference budgets. Their shared pattern is the gap between base-only correctness and success on both presentations.
Conditional Rescue Outcomes
Answer emission and conditional rescue
The saved RL bounded-thinking records mark 94 of 434 cases as successfully rescued; 54 of these are diagnostically rescued (57.4%). Among the 340 cases without that flag, 285 are correct (83.8%). At pair level, 82 pairs contain at least one successfully rescued case, and 40 are jointly correct (48.8%). The other 135 pairs contain 98 jointly correct outcomes (72.6%). Figure 8 shows these conditional quantities.

Figure 8. RL correctness by rescue status. Case denominators are 94 and 340; pair denominators are 82 and 135. Rescued pairs contain at least one rescued member. Both members must be correct for pair success.
The rescue flags coincide with the continuation markers and replayed first-pass failures. § 4.2 reports their readouts before continuation.
Trace Examples and Stratified Repetition
Reasoning trace analysis
Records contain raw generation, parsed prediction, visible rationale, verdict, error state, and rescue status. We check excerpt figures against the original saved text and separate first-pass thought from synthetic channel closure and continuation.
We distinguish three observable behaviors. Evidence-sensitive deliberation compares alternatives using case details and develops an evidential argument. Hypothesis revision changes an interpretation after reconsidering evidence; beneficial self-correction additionally requires an appropriate emitted answer. Rehearsal revisits substantially the same differential or conclusion without a substantive evidential update. These behaviors can coexist. Figure 10 presents a clear repetitive suffix, whereas Figure 9 illustrates revision mixed with later oscillation. These definitions concern visible text [26].

Figure 9. PAIR_183 and selected CASE_366 trace excerpts. Narratives are abbreviated; thought excerpts are noncontiguous. The counterfactual reference is meconium plug syndrome, whereas the rescue emits Hirschsprung disease.
A Base presentation CASE_365. Shared setting: term newborn with delayed meconium passage. Three days old; no meconium since birth; swollen, tense abdomen; feeding poorly. Benchmark reference: Hirschsprung’s disease.
B Counterfactual presentation CASE_366. Shared setting: term newborn with delayed meconium passage. Two days old; stool after mild interventions; soft, mildly distended abdomen; feeding well. Benchmark reference: meconium plug syndrome.
C Evidence-sensitive revision in the counterfactual thought channel. “Let’s reconsider: could it be meconium plug syndrome?” “This fits the ‘stool started after some mild interventions’ and ‘feeding well and generally active’ description better.”
D Later oscillation in the same first pass. “Let’s try Hirschsprung disease.” “Let’s try meconium plug syndrome.” The trace repeatedly reopens the alternatives before first-pass generation ends.
E Parsed answer after rescue. Hirschsprung disease. The continuation emits a diagnosis different from the benchmark reference for CASE_366.
For quantitative description, we lowercase first-pass thought, retain alphabetic words with internal apostrophes, and form the multiset of consecutive four-word windows. The repeated four-gram fraction is
Each bounded-thinking arm contains 434 nonempty traces. Table 14 stratifies by rescue and final correctness. For RL, rescued correct and incorrect responses have medians of 0.503 and 0.491; unrescued responses have medians of 0.047 and 0.055. Whole-trace varies with length as well as recurrence, so the suffix assertion analysis supplies a complementary view.
Evaluation Records and Verdict Consistency
The principal analysis uses the saved clarified-prompt SFT and RL direct-answer and bounded-thinking records, each with all 434 cases. The original exports are preserved. A derived analysis copy corrects the SFT direct-answer verdict for CASE_085 from incorrect to correct, matching the archived final decision for the identical prediction, “Acute otitis media with perforation.” This changes one pair from to : SFT direct answering has 113 rather than 112 jointly correct pairs. No other verdict is changed in the reported primary analysis.
Six identical-prediction cases differ in verdict between SFT and RL direct-answer snapshots, and five between the bounded-thinking snapshots. Within RL, CASE_407 is the sole identical-prediction disagreement between direct answering and bounded thinking; within SFT there are none. A case-label change can alter robust accuracy by at most 100/217 = 0.46 points. The first-pass-versus-continuation replay uses one set of bounded-thinking labels throughout.
We recompute pair metrics from case records and check odd/even base–counterfactual membership against MamaBench. Replay reproduces 94 RL and 93 SFT first-pass failures; every marked continuation is accepted. Quantitative readouts use all eligible records, while trace illustrations use selected, verified excerpts.
Replay also reproduces the historical direct-thinking run’s 193 error states. Published scores retain the source paper’s system labels and remain separate from the local paired resampling. Appendix A gives the training settings.
References
- [1]Di Jin et al. (2021). What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences, 11(14):6421.
- [2]Qiao Jin et al. (2019). PubMedQA: A Dataset for Biomedical Research Question Answering. EMNLP-IJCNLP, 2567–2577.arxiv.org/abs/1909.06146
- [3]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan SankarSubbu (2022). MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. CHIL, PMLR 174:248–260.arxiv.org/abs/2203.14371
- [4]Karan Singhal et al. (2023). Large language models encode clinical knowledge. Nature, 620:172–180.DOI
- [5]Rahul K. Arora et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775.arxiv.org/abs/2505.08775
- [6]Suhana Bedi et al. (2025). MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks. arXiv:2505.23802.arxiv.org/abs/2505.23802
- [7]Daniel McDuff et al. (2025). Towards accurate differential diagnosis with large language models. Nature, 642:451–457.DOI
- [8]Tao Tu et al. (2025). Towards conversational diagnostic artificial intelligence. Nature, 642:442–450.DOI
- [9]Matt Gardner et al. (2020). Evaluating Models’ Local Decision Boundaries via Contrast Sets. Findings of EMNLP, 1307–1323.arxiv.org/abs/2004.02709
- [10]Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL, 4902–4912.arxiv.org/abs/2005.04118
- [11]Thanni Adewuyi et al. (2026). MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation. arXiv:2607.14385.arxiv.org/abs/2607.14385
- [12]Rahul Thapa et al. (2026). Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs. EACL, 2450–2483.
- [13]Gemma Team (2026). Gemma 4 Technical Report. arXiv:2607.02770.
- [14]Niklas Münninghoff et al. (2025). s1: Simple test-time scaling. EMNLP, 20275–20321.
- [15]Tingxu Han et al. (2025). Token-Budget-Aware LLM Reasoning. Findings of ACL, 24842–24855.arxiv.org/abs/2412.18547
- [16]Hao Wen et al. (2025). BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens. arXiv:2508.17196.arxiv.org/abs/2508.17196
- [17]Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton (2020). Learning the Difference that Makes a Difference with Counterfactually-Augmented Data. ICLR.
- [18]Yizhong Wang et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL, 13484–13508.arxiv.org/abs/2212.10560
- [19]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman (2022). STaR: Bootstrapping Reasoning With Reasoning. NeurIPS, 35.arxiv.org/abs/2203.14465
- [20]Junying Chen et al. (2025). Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems. Findings of ACL, 14552–14573.DOI
- [21]Juncheng Wu et al. (2025). MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs. arXiv:2504.00993.arxiv.org/abs/2504.00993
- [22]Jason Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS, 35.arxiv.org/abs/2201.11903
- [23]Xuezhi Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR.arxiv.org/abs/2203.11171
- [24]Jie Huang et al. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. ICLR.arxiv.org/abs/2310.01798
- [25]Xingyu Chen et al. (2025). Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning Models. ICML, PMLR 267:9487–9499.
- [26]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS, 36.arxiv.org/abs/2305.04388
- [27]Yanda Chen et al. (2025). Reasoning Models Don’t Always Say What They Think. arXiv:2505.05410.arxiv.org/abs/2505.05410
- [28]Tamera Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702.arxiv.org/abs/2307.13702
- [29]Charlie Snell, Jaehon Lee, Kelvin Xu, and Alvır Kumar (2025). Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. ICLR.arxiv.org/abs/2408.03314
- [30]Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau (2021). PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. EMNLP, 9895–9901.
- [31]Brandon T. Willard and Rémi Louf (2023). Efficient Guided Generation for Large Language Models. arXiv:2307.09702.arxiv.org/abs/2307.09702
- [32]Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS, 35.arxiv.org/abs/2203.02155
- [33]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.arxiv.org/abs/1707.06347
- [34]Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.arxiv.org/abs/2402.03300
- [35]Hunter Lightman et al. (2024). Let’s Verify Step by Step. ICLR.arxiv.org/abs/2305.20050
- [36]Zichen Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. COLM.arxiv.org/abs/2503.20783
- [37]Qiying Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.arxiv.org/abs/2503.14476
- [38]Lianmin Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 36.arxiv.org/abs/2306.05685
- [39]Jack Gallifant et al. (2025). The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine, 31:60–69.
- [40]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 36:10088–10115.arxiv.org/abs/2305.14314
- [41]Edward J. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.arxiv.org/abs/2106.09685
- [42]Bradley Efron (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1):1–26.
- [43]Hugging Face. TRL GRPO Trainer, version 1.9.2. Versioned implementation documentation.
- [44]Jacob Cohen (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46.DOI
- [45]Jacob Cohen (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220.pubmed.ncbi.nlm.nih.gov/19673146