Integrate the Reward, Don't Sample It: Window Likelihood for Interruptions and Backchannels in Full-Duplex Speech Models

Abstract

When a user talks over a full-duplex spoken dialogue model, the model must decide whether the user is interrupting (stop) or only backchannelling (carry on). Post-training can make the model better at telling the two apart, or merely change how readily it stops. The whole-rollout GRPO runs we examine never lower both error rates: any gain in discrimination is offset by a threshold shift, and two released checkpoints discriminate no better. We trace this partly to how the reward is used. The probability that the model stops inside a short window is a known function of a few stop probabilities that the policy itself emits, yet GRPO estimates it by sampling. A window on which all rollouts agree then gives that update no gradient, even when all fail, and whole-rollout credit reaches decisions the reward never judges. Window likelihood (WL) instead samples a continuation and integrates over its stop decisions: the probability of the correct outcome is computed in closed form and its logarithm maximised. On HumDial-FDBench, across four open models with different stop interfaces, WL gives the best F1 and lowers both error rates beyond their confidence intervals, which no GRPO run does, without adding talking into silence.