betaFeedback

WorksComputer Science

Grounding Language Models in Speech Does Not Fix "AI Flavor": Calibrated LLM Judges, Goodhart Traps, and a Preference-Tuning Fix

Abstract

Chinese outputs of large language models carry a perceptible "AI flavor": formulaic sentence frames, stacked written-register connectives, repetition loops, and layout leakage. A popular hypothesis blames text-only training for the missing inner-speech layer of human writing; injecting speech or prosody signals should therefore reduce it. We test it with controlled interventions on a 0.6B model; all three fail: (i) phoneme-interleaved continued pretraining yields no spokenness gain under text-matched controls while raising repetition (+0.13 [+0.03, +0.23]); (ii) prosody-plan conditional generation reaches an adherence of only 0.257 against a pre-registered 0.50 gate; (iii) a rule-based reward built on TTS prosody-feature distance is imperceptible in double-blind evaluation by both humans (8/16) and a calibrated LLM judge (9/16). A root cause is evaluation: single-rater blind evaluation at n~20 only detects extreme (>=85%) preferences. We therefore build a calibrated LLM-judge protocol: the judge passes catch trials on known-answer pairs (16/16; single-text 22/24, same-topic pairs 24/24, versus 0.75 for the human rater), then annotates with two independent judges under A/B swapping, counting disagreement as a tie. Using the judge's agree labels directly as DPO pairs (no reward model), we obtain the first positive result: held-out blind win rate 0.833 [0.699, 0.972], repetition 0.114 to 0.000, unchanged MMLU. We further identify a Goodhart channel: the judge's length preference transduces into output-length collapse via pairwise DPO (median 75 chars vs a 300-char prompt); an SFT-anchor repairs it (w is a style knob: w=0.1 preserves gains, w=0.3 does not). Scaling preference data from 89 to 240 pairs yields no additional judge-visible gain. We release RA-Eval v0.1: a three-layer Chinese AI-flavor evaluation framework with 382 judge-annotated preference pairs.

data from 89 to 240 pairs yields no additional judge-visible gain. We release RA-Eval v0.1: a three-layer Chinese AI-flavor evaluation framework (validated feature library, human blind-eval protocol, calibratable judge pipeline) together with 382 judge-annotated preference pairs.

Introduction

“AI flavor” is a real, measurable phenomenon

Chinese readers hold stable intuitions about AI-generated text: tidy parallel structures, “not…but rather” frames, “in summary” closings, and paragraph layouts full of blank lines. Prior work of ours (the M0 baseline) operationalizes these as objective features and validates them: across 24 model outputs versus 86 human college-entrance-examination (gaokao) essays, the “不仅……更” (lit. “not only…but even more”) frame density is 3.24 vs 0.12 per 1,000 words (coverage 100% vs 10%), written-register connectives 7.90 vs 3.21 per 1,000 characters, and inter-paragraph blank lines 9.4 vs ≈0\approx0 per 1,000 characters (d=1.05d=1.05). These features are directionally consistent across 0.6B/1.7B/4B scales.

The speech-grounding hypothesis

A popular explanation: human writing passes through a subvocal/inner-speech layer, so human text naturally carries prosody; LLMs trained purely on text lack this layer, producing AI flavor. Injecting speech, phoneme, or prosody signals into training should help. This paper tests the hypothesis with controlled interventions—the answer is no at the 0.6B text-only scale, across three routes (§ 4). The negative results are not empty: they move the question from “should we add speech signals” to “can evaluation perceive the difference,” which motivates the methodological core of this paper.

Three-layer evaluation and calibrated judges

Evaluating AI-flavor interventions faces a dilemma: automatic features can detach from perception, and human blind evaluation is power-limited (§ 3) and fatiguing. We propose a three-layer framework, RA-Eval: an objective feature library, a human blind-eval protocol, and a calibrated LLM-judge pipeline—the judge must prove its sensitivity on known-answer calibration sets before annotating, under a fixed protocol (pairwise blind judgment, A/B swap double-pass, two independent judges, disagreement counted as a tie). The same framework uses judges both as evaluators and as preference annotators, making the evaluation signal and the training signal commensurate.

Contributions

  1. Three preregistered negative results with mechanistic diagnoses: phoneme-interleaved CPT, prosody-plan conditioning, and rule-based prosody rewards (§ 4);

  1. Evaluation methodology: a power analysis of single-rater blind evaluation; a calibrated judge protocol (sensitivity gate →\to double-pass annotation →\to disagreement handling), with judge sensitivity of 91.7–100%, substantially above the human rater’s 0.75;

  1. The first positive training result: judge agree labels wired directly into DPO (held-out blind win rate 0.833 [0.699, 0.972]), plus a controlled characterization of the Goodhart channel by which judge length preference transduces into length collapse, and an SFT-anchor repair with a two-point weight comparison;

  1. RA-Eval v0.1: the feature library, calibration sets, 382 judge-annotated preference pairs, and all preregistration documents.

Related Work

Evaluation and reduction of AI flavor. “AI flavor” became an explicit research term in 2026: CraftAlign [4] predicts 304 writing features with a 9B estimator, scores human-vs-AI patterns with an energy model, and converts feature shifts into inference-time revision guidance for an external editor, in the English-story domain. Recent HCI work studies users’ desire to control AI-flavor intensity in co-writing. Shortcut-Controlled Evaluation [5] shows that most AIGT detectors rely on stylistic “shortcut” features and degrade sharply under a controlled protocol; perplexity-ratio detectors such as Binoculars [15] exploit the same distributional signals—supporting our claim that automatic signals are not perception. Relative to these, this work operates in Chinese, intervenes through training (preference tuning) rather than inference-time rewriting, is fully open at 0.6B scale, and contributes the calibration and Goodhart-transduction analyses.

Prosody and language models. Plan-first speech generation systems established prosody intermediate representations on the speech side. In TTS, 2025–2026 saw a shift from verifiable prosody rewards to preference optimization [9], after automatic prosody rewards collapsed prosody into monotonicity; our three negative results in the text domain (P1/P3) provide the symmetric evidence. Interleaved Speech LMs [10] find an “implicit transcription phase” inside speech–text models, corroborating our P0 probe finding that phonology is already linearly decodable in hidden states—what is missing is not the representation but the reason to use it.

Chinese AI-text detection and humanization. Detection is active (LLM-Detector [11], a bilingual Chinese–English detection benchmark, LoRA-based robust detectors [12]); community humanizer-zh practices exist without academic evaluation. Training-side AI-flavor intervention in Chinese was, to our knowledge at submission (2026-11), absent from arXiv; PersonaSteer [13] suppresses AI-flavored replies (overlong, overly formal) in personalized dialogue and is the closest work.

LLM-as-judge. Position bias, verbosity bias, self-preference, and agreeableness bias are systematically documented [8]; the LLM-as-a-Judge survey [6] explicitly warns that biases “risk being amplified and propagated” when judgments feed training; 2026 mitigations include RL-trained robust judges [7]. We do not propose a new judge; we contribute (i) a practical protocol (catch-trial sensitivity gate, double independent pass, disagreement-as-tie), (ii) an end-to-end controlled experiment of bias transduction (§ 5.3), and (iii) a knob-based repair.

Preference optimization. Preference tuning descends from RLHF instruction tuning [16] and DPO [1]. DPO’s length bias is well studied: LD-DPO [2] attributes it to length-agnostic token-level log-likelihood increments, and Length-Controlled DPO [3] shows length biases in preference data propagate into models. Our difference is the source of bias: not the DPO objective but judge annotation bias (68% of agree pairs have a shorter chosen), which can be measured at the data-construction layer and anchored at the training layer. Reference-free, length-normalized objectives such as SimPO [14] target the same length effects; we give a two-point comparison (w=0.1/0.3w=0.1/0.3) of the chosen-CE (SFT) anchor in the DPO objective, used for the non-standard purpose of cancelling annotator bias.

RA-Eval: Three-Layer Evaluation

Layer 1: validated objective features

Features are computed per text, normalized per 1,000 words or characters, and carry effect sizes from 24 model vs 86 human essays:

  • AI frame constructions (中文“不仅……更” “不是……而是”; ai_frame);

  • register markers: written connectives (written_marker) vs human connectives (但是/不过, “but/however”; human_connective), sentence-final particles (语气词; particle), second-person address (address);

  • lexical diversity (distinct_1/2, token_entropy) and adjacent-sentence isomorphism (adjacent_iso);

  • layout leakage (blank lines, blank_line; markdown leak) and English leakage (en_leak);

  • repetition rate rep (10-gram).

Audio: edge-TTS rendering + librosa prosody (pause ratio, speech rate, F0 variance), compared to human references via standardized ∣z∣|z| distances. The feature library is Chinese-specific and validated on Chinese text; the three-layer structure and the judge protocol are language-agnostic.

Layer 2: the power wall of human blind evaluation

For pairwise blind evaluation at α=0.05\alpha=0.05 (two-sided) and 80% power, detecting a true preference pp requires roughly: p=0.70p=0.70 →60 pairs; p=0.65p=0.65 →100 pairs; p=0.60p=0.60 →300 pairs. Our two human rounds (n=24/16n=24/16) could only detect ≥85%\ge85\% extremes—“not detected” does not mean “not there”.

Layer 3: the calibrated judge protocol

  1. Qualification (catch trials): blind-judge known-answer pairs; sensitivity ≥0.75\ge0.75 required (actual: 16/16, production judge of isolated sub-agents);

  2. Calibration report: single-text detection 22/24 [0.74, 0.98]; same-topic pairs 24/24 (clean subset 7/7) – both from the in-session pilot judge; the human rater’s best on the same material is 0.75;

  3. Annotation: each pair is judged by two isolated judges under swapped A/B order; disagreement is scored 0.5 (tie) and stratified in analysis;

  4. Transduction check: after any training, verify that annotation bias did not enter the model (see the length collapse in § 5.3).

All tools (blind batch generation, label joining, judge scoring) ship with RA-Eval.

Experiment I: Three Negative Results

We report three speech-grounding negatives (§ 4.1–§ 4.3) and a fourth, non-speech negative (§ 4.4) that motivated the positive experiment of § 5.

Phoneme-interleaved continued pretraining (Tier1c)

Two arms train on exactly the same 50M text tokens; the treatment arm interleaves 26% pinyin tokens (1,305 new vocabulary items) into the text stream, the control arm is pure-text CPT. Spokenness of the treatment arm drops relative to control (−4.86 [−10.50, +0.31]), repetition rises (+0.13 [+0.03, +0.23]), length runs away (+46.7 chars [+21.5, +72.3]); MMLU is unchanged, ruling out “learned less”. Phoneme interleaving brings no benefit and real costs at 0.6B. Two earlier pilots (v1 without text-matched controls; v1b with an equal-steps confound) agree directionally.

Prosody-plan conditional generation (P1a)

Four-arm pretraining (noplan / true / shuffle / lenonly), with plan tokens (length buckets + pause strength) loss-masked during training. At inference, plans fed to the chat model yield an adherence of 0.257 (gate 0.50); zero-shot plan input yields empty output in 32/32; the true-plan arm loses diversity relative to noplan (distinct2_2 −0.136) and is worse in audio prosody (+0.255 [−0.183, +0.715]). Diagnosis: plan tokens appear only at document starts during training but after the chat assistant header at inference, triggering document continuation; and post-hoc plans are redundant for the token-level loss, so the model learns to ignore them. Without an architectural or semantic mechanism forcing plan use, conditioning signals are ignored.

Rule-based prosody reward, double-blinded (P3)

A reward built on ∣z∣|z|-distance of TTS prosody features to human references “works” on automatic metrics (rerank zz=1.711 vs 2.284). But double-blind evaluation on length-matched candidate pairs gives: human raters pick the low-zz side 8/16; the calibrated judge 9/16—both at chance, while the same judge detects human-vs-model text at 91.7–100%. The gap this reward optimizes does not exist perceptually. Automatic metrics won; humans and judges heard nothing—the inverse of the power-wall warning: not insufficient power, but the metric was constructed on the wrong object.

Feature-weighted reward: the calibrated middle route (negative)

Before moving to direct preference tuning, we tried a feature route: a linear reward over 18 objective features as direction-aligned deviations from human reference distributions, with weights calibrated by the judge via Bradley–Terry regression on 182 pairs (LOPO-CV AUC 0.661, above the pre-set gate). The frozen validation on 8 held-out prompts failed: the separation pair (argmax vs argmin) reached only 0.625 [0.39, 0.84] blind win rate, and the judge could not distinguish it from the old prosody reward (0.500). Judge reason analysis shows its criteria include dimensions (fluency/coherence, factual correctness) with no locally computable proxy—the ceiling of the feature route. This negative has two uses: it motivates the direct route (§ 5), and it provides counterevidence for feature-scoring approaches such as CraftAlign [4]: features can discriminate human vs AI, but feature distance is not the perceptual target—optimizing it optimizes a blind spot.

Experiment II: Judge-Driven RLAIF (Positive)

Data: judge agree preference pairs

One policy (the P1a noplan CPT arm of Qwen3-0.6B, the base for all DPO iterations) samples 8 candidates per prompt over 32 prompts; within-prompt pairs (constraints: ∣Δchars∣≤100|\Delta\mathrm{chars}| \le100, not both degenerate) total 382 and are annotated double-pass by the calibrated judge (position consistency 79.5–83.3%). Agree pairs (both passes consistent; 307/382 across two annotation rounds, 116+191) become preference pairs: 240 for training (24 prompts) and 67 held out (8 prompts, 27+40, never used in any fitting). No reward model—judge labels are wired directly as chosen/rejected into DPO.

First positive result (iter-1)

89 training pairs, LoRA r=32r=32 on all projections, lr 10−510^{-5}, β=0.1\beta=0.1, grad-accum 1, 300 steps (3.4 epochs), 4.5 minutes of GPU time. Held-out blind evaluation: soft win rate 0.833 [0.699, 0.972]; repetition 0.114 →\to 0.000 (the judges’ top-cited signal removed); spokenness 2.06 →\to 3.67; MMLU 0.4982 →\to 0.4895 (within noise). On held-out evaluation pairs with ∣Δchars∣≤100|\Delta\mathrm{chars}| \le100 (n=16n=16): 0.765 [0.58, 0.96].

The Goodhart channel: length collapse

Retraining directly on the expanded 240 pairs (iter-2) collapses outputs to a median of 75 characters (the prompt asks for “about 300 characters”). Mechanism: in 68% of agree pairs the chosen is shorter (judge length preference); pairwise DPO learns “shorter” as a direction; global length balancing of the dataset does not help (DPO learns per-pair); even per-pair length-neutral data (±30\pm30 chars, iter-3) shortens to 170 characters, because colloquial style covaries with brevity. Unlike the objective-intrinsic bias studied by LD-DPO/LC-DPO, the source here is judge annotation—measurable at data construction, and an end-to-end, controlled, reproducible instantiation of the transduction risk the judge survey warns about [6]. The failure is caught by a pre-registered secondary gate (length band), before any human evaluation.

Repair: the DPO+SFT anchor

We add a chosen-CE anchor (RPO-style): L=LDPO−w⋅1∣y+∣log⁡p(y+∣x)L=L_{\mathrm{DPO}}-w\cdot\frac{1}{\lvert y^{+}\rvert}\log p(y^{+}|x). The weight ww trades off fidelity-to-judge versus fidelity-to-data-marginals: w=0.3w=0.3 repairs length (246 chars) but washes out style (spokenness −0.66-0.66); w=0.1 preserves both: 213 characters (in band), spokenness 6.00, AI frames 0.000, rep 0.003, MMLU 0.4912. Held-out blind evaluation: 0.771 [0.600, 0.923] (passes the pre-registered gate); on held-out evaluation pairs with ∣Δchars∣≤100|\Delta\mathrm{chars}| \le100 (n=16n=16): 0.688.

Data saturation

Head-to-head, v5 (240 pairs) vs v1 (89 pairs) is 0.479 [0.283, 0.676]—statistically indistinguishable. On the judge-win-rate axis, 89 pairs are near saturation; the expansion’s gains appear in objective style metrics (spokenness 3.67 →\to 6.00, AI frames 0.203 →\to 0.000). Further gains require new prompts and new candidate pools, not denser sampling of the same pool.

ModelDataRecipeMedian charsHeld-out blind win rate
base––237–
iter-1 (v1)89DPO 3001510.833 [0.699, 0.972]
iter-2240DPO 48075rejected: length collapse
iter-390 (∣Δ∣≤30\lvert\Delta\rvert\le30)DPO 270121rejected: still short
iter-4240DPO+SFT w=0.3w=0.3173rejected: style washed out
iter-5 (v5)240DPO+SFT w=0.1w=0.11560.771 [0.600, 0.923]

Table 1. Iteration chain. Median characters (directly measured on identical generation configs (32 prompts). Blind pairwise win rates vs base on 8 held-out prompts (24 pairs, double-pass, disagreement-as-tie) apply to iter-1 and iter-5; iter-2/3/4 were rejected by pre-registered secondary gates (length/style) without blind evaluation.

Discussion

Why does speech grounding fail in the text domain? The three negatives share a pattern: the injected signal (phonemes, plans, prosody zz) is either redundant or semantically unrelated for next-token prediction. P0 probes show phonology is already linearly decodable in 0.6B hidden states (tone 0.63–0.79 vs majority class 0.31)—what is missing is not the representation but the reason to use it. This is consistent with the plan-ignoring mechanism of P1. In other words, AI flavor likely originates at the distribution level (register composition of the training corpus), not from a missing modality—which is why distribution-level interventions (pure-text CPT, preference tuning) work while modality injection does not.

Judge methodology: value and risk. Calibrated judges decouple the “cannot perceive” problem from the power layer (60–300 pairs needed) to the sensitivity layer (92–100% on calibration), making n=24n=24 evaluations viable. But § 5.3 shows their biases amplify along the optimization pipeline. Our position: judges may serve as evaluators and annotators, but every training run must be followed by a bias-transduction check (here: the length distribution), and an anchoring term should be standard equipment.

Limitations

A single judge model (GLM-5.3-Flash); a single Chinese short-essay genre (300-character tasks); 0.6B scale; the judge’s length preference is suppressed, not eliminated, by the SFT anchor (within ±30\pm30 chars, 62% of picks still favor the shorter side); the correlation between judge preference and human preference is not finally confirmed (human evaluation is power-limited); the weight comparison in § 5.4 uses two ww points, not a systematic sweep; the v5-vs-v1 comparison rests on n=24n=24; held-out covers 8 prompts; English and larger scales are untested.

Conclusion

The speech-grounding hypothesis does not survive controlled testing (three negatives), but the three-layer evaluation and calibrated-judge protocol built along the way enabled a previously infeasible positive experiment: wiring calibrated judge preference labels directly into DPO clears repetition loops to zero, raises spokenness, and achieves a 0.833 blind win rate—in 4.5 minutes of training. We also map the traps of this route: Goodhart transduction of annotator bias, and its repair. All tools, calibration sets, 382 judge-annotated preference pairs, and preregistration documents are released with RA-Eval v0.1.

Hyperparameters

BaseQwen3-0.6B (instruct) → 50M-token text-only CPT (P1a noplan arm) → DPO
TrainingLoRA r=32r=32 (α=64\alpha=64) on all projections, fp32
DPOlr 10−510^{-5}, β=0.1\beta=0.1, batch 1, accum 1, cosine to 0, seq 1024
SFT anchor−w⋅1∣y+∣log⁡p(y+∣x)-w \cdot\frac{1}{|y^{+}|}\log p(y^{+}|x), w∈{0,0.1,0.3}w \in\{0, 0.1, 0.3\}
Data240 agree pairs (train) / 67 (held-out); ∣Δchars∣≤100|\Delta\mathrm{chars}| \le100; not both degenerate
JudgeGLM-5.3-Flash, isolated sub-agents, A/B swap double-pass, disagreement =0.5=0.5

Table 2. Main hyperparameters.

Data Statement and Licenses

Human reference corpus. The human reference corpus consists of 86 Chinese college-entrance-examination (gaokao) essays collected from publicly accessible web archives, plus human texts embedded in the judge-calibration sets. Copyright remains with the original (anonymous) student authors and their publishers. The corpus is distributed for non-commercial research use only, with attribution, and under a takedown policy: any rights holder may request removal and the texts will be removed from all distributions. Contact: [email protected]. The full-text corpus is hosted on Zenodo with this license (DOI: 10.5281/zenodo.24237533) and is not part of the arXiv source package; the arXiv package contains only derived statistics and short excerpts.

Model outputs and annotations. All candidate texts are outputs of openly licensed 0.6B models; judge annotations (preference labels with per-pair rationales) are released under CC-BY-4.0. Model weights (base and iter-5) are released on Zenodo under the base model’s license.

Code. RA-Eval v0.1 is released under MIT.

Release Checklist

The release bundle (RA-Eval v0.1) contains: benchmark code; the judge pipeline (blind batch generation, label joining, judge scoring); calibration sets with answer keys; 382 preference pairs with soft labels and per-pair rationales; the 240-pair DPO training set; feature reference statistics; preregistration and result documents. Model weights and the full-text human corpus are distributed via Zenodo (see the data statement). Weights are not included in the arXiv source.

References

  1. [1]R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, C. Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290, 2023.arxiv.org/abs/2305.18290
  2. [2]W. Liu et al. Length Desensitization in Direct Preference Optimization. arXiv:2409.06411, 2024.arxiv.org/abs/2409.06411
  3. [3]R. Park et al. Disentangling Length from Quality in Direct Preference Optimization. Findings of ACL 2024 (arXiv:2403.19159).arxiv.org/abs/2403.19159
  4. [4]Y. Yang et al. CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories. arXiv:2608.01377, 2026.arxiv.org/abs/2608.01377
  5. [5]H. Y. Jung et al. Shortcut-Controlled Evaluation for AI-Generated Text Detection. EMNLP 2026.DOI
  6. [6]J. Gu et al. A Survey on LLM-as-a-Judge. The Innovation 7(6):101253, 2026 (arXiv:2411.15594).arxiv.org/abs/2411.15594
  7. [7]Q. Wang, X. Zhao, Z. Zhang, Z. Lou, N. Chen, D. Song, B. He. Making Bias Non-Predictive: Training Robust LLM Reasoning via Reinforcement Learning. arXiv:2602.01528, 2026.arxiv.org/abs/2602.01528
  8. [8]L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, S. Vosoughi. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. arXiv:2406.07791, AACL-IJCNLP 2025.arxiv.org/abs/2406.07791
  9. [9]S. Shin et al. No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS. arXiv:2509.18531, 2025.arxiv.org/abs/2509.18531
  10. [10]T. Sternberg, G. Maimon, Y. Adi. Interleaved Speech Language Models Latently Work In Text. arXiv:2606.22473, 2026.arxiv.org/abs/2606.22473
  11. [11]R. Wang et al. LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning. arXiv:2402.01158, 2024.arxiv.org/abs/2402.01158
  12. [12]H. Jin et al. LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA. arXiv:2509.00731, 2025.arxiv.org/abs/2509.00731
  13. [13]S. Ren et al. PersonaSteer: Dynamic Activation Steering for Personalized Multi-turn LLM Conversations. Findings of EMNLP 2026.DOI
  14. [14]P. Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv:2405.14734, 2024.arxiv.org/abs/2405.14734
  15. [15]A. Hans et al. Spotting LLMs with Binoculars: Zero-Shot Detection of Machine-Generated Text. arXiv:2401.12070, 2024.arxiv.org/abs/2401.12070
  16. [16]L. Ouyang et al. Training language models to follow instructions with human feedback. arXiv:2203.02155, 2022.arxiv.org/abs/2203.02155

Paper details

vlnc:W_jaqhrqsz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents