betaFeedback

WorksComputer Science

Pair-Dependent Judge Errors Change the Minimax Rate of Semantic Entropy

Abstract

Semantic entropy, the entropy of the meanings among a language model's sampled answers, is estimated from the verdicts of a judge that declares pairs of answers equivalent or not, and automatic judges err in ways that depend on the answers they compare. The existing minimax theory covers only judges whose false-equivalence probability is the same for all pairs. We model a known judge that declares two answers with meanings aa and bb equivalent with probability q+ΔMabq+ΔM_{ab}, for a symmetric matrix MM and qq bounded away from 00 and 11, and show that for three meanings the rate is set by tree statistics on four answers, while on an open set of channels the consistency threshold is set by trees on five. For every channel in an explicit open class, the minimax squared risk is of order N−1/3N^{-1/3} with N=n4Δ6N=n^4Δ^6 on polynomial weak windows on which n5Δ8n^5Δ^8 exceeds a multiple of log⁡n\log n, and of order min⁡{1,N−1/3}\min\{1,N^{-1/3}\} on every polynomial weak window when the tree functions on at most four vertices are injective. The exponent comes from a cubic degeneracy at the equitable point. On an open set of channels in the class, the four-vertex tree functions take equal values at two distributions with different entropies. For such a channel, on every polynomial weak window, the minimax risk is of the order of the sum of min⁡{1,N−1/3}\min\{1,N^{-1/3}\} and a term that decays exponentially in n5Δ8n^5Δ^8, so no estimator is consistent while n5Δ8n^5Δ^8 stays bounded, even as N→∞N\to\infty. For Δ=n−βΔ=n^{-β} with β>1/2β>1/2, channels of the class with injective four-vertex tree functions allow consistent estimation exactly for β<2/3β<2/3, and the channels of this open set exactly for β<5/8β<5/8, at which the risk exponent drops from 1/121/12 to 00. Computer-assisted certificates show that both kinds of channels occur on open sets.

Paper details

vlnc:W_yag8f8pw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…