betaFeedback

WorksComputer Science

QESChunker: A Single Objective Unifies Overlapping and Non-Overlapping Chunking for RAG

Abstract

Chunking is a fundamental preprocessing step in retrieval-augmented generation (RAG), yet most existing chunkers rely primarily on heuristics. We introduce the Question–Evidence-Supervised Chunker (QESChunker), to our knowledge the first method for constructing static chunk indices from question–evidence supervision. QESChunker constructs document-specific chunk indices by optimizing over human-annotated or LLM-generated question–evidence pairs, without access to held-out test queries. We formulate non-overlapping and overlapping index construction through a unified question–evidence utility-maximization objective, turning chunking into a supervised combinatorial optimization problem. Under the non-overlap constraint, the objective decomposes over chunks and can be optimized exactly in polynomial time by dynamic programming. When overlap is allowed, the objective becomes monotone submodular; exact optimization is NP-hard, but our greedy achieves a (1−1/e)(1-1/\sqrt{e})-approximation. Across two self-constructed and three public datasets, both variants outperform existing chunking methods in complete-evidence recall in nearly all evaluated settings, under the same overlap regime and a fixed hybrid retriever. These results recast chunking as supervised, budget-aware index construction for evidence retrieval.

Paper details

vlnc:W_8uw9gdj6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI

View statement
Looking for related papers…