betaFeedback

WorksComputer Science

Edit Recommendation: Can LLMs Anticipate Edit Requests from Users, and When?

Abstract

After a large language model (LLM) agent generates an artifact, users often send edit requests for it. This paper studies whether an LLM can anticipate these requests before they are sent. Such anticipation is useful in practice because it reduces the effort of user review and feedback. We formalize this task as edit recommendation and build EditRecBench, a benchmark for evaluating the performance of LLMs on this task, from real-world conversation logs in WildChat-4.8M. Through benchmarking and failure analysis, we identify a practical challenge that some cases are inherently difficult to predict in real-world settings. To address this, we introduce decision model-based gating to filter out cases where predictions are unlikely to be correct before they are presented to the user. Our trained model outperforms existing decision models and raises pass@5 among shown suggestions by 12–25 points at a gating threshold. Our code is available at https://github.com/ntt-dkiku/llm-edit-recommendation.

Introduction

Creating artifacts such as code and documents with a large language model (LLM)-based chat assistant often involves an iterative process: users review the assistant’s output, request changes, and repeat until the artifact meets their expectations. In this process, the assistant typically waits for users to send an edit request, but what if it could anticipate that request before they send it? With several likely edit requests shown as clickable suggestions in the chat interface, users could send their intended request with a single click, reducing review and feedback effort.

Prior work has studied user utterance prediction in the broader context of human–LLM conversations. Naous et al. [17] have proposed an LLM reproducing user turns to support realistic user simulation, while other studies have explored predicting next user prompts from conversational

Figure 1. Illustration of edit recommendation. LLMs predict the user’s next edit request for the most recent artifact from the preceding context.

context ([7, 3, 14]). In production, Codex and Claude Code have introduced features that suggest users’ next prompts ([20, 1]).

In contrast to these general prompt predictions, this paper focuses on predicting edit requests, which is practically useful in conversational artifact generation. We formalize this prediction task as edit recommendation and build EditRecBench, a benchmark for evaluating whether an LLM can predict users’ actual edit requests from the preceding context. The data is sampled from real-world conversation logs in WildChat-4.8M ([29]). We also propose a new evaluation criterion specifically designed for edit requests that compares the edit locations and content of predicted and actual edit requests. We evaluate eight LLMs (gpt-oss-20b/120b, Qwen3.5-9B, Gemma 4 12B, Llama 3.1, gpt-6-luna, etc.) on the benchmark, using Claude Haiku 5.5 for LLM-as-a-Judge. The results show that the models follow a similar ranking to that observed on other benchmarks, with pass@5 ranging from 10% to 47%.

Furthermore, our failure analysis reveals that some real-world edit requests are inherently difficult to anticipate from context alone. This indicates that simply offering suggestions every time may be counterproductive, as frequent incorrect suggestions could degrade the user experience in practice. We therefore introduce decision model-based gating to skip suggestions for such difficult samples, thereby improving pass@5 of the shown suggestions. Specifically, we fine-tuned Qwen3.5-4B as a decision model using the training split. Our gating model outperforms other statistical methods and decision models, and improves pass@5 on shown suggestions by 12–25 points at the threshold showing 20% of dev samples.

In summary, our contributions are threefold:

  • We propose a new benchmark for evaluating the ability of LLMs to anticipate edit requests from users in real-world datasets (§ 2).

  • We evaluate eight LLMs on the benchmark and validate the task feasibility. We also find that some real-world edit requests are inherently difficult to anticipate from context alone (§ 3).

  • We propose decision model-based gating to skip suggestions for inherently difficult samples, thereby improving pass@5 of the shown suggestions (§ 4).

Proposed Benchmark: EditRecBench

In this section, we describe the details of EditRecBench, including the task definition, evaluation metrics, and data curation pipeline.

Edit recommendation

Task definition. Edit recommendation is a task to predict the user’s next edit request from the preceding context. Let c=(u1,a1,…,un,an)c=(u_1,a_1,\ldots,u_n,a_n) be a conversation between a user and an LLM, where uiu_i and aia_i are the user’s and the assistant’s turns, and ana_n is the assistant’s latest response that includes an artifact. Given the preceding context cc, LLM predictors predict the user’s next edit request e(=un+1)e(=u_{n+1}) for ana_n. When generating multiple predictions, LLM predictors output the top KK edit requests e′=(e1′,…,eK′)e'=(e'_1,\ldots,e'_K) in descending order of likelihood.

Metrics. To evaluate whether a predicted edit request e′e' asks for the same edit as the actual edit request ee, we use LLM-as-a-Judge (Zheng et al. [30]). An LLM judge decomposes ee into a set of requirements R(e)={ρ1,…,ρm}R(e)=\{\rho_1,\ldots,\rho_m\}, each an atomic change ρj=(targetj,changej)\rho_j=(\mathrm{target}_j,\mathrm{change}_j), and labels every pair (e′,ρj)(e',\rho_j) with ℓ(e′,ρj)∈{satisfied,contradicted,absent}\ell(e',\rho_j)\in\{\text{satisfied},\text{contradicted},\text{absent}\}. A predicted edit request matches the actual one when it satisfies every requirement:

match⁡(e′,e)=I[∀ρ∈R(e):ℓ(e′,ρ)=satisfied].(1)\operatorname{match}(e',e)=\mathbb{I}\left[\forall\rho\in R(e):\ell(e',\rho)=\text{satisfied}\right]. \tag*{(1)}

An LLM predictor is evaluated by pass@kk (k≤Kk\le K), the share of samples in which at least one of its first kk predicted edit requests matches:

pass⁡@k=1∣D∣∑(c,e)∈Dmax⁡i≤kmatch⁡(ei′,e).(2)\operatorname{pass}@k=\frac{1}{|\mathcal{D}|}\sum_{(c,e)\in\mathcal{D}}\max_{i\le k}\operatorname{match}(e'_i,e). \tag*{(2)}

where D\mathcal{D} is the set of test samples.

Data curation

We curate data samples from WildChat-4.8M (Zhao et al. [29]), which includes real-world conversation logs between ChatGPT models and users collected between April 2023 and July 2025. The curation pipeline consists of multiple stages: rule-based pre-screening, LLM-based sample extraction, sample refinement, and data split. In the following, we describe the overview of each stage.

Pre-screening. We apply rule-based pre-screening to retain only samples that are mostly in English, contain no toxic or redacted content, and include a user turn following an assistant turn. We also exclude users whose traffic suggests automated activity or account sharing. To prevent highly active users from dominating the dataset, we retain at most three conversations per user. After pre-screening, 97,965 conversations remain.

Extracting edit requests. For each conversation, an LLM (gpt-oss-120b in this paper) reads the whole conversation, lists every edit request, and takes the last one as the edit request ee of a sample. The turns before it form the preceding context cc, and the turns after it are discarded. To help the LLM identify ee, we prompt it to extract ee along with the artifacts appearing immediately before and after ee. We repeat this process three times and retain only samples (c,e)(c,e) for which all three runs identify the same ee. This yields 18,992 samples.

Figure 2. Overview of EditRecBench. Left: illustration of data curation from WildChat-4.8M and task. Right: statistics for each split: (a) artifact types, (b) number of turns in context cc, and (c) artifact size in ana_n.

Sample refinement. We further filter out samples (c,e)(c, e) if either of the artifacts extracted from immediately before and after ee is empty, the two artifacts are identical, either artifact differs by more than 10% from its corresponding assistant response, or the artifact preceding ee contains fewer than 300 characters. With assistance from Claude Fable 5.1 and Opus 5.5, we also manually remove any ethically problematic and ambiguous samples remaining after the preceding processes. Finally, we draw one sample per user at random, which yields 12,298 samples. See Appendix § 13 for data validation.

Requirement annotation. We then define the requirements for each finalized sample using an LLM (Claude Haiku 5.5 in this paper). For the edit request ee of each sample, the LLM first generates a set of requirements. It then evaluates whether e^\hat{e}, a paraphrase of ee, satisfies these requirements. Since e^\hat{e} is essentially the same request as ee, it is expected to satisfy them. If the requirements are satisfied, we finalize the set. Otherwise, the LLM keeps revising and evaluating it until the requirements are satisfied or a total of three iterations is reached. If the requirements remain unsatisfied after the third iteration, we adopt the final set.

Splits and statistics. To prevent temporal leakage, we split all samples chronologically into training (April 2023–October 2024; 10,277 samples), development (November–December 2024; 885 samples), and test (January–July 2025; 1,136 samples) splits.

Benchmark Evaluation

In this section, we describe evaluation settings on the benchmark, show the results, and discuss real-world challenges through failure analysis.

Settings

Evaluated models. We evaluate eight LLMs: Llama 3.1 8B (Grattafiori et al., [8]), UserLM-8b (Naous et al., [17]), Qwen3.5-9B (Qwen Team, [24]), Gemma 4 12B (Gemma Team et al., [6]), gpt-oss-20b/120b (OpenAI et al., [18]), gpt-6-luna (OpenAI, [21]), gpt-6.1-sol (OpenAI, [22]). We enable reasoning and set the effort to medium for supported models. For all other parameters, we use the recommended settings. Each open model is served with vLLM (Kwon et al., [15]) on one or two RTX A6000 GPUs (48GB). In addition to LLMs, we also evaluate two rule-based baselines: Shuffled returns the five predictions that gpt-oss-120b made for another, randomly assigned sample, which gives the chance level of matching; Stock requests returns the same five generic or popular edit requests for every sample. See Appendix § 10 for details of models and parameters.

LLM-as-a-Judge. We use Claude Haiku 5.5 [2] with medium reasoning effort as an LLM judge. Given the preceding context, the actual edit request, and kk predicted edit requests, the judge evaluates whether any of the predictions are essentially equivalent to the actual request based on the annotated requirements. Extra details in a predicted edit request are allowed as long as they do not conflict with the actual request. We repeat this evaluation three times and set max⁡i≤kmatch⁡(ei′,e)=1\max_{i\leq k} \operatorname{match}(e_i', e) = 1 in Eq. (2) if at least two votes are positive. As a result, 98.8% of individual votes agree with the majority decision, and all three votes agree on 96.3% of samples, indicating high consistency of the LLM-as-a-Judge evaluations. See Appendix § 14 for the validation of LLM-as-a-Judge.

Results

The left-hand side of Table 1 shows pass@{1,3,5}\{1,3,5\} for each predictor. In terms of baselines, the very low pass@kk for shuffled requests indicates that our evaluation reliably distinguishes incorrect predictions. In contrast, while the two types of stock requests achieve higher pass@kk, it remains lower than that of most LLM predictors. This suggests that the task is not trivial and highlights the limitations of a rule-based approach.

pass@1pass@3pass@5pass@5 (no new info)pass@5 (new info)
Empirical ceiling.40 [.38,.43].53 [.51,.56].61 [.58,.63].70 [.67,.73].30 [.25,.36]
Shuffled.01 [.00,.01].01 [.01,.02].02 [.01,.03].03 [.02,.04].00 [.00,.01]
Stock requests (generic).06 [.05,.07].10 [.08,.11].11 [.09,.13].14 [.12,.16].02 [.00,.04]
Stock requests (popular).06 [.04,.07].13 [.12,.15].17 [.15,.20].20 [.17,.23].09 [.06,.13]
Llama 3.1 8B.04 [.03,.06].07 [.06,.09].10 [.08,.12].13 [.11,.15].01 [.00,.02]
UserLM-8b.03 [.02,.04].08 [.06,.09].11 [.09,.13].14 [.12,.16].02 [.00,.04]
Qwen3.5-9B.10 [.09,.12].15 [.13,.17].18 [.16,.20].22 [.19,.25].06 [.03,.09]
Gemma 4 12B.14 [.12,.16].23 [.20,.25].26 [.24,.29].33 [.29,.36].08 [.05,.11]
gpt-oss-20b.12 [.10,.14].19 [.16,.21].22 [.20,.25].27 [.24,.30].07 [.04,.10]
gpt-oss-120b.13 [.11,.15].22 [.20,.25].26 [.23,.28].32 [.29,.35].07 [.04,.10]
gpt-6-luna.24 [.21,.26].33 [.30,.36].39 [.36,.41].47 [.43,.50].14 [.10,.18]
gpt-6.1-sol.28 [.26,.31].41 [.38,.44].47 [.45,.50].57 [.54,.60].18 [.14,.23]

Table 1. Left: pass@kk of predictors on the test split. Right: pass@5 for new info and no new info samples. Brackets denote 95% bootstrap intervals over the test samples.

Among the LLM predictors, gpt-6.1-sol performs best, followed by gpt-6-luna, Gemma 4 12B, and gpt-oss-120b. Overall, this ranking is roughly consistent with their performance on other benchmarks. The empirical ceiling, defined as the fraction of samples matched by at least one predictor, is 61% at pass@5. This leaves approximately 14 points of headroom even for the best predictor, gpt-6.1-sol, while it remains unknown whether the remaining 39% of samples can feasibly be matched.

Failure analysis & discussion

To analyze failure modes, we categorize samples into two types: new info, where ee contains information not present in cc, and no new info, where ee introduces no information beyond what can be inferred from cc. For example, the former includes samples that introduce new names, constraints, goals, or topics, while the latter includes shortening the text, correcting errors, and translating.

The right-hand side of Table 1 shows pass@5 for each category, demonstrating that new info samples have a significantly lower pass@5. For example, after the assistant polishes an e-mail requesting a training session, the user asks to add that the team’s regular Monday meeting could host it, a fact that appears nowhere in cc. We observe such very hard samples in the real-world settings. This raises a practical question:

Practical question in deployment

Some edit requests are inherently difficult to predict. When deploying edit recommendation in a chat system, should we blindly show suggestions after every response?

Since frequent incorrect suggestions can harm the user experience, we argue that suggestions should be shown selectively.

Decision Model-Based Gating

To address the problem, we introduce decision model-based gating. After LLM predictors predict KK edit requests, it evaluates whether they include the actual edit request to decide whether to show them. They are shown to the user only if they pass the gate. This enables selective suggestions, increasing the likelihood of a match when they are shown. In the following, we model the gating mechanism and describe how to train a decision model for it.

Gating model

Given the context cc and KK predicted edit requests e′=(e1′,…,eK′)e' = (e'_1, \ldots, e'_K), a gating model returns a score g(c,e′)g(c,e') that estimates the probability of at least one of e′e' matching the user’s actual edit request ee:

g(c,e′)≈P(max⁡i≤Kmatch⁡(ei′,e)=1∣c,e′)(3)g(c,e') \approx P\left(\max_{i\leq K} \operatorname{match}(e'_i,e)=1 \mid c,e'\right) \tag*{(3)}

where 0≤g(c,e′)≤10 \leq g(c,e') \leq1. Let τ\tau be a confidence threshold; the list e′e' is shown to the user only when g(c,e′)≥τg(c,e') \geq\tau.

Decision model for gating

We implement the gating model using a decision model ([26, 19]), which enables cost-efficient classification.

Input context and output schema. A decision model takes natural-language instructions and outputs a probability between 0 and 1 for each option specified in the output schema. In our case, it takes (c,e′)(c,e') and outputs K+1K+1 yes/no probabilities: one indicating whether at least one of e′e' matches ee (match), and one for each ei′e'_i indicating whether it matches ee (sis_i). The probability of match being true serves as the score gg, while the sis_i are used as auxiliary training targets.

Training. We hypothesize that difficult samples share common underlying patterns, such as the new info discussed in § 3.3. To learn these patterns, we train a decision model using supervised learning. Specifically, we freeze the weights of the base model and train only LoRA adapters ([9]) on it and a classification head, minimizing the cross-entropy between the predicted probabilities and the labels given by the majority of three votes of the LLM judge.

Gating Evaluation

Settings

Training. We use Qwen3.5-2B/4B as the base model and train on outputs from gpt-oss-120b, gpt-6-luna, Gemma 4 12B, and Qwen3.5-9B in the training split (40,711 samples). The LoRA adapters have rank 16 and α=16\alpha= 16. The classification head consists of 2 routing layers and 4 decoder layers, with a hidden dimension of 512 and 8 attention heads. We train for one epoch with a learning rate of 2×10−42 \times10^{-4} (10−410^{-4} for the head), an effective batch size of 8, a cosine schedule with 30 warm-up steps, and a maximum input length of 32,768 tokens.

Baselines. We compare our gating model with three existing decision models without additional fine-tuning: Decisions API with gpt-6-luna ([19]), d3-mini (vLLM Semantic Router Team, 2026), and pplx-decider-flash-v1.2 ([23]). As a trained baseline, we also compare against logistic regression on TF-IDF and length features of the context and the predicted requests.

Metrics. We evaluate the gating models by AUROC ([5]), which measures how well gg predicts whether e′e' includes the user’s actual request across all thresholds τ\tau, with K=5K = 5. To show the effect of τ\tau, we set thresholds that retain the top 5–100% of the dev samples by gg and report the pass@5 among the test samples that pass the gate.

Results

Gating accuracy. Table 2 shows each gating model’s AUROC for LLMs whose predictions on the training split are used to train the gating model. Our models consistently outperform the baselines, with the Qwen3.5-4B-based model performing best, by .10 on average over the best baseline (.690 vs. .587). On the other hand, Table 3 shows the corresponding results for LLMs not used in training. Our models still consistently outperform the baselines, by .09 on average (.675 vs. .583), demonstrating generalization to predictions from LLMs not used in training. Across LLM predictors, the decision models tend to achieve higher AUROC on predictions from LLMs with lower original pass rates.

Gating modelgpt-6-lunagpt-oss-120bGemma 4 12BQwen3.5-9Bavg.
Zero-shot
d3-mini.549 [.52,.58].549 [.51,.59].564 [.53,.60].605 [.56,.65].567 [.55,.59]
pplx-decider-flash-v1.2.555 [.52,.59].574 [.54,.61].599 [.56,.64].622 [.58,.66].587 [.57,.61]
gpt-6-luna (Decisions API).508 [.47,.54].569 [.53,.61].563 [.52,.60].549 [.51,.59].547 [.53,.57]
Trained
Logistic regression.563 [.53,.60].564 [.53,.60].598 [.56,.64].589 [.55,.63].579 [.56,.60]
Ours (base: Qwen3.5-2B).657 [.62,.69].631 [.59,.67].678 [.64,.71].687 [.64,.73].663 [.64,.68]
Ours (base: Qwen3.5-4B).662 [.63,.69].665 [.63,.70].710 [.67,.74].723 [.68,.76].690 [.67,.71]

Table 2. AUROC of gating models for LLM predictors seen in training, which demonstrates generalization from the training split to the test split. Brackets denote 95% bootstrap intervals over the test samples.

Gating modelgpt-6.1-solgpt-oss-20bLlama 3.1 8BUserLM-8bavg.
Zero-shot
d3-mini.547 [.51,.58].575 [.53,.61].585 [.53,.64].566 [.51,.62].568 [.54,.59]
pplx-decider-flash-v1.2.546 [.51,.58].580 [.54,.62].616 [.56,.67].587 [.53,.64].583 [.56,.61]
gpt-6-luna (Decisions API).531 [.50,.56].568 [.53,.61].593 [.53,.64].520 [.47,.57].553 [.53,.57]
Trained
Logistic regression.570 [.54,.60].543 [.50,.58].564 [.51,.62].540 [.48,.59].554 [.53,.58]
Ours (base: Qwen3.5-2B).617 [.58,.65].666 [.63,.70].611 [.55,.66].591 [.54,.64].621 [.60,.64]
Ours (base: Qwen3.5-4B).640 [.61,.67].677 [.64,.71].723 [.67,.77].663 [.61,.72].675 [.66,.70]

Table 3. AUROC of gating models for LLM predictors not seen in training, which demonstrates generalization to unseen predictors. Brackets denote 95% bootstrap intervals over the test samples.

Pass rate improvement. Figure 3 shows how pass@5 on samples that pass the gate varies with the threshold τ\tau. In Figure 3, τ\tau increases from left to right, and only higher-confidence suggestions pass through the gate, so the pass rate rises. Our models consistently achieve the largest improvements in pass@5 across all LLM predictors.

Figure 3. pass@5 on test among the samples shown against the percentage of test samples shown. (a): predictors used for gating training. (b): predictors not used. We choose thresholds τ\tau that let 100%, 95%, ..., 5% of dev samples pass the gate, then plot the actual percentage of test samples passing each threshold on the x-axis.

Discussion. The threshold trades off precision against missed opportunities. For example, setting τ\tau to pass 20% of dev samples raises pass@5 by 12–25 points (from 47% to 62% for gpt-6.1-sol), but reduces suggestion frequency by 72% and retains 37% of all matches. In practice, a default could be τ=0.3\tau= 0.3 (showing suggestions one in five times, with about half matching), though the best choice depends on the application and the LLM predictor.

Conclusion

In this paper, we introduced a benchmark for evaluating LLMs’ ability to predict users’ next edit requests. Benchmarking and failure analysis revealed that some requests are inherently difficult to predict, motivating decision model-based gating to selectively show suggestions. Our trained gating model improved pass@5 among shown suggestions over baselines. We believe this paper contributes from both a research perspective, by revealing LLMs’ ability on the new task, and a practical perspective, by proposing a deployment strategy for edit recommendation in chat systems.

Limitations

Our gating model requires both the preceding context and the predicted edit requests, so it cannot avoid the LLM predictor’s inference cost even when suggestions are not shown to the user. As shown in Appendix § 16, the preceding context alone provides insufficient information for reliable gating, highlighting the importance of the predicted edit requests. Nevertheless, we consider that these costs are modest: with gpt-6-luna as the predictor, one list of five predicted edit requests costs about USD 0.0006 and takes a median of 4.0 s, and the gating model adds only 0.43 s on a single GPU.

We do not conduct a user study to quantify how the fraction of samples shown and pass@5 affect user experience. Instead, we evaluate our method using objective metrics, including pass@k and the fraction of samples shown. These results demonstrate its effectiveness under our offline evaluation but do not establish its impact on user experience.

Ethical Considerations

EditRecBench is built from WildChat-4.8M, which is released under the ODC-BY license and deidentified by its authors. We use the data in accordance with its ODC-BY license, cite its source, and refer to the original conversations by their hashes so that the dataset’s removal requests also apply to our samples. In addition to the dataset’s filter, we apply our own additional filters using LLM checks, keyword lists, and scans for credentials and personal secrets, manually reviewing all scan matches. This removes any sample with ethically problematic content. When in doubt, we removed the sample. In terms of the risks of the task itself, this study only asks LLMs to predict filtered edit requests for artifacts, so we do not expect the prediction task itself to cause direct harm.

Related Work

User prompt prediction and recommendation. Predicting user utterances supports efficient interaction with conversational systems. ChaI-TeA (Goren et al., [7] studies message autocompletion from conversation history and a typed prefix. OnePred (Chen et al., [3] predicts subsequent queries through recursive intent memory, while UserLM (Naous et al., [17] learns to reproduce user turns for realistic simulation. PromptHelper Kim et al., [14] recommends diverse follow-up prompts to support exploration during writing. On the other hand, our work focuses on edit-request prediction, enabling a concrete evaluation criterion based on where and how users request changes. EditRecBench also includes requests introducing previously unstated information, whereas OnePred’s benchmark selects queries whose intent and key information can be inferred from the preceding context. In addition to evaluating prediction capabilities, we identify challenges in deploying edit recommendation in chat systems and propose decision model-based gating for selective recommendation.

Artifact revision. Artifact revision has been studied across various domains. IteraTeR Du et al., [4] models iterative text revisions across domains and edit intentions, while CoEdIT Raheja et al., [25] edits text according to natural-language instructions. For long, structured documents, LEDGER Wang et al., [28] uses dependency-aware graph retrieval to select relevant context and preserve consistency during editing. In software engineering, SWE-bench Jimenez et al., [10] evaluates repository-level code changes for resolving real-world issues. For conversationally generated artifacts, Kikuta [13] studies the ability of LLMs to propagate revisions to dependent elements of artifacts. These studies address how to revise artifacts and maintain their quality or consistency. Our work addresses the complementary question of which revision the user will request next.

Selective recommendation. Selective prediction improves reliability by withholding likely errors. Kamath et al. [11] train a calibrator for selective question answering, while Smart Reply Kannan et al., [12] uses a triggering model to identify emails likely to receive replies. We apply selective prediction to edit recommendation by learning

whether predicted edit requests will cover the user’s next edit request.

Details of Models and Parameters

LLM predictors. Table 4 lists the predictors and their recommended settings. Every predictor receives the same prompts (Appendix § 17) and returns five predicted edit requests in a JSON object. An output that does not parse or validate is retried up to three times with the error fed back, and a sample still without a valid output counts as no match. The maximum output length is set to 30,000 tokens.

PredictorReasoningSampling
gpt-6.1-solmedium effortAPI defaults
gpt-6-lunamedium effortAPI defaults
gpt-oss-120bmedium effortTT 1.0, top-pp 1.0
gpt-oss-20bmedium effortTT 1.0, top-pp 1.0
Gemma 4 12Bthinking onTT 1.0, top-pp .95, top-kk 64
Qwen3.5-9Bthinking onTT 0.6, top-pp .95, top-kk 20
Llama 3.1 8B–TT 0.6, top-pp 0.9
UserLM-8b–TT 1.0, top-pp 0.8

Table 4. Recommended settings of the LLM predictors.

In terms of UserLM-8b, we apply a different approach to predict edit requests because it simulates the user’s next turn rather than following a task prompt. We give it the conversation with the intent “You are a user who wants the assistant to revise its last response.” as the system message and draw five samples of the next user turn as its five predicted edit requests.

LLM-as-a-Judge. We use Claude Haiku 5.5 with medium effort for defining the requirements and LLM-as-a-Judge.

Baselines. Table 5 shows the generic and popular stock requests. The generic requests are five hand-written requests of common kinds. For the popular requests, we embed the requirement changes of the training samples with bge-base-en-v1.5, cluster them with kk-means (k=40k = 40), let Claude Haiku 5.5 write one generic edit request per cluster, and choose five of these 40 and the five generic requests greedily by their judged matches on 400 training samples.

Stock requests (generic)Stock requests (popular)
1Make it shorter and more concise.Make it shorter and more concise.
2Add more detail and examples.Fix the errors the user reported in the code so that it runs without error, and provide the complete corrected version.
3Fix the mistakes and errors in it.Please revise your last response so that it adds the specific content, fields, or details that are currently missing, expanding each item the way the request calls for while keeping everything else as it is.
4Rewrite it in a different tone and style.Please simplify your last response so it is easier to read and understand, using plainer words and shorter, less complicated sentences while keeping the same meaning.
5Continue and expand it with the next part.Please rewrite the entire response in Chinese, translating all of the content rather than only parts of it.

Table 5. The five stock requests of each baseline, in the order they are given to the judge. The popular requests are in the order of the greedy choice.

For the gating baselines, d3-mini and pplx-decider-flash-v1.2 are used without fine-tuning through their own inference code, and the OpenAI Decisions API with gpt-6-luna. All three receive the same JSON state and questions as our gating model. The logistic regression (L2L_2 regularization, C=0.3C = 0.3) uses TF-IDF features from the last user turn (unigrams and bigrams), the artifact (unigrams), and the five predicted edit requests (unigrams and bigrams). TF-IDF vectorization uses sublinear term frequency, includes only terms that appear in at least three training samples, and limits each vocabulary to 50,000 terms. Additional features include one-hot encodings of the predictor and artifact type, as well as the logarithms of the number of turns, the artifact size, and the length of the last user turn.

Details of Data Curation

In this section, we describe the detailed settings of each curation stage.

Pre-screening. We use the conversation-level fields provided in WildChat-4.8M to filter samples. The language field indicates the most common language detected across the turns in a conversation, and we keep only samples with language = English. The toxic field is a boolean indicating whether a conversation contains toxic content, and we keep only samples with toxic = false. The redacted field is a boolean indicating whether a conversation contains any turns in which personal information or secrets, such as API keys, have been detected and redacted. We keep only samples with redacted = false. We also keep only conversations that contain at least one user turn that directly follows an assistant turn, which is a candidate for an edit request ee in the next stage. To avoid automated traffic and shared accounts, we identify a user by the pair of the hashed_ip field and the user agent in the header field. We then exclude every

conversation of a user who meets any of the following conditions: i) four or more users share the same IP address; ii) the IP address has more than 300 conversations; iii) the user has more than six conversations per active day (above the 99th percentile of users); iv) the user’s conversations are in three or more languages; v) the user has more than 200 conversations in total; or vi) one of the user’s conversations has exactly the same content as another conversation in WildChat-4.8M (e.g., templated or repeated inputs). From each remaining user, we keep three conversations at random.

Extracting edit requests. We use gpt-oss-120b to extract edit requests from each pre-screened conversation in the format shown in Figure 4. The last listed turn is the edit request ee. We run the extraction three times and keep a sample only if all three runs identify the same turn as ee.

Edit-request extraction format

Figure 4. Output format for extracting edit requests. revision_turns lists every candidate turn that asks to revise the preceding artifact, in order; the last one is the edit request ee. All fields are empty when the conversation has no edit request, and after_artifact is empty when the response to ee has no new version.

Sample refinement. We drop a sample if after_artifact is empty or identical to before_artifact. We also drop the sample if more than 10% of the lines in either artifact do not appear in the conversation up to the corresponding assistant turn.

Statistics and Results Breakdown

In this section, we present additional statistics and breakdown of pass@5 by artifact type, number of requirements, and request length.

Additional statistics. Figure 5 (a)–(c) shows additional statistics on the number of items in the annotated requirements, the length of edit requests, and the relationship between edit request length and the number of requirement items. In all three splits, about half of the requests have one requirement, one third have two, and the rest have three or more (panel (a)). The median request lengths are 50, 41, and 41 characters in the training, dev, and test splits, respectively (panel (b)). Longer requests tend to contain more requirement items (panel (c)).

Figure 5. Top: Additional statistics on the number of items in the annotated requirements, the length of edit requests, and the relationship between edit request length and the number of requirement items. Bottom: breakdown of pass@5 on the test split by artifact type, number of requirements, and request length.

Results breakdown. Figure 5 (d)–(f) shows pass@5 on the test split for each artifact type, number of requirements, and request length. pass@5 varies little across the common artifact types: for the six most common types, gpt-6.1-sol achieves 40–50% and the empirical ceiling is 57–65%. On the other hand, pass@5 decreases as the number of requirements increases. gpt-6.1-sol matches 66% of requests containing one requirement item, but only 13% of those containing three or more. pass@5 also decreases for longer requests, from 64% for requests of up to 30 characters to 26% for those of more than 70 characters, which is consistent with longer requests having more requirements.

Validation of Data Samples

To validate whether curated samples are aligned with the task of edit recommendation, we manually inspect a randomly selected subset of the dev/test samples: 100 LLM-accepted samples to estimate precision with a sufficient sample size, plus 50 LLM-rejected samples to compute Cohen’s κ\kappa between the LLM and the human annotator.

Given the preceding context cc, the actual edit request ee, and the assistant’s response to ee, a human annotator labels each sample as clear (a valid edit request), borderline, unrelated (not an edit request for an ana_n) or ethics (content that should not be included for ethical reasons). The labels of the LLM review are hidden during this annotation.

Table 6 shows the comparison between LLM’s labels and one human annotator’s labels for the 150 samples. Of the 100 samples accepted by the LLM, 93 are labeled clear by the human annotator (i.e., 93% precision). Of the other seven, five are borderline and two are unrelated. Agreement between the LLM and the human annotator over all 150 samples is moderate ([16]): 77% for clear vs. not clear (Cohen’s κ=.41\kappa= .41). The disagreement comes mostly from samples the LLM rejects as borderline but the human annotator finds clear (28 of 49), which reflects our data curation design: it trades recall for precision, so that the samples it keeps are valid edit requests. These results confirm that our pipeline successfully curates valid edit requests.

Human
LLMclearborderlineunrelatedethics
clear (accepted)93520
borderline (rejected)281830
unrelated (rejected)0100

Table 6. Cross-tabulation of human and LLM labels for the 150 samples.

Validation of LLM-as-a-Judge

To evaluate the validity of our LLM-as-a-Judge, we also manually inspect 100 pairs of a sample and a predictor: 50 that the judge counts as a match (the majority of its three votes finds e′e' satisfying every requirement) and 50 that it does not, sampled evenly from the eight LLM predictors.

Given the preceding context cc, the actual edit request ee, and the predicted edit request e′e', the human annotator answers yes, no, or unsure to whether sending e′e' would get the same edit as ee. The LLM’s judge results are hidden during this annotation.

Table 7 shows high agreement of the LLM judge with the human annotator on 100 pairs. The LLM judge agrees with the human annotator on 93 of the 100 pairs (Cohen’s κ=.86\kappa= .86). Agreement is 95% where the three votes are unanimous and 50% on the four pairs decided by two votes to one. These results confirm the validity of our LLM-as-a-Judge.

countvalue [95% CI]
Agreement93 / 100.93 [.86, .97]
unanimous votes91 / 96.95 [.88, .98]
split votes2 / 4.50 [.15, .85]
Precision of matches46 / 50.92 [.81, .97]
Precision of non-matches47 / 50.94 [.84, .98]
Recall46 / 49.94 [.84, .98]
Cohen’s κ\kappa.86 [.76, .96]

Table 7. Agreement of the LLM judge with the human annotator on 100 pairs.

Table 8 shows examples of actual edit requests, predicted requests, and their judge results. This also provides qualitative evidence of the judge’s validity.

ArtifactActual edit request eePredicted edit request e′e'
MatchCoderemove commentsJust give me the interface without the comments, example, or explanation.
MatchListNo just first and last nameWrite names without middle names
MatchDocumentplease include beverage pairings and sweet treats as wellPlease add a dessert and beverage options to each of the six menus.
MatchTranslationUse devnagriWrite it in Hindi script instead of Roman Hindi.
No matchListno bullet pointremove the intro, just give me the charges and statutes
No matchListnot so conversational the description, make it something like “direct, fresh..Can you make it even simpler and more natural? Some of the wording still sounds a bit robotic, especially “holistic” and “visual focal point”.
No matchDocumentmaybe more on the design aspectCan you shorten the description to a single punchy tagline for my portfolio?
No matchCodeOk, excellent, but I want it to be a script that runs this and I can plot it too!It would be nice to return the solution matrix from wave1D_spectral rather than just plot. Could you modify the function signature to something like [x, u] = wave1D_spectral(L, Nx, c, tmax, dt, u0, v0) and make it more reusable?

Table 8. Examples of actual edit requests, predicted requests, and their judge results.

Trade-off of Gating

Figure 6 shows three metrics for each predictor as the threshold τ\tau varies for our model (base: Qwen3.5-4B): pass@5 among the samples shown, the fraction of all matches retained, and the fraction of samples shown. As τ\tau increases, pass@5 improves for all predictors, while fewer matches and samples are retained. This illustrates the trade-off between improving pass@5 among shown suggestions and filtering out some matches as the threshold increases. A random gate retains matches at the same rate as it shows samples. The more the fraction of matches retained exceeds the fraction of samples shown, the better the gate is at selecting samples with matches.

Figure 6. Effect of the threshold τ\tau in our gating model on pass@5 among shown suggestions, the fraction of matches retained, and the fraction of samples shown. Shaded areas are 95% bootstrap confidence intervals.

Gating without Predicted Requests

Our gating model takes both the preceding context and the predicted edit requests as input, but could gating be performed using only the preceding context? If so, it could skip LLM predictor inference and reduce inference costs. To test whether such a gate is feasible, we train the decision model under the same setting as in § 5.1, but exclude the predicted edit requests from its input. We compare it with existing decision models given the same context-only input and with our gating model given both the context and the predicted edit requests.

Table 9 and Table 10 show the results. Without the predicted edit requests, the existing decision models are at chance level, and even the fine-tuned

Gating modelgpt-6-lunagpt-oss-120bGemma 4 12BQwen3.5-9Bavg.
Context only
d3-mini.495 [.46,.53].496 [.46,.54].475 [.44,.51].485 [.44,.53].488 [.47,.51]
pplx-decider-flash-v1.2.510 [.48,.55].510 [.47,.55].558 [.52,.60].536 [.49,.58].528 [.51,.55]
Ours (base: Qwen3.5-2B).545 [.51,.58].553 [.51,.59].562 [.52,.60].537 [.49,.58].549 [.53,.57]
Ours (base: Qwen3.5-4B).574 [.54,.61].591 [.55,.63].596 [.56,.63].555 [.51,.60].579 [.56,.60]
With predicted edit requests
Ours (base: Qwen3.5-4B).662 [.63,.69].665 [.63,.70].710 [.67,.74].723 [.68,.76].690 [.67,.71]

Table 9. Test AUROC of gating models that take only the preceding context and the predictor’s name, for LLM predictors seen in training. Brackets denote 95% bootstrap intervals over the test samples. Last row: our gating model that also takes the predicted edit requests.

Gating modelgpt-6.1-solgpt-oss-20bLlama 3.1 8BUserLM-8bavg.
Context only
d3-mini.534 [.50,.57].503 [.46,.54].531 [.48,.58].517 [.47,.57].521 [.50,.54]
pplx-decider-flash-v1.2.506 [.47,.54].546 [.51,.58].544 [.49,.60].515 [.46,.57].528 [.50,.55]
Ours (base: Qwen3.5-2B).530 [.50,.56].557 [.52,.60].509 [.45,.57].498 [.44,.55].523 [.50,.55]
Ours (base: Qwen3.5-4B).547 [.51,.58].610 [.57,.65].567 [.52,.62].544 [.48,.60].567 [.54,.59]
With predicted edit requests
Ours (base: Qwen3.5-4B).640 [.61,.67].677 [.64,.71].723 [.67,.77].663 [.61,.72].675 [.66,.70]

Table 10. Test AUROC of gating models that take only the preceding context and the predictor’s name, for LLM predictors not seen in training. Brackets denote 95% bootstrap intervals over the test samples. Last row: our gating model that also takes the predicted edit requests.

Qwen3.5-4B reaches average AUROCs of only .579 and .567, approximately .11 below the same model with the predicted edit requests. This indicates that the preceding context alone carries little information on whether the predictions will match. Therefore, the predicted edit requests are necessary for the gating models.

Prompts

In this section, we present prompts given to LLMs. Figure 7 is used for extracting edit requests, Figure 8 for predicting edit requests, Figure 9–Figure 11 for the LLM-as-a-Judge (decomposing the actual request into requirements, judging the predicted requests, and checking the requirements), and Figure 12 for the input of the decision model. Tokens in curly braces such as {conversation} are placeholders replaced with specific values. {format_instructions} specifies the JSON output format and schema, including field names and descriptions. Responses are validated against the schema, and validation errors are fed back for retries.

Prompts for extracting edit requests

Figure 7. Prompts for extracting edit requests.

Prompts for predicting edit request

Figure 8. Prompts for predicting edit request

Prompts for decomposing an edit request into requirements

Figure 9. Prompts for decomposing an edit request into requirements.

Prompts for checking the requirements with a rewording of the edit request

Figure 11. Prompts for checking the requirements with a rewording of the edit request.

Input state and questions of the decision model

Figure 12. Input state and questions of the decision model.

AI Usage

We implemented code with the assistance of coding agents (e.g., claude code, codex). However, the methodology and evaluation were designed by the authors, who also carefully reviewed the implemented code. We also used LLMs to refine the phrasing in the manuscript.

Prompts for judging predicted edit requests against the requirements

Figure 10. Prompts for judging predicted edit requests against the requirements.

Judge (stage 1, check): Rewording prompt

Rewrite the following message in different words, keeping exactly the same meaning and level of detail. Reply with the rewritten message only.

<<<
{text}
>>>

Rewrite prompt

A rewording of the edit request, with the same meaning, was checked against your requirements:

<<<
{rephrased}
>>>

It did not satisfy these requirements:
{failures}

Rewrite the requirements so that any request with the same meaning as the edit request, including this rewording, satisfies every one of them. Follow the same rules: write each requirement only from what the edit request itself says, and keep every specific it states (names, values, places, styles). If the rewording does not mean the same as the edit request, do not follow it: keep the requirements that the edit request itself supports. Return the full set of requirements again.
{format_instructions}
Decision model: state

{"conversation": [{"role": "user", "content": {turn_1}},
{"role": "assistant", "content": {turn_2}}, ...],
"output_to_revise": {artifact},
"suggestions": [{suggestion_1}, ..., {suggestion_5}]}

Decision model: questions

match: Will at least one of the suggestions explicitly ask for every change (same target, direction and specifics) that the user's next request to revise output_to_revise will ask for, and for nothing incompatible with it? A broader instruction that does not state a change does not count.

s{i}: Will suggestion {i} (item {i} of suggestions) explicitly ask for every change (same target, direction and specifics) that the user's next request to revise output_to_revise will ask for, and for nothing incompatible with it? A broader instruction that does not state a change does not count.

References

  1. [1]Anthropic. 2025. Interactive mode — prompt suggestions. Accessed 2026-10-25.
  2. [2]Anthropic. 2026. Introducing Claude Haiku 5.5. Accessed 2026-10-30.
  3. [3]Jiangwang Chen, Bowen Zhang, Zixin Song, Jiazheng Kang, Xiao Yang, Da Zhu, and Guanjun Jiang. 2026. Onpred: Next-query prediction via recursive intent memory in multi-turn conversations. Preprint, arXiv:2605.23668.arxiv.org/abs/2605.23668
  4. [4]Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022. Understanding iterative revision from human-written text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3573–3590, Dublin, Ireland. Association for Computational Linguistics.
  5. [5]Tom Fawcett. 2006. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861–874. ROC Analysis in Pattern Recognition.DOI
  6. [6]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, and 304 others. 2026. Gemma 4 technical report. Preprint, arXiv:2607.02770.
  7. [7]Shani Goren, Oren Kalinsky, Tomer Stav, Yuri Rapoport, Yaron Fairstein, Ram Yazdi, Nachshon Cohen, Alexander Libov, and Guy Kushilevitz. 2025. Chal-TeA: A benchmark for evaluating autocompletion of interactions with LLM-based chatbots. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 18–32, Albuquerque, New Mexico. Association for Computational Linguistics.
  8. [8]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Srivankumar, Artem Koreney, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  9. [9]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  10. [10]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.06770
  11. [11]Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684–5696. Online. Association for Computational Linguistics.arxiv.org/abs/2006.09462
  12. [12]Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart reply: Automated response suggestion for email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 955–964, New York, NY, USA. Association for Computing Machinery.
  13. [13]Daisuke Kikuta. 2026. What else needs fixing? exploring cost-effective test-time compute for revision propagation in artifacts generated through conversation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 4161–4177, Budapest, Hungary. Association for Computational Linguistics.arxiv.org/abs/2609.03254
  14. [14]Jason Kim, Maria Teleki, and James Carverlee. 2026. Prompthelper: A prompt recommender system for encouraging creativity in ai chatbot interactions. In Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, CHI EA ’26, New York, NY, USA. Association for Computing Machinery.
  15. [15]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA. Association for Computing Machinery.arxiv.org/abs/2309.06180
  16. [16]J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174.pubmed.ncbi.nlm.nih.gov/843571
  17. [17]Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. 2026. Flipping the dialogue: Training and evaluating user language models. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2510.06552
  18. [18]OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  19. [19]OpenAI. 2026a. Build more responsive apps with the Decisions API. Accessed 2026-10-30.
  20. [20]OpenAI. 2026b. Composer predictions. Accessed 2026-10-25.
  21. [21]OpenAI. 2026c. Introducing GPT-6 Sol and Luna. Accessed 2026-10-30.
  22. [22]OpenAI. 2026d. Near-astra intelligence for a fifth of the price. Accessed 2026-10-30.
  23. [23]Perplexity. 2026. pplx-decider-flash-v1.2. https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2.
  24. [24]Qwen Team. 2026. Qwen3.5: Towards native multimodal agents.
  25. [25]Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEdIT: Text editing by task-specific instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5274–5291, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.09857
  26. [26]TypeSafe. 2026. Introducing System One Models & Jev. Accessed 2026-10-30.
  27. [28]Hang Wang, Utkarsh Garg, Reza Davari, Huitian Jiao, Hao Cheng, Baolin Peng, Si-Qing Chen, and Tao Ge. 2026. LEDGER: Scaling agentic document editing with dependency-aware graph retrieval. In Findings of the Association for Computational Linguistics: ACL 2026, pages 10614–10644, San Diego, California, United States. Association for Computational Linguistics.
  28. [29]Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2405.01470
  29. [30]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.arxiv.org/abs/2306.05685

Paper details

vlnc:W_9j63bz5q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, Codex, LLMs

View statement
Looking for related papers…
Contents