tools.py)Turn-1 Standard Implementation FlowCore Execution PatternsClaude Opus 5.5 EfficientSkill Package on SpreadsheetBenchInterpretation rulesEfficiencyGemini 3.8 Flash EfficientSkill Package on ALFWorldTarget objectWorkflow (4-6 turns)Destination (head noun)Special patternsSide-by-Side Trajectory Comparisons Across BenchmarksSpreadsheetBench: Stopping No-Code Loops & Unsaved SubmissionsALFWorld: Targeted Object Search vs. Wandering Across RoomsMATH-500: Direct Symbolic Solution vs. Repeated Code RetriesReferencesWorksComputer Science
EfficientSkill: Learning Skills for Token-Efficient Multi-Turn Agents
Abstract
Reflective skill optimizers improve multi-turn LLM agents by revising instructions from failed and successful rollouts, yet accumulating edge-case rules bloats prompts, inflates token use, and pushes agents into long, error-prone loops. We introduce EfficientSkill, a two-step framework that first discovers a reliable skill without token penalties, and then warm-starts accuracy-guarded pruning from that skill to compress instructions, bound agent turns, and offload reusable code into workspace utilities with zero implementation prompt cost. Across four frontier, mid-tier, and open-weight models on MATH-500, ALFWorld, and SpreadsheetBench, EfficientSkill cuts output tokens over unconstrained EvoSkill on 11 of 12 configurations (by up to 80%) while raising mean accuracy on every benchmark, including +14.3 and +17.5 percentage-point gains on SpreadsheetBench with Gemini 3.8 Flash and Grok 4.1 Fast by replacing stalled loops with compact workflows. By shrinking both generated tokens and replayed history, EfficientSkill lowers total token usage and serving cost across every evaluated configuration, with or without full-prefix KV caching.
Introduction
Coding and tool-use agents increasingly organize domain knowledge into modular skill packages that pair natural-language instruction documents with executable workspace scripts and utilities ([2] (Li et al. 2026)). Recent reflective optimizers automate skill authoring by iteratively revising instructions from failed execution traces ([1, 2] (Yuksekgonul et al. 2025; Moll et al. 2026)), yet optimize almost exclusively for task accuracy.
In multi-turn deployments, optimizing skills for accuracy alone creates two compounding problems:
High Output Cost and Quadratic History Growth: Output and reasoning tokens cost up to input rates and drive up per-turn billing. Moreover, because multi-turn APIs re-send the growing interaction history at every turn, cumulative input tokens grow quadratically ((Sayed Ali et al., 2026; Liu, 2026)).
Prompt Over-Fitting and Multi-Turn Drift: Repeatedly patching instructions for edge cases yields over-fitted prompts. Rather than acting decisively, agents fall into hesitant re-inspection loops, unproductive deliberation turns, and cascading execution errors ((Laban et al., 2026; Khanal et al., 2026)).
Efficiency methods for prompts ([8] (Nayab et al. 2026; Xu et al. 2025)) and single-stage cost-regularized optimizers ((Zhao et al., 2025; Shah et al., 2026)) penalize token usage from the start of search. When an agent begins with a low-accuracy prompt on an unfamiliar environment, solving the task first requires exploratory instructions and diagnostic steps; penalizing tokens too early suppresses and stalls optimization near its starting accuracy. Conversely, post-hoc compressors such as SkillReducer ([7]) trim static skill instructions ( input tokens) without shortening the reasoning or actions an agent generates at runtime.
We introduce EFFICIENTSKILL (Figure 1, Algorithm 1), a sequential two-phase framework that first discovers a reliable skill () and then learns a token-efficient skill (). From minimal seed , Phase 1 repairs failed rollouts without length penalties to establish target-model validation accuracy . Phase 2 starts from and evolves to minimize output tokens while ensuring that candidate accuracy is only allowed to be slightly below validation accuracy .

Figure 1. Overview of EFFICIENTSKILL. From minimal seed , Phase 1 repairs failed rollouts on frozen agent without token penalties to discover anchor skill . Warm-starting from , Phase 2 prunes episodic output tokens () under a validation gate, yielding concise rules and zero-body-cost workspace tools ().
Evaluating EFFICIENTSKILL across four frontier, mid-tier, and open-weight models on MATH500, ALFWORLD, and SPREADSHEETBENCH against unconstrained skill evolution (EVOSKILL) [2] yields four main findings:
Substantial Output Compression (§5.1): Across MATH-500, ALFWORLD, and SPREADSHEETBENCH, EFFICIENTSKILL cuts output tokens on 11 of 12 pairs (38–80% on 9 of 12), while preserving or improving accuracy.
EFFICIENTSKILL Acts as Regularization (§5.2): Enforcing token efficiency trims bloated rules and ensures skills are generalized. On SPREADSHEETBENCH, it raises held-out accuracy by +14.3 points on Gemini 3.8 Flash and +17.5 points on Grok 4.1 Fast by curbing timeouts and early exits.
Why the Starting Point Matters (§5.3): Pruning directly from seed stalls on unfamiliar tasks since it struggles to find the successful trajectories required to learn the skills in the first place.
Lower Serving Cost (§5.4): Shorter horizons and offloaded utilities cut billed input tokens by 8–66% and lower serving cost across all 12 pairs (25–67% uncached and 31–75% under full-prefix provider KV caching on 10 of 12).
Related Work
Agent Skills & Workspace Tool Libraries. Early agent memory and tool-learning systems accumulate trajectories or reusable functions ([3] (Shinn et al. 2023; Wang et al. 2024a; Qian et al. 2023)), while TroVE (Wang et al. (2024c)) and AWM (Wang et al. (2025)) induce code toolboxes and workflows. Recent work standardizes modular skill packages pairing Markdown instructions with auxiliary scripts: SkillsBench (Li et al. (2026)) finds that focused multi-module skills out-perform exhaustive manuals, whereas EVOSKILL (Alzubi et al. [2]), GRASP (Moll et al. (2026)), SkillOpt (Yang et al. (2026)), and SkillLens (Huang et al. [10]) evolve skills from failures solely for accuracy. SkillReducer (Gao et al. [7]) trims static instructions (−27% input tokens) without reducing output (+1.7%). In contrast, EFFICIENTSKILL jointly optimizes prompt instructions and zero-prompt-cost workspace utilities to cut multi-turn output tokens and turns.
Prompt Optimization & Reasoning Efficiency. Prompt and trace optimizers such as DSPy (Khattab et al. (2024)), MIPROv2 (Opsahl-Ong et al. (2024)), TextGrad (Yuksekgonul et al. (2025)), Trace (Cheng et al. [6]), and GEPA (Agrawal et al. [1]) refine instructions over multi-step programs and agent rollouts, yet accuracy-only search accumulates long prompts without bounding generation cost. Compile-time structural compression (Schnabel and Neville (2024)) shrinks static prompts rather than multi-turn trajectories, whereas brevity prompts (Nayab et al. (2026); Chen et al. [5]; Han et al. [8]; Xu et al. (2025)) and single-stage cost-regularized optimizers (Zhao et al. (2025; Shah et al., 2026)) penalize generation from the start of search, suppressing task learning when initial accuracy is low. EFFICIENTSKILL avoids this cold-start failure via unconstrained discovery followed by accuracy-guarded pruning.
Multi-Turn Reliability & Context Cost. Multi-turn accuracy degrades over long horizons as early mistakes and tracebacks accumulate in context [4] (Wang et al. 2024b; Yang et al. 2024; Laban et al. 2026; Khanal et al. 2026). Stateless APIs also replay history at input cost (Sayed Ali et al., 2026; Pan et al., 2026), while mid-episode context compression [11] (Pan et al. 2024) can drop critical state and trigger recovery turns (Liu, 2026). Like codified workflows (Yang et al., 2025), EFFICIENTSKILL improves both cost and reliability at the source via concise instructions and workspace utilities.
EfficientSkill
Algorithm 1: EFFICIENTSKILL
Input: Agent , seed , , , ,
Output: Token-efficient skill
Phase 1: Discover Anchor // unconstrained
;
for mini-batch until patience do
// fix failures
at ckpt: if then
else
Phase 2: Learn Efficient // warm-start from
;
for mini-batch until patience do
// cut , add
at ckpt: if and
then else
return
†Requires valid ; also commits Pareto/acc. gains (Appendix § 10.§ 2).
Sequential Optimization. Let and denote validation pass rate and mean episodic output tokens on . After each training mini-batch, a reflective meta-optimizer proposes provisional edits , evaluated on at periodic checkpoints (4/epoch) and committed or reverted to the incumbent (Appendix § 10.§ 2). Because penalizing tokens directly from suppresses early task learning, EFFICIENTSKILL runs two sequential phases (Figure 1, Algorithm 1):
Discover Optimized Skill (Unconstrained): Starting from , Phase 1 (REPAIR) fixes failed rollouts without token penalties, committing improvements to establish anchor skill with validation accuracy .
Learn Efficient : Warm-starting under fixed discrete floor (Appendix § 10.§ 2), Phase 2 (PRUNE) tightens prose, bounds turns, and offloads code into compilation-checked . Checkpoints commit under when drops by or Pareto-improves .
Experimental Setup {#sec:4}
Benchmarks & Verifiers. We evaluate three interactive benchmarks with deterministic binary verifiers (; Appendix § 10):
MATH-500 ([9]): Competition mathematics with optional Python execution, graded by symbolic equivalence.
ALFWorld (Shridhar et al., 2021): Multi-turn household navigation and object manipulation, graded by simulator goal predicates.
SpreadsheetBench (Ma et al., 2024): Real-world spreadsheet editing via Python scripts, graded by cell-level value matching against reference workbooks.
Models & Evaluation Protocol. All methods run in a shared stateless execution environment with isolated per-episode workspaces and exact per-turn token logging (Appendix § 10). Benchmarks use frozen 20%/10%/70% train/validation/test splits. Each selected skill is evaluated once on across four models: Claude Opus 5.5, Gemini 3.8 Flash, Grok 4.1 Fast, and open-weight Gemma 4 26B.
Optimization Schedule & Baseline. We instantiate with empty Markdown instruc-
| Model | Test Pass (%) | Mean Turns () | Output Tokens / Episode () | ||||
| EvoSkill | Ours | EvoSkill | Ours | EvoSkill | Ours | ||
| MATH-500 (Competition Math with Optional Python) | |||||||
| Claude Opus 5.5 | 86.4 | 87.5 | 1.00 | 1.00 | 426 | 102 | −76.0% |
| Gemini 3.8 Flash | 85.0 | 87.5 | 1.76 | 1.04 | 1,734 | 543 | −68.7% |
| Grok 4.1 Fast | 72.5 | 72.5 | 1.01 | 1.00 | 1,937 | 1,665 | −14.1% |
| Gemma 4 26B | 80.4 | 83.6 | 1.70 | 1.19 | 1,349 | 806 | −40.3% |
| ALFWORLD (Embodied Household Navigation) | |||||||
| Claude Opus 5.5 | 88.9 | 93.2 | 10.21 | 7.33 | 933 | 376 | −59.7% |
| Gemini 3.8 Flash | 92.1 | 93.6 | 9.85 | 7.16 | 5,704 | 1,155 | −79.7% |
| Grok 4.1 Fast | 89.3 | 88.2 | 9.15 | 8.71 | 12,761 | 12,350 | −3.2% |
| Gemma 4 26B | 81.4 | 81.1 | 10.74 | 9.12 | 95 | 99 | +4.4%§ |
| SPREADSHEETBENCH (Multi-Turn Programmatic Workbook Editing) | |||||||
| Claude Opus 5.5 | 88.6 | 88.6 | 2.80 | 3.04 | 1,466 | 761 | −48.1% |
| Gemini 3.8 Flash | 58.2 | 72.5 | 10.44 | 8.56 | 13,376 | 8,364 | −37.5% |
| Grok 4.1 Fast | 57.5 | 75.0 | 2.71 | 1.97 | 10,056 | 4,271 | −57.5% |
| Gemma 4 26B | 47.1 | 45.7 | 5.41 | 3.39 | 4,677 | 2,174 | −53.5% |
Table 1. Held-out test performance () comparing EVOSKILL against EFFICIENTSKILL in task pass rate (%), mean turns (), and mean output tokens per episode (, including reasoning). Notes: Bold marks strictly better values per pair ( from unrounded totals; 1 test task = 0.36 pts, binomial SE pts; validation progression and serving costs in Tables 2 and Table 6). Under a 5%-pt slack floor, Gemma reaches 47.9% on SPREADSHEETBENCH (−57.9% OutTok). §On ALFWORLD, Gemma emits 0 reasoning tokens (95.2 → 99.4 action tokens), while fewer turns cut billed input tokens by 36.9% and serving cost by 36.7% (Table 2).
tions (SKILL.md; Appendix D.§ 1) and a Python module (tools.py), using Claude Opus 5.5 as meta-optimizer. Gemini Argon as meta-optimizer showed similar performance. Each phase runs mini-batches of training tasks for up to 10 epochs (4 validation checkpoints/epoch, patience 6, , ; Appendix § 10.§ 2). EVOSKILL [2] runs unconstrained accuracy-only search under Phase 1’s identical schedule (); Phase 2 adds warm-started pruning (extended-budget EVOSKILL control in Appendix § 11.§ 2; marginal spend in Table 9).
Table 9: Phase 2 optimization spend and deployment break-even episodes (). Columns report Phase 2 target-model evaluation spend (), reflection batch count (), estimated Claude Opus 5.5 meta-optimizer spend (), per-episode serving savings () under standard uncached billing vs. provider prefix caching (“KV-Prov”), and for target-model spend alone vs. all-in ().
| Benchmark | Model | Target Spend () | Batches () | Est. Meta () | Savings / Ep ($/ep) Std | Savings / Ep ($/ep) KV-Prov | Target Std | Target KV-Prov | All-In (incl. Meta) Std | All-In (incl. Meta) KV-Prov |
| MATH-500 | Claude Opus 5.5 | $46.07 | 53 | $6.36 | $0.04193 | $0.04193 | 1,099 | 1,099 | 1,251 | 1,251 | ||
| MATH-500 | Gemini 3.8 Flash | $0.61 | 78 | $9.36 | $0.00054 | $0.00042 | 1,128 | 1,449 | 18,561 | 23,852 | ||
| MATH-500 | Grok 4.1 Fast | $0.78 | 28 | $3.36 | $0.00018 | $0.00018 | 4,326 | 4,363 | 22,938 | 23,138 | ||
| MATH-500 | Gemma 4 26B | $0.42 | 33 | $3.96 | $0.00085 | $0.00057 | 492 | 734 | 5,142 | 7,676 | ||
| ALFWORLD | Claude Opus 5.5 | $95.15 | 18 | $2.16 | $0.37101 | $0.09847 | 257 | 967 | 263 | 989 | ||
| ALFWORLD | Gemini 3.8 Flash | $1.22 | 30 | $3.60 | $0.00227 | $0.00152 | 537 | 806 | 2,120 | 3,181 | ||
| ALFWORLD | Grok 4.1 Fast | $10.60 | 35 | $4.20 | $0.00176 | $0.00071 | 6,039 | 14,978 | 8,431 | 20,913 | ||
| ALFWORLD | Gemma 4 26B | $2.32 | 23 | $2.76 | $0.00210 | $0.00112 | 1,102 | 2,060 | 2,414 | 4,515 | ||
| SPREADSHEETBENCH | Claude Opus 5.5 | $231.28 | 30 | $3.60 | $0.11215 | $0.08087 | 2,063 | 2,860 | 2,095 | 2,905 | ||
| SPREADSHEETBENCH | Gemini 3.8 Flash | $3.56 | 15 | $1.80 | $0.00250 | $0.00161 | 1,426 | 2,206 | 2,146 | 3,321 | ||
| SPREADSHEETBENCH | Grok 4.1 Fast | $2.70 | 25 | $3.00 | $0.00375 | $0.00324 | 720 | 833 | 1,520 | 1,758 | ||
| SPREADSHEETBENCH | Gemma 4 26B | $2.63 | 20 | $2.40 | $0.00508 | $0.00306 | 518 | 861 | 990 | 1,647 |
Table 9. Notes: Both “Savings / Ep” (Cost) and break-even episode counts () are computed from exact unrounded per-episode API totals prior to 5-decimal display rounding in Table 8. Onboarding from additionally incurs Phase 1 discovery cost (shared identically with EvoSKILL).
Results
Output-Token Compression
Table 1 reports held-out test performance across all 12 model–benchmark pairs. Across all three benchmarks, EFFICIENTSKILL cuts output generation on 11 of 12 pairs while raising mean benchmark accuracy:
MATH-500: EFFICIENTSKILL cuts output tokens by 40–76% on Claude Opus 5.5, Gemini 3.8 Flash, and Gemma 4 26B (14% on Grok 4.1 Fast) while matching or slightly exceeding EVOSKILL accuracy. Learned skills replace multi-turn code retries with direct symbolic derivation, cutting turns from 1.76 → 1.04 on Gemini and 1.70 → 1.19 on Gemma (App. § 16.§ 3).
ALFWORLD: On Opus and Gemini, where
output is dominated by deliberation, distilling compact object-location priors cuts output tokens by 60% and 80% and lifts Opus accuracy by +4.3 points (88.9% → 93.2%). On Gemma 4 (0 reasoning tokens), finishing in fewer turns (10.74 → 9.12) still cuts billed input tokens and cost by 37% (Table 2).
SPREADSHEETBENCH: EFFICIENTSKILL cuts output tokens by 38–58% across all four models (48% on Opus, 38% on Gemini, 58% on Grok, and 54% on Gemma) by shortening multi-turn editing loops and moving recurring code into workspace utilities.
Pruning Acts as Regularization, Leading to Better Performance
Across all three benchmarks, guarded pruning raises four-model mean test accuracy (81.1% 82.8% on MATH-500, 87.9% 89.0% on ALFWorld, and 62.9% 70.4% on SpreadsheetBench; Figure 2). Gains are largest on SpreadsheetBench, where unconstrained EvoSkill expands prompts to 908–1,521 words with empty tools.py modules (Appendix § 11.§ 2):
Figure 2. Pruning Acts as Regularization. Mean held-out pass rate across all four models improves on every benchmark under EFFICIENTSKILL, led by +7.5 points on SPREADSHEETBENCH (+7.6 unrounded).
Fewer Timeouts and Early Exits: EfficientSkill lifts test accuracy by +14.3 points on Gemini 3.8 Flash (58.2% 72.5%) and +17.5 points on Grok 4.1 Fast (57.5% 75.0%) while matching Opus (88.6%; −1.4 points on Gemma), cutting combined timeout and deliberation-drift failures on Gemini (35.7% 19.6%; 20-turn timeouts 16.1% 6.4%) and premature Turn-1 exits on Grok (18.2% 3.2%).
Shorter Prompts and Reusable Helpers: Phase 2 acts jointly on both components, shrinking prompt instructions by 23–59% (Table 2) while populating 241–554 words of reusable inspection, parsing, and verification helpers in
tools.py(Table 5; Appendix D.§ 2).
| Model | Method | Package Words | Validation () | Post-Peak EVOSKILL | Held-Out Test () | ||||
| SKILL.md | tools.py | Pass | OutTok | Next 6 Checkpoints | Pass | OutTok | |||
| Claude Opus 5.5 | Seed () | 20 | 0 | 17.5% | 72.1k | – | – | – | |
| EVOSKILL () | 1,521 | 0 | 82.5% | 71.7k | 72.5–80.0% (66k–85k) | 88.57% | 410.4k | ||
| EFFICIENTSKILL | 785 | 554 | 75.0% | 35.7k | – | 88.57% | 212.9k | ||
| Gemini 3.8 Flash | Seed () | 20 | 0 | 2.5% | 496.5k | – | – | – | |
| EVOSKILL () | 1,271 | 0 | 50.0% | 599.4k | 2.5–45.0% (568k–700k) | 58.21% | 3,745.1k | ||
| EFFICIENTSKILL | 521 | 241 | 50.0% | 540.5k | – | 72.50% | 2,342.0k | ||
| Grok 4.1 Fast | Seed () | 20 | 0 | 15.0% | 201.3k | – | – | – | |
| EVOSKILL () | 908 | 0 | 67.5% | 445.3k | 10.0–57.5% (238k–527k) | 57.50% | 2,815.8k | ||
| EFFICIENTSKILL | 703 | 520 | 67.5% | 197.8k | – | 75.00% | 1,195.8k | ||
| Gemma 4 26B | Seed () | 20 | 0 | 17.5% | 108.2k | – | – | – | |
| EVOSKILL () | 1,198 | 0 | 42.5% | 156.4k | 32.5–40.0% (144k–204k) | 47.14% | 1,309.5k | ||
| EFFICIENTSKILL | 698 | 407 | 37.5% | 92.6k | – | 45.71% | 608.6k |
Table 5. Skill-package word counts and validation/test progression on SPREADSHEETBENCH (, ). “Next 6 Checkpoints” shows the validation pass rate and output-token range across candidate skills evaluated after ; none beat , so EVOSKILL retains its skill. On Gemma 4 26B, (42.5%), and rounding the 10% allowance ( tasks) sets the discrete validation floor to 15/40 (37.5%).
Why the Starting Point for Efficiency Optimization Matters
What if Phase 2’s pruning prompt and static 0.90 gate are applied directly from without Phase 1 warm-starting? In Figure 3 (Appendix § 11.§ 1), early length pressure suppresses exploratory task rules: even the best candidate proposals prior to gate rejection stall at 20.0% on SpreadsheetBench (vs. 50.0% unconstrained) and 71.8% on ALFWorld (vs. 89.7%), failing the 0.90 gate until patience reverts to (3.6% and 43.2% test pass, even at , vs. 72.5% and 81.1% warm-started).
Figure 3. Starting Point Matters. Pruning directly from seed stalls and reverts (), whereas warm-starting from (EfficientSkill) reaches 72.5% and 81.1%.
Input Tokens & Serving Cost
Table 2 summarizes skill instruction compression, billed input-token reductions, and serving-cost savings across all 12 pairs (Appendix § 11.4):
Lower Input Tokens and Serving Cost: Shorter skill instructions (20–59% fewer words in
SKILL.md), offloadedtools.pyhelpers, and fewer turns per trajectory cut cumulative input tokens by 8–66% and lower standard serving cost across all 12 pairs (25–67% on 10 of 12).
Savings Hold Under Prefix Caching: Even when discounting reused conversation prefixes at provider KV-cache rates (Appendix § 11.4, Table 8), serving cost still drops on all 12 pairs (31–75% on 10 of 12) because pricier output tokens and newly appended turn observations remain uncached (Phase 2 payback in Table 9).
Table 8: Held-out test token usage and per-episode serving cost under unconstrained EVOSKILL vs. EFFICIENTSKILL ( tasks per benchmark), expanding Table 2. “Turn-1” () is mean input tokens on Turn 1; “Provider KV (KV-Prov)” discounts the reused conversation prefix on turns at each provider’s published cached-read rate (see pricing note below). Bold marks strictly lower values per pair.
| Benchmark | Model | Method | Turns () | Turn-1 () | Standard Billing $/ep | | Provider KV (KV-Prov) $/ep | ||||
| MATH-500 | Claude Opus 5.5 | EVOSKILL | 1.00 | 3,260 | 3,260 | 426 | 3,686 | $0.08085 | – | $0.08085 | – |
| MATH-500 | Claude Opus 5.5 | EFFICIENTSKILL | 1.00 | 2,084 | 2,084 | 102 | 2,186 | $0.03892 | −51.9% | $0.03892 | −51.9% |
| MATH-500 | Gemini 3.8 Flash | EVOSKILL | 1.76 | 2,307 | 4,264 | 1,734 | 5,998 | $0.00084 | – | $0.00072 | – |
| MATH-500 | Gemini 3.8 Flash | EFFICIENTSKILL | 1.04 | 1,781 | 1,867 | 543 | 2,410 | $0.00030 | −63.9% | $0.00030 | −58.4% |
| MATH-500 | Grok 4.1 Fast | EVOSKILL | 1.01 | 2,599 | 2,620 | 1,937 | 4,557 | $0.00149 | – | $0.00149 | – |
| MATH-500 | Grok 4.1 Fast | EFFICIENTSKILL | 1.00 | 2,389 | 2,398 | 1,665 | 4,063 | $0.00131 | −12.1% | $0.00131 | −12.0% |
| MATH-500 | Gemma 4 26B | EVOSKILL | 1.70 | 2,263 | 6,385 | 1,349 | 7,734 | $0.00155 | – | $0.00119 | – |
| MATH-500 | Gemma 4 26B | EFFICIENTSKILL | 1.19 | 1,643 | 2,670 | 806 | 3,476 | $0.00070 | −55.1% | $0.00062 | −47.8% |
| ALFWORLD | Claude Opus 5.5 | EVOSKILL | 10.21 | 2,972 | 33,627 | 933 | 34,560 | $0.57438 | – | $0.16606 | – |
| ALFWORLD | Claude Opus 5.5 | EFFICIENTSKILL | 7.33 | 1,333 | 11,680 | 376 | 12,056 | $0.20337 | −64.6% | $0.06759 | −59.3% |
| ALFWORLD | Gemini 3.8 Flash | EVOSKILL | 9.85 | 2,050 | 22,207 | 5,704 | 27,911 | $0.00338 | – | $0.00203 | – |
| ALFWORLD | Gemini 3.8 Flash | EFFICIENTSKILL | 7.16 | 1,231 | 10,086 | 1,155 | 11,241 | $0.00110 | −67.3% | $0.00052 | −74.5% |
| ALFWORLD | Grok 4.1 Fast | EVOSKILL | 9.15 | 2,640 | 25,922 | 12,761 | 38,683 | $0.01156 | – | $0.00811 | – |
| ALFWORLD | Grok 4.1 Fast | EFFICIENTSKILL | 8.71 | 1,884 | 18,170 | 12,350 | 30,520 | $0.00981 | −15.2% | $0.00740 | −8.7% |
| ALFWORLD | Gemma 4 26B | EVOSKILL | 10.74 | 2,367 | 28,534 | 95 | 28,629 | $0.00573 | – | $0.00314 | – |
| ALFWORLD | Gemma 4 26B | EFFICIENTSKILL | 9.12 | 1,667 | 18,011 | 99 | 18,110 | $0.00362 | −36.7% | $0.00202 | −35.8% |
| SPREADSHEET-BENCH | Claude Opus 5.5 | EVOSKILL | 2.80 | 4,062 | 15,475 | 1,466 | 16,941 | $0.34206 | – | $0.21998 | – |
| SPREADSHEET-BENCH | Claude Opus 5.5 | EFFICIENTSKILL | 3.04 | 2,165 | 11,525 | 761 | 12,286 | $0.22991 | −32.8% | $0.13911 | −36.8% |
| SPREADSHEET-BENCH | Gemini 3.8 Flash | EVOSKILL | 10.44 | 2,301 | 78,016 | 13,376 | 91,392 | $0.00986 | – | $0.00521 | – |
| SPREADSHEET-BENCH | Gemini 3.8 Flash | EFFICIENTSKILL | 8.56 | 2,302 | 64,759 | 8,364 | 73,123 | $0.00737 | −25.3% | $0.00360 | −31.0% |
| SPREADSHEET-BENCH | Grok 4.1 Fast | EVOSKILL | 2.71 | 2,195 | 9,440 | 10,056 | 19,496 | $0.00692 | – | $0.00613 | – |
| SPREADSHEET-BENCH | Grok 4.1 Fast | EFFICIENTSKILL | 1.97 | 1,915 | 5,140 | 4,271 | 9,411 | $0.00316 | −54.3% | $0.00289 | −52.9% |
| SPREADSHEET-BENCH | Gemma 4 26B | EVOSKILL | 5.41 | 2,314 | 34,882 | 4,677 | 39,559 | $0.00791 | – | $0.00512 | – |
| SPREADSHEET-BENCH | Gemma 4 26B | EFFICIENTSKILL | 3.39 | 1,339 | 11,986 | 2,174 | 14,160 | $0.00283 | −64.2% | $0.00207 | −59.6% |
Table 8. Pricing (per 1M uncached in / cached in / out tokens): Claude Opus 5.5 ($15.00 / $1.50 / $75.00, 90% cache discount); Gemini 3.8 Flash ($0.075 / $0.0075 / $0.30, 90% cache discount); Grok 4.1 Fast ($0.20 / $0.05 / $0.50, 75% cache discount); Gemma 4 26B ($0.20 / $0.10 / $0.20, 50% hosted cache discount). Relative changes use exact unrounded API costs. Under a uniform 90% prefix-cache discount across all four models, savings remain 31.0–74.5% on 10/12 pairs (−12.0%, −6.7%, −52.6% on Grok and −38.0%, −31.7%, −49.6% on Claude across MATH-500, ALFWORLD, and SPREADSHEET-BENCH).
Trajectory Analysis
Paired test traces clarify how guarded pruning reshapes multi-turn behavior (Appendix § 16):
Stopping No-Code Loops (Gemini 3.8 Flash): Under EvoSkill (1,271-word prompt, empty
tools.py), once Turn 1–2 inspection dumps cell layouts, reconciling long prose rules without callable utilities leaves 26.1% of turns (763/2,923) without executable code. Unexecuted turns leave the file unchanged in context, triggering self-reinforcing no-code loops (27% follow another no-code turn) and compounding with inline scripting errors (5.8% 10.4% next-turn error rate). EfficientSkill pairs concise rules with working utilities, cutting total unexecuted turns to 14.1% (338/2,398, mostly code-free completion turns; non-empty prose turns fall to 1.1%, 27/2,398) and 20-turn timeouts by 60% (45 18).
Preventing Unsaved Submissions (Grok 4.1 Fast): Under EvoSkill, verbose Turn-1 plans often emit the completion signal before saving edits, ending 51/280 runs (18.2%, 0% pass) on untouched files. EfficientSkill requires verified edits first, cutting early exits to 9/280 (3.2%) and recovering 40 of those 51 tasks (+49-task net gain).
Solving Tasks in Fewer Steps: Rather than
| Model | SKILL.md Words (EVOSKILL → Ours) | Input Tok. Change | Serving Cost Change |
| MATH-500 (Direct math derivation + fewer code retries) | |||
| Claude Opus 5.5 | 1,157 → 649 | −36.1% | −51.9% |
| Gemini 3.8 Flash | 1,073 → 863 | −56.2% | −63.9% |
| Grok 4.1 Fast | 1,111 → 781 | −8.4% | −12.1% |
| Gemma 4 26B | 1,134 → 757 | −58.2% | −55.1% |
| ALFWORLD (Compact search rules + fewer navigation steps) | |||
| Claude Opus 5.5 | 1,175 → 500 | −65.3% | −64.6% |
| Gemini 3.8 Flash | 1,135 → 656 | −54.6% | −67.3% |
| Grok 4.1 Fast | 1,265 → 753 | −29.9% | −15.2% |
| Gemma 4 26B | 1,341 → 880 | −36.9% | −36.7% |
| SPREADSHEETBENCH (Reusable helper functions + fewer editing turns) | |||
| Claude Opus 5.5 | 1,521 → 785 | −25.5% | −32.8% |
| Gemini 3.8 Flash | 1,271 → 521 | −17.0% | −25.3% |
| Grok 4.1 Fast | 908 → 703 | −45.6% | −54.3% |
| Gemma 4 26B | 1,198 → 698 | −65.6% | −64.2% |
Table 2. SKILL.md word counts (EVOSKILL → EFFICIENTSKILL), billed input-token change, and standard serving-cost change across all 12 held-out test configurations ().
Notes: Full-prefix provider KV-caching costs (31–75% savings on 10/12 pairs), tools.py word counts, and Phase 2 optimization payback appear in Appendices § 11.§ 2–§ 11.4 (Tables 5, 8, and 9).
piecemeal probing, EFFICIENTSKILL converges on SPREADSHEETBENCH to an inspect → compute-and-verify → submit loop (93.2% of Opus runs finish in 3 turns) backed by verification helpers. Across non-Opus SPREADSHEETBENCH models, this cuts both turns (18–37%) and tokens per turn (24–42%), while Opus halves per-turn output (−52%; App. § 11.§ 3).
Practical Takeaways
Get the skill working before compressing: First run unconstrained repair to reach validation accuracy , then prune under .
Move repeated code into helper scripts: On programmatic tasks, moving deterministic inspection, parsing, and verification code into workspace modules avoids prompt-body bloat and curbs inline scripting errors.
Cap output tokens to curb multi-turn drift: Bounding episodic output enforces compact multi-turn workflows, curbing timeout spirals, uncached output generation, and residual cached-read history costs.
Re-optimize for each target model: Because learned skills adapt to each model’s verbosity and error profile at modest Phase 2 cost, re-run guarded pruning when switching models.
Conclusion
When skill optimizers chase accuracy alone, they keep patching edge cases until over-fitted skills pull agents into hesitant deliberation and costly retries. Yet penalizing token use too early backfires, because learning an unfamiliar task initially requires both success and failure trajectories to see what works well. EFFICIENTSKILL resolves this in two phases: Phase 1 learns a working skill without token limits, and Phase 2 refines that skill and its helper tools to produce shorter, token-efficient trajectories while keeping validation accuracy near its peak. Across four models and three benchmarks, replacing brittle patches with compact skills and reusable code helps agents generalize, raising mean test accuracy on every benchmark. At the same time, trajectories with fewer turns and less generated output cut serving cost in every setting, even under full-prefix caching, which discounts only reread history. Optimizing skills for token-efficient execution improves reliability rather than trading it away, once optimization first learns how to solve a task before learning how to solve it concisely.
Limitations
Optimizing Output Tokens Instead of Dollar Cost: Guarded pruning gates candidate acceptance on episodic output tokens so learned packages remain pricing-agnostic across providers, rather than optimizing a provider-specific dollar-cost objective. Although output compression and turn reduction lowered both uncached and full-prefix cached serving costs across all 12 evaluated pairs, fixed-horizon workflows where a candidate substantially expands prompt instructions could in principle raise uncached input replay cost; gating directly on weighted API spend is a straightforward extension.
Meta-Optimizer Cost, Output Cap, and Baseline Scope: Both phases query Claude Opus 5.5 as the reflective meta-optimizer under a shared 5,000-token output cap. While target-model rollout spend () is under $3.60 on 8 of 9 non-Opus runs, Opus meta-optimizer calls add $1.80–$9.36 per run, dominating offline dollar cost on inexpensive single-turn runs (Table 9); moreover, when unconstrained EVOSKILL expands SKILL.md past 900 words on SPREADSHEETBENCH, the 5,000-token cap truncates trailing tools_py drafts (though trace audits in Appendix D.§ 2 show missing-tools errors affect 1.1 accuracy points). Finally, Phase 2 jointly compresses SKILL.md and populates tools.py; evaluating larger reflection token budgets, isolated prompt-only pruning (tools.py disabled), post-hoc static compressors ([7]), and soft running-incumbent regularizers ((Zhao et al., 2025; Shah et al., 2026)), as well as open-ended tasks graded by noisy model judges, remains future work.
Evaluation Environments, Verifiers & Hyperparameters
Benchmarks, Execution Environment & Grading
Table 3: Benchmark specifications, frozen 20%/10%/70% dataset splits (seed 42; 1,198 unique tasks: 400-task SPREADSHEETBENCH-Verified, first 80/40/280 of shuffled MATH-500 for matched , and 398 ALFWORLD games balanced across all 6 task types), action spaces with per-episode turn caps, and deterministic binary verifiers.
| Benchmark | Train / Val / Test | Action Space & Turn Cap | Deterministic Verifier () |
| MATH-500 | 80 / 40 / 280 | Optional Python code block or final (up to 10 turns) | Exact string match or symbolic equivalence via SymPy (3 s timeout). |
| ALFWORLD | 79 / 39 / 280 | One TextWorld command per turn (up to 25 turns) | Simulator goal check run after every step. |
| SPREADSHEETBENCH | 80 / 40 / 280 | Python script (openpyxl) editing target workbook, or completion signal (up to 20 turns) | Cell-by-cell cached-value match against reference workbook ( for floats; case-insensitive trimmed match for strings). |
Table 3. Benchmark specifications, frozen 20%/10%/70% dataset splits (seed 42; 1,198 unique tasks: 400-task SPREADSHEETBENCH-Verified, first 80/40/280 of shuffled MATH-500 for matched , and 398 ALFWORLD games balanced across all 6 task types), action spaces with per-episode turn caps, and deterministic binary verifiers.
Stateless Evaluation Environment: Table 3 summarizes all three benchmarks (10, 25, and 20 maximum turns per episode on MATH-500, ALFWORLD, and SPREADSHEETBENCH). Rather than wrapping vendor-specific CLI agents (such as Claude Code or Codex), we run all four models in the same stateless multi-turn loop. Skill instructions (SKILL.md) pass as the system prompt, while helper scripts (tools.py) sit on disk in the episode workspace; full function bodies never enter the system prompt, whereas short call signatures in SKILL.md and printed runtime output count toward Turn-1 and cumulative input tokens ( and ; API and cache pricing in Table 8). Each turn resends the prior action and observation history (without past internal reasoning tokens, which providers strip automatically) and logs exact input, reasoning, and completion tokens. Code benchmarks extract fenced Python blocks from model replies, while ALFWORLD extracts a single TextWorld command.
Isolated Code Sandboxes: Each MATH-500 and SPREADSHEETBENCH episode runs in a fresh temporary working directory with standard input disabled. Subprocesses exceeding the per-turn timeout (10 s on MATH-500; 15 s on SPREADSHEETBENCH) are killed, and standard output and error streams are each capped at 4,000 and 8,000 characters, respectively.
SPREADSHEETBENCH Grading: Each episode places the initial workbook at input.xlsx and instructs the agent to edit and save input.xlsx in place before outputting SUBMIT: DONE. At submission, the grader opens input.xlsx via openpyxl (data_only=True) without running Excel or LibreOffice. Newly written formulas strings carry no cached value and read back as None, and edits saved only under another filename (such as output.xlsx) leave input.xlsx unchanged.
Optimization Schedule & State Updates
Mini-Batch Schedule & Validation Checkpoints: Both phases (Algorithm 1, § 3) train on mini-batches of tasks for up to epochs (10 batches per 80-task epoch; on ALFWORLD, batch 10 has 7 tasks). Four validation checkpoints run per epoch after batch indices ( for , spaced every 3, 2, 3, 2 batches), with early-stopping patience checkpoints (15 training batches). After every batch, the Claude Opus 5.5 meta-optimizer (5,000 max output tokens; Appendix § 12) proposes a JSON edit to the working skill without receiving prior rejection history; episodes with transient API or network errors are dropped before reflection. Between checkpoints (), carries forward directly to the next training batch. At checkpoints, accepted candidates update the incumbent ( or ) and reset patience to 0, whereas rejected candidates roll back to the incumbent and add 1 to the patience counter.
Code Compilation Check & Checkpoint Acceptance Rules: Whenever a proposed edit includes non-empty tools.py code, syntax and bytecode compilation are checked in an isolated subprocess
(5 s timeout), and invalid tools.py drafts are discarded before evaluation. Phase 1 (and unconstrained EVOSKILL) commits at a checkpoint when (or while still at the initial seed ), updating . Phase 2 warm-starts and sets a fixed discrete accuracy floor with . For example, on Gemma 4 26B SPREADSHEETBENCH (, ), tasks, giving . At each Phase 2 checkpoint, a candidate with valid tools.py and is committed if and only if (i) (), (ii) and , or (iii) and . Accepted candidates update the running incumbent , while stays fixed to Phase 1’s .
Ablations, Turn-vs.-Token Breakdown & Serving Cost
Pruning Directly from the Seed vs. Starting from a Learned Skill
Setup and Why Pruning from the Seed Stalls: This ablation applies Phase 2’s pruning meta-prompt and fixed 0.90 validation gate directly to the minimal seed skill with , skipping Phase 1 discovery. Note that this isolates the effect of warm-starting under our fixed-gate protocol rather than benchmarking soft running-incumbent penalties (Zhao et al., 2025; Shah et al., 2026). Two failure modes compound when starting accuracy is low. First, penalizing output length from the start discourages the exploratory rules needed to learn environment conventions, so even the best candidate skills evaluated prior to gate rejection stall well below unconstrained Phase 1 discovery (20.0% vs. 50.0% on SPREADSHEETBENCH; 71.8% vs. 89.7% on ALFWORLD). Second, because updates from cannot clear the 0.90 gate within 2–3 provisional batches, every checkpoint rejects the candidate and rolls back to until patience () runs out.
Starting from vs. Starting from (Table 4): On ALFWORLD with Gemma 4 26B, pruning directly from peaks at 71.8% () and 64.1% () among candidate proposals before reverting to (43.2% test accuracy), versus 81.1% when warm-started from . On SPREADSHEETBENCH with Gemini 3.8 Flash, candidate proposals from peak at 20.0% on validation before reverting to (3.6% test accuracy), whereas warm-starting from reaches 72.5%.
| Benchmark / Model | Optimization Method | Batch | Val Pass | Test Pass | Test OutTok (Per-Ep) |
| ALFWORLD (Gemma 4 26B, 0.9 ) | EVOSKILL (Accuracy Only, ) | 8 | 89.7% | 81.4% | 26.7k (95.2) |
| ALFWORLD (Gemma 4 26B, 0.9 ) | Prune Directly from Seed () | 8 | 71.8% | 43.2% | – (reverts to ) |
| ALFWORLD (Gemma 4 26B, 0.9 ) | Prune Directly from Seed () | 24 | 64.1%* | 43.2% | – (reverts to ) |
| ALFWORLD (Gemma 4 26B, 0.9 ) | EFFICIENTSKILL (Start from ) | 8 | 84.6% | 81.1% | 27.8k (99.4) |
| SPREADSHEETBENCH (Gemini 3.8 Flash, 0.9 ) | EVOSKILL (Accuracy Only, ) | 8 | 50.0% | 58.2% | 3,745.1k (13,376) |
| SPREADSHEETBENCH (Gemini 3.8 Flash, 0.9 ) | Prune Directly from Seed () | 8 | 20.0%* | 3.6% | – (reverts to ) |
| SPREADSHEETBENCH (Gemini 3.8 Flash, 0.9 ) | EFFICIENTSKILL (Start from ) | 8 | 50.0% | 72.5% | 2,342.0k (8,364) |
Table 4. Pruning directly from the minimal seed skill () vs. warm-starting from (EFFICIENTSKILL). “Val Pass” shows the selected skill’s validation accuracy; for seed-pruning runs that roll back to , * marks the highest validation accuracy reached by any rejected checkpoint candidate. “Test OutTok” reports cumulative output tokens across tasks, with the per-episode mean in parentheses.
Search Progression, Helper Functions & Accuracy-Floor Sensitivity
Running EVOSKILL Longer Does Not Help (Table 5): On SPREADSHEETBENCH, running unconstrained EVOSKILL for 6 extra validation checkpoints past (15 training batches, one full patience window ) fails to improve validation accuracy on any model while SKILL.md remains bloated at 908–1,521 words with an empty
tools.py. Under the shared 5,000-token reflection cap, long SKILL.md drafts crowd out the trailingtools_pyJSON field, whereas trimming SKILL.md in Phase 2 frees budget to populatetools.py. Held-out trace audits confirm that uncaught tools import errors occur on only 0–6.1% of EVOSKILL test tasks, with negligible accuracy impact (Appendix D.§ 2).
Compressing SKILL.md and Populating tools.py: On SPREADSHEETBENCH, guarded pruning trims SKILL.md to 521–785 words and populates 241–554 words of reusable helpers in tools.py (Table 5). During Claude Opus 5.5 Phase 2 search on SPREADSHEETBENCH, adding a 305-word tools.py module between consecutive Epoch 1 checkpoints lifted validation accuracy from 62.5% to 77.5% while cutting validation output tokens by 44.8% (78.7k → 43.5k). On MATH-500, where Python is optional, tools.py moves from 197–403 to 352–502 words on Gemini, Grok, and Gemma, and drops from 252 to 0 words on Opus as its skill shifts to direct symbolic derivation (ALFWORLD executes text commands rather than Python; Appendix D.4).
Accuracy-Floor Sensitivity (Table 6): Across all 12 model–benchmark pairs, the default 10% relative retention floor () and a 5-point absolute slack floor () select identical or adjacent checkpoints within test accuracy points and 0–5.3 percentage points of output-token savings.
| Benchmark | Model | Val (%) | EVOSKILL () Test (%) | EVOSKILL () OutTok | 10% Relative Floor Test (%) | 10% Relative Floor OutTok | 5%-Point Slack Floor Test (%) | 5%-Point Slack Floor OutTok |
| MATH-500 | Claude Opus 5.5 | 67.5 → 75.0 | 86.43 | 119.2k | 87.50 | 28.6k | 87.86 | 32.4k |
| Gemini 3.8 Flash | 75.0 → 77.5 | 85.00 | 485.5k | 87.50 | 152.1k | 86.79 | 149.7k | |
| Grok 4.1 Fast | 57.5 → 60.0 | 72.50 | 542.3k | 72.50 | 466.1k | 73.57 | 471.4k | |
| Gemma 4 26B | 72.5 → 72.5 | 80.36 | 377.8k | 83.57 | 225.6k | 83.57 | 205.5k | |
| ALFWORLD | Claude Opus 5.5 | 84.6 → 92.3 | 88.93 | 261.3k | 93.21 | 105.2k | 92.86 | 97.9k |
| Gemini 3.8 Flash | 89.7 → 92.3 | 92.14 | 1,597.1k | 93.57 | 323.5k | 93.21 | 272.6k | |
| Grok 4.1 Fast | 61.5 → 89.7 | 89.29 | 3,573.0k | 88.21 | 3,458.1k | 88.21 | 3,359.4k | |
| Gemma 4 26B | 43.6 → 89.7 | 81.43 | 26.7k | 81.07 | 27.8k | 81.07 | 27.8k | |
| SPREADSHEETBENCH | Claude Opus 5.5 | 17.5 → 82.5 | 88.57 | 410.4k | 88.57 | 212.9k | 88.57 | 212.9k |
| Gemini 3.8 Flash | 2.5 → 50.0 | 58.21 | 3,745.1k | 72.50 | 2,342.0k | 69.29 | 2,285.8k | |
| Grok 4.1 Fast | 15.0 → 67.5 | 57.50 | 2,815.8k | 75.00 | 1,195.8k | 74.64 | 1,239.6k | |
| Gemma 4 26B | 17.5 → 42.5 | 47.14 | 1,309.5k | 45.71 | 608.6k | 47.86 | 551.0k |
Table 6. Validation progression () and held-out test accuracy and cumulative output tokens () under unconstrained EVOSKILL (), the default 10% relative floor (), and the 5%-point absolute slack floor ().
Turn Reduction vs. Per-Turn Output Compression
Define mean output tokens per turn as , so mean output tokens per episode factor as . Table 7 breaks each per-episode output reduction () into fewer turns per trajectory ( Turns) and shorter responses per turn ( Tok/T).
| Benchmark | Model | (Tok/T) | Turns | Tok/T | ||
| MATH-500 | Claude Opus 5.5 | 1.00 → 1.00 | 426 → 102 | 0.0% | −76.0% | −76.0% |
| Gemini 3.8 Flash | 1.76 → 1.04 | 985 → 522 | −40.9% | −47.0% | −68.7% | |
| Grok 4.1 Fast | 1.01 → 1.00 | 1,918 → 1,665 | −0.4% | −13.8% | −14.1% | |
| Gemma 4 26B | 1.70 → 1.19 | 794 → 677 | −30.2% | −14.5% | −40.3% | |
| ALFWORLD | Claude Opus 5.5 | 10.21 → 7.33 | 91 → 51 | −28.2% | −43.9% | −59.7% |
| Gemini 3.8 Flash | 9.85 → 7.16 | 579 → 161 | −27.2% | −72.2% | −79.7% | |
| Grok 4.1 Fast | 9.15 → 8.71 | 1,395 → 1,418 | −4.8% | +1.6% | −3.2% | |
| Gemma 4 26B | 10.74 → 9.12 | 8.9 → 10.9 | −15.1% | +22.9% | +4.4%1 | |
| SPREADSHEETBENCH | Claude Opus 5.5 | 2.80 → 3.04 | 524 → 250 | +8.7% | −52.3% | −48.1% |
| Gemini 3.8 Flash | 10.44 → 8.56 | 1,281 → 977 | −18.0% | −23.8% | −37.5% | |
| Grok 4.1 Fast | 2.71 → 1.97 | 3,711 → 2,168 | −27.3% | −41.6% | −57.5% | |
| Gemma 4 26B | 5.41 → 3.39 | 865 → 641 | −37.3% | −25.8% | −53.5% |
Table 7. Breakdown of per-episode output-token changes () from unconstrained EVOSKILL to EFFICIENTSKILL into mean turns () and mean output tokens per turn (). Relative changes are computed from exact unrounded trajectory averages prior to display rounding.
Multi-Turn Input Billing, Prefix Caching & Optimization Payback
Input-Token Billing With and Without Prefix Caching (Table 8): Table 8 provides the per-episode token totals and dollar costs behind Table 2 (§ 5.4). Over an episode of turns, let denote the input tokens sent at turn . Turn 1 sends tokens (skill instructions plus the task description), and each turn appends new action–observation tokens (– per turn on ALFWORLD vs. – on SPREADSHEETBENCH), so cumulative input grows linearly with and quadratically with . Given per-token rates , we compare two serving regimes across tasks:
Standard Uncached Billing: .
Provider Prefix KV Caching (KV-Prov): Because our loop is append-only, the entire prior prefix hits the KV cache on turns , totaling cached-read tokens and leaving tokens (Turn 1 plus new turn observations) billed at the full input rate. Applying each provider’s published cache discount gives . Because generated output (1×–5× pricier than uncached input) and newly appended observations are never cached, token-efficient trajectories cut on all 12 pairs, and by 31.0–74.5% on 10 of 12 (matching the 31.0–74.5% range under a uniform 90% discount in Table 8’s note).
Phase 2 Optimization Cost and Payback (Table 9): Because Phase 1 discovery is shared with EvoSKILL, measures the extra Phase 2 target-model spend across training mini-batches and validation checkpoints. Dividing by per-episode serving savings yields break-even volume , which is 257–2,063 standard episodes (734–2,860 under KV-Prov) on 10 of 12 pairs. Adding estimated Claude Opus 5.5 meta-optimizer spend across all Phase 2 batches (–), Phase 2 still pays back within 263–2,414 standard episodes (989–4,515 under KV-Prov) on 8 of 12 pairs (7 of 8 multi-turn runs plus Opus on MATH-500 at 1,251). The three non-Opus MATH-500 runs save under $0.001 per episode and therefore need 5.1k–23.9k episodes to amortize Opus reflection calls, while Grok on ALFWORLD (−3.2% output tokens) requires 8.4k standard or 20.9k KV-Prov episodes.
Optimizer Meta-Prompts
EfficientSkill (§ 3) and our matched EvoSkill baseline call the same Claude Opus 5.5 meta-optimizer (5,000 max output tokens) after every 8-task training batch (). Proposed edits carry forward between batches and are accepted or rolled back on at 4 checkpoints per epoch (Appendix § 10.§ 2). Phase 1 (REPAIR, Skill Discovery, which uses the same prompt as EvoSkill) and Phase 2 (PRUNE, Accuracy-Guarded Pruning) apply the identical fixed template across all three benchmarks, sharing the same anti-memorization rules, tools.py instructions, and JSON output schema so only the objective block changes between phases.
Skill Discovery & EvoSkill Meta-Prompt — Unconstrained Accuracy Objective
SYSTEM PROMPT
You are an expert agent skill optimizer.
OBJECTIVE: ACCURACY-ONLY MAXIMIZATION (UNCONSTRAINED)
Your objective is unconstrained task success. Maximize domain coverage and pass rate by diagnosing all failure modes in the batch. Provide clear, robust procedural checklists, edge-case guardrails, and error recovery protocols so failed tasks succeed. Output tokens are unconstrained; prioritize functional correctness and domain robustness.
ANTI-MEMORIZATION RULES (BENCHMARK-AGNOSTIC):
GENERALIZE, NEVER MEMORIZE: Do NOT include task-specific coordinates, exact cell names/ranges, sheet names, hardcoded column letters, or one-off code snippets from specific tasks.
ABSTRACT INVARIANT PRINCIPLES: Formulate principles as general reusable engineering workflows (e.g. dynamic column inspection, schema checks, null/type coercion, defensive range boundaries, verification of written files).
COMPACT & DENSE: Keep
SKILL.mdconcise. Prune redundant verbosity; every sentence must convey transferable operational knowledge.
REUSABLE CODE IN tools.py: Put reusable executable helper routines in the JSON field "tools_py" (materialized on disk as tools.py in the agent’s working directory when non-empty) rather than inlining large code blocks into SKILL.md. Keep SKILL.md focused on high-level instructions and workflows.
USER PAYLOAD & STRICT JSON OUTPUT SCHEMA:
# Current SKILL.md: {current_skill} | # Current tools.py: {current_tools}
# Batch Execution Trajectories (8 tasks): [{task_id, instruction, success,
output_tokens, thinking_tokens, turns_taken,
steps: [{turn, action, observation}]}]
{"feedback": "<root causes & fixes>", "skill_md": "<updated SKILL.md>", "tools_py": "<optional
Python>"}
Accuracy-Guarded Pruning & Tool Offloading Meta-Prompt — EfficientSkill Pruning Objective
PRUNING OBJECTIVE BLOCK (replaces Unconstrained Accuracy Objective; anti-memorization & tools.py schema identical):
OBJECTIVE: MULTI-OBJECTIVE PARETO EFFICIENCY (ACCURACY + TOKEN REDUCTION)
Your objective is to optimize the Pareto frontier: maintain high accuracy while aggressively compressing instructions and reducing agent execution tokens.
Eliminate procedural redundancy, verbose conversational scaffolding, and excessive interaction turns.
Encourage minimal-turn solutions (e.g. inspect and solve in 2 turns instead of 5). Prune verbal fluff while preserving critical constraints.
External Harness Checks: Bytecode compilation of tools.py and validation acceptance gates (, ) are enforced by the evaluation harness outside the meta-prompt; see Appendix § 10.§ 2 for exact acceptance rules.
Learned Skill Packages (SKILL.md and tools.py)
Minimal Seed Package (S0)
Every optimization run initializes from an empty tools.py (0 words) and a minimal 20-word SKILL.md stub containing only YAML metadata and the benchmark submission line (zero task heuristics):
SKILL.md
---
name: spreadsheet-bench_model
description: Evolved skill for spreadsheet-bench.
---
Active Skill: spreadsheet-bench_model
Submission Protocol: When finished, output: SUBMIT: DONE
Benchmark protocol lines: MATH-500: Enclose final answer in \boxed{...}; ALFWORLD: Complete household task by navigating receptacles and manipulating objects.
Gemini 3.8 Flash on SpreadsheetBench: EvoSkill vs. EfficientSkill
Without token penalties, EvoSkill expands SKILL.md to 1,271 words of checklists and 11 Excel recipes while leaving tools.py empty (0 words across all four SpreadsheetBench models). EfficientSkill shrinks SKILL.md by 59% (521 words) and builds a 241-word tools.py module, lifting test accuracy from 58.2% to 72.5% and cutting output tokens by 37.5%.
Why EvoSkill Leaves tools.py Empty (Trace Audit): Both phases share a 5,000-token meta-optimizer cap covering reasoning plus JSON fields ordered feedback, skill_md, and tools_py (Appendix C). When EvoSkill grows SKILL.md to 908–1,521 words, trailing tools_py drafts hit the cap and parse as empty; shrinking SKILL.md to 521–785 words in Phase 2 leaves room for 241–554 words of helpers. The leftover from tools import * line in Gemini’s EvoSkill SKILL.md (paired with an inline openpyxl fallback) did not handicap the baseline: uncaught tools import errors occur on 8/280 Gemini test tasks (2.9%), passing at 62.5% (5/8) vs. 58.1% (158/272) on the other 272, and on 0/280 (Opus), 1/280 (Gemma), and 17/280 (6.1%, 52.9% vs. 57.8% pass) on Grok (≤ 1.1 points impact across models). EvoSkill’s losses instead come from no-code loops and unsaved submissions (Appendix E).
SKILL.md (Excerpt: lines 1–29 & recipe headers) Gemini 3.8 Flash | EvoSkill | 1,271 words | tools.py: 0 words
---
name: spreadsheet-bench_gemini_3.8_accuracy
description: Workflow for spreadsheet-bench tasks (formula/VBA/macro questions answered by
modifying input.xlsx and saving the solved workbook with real values). Use for any
Excel manipulation task.
---
# Spreadsheet-Bench Solver
## RULE 0: Every reply = one python block; write the answer early
Failures in practice come from replies with NO code and from re-inspecting forever. Both score
zero.
- Every reply MUST contain exactly one fenced ```python block, until the final reply.
- Start the visible reply with the code block. Do not deliberate at length, then emit only
prose.
- If unsure, still emit code that writes your best guess.
**Turn budget (hard)**
- **Turn 1:** `from tools import *`, then `inspect_wb('input.xlsx')`, then
`shutil.copy('input.xlsx', out)`.
- **Turn 2:** compute the answer, write it, save, and verify.
- **Turn 3:** fix only what verification showed was wrong.
- **Final reply:** `SUBMIT: DONE` alone, with no code.
- If `import tools` fails, inline a minimal openpyxl version and continue.
## Core facts about grading
- There is NO LibreOffice or recalc engine. A formula written with openpyxl has no cached value
and reads back as None, which counts as wrong. Always write the COMPUTED static value.
... [Lines 31–73: 6-step Workflow + 6 Default deliberation decisions omitted for brevity] ...
## Recipes
- **TEXT/weekday formula fix:** Target cells = stated range ... (`weekday_name(d,
'ddd'|'dddd')`) ...
- **IF over mixed cells (date / 'Yes' / 'N/A' / blank):** Classify each source cell ...
- **SUMPRODUCT/2-D matrix (date window × header category × row key):** Build boolean masks ...
- **Consecutive negative runs → merge:** Group by key ... maximal runs with change < 0 ...
- **Recreate sheet / formatting task:** Copy values, formulas and styles with `copy_cell_style`
...
````
Lookup (INDEX/MATCH, 2-D) | Durations | Modify an array formula | Transpose blocks | Delete columns/rows | Sorting | Suffix numbering ... [540 words of prose recipes]
SKILL.md (Complete Verbatim) Gemini 3.8 Flash | EFFICIENTSKILL | 521 words
````text
name: spreadsheet-bench_gemini_3.8_pareto description: Minimal-turn workflow for Excel benchmark tasks. Computes static values and structure edits in Python and saves verified workbooks in 1-2 turns.
Spreadsheet-Bench Solver
Mandatory Execution Rules
Always Output Code: Every turn must contain a single executable Python code block. Never output commentary without code or leave actions empty.
Target 1-2 Turn Completion:
Turn 1: Inspect workbook structure, run required transformations/computations in Python, write static values or formulas to the worksheet, save the workbook, and print a summary verification.
Turn 2: If Turn 1 succeeded and output was saved correctly, output
SUBMIT: DONE. If minor corrections are needed, execute the fix and submit.
Handle Macro/VBA Instructions in Python: Instructions asking for macros or VBA code require applying the described data manipulation directly to the Excel output file using Python.
Write Computed Static Values: Excel evaluators check computed cell values. Unless explicit formula strings are demanded, compute values (numbers, strings, dates) directly in Python and write them to cells. Always preserve existing formatting.
Data Preservation: Do not open workbooks with
data_only=Truewhen saving edits. Usedata_only=Truesolely for inspecting pre-calculated values.
Helper Toolkit (tools.py)
Import pre-built routines from tools:
from tools import inspect_wb, output_path, load_both, norm, to_date, save_updates,
copy_cell_style
inspect_wb(path='input.xlsx'): Summarizes sheets, dimension, non-empty rows, formulas, and values.output_path(instruction_text): Extracts output filename (defaults tooutput.xlsx).load_both(path): Returns(wb_formula, wb_value)objects.norm(val): Strips outer whitespace and normalizes internal spaces.to_date(val): Converts strings/datetimes intodatetime.dateobjects.save_updates(src_path, out_path, update_dict): Batch writes{sheet: {cell_ref: value}}.
Turn-1 Standard Implementation Flow
import openpyxl
from tools import inspect_wb, output_path, norm, to_date
out_file = output_path("TASK_INSTRUCTION")
print("Target file:", out_file)
print(inspect_wb("input.xlsx"))
wb = openpyxl.load_workbook("input.xlsx")
ws = wb.active # or select sheet by name
# Perform computation / row deletions / header lookups in Python, then save:
wb.save(out_file)
print(f"Successfully saved to {out_file}")
Core Execution Patterns
Row / Header Matching & Column Lookups:
Map header text (row 1 or header row) to column indices using
norm(cell.value).Perform exact string or numerical match in Python dictionary/lists rather than relying on ````
→ un-evaluated Excel lookup formulas.
Min/Max/Conditional Filtering Across Columns:
Iterate over cell values in candidate ranges (e.g. columns B to N).
Parse numeric values safely (
float(v)ifisinstance(v, (int, float))andv > 0).Join matching header names with specified delimiters (e.g.,
,) if multiple candidates → match.
Row Deletion & Table Structure Re-ordering:
When deleting rows (e.g. clearing blocks before specific markers), iterate rows in → reverse order (
range(ws.max_row, 0, -1)) or build a new list of desired rows to → prevent index shifting during deletion.
Group Aggregations & Rank Computing:
Read non-empty data into a list of tuples/dicts.
Group by category/person/block, compute relative ranks or max values in Python, and assign → target outputs directly to corresponding cells.
tools.py (Complete Verbatim) Gemini 3.8 Flash | EFFICIENTSKILL | 241 words | 7 helpers
import os
import re
import datetime
import openpyxl
from openpyxl.styles import PatternFill, Font, Alignment, Border
def norm(v):
if v is None:
return ""
s = str(v).strip()
return re.sub(r'\s+', ' ', s)
def to_date(v):
if isinstance(v, datetime.datetime):
return v.date()
if isinstance(v, datetime.date):
return v
if isinstance(v, str):
for fmt in ('%Y-%m-%d', '%m/%d/%Y', '%d/%m/%Y', '%Y/%m/%d'):
try:
return datetime.datetime.strptime(v.strip(), fmt).date()
except ValueError:
pass
return v
def output_path(text=""):
if text:
m = re.search(r'[\'"]?([a-zA-Z0-9_\-]+\.xlsx)[\'"]?', text)
if m and m.group(1).lower() != 'input.xlsx':
return m.group(1)
return 'output.xlsx'
def load_both(path='input.xlsx'):
wb_f = openpyxl.load_workbook(path, data_only=False)
wb_v = openpyxl.load_workbook(path, data_only=True)
return wb_f, wb_v
def inspect_wb(path='input.xlsx'):
wb_f, wb_v = load_both(path)
res = []
for sheet in wb_f.sheetnames:
ws_f = wb_f[sheet]
ws_v = wb_v[sheet]
res.append(f"=== Sheet: {sheet} (max_row={ws_f.max_row}, max_col={ws_f.max_column}) ===")
for r in range(1, min(ws_f.max_row + 1, 40)):
row_f = [ws_f.cell(r, c).value for c in range(1, min(ws_f.max_column + 1, 20))]
row_v = [ws_v.cell(r, c).value for c in range(1, min(ws_v.max_column + 1, 20))]
if any(x is not None for x in row_f) or any(x is not None for x in row_v):
res.append(f"R{r} Form: {row_f}")
res.append(f"R{r} Val : {row_v}")
return "\n".join(res)
def save_updates(src_path, out_path, update_dict):
wb = openpyxl.load_workbook(src_path)
for sheet_name, cell_dict in update_dict.items():
if sheet_name in wb.sheetnames:
ws = wb[sheet_name]
for cell_ref, val in cell_dict.items():
ws[cell_ref] = val
wb.save(out_path)
return f"Saved updates to {out_path}"
63 def copy_cell_style(src_cell, dst_cell):
64 if src_cell.has_style:
65 dst_cell.font = Font(name=src_cell.font.name, size=src_cell.font.size, bold=src_cell.font.bold,
↳ italic=src_cell.font.italic, color=src_cell.font.color)
66 dst_cell.fill = PatternFill(fill_type=src_cell.fill.fill_type, start_color=src_cell.fill.start_color,
↳ end_color=src_cell.fill.end_color)
67 dst_cell.border = Border(left=src_cell.border.left, right=src_cell.border.right, top=src_cell.border.top,
↳ bottom=src_cell.border.bottom)
68 dst_cell.alignment = Alignment(horizontal=src_cell.alignment.horizontal, vertical=src_cell.alignment.vertical,
↳ wrap_text=src_cell.alignment.wrap_text)
69 dst_cell.number_format = src_cell.number_format
Claude Opus 5.5 EfficientSkill Package on SpreadsheetBench
For Claude Opus 5.5, EfficientSkill (785-word SKILL.md, 554-word tools.py) learns a 3-turn loop (dump() → compute, write, and verify() → SUBMIT: DONE), finishing 93.2% (261/280) of test tasks in 3 turns at 88.6% accuracy (−48.1% output tokens).
---
name: spreadsheet-bench_claude_opus_pareto
description: Solve spreadsheet manipulation tasks in exactly 3 turns (inspect, one
compute/write/verify script, then SUBMIT: DONE) by writing computed static values into
the correct target cells with openpyxl.
---
# Spreadsheet Task Skill
## Hard rules
1. **Turn 3 is exactly `SUBMIT: DONE`.**
- After the verified save, send only that line.
- Never send a formula, VBA or explanation. Any prose is run as Python and causes a
SyntaxError.
2. **Every action before submitting is a Python script.**
- Put notes only in `#` comments.
- Each script is a fresh process, so re-import and reload every time.
3. **Treat the question as a task.** "How do I write a formula/macro/VBA..." means: put the
resulting values into the workbook.
4. **Write values, not formulas.**
- openpyxl does not cache formula results, so the grader would read None.
- Compute the result in Python and write literals, overwriting any existing (array) formulas
in the target region.
5. **Save to `input.xlsx`** unless told otherwise.
## Workflow
**Turn 1:** `from tools import *; dump()`.
Identify:
- the header row;
- the real last row (use `last_row`, because `max_row` can be inflated);
- worked examples, meaning already-filled target cells;
- the target region: columns named in the instruction, an empty or header-only column, a
labelled blank area, a formula column, or a named cell;
- the logic implied by existing formulas;
- the source of the input values.
Re-read the instruction for column letters, sheets, labels, thresholds, signs and formatting
requests.
**Turn 2 (a single script):**
1. Reload and compute the results.
2. Compare against the examples with `check_examples(ws, {coord: value})`. If they mismatch,
fix the logic in the same script before writing.
3. Write only the target cells.
4. Save.
5. Reload and print the target range compactly with `verify`.
Do not run a separate re-check turn.
Turn 3: SUBMIT: DONE
Interpretation rules
Formula-derived inputs.
dumpmay show-> Noneor stale cached values for formula cells.Recompute these from the underlying source data, using
norm_key/build_lookup.Never skip rows just because a cached value is None.
Fill the full natural range.
Write every row or group that has input keys, including the example rows, recomputed consistently.
"Only works for the first row" or "copy it down" means the missing rows are the real target.
Copy the examples' layout, orientation, separators and ordering.
Take strings from the canonical source (stripped), not from typos in the examples.
Location.
Use the columns the instruction names, the example columns, the empty-header column, or the named cell.
"New column in X", or writing to an occupied position, means
insert_col, which shifts existing data right.Never invent locations.
Blank vs. zero; no filler.
Follow the examples.
Otherwise use what the equivalent Excel formula would return; for example, SUMIFS and SUMPRODUCT give 0 when nothing matches.
Otherwise use None.
Never write '-', 'N/A' or '' unless the examples show exactly that.
Exactness.
Preserve source signs and magnitudes.
Do not round unless asked or the examples are rounded.
Implement the stated comparisons exactly (< vs <=) and apply multipliers and branches literally.
Use label strings exactly as given, including case.
For ties, join matches with the separator the instruction or examples specify.
Lookups.
Apply
norm_keyto both sides.With no match, leave the cell blank unless the examples show a fallback.
For aggregates and 2D matches, use explicit loops over the real data range.
For cross-sheet aggregation, iterate over sheets dynamically and skip the summary sheet.
Macro-style outputs (restructure, pivot, sort, filter, copy, delete rows):
Clear the target region, write compactly from the top, and set leftover cells to None.
For a sort, keep the header in place.
When copying, copy whole rows and preserve types.
When deleting rows, work bottom-up and check the other columns of the rows being deleted.
Groups and blocks: detect group boundaries dynamically, for example blank rows or key changes.
Merged cells: write to the top-left cell. Unmerge only if asked.
Types.
Write numbers as int or float, not numeric strings, unless the source or examples use text.
Write dates as datetime objects.
For malformed or None inputs, leave the result blank.
Formatting requests.
Apply them to every relevant cell. "Entire sheet" means all used cells; use
set_fontfor fonts.Do these in the same script as the value writes.
Data-validation warnings are harmless.
Efficiency
No exploratory turns.
Keep prints short.
Nothing after the save except
SUBMIT: DONE.
tools.py (Complete Verbatim)
import openpyxl
import re
from copy import copy
from openpyxl.worksheet.formula import ArrayFormula
def dump(path='input.xlsx', per_sheet=1500):
"""Print non-empty cells of every sheet (capped). Formulas are shown with their cached values."""
wb = openpyxl.load_workbook(path)
wv = openpyxl.load_workbook(path, data_only=True)
for ws in wb:
print('==', ws.title, ws.dimensions, 'merged:', [str(m) for m in list(ws.merged_cells.ranges)[:10]])
n = 0
for row in ws.iter_rows():
for c in row:
v = c.value
if v is None:
continue
if isinstance(v, ArrayFormula):
v = '{' + str(v.text) + '}'
if isinstance(v, str) and (v.startswith('=') or v.startswith('{=')):
print(c.coordinate, repr(v), '->', repr(wv[ws.title][c.coordinate].value))
else:
print(c.coordinate, repr(v))
n += 1
if n >= per_sheet:
break
if n >= per_sheet:
print('...truncated', ws.title)
break
def last_row(ws, col=None):
"""Real last non-empty row (optionally only in 1-based column col)."""
for r in range(ws.max_row, 0, -1):
cells = [ws.cell(r, col)] if col else ws[r]
if any(c.value is not None and not (isinstance(c.value, str) and not c.value.strip()) for c in cells):
return r
return 0
def norm_key(v):
"""Trim, collapse whitespace/NBSP, casefold, and turn 12.0 into '12'."""
if v is None:
return None
if isinstance(v, float) and v.is_integer():
v = int(v)
s = re.sub(r'\s+', ' ', str(v).replace('\xa0', ' ')).strip().casefold()
try:
f = float(s)
s = str(int(f)) if f.is_integer() else str(f)
except ValueError:
pass
return s
def find_col(ws, header, row=1):
"""1-based column whose header matches (normalized), else None."""
h = norm_key(header)
for c in range(1, ws.max_column + 1):
if norm_key(ws.cell(row, c).value) == h:
return c
return None
def build_lookup(ws, key_cols, val_col, start_row=2, end_row=None):
"""Map a tuple of normalized key-column values to the val_col value. The first occurrence wins."""
end_row = end_row or last_row(ws)
d = {}
for r in range(start_row, end_row + 1):
k = tuple(norm_key(ws.cell(r, c).value) for c in key_cols)
d.setdefault(k, ws.cell(r, val_col).value)
return d
def check_examples(ws, computed, tol=1e-6):
"""Compare computed {coord: value} against existing non-empty cells. Print mismatches; return their count."""
bad = 0
for coord, v in computed.items():
cur = ws[coord].value
if cur is None or (isinstance(cur, str) and cur.startswith('=')):
continue
83 ok = (abs(float(cur) - float(v)) <= tol) if isinstance(cur, (int, float)) and isinstance(v, (int, float)) else
norm_key(cur) == norm_key(v)
84 if not ok:
85 bad += 1
86 if bad <= 10:
87 print('MISMATCH', coord, 'have', repr(cur), 'got', repr(v))
88 print('example mismatches:', bad)
89 return bad
92 def clear_range(ws, min_row, max_row, min_col, max_col):
93 """Set every cell in the range to None."""
94 for r in range(min_row, max_row + 1):
95 for c in range(min_col, max_col + 1):
96 ws.cell(r, c).value = None
99 def set_font(ws, name=None, **kw):
100 """Apply a font name (and other attributes such as bold or size) to every cell in the used range."""
101 for row in ws.iter_rows(min_row=1, max_row=ws.max_row, max_col=ws.max_column):
102 for c in row:
103 f = copy(c.font)
104 if name:
105 f.name = name
106 for k, v in kw.items():
107 setattr(f, k, v)
108 c.font = f
111 def insert_col(ws, idx, header=None, header_row=1, fmt_from=None):
112 """Insert an empty column at 1-based idx (data shifts right) and optionally set its header.
113 fmt_from: source column index, read after the shift, whose header style and number formats are copied."""
114 ws.insert_cols(idx)
115 if header is not None:
116 ws.cell(header_row, idx).value = header
117 if fmt_from:
118 for r in range(1, ws.max_row + 1):
119 s, d = ws.cell(r, fmt_from), ws.cell(r, idx)
120 d.number_format = s.number_format
121 if r == header_row and s.has_style:
122 d.font = copy(s.font)
123 d.fill = copy(s.fill)
124 d.alignment = copy(s.alignment)
127 def verify(path, sheet, min_row, max_row, min_col, max_col):
128 """Reload the saved file and print the range compactly."""
129 ws = openpyxl.load_workbook(path)[sheet]
130 for row in ws.iter_rows(min_row=min_row, max_row=max_row, min_col=min_col, max_col=max_col, values_only=True):
131 print(row)
Gemini 3.8 Flash EfficientSkill Package on ALFWorld
On ALFWorld, agents issue short text commands rather than Python code, so EFFICIENTSKILL keeps tools.py empty (0 words) and compresses SKILL.md to 656 words covering search order and object aliases, cutting output tokens by 79.7% (5,704 → 1,155) at 93.6% test accuracy.
SKILL.md (Complete Verbatim) Gemini 3.8 Flash | ALFWorld EFFICIENTSKILL | 656 words | tools.py: 0 words
---
name: alfworld_gemini_3.8_pareto
description: Minimal-turn policy for text household tasks (find object, optionally
heat/cool/clean, place on target receptacle).
---
# ALFWorld Minimal-Turn Policy
Output exactly one valid action per turn.
- Keep thought to one line at most. Do not deliberate.
- Every pattern below is already solved, so apply it mechanically.
## Valid actions (exact syntax)
- `go to X`, `open X`, `close X`
- `take O from X`
- `put O in/on X` (always the literal "in/on")
- `heat O with microwave N`, `cool O with fridge N`, `clean O with sinkbasin N`
Nothing else works: no slice, use, toggle, examine or look.
Target object
Only the movable object that ends up on the final receptacle matters.
Containers: in phrases like "pan with Y", "bowl of Y" or "O in the pan", move the contents (Y/O), not the vessel. Fetch the vessel only if it is the only object named.
Slices: slice/piece/cut X means the whole X.
Ignore: adjectives, colors and source phrases.
Aliases: take the first match on sight, including aliases below. If there is no exact name, take the closest synonym on first sight. Never walk past it.
| Instruction word | Object name |
| glass | cup |
| keys | keychain |
| magazine/novel | book |
| tissues | tissuebox |
| sponge | dishsponge |
| soap/bar | soapbar |
| green vegetable | lettuce |
| cell phone | cellphone |
| remote | remotecontrol |
| spray bottle | spraybottle |
| CD | cd |
| TV | television |
| large spoon | ladle/spoon |
| computer | laptop |
Table 1 (PDF p. 23).
Workflow (4-6 turns)
Find. Turn 1 is
go to countertop 1.If the object is not there, visit each location once: fridge (open), diningtable, sidetable/desk, cabinet/drawer (open), microwave, sinkbasin, then other receptacles.
Use only receptacle names you have observed.
State change right after
take, in place, with no travel:hot/heated/warm/cooked:
heat O with microwave 1cold/chilled/cooled:
cool O with fridge 1clean/wash/rinse:
clean O with sinkbasin 1
Place.
go tothe destination (open it if closed), thenputimmediately. Stop on "Goal completed successfully!"
Destination (head noun)
Use the literal receptacle if it exists (cabinet, fridge, sofa, toilet, shelf, etc.). Otherwise map it:
| Instruction word | Receptacle(s), in order |
| table | diningtable, sidetable, desk, coffeetable |
| nightstand/end table | sidetable, diningtable |
| coffee table | coffeetable |
| chair/couch | armchair/sofa |
| bin/trash | garbagecan |
| rack | cart |
| tub | bathtubbasin |
| toilet paper holder | toiletpaperhanger |
| sink | sinkbasin |
| counter | countertop |
| cupboard | cabinet |
| refrigerator | fridge |
| coffee maker | coffeemachine |
| stove | stoveburner |
Table 2 (PDF p. 23).
If nothing matches, use countertop, then diningtable.
Special patterns
Lamp/light tasks: take X, then put it on diningtable. Never search for a lamp.
Two X: put the first one. Fetch a second only if the goal did not complete.
## Failure handling (strict, no loops)
- **"You can't see any X":** X does not exist. Never retry it or guess other numbers.
- **A put did not complete the goal:** take the object back and go down this ladder once each:
1. The next mapped destination (another instance or type).
2. A different state change in place (the required one again if skipped; otherwise cool for food, clean otherwise), then put.
3. If the object was originally taken from the destination itself, look for another matching object elsewhere, apply the state change, then put.
4. Stop.
- **Never repeat** a put at the same receptacle more than twice.
- **Never cycle** close/open, re-heat, or revisit an already-checked location.
- **Budget:** about 12 turns. After that, or once every location has been checked, stop.
Side-by-Side Trajectory Comparisons Across Benchmarks
We compare held-out test trajectories under EVOSKILL () and EFFICIENTSKILL () on the same tasks across SPREADSHEETBENCH, ALFWORLD, and MATH-500 (§ 6).
SpreadsheetBench: Stopping No-Code Loops & Unsaved Submissions
Across 12,130 evaluated SPREADSHEETBENCH turns (10,730 in the main test matrix plus seed and ablation runs), two failure modes drive most accuracy losses under verbose, helper-free skills. First, on Gemini 3.8 Flash, EVOSKILL’s 1,271-word SKILL.md and empty tools.py leave 26.1% of turns (763/2,923) without executable code (27% following another no-code turn) while inline scripting errors raise next-turn error rates from 5.8% to 10.4%; EFFICIENTSKILL cuts unexecuted turns to 14.1% (338/2,398: 311 code-free completion turns and 27, or 1.1%, non-empty prose turns) and cuts 20-turn timeouts by 60% (45 18). Second, on Grok 4.1 Fast, long Turn-1 plans under EVOSKILL emit the completion signal before saving on 51/280 runs (18.2%, 0% pass); EFFICIENTSKILL cuts unsaved exits to 9/280 (3.2%) and recovers 40 of those 51 tasks (+49-task net gain).
No-Code Loop vs. Clean 6-Turn Solve (Gemini 3.8 Flash, Task 15380). Task 15380 extracts two-digit campaign codes from A3:A14 (e.g., "2014_09_D2_Reminder_EN" "09"), maps them via named ranges codes2 (D3:D14) and month (E3:E14), and writes month labels ("SEP", . . . , "NOV") to B3:B14; a naive code in text check tests "01" first and falsely matches "2014" ("JAN"). In Table 10, Gemini 3.8 Flash under EVOSKILL hits this bug at Turn 6, stalls across 9 prose-only turns (22,852 output tokens), and times out at Turn 20 without saving. Under EFFICIENTSKILL, mandatory-code rules keep every turn executable (even when Turn 1 queries openpyxl directly rather than inspect_wb), catching the "2014" collision at Turn 3 and saving verified labels by Turn 6 (−68.1% output, −81.0% input tokens). On the same task, Claude Opus 5.5 with EFFICIENTSKILL (Appendix D.§ 3) finishes in 3 turns (468 output, 7,954 input tokens: 16-tok inspect, 440-tok compute-and-verify, 12-tok submit).
| Turn | EvoSkill (Unconstrained, ): Action & Workspace Outcome | Out Tok | EfficientSkill (Ours, ): Action & Workspace Outcome | Out Tok |
| FAIL: 20 turns, 36,669 out tok, 134,553 in tok | PASS: 6 turns, 11,708 out tok, 25,629 in tok | |||
| T1 | Prints sheet names and rows 2–5 of Sheet3. | 595 | Loads workbook; queries defined_names. definedName → AttributeError. | 327 |
| T2 | Re-prints A1:G20 coordinate tuples. | 355 | Fixes query; resolves codes2 (D3:D14) and month (E3:E14). | 615 |
| T3 | Inspects font, fill, and borders of empty col B. | 1,240 | Builds ’01’..’12’ map; sees naive match hit ’01’ in ’2014’ (JAN). | 7,480 |
| T4 | Prose only (no code executed). | 3,565 | Verifies str types across D2:E14 before writing. | 932 |
| T5 | Prose only (no code executed). | 5,337 | Extracts exact code, writes ’SEP’..’NOV’ to B3:B14, saves workbook. | 1,995 |
| T6 | Tests naive match (JAN); prints dict without saving. | 1,201 | Issues completion signal → Pass (verified). | 359 |
| T7–T8 | Re-checks type(D3:D14); T8 prose only. | 2,074 | (Episode finished at Turn 6) | |
| T9–T11 | Re-dumps MID slices, cell styles, and named ranges. | 4,137 | ||
| T12–T15 | Four consecutive prose-only turns (no code run). | 6,643 | ||
| T16–T17 | Re-prints A2:B10; T17 prose only. | 3,629 | ||
| T18–T19 | T18 AttributeError on definedName; T19 prose only. | 4,911 | ||
| T20 | Script prints nothing, never saves → Timeout. | 2,982 | ||
| Total | 32,027 reasoning + 4,642 visible (9 unexecuted turns) | 36,669 | 10,070 reasoning + 1,638 visible (0 unexecuted turns) | 11,708 |
Table 10. Paired trajectory on SPREADSHEETBENCH Task 15380 for Gemini 3.8 Flash. “Out Tok” reports total output tokens (reasoning + visible completion) per turn or turn range.
Unsaved Turn-1 Exit vs. Verified Save (Grok 4.1 Fast, Task 36191). Task 36191 strips trailing emails from names in Column B (e.g., "Sam Landon [email protected]" "Sam Landon") and writes them to Column C. Under EVOSKILL, Grok 4.1 Fast generates 14,649 Turn-1 output tokens (14,152 visible): its inline inspection script crashes, yet its trailing plan emits the completion signal and ends on an untouched file (Table 11). Under EFFICIENTSKILL, the tools.py inspection helper prints A1:C3 and reminds the agent that no edits are saved yet, yielding a verified 2-turn solve (−82.3% output tokens).
| Turn | EvoSkill (Unconstrained, ): Action & Workspace Outcome | Out Tok | EfficientSkill (Ours, ): Action & Workspace Outcome | Out Tok |
| FAIL: 1 turn, 14,649 out tok, 2,173 in tok | PASS: 2 turns, 2,597 out tok, 3,940 in tok | |||
| T1 | Inlines custom inspection script (Traceback) and narrates full plan ending in completion signal → Early exit (0 saves). | 14,649 | Calls tools.py inspection helper; prints A1:C3 and reminder that no edits are saved yet. | 226 |
| T2 | (Ended without saving at Turn 1) | Strips email suffix, writes ”Sharon Smith” and ”Sam Landon” to C2:C3, saves, verifies, completes → Pass. | 2,371 | |
| Total | 497 reasoning + 14,152 visible completion | 14,649 | 2,429 reasoning + 168 visible completion | 2,597 |
Table 11. Paired trajectory on SPREADSHEETBENCH Task 36191 for Grok 4.1 Fast.
ALFWorld: Targeted Object Search vs. Wandering Across Rooms
On ALFWORLD, actions are short (4–15 tokens), so token cost comes mainly from per-turn reasoning and extra room visits. In Table 12 (T20190911_131350_027076, “put clean spoon on the table”), Gemini 3.8 Flash under EVOSKILL searches table 1 on countertop 1 at Turn 2 but walks past it looking for a literal spoon, checks 7 other receptacles (Turns 3–11) and 4 nonexistent IDs (Turns 12–15), and spends 5,083 reasoning tokens before returning at Turn 16. Under EFFICIENTSKILL (Appendix D.4), three simple rules (start at countertop 1, match aliases like spoon ladle on sight, and run clean in place) finish in 5 turns, cutting output tokens by 96.3% (5,206 195) and input tokens by 86.8% (49,167 6,479).
| EVOSKILL (Unconstrained, ) | EFFICIENTSKILL (Ours, ) | |||
| PASS: 21 turns, 5,206 out tok, 49,167 in tok | PASS: 5 turns, 195 out tok, 6,479 in tok | |||
| Turn | Action & Environment Observation | Out Tok | Action & Environment Observation | Out Tok |
| T1 | go to diningtable 1 → empty. | 115 | go to countertop 1 → sees knife 1, mug 1, ladle 1. | 24 |
| T2 | go to countertop 1 → sees ladle 1, walks past it. | 75 | take ladle 1 from countertop 1 (spoon → ladle alias). | 44 |
| T3 | go to sinkbasin 1 → empty. | 138 | clean ladle 1 with sinkbasin 1 (in-place state change). | 41 |
| T4–T5 | Visits and opens cabinet 1 → sees plate 1. | 159 | T4: go to diningtable 1 (table → diningtable). T5: put ladle 1 in/on diningtable 1 → Goal completed! (Episode finished at Turn 5; T4: 61 tok, T5: 25 tok) | 86 |
| T6–T11 | Opens/checks fridge 1, microwave 1, coffeemachine 1, and stove 1 without finding a literal spoon. | 1,475 | ||
| T12–T15 | Guesses non-existent drawer 1, cabinet 2, garbagecan 1, countertop 2 (“can’t see any . . . ”). | 1,495 | ||
| T16–T17 | Returns to countertop 1 and takes ladle 1 after long reasoning. | 1,284 | ||
| T18–T19 | Travels to sinkbasin 1; cleans ladle 1. | 245 | ||
| T20–T21 | Travels to diningtable 1; places ladle 1 → Goal completed! | 220 | ||
| Total | 5,083 reasoning + 123 action tokens | 5,206 | 156 reasoning + 39 action tokens | 195 |
Table 12. Paired trajectory on ALFWORLD task T20190911_131350_027076 (“put clean spoon on the table”) for Gemini 3.8 Flash.
Ordered fallback locations also lift Claude Opus 5.5 accuracy from 88.9% to 93.2% (12 failed test tasks recovered, 0 lost). On T20190908_210847_591144 (“Place a warm cup on the counter where the microwave is”, with the microwave on sidetable 1), EVOSKILL repeatedly re-heats cup 1 and mug 1 until timing out at Turn 25 (3,448 output, 87,486 input tokens), whereas EFFICIENTSKILL heats cup 1 once and checks countertop 1 → diningtable 1 → sidetable 1 in order to pass in 11 turns (590 output, 17,236 input tokens).
Table 12: Paired trajectory on ALFWORLD task T20190911_131350_027076 (“put clean spoon on the table”) for Gemini 3.8 Flash.
MATH-500: Direct Symbolic Solution vs. Repeated Code Retries
Across 10,304 evaluated MATH-500 runs (2,240 in the test matrix plus validation and ablation runs), single-turn symbolic solutions pass 78.7% at 3,363 total tokens per solve, whereas multi-turn Python/SymPy runs pass 75.4% at 15,316 tokens (4.6× higher, reflecting both harder problems when models call Python and retry overhead). EFFICIENTSKILL shifts Gemini 3.8 Flash () and Gemma 4 26B () from code retries to direct Turn-1 derivations while trimming prose restatements.
On test_precalculus_190 (; Table 13), Gemini 3.8 Flash under EVOSKILL spends 4 turns across unreduced SymPy output, a no-code pause, a float check, and a long write-up (2,496 output, 9,946 input tokens); under EFFICIENTSKILL, it applies trigonometric identities directly in 1 turn (471 output, 1,775 input tokens; −81.1% and −82.2%). This also avoids sandbox syntax-error loops on open-weight models: on test_intermediate_algebra_1354 ( for , ), Gemma 4 26B fails across 10 malformed tool calls under EVOSKILL (2,336 output, 45,580 input tokens) versus a 1-turn symbolic solve under EFFICIENTSKILL (544 output, 1,643 input tokens). On single-turn Claude Opus 5.5, EFFICIENTSKILL strips extra prose (792 → 100 output tokens on test_precalculus_190; 522 → 33 on test_intermediate_algebra_1247).
| Turn | EvoSkill (Unconstrained, )<br>PASS: 4 turns, 2,496 out tok, 9,946 in tok | Out Tok | EfficientSkill (Ours, )<br>PASS: 1 turn, 471 out tok, 1,775 in tok | Out Tok |
| T1 | Runs SymPy sp.simplify(sp.prod(...)) → returns unreduced radian product. | 570 | Pairs into , applies identity → → Pass. | 471 |
| T2 | Emits prose without code block → no output. | 166 | (Episode finished at Turn 1) | |
| T3 | Runs Python float product (0.03515625) and checks Fraction(3,16)**2 = 9/256. | 696 | ||
| T4 | Writes full multi-paragraph prose proof and final answer → Pass. | 1,064 | ||
| Total | 1,527 reasoning + 969 visible completion | 2,496 | 325 reasoning + 146 visible completion | 471 |
Table 13. Paired trajectory on MATH-500 problem test_precalculus_190 (, Gemini 3.8 Flash).
References
- [1]Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations (ICLR), pages 8479–8565.arxiv.org/abs/2507.19457
- [2]Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Christian A. Calvo, Weiyuan Chen, and Tu Vu. 2026. EvoSkill: Automated skill discovery for multi-agent systems. In Proceedings of the Conference on Language Modeling (COLM).arxiv.org/abs/2603.02766
- [3]Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. Large language models as tool makers. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2305.17126
- [4]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 38.arxiv.org/abs/2503.13657
- [5]Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Do NOT think that much for 2+3=? On the overthinking of long reasoning models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 9487–9499.
- [6]Ching-An Cheng, Allen Nie, and Adith Swaminathan. 2024. Trace is the next AutoDiff: Generative optimization with rich feedback, execution traces, and LLMs. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 71596–71642.
- [7]Yudong Gao, Zongjie Li, Yuanyuan Yuan, Zimo Ji, Pingchuan Ma, and Shuai Wang. 2026. SkillReducer: Optimizing LLM agent skills for token efficiency. arXiv preprint arXiv:2603.29919.
- [8]Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. 2025. Token-budget-aware LLM reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24842–24855.arxiv.org/abs/2412.18547
- [9]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS).
- [10]Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Xue Yang, Dongdong Chen, Xiaoqing Zheng, and Chong Luo. 2026. From raw experience to skill consumption: A systematic study of model-generated agent skills. In Advances in Neural Information Processing Systems (NeurIPS).
- [11]Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 13358–13376.

