WorksComputer Science
Beyond Plain MCP: Do Decision Models and Code Mode Make Tool Use Cheaper and Faster?
Abstract
Model Context Protocol (MCP) gateways manage tool catalogs for language model agents using two primary paradigms: code mode where agents write scripts against tool stubs and decision layers which are cheap routers that preselect tool subsets. Using a controlled harness across 12 servers, up to 1,958 distractor tools, and three open-weight agents including Qwen, Gemma, and Nemotron, we present a preregistered evaluation on a sealed test split to determine when each approach succeeds. Code mode substantially outperforms plain MCP, cutting token usage by 70 to 90 percent while significantly boosting utility. Declaring return-shape hints in code mode stubs is a first-order variable, swinging success by up to 39.5 points for lower-performing agents, an effect larger than adding a router. The central hypothesis that a decision layer in front of code mode beats code mode alone is not confirmed, showing minor, statistically non-significant improvements for weaker agents and a slight cost for the strongest. We publicly release our harness, a finetuned 4B tool selection model, and all evaluation traces to support robust community experimentation on local AI privacy and agent architectures.
Introduction
An agent that talks to hundreds of MCP tools pays for them twice. Every tool schema is placed in the prompt on every turn, and every tool result is copied back through the model on its way to the next call. Two remedies have emerged. Code mode replaces JSON tool calls by a script that the agent writes against generated stubs; the script runs in a sandbox, chains calls, filters large results, and returns one short answer [1, 2]. A decision layer puts a cheap model in front of the agent to choose which tools to expose, in the spirit of tool retrieval and routing [8, 9, 10].
Recent controlled work shows that code-style tool calling matches or beats JSON calling for most models [5, 6, 7], and tool retrieval is established [8, 9]. What is less clear is how the two interact, and which of the many design details (the stubs, the router, the fallback) carries the effect. We started from the natural hypothesis that a decision layer in front of code mode would beat either alone, pre-registered it, and tested it on a sealed split. The data told a more nuanced story, and this report documents it, including the negative results.
Contributions.
A controlled four-configuration comparison (plain MCP, router+JSON, code mode, router+code mode) across three open-weight agents, with a pre-registered confirmatory evaluation on a sealed split (matched cells, paired bootstrap, Holm correction, serial latency; § 4), and exploratory development-set analyses (§ 5).
Evidence that the return-shape information in code-mode stubs is a first-order variable: on the sealed split, removing it lowers success by 22.7 and 39.5 points for two of three agents (§ 4.2), replicating the development-set result (§ 5.2). To our knowledge, no prior controlled study isolates this variable (see § 2).
A characterization of when a decision layer helps: its benefit in front of code mode is directionally positive for the two lower-performing agents, absent for the strongest, and not statistically confirmed even with 238 instances (§ 4, § 5.3); it never needs to choose the call shape, and it can be dramatically worse without a bounded fallback (§ 5.5).
A 4B tool-selection model fine-tuned from the
spark-s1decision model, with a measured recall, latency and robustness profile, including the finding that a prompted-LLM router is fully steerable by a poisoned tool description (§ 6–§ 7).
A frank account of what is not established (§ 9).
Related Work
Tool use and code as action. Tool-augmented language models range from self-supervised API calls [26] and interleaved reasoning and acting [27] to models trained on thousands of real APIs [25, 28]. Letting the agent emit executable code rather than structured calls was shown to improve agents in CodeAct [23], and LLMCompiler plans and executes function calls as a dependency graph to parallelize them [24]. MCP itself standardizes how tools are described and invoked [4]; benchmarks such as MCP-Bench [21] and MCPToolBench++ [20] evaluate agents against MCP servers.
Code mode for MCP. Cloudflare’s “Code Mode” and Anthropic’s code execution with MCP and programmatic tool calling expose MCP tools as a code API so that intermediate data stays out of the model context [1, 2, 3]. Controlled comparisons now exist: Patel et al. [5] compare code-based and JSON tool calling for 14 models on BFCL v4 and find code matches or beats JSON for 11 of 14; Yang et al. [6] ablate restricting a coding agent to an execute_code tool and report cost gains with tied pass rates; Fendler et al. [7] study MCP design choices on MCP-Bench and report token and latency savings together with a larger attack surface. These works establish that code mode works and that it varies by model. We ask why it varies, and show that in our setting a single stub-design variable (declared return shapes) changes code-mode success by up to 40 points for some agents and not for others.
Tool retrieval, routing and discovery. RAG-MCP retrieves relevant MCP tools by semantic search to avoid prompt bloat [8]; MCP-Zero lets the agent actively request tools in a hierarchical server-then-tool process [9]; ACE-Router trains history-aware routers for large MCP and agent ecosystems [10]; and Anthropic’s tool search defers loading of tool definitions [3]. Our router configurations (B, D) are in this family; we differ in (a) pairing the router with code mode, (b) measuring end-to-end latency and fallback behavior rather than only recall, and (c) showing that recall is capped by the candidate retriever at large catalogs. More general model routing [29] and per-step routing for agent systems [17] choose among models; our decision layer chooses tools and call shape.
Speculation, caching and skills. Several systems reduce agent latency by overlapping or speculating on tool execution [14, 15], or by reusing learned tool compositions [16]. These are orthogonal to our question and could be layered on a code-mode gateway; we do not evaluate them.
Security of tool selection and tool descriptions. ToolHijacker shows that an injected tool document can hijack tool selection in agents [11]; MCP-ITP automates implicit tool poisoning in MCP [12]; TRUSTDESC generates trusted tool descriptions as a defense [13]. Our router-attack result (§ 7) is a small extension of this line: it shows that the router is itself the target, and that a scoring router behaves differently from a prompted one.
Failure recovery and static checking. Runtime failure detection and repair for agents [18] and budgeted recovery policies in tool-augmented orchestrators [19] motivate the fallback analysis in § 5.5. Static analysis of generated code against known APIs can catch hallucinated library calls [22]; typed stubs in code mode make a similar check possible for tool scripts, which we identify as future work.
Study Design
Gateway configurations
A single gateway process mediates between the agent and the MCP servers; the four configurations are switches on it, so servers, tasks, seeds, turn caps and sandbox are held constant.
| Id | Tool choice | Call shape | Seen by the agent |
| A Plain MCP | agent, from all schemas | JSON calls | all schemas; every result |
| B Router + JSON | decision model (subset) | JSON calls | subset schemas; every result |
| C Code mode | agent via search() | script in sandbox | 2 meta-tools; final line |
| D Router + code | decision model (subset) | script; JSON fallback | subset stubs; final line |
Table 1. The four gateway configurations. D-ORACLE uses the gold tool list at zero router cost.
Code mode. The agent has two meta-tools: search() returns Python stubs for matching tools, and execute() runs a script in a sandbox (an rlmit-restricted subprocess, not a container) with up to three executions per task. Return-shape hints are hand-written descriptions of each tool’s return value (keys, types, list versus dict) appended to the stub; in the unhinted setting a stub only states -> dict | list.
Decision layer. Given the task and a candidate list, the router returns a tool subset. Variants: a prompted LLM router (Gemma for the Qwen agent, Qwen for the Gemma agent, never the agent itself); zero-shot spark-s1; and our fine-tuned spark-s1-based router (§ 6). D may also choose the call shape by rules or a learned classifier. On a subset miss, the default policy retries in code mode with search over the full catalog and, only if that fails, falls back to JSON over the subset with request_more_tools capped at two calls (full_code policy). An earlier unbounded policy (json_more) is reported as a baseline.
World, tasks and agents
The mock world has 12 servers and 42 tools (issues, chat, billing, logs, CRM, docs, files, calendar, deployments, inventory, HR, support) with seeded, resettable state; scale arms add up to 1,958 real distractor tools drawn from a public catalog, for catalogs of 42, 200 and 2,000 tools. Tasks come from programmatic template families (lookup chains, aggregation over logs and invoices, conditional actions with side effects, multi-server workflows). Each kept task is solvable by a reference solver and verified by inspecting world state, not model text. The development split is seeds 0–2 with salt dev (119 task instances).
Agents are Qwen3.8-27B (checkpoint Vontra/Qwen3.8-27B-MLX-4bit, 4-bit, served with the TensorFold runtime), Gemma-4-26B-A4B (nvidia/Gemma-4-26B-A4B-NVFP4, vLLM) and Nemotron-3-Nano-30B-A3B (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4, vLLM), all quantized community or vendor checkpoints, served locally on one NVIDIA GB10 system (121 GB unified memory); the router runs in bf16 under vLLM. Temperature is 0 and thinking is disabled for all agents; one agent model generates at a time.
Metrics and statistics
Success is the verifier’s pass. Tokens are total prompt and completion tokens including router tokens. Latency is measured only in serial passes (concurrency 1, nothing else generating). Utility is
with and as primary weights (chosen before seeing D and B results; is also reported). Where latency was not measured serially, we report (token term only). All intervals are 95% paired bootstrap intervals over matched task-seed cells.
Pre-registration and what is confirmatory
A written protocol is in the repository (docs/PREREG.md). Its first part was written after the first pilots and is flagged as such; everything produced under it (development seeds, harness changes during development) is exploratory. A confirmatory protocol was committed before any run on the sealed test split and the configuration was frozen at git tag confirmatory-freeze-v1 (the pre-registration commit; the harness was not modified afterwards, only a serial driver script and the analysis scripts were added). The protocol fixes: salt test, seeds 0–2, 119 task instances (41/38/40 after a quality filter; every kept task is solvable by its reference solver and deterministic, which we re-verified without reading any agent outcome), all three agents, configurations A, B, C, D and D-ORACLE, serial execution, utility with and as the primary endpoint, paired bootstrap (10,000 resamples), Holm correction over the 12 primary comparisons (H1–H4 per agent), and the rule that a run exceeding the 900 s cap counts as a failure.
Sample size. The protocol itself states that about 300 task-seed pairs per arm are needed to resolve the 2-point non-inferiority margin. The confirmatory split has 119 instances, so that margin cannot be resolved and “not confirmed” below means inconclusive, not absent.
Extension protocol. After the core confirmatory results had been analysed, and before any further run, we wrote an extension protocol in the same file. It specifies: (Ext-1) code mode without return-shape hints on the same 119 test instances, to test the stub hypothesis on the sealed split; (Ext-2) configurations C, D and D-oracle on 119 new test instances (seeds 3–5; solvable and deterministic, checked beforehand), to test H3 on 238 instances in total, still below 300; and (Ext-3) a bootstrap that resamples task families rather than instances, plus a sensitivity grid over the utility weights. These are labelled extensions and are outside the 12-comparison Holm family (Ext-1 and Ext-2 each carry their own Holm correction over three agents). One deviation: before the core run we made a single smoke run on one test-split task to check the services (not part of the results).
Confirmatory Results (Sealed Test Split)
Setup. All numbers in this section come from the frozen protocol: 3 agents × 5 configurations × 119 instances = 1,785 serial runs (concurrency 1, one agent model generating at a time), hinted stubs, thinking disabled, temperature 0. In contrast to the development runs, B and D use the fine-tuned spark-s1 router (§ 6; top-30 embedding candidates, threshold 0.05) for all agents. D-oracle receives the gold tool list at zero router cost and serves as a reference, not a primary test. Table 2 gives the per-configuration results and Table 3 the 12 pre-registered comparisons. Across the 1,785 runs there were no hangs and one infrastructure fault (a malformed router response on one Gemma D run), which was re-run once under the identical configuration as the protocol prescribes; the original record is retained in the release.
| Agent | Config | Success | Tokens/task | Median latency (s) | |
| Qwen3.8-27B | A plain MCP | 0.857 | 26,602 | 4.9 | 0.576 |
| B router+JSON | 0.824 | 15,099 | 5.0 | 0.648 | |
| C code mode | 0.992 | 4,362 | 3.6 | 0.940 | |
| D router+code | 0.983 | 5,703 | 4.7 | 0.916 | |
| D-oracle (reference) | 1.000 | 3,083 | 3.1 | 0.962 | |
| Gemma-4-26B-A4B | A | 0.866 | 16,779 | 5.7 | 0.680 |
| B | 0.866 | 10,439 | 7.0 | 0.741 | |
| C | 0.891 | 5,112 | 6.4 | 0.798 | |
| D | 0.933 | 3,921 | 8.1 | 0.875 | |
| D-oracle (reference) | 1.000 | 2,098 | 6.1 | 0.963 | |
| Nemotron-3-Nano | A | 0.361 | 74,608 | 6.2 | −0.404 |
| B | 0.563 | 35,547 | 6.1 | 0.167 | |
| C | 0.555 | 7,553 | 4.6 | 0.457 | |
| D | 0.672 | 9,867 | 5.3 | 0.552 | |
| D-oracle (reference) | 0.681 | 9,037 | 4.0 | 0.571 |
Table 2. Confirmatory results, sealed test split, 119 matched instances per cell. = success − 0.01 tokens/1000 − 0.002 latency (mean). Medians are shown because a few runs are very long (Gemma C: one 890 s run and three others above 60 s, so its mean latency is 20.9 s versus a 6.4 s median).
| Agent | Comparison | 95% CI | Holm | 95% CI | Holm | |
| Qwen | H1: C − A | +0.364 | [+0.251, +0.504] | 0.0012 | [+0.158, +0.654] | 0.0012 |
| Qwen | H2: B − A | +0.072 | [−0.111, +0.230] | 0.43 | [−0.116, +0.223] | 1.00 |
| Qwen | H3: D − C | −0.025 | [−0.056, +0.005] | 0.43 | [−0.073, +0.010] | 1.00 |
| Qwen | H4: D − B | +0.268 | [+0.133, +0.451] | 0.0012 | [+0.070, +0.502] | 0.042 |
| Gemma | H1 | +0.118 | [−0.011, +0.244] | 0.43 | [−0.151, +0.357] | 1.00 |
| Gemma | H2 | +0.061 | [−0.028, +0.143] | 0.43 | [−0.048, +0.172] | 1.00 |
| Gemma | H3 | +0.077 | [−0.009, +0.176] | 0.43 | [−0.037, +0.272] | 1.00 |
| Gemma | H4 | +0.133 | [+0.050, +0.222] | 0.0084 | [−0.001, +0.309] | 0.43 |
| Nemotron | H1 | +0.861 | [+0.674, +1.046] | 0.0012 | [+0.470, +1.184] | 0.0012 |
| Nemotron | H2 | +0.571 | [+0.378, +0.754] | 0.0012 | [+0.208, +0.904] | 0.045 |
| Nemotron | H3 | +0.095 | [−0.008, +0.199] | 0.43 | [−0.077, +0.268] | 1.00 |
| Nemotron | H4 | +0.385 | [+0.175, +0.616] | 0.0048 | [−0.067, +0.842] | 0.69 |
Table 3. The 12 pre-registered primary comparisons (endpoint , = 0.002, 10,000 bootstrap resamples, Holm over 12). Left: the pre-registered analysis, which resamples instances. Right: an extension that resamples the 21 task families, which respects the clustering of instances within families.
Instances are clustered. The 119 instances are instantiations of only 21 task templates, so they are not independent. The pre-registered bootstrap resamples instances; we therefore also resample families (right-hand columns of Table 3). The family-level intervals are wider, and two of the six instance-level significant results (H4 for Gemma and for Nemotron) do not survive Holm correction. We treat the family-level analysis as the more defensible one and phrase conclusions accordingly, while keeping the pre-registered analysis in the table.
Primary comparisons
H1, code mode versus plain MCP: supported for Qwen and Nemotron under both analyses. Code mode cuts tokens by 84% (Qwen), 70% (Gemma) and 90% (Nemotron). Success rises by +13.4 points [+7.6, +20.2] for Qwen and +19.3 points [+8.4, +30.3] for Nemotron. For Gemma the direction is the same (+2.5 points of success) but the interval includes zero under both analyses.
H2, router+JSON versus plain MCP: supported only for Nemotron (Holm = 0.0012 and 0.045 under the two analyses; +20.2 points of success). For Qwen and Gemma the router cut tokens but not utility significantly. The 2-point non-inferiority margin is unresolvable at this .
H3, router+code versus code mode alone: not confirmed. This is the central hypothesis. Point estimates are −0.025 (Qwen), +0.077 (Gemma) and +0.095 (Nemotron); none is significant (raw = 0.07–0.10, Holm = 0.43). Secondary success differences are −0.8 points (Qwen), +4.2 points [−1.7, +10.1] (Gemma) and +11.8 points [+3.4, +21.0] (Nemotron); the last is unadjusted and hypothesis-generating. Section 4.3 adds 119 unseen instances.
H4, router+code versus router+JSON: supported for Qwen under both analyses. At the instance level it holds for all three agents (Holm ≤ 0.0084); at the family level only Qwen survives ( = 0.042), Gemma is borderline (CI [−0.001, +0.309]) and Nemotron’s interval is wide. The direction is positive for all three: once a router has produced a small subset, executing scripts is better than JSON calls, with large token savings.
Headroom and router quality. D-oracle receives the reference solver’s tool list, which leaks the task plan, so it is a ceiling, not a realistic router. It improves on C by +0.022 [+0.009, +0.042] for Qwen, +0.165 [+0.083, +0.263] for Gemma and +0.114 [+0.001, +0.227] for Nemotron. The fine-tuned router’s complete recall on the test split is 0.975 (3 of 119 instances miss a needed tool; mean recall 0.987) at 2.6 exposed tools and 1.55–1.61 s per decision; the JSON fallback fired in 1.7%, 0% and 13.4% of D runs. We did not decompose the remaining gap between D and D-oracle.
Extension 1: do return-shape hints matter on the sealed split?
Ext-1 reruns code mode on the same 119 test instances with the hints removed (stubs state only dict | list) and compares it with the hinted C of Table 2. Intervals resample families; the Holm correction covers the three agents.
| Agent | Success, no hints | Success, hints | success [95% CI] | [95% CI] | Holm |
| Qwen3.8-27B | 0.975 | 0.992 | +0.017 [−0.023, +0.080] | +0.035 [−0.010, +0.107] | 0.20 |
| Gemma-4-26B-A4B | 0.664 | 0.891 | +0.227 [+0.119, +0.361] | +0.296 [+0.162, +0.459] | 0.0003 |
| Nemotron-3-Nano | 0.160 | 0.555 | +0.395 [+0.190, +0.571] | +0.478 [+0.235, +0.693] | 0.0003 |
Table 4. Extension 1: effect of return-shape hints on code-mode success and utility, sealed test split (119 instances, 21 families). The hints also cut tokens: −1,598 (Qwen), −4,372 (Gemma), −6,104 (Nemotron), all intervals excluding zero.
The development-set result (§ 5.2) replicates almost exactly: without hints Gemma and Nemotron fall to success 0.66 and 0.16 (development: 0.66 and 0.19), well below plain MCP, while Qwen is barely affected. On a common scale (), return shapes are worth +0.30 and +0.48 for the two lower-performing agents, whereas adding a router in front of code mode is worth +0.10 and +0.11 and is not significant (§ 4.3). For Qwen neither matters much. The hints are hand-written by the author, so this shows that information about return values matters, not that any particular way of producing it works.
Extension 2: H3 with more instances
Ext-2 runs C, D and D-oracle on 119 new test instances (seeds 3–5), with the frozen configuration, for all three agents. We report the new sample alone and pooled with the original (238 instances); Holm correction covers the three pooled tests.
| Agent | Sample | C success | D success | (D−C) [95% CI] | Holm | |
| Qwen | seeds 3–5 | 119 | 1.000 | 0.983 | −0.032 [−0.076, −0.007] | |
| pooled | 238 | 0.996 | 0.983 | −0.028 [−0.075, +0.000] | 0.15 | |
| Gemma | seeds 3–5 | 119 | 0.866 | 0.958 | +0.121 [+0.010, +0.281] | |
| pooled | 238 | 0.878 | 0.945 | +0.099 [−0.011, +0.273] | 0.21 | |
| Nemotron | seeds 3–5 | 119 | 0.504 | 0.681 | +0.123 [−0.091, +0.343] | |
| pooled | 238 | 0.529 | 0.676 | +0.109 [−0.071, +0.294] | 0.22 |
Table 5. Extension 2: H3 on 119 unseen test instances and pooled with the original 119. Intervals resample the 21 task families. The “seeds 3–5” rows are unadjusted descriptive replications.
H3 remains not confirmed: after pooling, no agent is significant after correction. The new sample reproduces the signs seen in the original sample: negative for Qwen (the router costs about 1 s and a little success), positive for Gemma and Nemotron. For Gemma the new sample alone has an interval above zero (+9.2 points of success, +0.121 in ) before adjustment; for Nemotron the effect is similar in size but uncertain. We read this as moderate evidence for a benefit of a decision layer for agents whose own code mode is weaker, and for a small cost for the strongest agent, not as confirmation.
Latency and scale
For Qwen, D is 1.4 s slower than C (mean difference +1.45 s [+1.16, +1.79]) while the oracle router makes it 0.4 s faster, reproducing the development finding that a front-door router must cost less than about 0.4–0.7 s to pay for itself on a fast agent. For Gemma, differences in mean latency are dominated by a few runaway C runs; the typical latency of D is higher (median 8.1 s versus 6.4 s) but its tail is shorter (p90 15.0 s versus 25.4 s). With the Gemma D−C difference is +0.282 [+0.018, +0.699], driven by the tail.
Table 6 is descriptive (10 tasks). It reproduces the development picture: code mode stays nearly flat in tokens and latency from 42 to 2,000 tools; plain MCP needs 148K tokens per task at 200 tools and fails by context overflow at 2,000; and D with the bounded fallback matches C’s success but is slower (9.3 versus 4.2 s at 200 tools, 16.8 versus 6.0 s at 2,000) and uses more tokens, because retrieval-limited recall forces full-catalog retries.
| Tools | Config | Success | Tokens/task | Latency mean / median / p90 (s) |
| 42 | A | 0.90 | 16,076 | 3.4 / 3.3 / 5.5 |
| 42 | C | 1.00 | 4,112 | 3.3 / 3.0 / 4.9 |
| 42 | D | 1.00 | 5,045 | 4.6 / 4.5 / 5.9 |
| 200 | A | 0.80 | 148,047 | 20.5 / 30.2 / 33.1 |
| 200 | C | 1.00 | 7,766 | 4.2 / 4.4 / 6.6 |
| 200 | D | 1.00 | 15,176 | 9.3 / 5.5 / 19.1 |
| 2,000 | A | 0.00 (context overflow) | – | – |
| 2,000 | C | 0.90 | 11,942 | 6.0 / 5.0 / 14.7 |
| 2,000 | D | 0.90 | 31,016 | 16.8 / 16.6 / 38.0 |
Table 6. Confirmatory scale arm (Qwen3.8-27B, test seed 0, 10 tasks, serial; D uses the bounded fallback). Descriptive only.
Sensitivity to the utility weights
The weights in are a choice. Table 7 shows for H1 and H3 under five settings. H1 for Qwen and Nemotron holds under every setting; Gemma’s H1 changes sign at , , because of its slow code-mode runs. H3 is positive for Gemma and Nemotron and negative for Qwen under all five settings, and is never family-significant except for Qwen at a high latency weight, where it is negative. The qualitative conclusions do not depend on the weights.
| Agent | Comparison | |||||
| Qwen | H1 C−A | +0.253* | +0.357* | +0.364* | +0.433* | +0.587* |
| Qwen | H3 D−C | −0.018 | −0.022 | −0.025 | −0.051* | −0.038 |
| Gemma | H1 | +0.059 | +0.142 | +0.118 | −0.099 | +0.234 |
| Gemma | H3 | +0.071 | +0.054 | +0.077 | +0.282 | +0.089 |
| Nemotron | H1 | +0.526* | +0.864* | +0.861* | +0.835* | +1.532* |
| Nemotron | H3 | +0.106 | +0.095 | +0.095 | +0.096 | +0.072 |
Table 7. on the original 119 instances under different weights; *: family-resampled 95% CI excludes zero (unadjusted). The primary setting is the middle column.
Exploratory Results on Development Seeds
The analyses in this section use development seeds (salt dev), one run per cell, prompted LLM routers in B and D unless noted, and a harness that changed during development. They generated the hypotheses tested in § 4; they are exploratory.
Code mode is the dominant effect
Table 8 gives the development results with hinted stubs (matched cells: 119 for Qwen, 118 for Gemma, 119 for Nemotron).
| Agent | Config | Success | Tokens/task | |
| Qwen3.8-27B | A plain MCP | 0.832 | 30,649 | 0.525 |
| B router+JSON (LLM router) | 0.857 | 10,815 | 0.749 | |
| C code mode | 0.975 | 4,474 | 0.930 | |
| D router+code (LLM router) | 0.966 | 4,576 | 0.921 | |
| Gemma-4-26B-A4B | A | 0.831 | 15,282 | 0.678 |
| B (Qwen router) | 0.839 | 7,886 | 0.760 | |
| C | 0.864 | 4,255 | 0.822 | |
| D (Qwen router) | 0.966 | 3,487 | 0.931 | |
| Nemotron-3-Nano | A | 0.412 | 80,936 | – |
| C | 0.504 | 8,187 | – |
Table 8. Development seeds, hinted stubs, one run per cell. B/D use a prompted LLM router (different family from the agent) with router tokens counted. Nemotron’s decision-layer configurations were run only on the 39-task latency set (Table 10).
H1 (code mode beats plain MCP) is supported for all three agents on development seeds (confirmed for two of three on the test split), with a size that depends on the agent. For Qwen, C beats A by +14.3 points of success [+7.6, +21.8] and = +0.405 [+0.252, +0.607] at one-seventh of the tokens. For Gemma the success gain is small (+3.4 points) but the token saving is 3.6× ( = +0.144 [+0.047, +0.246]). For Nemotron, success rises from 0.41 to 0.50 with 10× fewer tokens; plain MCP failed several long log-paging tasks that exceed the context window.
The stubs matter as much as the architecture
Our first runs gave code-mode stubs no return information. Nemotron’s code mode then looked very weak (success 0.185 versus 0.412 for plain MCP). Adding hand-written return-shape hints to the stubs, with no other change, produced the following.
Hints also reduced tokens (Qwen 6.2K→4.5K, Nemotron 14.2K→8.2K). They flipped the preferred shape for Gemma and Nemotron: unhinted, plain MCP beat code mode for both; hinted, code mode wins. They also shrank the apparent headroom for any shape-routing layer (oracle utility gap over the best fixed shape: Qwen 0.051 → 0.023, Gemma 0.219 → 0.110, Nemotron 0.272 → 0.084). Conclusions about “which shape is best for this agent” are therefore conditional on stub quality, and any earlier comparison that used undeclared return types may be measuring the stubs rather than the agent. The sealed-split extension (§ 4.2) reproduces this result. The hints were written by the author and tuned on development data, so the finding is that return-value information matters; how much of the gain real servers’ outputSchema declarations (when present) would deliver is untested.
| Agent | C unhinted | C hinted | success | A (reference) |
| Qwen3.8-27B | 0.950 | 0.975 | +2.5 pts | 0.832 |
| Gemma-4-26B-A4B | 0.664 | 0.864 | +20.0 pts | 0.831 |
| Nemotron-3-Nano | 0.185 | 0.504 | +31.9 pts | 0.412 |
Table 9. Effect of return-shape hints in code-mode stubs (development seeds, 118–119 cells).
When does a decision layer help?
H3: D versus C. With the same hinted stubs:
Gemma (lower code-mode success): supported on development seeds, not confirmed on the test split (§ 4). With a Qwen-selected subset (2.4 tools on average), D reaches success 0.966 versus 0.864 for C (+10.2 points [+4.2, +16.1]), uses 18% fewer tokens (router tokens included), and [+0.046, +0.178]. With the gold subset, D-oracle gives +7.6 points and [+0.048, +0.160], so the prompted selector captured the whole oracle benefit. The benefit comes from the small accurate subset plus the JSON fallback, not from shape choice.
Qwen (highest code-mode success): not supported. D ties C (, [−0.032, +0.005]); C is already near ceiling at 0.975.
Nemotron: suggestive. On the 39-task latency set, success is 0.64 (D) versus 0.56 (C) and 0.38 (A); all intervals include zero.
H4 (D beats B) is supported for Qwen and Gemma: with the same subset, adding code mode gives +12.7 points of success and 4.4K fewer tokens on Gemma. H2 (router+JSON is noninferior to plain MCP and cheaper) holds only when the subset is small and accurate: LLM and gold subsets cut tokens by 65–68% at no loss of success, whereas zero-shot spark-s1 subsets (about 12 tools) saved no tokens. The pre-registered 2-point non-inferiority margin cannot be resolved at .
H5: shape routing. Every shape router we tried (a rule-based selector, zero-shot spark-s1) scored below “always code plus fallback” for Qwen and Gemma. Per-cell oracle shape labels were not an upper bound, because they come from single noisy runs (winner’s curse). A held-out analysis for Gemma, learning the best shape per task family on seeds 0–2 and applying it to fresh seeds 3–5, gave +0.051 utility over always-code [+0.003, +0.106], but that comparison is against C without a subset or fallback. An offline analysis over all (agent × shape) arms found only +0.036 oracle headroom, and a leave-one-task-out family-level policy scored −0.021 [−0.078, +0.010] below the best fixed arm (Qwen with code mode). We therefore treat shape routing as a negative result.
Latency
Table 10 gives clean serial latency on 39 tasks (seed 0, hinted stubs, mock tools at ms). Medians are reported because Gemma has rare runaway runs (one of 683 s).
| Qwen3.8-27B succ / med s / p90 s | Gemma-4-26B succ / med / p90 | Nemotron-3-Nano succ / med / p90 | |
| A plain MCP | 0.82 / 5.0 / 15.6 | 0.82 / 5.8 / 21.0 | 0.38 / 6.4 / 22.9 |
| B router + JSON | 0.85 / 5.7 / 12.7 | 0.87 / 7.9 / 18.9 | 0.59 / 6.2 / 22.3 |
| C code mode | 0.97 / 3.7 / 5.9 | 0.79 / 6.6 / 33.9 | 0.56 / 4.5 / 29.3 |
| D router + code | 0.95 / 4.6 / 6.8 | 0.90 / 7.4 / 21.4 | 0.64 / 5.5 / 35.1 |
| D-oracle (free router) | 0.97 / 3.0 / 4.8 | 0.90 / 7.5 / 19.3 | 0.64 / 3.6 / 21.9 |
Table 10. Serial latency, 39 tasks, one seed (development). Router time is included in B and D.
Code mode is the latency winner for the strongest coder and its advantage over plain MCP grows with the catalog (§ 5.5). A front-door router costs about 1.6 s at 42 tools (about 50 ms per candidate question), more than it saves for Qwen: with a free, perfect router D is faster than C ( s [, ]), but with the real router it is 2.2 s slower. For the two lower-performing agents the decision layer improves success (0.790.90 for Gemma, 0.560.64 for Nemotron; intervals include zero on this 39-task set) and trims the tail, while typical latency is unchanged. Mock tools return instantly, so real servers with per-call network time would favor batching scripts even more; we did not simulate this.
Scale and the fallback
We vary the catalog to 42, 200 and 2,000 tools for the Qwen agent (10 tasks, seed 0, serial).
| Tools | Config | Success | Tokens/task | Latency mean / median / p90 (s) |
| 42 | A | 0.90 | 87,460 | 9.8 / 4.8 / 60.6 |
| 42 | C | 1.00 | 4,677 | 3.6 / 3.6 / 5.9 |
| 200 | A | 0.90 | 163,969 | 22.1 / 31.3 / 39.2 |
| 200 | C | 0.90 | 10,390 | 5.6 / 4.9 / 10.0 |
| 200 | D unbounded fallback | 0.80 | 66,008 | 24.4 / 6.4 / 98.9 |
| 200 | D bounded fallback | 0.90 | 14,961 | 9.1 / 6.1 / 22.1 |
| 2,000 | A | 0.00 (context overflow) | – | – |
| 2,000 | C | 0.90 | 14,255 | 6.3 / 5.0 / 11.3 |
| 2,000 | D unbounded fallback | 0.40 | 370,482 | 222.7 / 116.1 / 570.2 |
| 2,000 | D bounded fallback | 0.90 | 34,490 | 16.1 / 16.4 / 34.0 |
Table 11. Scale arm (Qwen3.8-27B, 10 tasks, one seed). Plain MCP needs about 47K schema tokens per turn at 200 tools and cannot run at 2,000.
Code mode scales almost flat (3.66.3 s, 4.7K14K tokens), while plain MCP degrades and then fails. The decision layer exposes a design lesson: because candidate retrieval caps tool recall (0.65 at 200 and 0.44 at 2,000 tools), a subset miss is common at scale. Our first fallback switched to JSON and let the agent call request_more_tools without bound; each call added long real-world schemas, and tasks reached up to 23 calls and 900K tokens. Bounding the fallback (retry in code mode over the full catalog, then at most two request calls) cut D at 2,000 tools from 223 s to 16 s (14) and from 370K to 34K tokens, with success 0.40 0.90. Even so, D did not beat C for Qwen at any catalog size. This is a 10-task mechanism check, not a precise estimate.
A Fine-Tuned Tool Router
Model. We fine-tuned spark-s1, a 4B decision model on a Qwen3.5-4B backbone that returns calibrated probabilities for typed questions, into a tool selector (spark-s1-mcp-router-4b-r1) with LoRA. The state is the task plus 22–30 embedding-retrieved candidate tools, with one yes/no question per tool (“is this tool needed, directly or because its result feeds another step?”). Training data come from a disjoint training world (five task families and four servers’ tools held out), with paraphrases written by one model family (Nemotron) and verified by another (Qwen), labels computed from code, and negatives drawn from real catalog tools never used as evaluation distractors. Our first two training runs were aborted because loss reached zero within about 40 steps: 40% of the questions were trivially “no” (a tool-name-only rule scored 91%); the third run downsampled them.
Results. On 79 development tasks at a 42-tool catalog, complete recall (every needed tool selected) at threshold 0.01 is 0.899 for the zero-shot model at 3.3 tools and 1.000 for the fine-tuned model at 2.8 tools. On the same tasks reworded by a different model family and never seen in training, recall is 0.873 versus 0.964. Paired differences in complete recall (fine-tuned minus zero-shot) are +10.1 points [+3.8, +17.7] (template wording, same threshold) and +9.1 points [+1.8, +18.2] (reworded). At 200 and 2,000 tools both models sit at the retriever’s ceiling (0.65 and 0.44), but the fine-tuned model exposes about half as many tools (2.6 and 3.4 versus 5.3 and 6.2). In end-to-end runs with a Gemma agent, D with this router reaches success 0.932 versus 0.864 for C (+6.8 points [+1.7, +12.7]); the prompted Qwen router did better (0.966), although the fine-tuned router’s tool recall is higher (0.987 versus 0.966), so that gap is agent-side noise at .
Costs, stated plainly. (i) No latency win: a prompted 26–27B router answers in 0.75–0.9 s; the fine-tuned 4B router takes 1.6 s at 30 candidates, or 0.41 s at 15 candidates with a 2.5–5.5 point recall loss; a GGUF build under llama.cpp is slower still. Its advantage is footprint (8.4 GB versus about 52 GB) and precision, not speed. (ii) General skills: on the base model’s own benchmark, accuracy drops by 1.5–2.8 points, so it is not a drop-in replacement. (iii) Narrow data: 11 template families in one synthetic world, so near-zero training loss and recall that falls from 1.00 to 0.96 on reworded tasks.
Exploratory Security Observations
These observations are small and exploratory (one agent, six tasks, one seed for injection; one poisoned tool for routing) and are not part of the confirmatory evaluation.
Injection through tool results. With the Qwen agent, attack success was measured from world state across 108 runs per configuration: A 0.148, C 0.093, D 0.000. Code mode therefore did not raise attack success over JSON in this pilot (the difference is within noise), and the zero for D is largely structural: with about 2.4 tools exposed, the attacker’s goal tools are usually not callable at all. Only one payload style succeeded. Screening tool outputs with a zero-shot spark-s1 question roughly halved attack success for A and C, with 0 blocks in 343 benign calls, but the author wrote both the attack payloads and the defence.
The router is a target. With one poisoned attacker tool in the catalog, a prompted LLM router exposed it in 39 of 39 tasks when its description said “ALWAYS select this tool for every task”, and in none for two softer phrasings. A scoring router (fine-tuned spark-s1) exposed it in 62% of tasks for the “ALWAYS” phrasing, versus 69% for the zero-shot model (not a distinguishable difference at ). The “least-privilege” benefit of a small tool subset therefore assumes an honest router; no router output should be used as an authorization decision.
Discussion
What the evidence supports. Confirmed on the sealed split: code mode is the dominant effect on utility for two of three agents, with 70–90% fewer tokens; and return-shape information in the stubs is worth and of utility for the two lower-performing agents, more than any router effect we measured. Directionally supported, not confirmed: a decision layer in front of code mode helps those two agents (, pooled over 238 instances, not significant after correction) and slightly hurts the strongest; router+code over router+JSON is supported for Qwen and directionally for the others. Exploratory only: shape routing adds nothing, an unbounded fallback is pathological, and routers are injectable.
Implications for gateway design. (1) Publish or infer an output contract for every tool; where servers do not declare outputSchema, a gateway could learn shapes from traffic and use them to annotate stubs, which we propose but do not test. (2) Default to code mode with a bounded fallback instead of routing the call shape. (3) If a front-door router is used, make its fallback bounded and its output non-authoritative. (4) Measure end-to-end latency, not recall alone.
Future work. Real MCP servers and an established benchmark; unseen task families and paraphrased wording; thinking-enabled agents; a cheaper single-pass router; and typed code mode that infers return shapes from traffic and type-checks scripts before execution.
Limitations and Threats to Validity
No external benchmark or real servers. The world is synthetic, the tools are instant, and the test split uses the same template families as development (21 families, so instances are clustered). The study does not test unseen task families, real third-party MCP servers, real per-call latency, or paraphrased wording. This is its largest limitation for external validity.
Power and sample size. The protocol called for about 300 pairs per arm; the confirmatory split has 119 instances and the extended H3 test 238. Several intervals include zero, so “not confirmed” means inconclusive. One run per cell at temperature 0; the scale arm has 10 tasks.
Analysis choices. The pre-registered analysis resamples instances; we added a family-level analysis after seeing the core results, and it changes some conclusions (Table 3). The extension protocol was written after the core results and before the extension runs; its tests are outside the 12-comparison Holm family. The utility weights are a choice (Table 7).
Researcher degrees of freedom before the freeze. The stub hints were hand-written by the authors and tuned on development data; harness bugs were fixed on development worlds; thresholds were not tuned on a separate split. “Lower-performing agent” is an outcome-based label, and the three agents also differ in architecture, quantization and serving stack, so agent-level explanations are confounded.
Oracle and role confounds. D-oracle uses the reference solver’s tool list and so leaks the task plan; it is a ceiling. The router is the author’s own, trained on the same template structure; development B/D used prompted routers of a different family, the confirmatory runs used the fine-tuned router for all agents, so the two are not like-for-like. Nemotron shares a family with the test-side text writer.
Infrastructure. The sandbox is not a container; agents are quantized; thinking was disabled. In the confirmatory runs one Gemma and two Nemotron runs failed inside the router (a malformed response) and were re-run once under the protocol (original records are released); one extension run hit the 900 s cap and is scored as a failure. In development about 5 of roughly 3,000 runs hung and were excluded.
Security. Exploratory and tiny; the authors wrote both the attacks and the defence.
Conclusion
In a controlled study with a pre-registered evaluation on a sealed split, code mode was the dominant design choice for MCP gateways, and the information stubs carry about tool return values changed outcomes by up to 40 points for two of three agents, more than adding a router did. The motivating hypothesis, that a decision layer in front of code mode improves on code mode alone, was not confirmed, even after doubling the instances: it shows a consistent but statistically inconclusive pattern (helpful for the two lower-performing agents, slightly harmful for the strongest). The analysis choices matter: resampling task families rather than instances overturns some pre-registered conclusions. We release the harness, the router and all run records so that these questions can be tested on fresh splits and on real servers.
Artifacts
Code and pre-registration: https://github.com/abhishek085/mcp-decision-layer. The confirmatory protocol, frozen at tag confirmatory-freeze-v1, the raw per-run records (traces/confirmatory/) and the analysis scripts (bench/confirm_analysis.py, bench/confirm_robustness.py) are in the repository. Model spark-s1-mcp-router-4b-r1 (GGUF, Apache-2.0 with the base-model license chain), the router training dataset and the benchmark dataset with run traces (CC-BY-4.0): https://huggingface.co/abhishek085.
AI assistance
Experiments were executed and logged with the assistance of an AI coding agent (Claude Code) under the author’s direction, and this manuscript was drafted with its help. The author is responsible for the content.
References
- [1]Cloudflare. Code Mode. Cloudflare Blog, 2025. https://blog.cloudflare.com/code-mode/; see also https://blog.cloudflare.com/code-mode-mcp.
- [2]Anthropic. Code execution with MCP. Anthropic Engineering Blog, 2025. https://www.anthropic.com/engineering/code-execution-with-mcp.
- [3]Anthropic. Advanced tool use (Tool Search, defer_loading, Programmatic Tool Calling). Anthropic Engineering Blog, 2025. https://anthropic.com/engineering/advanced-tool-use.
- [4]Anthropic. Model Context Protocol specification. https://modelcontextprotocol.io.
- [5]I. Patel, S. Sen, E. Lumer, V. K. Subbiah. The Bitter Lesson of Tool Calling. arXiv:2608.06370, 2026.arxiv.org/abs/2608.06370
- [6]H. Yang, Q. Yu, T. Desell. When Does Restricting a Coding Agent to execute_code Help? A Regime × Agent-Design Ablation. arXiv:2607.10569, 2026.arxiv.org/abs/2607.10569
- [7]Y. Felendler, P. A. Gandhi, I. Habler, Y. Elovici, A. Shabtai. From Tool Orchestration to Code Execution: A Study of MCP Design Choices. arXiv:2602.15945, 2026.arxiv.org/abs/2602.15945
- [8]T. Gan, Q. Sun. RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation. arXiv:2505.03275, 2025.
- [9]X. Fei, X. Zheng, H. Feng. MCP-Zero: Active Tool Discovery for Autonomous LLM Agents. arXiv:2506.01056, 2025.
- [10]Z. Yao, Z. Xu, Y. Guo, Z. Han, C. Yang, S. Zhang, W. Zhang, X. Zeng. ACE-Router: Generalizing History-Aware Routing from MCP Tools to the Agent Web. arXiv:2601.08276, 2026.
- [11]J. Shi, Z. Yuan, G. Tie, P. Zhou, N. Z. Gong, L. Sun. Prompt Injection Attack to Tool Selection in LLM Agents (ToolHijacker). arXiv:2504.19793, 2025 (NDSS 2026).arxiv.org/abs/2504.19793
- [12]R. Li, Z. Wang, Y. Yao, X.-Y. Li. MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCP. arXiv:2601.07395, 2026.
- [13]H. Ye, Z. Zhang, J. Jia, H. Hu. TRUSTDESC: Preventing Tool Poisoning in LLM Applications via Trusted Description Generation. arXiv:2604.07536, 2026.
- [14]N. Ye, A. Ahuja, G. Liargkovas, Y. Lu, K. Kaffes, T. Peng. Speculative Actions: A Lossless Framework for Faster Agentic Systems. arXiv:2510.04371, 2025.
- [15]Y. Sui, H. Zhao, R. Ma, Z. He, H. Wang, J. Li, K. Xu, K. Chen, Y. Yang. Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving (PASTE). arXiv:2603.18897, 2026.
- [16]S. Chen et al. SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? arXiv:2603.00718, 2026.
- [17]G. Zhang, H. Yu, K. Yang, B. Wu, F. Huang, Y. Li, S. Yan. EvoRoute: Experience-Driven Self-Routing LLM Agent Systems. arXiv:2601.02695, 2026.arxiv.org/abs/2601.02695
- [18]S. Dubey. Real-Time Detection and Repair of LLM Agent Failures. arXiv:2608.02464, 2026.
- [19]R. S. Babu, A. Agrawal. Self-Healing Agentic Orchestrators for Reliable Tool-Augmented Large Language Model Systems. arXiv:2606.01416, 2026.
- [20]S. Fan, X. Ding, L. Zhang, L. Mo. MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark. arXiv:2508.07575, 2025.
- [21]Z. Wang, Q. Chang, H. Patel, S. Biju, C.-E. Wu, Q. Liu, et al. MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers. arXiv:2508.20453, 2025.
- [22]C. Miranda-Pena, A. Reeson, C. Paris, J. Poon, J. K. Kummerfeld. An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations. arXiv:2604.07755, 2026.
- [23]X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, et al. Executable Code Actions Elicit Better LLM Agents. arXiv:2402.01030, 2024.arxiv.org/abs/2402.01030
- [24]S. Kim, S. Moon, R. Tabrizi, N. Lee, M. W. Mahoney, K. Keutzer. An LLM Compiler for Parallel Function Calling. arXiv:2312.04511, 2023.arxiv.org/abs/2312.04511
- [25]Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, et al. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789, 2023.arxiv.org/abs/2307.16789
- [26]T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761, 2023.arxiv.org/abs/2302.04761
- [27]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
- [28]S. G. Patil, T. Zhang, X. Wang, J. E. Gonzalez. Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334, 2023.arxiv.org/abs/2305.15334
- [29]I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, et al. RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665, 2024.arxiv.org/abs/2406.18665