WorksComputer Science
Exstill: The Most Efficient Single-Chip Processor for the Next Token
Abstract
Generating a token with a language model requires a fixed amount of arithmetic, whose energy sets a floor beneath any hardware. For Qwen3-1.7B at a 16K-token context, gate-level measurements at 5 nm place this floor at 0.89 mJ per token, yet an NVIDIA B300 spends 391-1,120x that. The excess is the price of generality, software layers, programmable hardware and moving weights and key-value (KV) caches from external memory, and it widens with each process generation. We present a co-design approach that brings token generation close to this floor and then lowers the floor itself, optimizing silicon and model together in a gate-level search loop. Fixing one model architecture in the datapath, with weights still loadable, and holding weights and every conversation's KV cache on-die, a pre-silicon 5 nm design brings an unchanged Qwen3-1.7B token to 3.4x its floor. Co-designing the model with the circuits at fixed quality lowers the floor to 0.29 mJ and the token to 0.52 mJ, 1.8x the new floor and less than the baseline model's arithmetic alone, while on-die capacity grows from one user to twelve. Against a measured B300 serving the same twelve users, the design decodes each user's tokens 110x faster at 690x lower energy. To show that one fixed architecture can serve, as a co-processor, the latency-critical workloads around a frontier model, we introduce Polyglot-DC (1.7B and 4B, each with its own design), specialized for each job by distillation and task-specific training into its own weights. As a drafter it speeds single-user speculative decoding of large open models by up to 6.6x at the maximum reasoning effort. Its guard returns multilingual verdicts in a millisecond or less, within a point of the strongest open guard. Its agent and financial-reasoning models lead comparable open models on established benchmarks, with a reasoning turn falling from seconds to tens of milliseconds.
exstill.ai/co-processor
Introduction
Much of modern AI works one token at a time, and someone usually waits on each one: a person reading a reply, or a program waiting on an agent’s next action. Often the wait is on the smaller models that surround a frontier model, the large general model that composes the answer. Before and during its response, they screen prompts and outputs for safety, draft tokens ahead for speculative decoding, select the next tool call in an agent’s loop, and reason through domain-specific questions. These models typically serve one user or a few at a time, yet they run on general-purpose hardware built to batch very large models across many users. This report asks what a token from a small model needs to cost, why it costs hundreds of times more today, and how designing the silicon and the model together can both close that gap and lower the limit itself.
What a token needs to cost is set by its arithmetic. A model’s architecture fixes the multiply-accumulate operations each of its tokens requires, and the energy of those operations is a floor that no hardware implementation can go below: a limit that belongs to the model, whatever chip runs it. Measured on circuits placed and routed at the 5 nm class, the floor for Qwen3-1.7B (Yang et al., 2025), a representative small model writing against a 16K-token context, is 0.89 mJ per token.
Today’s best hardware spends hundreds of times more. Serving one user, an NVIDIA B300 spends 994 mJ on the same token, 1,120x the floor. Sharing the GPU among sixteen users, the conventional remedy known as batching, lowers the energy to 347 mJ, still 391x the floor, but each user then waits three times as long for a token. Batching is thus a dial that trades the operator’s energy against the user’s time, and it leaves their product, the energy-delay, almost unchanged.
More than 99.7% of what the GPU spends is therefore not arithmetic, and this excess has two causes. The first is the layers of software and programmable hardware between a model and a general-purpose chip’s arithmetic: frameworks, serving engines, kernel launches and instruction decoding, to which we attribute a fixed charge of more than a millisecond on every token, however small the model. The second is the movement of data. A chip that must run any model at any size keeps the weights and each conversation’s stored state, its key-value (KV) cache, in external high-bandwidth memory, and fetches gigabytes of them across the memory interface for every token. Both are the price of generality, paid by hardware designed before its model existed, and neither is a matter of physics. Nor does the usual progress remove them: better software reduces only the first, a smaller model only the second, and a newer process improves logic much faster than wires and memory, so the distance between a token and its floor widens with each generation.
A token’s energy on any hardware is thus the product of two factors: the energy of the arithmetic its model requires, and the multiple of that energy the hardware spends to carry it out. Our approach makes both small by co-designing the architecture of the silicon with the architecture of the model, and it rests on three decisions. The first removes the layers. We fix one model architecture in the silicon’s datapath, so that there are no kernels to launch and no instructions to schedule; the weights remain loadable, so a new model for a new task is a training run rather than a new chip. The second removes the movement. Everything a token needs, the weights and the KV cache of every active conversation, is kept on a single die beside the arithmetic, and nothing crosses an external memory interface. The third lowers the floor itself. How much fits on the die and how much arithmetic a token takes both depend on the model’s structure, so we treat the model’s architecture as a design variable, optimized together with the silicon at a fixed quality.
Each of these ingredients has precedent. Their combination in one closed loop does not: model architectures are explored and trained while the chip that would run each is costed at the gate level, on placed-and-routed circuits and over every token its model would write. We have costed more than ten million whole designs this way, and it is that loop, rather than any one decision, on which the results of this report rest.
Two pre-silicon designs returned by the search, both within a data-center envelope of 125 W and 575 mm², isolate what each half of the co-design contributes (Figure 1). In the first, Qwen3-1.7B is held exactly as released and only the silicon is designed. The token comes within 3.4x of its floor, removing nearly all of the GPU’s excess, but silicon alone can do little more: it cannot go below the floor of a model chosen in advance, and this model leaves the die room for only one 16K-token conversation. In the second, the model’s architecture is freed and co-designed with the silicon at the same quality. The floor falls by two-thirds, the hardware stays within 1.8x of the new floor, and the whole token costs less than the arithmetic alone of the model it replaces, while the same envelope holds twelve conversations.

Figure 1. A token’s energy against the floor of its model, at a 16K-token context, on a logarithmic scale. Each bar runs from the floor, the energy of the arithmetic the model requires, measured on our gate-level flow (§ 2.1), to the energy the hardware spends, so its length is the hardware’s multiple. The B300 serves Qwen3-1.7B and is measured (Table 1), from sixteen users, its most efficient setting, to one. The Exstill designs are pre-silicon at the 5 nm class, both within 125 W and 575 mm², and the co-designed model is held to Qwen3-1.7B’s quality (§ 3.4 and § 3.5).
The second design is the frontier model’s co-processor: a data-center chip that sits beside the accelerator running the frontier model, on a standard link, and takes over the small, latency-critical models around it. It serves twelve users at 19,921 tokens per second each, about 239,000 in total, and against a measured NVIDIA B300 serving the same twelve users the pre-silicon design decodes each user’s next token 110x faster at 690x lower energy. The floor gives that comparison a wider reach, and it is the sense in which the claim in the report’s title is meant. No token can cost less than its arithmetic, so a chip running the same model in the same arithmetic could improve on the design’s energy by less than a factor of two, and a chip running Qwen3-1.7B as released could not match it at all, since that model’s arithmetic alone costs more than the design’s whole token.
The same loop, run for a 4B-parameter model at a tighter quality bar within 150 W and 750 mm, returns a second and larger design, the DC2 design (§ 3.8). It serves four users at 13,998 tokens per second each, 58x faster and at 360x lower energy than a B300 serving the same four.
Fixing one architecture in silicon is worthwhile only if that architecture can do enough useful work, and the second half of the report shows that it can. Polyglot-DC is a family of models on the co-designed architecture, at 1.7B and 4B parameters, specialized by distillation into four roles that differ only in their weights. The first is drafting for speculative decoding, in which a small model proposes the next few tokens and a large model verifies them in a single pass. Polyglot-DC1-Draft runs on its own chip, apart from the GPU, and is aligned to five open models of 80B to 2.8T parameters, whose single-user decoding it speeds by up to 6.6x.
In the other three roles the user waits on the small model itself. As a guard screening prompts and streaming replies in six languages, the 4B model comes within one F1 point of the strongest open guard while flagging fewer than half as many harmless prompts, and a verdict at either size takes a millisecond or less, 38-39x faster than on a B300. As an agent, the 4B model leads eight open models of comparable size on -bench, a test of multi-turn tool use under a written policy; replayed at the 1.7B size’s decoding rates, an average reasoning turn from its episodes takes 57 ms where the B300 takes 2.46 s. In financial reasoning, the 4B model leads seven open models on nine of ten benchmarks, and a reasoned answer from the 1.7B model that takes the B300 1.68 s on average takes 44 ms.
The chip has not yet been fabricated, so the report states how each number is known. The GPU’s figures are measured on our bench, on one NVIDIA B300 running vLLM as shipped, and the models’ quality is measured on public benchmarks. The chip’s figures are pre-silicon estimates from the gate-level costing described above, which covers every layer and every token of the whole model rather than one kernel scaled up. Results that combine the two, such as the drafting speedups, are simulated. We estimate that a hand-tuned serving stack would narrow each ratio by about half, though it would leave the memory traffic that dominates when several users share the GPU. Against other specialized chips, which we have not measured, the claim rests on the argument from the floor, not on a bench.
The rest of the report follows this order. § 2 establishes the floor, measures the B300’s distance from it and traces that distance to the layers and the movement of data. § 3 describes the three decisions and the search, sets the unchanged and co-designed results against the B300, places both designs beside two others that keep a model on the chip, and states their limitations. § 4 introduces Polyglot-DC and its distillations, § 5 to § 8 examine its four roles in turn, and § 9 draws the argument together. The data center is the first setting for the approach; others, from robots to devices at the edge, are left to later work.
Where a Token’s Cost Lies
This section and the next make the case for the claim in this report’s title. The case begins from a limit rather than from an existing architecture: the energy of the arithmetic a token requires, measured on real circuits, which no implementation of its model can go below. A token’s energy on any hardware is the product of that limit and the multiple of it the hardware actually spends, and a processor is efficient to the extent that both factors are small. This section establishes the limit, measures the multiple on the best chip available and traces where it comes from; § 3 returns by the same steps in reverse.
The Limit: The Arithmetic Alone
To say how far a token’s cost is from what it needs to be, we need a reference that does not depend on how any particular chip is organized. We use the energy of the token’s arithmetic alone: the multiply-accumulate operations the model requires to produce one token, each a multiplication whose result is added to a running sum, with nothing moved, nothing scheduled and nothing idle. We call this the floor, and it is the limit against which every token in this report is measured.
We measured the arithmetic directly rather than taking it from a published estimate. To do so we built the datapath that a standard accelerator uses for 8-bit floating point (FP8), the circuit that carries out the arithmetic: a dot product that forms sixteen FP8 products per cycle and accumulates them in 16-bit floating point. The circuit was carried through placement and routing at the 5 nm class, the stage of physical design at which wire lengths and timing become known, and then simulated gate by gate on a real layer’s data, counting every transition, including glitches, the spurious transitions a signal makes before it settles. Each multiply-accumulate costs 246 fJ, its share of the accumulation and rounding included. We call this procedure our gate-level flow.
The reference model throughout this report is Qwen3-1.7B (Yang et al., 2025), representative of the small models that work around a frontier model. The number of multiply-accumulates it needs per token follows from its public configuration. In a decoder-only transformer, each generated token multiplies every weight once and, in every layer, attends over every earlier token of the context:
where is the number of weights used per token, the number of layers, the number of query heads, the head dimension, the context length and the energy of one multiply-accumulate. The factor of two counts the two products attention forms against the context, the query against each key and the attention weights against each value. Qwen3-1.7B uses about weights per token, including its output projection, and has 28 layers of 16 query heads of dimension 128. At a 16K-token context, attention adds multiply-accumulates, for in all, and at 246 fJ each the floor is 0.89 mJ. We chose FP8 because it is the format of the model’s published checkpoint and, in our measurements, the B300’s lowest-energy setting for it, so the floor and the GPU measurement that follows use the same arithmetic.
The floor is the limit for this model in this arithmetic, not for the task. No chip running Qwen3-1.7B unchanged in an FP8 datapath of this kind can spend less than 0.89 mJ on a token at this context. A different model that reaches the same quality with less arithmetic, or with cheaper arithmetic, has a lower floor of its own. A model built for every task has a higher one than a narrow question needs, since it carries arithmetic and state the question does not use. The floor, the first of the two factors, therefore belongs to the model rather than to the chip, and § 3.5 returns to it.
The Distance: A GPU Token Against the Limit
The small models around a frontier model usually serve one person or a few at a time, with someone waiting on every token. We measured that workload on one NVIDIA B300 running Qwen3-1.7B from its published FP8 checkpoint under the vLLM serving engine (Kwon et al., 2023) as shipped. Each user is given a 15K-token prompt followed by 1K generated tokens, so that every token is written against a context of about 16K, and the number of users served at once is varied from one to sixteen. We report decoding only, the writing of the generated tokens once the prompt has been read. Table 1 gives the results of this sweep, with each quantity defined in its caption.
| Users | Time between tokens | Energy per token | Board power | Total tokens/s | Energy-delay | Over the floor |
| 1 | 2.15 ms | 994 mJ | 461 W | 445 | 2.14 J·ms | 1,120x |
| 2 | 2.41 ms | 630 mJ | 522 W | 790 | 1.52 J·ms | 711x |
| 4 | 2.95 ms | 459 mJ | 623 W | 1,284 | 1.35 J·ms | 518x |
| 8 | 4.09 ms | 382 mJ | 747 W | 1,830 | 1.56 J·ms | 431x |
| 12 | 5.35 ms | 361 mJ | 810 W | 2,094 | 1.93 J·ms | 407x |
| 16 | 6.35 ms | 347 mJ | 869 W | 2,346 | 2.20 J·ms | 391x |
Table 1. Qwen3-1.7B in FP8 on one NVIDIA B300 with vLLM as shipped, 15K tokens of prompt and 1K generated per user, decoding only, measured on 16 October 2026. Energy per token is mean board power over decoding time divided by tokens; the time between a user’s tokens is the median. Energy-delay is their product (§ 2.3). The last column divides energy per token by the 0.89 mJ floor of § 2.1.
Serving one user, the B300 writes a token every 2.15 ms and spends 994 mJ on it, 1,120x the floor. Serving sixteen, each user waits 6.35 ms between tokens, three times as long, and each token costs 347 mJ, about a third as much. The arithmetic is the same throughout, yet it is less than a tenth of a percent of a token’s energy with one user and about a quarter of a percent with sixteen. Figure 2 draws the sweep against the floor: as more users share the GPU, the energy per token falls and then levels off, more than two orders of magnitude above the arithmetic it pays for.

Figure 2. Energy per token on one NVIDIA B300 serving Qwen3-1.7B in FP8 at a 16K-token context, decoding only, as one to sixteen users share it (measured, Table 1), against the floor, the energy of the same token’s arithmetic alone (measured on our gate-level flow, § 2.1). The shaded region between them is energy spent on something other than the token’s arithmetic. The energy scale is logarithmic.
The same distance appears in time, because energy is power multiplied by time. A chip that drew the the board draws for one user and spent all of it on the arithmetic would finish a token of Qwen3-1.7B in , where the B300 takes . The factor by which the operator of the chip overpays in energy is the factor by which the user overpays in waiting.
Batching Is a Dial, and How to Score a Token
Batching is the conventional remedy for a costly token. More users share each decoding step, the pass through the model that writes one token for every user being served, and the sweep shows the result: total output rises more than fivefold, from 445 to 2,346 tokens per second, and the operator’s energy per token falls. The saving is paid for entirely by the users, in time.
Whether that counts as an improvement depends on the measure. By the figure of merit most AI chips lead with, tokens per joule (equivalently, tokens per second per watt), the B300 improves nearly threefold, from 1.0 to 2.9. But tokens per joule rewards any change that lowers energy per token whatever it costs in time, and it can be raised in ways that serve no one waiting: by batching more users, by running the chip more slowly, by compressing the model past the point where its answers degrade, or by adopting a tokenizer that splits the same text into more tokens. Benchmarks that add latency limits and accuracy targets help, but they are pass/fail: they report whether a system met a bound, not what the wait cost.
Gonzalez and Horowitz [42] met the same problem in comparing microprocessors and resolved it by scoring a design on the product of energy and delay. A change that saves energy by making the user wait, or saves time by spending more power, has to pay for itself in the product. Scored this way, a token on the B300 costs with one user
Two Sweeps, Two Straight Lines
and with sixteen, essentially where it began; it is lowest at four users, at , and the whole range spans a factor of 1.6. In Figure 3 a token that is better for both the user and the operator lies toward the lower left, and as users are added the B300 does not move that way: it slides down and to the right, inside a band of nearly constant energy-delay. Batching is therefore a dial rather than a remedy. It moves the cost between the operator and the user without removing it, and at its most efficient setting the GPU still sits nearly four hundred times above the floor.

Figure 3. The batching dial. Energy per token against the time between a user’s tokens on one NVIDIA B300 serving Qwen3-1.7B in FP8 at a 16K-token context, one to sixteen users (measured, Table 1). Both scales are logarithmic, so a line of constant energy-delay is a straight diagonal; the shaded band lies between the least energy-delay of the sweep, at four users, and the greatest, at sixteen. Batching trades the user’s time for the operator’s energy and leaves their product within a factor of 1.6.
Energy-delay still rewards a model that answers worse but faster, so we fix quality first. We measure it in bits per byte: the model’s total negative log-likelihood on a text, in bits, divided by the text’s length in bytes, so that fewer bits per byte means a model that predicts the text better. Normalized by bytes rather than tokens, it cannot be improved by a finer tokenizer. Every design in this report is therefore the answer to one question: the least energy-delay per token that clears a stated quality bar, within a stated budget of power and area.
Insight 1: Score a token by energy times delay, at a fixed quality
Tokens per joule improves whenever users wait longer, so it cannot distinguish a better design from a more heavily batched one. Energy-delay at a fixed quality bar can, and by that measure batching a GPU leaves a small model’s token almost exactly where it started.
Two Sweeps, Two Straight Lines
No setting of the dial, then, brings the multiple below several hundred. To find where it comes from, we watch how the time between tokens changes as one thing at a time is varied: first the size of the model, then the number of users.
If a token’s time were spent reading and multiplying weights, a model sixteen times smaller would decode sixteen times faster. Figure 4 plots the time between tokens for seven open models between 0.5B and 8B parameters, each served to one user. Qwen2.5-0.5B is sixteen times smaller than Llama-3.1-8B and decodes only about three times as fast. A straight line through those two models has a slope of about 0.17 ms for every gigabyte of weights and meets the axis at about 1.1 ms, the time a token would take if the model had no weights at all, and the five models between them lie within about 20% of it.

Figure 4. The time between tokens for one user on one NVIDIA B300 with vLLM as shipped, by model size: Qwen2.5-0.5B, Qwen3-0.6B, Llama-3.2-1B, Qwen2.5-1.5B, Qwen3-1.7B, Llama-3.2-3B and Llama-3.1-8B, each in 16-bit precision with 1K tokens of prompt and 1K generated, measured on 14 October 2026. The solid line passes through the smallest and largest models and meets the axis at 1.1 ms; a least-squares line through all seven is nearly the same, 0.17 ms per gigabyte and 1.2 ms. The dashed line is where the models would lie if time followed size, and the shaded region between the two is time that making the model smaller does not recover.
For the number of users we return to the sweep of Table 1, at its 16K-token context. A straight line fitted to the time between tokens in its six rows, about 1.8 ms plus 0.28 ms for each user, reproduces every one of them within 2%.
Both lines have a part that grows, with the model’s size or the number of users, and a part that does not: more than a millisecond that every token pays regardless. The next two subsections take the two parts in turn.
What Grows With Load: The Movement of Data
The part that grows is the time spent moving data. A chip that must run any model cannot keep a particular model’s weights beside its arithmetic, and one built to serve large models to many users must hold more weights and more conversation state than fit on a die. It therefore keeps them in external high-bandwidth memory (HBM) stacks beside the processor, and every token pays to bring what it needs across that interface and through the on-die memory hierarchy to the arithmetic. The caches in that hierarchy, built for any pattern of access, must guess what to keep. A model’s weights and a conversation’s state are far larger than any cache, so most of what a token needs is fetched again from external memory for every token. Storage costs something even when nothing moves: memory takes area that could hold logic, and it leaks power whether or not its contents are used.
How much must be brought is easy to size from the model’s public configuration. A token needs the model’s weights and, for its user, the keys and values the model has stored for every earlier token of the conversation, its key-value (KV) cache. Serving the FP8 checkpoint of Qwen3-1.7B means reading about 2 GB of weights in each decoding step, once, for all the users in the step. The KV cache holds, for each token of context, a key and a value for each of 8 key-value heads of dimension 128 in each of 28 layers. At two bytes per number in vLLM’s default 16-bit cache, that is 112 KiB per token, or about 1.9 GB for a 16K-token conversation. Unlike the weights, it is read separately for every user.
At the B300’s published memory bandwidth of 8 TB/s those bytes take a known time to arrive, and the times match the slopes of the two lines. A gigabyte takes 0.125 ms, close to the 0.17 ms per gigabyte of Figure 4. One user’s cache takes 0.23 ms, more than four fifths of the 0.28 ms that each added user is measured to cost, which identifies the cost of adding a user as the cost of reading that user’s context. Figure 5 sets this calculated traffic, the time a step spends reading from memory, beneath the measured time between tokens: 0.25 ms for the weights and 0.23 ms for each user’s cache, which comes to about 0.5 ms for one user’s step and about 4 ms for a step shared by sixteen. The calculation assumes the full published bandwidth, so it is the least the traffic can take, and what remains above it is the most that everything else can account for.

Figure 5. Where the time between tokens goes on one NVIDIA B300 serving Qwen3-1.7B in FP8 at a 16K-token context. The line is the measured time between a user’s tokens (Table 1), which grows by 0.28 ms for each user added. The two lower bands are calculated, not measured: the bytes each step reads, from the model’s public configuration, at the B300’s published memory bandwidth of 8 TB/s. The upper band is the remainder. Labels on the right give each part at sixteen users.
Sharing a step helps with one of the two reads and not the other. With one user, a token reads about 3.9 GB, half of it weights and half conversation. With sixteen, each step reads about 32 GB, of which 30 GB is conversation: each user’s share of the weights has shrunk to an eighth of a gigabyte, while that user’s 1.9 GB of conversation is read in full however many share the step. Longer contexts make it worse. At 128K tokens one conversation’s cache is about 15 GB, more than seven times the model, so the share of each step spent moving context grows with exactly the long conversations and agent sessions in which small models do more and more of the work.
What Does Not Grow: The Layers Between the Model and Its Arithmetic
The part that does not grow is not explained by traffic. In the sweep of users, reading the weights accounts for 0.25 ms of the 1.8 ms that every step pays, and the remaining 1.6 ms or so is spent whether the step serves one user or sixteen. It is the counterpart, at a 16K-token context, of the 1.1 ms that Figure 4 found at 1K.
A fixed charge of that kind is what one would expect from the layers that stand between a model and a GPU’s arithmetic, layers of software and hardware that exist so that the GPU can run anything. A framework expresses the model as a graph of operations. A serving engine schedules requests, batches them and manages the memory that holds their state. Each operation is carried out by a kernel from a library, a program launched onto the device, and each kernel runs as a stream of instructions that thousands of programmable cores fetch, decode and schedule.
Generality of shape and of load adds charges of its own. The arithmetic units are built for blocks of a fixed shape, or tiles, so an operation that does not fill one, such as the matrix-vector products of a single user’s decoding step, is padded to the shape the hardware expects, and part of the arithmetic is spent on zeros or left idle. The machine is also built for any number of users, so it is organized around the batch: the whole chip, sized for a crowd, is stepped once for every token, and each user’s token waits on that step. None of this is arithmetic, and some of it is paid once per step whatever the size of the model.
We attribute most of the fixed charge to these layers: the cost of stepping a chip sized for a crowd through a long chain of small, dependent operations for every token. The attribution rests on the form of the charge, fixed per step and indifferent to how many users share it, rather than on a profile of the software.
How a token’s time divides between the two causes therefore depends on how many users share the chip. For one user, memory traffic is about a quarter of the 2.15 ms token and nearly all of the rest is the fixed charge, so the single-user token is slow because of the layers, not because of memory. With sixteen users the proportions reverse: traffic is about 4 ms of the 6.35 ms step, nearly all of it KV caches. Since the board draws its power throughout the step, a token’s energy divides in roughly the same proportions as its time. The division also explains the dial of § 2.3. Sharing a step spreads its fixed charge and the reading of the weights across the users, but each of them adds a conversation that must be read in full, so the step lengthens for all.
Why the Distance Persists
Neither part is arithmetic, which is why three usual ways of making a token cheaper leave most of the distance in place. The first is a smaller model, which shortens only the part that grows with size. For Llama-3.1-8B, whose weights occupy 16 GB at 16-bit precision, that part is most of the token, about 2.8 of its 3.85 ms. Under 2B parameters the fixed charge is most of the token, so small models are the case in which a general-purpose chip recovers least of what the model’s size should give.
The second is better software, which shortens only the fixed part. Hand-tuned kernels can remove much of it for a single user (§ 3.10), but the model still reaches its arithmetic through a general-purpose machine that does not know its structure in advance, and the traffic is untouched.
The third is a new process node, the next generation of chip manufacturing, which repairs neither part, because neither falls at the rate the cost of arithmetic does. For the movement and storage of data the evidence is published. Horowitz [49] tabulated the energy of arithmetic and memory operations at 45 nm, and Jouppi et al. (2021), drawing on three generations of Google’s data-center accelerators, brought the table forward to 7 nm and found the improvement uneven. Arithmetic operations became 2.1-4.3x cheaper, while an access to a small block of on-chip static memory (SRAM) improved by 1.3-2.4x and the energy of driving a wire, per unit of length, by less than 2x. SRAM density lagged as well: at 7 nm a square millimeter held about a fifth of what ideal scaling from 65 nm would have given, and by their estimate the step to 5 nm added only about 13%. Reaching external memory over a conventional interface cost the same at both nodes. The gain that did arrive came from packaging rather than from the process, when high-bandwidth memory placed stacks of memory dies close to the processor, and the B300 already has it. Their summary is that logic improves much faster than wires and SRAM.
For the layers the reasoning is our own. Their charge is set not by the energy of a transistor but by the time a general-purpose machine takes to sequence a chain of small, dependent operations, and the additional transistors of a new generation, spent as they usually are on more parallel arithmetic and more memory bandwidth, do little to shorten such a chain. A token whose cost lies in the layers and in moving and storing data therefore improves more slowly with each node than its arithmetic does, and its distance from the floor widens from one generation to the next.
Insight 2: Almost none of a token’s cost is its arithmetic
On a GPU serving a small model, the arithmetic is a fraction of a percent of a token’s energy. For one user, most of the rest is the fixed charge of the layers between the model and its arithmetic; as users are added, it becomes the movement of each user’s context from where it is stored. Neither falls at the rate the cost of arithmetic does, so a design that only makes the arithmetic cheaper leaves almost all of the cost in place.
The industry tends to treat the distance between a token and its arithmetic as physics. The measurements of this section point instead to design choices. Almost every model today runs on hardware designed before the model existed, and such hardware has to be general: able to run any program, on any shape of computation, for any number of users, with any pattern of memory access, for any model. The layers are the price of the first three and the movement of data the price of the last two, while a model built for every task raises the arithmetic itself. Choices can be made differently, and the next section describes how we make them.
Approaching the Limit Through Model and Silicon Co-Design
Our approach is to remove the generality a token does not need, from both factors of its cost: the multiple the hardware spends, and the arithmetic the model requires. Two decisions about the silicon bring down the multiple. We build the chip around the model’s architecture, so that no layers stand between the model and its arithmetic (§ 3.1), and we keep everything a token needs on the die, beside the arithmetic, so that nothing has to be brought to it (§ 3.2). A third decision lowers the arithmetic itself: since the floor is a property of the model, we treat the model’s architecture as a design variable, chosen together with the silicon (§ 3.5). Table 2 sets each form of generality beside what it costs a token and the decision that answers it.
| Built for | What every token pays | The Exstill design |
| Any program | Framework, serving engine, kernels; instructions fetched, decoded and scheduled | The architecture is fixed in the silicon; there is nothing to program (§ 3.1) |
| Any shape | Arithmetic padded to the hardware’s tile shapes, spent on zeros or left idle | Engines and data paths sized for the model’s architecture (§ 3.1) |
| Any number of users | A crowd-sized machine stepped once per token; each token waits on the batch | Sized for one user’s next token, a few users at a time (§ 3.1) |
| Any access pattern | Caches that guess what to keep; most data arrives from external memory | Nothing to guess: all of a token’s data is on the die (§ 3.2) |
| Any model | Every weight fetched again from memory for every token | Weights loaded once and kept beside the arithmetic (§ 3.2) |
| Any task | A general model carries arithmetic and state the question does not use | A model distilled for its job, its architecture chosen with the silicon (§ 3.5) |
Table 2. The forms of generality of § 2, what each costs a token on a conventional chip, and how the Exstill design answers it. The first three are the layers, the next two data movement and storage, and the last the generality of the model itself.
The first instance of this approach is the frontier model’s co-processor, a chip that runs the small models working around a frontier model and sits beside the processor that runs it. The same approach applies to other environments and power envelopes, from robots and factory equipment to devices at the edge, but those designs are a subject for later work, and this report covers the data-center design. The section retraces § 2 in reverse, removing the causes, then closing the distance, then moving the limit. It ends with a second design for a larger model (§ 3.8) and with two other designs that keep a model on the chip, set beside ours from their published figures (§ 3.9).
Removing the Layers: A Chip Built Around the Model
A model is two things: its architecture, the shape of the computation it performs for each token, and its weights, the values learned in training that decide what its tokens mean. A GPU is built knowing neither. It receives the architecture at run time, operation by operation, through the layers of § 2.7, and it reads the weights from external memory as each operation needs them. That is what lets a GPU run any model, and it is where much of a small model’s token is paid for.
We fix the architecture in the silicon and leave the weights free. The chip’s memory, compute engines and data paths are organized around one model architecture, chosen together with the silicon, so the computation a token requires is laid out in the hardware rather than assembled at run time by a framework, a serving engine and a library of kernels. There is nothing to program: tokens go in and tokens come out. The weights are loaded rather than built in, so a new model of the same architecture, whether a new task or a new release, is a training run rather than a new chip. The chip is built for one user’s next token rather than for a batch, so it serves a few users at a time, and no crowd-sized machine is stepped through each token.
The usual escape from the cost of generality is dedicated silicon, a chip built for one model: lean, fast, and obsolete the moment the model changes. For the small models around a frontier model that is a poor trade, because the models change every few months while their jobs do not. Fixing the architecture and leaving the weights free keeps most of what dedicated silicon offers, a chip organized around the computation it performs with no layers in between, without tying the chip to the model it was built for. What it gives up is freedom in the architecture, so the chip’s value depends on how much useful work one architecture can do. § 4 answers that question with Polyglot-DC, a single architecture distilled into separate weights for drafting, guardrails, agents and real-time reasoning.
Removing the Movement: Everything the Token Needs, Beside the Arithmetic
The answer to the movement of data is to change where the data lives. The data-center design keeps the model’s weights and the context of every conversation it serves on a single die, beside the arithmetic that uses them, with no external memory stacks. It holds twelve conversations of 16K tokens each. For the same twelve users, each step of the B300 brings about 25 GB across its memory interface (§ 2.6); here no weight or key-value pair crosses a chip boundary on its way to being used, and no token waits on a trip to external memory. This report says where the weights and the context are kept and what that achieves. How they are stored and read, and how contexts longer than 16K tokens are handled (§ 5.4), are part of the co-design and are not described.
Keeping everything on the die is possible only if there is little enough of it, which is why this decision cannot be separated from the model. The principle that governs the silicon and the model alike is to move as few bits as the quality bar allows, over the shortest distance to the arithmetic. The silicon sets the distance; the model, chosen together with it, sets how many bits must be stored and moved to reach the required quality.
The chip does not replace the processor that runs the frontier model. It sits beside it on a standard link, and only tokens cross between the two. At the design’s full output of about 239,000 tokens per second (§ 3.5), that traffic is about a megabyte per second in each direction, a few bytes per token and under a ten-thousandth of the bandwidth of a PCIe slot, so the link constrains neither side. Figure 6 sets the arrangement that results from the two decisions beside a GPU’s.

Figure 6. What stands between a model and its arithmetic, and where a token’s data lives. On a GPU, the model reaches the arithmetic through layers of software and programmable hardware, and every token brings the weights and the user’s conversation in from external memory: for Qwen3-1.7B at a 16K-token context, about 2 GB of weights and 1.9 GB for each user (§ 2.6). On the Exstill design, the architecture is fixed in the silicon, the weights and every conversation are kept on the die beside the arithmetic, and only tokens enter and leave, a few bytes each (§ 3.2). Illustrative, not a floorplan.
Other designs have removed parts of the same overhead. Keeping a model in on-chip memory is not new: Groq’s tensor streaming processors ([2]), NVIDIA’s LPX decode racks (NVIDIA, 2026b) and Cerebras’s wafer-scale engine (Lie, 2023) serve large models quickly from on-chip memory spread across many dies or a whole wafer. Taalas builds a single model into its chip (Taalas, 2026), the dedicated-silicon end of the trade described in § 3.1, and d-Matrix’s cards already sit beside GPUs. Our design is fitted differently: the whole of a small model on one die sized for it, the silicon and the model chosen together, placed beside any accelerator, with a new model of the same architecture loaded as weights.
None of the ingredients is ours alone. Keeping weights beside the arithmetic, building silicon for one kind of computation, distilling a large model into a small one for a narrow job, and shaping a model to the hardware it will run on are each established directions, and each of the designs above takes one or two of them further than we do. What has, as far as we know, no precedent is taking all of them at once, inside a single loop: model architectures explored and trained on one side, and on the other every candidate chip costed on placed and routed circuits rather than from a spreadsheet, with the two sides scored together on the cost of a token at a fixed quality. That loop is the capability this report rests on, and it is harder to assemble than any of its parts.
Designing From the Circuits Up
The three decisions interact, so we make them in one search over whole designs, each a model and its silicon together. The search needs the time and energy of every candidate’s tokens, which we call costing the design, and because the limit is measured on circuits, the candidates must be costed on circuits too. Before a chip exists there are four levels at which it can be costed (Table 3), and only the last, the gate level, grounds both time and energy in the circuits as they will be fabricated.
| Level | What is run | Time | Energy |
| Paper | The chip’s arithmetic units multiplied by its clock | A ceiling | A ceiling |
| Architecture | The workload on a software model of the chip, with an assumed cost for each operation | Estimated | Estimated |
| Cycle | The real design, simulated clock by clock, running its actual program | Exact | Estimated |
| Gate | Circuits placed and routed as they will be fabricated, every switching gate counted | Exact | Measured on the flow |
Table 3. The four levels at which a chip can be costed before silicon, and what each grounds. Silicon follows, where anything the levels above missed appears after the money is spent.
Published pre-silicon numbers tend to fall short in one of two ways. Those that cover a whole model usually stop at the architecture level, on assumed costs. Those that reach the cycle or gate level cover only part of the model, and compilers are the reason. Below the architecture level the simulated chip must run its actual program, and turning a whole model into that program takes a compiler and a library of kernels that many chips do not have before silicon. The common route therefore carries one hand-written kernel, typically the large matrix multiplication the chip was built for, to the cycle or gate level and scales it up as though the rest of the model ran equally well. It rarely does: the rest of the model, and the data moving between layers, is where chips lose time and energy. Even a company [after one example] Text: Kenji Watanabe, 51, is a ship’s pilot in Osaka and speaks Japanese and Korean. JSON: {"name": "Kenji Watanabe", "age": 51, "job": "ship's pilot ", "city": "Osaka", "languages": ["Japanese", "Korean"]}
[16,013 tokens of The Time Machine, H. G. Wells] A summary of the story so far, in a few sentences: The Time Traveller, a scientist, has invented a machine that allows him to travel through time. He travels back in time to a future society where he encounters a beautiful and fragile race of people. He learns about their society and its decline, and decides to return to his own time to prevent further changes.
with shipping silicon often cannot fully cost a new architecture, because every change to it means updating the compiler, recompiling the model and sometimes rewriting kernels by hand.
Chips are conventionally designed from the top of Table 3 down: an architecture is chosen early from paper estimates and a software model, and gate-level costs arrive only after it is frozen, when a disagreement between the gates and the model can usually be corrected only in the next chip. We reverse the order and begin with the circuits, 304 of them to date, each placed, routed and simulated gate by gate at the 5 nm class, glitches included. Whole designs are then costed on those circuits. For each design we measure every token its model would write, at every layer and every position in the context, in clock cycles for time and in picojoules for energy, including the token’s share of the chip’s leakage, the power it draws even when idle. More than ten million designs have been costed this way: not ten million gate-level simulations, but ten million whole designs, each with every token measured on the same placed and routed circuits, before any one of them is chosen. Figure 7 shows the measure for two of those tokens, one early in a short exchange and one deep in a long one.

Figure 7. A receipt for every word. Each card is one token the co-designed model wrote on the pre-silicon data-center design at the 5 nm class, with the clock cycles and the energy the token took, split into the arithmetic the model requires and everything else, leakage included. The word ‘pilot’ comes early in a short exchange and ‘machine’ 16,000 tokens into a long one, and the difference between them is what the context behind the second costs. The lower bar is a token of Qwen3-1.7B on one NVIDIA B300 serving twelve users in FP8, measured on our bench, with its arithmetic measured on our flow, a quarter of one percent of the token.
Removing the layers is what makes this order practical. Each design is built around its model’s architecture and has no program to compile, so a candidate can be costed on the whole model without first writing a compiler and kernels for it. The property that removes the layers from every token at run time removes them from every design during the search.
The chip’s figures in this report are therefore pre-silicon in a specific sense: they rest on gate-level measurement of every circuit and cover the whole model rather than one kernel scaled up. Throughout the report, each number states how it is known: measured on an NVIDIA B300 on our bench with the stated setup; measured on a named benchmark or on a named circuit block on our gate-level flow; calculated from a public configuration or specification; pre-silicon for the chip’s own figures at the 5 nm class; and simulated for a result that combines a B300 measurement with a pre-silicon figure, such as the speedups of § 5.
Every design the search returns answers the question set in § 2.3: the least energy-delay per token that clears a quality bar, within a budget of power and area. For the data center the budget, which we call the envelope, is 125 W and , with every user holding a 16K-token context on the chip. The bar is Qwen3-1.7B’s own quality in bits per byte on our probe texts, so each design below is held to the quality of the model the B300 serves in § 2.2.
The Distance Closed: The Same Model on Our Silicon
To see what the silicon alone achieves, we hold the model fixed. We asked the search its question for Qwen3-1.7B exactly as released, on our silicon, within the envelope. On the design with the lowest energy-delay that fits, a token costs 3.0 mJ, 3.4x the model’s floor, and arrives every 57 s. The B300 serving the same model sits between 391x and 1,120x above that floor, depending on how many users share it, and at its fastest takes 2.15 ms per token. Removing the layers and the movement, with no change to the model, brings the token within a small multiple of its arithmetic. Figure 8 places the two on one scale, with the design of the next subsection beneath them.

Figure 8. How close each design comes to the limit. Energy per token at a 16K-token context on a logarithmic scale, against the floor of Qwen3-1.7B as released and the floor of the co-designed model, both the arithmetic alone measured on our gate-level flow. Each bar runs from a design’s own floor to its token, so its length is the multiple beside it. The B300 is measured (Table 1); the Exstill points are pre-silicon at the 5 nm class. Both Exstill points lie within the same envelope of 125 W and 575 mm²; the model-unchanged point is the design with the lowest energy-delay for Qwen3-1.7B as released that fits it, and it serves one user (§ 3.6).
Silicon alone can go little further. A chip built for a model chosen in advance can at best approach that model’s floor; it cannot take a token below it. And this model leaves the die little room: within the envelope the chip holds the weights of Qwen3-1.7B and one 16K-token conversation, and has no room for a second. Both limits, the arithmetic a token requires and the bytes the chip must store, are properties of the model’s architecture.
The Limit Moved: The Model as a Design Variable
When the model’s architecture is itself a design variable, chosen together with the silicon and held to the same quality, the floor moves with it. A model that reaches the quality bar with less arithmetic, or with arithmetic that costs less on the silicon it is paired with, or with less state to store, has a lower floor, and a chip built around it can go below the floor of the model it replaces. This also answers the generality of the model itself: a model trained for one job by distillation (§ 4.1) does not carry the arithmetic and state that a general model brings unused to a narrow question.
This is where our approach parts from hardware-aware model design as it is usually practiced, which searches for the model that runs best on a fixed accelerator (§ 4.2). We vary both. Neither side is optimized first and handed to the other; each candidate is a model and its silicon together, scored on the cost of its tokens at a fixed quality. The difficulty is not in either side but in holding them to the same pace and the same fidelity: a candidate architecture must be trained and measured in hours for the search to move, and the silicon that would run it must be costed at the gate level in the same loop. The 304 circuits, and the ten million designs costed on them, are what make that possible. What the search changes in the model and in the silicon is proprietary to Exstill.
With the model free, the search within the same envelope evaluated 522,675 designs. Of these, 322,941 fit the budget, and 313 lie on the Pareto front, the designs that no other design in the budget beats on both quality and energy-delay. The quality bar selects the design on the front with the lowest energy-delay that meets it, and Figure 9 shows the narrowing.

Figure 9. The search for the data-center envelope, from the designs evaluated to the one returned: within the budget of 125 W and 575 mm, on the Pareto front of quality against energy-delay, and the lowest energy-delay on the front that meets the quality bar. Every design is costed token by token on placed and routed circuits; pre-silicon at the 5 nm class. Bar lengths are on a logarithmic scale. Across all envelopes and contexts, more than ten million designs have been costed this way.
The design returned is the data-center design, summarized below. It uses 526 mm² of the 575 available and the full 125 W, and each of its twelve users, with a 16K-token context held on the chip, receives 19,921 tokens per second. Across the twelve the chip produces about 239,000 tokens per second, or 1,912 tokens per joule.
50 μs — to decode a token, per user
0.52 mJ — per token, leakage included
12 users — 16K tokens of context each
125 W — and 526 mm², pre-silicon
Its model, which we call the co-designed model, has a floor of its own of 0.29 mJ, about a third of Qwen3-1.7B’s, measured the same way on our flow. The design’s full token, leakage included, costs 0.52 mJ, 1.8x that floor, so the design stands within a factor of two of limit set by its own model. It also stands below the 0.89 mJ floor of Qwen3-1.7B, the model whose quality it matches, where no chip running that model unchanged in that arithmetic can go.
A leaner model alone would not have reached this either: on conventional hardware it would still pay for the layers and the movement, which Figure 4 showed to be most of a small model’s token whatever its size. The result depends on both halves of the co-design.
Insight 3: Silicon approaches the floor; the model moves it Silicon that removes the layers between a model and its arithmetic, and keeps the model and every conversation beside that arithmetic, brings an unchanged model within a few times its floor. Going below that floor at the same quality requires changing the model, which is why the model and the silicon are searched together rather than designed in sequence.
Beyond Energy: How Many Users One Die Serves
Energy per token understates the model’s half of the co-design. On the scale of Figure 8 the two Exstill points lie close together, 3.0 mJ for Qwen3-1.7B unchanged and 0.52 mJ for the co-designed model, a factor of 5.8 that is small beside the factor of more than a hundred separating either from the GPU. What that axis leaves out is how much each design delivers for its energy. The two designs answer the same question within the same envelope and differ only in whether the model is free to change, yet the first serves one user and the second serves twelve, each of them sooner: a token every 50 s, against 57 s for the unchanged model’s single user.
The number of users is a consequence of the model rather than a setting of ours. A chip that keeps every conversation beside the arithmetic serves as many users as it has room to hold, and the room a conversation takes, like the room the weights take, is a property of the model’s architecture. The released model leaves room for one conversation within the envelope (§ 3.4); the co-designed model, held to the same quality, leaves room for twelve.
Comparing designs that serve different numbers of users calls for a measure that counts the users. The energy-delay of § 2.3 does not: it multiplies a token’s energy by the wait of the one user who receives it. That is the right delay when both sides serve the same number of users, as in the comparison with the B300 that follows, but it gives a design no credit for serving several users at once. By it the data-center design is 6.6x better than the unchanged model’s, little more than energy per token already showed.
The delay that describes a chip, rather than one of its users, is the time between the tokens it produces for all of them together, which is one divided by its total tokens per second. Energy per token multiplied by that delay is the energy-delay of the chip’s whole output, and it is the form in which the search of § 3.3 scores a design. For a fixed number of users the two forms differ only by that number, so the ratios against the B300 in § 3.7 do not depend on the choice.
Counting users in the delay is legitimate only where they are not paid for with each user’s time. On a GPU they are, since every added user lengthens the wait of all the others (§ 2.3), which is why the B300 was scored on the per-user delay. Here the design that serves twelve users gives each of them a token sooner than the design that serves one, so none of its greater output is bought with a longer wait.
Figure 10 sets the two designs side by side on that measure. The design for the unchanged model produces 17,512 tokens per second for its one user, and its energy-delay is 174 per token. The data-center design produces about 239,000 tokens per second across its twelve users, and its energy-delay is 2.2 , 79x lower. The factor is the product of

Figure 10. What the co-designed model adds, like for like: the two Exstill points of Figure 8, on Exstill silicon at a 16K-token context within the same envelope of 125 W and 575 . The first is the design with the lowest energy-delay for Qwen3-1.7B unchanged, which serves one user; the second is the data-center design with its co-designed model, which serves twelve. Energy-delay per token is energy per token divided by the chip’s total tokens per second, so its 79x is the product of the 5.8x and the 13.7x above it. The bars of each group share a linear scale from zero. Both designs are pre-silicon at the 5 nm class.
Table 4 | The Exstill data-center design against one NVIDIA B300 serving Qwen3-1.7B in FP8, each user with a 16K-token context, decoding only. The design is pre-silicon at the 5 nm class with its model held to Qwen3-1.7B’s quality; the B300 rows are measured (Table 1). Ratios are the B300’s figure over the design’s. The comparison is the row with the same twelve users; the other two rows give only the bound each setting is best at, the B300’s fastest time and its lowest energy.
| Users | Time between tokens | Energy per token | Faster | Less energy | |
| Exstill data-center design | 12 | ||||
| B300, the same twelve users | 12 | 110x | 690x | ||
| B300, fastest setting | 1 | x | |||
| B300, most efficient setting | 16 | x |
Table 4. The Exstill data-center design against one NVIDIA B300 serving Qwen3-1.7B in FP8, each user with a 16K-token context, decoding only. The design is pre-silicon at the 5 nm class with its model held to Qwen3-1.7B’s quality; the B300 rows are measured (Table 1). Ratios are the B300’s figure over the design’s. The comparison is the row with the same twelve users; the other two rows give only the bound each setting is best at, the B300’s fastest time and its lowest energy.
5.8x in energy per token and 13.7x in total output, so the larger part of what the model contributes lies in the second term, which an energy axis does not show.
Nor does the result rest on the area the envelope allows. Even on a full reticle, the largest die that can be printed in one exposure and half again the envelope’s area, and with no limit on its power, the unchanged model holds two conversations and has no room for a third, and its energy-delay falls to , still 31x the data-center design’s.
Insight 4: Energy per token understates what the model contributes
On silicon that keeps every conversation beside the arithmetic, the model decides how many conversations fit. Within the same budget of power and area, the co-designed model lowers a token’s energy 5.8x and raises the die’s total output 13.7x, with each user’s token arriving sooner, so the energy-delay of the chip’s whole output falls 79x. A comparison of the two models on energy per token alone shows the first of these and misses the second.
The Claim: Against the Best Single Chip
What remains is the practical comparison, against the best single chip available. Both chips’ figures depend on load, so it must be made on the same workload, at the same quality and for the same number of users. Table 4 sets the data-center design against the B300 serving the same twelve users from Table 1: each user’s next token is decoded 110x faster and costs 690x less energy, so energy-delay per token is about 74,000x lower, against . That like-for-like comparison is the one we use throughout. The table’s other rows show that the result does not depend on choosing a convenient setting for the GPU: at its fastest, serving a single user, the B300 is still 43x slower, and at its most efficient, serving sixteen, it still spends 660x the energy per token.
Figure 11 places the design on the plane of Figure 3. Batching moves the B300 within a band of nearly constant energy-delay, trading one axis for the other, and the design lies far outside that band, toward the corner of the plane where a token is both sooner and cheaper, so no setting of the dial closes either gap. A hand-tuned serving stack would narrow the ratios; § 3.10 estimates by how much.

Figure 11. The data-center design on the plane of Figure 3: energy per token against the time between a user’s tokens, both on logarithmic scales. The B300 points are the measured sweep of Table 1, one to sixteen users, and the shaded band is the band of Figure 3, the range of energy-delay that batching moves the B300 within. The Exstill point is pre-silicon at the 5 nm class, serving twelve users with its model held to Qwen3-1.7B’s quality. The ratios are against the B300 serving the same twelve users.
The title’s claim is meant in the following sense. A token’s energy is the energy of the arithmetic its model requires, multiplied by the multiple of it that the hardware actually spends, and the data-center design, on one die, makes both factors small. Its token costs 1.8x its arithmetic, where the B300’s costs several hundred times, and its arithmetic is about a third of Qwen3-1.7B’s, the model whose quality it matches. No token can cost less than its arithmetic. A chip running the same model in the same arithmetic could therefore improve on the design’s energy by less than a factor of two, and a chip running Qwen3-1.7B as released, in the arithmetic of § 2.1, could not match it at all, because that model’s arithmetic alone costs more than the design’s whole token. A larger improvement requires a model that needs less arithmetic at the same quality, on silicon that stays as close to that arithmetic, which is the co-design this section has described. The figures are pre-silicon, and § 3.10 sets out what else the claim rests on.
A Second Design: The Larger Model
The question of § 3.3 can be asked for any bar and any budget, and we have asked it a second time, for work in which accuracy comes first. The second design, which we call the DC2 design, is built around the 4B size of the co-designed architecture. Its bar is 0.585 bits per byte on the same probe texts, tighter than the first design’s, and its envelope is 150 W and 750 mm, with every user again holding a 16K-token context on the chip. The design returned uses 746 mm and draws 106 W, less than the envelope allows, and each of its four users receives 13,998 tokens per second. Across the four the chip produces about 56,000 tokens per second, or 531 tokens per joule.
71 μs
to decode a token, per user
1.88 mJ
per token, leakage included
4 users
16K tokens of context each
106 W
and 746 mm², pre-silicon
Both factors of the token’s cost behave as they did in the first design. The released model of this size is Qwen3-4B (Yang et al., 2025), and Equation (1) applied to its public configuration, about weights per token and 36 layers of 32 query heads of dimension 128, gives multiply-accumulates per token at a 16K-token context, a floor of 2.18 mJ. The DC2 design’s model has a floor of 0.74 mJ, measured the same way on our flow and again about a third of the released model’s. Its full token costs 1.88 mJ, 2.5x its own floor and less than the arithmetic alone of Qwen3-4B. What the larger model changes is the room it leaves: the die holds four conversations where the first design holds twelve (§ 3.6), and the token stands somewhat further from its floor.
Table 5 | The DC2 design against one NVIDIA B300 serving Qwen3-4B-Thinking-2507 in FP8 with vLLM as shipped, each user with a 15K-token prompt and 1K generated tokens, decoding only. The design is pre-silicon at the 5 nm class; the B300 rows were measured on 28 October 2026, with the time between tokens and the energy per token defined as in Table 1. Ratios are the B300’s figure over the design’s, and the rows are those of Table 4.
Table 5 makes the practical comparison in the form of Table 4, against one B300 serving Qwen3-4B-Thinking-2507 from its published FP8 checkpoint on the workload of § 2.2. For the same four users, each user’s next token is decoded 58x faster and costs 360x less energy, and at its most efficient setting the B300 still spends 222x the floor of the model it serves. The comparison is matched in model size, load and context. The bar is a value we set, and we have not measured Qwen3-4B against it on our probe texts; § 6 to § 8 report the task-level quality of the models built for this design.
| Users | Time between tokens | Energy per token | Faster | Less energy | |
| Exstill DC2 design | 4 | 71 μs | 1.88 mJ | ||
| B300, the same four users | 4 | 4.13 ms | 683 mJ | 58x | 360x |
| B300, fastest setting | 1 | 3.06 ms | ≥43x | ||
| B300, most efficient setting | 16 | 483 mJ | ≥260x |
Table 5.
Other Designs That Keep the Model on the Chip
Two of the designs named in § 3.2 have published enough to be set beside ours: Taalas’s HC1 [12] (Taalas 2026) and NVIDIA’s Groq 3 LPX (NVIDIA, 2026a,b). Table 6 collects what each states. The rows are not like for like. Apart from one estimate of our own, the other designs’ figures are their makers’ or a third party’s and describe larger models on other loads, and both are fabricated silicon where our designs are pre-silicon.
| Design | Model on the chip | Published speed | Silicon | Power |
| Exstill DC1 design | 1.7B, weights loadable, with twelve 16K conversations | 19,921 tokens/s for each of twelve users | One die, 526 mm, 5 nm class, pre-silicon | 125 W |
| Exstill DC2 design | 4B, weights loadable, with four 16K conversations | 13,998 tokens/s for each of four users | One die, 746 mm, 5 nm class, pre-silicon | 106 W |
| Taalas HC1 | Llama 3.1 8B, built into the chip, parameters at 3 and 6 bits | 17,000 tokens/s for one user, at 1K tokens in and 1K out | One die, 815 mm, TSMC 6 nm | 2.5 kW a server |
| NVIDIA Groq 3 LPX | Any model, spread over LPUs of 500 MB of SRAM each | 3,400 tokens/s on Gemma 4 31B at a 100K context | A rack of 256 LPUs, 128 GB of SRAM, beside Vera Rubin GPUs | Not stated |
Table 6. Our two designs, the data-center design (DC1) and the DC2 design, beside two published designs that keep a model on the chip. The figures for HC1 are Taalas’s own ([12] (Taalas 2026)); those for the LPX are NVIDIA’s (NVIDIA, 2026b), with the token rate Artificial Analysis’s measurement of one rack as reported by The Decoder (2026). The rows differ in model, load and maturity and are not a like-for-like comparison.
What can be compared is what each design keeps on the chip, and what that choice buys and costs.
HC1 takes the dedicated-silicon end of the trade described in § 3.1. With Llama 3.1 8B built into the chip, weights included, Taalas reports 17,000 tokens per second for one user, the same order as our designs on a model two to five times the size of ours, and on silicon that exists. It pays for this in three places. The model is fixed: a new model is a new chip, which Taalas expects to produce in about two months, with low-rank adapters as the adjustment in between, where our designs load a new model of their architecture as weights. The parameters are quantized to 3 and 6 bits, which Taalas says costs some quality against GPU results. And the published figures do not settle a token’s cost. The rate is for 1K tokens in and 1K out, and Taalas gives a server’s power, 2.5 kW, but neither a chip’s power nor the number of users a chip serves at once, so neither energy per token nor the cost of a long context can be derived from them.
We estimated what the quantization costs by testing Taalas’s public endpoint. The service returns no token probabilities, so its bits per byte cannot be measured directly. We asked it to continue the openings of public documents, read its answers back through Llama 3.1 8B in 16-bit precision and through 35 quantized versions of that model whose bits per byte we had measured, and predicted the service’s from how often its choice of token departs from the 16-bit model’s. The estimate is 14.6% more bits per byte than the 16-bit model, with a 95% interval of 12.5% to 16.9%. On our probe texts at a 16K-token context that is 0.63 bits per byte against 0.55, a projection, since the service holds at most 6,144 tokens; at 4K, within what it holds, it is 0.62 against 0.54. By this estimate the 8B model built into HC1 answers at about the quality of a model of the 1.7B size, comparable to that of the models our data-center design runs and if anything slightly better.
The LPX rests on the premise of this report, that decoding deserves memory beside its arithmetic, at the other end of model size. Each of its chips, a language processing unit (LPU), carries 500 MB of SRAM and a rack carries 256 of them, so a model is spread over as many chips as its weights fill: Gemma 4 31B occupies at least 64 (The Decoder, 2026). By the same count, the published FP8 checkpoint of Qwen3-1.7B, 2.7 GB, fills six LPUs, and that of Qwen3-4B-Thinking-2507, 5.2 GB, eleven. The weights, however, are the smaller part of what a token needs, and whether the rest must also be held on the LPUs depends on how the rack is used. NVIDIA describes two uses (NVIDIA, 2026a). In one, the LPUs run the feed-forward part of each layer while attention over the context stays on the Vera Rubin GPUs, and the activations of every token cross between the two. In the other, the LPX generates draft tokens for speculative decoding, the role of § 5, with its model’s whole working set, weights and key-value state, in on-chip memory. In that use the count must include the context. The twelve 16K conversations of the data-center design’s workload add about 23 GB at the cache precision of § 2.6, for some fifty LPUs in all, and the four of the DC2 design’s add about 10 GB, for thirty.
The same workload is thus one die in our designs and tens of chips in the other, or a few chips beside a GPU that keeps the context. Both keep the model in memory on the chip. The difference lies in how much co-design leaves to be stored (§ 3.2), and in building one die for a small model rather than a rack for models of tens of billions of parameters and more, the work the LPX is built for. On speed the public record does not reach our workload: the one measured rate is 3,400 tokens per second on Gemma 4 31B at a 100K-token context (The Decoder, 2026), and NVIDIA’s own projections are for models of two trillion parameters, about 400 tokens per second per user (NVIDIA, 2026a).
Limitations
The chip’s figures are pre-silicon. Its circuits are measured on our gate-level flow, not on fabricated silicon, and the whole-chip figures are our estimate until silicon is measured; effects that a gate-level flow does not capture will appear only then.
The comparison is also bounded on the GPU side. The B300 figures are for one GPU running vLLM as shipped, which is what most operators run, and cover decoding only, with prefill, the reading of the prompt, excluded on both sides. Of the 2.15 ms the B300 takes per token for one user, roughly 1.7 ms is not memory traffic, most of it the fixed charge that § 2.7 attributes to the layers, and a hand-tuned serving stack would remove part of it. We estimate that such a stack would narrow each ratio against the B300 by about half. It would not close any of them, because the memory traffic of § 2.6 is set by the model and its context rather than by the software. With the caches as configured here, reading the weights and twelve conversations occupies about 3 ms of the twelve-user step at the published bandwidth, however well the layers above it are tuned.
The design is also fixed where the GPU is not. It holds twelve conversations and runs models of one architecture, whereas a B300 can be loaded with any model that fits its memory and, as Table 1 shows, with more users at a further cost in each user’s time. The like-for-like comparison is made at twelve users because that is the design’s full load; serving more means more chips, and what one architecture can do is the subject of § 4.
The floor is specific to the datapath we measured, a 16-wide FP8 dot product accumulating into 16-bit floating point at the 5 nm class. A different datapath would place it somewhat higher or lower without changing the size of the gap that § 2 describes. The search measures quality in bits per byte on our probe texts, which makes designs comparable across models and tokenizers but is not a task-level evaluation; the role sections that follow report task-level quality for each model on the benchmarks named there.
Finally, the measured comparison is with one GPU. We have not measured the specialized designs named in § 3.2 on this workload, and against them the claim rests on the limit, in the form § 3.7 gives it, rather than on a bench. § 3.9 sets two of them beside ours from their published figures, which are not measurements of this workload.
Polyglot-DC: One Architecture, Many Roles
Every frontier model operates within a crew of smaller models. Along the path of a single request, one model screens the prompt before the frontier model sees it, another drafts tokens ahead while the frontier model writes, a third selects an agent’s next tool call, and a fourth reasons through a document while a person waits for the answer. The frontier model determines what the answer says, but these surrounding models largely determine how long the user waits for it. Today each of them is a different architecture, built, tuned and served on its own.
This section introduces Polyglot-DC, a family of small language models co-designed with Exstill silicon. It answers the question that § 3.1 left open, whether one architecture can do enough useful work to justify a chip with that architecture fixed in the silicon. Rather than designing hardware to accelerate four unrelated models, we co-designed a single model architecture together with the silicon it runs on, and then specialized it through distillation for the four workloads this report studies. Three principles guided this approach. The first is specialization, in which a small model is trained for the task it performs instead of serving as a scaled-down generalist. The second is co-design, the model architecture and the hardware optimized together rather than in turn. The third is a shared foundation, one architecture supporting many workloads, so that adding a workload does not require a new chip.
The name expresses these principles. Polyglot refers to one model architecture fluent in many roles, identical for every job, with only the weights changing. DC refers to the silicon the models are built for and designed with, our data-center designs. The architecture comes in two sizes, each with a design of its own: Polyglot-DC1, with 1.7B parameters, on the data-center design of § 3.5, and Polyglot-DC2, with 4B, on the DC2 design of § 3.8. Each role is a set of weights for one of the two, named for the role, as in Polyglot-DC1-Draft.
Small Models, Trained for the Job
A frontier model is a generalist, and to be prepared for any request it carries hundreds of billions or even trillions of parameters. The jobs that surround it have the opposite character. A guard assigns each message to one of a fixed set of labels, a draft model predicts the next few tokens of one specific target model, and an agent step usually amounts to a single tool call in a fixed format, work that is narrow, repetitive and well defined. On tasks of this kind, a small model trained for the task can match a far larger general model at a fraction of the cost to serve it.
Belcak et al. [14] develop this argument for agentic systems. They observe that agents exercise only a narrow slice of what a language model can do, while requiring strict and consistent output formats, a combination that a model below roughly 10B parameters can be fine-tuned to deliver. On this basis they estimate that serving a 7B model is 10-30x cheaper than serving a model of 70-175B parameters, measured in latency, energy and compute, and that 40-70% of the large-model calls in three open agent frameworks could be served by a small model instead. Subsequent work supports this view. Zhou et al. (2026) post-train an 8B model to select tools from a library of more than ten thousand, and it outperforms a 402B model on ToolBench (Qin et al., 2024), a benchmark built from more than 16,000 real-world application programming interfaces (APIs). The same pattern recurs in this report’s own results for safety classification, drafting and financial reasoning.
The Model and the Silicon, Designed Together
A smaller parameter count does not by itself produce lower latency, as Figure 4 showed on a GPU. The performance a model actually achieves depends on how efficiently its architecture maps onto the hardware that runs it, and a growing body of work therefore designs the two together. The prevailing method is an architecture search in which each candidate model is scored not only on accuracy but also on its speed on the target hardware. Puzzle (Bercovich et al., [15]) applies this method to derive, from the 70B-parameter Llama model, a model that fits on a single NVIDIA H100 GPU and runs 2.17x faster while retaining 98.4% of its parent’s accuracy. Jet-Nemotron (Gu et al., [45]) reaches the accuracy of leading open models at 2B parameters while generating tokens up to 53.6x faster. The same idea has been applied on phones, where MobileLLM-Flash (Huang et al., [50]) searches against latency measured on the device itself and MobileMoE (Chen et al., [21]) sizes a mixture-of-experts model to the device’s memory and compute. Sun et al. (2026) take this direction further by deriving co-design scaling laws from a roofline model of the target chip, an analytical model that bounds achievable performance by either the chip’s arithmetic throughput or its memory bandwidth.
These studies demonstrate the value of hardware-aware model design, but they share one constraint, that the hardware is fixed and only the model may change. We remove it. Rather than fitting a model to an existing accelerator, we treat the model architecture and the silicon architecture as variables in a single design space, so that decisions on each side inform the other from the outset. As § 3.3 describes, the search scores every candidate pairing by the time and energy of each token it generates, measured on placed and routed circuits. Hardware-aware model design and model-aware hardware design are each well explored. What is harder, and what the rest of this section relies on, is doing both in one loop without giving up fidelity on either side: an architecture trained and judged in hours while the silicon for it is costed at the gate level.
The chip that emerges from this search is shaped around one architecture, with its memory, compute engines and data paths sized for that architecture and no other. Designing silicon separately for every model would fragment the platform into one chip per model, whereas a shared architecture lets the same silicon take on a new task as a new set of weights. Polyglot-DC is that shared architecture, and each of its jobs is a separate set of weights produced through distillation.
Insight 5: Fix the architecture, vary the weights Co-designing silicon around a single shared architecture captures the efficiency of hardware built for one model while retaining the flexibility of a programmable part. A new workload becomes a training run on silicon that already exists, rather than a new chip.
Four Jobs, Four Distillations
Although they share an architecture, the four jobs are distinct machine-learning tasks, summarized in Table 7 and illustrated in Figure 12 with one request of each kind and the output its role returns. Drafting for speculative decoding, a technique in which a small model proposes tokens that a large model then verifies, is next-token prediction aligned to one target model, and it is judged by how many of the draft’s proposals the target accepts. Guardrails are a classification task in which, for each prompt and each response, the model produces a safety label, the category of harm and whether the assistant refused, across six languages. Agentic work is multi-turn tool use, in which the model reads a policy and a conversation, chooses the right tool call with the right arguments, and reasons between calls. Real-time reasoning is chain-of-thought question answering over financial text and tables, in which the model writes out its reasoning before giving an answer.
| Role | Task | Output | Measure of success |
| Speculative decoding | Next-token prediction aligned to one target model | A short chain of proposed tokens | Tokens accepted by the target per verification |
| Guardrails | Classification of prompts and responses | Safety label, harm category, refusal flag | F1 against reference labels, in six languages |
| Agentic | Multi-turn tool use under a policy | Reasoning, then a tool call with arguments | Correct actions over a task |
| Real-time reasoning | Chain-of-thought question answering over financial text and tables | Reasoning, then an answer | Answer accuracy |
Table 7. The four Polyglot-DC roles. Each is a separate set of weights for the same architecture. F1 is the harmonic mean of precision and recall.

Figure 12. The four roles of Polyglot-DC. One request of each kind, the chip holding the role’s weights, and the output that comes back: a chain of tokens proposed for a larger target model, a safety verdict, a tool call, a reasoned answer. The requests and the guard’s, agent’s and finance model’s outputs are the models’ own; the draft’s tokens are in the target model’s vocabulary and are drawn as a chain.
With each role its own set of weights, the central challenge is not simply to compress a larger model into a smaller one but to preserve the capabilities a particular workload needs while discarding the general-purpose capacity it does not. Knowledge distillation provides the foundation for this kind of specialization. In distillation, a larger model that already performs the job well, the teacher, supervises the training of a small model, the student. In its original form [48], the student learns to match the teacher’s full probability distribution over the next token rather than a single correct label, so that every training token conveys how the teacher weighs each alternative and carries far more signal than a hard label can. For language models, the teacher can also generate complete sequences for the student to imitate (Kim and Rush, 2016). More recent methods go a step further and train on sequences generated by the student itself, scored against the teacher, so that the student is corrected on the mistakes it actually makes rather than on text it would never have produced [3]. The measure used to compare the two distributions also shapes the result. Under the forward Kullback-Leibler (KL) divergence, the student spreads its probability across the teacher’s whole distribution, whereas under the reverse KL divergence it concentrates on the teacher’s most likely outputs.
The literature on distillation is extensive, and its central finding is consistent: a small model distilled for one task can reach, and sometimes surpass, models several times its size. Distillation scaling laws show that when a capable teacher already exists, distilling a student outperforms training it from scratch, up to a compute budget that grows with the student’s size [17]. DistiLLM-2 brings 1.5B students to within two points of their 7B teacher on MATH, and above the previous generation’s 7B teacher (Ko et al., 2025), and agent distillation transfers the full behavior of a 32B agent, including its use of retrieval and code tools, into students of 0.5B to 3B parameters (Kang et al., 2025b). Distillation also benefits drafting directly: training the draft model on the target it serves aligns its proposals with the target’s choices and increases the speedup of speculative decoding (Zhou et al., 2024). On-policy self-distillation improves reasoning with far fewer training tokens than reinforcement learning requires (Zhao et al., 2026).
Polyglot-DC applies the technique role by role, distilling each of the four roles from systems that already perform it well into the one architecture the chip is built for. The difficulty lies not in distillation itself but in the requirement that a single fixed architecture, chosen for the silicon rather than for any one task, match purpose-built models on tasks as different as these. Conventional distillation alone did not meet it, and the task-specific training we developed is what specializes Polyglot-DC for each role while leaving its common architecture untouched.
Beneath their differences, the four roles share one property that matters for the hardware. In each, a person or an agent waits while the model writes its output one token at a time, which places the workload in the regime of single-stream decoding that § 2.2 measured on a GPU and that the chip is built for. Throughout the role sections, reported times are therefore decode times, meaning the time to generate the output tokens. Reading the prompt into the model, network transfer and the rest of an application add to these times, on our chip and on the GPU alike.
Outline of the Role Sections
Outline of the Role Sections
Each of the four sections that follow takes one role from the problem it solves in a production system, and why that problem benefits from dedicated silicon, to the Polyglot-DC model built for it and an evaluation of that model’s accuracy and latency.
Speculative decoding. A small draft model proposes several tokens and the target checks them all in a single pass, so the target writes the same text in fewer passes. Polyglot-DC1-Draft is a standalone draft built for five open targets. The section evaluates acceptance and end-to-end decoding speedup.
Guardrails. A guard screens each prompt before the frontier model sees it and each reply before the user does, and a slow guard invites screening less. Polyglot-DC1-Guard and Polyglot-DC2-Guard give their verdict in a handful of tokens and operate in six languages. The section evaluates their accuracy and false-alarm rates against purpose-built guards, together with their latency.
Agentic. An agent reasons, calls a tool, reads the result and decides what to do next, many times within a single task, and every step waits for the one before it. Polyglot-DC2-Agent is a reasoning model for multi-turn tool use under a written policy. The section measures it against open models of its size on function-calling, coding and customer-service benchmarks, then replays eight of its completed cases to compare its decode time on the chip and on a GPU.
Real-time reasoning. A reasoning model writes thousands of tokens before it answers, and in finance an answer can lose its value within seconds. Polyglot-DC1-Finance and Polyglot-DC2-Finance reason through filings, valuation, risk and derivatives. The section evaluates them on ten finance benchmarks against open models of comparable size and measures the decode time of long chains of thought.
Model sizes. Polyglot-DC1, the 1.7B model, is intended for jobs that run on every request, and Polyglot-DC2, the 4B model, for jobs in which accuracy comes first. Each role section reports both sizes where they apply and discusses the trade-off between accuracy, single-user speed and the number of conversations a chip can hold at once.
Speculative Decoding on Its Own Silicon
A large language model writes one token at a time. For a single user, each token is a full forward pass through hundreds of billions of parameters, and at that batch size the pass is bound by reading the weights from memory rather than by arithmetic, so the GPU’s compute sits mostly idle while the weights stream past. Speculative decoding puts that idle arithmetic to work. A small draft model proposes the next few tokens, and the large model, the target, verifies all of them in one pass, keeping those that match what it would have written itself. Every token the user sees is still the target’s own; the answer simply arrives in fewer passes. The technique is now standard at the frontier, where the major closed models are served with a drafter of some kind.
Where the draft runs receives less attention, even though it determines how much of the gain survives. In the GPU designs in production today, the drafter shares the target’s accelerator and often depends on the target’s internal state. We take a different approach. Polyglot-DC1-Draft, a standalone draft on the shared Polyglot-DC architecture of § 4, is distilled separately for each of five open targets and runs on its own chip while the target stays on the GPU.
How Speculative Decoding Works
Each round of speculative decoding has four steps, which Figure 13 follows through an example. The draft model proposes speculative tokens, one at a time, and sends them to the GPU, where the target verifies all in a single forward pass. Verification accepts a prefix of the proposals and stops at the first token that does not match. The same pass also yields one token from the target itself. At a miss, the target’s own token replaces the rejected proposal, and when all proposals match, the target supplies the token after them. Each pass therefore yields the accepted draft tokens plus one.

Figure 13. One round of speculative decoding as it is usually run, on one GPU, with a chain of drafted tokens, the steps numbered in order. The draft model writes six tokens one after another, a small pass each. One forward pass of the target then scores all six positions at once, accepts the three that match what it would have written, replaces the first that does not with its own token, and discards the rest. The pass yields four tokens where the target alone would have written one, an accepted length of 4, and the next round drafts from there. On a GPU the draft and the target take turns on the same device, so the draft’s passes are added to every round. The words are illustrative.
The result is fewer target forward passes without any change to the target’s behavior. At temperature zero, verification is a greedy comparison, and the output is identical to what the target would have produced alone. At non-zero temperatures, speculative sampling [18] (Leviathan et al. 2023) accepts and rejects proposals with probabilities chosen so that the output follows exactly the target’s own sampling distribution, though not necessarily as the same sequence.
The measure of a draft is therefore its accepted length, the number of tokens each verification pass yields, including the target’s own token, averaged over the passes of a workload. A draft that averages 4.4 accepted tokens per pass lets the target advance by 4.4 tokens for each pass it makes, before the cost of drafting and the extra work of verification are counted.
Both costs matter. For one user, verifying tokens costs little more than writing one, because the pass is limited by reading the target’s weights rather than by arithmetic (Mitra et al., 2026); as users are added, and on mixture-of-experts targets where each verified token routes to experts the plain step did not load, the pass grows more expensive [51] (Mitra et al. 2026; Ranjan et al. 2026). The three quantities combine in one expression (Leviathan et al., 2023; Mitra et al., 2026). In units of the target’s own decode step, a round of speculative decoding speeds decoding by
where is the accepted length, the cost of the verification pass, equal to 1 when verifying tokens costs no more than writing one and larger as users and experts are added, and the drafting and transfer time the target waits for. The accepted length is a property of the draft and its target together, and § 5.5 and § 5.6 measure it. The drafting cost is a property of where the draft runs, and it is what the Exstill chip removes (§ 5.4).
How Speculative Decoding Is Done on GPUs Today
The first speculative decoding papers [18] (Leviathan et al. 2023) used standalone draft models, small language models, often from the target’s own family, run before each verification pass. On a GPU, however, even a small standalone drafter pays the per-token overhead of a full decode step, the fixed charge that § 2.7 found beneath every small model. In our measurement on an NVIDIA B300 with vLLM (Kwon et al., 2023), Qwen2.5-0.5B decodes at 793 tokens per second for one user, or 1.3 ms per token. Six drafted tokens therefore take about 7.6 ms, four times longer than the entire decode step of gpt-oss-120b, which SemiAnalysis’s InferenceX benchmark (SemiAnalysis, 2026) measures at about 1.9 ms for one user on a GB200 system. In this arrangement a standalone GPU drafter costs more than it saves, which is why production GPU drafters took another shape and attached themselves to the target. They did not do so for want of acceptance. NVIDIA’s own comparison of drafting mechanisms finds that standalone draft models reach the highest accepted length of any it tests, above 5 tokens at a chain of nine for a 4B draft of Qwen3.5-122B-A10B, where its multi-token-prediction (MTP) and DFlash heads level off earlier, and it prices that acceptance at a full forward pass of the draft per token (Mitra et al., 2026).
EAGLE-3 (Li et al., 2025b) drafts from the target’s own hidden state. Its drafter is a single decoder layer with a language-model head that reads features fused from several of the target’s layers, so it reuses computation the target has already done and a draft step costs a fraction of a millisecond. The drafter is trained to draft several steps ahead, and the EAGLE-3 paper reports speedups of up to 6.5x for a single user. EAGLE-3.1 keeps the design and adds a normalization scheme [35] that stops the drafter’s hidden state from growing with the depth of the chain, which raises acceptance and slows its decay at depth and at long context. NVIDIA’s EAGLE-3 head for gpt-oss-120b reaches an accepted length of 3.28 on SPEED-Bench coding at a speculation length of , and LightSeek’s EAGLE-3.1 head for Kimi K3, built with multi-head latent attention to match its target, reaches 3.10 at , both according to their model cards. On LightSeek’s own serving stack the Kimi K3 head decodes 2.15x faster for one user and 1.51x for sixteen.
Two more recent designs change the drafter itself. DFlash [19] replaces the token-by-token drafter with a block diffusion model that proposes a whole block in a single forward pass, conditioned on context features from the target, and its paper reports up to 2.5x the speedup of EAGLE-3. DSpark [24] pairs a parallel drafting backbone with a light sequential module and schedules the verification length per request from the acceptance it expects and the engine’s throughput profile; its paper reports that in DeepSeek-V4’s serving system it raises per-user speed by 60-85% over the single-token MTP baseline at matched throughput. Both are designed for the GPU’s economics, with fewer, wider passes and verification sized to the batch.
Across these designs two features recur. The drafter is tied to its target, through the target’s hidden states or its serving engine, so supporting a new target generally calls for a new drafter. And draft generation runs on the target’s GPUs, sharing their compute and scheduling with verification. The first limits reuse across models. The second adds overhead precisely where speculative decoding is supposed to save time, and increasingly so as more requests share the GPU. Both motivate a different design.
Polyglot-DC1-Draft: One Architecture, Five Open Models
Polyglot-DC1-Draft is that different design, a standalone draft model built on Polyglot-DC1, the 1.7B size of the shared architecture. We co-designed the architecture with Exstill silicon, then distilled and aligned its weights separately for each target. Each draft operates in its target’s tokenizer and vocabulary without reading the target’s internal state, through a proprietary method of Exstill’s. The architecture is the same across the five targets in Table 8, which span 80B to 2.8T parameters, 3B to 104B active parameters per token, and five different tokenizers and reasoning behaviors.
| Target | Parameters | Active | Reasoning effort |
| gpt-oss-120b | 117B | 5.1B | high |
| Qwen3-Next-80B-A3B-Thinking | 80B | 3B | thinking on |
| DeepSeek-V4-Pro-0813 | 1.6T | 49B | max |
| Kimi K3 | 2.8T | 104B | max |
| Muse Spark Open | 816B | 42B | xhigh |
Table 8. The five open target models Polyglot-DC1-Draft is distilled for. Each has its own tokenizer; the draft architecture is the same for all five, with a separate set of weights per target. Reasoning effort is the setting at which each target’s answers were generated for the evaluations in this section.
With the architecture and the serving integration fixed, switching targets means loading different distilled weights and the corresponding tokenizer, and a sixth target is a training run rather than a new hardware design. The draft reads nothing inside the target, so the first of the two features that recur in the GPU designs is gone.
Why Draft on the Exstill Chip
The second recurring feature, drafting on the target’s GPUs, is what the chip removes. In our simulation of the pre-silicon data-center design, each stream decodes at about 50 per token, so a chain of a dozen tokens is drafted in well under a millisecond, a small fraction of the time the target takes to verify it. Figure 14 sets the two ways of drafting against the pass they serve, where six tokens take four times as long as the pass on a GPU and about a sixth of it on the chip. The chip can also prepare the next draft while the GPU verifies the current one, overlapping much of the drafting with verification. In the terms of Equation (2), is about 4 for the GPU drafter of § 5.2 and about 0.16 for the chip, and the overlap takes it toward zero. Figure 15 draws the arrangement, with the draft on its own card, the target on whatever system serves it, and only token identifiers crossing between them each round, the proposals in one direction and the verdict in the other, as the count accepted and the target’s own token. Table 9 sets this arrangement beside the GPU drafters of § 5.2, in the terms of NVIDIA’s own comparison (Mitra et al., 2026). An independent analysis of draft disaggregation arrives at the same arrangement, the draft decoding on its own accelerator while the target stays on the GPU (Kuznetsov and Azizi, 2026).

Figure 14. What drafting costs against the pass it serves, on one linear time scale. The target’s pass is one decode step of gpt-oss-120b for one user, as measured by InferenceX (SemiAnalysis, 2026), which verifying a short chain barely lengthens (§ 5.1); the dashed line carries it down for comparison. Six tokens drafted on a B300 is our measurement of Qwen2.5-0.5B under vLLM (§ 5.2), a model less than a third the size of Polyglot-DC1-Draft. Six tokens drafted on the Exstill data-center design is pre-silicon, at 50 per token.

Figure 15. Speculative decoding across two machines, in the draft-then-verify flow of Xia et al. (2024), the steps numbered in the order of one round. Each machine reads the prompt for itself and keeps its own cache, so no cache ever crosses the link between them. The Exstill card drafts six tokens one after another at about 50 s each, every token conditioned on the ones before it, and the six token identifiers cross to the target, gpt-oss-120b, Qwen3-Next, Kimi K3, DeepSeek-V4-Pro or Muse Spark Open, on whatever system serves it. One forward pass of the target writes its own token for every position. The drafts are kept while they match: four do, the fifth differs, so the target’s own token takes its place and the sixth is discarded. What returns to the card is the count of drafts accepted and that one token; both machines append the same five tokens to their caches, and the next round drafts from there. Had all six matched, the target’s seventh token would have come back as a bonus and the round would have yielded seven.
| Polyglot-DC1-Draft on the Exstill card | EAGLE-3 | MTP | DFlash | DSpark | |
| Draft architecture | A standalone small model, one architecture, weights per target | A decoder layer with a linear projection | A decoder layer with a linear projection | Decoder layers with linear KV fusion | Decoder layers with KV fusion and a light sequential head |
| Inputs | Token identifiers | The previous token’s embedding and the target’s hidden states | The target’s final hidden state | The target’s fused hidden states, as the draft’s KV cache | The target’s fused hidden states, as the draft’s KV cache |
| Generation | autoregressive passes, as a chain or a tree, pre-drafted while the target verifies | The module run times, serially | The module run times, serially | tokens in one parallel pass | tokens in one parallel pass, then a light serial correction |
| Runs on | The Exstill card, beside the target’s system | The target’s GPUs | The target’s GPUs | The target’s GPUs | The target’s GPUs |
| Release method | A new set of weights, the same draft architecture and silicon for any target | A separate head, trained after the target | Part of the target’s checkpoint | A separate head, trained after the target | A separate head, trained after the target |
| Activation memory cost | Draft weights and a sparse KV cache | Weights and a small KV cache | Weights and a small KV cache | Weights and a small KV cache built from the target’s | Weights and a small KV cache built from the target’s |
| Speculation cost per tokens | A full draft forward per token, on the card | One layer, serial | One layer, serial | About five layers, once | About five layers once, then the light head serially |
| Best for | Exstill beside a GPU target. Not recommended on GPUs alone | Not recommended, lower acceptance than MTP | Larger models on GPUs | Smaller models at batch size 1 | Smaller models at batch size 1 |
Table 9. Draft mechanisms compared, after NVIDIA’s table (Mitra et al., 2026), with the Exstill arrangement added as the first column. The GPU mechanisms are as NVIDIA characterizes them. The first column is Polyglot-DC1-Draft on its own card, which needs only the target’s token identifiers and drafts off the GPU. In NVIDIA’s own measurements the standalone draft also reaches the highest accepted length of these mechanisms (§ 5.2).
The draft needs none of the target’s hidden states, so the traffic between the two machines is tens of bytes per round and drafting stays off the GPU entirely. To measure what sharing the target’s GPU costs instead, we ran a 1.7B draft as the draft-model speculator of vLLM 0.31 alongside gpt-oss-120b on one B300. Decoding was 4.7-8.0x slower than the target alone for one user and 2.8-4.4x slower for eight, and slower still with the two models split across two B300s. The time went to the draft’s serial passes inside the serving loop, each costing several times what the same pass costs when the draft is served alone, not to the target’s own step.
In the same simulation, one card holds twelve streams at once, so a single card drafts for twelve people, for any of the five targets, by loading the weights for that target. Each stream’s slot is sized for a 16K-token context, but the draft is not bound by it. Exstill’s proprietary long-context method lets it keep drafting at contexts many times longer, for agents and long conversations, with its memory per stream fixed, so twelve long sessions fit the card as well as twelve short ones.
Off the Shelf, or Co-Designed and Distilled for the Target
Moving drafting off the GPU addresses where the work runs. The next question is how well the draft predicts its target, and the natural alternative to a purpose-built drafter is a small open model from the target’s own family. For Qwen3-Next-80B-A3B-Thinking, Qwen3-1.7B is a particularly strong candidate. It shares the tokenizer, and it was distilled by Qwen from Qwen3-32B and Qwen3-235B-A22B, first off-policy and then on-policy, on top of 36 trillion tokens of pretraining (Yang et al., 2025). No off-the-shelf model is better placed to draft for this target.
Polyglot-DC1-Draft for Qwen3-Next has the same parameter count as Qwen3-1.7B and shares nothing else with it. Its architecture is Polyglot-DC’s, found by exploring the model and the chip as one design space, and its weights are distilled from Qwen3-Next’s own outputs and aligned to its choices. Table 10 sets the two side by side, scored by us on the same cached answers in the same way with our evaluation harness, on SPEED-Bench [1], a benchmark for speculative decoding that pools eighteen public datasets into eleven task categories, and on the nine datasets that DSpark reports on [24]: GSM8K [26], the 500-problem subset of MATH [47] (Lightman et al. 2024), the 2025 American Invitational Mathematics Examination (AIME), HumanEval [20], MBPP [11], LiveCodeBench [53], MT-Bench (Zheng et al., 2023), AlpacaEval [34] and Arena-Hard v2 (Li et al., 2025a).
Table 10 | The same size, off the shelf or co-designed. Accepted length per verification pass for Qwen3-Next-80B-A3B-Thinking with a chain of drafted tokens, the target’s own token included, under verification of the target’s cached answers. Both drafts have 1.7B parameters and the target’s tokenizer. The last column is the relative gain of Polyglot-DC1-Draft over Qwen3-1.7B.
| Benchmark | Polyglot-DC1-Draft | Qwen3-1.7B | Gain |
| SPEED-Bench, all | 3.68 | 3.16 | +16% |
| SPEED-Bench, coding | 4.86 | 3.81 | +28% |
| Mathematical reasoning | |||
| GSM8K | 5.24 | 4.45 | +18% |
| MATH-500 | 5.12 | 4.18 | +22% |
| AIME 2025 | 4.57 | 3.43 | +33% |
| Coding | |||
| HumanEval | 4.81 | 3.58 | +34% |
| MBPP | 4.58 | 3.09 | +48% |
| LiveCodeBench | 4.32 | 3.07 | +41% |
| General chat | |||
| MT-Bench | 3.65 | 3.02 | +21% |
| AlpacaEval | 3.33 | 2.76 | +21% |
| Arena-Hard v2 | 3.62 | 3.17 | +14% |
| All nine datasets | 4.18 | 3.32 | +26% |
Table 10.
Polyglot-DC1-Draft yields more tokens per pass on every row. Pooled over the nine datasets it averages 4.18 tokens per pass against 3.32, a gain of 26%, and the gap is widest on code and competition mathematics, at 48% on MBPP, 41% on LiveCodeBench, 34% on HumanEval and 33% on AIME 2025. Both drafts are the same size and run at the same speed on the same chip, so every one of these differences carries straight through to the speedup.
Table 11 shows how far. It prices each draft on the Exstill data-center design with our simulator at the chain length that gives it the highest end-to-end speedup, with the chip pre-drafting the next round while the GPU verifies the current one. Qwen3-Next alone decodes at 274 tokens per second for one user, measured by us on one B300 with vLLM and taken for a DGX B300 at tensor parallel 8. The gains in speed exceed the gains in accepted length because a draft whose proposals survive longer can profitably run a longer chain, and because the fixed cost of a round is spread over more accepted tokens. Pooled over the nine datasets, Polyglot-DC1-Draft decodes 4.37x faster than the target alone against 3.13x for Qwen3-1.7B, a 40% advantage, and on code the advantage reaches 77% on MBPP and 62% on LiveCodeBench. Figure 16 draws the two tables on a common scale, where the widening from accepted length to speed is visible benchmark by benchmark.
| Benchmark | Polyglot-DC1-Draft Speedup | Polyglot-DC1-Draft | Qwen3-1.7B Speedup | Qwen3-1.7B | Gain |
| SPEED-Bench, all | 3.57x | 13 | 2.94x | 11 | +21% |
| SPEED-Bench, coding | 5.68x | 16 | 3.81x | 14 | +49% |
| Mathematical reasoning | |||||
| GSM8K | 6.57x | 16 | 4.83x | 15 | +36% |
| MATH-500 | 6.27x | 16 | 4.42x | 15 | +42% |
| AIME 2025 | 5.06x | 15 | 3.29x | 14 | +54% |
| Coding | |||||
| HumanEval | 5.55x | 16 | 3.48x | 14 | +59% |
| MBPP | 5.06x | 15 | 2.86x | 11 | +77% |
| LiveCodeBench | 4.59x | 15 | 2.84x | 11 | +62% |
| General chat | |||||
| MT-Bench | 3.53x | 14 | 2.77x | 11 | +27% |
| AlpacaEval | 3.10x | 11 | 2.48x | 9 | +25% |
| Arena-Hard v2 | 3.50x | 14 | 2.94x | 11 | +19% |
| All nine datasets | 4.37x | 15 | 3.13x | 12 | +40% |
Table 11. End-to-end speedup over Qwen3-Next-80B-A3B-Thinking alone for one user, simulated on the pre-silicon Exstill data-center design, with each draft at the chain length that gives it the highest speedup, the next round pre-drafted during verification, and the GPU’s verification cost taken from our B300 measurement. A request is a 2,000-token prompt and a 512-token answer. Accepted lengths are measured as in Table 10; the speedups are simulated. The last column is the relative gain of Polyglot-DC1-Draft over Qwen3-1.7B.

Figure 16. Off the shelf or co-designed, benchmark by benchmark, for Qwen3-Next-80B-A3B-Thinking. Left, accepted length per verification pass at a chain of (Table 10, measured), on a scale that runs from 1, the target’s own token alone, to 7, all six proposals accepted. Right, end-to-end speedup over the target alone at each draft’s best chain length (Table 11, simulated). The benchmarks are grouped as in the tables, and the printed gains are those of the pooled row and of MBPP, the largest.
With the same size, the same tokenizer and the same target, what separates the two is the model, its architecture and what it was trained to do. A draft is not judged by its own quality but by how often its proposals survive the target’s verification, which is a property of the pair. Qwen3-1.7B was distilled to be a good model in its own right. A drafter is distilled to agree with one target, token by token, which the literature frames as aligning the draft’s distribution to the target’s under the verification rule (Zhou et al., 2024). Exstill’s own distillation and alignment methods are built for that objective and remain proprietary.
Co-design changes not only what the draft proposes but what silicon can be built around it. In our pre-silicon estimates, a chip built around Qwen3-1.7B as released would spend per drafted token, against for the Exstill data-center design running Polyglot-DC1-Draft, or 5.8x as much energy. These are the two Exstill points of Figure 8, costed within the same envelope, and § 3.6 sets out what else separates them.
Insight 6: A draft is judged by agreement, not by quality Two drafts of the same size and tokenizer can differ by a quarter to a half in the tokens their target accepts. What matters is not how good the draft is as a language model but how closely its next-token choices track one specific target, and that is what target-specific distillation and alignment buy.
Accepted Length Across Benchmarks
Table 12 reports the same measurement, made by us with the same harness, for all five targets. Every figure is an accepted length at . Each target’s answers are generated once and cached, then the draft’s proposals are verified against them token for token, and the tokens each pass yields are averaged over every pass, the target’s own token included. SPEED-Bench is its first hundred qualitative conversations, with responses of up to 4,096 tokens; the nine datasets are sampled as DSpark’s evaluation samples them, 3,030 prompts in all, with responses of up to 2,048 tokens. Each target answers at the reasoning effort in Table 8.
| Benchmark | gpt-oss-120b | Qwen3-Next | DeepSeek-V4-Pro-0813 | Kimi K3 | Muse-Spark-Open |
| SPEED-Bench, all | 4.04 | 3.68 | 3.22 | 2.92 | 2.97 |
| SPEED-Bench, coding | 4.36 | 4.86 | 4.04 | 4.18 | 3.92 |
| Mathematical reasoning | |||||
| GSM8K | 4.90 | 5.24 | 4.41 | 4.96 | 4.32 |
| MATH-500 | 4.78 | 5.12 | 4.23 | 3.55 | 4.12 |
| AIME 2025 | 4.31 | 4.57 | 3.76 | 2.92 | 3.75 |
| Coding | |||||
| HumanEval | 4.15 | 4.81 | 3.84 | 4.91 | 4.07 |
| MBPP | 4.49 | 4.58 | 4.01 | 4.49 | 4.16 |
| LiveCodeBench | 3.84 | 4.32 | 3.25 | 3.26 | 3.26 |
| General chat | |||||
| MT-Bench | 3.98 | 3.65 | 3.19 | 3.08 | 3.28 |
| AlpacaEval | 3.88 | 3.33 | 3.06 | 3.07 | 3.03 |
| Arena-Hard v2 | 3.68 | 3.62 | 3.08 | 2.90 | 2.89 |
Table 12. Accepted length per verification pass with a chain of drafted tokens, the target’s own token included, for Polyglot-DC1-Draft against each of its five targets, under verification of the target’s cached answers. The targets are those of Table 8, with Qwen3-Next-80B-A3B-Thinking abbreviated. SPEED-Bench’s rows are its qualitative split; the nine datasets below are those of Table 10.
Three patterns hold across the table. First, every draft is strongest on mathematics and code and weakest on open-ended conversation. On GSM8K the five drafts yield 4.3 to 5.2 tokens per pass, and on HumanEval and MBPP 3.8 to 4.9, whereas on AlpacaEval and Arena-Hard v2 they yield 2.9 to 3.9. Structured text, with its formulas, identifiers and boilerplate, is more predictable than free prose, and a chain of six proposals is correspondingly more often accepted in full. Second, the spread between benchmarks differs by target. Kimi K3 shows the widest range, from 4.9 tokens on HumanEval and GSM8K to 2.9 on AIME 2025 and Arena-Hard v2, while gpt-oss-120b is the most even, within 3.7 to 4.9 everywhere. Third, no single target leads throughout. Qwen3-Next is drafted best on most of the datasets, Kimi K3 edges it on GSM8K and HumanEval, and gpt-oss-120b leads on each of the three conversational benchmarks. These differences reflect the targets’ own regularity as much as the drafts, since a target whose next token is more often determined by the preceding ones is easier to draft for, whatever the draft.
Two conventions bear on how the table is read. The SPEED-Bench and nine-dataset answers for Muse Spark Open and for Qwen3-Next were generated on different serving stacks, whose outputs differ after a few tokens, so each target’s two blocks were scored against answers that are not identical in every token. And accepted length depends on the sampling settings at which a target answers, so figures from other settings are not directly comparable with these.
Table 13 turns those accepted lengths into decoding speed. As in Table 11, each draft runs at the chain length that gives it the highest end-to-end speedup for one user, with the next round pre-drafted while the GPU verifies the current one. That is the rule NVIDIA’s guidance gives for the latency-bound regime, to lengthen the chain only while the gain in accepted length justifies the added draft cost (Mitra et al., 2026). Each target alone decodes at the speed its own measurements give for one user: gpt-oss-120b at 526 tokens per second, measured without speculation on a GB200 system by SemiAnalysis’s InferenceX (SemiAnalysis, 2026); Qwen3-Next at 274, measured by us on one B300 with vLLM; DeepSeek-V4-Pro-0813 at 100, measured on a DGX B300 with vLLM by InferenceX, and Kimi K3 at 101, measured the same way by Open Zeka (Open Zeka, 2026); and Muse Spark Open at 150, derived from the two DGX B300 measurements.
| Benchmark | gpt-oss-120b | Qwen3-Next | DeepSeek-V4-Pro-0813 | Kimi K3 | Muse-Spark-Open |
| SPEED-Bench, all | 3.77x | 3.57x | 2.92x | 2.33x | 2.50x |
| SPEED-Bench, coding | 4.19x | 5.68x | 3.90x | 3.53x | 3.48x |
| Mathematical reasoning | |||||
| GSM8K | 5.06x | 6.57x | 4.50x | 4.57x | 4.02x |
| MATH-500 | 4.86x | 6.27x | 4.23x | 2.88x | 3.76x |
| AIME 2025 | 4.11x | 5.06x | 3.57x | 2.33x | 3.29x |
| Coding | |||||
| HumanEval | 3.88x | 5.55x | 3.63x | 4.51x | 3.68x |
| MBPP | 4.40x | 5.06x | 3.88x | 3.89x | 3.78x |
| LiveCodeBench | 3.49x | 4.59x | 2.95x | 2.61x | 2.76x |
| General chat | |||||
| MT-Bench | 3.68x | 3.53x | 2.88x | 2.45x | 2.79x |
| AlpacaEval | 3.57x | 3.10x | 2.74x | 2.45x | 2.55x |
| Arena-Hard v2 | 3.33x | 3.50x | 2.77x | 2.31x | 2.43x |
| All nine datasets | 3.82x | 4.37x | 3.13x | 2.65x | 2.92x |
Table 13. End-to-end speedup over each target alone for one user, simulated on the pre-silicon Exstill data-center design, with Polyglot-DC1-Draft at the chain length that gives the highest speedup and the next round pre-drafted during verification. For gpt-oss-120b and Qwen3-Next the GPU’s verification cost is taken from our B300 measurement; for the three targets no single B300 can hold it is modeled from their published expert counts and sizes. A request is a 2,000-token prompt and a 512-token answer. Accepted lengths are measured as in Table 12; the speedups are simulated. Target names are abbreviated as in Table 12.
The speedups follow the accepted lengths, with two further patterns. The first is that the best chain length tracks acceptance: where a draft is accepted deep into the chain, on mathematics and code, the optimum lies at 12 to 16 tokens, and where acceptance falls off early, on conversation, it lies at 6 to 11. The second is that the optimum is shorter for the trillion-parameter targets at any acceptance, because each of their verified tokens routes to more expert weights than a plain step loads, so beyond a point an extra proposal costs the GPU more than it is likely to earn. Kimi K3 is the clearest case, at a chain of 6 on every conversational benchmark and 15 on GSM8K and HumanEval, where it accepts most. For one user the gains run from 2.3x to 6.6x, with every target above 3.4x on coding and above 2.3x across SPEED-Bench as a whole.
These gains bring together the two quantities the section set out to separate. The accepted lengths belong to each draft and its target, the result of distilling one architecture for each of the five, and they carry through to decoding speed because the draft runs on its own silicon, where drafting adds little to the rounding it serves.
Guardrails in the Critical Path
A language model in production rarely runs alone. Beside it runs a second model whose only job is to decide what the first may read and what it may write. The reason is that safety training is a tendency, not a guarantee. Simple adaptive attacks, a random search over an appended suffix or a reply begun on the model’s behalf, reached 100% attack success against a dozen leading safety-aligned models, GPT-4o and every Claude model among them [6], and published defenses have not closed the gap. An adaptive attacker bypassed twelve recent ones, most of them at attack success above 90%, though most had originally reported attack success near zero (Nasr et al., 2025). Agents raise the stakes. An agent that reads web pages, email or documents on a user’s behalf can be steered by instructions planted in them, and agents are open to such prompt injection whenever they handle untrusted data [30]. The danger is greatest when the agent can call tools or touch sensitive information [16], because a manipulated answer becomes a manipulated action. OWASP’s 2026 list ranks prompt injection first among the risks of language-model applications, noting that the model itself draws no built-in line between instructions and data (OWASP GenAI Security Project, 2026). Even the newest frontier models are exposed. In Gray Swan’s indirect prompt-injection arena, OpenAI’s GPT-6 Astra, with its safeguards enabled, still fell to an estimated 8.5% of attacks across fifteen attempts per scenario, against 27.0% for its predecessor, GPT-5.6 Sol, according to its system card (OpenAI, 2026a).
What the main model’s training cannot guarantee is left to the guard, a separate model that reads each prompt before the main model does and each reply before the user does. Its job has three parts. It must catch harm, including the attacks the main model’s own training misses. It must do so in every language the product’s users write, whereas multilingual moderation has long concentrated on a small set of languages, English and Chinese chief among them (Kumar et al., 2025). And it must leave harmless requests alone. Over-refusal is now measured as carefully as harm, with OR-Bench’s 80,000 prompts that only sound harmful, tested across 32 models [27], and FalseReject’s 16,000, on which 29 current models still over-refuse (Zhang et al., 2025). A guard that flags too many harmless requests quickly becomes a liability.
The guard also faces a constraint unrelated to what it detects. It sits in the inference critical path, where every check adds compute and delays the response it screens, and much of how production screening is built follows from that cost.
How the Frontier Screens Traffic Today
Anthropic, Google DeepMind and OpenAI have each described how they screen production traffic, and the open guards follow the same pattern. Anthropic guards Claude with Constitutional Classifiers, classifiers trained on synthetic data generated from a written constitution of what is allowed and what is not, one screening the input and one the output (Sharma et al., 2025). The output classifier scores the reply token by token as it streams and halts generation the moment the score crosses a threshold; preserving time to first token and streaming was a design requirement. In Anthropic’s evaluation, jailbreak success on 10,000 synthetic jailbreaks fell from 86% to 4.4%, at the price of a 0.38% absolute rise in refusals on production Claude.ai traffic and a 23.7% inference overhead. Anthropic deployed the classifiers with Claude Opus 4 under its AI Safety Level 3 protections in May 2025 [7]. The next generation is built around cost. A linear probe on Claude’s own activations screens all traffic and escalates what it flags to an ensemble of a probe and a classifier that judges each reply together with its prompt. Anthropic estimates the overhead at roughly 1% of compute on Claude Opus 4.0 traffic, and over a month on Claude Sonnet 4.5 traffic the system refused 0.05% of harmless queries [9, 28].
Google DeepMind has deployed probes on Gemini’s activations to catch cyber-offensive misuse in user-facing Gemini. In its cascade, the probe defers to Gemini 2.5 Flash on about 8% of traffic and beats Flash alone at roughly a fiftieth of the cost; the hardest part was the move from short inputs to long ones (Kramár et al., 2026). OpenAI describes two tiers of real-time oversight against biological misuse on GPT-5’s thinking models, a fast topical classifier followed by a reasoning model that decides whether a response can be shown, running on all of that traffic and scanning user messages, tool calls and final output (OpenAI, 2025a). OpenAI’s open-weight gpt-oss-safeguard reasons from a policy given in its prompt, and its report is direct about the cost. Such reasoning is time- and compute-intensive, so OpenAI’s own Safety Reasoner relies on smaller, faster classifiers to decide what to assess and is sometimes run asynchronously to keep latency low (OpenAI, 2025c). With GPT-6 Astra the screened surface grew again. Alongside classifiers in production, OpenAI added misalignment monitoring to all tool-using inference in Astra’s external deployment and notes that it comes with significant compute cost (OpenAI, 2026a).
The open guards are built the same way. Meta’s Llama Guard 4, a 12B model, classifies prompts and responses (Meta, 2025a), and its Prompt Guard 2 catches injections and jailbreaks with 86M and 22M models that take 92.4 ms and 19.3 ms per check on an A100 at 512 tokens, according to its model card (Meta, 2025b). Qwen3Guard, from 0.6B to 8B parameters, labels content Safe, Controversial or Unsafe, and its stream variant classifies every token of a reply as it is generated (Zhao et al., 2025). NVIDIA’s NeMo Guardrails checks a streamed reply in chunks, 200 tokens by default, and its documentation states the trade-off plainly, that larger chunks mean fewer checks but a longer wait for the first output (NVIDIA, 2026c).
Despite their differences, these systems converge on one shape. Something cheap screens everything; something stronger judges what it flags; and both the prompt and the streaming reply are screened, increasingly along with every tool call an agent makes. Nearly every design choice serves the latency and compute budget, and each buys its saving with a loss elsewhere. A cascade’s strong classifier sees only the fraction of traffic the cheap screen flags. A larger chunk either delays the reply or lets more of it reach the user before it is checked. An asynchronous check rules after the user may already have read the text. Probes, the cheapest screen of all, are cheap because they reuse activations the model has already computed, and that brings limits of its own. A probe needs access to the served model’s internals and must be trained for each model it guards, so an application built on another provider’s API has nothing to read, and Google DeepMind reports that its probe techniques do not significantly reduce the success of adaptive attacks. We asked instead what screening would look like if it were fast enough that its cost no longer dictated when and how often it could run.
Why the Guard Belongs on Its Own Silicon
A guard sits in each user’s critical path twice, before the model sees the prompt and on every chunk of the reply before the user sees it. Its verdict is short, typically 10 to 15 tokens, so each check is a small burst of decoding for a single conversation. That is the regime in which a GPU is weakest. To write each token it pays the fixed charge of the layers and reads the whole model in from memory, as § 2 showed, and a short verdict leaves no time to spread either across a batch. Table 14 puts numbers on the gap. The B300 figures are measured, on one GPU serving an open model of each guard’s size with vLLM in FP8 on a screening-shaped load, and the Exstill figures are pre-silicon, for the design built for a model of that size: the data-center design of § 3.5 for the 1.7B and the DC2 design of § 3.8 for the 4B.
| Exstill design | NVIDIA B300 | Ratio | |
| A model of Polyglot-DC1-Guard’s size (1.7B) | |||
| One user, tokens per second | 19,921 | 522.9 | 38x |
| Twelve users at once, tokens per second each | 19,921 | 342.5 | 58x |
| Twelve users at once, tokens per second in total | 239,056 | 3,654.9 | 65x |
| Tokens per joule, twelve users | 1,912 | 6.2 | |
| Power, twelve users | 125 W | 585 W | |
| One 14-token verdict, one user | 0.70 ms | 26.8 ms | 38x |
| A model of Polyglot-DC2-Guard’s size (4B) | |||
| One user, tokens per second | 13,998 | 359.1 | 39x |
| Four users at once, tokens per second each | 13,998 | 320.7 | 44x |
| Four users at once, tokens per second in total | 55,992 | 1,162.8 | 48x |
| Tokens per joule, four users | 531 | 2.3 | |
| Power, four users | 106 W | 506 W | |
| One 14-token verdict, one user | 1.00 ms | 39.0 ms | 39x |
Table 14. Decoding a model of each guard’s size on the pre-silicon Exstill design built for it and on one NVIDIA B300. The B300 serves Qwen3-1.7B and Qwen3-4B-Thinking-2507 in FP8 with vLLM, measured on a screening-shaped load of 4K tokens in and 256 out; the data-center design serves twelve users for the 1.7B’s size and the DC2 design four users for the 4B’s, each with a 16K-token context. A verdict is 14 tokens. Decoding only.
For one user, the B300 decodes a model of the 1.7B guard’s size at 522.9 tokens per second, so a 14-token verdict takes 26.8 ms and four checks on one long answer take 107 ms. The data-center design decodes a model of the same size at 19,921 tokens per second for each of twelve conversations at once, 0.70 ms per verdict and 2.8 ms for the four. A model of the 4B guard’s size is slower on both, 39.0 ms per verdict on the B300 against 1.00 ms on the DC2 design, a ratio of 39x. Screening everything is cheap in energy as well as in time. At twelve conversations at once the data-center design draws 125 W and delivers 1,912 tokens per joule, against 6.2 for the B300 at 585 W, roughly 300x less energy per verdict. The DC2 design, at four conversations at once, draws 106 W and delivers 531 tokens per joule against 2.3, roughly 230x less.
At about a millisecond per check, a guard can screen every prompt and every chunk of every reply rather than the fraction a cheaper screen escalates, the chunks can be small, and no check has to rule after the user has read the text. A deployment may still send hard cases to a larger model, as Anthropic and Google DeepMind do, but its first line no longer has to be a compromise.
Insight 7: Screening is rationed by its cost, and the cost is where it runs Every production safeguard cascades, chunks or defers its checks because each check is a short single-user decode, the workload a GPU serves worst. On silicon built for that workload a verdict costs about a millisecond, and the rationing is no longer necessary.
Polyglot-DC1-Guard and Polyglot-DC2-Guard
The guard is the safety role of the Polyglot-DC architecture of § 4, in two sizes, Polyglot-DC1-Guard with 1.7B parameters and Polyglot-DC2-Guard with 4B. Each reads a conversation, classifies its last turn, the user’s message or the assistant’s reply, and answers in three short lines:
Safety: Safe | Controversial | Unsafe
Categories: None | one or more categories
Refusal: Yes | No (replies only)
The three-level safety label follows Qwen3Guard’s (Zhao et al., 2025). Controversial marks harm that depends on context, audience or jurisdiction, for a deployment to send to review or to resolve under its own policy, and it keeps borderline content from being forced into a binary verdict in either direction. The thirteen categories run from violence, weapons and illegal acts through personal data, self-harm, hate, political sensitivity, copyright and specialized advice to jailbreak attempts, the last for prompts only. The refusal line records whether the assistant declined, which separates a safe refusal from a safe answer and gives a deployment the signal for auditing an assistant that refuses too much or too little. The guard reads English, French, Spanish, Portuguese, German and Chinese with no translation step.
In deployment a single message, a prompt or a complete reply, is checked once, and a streamed reply is checked as it is written. A long answer from a reasoning model, its thinking included, streams to the user and is re-checked at intervals of a few hundred tokens; an answer allowed at every check reaches the user whole, one sent for review at its first check can be held, and one judged unsafe at a later check stops there, with nothing after it shown.
Detection Quality and False Alarms
Speed alone does not make a production guard, so we evaluated both sizes against four public guard models on 72,229 held-out items from 21 public benchmarks in six languages, 32,519 of them English and the rest French, Spanish, Portuguese, German and Chinese. The English sets are the fourteen on which Qwen3Guard reports, named with Table 15. The other five languages are covered by the test split of NVIDIA’s Nemotron Safety Guard dataset, the Aegis 2.0 test adapted into twelve languages with the CultureGuard pipeline (Joshi et al., 2025; NVIDIA, 2026d), by PolyGuardPrompts, which is WildGuardTest translated into seventeen languages (Kumar et al., 2025), by RTP-LX, a set of toxic prompts translated and adapted by native speakers in 28 languages [29], and, for Chinese, by MultiJail [33]. Portuguese, which the Nemotron set omits, rests on PolyGuardPrompts and RTP-LX alone. Every model runs on the same items, each in its own prompt format. Scores are F1, the harmonic mean of precision and recall, on the Unsafe class in best mode, meaning that for each benchmark the better of two readings is taken, one counting Controversial as unsafe and one counting it as safe, as Qwen3Guard reports its own results. The baselines are Qwen3Guard-Gen-0.6B and Qwen3Guard-Gen-4B (Zhao et al., 2025), Meta’s Llama Guard 4 (Meta, 2025a), a 12B model which answers safe or unsafe only and has no refusal output, and OpenAI’s gpt-oss-safeguard-20b (OpenAI, 2025b,c), a 21B-parameter mixture-of-experts model with 3.6B active, given the same policy in its prompt and run at medium reasoning effort.
| Benchmark | Polyglot-DC1-Guard (1.7B) | Polyglot-DC2-Guard (4B) | Qwen3Guard-Gen-0.6B | Qwen3Guard-Gen-4B | Llama Guard 4 (12B) | gpt-oss-safeguard-20b |
| Prompt classification | ||||||
| ToxicChat | 79.5 | 82.0 | 77.9 | 83.1 | 50.9 | 74.0 |
| OpenAI Moderation | 79.9 | 80.5 | 77.5 | 80.8 | 73.9 | 78.5 |
| Aegis 1.0 | 85.3 | 87.6 | 90.9 | 91.1 | 67.8 | 84.4 |
| Aegis 2.0 | 86.3 | 86.5 | 85.0 | 85.8 | 70.8 | 82.4 |
| SimpleSafetyTests | 99.0 | 99.0 | 99.0 | 99.5 | 98.5 | 99.5 |
| HarmBench | 98.1 | 98.3 | 98.3 | 100.0 | 97.6 | 100.0 |
| WildGuardTest | 87.9 | 88.2 | 88.5 | 89.2 | 71.1 | 85.4 |
| Response classification | ||||||
| HarmBench | 86.5 | 87.7 | 84.7 | 86.7 | 82.4 | 86.4 |
| SafeRLHF | 65.1 | 67.7 | 66.7 | 69.6 | 43.7 | 64.9 |
| BeaverTails | 85.2 | 85.9 | 86.3 | 86.5 | 69.8 | 83.2 |
| XSTest | 94.7 | 94.9 | 90.6 | 93.0 | 89.0 | 90.6 |
| Aegis 2.0 | 82.3 | 82.7 | 84.4 | 86.5 | 64.7 | 79.3 |
| WildGuardTest | 79.0 | 79.9 | 76.9 | 79.9 | 66.4 | 75.4 |
| Qwen3GuardTest, thinking | 85.4 | 86.1 | 86.1 | 84.0 | 60.6 | 85.7 |
Table 15. F1 on the Unsafe class, in best mode, on the English benchmarks, measured by us on the same held-out items for every model. Prompt classification judges the user’s message; response classification judges the assistant’s reply given its prompt. Bold marks the best value in each row.
Table 16 gives the four aggregates and the two false-alarm measures. Polyglot-DC2-Guard comes within one F1 point of Qwen3Guard-Gen-4B, the strongest model in the comparison, on all four aggregates, trailing it by 1.0 on English prompts, 0.2 on English responses, 0.6 on prompts in the other five languages and 1.0 on their responses, and it is ahead of Qwen3Guard-Gen-0.6B, Llama Guard 4 and gpt-oss-safeguard-20b on every one of them. Polyglot-DC1-Guard, with under half of Qwen3Guard-Gen-4B’s parameters, is level with Qwen3Guard-Gen-0.6B in English, 88.0 and 82.6 against 88.1 and 82.2, and ahead of it in the other five languages. Llama Guard 4, at 12B parameters, trails the 4B guard by 13 to 15 F1 points in English and by more outside it.
| Polyglot-DC1-Guard (1.7B) | Polyglot-DC2-Guard (4B) | Qwen3Guard-Gen-0.6B | Qwen3Guard-Gen-4B | Llama Guard 4 (12B) | gpt-oss-safeguard-20b | |
| F1 on unsafe content, higher is better | ||||||
| English prompts | 88.0 | 88.9 | 88.1 | 89.9 | 75.8 | 86.3 |
| English responses | 82.6 | 83.5 | 82.2 | 83.7 | 68.1 | 80.8 |
| Other five languages, prompts | 86.5 | 87.7 | 85.5 | 88.3 | 56.3 | 83.4 |
| Other five languages, responses | 80.4 | 81.5 | 79.6 | 82.5 | 60.1 | 79.0 |
| Harmless prompts flagged, lower is better | ||||||
| PHTest | 20.0% | 9.8% | 29.2% | 23.8% | 6.8% | 17.8% |
| OR-Bench hard-1k | 14.9% | 7.2% | 72.7% | 82.2% | 10.6% | 90.5% |
Table 16. Detection quality and false alarms of the Polyglot-DC guards against four public guards, measured by us on the same held-out items. F1 is on the Unsafe class in best mode, averaged over the benchmarks of each group; the English averages are over the prompt and response benchmarks that Qwen3Guard reports. The public guards are Qwen3Guard-Gen (Zhao et al., 2025), Llama Guard 4 (Meta, 2025a) and gpt-oss-safeguard-20b (OpenAI, 2025c). The last two rows are the share of harmless prompts flagged as unsafe, lower being better.
The more striking difference is how often the models flag harmless content. PHTest is a set of harmless prompts that sound harmful [5]. Polyglot-DC2-Guard flags 9.8% of them, against 17.8% to 29.2% for gpt-oss-safeguard and the two Qwen3Guard models, under half the rate of Qwen3Guard-Gen-4B, the guard it matches within a point on harm. Only Llama Guard 4 flags fewer, at 6.8%, and its English F1 is 13 to 15 points lower. On OR-Bench hard-1k [27], the thousand hardest of OR-Bench’s seemingly toxic prompts, the gap widens further, with the Polyglot-DC guards flagging 7.2% and 14.9% where Qwen3Guard flags 72.7% to 82.2% and gpt-oss-safeguard 90.5%.
Table 15 breaks the English aggregates into their fourteen benchmarks: for prompts, ToxicChat (Lin et al., 2023), the OpenAI Moderation set (Markov et al., 2023), Aegis 1.0 and 2.0 [40, 41], SimpleSafetyTests (Vidgen et al., 2023), HarmBench (Mazeika et al., 2024) and WildGuardTest [46]; for responses, HarmBench, SafeRLHF [55], BeaverTails [54], XSTest (Röttger et al., 2024), Aegis 2.0, WildGuardTest and the thinking split of Qwen3GuardTest (Zhao et al., 2025). The pattern is consistent rather than dominated by any one set. Qwen3Guard-Gen-4B leads on six of the seven prompt sets, mostly by about a point. Polyglot-DC2-Guard leads on Aegis 2.0 prompts and on the HarmBench and XSTest responses, XSTest being the set on which exaggerated safety is the failure under test. It ties the leader on Qwen3GuardTest responses, and the two 4B guards are tied on WildGuardTest responses. The places the Polyglot-DC guards trail clearly are Aegis 1.0 prompts and Aegis 2.0 responses, by about four points. Llama Guard 4 trails everywhere, most of all on the conversational sets, and gpt-oss-safeguard-20b, despite reasoning over an explicit policy at every check, sits between it and the smaller Qwen3Guard.
Table 17 shows the same comparison in the other five languages. The Polyglot-DC guards hold their English standing in each of them. Polyglot-DC2-Guard leads on French, Spanish and Chinese prompts and is within a point or two of Qwen3Guard-Gen-4B everywhere else, and the ordering of the public guards is unchanged, with Llama Guard 4 falling furthest, by roughly 30 points on prompts, once the language is not English. Portuguese responses, which come from PolyGuardPrompts alone, are the weakest set for five of the six models, and the only language in which Polyglot-DC1-Guard drops below 80 F1. The refusal flag, which only the Polyglot-DC guards, Qwen3Guard and gpt-oss-safeguard produce, is detected at 89 to 95 F1 on WildGuardTest, XSTest and PolyGuardPrompts (Kumar et al., 2025) by both sizes, level with Qwen3Guard, and within one to three points of gpt-oss-safeguard, which leads on two of the three sets and trails on XSTest.
| Language | Task | Polyglot-DC1-Guard (1.7B) | Polyglot-DC2-Guard (4B) | Qwen3Guard-Gen-0.6B | Qwen3Guard-Gen-4B | Llama Guard 4 (12B) | gpt-oss-safeguard-20b |
| French | prompts | 86.4 | 87.9 | 85.2 | 87.7 | 56.6 | 82.4 |
| French | responses | 81.3 | 82.4 | 80.5 | 83.6 | 60.1 | 80.3 |
| Spanish | prompts | 85.8 | 87.1 | 85.1 | 86.8 | 56.2 | 83.2 |
| Spanish | responses | 81.6 | 82.4 | 81.8 | 83.8 | 60.7 | 79.6 |
| Portuguese | prompts | 86.7 | 87.5 | 84.7 | 89.3 | 50.2 | 82.0 |
| Portuguese | responses | 77.0 | 78.2 | 75.3 | 78.6 | 59.9 | 77.7 |
| German | prompts | 85.2 | 86.0 | 84.3 | 87.6 | 57.5 | 82.5 |
| German | responses | 80.4 | 82.2 | 79.9 | 83.3 | 61.4 | 79.0 |
| Chinese | prompts | 88.6 | 90.0 | 87.9 | 89.9 | 61.0 | 86.7 |
| Chinese | responses | 81.6 | 82.5 | 80.5 | 83.1 | 58.7 | 78.6 |
Table 17. F1 on the Unsafe class in the five languages other than English, in best mode, averaged over the benchmarks in each language, measured by us on the same held-out items for every model. The benchmarks are the Nemotron Safety Guard test split ((Joshi et al., 2025)), PolyGuardPrompts ((Kumar et al., 2025)) and RTP-LX ([29]) in every language, and MultiJail ([33]) in Chinese; Portuguese has PolyGuardPrompts and RTP-LX only. Bold marks the best value in each row.
Choosing the Size
Of the two sizes, Polyglot-DC2-Guard is the more accurate. It gains about a point of F1 on every aggregate over Polyglot-DC1-Guard, and it flags under half as many of PHTest’s harmless prompts, 9.8% against 20.0%, which for a deployment is the difference between a guard that is occasionally over-cautious and one that gets in the way. On a GPU, choosing the more accurate size also raises the latency of every check. One B300 takes 39.0 ms for a 14-token verdict from a model of the 4B’s size, against 26.8 ms from one of the 1.7B’s, an extra 12.2 ms on each of the several checks a long answer receives. On the Exstill designs the same verdicts decode in 1.00 ms and 0.70 ms, so the more accurate guard costs 0.30 ms more per check, and a deployment can choose its guard for accuracy rather than for speed.
Insight 8: Choose the guard for accuracy, not for latency
On a GPU, the more accurate of two guards adds over ten milliseconds to every check, so the smaller one is chosen. On silicon where either verdict takes about a millisecond or less, the trade disappears and the more accurate guard is the obvious choice.
The frontier labs have shown what a production guard must do, which is to screen the prompt and the streaming reply, catch the attacks the model’s own training misses, and leave harmless requests alone. They have also shown that every safeguard is shaped by its cost. On Exstill silicon a deployment can screen every prompt, every chunk of every reply and every step of an agent’s work without the screening becoming the inference bottleneck.
Agents and the Cost of a Turn
An agent is a language model in a loop. It reads a policy and a conversation, reasons about what to do, calls a tool, reads the result and decides again, and a single customer’s request can take it through a dozen or more such turns before anything is resolved. Every turn is decoding, the model writing its reasoning and its action one token at a time, and every turn must finish before the next can begin. The latency of an agent is therefore the sum of its turns. § 4 made the case that an agent’s work is a narrow slice of what a language model can do, with strict output formats and a consistent shape, which is what makes it a job for a small model trained for it.
The Loop and Its Latency
A turn of a tool-using agent has three parts. The model reads its context, which holds the policy it must follow, the conversation so far, the tools it may call and the results of the calls it has made. It reasons about what the customer wants, what the policy allows and which tool would settle the question, and a reasoning model does so at some length. Then it acts, either by calling a tool with arguments or by writing a reply to the customer. Only the third part is visible to anyone outside the loop, and in a reasoning model it is the smallest. In the eight episodes examined in § 7.4, reasoning accounts for 86% of the tokens the agent wrote; the tool calls and replies that the customer and the tools actually receive are the remaining 14%.
This shape is what makes an agent’s latency compound. A chat assistant writes one answer and the person reads it as it streams, so the first token matters most and the rest arrive while they read. An agent’s turn is read by nobody until it is complete, because the tool cannot be called until the arguments are written and the reply cannot be sent until the reasoning is done. On a GPU serving one conversation, a token of a small model takes about two milliseconds (Table 1), so a turn of a thousand tokens is two to three seconds and a case of fifteen turns is half a minute of waiting, before the tools’ own latency and the customer’s replies are added. Batching does not help the customer. A GPU serving sixteen agents at once writes more tokens per second in total, but each agent’s turn takes longer, not shorter, so the operator’s throughput improves while the customer’s half-minute case grows.
Polyglot-DC2-Agent
The agent role of the Polyglot-DC architecture of § 4 is Polyglot-DC2-Agent, built on the 4B size. It is a reasoning model for multi-turn tool use under a written policy. Given the policy, the conversation and a set of tools, it reasons about the customer’s request, then either calls a tool, with structured arguments in the architecture’s native tool-call format, or writes a reply, and it continues until the request is resolved, refused under the policy or handed to a human. Its conversations grow with every turn, since each tool result and each exchange is appended to its context, and in the cases examined below the context reaches 68,000 tokens where a knowledge-base search returns long documents. It is a standalone model, and a deployment runs it as it would any agent model, behind the same tool schemas the frontier models use.
Results Against Open Models of Its Size
We measured Polyglot-DC2-Agent on five public benchmarks against eight open models from 1.2B to 4.5B parameters: Qwen3.5-4B and Qwen3.5-2B (Qwen Team, 2026a,b), Qwen3-4B-Thinking-2507 and Qwen3-1.7B (Qwen Team, 2025; Yang et al., 2025), Gemma 4 E4B and E2B [39], and LFM2.5-2.6B and LFM2.5-1.2B-Thinking (Liquid AI, 2026a,c). No vendor’s published number is used. Every model was served the same way, with vLLM at its full context length and 32,768 tokens per reply, its thinking mode on, the sampling settings its model card recommends, and one sample per item, so that the comparison measures the models and not the setups. Polyglot-DC1-Agent, the 1.7B size, is in the tables for scale.
The first two benchmarks are versions 3 and 4 of the Berkeley Function Calling Leaderboard (BFCL) (Patil et al., 2025), the latter with the memory group (Mao et al., 2025), each scored as the leaderboard’s overall accuracy. The third is LiveCodeBench [53], scored as pass@1, the share of problems solved at the first attempt, on the 182 problems published between January and April 2025. The last two are -bench [13] and -bench (Sierra Research, 2026), the customer-service benchmarks descended from -bench (Yao et al., 2024), in which the agent follows a domain policy and uses tools on behalf of a simulated customer. They are scored as pass, the share of trials in which the task is completed, over 20 tasks in each domain and four trials of each task, with GPT-4.1 as the customer. -bench adds a banking domain in which the agent must first retrieve the relevant documents from a knowledge base of 698 documents.
Table 18 and Table 19 give the results. The two customer-service benchmarks are the closest of the five to an agent’s work in deployment. On -bench Polyglot-DC2-Agent is the strongest model in the table at 66.9, six points ahead of LFM2.5-2.6B and 25 ahead of every other model, with the three service domains at 72.5, 86.2 and 97.5. On -bench it scores 79.6, within four points of Qwen3.5-4B’s 83.2 and ahead of the rest. The gap between the two benchmarks is instructive. -bench’s revised airline and retail tasks are harder, and most of the models that do well on -bench lose half their score on them, while Polyglot-DC2-Agent keeps its airline and retail scores and leads both domains. Banking, the new retrieval domain, is weak for every model at this size, with no score above 18.8.
Table 18 | Polyglot-DC2-Agent against eight open models of its size, measured by us under one protocol, with Polyglot-DC1-Agent for scale. BFCL v3 and v4 are the leaderboard’s overall accuracy; the v4 web-search group was not run, so its agentic term is the memory group. LiveCodeBench is pass on the 182 problems of January to April 2025. -bench and -bench are pass, the mean over their domains. Bold marks the best value in each column.
| Model | BFCL v3 | BFCL v4 | LiveCodeBench | -bench | -bench |
| Polyglot-DC2-Agent (4B) | 63.2 | 65.9 | 44.0 | 79.6 | 66.9 |
| Polyglot-DC1-Agent (1.7B) | 50.2 | 48.2 | 26.9 | 71.2 | 57.5 |
| Qwen3.5-4B | 64.2 | 56.6 | 50.0 | 83.2 | 39.7 |
| Qwen3-4B-Thinking-2507 | 66.5 | 48.0 | 53.8 | 47.9 | 42.2 |
| Gemma 4 E4B | 62.7 | 44.4 | 51.6 | 42.9 | 38.8 |
| LFM2.5-2.6B | 61.6 | 56.3 | 40.1 | 72.9 | 60.9 |
| Qwen3.5-2B | 53.6 | 35.9 | 18.1 | 78.0 | 40.9 |
| Gemma 4 E2B | 58.6 | 32.9 | 44.5 | 28.3 | 21.9 |
| Qwen3-1.7B | 55.6 | 31.3 | 30.2 | 16.7 | 15.3 |
| LFM2.5-1.2B-Thinking | 52.1 | 29.8 | 20.9 | 24.3 | 19.4 |
Table 18.
Table 19 | -bench and -bench by domain, pass over 20 tasks and four trials in each, measured by us with GPT-4.1 as the simulated customer. The mean is over the benchmark’s domains. Bold marks the best value in each column.
| Model | Airline | Retail | Telecom | Mean | Airline | Retail | Telecom | Banking | Mean |
| Polyglot-DC2-Agent (4B) | 68.8 | 75.0 | 95.0 | 79.6 | 72.5 | 86.2 | 97.5 | 11.2 | 66.9 |
| Polyglot-DC1-Agent (1.7B) | 56.2 | 57.5 | 100.0 | 71.2 | 61.3 | 63.7 | 91.2 | 13.8 | 57.5 |
| Qwen3.5-4B | 73.2 | 76.5 | 100.0 | 83.2 | 31.2 | 30.0 | 85.0 | 12.5 | 39.7 |
| Qwen3-4B-Thinking-2507 | 51.2 | 71.2 | 21.2 | 47.9 | 65.0 | 67.5 | 18.8 | 17.5 | 42.2 |
| Gemma 4 E4B | 48.8 | 72.5 | 7.5 | 42.9 | 58.8 | 73.8 | 6.2 | 16.2 | 38.8 |
| LFM2.5-2.6B | 63.7 | 73.8 | 81.2 | 72.9 | 62.5 | 77.5 | 85.0 | 18.8 | 60.9 |
| Qwen3.5-2B | 64.1 | 78.8 | 91.2 | 78.0 | 25.0 | 57.5 | 76.2 | 5.0 | 40.9 |
| Gemma 4 E2B | 37.5 | 36.2 | 11.2 | 28.3 | 36.2 | 37.5 | 6.2 | 7.5 | 21.9 |
| Qwen3-1.7B | 16.2 | 8.8 | 25.0 | 16.7 | 15.0 | 12.5 | 27.5 | 6.2 | 15.3 |
| LFM2.5-1.2B-Thinking | 30.4 | 15.0 | 27.6 | 24.3 | 32.5 | 11.2 | 26.2 | 7.5 | 19.4 |
Table 19.
On BFCL v4, whose overall score weights multi-turn tool use and the memory group most heavily, the agent leads at 65.9, nine points ahead of Qwen3.5-4B and LFM2.5-2.6B. On BFCL v3, which weights single-turn function calling more, it scores 63.2, three points behind Qwen3-4B-Thinking-2507 and one behind Qwen3.5-4B, and on LiveCodeBench, a coding benchmark outside its role, 44.0, in the middle of the group. Single-turn function calling is the clearest room to improve, and retrieval over a large knowledge base the next piece of work.
Eight Episodes, Turn by Turn
Benchmarks score outcomes. To see what an agent’s work costs in time, we took eight -bench episodes that Polyglot-DC2-Agent completed, two from each domain, chosen for how much they reason, and replayed every agent turn at two decode speeds, the Exstill data-center design’s and one NVIDIA B300’s. Each is a full case. In the banking episodes the agent searches the bank’s knowledge base card by card to find an everyday card for a cryptocurrency trader, eleven turns in which the search results take its context to 68,000 tokens, and chases missing referral bonuses. In the telecom episodes it walks a customer abroad through data-saver mode, a VPN and a network reset before finding roaming disabled on the line and enabling it, eighteen turns, and resolves a line that has run out of data. The airline episodes rebook to the cheapest Economy fare a day later and price three changes to one booking before making any; the retail episodes handle a return against two possible exchanges and exchange two items in one request. Every dollar figure the agent states was checked against its tool results and the policy; a passing grade was not treated as enough.
Table 20 gives the eight episodes. Together they are 108 agent turns and 123,677 tokens written, of which 106,203 are reasoning. The B300 time is the one-user decode of Table 1, 2.15 ms per token; the chip time is the data-center design’s 19,921 tokens per second. Both rates are those of a model of the 1.7B size on the same load of 15K tokens in and 1K out, not of the 4B that wrote the episodes, and both count decoding only, with no prefill, tool latency or customer time.
| Domain | Case | Agent turns | Tokens written | Reasoning share | Longest prompt | B300, one user | Exstill design |
| Banking | A card for a crypto trader | 11 | 21,081 | 89% | 67.9K | 45.3 s | 1.06 s |
| Banking | Missing referral bonuses | 10 | 17,810 | 89% | 19.8K | 38.3 s | 0.89 s |
| Telecom | Slow data abroad | 18 | 26,550 | 91% | 11.7K | 57.1 s | 1.33 s |
| Telecom | Out of data abroad | 18 | 12,296 | 74% | 11.6K | 26.4 s | 0.62 s |
| Airline | Cheapest Economy, a day later | 13 | 17,674 | 85% | 13.3K | 38.0 s | 0.89 s |
| Airline | Bags, a name and an upgrade | 14 | 14,889 | 88% | 9.7K | 32.0 s | 0.75 s |
| Retail | Return or exchange | 11 | 8,753 | 79% | 9.6K | 18.8 s | 0.44 s |
| Retail | Two exchanges at once | 13 | 4,624 | 73% | 10.9K | 9.9 s | 0.23 s |
| All eight | 108 | 123,677 | 86% | 67.9K | 266 s | 6.2 s | |
| One turn, on average | 1,145 | 86% | 2.46 s | 57 ms |
Table 20. Eight -bench episodes that Polyglot-DC2-Agent completed, replayed turn by turn. Tokens are every token the agent wrote across its turns, reasoning included; the reasoning share is the fraction of those that were reasoning. The longest prompt is the largest context the agent read in any one turn. Times are decoding only, at the one-user B300 rate of Table 1 and the data-center design’s rate, both for a model of the 1.7B size.
An average turn is 1,145 tokens, which the B300 decodes in 2.46 s and the chip in 57 ms. The longest case, the telecom customer with slow data abroad, is 57 s of decoding on the GPU and 1.3 s on the chip; the shortest, the two retail exchanges, is 10 s against 0.23 s. Across all eight cases the GPU spends 266 s writing the agent’s tokens and the chip 6.2 s, a ratio of 43x that holds turn for turn, because it is the ratio of two decode rates and the agent’s turns are nothing but decoding. The practical difference is in what the customer notices. A turn of 57 ms is below the time a person takes to read the agent’s previous reply, so the agent’s thinking disappears into the conversation, and a case of eleven to eighteen turns is a second or so of machine time in a conversation that is otherwise paced by the customer.
Insight 9: An agent’s latency is the sum of its turns, and each turn is a decode
Nothing an agent does between tool calls is visible until the turn is complete, so its latency is the whole turn, not the first token. On a GPU a reasoning turn costs seconds and a case costs tens of seconds, which is why deployments shorten the reasoning or choose a model that reasons less. On silicon that decodes 43x faster, the same turns cost tens of milliseconds, and the accuracy that reasoning buys comes without the wait.
Polyglot-DC2-Agent reasons at length, and that is what puts it first among the models of its size on -bench and near the front on -bench. On the Exstill design that reasoning is paid for at 57 ms per turn, and a conversation’s pace is set by the customer again.
Reasoning in Milliseconds
A language model that reasons before it answers writes its working first, hundreds or thousands of tokens of it, and the answer only after. That habit is what has made small models capable of hard problems, and it is paid for in time, since every token of working is decoded in sequence before the first token of the answer appears. In most uses the wait is an inconvenience. In finance it decides what reasoning can be used for, because the question is asked while the market is moving and the answer is worth most in the moments after a filing, a headline or a quote lands.
Thinking Is Sold by the Effort
Wei et al. (2022) showed that asking a model to write out its intermediate steps improves its answers on arithmetic, commonsense and symbolic reasoning, and by the end of 2024 OpenAI’s o1 models had been trained by reinforcement learning to think before they answer, producing a long chain of thought in which they try strategies and recognize their own mistakes (OpenAI, 2024). Compute spent at answer time became a second axis of scale. Spent well, it lets a small model outperform one fourteen times its size on problems it can partly solve (Snell et al., 2025). Every frontier model now reasons. GPT-5 routes each request between a fast model and a deeper reasoning model according to its complexity and the tools it needs (OpenAI, 2025a), Gemini 2.5 is a hybrid reasoning model whose scores on competition mathematics, live coding and graduate-level science rise with its thinking budget [38], and the strongest open models think by default [31] (Moonshot AI 2026; Z.ai 2026b).
Small models have followed. Gemma 4 has a thinking mode from 2.3B parameters up [39], and Qwen3.5-4B, Nanbeige4.2-3B and LFM3-1.7B all reason unless told not to (Liquid AI, 2026d; Qwen Team, 2026a; Yang et al., 2026a). Where a model reports both modes the gap is wide: Qwen3’s 4B scores 55.9 on graduate-level science with thinking on and 41.7 with it off (Yang et al., 2025). Reasoning does not transfer freely across domains, however. Models trained to reason on general tasks lose ground on financial problems (Qian et al., 2025), which is why finance now has reasoning models of its own (Liu et al., 2025).
Thinking costs latency and compute, so every provider sells it as a setting. OpenAI’s reasoning models take an effort from none to max, with lower effort favoring speed and fewer tokens, and reasoning tokens billed as output (OpenAI, 2026b). Claude takes an effort level from low to max that trades thoroughness against token efficiency [9], and at medium effort Claude Opus 4.5 matched Sonnet 4.5’s best score on SWE-bench Verified with 76% fewer output tokens [8]. Gemini takes a thinking level whose lowest setting minimizes latency and cost [43]. The open models expose the same dial and often default to its top. Kimi K3 and GLM-5.3 offer low, high and max and default to max, and Muse Spark Open offers low to xhigh and defaults to xhigh (Meta, 2026; Moonshot AI, 2026; Z.ai, 2026a). The published curves show what each step buys and costs. Raising DeepSeek-V4.1-Flash’s effort from 25 to 100 lifts its average over eight reasoning benchmarks from 67.1% to 76.3% for roughly 2.5x the output tokens, and on AIME 2026 an answer grows from 4.6K to 11.4K tokens [32]. On Z.ai’s coding benchmark GLM-5.3 scores 31.4% at high effort with about 50K output tokens a task and 34.5% at max with about 75K (Z.ai, 2026b). Every setting is the same bargain, a better answer paid for in tokens, and every token is time the user waits.
Why Reasoning Is Slow on a GPU
Reasoning is decoding, one token at a time, and § 2 showed that on a GPU serving one user each token means reading the whole model in from memory. A long chain of thought makes that traffic heavier, not lighter. Measured by the memory traffic it causes rather than by its arithmetic, the cost of test-time reasoning is mostly memory access, and as the chain grows it is the attention over the chain, rather than the parameter count, that comes to dominate (Sadhukhan et al., 2025). The rates measured for the guard in Table 14 apply unchanged here, since a reasoning model of a given size decodes at the same speed whatever it is reasoning about. One B300 decodes a model of Polyglot-DC1-Finance’s size at 522.9 tokens per second for one user, so a 2,000-token chain of thought takes about 3.8 s before the answer begins, and a model of Polyglot-DC2-Finance’s size at 359.1 tokens per second, so the same chain takes 5.6 s. The Exsil data-center design decodes the first at 19,921 tokens per second for each of twelve users, and the DC2 design of § 3.8 decodes the second at 13,998 tokens per second for each of four, so the same 2,000 tokens take about 0.10 s and 0.14 s. The ratio is 38x for the smaller model and 39x for the larger.
Finance Runs on the Clock
The fastest trading involves no language model at all. Trading firms place their servers in the exchange’s own data center and their logic on field-programmable gate arrays (FPGAs), and the record from the last bit of market data a decision needs to the first bit of the order stands at 13.9 nanoseconds, on an AMD FPGA card [4] (STAC 2024).
Language models are not candidates for that loop. Their place is the work around it, which is where firms are putting them. Summarization and information extraction are the leading generative-AI uses among the broker-dealers FINRA oversees [36]. JPMorganChase has put its LLM Suite in front of more than 200,000 employees in a controlled environment that protects customer and company data (JPMorganChase, 2025), and Goldman Sachs runs its assistant firm-wide behind its own firewall, drawing on OpenAI’s and Google’s models and on Meta’s open Llama [25, 37].
Two results tie the value of that work to its speed. A language model’s reading of a headline predicts the stock’s initial reaction on about 90% of portfolio-days, the ability grows with model size, and the returns from trading on it shrink as more traders use such models (Lopez-Lira and Tang, 2023). In a simulated high-frequency trading benchmark a 14B model took 0.71 to 1.30 s a decision on a GPU, and its trading results improved as its latency fell, even at some cost in quality (Kang et al., 2025a). A reasoned answer that takes seconds can follow a desk’s questions. It cannot screen every headline as it lands or chain a dozen calls in an agent before the moment has passed. An answer decoded in tens of milliseconds can, though a complete system must also budget for reading its input and ingesting the data. Finance also keeps its data close, so a small model on hardware the firm controls is one a trading floor can deploy.
Polyglot-DC1-Finance and Polyglot-DC2-Finance
The finance role of the Polyglot-DC architecture of § 4 comes in both sizes, Polyglot-DC1-Finance with 1.7B parameters and Polyglot-DC2-Finance with 4B. Each is a reasoning model that writes its working before its answer, and each covers four kinds of analyst work. It classifies the sentiment of financial news, headlines and posts, extracts the people, places and companies named in financial text, reasons numerically over the tables and text of filings and earnings reports across the turns of a conversation, and answers quantitative questions on calculations, valuation and analyst judgment, with a document as context where one is given. The reasoning is part of the product rather than a cost of it, since the written steps are what let a desk check an answer before acting on it.
Ten Benchmarks Against Seven Open Models
Speed is worth having only if the answers hold, so both sizes were evaluated on ten public financial benchmarks against seven open models from 1.2B to 8B parameters, every model run in the same harness with reasoning on, one sample an item and the same sampling settings. The benchmarks cover the four kinds of work: for sentiment, FinancialPhrase-Bank (Malo et al., 2014) and the Gold Headlines set (Sinha and Khandait, 2021); for extraction, FiNER-ORD (Shah et al., 2023), scored by entity F1; for numerical reasoning over filings, FinQA [22], ConvFinQA [23] and TAT-QA (Zhu et al., 2021); and for quantitative finance, FinanceReasoning (Tang et al., 2025), FinanceMath (Zhao et al., 2024), FinanceBench with its oracle evidence pages [52] and
FinanceQA (Mateega et al., 2025). FinanceBench and FinanceQA are graded by a language-model judge, DeepSeek-V4.1-Flash, and an item whose verdict could not be parsed counts as wrong, so those two scores are lower bounds for every model. The baselines are LFM2.5-1.2B-Thinking, LFM2.5-2.6B and LFM2.5-8B-A1B (Liquid AI, 2026a,b,c), Qwen3-1.7B (Yang et al., 2025), MiniCPM5-2B (OpenBMB, 2026), Nanbeige4.1-3B (Yang et al., 2026b) and Gemma-4-E4B-it [44], the last two from the small reasoning families of § 8.1.
Table 21 gives the results. Polyglot-DC2-Finance averages 72.2 over the ten benchmarks, 4.0 above Gemma-4-E4B-it at 68.2, the strongest of the open models, and 6.1 above Nanbeige4.1-3B. It leads all seven on nine of the ten, and on the tenth, FinanceMath, it scores 57.7 against Gemma-4-E4B-it’s 58.1. Its largest margins are where reasoning does the most work. On FinanceReasoning it scores 81.5 against 78.3 for the best open model, on FinanceBench 80.7 against 76.7, and on FiNER-ORD, the extraction task on which most of the small models struggle, 70.7 against 62.1. Polyglot-DC1-Finance averages 63.6, 1.6 above MiniCPM5-2B and 8.3 above Qwen3-1.7B, the open model nearest its size. It leads every open model on FinQA, TAT-QA and both sentiment sets, with 82.7 on TAT-QA level with the 4B, and it trails the larger open models on the hardest calculations, 34.9 on FinanceMath against Gemma-4-E4B-it’s 58.1 and 65.5 on FinanceReasoning against Nanbeige4.1-3B’s 78.3. The pattern is the one § 4 predicted. A small model distilled for one domain reaches the models two to four times its size on the parts of the domain that are read and structured, and the extra parameters of the 4B are what the open-ended quantitative questions need.
| Benchmark | Polyglot-DC1-Fin. (1.7B) | Polyglot-DC2-Fin. (4B) | LFM2.5-1.2B-Thinking | Qwen3-1.7B | MiniCPM5-2B | LFM2.5-2.6B | Nanbeige 4.1-3B | Gemma-4-E4B-it | LFM2.5-8B-A1B |
| Sentiment and classification | |||||||||
| FinancialPhraseBank | 77.4 | 78.1 | 69.6 | 71.9 | 76.0 | 76.8 | 76.0 | 73.3 | 73.5 |
| Gold Headlines | 79.6 | 80.2 | 71.0 | 76.2 | 70.1 | 76.5 | 79.4 | 77.8 | 78.9 |
| Extraction | |||||||||
| FiNER-ORD | 53.7 | 70.7 | 5.4 | 20.0 | 24.8 | 43.2 | 56.4 | 62.1 | 23.6 |
| Numerical reasoning over filings | |||||||||
| FinQA | 71.0 | 72.7 | 32.6 | 62.6 | 69.1 | 66.0 | 64.4 | 69.3 | 48.1 |
| ConvFinQA | 80.5 | 82.8 | 30.7 | 65.6 | 78.5 | 71.3 | 74.9 | 81.0 | 55.8 |
| TAT-QA | 82.7 | 82.7 | 42.1 | 67.6 | 77.2 | 74.2 | 74.2 | 80.0 | 58.9 |
| Quantitative finance and analyst questions | |||||||||
| FinanceReasoning | 65.5 | 81.5 | 39.0 | 68.0 | 74.0 | 51.2 | 78.3 | 77.2 | 65.3 |
| FinanceMath | 34.9 | 57.7 | 22.7 | 42.4 | 45.0 | 27.2 | 56.5 | 58.1 | 40.8 |
| FinanceBench | 70.0 | 80.7 | 36.7 | 61.3 | 76.7 | 60.7 | 72.7 | 70.7 | 52.0 |
| FinanceQA | 20.9 | 35.1 | 14.2 | 17.6 | 29.1 | 23.6 | 28.4 | 32.4 | 22.3 |
| Mean of the ten | 63.6 | 72.2 | 36.4 | 55.3 | 62.0 | 57.1 | 66.1 | 68.2 | 51.9 |
Table 21. Accuracy on ten financial benchmarks, in percent, measured by us with the same harness and settings for every model, reasoning on and one sample an item. FiNER-ORD is entity F1. FinanceBench and FinanceQA are judged by a language model, with an unparsed verdict counted as wrong. The mean is unweighted over the ten. Bold marks the best value in each row.
Eight Answers, Timed
To see what the chip does to a reasoned answer, eight of Polyglot-DC1-Finance’s correct answers from the evaluation were replayed at the two decode rates, two for each kind of work. Table 22 lists them. The shortest is a sentiment call on a line of earnings news, 210 tokens of working and answer. The longest is a question about a put-option position, in which the model reasons through 1,995 tokens to the payoff. In between are a trader’s post, a segment-cost figure read from a TAT-QA table, a question about the effect of a buyback on equity value and enterprise value, a REIT’s funds from operations, and Costco’s EBITDA margin, for which the model first reads 10,762 tokens of the company’s 10-K and then reasons through 1,286 tokens to the answer.
| Kind of work | Case (benchmark) | Prompt tokens | Answer tokens | B300, one user | Exstill design |
| Sentiment | Earnings news (FinancialPhraseBank) | 103 | 210 | 0.40 s | 11 ms |
| Sentiment | Trader’s post (FiQA-SA) | 102 | 348 | 0.67 s | 17 ms |
| Filings | Segment costs (TAT-QA) | 465 | 278 | 0.53 s | 14 ms |
| Filings | EBITDA margin, 10-K (FinanceQA) | 10,762 | 1,286 | 2.46 s | 65 ms |
| Quantitative | Put-option payoff (Fin.Reasoning) | 248 | 1,995 | 3.82 s | 100 ms |
| Quantitative | REIT funds from operations (Fin.Reasoning) | 276 | 703 | 1.34 s | 35 ms |
| Analyst | Buyback, equity value (FinanceQA) | 79 | 711 | 1.36 s | 36 ms |
| Analyst | Buyback, enterprise value (FinanceQA) | 80 | 1,476 | 2.82 s | 74 ms |
| All eight | 7,007 | 13.4 s | 352 ms | ||
| One answer, on average | 876 | 1.68 s | 44 ms |
Table 22. Eight correct answers of Polyglot-DC1-Finance from the evaluation, replayed at the one-user B300 rate for a model of its size (522.9 tokens per second, Table 14) and the data-center design’s rate (19,921). Prompt tokens are what the model read; answer tokens are its reasoning and answer together. Times are decoding only.
An answer averages 876 tokens, 1.68 s of decoding on the GPU and 44 ms on the chip, and the longest takes 3.82 s against 100 ms. All eight back to back take 13.4 s on the GPU and 352 ms on the chip. The difference is not in the answers, which are the same tokens, but in what they can be used for. At 44 ms a reasoned answer arrives inside the time a person takes to glance at a screen.
Insight 10: Reasoning stops being rationed by the wait Every provider prices thinking as effort because on a GPU each reasoning token is time the user waits, so deployments buy less of it than the answer deserves. On silicon that decodes a small model 38x faster, a two-thousand-token chain of thought costs a tenth of a second, and the effort dial can sit where accuracy wants it.
The two sizes divide the work as they do for the guard. Polyglot-DC2-Finance is the more accurate, and on the DC2 design its answers decode at 13,998 tokens per second on each of four streams, about seven tenths of the smaller model’s rate. Polyglot-DC1-Finance serves twelve streams at once at 19,921 tokens per second each, so the data-center design can read and reason over a firm’s whole flow of filings and headlines. Finance is among the places where the seconds a reasoned answer takes on a GPU cost most, and it is the case in which this report shows a reasoning model on its own silicon turning the effort dial from a cost into a setting.
Conclusion
The measurements that open this report locate the cost of a token away from where it is usually assumed to lie. The arithmetic a token requires is a fraction of a percent of what a GPU spends on it; the rest is the price of generality, paid in the layers between a model and its computation and in the movement of weights and conversation state from memory beside the chip. Neither price is set by physics. Both follow from running every model on hardware designed before the model existed, and both fall more slowly with each process generation than the arithmetic they surround, so the distance between a token and its limit widens if nothing changes.
Designing the model and the silicon together closes that distance by lowering both factors of a token’s cost at once. Silicon organized around one neural network architecture removes the layers and the movement, while the trained weights stay loadable, so the same hardware carries whatever a new set of weights teaches it. A model whose architecture is chosen with that silicon, and held to a fixed quality, needs less arithmetic than the general-purpose model it replaces, and distillation turns the one architecture into the specialized models a deployment needs. The data-center design that results is a pre-silicon estimate, costed gate by gate across the whole model, and against a measured NVIDIA B300 serving the same users at the same quality it decodes each user’s next token 110x faster at 690x less energy. The models built for it complete the argument: one architecture, specialized by its weights alone, holds its own against open models of its size across the latency-critical work around a frontier model, on silicon where reasoning at length no longer costs the user the wait.
That co-processor is the first instance of the approach, not its extent. None of its ingredients is new on its own; weights beside the arithmetic, silicon for one kind of computation, distillation into a narrow job and models shaped to their hardware are all established. What is difficult to reproduce is their combination in a single loop, in which model architectures are explored and trained while the chip for each is costed at circuit-level fidelity, and that loop does not depend on the data center’s power envelope or on the frontier model beside it. The same search can be put to other workloads and other envelopes, in robotics, at the edge and on mobile devices, where a token’s cost in energy and time binds more tightly still. Those designs remain to be made, as this one remains to be measured in silicon; what the report establishes is the principle they would rest on. The distance between a token and its arithmetic is a matter of design, and when the model and the silicon are designed together, both the computation a token requires and the cost of carrying it out can be brought down to meet it.
References
References
- [1]Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman. SPEED-Bench: A unified and diverse benchmark for speculative decoding. In International Conference on Machine Learning, 2026.arxiv.org/abs/2604.09557
- [2]Dennis Abts, Garrin Kimmell, Andrew Ling, John Kim, Matt Boyd, Andrew Bitar, Sahil Parmar, Ibrahim Ahmed, Roberto DiCecco, David Han, John Thompson, Michael Bye, Jennifer Hwang, Jeremy Fowers, Peter Lillian, Ashwin Murthy, Elyas Mehtabuddin, Chetan Tekur, Thomas Sohmers, Kris Kang, Stephen Maresh, and Jonathan Ross. A software-defined tensor streaming multiprocessor for large-scale machine learning. In Proceedings of the 49th Annual International Symposium on Computer Architecture, 2022.DOI
- [3]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.13649
- [4]AMD. AMD expands Alveo portfolio with launch of world’s fastest electronic trading accelerator. https://ir.amd.com/news-events/press-releases/detail/1222/amd-expands-alveo-portfolio-with-launch-of-worlds-fastest-electronic-trading-accelerator-in-slim-form-factor-for-broad-cost-effective-server-deployments, 2024.
- [5]Bang An, Sicheng Zhu, Ruiji Zhang, Michael-Andrei Panaitescu-Liess, Yuancheng Xu, and Furong Huang. Automatic pseudo-harmful prompt generation for evaluating false refusals in large language models. In First Conference on Language Modeling, 2024.arxiv.org/abs/2409.00598
- [6]Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. In International Conference on Learning Representations, 2025.arxiv.org/abs/2404.02151
- [7]Anthropic. Activating AI safety level 3 protections. https://www.anthropic.com/news/activating-asl3-protections, 2025a.
- [8]Anthropic. Introducing Claude Opus 4.5. https://www.anthropic.com/news/claude-opus-4-5, 2025b.
- [9]Anthropic. Next-generation constitutional classifiers: More efficient protection against universal jailbreaks. https://www.anthropic.com/research/next-generation-constitutional-classifiers, 2026a.
- [11]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
- [12]Ljubisa Bajic. The path to ubiquitous AI. https://taalas.com/the-path-to-ubiquitous-ai/, 2026.
- [13]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-bench: Evaluating conversational agents in a dual-control environment. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.07982
- [14]Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyuan Geline Lin, and Pavlo Molchanov. Small language models are the future of agentic AI. arXiv preprint arXiv:2506.02153, 2025.arxiv.org/abs/2506.02153
- [15]Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, Ehud Karpas, Roi Koren, Itay Levy, Pavlo Molchanov, Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ullman Argov, Ran Zilberstein, and Ran El-Yaniv. Puzzle: Distillation-based NAS for inference-optimized LLMs. In International Conference on Machine Learning, 2025.
- [16]Luca Beurer-Kellner, Beat Buesser, Ana-Maria Crețu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, Ezinwanne Ozoani, Andrew Pawerda, Florian Tramèr, and Václav Volhej́n. Design patterns for securing LLM agents against prompt injections. arXiv preprint arXiv:2506.08837, 2025.
- [17]Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. In International Conference on Machine Learning, 2025.
- [18]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.arxiv.org/abs/2302.01318
- [19]Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding. In International Conference on Machine Learning, 2026a.arxiv.org/abs/2602.06036
- [20]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021a.arxiv.org/abs/2107.03374
- [21]Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, and Raghuraman Krishnamoorthi. MobileMoE: Scaling on-device mixture of experts. arXiv preprint arXiv:2605.27358, 2026b.arxiv.org/abs/2605.27358
- [22]Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021b.arxiv.org/abs/2109.00122
- [23]Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.arxiv.org/abs/2210.03849
- [24]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, and Wenfeng Liang. DSPark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147, 2026.arxiv.org/abs/2607.05147
- [25]CNBC. Goldman Sachs rolls out an AI assistant for its employees as artificial intelligence sweeps Wall Street. https://www.cnbc.com/2025/01/21/goldman-sachs-launches-ai-assistant.html, 2025.
- [26]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
- [27]Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An over-refusal benchmark for large language models. In International Conference on Machine Learning, 2025.arxiv.org/abs/2405.20947
- [28]Hoagy Cunningham, Jerry Wei, Zihan Wang, Andrew Persic, Alwin Peng, Jordan Abderrachid, Raj Agarwal, Bobby Chen, Andy Dau, Alek Dimitriev, Rob Gilson, Logan Howard, Yijin Hua, Mu Lin, Christopher Liu, Vladimir Mikulik, Rohit Mittapalli, Clare O’Hara, Jin Pan, Nikhil Saxena, Alex Silverstein, Yue Song, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez, and Mrinank Sharma. Constitutional classifiers++: Efficient production-grade defenses against universal jailbreaks. In International Conference on Learning Representations, 2026.arxiv.org/abs/2601.04603
- [29]Adrian de Wynter, Ishaan Watts, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Nektar Ege Altintoprak, Lena Baur, Samantha Claudet, Pavel Gajdusek, Can Gören, Qilong Gu, Anna Kaminska, Tomasz Kaminski, Ruby Kuo, Akiko Kyuba, Jongho Lee, Kartik Mathur, Petter Merok, Ivana Milovanovic, Nani Paananen, Vesa-Matti Paananen, Anna Pavlenko, Bruno Pereira Vidal, Luciano Strika, Yueh Tsao, Davide Turcato, Oleksandr Vakhno, Judit Velcsov, Anna Vickers, Stéphanie Visser, Herdyan Widarmanto, Andrey Zaikin, and Si-Qing Chen. RTP-LX: Can LLMs evaluate toxicity in multilingual scenarios? In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
- [30]Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr. Defeating prompt injections by design. In IEEE Conference on Secure and Trustworthy Machine Learning, 2026.arxiv.org/abs/2503.18813
- [31]DeepSeek. Thinking mode. https://api-docs.deepseek.com/guides/thinking_mode/, 2026.
- [32]DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026.
- [33]Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.06474
- [34]Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. In First Conference on Language Modeling, 2024.arxiv.org/abs/2404.04475
- [35]Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek, Kaan Oktay, Hongyang Zhang, and Stephen Xia. Attention drift: What autoregressive speculative decoding models learn. arXiv preprint arXiv:2605.09992, 2026.arxiv.org/abs/2605.09992
- [36]FINRA. 2026 annual regulatory oversight report. https://www.finra.org/sites/default/files/2025-12/2026-annual-regulatory-oversight-report.pdf, 2025.
- [37]Fortune. Goldman Sachs rolls out an internal AI assistant firm-wide. https://fortune.com/2025/06/24/goldman-sachs-internal-ai-assistant/, 2025.
- [38]Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
- [39]Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
- [40]Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AEGIS: Online adaptive AI content safety moderation with ensemble of LLM experts. arXiv preprint arXiv:2404.05993, 2024.arxiv.org/abs/2404.05993
- [41]Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. AEGIS2.0: A diverse AI safety dataset and risks taxonomy for alignment of LLM guardrails. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025.
- [42]Ricardo Gonzalez and Mark Horowitz. Energy dissipation in general purpose microprocessors. IEEE Journal of Solid-State Circuits, 1996.DOI
- [43]Google. Gemini thinking. https://aistudio.google.com/docs/thinking, 2026.
- [44]Google DeepMind. Gemma 4 E4B model card. https://huggingface.co/google/gemma-4-E4B-it, 2026.
- [45]Yuxian Gu, Qinghao Hu, Haocheng Xi, Junyu Chen, Shang Yang, Song Han, and Han Cai. Jet-Nemotron: Efficient language model with post neural architecture search. In Advances in Neural Information Processing Systems, 2025.
- [46]Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.18495
- [47]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2021.
- [48]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
- [49]Mark Horowitz. Computing’s energy problem (and what we can do about it). In IEEE International Solid-State Circuits Conference Digest of Technical Papers, 2014.
- [50]Hanxian Huang, Igor Fedorov, Andrey Gromov, Bernard Beckerman, Naveen Suda, David Eriksson, Maximilian Balandat, Rylan Conway, Patrick Huber, Chinnadhurai Sankar, Ayushi Dalmia, Zechun Liu, Lemeng Wu, Tarek Elgamal, Adithya Sagar, Vikas Chandra, and Raghuraman Krishnamoorthi. MobileLLM-Flash: Latency-guided on-device LLM design for industry scale deployment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics: Industry Track, 2026.
- [51]Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, and Tianyu Zhang. MoESD: Unveil speculative decoding’s potential for accelerating sparse MoE. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.19645
- [52]Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. FinanceBench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023.arxiv.org/abs/2311.11944
- [53]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.07974
- [54]Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2307.04657
- [55]Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, and Yaodong Yang. PKU-SafeRLHF: Towards multi-level safety alignment for LLMs with human preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.arxiv.org/abs/2406.15513